1 .TH "LTTNG" "1" "December 3rd, 2012" "" ""
4 lttng \(em LTTng 2.x tracer control command line tool
10 lttng [OPTIONS] <COMMAND>
15 The LTTng project aims at providing highly efficient tracing tools for Linux.
16 It's tracers help tracking down performance issues and debugging problems
17 involving multiple concurrent processes and threads. Tracing across multiple
18 systems is also possible.
20 The \fBlttng\fP command line tool from the lttng-tools package is used to control
21 both kernel and user-space tracing. Every interactions with the tracer should
22 be done by this tool or by the liblttng-ctl provided with the lttng-tools
25 LTTng uses a session daemon (lttng-sessiond(8)), acting as a tracing registry,
26 which allows you to interact with multiple tracers (kernel and user-space)
27 inside the same container, a tracing session. Traces can be gathered from the
28 kernel and/or instrumented applications (lttng-ust(3)). Aggregating and reading
29 those traces is done using the babeltrace(1) text viewer.
31 We introduce the notion of \fBtracing domains\fP which is essentially a type of
32 tracer (kernel or user space for now). In the future, we could see a third
33 tracer being for instance an hypervisor. For some commands, you'll need to
34 specify on which domain the command applies (-u or -k). For instance, enabling
35 a kernel event, you must specify the kernel domain to the command so we know
36 for which tracer this event is for.
38 In order to trace the kernel, the session daemon needs to be running as root.
39 LTTng provides the use of a \fBtracing group\fP (default: tracing). Whomever is
40 in that group can interact with the root session daemon and thus trace the
41 kernel. Session daemons can co-exist meaning that you can have a session daemon
42 running as Alice that can be used to trace her applications along side with a
43 root daemon or even a Bob daemon. We highly recommend to start the session
44 daemon at boot time for stable and long term tracing.
46 Every user-space applications instrumented with lttng-ust(3), will
47 automatically register to the session daemon. This feature gives you the
48 ability to list available traceable applications and tracepoints on a per user
49 basis. (See \fBlist\fP command).
53 This program follow the usual GNU command line syntax with long options starting with
54 two dashes. Below is a summary of the available options.
59 Show summary of possible options and commands.
61 .BR "\-v, \-\-verbose"
63 Three levels of verbosity are available which are triggered by putting additional v to
64 the option (\-vv or \-vvv)
67 Suppress all messages (even errors).
69 .BR "\-g, \-\-group NAME"
70 Set unix tracing group name. (default: tracing)
72 .BR "\-n, \-\-no-sessiond"
73 Don't automatically spawn a session daemon.
75 .BR "\-\-sessiond\-path PATH"
76 Set session daemon full binary path.
78 .BR "\-\-list\-options"
79 Simple listing of lttng options.
81 .BR "\-\-list\-commands"
82 Simple listing of lttng commands.
88 Add context to event(s) and/or channel(s).
90 A context is basically extra information appended to a channel. For instance,
91 you could ask the tracer to add the PID information for all events in a
92 channel. You can also add performance monitoring unit counters (perf PMU) using
95 For example, this command will add the context information 'prio' and two perf
96 counters (hardware branch misses and cache misses), to all events in the trace
99 # lttng add-context \-k \-t prio \-t perf:branch-misses \-t perf:cache-misses
101 Please take a look at the help (\-h/\-\-help) for a detailed list of available
104 If no channel is given (\-c), the context is added to all channels that were
105 already enabled. If the session has no channel, a default channel is created.
106 Otherwise the context will be added only to the given channel (\-c).
108 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
116 Show summary of possible options and commands.
117 \-s, \-\-session NAME
118 Apply on session name.
119 \-c, \-\-channel NAME
120 Apply on channel name.
122 Apply for the kernel tracer
124 Apply for the user-space tracer
126 Context type. You can repeat this option on the command line. Please
127 use "lttng add-context \-h" to list all available types.
132 .IP "\fBcalibrate\fP"
134 Quantify LTTng overhead
136 The LTTng calibrate command can be used to find out the combined average
137 overhead of the LTTng tracer and the instrumentation mechanisms used. This
138 overhead can be calibrated in terms of time or using any of the PMU performance
139 counter available on the system.
141 For now, the only calibration implemented is that of the kernel function
142 instrumentation (kretprobes).
144 * Calibrate kernel function instrumentation
146 Let's use an example to show this calibration. We use an i7 processor with 4
147 general-purpose PMU registers. This information is available by issuing dmesg,
148 looking for "generic registers".
150 This sequence of commands will gather a trace executing a kretprobe hooked on
151 an empty function, gathering PMU counters LLC (Last Level Cache) misses
152 information (see lttng add-context \-\-help to see the list of available PMU
155 # lttng create calibrate-function
156 # lttng enable-event calibrate \-\-kernel \-\-function lttng_calibrate_kretprobe
157 # lttng add-context \-\-kernel \-t perf:LLC-load-misses \-t perf:LLC-store-misses \\
158 \-t perf:LLC-prefetch-misses
160 # for a in $(seq 1 10); do \\
161 lttng calibrate \-\-kernel \-\-function;
164 # babeltrace $(ls \-1drt ~/lttng-traces/calibrate-function-* | tail \-n 1)
166 The output from babeltrace can be saved to a text file and opened in a
167 spreadsheet (e.g. oocalc) to focus on the per-PMU counter delta between
168 consecutive "calibrate_entry" and "calibrate_return" events. Note that these
169 counters are per-CPU, so scheduling events would need to be present to account
170 for migration between CPU. Therefore, for calibration purposes, only events
171 staying on the same CPU must be considered.
173 The average result, for the i7, on 10 samples:
176 perf_LLC_load_misses: 5.0 0.577
177 perf_LLC_store_misses: 1.6 0.516
178 perf_LLC_prefetch_misses: 9.0 14.742
180 As we can notice, the load and store misses are relatively stable across runs
181 (their standard deviation is relatively low) compared to the prefetch misses.
182 We can conclude from this information that LLC load and store misses can be
183 accounted for quite precisely, but prefetches within a function seems to behave
184 too erratically (not much causality link between the code executed and the CPU
185 prefetch activity) to be accounted for.
192 Show summary of possible options and commands.
194 Apply for the kernel tracer
196 Apply for the user-space tracer
198 Dynamic function entry/return probe (default)
203 .IP "\fBcreate\fP [NAME] [OPTIONS]
205 Create tracing session.
207 A tracing session contains channel(s) which contains event(s). It is domain
208 agnostic meaning that you can enable channels and events for either the
209 user-space tracer and/or the kernel tracer. It acts like a container
210 aggregating multiple tracing sources.
212 On creation, a \fB.lttngrc\fP file is created in your $HOME directory
213 containing the current session name. If NAME is omitted, a session name is
214 automatically created having this form: 'auto-yyyymmdd-hhmmss'.
216 If no \fB\-o, \-\-output\fP is specified, the traces will be written in
219 The $HOME environment variable can be overridden by defining the environment
220 variable LTTNG_HOME. This is useful when the user running the commands has
221 a non-writeable home directory.
228 Show summary of possible options and commands.
230 Simple listing of options
232 Specify output path for traces
234 Traces will not be outputed
236 Set the session in snapshot mode. Created in no-output mode
237 and uses the URL, if one, as the default snapshot output.
238 Every channel will be set in overwrite mode and with mmap
239 output (splice not supported).
241 Using these options, each API call can be controlled individually. For
242 instance, \-C does not enable the consumer automatically. You'll need the \-e
246 Set URL for the consumer output destination. It is persistent for the
247 session lifetime. Redo the command to change it. This will set both
248 data and control URL for network.
249 \-C, \-\-ctrl-url=URL
250 Set control path URL. (Must use -D also)
251 \-D, \-\-data-url=URL
252 Set data path URL. (Must use -C also)
256 proto://[HOST|IP][:PORT1[:PORT2]][/TRACE_PATH]
258 Supported protocols are (proto):
260 Local filesystem full path.
263 This will use the default network transport layer which is TCP for both
264 control (PORT1) and data port (PORT2). The default ports are
265 respectively 5342 and 5343. Note that net[6]:// is not yet supported.
268 Can only be used with -C and -D together
270 NOTE: IPv6 address MUST be enclosed in brackets '[]' (rfc2732)
274 # lttng create -U net://192.168.1.42
275 Uses TCP and default ports for the given destination.
277 # lttng create -U net6://[fe80::f66d:4ff:fe53:d220]
278 Uses TCP, default ports and IPv6.
280 # lttng create s1 -U net://myhost.com:3229
281 Create session s1 and set its consumer to myhost.com on port 3229 for control.
286 .IP "\fBdestroy\fP [OPTIONS] [NAME]"
288 Teardown tracing session
290 Free memory on the session daemon and tracer side. It's gone!
292 If NAME is omitted, the session name is taken from the .lttngrc file.
299 Show summary of possible options and commands.
303 Simple listing of options
308 .IP "\fBenable-channel\fP NAME[,NAME2,...] [-k|-u] [OPTIONS]"
310 Enable tracing channel
312 To enable an event, you must enable both the event and the channel that
315 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
318 It is important to note that if a certain type of buffers is used, the session
319 will be set with that type and all other subsequent channel needs to have the
322 Note that once the session has been started and enabled on the tracer side,
323 it's not possible anymore to enable a new channel for that session.
332 Simple listing of options
333 \-s, \-\-session NAME
334 Apply on session name
336 Apply to the kernel tracer
338 Apply to the user-space tracer
341 Discard event when subbuffers are full (default)
343 Flight recorder mode : overwrites events when subbuffers are full
345 Subbuffer size in bytes {+k,+M,+G}
346 (default UST uid: 131072, UST pid: 4096, kernel: 262144, metadata: 4096)
347 Rounded up to the next power of 2.
349 The minimum subbuffer size, for each tracer, is the max value between
350 the default above and the system page size. You can issue this command
351 to get the current page size on your system: \fB$ getconf PAGE_SIZE\fP
353 Number of subbuffers. (default UST uid: 4, UST pid: 4, kernel: 4, metadata: 2)
354 Rounded up to the next power of 2.
355 \-\-switch-timer USEC
356 Switch subbuffer timer interval in µsec.
357 (default UST uid: 0, UST pid: 0, kernel: 0, metadata: 0)
359 Read timer interval in µsec.
360 (default UST uid: 0, UST pid: 0, kernel: 200000, metadata: 0)
362 Channel output type. Possible values: mmap, splice
363 (default UST uid: mmap, UST pid: mmap, kernel: splice, metadata: mmap)
365 Use per UID buffer (\-u only). Buffers are shared between applications
366 that have the same UID.
368 Use per PID buffer (\-u only). Each application has its own buffers.
370 Use shared buffer for the whole system (\-k only)
371 \-C, \-\-tracefile-size SIZE
372 Maximum size of each tracefile within a stream (in bytes).
373 0 means unlimited. (default: 0)
374 \-W, \-\-tracefile-count COUNT
375 Used in conjunction with \-C option, this will limit the number
376 of files created to the specified count. 0 means unlimited. (default: 0)
380 $ lttng enable-channel -C 4096 -W 32 chan1
381 For each stream, the maximum size of each trace file will be 4096 bytes, and
382 there will be a maximum of 32 different files. The file count is appended after
383 the stream number as seen in the following example. The last trace file is
384 smaller than 4096 since it was not completely filled.
386 ~/lttng-traces/[...]/chan1_0_0 (4096)
387 ~/lttng-traces/[...]/chan1_0_1 (4096)
388 ~/lttng-traces/[...]/chan1_0_2 (3245)
389 ~/lttng-traces/[...]/chan1_1_0 (4096)
392 $ lttng enable-channel -C 4096
393 This will create trace files of 4096 bytes and will create new ones as long as
394 there is data available.
399 .IP "\fBenable-event\fP NAME[,NAME2,...] [-k|-u] [OPTIONS]"
403 A tracing event is always assigned to a channel. If \fB\-c, \-\-channel\fP is
404 omitted, a default channel named '\fBchannel0\fP' is created and the event is
405 added to it. For the user-space tracer, using \fB\-a, \-\-all\fP is the same as
406 using the wildcard "*".
408 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
416 Show summary of possible options and commands.
418 Simple listing of options
419 \-s, \-\-session NAME
420 Apply on session name
421 \-c, \-\-channel NAME
422 Apply on channel name
424 Enable all tracepoints and syscalls. This actually enable a single
427 Apply for the kernel tracer
429 Apply for the user-space tracer
432 Tracepoint event (default)
433 - userspace tracer supports wildcards at end of string. Don't forget to
434 quote to deal with bash expansion.
439 Tracepoint loglevel range from 0 to loglevel. Listed in the help (\-h).
440 \-\-loglevel-only NAME
441 Tracepoint loglevel (only this loglevel).
443 The loglevel or loglevel-only options should be combined with a
444 tracepoint name or tracepoint wildcard.
445 \-\-probe [addr | symbol | symbol+offset]
446 Dynamic probe. Addr and offset can be octal (0NNN...), decimal (NNN...)
447 or hexadecimal (0xNNN...)
448 \-\-function [addr | symbol | symbol+offset]
449 Dynamic function entry/return probe. Addr and offset can be octal
450 (0NNN...), decimal (NNN...) or hexadecimal (0xNNN...)
452 System call event. Enabling syscalls tracing (kernel tracer), you will
453 not be able to disable them with disable-event. This is a known
454 limitation. You can disable the entire channel to do the trick.
456 \-\-filter 'expression'
457 Set a filter on a newly enabled event. Filter expression on event
458 fields and context. Event recording depends on evaluation. Only
459 specify on first activation of a given event within a session.
460 Filter only allowed when enabling events within a session before
461 tracing is started. If the filter fails to link with the event
462 within the traced domain, the event will be discarded.
463 Currently, filter is only implemented for the user-space tracer.
467 'intfield > 500 && intfield < 503'
468 '(stringfield == "test" || intfield != 10) && intfield > 33'
469 'doublefield > 1.1 && intfield < 5.3'
471 Wildcards are allowed at the end of strings:
473 In string literals, the escape character is a '\\'. Use '\\*' for
474 the '*' character, and '\\\\' for the '\\' character. Wildcard
475 match any sequence of characters, including an empty sub-string
476 (match 0 or more characters).
478 Context information can be used for filtering. The examples
479 below show usage of context filtering on process name (with a
480 wildcard), process ID range, and unique thread ID for filtering.
481 The process and thread ID of running applications can be found
482 under columns "PID" and "LWP" of the "ps -eLf" command.
484 '$ctx.procname == "demo*"'
485 '$ctx.vpid >= 4433 && $ctx.vpid < 4455'
489 .IP "\fBdisable-channel\fP NAME[,NAME2,...] [\-k|\-u] [OPTIONS]"
491 Disable tracing channel
493 Disabling a channel makes all event(s) in that channel to stop tracing. You can
494 enable it back by calling \fBlttng enable-channel NAME\fP again.
496 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
504 Show summary of possible options and commands.
506 Simple listing of options
507 \-s, \-\-session NAME
508 Apply on session name
510 Apply for the kernel tracer
512 Apply for the user-space tracer
515 .IP "\fBdisable-event\fP NAME[,NAME2,...] [\-k|\-u] [OPTIONS]"
517 Disable tracing event
519 The event, once disabled, can be re-enabled by calling \fBlttng enable-event
522 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
530 Show summary of possible options and commands.
532 Simple listing of options
533 \-s, \-\-session NAME
534 Apply on session name
536 Disable all events. This does NOT disable "*" but rather
537 every known events of the session.
539 Apply for the kernel tracer
541 Apply for the user-space tracer
544 .IP "\fBlist\fP [\-k|\-u] [SESSION [SESSION_OPTIONS]]"
546 List tracing session information.
548 With no arguments, it will list available tracing session(s).
550 With the session name, it will display the details of the session including
551 the trace file path, the associated channels and their state (activated
552 and deactivated), the activated events and more.
554 With \-k alone, it will list all available kernel events (except the system
556 With \-u alone, it will list all available user-space events from registered
557 applications. Here is an example of 'lttng list \-u':
559 PID: 7448 - Name: /tmp/lttng-ust/tests/hello/.libs/lt-hello
560 ust_tests_hello:tptest_sighandler (type: tracepoint)
561 ust_tests_hello:tptest (type: tracepoint)
563 You can now enable any event listed by using the name :
564 \fBust_tests_hello:tptest\fP.
571 Show summary of possible options and commands.
573 Simple listing of options
577 Select user-space domain.
581 \-c, \-\-channel NAME
582 List details of a channel
584 List available domain(s)
587 .IP "\fBset-session\fP NAME"
589 Set current session name
591 Will change the session name in the .lttngrc file.
598 Show summary of possible options and commands.
600 Simple listing of options
605 .IP "\fBsnapshot\fP ACTION"
607 Snapshot command for LTTng session.
614 Show summary of possible options and commands.
616 Simple listing of options
622 \fBadd-output\fP [-m <SIZE>] [-s <NAME>] [-n <NAME>] <URL> | -C <URL> -D <URL>
624 Setup and add an snapshot output for a session. Output are the destination
625 where the snapshot will be sent. Only one output is permitted. To change it,
626 you'll need to delete it and add back the new one.
628 \fBdel-output\fP ID | NAME [-s <NAME>]
630 Delete an output for a session using the ID. You can either specify the
631 output's ID that can be found with list-output or the name.
633 \fBlist-output\fP [-s <NAME>]
635 List the output of a session. Attributes of the output are printed.
637 \fBrecord\fP [-m <SIZE>] [-s <NAME>] [-n <NAME>] [<URL> | -C <URL> -D <URL>]
639 Snapshot a session's buffer(s) for all domains. If an URL is specified, it is
640 used instead of a previously added output. Specifying only a name or/and a max
641 size will override the current output values. For instance, you can record a
642 snapshot with a custom maximum size or with a different name.
644 $ lttng add-output -n mysnapshot file:///data/snapshot
646 $ lttng snapshot record -n new_name_snapshot
648 The above will create a snapshot in /data/snapshot/new_name_snapshot* directory
649 rather then in mysnapshot*/
655 \-s, \-\-session NAME
656 Apply to session name.
658 Name of the snapshot's output.
659 \-m, \-\-max-size SIZE
660 Maximum size in bytes of the snapshot. The maxium size does not
661 include the metadata file.
662 \-C, \-\-ctrl-url URL
663 Set control path URL. (Must use -D also)
664 \-D, \-\-data-url URL
665 Set data path URL. (Must use -C also)
670 .IP "\fBstart\fP [NAME] [OPTIONS]"
674 It will start tracing for all tracers for a specific tracing session.
676 If NAME is omitted, the session name is taken from the .lttngrc file.
683 Show summary of possible options and commands.
685 Simple listing of options
690 .IP "\fBstop\fP [NAME] [OPTIONS]"
694 It will stop tracing for all tracers for a specific tracing session. Before
695 returning, the command checks for data availability meaning that it will wait
696 until the trace is readable for the session. Use \-\-no-wait to avoid this
699 If NAME is omitted, the session name is taken from the .lttngrc file.
706 Show summary of possible options and commands.
708 Simple listing of options
710 Don't wait for data availability.
717 Show version information
724 Show summary of possible options and commands.
726 Simple listing of options
731 .IP "\fBview\fP [SESSION_NAME] [OPTIONS]"
733 View traces of a tracing session
735 By default, the babeltrace viewer will be used for text viewing.
737 If SESSION_NAME is omitted, the session name is taken from the .lttngrc file.
747 Simple listing of options
748 \-t, \-\-trace-path PATH
749 Trace directory path for the viewer
751 Specify viewer and/or options to use
752 This will completely override the default viewers so
753 please make sure to specify the full command. The trace
754 directory path of the session will be appended at the end
759 On success 0 is returned and a positive value on error. Value of 1 means a command
760 error, 2 an undefined command, 3 a fatal error and 4 a command warning meaning that
761 something went wrong during the command.
763 Any other value above 10, please refer to
764 .BR <lttng/lttng-error.h>
765 for a detailed list or use lttng_strerror() to get a human readable string of
769 .SH "ENVIRONMENT VARIABLES"
772 Note that all command line options override environment variables.
776 .IP "LTTNG_SESSIOND_PATH"
777 Allows one to specify the full session daemon binary path to lttng command line
778 tool. You can also use \-\-sessiond-path option having the same effect.
782 .BR lttng-sessiond(8),
784 .BR lttng-health-check(3)
787 If you encounter any issues or usability problem, please report it on our
788 mailing list <lttng-dev@lists.lttng.org> to help improve this project or
789 at https://bugs.lttng.org which is a bugtracker.
793 lttng is distributed under the GNU General Public License version 2. See the file
796 A Web site is available at http://lttng.org for more information on the LTTng
799 You can also find our git tree at http://git.lttng.org.
801 Mailing lists for support and development: <lttng-dev@lists.lttng.org>.
803 You can find us on IRC server irc.oftc.net (OFTC) in #lttng.
808 Thanks to Yannick Brosseau without whom this project would never have been so
809 lean and mean! Also thanks to the Ericsson teams working on tracing which
810 helped us greatly with detailed bug reports and unusual test cases.
812 Thanks to our beloved packager Alexandre Montplaisir-Goncalves (Ubuntu and PPA
813 maintainer) and Jon Bernard for our Debian packages.
815 Special thanks to Michel Dagenais and the DORSAL laboratory at Polytechnique de
816 Montreal for the LTTng journey.
821 lttng-tools was originally written by Mathieu Desnoyers, Julien Desfossez and
822 David Goulet. More people have since contributed to it. It is currently
823 maintained by David Goulet <dgoulet@efficios.com>.