1 .TH "LTTNG" "1" "July 18th, 2013" "" ""
4 lttng \(em LTTng 2.x tracer control command line tool
9 lttng [OPTIONS] <COMMAND>
13 The LTTng project aims at providing highly efficient tracing tools for Linux.
14 Its tracers help track down performance issues and debug problems
15 involving multiple concurrent processes and threads. Tracing across multiple
16 systems is also possible.
18 The \fBlttng\fP command line tool from the lttng-tools package is used to control
19 both kernel and user-space tracing. Every interaction with the tracer should
20 be done by this tool or by the liblttng-ctl library provided by the lttng-tools
23 LTTng uses a session daemon (lttng-sessiond(8)), acting as a tracing registry,
24 which allows you to interact with multiple tracers (kernel and user-space)
25 inside the same container, a tracing session. Traces can be gathered from the
26 kernel and/or instrumented applications (lttng-ust(3)). Aggregating and reading
27 those traces is done using the babeltrace(1) text viewer.
29 We introduce the notion of \fBtracing domains\fP which is essentially a type of
30 tracer (kernel or user space for now). In the future, we could see a third
31 tracer being for instance an hypervisor. For some commands, you'll need to
32 specify on which domain the command operates (-u or -k). For instance, the
33 kernel domain must be specified when enabling a kernel event.
35 In order to trace the kernel, the session daemon needs to be running as root.
36 LTTng provides the use of a \fBtracing group\fP (default: tracing). Whomever is
37 in that group can interact with the root session daemon and thus trace the
38 kernel. Session daemons can co-exist, meaning that you can have a session daemon
39 running as Alice that can be used to trace her applications along side with a
40 root daemon or even a Bob daemon. We highly recommend starting the session
41 daemon at boot time for stable and long term tracing.
43 All user-space applications instrumented with lttng-ust(3) will
44 automatically register to the session daemon. This feature gives you the
45 ability to list available traceable applications and tracepoints on a per user
46 basis. (See \fBlist\fP command).
50 This program follow the usual GNU command line syntax with long options starting with
51 two dashes. Below is a summary of the available options.
56 Show summary of possible options and commands.
58 .BR "\-v, \-\-verbose"
60 Three levels of verbosity are available which are triggered by putting additional v to
61 the option (\-vv or \-vvv)
64 Suppress all messages (even errors).
66 .BR "\-g, \-\-group NAME"
67 Set unix tracing group name. (default: tracing)
69 .BR "\-n, \-\-no-sessiond"
70 Don't automatically spawn a session daemon.
72 .BR "\-\-sessiond\-path PATH"
73 Set session daemon full binary path.
75 .BR "\-\-list\-options"
76 Simple listing of lttng options.
78 .BR "\-\-list\-commands"
79 Simple listing of lttng commands.
83 \fBadd-context\fP [OPTIONS]
85 Add context to event(s) and/or channel(s).
87 A context is basically extra information appended to a channel. For instance,
88 you could ask the tracer to add the PID information for all events in a
89 channel. You can also add performance monitoring unit counters (perf PMU) using
92 For example, this command will add the context information 'prio' and two perf
93 counters (hardware branch misses and cache misses), to all events in the trace
97 # lttng add-context \-k \-t prio \-t perf:branch-misses \\
101 Please take a look at the help (\-h/\-\-help) for a detailed list of available
104 If no channel is given (\-c), the context is added to all channels that were
105 already enabled. If the session has no channel, a default channel is created.
106 Otherwise the context will be added only to the given channel (\-c).
108 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
115 Show summary of possible options and commands.
117 .BR "\-s, \-\-session NAME"
118 Apply on session name.
120 .BR "\-c, \-\-channel NAME"
121 Apply on channel name.
123 .BR "\-k, \-\-kernel"
124 Apply for the kernel tracer
126 .BR "\-u, \-\-userspace"
127 Apply for the user-space tracer
129 .BR "\-t, \-\-type TYPE"
130 Context type. You can repeat this option on the command line. Please
131 use "lttng add-context \-h" to list all available types.
136 \fBcalibrate\fP [OPTIONS]
138 Quantify LTTng overhead
140 The LTTng calibrate command can be used to find out the combined average
141 overhead of the LTTng tracer and the instrumentation mechanisms used. This
142 overhead can be calibrated in terms of time or using any of the PMU performance
143 counter available on the system.
145 For now, the only calibration implemented is that of the kernel function
146 instrumentation (kretprobes).
148 * Calibrate kernel function instrumentation
150 Let's use an example to show this calibration. We use an i7 processor with 4
151 general-purpose PMU registers. This information is available by issuing dmesg,
152 looking for "generic registers".
154 This sequence of commands will gather a trace executing a kretprobe hooked on
155 an empty function, gathering PMU counters LLC (Last Level Cache) misses
156 information (see lttng add-context \-\-help to see the list of available PMU
160 # lttng create calibrate-function
161 # lttng enable-event calibrate \-\-kernel \\
162 \-\-function lttng_calibrate_kretprobe
163 # lttng add-context \-\-kernel \-t perf:LLC-load-misses \\
164 \-t perf:LLC-store-misses \\
165 \-t perf:LLC-prefetch-misses
167 # for a in $(seq 1 10); do \\
168 lttng calibrate \-\-kernel \-\-function;
171 # babeltrace $(ls \-1drt ~/lttng-traces/calibrate-function-* \\
175 The output from babeltrace can be saved to a text file and opened in a
176 spreadsheet (e.g. oocalc) to focus on the per-PMU counter delta between
177 consecutive "calibrate_entry" and "calibrate_return" events. Note that these
178 counters are per-CPU, so scheduling events would need to be present to account
179 for migration between CPU. Therefore, for calibration purposes, only events
180 staying on the same CPU must be considered.
182 The average result, for the i7, on 10 samples:
186 perf_LLC_load_misses: 5.0 0.577
187 perf_LLC_store_misses: 1.6 0.516
188 perf_LLC_prefetch_misses: 9.0 14.742
191 As we can notice, the load and store misses are relatively stable across runs
192 (their standard deviation is relatively low) compared to the prefetch misses.
193 We can conclude from this information that LLC load and store misses can be
194 accounted for quite precisely, but prefetches within a function seems to behave
195 too erratically (not much causality link between the code executed and the CPU
196 prefetch activity) to be accounted for.
202 Show summary of possible options and commands.
204 .BR "\-k, \-\-kernel"
205 Apply for the kernel tracer
207 .BR "\-u, \-\-userspace"
208 Apply for the user-space tracer
211 Dynamic function entry/return probe (default)
216 \fBcreate\fP [NAME] [OPTIONS]
218 Create tracing session.
220 A tracing session contains channel(s) which contains event(s). It is domain
221 agnostic, meaning that channels and events can be enabled for the
222 user-space tracer and/or the kernel tracer. It acts like a container
223 aggregating multiple tracing sources.
225 On creation, a \fB.lttngrc\fP file is created in your $HOME directory
226 containing the current session name. If NAME is omitted, a session name is
227 automatically created having this form: 'auto-yyyymmdd-hhmmss'.
229 If no \fB\-o, \-\-output\fP is specified, the traces will be written in
232 The $HOME environment variable can be overridden by defining the environment
233 variable LTTNG_HOME. This is useful when the user running the commands has
234 a non-writeable home directory.
240 Show summary of possible options and commands.
242 .BR "\-\-list-options"
243 Simple listing of options
245 .BR "\-o, \-\-output PATH"
246 Specify output path for traces
249 Traces will not be output
252 Set the session in snapshot mode. Created in no-output mode and uses the
253 URL, if one is specified, as the default snapshot output. Every channel will be set
254 in overwrite mode and with mmap output (splice not supported).
257 .BR "\-U, \-\-set-url=URL"
258 Set URL for the consumer output destination. It is persistent for the
259 session lifetime. Redo the command to change it. This will set both data
260 and control URL for network.
262 .BR "\-C, \-\-ctrl-url=URL"
263 Set control path URL. (Must use -D also)
265 .BR "\-D, \-\-data-url=URL"
266 Set data path URL. (Must use -C also)
268 Using these options, each API call can be controlled individually. For
269 instance, \-C does not enable the consumer automatically. You'll need the \-e
274 proto://[HOST|IP][:PORT1[:PORT2]][/TRACE_PATH]
276 Supported protocols are (proto):
279 Local filesystem full path.
283 This will use the default network transport layer which is TCP for both
284 control (PORT1) and data port (PORT2). The default ports are
285 respectively 5342 and 5343. Note that net[6]:// is not yet supported.
289 Can only be used with -C and -D together
291 NOTE: IPv6 address MUST be enclosed in brackets '[]' (rfc2732)
296 # lttng create -U net://192.168.1.42
298 Uses TCP and default ports for the given destination.
301 # lttng create -U net6://[fe80::f66d:4ff:fe53:d220]
303 Uses TCP, default ports and IPv6.
306 # lttng create s1 -U net://myhost.com:3229
308 Create session s1 and set its consumer to myhost.com on port 3229 for control.
313 \fBdestroy\fP [NAME] [OPTIONS]
315 Teardown tracing session
317 Free memory on the session daemon and tracer side. It's gone!
319 If NAME is omitted, the session name is taken from the .lttngrc file.
325 Show summary of possible options and commands.
330 .BR "\-\-list-options"
331 Simple listing of options
336 \fBenable-channel\fP NAME[,NAME2,...] (\-k | \-u) [OPTIONS]
338 Enable tracing channel
340 To enable an event, you must enable both the event and the channel that
343 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
346 Exactly one of \-k or -u must be specified.
348 It is important to note that if a certain type of buffers is used, the session
349 will be set with that type and all other subsequent channel needs to have the
352 Note that once the session has been started and enabled on the tracer side,
353 it's not possible anymore to enable a new channel for that session.
361 .BR "\-\-list-options"
362 Simple listing of options
364 .BR "\-s, \-\-session NAME"
365 Apply on session name
367 .BR "\-k, \-\-kernel"
368 Apply to the kernel tracer
370 .BR "\-u, \-\-userspace"
371 Apply to the user-space tracer
374 Discard event when subbuffers are full (default)
377 Flight recorder mode : overwrites events when subbuffers are full
379 .BR "\-\-subbuf-size SIZE"
380 Subbuffer size in bytes {+k,+M,+G}.
381 (default UST uid: 131072, UST pid: 4096, kernel: 262144, metadata: 4096)
382 Rounded up to the next power of 2.
384 The minimum subbuffer size, for each tracer, is the max value between
385 the default above and the system page size. You can issue this command
386 to get the current page size on your system: \fB$ getconf PAGE_SIZE\fP
388 .BR "\-\-num-subbuf NUM"
389 Number of subbuffers. (default UST uid: 4, UST pid: 4, kernel: 4,
390 metadata: 2) Rounded up to the next power of 2.
392 .BR "\-\-switch-timer USEC"
393 Switch subbuffer timer interval in µsec.
394 (default UST uid: 0, UST pid: 0, kernel: 0, metadata: 0)
396 .BR "\-\-read-timer USEC"
397 Read timer interval in µsec.
398 (default UST uid: 0, UST pid: 0, kernel: 200000, metadata: 0)
400 .BR "\-\-output TYPE"
401 Channel output type. Possible values: mmap, splice
402 (default UST uid: mmap, UST pid: mmap, kernel: splice, metadata: mmap)
404 .BR "\-\-buffers-uid"
405 Use per UID buffer (\-u only). Buffers are shared between applications
406 that have the same UID.
408 .BR "\-\-buffers-pid"
409 Use per PID buffer (\-u only). Each application has its own buffers.
411 .BR "\-\-buffers-global"
412 Use shared buffer for the whole system (\-k only)
414 .BR "\-C, \-\-tracefile-size SIZE"
415 Maximum size of each tracefile within a stream (in bytes).
416 0 means unlimited. (default: 0)
418 .BR "\-W, \-\-tracefile-count COUNT"
419 Used in conjunction with \-C option, this will limit the number of files
420 created to the specified count. 0 means unlimited. (default: 0)
425 $ lttng enable-channel -k -C 4096 -W 32 chan1
427 For each stream, the maximum size of each trace file will be 4096 bytes and
428 there will be a maximum of 32 different files. The file count is appended after
429 the stream number as seen in the following example. The last trace file is
430 smaller than 4096 since it was not completely filled.
433 ~/lttng-traces/[...]/chan1_0_0 (4096)
434 ~/lttng-traces/[...]/chan1_0_1 (4096)
435 ~/lttng-traces/[...]/chan1_0_2 (3245)
436 ~/lttng-traces/[...]/chan1_1_0 (4096)
441 $ lttng enable-channel -k -C 4096
443 This will create trace files of 4096 bytes and will create new ones as long as
444 there is data available.
449 \fBenable-event\fP NAME[,NAME2,...] [-k|-u] [OPTIONS]
453 A tracing event is always assigned to a channel. If \fB\-c, \-\-channel\fP is
454 omitted, a default channel named '\fBchannel0\fP' is created and the event is
455 added to it. If \fB\-c, \-\-channel\fP is omitted, but a non-default
456 channel already exists within the session, an error is returned. For the
457 user-space tracer, using \fB\-a, \-\-all\fP is the same as using the
460 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
467 Show summary of possible options and commands.
469 .BR "\-\-list-options"
470 Simple listing of options
472 .BR "\-s, \-\-session NAME"
473 Apply on session name
475 .BR "\-c, \-\-channel NAME"
476 Apply on channel name
479 Enable all tracepoints and syscalls. This actually enables a single
482 .BR "\-k, \-\-kernel"
483 Apply for the kernel tracer
485 .BR "\-u, \-\-userspace"
486 Apply for the user-space tracer
489 Tracepoint event (default). Userspace tracer supports wildcards at the end
490 of string. Don't forget to quote to deal with bash expansion.
497 .BR "\-\-loglevel NAME"
498 Tracepoint loglevel range from 0 to loglevel. Listed in the help (\-h).
500 .BR "\-\-loglevel-only NAME"
501 Tracepoint loglevel (only this loglevel).
502 The loglevel or loglevel-only options should be combined with a
503 tracepoint name or tracepoint wildcard.
505 .BR "\-\-probe (addr | symbol | symbol+offset)"
506 Dynamic probe. Addr and offset can be octal (0NNN...), decimal (NNN...)
507 or hexadecimal (0xNNN...)
509 .BR "\-\-function (addr | symbol | symbol+offset)"
510 Dynamic function entry/return probe. Addr and offset can be octal
511 (0NNN...), decimal (NNN...) or hexadecimal (0xNNN...)
514 System call event. Enabling syscalls tracing (kernel tracer), you will
515 not be able to disable them with disable-event. This is a known
516 limitation. You can disable the entire channel to do the trick.
518 .BR "\-\-filter 'expression'"
519 Set a filter on a newly enabled event. Filter expression on event
520 fields and context. The event will be recorded if the filter's
521 expression evaluates to TRUE. Only specify on first activation of a
522 given event within a session.
523 Specifying a filter is only allowed when enabling events within a session before
524 tracing is started. If the filter fails to link with the event
525 within the traced domain, the event will be discarded.
526 Filtering is currently only implemented for the user-space tracer.
531 'intfield > 500 && intfield < 503'
532 '(strfield == "test" || intfield != 10) && intfield > 33'
533 'doublefield > 1.1 && intfield < 5.3'
536 Wildcards are allowed at the end of strings:
538 In string literals, the escape character is a '\\'. Use '\\*' for
539 the '*' character, and '\\\\' for the '\\' character sequence. Wildcard
540 matches any sequence of characters, including an empty sub-string
541 (matches 0 or more characters).
543 Context information can be used for filtering. The examples below shows
544 usage of context filtering on the process name (using a wildcard), process ID
545 range, and unique thread ID. The process and thread IDs of
546 running applications can be found under columns "PID" and "LWP" of the
550 '$ctx.procname == "demo*"'
551 '$ctx.vpid >= 4433 && $ctx.vpid < 4455'
559 \fBdisable-channel\fP NAME[,NAME2,...] (\-k | \-u) [OPTIONS]
561 Disable tracing channel
563 Disabling a channel disables the tracing of all of the channel's events. A channel
564 can be reenabled by calling \fBlttng enable-channel NAME\fP again.
566 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
573 Show summary of possible options and commands.
575 .BR "\-\-list-options"
576 Simple listing of options
578 .BR "\-s, \-\-session NAME"
579 Apply on session name
581 .BR "\-k, \-\-kernel"
582 Apply for the kernel tracer
584 .BR "\-u, \-\-userspace"
585 Apply for the user-space tracer
590 \fBdisable-event\fP NAME[,NAME2,...] (\-k | \-u) [OPTIONS]
592 Disable tracing event
594 The event, once disabled, can be re-enabled by calling \fBlttng enable-event
597 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
600 If \fB\-c, \-\-channel\fP is omitted, the default channel name is used.
601 If \fB\-c, \-\-channel\fP is omitted, but a non-default channel already
602 exists within the session, an error is returned.
608 Show summary of possible options and commands.
610 .BR "\-\-list-options"
611 Simple listing of options
613 .BR "\-s, \-\-session NAME"
614 Apply on session name
616 .BR "\-c, \-\-channel NAME"
617 Apply on channel name
619 .BR "\-a, \-\-all-events"
620 Disable all events. This does NOT disable "*" but rather every known
621 events of the session.
623 .BR "\-k, \-\-kernel"
624 Apply for the kernel tracer
626 .BR "\-u, \-\-userspace"
627 Apply for the user-space tracer
632 \fBlist\fP [OPTIONS] [SESSION [SESSION OPTIONS]]
634 List tracing session information.
636 With no arguments, it will list available tracing session(s).
638 With the session name, it will display the details of the session including
639 the trace file path, the associated channels and their state (activated
640 and deactivated), the activated events and more.
642 With \-k alone, it will list all available kernel events (except the system
644 With \-u alone, it will list all available user-space events from registered
645 applications. Here is an example of 'lttng list \-u':
648 PID: 7448 - Name: /tmp/lttng-ust/tests/hello/.libs/lt-hello
649 ust_tests_hello:tptest_sighandler (type: tracepoint)
650 ust_tests_hello:tptest (type: tracepoint)
653 You can now enable any event listed by using the name :
654 \fBust_tests_hello:tptest\fP.
660 Show summary of possible options and commands.
662 .BR "\-\-list-options"
663 Simple listing of options
665 .BR "\-k, \-\-kernel"
668 .BR "\-u, \-\-userspace"
669 Select user-space domain.
675 .BR "\-c, \-\-channel NAME"
676 List details of a channel
678 .BR "\-d, \-\-domain"
679 List available domain(s)
684 \fBset-session\fP NAME [OPTIONS]
686 Set current session name
688 Will change the session name in the .lttngrc file.
694 Show summary of possible options and commands.
696 .BR "\-\-list-options"
697 Simple listing of options
702 \fBsnapshot\fP [OPTIONS] ACTION
704 Snapshot command for LTTng session.
710 Show summary of possible options and commands.
712 .BR "\-\-list-options"
713 Simple listing of options
719 \fBadd-output\fP [-m <SIZE>] [-s <NAME>] [-n <NAME>] <URL> | -C <URL> -D <URL>
721 Setup and add an snapshot output for a session. Output are the destination
722 where the snapshot will be sent. Only one output is permitted. To change it,
723 you'll need to delete it and add back the new one.
726 \fBdel-output\fP ID | NAME [-s <NAME>]
728 Delete an output for a session using the ID. You can either specify the
729 output's ID that can be found with list-output or the name.
732 \fBlist-output\fP [-s <NAME>]
734 List the output of a session. Attributes of the output are printed.
737 \fBrecord\fP [-m <SIZE>] [-s <NAME>] [-n <NAME>] [<URL> | -C <URL> -D <URL>]
739 Snapshot a session's buffer(s) for all domains. If an URL is specified, it is
740 used instead of a previously added output. Specifying only a name or/and a max
741 size will override the current output values. For instance, you can record a
742 snapshot with a custom maximum size or with a different name.
745 $ lttng snapshot add-output -n mysnapshot file:///data/snapshot
747 $ lttng snapshot record -n new_name_snapshot
750 The above will create a snapshot in /data/snapshot/new_name_snapshot* directory
751 rather then in mysnapshot*/
754 .B DETAILED ACTION OPTIONS
757 .BR "\-s, \-\-session NAME"
758 Apply to session name.
760 .BR "\-n, \-\-name NAME"
761 Name of the snapshot's output.
763 .BR "\-m, \-\-max-size SIZE"
764 Maximum size in bytes of the snapshot. The maxium size does not include the
765 metadata file. Human readable format is accepted: {+k,+M,+G}. For instance,
768 .BR "\-C, \-\-ctrl-url URL"
769 Set control path URL. (Must use -D also)
771 .BR "\-D, \-\-data-url URL"
772 Set data path URL. (Must use -C also)
777 \fBstart\fP [NAME] [OPTIONS]
781 It will start tracing for all tracers for a specific tracing session.
782 If NAME is omitted, the session name is taken from the .lttngrc file.
788 Show summary of possible options and commands.
790 .BR "\-\-list-options"
791 Simple listing of options
796 \fBstop\fP [NAME] [OPTIONS]
800 It will stop tracing for all tracers for a specific tracing session. Before
801 returning, the command checks for data availability meaning that it will wait
802 until the trace is readable for the session. Use \-\-no-wait to avoid this
805 If NAME is omitted, the session name is taken from the .lttngrc file.
811 Show summary of possible options and commands.
813 .BR "\-\-list-options"
814 Simple listing of options
816 Don't wait for data availability.
823 Show version information
829 Show summary of possible options and commands.
831 .BR "\-\-list-options"
832 Simple listing of options
837 \fBview\fP [SESSION_NAME] [OPTIONS]
839 View traces of a tracing session. By default, the babeltrace viewer
840 will be used for text viewing. If SESSION_NAME is omitted, the session
841 name is taken from the .lttngrc file.
849 .BR "\-\-list-options"
850 Simple listing of options
852 .BR "\-t, \-\-trace-path PATH"
853 Trace directory path for the viewer
855 .BR "\-e, \-\-viewer CMD"
856 Specify viewer and/or options to use This will completely override the
857 default viewers so please make sure to specify the full command. The
858 trace directory path of the session will be appended at the end to the
865 On success 0 is returned and a positive value on error. Value of 1 means a command
866 error, 2 an undefined command, 3 a fatal error and 4 a command warning meaning that
867 something went wrong during the command.
869 Any other value above 10, please refer to
870 .BR "<lttng/lttng-error.h>"
871 for a detailed list or use lttng_strerror() to get a human readable string of
875 .SH "ENVIRONMENT VARIABLES"
878 Note that all command line options override environment variables.
882 .IP "LTTNG_SESSIOND_PATH"
883 Allows one to specify the full session daemon binary path to lttng command line
884 tool. You can also use \-\-sessiond-path option having the same effect.
890 .BR lttng-sessiond(8),
892 .BR lttng-health-check(3)
897 If you encounter any issues or usability problem, please report it on our
898 mailing list <lttng-dev@lists.lttng.org> to help improve this project or
899 at https://bugs.lttng.org which is a bugtracker.
905 lttng is distributed under the GNU General Public License version 2. See the file
908 A Web site is available at http://lttng.org for more information on the LTTng
911 You can also find our git tree at http://git.lttng.org.
913 Mailing lists for support and development: <lttng-dev@lists.lttng.org>.
915 You can find us on IRC server irc.oftc.net (OFTC) in #lttng.
920 Thanks to Yannick Brosseau without whom this project would never have been so
921 lean and mean! Also thanks to the Ericsson teams working on tracing which
922 helped us greatly with detailed bug reports and unusual test cases.
924 Thanks to our beloved packager Alexandre Montplaisir-Goncalves (Ubuntu and PPA
925 maintainer) and Jon Bernard for our Debian packages.
927 Special thanks to Michel Dagenais and the DORSAL laboratory at Polytechnique de
928 Montreal for the LTTng journey.
933 lttng-tools was originally written by Mathieu Desnoyers, Julien Desfossez and
934 David Goulet. More people have since contributed to it. It is currently
935 maintained by David Goulet <dgoulet@efficios.com>.