1 .TH "LTTNG" "1" "July 17, 2012" "" ""
4 lttng \(em LTTng 2.0 tracer control command line tool
10 lttng [OPTIONS] <COMMAND>
15 The LTTng project aims at providing highly efficient tracing tools for Linux.
16 It's tracers help tracking down performance issues and debugging problems
17 involving multiple concurrent processes and threads. Tracing across multiple
18 systems is also possible.
20 The \fBlttng\fP command line tool from the lttng-tools package is used to control
21 both kernel and user-space tracing. Every interactions with the tracer should
22 be done by this tool or by the liblttng-ctl provided with the lttng-tools
25 LTTng uses a session daemon (lttng-sessiond(8)), acting as a tracing registry,
26 which permits you to interact with multiple tracers (kernel and user-space)
27 inside the same container, a tracing session. Traces can be gathered from the
28 kernel and/or instrumented applications (lttng-ust(3)). Aggregating and reading
29 those traces is done using the babeltrace(1) text viewer.
31 In order to trace the kernel, the session daemon needs to be running as root.
32 LTTng provides the use of a \fBtracing group\fP (default: tracing). Whomever is
33 in that group can interact with the root session daemon and thus trace the
34 kernel. Session daemons can co-exist meaning that you can have a session daemon
35 running as Alice that can be used to trace her applications along side with a
36 root daemon or even a Bob daemon. We highly recommend to start the session
37 daemon at boot time for stable and long term tracing.
39 Every user-space applications instrumented with lttng-ust(3), will
40 automatically register to the session daemon. This feature gives you the
41 ability to list available traceable applications and tracepoints on a per user
42 basis. (See \fBlist\fP command).
46 This program follow the usual GNU command line syntax with long options starting with
47 two dashes. Below is a summary of the available options.
52 Show summary of possible options and commands.
54 .BR "\-v, \-\-verbose"
56 Three levels of verbosity are available which are triggered by putting additional v to
57 the option (\-vv or \-vvv)
60 Suppress all messages (even errors).
62 .BR "\-g, \-\-group NAME"
63 Set unix tracing group name. (default: tracing)
65 .BR "\-n, \-\-no-sessiond"
66 Don't automatically spawn a session daemon.
68 .BR "\-\-sessiond\-path"
69 Set session daemon full binary path.
71 .BR "\-\-list\-options"
72 Simple listing of lttng options.
74 .BR "\-\-list\-commands"
75 Simple listing of lttng commands.
81 Add context to event(s) and/or channel(s).
83 A context is basically extra information appended to a channel or event. For
84 instance, you could ask the tracer to add the PID information within the
85 "sched_switch" kernel event. You can also add performance monitoring unit
86 counters (perf PMU) using the perf kernel API).
88 For example, this command will add the context information 'prio' and two perf
89 counters (hardware branch misses and cache misses), to all events in the trace
92 # lttng add-context \-k \-t prio \-t perf:branch-misses \-t perf:cache-misses
94 Please take a look at the help (\-h/\-\-help) for a detailed list of available
97 If no channel and no event is given (\-c/\-e), the context is added to all
98 channels (which applies automatically to all events in that channel). Otherwise
99 the context will be added only to the channel (\-c) and/or event (\-e) indicated.
101 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
109 Show summary of possible options and commands.
110 \-s, \-\-session NAME
111 Apply on session name.
112 \-c, \-\-channel NAME
113 Apply on channel name.
117 Apply for the kernel tracer
119 Apply for the user-space tracer
121 Context type. You can repeat this option on the command line. Please
122 use "lttng add-context \-h" to list all available types.
127 .IP "\fBcalibrate\fP"
129 Quantify LTTng overhead
131 The LTTng calibrate command can be used to find out the combined average
132 overhead of the LTTng tracer and the instrumentation mechanisms used. This
133 overhead can be calibrated in terms of time or using any of the PMU performance
134 counter available on the system.
136 For now, the only calibration implemented is that of the kernel function
137 instrumentation (kretprobes).
139 * Calibrate kernel function instrumentation
141 Let's use an example to show this calibration. We use an i7 processor with 4
142 general-purpose PMU registers. This information is available by issuing dmesg,
143 looking for "generic registers".
145 This sequence of commands will gather a trace executing a kretprobe hooked on
146 an empty function, gathering PMU counters LLC (Last Level Cache) misses
147 information (see lttng add-context \-\-help to see the list of available PMU
150 # lttng create calibrate-function
151 # lttng enable-event calibrate \-\-kernel \-\-function lttng_calibrate_kretprobe
152 # lttng add-context \-\-kernel \-t perf:LLC-load-misses \-t perf:LLC-store-misses \\
153 \-t perf:LLC-prefetch-misses
155 # for a in $(seq 1 10); do \\
156 lttng calibrate \-\-kernel \-\-function;
159 # babeltrace $(ls \-1drt ~/lttng-traces/calibrate-function-* | tail \-n 1)
161 The output from babeltrace can be saved to a text file and opened in a
162 spreadsheet (e.g. oocalc) to focus on the per-PMU counter delta between
163 consecutive "calibrate_entry" and "calibrate_return" events. Note that these
164 counters are per-CPU, so scheduling events would need to be present to account
165 for migration between CPU. Therefore, for calibration purposes, only events
166 staying on the same CPU must be considered.
168 The average result, for the i7, on 10 samples:
171 perf_LLC_load_misses: 5.0 0.577
172 perf_LLC_store_misses: 1.6 0.516
173 perf_LLC_prefetch_misses: 9.0 14.742
175 As we can notice, the load and store misses are relatively stable across runs
176 (their standard deviation is relatively low) compared to the prefetch misses.
177 We can conclude from this information that LLC load and store misses can be
178 accounted for quite precisely, but prefetches within a function seems to behave
179 too erratically (not much causality link between the code executed and the CPU
180 prefetch activity) to be accounted for.
187 Show summary of possible options and commands.
189 Apply for the kernel tracer
191 Apply for the user-space tracer
193 Dynamic function entry/return probe (default)
198 .IP "\fBcreate\fP [NAME] [OPTIONS]
200 Create tracing session.
202 A tracing session contains channel(s) which contains event(s). It is domain
203 agnostic meaning that you can enable channels and events for either the
204 user-space tracer and/or the kernel tracer. It acts like a container
205 aggregating multiple tracing sources.
207 On creation, a \fB.lttngrc\fP file is created in your $HOME directory
208 containing the current session name. If NAME is omitted, a session name is
209 automatically created having this form: 'auto-yyyymmdd-hhmmss'.
211 If no \fB\-o, \-\-output\fP is specified, the traces will be written in
219 Show summary of possible options and commands.
221 Simple listing of options
223 Specify output path for traces
225 Using these options, each API call can be controlled individually. For
226 instance, \-C does not enable the consumer automatically. You'll need the \-e
230 Set URL for the enable-consumer destination. It is persistent for the
231 session lifetime. Redo the command to change it. This will set both
232 data and control URL for network.
233 \-C, \-\-ctrl-url=URL
234 Set control path URL. (Must use -D also)
235 \-D, \-\-data-url=URL
236 Set data path URL. (Must use -C also)
238 Don't activate a consumer for this session.
240 Disable consumer for this session.
242 See \fBenable-consumer\fP command below for the supported URL format.
246 # lttng create -U net://192.168.1.42
247 Uses TCP and default ports for the given destination.
249 # lttng create -U net6://[fe80::f66d:4ff:fe53:d220]
250 Uses TCP, default ports and IPv6.
252 # lttng create s1 -U net://myhost.com:3229
253 Create session s1 and set its consumer to myhost.com on port 3229 for control.
258 .IP "\fBdestroy\fP [OPTIONS] [NAME]"
260 Teardown tracing session
262 Free memory on the session daemon and tracer side. It's gone!
264 If NAME is omitted, the session name is taken from the .lttngrc file.
271 Show summary of possible options and commands.
275 Simple listing of options
280 .IP "\fBenable-channel\fP NAME[,NAME2,...] [-k|-u] [OPTIONS]"
282 Enable tracing channel
284 To enable event, you must first enable a channel which contains event(s).
286 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
296 Simple listing of options
298 Apply on session name
300 Apply to the kernel tracer
302 Apply to the user-space tracer
305 Discard event when subbuffers are full (default)
307 Flight recorder mode : overwrites events when subbuffers are full
309 Subbuffer size in bytes (default: 4096, kernel default: 262144)
311 Number of subbuffers (default: 4)
312 Needs to be a power of 2 for kernel and ust tracers
314 Switch subbuffer timer interval in usec (default: 0)
315 Needs to be a power of 2 for kernel and ust tracers
317 Read timer interval in usec (default: 200)
322 .IP "\fBenable-consumer\fP [-u|-k] [URL] [OPTIONS]"
324 Enable a consumer for the tracing session and domain.
326 By default, every tracing session has a consumer attached to it using the local
327 filesystem as output. The trace is written in $HOME/lttng-traces. This command
328 allows the user to specify a specific URL after the session was created for a
329 specific domain. If no domain is specified, the consumer is applied on all
332 Without options, the behavior is to enable a consumer to the current URL. The
333 default URL is the local filesystem at the path of the session mentioned above.
335 The enable-consumer feature supports both local and network transport. You must
336 have a running \fBlttng-relayd(8)\fP for network transmission or any other daemon
337 that can understand the streaming protocol of LTTng.
344 Show summary of possible options and commands.
346 Simple listing of options
348 Apply on session name
350 Apply for the kernel tracer
352 Apply for the user-space tracer
354 Using these options, each API call can be controlled individually. For
355 instance, \-C does not enable the consumer automatically. You'll need the \-e
359 Set URL for the enable-consumer destination. It is persistent for the
360 session lifetime. Redo the command to change it. This will set both
361 data and control URL for network.
362 \-C, \-\-ctrl-url=URL
363 Set control path URL. (Must use -D also)
364 \-D, \-\-data-url=URL
365 Set data path URL. (Must use -C also)
371 proto://[HOST|IP][:PORT1[:PORT2]][/TRACE_PATH]
373 Supported protocols are (proto):
375 Local filesystem full path.
378 This will use the default network transport layer which is TCP for both
379 control (PORT1) and data port (PORT2). The default ports are
380 respectively 5342 and 5343.
383 Can only be used with -C and -D together
385 NOTE: IPv6 address MUST be enclosed in brackets '[]' (rfc2732)
389 $ lttng enable-consumer -u net://192.168.1.42
391 Uses TCP and default ports for user space tracing (-u) where the IP address
392 above is the destination machine where the traces will be streamed and a
393 \fBlttng-relayd(8)\fP is listening.
396 .IP "\fBenable-event\fP NAME[,NAME2,...] [-k|-u] [OPTIONS]"
400 A tracing event is always assigned to a channel. If \fB\-c, \-\-channel\fP is
401 omitted, a default channel named '\fBchannel0\fP' is created and the event is
402 added to it. For the user-space tracer, using \fB\-a, \-\-all\fP is the same as
403 using the wildcard "*".
405 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
413 Show summary of possible options and commands.
415 Simple listing of options
417 Apply on session name
419 Apply on channel name
421 Enable all tracepoints and syscalls
423 Apply for the kernel tracer
425 Apply for the user-space tracer
428 Tracepoint event (default)
429 - userspace tracer supports wildcards at end of string. Don't forget to
430 quote to deal with bash expansion.
436 \-\-probe [addr | symbol | symbol+offset]
437 Dynamic probe. Addr and offset can be octal (0NNN...), decimal (NNN...)
438 or hexadecimal (0xNNN...)
439 \-\-function [addr | symbol | symbol+offset]
440 Dynamic function entry/return probe. Addr and offset can be octal
441 (0NNN...), decimal (NNN...) or hexadecimal (0xNNN...)
443 System call event. Enabling syscalls tracing (kernel tracer), you will
444 not be able to disable them with disable-event. This is a known
445 limitation. You can disable the entire channel to do the trick.
447 \-\-filter 'expression'
448 Set a filter on a newly enabled event. Filter expression on event
449 fields, event recording depends on evaluation. Only specify on first
450 activation of a given event within a session. Filter only allowed when
451 enabling events within a session before tracing is started. If the
452 filter fails to link with the event within the traced domain, the event
453 will be discarded. Currently, filter is only implemented for the
458 'intfield > 500 && intfield < 503'
459 '(stringfield == "test" || intfield != 10) && intfield > 33'
460 'doublefield > 1.1 && intfield < 5.3'
462 Wildcards are allowed at the end of strings:
464 In string literals, the escape character is a '\\'. Use '\\*' for
465 the '*' character, and '\\\\' for the '\\' character.
468 .IP "\fBdisable-channel\fP NAME[,NAME2,...] [\-k|\-u] [OPTIONS]"
470 Disable tracing channel
472 Disabling a channel makes all event(s) in that channel to stop tracing. You can
473 enable it back by calling \fBlttng enable-channel NAME\fP again.
475 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
483 Show summary of possible options and commands.
485 Simple listing of options
486 \-s, \-\-session NAME
487 Apply on session name
489 Apply for the kernel tracer
491 Apply for the user-space tracer
494 .IP "\fBdisable-consumer\fP [\-k|\-u] [OPTIONS]"
496 Disable the consumer of a tracing session.
498 This call MUST be done BEFORE tracing has started.
505 Show summary of possible options and commands.
507 Simple listing of options
508 \-s, \-\-session NAME
509 Apply on session name
511 Apply for the kernel tracer
513 Apply for the user-space tracer
516 .IP "\fBdisable-event\fP NAME[,NAME2,...] [\-k|\-u] [OPTIONS]"
518 Disable tracing event
520 The event, once disabled, can be re-enabled by calling \fBlttng enable-event
523 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
531 Show summary of possible options and commands.
533 Simple listing of options
535 Apply on session name
537 Apply for the kernel tracer
539 Apply for the user-space tracer
542 .IP "\fBlist\fP [\-k|\-u] [SESSION [SESSION_OPTIONS]]"
544 List tracing session information.
546 With no arguments, it will list available tracing session(s).
548 With the session name, it will display the details of the session including
549 the trace file path, the associated channels and their state (activated
550 and deactivated), the activated events and more.
552 With \-k alone, it will list all available kernel events (except the system
554 With \-u alone, it will list all available user-space events from registered
555 applications. Here is an example of 'lttng list \-u':
557 PID: 7448 - Name: /tmp/lttng-ust/tests/hello/.libs/lt-hello
558 ust_tests_hello:tptest_sighandler (type: tracepoint)
559 ust_tests_hello:tptest (type: tracepoint)
561 You can now enable any event listed by using the name :
562 \fBust_tests_hello:tptest\fP.
569 Show summary of possible options and commands.
571 Simple listing of options
575 Select user-space domain.
579 \-c, \-\-channel NAME
580 List details of a channel
582 List available domain(s)
585 .IP "\fBset-session\fP NAME"
587 Set current session name
589 Will change the session name in the .lttngrc file.
596 Show summary of possible options and commands.
598 Simple listing of options
603 .IP "\fBstart\fP [NAME] [OPTIONS]"
607 It will start tracing for all tracers for a specific tracing session.
609 If NAME is omitted, the session name is taken from the .lttngrc file.
616 Show summary of possible options and commands.
618 Simple listing of options
623 .IP "\fBstop\fP [NAME] [OPTIONS]"
627 It will stop tracing for all tracers for a specific tracing session.
629 If NAME is omitted, the session name is taken from the .lttngrc file.
636 Show summary of possible options and commands.
638 Simple listing of options
645 Show version information
652 Show summary of possible options and commands.
654 Simple listing of options
659 .IP "\fBview\fP [SESSION_NAME] [OPTIONS]"
661 View traces of a tracing session
663 By default, the babeltrace viewer will be used for text viewing.
665 If SESSION_NAME is omitted, the session name is taken from the .lttngrc file.
675 Simple listing of options
676 \-t, \-\-trace-path PATH
677 Trace directory path for the viewer
679 Specify viewer and/or options to use
680 This will completely override the default viewers so
681 please make sure to specify the full command. The trace
682 directory path of the session will be appended at the end
687 On success 0 is returned and a positive value on error. Value of 1 means a command
688 error, 2 an undefined command, 3 a fatal error and 4 a command warning meaning that
689 something went wrong during the command.
691 Any other value above 10, please refer to
692 .BR <lttng/lttng-error.h>
693 for a detailed list or use lttng_strerror() to get a human readable string of
697 .SH "ENVIRONMENT VARIABLES"
700 Note that all command line options override environment variables.
704 .IP "LTTNG_SESSIOND_PATH"
705 Allows one to specify the full session daemon binary path to lttng command line
706 tool. You can also use \-\-sessiond-path option having the same effect.
710 .BR lttng-sessiond(8),
712 .BR lttng-health-check(3)
715 With version 2.1 and earlier, if you start a tracing session and than enable
716 kernel events, they are not recorded and the tracing session fails to stop. To
717 fix this, simply enable events before starting the session.
719 If you encounter any issues or usability problem, please report it on our
720 mailing list <lttng-dev@lists.lttng.org> to help improve this project or
721 at https://bugs.lttng.org which is a bugtracker.
725 lttng is distributed under the GNU General Public License version 2. See the file
728 A Web site is available at http://lttng.org for more information on the LTTng
731 You can also find our git tree at http://git.lttng.org.
733 Mailing lists for support and development: <lttng-dev@lists.lttng.org>.
735 You can find us on IRC server irc.oftc.net (OFTC) in #lttng.
740 Thanks to Yannick Brosseau without whom this project would never have been so
741 lean and mean! Also thanks to the Ericsson teams working on tracing which
742 helped us greatly with detailed bug reports and unusual test cases.
744 Thanks to our beloved packager Alexandre Montplaisir-Goncalves (Ubuntu and PPA
745 maintainer) and Jon Bernard for our Debian packages.
747 Special thanks to Michel Dagenais and the DORSAL laboratory at Polytechnique de
748 Montreal for the LTTng journey.
753 lttng-tools was originally written by Mathieu Desnoyers, Julien Desfossez and
754 David Goulet. More people have since contributed to it. It is currently
755 maintained by David Goulet <dgoulet@efficios.com>.