1 .TH "LTTNG" "1" "December 3rd, 2012" "" ""
4 lttng \(em LTTng 2.1.x tracer control command line tool
10 lttng [OPTIONS] <COMMAND>
15 The LTTng project aims at providing highly efficient tracing tools for Linux.
16 It's tracers help tracking down performance issues and debugging problems
17 involving multiple concurrent processes and threads. Tracing across multiple
18 systems is also possible.
20 The \fBlttng\fP command line tool from the lttng-tools package is used to control
21 both kernel and user-space tracing. Every interactions with the tracer should
22 be done by this tool or by the liblttng-ctl provided with the lttng-tools
25 LTTng uses a session daemon (lttng-sessiond(8)), acting as a tracing registry,
26 which allows you to interact with multiple tracers (kernel and user-space)
27 inside the same container, a tracing session. Traces can be gathered from the
28 kernel and/or instrumented applications (lttng-ust(3)). Aggregating and reading
29 those traces is done using the babeltrace(1) text viewer.
31 We introduce the notion of \fBtracing domains\fP which is essentially a type of
32 tracer (kernel or user space for now). In the future, we could see a third
33 tracer being for instance an hypervisor. For some commands, you'll need to
34 specify on which domain the command applies (-u or -k). For instance, enabling
35 a kernel event, you must specify the kernel domain to the command so we know
36 for which tracer this event is for.
38 In order to trace the kernel, the session daemon needs to be running as root.
39 LTTng provides the use of a \fBtracing group\fP (default: tracing). Whomever is
40 in that group can interact with the root session daemon and thus trace the
41 kernel. Session daemons can co-exist meaning that you can have a session daemon
42 running as Alice that can be used to trace her applications along side with a
43 root daemon or even a Bob daemon. We highly recommend to start the session
44 daemon at boot time for stable and long term tracing.
46 Every user-space applications instrumented with lttng-ust(3), will
47 automatically register to the session daemon. This feature gives you the
48 ability to list available traceable applications and tracepoints on a per user
49 basis. (See \fBlist\fP command).
53 This program follow the usual GNU command line syntax with long options starting with
54 two dashes. Below is a summary of the available options.
59 Show summary of possible options and commands.
61 .BR "\-v, \-\-verbose"
63 Three levels of verbosity are available which are triggered by putting additional v to
64 the option (\-vv or \-vvv)
67 Suppress all messages (even errors).
69 .BR "\-g, \-\-group NAME"
70 Set unix tracing group name. (default: tracing)
72 .BR "\-n, \-\-no-sessiond"
73 Don't automatically spawn a session daemon.
75 .BR "\-\-sessiond\-path PATH"
76 Set session daemon full binary path.
78 .BR "\-\-list\-options"
79 Simple listing of lttng options.
81 .BR "\-\-list\-commands"
82 Simple listing of lttng commands.
88 Add context to event(s) and/or channel(s).
90 A context is basically extra information appended to a channel. For instance,
91 you could ask the tracer to add the PID information for all events in a
92 channel. You can also add performance monitoring unit counters (perf PMU) using
95 For example, this command will add the context information 'prio' and two perf
96 counters (hardware branch misses and cache misses), to all events in the trace
99 # lttng add-context \-k \-t prio \-t perf:branch-misses \-t perf:cache-misses
101 Please take a look at the help (\-h/\-\-help) for a detailed list of available
104 If no channel is given (\-c), the context is added to all channels. Otherwise
105 the context will be added only to the given channel (\-c).
107 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
115 Show summary of possible options and commands.
116 \-s, \-\-session NAME
117 Apply on session name.
118 \-c, \-\-channel NAME
119 Apply on channel name.
121 Apply for the kernel tracer
123 Apply for the user-space tracer
125 Context type. You can repeat this option on the command line. Please
126 use "lttng add-context \-h" to list all available types.
131 .IP "\fBcalibrate\fP"
133 Quantify LTTng overhead
135 The LTTng calibrate command can be used to find out the combined average
136 overhead of the LTTng tracer and the instrumentation mechanisms used. This
137 overhead can be calibrated in terms of time or using any of the PMU performance
138 counter available on the system.
140 For now, the only calibration implemented is that of the kernel function
141 instrumentation (kretprobes).
143 * Calibrate kernel function instrumentation
145 Let's use an example to show this calibration. We use an i7 processor with 4
146 general-purpose PMU registers. This information is available by issuing dmesg,
147 looking for "generic registers".
149 This sequence of commands will gather a trace executing a kretprobe hooked on
150 an empty function, gathering PMU counters LLC (Last Level Cache) misses
151 information (see lttng add-context \-\-help to see the list of available PMU
154 # lttng create calibrate-function
155 # lttng enable-event calibrate \-\-kernel \-\-function lttng_calibrate_kretprobe
156 # lttng add-context \-\-kernel \-t perf:LLC-load-misses \-t perf:LLC-store-misses \\
157 \-t perf:LLC-prefetch-misses
159 # for a in $(seq 1 10); do \\
160 lttng calibrate \-\-kernel \-\-function;
163 # babeltrace $(ls \-1drt ~/lttng-traces/calibrate-function-* | tail \-n 1)
165 The output from babeltrace can be saved to a text file and opened in a
166 spreadsheet (e.g. oocalc) to focus on the per-PMU counter delta between
167 consecutive "calibrate_entry" and "calibrate_return" events. Note that these
168 counters are per-CPU, so scheduling events would need to be present to account
169 for migration between CPU. Therefore, for calibration purposes, only events
170 staying on the same CPU must be considered.
172 The average result, for the i7, on 10 samples:
175 perf_LLC_load_misses: 5.0 0.577
176 perf_LLC_store_misses: 1.6 0.516
177 perf_LLC_prefetch_misses: 9.0 14.742
179 As we can notice, the load and store misses are relatively stable across runs
180 (their standard deviation is relatively low) compared to the prefetch misses.
181 We can conclude from this information that LLC load and store misses can be
182 accounted for quite precisely, but prefetches within a function seems to behave
183 too erratically (not much causality link between the code executed and the CPU
184 prefetch activity) to be accounted for.
191 Show summary of possible options and commands.
193 Apply for the kernel tracer
195 Apply for the user-space tracer
197 Dynamic function entry/return probe (default)
202 .IP "\fBcreate\fP [NAME] [OPTIONS]
204 Create tracing session.
206 A tracing session contains channel(s) which contains event(s). It is domain
207 agnostic meaning that you can enable channels and events for either the
208 user-space tracer and/or the kernel tracer. It acts like a container
209 aggregating multiple tracing sources.
211 On creation, a \fB.lttngrc\fP file is created in your $HOME directory
212 containing the current session name. If NAME is omitted, a session name is
213 automatically created having this form: 'auto-yyyymmdd-hhmmss'.
215 If no \fB\-o, \-\-output\fP is specified, the traces will be written in
223 Show summary of possible options and commands.
225 Simple listing of options
227 Specify output path for traces
229 Using these options, each API call can be controlled individually. For
230 instance, \-C does not enable the consumer automatically. You'll need the \-e
234 Set URL for the enable-consumer destination. It is persistent for the
235 session lifetime. Redo the command to change it. This will set both
236 data and control URL for network.
237 \-C, \-\-ctrl-url=URL
238 Set control path URL. (Must use -D also)
239 \-D, \-\-data-url=URL
240 Set data path URL. (Must use -C also)
242 Don't activate a consumer for this session.
244 Disable consumer for this session.
246 See \fBenable-consumer\fP command below for the supported URL format.
250 # lttng create -U net://192.168.1.42
251 Uses TCP and default ports for the given destination.
253 # lttng create -U net6://[fe80::f66d:4ff:fe53:d220]
254 Uses TCP, default ports and IPv6.
256 # lttng create s1 -U net://myhost.com:3229
257 Create session s1 and set its consumer to myhost.com on port 3229 for control.
262 .IP "\fBdestroy\fP [OPTIONS] [NAME]"
264 Teardown tracing session
266 Free memory on the session daemon and tracer side. It's gone!
268 If NAME is omitted, the session name is taken from the .lttngrc file.
275 Show summary of possible options and commands.
279 Simple listing of options
284 .IP "\fBenable-channel\fP NAME[,NAME2,...] [-k|-u] [OPTIONS]"
286 Enable tracing channel
288 To enable event, you must first enable a channel which contains event(s).
290 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
300 Simple listing of options
301 \-s, \-\-session NAME
302 Apply on session name
304 Apply to the kernel tracer
306 Apply to the user-space tracer
309 Discard event when subbuffers are full (default)
311 Flight recorder mode : overwrites events when subbuffers are full
313 Subbuffer size in bytes (default: 4096, kernel default: 262144)
315 Number of subbuffers (default: 4)
316 Needs to be a power of 2 for kernel and ust tracers
317 \-\-switch-timer USEC
318 Switch subbuffer timer interval in usec (default: 0)
319 Needs to be a power of 2 for kernel and ust tracers
321 Read timer interval in usec (default: 200)
323 Channel output type. Possible values: mmap, splice
328 .IP "\fBenable-consumer\fP [-u|-k] [URL] [OPTIONS]"
330 Enable a consumer for the tracing session and domain.
332 By default, every tracing session has a consumer attached to it using the local
333 filesystem as output. The trace is written in $HOME/lttng-traces. This command
334 allows the user to specify a specific URL after the session was created for a
335 specific domain. If no domain is specified, the consumer is applied on all
338 Without options, the behavior is to enable a consumer to the current URL. The
339 default URL is the local filesystem at the path of the session mentioned above.
341 The enable-consumer feature supports both local and network transport. You must
342 have a running \fBlttng-relayd(8)\fP for network transmission or any other daemon
343 that can understand the streaming protocol of LTTng.
350 Show summary of possible options and commands.
352 Simple listing of options
353 \-s, \-\-session NAME
354 Apply on session name
356 Apply for the kernel tracer
358 Apply for the user-space tracer
360 Using these options, each API call can be controlled individually. For
361 instance, \-C does not enable the consumer automatically. You'll need the \-e
365 Set URL for the enable-consumer destination. It is persistent for the
366 session lifetime. Redo the command to change it. This will set both
367 data and control URL for network.
368 \-C, \-\-ctrl-url=URL
369 Set control path URL. (Must use -D also)
370 \-D, \-\-data-url=URL
371 Set data path URL. (Must use -C also)
377 proto://[HOST|IP][:PORT1[:PORT2]][/TRACE_PATH]
379 Supported protocols are (proto):
381 Local filesystem full path.
384 This will use the default network transport layer which is TCP for both
385 control (PORT1) and data port (PORT2). The default ports are
386 respectively 5342 and 5343. Note that net[6]:// is not yet supported.
389 Can only be used with -C and -D together
391 NOTE: IPv6 address MUST be enclosed in brackets '[]' (rfc2732)
395 $ lttng enable-consumer -u net://192.168.1.42
397 Uses TCP and default ports for user space tracing (-u) where the IP address
398 above is the destination machine where the traces will be streamed and a
399 \fBlttng-relayd(8)\fP is listening.
402 .IP "\fBenable-event\fP NAME[,NAME2,...] [-k|-u] [OPTIONS]"
406 A tracing event is always assigned to a channel. If \fB\-c, \-\-channel\fP is
407 omitted, a default channel named '\fBchannel0\fP' is created and the event is
408 added to it. For the user-space tracer, using \fB\-a, \-\-all\fP is the same as
409 using the wildcard "*".
411 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
419 Show summary of possible options and commands.
421 Simple listing of options
422 \-s, \-\-session NAME
423 Apply on session name
424 \-c, \-\-channel NAME
425 Apply on channel name
427 Enable all tracepoints and syscalls. This actually enable a single
430 Apply for the kernel tracer
432 Apply for the user-space tracer
435 Tracepoint event (default)
436 - userspace tracer supports wildcards at end of string. Don't forget to
437 quote to deal with bash expansion.
442 Tracepoint loglevel range from 0 to loglevel. Listed in the help (\-h).
443 \-\-loglevel-only NAME
444 Tracepoint loglevel (only this loglevel).
446 The loglevel or loglevel-only options should be combined with a
447 tracepoint name or tracepoint wildcard.
448 \-\-probe [addr | symbol | symbol+offset]
449 Dynamic probe. Addr and offset can be octal (0NNN...), decimal (NNN...)
450 or hexadecimal (0xNNN...)
451 \-\-function [addr | symbol | symbol+offset]
452 Dynamic function entry/return probe. Addr and offset can be octal
453 (0NNN...), decimal (NNN...) or hexadecimal (0xNNN...)
455 System call event. Enabling syscalls tracing (kernel tracer), you will
456 not be able to disable them with disable-event. This is a known
457 limitation. You can disable the entire channel to do the trick.
459 \-\-filter 'expression'
460 Set a filter on a newly enabled event. Filter expression on event
461 fields, event recording depends on evaluation. Only specify on first
462 activation of a given event within a session. Filter only allowed when
463 enabling events within a session before tracing is started. If the
464 filter fails to link with the event within the traced domain, the event
465 will be discarded. Currently, filter is only implemented for the
470 'intfield > 500 && intfield < 503'
471 '(stringfield == "test" || intfield != 10) && intfield > 33'
472 'doublefield > 1.1 && intfield < 5.3'
474 Wildcards are allowed at the end of strings:
476 In string literals, the escape character is a '\\'. Use '\\*' for
477 the '*' character, and '\\\\' for the '\\' character.
480 .IP "\fBdisable-channel\fP NAME[,NAME2,...] [\-k|\-u] [OPTIONS]"
482 Disable tracing channel
484 Disabling a channel makes all event(s) in that channel to stop tracing. You can
485 enable it back by calling \fBlttng enable-channel NAME\fP again.
487 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
495 Show summary of possible options and commands.
497 Simple listing of options
498 \-s, \-\-session NAME
499 Apply on session name
501 Apply for the kernel tracer
503 Apply for the user-space tracer
506 .IP "\fBdisable-consumer\fP [\-k|\-u] [OPTIONS]"
508 Disable the consumer of a tracing session.
510 This call MUST be done BEFORE tracing has started.
517 Show summary of possible options and commands.
519 Simple listing of options
520 \-s, \-\-session NAME
521 Apply on session name
523 Apply for the kernel tracer
525 Apply for the user-space tracer
528 .IP "\fBdisable-event\fP NAME[,NAME2,...] [\-k|\-u] [OPTIONS]"
530 Disable tracing event
532 The event, once disabled, can be re-enabled by calling \fBlttng enable-event
535 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
543 Show summary of possible options and commands.
545 Simple listing of options
546 \-s, \-\-session NAME
547 Apply on session name
549 Disable all events. This does NOT disable "*" but rather
550 every known events of the session.
552 Apply for the kernel tracer
554 Apply for the user-space tracer
557 .IP "\fBlist\fP [\-k|\-u] [SESSION [SESSION_OPTIONS]]"
559 List tracing session information.
561 With no arguments, it will list available tracing session(s).
563 With the session name, it will display the details of the session including
564 the trace file path, the associated channels and their state (activated
565 and deactivated), the activated events and more.
567 With \-k alone, it will list all available kernel events (except the system
569 With \-u alone, it will list all available user-space events from registered
570 applications. Here is an example of 'lttng list \-u':
572 PID: 7448 - Name: /tmp/lttng-ust/tests/hello/.libs/lt-hello
573 ust_tests_hello:tptest_sighandler (type: tracepoint)
574 ust_tests_hello:tptest (type: tracepoint)
576 You can now enable any event listed by using the name :
577 \fBust_tests_hello:tptest\fP.
584 Show summary of possible options and commands.
586 Simple listing of options
590 Select user-space domain.
594 \-c, \-\-channel NAME
595 List details of a channel
597 List available domain(s)
600 .IP "\fBset-session\fP NAME"
602 Set current session name
604 Will change the session name in the .lttngrc file.
611 Show summary of possible options and commands.
613 Simple listing of options
618 .IP "\fBstart\fP [NAME] [OPTIONS]"
622 It will start tracing for all tracers for a specific tracing session.
624 If NAME is omitted, the session name is taken from the .lttngrc file.
631 Show summary of possible options and commands.
633 Simple listing of options
638 .IP "\fBstop\fP [NAME] [OPTIONS]"
642 It will stop tracing for all tracers for a specific tracing session. Before
643 returning, the command checks for data availability meaning that it will wait
644 until the trace is readable for the session. Use \-\-no-wait to avoid this
647 If NAME is omitted, the session name is taken from the .lttngrc file.
654 Show summary of possible options and commands.
656 Simple listing of options
658 Don't wait for data availability.
665 Show version information
672 Show summary of possible options and commands.
674 Simple listing of options
679 .IP "\fBview\fP [SESSION_NAME] [OPTIONS]"
681 View traces of a tracing session
683 By default, the babeltrace viewer will be used for text viewing.
685 If SESSION_NAME is omitted, the session name is taken from the .lttngrc file.
695 Simple listing of options
696 \-t, \-\-trace-path PATH
697 Trace directory path for the viewer
699 Specify viewer and/or options to use
700 This will completely override the default viewers so
701 please make sure to specify the full command. The trace
702 directory path of the session will be appended at the end
707 On success 0 is returned and a positive value on error. Value of 1 means a command
708 error, 2 an undefined command, 3 a fatal error and 4 a command warning meaning that
709 something went wrong during the command.
711 Any other value above 10, please refer to
712 .BR <lttng/lttng-error.h>
713 for a detailed list or use lttng_strerror() to get a human readable string of
717 .SH "ENVIRONMENT VARIABLES"
720 Note that all command line options override environment variables.
724 .IP "LTTNG_SESSIOND_PATH"
725 Allows one to specify the full session daemon binary path to lttng command line
726 tool. You can also use \-\-sessiond-path option having the same effect.
730 .BR lttng-sessiond(8),
732 .BR lttng-health-check(3)
735 If you encounter any issues or usability problem, please report it on our
736 mailing list <lttng-dev@lists.lttng.org> to help improve this project or
737 at https://bugs.lttng.org which is a bugtracker.
741 lttng is distributed under the GNU General Public License version 2. See the file
744 A Web site is available at http://lttng.org for more information on the LTTng
747 You can also find our git tree at http://git.lttng.org.
749 Mailing lists for support and development: <lttng-dev@lists.lttng.org>.
751 You can find us on IRC server irc.oftc.net (OFTC) in #lttng.
756 Thanks to Yannick Brosseau without whom this project would never have been so
757 lean and mean! Also thanks to the Ericsson teams working on tracing which
758 helped us greatly with detailed bug reports and unusual test cases.
760 Thanks to our beloved packager Alexandre Montplaisir-Goncalves (Ubuntu and PPA
761 maintainer) and Jon Bernard for our Debian packages.
763 Special thanks to Michel Dagenais and the DORSAL laboratory at Polytechnique de
764 Montreal for the LTTng journey.
769 lttng-tools was originally written by Mathieu Desnoyers, Julien Desfossez and
770 David Goulet. More people have since contributed to it. It is currently
771 maintained by David Goulet <dgoulet@efficios.com>.