.TH "LTTNG" "1" "December 3rd, 2012" "" ""

.SH "NAME"
lttng \(em LTTng 2.x tracer control command line tool

.SH "SYNOPSIS"

.PP
.nf
lttng [OPTIONS] <COMMAND>
.fi
.SH "DESCRIPTION"

.PP
The LTTng project aims at providing highly efficient tracing tools for Linux.
It's tracers help tracking down performance issues and debugging problems
involving multiple concurrent processes and threads. Tracing across multiple
systems is also possible.

The \fBlttng\fP command line tool from the lttng-tools package is used to control
both kernel and user-space tracing. Every interactions with the tracer should
be done by this tool or by the liblttng-ctl provided with the lttng-tools
package.

LTTng uses a session daemon (lttng-sessiond(8)), acting as a tracing registry,
which allows you to interact with multiple tracers (kernel and user-space)
inside the same container, a tracing session. Traces can be gathered from the
kernel and/or instrumented applications (lttng-ust(3)). Aggregating and reading
those traces is done using the babeltrace(1) text viewer.

We introduce the notion of \fBtracing domains\fP which is essentially a type of
tracer (kernel or user space for now). In the future, we could see a third
tracer being for instance an hypervisor. For some commands, you'll need to
specify on which domain the command applies (-u or -k). For instance, enabling
a kernel event, you must specify the kernel domain to the command so we know
for which tracer this event is for.

In order to trace the kernel, the session daemon needs to be running as root.
LTTng provides the use of a \fBtracing group\fP (default: tracing). Whomever is
in that group can interact with the root session daemon and thus trace the
kernel. Session daemons can co-exist meaning that you can have a session daemon
running as Alice that can be used to trace her applications along side with a
root daemon or even a Bob daemon. We highly recommend to start the session
daemon at boot time for stable and long term tracing.

Every user-space applications instrumented with lttng-ust(3), will
automatically register to the session daemon. This feature gives you the
ability to list available traceable applications and tracepoints on a per user
basis. (See \fBlist\fP command).
.SH "OPTIONS"

.PP
This program follow the usual GNU command line syntax with long options starting with
two dashes. Below is a summary of the available options.
.PP

.TP
.BR "\-h, \-\-help"
Show summary of possible options and commands.
.TP
.BR "\-v, \-\-verbose"
Increase verbosity.
Three levels of verbosity are available which are triggered by putting additional v to
the option (\-vv or \-vvv)
.TP
.BR "\-q, \-\-quiet"
Suppress all messages (even errors).
.TP
.BR "\-g, \-\-group NAME"
Set unix tracing group name. (default: tracing)
.TP
.BR "\-n, \-\-no-sessiond"
Don't automatically spawn a session daemon.
.TP
.BR "\-\-sessiond\-path PATH"
Set session daemon full binary path.
.TP
.BR "\-\-list\-options"
Simple listing of lttng options.
.TP
.BR "\-\-list\-commands"
Simple listing of lttng commands.
.SH "COMMANDS"

.TP
\fBadd-context\fP
.nf
Add context to event(s) and/or channel(s).

A context is basically extra information appended to a channel. For instance,
you could ask the tracer to add the PID information for all events in a
channel. You can also add performance monitoring unit counters (perf PMU) using
the perf kernel API).

For example, this command will add the context information 'prio' and two perf
counters (hardware branch misses and cache misses), to all events in the trace
data output:

# lttng add-context \-k \-t prio \-t perf:branch-misses \-t perf:cache-misses

Please take a look at the help (\-h/\-\-help) for a detailed list of available
contexts.

If no channel is given (\-c), the context is added to all channels that were
already enabled. If the session has no channel, a default channel is created.
Otherwise the context will be added only to the given channel (\-c).

If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
file.
.fi

.B OPTIONS:

.nf
\-h, \-\-help
        Show summary of possible options and commands.
\-s, \-\-session NAME
        Apply on session name.
\-c, \-\-channel NAME
        Apply on channel name.
\-k, \-\-kernel
        Apply for the kernel tracer
\-u, \-\-userspace
        Apply for the user-space tracer
\-t, \-\-type TYPE
        Context type. You can repeat this option on the command line. Please
        use "lttng add-context \-h" to list all available types.
.fi

.IP

.IP "\fBcalibrate\fP"
.nf
Quantify LTTng overhead

The LTTng calibrate command can be used to find out the combined average
overhead of the LTTng tracer and the instrumentation mechanisms used. This
overhead can be calibrated in terms of time or using any of the PMU performance
counter available on the system.

For now, the only calibration implemented is that of the kernel function
instrumentation (kretprobes).

* Calibrate kernel function instrumentation

Let's use an example to show this calibration. We use an i7 processor with 4
general-purpose PMU registers. This information is available by issuing dmesg,
looking for "generic registers".

This sequence of commands will gather a trace executing a kretprobe hooked on
an empty function, gathering PMU counters LLC (Last Level Cache) misses
information (see lttng add-context \-\-help to see the list of available PMU
counters).

# lttng create calibrate-function
# lttng enable-event calibrate \-\-kernel \-\-function lttng_calibrate_kretprobe
# lttng add-context \-\-kernel \-t perf:LLC-load-misses \-t perf:LLC-store-misses \\
                  \-t perf:LLC-prefetch-misses
# lttng start
# for a in $(seq 1 10); do \\
        lttng calibrate \-\-kernel \-\-function;
  done
# lttng destroy
# babeltrace $(ls \-1drt ~/lttng-traces/calibrate-function-* | tail \-n 1)

The output from babeltrace can be saved to a text file and opened in a
spreadsheet (e.g. oocalc) to focus on the per-PMU counter delta between
consecutive "calibrate_entry" and "calibrate_return" events. Note that these
counters are per-CPU, so scheduling events would need to be present to account
for migration between CPU. Therefore, for calibration purposes, only events
staying on the same CPU must be considered.

The average result, for the i7, on 10 samples:

                          Average     Std.Dev.
perf_LLC_load_misses:       5.0       0.577
perf_LLC_store_misses:      1.6       0.516
perf_LLC_prefetch_misses:   9.0      14.742

As we can notice, the load and store misses are relatively stable across runs
(their standard deviation is relatively low) compared to the prefetch misses.
We can conclude from this information that LLC load and store misses can be
accounted for quite precisely, but prefetches within a function seems to behave
too erratically (not much causality link between the code executed and the CPU
prefetch activity) to be accounted for.
.fi

.B OPTIONS:

.nf
\-h, \-\-help
        Show summary of possible options and commands.
\-k, \-\-kernel
        Apply for the kernel tracer
\-u, \-\-userspace
        Apply for the user-space tracer
\-\-function
        Dynamic function entry/return probe (default)
.fi

.IP

.IP "\fBcreate\fP [NAME] [OPTIONS]
.nf
Create tracing session.

A tracing session contains channel(s) which contains event(s). It is domain
agnostic meaning that you can enable channels and events for either the
user-space tracer and/or the kernel tracer. It acts like a container
aggregating multiple tracing sources.

On creation, a \fB.lttngrc\fP file is created in your $HOME directory
containing the current session name. If NAME is omitted, a session name is
automatically created having this form: 'auto-yyyymmdd-hhmmss'.

If no \fB\-o, \-\-output\fP is specified, the traces will be written in
$HOME/lttng-traces.

The $HOME environment variable can be overridden by defining the environment
variable LTTNG_HOME. This is useful when the user running the commands has
a non-writeable home directory.
.fi

.B OPTIONS:

.nf
\-h, \-\-help
        Show summary of possible options and commands.
\-\-list-options
        Simple listing of options
\-o, \-\-output PATH
        Specify output path for traces

Using these options, each API call can be controlled individually. For
instance, \-C does not enable the consumer automatically. You'll need the \-e
option for that.

\-U, \-\-set-url=URL
        Set URL for the consumer output destination. It is persistent for the
        session lifetime. Redo the command to change it. This will set both
        data and control URL for network.
\-C, \-\-ctrl-url=URL
        Set control path URL. (Must use -D also)
\-D, \-\-data-url=URL
        Set data path URL. (Must use -C also)

.B URL FORMAT:

proto://[HOST|IP][:PORT1[:PORT2]][/TRACE_PATH]

Supported protocols are (proto):
> file://...
        Local filesystem full path.

> net://...
        This will use the default network transport layer which is TCP for both
        control (PORT1) and data port (PORT2). The default ports are
        respectively 5342 and 5343. Note that net[6]:// is not yet supported.

> tcp[6]://...
        Can only be used with -C and -D together

NOTE: IPv6 address MUST be enclosed in brackets '[]' (rfc2732)

.B EXAMPLES:

# lttng create -U net://192.168.1.42
Uses TCP and default ports for the given destination.

# lttng create -U net6://[fe80::f66d:4ff:fe53:d220]
Uses TCP, default ports and IPv6.

# lttng create s1 -U net://myhost.com:3229
Create session s1 and set its consumer to myhost.com on port 3229 for control.
.fi

.IP

.IP "\fBdestroy\fP [OPTIONS] [NAME]"
.nf
Teardown tracing session

Free memory on the session daemon and tracer side. It's gone!

If NAME is omitted, the session name is taken from the .lttngrc file.
.fi

.B OPTIONS:

.nf
\-h, \-\-help
        Show summary of possible options and commands.
\-a, \-\-all
        Destroy all sessions
\-\-list-options
        Simple listing of options
.fi

.IP

.IP "\fBenable-channel\fP NAME[,NAME2,...] [-k|-u] [OPTIONS]"
.nf
Enable tracing channel

To enable an event, you must enable both the event and the channel that
contains it.

If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
file.

It is important to note that if a certain type of buffers is used, the session
will be set with that type and all other subsequent channel needs to have the
same type.

Note that once the session has been started and enabled on the tracer side,
it's not possible anymore to enable a new channel for that session.
.fi

.B OPTIONS:

.nf
\-h, \-\-help
        Show this help
\-\-list-options
        Simple listing of options
\-s, \-\-session NAME
        Apply on session name
\-k, \-\-kernel
        Apply to the kernel tracer
\-u, \-\-userspace
        Apply to the user-space tracer

\-\-discard
        Discard event when subbuffers are full (default)
\-\-overwrite
        Flight recorder mode : overwrites events when subbuffers are full
\-\-subbuf-size SIZE
        Subbuffer size in bytes {+k,+M,+G}
        (default UST uid: 131072, UST pid: 4096, kernel: 262144, metadata: 4096)
        Rounded up to the next power of 2.

        The minimum subbuffer size, for each tracer, is the max value between
        the default above and the system page size. You can issue this command
        to get the current page size on your system: \fB$ getconf PAGE_SIZE\fP
\-\-num-subbuf NUM
        Number of subbuffers. (default UST uid: 4, UST pid: 4, kernel: 4, metadata: 2)
        Rounded up to the next power of 2.
\-\-switch-timer USEC
        Switch subbuffer timer interval in µsec.
        (default UST uid: 0, UST pid: 0, kernel: 0, metadata: 0)
\-\-read-timer USEC
        Read timer interval in µsec.
        (default UST uid: 0, UST pid: 0, kernel: 200000, metadata: 0)
\-\-output TYPE
        Channel output type. Possible values: mmap, splice
        (default UST uid: mmap, UST pid: mmap, kernel: splice, metadata: mmap)
\-\-buffers-uid
        Use per UID buffer (\-u only). Buffers are shared between applications
        that have the same UID.
\-\-buffers-pid
        Use per PID buffer (\-u only). Each application has its own buffers.
\-\-buffers-global
        Use shared buffer for the whole system (\-k only)
\-C, \-\-tracefile-size SIZE
        Maximum size of each tracefile within a stream (in bytes).
		0 means unlimited. (default: 0)
\-W, \-\-tracefile-count COUNT
        Used in conjunction with \-C option, this will limit the number
        of files created to the specified count. 0 means unlimited. (default: 0)

.B EXAMPLES:

$ lttng enable-channel -C 4096 -W 32 chan1
For each stream, the maximum size of each trace file will be 4096 bytes, and
there will be a maximum of 32 different files. The file count is appended after
the stream number as seen in the following example. The last trace file is
smaller than 4096 since it was not completely filled.

        ~/lttng-traces/[...]/chan1_0_0 (4096)
        ~/lttng-traces/[...]/chan1_0_1 (4096)
        ~/lttng-traces/[...]/chan1_0_2 (3245)
        ~/lttng-traces/[...]/chan1_1_0 (4096)
        ...

$ lttng enable-channel -C 4096
This will create trace files of 4096 bytes and will create new ones as long as
there is data available.
.fi

.IP

.IP "\fBenable-event\fP NAME[,NAME2,...] [-k|-u] [OPTIONS]"
.nf
Enable tracing event

A tracing event is always assigned to a channel. If \fB\-c, \-\-channel\fP is
omitted, a default channel named '\fBchannel0\fP' is created and the event is
added to it. For the user-space tracer, using \fB\-a, \-\-all\fP is the same as
using the wildcard "*".

If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
file.
.fi

.B OPTIONS:

.nf
\-h, \-\-help
        Show summary of possible options and commands.
\-\-list-options
        Simple listing of options
\-s, \-\-session NAME
        Apply on session name
\-c, \-\-channel NAME
        Apply on channel name
\-a, \-\-all
        Enable all tracepoints and syscalls. This actually enable a single
        wildcard event "*".
\-k, \-\-kernel
        Apply for the kernel tracer
\-u, \-\-userspace
        Apply for the user-space tracer

\-\-tracepoint
        Tracepoint event (default)
        - userspace tracer supports wildcards at end of string. Don't forget to
        quote to deal with bash expansion.
        e.g.:
        "*"
        "app_component:na*"
\-\-loglevel NAME
        Tracepoint loglevel range from 0 to loglevel. Listed in the help (\-h).
\-\-loglevel-only NAME
        Tracepoint loglevel (only this loglevel).

        The loglevel or loglevel-only options should be combined with a
        tracepoint name or tracepoint wildcard.
\-\-probe [addr | symbol | symbol+offset]
        Dynamic probe. Addr and offset can be octal (0NNN...), decimal (NNN...)
        or hexadecimal (0xNNN...)
\-\-function [addr | symbol | symbol+offset]
        Dynamic function entry/return probe. Addr and offset can be octal
        (0NNN...), decimal (NNN...) or hexadecimal (0xNNN...)
\-\-syscall
        System call event. Enabling syscalls tracing (kernel tracer), you will
        not be able to disable them with disable-event. This is a known
        limitation. You can disable the entire channel to do the trick.

\-\-filter 'expression'
        Set a filter on a newly enabled event. Filter expression on event
	fields and context. Event recording depends on evaluation. Only
	specify on first activation of a given event within a session.
	Filter only allowed when enabling events within a session before
	tracing is started. If the filter fails to link with the event
	within the traced domain, the event will be discarded.
	Currently, filter is only implemented for the user-space tracer.

        Expression examples:

        'intfield > 500 && intfield < 503'
        '(stringfield == "test" || intfield != 10) && intfield > 33'
        'doublefield > 1.1 && intfield < 5.3'

        Wildcards are allowed at the end of strings:
        'seqfield1 == "te*"'
        In string literals, the escape character is a '\\'. Use '\\*' for
        the '*' character, and '\\\\' for the '\\' character. Wildcard
	match any sequence of characters, including an empty sub-string
	(match 0 or more characters).

	Context information can be used for filtering. The examples
	below show usage of context filtering on process name (with a
	wildcard), process ID range, and unique thread ID for filtering.
	The process and thread ID of running applications can be found
	under columns "PID" and "LWP" of the "ps -eLf" command.

	'$ctx.procname == "demo*"'
	'$ctx.vpid >= 4433 && $ctx.vpid < 4455'
	'$ctx.vtid == 1234'
.fi

.IP "\fBdisable-channel\fP NAME[,NAME2,...] [\-k|\-u] [OPTIONS]"
.nf
Disable tracing channel

Disabling a channel makes all event(s) in that channel to stop tracing. You can
enable it back by calling \fBlttng enable-channel NAME\fP again.

If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
file.
.fi

.B OPTIONS:

.nf
\-h, \-\-help
        Show summary of possible options and commands.
\-\-list-options
        Simple listing of options
\-s, \-\-session NAME
        Apply on session name
\-k, \-\-kernel
        Apply for the kernel tracer
\-u, \-\-userspace
        Apply for the user-space tracer
.fi

.IP "\fBdisable-event\fP NAME[,NAME2,...] [\-k|\-u] [OPTIONS]"
.nf
Disable tracing event

The event, once disabled, can be re-enabled by calling \fBlttng enable-event
NAME\fP again.

If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
file.
.fi

.B OPTIONS:

.nf
\-h, \-\-help
        Show summary of possible options and commands.
\-\-list-options
        Simple listing of options
\-s, \-\-session NAME
        Apply on session name
\-a, \-\-all-events
        Disable all events. This does NOT disable "*" but rather
        every known events of the session.
\-k, \-\-kernel
        Apply for the kernel tracer
\-u, \-\-userspace
        Apply for the user-space tracer
.fi

.IP "\fBlist\fP [\-k|\-u] [SESSION [SESSION_OPTIONS]]"
.nf
List tracing session information.

With no arguments, it will list available tracing session(s).

With the session name, it will display the details of the session including
the trace file path, the associated channels and their state (activated
and deactivated), the activated events and more.

With \-k alone, it will list all available kernel events (except the system
calls events).
With \-u alone, it will list all available user-space events from registered
applications. Here is an example of 'lttng list \-u':

PID: 7448 - Name: /tmp/lttng-ust/tests/hello/.libs/lt-hello
      ust_tests_hello:tptest_sighandler (type: tracepoint)
      ust_tests_hello:tptest (type: tracepoint)

You can now enable any event listed by using the name :
\fBust_tests_hello:tptest\fP.
.fi

.B OPTIONS:

.nf
\-h, \-\-help
        Show summary of possible options and commands.
\-\-list-options
        Simple listing of options
\-k, \-\-kernel
        Select kernel domain
\-u, \-\-userspace
        Select user-space domain.

.B SESSION OPTIONS:

\-c, \-\-channel NAME
        List details of a channel
\-d, \-\-domain
        List available domain(s)
.fi

.IP "\fBset-session\fP NAME"
.nf
Set current session name

Will change the session name in the .lttngrc file.
.fi

.B OPTIONS:

.nf
\-h, \-\-help
        Show summary of possible options and commands.
\-\-list-options
        Simple listing of options
.fi

.IP

.IP "\fBstart\fP [NAME] [OPTIONS]"
.nf
Start tracing

It will start tracing for all tracers for a specific tracing session.

If NAME is omitted, the session name is taken from the .lttngrc file.
.fi

.B OPTIONS:

.nf
\-h, \-\-help
        Show summary of possible options and commands.
\-\-list-options
        Simple listing of options
.fi

.IP

.IP "\fBstop\fP [NAME] [OPTIONS]"
.nf
Stop tracing

It will stop tracing for all tracers for a specific tracing session. Before
returning, the command checks for data availability meaning that it will wait
until the trace is readable for the session. Use \-\-no-wait to avoid this
behavior.

If NAME is omitted, the session name is taken from the .lttngrc file.
.fi

.B OPTIONS:

.nf
\-h, \-\-help
        Show summary of possible options and commands.
\-\-list-options
        Simple listing of options
\-\-no-wait
        Don't wait for data availability.
.fi

.IP

.IP "\fBversion\fP"
.nf
Show version information
.fi

.B OPTIONS:

.nf
\-h, \-\-help
        Show summary of possible options and commands.
\-\-list-options
        Simple listing of options
.fi

.IP

.IP "\fBview\fP [SESSION_NAME] [OPTIONS]"
.nf
View traces of a tracing session

By default, the babeltrace viewer will be used for text viewing.

If SESSION_NAME is omitted, the session name is taken from the .lttngrc file.

.fi

.B OPTIONS:

.nf
\-h, \-\-help
        Show this help
\-\-list-options
        Simple listing of options
\-t, \-\-trace-path PATH
        Trace directory path for the viewer
\-e, \-\-viewer CMD
        Specify viewer and/or options to use
        This will completely override the default viewers so
        please make sure to specify the full command. The trace
        directory path of the session will be appended at the end
        to the arguments
.fi

.SH "EXIT VALUES"
On success 0 is returned and a positive value on error. Value of 1 means a command
error, 2 an undefined command, 3 a fatal error and 4 a command warning meaning that
something went wrong during the command.

Any other value above 10, please refer to
.BR <lttng/lttng-error.h>
for a detailed list or use lttng_strerror() to get a human readable string of
the error code.

.PP
.SH "ENVIRONMENT VARIABLES"

.PP
Note that all command line options override environment variables.
.PP

.PP
.IP "LTTNG_SESSIOND_PATH"
Allows one to specify the full session daemon binary path to lttng command line
tool. You can also use \-\-sessiond-path option having the same effect.
.SH "SEE ALSO"
.BR babeltrace(1),
.BR lttng-ust(3),
.BR lttng-sessiond(8),
.BR lttng-relayd(8),
.BR lttng-health-check(3)
.SH "BUGS"

If you encounter any issues or usability problem, please report it on our
mailing list <lttng-dev@lists.lttng.org> to help improve this project or
at https://bugs.lttng.org which is a bugtracker.
.SH "CREDITS"

.PP
lttng is distributed under the GNU General Public License version 2. See the file
COPYING for details.
.PP
A Web site is available at http://lttng.org for more information on the LTTng
project.
.PP
You can also find our git tree at http://git.lttng.org.
.PP
Mailing lists for support and development: <lttng-dev@lists.lttng.org>.
.PP
You can find us on IRC server irc.oftc.net (OFTC) in #lttng.
.PP
.SH "THANKS"

.PP
Thanks to Yannick Brosseau without whom this project would never have been so
lean and mean! Also thanks to the Ericsson teams working on tracing which
helped us greatly with detailed bug reports and unusual test cases.

Thanks to our beloved packager Alexandre Montplaisir-Goncalves (Ubuntu and PPA
maintainer) and Jon Bernard for our Debian packages.

Special thanks to Michel Dagenais and the DORSAL laboratory at Polytechnique de
Montreal for the LTTng journey.
.PP
.SH "AUTHORS"

.PP
lttng-tools was originally written by Mathieu Desnoyers, Julien Desfossez and
David Goulet. More people have since contributed to it. It is currently
maintained by David Goulet <dgoulet@efficios.com>.
.PP