1 The LTTng Documentation
2 =======================
3 Philippe Proulx <pproulx@efficios.com>
7 include::../common/copyright.txt[]
10 include::../common/welcome.txt[]
13 include::../common/audience.txt[]
17 === What's in this documentation?
19 The LTTng Documentation is divided into the following sections:
21 * **<<nuts-and-bolts,Nuts and bolts>>** explains the
22 rudiments of software tracing and the rationale behind the
25 You can skip this section if you’re familiar with software tracing and
26 with the LTTng project.
28 * **<<installing-lttng,Installation>>** describes the steps to
29 install the LTTng packages on common Linux distributions and from
32 You can skip this section if you already properly installed LTTng on
35 * **<<getting-started,Quick start>>** is a concise guide to
36 getting started quickly with LTTng kernel and user space tracing.
38 We recommend this section if you're new to LTTng or to software tracing
41 You can skip this section if you're not new to LTTng.
43 * **<<core-concepts,Core concepts>>** explains the concepts at
46 It's a good idea to become familiar with the core concepts
47 before attempting to use the toolkit.
49 * **<<plumbing,Components of LTTng>>** describes the various components
50 of the LTTng machinery, like the daemons, the libraries, and the
51 command-line interface.
52 * **<<instrumenting,Instrumentation>>** shows different ways to
53 instrument user applications and the Linux kernel.
55 Instrumenting source code is essential to provide a meaningful
58 You can skip this section if you do not have a programming background.
60 * **<<controlling-tracing,Tracing control>>** is divided into topics
61 which demonstrate how to use the vast array of features that
62 LTTng{nbsp}{revision} offers.
63 * **<<reference,Reference>>** contains reference tables.
64 * **<<glossary,Glossary>>** is a specialized dictionary of terms related
65 to LTTng or to the field of software tracing.
68 include::../common/convention.txt[]
71 include::../common/acknowledgements.txt[]
75 == What's new in LTTng{nbsp}{revision}?
77 LTTng{nbsp}{revision} bears the name _Lafontaine_. This modern
78 https://en.wikipedia.org/wiki/saison[saison] from the
79 https://oshlag.com/[Oshlag] microbrewery is a refreshing--zesty--rice
80 beer with hints of fruit and spices. Some even say it makes for a great
81 https://en.wikipedia.org/wiki/Somaek[Somaek] when mixed with
82 Chamisul Soju, not that we've tried!
84 New features and changes in LTTng{nbsp}{revision}:
86 * Just like you can typically perform
87 https://en.wikipedia.org/wiki/Log_rotation[log rotation], you can
88 now <<session-rotation,_rotate_ a tracing session>>, that
89 is, according to man:lttng-rotate(1), archive the current trace
90 chunk (all the tracing session's trace data since the last rotation
91 or since its inception) so that LTTng does not manage it anymore.
93 Once LTTng archives a trace chunk, you are free to read it, modify it,
94 move it, or remove it.
96 You can rotate a tracing session immediately or set a rotation schedule
97 to automate rotations.
99 * When you <<enabling-disabling-events,create an event rule>>, the
100 filter expression syntax now supports the following new operators:
104 ** `<<` (bitwise left shift)
105 ** `>>` (bitwise right shift)
111 The syntax also supports array indexing with the usual square brackets:
114 regs[3][1] & 0xff7 == 0x240
117 There are peculiarities for both the new operators and the array
118 indexing brackets, like a custom precedence table and implicit casting.
119 See man:lttng-enable-event(1) to get all the details about the filter
122 * You can now dynamically instrument any application's or library's
123 function entry by symbol name thanks to the new
124 opt:lttng-enable-event(1):--userspace-probe option of
125 the `lttng enable-event` command:
129 $ lttng enable-event --kernel \
130 --userspace-probe=/usr/lib/libc.so.6:malloc libc_malloc
133 The option also supports tracing existing
134 https://www.sourceware.org/systemtap/wiki/AddingUserSpaceProbingToApps[SystemTap
135 Statically Defined Tracing] (USDT) probe (DTrace-style marker). For
136 example, given the following probe:
140 DTRACE_PROBE2("server", "accept-request", request_id, ip_addr);
143 You can trace this probe with:
146 $ lttng enable-event --kernel \
147 --userspace-probe=sdt:/path/to/server:server:accept-request \
148 server_accept_request
151 This feature makes use of Linux's
152 https://www.kernel.org/doc/Documentation/trace/uprobetracer.txt[uprobe]
153 mechanism, therefore you must use the `--userspace-probe`
154 instrumentation option with the opt:lttng-enable-event(1):--kernel
157 NOTE: As of LTTng{nbsp}{revision}, LTTng does not record function
158 parameters with the opt:lttng-enable-event(1):--userspace-probe option.
160 * Two new <<adding-context,context>> fields are available for Linux
161 kernel <<channel,channels>>:
164 ** `callstack-kernel`
168 Thanks to those, you can record the Linux kernel and user call stacks
169 when a kernel event occurs. For example:
173 $ lttng enable-event --kernel --syscall open
174 $ lttng add-context --kernel --type=callstack-kernel --type=callstack-user
177 When an man:open(2) system call occurs, LTTng attaches the kernel and
178 user call stacks to the recorded event.
180 NOTE: LTTng cannot always sample the user space call stack reliably.
181 For instance, LTTng cannot sample the call stack of user applications
182 and libraries compiled with the
183 https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html[`-fomit-frame-pointer`]
184 option. In such a case, the tracing is not affected, but the sampled
185 user space call stack may only contain the user call stack's topmost
188 * User applications and libraries instrumented with
189 <<lttng-ust,LTTng-UST>> can now safely unload (man:dlclose(3)) a
191 <<building-tracepoint-providers-and-user-application,tracepoint
194 * The <<lttng-relayd,relay daemon>> is more efficient and presents fewer
195 connectivity issues, especially when a large number of targets send
196 trace data to a given relay daemon.
198 * LTTng-UST uses https://github.com/numactl/numactl[libnuma]
199 when available to allocate <<def-sub-buffer,sub-buffers>>, making them
201 https://en.wikipedia.org/wiki/Non-uniform_memory_access[NUMA] node.
203 This change makes the tracer more efficient on NUMA systems.
205 * The <<proc-lttng-logger-abi,LTTng logger>> kernel module now also
206 creates a ``misc'' device named `lttng-logger`, which udev will
207 make accessible as the path:{/dev/lttng-logger} special file.
209 The `lttng-logger` device shares the `/proc/lttng-logger` file's ABI,
210 but it works from within containers when the path is made accessible to
217 What is LTTng? As its name suggests, the _Linux Trace Toolkit: next
218 generation_ is a modern toolkit for tracing Linux systems and
219 applications. So your first question might be:
226 As the history of software engineering progressed and led to what
227 we now take for granted--complex, numerous and
228 interdependent software applications running in parallel on
229 sophisticated operating systems like Linux--the authors of such
230 components, software developers, began feeling a natural
231 urge to have tools that would ensure the robustness and good performance
232 of their masterpieces.
234 One major achievement in this field is, inarguably, the
235 https://www.gnu.org/software/gdb/[GNU debugger (GDB)],
236 an essential tool for developers to find and fix bugs. But even the best
237 debugger won't help make your software run faster, and nowadays, faster
238 software means either more work done by the same hardware, or cheaper
239 hardware for the same work.
241 A _profiler_ is often the tool of choice to identify performance
242 bottlenecks. Profiling is suitable to identify _where_ performance is
243 lost in a given software. The profiler outputs a profile, a statistical
244 summary of observed events, which you may use to discover which
245 functions took the most time to execute. However, a profiler won't
246 report _why_ some identified functions are the bottleneck. Bottlenecks
247 might only occur when specific conditions are met, conditions that are
248 sometimes impossible to capture by a statistical profiler, or impossible
249 to reproduce with an application altered by the overhead of an
250 event-based profiler. For a thorough investigation of software
251 performance issues, a history of execution is essential, with the
252 recorded values of variables and context fields you choose, and
253 with as little influence as possible on the instrumented software. This
254 is where tracing comes in handy.
256 _Tracing_ is a technique used to understand what goes on in a running
257 software system. The software used for tracing is called a _tracer_,
258 which is conceptually similar to a tape recorder. When recording,
259 specific instrumentation points placed in the software source code
260 generate events that are saved on a giant tape: a _trace_ file. You
261 can trace user applications and the operating system at the same time,
262 opening the possibility of resolving a wide range of problems that would
263 otherwise be extremely challenging.
265 Tracing is often compared to _logging_. However, tracers and loggers are
266 two different tools, serving two different purposes. Tracers are
267 designed to record much lower-level events that occur much more
268 frequently than log messages, often in the range of thousands per
269 second, with very little execution overhead. Logging is more appropriate
270 for a very high-level analysis of less frequent events: user accesses,
271 exceptional conditions (errors and warnings, for example), database
272 transactions, instant messaging communications, and such. Simply put,
273 logging is one of the many use cases that can be satisfied with tracing.
275 The list of recorded events inside a trace file can be read manually
276 like a log file for the maximum level of detail, but it is generally
277 much more interesting to perform application-specific analyses to
278 produce reduced statistics and graphs that are useful to resolve a
279 given problem. Trace viewers and analyzers are specialized tools
282 In the end, this is what LTTng is: a powerful, open source set of
283 tools to trace the Linux kernel and user applications at the same time.
284 LTTng is composed of several components actively maintained and
285 developed by its link:/community/#where[community].
288 [[lttng-alternatives]]
289 === Alternatives to noch:{LTTng}
291 Excluding proprietary solutions, a few competing software tracers
294 * https://github.com/dtrace4linux/linux[dtrace4linux] is a port of
295 Sun Microsystems's DTrace to Linux. The cmd:dtrace tool interprets
296 user scripts and is responsible for loading code into the
297 Linux kernel for further execution and collecting the outputted data.
298 * https://en.wikipedia.org/wiki/Berkeley_Packet_Filter[eBPF] is a
299 subsystem in the Linux kernel in which a virtual machine can execute
300 programs passed from the user space to the kernel. You can attach
301 such programs to tracepoints and kprobes thanks to a system call, and
302 they can output data to the user space when executed thanks to
303 different mechanisms (pipe, VM register values, and eBPF maps, to name
305 * https://www.kernel.org/doc/Documentation/trace/ftrace.txt[ftrace]
306 is the de facto function tracer of the Linux kernel. Its user
307 interface is a set of special files in sysfs.
308 * https://perf.wiki.kernel.org/[perf] is
309 a performance analysis tool for Linux which supports hardware
310 performance counters, tracepoints, as well as other counters and
311 types of probes. perf's controlling utility is the cmd:perf command
313 * http://linux.die.net/man/1/strace[strace]
314 is a command-line utility which records system calls made by a
315 user process, as well as signal deliveries and changes of process
316 state. strace makes use of https://en.wikipedia.org/wiki/Ptrace[ptrace]
317 to fulfill its function.
318 * http://www.sysdig.org/[sysdig], like SystemTap, uses scripts to
319 analyze Linux kernel events. You write scripts, or _chisels_ in
320 sysdig's jargon, in Lua and sysdig executes them while it traces the
321 system or afterwards. sysdig's interface is the cmd:sysdig
322 command-line tool as well as the curses-based cmd:csysdig tool.
323 * https://sourceware.org/systemtap/[SystemTap] is a Linux kernel and
324 user space tracer which uses custom user scripts to produce plain text
325 traces. SystemTap converts the scripts to the C language, and then
326 compiles them as Linux kernel modules which are loaded to produce
327 trace data. SystemTap's primary user interface is the cmd:stap
330 The main distinctive features of LTTng is that it produces correlated
331 kernel and user space traces, as well as doing so with the lowest
332 overhead amongst other solutions. It produces trace files in the
333 http://diamon.org/ctf[CTF] format, a file format optimized
334 for the production and analyses of multi-gigabyte data.
336 LTTng is the result of more than 10{nbsp}years of active open source
337 development by a community of passionate developers.
338 LTTng{nbsp}{revision} is currently available on major desktop and server
341 The main interface for tracing control is a single command-line tool
342 named cmd:lttng. The latter can create several tracing sessions, enable
343 and disable events on the fly, filter events efficiently with custom
344 user expressions, start and stop tracing, and much more. LTTng can
345 record the traces on the file system or send them over the network, and
346 keep them totally or partially. You can view the traces once tracing
347 becomes inactive or in real-time.
349 <<installing-lttng,Install LTTng now>> and
350 <<getting-started,start tracing>>!
356 **LTTng** is a set of software <<plumbing,components>> which interact to
357 <<instrumenting,instrument>> the Linux kernel and user applications, and
358 to <<controlling-tracing,control tracing>> (start and stop
359 tracing, enable and disable event rules, and the rest). Those
360 components are bundled into the following packages:
363 Libraries and command-line interface to control tracing.
366 Linux kernel modules to instrument and trace the kernel.
369 Libraries and Java/Python packages to instrument and trace user
372 Most distributions mark the LTTng-modules and LTTng-UST packages as
373 optional when installing LTTng-tools (which is always required). In the
374 following sections, we always provide the steps to install all three,
377 * You only need to install LTTng-modules if you intend to trace the
379 * You only need to install LTTng-UST if you intend to trace user
383 .Availability of LTTng{nbsp}{revision} for major Linux distributions as of 30 March 2020.
385 |Distribution |Available in releases
387 |https://www.ubuntu.com/[Ubuntu]
388 |Ubuntu{nbsp}16.04 _Xenial Xerus_ and Ubuntu{nbsp}18.04 _Bionic Beaver_:
389 <<ubuntu-ppa,use the LTTng Stable{nbsp}{revision} PPA>>.
391 |https://www.debian.org/[Debian]
392 |<<debian,Debian "bullseye" (testing)>>.
394 |https://www.archlinux.org/[Arch Linux]
395 |<<arch-linux,_Community_ repository and AUR>>.
397 |https://getfedora.org/[Fedora]
398 |xref:fedora[Fedora{nbsp}32 and Fedora{nbsp}33].
400 |https://www.redhat.com/[RHEL] and https://www.suse.com/[SLES]
401 |See http://packages.efficios.com/[EfficiOS Enterprise Packages].
403 |https://buildroot.org/[Buildroot]
404 |xref:buildroot[Buildroot{nbsp}2019.11 and Buildroot{nbsp}2020.02].
409 === Ubuntu: noch:{LTTng} Stable {revision} PPA
411 The https://launchpad.net/~lttng/+archive/ubuntu/stable-{revision}[LTTng
412 Stable{nbsp}{revision} PPA] offers the latest stable
413 LTTng{nbsp}{revision} packages for Ubuntu{nbsp}16.04 _Xenial Xerus_ and
414 Ubuntu{nbsp}18.04 _Bionic Beaver_.
416 To install LTTng{nbsp}{revision} from the LTTng Stable{nbsp}{revision}
419 . Add the LTTng Stable{nbsp}{revision} PPA repository and update the
425 # apt-add-repository ppa:lttng/stable-2.10
430 . Install the main LTTng{nbsp}{revision} packages:
435 # apt-get install lttng-tools
436 # apt-get install lttng-modules-dkms
437 # apt-get install liblttng-ust-dev
441 . **If you need to instrument and trace
442 <<java-application,Java applications>>**, install the LTTng-UST
448 # apt-get install liblttng-ust-agent-java
452 . **If you need to instrument and trace
453 <<python-application,Python{nbsp}3 applications>>**, install the
454 LTTng-UST Python agent:
459 # apt-get install python3-lttngust
467 To install LTTng{nbsp}{revision} on Debian "bullseye" (testing):
469 . Install the main LTTng{nbsp}{revision} packages:
474 # apt-get install lttng-modules-dkms
475 # apt-get install liblttng-ust-dev
476 # apt-get install lttng-tools
480 . **If you need to instrument and trace <<java-application,Java
481 applications>>**, install the LTTng-UST Java agent:
486 # apt-get install liblttng-ust-agent-java
490 . **If you need to instrument and trace <<python-application,Python
491 applications>>**, install the LTTng-UST Python agent:
496 # apt-get install python3-lttngust
504 LTTng-UST{nbsp}{revision} is available in Arch Linux's _community_
505 repository, while LTTng-tools{nbsp}{revision} and
506 LTTng-modules{nbsp}{revision} are available in the
507 https://aur.archlinux.org/[AUR].
509 To install LTTng{nbsp}{revision} on Arch Linux, using
510 https://github.com/Jguer/yay[yay] for the AUR packages:
512 . Install the main LTTng{nbsp}{revision} packages:
517 # pacman -Sy lttng-ust
518 $ yay -Sy lttng-tools
519 $ yay -Sy lttng-modules
523 . **If you need to instrument and trace <<python-application,Python
524 applications>>**, install the LTTng-UST Python agent:
529 # pacman -Sy python-lttngust
530 # pacman -Sy python2-lttngust
538 To install LTTng{nbsp}{revision} on Fedora{nbsp}32 and Fedora{nbsp}33:
540 . Install the LTTng-tools{nbsp}{revision} and LTTng-UST{nbsp}{revision}
546 # yum install lttng-tools
547 # yum install lttng-ust
551 . Download, build, and install the latest LTTng-modules{nbsp}{revision}:
557 wget http://lttng.org/files/lttng-modules/lttng-modules-latest-2.11.tar.bz2 &&
558 tar -xf lttng-modules-latest-2.11.tar.bz2 &&
559 cd lttng-modules-2.11.* &&
561 sudo make modules_install &&
567 .Java and Python application instrumentation and tracing
569 If you need to instrument and trace <<java-application,Java
570 applications>> on Fedora, you need to build and install
571 LTTng-UST{nbsp}{revision} <<building-from-source,from source>> and pass
572 the `--enable-java-agent-jul`, `--enable-java-agent-log4j`, or
573 `--enable-java-agent-all` options to the `configure` script, depending
574 on which Java logging framework you use.
576 If you need to instrument and trace <<python-application,Python
577 applications>> on Fedora, you need to build and install
578 LTTng-UST{nbsp}{revision} from source and pass the
579 `--enable-python-agent` option to the `configure` script.
586 To install LTTng{nbsp}{revision} on Buildroot{nbsp}2019.11 and
587 Buildroot{nbsp}2020.02:
589 . Launch the Buildroot configuration tool:
598 . In **Kernel**, check **Linux kernel**.
599 . In **Toolchain**, check **Enable WCHAR support**.
600 . In **Target packages**{nbsp}→ **Debugging, profiling and benchmark**,
601 check **lttng-modules** and **lttng-tools**.
602 . In **Target packages**{nbsp}→ **Libraries**{nbsp}→
603 **Other**, check **lttng-libust**.
606 [[building-from-source]]
607 === Build from source
609 To build and install LTTng{nbsp}{revision} from source:
611 . Using your distribution's package manager, or from source, install
612 the following dependencies of LTTng-tools and LTTng-UST:
615 * https://sourceforge.net/projects/libuuid/[libuuid]
616 * http://directory.fsf.org/wiki/Popt[popt]
617 * http://liburcu.org/[Userspace RCU]
618 * http://www.xmlsoft.org/[libxml2]
619 * **Optional**: https://github.com/numactl/numactl[numactl]
622 . Download, build, and install the latest LTTng-modules{nbsp}{revision}:
628 wget http://lttng.org/files/lttng-modules/lttng-modules-latest-2.11.tar.bz2 &&
629 tar -xf lttng-modules-latest-2.11.tar.bz2 &&
630 cd lttng-modules-2.11.* &&
632 sudo make modules_install &&
637 . Download, build, and install the latest LTTng-UST{nbsp}{revision}:
643 wget http://lttng.org/files/lttng-ust/lttng-ust-latest-2.11.tar.bz2 &&
644 tar -xf lttng-ust-latest-2.11.tar.bz2 &&
645 cd lttng-ust-2.11.* &&
653 Add `--disable-numa` to `./configure` if you don't have
654 https://github.com/numactl/numactl[numactl].
658 .Java and Python application tracing
660 If you need to instrument and trace <<java-application,Java
661 applications>>, pass the `--enable-java-agent-jul`,
662 `--enable-java-agent-log4j`, or `--enable-java-agent-all` options to the
663 `configure` script, depending on which Java logging framework you use.
665 If you need to instrument and trace <<python-application,Python
666 applications>>, pass the `--enable-python-agent` option to the
667 `configure` script. You can set the `PYTHON` environment variable to the
668 path to the Python interpreter for which to install the LTTng-UST Python
676 By default, LTTng-UST libraries are installed to
677 dir:{/usr/local/lib}, which is the de facto directory in which to
678 keep self-compiled and third-party libraries.
680 When <<building-tracepoint-providers-and-user-application,linking an
681 instrumented user application with `liblttng-ust`>>:
683 * Append `/usr/local/lib` to the env:LD_LIBRARY_PATH environment
685 * Pass the `-L/usr/local/lib` and `-Wl,-rpath,/usr/local/lib` options to
686 man:gcc(1), man:g++(1), or man:clang(1).
690 . Download, build, and install the latest LTTng-tools{nbsp}{revision}:
696 wget http://lttng.org/files/lttng-tools/lttng-tools-latest-2.11.tar.bz2 &&
697 tar -xf lttng-tools-latest-2.11.tar.bz2 &&
698 cd lttng-tools-2.11.* &&
706 TIP: The https://github.com/eepp/vlttng[vlttng tool] can do all the
707 previous steps automatically for a given version of LTTng and confine
708 the installed files in a specific directory. This can be useful to test
709 LTTng without installing it on your system.
715 This is a short guide to get started quickly with LTTng kernel and user
718 Before you follow this guide, make sure to <<installing-lttng,install>>
721 This tutorial walks you through the steps to:
723 . <<tracing-the-linux-kernel,Trace the Linux kernel>>.
724 . <<tracing-your-own-user-application,Trace a user application>> written
726 . <<viewing-and-analyzing-your-traces,View and analyze the
730 [[tracing-the-linux-kernel]]
731 === Trace the Linux kernel
733 The following command lines start with the `#` prompt because you need
734 root privileges to trace the Linux kernel. You can also trace the kernel
735 as a regular user if your Unix user is a member of the
736 <<tracing-group,tracing group>>.
738 . Create a <<tracing-session,tracing session>> which writes its traces
739 to dir:{/tmp/my-kernel-trace}:
744 # lttng create my-kernel-session --output=/tmp/my-kernel-trace
748 . List the available kernel tracepoints and system calls:
753 # lttng list --kernel
754 # lttng list --kernel --syscall
758 . Create <<event,event rules>> which match the desired instrumentation
759 point names, for example the `sched_switch` and `sched_process_fork`
760 tracepoints, and the man:open(2) and man:close(2) system calls:
765 # lttng enable-event --kernel sched_switch,sched_process_fork
766 # lttng enable-event --kernel --syscall open,close
770 You can also create an event rule which matches _all_ the Linux kernel
771 tracepoints (this will generate a lot of data when tracing):
776 # lttng enable-event --kernel --all
780 . <<basic-tracing-session-control,Start tracing>>:
789 . Do some operation on your system for a few seconds. For example,
790 load a website, or list the files of a directory.
791 . <<creating-destroying-tracing-sessions,Destroy>> the current
801 The man:lttng-destroy(1) command does not destroy the trace data; it
802 only destroys the state of the tracing session.
804 The man:lttng-destroy(1) command also runs the man:lttng-stop(1) command
805 implicitly (see <<basic-tracing-session-control,Start and stop a tracing
806 session>>). You need to stop tracing to make LTTng flush the remaining
807 trace data and make the trace readable.
809 . For the sake of this example, make the recorded trace accessible to
815 # chown -R $(whoami) /tmp/my-kernel-trace
819 See <<viewing-and-analyzing-your-traces,View and analyze the
820 recorded events>> to view the recorded events.
823 [[tracing-your-own-user-application]]
824 === Trace a user application
826 This section steps you through a simple example to trace a
827 _Hello world_ program written in C.
829 To create the traceable user application:
831 . Create the tracepoint provider header file, which defines the
832 tracepoints and the events they can generate:
838 #undef TRACEPOINT_PROVIDER
839 #define TRACEPOINT_PROVIDER hello_world
841 #undef TRACEPOINT_INCLUDE
842 #define TRACEPOINT_INCLUDE "./hello-tp.h"
844 #if !defined(_HELLO_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
847 #include <lttng/tracepoint.h>
857 ctf_string(my_string_field, my_string_arg)
858 ctf_integer(int, my_integer_field, my_integer_arg)
862 #endif /* _HELLO_TP_H */
864 #include <lttng/tracepoint-event.h>
868 . Create the tracepoint provider package source file:
874 #define TRACEPOINT_CREATE_PROBES
875 #define TRACEPOINT_DEFINE
877 #include "hello-tp.h"
881 . Build the tracepoint provider package:
886 $ gcc -c -I. hello-tp.c
890 . Create the _Hello World_ application source file:
897 #include "hello-tp.h"
899 int main(int argc, char *argv[])
903 puts("Hello, World!\nPress Enter to continue...");
906 * The following getchar() call is only placed here for the purpose
907 * of this demonstration, to pause the application in order for
908 * you to have time to list its tracepoints. It is not
914 * A tracepoint() call.
916 * Arguments, as defined in hello-tp.h:
918 * 1. Tracepoint provider name (required)
919 * 2. Tracepoint name (required)
920 * 3. my_integer_arg (first user-defined argument)
921 * 4. my_string_arg (second user-defined argument)
923 * Notice the tracepoint provider and tracepoint names are
924 * NOT strings: they are in fact parts of variables that the
925 * macros in hello-tp.h create.
927 tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");
929 for (x = 0; x < argc; ++x) {
930 tracepoint(hello_world, my_first_tracepoint, x, argv[x]);
933 puts("Quitting now!");
934 tracepoint(hello_world, my_first_tracepoint, x * x, "x^2");
941 . Build the application:
950 . Link the application with the tracepoint provider package,
951 `liblttng-ust`, and `libdl`:
956 $ gcc -o hello hello.o hello-tp.o -llttng-ust -ldl
960 Here's the whole build process:
963 .User space tracing tutorial's build steps.
964 image::ust-flow.png[]
966 To trace the user application:
968 . Run the application with a few arguments:
973 $ ./hello world and beyond
982 Press Enter to continue...
986 . Start an LTTng <<lttng-sessiond,session daemon>>:
991 $ lttng-sessiond --daemonize
995 Note that a session daemon might already be running, for example as
996 a service that the distribution's service manager started.
998 . List the available user space tracepoints:
1003 $ lttng list --userspace
1007 You see the `hello_world:my_first_tracepoint` tracepoint listed
1008 under the `./hello` process.
1010 . Create a <<tracing-session,tracing session>>:
1015 $ lttng create my-user-space-session
1019 . Create an <<event,event rule>> which matches the
1020 `hello_world:my_first_tracepoint` event name:
1025 $ lttng enable-event --userspace hello_world:my_first_tracepoint
1029 . <<basic-tracing-session-control,Start tracing>>:
1038 . Go back to the running `hello` application and press Enter. The
1039 program executes all `tracepoint()` instrumentation points and exits.
1040 . <<creating-destroying-tracing-sessions,Destroy>> the current
1050 The man:lttng-destroy(1) command does not destroy the trace data; it
1051 only destroys the state of the tracing session.
1053 The man:lttng-destroy(1) command also runs the man:lttng-stop(1) command
1054 implicitly (see <<basic-tracing-session-control,Start and stop a tracing
1055 session>>). You need to stop tracing to make LTTng flush the remaining
1056 trace data and make the trace readable.
1058 By default, LTTng saves the traces in
1059 +$LTTNG_HOME/lttng-traces/__name__-__date__-__time__+,
1060 where +__name__+ is the tracing session name. The
1061 env:LTTNG_HOME environment variable defaults to `$HOME` if not set.
1063 See <<viewing-and-analyzing-your-traces,View and analyze the
1064 recorded events>> to view the recorded events.
1067 [[viewing-and-analyzing-your-traces]]
1068 === View and analyze the recorded events
1070 Once you have completed the <<tracing-the-linux-kernel,Trace the Linux
1071 kernel>> and <<tracing-your-own-user-application,Trace a user
1072 application>> tutorials, you can inspect the recorded events.
1074 There are many tools you can use to read LTTng traces:
1076 * **cmd:babeltrace** is a command-line utility which converts trace
1077 formats; it supports the format that LTTng produces, CTF, as well as a
1078 basic text output which can be ++grep++ed. The cmd:babeltrace command
1079 is part of the http://diamon.org/babeltrace[Babeltrace] project.
1080 * Babeltrace also includes
1081 **https://www.python.org/[Python{nbsp}3] bindings** so
1082 that you can easily open and read an LTTng trace with your own script,
1083 benefiting from the power of Python.
1084 * http://tracecompass.org/[**Trace Compass**]
1085 is a graphical user interface for viewing and analyzing any type of
1086 logs or traces, including LTTng's.
1087 * https://github.com/lttng/lttng-analyses[**LTTng analyses**] is a
1088 project which includes many high-level analyses of LTTng kernel
1089 traces, like scheduling statistics, interrupt frequency distribution,
1090 top CPU usage, and more.
1092 NOTE: This section assumes that LTTng saved the traces it recorded
1093 during the previous tutorials to their default location, in the
1094 dir:{$LTTNG_HOME/lttng-traces} directory. The env:LTTNG_HOME
1095 environment variable defaults to `$HOME` if not set.
1098 [[viewing-and-analyzing-your-traces-bt]]
1099 ==== Use the cmd:babeltrace command-line tool
1101 The simplest way to list all the recorded events of a trace is to pass
1102 its path to cmd:babeltrace with no options:
1106 $ babeltrace ~/lttng-traces/my-user-space-session*
1109 cmd:babeltrace finds all traces recursively within the given path and
1110 prints all their events, merging them in chronological order.
1112 You can pipe the output of cmd:babeltrace into a tool like man:grep(1) for
1117 $ babeltrace /tmp/my-kernel-trace | grep _switch
1120 You can pipe the output of cmd:babeltrace into a tool like man:wc(1) to
1121 count the recorded events:
1125 $ babeltrace /tmp/my-kernel-trace | grep _open | wc --lines
1129 [[viewing-and-analyzing-your-traces-bt-python]]
1130 ==== Use the Babeltrace{nbsp}1 Python bindings
1132 The <<viewing-and-analyzing-your-traces-bt,text output of cmd:babeltrace>>
1133 is useful to isolate events by simple matching using man:grep(1) and
1134 similar utilities. However, more elaborate filters, such as keeping only
1135 event records with a field value falling within a specific range, are
1136 not trivial to write using a shell. Moreover, reductions and even the
1137 most basic computations involving multiple event records are virtually
1138 impossible to implement.
1140 Fortunately, Babeltrace{nbsp}1 ships with Python{nbsp}3 bindings which
1141 makes it easy to read the event records of an LTTng trace sequentially
1142 and compute the desired information.
1144 The following script accepts an LTTng Linux kernel trace path as its
1145 first argument and prints the short names of the top five running
1146 processes on CPU{nbsp}0 during the whole trace:
1151 from collections import Counter
1157 if len(sys.argv) != 2:
1158 msg = 'Usage: python3 {} TRACEPATH'.format(sys.argv[0])
1159 print(msg, file=sys.stderr)
1162 # A trace collection contains one or more traces
1163 col = babeltrace.TraceCollection()
1165 # Add the trace provided by the user (LTTng traces always have
1167 if col.add_trace(sys.argv[1], 'ctf') is None:
1168 raise RuntimeError('Cannot add trace')
1170 # This counter dict contains execution times:
1172 # task command name -> total execution time (ns)
1173 exec_times = Counter()
1175 # This contains the last `sched_switch` timestamp
1179 for event in col.events:
1180 # Keep only `sched_switch` events
1181 if event.name != 'sched_switch':
1184 # Keep only events which happened on CPU 0
1185 if event['cpu_id'] != 0:
1189 cur_ts = event.timestamp
1195 # Previous task command (short) name
1196 prev_comm = event['prev_comm']
1198 # Initialize entry in our dict if not yet done
1199 if prev_comm not in exec_times:
1200 exec_times[prev_comm] = 0
1202 # Compute previous command execution time
1203 diff = cur_ts - last_ts
1205 # Update execution time of this command
1206 exec_times[prev_comm] += diff
1208 # Update last timestamp
1212 for name, ns in exec_times.most_common(5):
1214 print('{:20}{} s'.format(name, s))
1219 if __name__ == '__main__':
1220 sys.exit(0 if top5proc() else 1)
1227 $ python3 top5proc.py /tmp/my-kernel-trace/kernel
1233 swapper/0 48.607245889 s
1234 chromium 7.192738188 s
1235 pavucontrol 0.709894415 s
1236 Compositor 0.660867933 s
1237 Xorg.bin 0.616753786 s
1240 Note that `swapper/0` is the "idle" process of CPU{nbsp}0 on Linux;
1241 since we weren't using the CPU that much when tracing, its first
1242 position in the list makes sense.
1246 == [[understanding-lttng]]Core concepts
1248 From a user's perspective, the LTTng system is built on a few concepts,
1249 or objects, on which the <<lttng-cli,cmd:lttng command-line tool>>
1250 operates by sending commands to the <<lttng-sessiond,session daemon>>.
1251 Understanding how those objects relate to eachother is key in mastering
1254 The core concepts are:
1256 * <<tracing-session,Tracing session>>
1257 * <<domain,Tracing domain>>
1258 * <<channel,Channel and ring buffer>>
1259 * <<"event","Instrumentation point, event rule, event, and event record">>
1265 A _tracing session_ is a stateful dialogue between you and
1266 a <<lttng-sessiond,session daemon>>. You can
1267 <<creating-destroying-tracing-sessions,create a new tracing
1268 session>> with the `lttng create` command.
1270 Anything that you do when you control LTTng tracers happens within a
1271 tracing session. In particular, a tracing session:
1274 * Has its own set of trace files.
1275 * Has its own state of activity (started or stopped).
1276 * Has its own <<tracing-session-mode,mode>> (local, network streaming,
1278 * Has its own <<channel,channels>> to which are associated their own
1279 <<event,event rules>>.
1282 .A _tracing session_ contains <<channel,channels>> that are members of <<domain,tracing domains>> and contain <<event,event rules>>.
1283 image::concepts.png[]
1285 Those attributes and objects are completely isolated between different
1288 A tracing session is analogous to a cash machine session:
1289 the operations you do on the banking system through the cash machine do
1290 not alter the data of other users of the same system. In the case of
1291 the cash machine, a session lasts as long as your bank card is inside.
1292 In the case of LTTng, a tracing session lasts from the `lttng create`
1293 command to the `lttng destroy` command.
1296 .Each Unix user has its own set of tracing sessions.
1297 image::many-sessions.png[]
1300 [[tracing-session-mode]]
1301 ==== Tracing session mode
1303 LTTng can send the generated trace data to different locations. The
1304 _tracing session mode_ dictates where to send it. The following modes
1305 are available in LTTng{nbsp}{revision}:
1308 LTTng writes the traces to the file system of the machine it traces
1311 Network streaming mode::
1312 LTTng sends the traces over the network to a
1313 <<lttng-relayd,relay daemon>> running on a remote system.
1316 LTTng does not write the traces by default. Instead, you can request
1317 LTTng to <<taking-a-snapshot,take a snapshot>>, that is, a copy of the
1318 tracing session's current sub-buffers, and to write it to the
1319 target's file system or to send it over the network to a
1320 <<lttng-relayd,relay daemon>> running on a remote system.
1323 This mode is similar to the network streaming mode, but a live
1324 trace viewer can connect to the distant relay daemon to
1325 <<lttng-live,view event records as LTTng generates them>>.
1331 A _tracing domain_ is a namespace for event sources. A tracing domain
1332 has its own properties and features.
1334 There are currently five available tracing domains:
1338 * `java.util.logging` (JUL)
1342 You must specify a tracing domain when using some commands to avoid
1343 ambiguity. For example, since all the domains support named tracepoints
1344 as event sources (instrumentation points that you manually insert in the
1345 source code), you need to specify a tracing domain when
1346 <<enabling-disabling-events,creating an event rule>> because all the
1347 tracing domains could have tracepoints with the same names.
1349 You can create <<channel,channels>> in the Linux kernel and user space
1350 tracing domains. The other tracing domains have a single default
1355 === Channel and ring buffer
1357 A _channel_ is an object which is responsible for a set of ring buffers.
1358 Each ring buffer is divided into multiple sub-buffers. When an LTTng
1359 tracer emits an event, it can record it to one or more
1360 sub-buffers. The attributes of a channel determine what to do when
1361 there's no space left for a new event record because all sub-buffers
1362 are full, where to send a full sub-buffer, and other behaviours.
1364 A channel is always associated to a <<domain,tracing domain>>. The
1365 `java.util.logging` (JUL), log4j, and Python tracing domains each have
1366 a default channel which you cannot configure.
1368 A channel also owns <<event,event rules>>. When an LTTng tracer emits
1369 an event, it records it to the sub-buffers of all
1370 the enabled channels with a satisfied event rule, as long as those
1371 channels are part of active <<tracing-session,tracing sessions>>.
1374 [[channel-buffering-schemes]]
1375 ==== Per-user vs. per-process buffering schemes
1377 A channel has at least one ring buffer _per CPU_. LTTng always
1378 records an event to the ring buffer associated to the CPU on which it
1381 Two _buffering schemes_ are available when you
1382 <<enabling-disabling-channels,create a channel>> in the
1383 user space <<domain,tracing domain>>:
1385 Per-user buffering::
1386 Allocate one set of ring buffers--one per CPU--shared by all the
1387 instrumented processes of each Unix user.
1391 .Per-user buffering scheme.
1392 image::per-user-buffering.png[]
1395 Per-process buffering::
1396 Allocate one set of ring buffers--one per CPU--for each
1397 instrumented process.
1401 .Per-process buffering scheme.
1402 image::per-process-buffering.png[]
1405 The per-process buffering scheme tends to consume more memory than the
1406 per-user option because systems generally have more instrumented
1407 processes than Unix users running instrumented processes. However, the
1408 per-process buffering scheme ensures that one process having a high
1409 event throughput won't fill all the shared sub-buffers of the same
1412 The Linux kernel tracing domain has only one available buffering scheme
1413 which is to allocate a single set of ring buffers for the whole system.
1414 This scheme is similar to the per-user option, but with a single, global
1415 user "running" the kernel.
1418 [[channel-overwrite-mode-vs-discard-mode]]
1419 ==== Overwrite vs. discard event record loss modes
1421 When an event occurs, LTTng records it to a specific sub-buffer (yellow
1422 arc in the following animations) of a specific channel's ring buffer.
1423 When there's no space left in a sub-buffer, the tracer marks it as
1424 consumable (red) and another, empty sub-buffer starts receiving the
1425 following event records. A <<lttng-consumerd,consumer daemon>>
1426 eventually consumes the marked sub-buffer (returns to white).
1429 [role="docsvg-channel-subbuf-anim"]
1434 In an ideal world, sub-buffers are consumed faster than they are filled,
1435 as it is the case in the previous animation. In the real world,
1436 however, all sub-buffers can be full at some point, leaving no space to
1437 record the following events.
1439 By default, LTTng-modules and LTTng-UST are _non-blocking_ tracers: when
1440 no empty sub-buffer is available, it is acceptable to lose event records
1441 when the alternative would be to cause substantial delays in the
1442 instrumented application's execution. LTTng privileges performance over
1443 integrity; it aims at perturbing the target system as little as possible
1444 in order to make tracing of subtle race conditions and rare interrupt
1447 Since LTTng{nbsp}2.10, the LTTng user space tracer, LTTng-UST, supports
1448 a _blocking mode_. See the <<blocking-timeout-example,blocking timeout
1449 example>> to learn how to use the blocking mode.
1451 When it comes to losing event records because no empty sub-buffer is
1452 available, or because the <<opt-blocking-timeout,blocking timeout>> is
1453 reached, the channel's _event record loss mode_ determines what to do.
1454 The available event record loss modes are:
1457 Drop the newest event records until the tracer releases a sub-buffer.
1459 This is the only available mode when you specify a
1460 <<opt-blocking-timeout,blocking timeout>>.
1463 Clear the sub-buffer containing the oldest event records and start
1464 writing the newest event records there.
1466 This mode is sometimes called _flight recorder mode_ because it's
1468 https://en.wikipedia.org/wiki/Flight_recorder[flight recorder]:
1469 always keep a fixed amount of the latest data.
1471 Which mechanism you should choose depends on your context: prioritize
1472 the newest or the oldest event records in the ring buffer?
1474 Beware that, in overwrite mode, the tracer abandons a _whole sub-buffer_
1475 as soon as a there's no space left for a new event record, whereas in
1476 discard mode, the tracer only discards the event record that doesn't
1479 In discard mode, LTTng increments a count of lost event records when an
1480 event record is lost and saves this count to the trace. In overwrite
1481 mode, since LTTng{nbsp}2.8, LTTng increments a count of lost sub-buffers
1482 when a sub-buffer is lost and saves this count to the trace. In this
1483 mode, LTTng does not write to the trace the exact number of lost event
1484 records in those lost sub-buffers. Trace analyses can use the trace's
1485 saved discarded event record and sub-buffer counts to decide whether or
1486 not to perform the analyses even if trace data is known to be missing.
1488 There are a few ways to decrease your probability of losing event
1490 <<channel-subbuf-size-vs-subbuf-count,Sub-buffer count and size>> shows
1491 how you can fine-tune the sub-buffer count and size of a channel to
1492 virtually stop losing event records, though at the cost of greater
1496 [[channel-subbuf-size-vs-subbuf-count]]
1497 ==== Sub-buffer count and size
1499 When you <<enabling-disabling-channels,create a channel>>, you can
1500 set its number of sub-buffers and their size.
1502 Note that there is noticeable CPU overhead introduced when
1503 switching sub-buffers (marking a full one as consumable and switching
1504 to an empty one for the following events to be recorded). Knowing this,
1505 the following list presents a few practical situations along with how
1506 to configure the sub-buffer count and size for them:
1508 * **High event throughput**: In general, prefer bigger sub-buffers to
1509 lower the risk of losing event records.
1511 Having bigger sub-buffers also ensures a lower
1512 <<channel-switch-timer,sub-buffer switching frequency>>.
1514 The number of sub-buffers is only meaningful if you create the channel
1515 in overwrite mode: in this case, if a sub-buffer overwrite happens, the
1516 other sub-buffers are left unaltered.
1518 * **Low event throughput**: In general, prefer smaller sub-buffers
1519 since the risk of losing event records is low.
1521 Because events occur less frequently, the sub-buffer switching frequency
1522 should remain low and thus the tracer's overhead should not be a
1525 * **Low memory system**: If your target system has a low memory
1526 limit, prefer fewer first, then smaller sub-buffers.
1528 Even if the system is limited in memory, you want to keep the
1529 sub-buffers as big as possible to avoid a high sub-buffer switching
1532 Note that LTTng uses http://diamon.org/ctf/[CTF] as its trace format,
1533 which means event data is very compact. For example, the average
1534 LTTng kernel event record weights about 32{nbsp}bytes. Thus, a
1535 sub-buffer size of 1{nbsp}MiB is considered big.
1537 The previous situations highlight the major trade-off between a few big
1538 sub-buffers and more, smaller sub-buffers: sub-buffer switching
1539 frequency vs. how much data is lost in overwrite mode. Assuming a
1540 constant event throughput and using the overwrite mode, the two
1541 following configurations have the same ring buffer total size:
1544 [role="docsvg-channel-subbuf-size-vs-count-anim"]
1549 * **Two sub-buffers of 4{nbsp}MiB each**: Expect a very low sub-buffer
1550 switching frequency, but if a sub-buffer overwrite happens, half of
1551 the event records so far (4{nbsp}MiB) are definitely lost.
1552 * **Eight sub-buffers of 1{nbsp}MiB each**: Expect four times the tracer's
1553 overhead as the previous configuration, but if a sub-buffer
1554 overwrite happens, only the eighth of event records so far are
1557 In discard mode, the sub-buffers count parameter is pointless: use two
1558 sub-buffers and set their size according to the requirements of your
1562 [[channel-switch-timer]]
1563 ==== Switch timer period
1565 The _switch timer period_ is an important configurable attribute of
1566 a channel to ensure periodic sub-buffer flushing.
1568 When the _switch timer_ expires, a sub-buffer switch happens. You can
1569 set the switch timer period attribute when you
1570 <<enabling-disabling-channels,create a channel>> to ensure that LTTng
1571 consumes and commits trace data to trace files or to a distant relay
1572 daemon periodically in case of a low event throughput.
1575 [role="docsvg-channel-switch-timer"]
1580 This attribute is also convenient when you use big sub-buffers to cope
1581 with a sporadic high event throughput, even if the throughput is
1585 [[channel-read-timer]]
1586 ==== Read timer period
1588 By default, the LTTng tracers use a notification mechanism to signal a
1589 full sub-buffer so that a consumer daemon can consume it. When such
1590 notifications must be avoided, for example in real-time applications,
1591 you can use the channel's _read timer_ instead. When the read timer
1592 fires, the <<lttng-consumerd,consumer daemon>> checks for full,
1593 consumable sub-buffers.
1596 [[tracefile-rotation]]
1597 ==== Trace file count and size
1599 By default, trace files can grow as large as needed. You can set the
1600 maximum size of each trace file that a channel writes when you
1601 <<enabling-disabling-channels,create a channel>>. When the size of
1602 a trace file reaches the channel's fixed maximum size, LTTng creates
1603 another file to contain the next event records. LTTng appends a file
1604 count to each trace file name in this case.
1606 If you set the trace file size attribute when you create a channel, the
1607 maximum number of trace files that LTTng creates is _unlimited_ by
1608 default. To limit them, you can also set a maximum number of trace
1609 files. When the number of trace files reaches the channel's fixed
1610 maximum count, the oldest trace file is overwritten. This mechanism is
1611 called _trace file rotation_.
1615 Even if you don't limit the trace file count, you cannot assume that
1616 LTTng doesn't manage any trace file.
1618 In other words, there is no safe way to know if LTTng still holds a
1619 given trace file open with the trace file rotation feature.
1621 The only way to obtain an unmanaged, self-contained LTTng trace before
1622 you <<creating-destroying-tracing-sessions,destroy>> the tracing session
1623 is with the <<session-rotation,tracing session rotation>> feature
1624 (available since LTTng{nbsp}2.11).
1629 === Instrumentation point, event rule, event, and event record
1631 An _event rule_ is a set of conditions which must be **all** satisfied
1632 for LTTng to record an occuring event.
1634 You set the conditions when you <<enabling-disabling-events,create
1637 You always attach an event rule to a <<channel,channel>> when you create
1640 When an event passes the conditions of an event rule, LTTng records it
1641 in one of the attached channel's sub-buffers.
1643 The available conditions, as of LTTng{nbsp}{revision}, are:
1645 * The event rule _is enabled_.
1646 * The instrumentation point's type _is{nbsp}T_.
1647 * The instrumentation point's name (sometimes called _event name_)
1648 _matches{nbsp}N_, but _is not{nbsp}E_.
1649 * The instrumentation point's log level _is as severe as{nbsp}L_, or
1650 _is exactly{nbsp}L_.
1651 * The fields of the event's payload _satisfy_ a filter
1652 expression{nbsp}__F__.
1654 As you can see, all the conditions but the dynamic filter are related to
1655 the event rule's status or to the instrumentation point, not to the
1656 occurring events. This is why, without a filter, checking if an event
1657 passes an event rule is not a dynamic task: when you create or modify an
1658 event rule, all the tracers of its tracing domain enable or disable the
1659 instrumentation points themselves once. This is possible because the
1660 attributes of an instrumentation point (type, name, and log level) are
1661 defined statically. In other words, without a dynamic filter, the tracer
1662 _does not evaluate_ the arguments of an instrumentation point unless it
1663 matches an enabled event rule.
1665 Note that, for LTTng to record an event, the <<channel,channel>> to
1666 which a matching event rule is attached must also be enabled, and the
1667 <<tracing-session,tracing session>> owning this channel must be active
1671 .Logical path from an instrumentation point to an event record.
1672 image::event-rule.png[]
1674 .Event, event record, or event rule?
1676 With so many similar terms, it's easy to get confused.
1678 An **event** is the consequence of the execution of an _instrumentation
1679 point_, like a tracepoint that you manually place in some source code,
1680 or a Linux kernel kprobe. An event is said to _occur_ at a specific
1681 time. Different actions can be taken upon the occurrence of an event,
1682 like record the event's payload to a buffer.
1684 An **event record** is the representation of an event in a sub-buffer. A
1685 tracer is responsible for capturing the payload of an event, current
1686 context variables, the event's ID, and the event's timestamp. LTTng
1687 can append this sub-buffer to a trace file.
1689 An **event rule** is a set of conditions which must _all_ be satisfied
1690 for LTTng to record an occuring event. Events still occur without
1691 satisfying event rules, but LTTng does not record them.
1696 == Components of noch:{LTTng}
1698 The second _T_ in _LTTng_ stands for _toolkit_: it would be wrong
1699 to call LTTng a simple _tool_ since it is composed of multiple
1700 interacting components. This section describes those components,
1701 explains their respective roles, and shows how they connect together to
1702 form the LTTng ecosystem.
1704 The following diagram shows how the most important components of LTTng
1705 interact with user applications, the Linux kernel, and you:
1708 .Control and trace data paths between LTTng components.
1709 image::plumbing.png[]
1711 The LTTng project incorporates:
1713 * **LTTng-tools**: Libraries and command-line interface to
1714 control tracing sessions.
1715 ** <<lttng-sessiond,Session daemon>> (man:lttng-sessiond(8)).
1716 ** <<lttng-consumerd,Consumer daemon>> (cmd:lttng-consumerd).
1717 ** <<lttng-relayd,Relay daemon>> (man:lttng-relayd(8)).
1718 ** <<liblttng-ctl-lttng,Tracing control library>> (`liblttng-ctl`).
1719 ** <<lttng-cli,Tracing control command-line tool>> (man:lttng(1)).
1720 * **LTTng-UST**: Libraries and Java/Python packages to trace user
1722 ** <<lttng-ust,User space tracing library>> (`liblttng-ust`) and its
1723 headers to instrument and trace any native user application.
1724 ** <<prebuilt-ust-helpers,Preloadable user space tracing helpers>>:
1725 *** `liblttng-ust-libc-wrapper`
1726 *** `liblttng-ust-pthread-wrapper`
1727 *** `liblttng-ust-cyg-profile`
1728 *** `liblttng-ust-cyg-profile-fast`
1729 *** `liblttng-ust-dl`
1730 ** User space tracepoint provider source files generator command-line
1731 tool (man:lttng-gen-tp(1)).
1732 ** <<lttng-ust-agents,LTTng-UST Java agent>> to instrument and trace
1733 Java applications using `java.util.logging` or
1734 Apache log4j{nbsp}1.2 logging.
1735 ** <<lttng-ust-agents,LTTng-UST Python agent>> to instrument
1736 Python applications using the standard `logging` package.
1737 * **LTTng-modules**: <<lttng-modules,Linux kernel modules>> to trace
1739 ** LTTng kernel tracer module.
1740 ** Tracing ring buffer kernel modules.
1741 ** Probe kernel modules.
1742 ** LTTng logger kernel module.
1746 === Tracing control command-line interface
1749 .The tracing control command-line interface.
1750 image::plumbing-lttng-cli.png[]
1752 The _man:lttng(1) command-line tool_ is the standard user interface to
1753 control LTTng <<tracing-session,tracing sessions>>. The cmd:lttng tool
1754 is part of LTTng-tools.
1756 The cmd:lttng tool is linked with
1757 <<liblttng-ctl-lttng,`liblttng-ctl`>> to communicate with
1758 one or more <<lttng-sessiond,session daemons>> behind the scenes.
1760 The cmd:lttng tool has a Git-like interface:
1764 $ lttng <GENERAL OPTIONS> <COMMAND> <COMMAND OPTIONS>
1767 The <<controlling-tracing,Tracing control>> section explores the
1768 available features of LTTng using the cmd:lttng tool.
1771 [[liblttng-ctl-lttng]]
1772 === Tracing control library
1775 .The tracing control library.
1776 image::plumbing-liblttng-ctl.png[]
1778 The _LTTng control library_, `liblttng-ctl`, is used to communicate
1779 with a <<lttng-sessiond,session daemon>> using a C API that hides the
1780 underlying protocol's details. `liblttng-ctl` is part of LTTng-tools.
1782 The <<lttng-cli,cmd:lttng command-line tool>>
1783 is linked with `liblttng-ctl`.
1785 You can use `liblttng-ctl` in C or $$C++$$ source code by including its
1790 #include <lttng/lttng.h>
1793 Some objects are referenced by name (C string), such as tracing
1794 sessions, but most of them require to create a handle first using
1795 `lttng_create_handle()`.
1797 As of LTTng{nbsp}{revision}, the best available developer documentation for
1798 `liblttng-ctl` is its installed header files. Every function and structure is
1799 thoroughly documented.
1803 === User space tracing library
1806 .The user space tracing library.
1807 image::plumbing-liblttng-ust.png[]
1809 The _user space tracing library_, `liblttng-ust` (see man:lttng-ust(3)),
1810 is the LTTng user space tracer. It receives commands from a
1811 <<lttng-sessiond,session daemon>>, for example to
1812 enable and disable specific instrumentation points, and writes event
1813 records to ring buffers shared with a
1814 <<lttng-consumerd,consumer daemon>>.
1815 `liblttng-ust` is part of LTTng-UST.
1817 Public C header files are installed beside `liblttng-ust` to
1818 instrument any <<c-application,C or $$C++$$ application>>.
1820 <<lttng-ust-agents,LTTng-UST agents>>, which are regular Java and Python
1821 packages, use their own library providing tracepoints which is
1822 linked with `liblttng-ust`.
1824 An application or library does not have to initialize `liblttng-ust`
1825 manually: its constructor does the necessary tasks to properly register
1826 to a session daemon. The initialization phase also enables the
1827 instrumentation points matching the <<event,event rules>> that you
1831 [[lttng-ust-agents]]
1832 === User space tracing agents
1835 .The user space tracing agents.
1836 image::plumbing-lttng-ust-agents.png[]
1838 The _LTTng-UST Java and Python agents_ are regular Java and Python
1839 packages which add LTTng tracing capabilities to the
1840 native logging frameworks. The LTTng-UST agents are part of LTTng-UST.
1842 In the case of Java, the
1843 https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[`java.util.logging`
1844 core logging facilities] and
1845 https://logging.apache.org/log4j/1.2/[Apache log4j{nbsp}1.2] are supported.
1846 Note that Apache Log4{nbsp}2 is not supported.
1848 In the case of Python, the standard
1849 https://docs.python.org/3/library/logging.html[`logging`] package
1850 is supported. Both Python{nbsp}2 and Python{nbsp}3 modules can import the
1851 LTTng-UST Python agent package.
1853 The applications using the LTTng-UST agents are in the
1854 `java.util.logging` (JUL),
1855 log4j, and Python <<domain,tracing domains>>.
1857 Both agents use the same mechanism to trace the log statements. When an
1858 agent initializes, it creates a log handler that attaches to the root
1859 logger. The agent also registers to a <<lttng-sessiond,session daemon>>.
1860 When the application executes a log statement, the root logger passes it
1861 to the agent's log handler. The agent's log handler calls a native
1862 function in a tracepoint provider package shared library linked with
1863 <<lttng-ust,`liblttng-ust`>>, passing the formatted log message and
1864 other fields, like its logger name and its log level. This native
1865 function contains a user space instrumentation point, hence tracing the
1868 The log level condition of an
1869 <<event,event rule>> is considered when tracing
1870 a Java or a Python application, and it's compatible with the standard
1871 JUL, log4j, and Python log levels.
1875 === LTTng kernel modules
1878 .The LTTng kernel modules.
1879 image::plumbing-lttng-modules.png[]
1881 The _LTTng kernel modules_ are a set of Linux kernel modules
1882 which implement the kernel tracer of the LTTng project. The LTTng
1883 kernel modules are part of LTTng-modules.
1885 The LTTng kernel modules include:
1887 * A set of _probe_ modules.
1889 Each module attaches to a specific subsystem
1890 of the Linux kernel using its tracepoint instrument points. There are
1891 also modules to attach to the entry and return points of the Linux
1892 system call functions.
1894 * _Ring buffer_ modules.
1896 A ring buffer implementation is provided as kernel modules. The LTTng
1897 kernel tracer writes to the ring buffer; a
1898 <<lttng-consumerd,consumer daemon>> reads from the ring buffer.
1900 * The _LTTng kernel tracer_ module.
1901 * The _LTTng logger_ module.
1903 The LTTng logger module implements the special path:{/proc/lttng-logger}
1904 (and path:{/dev/lttng-logger} since LTTng{nbsp}2.11) files so that any
1905 executable can generate LTTng events by opening and writing to those
1908 See <<proc-lttng-logger-abi,LTTng logger>>.
1910 Generally, you do not have to load the LTTng kernel modules manually
1911 (using man:modprobe(8), for example): a root <<lttng-sessiond,session
1912 daemon>> loads the necessary modules when starting. If you have extra
1913 probe modules, you can specify to load them to the session daemon on
1916 The LTTng kernel modules are installed in
1917 +/usr/lib/modules/__release__/extra+ by default, where +__release__+ is
1918 the kernel release (see `uname --kernel-release`).
1925 .The session daemon.
1926 image::plumbing-sessiond.png[]
1928 The _session daemon_, man:lttng-sessiond(8), is a daemon responsible for
1929 managing tracing sessions and for controlling the various components of
1930 LTTng. The session daemon is part of LTTng-tools.
1932 The session daemon sends control requests to and receives control
1935 * The <<lttng-ust,user space tracing library>>.
1937 Any instance of the user space tracing library first registers to
1938 a session daemon. Then, the session daemon can send requests to
1939 this instance, such as:
1942 ** Get the list of tracepoints.
1943 ** Share an <<event,event rule>> so that the user space tracing library
1944 can enable or disable tracepoints. Amongst the possible conditions
1945 of an event rule is a filter expression which `liblttng-ust` evalutes
1946 when an event occurs.
1947 ** Share <<channel,channel>> attributes and ring buffer locations.
1950 The session daemon and the user space tracing library use a Unix
1951 domain socket for their communication.
1953 * The <<lttng-ust-agents,user space tracing agents>>.
1955 Any instance of a user space tracing agent first registers to
1956 a session daemon. Then, the session daemon can send requests to
1957 this instance, such as:
1960 ** Get the list of loggers.
1961 ** Enable or disable a specific logger.
1964 The session daemon and the user space tracing agent use a TCP connection
1965 for their communication.
1967 * The <<lttng-modules,LTTng kernel tracer>>.
1968 * The <<lttng-consumerd,consumer daemon>>.
1970 The session daemon sends requests to the consumer daemon to instruct
1971 it where to send the trace data streams, amongst other information.
1973 * The <<lttng-relayd,relay daemon>>.
1975 The session daemon receives commands from the
1976 <<liblttng-ctl-lttng,tracing control library>>.
1978 The root session daemon loads the appropriate
1979 <<lttng-modules,LTTng kernel modules>> on startup. It also spawns
1980 a <<lttng-consumerd,consumer daemon>> as soon as you create
1981 an <<event,event rule>>.
1983 The session daemon does not send and receive trace data: this is the
1984 role of the <<lttng-consumerd,consumer daemon>> and
1985 <<lttng-relayd,relay daemon>>. It does, however, generate the
1986 http://diamon.org/ctf/[CTF] metadata stream.
1988 Each Unix user can have its own session daemon instance. The
1989 tracing sessions which different session daemons manage are completely
1992 The root user's session daemon is the only one which is
1993 allowed to control the LTTng kernel tracer, and its spawned consumer
1994 daemon is the only one which is allowed to consume trace data from the
1995 LTTng kernel tracer. Note, however, that any Unix user which is a member
1996 of the <<tracing-group,tracing group>> is allowed
1997 to create <<channel,channels>> in the
1998 Linux kernel <<domain,tracing domain>>, and thus to trace the Linux
2001 The <<lttng-cli,cmd:lttng command-line tool>> automatically starts a
2002 session daemon when using its `create` command if none is currently
2003 running. You can also start the session daemon manually.
2010 .The consumer daemon.
2011 image::plumbing-consumerd.png[]
2013 The _consumer daemon_, cmd:lttng-consumerd, is a daemon which shares
2014 ring buffers with user applications or with the LTTng kernel modules to
2015 collect trace data and send it to some location (on disk or to a
2016 <<lttng-relayd,relay daemon>> over the network). The consumer daemon
2017 is part of LTTng-tools.
2019 You do not start a consumer daemon manually: a consumer daemon is always
2020 spawned by a <<lttng-sessiond,session daemon>> as soon as you create an
2021 <<event,event rule>>, that is, before you start tracing. When you kill
2022 its owner session daemon, the consumer daemon also exits because it is
2023 the session daemon's child process. Command-line options of
2024 man:lttng-sessiond(8) target the consumer daemon process.
2026 There are up to two running consumer daemons per Unix user, whereas only
2027 one session daemon can run per user. This is because each process can be
2028 either 32-bit or 64-bit: if the target system runs a mixture of 32-bit
2029 and 64-bit processes, it is more efficient to have separate
2030 corresponding 32-bit and 64-bit consumer daemons. The root user is an
2031 exception: it can have up to _three_ running consumer daemons: 32-bit
2032 and 64-bit instances for its user applications, and one more
2033 reserved for collecting kernel trace data.
2041 image::plumbing-relayd.png[]
2043 The _relay daemon_, man:lttng-relayd(8), is a daemon acting as a bridge
2044 between remote session and consumer daemons, local trace files, and a
2045 remote live trace viewer. The relay daemon is part of LTTng-tools.
2047 The main purpose of the relay daemon is to implement a receiver of
2048 <<sending-trace-data-over-the-network,trace data over the network>>.
2049 This is useful when the target system does not have much file system
2050 space to record trace files locally.
2052 The relay daemon is also a server to which a
2053 <<lttng-live,live trace viewer>> can
2054 connect. The live trace viewer sends requests to the relay daemon to
2055 receive trace data as the target system emits events. The
2056 communication protocol is named _LTTng live_; it is used over TCP
2059 Note that you can start the relay daemon on the target system directly.
2060 This is the setup of choice when the use case is to view events as
2061 the target system emits them without the need of a remote system.
2065 == [[using-lttng]]Instrumentation
2067 There are many examples of tracing and monitoring in our everyday life:
2069 * You have access to real-time and historical weather reports and
2070 forecasts thanks to weather stations installed around the country.
2071 * You know your heart is safe thanks to an electrocardiogram.
2072 * You make sure not to drive your car too fast and to have enough fuel
2073 to reach your destination thanks to gauges visible on your dashboard.
2075 All the previous examples have something in common: they rely on
2076 **instruments**. Without the electrodes attached to the surface of your
2077 body's skin, cardiac monitoring is futile.
2079 LTTng, as a tracer, is no different from those real life examples. If
2080 you're about to trace a software system or, in other words, record its
2081 history of execution, you better have **instrumentation points** in the
2082 subject you're tracing, that is, the actual software.
2084 Various ways were developed to instrument a piece of software for LTTng
2085 tracing. The most straightforward one is to manually place
2086 instrumentation points, called _tracepoints_, in the software's source
2087 code. It is also possible to add instrumentation points dynamically in
2088 the Linux kernel <<domain,tracing domain>>.
2090 If you're only interested in tracing the Linux kernel, your
2091 instrumentation needs are probably already covered by LTTng's built-in
2092 <<lttng-modules,Linux kernel tracepoints>>. You may also wish to trace a
2093 user application which is already instrumented for LTTng tracing.
2094 In such cases, you can skip this whole section and read the topics of
2095 the <<controlling-tracing,Tracing control>> section.
2097 Many methods are available to instrument a piece of software for LTTng
2100 * <<c-application,User space instrumentation for C and $$C++$$
2102 * <<prebuilt-ust-helpers,Prebuilt user space tracing helpers>>.
2103 * <<java-application,User space Java agent>>.
2104 * <<python-application,User space Python agent>>.
2105 * <<proc-lttng-logger-abi,LTTng logger>>.
2106 * <<instrumenting-linux-kernel,LTTng kernel tracepoints>>.
2110 === [[cxx-application]]User space instrumentation for C and $$C++$$ applications
2112 The procedure to instrument a C or $$C++$$ user application with
2113 the <<lttng-ust,LTTng user space tracing library>>, `liblttng-ust`, is:
2115 . <<tracepoint-provider,Create the source files of a tracepoint provider
2117 . <<probing-the-application-source-code,Add tracepoints to
2118 the application's source code>>.
2119 . <<building-tracepoint-providers-and-user-application,Build and link
2120 a tracepoint provider package and the user application>>.
2122 If you need quick, man:printf(3)-like instrumentation, you can skip
2123 those steps and use <<tracef,`tracef()`>> or <<tracelog,`tracelog()`>>
2126 IMPORTANT: You need to <<installing-lttng,install>> LTTng-UST to
2127 instrument a user application with `liblttng-ust`.
2130 [[tracepoint-provider]]
2131 ==== Create the source files of a tracepoint provider package
2133 A _tracepoint provider_ is a set of compiled functions which provide
2134 **tracepoints** to an application, the type of instrumentation point
2135 supported by LTTng-UST. Those functions can emit events with
2136 user-defined fields and serialize those events as event records to one
2137 or more LTTng-UST <<channel,channel>> sub-buffers. The `tracepoint()`
2138 macro, which you <<probing-the-application-source-code,insert in a user
2139 application's source code>>, calls those functions.
2141 A _tracepoint provider package_ is an object file (`.o`) or a shared
2142 library (`.so`) which contains one or more tracepoint providers.
2143 Its source files are:
2145 * One or more <<tpp-header,tracepoint provider header>> (`.h`).
2146 * A <<tpp-source,tracepoint provider package source>> (`.c`).
2148 A tracepoint provider package is dynamically linked with `liblttng-ust`,
2149 the LTTng user space tracer, at run time.
2152 .User application linked with `liblttng-ust` and containing a tracepoint provider.
2153 image::ust-app.png[]
2155 NOTE: If you need quick, man:printf(3)-like instrumentation, you can
2156 skip creating and using a tracepoint provider and use
2157 <<tracef,`tracef()`>> or <<tracelog,`tracelog()`>> instead.
2161 ===== Create a tracepoint provider header file template
2163 A _tracepoint provider header file_ contains the tracepoint
2164 definitions of a tracepoint provider.
2166 To create a tracepoint provider header file:
2168 . Start from this template:
2172 .Tracepoint provider header file template (`.h` file extension).
2174 #undef TRACEPOINT_PROVIDER
2175 #define TRACEPOINT_PROVIDER provider_name
2177 #undef TRACEPOINT_INCLUDE
2178 #define TRACEPOINT_INCLUDE "./tp.h"
2180 #if !defined(_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
2183 #include <lttng/tracepoint.h>
2186 * Use TRACEPOINT_EVENT(), TRACEPOINT_EVENT_CLASS(),
2187 * TRACEPOINT_EVENT_INSTANCE(), and TRACEPOINT_LOGLEVEL() here.
2192 #include <lttng/tracepoint-event.h>
2198 * `provider_name` with the name of your tracepoint provider.
2199 * `"tp.h"` with the name of your tracepoint provider header file.
2201 . Below the `#include <lttng/tracepoint.h>` line, put your
2202 <<defining-tracepoints,tracepoint definitions>>.
2204 Your tracepoint provider name must be unique amongst all the possible
2205 tracepoint provider names used on the same target system. We
2206 suggest to include the name of your project or company in the name,
2207 for example, `org_lttng_my_project_tpp`.
2209 TIP: [[lttng-gen-tp]]You can use the man:lttng-gen-tp(1) tool to create
2210 this boilerplate for you. When using cmd:lttng-gen-tp, all you need to
2211 write are the <<defining-tracepoints,tracepoint definitions>>.
2214 [[defining-tracepoints]]
2215 ===== Create a tracepoint definition
2217 A _tracepoint definition_ defines, for a given tracepoint:
2219 * Its **input arguments**. They are the macro parameters that the
2220 `tracepoint()` macro accepts for this particular tracepoint
2221 in the user application's source code.
2222 * Its **output event fields**. They are the sources of event fields
2223 that form the payload of any event that the execution of the
2224 `tracepoint()` macro emits for this particular tracepoint.
2226 You can create a tracepoint definition by using the
2227 `TRACEPOINT_EVENT()` macro below the `#include <lttng/tracepoint.h>`
2229 <<tpp-header,tracepoint provider header file template>>.
2231 The syntax of the `TRACEPOINT_EVENT()` macro is:
2234 .`TRACEPOINT_EVENT()` macro syntax.
2237 /* Tracepoint provider name */
2240 /* Tracepoint name */
2243 /* Input arguments */
2248 /* Output event fields */
2257 * `provider_name` with your tracepoint provider name.
2258 * `tracepoint_name` with your tracepoint name.
2259 * `arguments` with the <<tpp-def-input-args,input arguments>>.
2260 * `fields` with the <<tpp-def-output-fields,output event field>>
2263 This tracepoint emits events named `provider_name:tracepoint_name`.
2266 .Event name's length limitation
2268 The concatenation of the tracepoint provider name and the
2269 tracepoint name must not exceed **254{nbsp}characters**. If it does, the
2270 instrumented application compiles and runs, but LTTng throws multiple
2271 warnings and you could experience serious issues.
2274 [[tpp-def-input-args]]The syntax of the `TP_ARGS()` macro is:
2277 .`TP_ARGS()` macro syntax.
2286 * `type` with the C type of the argument.
2287 * `arg_name` with the argument name.
2289 You can repeat `type` and `arg_name` up to 10{nbsp}times to have
2290 more than one argument.
2292 .`TP_ARGS()` usage with three arguments.
2304 The `TP_ARGS()` and `TP_ARGS(void)` forms are valid to create a
2305 tracepoint definition with no input arguments.
2307 [[tpp-def-output-fields]]The `TP_FIELDS()` macro contains a list of
2308 `ctf_*()` macros. Each `ctf_*()` macro defines one event field. See
2309 man:lttng-ust(3) for a complete description of the available `ctf_*()`
2310 macros. A `ctf_*()` macro specifies the type, size, and byte order of
2313 Each `ctf_*()` macro takes an _argument expression_ parameter. This is a
2314 C expression that the tracer evalutes at the `tracepoint()` macro site
2315 in the application's source code. This expression provides a field's
2316 source of data. The argument expression can include input argument names
2317 listed in the `TP_ARGS()` macro.
2319 Each `ctf_*()` macro also takes a _field name_ parameter. Field names
2320 must be unique within a given tracepoint definition.
2322 Here's a complete tracepoint definition example:
2324 .Tracepoint definition.
2326 The following tracepoint definition defines a tracepoint which takes
2327 three input arguments and has four output event fields.
2331 #include "my-custom-structure.h"
2337 const struct my_custom_structure*, my_custom_structure,
2342 ctf_string(query_field, query)
2343 ctf_float(double, ratio_field, ratio)
2344 ctf_integer(int, recv_size, my_custom_structure->recv_size)
2345 ctf_integer(int, send_size, my_custom_structure->send_size)
2350 You can refer to this tracepoint definition with the `tracepoint()`
2351 macro in your application's source code like this:
2355 tracepoint(my_provider, my_tracepoint,
2356 my_structure, some_ratio, the_query);
2360 NOTE: The LTTng tracer only evaluates tracepoint arguments at run time
2361 if they satisfy an enabled <<event,event rule>>.
2364 [[using-tracepoint-classes]]
2365 ===== Use a tracepoint class
2367 A _tracepoint class_ is a class of tracepoints which share the same
2368 output event field definitions. A _tracepoint instance_ is one
2369 instance of such a defined tracepoint class, with its own tracepoint
2372 The <<defining-tracepoints,`TRACEPOINT_EVENT()` macro>> is actually a
2373 shorthand which defines both a tracepoint class and a tracepoint
2374 instance at the same time.
2376 When you build a tracepoint provider package, the C or $$C++$$ compiler
2377 creates one serialization function for each **tracepoint class**. A
2378 serialization function is responsible for serializing the event fields
2379 of a tracepoint to a sub-buffer when tracing.
2381 For various performance reasons, when your situation requires multiple
2382 tracepoint definitions with different names, but with the same event
2383 fields, we recommend that you manually create a tracepoint class
2384 and instantiate as many tracepoint instances as needed. One positive
2385 effect of such a design, amongst other advantages, is that all
2386 tracepoint instances of the same tracepoint class reuse the same
2387 serialization function, thus reducing
2388 https://en.wikipedia.org/wiki/Cache_pollution[cache pollution].
2390 .Use a tracepoint class and tracepoint instances.
2392 Consider the following three tracepoint definitions:
2404 ctf_integer(int, userid, userid)
2405 ctf_integer(size_t, len, len)
2417 ctf_integer(int, userid, userid)
2418 ctf_integer(size_t, len, len)
2430 ctf_integer(int, userid, userid)
2431 ctf_integer(size_t, len, len)
2436 In this case, we create three tracepoint classes, with one implicit
2437 tracepoint instance for each of them: `get_account`, `get_settings`, and
2438 `get_transaction`. However, they all share the same event field names
2439 and types. Hence three identical, yet independent serialization
2440 functions are created when you build the tracepoint provider package.
2442 A better design choice is to define a single tracepoint class and three
2443 tracepoint instances:
2447 /* The tracepoint class */
2448 TRACEPOINT_EVENT_CLASS(
2449 /* Tracepoint provider name */
2452 /* Tracepoint class name */
2455 /* Input arguments */
2461 /* Output event fields */
2463 ctf_integer(int, userid, userid)
2464 ctf_integer(size_t, len, len)
2468 /* The tracepoint instances */
2469 TRACEPOINT_EVENT_INSTANCE(
2470 /* Tracepoint provider name */
2473 /* Tracepoint class name */
2476 /* Tracepoint name */
2479 /* Input arguments */
2485 TRACEPOINT_EVENT_INSTANCE(
2494 TRACEPOINT_EVENT_INSTANCE(
2507 [[assigning-log-levels]]
2508 ===== Assign a log level to a tracepoint definition
2510 You can assign an optional _log level_ to a
2511 <<defining-tracepoints,tracepoint definition>>.
2513 Assigning different levels of severity to tracepoint definitions can
2514 be useful: when you <<enabling-disabling-events,create an event rule>>,
2515 you can target tracepoints having a log level as severe as a specific
2518 The concept of LTTng-UST log levels is similar to the levels found
2519 in typical logging frameworks:
2521 * In a logging framework, the log level is given by the function
2522 or method name you use at the log statement site: `debug()`,
2523 `info()`, `warn()`, `error()`, and so on.
2524 * In LTTng-UST, you statically assign the log level to a tracepoint
2525 definition; any `tracepoint()` macro invocation which refers to
2526 this definition has this log level.
2528 You can assign a log level to a tracepoint definition with the
2529 `TRACEPOINT_LOGLEVEL()` macro. You must use this macro _after_ the
2530 <<defining-tracepoints,`TRACEPOINT_EVENT()`>> or
2531 <<using-tracepoint-classes,`TRACEPOINT_INSTANCE()`>> macro for a given
2534 The syntax of the `TRACEPOINT_LOGLEVEL()` macro is:
2537 .`TRACEPOINT_LOGLEVEL()` macro syntax.
2539 TRACEPOINT_LOGLEVEL(provider_name, tracepoint_name, log_level)
2544 * `provider_name` with the tracepoint provider name.
2545 * `tracepoint_name` with the tracepoint name.
2546 * `log_level` with the log level to assign to the tracepoint
2547 definition named `tracepoint_name` in the `provider_name`
2548 tracepoint provider.
2550 See man:lttng-ust(3) for a list of available log level names.
2552 .Assign the `TRACE_DEBUG_UNIT` log level to a tracepoint definition.
2556 /* Tracepoint definition */
2565 ctf_integer(int, userid, userid)
2566 ctf_integer(size_t, len, len)
2570 /* Log level assignment */
2571 TRACEPOINT_LOGLEVEL(my_app, get_transaction, TRACE_DEBUG_UNIT)
2577 ===== Create a tracepoint provider package source file
2579 A _tracepoint provider package source file_ is a C source file which
2580 includes a <<tpp-header,tracepoint provider header file>> to expand its
2581 macros into event serialization and other functions.
2583 You can always use the following tracepoint provider package source
2587 .Tracepoint provider package source file template.
2589 #define TRACEPOINT_CREATE_PROBES
2594 Replace `tp.h` with the name of your <<tpp-header,tracepoint provider
2595 header file>> name. You may also include more than one tracepoint
2596 provider header file here to create a tracepoint provider package
2597 holding more than one tracepoint providers.
2600 [[probing-the-application-source-code]]
2601 ==== Add tracepoints to an application's source code
2603 Once you <<tpp-header,create a tracepoint provider header file>>, you
2604 can use the `tracepoint()` macro in your application's
2605 source code to insert the tracepoints that this header
2606 <<defining-tracepoints,defines>>.
2608 The `tracepoint()` macro takes at least two parameters: the tracepoint
2609 provider name and the tracepoint name. The corresponding tracepoint
2610 definition defines the other parameters.
2612 .`tracepoint()` usage.
2614 The following <<defining-tracepoints,tracepoint definition>> defines a
2615 tracepoint which takes two input arguments and has two output event
2619 .Tracepoint provider header file.
2621 #include "my-custom-structure.h"
2628 const char*, cmd_name
2631 ctf_string(cmd_name, cmd_name)
2632 ctf_integer(int, number_of_args, argc)
2637 You can refer to this tracepoint definition with the `tracepoint()`
2638 macro in your application's source code like this:
2641 .Application's source file.
2645 int main(int argc, char* argv[])
2647 tracepoint(my_provider, my_tracepoint, argc, argv[0]);
2653 Note how the application's source code includes
2654 the tracepoint provider header file containing the tracepoint
2655 definitions to use, path:{tp.h}.
2658 .`tracepoint()` usage with a complex tracepoint definition.
2660 Consider this complex tracepoint definition, where multiple event
2661 fields refer to the same input arguments in their argument expression
2665 .Tracepoint provider header file.
2667 /* For `struct stat` */
2668 #include <sys/types.h>
2669 #include <sys/stat.h>
2681 ctf_integer(int, my_constant_field, 23 + 17)
2682 ctf_integer(int, my_int_arg_field, my_int_arg)
2683 ctf_integer(int, my_int_arg_field2, my_int_arg * my_int_arg)
2684 ctf_integer(int, sum4_field, my_str_arg[0] + my_str_arg[1] +
2685 my_str_arg[2] + my_str_arg[3])
2686 ctf_string(my_str_arg_field, my_str_arg)
2687 ctf_integer_hex(off_t, size_field, st->st_size)
2688 ctf_float(double, size_dbl_field, (double) st->st_size)
2689 ctf_sequence_text(char, half_my_str_arg_field, my_str_arg,
2690 size_t, strlen(my_str_arg) / 2)
2695 You can refer to this tracepoint definition with the `tracepoint()`
2696 macro in your application's source code like this:
2699 .Application's source file.
2701 #define TRACEPOINT_DEFINE
2708 stat("/etc/fstab", &s);
2709 tracepoint(my_provider, my_tracepoint, 23, "Hello, World!", &s);
2715 If you look at the event record that LTTng writes when tracing this
2716 program, assuming the file size of path:{/etc/fstab} is 301{nbsp}bytes,
2717 it should look like this:
2719 .Event record fields
2721 |Field's name |Field's value
2722 |`my_constant_field` |40
2723 |`my_int_arg_field` |23
2724 |`my_int_arg_field2` |529
2726 |`my_str_arg_field` |`Hello, World!`
2727 |`size_field` |0x12d
2728 |`size_dbl_field` |301.0
2729 |`half_my_str_arg_field` |`Hello,`
2733 Sometimes, the arguments you pass to `tracepoint()` are expensive to
2734 compute--they use the call stack, for example. To avoid this
2735 computation when the tracepoint is disabled, you can use the
2736 `tracepoint_enabled()` and `do_tracepoint()` macros.
2738 The syntax of the `tracepoint_enabled()` and `do_tracepoint()` macros
2742 .`tracepoint_enabled()` and `do_tracepoint()` macros syntax.
2744 tracepoint_enabled(provider_name, tracepoint_name)
2745 do_tracepoint(provider_name, tracepoint_name, ...)
2750 * `provider_name` with the tracepoint provider name.
2751 * `tracepoint_name` with the tracepoint name.
2753 `tracepoint_enabled()` returns a non-zero value if the tracepoint named
2754 `tracepoint_name` from the provider named `provider_name` is enabled
2757 `do_tracepoint()` is like `tracepoint()`, except that it doesn't check
2758 if the tracepoint is enabled. Using `tracepoint()` with
2759 `tracepoint_enabled()` is dangerous since `tracepoint()` also contains
2760 the `tracepoint_enabled()` check, thus a race condition is
2761 possible in this situation:
2764 .Possible race condition when using `tracepoint_enabled()` with `tracepoint()`.
2766 if (tracepoint_enabled(my_provider, my_tracepoint)) {
2767 stuff = prepare_stuff();
2770 tracepoint(my_provider, my_tracepoint, stuff);
2773 If the tracepoint is enabled after the condition, then `stuff` is not
2774 prepared: the emitted event will either contain wrong data, or the whole
2775 application could crash (segmentation fault, for example).
2777 NOTE: Neither `tracepoint_enabled()` nor `do_tracepoint()` have an
2778 `STAP_PROBEV()` call. If you need it, you must emit
2782 [[building-tracepoint-providers-and-user-application]]
2783 ==== Build and link a tracepoint provider package and an application
2785 Once you have one or more <<tpp-header,tracepoint provider header
2786 files>> and a <<tpp-source,tracepoint provider package source file>>,
2787 you can create the tracepoint provider package by compiling its source
2788 file. From here, multiple build and run scenarios are possible. The
2789 following table shows common application and library configurations
2790 along with the required command lines to achieve them.
2792 In the following diagrams, we use the following file names:
2795 Executable application.
2798 Application's object file.
2801 Tracepoint provider package object file.
2804 Tracepoint provider package archive file.
2807 Tracepoint provider package shared object file.
2810 User library object file.
2813 User library shared object file.
2815 We use the following symbols in the diagrams of table below:
2818 .Symbols used in the build scenario diagrams.
2819 image::ust-sit-symbols.png[]
2821 We assume that path:{.} is part of the env:LD_LIBRARY_PATH environment
2822 variable in the following instructions.
2824 [role="growable ust-scenarios",cols="asciidoc,asciidoc"]
2825 .Common tracepoint provider package scenarios.
2827 |Scenario |Instructions
2830 The instrumented application is statically linked with
2831 the tracepoint provider package object.
2833 image::ust-sit+app-linked-with-tp-o+app-instrumented.png[]
2836 include::../common/ust-sit-step-tp-o.txt[]
2838 To build the instrumented application:
2840 . In path:{app.c}, before including path:{tpp.h}, add the following line:
2845 #define TRACEPOINT_DEFINE
2849 . Compile the application source file:
2858 . Build the application:
2863 $ gcc -o app app.o tpp.o -llttng-ust -ldl
2867 To run the instrumented application:
2869 * Start the application:
2879 The instrumented application is statically linked with the
2880 tracepoint provider package archive file.
2882 image::ust-sit+app-linked-with-tp-a+app-instrumented.png[]
2885 To create the tracepoint provider package archive file:
2887 . Compile the <<tpp-source,tracepoint provider package source file>>:
2896 . Create the tracepoint provider package archive file:
2901 $ ar rcs tpp.a tpp.o
2905 To build the instrumented application:
2907 . In path:{app.c}, before including path:{tpp.h}, add the following line:
2912 #define TRACEPOINT_DEFINE
2916 . Compile the application source file:
2925 . Build the application:
2930 $ gcc -o app app.o tpp.a -llttng-ust -ldl
2934 To run the instrumented application:
2936 * Start the application:
2946 The instrumented application is linked with the tracepoint provider
2947 package shared object.
2949 image::ust-sit+app-linked-with-tp-so+app-instrumented.png[]
2952 include::../common/ust-sit-step-tp-so.txt[]
2954 To build the instrumented application:
2956 . In path:{app.c}, before including path:{tpp.h}, add the following line:
2961 #define TRACEPOINT_DEFINE
2965 . Compile the application source file:
2974 . Build the application:
2979 $ gcc -o app app.o -ldl -L. -ltpp
2983 To run the instrumented application:
2985 * Start the application:
2995 The tracepoint provider package shared object is preloaded before the
2996 instrumented application starts.
2998 image::ust-sit+tp-so-preloaded+app-instrumented.png[]
3001 include::../common/ust-sit-step-tp-so.txt[]
3003 To build the instrumented application:
3005 . In path:{app.c}, before including path:{tpp.h}, add the
3011 #define TRACEPOINT_DEFINE
3012 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3016 . Compile the application source file:
3025 . Build the application:
3030 $ gcc -o app app.o -ldl
3034 To run the instrumented application with tracing support:
3036 * Preload the tracepoint provider package shared object and
3037 start the application:
3042 $ LD_PRELOAD=./libtpp.so ./app
3046 To run the instrumented application without tracing support:
3048 * Start the application:
3058 The instrumented application dynamically loads the tracepoint provider
3059 package shared object.
3061 image::ust-sit+app-dlopens-tp-so+app-instrumented.png[]
3064 include::../common/ust-sit-step-tp-so.txt[]
3066 To build the instrumented application:
3068 . In path:{app.c}, before including path:{tpp.h}, add the
3074 #define TRACEPOINT_DEFINE
3075 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3079 . Compile the application source file:
3088 . Build the application:
3093 $ gcc -o app app.o -ldl
3097 To run the instrumented application:
3099 * Start the application:
3109 The application is linked with the instrumented user library.
3111 The instrumented user library is statically linked with the tracepoint
3112 provider package object file.
3114 image::ust-sit+app-linked-with-lib+lib-linked-with-tp-o+lib-instrumented.png[]
3117 include::../common/ust-sit-step-tp-o-fpic.txt[]
3119 To build the instrumented user library:
3121 . In path:{emon.c}, before including path:{tpp.h}, add the
3127 #define TRACEPOINT_DEFINE
3131 . Compile the user library source file:
3136 $ gcc -I. -fpic -c emon.c
3140 . Build the user library shared object:
3145 $ gcc -shared -o libemon.so emon.o tpp.o -llttng-ust -ldl
3149 To build the application:
3151 . Compile the application source file:
3160 . Build the application:
3165 $ gcc -o app app.o -L. -lemon
3169 To run the application:
3171 * Start the application:
3181 The application is linked with the instrumented user library.
3183 The instrumented user library is linked with the tracepoint provider
3184 package shared object.
3186 image::ust-sit+app-linked-with-lib+lib-linked-with-tp-so+lib-instrumented.png[]
3189 include::../common/ust-sit-step-tp-so.txt[]
3191 To build the instrumented user library:
3193 . In path:{emon.c}, before including path:{tpp.h}, add the
3199 #define TRACEPOINT_DEFINE
3203 . Compile the user library source file:
3208 $ gcc -I. -fpic -c emon.c
3212 . Build the user library shared object:
3217 $ gcc -shared -o libemon.so emon.o -ldl -L. -ltpp
3221 To build the application:
3223 . Compile the application source file:
3232 . Build the application:
3237 $ gcc -o app app.o -L. -lemon
3241 To run the application:
3243 * Start the application:
3253 The tracepoint provider package shared object is preloaded before the
3256 The application is linked with the instrumented user library.
3258 image::ust-sit+tp-so-preloaded+app-linked-with-lib+lib-instrumented.png[]
3261 include::../common/ust-sit-step-tp-so.txt[]
3263 To build the instrumented user library:
3265 . In path:{emon.c}, before including path:{tpp.h}, add the
3271 #define TRACEPOINT_DEFINE
3272 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3276 . Compile the user library source file:
3281 $ gcc -I. -fpic -c emon.c
3285 . Build the user library shared object:
3290 $ gcc -shared -o libemon.so emon.o -ldl
3294 To build the application:
3296 . Compile the application source file:
3305 . Build the application:
3310 $ gcc -o app app.o -L. -lemon
3314 To run the application with tracing support:
3316 * Preload the tracepoint provider package shared object and
3317 start the application:
3322 $ LD_PRELOAD=./libtpp.so ./app
3326 To run the application without tracing support:
3328 * Start the application:
3338 The application is linked with the instrumented user library.
3340 The instrumented user library dynamically loads the tracepoint provider
3341 package shared object.
3343 image::ust-sit+app-linked-with-lib+lib-dlopens-tp-so+lib-instrumented.png[]
3346 include::../common/ust-sit-step-tp-so.txt[]
3348 To build the instrumented user library:
3350 . In path:{emon.c}, before including path:{tpp.h}, add the
3356 #define TRACEPOINT_DEFINE
3357 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3361 . Compile the user library source file:
3366 $ gcc -I. -fpic -c emon.c
3370 . Build the user library shared object:
3375 $ gcc -shared -o libemon.so emon.o -ldl
3379 To build the application:
3381 . Compile the application source file:
3390 . Build the application:
3395 $ gcc -o app app.o -L. -lemon
3399 To run the application:
3401 * Start the application:
3411 The application dynamically loads the instrumented user library.
3413 The instrumented user library is linked with the tracepoint provider
3414 package shared object.
3416 image::ust-sit+app-dlopens-lib+lib-linked-with-tp-so+lib-instrumented.png[]
3419 include::../common/ust-sit-step-tp-so.txt[]
3421 To build the instrumented user library:
3423 . In path:{emon.c}, before including path:{tpp.h}, add the
3429 #define TRACEPOINT_DEFINE
3433 . Compile the user library source file:
3438 $ gcc -I. -fpic -c emon.c
3442 . Build the user library shared object:
3447 $ gcc -shared -o libemon.so emon.o -ldl -L. -ltpp
3451 To build the application:
3453 . Compile the application source file:
3462 . Build the application:
3467 $ gcc -o app app.o -ldl -L. -lemon
3471 To run the application:
3473 * Start the application:
3483 The application dynamically loads the instrumented user library.
3485 The instrumented user library dynamically loads the tracepoint provider
3486 package shared object.
3488 image::ust-sit+app-dlopens-lib+lib-dlopens-tp-so+lib-instrumented.png[]
3491 include::../common/ust-sit-step-tp-so.txt[]
3493 To build the instrumented user library:
3495 . In path:{emon.c}, before including path:{tpp.h}, add the
3501 #define TRACEPOINT_DEFINE
3502 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3506 . Compile the user library source file:
3511 $ gcc -I. -fpic -c emon.c
3515 . Build the user library shared object:
3520 $ gcc -shared -o libemon.so emon.o -ldl
3524 To build the application:
3526 . Compile the application source file:
3535 . Build the application:
3540 $ gcc -o app app.o -ldl -L. -lemon
3544 To run the application:
3546 * Start the application:
3556 The tracepoint provider package shared object is preloaded before the
3559 The application dynamically loads the instrumented user library.
3561 image::ust-sit+tp-so-preloaded+app-dlopens-lib+lib-instrumented.png[]
3564 include::../common/ust-sit-step-tp-so.txt[]
3566 To build the instrumented user library:
3568 . In path:{emon.c}, before including path:{tpp.h}, add the
3574 #define TRACEPOINT_DEFINE
3575 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3579 . Compile the user library source file:
3584 $ gcc -I. -fpic -c emon.c
3588 . Build the user library shared object:
3593 $ gcc -shared -o libemon.so emon.o -ldl
3597 To build the application:
3599 . Compile the application source file:
3608 . Build the application:
3613 $ gcc -o app app.o -L. -lemon
3617 To run the application with tracing support:
3619 * Preload the tracepoint provider package shared object and
3620 start the application:
3625 $ LD_PRELOAD=./libtpp.so ./app
3629 To run the application without tracing support:
3631 * Start the application:
3641 The application is statically linked with the tracepoint provider
3642 package object file.
3644 The application is linked with the instrumented user library.
3646 image::ust-sit+app-linked-with-tp-o+app-linked-with-lib+lib-instrumented.png[]
3649 include::../common/ust-sit-step-tp-o.txt[]
3651 To build the instrumented user library:
3653 . In path:{emon.c}, before including path:{tpp.h}, add the
3659 #define TRACEPOINT_DEFINE
3663 . Compile the user library source file:
3668 $ gcc -I. -fpic -c emon.c
3672 . Build the user library shared object:
3677 $ gcc -shared -o libemon.so emon.o
3681 To build the application:
3683 . Compile the application source file:
3692 . Build the application:
3697 $ gcc -o app app.o tpp.o -llttng-ust -ldl -L. -lemon
3701 To run the instrumented application:
3703 * Start the application:
3713 The application is statically linked with the tracepoint provider
3714 package object file.
3716 The application dynamically loads the instrumented user library.
3718 image::ust-sit+app-linked-with-tp-o+app-dlopens-lib+lib-instrumented.png[]
3721 include::../common/ust-sit-step-tp-o.txt[]
3723 To build the application:
3725 . In path:{app.c}, before including path:{tpp.h}, add the following line:
3730 #define TRACEPOINT_DEFINE
3734 . Compile the application source file:
3743 . Build the application:
3748 $ gcc -Wl,--export-dynamic -o app app.o tpp.o \
3753 The `--export-dynamic` option passed to the linker is necessary for the
3754 dynamically loaded library to ``see'' the tracepoint symbols defined in
3757 To build the instrumented user library:
3759 . Compile the user library source file:
3764 $ gcc -I. -fpic -c emon.c
3768 . Build the user library shared object:
3773 $ gcc -shared -o libemon.so emon.o
3777 To run the application:
3779 * Start the application:
3790 [[using-lttng-ust-with-daemons]]
3791 ===== Use noch:{LTTng-UST} with daemons
3793 If your instrumented application calls man:fork(2), man:clone(2),
3794 or BSD's man:rfork(2), without a following man:exec(3)-family
3795 system call, you must preload the path:{liblttng-ust-fork.so} shared
3796 object when you start the application.
3800 $ LD_PRELOAD=liblttng-ust-fork.so ./my-app
3803 If your tracepoint provider package is
3804 a shared library which you also preload, you must put both
3805 shared objects in env:LD_PRELOAD:
3809 $ LD_PRELOAD=liblttng-ust-fork.so:/path/to/tp.so ./my-app
3815 ===== Use noch:{LTTng-UST} with applications which close file descriptors that don't belong to them
3817 If your instrumented application closes one or more file descriptors
3818 which it did not open itself, you must preload the
3819 path:{liblttng-ust-fd.so} shared object when you start the application:
3823 $ LD_PRELOAD=liblttng-ust-fd.so ./my-app
3826 Typical use cases include closing all the file descriptors after
3827 man:fork(2) or man:rfork(2) and buggy applications doing
3831 [[lttng-ust-pkg-config]]
3832 ===== Use noch:{pkg-config}
3834 On some distributions, LTTng-UST ships with a
3835 https://www.freedesktop.org/wiki/Software/pkg-config/[pkg-config]
3836 metadata file. If this is your case, then you can use cmd:pkg-config to
3837 build an application on the command line:
3841 $ gcc -o my-app my-app.o tp.o $(pkg-config --cflags --libs lttng-ust)
3845 [[instrumenting-32-bit-app-on-64-bit-system]]
3846 ===== [[advanced-instrumenting-techniques]]Build a 32-bit instrumented application for a 64-bit target system
3848 In order to trace a 32-bit application running on a 64-bit system,
3849 LTTng must use a dedicated 32-bit
3850 <<lttng-consumerd,consumer daemon>>.
3852 The following steps show how to build and install a 32-bit consumer
3853 daemon, which is _not_ part of the default 64-bit LTTng build, how to
3854 build and install the 32-bit LTTng-UST libraries, and how to build and
3855 link an instrumented 32-bit application in that context.
3857 To build a 32-bit instrumented application for a 64-bit target system,
3858 assuming you have a fresh target system with no installed Userspace RCU
3861 . Download, build, and install a 32-bit version of Userspace RCU:
3866 $ cd $(mktemp -d) &&
3867 wget http://lttng.org/files/urcu/userspace-rcu-latest-0.9.tar.bz2 &&
3868 tar -xf userspace-rcu-latest-0.9.tar.bz2 &&
3869 cd userspace-rcu-0.9.* &&
3870 ./configure --libdir=/usr/local/lib32 CFLAGS=-m32 &&
3872 sudo make install &&
3877 . Using your distribution's package manager, or from source, install
3878 the following 32-bit versions of the following dependencies of
3879 LTTng-tools and LTTng-UST:
3882 * https://sourceforge.net/projects/libuuid/[libuuid]
3883 * http://directory.fsf.org/wiki/Popt[popt]
3884 * http://www.xmlsoft.org/[libxml2]
3887 . Download, build, and install a 32-bit version of the latest
3888 LTTng-UST{nbsp}{revision}:
3893 $ cd $(mktemp -d) &&
3894 wget http://lttng.org/files/lttng-ust/lttng-ust-latest-2.11.tar.bz2 &&
3895 tar -xf lttng-ust-latest-2.11.tar.bz2 &&
3896 cd lttng-ust-2.11.* &&
3897 ./configure --libdir=/usr/local/lib32 \
3898 CFLAGS=-m32 CXXFLAGS=-m32 \
3899 LDFLAGS='-L/usr/local/lib32 -L/usr/lib32' &&
3901 sudo make install &&
3908 Depending on your distribution,
3909 32-bit libraries could be installed at a different location than
3910 `/usr/lib32`. For example, Debian is known to install
3911 some 32-bit libraries in `/usr/lib/i386-linux-gnu`.
3913 In this case, make sure to set `LDFLAGS` to all the
3914 relevant 32-bit library paths, for example:
3918 $ LDFLAGS='-L/usr/lib/i386-linux-gnu -L/usr/lib32'
3922 . Download the latest LTTng-tools{nbsp}{revision}, build, and install
3923 the 32-bit consumer daemon:
3928 $ cd $(mktemp -d) &&
3929 wget http://lttng.org/files/lttng-tools/lttng-tools-latest-2.11.tar.bz2 &&
3930 tar -xf lttng-tools-latest-2.11.tar.bz2 &&
3931 cd lttng-tools-2.11.* &&
3932 ./configure --libdir=/usr/local/lib32 CFLAGS=-m32 CXXFLAGS=-m32 \
3933 LDFLAGS='-L/usr/local/lib32 -L/usr/lib32' \
3934 --disable-bin-lttng --disable-bin-lttng-crash \
3935 --disable-bin-lttng-relayd --disable-bin-lttng-sessiond &&
3937 cd src/bin/lttng-consumerd &&
3938 sudo make install &&
3943 . From your distribution or from source,
3944 <<installing-lttng,install>> the 64-bit versions of
3945 LTTng-UST and Userspace RCU.
3946 . Download, build, and install the 64-bit version of the
3947 latest LTTng-tools{nbsp}{revision}:
3952 $ cd $(mktemp -d) &&
3953 wget http://lttng.org/files/lttng-tools/lttng-tools-latest-2.11.tar.bz2 &&
3954 tar -xf lttng-tools-latest-2.11.tar.bz2 &&
3955 cd lttng-tools-2.11.* &&
3956 ./configure --with-consumerd32-libdir=/usr/local/lib32 \
3957 --with-consumerd32-bin=/usr/local/lib32/lttng/libexec/lttng-consumerd &&
3959 sudo make install &&
3964 . Pass the following options to man:gcc(1), man:g++(1), or man:clang(1)
3965 when linking your 32-bit application:
3968 -m32 -L/usr/lib32 -L/usr/local/lib32 \
3969 -Wl,-rpath,/usr/lib32,-rpath,/usr/local/lib32
3972 For example, let's rebuild the quick start example in
3973 <<tracing-your-own-user-application,Trace a user application>> as an
3974 instrumented 32-bit application:
3979 $ gcc -m32 -c -I. hello-tp.c
3980 $ gcc -m32 -c hello.c
3981 $ gcc -m32 -o hello hello.o hello-tp.o \
3982 -L/usr/lib32 -L/usr/local/lib32 \
3983 -Wl,-rpath,/usr/lib32,-rpath,/usr/local/lib32 \
3988 No special action is required to execute the 32-bit application and
3989 to trace it: use the command-line man:lttng(1) tool as usual.
3996 man:tracef(3) is a small LTTng-UST API designed for quick,
3997 man:printf(3)-like instrumentation without the burden of
3998 <<tracepoint-provider,creating>> and
3999 <<building-tracepoint-providers-and-user-application,building>>
4000 a tracepoint provider package.
4002 To use `tracef()` in your application:
4004 . In the C or C++ source files where you need to use `tracef()`,
4005 include `<lttng/tracef.h>`:
4010 #include <lttng/tracef.h>
4014 . In the application's source code, use `tracef()` like you would use
4022 tracef("my message: %d (%s)", my_integer, my_string);
4028 . Link your application with `liblttng-ust`:
4033 $ gcc -o app app.c -llttng-ust
4037 To trace the events that `tracef()` calls emit:
4039 * <<enabling-disabling-events,Create an event rule>> which matches the
4040 `lttng_ust_tracef:*` event name:
4045 $ lttng enable-event --userspace 'lttng_ust_tracef:*'
4050 .Limitations of `tracef()`
4052 The `tracef()` utility function was developed to make user space tracing
4053 super simple, albeit with notable disadvantages compared to
4054 <<defining-tracepoints,user-defined tracepoints>>:
4056 * All the emitted events have the same tracepoint provider and
4057 tracepoint names, respectively `lttng_ust_tracef` and `event`.
4058 * There is no static type checking.
4059 * The only event record field you actually get, named `msg`, is a string
4060 potentially containing the values you passed to `tracef()`
4061 using your own format string. This also means that you cannot filter
4062 events with a custom expression at run time because there are no
4064 * Since `tracef()` uses the C standard library's man:vasprintf(3)
4065 function behind the scenes to format the strings at run time, its
4066 expected performance is lower than with user-defined tracepoints,
4067 which do not require a conversion to a string.
4069 Taking this into consideration, `tracef()` is useful for some quick
4070 prototyping and debugging, but you should not consider it for any
4071 permanent and serious applicative instrumentation.
4077 ==== Use `tracelog()`
4079 The man:tracelog(3) API is very similar to <<tracef,`tracef()`>>, with
4080 the difference that it accepts an additional log level parameter.
4082 The goal of `tracelog()` is to ease the migration from logging to
4085 To use `tracelog()` in your application:
4087 . In the C or C++ source files where you need to use `tracelog()`,
4088 include `<lttng/tracelog.h>`:
4093 #include <lttng/tracelog.h>
4097 . In the application's source code, use `tracelog()` like you would use
4098 man:printf(3), except for the first parameter which is the log
4106 tracelog(TRACE_WARNING, "my message: %d (%s)",
4107 my_integer, my_string);
4113 See man:lttng-ust(3) for a list of available log level names.
4115 . Link your application with `liblttng-ust`:
4120 $ gcc -o app app.c -llttng-ust
4124 To trace the events that `tracelog()` calls emit with a log level
4125 _as severe as_ a specific log level:
4127 * <<enabling-disabling-events,Create an event rule>> which matches the
4128 `lttng_ust_tracelog:*` event name and a minimum level
4134 $ lttng enable-event --userspace 'lttng_ust_tracelog:*'
4135 --loglevel=TRACE_WARNING
4139 To trace the events that `tracelog()` calls emit with a
4140 _specific log level_:
4142 * Create an event rule which matches the `lttng_ust_tracelog:*`
4143 event name and a specific log level:
4148 $ lttng enable-event --userspace 'lttng_ust_tracelog:*'
4149 --loglevel-only=TRACE_INFO
4154 [[prebuilt-ust-helpers]]
4155 === Prebuilt user space tracing helpers
4157 The LTTng-UST package provides a few helpers in the form or preloadable
4158 shared objects which automatically instrument system functions and
4161 The helper shared objects are normally found in dir:{/usr/lib}. If you
4162 built LTTng-UST <<building-from-source,from source>>, they are probably
4163 located in dir:{/usr/local/lib}.
4165 The installed user space tracing helpers in LTTng-UST{nbsp}{revision}
4168 path:{liblttng-ust-libc-wrapper.so}::
4169 path:{liblttng-ust-pthread-wrapper.so}::
4170 <<liblttng-ust-libc-pthread-wrapper,C{nbsp}standard library
4171 memory and POSIX threads function tracing>>.
4173 path:{liblttng-ust-cyg-profile.so}::
4174 path:{liblttng-ust-cyg-profile-fast.so}::
4175 <<liblttng-ust-cyg-profile,Function entry and exit tracing>>.
4177 path:{liblttng-ust-dl.so}::
4178 <<liblttng-ust-dl,Dynamic linker tracing>>.
4180 To use a user space tracing helper with any user application:
4182 * Preload the helper shared object when you start the application:
4187 $ LD_PRELOAD=liblttng-ust-libc-wrapper.so my-app
4191 You can preload more than one helper:
4196 $ LD_PRELOAD=liblttng-ust-libc-wrapper.so:liblttng-ust-dl.so my-app
4202 [[liblttng-ust-libc-pthread-wrapper]]
4203 ==== Instrument C standard library memory and POSIX threads functions
4205 The path:{liblttng-ust-libc-wrapper.so} and
4206 path:{liblttng-ust-pthread-wrapper.so} helpers
4207 add instrumentation to some C standard library and POSIX
4211 .Functions instrumented by preloading path:{liblttng-ust-libc-wrapper.so}.
4213 |TP provider name |TP name |Instrumented function
4215 .6+|`lttng_ust_libc` |`malloc` |man:malloc(3)
4216 |`calloc` |man:calloc(3)
4217 |`realloc` |man:realloc(3)
4218 |`free` |man:free(3)
4219 |`memalign` |man:memalign(3)
4220 |`posix_memalign` |man:posix_memalign(3)
4224 .Functions instrumented by preloading path:{liblttng-ust-pthread-wrapper.so}.
4226 |TP provider name |TP name |Instrumented function
4228 .4+|`lttng_ust_pthread` |`pthread_mutex_lock_req` |man:pthread_mutex_lock(3p) (request time)
4229 |`pthread_mutex_lock_acq` |man:pthread_mutex_lock(3p) (acquire time)
4230 |`pthread_mutex_trylock` |man:pthread_mutex_trylock(3p)
4231 |`pthread_mutex_unlock` |man:pthread_mutex_unlock(3p)
4234 When you preload the shared object, it replaces the functions listed
4235 in the previous tables by wrappers which contain tracepoints and call
4236 the replaced functions.
4239 [[liblttng-ust-cyg-profile]]
4240 ==== Instrument function entry and exit
4242 The path:{liblttng-ust-cyg-profile*.so} helpers can add instrumentation
4243 to the entry and exit points of functions.
4245 man:gcc(1) and man:clang(1) have an option named
4246 https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html[`-finstrument-functions`]
4247 which generates instrumentation calls for entry and exit to functions.
4248 The LTTng-UST function tracing helpers,
4249 path:{liblttng-ust-cyg-profile.so} and
4250 path:{liblttng-ust-cyg-profile-fast.so}, take advantage of this feature
4251 to add tracepoints to the two generated functions (which contain
4252 `cyg_profile` in their names, hence the helper's name).
4254 To use the LTTng-UST function tracing helper, the source files to
4255 instrument must be built using the `-finstrument-functions` compiler
4258 There are two versions of the LTTng-UST function tracing helper:
4260 * **path:{liblttng-ust-cyg-profile-fast.so}** is a lightweight variant
4261 that you should only use when it can be _guaranteed_ that the
4262 complete event stream is recorded without any lost event record.
4263 Any kind of duplicate information is left out.
4265 Assuming no event record is lost, having only the function addresses on
4266 entry is enough to create a call graph, since an event record always
4267 contains the ID of the CPU that generated it.
4269 You can use a tool like man:addr2line(1) to convert function addresses
4270 back to source file names and line numbers.
4272 * **path:{liblttng-ust-cyg-profile.so}** is a more robust variant
4273 which also works in use cases where event records might get discarded or
4274 not recorded from application startup.
4275 In these cases, the trace analyzer needs more information to be
4276 able to reconstruct the program flow.
4278 See man:lttng-ust-cyg-profile(3) to learn more about the instrumentation
4279 points of this helper.
4281 All the tracepoints that this helper provides have the
4282 log level `TRACE_DEBUG_FUNCTION` (see man:lttng-ust(3)).
4284 TIP: It's sometimes a good idea to limit the number of source files that
4285 you compile with the `-finstrument-functions` option to prevent LTTng
4286 from writing an excessive amount of trace data at run time. When using
4287 man:gcc(1), you can use the
4288 `-finstrument-functions-exclude-function-list` option to avoid
4289 instrument entries and exits of specific function names.
4294 ==== Instrument the dynamic linker
4296 The path:{liblttng-ust-dl.so} helper adds instrumentation to the
4297 man:dlopen(3) and man:dlclose(3) function calls.
4299 See man:lttng-ust-dl(3) to learn more about the instrumentation points
4304 [[java-application]]
4305 === User space Java agent
4307 You can instrument any Java application which uses one of the following
4310 * The https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[**`java.util.logging`**]
4311 (JUL) core logging facilities.
4312 * http://logging.apache.org/log4j/1.2/[**Apache log4j{nbsp}1.2**], since
4313 LTTng{nbsp}2.6. Note that Apache Log4j{nbsp}2 is not supported.
4316 .LTTng-UST Java agent imported by a Java application.
4317 image::java-app.png[]
4319 Note that the methods described below are new in LTTng{nbsp}2.8.
4320 Previous LTTng versions use another technique.
4322 NOTE: We use http://openjdk.java.net/[OpenJDK]{nbsp}8 for development
4323 and https://ci.lttng.org/[continuous integration], thus this version is
4324 directly supported. However, the LTTng-UST Java agent is also tested
4325 with OpenJDK{nbsp}7.
4330 ==== Use the LTTng-UST Java agent for `java.util.logging`
4332 To use the LTTng-UST Java agent in a Java application which uses
4333 `java.util.logging` (JUL):
4335 . In the Java application's source code, import the LTTng-UST
4336 log handler package for `java.util.logging`:
4341 import org.lttng.ust.agent.jul.LttngLogHandler;
4345 . Create an LTTng-UST JUL log handler:
4350 Handler lttngUstLogHandler = new LttngLogHandler();
4354 . Add this handler to the JUL loggers which should emit LTTng events:
4359 Logger myLogger = Logger.getLogger("some-logger");
4361 myLogger.addHandler(lttngUstLogHandler);
4365 . Use `java.util.logging` log statements and configuration as usual.
4366 The loggers with an attached LTTng-UST log handler can emit
4369 . Before exiting the application, remove the LTTng-UST log handler from
4370 the loggers attached to it and call its `close()` method:
4375 myLogger.removeHandler(lttngUstLogHandler);
4376 lttngUstLogHandler.close();
4380 This is not strictly necessary, but it is recommended for a clean
4381 disposal of the handler's resources.
4383 . Include the LTTng-UST Java agent's common and JUL-specific JAR files,
4384 path:{lttng-ust-agent-common.jar} and path:{lttng-ust-agent-jul.jar},
4386 https://docs.oracle.com/javase/tutorial/essential/environment/paths.html[class
4387 path] when you build the Java application.
4389 The JAR files are typically located in dir:{/usr/share/java}.
4391 IMPORTANT: The LTTng-UST Java agent must be
4392 <<installing-lttng,installed>> for the logging framework your
4395 .Use the LTTng-UST Java agent for `java.util.logging`.
4400 import java.io.IOException;
4401 import java.util.logging.Handler;
4402 import java.util.logging.Logger;
4403 import org.lttng.ust.agent.jul.LttngLogHandler;
4407 private static final int answer = 42;
4409 public static void main(String[] argv) throws Exception
4412 Logger logger = Logger.getLogger("jello");
4414 // Create an LTTng-UST log handler
4415 Handler lttngUstLogHandler = new LttngLogHandler();
4417 // Add the LTTng-UST log handler to our logger
4418 logger.addHandler(lttngUstLogHandler);
4421 logger.info("some info");
4422 logger.warning("some warning");
4424 logger.finer("finer information; the answer is " + answer);
4426 logger.severe("error!");
4428 // Not mandatory, but cleaner
4429 logger.removeHandler(lttngUstLogHandler);
4430 lttngUstLogHandler.close();
4439 $ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar Test.java
4442 <<creating-destroying-tracing-sessions,Create a tracing session>>,
4443 <<enabling-disabling-events,create an event rule>> matching the
4444 `jello` JUL logger, and <<basic-tracing-session-control,start tracing>>:
4449 $ lttng enable-event --jul jello
4453 Run the compiled class:
4457 $ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar:. Test
4460 <<basic-tracing-session-control,Stop tracing>> and inspect the
4470 In the resulting trace, an <<event,event record>> generated by a Java
4471 application using `java.util.logging` is named `lttng_jul:event` and
4472 has the following fields:
4475 Log record's message.
4481 Name of the class in which the log statement was executed.
4484 Name of the method in which the log statement was executed.
4487 Logging time (timestamp in milliseconds).
4490 Log level integer value.
4493 ID of the thread in which the log statement was executed.
4495 You can use the opt:lttng-enable-event(1):--loglevel or
4496 opt:lttng-enable-event(1):--loglevel-only option of the
4497 man:lttng-enable-event(1) command to target a range of JUL log levels
4498 or a specific JUL log level.
4503 ==== Use the LTTng-UST Java agent for Apache log4j
4505 To use the LTTng-UST Java agent in a Java application which uses
4506 Apache log4j{nbsp}1.2:
4508 . In the Java application's source code, import the LTTng-UST
4509 log appender package for Apache log4j:
4514 import org.lttng.ust.agent.log4j.LttngLogAppender;
4518 . Create an LTTng-UST log4j log appender:
4523 Appender lttngUstLogAppender = new LttngLogAppender();
4527 . Add this appender to the log4j loggers which should emit LTTng events:
4532 Logger myLogger = Logger.getLogger("some-logger");
4534 myLogger.addAppender(lttngUstLogAppender);
4538 . Use Apache log4j log statements and configuration as usual. The
4539 loggers with an attached LTTng-UST log appender can emit LTTng events.
4541 . Before exiting the application, remove the LTTng-UST log appender from
4542 the loggers attached to it and call its `close()` method:
4547 myLogger.removeAppender(lttngUstLogAppender);
4548 lttngUstLogAppender.close();
4552 This is not strictly necessary, but it is recommended for a clean
4553 disposal of the appender's resources.
4555 . Include the LTTng-UST Java agent's common and log4j-specific JAR
4556 files, path:{lttng-ust-agent-common.jar} and
4557 path:{lttng-ust-agent-log4j.jar}, in the
4558 https://docs.oracle.com/javase/tutorial/essential/environment/paths.html[class
4559 path] when you build the Java application.
4561 The JAR files are typically located in dir:{/usr/share/java}.
4563 IMPORTANT: The LTTng-UST Java agent must be
4564 <<installing-lttng,installed>> for the logging framework your
4567 .Use the LTTng-UST Java agent for Apache log4j.
4572 import org.apache.log4j.Appender;
4573 import org.apache.log4j.Logger;
4574 import org.lttng.ust.agent.log4j.LttngLogAppender;
4578 private static final int answer = 42;
4580 public static void main(String[] argv) throws Exception
4583 Logger logger = Logger.getLogger("jello");
4585 // Create an LTTng-UST log appender
4586 Appender lttngUstLogAppender = new LttngLogAppender();
4588 // Add the LTTng-UST log appender to our logger
4589 logger.addAppender(lttngUstLogAppender);
4592 logger.info("some info");
4593 logger.warn("some warning");
4595 logger.debug("debug information; the answer is " + answer);
4597 logger.fatal("error!");
4599 // Not mandatory, but cleaner
4600 logger.removeAppender(lttngUstLogAppender);
4601 lttngUstLogAppender.close();
4607 Build this example (`$LOG4JPATH` is the path to the Apache log4j JAR
4612 $ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-log4j.jar:$LOG4JPATH Test.java
4615 <<creating-destroying-tracing-sessions,Create a tracing session>>,
4616 <<enabling-disabling-events,create an event rule>> matching the
4617 `jello` log4j logger, and <<basic-tracing-session-control,start tracing>>:
4622 $ lttng enable-event --log4j jello
4626 Run the compiled class:
4630 $ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-log4j.jar:$LOG4JPATH:. Test
4633 <<basic-tracing-session-control,Stop tracing>> and inspect the
4643 In the resulting trace, an <<event,event record>> generated by a Java
4644 application using log4j is named `lttng_log4j:event` and
4645 has the following fields:
4648 Log record's message.
4654 Name of the class in which the log statement was executed.
4657 Name of the method in which the log statement was executed.
4660 Name of the file in which the executed log statement is located.
4663 Line number at which the log statement was executed.
4669 Log level integer value.
4672 Name of the Java thread in which the log statement was executed.
4674 You can use the opt:lttng-enable-event(1):--loglevel or
4675 opt:lttng-enable-event(1):--loglevel-only option of the
4676 man:lttng-enable-event(1) command to target a range of Apache log4j log levels
4677 or a specific log4j log level.
4681 [[java-application-context]]
4682 ==== Provide application-specific context fields in a Java application
4684 A Java application-specific context field is a piece of state provided
4685 by the application which <<adding-context,you can add>>, using the
4686 man:lttng-add-context(1) command, to each <<event,event record>>
4687 produced by the log statements of this application.
4689 For example, a given object might have a current request ID variable.
4690 You can create a context information retriever for this object and
4691 assign a name to this current request ID. You can then, using the
4692 man:lttng-add-context(1) command, add this context field by name to
4693 the JUL or log4j <<channel,channel>>.
4695 To provide application-specific context fields in a Java application:
4697 . In the Java application's source code, import the LTTng-UST
4698 Java agent context classes and interfaces:
4703 import org.lttng.ust.agent.context.ContextInfoManager;
4704 import org.lttng.ust.agent.context.IContextInfoRetriever;
4708 . Create a context information retriever class, that is, a class which
4709 implements the `IContextInfoRetriever` interface:
4714 class MyContextInfoRetriever implements IContextInfoRetriever
4717 public Object retrieveContextInfo(String key)
4719 if (key.equals("intCtx")) {
4721 } else if (key.equals("strContext")) {
4722 return "context value!";
4731 This `retrieveContextInfo()` method is the only member of the
4732 `IContextInfoRetriever` interface. Its role is to return the current
4733 value of a state by name to create a context field. The names of the
4734 context fields and which state variables they return depends on your
4737 All primitive types and objects are supported as context fields.
4738 When `retrieveContextInfo()` returns an object, the context field
4739 serializer calls its `toString()` method to add a string field to
4740 event records. The method can also return `null`, which means that
4741 no context field is available for the required name.
4743 . Register an instance of your context information retriever class to
4744 the context information manager singleton:
4749 IContextInfoRetriever cir = new MyContextInfoRetriever();
4750 ContextInfoManager cim = ContextInfoManager.getInstance();
4751 cim.registerContextInfoRetriever("retrieverName", cir);
4755 . Before exiting the application, remove your context information
4756 retriever from the context information manager singleton:
4761 ContextInfoManager cim = ContextInfoManager.getInstance();
4762 cim.unregisterContextInfoRetriever("retrieverName");
4766 This is not strictly necessary, but it is recommended for a clean
4767 disposal of some manager's resources.
4769 . Build your Java application with LTTng-UST Java agent support as
4770 usual, following the procedure for either the <<jul,JUL>> or
4771 <<log4j,Apache log4j>> framework.
4774 .Provide application-specific context fields in a Java application.
4779 import java.util.logging.Handler;
4780 import java.util.logging.Logger;
4781 import org.lttng.ust.agent.jul.LttngLogHandler;
4782 import org.lttng.ust.agent.context.ContextInfoManager;
4783 import org.lttng.ust.agent.context.IContextInfoRetriever;
4787 // Our context information retriever class
4788 private static class MyContextInfoRetriever
4789 implements IContextInfoRetriever
4792 public Object retrieveContextInfo(String key) {
4793 if (key.equals("intCtx")) {
4795 } else if (key.equals("strContext")) {
4796 return "context value!";
4803 private static final int answer = 42;
4805 public static void main(String args[]) throws Exception
4807 // Get the context information manager instance
4808 ContextInfoManager cim = ContextInfoManager.getInstance();
4810 // Create and register our context information retriever
4811 IContextInfoRetriever cir = new MyContextInfoRetriever();
4812 cim.registerContextInfoRetriever("myRetriever", cir);
4815 Logger logger = Logger.getLogger("jello");
4817 // Create an LTTng-UST log handler
4818 Handler lttngUstLogHandler = new LttngLogHandler();
4820 // Add the LTTng-UST log handler to our logger
4821 logger.addHandler(lttngUstLogHandler);
4824 logger.info("some info");
4825 logger.warning("some warning");
4827 logger.finer("finer information; the answer is " + answer);
4829 logger.severe("error!");
4831 // Not mandatory, but cleaner
4832 logger.removeHandler(lttngUstLogHandler);
4833 lttngUstLogHandler.close();
4834 cim.unregisterContextInfoRetriever("myRetriever");
4843 $ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar Test.java
4846 <<creating-destroying-tracing-sessions,Create a tracing session>>
4847 and <<enabling-disabling-events,create an event rule>> matching the
4853 $ lttng enable-event --jul jello
4856 <<adding-context,Add the application-specific context fields>> to the
4861 $ lttng add-context --jul --type='$app.myRetriever:intCtx'
4862 $ lttng add-context --jul --type='$app.myRetriever:strContext'
4865 <<basic-tracing-session-control,Start tracing>>:
4872 Run the compiled class:
4876 $ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar:. Test
4879 <<basic-tracing-session-control,Stop tracing>> and inspect the
4891 [[python-application]]
4892 === User space Python agent
4894 You can instrument a Python{nbsp}2 or Python{nbsp}3 application which
4896 https://docs.python.org/3/library/logging.html[`logging`] package.
4898 Each log statement emits an LTTng event once the
4899 application module imports the
4900 <<lttng-ust-agents,LTTng-UST Python agent>> package.
4903 .A Python application importing the LTTng-UST Python agent.
4904 image::python-app.png[]
4906 To use the LTTng-UST Python agent:
4908 . In the Python application's source code, import the LTTng-UST Python
4918 The LTTng-UST Python agent automatically adds its logging handler to the
4919 root logger at import time.
4921 Any log statement that the application executes before this import does
4922 not emit an LTTng event.
4924 IMPORTANT: The LTTng-UST Python agent must be
4925 <<installing-lttng,installed>>.
4927 . Use log statements and logging configuration as usual.
4928 Since the LTTng-UST Python agent adds a handler to the _root_
4929 logger, you can trace any log statement from any logger.
4931 .Use the LTTng-UST Python agent.
4942 logging.basicConfig()
4943 logger = logging.getLogger('my-logger')
4946 logger.debug('debug message')
4947 logger.info('info message')
4948 logger.warn('warn message')
4949 logger.error('error message')
4950 logger.critical('critical message')
4954 if __name__ == '__main__':
4958 NOTE: `logging.basicConfig()`, which adds to the root logger a basic
4959 logging handler which prints to the standard error stream, is not
4960 strictly required for LTTng-UST tracing to work, but in versions of
4961 Python preceding{nbsp}3.2, you could see a warning message which indicates
4962 that no handler exists for the logger `my-logger`.
4964 <<creating-destroying-tracing-sessions,Create a tracing session>>,
4965 <<enabling-disabling-events,create an event rule>> matching the
4966 `my-logger` Python logger, and <<basic-tracing-session-control,start
4972 $ lttng enable-event --python my-logger
4976 Run the Python script:
4983 <<basic-tracing-session-control,Stop tracing>> and inspect the recorded
4993 In the resulting trace, an <<event,event record>> generated by a Python
4994 application is named `lttng_python:event` and has the following fields:
4997 Logging time (string).
5000 Log record's message.
5006 Name of the function in which the log statement was executed.
5009 Line number at which the log statement was executed.
5012 Log level integer value.
5015 ID of the Python thread in which the log statement was executed.
5018 Name of the Python thread in which the log statement was executed.
5020 You can use the opt:lttng-enable-event(1):--loglevel or
5021 opt:lttng-enable-event(1):--loglevel-only option of the
5022 man:lttng-enable-event(1) command to target a range of Python log levels
5023 or a specific Python log level.
5025 When an application imports the LTTng-UST Python agent, the agent tries
5026 to register to a <<lttng-sessiond,session daemon>>. Note that you must
5027 <<start-sessiond,start the session daemon>> _before_ you run the Python
5028 application. If a session daemon is found, the agent tries to register
5029 to it during five seconds, after which the application continues
5030 without LTTng tracing support. You can override this timeout value with
5031 the env:LTTNG_UST_PYTHON_REGISTER_TIMEOUT environment variable
5034 If the session daemon stops while a Python application with an imported
5035 LTTng-UST Python agent runs, the agent retries to connect and to
5036 register to a session daemon every three seconds. You can override this
5037 delay with the env:LTTNG_UST_PYTHON_REGISTER_RETRY_DELAY environment
5042 [[proc-lttng-logger-abi]]
5045 The `lttng-tracer` Linux kernel module, part of
5046 <<lttng-modules,LTTng-modules>>, creates the special LTTng logger files
5047 path:{/proc/lttng-logger} and path:{/dev/lttng-logger} (since
5048 LTTng{nbsp}2.11) when it's loaded. Any application can write text data
5049 to any of those files to emit an LTTng event.
5052 .An application writes to the LTTng logger file to emit an LTTng event.
5053 image::lttng-logger.png[]
5055 The LTTng logger is the quickest method--not the most efficient,
5056 however--to add instrumentation to an application. It is designed
5057 mostly to instrument shell scripts:
5061 $ echo "Some message, some $variable" > /dev/lttng-logger
5064 Any event that the LTTng logger emits is named `lttng_logger` and
5065 belongs to the Linux kernel <<domain,tracing domain>>. However, unlike
5066 other instrumentation points in the kernel tracing domain, **any Unix
5067 user** can <<enabling-disabling-events,create an event rule>> which
5068 matches its event name, not only the root user or users in the
5069 <<tracing-group,tracing group>>.
5071 To use the LTTng logger:
5073 * From any application, write text data to the path:{/dev/lttng-logger}
5076 The `msg` field of `lttng_logger` event records contains the
5079 NOTE: The maximum message length of an LTTng logger event is
5080 1024{nbsp}bytes. Writing more than this makes the LTTng logger emit more
5081 than one event to contain the remaining data.
5083 You should not use the LTTng logger to trace a user application which
5084 can be instrumented in a more efficient way, namely:
5086 * <<c-application,C and $$C++$$ applications>>.
5087 * <<java-application,Java applications>>.
5088 * <<python-application,Python applications>>.
5090 .Use the LTTng logger.
5095 echo 'Hello, World!' > /dev/lttng-logger
5097 df --human-readable --print-type / > /dev/lttng-logger
5100 <<creating-destroying-tracing-sessions,Create a tracing session>>,
5101 <<enabling-disabling-events,create an event rule>> matching the
5102 `lttng_logger` Linux kernel tracepoint, and
5103 <<basic-tracing-session-control,start tracing>>:
5108 $ lttng enable-event --kernel lttng_logger
5112 Run the Bash script:
5119 <<basic-tracing-session-control,Stop tracing>> and inspect the recorded
5130 [[instrumenting-linux-kernel]]
5131 === LTTng kernel tracepoints
5133 NOTE: This section shows how to _add_ instrumentation points to the
5134 Linux kernel. The kernel's subsystems are already thoroughly
5135 instrumented at strategic places for LTTng when you
5136 <<installing-lttng,install>> the <<lttng-modules,LTTng-modules>>
5140 There are two methods to instrument the Linux kernel:
5142 . <<linux-add-lttng-layer,Add an LTTng layer>> over an existing ftrace
5143 tracepoint which uses the `TRACE_EVENT()` API.
5145 Choose this if you want to instrumentation a Linux kernel tree with an
5146 instrumentation point compatible with ftrace, perf, and SystemTap.
5148 . Use an <<linux-lttng-tracepoint-event,LTTng-only approach>> to
5149 instrument an out-of-tree kernel module.
5151 Choose this if you don't need ftrace, perf, or SystemTap support.
5155 [[linux-add-lttng-layer]]
5156 ==== [[instrumenting-linux-kernel-itself]][[mainline-trace-event]][[lttng-adaptation-layer]]Add an LTTng layer to an existing ftrace tracepoint
5158 This section shows how to add an LTTng layer to existing ftrace
5159 instrumentation using the `TRACE_EVENT()` API.
5161 This section does not document the `TRACE_EVENT()` macro. You can
5162 read the following articles to learn more about this API:
5164 * http://lwn.net/Articles/379903/[Using the TRACE_EVENT() macro (Part{nbsp}1)]
5165 * http://lwn.net/Articles/381064/[Using the TRACE_EVENT() macro (Part{nbsp}2)]
5166 * http://lwn.net/Articles/383362/[Using the TRACE_EVENT() macro (Part{nbsp}3)]
5168 The following procedure assumes that your ftrace tracepoints are
5169 correctly defined in their own header and that they are created in
5170 one source file using the `CREATE_TRACE_POINTS` definition.
5172 To add an LTTng layer over an existing ftrace tracepoint:
5174 . Make sure the following kernel configuration options are
5180 * `CONFIG_HIGH_RES_TIMERS`
5181 * `CONFIG_TRACEPOINTS`
5184 . Build the Linux source tree with your custom ftrace tracepoints.
5185 . Boot the resulting Linux image on your target system.
5187 Confirm that the tracepoints exist by looking for their names in the
5188 dir:{/sys/kernel/debug/tracing/events/subsys} directory, where `subsys`
5189 is your subsystem's name.
5191 . Get a copy of the latest LTTng-modules{nbsp}{revision}:
5196 $ cd $(mktemp -d) &&
5197 wget http://lttng.org/files/lttng-modules/lttng-modules-latest-2.11.tar.bz2 &&
5198 tar -xf lttng-modules-latest-2.11.tar.bz2 &&
5199 cd lttng-modules-2.11.*
5203 . In dir:{instrumentation/events/lttng-module}, relative to the root
5204 of the LTTng-modules source tree, create a header file named
5205 +__subsys__.h+ for your custom subsystem +__subsys__+ and write your
5206 LTTng-modules tracepoint definitions using the LTTng-modules
5209 Start with this template:
5213 .path:{instrumentation/events/lttng-module/my_subsys.h}
5216 #define TRACE_SYSTEM my_subsys
5218 #if !defined(_LTTNG_MY_SUBSYS_H) || defined(TRACE_HEADER_MULTI_READ)
5219 #define _LTTNG_MY_SUBSYS_H
5221 #include "../../../probes/lttng-tracepoint-event.h"
5222 #include <linux/tracepoint.h>
5224 LTTNG_TRACEPOINT_EVENT(
5226 * Format is identical to TRACE_EVENT()'s version for the three
5227 * following macro parameters:
5230 TP_PROTO(int my_int, const char *my_string),
5231 TP_ARGS(my_int, my_string),
5233 /* LTTng-modules specific macros */
5235 ctf_integer(int, my_int_field, my_int)
5236 ctf_string(my_bar_field, my_bar)
5240 #endif /* !defined(_LTTNG_MY_SUBSYS_H) || defined(TRACE_HEADER_MULTI_READ) */
5242 #include "../../../probes/define_trace.h"
5246 The entries in the `TP_FIELDS()` section are the list of fields for the
5247 LTTng tracepoint. This is similar to the `TP_STRUCT__entry()` part of
5248 ftrace's `TRACE_EVENT()` macro.
5250 See <<lttng-modules-tp-fields,Tracepoint fields macros>> for a
5251 complete description of the available `ctf_*()` macros.
5253 . Create the LTTng-modules probe's kernel module C source file,
5254 +probes/lttng-probe-__subsys__.c+, where +__subsys__+ is your
5259 .path:{probes/lttng-probe-my-subsys.c}
5261 #include <linux/module.h>
5262 #include "../lttng-tracer.h"
5265 * Build-time verification of mismatch between mainline
5266 * TRACE_EVENT() arguments and the LTTng-modules adaptation
5267 * layer LTTNG_TRACEPOINT_EVENT() arguments.
5269 #include <trace/events/my_subsys.h>
5271 /* Create LTTng tracepoint probes */
5272 #define LTTNG_PACKAGE_BUILD
5273 #define CREATE_TRACE_POINTS
5274 #define TRACE_INCLUDE_PATH ../instrumentation/events/lttng-module
5276 #include "../instrumentation/events/lttng-module/my_subsys.h"
5278 MODULE_LICENSE("GPL and additional rights");
5279 MODULE_AUTHOR("Your name <your-email>");
5280 MODULE_DESCRIPTION("LTTng my_subsys probes");
5281 MODULE_VERSION(__stringify(LTTNG_MODULES_MAJOR_VERSION) "."
5282 __stringify(LTTNG_MODULES_MINOR_VERSION) "."
5283 __stringify(LTTNG_MODULES_PATCHLEVEL_VERSION)
5284 LTTNG_MODULES_EXTRAVERSION);
5288 . Edit path:{probes/KBuild} and add your new kernel module object
5289 next to the existing ones:
5293 .path:{probes/KBuild}
5297 obj-m += lttng-probe-module.o
5298 obj-m += lttng-probe-power.o
5300 obj-m += lttng-probe-my-subsys.o
5306 . Build and install the LTTng kernel modules:
5311 $ make KERNELDIR=/path/to/linux
5312 # make modules_install && depmod -a
5316 Replace `/path/to/linux` with the path to the Linux source tree where
5317 you defined and used tracepoints with ftrace's `TRACE_EVENT()` macro.
5319 Note that you can also use the
5320 <<lttng-tracepoint-event-code,`LTTNG_TRACEPOINT_EVENT_CODE()` macro>>
5321 instead of `LTTNG_TRACEPOINT_EVENT()` to use custom local variables and
5322 C code that need to be executed before the event fields are recorded.
5324 The best way to learn how to use the previous LTTng-modules macros is to
5325 inspect the existing LTTng-modules tracepoint definitions in the
5326 dir:{instrumentation/events/lttng-module} header files. Compare them
5327 with the Linux kernel mainline versions in the
5328 dir:{include/trace/events} directory of the Linux source tree.
5332 [[lttng-tracepoint-event-code]]
5333 ===== Use custom C code to access the data for tracepoint fields
5335 Although we recommended to always use the
5336 <<lttng-adaptation-layer,`LTTNG_TRACEPOINT_EVENT()`>> macro to describe
5337 the arguments and fields of an LTTng-modules tracepoint when possible,
5338 sometimes you need a more complex process to access the data that the
5339 tracer records as event record fields. In other words, you need local
5340 variables and multiple C{nbsp}statements instead of simple
5341 argument-based expressions that you pass to the
5342 <<lttng-modules-tp-fields,`ctf_*()` macros of `TP_FIELDS()`>>.
5344 You can use the `LTTNG_TRACEPOINT_EVENT_CODE()` macro instead of
5345 `LTTNG_TRACEPOINT_EVENT()` to declare custom local variables and define
5346 a block of C{nbsp}code to be executed before LTTng records the fields.
5347 The structure of this macro is:
5350 .`LTTNG_TRACEPOINT_EVENT_CODE()` macro syntax.
5352 LTTNG_TRACEPOINT_EVENT_CODE(
5354 * Format identical to the LTTNG_TRACEPOINT_EVENT()
5355 * version for the following three macro parameters:
5358 TP_PROTO(int my_int, const char *my_string),
5359 TP_ARGS(my_int, my_string),
5361 /* Declarations of custom local variables */
5364 unsigned long b = 0;
5365 const char *name = "(undefined)";
5366 struct my_struct *my_struct;
5370 * Custom code which uses both tracepoint arguments
5371 * (in TP_ARGS()) and local variables (in TP_locvar()).
5373 * Local variables are actually members of a structure pointed
5374 * to by the special variable tp_locvar.
5378 tp_locvar->a = my_int + 17;
5379 tp_locvar->my_struct = get_my_struct_at(tp_locvar->a);
5380 tp_locvar->b = my_struct_compute_b(tp_locvar->my_struct);
5381 tp_locvar->name = my_struct_get_name(tp_locvar->my_struct);
5382 put_my_struct(tp_locvar->my_struct);
5391 * Format identical to the LTTNG_TRACEPOINT_EVENT()
5392 * version for this, except that tp_locvar members can be
5393 * used in the argument expression parameters of
5394 * the ctf_*() macros.
5397 ctf_integer(unsigned long, my_struct_b, tp_locvar->b)
5398 ctf_integer(int, my_struct_a, tp_locvar->a)
5399 ctf_string(my_string_field, my_string)
5400 ctf_string(my_struct_name, tp_locvar->name)
5405 IMPORTANT: The C code defined in `TP_code()` must not have any side
5406 effects when executed. In particular, the code must not allocate
5407 memory or get resources without deallocating this memory or putting
5408 those resources afterwards.
5411 [[instrumenting-linux-kernel-tracing]]
5412 ==== Load and unload a custom probe kernel module
5414 You must load a <<lttng-adaptation-layer,created LTTng-modules probe
5415 kernel module>> in the kernel before it can emit LTTng events.
5417 To load the default probe kernel modules and a custom probe kernel
5420 * Use the opt:lttng-sessiond(8):--extra-kmod-probes option to give extra
5421 probe modules to load when starting a root <<lttng-sessiond,session
5425 .Load the `my_subsys`, `usb`, and the default probe modules.
5429 # lttng-sessiond --extra-kmod-probes=my_subsys,usb
5434 You only need to pass the subsystem name, not the whole kernel module
5437 To load _only_ a given custom probe kernel module:
5439 * Use the opt:lttng-sessiond(8):--kmod-probes option to give the probe
5440 modules to load when starting a root session daemon:
5443 .Load only the `my_subsys` and `usb` probe modules.
5447 # lttng-sessiond --kmod-probes=my_subsys,usb
5452 To confirm that a probe module is loaded:
5459 $ lsmod | grep lttng_probe_usb
5463 To unload the loaded probe modules:
5465 * Kill the session daemon with `SIGTERM`:
5470 # pkill lttng-sessiond
5474 You can also use man:modprobe(8)'s `--remove` option if the session
5475 daemon terminates abnormally.
5478 [[controlling-tracing]]
5481 Once an application or a Linux kernel is
5482 <<instrumenting,instrumented>> for LTTng tracing,
5485 This section is divided in topics on how to use the various
5486 <<plumbing,components of LTTng>>, in particular the <<lttng-cli,cmd:lttng
5487 command-line tool>>, to _control_ the LTTng daemons and tracers.
5489 NOTE: In the following subsections, we refer to an man:lttng(1) command
5490 using its man page name. For example, instead of _Run the `create`
5491 command to..._, we use _Run the man:lttng-create(1) command to..._.
5495 === Start a session daemon
5497 In some situations, you need to run a <<lttng-sessiond,session daemon>>
5498 (man:lttng-sessiond(8)) _before_ you can use the man:lttng(1)
5501 You will see the following error when you run a command while no session
5505 Error: No session daemon is available
5508 The only command that automatically runs a session daemon is
5509 man:lttng-create(1), which you use to
5510 <<creating-destroying-tracing-sessions,create a tracing session>>. While
5511 this is most of the time the first operation that you do, sometimes it's
5512 not. Some examples are:
5514 * <<list-instrumentation-points,List the available instrumentation points>>.
5515 * <<saving-loading-tracing-session,Load a tracing session configuration>>.
5517 [[tracing-group]] Each Unix user must have its own running session
5518 daemon to trace user applications. The session daemon that the root user
5519 starts is the only one allowed to control the LTTng kernel tracer. Users
5520 that are part of the _tracing group_ can control the root session
5521 daemon. The default tracing group name is `tracing`; you can set it to
5522 something else with the opt:lttng-sessiond(8):--group option when you
5523 start the root session daemon.
5525 To start a user session daemon:
5527 * Run man:lttng-sessiond(8):
5532 $ lttng-sessiond --daemonize
5536 To start the root session daemon:
5538 * Run man:lttng-sessiond(8) as the root user:
5543 # lttng-sessiond --daemonize
5547 In both cases, remove the opt:lttng-sessiond(8):--daemonize option to
5548 start the session daemon in foreground.
5550 To stop a session daemon, use man:kill(1) on its process ID (standard
5553 Note that some Linux distributions could manage the LTTng session daemon
5554 as a service. In this case, you should use the service manager to
5555 start, restart, and stop session daemons.
5558 [[creating-destroying-tracing-sessions]]
5559 === Create and destroy a tracing session
5561 Almost all the LTTng control operations happen in the scope of
5562 a <<tracing-session,tracing session>>, which is the dialogue between the
5563 <<lttng-sessiond,session daemon>> and you.
5565 To create a tracing session with a generated name:
5567 * Use the man:lttng-create(1) command:
5576 The created tracing session's name is `auto` followed by the
5579 To create a tracing session with a specific name:
5581 * Use the optional argument of the man:lttng-create(1) command:
5586 $ lttng create my-session
5590 Replace `my-session` with the specific tracing session name.
5592 LTTng appends the creation date to the created tracing session's name.
5594 LTTng writes the traces of a tracing session in
5595 +$LTTNG_HOME/lttng-trace/__name__+ by default, where +__name__+ is the
5596 name of the tracing session. Note that the env:LTTNG_HOME environment
5597 variable defaults to `$HOME` if not set.
5599 To output LTTng traces to a non-default location:
5601 * Use the opt:lttng-create(1):--output option of the man:lttng-create(1) command:
5606 $ lttng create my-session --output=/tmp/some-directory
5610 You may create as many tracing sessions as you wish.
5612 To list all the existing tracing sessions for your Unix user:
5614 * Use the man:lttng-list(1) command:
5623 When you create a tracing session, it is set as the _current tracing
5624 session_. The following man:lttng(1) commands operate on the current
5625 tracing session when you don't specify one:
5627 [role="list-3-cols"]
5628 * man:lttng-add-context(1)
5629 * man:lttng-destroy(1)
5630 * man:lttng-disable-channel(1)
5631 * man:lttng-disable-event(1)
5632 * man:lttng-disable-rotation(1)
5633 * man:lttng-enable-channel(1)
5634 * man:lttng-enable-event(1)
5635 * man:lttng-enable-rotation(1)
5637 * man:lttng-regenerate(1)
5638 * man:lttng-rotate(1)
5640 * man:lttng-snapshot(1)
5641 * man:lttng-start(1)
5642 * man:lttng-status(1)
5644 * man:lttng-track(1)
5645 * man:lttng-untrack(1)
5648 To change the current tracing session:
5650 * Use the man:lttng-set-session(1) command:
5655 $ lttng set-session new-session
5659 Replace `new-session` by the name of the new current tracing session.
5661 When you are done tracing in a given tracing session, you can destroy
5662 it. This operation frees the resources taken by the tracing session
5663 to destroy; it does not destroy the trace data that LTTng wrote for
5664 this tracing session.
5666 To destroy the current tracing session:
5668 * Use the man:lttng-destroy(1) command:
5677 The man:lttng-destroy(1) command also runs the man:lttng-stop(1)
5678 command implicitly (see <<basic-tracing-session-control,Start and stop a
5679 tracing session>>). You need to stop tracing to make LTTng flush the
5680 remaining trace data and make the trace readable.
5683 [[list-instrumentation-points]]
5684 === List the available instrumentation points
5686 The <<lttng-sessiond,session daemon>> can query the running instrumented
5687 user applications and the Linux kernel to get a list of available
5688 instrumentation points. For the Linux kernel <<domain,tracing domain>>,
5689 they are tracepoints and system calls. For the user space tracing
5690 domain, they are tracepoints. For the other tracing domains, they are
5693 To list the available instrumentation points:
5695 * Use the man:lttng-list(1) command with the requested tracing domain's
5699 * opt:lttng-list(1):--kernel: Linux kernel tracepoints (your Unix user
5700 must be a root user, or it must be a member of the
5701 <<tracing-group,tracing group>>).
5702 * opt:lttng-list(1):--kernel with opt:lttng-list(1):--syscall: Linux
5703 kernel system calls (your Unix user must be a root user, or it must be
5704 a member of the tracing group).
5705 * opt:lttng-list(1):--userspace: user space tracepoints.
5706 * opt:lttng-list(1):--jul: `java.util.logging` loggers.
5707 * opt:lttng-list(1):--log4j: Apache log4j loggers.
5708 * opt:lttng-list(1):--python: Python loggers.
5711 .List the available user space tracepoints.
5715 $ lttng list --userspace
5719 .List the available Linux kernel system call tracepoints.
5723 $ lttng list --kernel --syscall
5728 [[enabling-disabling-events]]
5729 === Create and enable an event rule
5731 Once you <<creating-destroying-tracing-sessions,create a tracing
5732 session>>, you can create <<event,event rules>> with the
5733 man:lttng-enable-event(1) command.
5735 You specify each condition with a command-line option. The available
5736 condition arguments are shown in the following table.
5738 [role="growable",cols="asciidoc,asciidoc,default"]
5739 .Condition command-line arguments for the man:lttng-enable-event(1) command.
5741 |Argument |Description |Applicable tracing domains
5747 . +--probe=__ADDR__+
5748 . +--function=__ADDR__+
5749 . +--userspace-probe=__PATH__:__SYMBOL__+
5750 . +--userspace-probe=sdt:__PATH__:__PROVIDER__:__NAME__+
5753 Instead of using the default _tracepoint_ instrumentation type, use:
5755 . A Linux system call (entry and exit).
5756 . A Linux https://lwn.net/Articles/132196/[kprobe] (symbol or address).
5757 . The entry and return points of a Linux function (symbol or address).
5758 . The entry point of a user application or library function (path to
5759 application/library and symbol).
5760 . A https://www.sourceware.org/systemtap/wiki/AddingUserSpaceProbingToApps[SystemTap
5761 Statically Defined Tracing] (USDT) probe (path to application/library,
5762 provider and probe names).
5766 |First positional argument.
5769 Tracepoint or system call name.
5771 With the opt:lttng-enable-event(1):--probe,
5772 opt:lttng-enable-event(1):--function, and
5773 opt:lttng-enable-event(1):--userspace-probe options, this is a custom
5774 name given to the event rule. With the JUL, log4j, and Python domains,
5775 this is a logger name.
5777 With a tracepoint, logger, or system call name, you can use the special
5778 `*` globbing character to match anything (for example, `sched_*`,
5786 . +--loglevel=__LEVEL__+
5787 . +--loglevel-only=__LEVEL__+
5790 . Match only tracepoints or log statements with a logging level at
5791 least as severe as +__LEVEL__+.
5792 . Match only tracepoints or log statements with a logging level
5793 equal to +__LEVEL__+.
5795 See man:lttng-enable-event(1) for the list of available logging level
5798 |User space, JUL, log4j, and Python.
5800 |+--exclude=__EXCLUSIONS__+
5803 When you use a `*` character at the end of the tracepoint or logger
5804 name (first positional argument), exclude the specific names in the
5805 comma-delimited list +__EXCLUSIONS__+.
5808 User space, JUL, log4j, and Python.
5810 |+--filter=__EXPR__+
5813 Match only events which satisfy the expression +__EXPR__+.
5815 See man:lttng-enable-event(1) to learn more about the syntax of a
5822 You attach an event rule to a <<channel,channel>> on creation. If you do
5823 not specify the channel with the opt:lttng-enable-event(1):--channel
5824 option, and if the event rule to create is the first in its
5825 <<domain,tracing domain>> for a given tracing session, then LTTng
5826 creates a _default channel_ for you. This default channel is reused in
5827 subsequent invocations of the man:lttng-enable-event(1) command for the
5828 same tracing domain.
5830 An event rule is always enabled at creation time.
5832 The following examples show how you can combine the previous
5833 command-line options to create simple to more complex event rules.
5835 .Create an event rule targetting a Linux kernel tracepoint (default channel).
5839 $ lttng enable-event --kernel sched_switch
5843 .Create an event rule matching four Linux kernel system calls (default channel).
5847 $ lttng enable-event --kernel --syscall open,write,read,close
5851 .Create event rules matching tracepoints with filter expressions (default channel).
5855 $ lttng enable-event --kernel sched_switch --filter='prev_comm == "bash"'
5860 $ lttng enable-event --kernel --all \
5861 --filter='$ctx.tid == 1988 || $ctx.tid == 1534'
5866 $ lttng enable-event --jul my_logger \
5867 --filter='$app.retriever:cur_msg_id > 3'
5870 IMPORTANT: Make sure to always quote the filter string when you
5871 use man:lttng(1) from a shell.
5874 .Create an event rule matching any user space tracepoint of a given tracepoint provider with a log level range (default channel).
5878 $ lttng enable-event --userspace my_app:'*' --loglevel=TRACE_INFO
5881 IMPORTANT: Make sure to always quote the wildcard character when you
5882 use man:lttng(1) from a shell.
5885 .Create an event rule matching multiple Python loggers with a wildcard and with exclusions (default channel).
5889 $ lttng enable-event --python my-app.'*' \
5890 --exclude='my-app.module,my-app.hello'
5894 .Create an event rule matching any Apache log4j logger with a specific log level (default channel).
5898 $ lttng enable-event --log4j --all --loglevel-only=LOG4J_WARN
5902 .Create an event rule attached to a specific channel matching a specific user space tracepoint provider and tracepoint.
5906 $ lttng enable-event --userspace my_app:my_tracepoint --channel=my-channel
5910 .Create an event rule matching the `malloc` function entry in path:{/usr/lib/libc.so.6}:
5914 $ lttng enable-event --kernel --userspace-probe=/usr/lib/libc.so.6:malloc \
5919 .Create an event rule matching the `server`/`accept_request` https://www.sourceware.org/systemtap/wiki/AddingUserSpaceProbingToApps[USDT probe] in path:{/usr/bin/serv}:
5923 $ lttng enable-event --kernel --userspace-probe=sdt:serv:server:accept_request \
5924 server_accept_request
5928 The event rules of a given channel form a whitelist: as soon as an
5929 emitted event passes one of them, LTTng can record the event. For
5930 example, an event named `my_app:my_tracepoint` emitted from a user space
5931 tracepoint with a `TRACE_ERROR` log level passes both of the following
5936 $ lttng enable-event --userspace my_app:my_tracepoint
5937 $ lttng enable-event --userspace my_app:my_tracepoint \
5938 --loglevel=TRACE_INFO
5941 The second event rule is redundant: the first one includes
5945 [[disable-event-rule]]
5946 === Disable an event rule
5948 To disable an event rule that you <<enabling-disabling-events,created>>
5949 previously, use the man:lttng-disable-event(1) command. This command
5950 disables _all_ the event rules (of a given tracing domain and channel)
5951 which match an instrumentation point. The other conditions are not
5952 supported as of LTTng{nbsp}{revision}.
5954 The LTTng tracer does not record an emitted event which passes
5955 a _disabled_ event rule.
5957 .Disable an event rule matching a Python logger (default channel).
5961 $ lttng disable-event --python my-logger
5965 .Disable an event rule matching all `java.util.logging` loggers (default channel).
5969 $ lttng disable-event --jul '*'
5973 .Disable _all_ the event rules of the default channel.
5975 The opt:lttng-disable-event(1):--all-events option is not, like the
5976 opt:lttng-enable-event(1):--all option of man:lttng-enable-event(1), the
5977 equivalent of the event name `*` (wildcard): it disables _all_ the event
5978 rules of a given channel.
5982 $ lttng disable-event --jul --all-events
5986 NOTE: You cannot delete an event rule once you create it.
5990 === Get the status of a tracing session
5992 To get the status of the current tracing session, that is, its
5993 parameters, its channels, event rules, and their attributes:
5995 * Use the man:lttng-status(1) command:
6005 To get the status of any tracing session:
6007 * Use the man:lttng-list(1) command with the tracing session's name:
6012 $ lttng list my-session
6016 Replace `my-session` with the desired tracing session's name.
6019 [[basic-tracing-session-control]]
6020 === Start and stop a tracing session
6022 Once you <<creating-destroying-tracing-sessions,create a tracing
6024 <<enabling-disabling-events,create one or more event rules>>,
6025 you can start and stop the tracers for this tracing session.
6027 To start tracing in the current tracing session:
6029 * Use the man:lttng-start(1) command:
6038 LTTng is very flexible: you can launch user applications before
6039 or after the you start the tracers. The tracers only record the events
6040 if they pass enabled event rules and if they occur while the tracers are
6043 To stop tracing in the current tracing session:
6045 * Use the man:lttng-stop(1) command:
6054 If there were <<channel-overwrite-mode-vs-discard-mode,lost event
6055 records>> or lost sub-buffers since the last time you ran
6056 man:lttng-start(1), warnings are printed when you run the
6057 man:lttng-stop(1) command.
6059 IMPORTANT: You need to stop tracing to make LTTng flush the remaining
6060 trace data and make the trace readable. Note that the
6061 man:lttng-destroy(1) command (see
6062 <<creating-destroying-tracing-sessions,Create and destroy a tracing
6063 session>>) also runs the man:lttng-stop(1) command implicitly.
6066 [[enabling-disabling-channels]]
6067 === Create a channel
6069 Once you create a tracing session, you can create a <<channel,channel>>
6070 with the man:lttng-enable-channel(1) command.
6072 Note that LTTng automatically creates a default channel when, for a
6073 given <<domain,tracing domain>>, no channels exist and you
6074 <<enabling-disabling-events,create>> the first event rule. This default
6075 channel is named `channel0` and its attributes are set to reasonable
6076 values. Therefore, you only need to create a channel when you need
6077 non-default attributes.
6079 You specify each non-default channel attribute with a command-line
6080 option when you use the man:lttng-enable-channel(1) command. The
6081 available command-line options are:
6083 [role="growable",cols="asciidoc,asciidoc"]
6084 .Command-line options for the man:lttng-enable-channel(1) command.
6086 |Option |Description
6092 <<channel-overwrite-mode-vs-discard-mode,event record loss mode>> instead
6093 of the default _discard_ mode.
6095 |`--buffers-pid` (user space tracing domain only)
6098 Use the per-process <<channel-buffering-schemes,buffering scheme>>
6099 instead of the default per-user buffering scheme.
6101 |+--subbuf-size=__SIZE__+
6104 Allocate sub-buffers of +__SIZE__+ bytes (power of two), for each CPU,
6105 either for each Unix user (default), or for each instrumented process.
6107 See <<channel-subbuf-size-vs-subbuf-count,Sub-buffer count and size>>.
6109 |+--num-subbuf=__COUNT__+
6112 Allocate +__COUNT__+ sub-buffers (power of two), for each CPU, either
6113 for each Unix user (default), or for each instrumented process.
6115 See <<channel-subbuf-size-vs-subbuf-count,Sub-buffer count and size>>.
6117 |+--tracefile-size=__SIZE__+
6120 Set the maximum size of each trace file that this channel writes within
6121 a stream to +__SIZE__+ bytes instead of no maximum.
6123 See <<tracefile-rotation,Trace file count and size>>.
6125 |+--tracefile-count=__COUNT__+
6128 Limit the number of trace files that this channel creates to
6129 +__COUNT__+ channels instead of no limit.
6131 See <<tracefile-rotation,Trace file count and size>>.
6133 |+--switch-timer=__PERIODUS__+
6136 Set the <<channel-switch-timer,switch timer period>>
6137 to +__PERIODUS__+{nbsp}µs.
6139 |+--read-timer=__PERIODUS__+
6142 Set the <<channel-read-timer,read timer period>>
6143 to +__PERIODUS__+{nbsp}µs.
6145 |[[opt-blocking-timeout]]+--blocking-timeout=__TIMEOUTUS__+
6148 Set the timeout of user space applications which load LTTng-UST
6149 in blocking mode to +__TIMEOUTUS__+:
6152 Never block (non-blocking mode).
6155 Block forever until space is available in a sub-buffer to record
6158 __n__, a positive value::
6159 Wait for at most __n__ µs when trying to write into a sub-buffer.
6161 Note that, for this option to have any effect on an instrumented
6162 user space application, you need to run the application with a set
6163 env:LTTNG_UST_ALLOW_BLOCKING environment variable.
6165 |+--output=__TYPE__+ (Linux kernel tracing domain only)
6168 Set the channel's output type to +__TYPE__+, either `mmap` or `splice`.
6172 You can only create a channel in the Linux kernel and user space
6173 <<domain,tracing domains>>: other tracing domains have their own channel
6174 created on the fly when <<enabling-disabling-events,creating event
6179 Because of a current LTTng limitation, you must create all channels
6180 _before_ you <<basic-tracing-session-control,start tracing>> in a given
6181 tracing session, that is, before the first time you run
6184 Since LTTng automatically creates a default channel when you use the
6185 man:lttng-enable-event(1) command with a specific tracing domain, you
6186 cannot, for example, create a Linux kernel event rule, start tracing,
6187 and then create a user space event rule, because no user space channel
6188 exists yet and it's too late to create one.
6190 For this reason, make sure to configure your channels properly
6191 before starting the tracers for the first time!
6194 The following examples show how you can combine the previous
6195 command-line options to create simple to more complex channels.
6197 .Create a Linux kernel channel with default attributes.
6201 $ lttng enable-channel --kernel my-channel
6205 .Create a user space channel with four sub-buffers or 1{nbsp}MiB each, per CPU, per instrumented process.
6209 $ lttng enable-channel --userspace --num-subbuf=4 --subbuf-size=1M \
6210 --buffers-pid my-channel
6214 .[[blocking-timeout-example]]Create a default user space channel with an infinite blocking timeout.
6216 <<creating-destroying-tracing-sessions,Create a tracing-session>>,
6217 create the channel, <<enabling-disabling-events,create an event rule>>,
6218 and <<basic-tracing-session-control,start tracing>>:
6223 $ lttng enable-channel --userspace --blocking-timeout=inf blocking-channel
6224 $ lttng enable-event --userspace --channel=blocking-channel --all
6228 Run an application instrumented with LTTng-UST and allow it to block:
6232 $ LTTNG_UST_ALLOW_BLOCKING=1 my-app
6236 .Create a Linux kernel channel which rotates eight trace files of 4{nbsp}MiB each for each stream
6240 $ lttng enable-channel --kernel --tracefile-count=8 \
6241 --tracefile-size=4194304 my-channel
6245 .Create a user space channel in overwrite (or _flight recorder_) mode.
6249 $ lttng enable-channel --userspace --overwrite my-channel
6253 You can <<enabling-disabling-events,create>> the same event rule in
6254 two different channels:
6258 $ lttng enable-event --userspace --channel=my-channel app:tp
6259 $ lttng enable-event --userspace --channel=other-channel app:tp
6262 If both channels are enabled, when a tracepoint named `app:tp` is
6263 reached, LTTng records two events, one for each channel.
6267 === Disable a channel
6269 To disable a specific channel that you <<enabling-disabling-channels,created>>
6270 previously, use the man:lttng-disable-channel(1) command.
6272 .Disable a specific Linux kernel channel.
6276 $ lttng disable-channel --kernel my-channel
6280 The state of a channel precedes the individual states of event rules
6281 attached to it: event rules which belong to a disabled channel, even if
6282 they are enabled, are also considered disabled.
6286 === Add context fields to a channel
6288 Event record fields in trace files provide important information about
6289 events that occured previously, but sometimes some external context may
6290 help you solve a problem faster. Examples of context fields are:
6292 * The **process ID**, **thread ID**, **process name**, and
6293 **process priority** of the thread in which the event occurs.
6294 * The **hostname** of the system on which the event occurs.
6295 * The Linux kernel and user call stacks (since
6297 * The current values of many possible **performance counters** using
6299 ** CPU cycles, stalled cycles, idle cycles, and the other cycle types.
6301 ** Branch instructions, misses, and loads.
6303 * Any context defined at the application level (supported for the
6304 JUL and log4j <<domain,tracing domains>>).
6306 To get the full list of available context fields, see
6307 `lttng add-context --list`. Some context fields are reserved for a
6308 specific <<domain,tracing domain>> (Linux kernel or user space).
6310 You add context fields to <<channel,channels>>. All the events
6311 that a channel with added context fields records contain those fields.
6313 To add context fields to one or all the channels of a given tracing
6316 * Use the man:lttng-add-context(1) command.
6318 .Add context fields to all the channels of the current tracing session.
6320 The following command line adds the virtual process identifier and
6321 the per-thread CPU cycles count fields to all the user space channels
6322 of the current tracing session.
6326 $ lttng add-context --userspace --type=vpid --type=perf:thread:cpu-cycles
6330 .Add performance counter context fields by raw ID
6332 See man:lttng-add-context(1) for the exact format of the context field
6333 type, which is partly compatible with the format used in
6338 $ lttng add-context --userspace --type=perf:thread:raw:r0110:test
6339 $ lttng add-context --kernel --type=perf:cpu:raw:r0013c:x86unhalted
6343 .Add context fields to a specific channel.
6345 The following command line adds the thread identifier and user call
6346 stack context fields to the Linux kernel channel named `my-channel` in
6347 the current tracing session.
6351 $ lttng add-context --kernel --channel=my-channel \
6352 --type=tid --type=callstack-user
6356 .Add an application-specific context field to a specific channel.
6358 The following command line adds the `cur_msg_id` context field of the
6359 `retriever` context retriever for all the instrumented
6360 <<java-application,Java applications>> recording <<event,event records>>
6361 in the channel named `my-channel`:
6365 $ lttng add-context --kernel --channel=my-channel \
6366 --type='$app:retriever:cur_msg_id'
6369 IMPORTANT: Make sure to always quote the `$` character when you
6370 use man:lttng-add-context(1) from a shell.
6373 NOTE: You cannot remove context fields from a channel once you add it.
6378 === Track process IDs
6380 It's often useful to allow only specific process IDs (PIDs) to emit
6381 events. For example, you may wish to record all the system calls made by
6382 a given process (Ă la http://linux.die.net/man/1/strace[strace]).
6384 The man:lttng-track(1) and man:lttng-untrack(1) commands serve this
6385 purpose. Both commands operate on a whitelist of process IDs. You _add_
6386 entries to this whitelist with the man:lttng-track(1) command and remove
6387 entries with the man:lttng-untrack(1) command. Any process which has one
6388 of the PIDs in the whitelist is allowed to emit LTTng events which pass
6389 an enabled <<event,event rule>>.
6391 NOTE: The PID tracker tracks the _numeric process IDs_. Should a
6392 process with a given tracked ID exit and another process be given this
6393 ID, then the latter would also be allowed to emit events.
6395 .Track and untrack process IDs.
6397 For the sake of the following example, assume the target system has
6398 16{nbsp}possible PIDs.
6401 <<creating-destroying-tracing-sessions,create a tracing session>>,
6402 the whitelist contains all the possible PIDs:
6405 .All PIDs are tracked.
6406 image::track-all.png[]
6408 When the whitelist is full and you use the man:lttng-track(1) command to
6409 specify some PIDs to track, LTTng first clears the whitelist, then it
6410 tracks the specific PIDs. After:
6414 $ lttng track --pid=3,4,7,10,13
6420 .PIDs 3, 4, 7, 10, and 13 are tracked.
6421 image::track-3-4-7-10-13.png[]
6423 You can add more PIDs to the whitelist afterwards:
6427 $ lttng track --pid=1,15,16
6433 .PIDs 1, 15, and 16 are added to the whitelist.
6434 image::track-1-3-4-7-10-13-15-16.png[]
6436 The man:lttng-untrack(1) command removes entries from the PID tracker's
6437 whitelist. Given the previous example, the following command:
6441 $ lttng untrack --pid=3,7,10,13
6444 leads to this whitelist:
6447 .PIDs 3, 7, 10, and 13 are removed from the whitelist.
6448 image::track-1-4-15-16.png[]
6450 LTTng can track all possible PIDs again using the
6451 opt:lttng-track(1):--all option:
6455 $ lttng track --pid --all
6458 The result is, again:
6461 .All PIDs are tracked.
6462 image::track-all.png[]
6465 .Track only specific PIDs
6467 A very typical use case with PID tracking is to start with an empty
6468 whitelist, then <<basic-tracing-session-control,start the tracers>>, and
6469 then add PIDs manually while tracers are active. You can accomplish this
6470 by using the opt:lttng-untrack(1):--all option of the
6471 man:lttng-untrack(1) command to clear the whitelist after you
6472 <<creating-destroying-tracing-sessions,create a tracing session>>:
6476 $ lttng untrack --pid --all
6482 .No PIDs are tracked.
6483 image::untrack-all.png[]
6485 If you trace with this whitelist configuration, the tracer records no
6486 events for this <<domain,tracing domain>> because no processes are
6487 tracked. You can use the man:lttng-track(1) command as usual to track
6488 specific PIDs, for example:
6492 $ lttng track --pid=6,11
6498 .PIDs 6 and 11 are tracked.
6499 image::track-6-11.png[]
6504 [[saving-loading-tracing-session]]
6505 === Save and load tracing session configurations
6507 Configuring a <<tracing-session,tracing session>> can be long. Some of
6508 the tasks involved are:
6510 * <<enabling-disabling-channels,Create channels>> with
6511 specific attributes.
6512 * <<adding-context,Add context fields>> to specific channels.
6513 * <<enabling-disabling-events,Create event rules>> with specific log
6514 level and filter conditions.
6516 If you use LTTng to solve real world problems, chances are you have to
6517 record events using the same tracing session setup over and over,
6518 modifying a few variables each time in your instrumented program
6519 or environment. To avoid constant tracing session reconfiguration,
6520 the man:lttng(1) command-line tool can save and load tracing session
6521 configurations to/from XML files.
6523 To save a given tracing session configuration:
6525 * Use the man:lttng-save(1) command:
6530 $ lttng save my-session
6534 Replace `my-session` with the name of the tracing session to save.
6536 LTTng saves tracing session configurations to
6537 dir:{$LTTNG_HOME/.lttng/sessions} by default. Note that the
6538 env:LTTNG_HOME environment variable defaults to `$HOME` if not set. Use
6539 the opt:lttng-save(1):--output-path option to change this destination
6542 LTTng saves all configuration parameters, for example:
6544 * The tracing session name.
6545 * The trace data output path.
6546 * The channels with their state and all their attributes.
6547 * The context fields you added to channels.
6548 * The event rules with their state, log level and filter conditions.
6550 To load a tracing session:
6552 * Use the man:lttng-load(1) command:
6557 $ lttng load my-session
6561 Replace `my-session` with the name of the tracing session to load.
6563 When LTTng loads a configuration, it restores your saved tracing session
6564 as if you just configured it manually.
6566 See man:lttng-load(1) for the complete list of command-line options. You
6567 can also save and load many sessions at a time, and decide in which
6568 directory to output the XML files.
6571 [[sending-trace-data-over-the-network]]
6572 === Send trace data over the network
6574 LTTng can send the recorded trace data to a remote system over the
6575 network instead of writing it to the local file system.
6577 To send the trace data over the network:
6579 . On the _remote_ system (which can also be the target system),
6580 start an LTTng <<lttng-relayd,relay daemon>> (man:lttng-relayd(8)):
6589 . On the _target_ system, create a tracing session configured to
6590 send trace data over the network:
6595 $ lttng create my-session --set-url=net://remote-system
6599 Replace `remote-system` by the host name or IP address of the
6600 remote system. See man:lttng-create(1) for the exact URL format.
6602 . On the target system, use the man:lttng(1) command-line tool as usual.
6603 When tracing is active, the target's consumer daemon sends sub-buffers
6604 to the relay daemon running on the remote system instead of flushing
6605 them to the local file system. The relay daemon writes the received
6606 packets to the local file system.
6608 The relay daemon writes trace files to
6609 +$LTTNG_HOME/lttng-traces/__hostname__/__session__+ by default, where
6610 +__hostname__+ is the host name of the target system and +__session__+
6611 is the tracing session name. Note that the env:LTTNG_HOME environment
6612 variable defaults to `$HOME` if not set. Use the
6613 opt:lttng-relayd(8):--output option of man:lttng-relayd(8) to write
6614 trace files to another base directory.
6619 === View events as LTTng emits them (noch:{LTTng} live)
6621 LTTng live is a network protocol implemented by the <<lttng-relayd,relay
6622 daemon>> (man:lttng-relayd(8)) to allow compatible trace viewers to
6623 display events as LTTng emits them on the target system while tracing is
6626 The relay daemon creates a _tee_: it forwards the trace data to both
6627 the local file system and to connected live viewers:
6630 .The relay daemon creates a _tee_, forwarding the trace data to both trace files and a connected live viewer.
6635 . On the _target system_, create a <<tracing-session,tracing session>>
6641 $ lttng create my-session --live
6645 This spawns a local relay daemon.
6647 . Start the live viewer and configure it to connect to the relay
6648 daemon. For example, with http://diamon.org/babeltrace[Babeltrace]:
6653 $ babeltrace --input-format=lttng-live \
6654 net://localhost/host/hostname/my-session
6661 * `hostname` with the host name of the target system.
6662 * `my-session` with the name of the tracing session to view.
6665 . Configure the tracing session as usual with the man:lttng(1)
6666 command-line tool, and <<basic-tracing-session-control,start tracing>>.
6668 You can list the available live tracing sessions with Babeltrace:
6672 $ babeltrace --input-format=lttng-live net://localhost
6675 You can start the relay daemon on another system. In this case, you need
6676 to specify the relay daemon's URL when you create the tracing session
6677 with the opt:lttng-create(1):--set-url option. You also need to replace
6678 `localhost` in the procedure above with the host name of the system on
6679 which the relay daemon is running.
6681 See man:lttng-create(1) and man:lttng-relayd(8) for the complete list of
6682 command-line options.
6686 [[taking-a-snapshot]]
6687 === Take a snapshot of the current sub-buffers of a tracing session
6689 The normal behavior of LTTng is to append full sub-buffers to growing
6690 trace data files. This is ideal to keep a full history of the events
6691 that occurred on the target system, but it can
6692 represent too much data in some situations. For example, you may wish
6693 to trace your application continuously until some critical situation
6694 happens, in which case you only need the latest few recorded
6695 events to perform the desired analysis, not multi-gigabyte trace files.
6697 With the man:lttng-snapshot(1) command, you can take a snapshot of the
6698 current sub-buffers of a given <<tracing-session,tracing session>>.
6699 LTTng can write the snapshot to the local file system or send it over
6703 .A snapshot is a copy of the current sub-buffers, which are not cleared after the operation.
6704 image::snapshot.png[]
6706 If you wish to create unmanaged, self-contained, non-overlapping
6707 trace chunk archives instead of a simple copy of the current
6708 sub-buffers, see the <<session-rotation,tracing session rotation>>
6709 feature (available since LTTng{nbsp}2.11).
6713 . Create a tracing session in _snapshot mode_:
6718 $ lttng create my-session --snapshot
6722 The <<channel-overwrite-mode-vs-discard-mode,event record loss mode>> of
6723 <<channel,channels>> created in this mode is automatically set to
6724 _overwrite_ (flight recorder mode).
6726 . Configure the tracing session as usual with the man:lttng(1)
6727 command-line tool, and <<basic-tracing-session-control,start tracing>>.
6729 . **Optional**: When you need to take a snapshot,
6730 <<basic-tracing-session-control,stop tracing>>.
6732 You can take a snapshot when the tracers are active, but if you stop
6733 them first, you are sure that the data in the sub-buffers does not
6734 change before you actually take the snapshot.
6741 $ lttng snapshot record --name=my-first-snapshot
6745 LTTng writes the current sub-buffers of all the current tracing
6746 session's channels to trace files on the local file system. Those trace
6747 files have `my-first-snapshot` in their name.
6749 There is no difference between the format of a normal trace file and the
6750 format of a snapshot: viewers of LTTng traces also support LTTng
6753 By default, LTTng writes snapshot files to the path shown by
6754 `lttng snapshot list-output`. You can change this path or decide to send
6755 snapshots over the network using either:
6757 . An output path or URL that you specify when you
6758 <<creating-destroying-tracing-sessions,create the tracing session>>.
6759 . A snapshot output path or URL that you add using
6760 `lttng snapshot add-output`.
6761 . An output path or URL that you provide directly to the
6762 `lttng snapshot record` command.
6764 Method{nbsp}3 overrides method{nbsp}2, which overrides method 1. When
6765 you specify a URL, a relay daemon must listen on a remote system (see
6766 <<sending-trace-data-over-the-network,Send trace data over the
6771 [[session-rotation]]
6772 === Archive the current trace chunk (rotate a tracing session)
6774 The <<taking-a-snapshot,snapshot user guide>> shows how you can dump
6775 a tracing session's current sub-buffers to the file system or send them
6776 over the network. When you take a snapshot, LTTng does not clear the
6777 tracing session's ring buffers: if you take another snapshot immediately
6778 after, both snapshots could contain overlapping trace data.
6780 Inspired by https://en.wikipedia.org/wiki/Log_rotation[log rotation],
6781 _tracing session rotation_ is a feature which appends the content of the
6782 ring buffers to what's already on the file system or sent over the
6783 network since the tracing session's creation or since the last
6784 rotation, and then clears those ring buffers to avoid trace data
6787 What LTTng is about to write when performing a tracing session rotation
6788 is called the _current trace chunk_. When this current trace chunk is
6789 written to the file system or sent over the network, it becomes a _trace
6790 chunk archive_. Therefore, a tracing session rotation _archives_ the
6791 current trace chunk.
6794 .A tracing session rotation operation _archives_ the current trace chunk.
6795 image::rotation.png[]
6797 A trace chunk archive is a self-contained LTTng trace which LTTng
6798 doesn't manage anymore: you can read it, modify it, move it, or remove
6801 There are two methods to perform a tracing session rotation: immediately
6802 or with a rotation schedule.
6804 To perform an immediate tracing session rotation:
6806 . <<creating-destroying-tracing-sessions,Create a tracing session>>
6807 in _normal mode_ or _network streaming mode_
6808 (only those two creation modes support tracing session rotation):
6813 $ lttng create my-session
6817 . <<enabling-disabling-events,Create one or more event rules>>
6818 and <<basic-tracing-session-control,start tracing>>:
6823 $ lttng enable-event --kernel sched_'*'
6828 . When needed, immediately rotate the current tracing session:
6837 The cmd:lttng-rotate command prints the path to the created trace
6838 chunk archive. See man:lttng-rotate(1) to learn about the format
6839 of trace chunk archive directory names.
6841 You can perform other immediate rotations while the tracing session is
6842 active. It is guaranteed that all the trace chunk archives do not
6843 contain overlapping trace data. You can also perform an immediate
6844 rotation once you have <<basic-tracing-session-control,stopped>> the
6847 . When you are done tracing,
6848 <<creating-destroying-tracing-sessions,destroy the current tracing
6858 The tracing session destruction operation creates one last trace
6859 chunk archive from the current trace chunk.
6861 A tracing session rotation schedule is a planned rotation which LTTng
6862 performs automatically based on one of the following conditions:
6864 * A timer with a configured period times out.
6866 * The total size of the flushed part of the current trace chunk
6867 becomes greater than or equal to a configured value.
6869 To schedule a tracing session rotation, set a _rotation schedule_:
6871 . <<creating-destroying-tracing-sessions,Create a tracing session>>
6872 in _normal mode_ or _network streaming mode_
6873 (only those two creation modes support tracing session rotation):
6878 $ lttng create my-session
6882 . <<enabling-disabling-events,Create one or more event rules>>:
6887 $ lttng enable-event --kernel sched_'*'
6891 . Set a tracing session rotation schedule:
6896 $ lttng enable-rotation --timer=10s
6900 In this example, we set a rotation schedule so that LTTng performs a
6901 tracing session rotation every ten seconds.
6903 See man:lttng-enable-rotation(1) to learn more about other ways to set a
6906 . <<basic-tracing-session-control,Start tracing>>:
6915 LTTng performs tracing session rotations automatically while the tracing
6916 session is active thanks to the rotation schedule.
6918 . When you are done tracing,
6919 <<creating-destroying-tracing-sessions,destroy the current tracing
6929 The tracing session destruction operation creates one last trace chunk
6930 archive from the current trace chunk.
6932 You can use man:lttng-disable-rotation(1) to unset a tracing session
6935 NOTE: man:lttng-rotate(1) and man:lttng-enable-rotation(1) list
6936 limitations regarding those two commands.
6941 === Use the machine interface
6943 With any command of the man:lttng(1) command-line tool, you can set the
6944 opt:lttng(1):--mi option to `xml` (before the command name) to get an
6945 XML machine interface output, for example:
6949 $ lttng --mi=xml enable-event --kernel --syscall open
6952 A schema definition (XSD) is
6953 https://github.com/lttng/lttng-tools/blob/stable-2.11/src/common/mi-lttng-3.0.xsd[available]
6954 to ease the integration with external tools as much as possible.
6958 [[metadata-regenerate]]
6959 === Regenerate the metadata of an LTTng trace
6961 An LTTng trace, which is a http://diamon.org/ctf[CTF] trace, has both
6962 data stream files and a metadata file. This metadata file contains,
6963 amongst other things, information about the offset of the clock sources
6964 used to timestamp <<event,event records>> when tracing.
6966 If, once a <<tracing-session,tracing session>> is
6967 <<basic-tracing-session-control,started>>, a major
6968 https://en.wikipedia.org/wiki/Network_Time_Protocol[NTP] correction
6969 happens, the trace's clock offset also needs to be updated. You
6970 can use the `metadata` item of the man:lttng-regenerate(1) command
6973 The main use case of this command is to allow a system to boot with
6974 an incorrect wall time and trace it with LTTng before its wall time
6975 is corrected. Once the system is known to be in a state where its
6976 wall time is correct, it can run `lttng regenerate metadata`.
6978 To regenerate the metadata of an LTTng trace:
6980 * Use the `metadata` item of the man:lttng-regenerate(1) command:
6985 $ lttng regenerate metadata
6991 `lttng regenerate metadata` has the following limitations:
6993 * Tracing session <<creating-destroying-tracing-sessions,created>>
6995 * User space <<channel,channels>>, if any, are using
6996 <<channel-buffering-schemes,per-user buffering>>.
7001 [[regenerate-statedump]]
7002 === Regenerate the state dump of a tracing session
7004 The LTTng kernel and user space tracers generate state dump
7005 <<event,event records>> when the application starts or when you
7006 <<basic-tracing-session-control,start a tracing session>>. An analysis
7007 can use the state dump event records to set an initial state before it
7008 builds the rest of the state from the following event records.
7009 http://tracecompass.org/[Trace Compass] is a notable example of an
7010 application which uses the state dump of an LTTng trace.
7012 When you <<taking-a-snapshot,take a snapshot>>, it's possible that the
7013 state dump event records are not included in the snapshot because they
7014 were recorded to a sub-buffer that has been consumed or overwritten
7017 You can use the `lttng regenerate statedump` command to emit the state
7018 dump event records again.
7020 To regenerate the state dump of the current tracing session, provided
7021 create it in snapshot mode, before you take a snapshot:
7023 . Use the `statedump` item of the man:lttng-regenerate(1) command:
7028 $ lttng regenerate statedump
7032 . <<basic-tracing-session-control,Stop the tracing session>>:
7041 . <<taking-a-snapshot,Take a snapshot>>:
7046 $ lttng snapshot record --name=my-snapshot
7050 Depending on the event throughput, you should run steps 1 and 2
7051 as closely as possible.
7053 NOTE: To record the state dump events, you need to
7054 <<enabling-disabling-events,create event rules>> which enable them.
7055 LTTng-UST state dump tracepoints start with `lttng_ust_statedump:`.
7056 LTTng-modules state dump tracepoints start with `lttng_statedump_`.
7060 [[persistent-memory-file-systems]]
7061 === Record trace data on persistent memory file systems
7063 https://en.wikipedia.org/wiki/Non-volatile_random-access_memory[Non-volatile random-access memory]
7064 (NVRAM) is random-access memory that retains its information when power
7065 is turned off (non-volatile). Systems with such memory can store data
7066 structures in RAM and retrieve them after a reboot, without flushing
7067 to typical _storage_.
7069 Linux supports NVRAM file systems thanks to either
7070 http://pramfs.sourceforge.net/[PRAMFS] or
7071 https://www.kernel.org/doc/Documentation/filesystems/dax.txt[DAX]{nbsp}+{nbsp}http://lkml.iu.edu/hypermail/linux/kernel/1504.1/03463.html[pmem]
7072 (requires Linux{nbsp}4.1+).
7074 This section does not describe how to operate such file systems;
7075 we assume that you have a working persistent memory file system.
7077 When you create a <<tracing-session,tracing session>>, you can specify
7078 the path of the shared memory holding the sub-buffers. If you specify a
7079 location on an NVRAM file system, then you can retrieve the latest
7080 recorded trace data when the system reboots after a crash.
7082 To record trace data on a persistent memory file system and retrieve the
7083 trace data after a system crash:
7085 . Create a tracing session with a sub-buffer shared memory path located
7086 on an NVRAM file system:
7091 $ lttng create my-session --shm-path=/path/to/shm
7095 . Configure the tracing session as usual with the man:lttng(1)
7096 command-line tool, and <<basic-tracing-session-control,start tracing>>.
7098 . After a system crash, use the man:lttng-crash(1) command-line tool to
7099 view the trace data recorded on the NVRAM file system:
7104 $ lttng-crash /path/to/shm
7108 The binary layout of the ring buffer files is not exactly the same as
7109 the trace files layout. This is why you need to use man:lttng-crash(1)
7110 instead of your preferred trace viewer directly.
7112 To convert the ring buffer files to LTTng trace files:
7114 * Use the opt:lttng-crash(1):--extract option of man:lttng-crash(1):
7119 $ lttng-crash --extract=/path/to/trace /path/to/shm
7125 [[notif-trigger-api]]
7126 === Get notified when a channel's buffer usage is too high or too low
7128 With LTTng's $$C/C++$$ notification and trigger API, your user
7129 application can get notified when the buffer usage of one or more
7130 <<channel,channels>> becomes too low or too high. You can use this API
7131 and enable or disable <<event,event rules>> during tracing to avoid
7132 <<channel-overwrite-mode-vs-discard-mode,discarded event records>>.
7134 .Have a user application get notified when an LTTng channel's buffer usage is too high.
7136 In this example, we create and build an application which gets notified
7137 when the buffer usage of a specific LTTng channel is higher than
7138 75{nbsp}%. We only print that it is the case in the example, but we
7139 could as well use the API of <<liblttng-ctl-lttng,`liblttng-ctl`>> to
7140 disable event rules when this happens.
7142 . Create the application's C source file:
7150 #include <lttng/domain.h>
7151 #include <lttng/action/action.h>
7152 #include <lttng/action/notify.h>
7153 #include <lttng/condition/condition.h>
7154 #include <lttng/condition/buffer-usage.h>
7155 #include <lttng/condition/evaluation.h>
7156 #include <lttng/notification/channel.h>
7157 #include <lttng/notification/notification.h>
7158 #include <lttng/trigger/trigger.h>
7159 #include <lttng/endpoint.h>
7161 int main(int argc, char *argv[])
7163 int exit_status = 0;
7164 struct lttng_notification_channel *notification_channel;
7165 struct lttng_condition *condition;
7166 struct lttng_action *action;
7167 struct lttng_trigger *trigger;
7168 const char *tracing_session_name;
7169 const char *channel_name;
7172 tracing_session_name = argv[1];
7173 channel_name = argv[2];
7176 * Create a notification channel. A notification channel
7177 * connects the user application to the LTTng session daemon.
7178 * This notification channel can be used to listen to various
7179 * types of notifications.
7181 notification_channel = lttng_notification_channel_create(
7182 lttng_session_daemon_notification_endpoint);
7185 * Create a "high buffer usage" condition. In this case, the
7186 * condition is reached when the buffer usage is greater than or
7187 * equal to 75 %. We create the condition for a specific tracing
7188 * session name, channel name, and for the user space tracing
7191 * The "low buffer usage" condition type also exists.
7193 condition = lttng_condition_buffer_usage_high_create();
7194 lttng_condition_buffer_usage_set_threshold_ratio(condition, .75);
7195 lttng_condition_buffer_usage_set_session_name(
7196 condition, tracing_session_name);
7197 lttng_condition_buffer_usage_set_channel_name(condition,
7199 lttng_condition_buffer_usage_set_domain_type(condition,
7203 * Create an action (get a notification) to take when the
7204 * condition created above is reached.
7206 action = lttng_action_notify_create();
7209 * Create a trigger. A trigger associates a condition to an
7210 * action: the action is executed when the condition is reached.
7212 trigger = lttng_trigger_create(condition, action);
7214 /* Register the trigger to LTTng. */
7215 lttng_register_trigger(trigger);
7218 * Now that we have registered a trigger, a notification will be
7219 * emitted everytime its condition is met. To receive this
7220 * notification, we must subscribe to notifications that match
7221 * the same condition.
7223 lttng_notification_channel_subscribe(notification_channel,
7227 * Notification loop. You can put this in a dedicated thread to
7228 * avoid blocking the main thread.
7231 struct lttng_notification *notification;
7232 enum lttng_notification_channel_status status;
7233 const struct lttng_evaluation *notification_evaluation;
7234 const struct lttng_condition *notification_condition;
7235 double buffer_usage;
7237 /* Receive the next notification. */
7238 status = lttng_notification_channel_get_next_notification(
7239 notification_channel, ¬ification);
7242 case LTTNG_NOTIFICATION_CHANNEL_STATUS_OK:
7244 case LTTNG_NOTIFICATION_CHANNEL_STATUS_NOTIFICATIONS_DROPPED:
7246 * The session daemon can drop notifications if
7247 * a monitoring application is not consuming the
7248 * notifications fast enough.
7251 case LTTNG_NOTIFICATION_CHANNEL_STATUS_CLOSED:
7253 * The notification channel has been closed by the
7254 * session daemon. This is typically caused by a session
7255 * daemon shutting down.
7259 /* Unhandled conditions or errors. */
7265 * A notification provides, amongst other things:
7267 * * The condition that caused this notification to be
7269 * * The condition evaluation, which provides more
7270 * specific information on the evaluation of the
7273 * The condition evaluation provides the buffer usage
7274 * value at the moment the condition was reached.
7276 notification_condition = lttng_notification_get_condition(
7278 notification_evaluation = lttng_notification_get_evaluation(
7281 /* We're subscribed to only one condition. */
7282 assert(lttng_condition_get_type(notification_condition) ==
7283 LTTNG_CONDITION_TYPE_BUFFER_USAGE_HIGH);
7286 * Get the exact sampled buffer usage from the
7287 * condition evaluation.
7289 lttng_evaluation_buffer_usage_get_usage_ratio(
7290 notification_evaluation, &buffer_usage);
7293 * At this point, instead of printing a message, we
7294 * could do something to reduce the channel's buffer
7295 * usage, like disable specific events.
7297 printf("Buffer usage is %f %% in tracing session \"%s\", "
7298 "user space channel \"%s\".\n", buffer_usage * 100,
7299 tracing_session_name, channel_name);
7300 lttng_notification_destroy(notification);
7304 lttng_action_destroy(action);
7305 lttng_condition_destroy(condition);
7306 lttng_trigger_destroy(trigger);
7307 lttng_notification_channel_destroy(notification_channel);
7313 . Build the `notif-app` application, linking it to `liblttng-ctl`:
7318 $ gcc -o notif-app notif-app.c -llttng-ctl
7322 . <<creating-destroying-tracing-sessions,Create a tracing session>>,
7323 <<enabling-disabling-events,create an event rule>> matching all the
7324 user space tracepoints, and
7325 <<basic-tracing-session-control,start tracing>>:
7330 $ lttng create my-session
7331 $ lttng enable-event --userspace --all
7336 If you create the channel manually with the man:lttng-enable-channel(1)
7337 command, you can control how frequently are the current values of the
7338 channel's properties sampled to evaluate user conditions with the
7339 opt:lttng-enable-channel(1):--monitor-timer option.
7341 . Run the `notif-app` application. This program accepts the
7342 <<tracing-session,tracing session>> name and the user space channel
7343 name as its two first arguments. The channel which LTTng automatically
7344 creates with the man:lttng-enable-event(1) command above is named
7350 $ ./notif-app my-session channel0
7354 . In another terminal, run an application with a very high event
7355 throughput so that the 75{nbsp}% buffer usage condition is reached.
7357 In the first terminal, the application should print lines like this:
7360 Buffer usage is 81.45197 % in tracing session "my-session", user space
7364 If you don't see anything, try modifying the condition in
7365 path:{notif-app.c} to a lower value (0.1, for example), rebuilding it
7366 (step{nbsp}2) and running it again (step{nbsp}4).
7373 [[lttng-modules-ref]]
7374 === noch:{LTTng-modules}
7378 [[lttng-tracepoint-enum]]
7379 ==== `LTTNG_TRACEPOINT_ENUM()` usage
7381 Use the `LTTNG_TRACEPOINT_ENUM()` macro to define an enumeration:
7385 LTTNG_TRACEPOINT_ENUM(name, TP_ENUM_VALUES(entries))
7390 * `name` with the name of the enumeration (C identifier, unique
7391 amongst all the defined enumerations).
7392 * `entries` with a list of enumeration entries.
7394 The available enumeration entry macros are:
7396 +ctf_enum_value(__name__, __value__)+::
7397 Entry named +__name__+ mapped to the integral value +__value__+.
7399 +ctf_enum_range(__name__, __begin__, __end__)+::
7400 Entry named +__name__+ mapped to the range of integral values between
7401 +__begin__+ (included) and +__end__+ (included).
7403 +ctf_enum_auto(__name__)+::
7404 Entry named +__name__+ mapped to the integral value following the
7405 last mapping's value.
7407 The last value of a `ctf_enum_value()` entry is its +__value__+
7410 The last value of a `ctf_enum_range()` entry is its +__end__+ parameter.
7412 If `ctf_enum_auto()` is the first entry in the list, its integral
7415 Use the `ctf_enum()` <<lttng-modules-tp-fields,field definition macro>>
7416 to use a defined enumeration as a tracepoint field.
7418 .Define an enumeration with `LTTNG_TRACEPOINT_ENUM()`.
7422 LTTNG_TRACEPOINT_ENUM(
7425 ctf_enum_auto("AUTO: EXPECT 0")
7426 ctf_enum_value("VALUE: 23", 23)
7427 ctf_enum_value("VALUE: 27", 27)
7428 ctf_enum_auto("AUTO: EXPECT 28")
7429 ctf_enum_range("RANGE: 101 TO 303", 101, 303)
7430 ctf_enum_auto("AUTO: EXPECT 304")
7438 [[lttng-modules-tp-fields]]
7439 ==== Tracepoint fields macros (for `TP_FIELDS()`)
7441 [[tp-fast-assign]][[tp-struct-entry]]The available macros to define
7442 tracepoint fields, which must be listed within `TP_FIELDS()` in
7443 `LTTNG_TRACEPOINT_EVENT()`, are:
7445 [role="func-desc growable",cols="asciidoc,asciidoc"]
7446 .Available macros to define LTTng-modules tracepoint fields
7448 |Macro |Description and parameters
7451 +ctf_integer(__t__, __n__, __e__)+
7453 +ctf_integer_nowrite(__t__, __n__, __e__)+
7455 +ctf_user_integer(__t__, __n__, __e__)+
7457 +ctf_user_integer_nowrite(__t__, __n__, __e__)+
7459 Standard integer, displayed in base{nbsp}10.
7462 Integer C type (`int`, `long`, `size_t`, ...).
7468 Argument expression.
7471 +ctf_integer_hex(__t__, __n__, __e__)+
7473 +ctf_user_integer_hex(__t__, __n__, __e__)+
7475 Standard integer, displayed in base{nbsp}16.
7484 Argument expression.
7486 |+ctf_integer_oct(__t__, __n__, __e__)+
7488 Standard integer, displayed in base{nbsp}8.
7497 Argument expression.
7500 +ctf_integer_network(__t__, __n__, __e__)+
7502 +ctf_user_integer_network(__t__, __n__, __e__)+
7504 Integer in network byte order (big-endian), displayed in base{nbsp}10.
7513 Argument expression.
7516 +ctf_integer_network_hex(__t__, __n__, __e__)+
7518 +ctf_user_integer_network_hex(__t__, __n__, __e__)+
7520 Integer in network byte order, displayed in base{nbsp}16.
7529 Argument expression.
7532 +ctf_enum(__N__, __t__, __n__, __e__)+
7534 +ctf_enum_nowrite(__N__, __t__, __n__, __e__)+
7536 +ctf_user_enum(__N__, __t__, __n__, __e__)+
7538 +ctf_user_enum_nowrite(__N__, __t__, __n__, __e__)+
7543 Name of a <<lttng-tracepoint-enum,previously defined enumeration>>.
7546 Integer C type (`int`, `long`, `size_t`, ...).
7552 Argument expression.
7555 +ctf_string(__n__, __e__)+
7557 +ctf_string_nowrite(__n__, __e__)+
7559 +ctf_user_string(__n__, __e__)+
7561 +ctf_user_string_nowrite(__n__, __e__)+
7563 Null-terminated string; undefined behavior if +__e__+ is `NULL`.
7569 Argument expression.
7572 +ctf_array(__t__, __n__, __e__, __s__)+
7574 +ctf_array_nowrite(__t__, __n__, __e__, __s__)+
7576 +ctf_user_array(__t__, __n__, __e__, __s__)+
7578 +ctf_user_array_nowrite(__t__, __n__, __e__, __s__)+
7580 Statically-sized array of integers.
7583 Array element C type.
7589 Argument expression.
7595 +ctf_array_bitfield(__t__, __n__, __e__, __s__)+
7597 +ctf_array_bitfield_nowrite(__t__, __n__, __e__, __s__)+
7599 +ctf_user_array_bitfield(__t__, __n__, __e__, __s__)+
7601 +ctf_user_array_bitfield_nowrite(__t__, __n__, __e__, __s__)+
7603 Statically-sized array of bits.
7605 The type of +__e__+ must be an integer type. +__s__+ is the number
7606 of elements of such type in +__e__+, not the number of bits.
7609 Array element C type.
7615 Argument expression.
7621 +ctf_array_text(__t__, __n__, __e__, __s__)+
7623 +ctf_array_text_nowrite(__t__, __n__, __e__, __s__)+
7625 +ctf_user_array_text(__t__, __n__, __e__, __s__)+
7627 +ctf_user_array_text_nowrite(__t__, __n__, __e__, __s__)+
7629 Statically-sized array, printed as text.
7631 The string does not need to be null-terminated.
7634 Array element C type (always `char`).
7640 Argument expression.
7646 +ctf_sequence(__t__, __n__, __e__, __T__, __E__)+
7648 +ctf_sequence_nowrite(__t__, __n__, __e__, __T__, __E__)+
7650 +ctf_user_sequence(__t__, __n__, __e__, __T__, __E__)+
7652 +ctf_user_sequence_nowrite(__t__, __n__, __e__, __T__, __E__)+
7654 Dynamically-sized array of integers.
7656 The type of +__E__+ must be unsigned.
7659 Array element C type.
7665 Argument expression.
7668 Length expression C type.
7674 +ctf_sequence_hex(__t__, __n__, __e__, __T__, __E__)+
7676 +ctf_user_sequence_hex(__t__, __n__, __e__, __T__, __E__)+
7678 Dynamically-sized array of integers, displayed in base{nbsp}16.
7680 The type of +__E__+ must be unsigned.
7683 Array element C type.
7689 Argument expression.
7692 Length expression C type.
7697 |+ctf_sequence_network(__t__, __n__, __e__, __T__, __E__)+
7699 Dynamically-sized array of integers in network byte order (big-endian),
7700 displayed in base{nbsp}10.
7702 The type of +__E__+ must be unsigned.
7705 Array element C type.
7711 Argument expression.
7714 Length expression C type.
7720 +ctf_sequence_bitfield(__t__, __n__, __e__, __T__, __E__)+
7722 +ctf_sequence_bitfield_nowrite(__t__, __n__, __e__, __T__, __E__)+
7724 +ctf_user_sequence_bitfield(__t__, __n__, __e__, __T__, __E__)+
7726 +ctf_user_sequence_bitfield_nowrite(__t__, __n__, __e__, __T__, __E__)+
7728 Dynamically-sized array of bits.
7730 The type of +__e__+ must be an integer type. +__s__+ is the number
7731 of elements of such type in +__e__+, not the number of bits.
7733 The type of +__E__+ must be unsigned.
7736 Array element C type.
7742 Argument expression.
7745 Length expression C type.
7751 +ctf_sequence_text(__t__, __n__, __e__, __T__, __E__)+
7753 +ctf_sequence_text_nowrite(__t__, __n__, __e__, __T__, __E__)+
7755 +ctf_user_sequence_text(__t__, __n__, __e__, __T__, __E__)+
7757 +ctf_user_sequence_text_nowrite(__t__, __n__, __e__, __T__, __E__)+
7759 Dynamically-sized array, displayed as text.
7761 The string does not need to be null-terminated.
7763 The type of +__E__+ must be unsigned.
7765 The behaviour is undefined if +__e__+ is `NULL`.
7768 Sequence element C type (always `char`).
7774 Argument expression.
7777 Length expression C type.
7783 Use the `_user` versions when the argument expression, `e`, is
7784 a user space address. In the cases of `ctf_user_integer*()` and
7785 `ctf_user_float*()`, `&e` must be a user space address, thus `e` must
7788 The `_nowrite` versions omit themselves from the session trace, but are
7789 otherwise identical. This means the `_nowrite` fields won't be written
7790 in the recorded trace. Their primary purpose is to make some
7791 of the event context available to the
7792 <<enabling-disabling-events,event filters>> without having to
7793 commit the data to sub-buffers.
7799 Terms related to LTTng and to tracing in general:
7802 The http://diamon.org/babeltrace[Babeltrace] project, which includes:
7804 * The cmd:babeltrace (Babeltrace{nbsp}1) or cmd:babeltrace2
7805 (Babeltrace{nbsp}2) command.
7806 * Libraries with a C{nbsp}API.
7807 * Python{nbsp}3 bindings.
7808 * Plugins (Babeltrace{nbsp}2).
7810 [[def-buffering-scheme]]<<channel-buffering-schemes,buffering scheme>>::
7811 A layout of <<def-sub-buffer,sub-buffers>> applied to a given channel.
7813 [[def-channel]]<<channel,channel>>::
7814 An entity which is responsible for a set of
7815 <<def-ring-buffer,ring buffers>>.
7817 <<def-event-rule,Event rules>> are always attached to a specific
7821 A source of time for a <<def-tracer,tracer>>.
7823 [[def-consumer-daemon]]<<lttng-consumerd,consumer daemon>>::
7824 A process which is responsible for consuming the full
7825 <<def-sub-buffer,sub-buffers>> and write them to a file system or
7826 send them over the network.
7828 [[def-current-trace-chunk]]current trace chunk::
7829 A <<def-trace-chunk,trace chunk>> which includes the current content
7830 of all the <<def-tracing-session-rotation,tracing session>>'s
7831 <<def-sub-buffer,sub-buffers>> and the stream files produced since the
7832 latest event amongst:
7834 * The creation of the <<def-tracing-session,tracing session>>.
7835 * The last tracing session rotation, if any.
7837 <<channel-overwrite-mode-vs-discard-mode,discard mode>>::
7838 The <<def-event-record-loss-mode,event record loss mode>> in which
7839 the <<def-tracer,tracer>> _discards_ new event records when there's no
7840 <<def-sub-buffer,sub-buffer>> space left to store them.
7842 [[def-event]]event::
7843 The consequence of the execution of an
7844 <<def-instrumentation-point,instrumentation point>>, like a
7845 <<def-tracepoint,tracepoint>> that you manually place in some source
7846 code, or a Linux kernel kprobe.
7848 An event is said to _occur_ at a specific time. <<def-lttng,LTTng>> can
7849 take various actions upon the occurrence of an event, like record the
7850 event's payload to a <<def-sub-buffer,sub-buffer>>.
7852 [[def-event-name]]event name::
7853 The name of an <<def-event,event>>, which is also the name of the
7854 <<def-event-record,event record>>.
7856 This is also called the _instrumentation point name_.
7858 [[def-event-record]]event record::
7859 A record, in a <<def-trace,trace>>, of the payload of an
7860 <<def-event,event>> which occured.
7862 [[def-event-record-loss-mode]]<<channel-overwrite-mode-vs-discard-mode,event record loss mode>>::
7863 The mechanism by which event records of a given
7864 <<def-channel,channel>> are lost (not recorded) when there is no
7865 <<def-sub-buffer,sub-buffer>> space left to store them.
7867 [[def-event-rule]]<<event,event rule>>::
7868 Set of conditions which must be satisfied for one or more occuring
7869 <<def-event,events>> to be recorded.
7871 `java.util.logging`::
7873 https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[core logging facilities].
7875 <<instrumenting,instrumentation>>::
7876 The use of <<def-lttng,LTTng>> probes to make a piece of software
7879 [[def-instrumentation-point]]instrumentation point::
7880 A point in the execution path of a piece of software that, when
7881 reached by this execution, can emit an <<def-event,event>>.
7883 instrumentation point name::
7884 See _<<def-event-name,event name>>_.
7887 A http://logging.apache.org/log4j/1.2/[logging library] for Java
7888 developed by the Apache Software Foundation.
7891 Level of severity of a log statement or user space
7892 <<def-instrumentation-point,instrumentation point>>.
7894 [[def-lttng]]LTTng::
7895 The _Linux Trace Toolkit: next generation_ project.
7897 <<lttng-cli,cmd:lttng>>::
7898 A command-line tool provided by the <<def-lttng-tools,LTTng-tools>>
7899 project which you can use to send and receive control messages to and
7900 from a <<def-session-daemon,session daemon>>.
7903 The https://github.com/lttng/lttng-analyses[LTTng analyses] project,
7904 which is a set of analyzing programs that you can use to obtain a
7905 higher level view of an <<def-lttng,LTTng>> <<def-trace,trace>>.
7907 cmd:lttng-consumerd::
7908 The name of the <<def-consumer-daemon,consumer daemon>> program.
7911 A utility provided by the <<def-lttng-tools,LTTng-tools>> project
7912 which can convert <<def-ring-buffer,ring buffer>> files (usually
7913 <<persistent-memory-file-systems,saved on a persistent memory file
7914 system>>) to <<def-trace,trace>> files.
7916 See man:lttng-crash(1).
7918 LTTng Documentation::
7921 <<lttng-live,LTTng live>>::
7922 A communication protocol between the <<lttng-relayd,relay daemon>> and
7923 live viewers which makes it possible to see <<def-event-record,event
7924 records>> "live", as they are received by the
7925 <<def-relay-daemon,relay daemon>>.
7927 <<lttng-modules,LTTng-modules>>::
7928 The https://github.com/lttng/lttng-modules[LTTng-modules] project,
7929 which contains the Linux kernel modules to make the Linux kernel
7930 <<def-instrumentation-point,instrumentation points>> available for
7931 <<def-lttng,LTTng>> tracing.
7934 The name of the <<def-relay-daemon,relay daemon>> program.
7936 cmd:lttng-sessiond::
7937 The name of the <<def-session-daemon,session daemon>> program.
7939 [[def-lttng-tools]]LTTng-tools::
7940 The https://github.com/lttng/lttng-tools[LTTng-tools] project, which
7941 contains the various programs and libraries used to
7942 <<controlling-tracing,control tracing>>.
7944 [[def-lttng-ust]]<<lttng-ust,LTTng-UST>>::
7945 The https://github.com/lttng/lttng-ust[LTTng-UST] project, which
7946 contains libraries to instrument
7947 <<def-user-application,user applications>>.
7949 <<lttng-ust-agents,LTTng-UST Java agent>>::
7950 A Java package provided by the <<def-lttng-ust,LTTng-UST>> project to
7951 allow the LTTng instrumentation of `java.util.logging` and Apache
7952 log4j{nbsp}1.2 logging statements.
7954 <<lttng-ust-agents,LTTng-UST Python agent>>::
7955 A Python package provided by the <<def-lttng-ust,LTTng-UST>> project
7956 to allow the <<def-lttng,LTTng>> instrumentation of Python logging
7959 <<channel-overwrite-mode-vs-discard-mode,overwrite mode>>::
7960 The <<def-event-record-loss-mode,event record loss mode>> in which new
7961 <<def-event-record,event records>> _overwrite_ older event records
7962 when there's no <<def-sub-buffer,sub-buffer>> space left to store
7965 <<channel-buffering-schemes,per-process buffering>>::
7966 A <<def-buffering-scheme,buffering scheme>> in which each instrumented
7967 process has its own <<def-sub-buffer,sub-buffers>> for a given user
7968 space <<def-channel,channel>>.
7970 <<channel-buffering-schemes,per-user buffering>>::
7971 A <<def-buffering-scheme,buffering scheme>> in which all the processes
7972 of a Unix user share the same <<def-sub-buffer,sub-buffers>> for a
7973 given user space <<def-channel,channel>>.
7975 [[def-relay-daemon]]<<lttng-relayd,relay daemon>>::
7976 A process which is responsible for receiving the <<def-trace,trace>>
7977 data which a distant <<def-consumer-daemon,consumer daemon>> sends.
7979 [[def-ring-buffer]]ring buffer::
7980 A set of <<def-sub-buffer,sub-buffers>>.
7983 See _<<def-tracing-session-rotation,tracing session rotation>>_.
7985 [[def-session-daemon]]<<lttng-sessiond,session daemon>>::
7986 A process which receives control commands from you and orchestrates
7987 the <<def-tracer,tracers>> and various <<def-lttng,LTTng>> daemons.
7989 <<taking-a-snapshot,snapshot>>::
7990 A copy of the current data of all the <<def-sub-buffer,sub-buffers>>
7991 of a given <<def-tracing-session,tracing session>>, saved as
7992 <<def-trace,trace>> files.
7994 [[def-sub-buffer]]sub-buffer::
7995 One part of an <<def-lttng,LTTng>> <<def-ring-buffer,ring buffer>>
7996 which contains <<def-event-record,event records>>.
7999 The time information attached to an <<def-event,event>> when it is
8002 [[def-trace]]trace (_noun_)::
8005 * One http://diamon.org/ctf/[CTF] metadata stream file.
8006 * One or more CTF data stream files which are the concatenations of one
8007 or more flushed <<def-sub-buffer,sub-buffers>>.
8009 [[def-trace-verb]]trace (_verb_)::
8010 The action of recording the <<def-event,events>> emitted by an
8011 application or by a system, or to initiate such recording by
8012 controlling a <<def-tracer,tracer>>.
8014 [[def-trace-chunk]]trace chunk::
8015 A self-contained <<def-trace,trace>> which is part of a
8016 <<def-tracing-session,tracing session>>. Each
8017 <<def-tracing-session-rotation, tracing session rotation>> produces a
8018 <<def-trace-chunk-archive,trace chunk archive>>.
8020 [[def-trace-chunk-archive]]trace chunk archive::
8021 The result of a <<def-tracing-session-rotation, tracing session rotation>>.
8023 <<def-lttng,LTTng>> does not manage any trace chunk archive, even if its
8024 containing <<def-tracing-session,tracing session>> is still active: you
8025 are free to read it, modify it, move it, or remove it.
8028 The http://tracecompass.org[Trace Compass] project and application.
8030 [[def-tracepoint]]tracepoint::
8031 An instrumentation point using the tracepoint mechanism of the Linux
8032 kernel or of <<def-lttng-ust,LTTng-UST>>.
8034 tracepoint definition::
8035 The definition of a single <<def-tracepoint,tracepoint>>.
8038 The name of a <<def-tracepoint,tracepoint>>.
8040 [[def-tracepoint-provider]]tracepoint provider::
8041 A set of functions providing <<def-tracepoint,tracepoints>> to an
8042 instrumented <<def-user-application,user application>>.
8044 Not to be confused with a <<def-tracepoint-provider-package,tracepoint
8045 provider package>>: many tracepoint providers can exist within a
8046 tracepoint provider package.
8048 [[def-tracepoint-provider-package]]tracepoint provider package::
8049 One or more <<def-tracepoint-provider,tracepoint providers>> compiled
8050 as an https://en.wikipedia.org/wiki/Object_file[object file] or as a
8051 link:https://en.wikipedia.org/wiki/Library_(computing)#Shared_libraries[shared
8054 [[def-tracer]]tracer::
8055 A software which records emitted <<def-event,events>>.
8057 <<domain,tracing domain>>::
8058 A namespace for <<def-event,event>> sources.
8060 <<tracing-group,tracing group>>::
8061 The Unix group in which a Unix user can be to be allowed to
8062 <<def-trace-verb,trace>> the Linux kernel.
8064 [[def-tracing-session]]<<tracing-session,tracing session>>::
8065 A stateful dialogue between you and a <<lttng-sessiond,session daemon>>.
8067 [[def-tracing-session-rotation]]<<session-rotation,tracing session rotation>>::
8068 The action of archiving the
8069 <<def-current-trace-chunk,current trace chunk>> of a
8070 <<def-tracing-session,tracing session>>.
8072 [[def-user-application]]user application::
8073 An application running in user space, as opposed to a Linux kernel
8074 module, for example.