1 The LTTng Documentation
2 =======================
3 Philippe Proulx <pproulx@efficios.com>
7 include::../common/copyright.txt[]
10 include::../common/welcome.txt[]
13 include::../common/audience.txt[]
17 === What's in this documentation?
19 The LTTng Documentation is divided into the following sections:
21 * **<<nuts-and-bolts,Nuts and bolts>>** explains the
22 rudiments of software tracing and the rationale behind the
25 You can skip this section if you’re familiar with software tracing and
26 with the LTTng project.
28 * **<<installing-lttng,Installation>>** describes the steps to
29 install the LTTng packages on common Linux distributions and from
32 You can skip this section if you already properly installed LTTng on
35 * **<<getting-started,Quick start>>** is a concise guide to
36 getting started quickly with LTTng kernel and user space tracing.
38 We recommend this section if you're new to LTTng or to software tracing
41 You can skip this section if you're not new to LTTng.
43 * **<<core-concepts,Core concepts>>** explains the concepts at
46 It's a good idea to become familiar with the core concepts
47 before attempting to use the toolkit.
49 * **<<plumbing,Components of LTTng>>** describes the various components
50 of the LTTng machinery, like the daemons, the libraries, and the
51 command-line interface.
52 * **<<instrumenting,Instrumentation>>** shows different ways to
53 instrument user applications and the Linux kernel.
55 Instrumenting source code is essential to provide a meaningful
58 You can skip this section if you do not have a programming background.
60 * **<<controlling-tracing,Tracing control>>** is divided into topics
61 which demonstrate how to use the vast array of features that
62 LTTng{nbsp}{revision} offers.
63 * **<<reference,Reference>>** contains reference tables.
64 * **<<glossary,Glossary>>** is a specialized dictionary of terms related
65 to LTTng or to the field of software tracing.
68 include::../common/convention.txt[]
71 include::../common/acknowledgements.txt[]
75 == What's new in LTTng {revision}?
77 * **Tracing control**:
78 ** You can attach <<java-application-context,Java application-specific
79 context fields>> to a <<channel,channel>> with the
80 man:lttng-add-context(1) command:
85 lttng add-context --jul --type='$app.retriever:cur_msg_id'
89 Here, `$app` is the prefix of all application-specific context fields,
90 `retriever` names a _context information retriever_ defined at the
91 application level, and `cur_msg_id` names a context field read from this
94 Both the `java.util.logging` and Apache log4j <<domain,tracing domains>>
97 ** You can use Java application-specific <<adding-context,context>>
98 fields in the <<enabling-disabling-events,filter expression>> of an
104 lttng enable-event --log4j my_logger \
105 --filter='$app.retriever:cur_msg_id == 23'
109 ** New `lttng status` command which is the equivalent of +lttng list
110 __CUR__+, where +__CUR__+ is the name of the current
111 <<tracing-session,tracing session>>.
113 See man:lttng-status(1).
115 ** New `lttng metadata regenerate` command to
116 <<metadata-regenerate,regenerate the metadata file of an LTTng
117 trace>> at any moment. This command is meant to be used to resample
118 the wall time following a major
119 https://en.wikipedia.org/wiki/Network_Time_Protocol[NTP] correction
120 so that a system which boots with an incorrect wall time can be
121 traced before its wall time is NTP-corrected.
123 See man:lttng-metadata(1).
125 ** New command-line interface warnings when <<event,event records>> or
126 whole sub-buffers are
127 <<channel-overwrite-mode-vs-discard-mode,lost>>. The warning messages
128 are printed when a <<tracing-session,tracing session>> is
129 <<basic-tracing-session-control,stopped>> (man:lttng-stop(1)
132 * **User space tracing**:
133 ** Shared object base address dump in order to map <<event,event
134 records>> to original source location (file and line number).
136 If you attach the `ip` and `vpid` <<adding-context,context fields>> to a
137 user space <<channel,channel>> and if you use the
138 <<liblttng-ust-dl,path:{liblttng-ust-dl.so} helper>>, you can retrieve
139 the source location where a given event record was generated.
141 The http://diamon.org/babeltrace/[Babeltrace] trace viewer supports this
142 state dump and those context fields since version 1.4 to print the
143 source location of a given event record. http://tracecompass.org/[Trace
144 Compass] also supports this since version 2.0.
146 ** A <<java-application,Java application>> which uses
147 `java.util.logging` now adds an LTTng-UST log handler to the desired
150 The previous workflow was to initialize the LTTng-UST Java agent
151 by calling `LTTngAgent.getLTTngAgent()`. This had the effect of adding
152 an LTTng-UST log handler to the root loggers.
154 ** A <<java-application,Java application>> which uses Apache log4j now
155 adds an LTTng-UST log appender to the desired log4j loggers.
157 The previous workflow was to initialize the LTTng-UST Java agent
158 by calling `LTTngAgent.getLTTngAgent()`. This had the effect of adding
159 an LTTng-UST appender to the root loggers.
161 ** Any <<java-application,Java application>> can provide
162 <<java-application-context,dynamic context fields>> while running
163 thanks to a new API provided by the <<lttng-ust-agents,LTTng-UST Java
164 agent>>. You can require LTTng to record specific context fields in
165 event records, and you can use them in the filter expression of
166 <<event,event rules>>.
168 * **Linux kernel tracing**:
169 ** The LTTng kernel modules can now be built into a Linux kernel image,
170 that is, not as loadable modules.
173 https://github.com/lttng/lttng-modules/blob/stable-{revision}/README.md#kernel-built-in-support[`README.md`]
176 ** New instrumentation:
177 *** ARM64 architecture support.
179 *** x86 `irq_vectors`.
180 ** New <<adding-context,context fields>>:
183 *** `need_reschedule`
184 *** `migratable` (specific to RT-Preempt)
185 ** Clock source plugin support for advanced cases where a custom source
186 of time is needed to timestamp LTTng event records.
188 See https://github.com/lttng/lttng-modules/blob/stable-{revision}/lttng-clock.h[`lttng-clock.h`]
189 for an overview of the small API.
192 ** The link:/man[man pages] of the man:lttng(1) command-line tool are
193 split into one man page per command (à la Git), for example:
198 man lttng-enable-event
202 You can also use the `--help` option of any man:lttng(1) command to
205 The content and formatting of all the LTTng man pages has improved
212 What is LTTng? As its name suggests, the _Linux Trace Toolkit: next
213 generation_ is a modern toolkit for tracing Linux systems and
214 applications. So your first question might be:
221 As the history of software engineering progressed and led to what
222 we now take for granted--complex, numerous and
223 interdependent software applications running in parallel on
224 sophisticated operating systems like Linux--the authors of such
225 components, software developers, began feeling a natural
226 urge to have tools that would ensure the robustness and good performance
227 of their masterpieces.
229 One major achievement in this field is, inarguably, the
230 https://www.gnu.org/software/gdb/[GNU debugger (GDB)],
231 an essential tool for developers to find and fix bugs. But even the best
232 debugger won't help make your software run faster, and nowadays, faster
233 software means either more work done by the same hardware, or cheaper
234 hardware for the same work.
236 A _profiler_ is often the tool of choice to identify performance
237 bottlenecks. Profiling is suitable to identify _where_ performance is
238 lost in a given software. The profiler outputs a profile, a statistical
239 summary of observed events, which you may use to discover which
240 functions took the most time to execute. However, a profiler won't
241 report _why_ some identified functions are the bottleneck. Bottlenecks
242 might only occur when specific conditions are met, conditions that are
243 sometimes impossible to capture by a statistical profiler, or impossible
244 to reproduce with an application altered by the overhead of an
245 event-based profiler. For a thorough investigation of software
246 performance issues, a history of execution is essential, with the
247 recorded values of variables and context fields you choose, and
248 with as little influence as possible on the instrumented software. This
249 is where tracing comes in handy.
251 _Tracing_ is a technique used to understand what goes on in a running
252 software system. The software used for tracing is called a _tracer_,
253 which is conceptually similar to a tape recorder. When recording,
254 specific instrumentation points placed in the software source code
255 generate events that are saved on a giant tape: a _trace_ file. You
256 can trace user applications and the operating system at the same time,
257 opening the possibility of resolving a wide range of problems that would
258 otherwise be extremely challenging.
260 Tracing is often compared to _logging_. However, tracers and loggers are
261 two different tools, serving two different purposes. Tracers are
262 designed to record much lower-level events that occur much more
263 frequently than log messages, often in the range of thousands per
264 second, with very little execution overhead. Logging is more appropriate
265 for a very high-level analysis of less frequent events: user accesses,
266 exceptional conditions (errors and warnings, for example), database
267 transactions, instant messaging communications, and such. Simply put,
268 logging is one of the many use cases that can be satisfied with tracing.
270 The list of recorded events inside a trace file can be read manually
271 like a log file for the maximum level of detail, but it is generally
272 much more interesting to perform application-specific analyses to
273 produce reduced statistics and graphs that are useful to resolve a
274 given problem. Trace viewers and analyzers are specialized tools
277 In the end, this is what LTTng is: a powerful, open source set of
278 tools to trace the Linux kernel and user applications at the same time.
279 LTTng is composed of several components actively maintained and
280 developed by its link:/community/#where[community].
283 [[lttng-alternatives]]
284 === Alternatives to noch:{LTTng}
286 Excluding proprietary solutions, a few competing software tracers
289 * https://github.com/dtrace4linux/linux[dtrace4linux] is a port of
290 Sun Microsystems's DTrace to Linux. The cmd:dtrace tool interprets
291 user scripts and is responsible for loading code into the
292 Linux kernel for further execution and collecting the outputted data.
293 * https://en.wikipedia.org/wiki/Berkeley_Packet_Filter[eBPF] is a
294 subsystem in the Linux kernel in which a virtual machine can execute
295 programs passed from the user space to the kernel. You can attach
296 such programs to tracepoints and KProbes thanks to a system call, and
297 they can output data to the user space when executed thanks to
298 different mechanisms (pipe, VM register values, and eBPF maps, to name
300 * https://www.kernel.org/doc/Documentation/trace/ftrace.txt[ftrace]
301 is the de facto function tracer of the Linux kernel. Its user
302 interface is a set of special files in sysfs.
303 * https://perf.wiki.kernel.org/[perf] is
304 a performance analyzing tool for Linux which supports hardware
305 performance counters, tracepoints, as well as other counters and
306 types of probes. perf's controlling utility is the cmd:perf command
308 * http://linux.die.net/man/1/strace[strace]
309 is a command-line utility which records system calls made by a
310 user process, as well as signal deliveries and changes of process
311 state. strace makes use of https://en.wikipedia.org/wiki/Ptrace[ptrace]
312 to fulfill its function.
313 * http://www.sysdig.org/[sysdig], like SystemTap, uses scripts to
314 analyze Linux kernel events. You write scripts, or _chisels_ in
315 sysdig's jargon, in Lua and sysdig executes them while the system is
316 being traced or afterwards. sysdig's interface is the cmd:sysdig
317 command-line tool as well as the curses-based cmd:csysdig tool.
318 * https://sourceware.org/systemtap/[SystemTap] is a Linux kernel and
319 user space tracer which uses custom user scripts to produce plain text
320 traces. SystemTap converts the scripts to the C language, and then
321 compiles them as Linux kernel modules which are loaded to produce
322 trace data. SystemTap's primary user interface is the cmd:stap
325 The main distinctive features of LTTng is that it produces correlated
326 kernel and user space traces, as well as doing so with the lowest
327 overhead amongst other solutions. It produces trace files in the
328 http://diamon.org/ctf[CTF] format, a file format optimized
329 for the production and analyses of multi-gigabyte data.
331 LTTng is the result of more than 10 years of active open source
332 development by a community of passionate developers.
333 LTTng{nbsp}{revision} is currently available on major desktop and server
336 The main interface for tracing control is a single command-line tool
337 named cmd:lttng. The latter can create several tracing sessions, enable
338 and disable events on the fly, filter events efficiently with custom
339 user expressions, start and stop tracing, and much more. LTTng can
340 record the traces on the file system or send them over the network, and
341 keep them totally or partially. You can view the traces once tracing
342 becomes inactive or in real-time.
344 <<installing-lttng,Install LTTng now>> and
345 <<getting-started,start tracing>>!
351 **LTTng** is a set of software <<plumbing,components>> which interact to
352 <<instrumenting,instrument>> the Linux kernel and user applications, and
353 to <<controlling-tracing,control tracing>> (start and stop
354 tracing, enable and disable event rules, and the rest). Those
355 components are bundled into the following packages:
357 * **LTTng-tools**: Libraries and command-line interface to
359 * **LTTng-modules**: Linux kernel modules to instrument and
361 * **LTTng-UST**: Libraries and Java/Python packages to instrument and
362 trace user applications.
364 Most distributions mark the LTTng-modules and LTTng-UST packages as
365 optional when installing LTTng-tools (which is always required). In the
366 following sections, we always provide the steps to install all three,
369 * You only need to install LTTng-modules if you intend to trace the
371 * You only need to install LTTng-UST if you intend to trace user
375 .Availability of LTTng{nbsp}{revision} for major Linux distributions as of 2 December 2016.
377 |Distribution |Available in releases |Alternatives
380 |<<ubuntu,Ubuntu{nbsp}16.10 _Yakkety Yak_>>.
381 |LTTng{nbsp}{revision} for Ubuntu{nbsp}14.04 _Trusty Tahr_
382 and Ubuntu{nbsp}16.04 _Xenial Xerus_:
383 <<ubuntu-ppa,use the LTTng Stable{nbsp}{revision} PPA>>.
385 LTTng{nbsp}2.9 for Ubuntu{nbsp}14.04 _Trusty Tahr_
386 and Ubuntu{nbsp}16.04 _Xenial Xerus_:
387 link:/docs/v2.9#doc-ubuntu-ppa[use the LTTng Stable{nbsp}2.9 PPA].
389 <<building-from-source,Build LTTng{nbsp}{revision} from source>> for
390 other Ubuntu releases.
393 |<<fedora,Fedora{nbsp}25>>.
394 |<<building-from-source,Build LTTng{nbsp}{revision} from source>> for
395 other Fedora releases.
398 |<<debian,Debian "stretch" (testing)>>.
399 |<<building-from-source,Build LTTng{nbsp}{revision} from source>> for
400 previous Debian releases.
404 |<<building-from-source,Build LTTng{nbsp}{revision} from source>>.
408 |link:/docs/v2.9#doc-arch-linux[LTTng{nbsp}2.9 from the AUR].
411 |<<alpine-linux,Alpine Linux "edge">>.
412 |LTTng{nbsp}{revision} for Alpine Linux{nbsp}3.5 (not released yet).
414 <<building-from-source,Build LTTng{nbsp}{revision} from source>> for
415 other Alpine Linux releases.
418 |See http://packages.efficios.com/[EfficiOS Enterprise Packages].
422 |<<buildroot,Buildroot 2016.11>>.
423 |<<building-from-source,Build LTTng{nbsp}{revision} from source>> for
424 other Buildroot releases.
426 |OpenEmbedded and Yocto
427 |<<oe-yocto,Yocto Project{nbsp}2.2 _Morty_>> (`openembedded-core` layer).
428 |<<building-from-source,Build LTTng{nbsp}{revision} from source>> for
429 other OpenEmbedded releases.
434 === [[ubuntu-official-repositories]]Ubuntu
436 LTTng{nbsp}{revision} is available on Ubuntu{nbsp}16.10 _Yakkety Yak_.
437 For previous releases of Ubuntu, <<ubuntu-ppa,use the LTTng
438 Stable{nbsp}{revision} PPA>>.
440 To install LTTng{nbsp}{revision} on Ubuntu{nbsp}16.10 _Yakkety Yak_:
442 . Install the main LTTng{nbsp}{revision} packages:
447 sudo apt-get install lttng-tools
448 sudo apt-get install lttng-modules-dkms
449 sudo apt-get install liblttng-ust-dev
453 . **If you need to instrument and trace
454 <<java-application,Java applications>>**, install the LTTng-UST
460 sudo apt-get install liblttng-ust-agent-java
464 . **If you need to instrument and trace
465 <<python-application,Python{nbsp}3 applications>>**, install the
466 LTTng-UST Python agent:
471 sudo apt-get install python3-lttngust
477 ==== noch:{LTTng} Stable {revision} PPA
479 The https://launchpad.net/~lttng/+archive/ubuntu/stable-{revision}[LTTng
480 Stable{nbsp}{revision} PPA] offers the latest stable
481 LTTng{nbsp}{revision} packages for:
483 * Ubuntu{nbsp}14.04 _Trusty Tahr_
484 * Ubuntu{nbsp}16.04 _Xenial Xerus_
486 To install LTTng{nbsp}{revision} from the LTTng Stable{nbsp}{revision} PPA:
488 . Add the LTTng Stable{nbsp}{revision} PPA repository and update the
494 sudo apt-add-repository ppa:lttng/stable-2.8
499 . Install the main LTTng{nbsp}{revision} packages:
504 sudo apt-get install lttng-tools
505 sudo apt-get install lttng-modules-dkms
506 sudo apt-get install liblttng-ust-dev
510 . **If you need to instrument and trace
511 <<java-application,Java applications>>**, install the LTTng-UST
517 sudo apt-get install liblttng-ust-agent-java
521 . **If you need to instrument and trace
522 <<python-application,Python{nbsp}3 applications>>**, install the
523 LTTng-UST Python agent:
528 sudo apt-get install python3-lttngust
536 To install LTTng{nbsp}{revision} on Fedora{nbsp}25:
538 . Install the LTTng-tools{nbsp}{revision} and LTTng-UST{nbsp}{revision}
544 sudo yum install lttng-tools
545 sudo yum install lttng-ust
549 . Download, build, and install the latest LTTng-modules{nbsp}{revision}:
555 wget http://lttng.org/files/lttng-modules/lttng-modules-latest-2.8.tar.bz2 &&
556 tar -xf lttng-modules-latest-2.8.tar.bz2 &&
557 cd lttng-modules-2.8.* &&
559 sudo make modules_install &&
565 .Java and Python application instrumentation and tracing
567 If you need to instrument and trace <<java-application,Java
568 applications>> on openSUSE, you need to build and install
569 LTTng-UST{nbsp}{revision} <<building-from-source,from source>> and pass
570 the `--enable-java-agent-jul`, `--enable-java-agent-log4j`, or
571 `--enable-java-agent-all` options to the `configure` script, depending
572 on which Java logging framework you use.
574 If you need to instrument and trace <<python-application,Python
575 applications>> on openSUSE, you need to build and install
576 LTTng-UST{nbsp}{revision} from source and pass the
577 `--enable-python-agent` option to the `configure` script.
584 To install LTTng{nbsp}{revision} on Debian "stretch" (testing):
586 . Install the main LTTng{nbsp}{revision} packages:
591 sudo apt-get install lttng-modules-dkms
592 sudo apt-get install liblttng-ust-dev
593 sudo apt-get install lttng-tools
597 . **If you need to instrument and trace <<java-application,Java
598 applications>>**, install the LTTng-UST Java agent:
603 sudo apt-get install liblttng-ust-agent-java
607 . **If you need to instrument and trace <<python-application,Python
608 applications>>**, install the LTTng-UST Python agent:
613 sudo apt-get install python3-lttngust
621 To install LTTng-tools{nbsp}{revision} and LTTng-UST{nbsp}{revision} on
624 . Make sure your system is
625 https://wiki.alpinelinux.org/wiki/Edge[configured for "edge"].
626 . Enable the _testing_ repository by uncommenting the corresponding
627 line in path:{/etc/apk/repositories}.
628 . Add the LTTng packages:
633 sudo apk add lttng-tools
634 sudo apk add lttng-ust-dev
638 To install LTTng-modules{nbsp}{revision} (Linux kernel tracing support)
639 on Alpine Linux "edge":
641 . Add the vanilla Linux kernel:
646 apk add linux-vanilla linux-vanilla-dev
650 . Reboot with the vanilla Linux kernel.
651 . Download, build, and install the latest LTTng-modules{nbsp}{revision}:
657 wget http://lttng.org/files/lttng-modules/lttng-modules-latest-2.8.tar.bz2 &&
658 tar -xf lttng-modules-latest-2.8.tar.bz2 &&
659 cd lttng-modules-2.8.* &&
661 sudo make modules_install &&
667 [[enterprise-distributions]]
668 === RHEL, SUSE, and other enterprise distributions
670 To install LTTng on enterprise Linux distributions, such as Red Hat
671 Enterprise Linux (RHEL) and SUSE Linux Enterprise Server (SUSE), please
672 see http://packages.efficios.com/[EfficiOS Enterprise Packages].
678 To install LTTng{nbsp}{revision} on Buildroot{nbsp}2016.11:
680 . Launch the Buildroot configuration tool:
689 . In **Kernel**, check **Linux kernel**.
690 . In **Toolchain**, check **Enable WCHAR support**.
691 . In **Target packages**{nbsp}→ **Debugging, profiling and benchmark**,
692 check **lttng-modules** and **lttng-tools**.
693 . In **Target packages**{nbsp}→ **Libraries**{nbsp}→
694 **Other**, check **lttng-libust**.
698 === OpenEmbedded and Yocto
700 LTTng{nbsp}{revision} recipes are available in the
701 http://layers.openembedded.org/layerindex/branch/master/layer/openembedded-core/[`openembedded-core`]
702 layer for Yocto Project{nbsp}2.2 _Morty_ under the following names:
708 With BitBake, the simplest way to include LTTng recipes in your target
709 image is to add them to `IMAGE_INSTALL_append` in path:{conf/local.conf}:
712 IMAGE_INSTALL_append = " lttng-tools lttng-modules lttng-ust"
717 . Select a machine and an image recipe.
718 . Click **Edit image recipe**.
719 . Under the **All recipes** tab, search for **lttng**.
720 . Check the desired LTTng recipes.
723 .Java and Python application instrumentation and tracing
725 If you need to instrument and trace <<java-application,Java
726 applications>> on openSUSE, you need to build and install
727 LTTng-UST{nbsp}{revision} <<building-from-source,from source>> and pass
728 the `--enable-java-agent-jul`, `--enable-java-agent-log4j`, or
729 `--enable-java-agent-all` options to the `configure` script, depending
730 on which Java logging framework you use.
732 If you need to instrument and trace <<python-application,Python
733 applications>> on openSUSE, you need to build and install
734 LTTng-UST{nbsp}{revision} from source and pass the
735 `--enable-python-agent` option to the `configure` script.
739 [[building-from-source]]
740 === Build from source
742 To build and install LTTng{nbsp}{revision} from source:
744 . Using your distribution's package manager, or from source, install
745 the following dependencies of LTTng-tools and LTTng-UST:
748 * https://sourceforge.net/projects/libuuid/[libuuid]
749 * http://directory.fsf.org/wiki/Popt[popt]
750 * http://liburcu.org/[Userspace RCU]
751 * http://www.xmlsoft.org/[libxml2]
754 . Download, build, and install the latest LTTng-modules{nbsp}{revision}:
760 wget http://lttng.org/files/lttng-modules/lttng-modules-latest-2.8.tar.bz2 &&
761 tar -xf lttng-modules-latest-2.8.tar.bz2 &&
762 cd lttng-modules-2.8.* &&
764 sudo make modules_install &&
769 . Download, build, and install the latest LTTng-UST{nbsp}{revision}:
775 wget http://lttng.org/files/lttng-ust/lttng-ust-latest-2.8.tar.bz2 &&
776 tar -xf lttng-ust-latest-2.8.tar.bz2 &&
777 cd lttng-ust-2.8.* &&
787 .Java and Python application tracing
789 If you need to instrument and trace <<java-application,Java
790 applications>>, pass the `--enable-java-agent-jul`,
791 `--enable-java-agent-log4j`, or `--enable-java-agent-all` options to the
792 `configure` script, depending on which Java logging framework you use.
794 If you need to instrument and trace <<python-application,Python
795 applications>>, pass the `--enable-python-agent` option to the
796 `configure` script. You can set the `PYTHON` environment variable to the
797 path to the Python interpreter for which to install the LTTng-UST Python
805 By default, LTTng-UST libraries are installed to
806 dir:{/usr/local/lib}, which is the de facto directory in which to
807 keep self-compiled and third-party libraries.
809 When <<building-tracepoint-providers-and-user-application,linking an
810 instrumented user application with `liblttng-ust`>>:
812 * Append `/usr/local/lib` to the env:LD_LIBRARY_PATH environment
814 * Pass the `-L/usr/local/lib` and `-Wl,-rpath,/usr/local/lib` options to
815 man:gcc(1), man:g++(1), or man:clang(1).
819 . Download, build, and install the latest LTTng-tools{nbsp}{revision}:
825 wget http://lttng.org/files/lttng-tools/lttng-tools-latest-2.8.tar.bz2 &&
826 tar -xf lttng-tools-latest-2.8.tar.bz2 &&
827 cd lttng-tools-2.8.* &&
835 TIP: The https://github.com/eepp/vlttng[vlttng tool] can do all the
836 previous steps automatically for a given version of LTTng and confine
837 the installed files in a specific directory. This can be useful to test
838 LTTng without installing it on your system.
844 This is a short guide to get started quickly with LTTng kernel and user
847 Before you follow this guide, make sure to <<installing-lttng,install>>
850 This tutorial walks you through the steps to:
852 . <<tracing-the-linux-kernel,Trace the Linux kernel>>.
853 . <<tracing-your-own-user-application,Trace a user application>> written
855 . <<viewing-and-analyzing-your-traces,View and analyze the
859 [[tracing-the-linux-kernel]]
860 === Trace the Linux kernel
862 The following command lines start with cmd:sudo because you need root
863 privileges to trace the Linux kernel. You can avoid using cmd:sudo if
864 your Unix user is a member of the <<lttng-sessiond,tracing group>>.
866 . Create a <<tracing-session,tracing session>>:
871 sudo lttng create my-kernel-session
875 . List the available kernel tracepoints and system calls:
884 . Create an <<event,event rule>> which matches the desired event names,
885 for example `sched_switch` and `sched_process_fork`:
890 sudo lttng enable-event --kernel sched_switch,sched_process_fork
894 You can also create an event rule which _matches_ all the Linux kernel
895 tracepoints (this will generate a lot of data when tracing):
900 sudo lttng enable-event --kernel --all
913 . Do some operation on your system for a few seconds. For example,
914 load a website, or list the files of a directory.
915 . Stop tracing and destroy the tracing session:
925 The man:lttng-destroy(1) command does not destroy the trace data; it
926 only destroys the state of the tracing session.
928 By default, LTTng saves the traces in
929 +$LTTNG_HOME/lttng-traces/__name__-__date__-__time__+,
930 where +__name__+ is the tracing session name. Note that the
931 env:LTTNG_HOME environment variable defaults to `$HOME` if not set.
933 See <<viewing-and-analyzing-your-traces,View and analyze the
934 recorded events>> to view the recorded events.
937 [[tracing-your-own-user-application]]
938 === Trace a user application
940 This section steps you through a simple example to trace a
941 _Hello world_ program written in C.
943 To create the traceable user application:
945 . Create the tracepoint provider header file, which defines the
946 tracepoints and the events they can generate:
952 #undef TRACEPOINT_PROVIDER
953 #define TRACEPOINT_PROVIDER hello_world
955 #undef TRACEPOINT_INCLUDE
956 #define TRACEPOINT_INCLUDE "./hello-tp.h"
958 #if !defined(_HELLO_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
961 #include <lttng/tracepoint.h>
971 ctf_string(my_string_field, my_string_arg)
972 ctf_integer(int, my_integer_field, my_integer_arg)
976 #endif /* _HELLO_TP_H */
978 #include <lttng/tracepoint-event.h>
982 . Create the tracepoint provider package source file:
988 #define TRACEPOINT_CREATE_PROBES
989 #define TRACEPOINT_DEFINE
991 #include "hello-tp.h"
995 . Build the tracepoint provider package:
1000 gcc -c -I. hello-tp.c
1004 . Create the _Hello World_ application source file:
1011 #include "hello-tp.h"
1013 int main(int argc, char *argv[])
1017 puts("Hello, World!\nPress Enter to continue...");
1020 * The following getchar() call is only placed here for the purpose
1021 * of this demonstration, to pause the application in order for
1022 * you to have time to list its tracepoints. It is not
1028 * A tracepoint() call.
1030 * Arguments, as defined in hello-tp.h:
1032 * 1. Tracepoint provider name (required)
1033 * 2. Tracepoint name (required)
1034 * 3. my_integer_arg (first user-defined argument)
1035 * 4. my_string_arg (second user-defined argument)
1037 * Notice the tracepoint provider and tracepoint names are
1038 * NOT strings: they are in fact parts of variables that the
1039 * macros in hello-tp.h create.
1041 tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");
1043 for (x = 0; x < argc; ++x) {
1044 tracepoint(hello_world, my_first_tracepoint, x, argv[x]);
1047 puts("Quitting now!");
1048 tracepoint(hello_world, my_first_tracepoint, x * x, "x^2");
1055 . Build the application:
1064 . Link the application with the tracepoint provider package,
1065 `liblttng-ust`, and `libdl`:
1070 gcc -o hello hello.o hello-tp.o -llttng-ust -ldl
1074 Here's the whole build process:
1077 .User space tracing tutorial's build steps.
1078 image::ust-flow.png[]
1080 To trace the user application:
1082 . Run the application with a few arguments:
1087 ./hello world and beyond
1096 Press Enter to continue...
1100 . Start an LTTng <<lttng-sessiond,session daemon>>:
1105 lttng-sessiond --daemonize
1109 Note that a session daemon might already be running, for example as
1110 a service that the distribution's service manager started.
1112 . List the available user space tracepoints:
1117 lttng list --userspace
1121 You see the `hello_world:my_first_tracepoint` tracepoint listed
1122 under the `./hello` process.
1124 . Create a <<tracing-session,tracing session>>:
1129 lttng create my-user-space-session
1133 . Create an <<event,event rule>> which matches the
1134 `hello_world:my_first_tracepoint` event name:
1139 lttng enable-event --userspace hello_world:my_first_tracepoint
1152 . Go back to the running `hello` application and press Enter. The
1153 program executes all `tracepoint()` instrumentation points and exits.
1154 . Stop tracing and destroy the tracing session:
1164 The man:lttng-destroy(1) command does not destroy the trace data; it
1165 only destroys the state of the tracing session.
1167 By default, LTTng saves the traces in
1168 +$LTTNG_HOME/lttng-traces/__name__-__date__-__time__+,
1169 where +__name__+ is the tracing session name. Note that the
1170 env:LTTNG_HOME environment variable defaults to `$HOME` if not set.
1172 See <<viewing-and-analyzing-your-traces,View and analyze the
1173 recorded events>> to view the recorded events.
1176 [[viewing-and-analyzing-your-traces]]
1177 === View and analyze the recorded events
1179 Once you have completed the <<tracing-the-linux-kernel,Trace the Linux
1180 kernel>> and <<tracing-your-own-user-application,Trace a user
1181 application>> tutorials, you can inspect the recorded events.
1183 Many tools are available to read LTTng traces:
1185 * **cmd:babeltrace** is a command-line utility which converts trace
1186 formats; it supports the format that LTTng produces, CTF, as well as a
1187 basic text output which can be ++grep++ed. The cmd:babeltrace command
1188 is part of the http://diamon.org/babeltrace[Babeltrace] project.
1189 * Babeltrace also includes
1190 **https://www.python.org/[Python] bindings** so
1191 that you can easily open and read an LTTng trace with your own script,
1192 benefiting from the power of Python.
1193 * http://tracecompass.org/[**Trace Compass**]
1194 is a graphical user interface for viewing and analyzing any type of
1195 logs or traces, including LTTng's.
1196 * https://github.com/lttng/lttng-analyses[**LTTng analyses**] is a
1197 project which includes many high-level analyses of LTTng kernel
1198 traces, like scheduling statistics, interrupt frequency distribution,
1199 top CPU usage, and more.
1201 NOTE: This section assumes that the traces recorded during the previous
1202 tutorials were saved to their default location, in the
1203 dir:{$LTTNG_HOME/lttng-traces} directory. Note that the env:LTTNG_HOME
1204 environment variable defaults to `$HOME` if not set.
1207 [[viewing-and-analyzing-your-traces-bt]]
1208 ==== Use the cmd:babeltrace command-line tool
1210 The simplest way to list all the recorded events of a trace is to pass
1211 its path to cmd:babeltrace with no options:
1215 babeltrace ~/lttng-traces/my-user-space-session*
1218 cmd:babeltrace finds all traces recursively within the given path and
1219 prints all their events, merging them in chronological order.
1221 You can pipe the output of cmd:babeltrace into a tool like man:grep(1) for
1226 babeltrace ~/lttng-traces/my-kernel-session* | grep sys_
1229 You can pipe the output of cmd:babeltrace into a tool like man:wc(1) to
1230 count the recorded events:
1234 babeltrace ~/lttng-traces/my-kernel-session* | grep sys_read | wc --lines
1238 [[viewing-and-analyzing-your-traces-bt-python]]
1239 ==== Use the Babeltrace Python bindings
1241 The <<viewing-and-analyzing-your-traces-bt,text output of cmd:babeltrace>>
1242 is useful to isolate events by simple matching using man:grep(1) and
1243 similar utilities. However, more elaborate filters, such as keeping only
1244 event records with a field value falling within a specific range, are
1245 not trivial to write using a shell. Moreover, reductions and even the
1246 most basic computations involving multiple event records are virtually
1247 impossible to implement.
1249 Fortunately, Babeltrace ships with Python 3 bindings which makes it easy
1250 to read the event records of an LTTng trace sequentially and compute the
1251 desired information.
1253 The following script accepts an LTTng Linux kernel trace path as its
1254 first argument and prints the short names of the top 5 running processes
1255 on CPU 0 during the whole trace:
1260 from collections import Counter
1266 if len(sys.argv) != 2:
1267 msg = 'Usage: python3 {} TRACEPATH'.format(sys.argv[0])
1268 print(msg, file=sys.stderr)
1271 # A trace collection contains one or more traces
1272 col = babeltrace.TraceCollection()
1274 # Add the trace provided by the user (LTTng traces always have
1276 if col.add_trace(sys.argv[1], 'ctf') is None:
1277 raise RuntimeError('Cannot add trace')
1279 # This counter dict contains execution times:
1281 # task command name -> total execution time (ns)
1282 exec_times = Counter()
1284 # This contains the last `sched_switch` timestamp
1288 for event in col.events:
1289 # Keep only `sched_switch` events
1290 if event.name != 'sched_switch':
1293 # Keep only events which happened on CPU 0
1294 if event['cpu_id'] != 0:
1298 cur_ts = event.timestamp
1304 # Previous task command (short) name
1305 prev_comm = event['prev_comm']
1307 # Initialize entry in our dict if not yet done
1308 if prev_comm not in exec_times:
1309 exec_times[prev_comm] = 0
1311 # Compute previous command execution time
1312 diff = cur_ts - last_ts
1314 # Update execution time of this command
1315 exec_times[prev_comm] += diff
1317 # Update last timestamp
1321 for name, ns in exec_times.most_common(5):
1323 print('{:20}{} s'.format(name, s))
1328 if __name__ == '__main__':
1329 sys.exit(0 if top5proc() else 1)
1336 python3 top5proc.py ~/lttng-traces/my-kernel-session*/kernel
1342 swapper/0 48.607245889 s
1343 chromium 7.192738188 s
1344 pavucontrol 0.709894415 s
1345 Compositor 0.660867933 s
1346 Xorg.bin 0.616753786 s
1349 Note that `swapper/0` is the "idle" process of CPU 0 on Linux; since we
1350 weren't using the CPU that much when tracing, its first position in the
1355 == [[understanding-lttng]]Core concepts
1357 From a user's perspective, the LTTng system is built on a few concepts,
1358 or objects, on which the <<lttng-cli,cmd:lttng command-line tool>>
1359 operates by sending commands to the <<lttng-sessiond,session daemon>>.
1360 Understanding how those objects relate to eachother is key in mastering
1363 The core concepts are:
1365 * <<tracing-session,Tracing session>>
1366 * <<domain,Tracing domain>>
1367 * <<channel,Channel and ring buffer>>
1368 * <<"event","Instrumentation point, event rule, event, and event record">>
1374 A _tracing session_ is a stateful dialogue between you and
1375 a <<lttng-sessiond,session daemon>>. You can
1376 <<creating-destroying-tracing-sessions,create a new tracing
1377 session>> with the `lttng create` command.
1379 Anything that you do when you control LTTng tracers happens within a
1380 tracing session. In particular, a tracing session:
1383 * Has its own set of trace files.
1384 * Has its own state of activity (started or stopped).
1385 * Has its own <<tracing-session-mode,mode>> (local, network streaming,
1387 * Has its own <<channel,channels>> which have their own
1388 <<event,event rules>>.
1391 .A _tracing session_ contains <<channel,channels>> that are members of <<domain,tracing domains>> and contain <<event,event rules>>.
1392 image::concepts.png[]
1394 Those attributes and objects are completely isolated between different
1397 A tracing session is analogous to a cash machine session:
1398 the operations you do on the banking system through the cash machine do
1399 not alter the data of other users of the same system. In the case of
1400 the cash machine, a session lasts as long as your bank card is inside.
1401 In the case of LTTng, a tracing session lasts from the `lttng create`
1402 command to the `lttng destroy` command.
1405 .Each Unix user has its own set of tracing sessions.
1406 image::many-sessions.png[]
1409 [[tracing-session-mode]]
1410 ==== Tracing session mode
1412 LTTng can send the generated trace data to different locations. The
1413 _tracing session mode_ dictates where to send it. The following modes
1414 are available in LTTng{nbsp}{revision}:
1417 LTTng writes the traces to the file system of the machine being traced
1420 Network streaming mode::
1421 LTTng sends the traces over the network to a
1422 <<lttng-relayd,relay daemon>> running on a remote system.
1425 LTTng does not write the traces by default. Instead, you can request
1426 LTTng to <<taking-a-snapshot,take a snapshot>>, that is, a copy of the
1427 current tracing buffers, and to write it to the target's file system
1428 or to send it over the network to a <<lttng-relayd,relay daemon>>
1429 running on a remote system.
1432 This mode is similar to the network streaming mode, but a live
1433 trace viewer can connect to the distant relay daemon to
1434 <<lttng-live,view event records as LTTng generates them>> by
1441 A _tracing domain_ is a namespace for event sources. A tracing domain
1442 has its own properties and features.
1444 There are currently five available tracing domains:
1448 * `java.util.logging` (JUL)
1452 You must specify a tracing domain when using some commands to avoid
1453 ambiguity. For example, since all the domains support named tracepoints
1454 as event sources (instrumentation points that you manually insert in the
1455 source code), you need to specify a tracing domain when
1456 <<enabling-disabling-events,creating an event rule>> because all the
1457 tracing domains could have tracepoints with the same names.
1459 Some features are reserved to specific tracing domains. Dynamic function
1460 entry and return instrumentation points, for example, are currently only
1461 supported in the Linux kernel tracing domain, but support for other
1462 tracing domains could be added in the future.
1464 You can create <<channel,channels>> in the Linux kernel and user space
1465 tracing domains. The other tracing domains have a single default
1470 === Channel and ring buffer
1472 A _channel_ is an object which is responsible for a set of ring buffers.
1473 Each ring buffer is divided into multiple sub-buffers. When an LTTng
1474 tracer emits an event, it can record it to one or more
1475 sub-buffers. The attributes of a channel determine what to do when
1476 there's no space left for a new event record because all sub-buffers
1477 are full, where to send a full sub-buffer, and other behaviours.
1479 A channel is always associated to a <<domain,tracing domain>>. The
1480 `java.util.logging` (JUL), log4j, and Python tracing domains each have
1481 a default channel which you cannot configure.
1483 A channel also owns <<event,event rules>>. When an LTTng tracer emits
1484 an event, it records it to the sub-buffers of all
1485 the enabled channels with a satisfied event rule, as long as those
1486 channels are part of active <<tracing-session,tracing sessions>>.
1489 [[channel-buffering-schemes]]
1490 ==== Per-user vs. per-process buffering schemes
1492 A channel has at least one ring buffer _per CPU_. LTTng always
1493 records an event to the ring buffer associated to the CPU on which it
1496 Two _buffering schemes_ are available when you
1497 <<enabling-disabling-channels,create a channel>> in the
1498 user space <<domain,tracing domain>>:
1500 Per-user buffering::
1501 Allocate one set of ring buffers--one per CPU--shared by all the
1502 instrumented processes of each Unix user.
1506 .Per-user buffering scheme.
1507 image::per-user-buffering.png[]
1510 Per-process buffering::
1511 Allocate one set of ring buffers--one per CPU--for each
1512 instrumented process.
1516 .Per-process buffering scheme.
1517 image::per-process-buffering.png[]
1520 The per-process buffering scheme tends to consume more memory than the
1521 per-user option because systems generally have more instrumented
1522 processes than Unix users running instrumented processes. However, the
1523 per-process buffering scheme ensures that one process having a high
1524 event throughput won't fill all the shared sub-buffers of the same
1527 The Linux kernel tracing domain has only one available buffering scheme
1528 which is to allocate a single set of ring buffers for the whole system.
1529 This scheme is similar to the per-user option, but with a single, global
1530 user "running" the kernel.
1533 [[channel-overwrite-mode-vs-discard-mode]]
1534 ==== Overwrite vs. discard event loss modes
1536 When an event occurs, LTTng records it to a specific sub-buffer (yellow
1537 arc in the following animation) of a specific channel's ring buffer.
1538 When there's no space left in a sub-buffer, the tracer marks it as
1539 consumable (red) and another, empty sub-buffer starts receiving the
1540 following event records. A <<lttng-consumerd,consumer daemon>>
1541 eventually consumes the marked sub-buffer (returns to white).
1544 [role="docsvg-channel-subbuf-anim"]
1549 In an ideal world, sub-buffers are consumed faster than they are filled,
1550 as is the case in the previous animation. In the real world,
1551 however, all sub-buffers can be full at some point, leaving no space to
1552 record the following events.
1554 By design, LTTng is a _non-blocking_ tracer: when no empty sub-buffer is
1555 available, it is acceptable to lose event records when the alternative
1556 would be to cause substantial delays in the instrumented application's
1557 execution. LTTng privileges performance over integrity; it aims at
1558 perturbing the traced system as little as possible in order to make
1559 tracing of subtle race conditions and rare interrupt cascades possible.
1561 When it comes to losing event records because no empty sub-buffer is
1562 available, the channel's _event loss mode_ determines what to do. The
1563 available event loss modes are:
1566 Drop the newest event records until a the tracer
1567 releases a sub-buffer.
1570 Clear the sub-buffer containing the oldest event records and start
1571 writing the newest event records there.
1573 This mode is sometimes called _flight recorder mode_ because it's
1575 https://en.wikipedia.org/wiki/Flight_recorder[flight recorder]:
1576 always keep a fixed amount of the latest data.
1578 Which mechanism you should choose depends on your context: prioritize
1579 the newest or the oldest event records in the ring buffer?
1581 Beware that, in overwrite mode, the tracer abandons a whole sub-buffer
1582 as soon as a there's no space left for a new event record, whereas in
1583 discard mode, the tracer only discards the event record that doesn't
1586 In discard mode, LTTng increments a count of lost event records when
1587 an event record is lost and saves this count to the trace. In
1588 overwrite mode, LTTng keeps no information when it overwrites a
1589 sub-buffer before consuming it.
1591 There are a few ways to decrease your probability of losing event
1593 <<channel-subbuf-size-vs-subbuf-count,Sub-buffer count and size>> shows
1594 how you can fine-une the sub-buffer count and size of a channel to
1595 virtually stop losing event records, though at the cost of greater
1599 [[channel-subbuf-size-vs-subbuf-count]]
1600 ==== Sub-buffer count and size
1602 When you <<enabling-disabling-channels,create a channel>>, you can
1603 set its number of sub-buffers and their size.
1605 Note that there is noticeable CPU overhead introduced when
1606 switching sub-buffers (marking a full one as consumable and switching
1607 to an empty one for the following events to be recorded). Knowing this,
1608 the following list presents a few practical situations along with how
1609 to configure the sub-buffer count and size for them:
1611 * **High event throughput**: In general, prefer bigger sub-buffers to
1612 lower the risk of losing event records.
1614 Having bigger sub-buffers also ensures a lower sub-buffer switching
1617 The number of sub-buffers is only meaningful if you create the channel
1618 in overwrite mode: in this case, if a sub-buffer overwrite happens, the
1619 other sub-buffers are left unaltered.
1621 * **Low event throughput**: In general, prefer smaller sub-buffers
1622 since the risk of losing event records is low.
1624 Because events occur less frequently, the sub-buffer switching frequency
1625 should remain low and thus the tracer's overhead should not be a
1628 * **Low memory system**: If your target system has a low memory
1629 limit, prefer fewer first, then smaller sub-buffers.
1631 Even if the system is limited in memory, you want to keep the
1632 sub-buffers as big as possible to avoid a high sub-buffer switching
1635 Note that LTTng uses http://diamon.org/ctf/[CTF] as its trace format,
1636 which means event data is very compact. For example, the average
1637 LTTng kernel event record weights about 32{nbsp}bytes. Thus, a
1638 sub-buffer size of 1{nbsp}MiB is considered big.
1640 The previous situations highlight the major trade-off between a few big
1641 sub-buffers and more, smaller sub-buffers: sub-buffer switching
1642 frequency vs. how much data is lost in overwrite mode. Assuming a
1643 constant event throughput and using the overwrite mode, the two
1644 following configurations have the same ring buffer total size:
1647 [role="docsvg-channel-subbuf-size-vs-count-anim"]
1652 * **2 sub-buffers of 4{nbsp}MiB each**: Expect a very low sub-buffer
1653 switching frequency, but if a sub-buffer overwrite happens, half of
1654 the event records so far (4{nbsp}MiB) are definitely lost.
1655 * **8 sub-buffers of 1{nbsp}MiB each**: Expect 4{nbsp}times the tracer's
1656 overhead as the previous configuration, but if a sub-buffer
1657 overwrite happens, only the eighth of event records so far are
1660 In discard mode, the sub-buffers count parameter is pointless: use two
1661 sub-buffers and set their size according to the requirements of your
1665 [[channel-switch-timer]]
1666 ==== Switch timer period
1668 The _switch timer period_ is an important configurable attribute of
1669 a channel to ensure periodic sub-buffer flushing.
1671 When the _switch timer_ expires, a sub-buffer switch happens. You can
1672 set the switch timer period attribute when you
1673 <<enabling-disabling-channels,create a channel>> to ensure that event
1674 data is consumed and committed to trace files or to a distant relay
1675 daemon periodically in case of a low event throughput.
1678 [role="docsvg-channel-switch-timer"]
1683 This attribute is also convenient when you use big sub-buffers to cope
1684 with a sporadic high event throughput, even if the throughput is
1688 [[channel-read-timer]]
1689 ==== Read timer period
1691 By default, the LTTng tracers use a notification mechanism to signal a
1692 full sub-buffer so that a consumer daemon can consume it. When such
1693 notifications must be avoided, for example in real-time applications,
1694 you can use the channel's _read timer_ instead. When the read timer
1695 fires, the <<lttng-consumerd,consumer daemon>> checks for full,
1696 consumable sub-buffers.
1699 [[tracefile-rotation]]
1700 ==== Trace file count and size
1702 By default, trace files can grow as large as needed. You can set the
1703 maximum size of each trace file that a channel writes when you
1704 <<enabling-disabling-channels,create a channel>>. When the size of
1705 a trace file reaches the channel's fixed maximum size, LTTng creates
1706 another file to contain the next event records. LTTng appends a file
1707 count to each trace file name in this case.
1709 If you set the trace file size attribute when you create a channel, the
1710 maximum number of trace files that LTTng creates is _unlimited_ by
1711 default. To limit them, you can also set a maximum number of trace
1712 files. When the number of trace files reaches the channel's fixed
1713 maximum count, the oldest trace file is overwritten. This mechanism is
1714 called _trace file rotation_.
1718 === Instrumentation point, event rule, event, and event record
1720 An _event rule_ is a set of conditions which must be **all** satisfied
1721 for LTTng to record an occuring event.
1723 You set the conditions when you <<enabling-disabling-events,create
1726 You always attach an event rule to <<channel,channel>> when you create
1729 When an event passes the conditions of an event rule, LTTng records it
1730 in one of the attached channel's sub-buffers.
1732 The available conditions, as of LTTng{nbsp}{revision}, are:
1734 * The event rule _is enabled_.
1735 * The instrumentation point's type _is{nbsp}T_.
1736 * The instrumentation point's name (sometimes called _event name_)
1737 _matches{nbsp}N_, but _is not{nbsp}E_.
1738 * The instrumentation point's log level _is as severe as{nbsp}L_, or
1739 _is exactly{nbsp}L_.
1740 * The fields of the event's payload _satisfy_ a filter
1741 expression{nbsp}__F__.
1743 As you can see, all the conditions but the dynamic filter are related to
1744 the event rule's status or to the instrumentation point, not to the
1745 occurring events. This is why, without a filter, checking if an event
1746 passes an event rule is not a dynamic task: when you create or modify an
1747 event rule, all the tracers of its tracing domain enable or disable the
1748 instrumentation points themselves once. This is possible because the
1749 attributes of an instrumentation point (type, name, and log level) are
1750 defined statically. In other words, without a dynamic filter, the tracer
1751 _does not evaluate_ the arguments of an instrumentation point unless it
1752 matches an enabled event rule.
1754 Note that, for LTTng to record an event, the <<channel,channel>> to
1755 which a matching event rule is attached must also be enabled, and the
1756 tracing session owning this channel must be active.
1759 .Logical path from an instrumentation point to an event record.
1760 image::event-rule.png[]
1762 .Event, event record, or event rule?
1764 With so many similar terms, it's easy to get confused.
1766 An **event** is the consequence of the execution of an _instrumentation
1767 point_, like a tracepoint that you manually place in some source code,
1768 or a Linux kernel KProbe. An event is said to _occur_ at a specific
1769 time. Different actions can be taken upon the occurance of an event,
1770 like record the event's payload to a buffer.
1772 An **event record** is the representation of an event in a sub-buffer. A
1773 tracer is responsible for capturing the payload of an event, current
1774 context variables, the event's ID, and the event's timestamp. LTTng
1775 can append this sub-buffer to a trace file.
1777 An **event rule** is a set of conditions which must all be satisfied for
1778 LTTng to record an occuring event. Events still occur without
1779 satisfying event rules, but LTTng does not record them.
1784 == Components of noch:{LTTng}
1786 The second _T_ in _LTTng_ stands for _toolkit_: it would be wrong
1787 to call LTTng a simple _tool_ since it is composed of multiple
1788 interacting components. This section describes those components,
1789 explains their respective roles, and shows how they connect together to
1790 form the LTTng ecosystem.
1792 The following diagram shows how the most important components of LTTng
1793 interact with user applications, the Linux kernel, and you:
1796 .Control and trace data paths between LTTng components.
1797 image::plumbing.png[]
1799 The LTTng project incorporates:
1801 * **LTTng-tools**: Libraries and command-line interface to
1802 control tracing sessions.
1803 ** <<lttng-sessiond,Session daemon>> (man:lttng-sessiond(8)).
1804 ** <<lttng-consumerd,Consumer daemon>> (man:lttng-consumerd(8)).
1805 ** <<lttng-relayd,Relay daemon>> (man:lttng-relayd(8)).
1806 ** <<liblttng-ctl-lttng,Tracing control library>> (`liblttng-ctl`).
1807 ** <<lttng-cli,Tracing control command-line tool>> (man:lttng(1)).
1808 * **LTTng-UST**: Libraries and Java/Python packages to trace user
1810 ** <<lttng-ust,User space tracing library>> (`liblttng-ust`) and its
1811 headers to instrument and trace any native user application.
1812 ** <<prebuilt-ust-helpers,Preloadable user space tracing helpers>>:
1813 *** `liblttng-ust-libc-wrapper`
1814 *** `liblttng-ust-pthread-wrapper`
1815 *** `liblttng-ust-cyg-profile`
1816 *** `liblttng-ust-cyg-profile-fast`
1817 *** `liblttng-ust-dl`
1818 ** User space tracepoint provider source files generator command-line
1819 tool (man:lttng-gen-tp(1)).
1820 ** <<lttng-ust-agents,LTTng-UST Java agent>> to instrument and trace
1821 Java applications using `java.util.logging` or
1822 Apache log4j 1.2 logging.
1823 ** <<lttng-ust-agents,LTTng-UST Python agent>> to instrument
1824 Python applications using the standard `logging` package.
1825 * **LTTng-modules**: <<lttng-modules,Linux kernel modules>> to trace
1827 ** LTTng kernel tracer module.
1828 ** Tracing ring buffer kernel modules.
1829 ** Probe kernel modules.
1830 ** LTTng logger kernel module.
1834 === Tracing control command-line interface
1837 .The tracing control command-line interface.
1838 image::plumbing-lttng-cli.png[]
1840 The _man:lttng(1) command-line tool_ is the standard user interface to
1841 control LTTng <<tracing-session,tracing sessions>>. The cmd:lttng tool
1842 is part of LTTng-tools.
1844 The cmd:lttng tool is linked with
1845 <<liblttng-ctl-lttng,`liblttng-ctl`>> to communicate with
1846 one or more <<lttng-sessiond,session daemons>> behind the scenes.
1848 The cmd:lttng tool has a Git-like interface:
1852 lttng <general options> <command> <command options>
1855 The <<controlling-tracing,Tracing control>> section explores the
1856 available features of LTTng using the cmd:lttng tool.
1859 [[liblttng-ctl-lttng]]
1860 === Tracing control library
1863 .The tracing control library.
1864 image::plumbing-liblttng-ctl.png[]
1866 The _LTTng control library_, `liblttng-ctl`, is used to communicate
1867 with a <<lttng-sessiond,session daemon>> using a C API that hides the
1868 underlying protocol's details. `liblttng-ctl` is part of LTTng-tools.
1870 The <<lttng-cli,cmd:lttng command-line tool>>
1871 is linked with `liblttng-ctl`.
1873 You can use `liblttng-ctl` in C or $$C++$$ source code by including its
1878 #include <lttng/lttng.h>
1881 Some objects are referenced by name (C string), such as tracing
1882 sessions, but most of them require to create a handle first using
1883 `lttng_create_handle()`.
1885 The best available developer documentation for `liblttng-ctl` is, as of
1886 LTTng{nbsp}{revision}, its installed header files. Every function and
1887 structure is thoroughly documented.
1891 === User space tracing library
1894 .The user space tracing library.
1895 image::plumbing-liblttng-ust.png[]
1897 The _user space tracing library_, `liblttng-ust` (see man:lttng-ust(3)),
1898 is the LTTng user space tracer. It receives commands from a
1899 <<lttng-sessiond,session daemon>>, for example to
1900 enable and disable specific instrumentation points, and writes event
1901 records to ring buffers shared with a
1902 <<lttng-consumerd,consumer daemon>>.
1903 `liblttng-ust` is part of LTTng-UST.
1905 Public C header files are installed beside `liblttng-ust` to
1906 instrument any <<c-application,C or $$C++$$ application>>.
1908 <<lttng-ust-agents,LTTng-UST agents>>, which are regular Java and Python
1909 packages, use their own library providing tracepoints which is
1910 linked with `liblttng-ust`.
1912 An application or library does not have to initialize `liblttng-ust`
1913 manually: its constructor does the necessary tasks to properly register
1914 to a session daemon. The initialization phase also enables the
1915 instrumentation points matching the <<event,event rules>> that you
1919 [[lttng-ust-agents]]
1920 === User space tracing agents
1923 .The user space tracing agents.
1924 image::plumbing-lttng-ust-agents.png[]
1926 The _LTTng-UST Java and Python agents_ are regular Java and Python
1927 packages which add LTTng tracing capabilities to the
1928 native logging frameworks. The LTTng-UST agents are part of LTTng-UST.
1930 In the case of Java, the
1931 https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[`java.util.logging`
1932 core logging facilities] and
1933 https://logging.apache.org/log4j/1.2/[Apache log4j 1.2] are supported.
1934 Note that Apache Log4{nbsp}2 is not supported.
1936 In the case of Python, the standard
1937 https://docs.python.org/3/library/logging.html[`logging`] package
1938 is supported. Both Python 2 and Python 3 modules can import the
1939 LTTng-UST Python agent package.
1941 The applications using the LTTng-UST agents are in the
1942 `java.util.logging` (JUL),
1943 log4j, and Python <<domain,tracing domains>>.
1945 Both agents use the same mechanism to trace the log statements. When an
1946 agent is initialized, it creates a log handler that attaches to the root
1947 logger. The agent also registers to a <<lttng-sessiond,session daemon>>.
1948 When the application executes a log statement, it is passed to the
1949 agent's log handler by the root logger. The agent's log handler calls a
1950 native function in a tracepoint provider package shared library linked
1951 with <<lttng-ust,`liblttng-ust`>>, passing the formatted log message and
1952 other fields, like its logger name and its log level. This native
1953 function contains a user space instrumentation point, hence tracing the
1956 The log level condition of an
1957 <<event,event rule>> is considered when tracing
1958 a Java or a Python application, and it's compatible with the standard
1959 JUL, log4j, and Python log levels.
1963 === LTTng kernel modules
1966 .The LTTng kernel modules.
1967 image::plumbing-lttng-modules.png[]
1969 The _LTTng kernel modules_ are a set of Linux kernel modules
1970 which implement the kernel tracer of the LTTng project. The LTTng
1971 kernel modules are part of LTTng-modules.
1973 The LTTng kernel modules include:
1975 * A set of _probe_ modules.
1977 Each module attaches to a specific subsystem
1978 of the Linux kernel using its tracepoint instrument points. There are
1979 also modules to attach to the entry and return points of the Linux
1980 system call functions.
1982 * _Ring buffer_ modules.
1984 A ring buffer implementation is provided as kernel modules. The LTTng
1985 kernel tracer writes to the ring buffer; a
1986 <<lttng-consumerd,consumer daemon>> reads from the ring buffer.
1988 * The _LTTng kernel tracer_ module.
1989 * The _LTTng logger_ module.
1991 The LTTng logger module implements the special path:{/proc/lttng-logger}
1992 file so that any executable can generate LTTng events by opening and
1993 writing to this file.
1995 See <<proc-lttng-logger-abi,LTTng logger>>.
1997 Generally, you do not have to load the LTTng kernel modules manually
1998 (using man:modprobe(8), for example): a root <<lttng-sessiond,session
1999 daemon>> loads the necessary modules when starting. If you have extra
2000 probe modules, you can specify to load them to the session daemon on
2003 The LTTng kernel modules are installed in
2004 +/usr/lib/modules/__release__/extra+ by default, where +__release__+ is
2005 the kernel release (see `uname --kernel-release`).
2012 .The session daemon.
2013 image::plumbing-sessiond.png[]
2015 The _session daemon_, man:lttng-sessiond(8), is a daemon responsible for
2016 managing tracing sessions and for controlling the various components of
2017 LTTng. The session daemon is part of LTTng-tools.
2019 The session daemon sends control requests to and receives control
2022 * The <<lttng-ust,user space tracing library>>.
2024 Any instance of the user space tracing library first registers to
2025 a session daemon. Then, the session daemon can send requests to
2026 this instance, such as:
2029 ** Get the list of tracepoints.
2030 ** Share an <<event,event rule>> so that the user space tracing library
2031 can enable or disable tracepoints. Amongst the possible conditions
2032 of an event rule is a filter expression which `liblttng-ust` evalutes
2033 when an event occurs.
2034 ** Share <<channel,channel>> attributes and ring buffer locations.
2037 The session daemon and the user space tracing library use a Unix
2038 domain socket for their communication.
2040 * The <<lttng-ust-agents,user space tracing agents>>.
2042 Any instance of a user space tracing agent first registers to
2043 a session daemon. Then, the session daemon can send requests to
2044 this instance, such as:
2047 ** Get the list of loggers.
2048 ** Enable or disable a specific logger.
2051 The session daemon and the user space tracing agent use a TCP connection
2052 for their communication.
2054 * The <<lttng-modules,LTTng kernel tracer>>.
2055 * The <<lttng-consumerd,consumer daemon>>.
2057 The session daemon sends requests to the consumer daemon to instruct
2058 it where to send the trace data streams, amongst other information.
2060 * The <<lttng-relayd,relay daemon>>.
2062 The session daemon receives commands from the
2063 <<liblttng-ctl-lttng,tracing control library>>.
2065 The root session daemon loads the appropriate
2066 <<lttng-modules,LTTng kernel modules>> on startup. It also spawns
2067 a <<lttng-consumerd,consumer daemon>> as soon as you create
2068 an <<event,event rule>>.
2070 The session daemon does not send and receive trace data: this is the
2071 role of the <<lttng-consumerd,consumer daemon>> and
2072 <<lttng-relayd,relay daemon>>. It does, however, generate the
2073 http://diamon.org/ctf/[CTF] metadata stream.
2075 Each Unix user can have its own session daemon instance. The
2076 tracing sessions managed by different session daemons are completely
2079 The root user's session daemon is the only one which is
2080 allowed to control the LTTng kernel tracer, and its spawned consumer
2081 daemon is the only one which is allowed to consume trace data from the
2082 LTTng kernel tracer. Note, however, that any Unix user which is a member
2083 of the <<tracing-group,tracing group>> is allowed
2084 to create <<channel,channels>> in the
2085 Linux kernel <<domain,tracing domain>>, and thus to trace the Linux
2088 The <<lttng-cli,cmd:lttng command-line tool>> automatically starts a
2089 session daemon when using its `create` command if none is currently
2090 running. You can also start the session daemon manually.
2097 .The consumer daemon.
2098 image::plumbing-consumerd.png[]
2100 The _consumer daemon_, man:lttng-consumerd(8), is a daemon which shares
2101 ring buffers with user applications or with the LTTng kernel modules to
2102 collect trace data and send it to some location (on disk or to a
2103 <<lttng-relayd,relay daemon>> over the network). The consumer daemon
2104 is part of LTTng-tools.
2106 You do not start a consumer daemon manually: a consumer daemon is always
2107 spawned by a <<lttng-sessiond,session daemon>> as soon as you create an
2108 <<event,event rule>>, that is, before you start tracing. When you kill
2109 its owner session daemon, the consumer daemon also exits because it is
2110 the session daemon's child process. Command-line options of
2111 man:lttng-sessiond(8) target the consumer daemon process.
2113 There are up to two running consumer daemons per Unix user, whereas only
2114 one session daemon can run per user. This is because each process can be
2115 either 32-bit or 64-bit: if the target system runs a mixture of 32-bit
2116 and 64-bit processes, it is more efficient to have separate
2117 corresponding 32-bit and 64-bit consumer daemons. The root user is an
2118 exception: it can have up to _three_ running consumer daemons: 32-bit
2119 and 64-bit instances for its user applications, and one more
2120 reserved for collecting kernel trace data.
2128 image::plumbing-relayd.png[]
2130 The _relay daemon_, man:lttng-relayd(8), is a daemon acting as a bridge
2131 between remote session and consumer daemons, local trace files, and a
2132 remote live trace viewer. The relay daemon is part of LTTng-tools.
2134 The main purpose of the relay daemon is to implement a receiver of
2135 <<sending-trace-data-over-the-network,trace data over the network>>.
2136 This is useful when the target system does not have much file system
2137 space to record trace files locally.
2139 The relay daemon is also a server to which a
2140 <<lttng-live,live trace viewer>> can
2141 connect. The live trace viewer sends requests to the relay daemon to
2142 receive trace data as the target system emits events. The
2143 communication protocol is named _LTTng live_; it is used over TCP
2146 Note that you can start the relay daemon on the target system directly.
2147 This is the setup of choice when the use case is to view events as
2148 the target system emits them without the need of a remote system.
2152 == [[using-lttng]]Instrumentation
2154 There are many examples of tracing and monitoring in our everyday life:
2156 * You have access to real-time and historical weather reports and
2157 forecasts thanks to weather stations installed around the country.
2158 * You know your heart is safe thanks to an electrocardiogram.
2159 * You make sure not to drive your car too fast and to have enough fuel
2160 to reach your destination thanks to gauges visible on your dashboard.
2162 All the previous examples have something in common: they rely on
2163 **instruments**. Without the electrodes attached to the surface of your
2164 body's skin, cardiac monitoring is futile.
2166 LTTng, as a tracer, is no different from those real life examples. If
2167 you're about to trace a software system or, in other words, record its
2168 history of execution, you better have **instrumentation points** in the
2169 subject you're tracing, that is, the actual software.
2171 Various ways were developed to instrument a piece of software for LTTng
2172 tracing. The most straightforward one is to manually place
2173 instrumentation points, called _tracepoints_, in the software's source
2174 code. It is also possible to add instrumentation points dynamically in
2175 the Linux kernel <<domain,tracing domain>>.
2177 If you're only interested in tracing the Linux kernel, your
2178 instrumentation needs are probably already covered by LTTng's built-in
2179 <<lttng-modules,Linux kernel tracepoints>>. You may also wish to trace a
2180 user application which is already instrumented for LTTng tracing.
2181 In such cases, you can skip this whole section and read the topics of
2182 the <<controlling-tracing,Tracing control>> section.
2184 Many methods are available to instrument a piece of software for LTTng
2187 * <<c-application,User space instrumentation for C and $$C++$$
2189 * <<prebuilt-ust-helpers,Prebuilt user space tracing helpers>>.
2190 * <<java-application,User space Java agent>>.
2191 * <<python-application,User space Python agent>>.
2192 * <<proc-lttng-logger-abi,LTTng logger>>.
2193 * <<instrumenting-linux-kernel,LTTng kernel tracepoints>>.
2197 === [[cxx-application]]User space instrumentation for C and $$C++$$ applications
2199 The procedure to instrument a C or $$C++$$ user application with
2200 the <<lttng-ust,LTTng user space tracing library>>, `liblttng-ust`, is:
2202 . <<tracepoint-provider,Create the source files of a tracepoint provider
2204 . <<probing-the-application-source-code,Add tracepoints to
2205 the application's source code>>.
2206 . <<building-tracepoint-providers-and-user-application,Build and link
2207 a tracepoint provider package and the user application>>.
2209 If you need quick, man:printf(3)-like instrumentation, you can skip
2210 those steps and use <<tracef,`tracef()`>> or <<tracelog,`tracelog()`>>
2213 IMPORTANT: You need to <<installing-lttng,install>> LTTng-UST to
2214 instrument a user application with `liblttng-ust`.
2217 [[tracepoint-provider]]
2218 ==== Create the source files of a tracepoint provider package
2220 A _tracepoint provider_ is a set of compiled functions which provide
2221 **tracepoints** to an application, the type of instrumentation point
2222 supported by LTTng-UST. Those functions can emit events with
2223 user-defined fields and serialize those events as event records to one
2224 or more LTTng-UST <<channel,channel>> sub-buffers. The `tracepoint()`
2225 macro, which you <<probing-the-application-source-code,insert in a user
2226 application's source code>>, calls those functions.
2228 A _tracepoint provider package_ is an object file (`.o`) or a shared
2229 library (`.so`) which contains one or more tracepoint providers.
2230 Its source files are:
2232 * One or more <<tpp-header,tracepoint provider header>> (`.h`).
2233 * A <<tpp-source,tracepoint provider package source>> (`.c`).
2235 A tracepoint provider package is dynamically linked with `liblttng-ust`,
2236 the LTTng user space tracer, at run time.
2239 .User application linked with `liblttng-ust` and containing a tracepoint provider.
2240 image::ust-app.png[]
2242 NOTE: If you need quick, man:printf(3)-like instrumentation, you can
2243 skip creating and using a tracepoint provider and use
2244 <<tracef,`tracef()`>> or <<tracelog,`tracelog()`>> instead.
2248 ===== Create a tracepoint provider header file template
2250 A _tracepoint provider header file_ contains the tracepoint
2251 definitions of a tracepoint provider.
2253 To create a tracepoint provider header file:
2255 . Start from this template:
2259 .Tracepoint provider header file template (`.h` file extension).
2261 #undef TRACEPOINT_PROVIDER
2262 #define TRACEPOINT_PROVIDER provider_name
2264 #undef TRACEPOINT_INCLUDE
2265 #define TRACEPOINT_INCLUDE "./tp.h"
2267 #if !defined(_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
2270 #include <lttng/tracepoint.h>
2273 * Use TRACEPOINT_EVENT(), TRACEPOINT_EVENT_CLASS(),
2274 * TRACEPOINT_EVENT_INSTANCE(), and TRACEPOINT_LOGLEVEL() here.
2279 #include <lttng/tracepoint-event.h>
2285 * `provider_name` with the name of your tracepoint provider.
2286 * `"tp.h"` with the name of your tracepoint provider header file.
2288 . Below the `#include <lttng/tracepoint.h>` line, put your
2289 <<defining-tracepoints,tracepoint definitions>>.
2291 Your tracepoint provider name must be unique amongst all the possible
2292 tracepoint provider names used on the same target system. We
2293 suggest to include the name of your project or company in the name,
2294 for example, `org_lttng_my_project_tpp`.
2296 TIP: [[lttng-gen-tp]]You can use the man:lttng-gen-tp(1) tool to create
2297 this boilerplate for you. When using cmd:lttng-gen-tp, all you need to
2298 write are the <<defining-tracepoints,tracepoint definitions>>.
2301 [[defining-tracepoints]]
2302 ===== Create a tracepoint definition
2304 A _tracepoint definition_ defines, for a given tracepoint:
2306 * Its **input arguments**. They are the macro parameters that the
2307 `tracepoint()` macro accepts for this particular tracepoint
2308 in the user application's source code.
2309 * Its **output event fields**. They are the sources of event fields
2310 that form the payload of any event that the execution of the
2311 `tracepoint()` macro emits for this particular tracepoint.
2313 You can create a tracepoint definition by using the
2314 `TRACEPOINT_EVENT()` macro below the `#include <lttng/tracepoint.h>`
2316 <<tpp-header,tracepoint provider header file template>>.
2318 The syntax of the `TRACEPOINT_EVENT()` macro is:
2321 .`TRACEPOINT_EVENT()` macro syntax.
2324 /* Tracepoint provider name */
2327 /* Tracepoint name */
2330 /* Input arguments */
2335 /* Output event fields */
2344 * `provider_name` with your tracepoint provider name.
2345 * `tracepoint_name` with your tracepoint name.
2346 * `arguments` with the <<tpp-def-input-args,input arguments>>.
2347 * `fields` with the <<tpp-def-output-fields,output event field>>
2350 This tracepoint emits events named `provider_name:tracepoint_name`.
2353 .Event name's length limitation
2355 The concatenation of the tracepoint provider name and the
2356 tracepoint name must not exceed **254 characters**. If it does, the
2357 instrumented application compiles and runs, but LTTng throws multiple
2358 warnings and you could experience serious issues.
2361 [[tpp-def-input-args]]The syntax of the `TP_ARGS()` macro is:
2364 .`TP_ARGS()` macro syntax.
2373 * `type` with the C type of the argument.
2374 * `arg_name` with the argument name.
2376 You can repeat `type` and `arg_name` up to 10 times to have
2377 more than one argument.
2379 .`TP_ARGS()` usage with three arguments.
2391 The `TP_ARGS()` and `TP_ARGS(void)` forms are valid to create a
2392 tracepoint definition with no input arguments.
2394 [[tpp-def-output-fields]]The `TP_FIELDS()` macro contains a list of
2395 `ctf_*()` macros. Each `ctf_*()` macro defines one event field. See
2396 man:lttng-ust(3) for a complete description of the available `ctf_*()`
2397 macros. A `ctf_*()` macro specifies the type, size, and byte order of
2400 Each `ctf_*()` macro takes an _argument expression_ parameter. This is a
2401 C expression that the tracer evalutes at the `tracepoint()` macro site
2402 in the application's source code. This expression provides a field's
2403 source of data. The argument expression can include input argument names
2404 listed in the `TP_ARGS()` macro.
2406 Each `ctf_*()` macro also takes a _field name_ parameter. Field names
2407 must be unique within a given tracepoint definition.
2409 Here's a complete tracepoint definition example:
2411 .Tracepoint definition.
2413 The following tracepoint definition defines a tracepoint which takes
2414 three input arguments and has four output event fields.
2418 #include "my-custom-structure.h"
2424 const struct my_custom_structure*, my_custom_structure,
2429 ctf_string(query_field, query)
2430 ctf_float(double, ratio_field, ratio)
2431 ctf_integer(int, recv_size, my_custom_structure->recv_size)
2432 ctf_integer(int, send_size, my_custom_structure->send_size)
2437 You can refer to this tracepoint definition with the `tracepoint()`
2438 macro in your application's source code like this:
2442 tracepoint(my_provider, my_tracepoint,
2443 my_structure, some_ratio, the_query);
2447 NOTE: The LTTng tracer only evaluates tracepoint arguments at run time
2448 if they satisfy an enabled <<event,event rule>>.
2451 [[using-tracepoint-classes]]
2452 ===== Use a tracepoint class
2454 A _tracepoint class_ is a class of tracepoints which share the same
2455 output event field definitions. A _tracepoint instance_ is one
2456 instance of such a defined tracepoint class, with its own tracepoint
2459 The <<defining-tracepoints,`TRACEPOINT_EVENT()` macro>> is actually a
2460 shorthand which defines both a tracepoint class and a tracepoint
2461 instance at the same time.
2463 When you build a tracepoint provider package, the C or $$C++$$ compiler
2464 creates one serialization function for each **tracepoint class**. A
2465 serialization function is responsible for serializing the event fields
2466 of a tracepoint to a sub-buffer when tracing.
2468 For various performance reasons, when your situation requires multiple
2469 tracepoint definitions with different names, but with the same event
2470 fields, we recommend that you manually create a tracepoint class
2471 and instantiate as many tracepoint instances as needed. One positive
2472 effect of such a design, amongst other advantages, is that all
2473 tracepoint instances of the same tracepoint class reuse the same
2474 serialization function, thus reducing
2475 https://en.wikipedia.org/wiki/Cache_pollution[cache pollution].
2477 .Use a tracepoint class and tracepoint instances.
2479 Consider the following three tracepoint definitions:
2491 ctf_integer(int, userid, userid)
2492 ctf_integer(size_t, len, len)
2504 ctf_integer(int, userid, userid)
2505 ctf_integer(size_t, len, len)
2517 ctf_integer(int, userid, userid)
2518 ctf_integer(size_t, len, len)
2523 In this case, we create three tracepoint classes, with one implicit
2524 tracepoint instance for each of them: `get_account`, `get_settings`, and
2525 `get_transaction`. However, they all share the same event field names
2526 and types. Hence three identical, yet independent serialization
2527 functions are created when you build the tracepoint provider package.
2529 A better design choice is to define a single tracepoint class and three
2530 tracepoint instances:
2534 /* The tracepoint class */
2535 TRACEPOINT_EVENT_CLASS(
2536 /* Tracepoint provider name */
2539 /* Tracepoint class name */
2542 /* Input arguments */
2548 /* Output event fields */
2550 ctf_integer(int, userid, userid)
2551 ctf_integer(size_t, len, len)
2555 /* The tracepoint instances */
2556 TRACEPOINT_EVENT_INSTANCE(
2557 /* Tracepoint provider name */
2560 /* Tracepoint class name */
2563 /* Tracepoint name */
2566 /* Input arguments */
2572 TRACEPOINT_EVENT_INSTANCE(
2581 TRACEPOINT_EVENT_INSTANCE(
2594 [[assigning-log-levels]]
2595 ===== Assign a log level to a tracepoint definition
2597 You can assign an optional _log level_ to a
2598 <<defining-tracepoints,tracepoint definition>>.
2600 Assigning different levels of severity to tracepoint definitions can
2601 be useful: when you <<enabling-disabling-events,create an event rule>>,
2602 you can target tracepoints having a log level as severe as a specific
2605 The concept of LTTng-UST log levels is similar to the levels found
2606 in typical logging frameworks:
2608 * In a logging framework, the log level is given by the function
2609 or method name you use at the log statement site: `debug()`,
2610 `info()`, `warn()`, `error()`, and so on.
2611 * In LTTng-UST, you statically assign the log level to a tracepoint
2612 definition; any `tracepoint()` macro invocation which refers to
2613 this definition has this log level.
2615 You can assign a log level to a tracepoint definition with the
2616 `TRACEPOINT_LOGLEVEL()` macro. You must use this macro _after_ the
2617 <<defining-tracepoints,`TRACEPOINT_EVENT()`>> or
2618 <<using-tracepoint-classes,`TRACEPOINT_INSTANCE()`>> macro for a given
2621 The syntax of the `TRACEPOINT_LOGLEVEL()` macro is:
2624 .`TRACEPOINT_LOGLEVEL()` macro syntax.
2626 TRACEPOINT_LOGLEVEL(provider_name, tracepoint_name, log_level)
2631 * `provider_name` with the tracepoint provider name.
2632 * `tracepoint_name` with the tracepoint name.
2633 * `log_level` with the log level to assign to the tracepoint
2634 definition named `tracepoint_name` in the `provider_name`
2635 tracepoint provider.
2637 See man:lttng-ust(3) for a list of available log level names.
2639 .Assign the `TRACE_DEBUG_UNIT` log level to a tracepoint definition.
2643 /* Tracepoint definition */
2652 ctf_integer(int, userid, userid)
2653 ctf_integer(size_t, len, len)
2657 /* Log level assignment */
2658 TRACEPOINT_LOGLEVEL(my_app, get_transaction, TRACE_DEBUG_UNIT)
2664 ===== Create a tracepoint provider package source file
2666 A _tracepoint provider package source file_ is a C source file which
2667 includes a <<tpp-header,tracepoint provider header file>> to expand its
2668 macros into event serialization and other functions.
2670 You can always use the following tracepoint provider package source
2674 .Tracepoint provider package source file template.
2676 #define TRACEPOINT_CREATE_PROBES
2681 Replace `tp.h` with the name of your <<tpp-header,tracepoint provider
2682 header file>> name. You may also include more than one tracepoint
2683 provider header file here to create a tracepoint provider package
2684 holding more than one tracepoint providers.
2687 [[probing-the-application-source-code]]
2688 ==== Add tracepoints to an application's source code
2690 Once you <<tpp-header,create a tracepoint provider header file>>, you
2691 can use the `tracepoint()` macro in your application's
2692 source code to insert the tracepoints that this header
2693 <<defining-tracepoints,defines>>.
2695 The `tracepoint()` macro takes at least two parameters: the tracepoint
2696 provider name and the tracepoint name. The corresponding tracepoint
2697 definition defines the other parameters.
2699 .`tracepoint()` usage.
2701 The following <<defining-tracepoints,tracepoint definition>> defines a
2702 tracepoint which takes two input arguments and has two output event
2706 .Tracepoint provider header file.
2708 #include "my-custom-structure.h"
2715 const char*, cmd_name
2718 ctf_string(cmd_name, cmd_name)
2719 ctf_integer(int, number_of_args, argc)
2724 You can refer to this tracepoint definition with the `tracepoint()`
2725 macro in your application's source code like this:
2728 .Application's source file.
2732 int main(int argc, char* argv[])
2734 tracepoint(my_provider, my_tracepoint, argc, argv[0]);
2740 Note how the application's source code includes
2741 the tracepoint provider header file containing the tracepoint
2742 definitions to use, path:{tp.h}.
2745 .`tracepoint()` usage with a complex tracepoint definition.
2747 Consider this complex tracepoint definition, where multiple event
2748 fields refer to the same input arguments in their argument expression
2752 .Tracepoint provider header file.
2754 /* For `struct stat` */
2755 #include <sys/types.h>
2756 #include <sys/stat.h>
2768 ctf_integer(int, my_constant_field, 23 + 17)
2769 ctf_integer(int, my_int_arg_field, my_int_arg)
2770 ctf_integer(int, my_int_arg_field2, my_int_arg * my_int_arg)
2771 ctf_integer(int, sum4_field, my_str_arg[0] + my_str_arg[1] +
2772 my_str_arg[2] + my_str_arg[3])
2773 ctf_string(my_str_arg_field, my_str_arg)
2774 ctf_integer_hex(off_t, size_field, st->st_size)
2775 ctf_float(double, size_dbl_field, (double) st->st_size)
2776 ctf_sequence_text(char, half_my_str_arg_field, my_str_arg,
2777 size_t, strlen(my_str_arg) / 2)
2782 You can refer to this tracepoint definition with the `tracepoint()`
2783 macro in your application's source code like this:
2786 .Application's source file.
2788 #define TRACEPOINT_DEFINE
2795 stat("/etc/fstab", &s);
2796 tracepoint(my_provider, my_tracepoint, 23, "Hello, World!", &s);
2802 If you look at the event record that LTTng writes when tracing this
2803 program, assuming the file size of path:{/etc/fstab} is 301{nbsp}bytes,
2804 it should look like this:
2806 .Event record fields
2808 |Field's name |Field's value
2809 |`my_constant_field` |40
2810 |`my_int_arg_field` |23
2811 |`my_int_arg_field2` |529
2813 |`my_str_arg_field` |`Hello, World!`
2814 |`size_field` |0x12d
2815 |`size_dbl_field` |301.0
2816 |`half_my_str_arg_field` |`Hello,`
2820 Sometimes, the arguments you pass to `tracepoint()` are expensive to
2821 compute--they use the call stack, for example. To avoid this
2822 computation when the tracepoint is disabled, you can use the
2823 `tracepoint_enabled()` and `do_tracepoint()` macros.
2825 The syntax of the `tracepoint_enabled()` and `do_tracepoint()` macros
2829 .`tracepoint_enabled()` and `do_tracepoint()` macros syntax.
2831 tracepoint_enabled(provider_name, tracepoint_name)
2832 do_tracepoint(provider_name, tracepoint_name, ...)
2837 * `provider_name` with the tracepoint provider name.
2838 * `tracepoint_name` with the tracepoint name.
2840 `tracepoint_enabled()` returns a non-zero value if the tracepoint named
2841 `tracepoint_name` from the provider named `provider_name` is enabled
2844 `do_tracepoint()` is like `tracepoint()`, except that it doesn't check
2845 if the tracepoint is enabled. Using `tracepoint()` with
2846 `tracepoint_enabled()` is dangerous since `tracepoint()` also contains
2847 the `tracepoint_enabled()` check, thus a race condition is
2848 possible in this situation:
2851 .Possible race condition when using `tracepoint_enabled()` with `tracepoint()`.
2853 if (tracepoint_enabled(my_provider, my_tracepoint)) {
2854 stuff = prepare_stuff();
2857 tracepoint(my_provider, my_tracepoint, stuff);
2860 If the tracepoint is enabled after the condition, then `stuff` is not
2861 prepared: the emitted event will either contain wrong data, or the whole
2862 application could crash (segmentation fault, for example).
2864 NOTE: Neither `tracepoint_enabled()` nor `do_tracepoint()` have an
2865 `STAP_PROBEV()` call. If you need it, you must emit
2869 [[building-tracepoint-providers-and-user-application]]
2870 ==== Build and link a tracepoint provider package and an application
2872 Once you have one or more <<tpp-header,tracepoint provider header
2873 files>> and a <<tpp-source,tracepoint provider package source file>>,
2874 you can create the tracepoint provider package by compiling its source
2875 file. From here, multiple build and run scenarios are possible. The
2876 following table shows common application and library configurations
2877 along with the required command lines to achieve them.
2879 In the following diagrams, we use the following file names:
2882 Executable application.
2885 Application's object file.
2888 Tracepoint provider package object file.
2891 Tracepoint provider package archive file.
2894 Tracepoint provider package shared object file.
2897 User library object file.
2900 User library shared object file.
2902 We use the following symbols in the diagrams of table below:
2905 .Symbols used in the build scenario diagrams.
2906 image::ust-sit-symbols.png[]
2908 We assume that path:{.} is part of the env:LD_LIBRARY_PATH environment
2909 variable in the following instructions.
2911 [role="growable ust-scenarios",cols="asciidoc,asciidoc"]
2912 .Common tracepoint provider package scenarios.
2914 |Scenario |Instructions
2917 The instrumented application is statically linked with
2918 the tracepoint provider package object.
2920 image::ust-sit+app-linked-with-tp-o+app-instrumented.png[]
2923 include::../common/ust-sit-step-tp-o.txt[]
2925 To build the instrumented application:
2927 . In path:{app.c}, before including path:{tpp.h}, add the following line:
2932 #define TRACEPOINT_DEFINE
2936 . Compile the application source file:
2945 . Build the application:
2950 gcc -o app app.o tpp.o -llttng-ust -ldl
2954 To run the instrumented application:
2956 * Start the application:
2966 The instrumented application is statically linked with the
2967 tracepoint provider package archive file.
2969 image::ust-sit+app-linked-with-tp-a+app-instrumented.png[]
2972 To create the tracepoint provider package archive file:
2974 . Compile the <<tpp-source,tracepoint provider package source file>>:
2983 . Create the tracepoint provider package archive file:
2992 To build the instrumented application:
2994 . In path:{app.c}, before including path:{tpp.h}, add the following line:
2999 #define TRACEPOINT_DEFINE
3003 . Compile the application source file:
3012 . Build the application:
3017 gcc -o app app.o tpp.a -llttng-ust -ldl
3021 To run the instrumented application:
3023 * Start the application:
3033 The instrumented application is linked with the tracepoint provider
3034 package shared object.
3036 image::ust-sit+app-linked-with-tp-so+app-instrumented.png[]
3039 include::../common/ust-sit-step-tp-so.txt[]
3041 To build the instrumented application:
3043 . In path:{app.c}, before including path:{tpp.h}, add the following line:
3048 #define TRACEPOINT_DEFINE
3052 . Compile the application source file:
3061 . Build the application:
3066 gcc -o app app.o -ldl -L. -ltpp
3070 To run the instrumented application:
3072 * Start the application:
3082 The tracepoint provider package shared object is preloaded before the
3083 instrumented application starts.
3085 image::ust-sit+tp-so-preloaded+app-instrumented.png[]
3088 include::../common/ust-sit-step-tp-so.txt[]
3090 To build the instrumented application:
3092 . In path:{app.c}, before including path:{tpp.h}, add the
3098 #define TRACEPOINT_DEFINE
3099 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3103 . Compile the application source file:
3112 . Build the application:
3117 gcc -o app app.o -ldl
3121 To run the instrumented application with tracing support:
3123 * Preload the tracepoint provider package shared object and
3124 start the application:
3129 LD_PRELOAD=./libtpp.so ./app
3133 To run the instrumented application without tracing support:
3135 * Start the application:
3145 The instrumented application dynamically loads the tracepoint provider
3146 package shared object.
3148 See the <<dlclose-warning,warning about `dlclose()`>>.
3150 image::ust-sit+app-dlopens-tp-so+app-instrumented.png[]
3153 include::../common/ust-sit-step-tp-so.txt[]
3155 To build the instrumented application:
3157 . In path:{app.c}, before including path:{tpp.h}, add the
3163 #define TRACEPOINT_DEFINE
3164 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3168 . Compile the application source file:
3177 . Build the application:
3182 gcc -o app app.o -ldl
3186 To run the instrumented application:
3188 * Start the application:
3198 The application is linked with the instrumented user library.
3200 The instrumented user library is statically linked with the tracepoint
3201 provider package object file.
3203 image::ust-sit+app-linked-with-lib+lib-linked-with-tp-o+lib-instrumented.png[]
3206 include::../common/ust-sit-step-tp-o-fpic.txt[]
3208 To build the instrumented user library:
3210 . In path:{emon.c}, before including path:{tpp.h}, add the
3216 #define TRACEPOINT_DEFINE
3220 . Compile the user library source file:
3225 gcc -I. -fpic -c emon.c
3229 . Build the user library shared object:
3234 gcc -shared -o libemon.so emon.o tpp.o -llttng-ust -ldl
3238 To build the application:
3240 . Compile the application source file:
3249 . Build the application:
3254 gcc -o app app.o -L. -lemon
3258 To run the application:
3260 * Start the application:
3270 The application is linked with the instrumented user library.
3272 The instrumented user library is linked with the tracepoint provider
3273 package shared object.
3275 image::ust-sit+app-linked-with-lib+lib-linked-with-tp-so+lib-instrumented.png[]
3278 include::../common/ust-sit-step-tp-so.txt[]
3280 To build the instrumented user library:
3282 . In path:{emon.c}, before including path:{tpp.h}, add the
3288 #define TRACEPOINT_DEFINE
3292 . Compile the user library source file:
3297 gcc -I. -fpic -c emon.c
3301 . Build the user library shared object:
3306 gcc -shared -o libemon.so emon.o -ldl -L. -ltpp
3310 To build the application:
3312 . Compile the application source file:
3321 . Build the application:
3326 gcc -o app app.o -L. -lemon
3330 To run the application:
3332 * Start the application:
3342 The tracepoint provider package shared object is preloaded before the
3345 The application is linked with the instrumented user library.
3347 image::ust-sit+tp-so-preloaded+app-linked-with-lib+lib-instrumented.png[]
3350 include::../common/ust-sit-step-tp-so.txt[]
3352 To build the instrumented user library:
3354 . In path:{emon.c}, before including path:{tpp.h}, add the
3360 #define TRACEPOINT_DEFINE
3361 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3365 . Compile the user library source file:
3370 gcc -I. -fpic -c emon.c
3374 . Build the user library shared object:
3379 gcc -shared -o libemon.so emon.o -ldl
3383 To build the application:
3385 . Compile the application source file:
3394 . Build the application:
3399 gcc -o app app.o -L. -lemon
3403 To run the application with tracing support:
3405 * Preload the tracepoint provider package shared object and
3406 start the application:
3411 LD_PRELOAD=./libtpp.so ./app
3415 To run the application without tracing support:
3417 * Start the application:
3427 The application is linked with the instrumented user library.
3429 The instrumented user library dynamically loads the tracepoint provider
3430 package shared object.
3432 See the <<dlclose-warning,warning about `dlclose()`>>.
3434 image::ust-sit+app-linked-with-lib+lib-dlopens-tp-so+lib-instrumented.png[]
3437 include::../common/ust-sit-step-tp-so.txt[]
3439 To build the instrumented user library:
3441 . In path:{emon.c}, before including path:{tpp.h}, add the
3447 #define TRACEPOINT_DEFINE
3448 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3452 . Compile the user library source file:
3457 gcc -I. -fpic -c emon.c
3461 . Build the user library shared object:
3466 gcc -shared -o libemon.so emon.o -ldl
3470 To build the application:
3472 . Compile the application source file:
3481 . Build the application:
3486 gcc -o app app.o -L. -lemon
3490 To run the application:
3492 * Start the application:
3502 The application dynamically loads the instrumented user library.
3504 The instrumented user library is linked with the tracepoint provider
3505 package shared object.
3507 See the <<dlclose-warning,warning about `dlclose()`>>.
3509 image::ust-sit+app-dlopens-lib+lib-linked-with-tp-so+lib-instrumented.png[]
3512 include::../common/ust-sit-step-tp-so.txt[]
3514 To build the instrumented user library:
3516 . In path:{emon.c}, before including path:{tpp.h}, add the
3522 #define TRACEPOINT_DEFINE
3526 . Compile the user library source file:
3531 gcc -I. -fpic -c emon.c
3535 . Build the user library shared object:
3540 gcc -shared -o libemon.so emon.o -ldl -L. -ltpp
3544 To build the application:
3546 . Compile the application source file:
3555 . Build the application:
3560 gcc -o app app.o -ldl -L. -lemon
3564 To run the application:
3566 * Start the application:
3576 The application dynamically loads the instrumented user library.
3578 The instrumented user library dynamically loads the tracepoint provider
3579 package shared object.
3581 See the <<dlclose-warning,warning about `dlclose()`>>.
3583 image::ust-sit+app-dlopens-lib+lib-dlopens-tp-so+lib-instrumented.png[]
3586 include::../common/ust-sit-step-tp-so.txt[]
3588 To build the instrumented user library:
3590 . In path:{emon.c}, before including path:{tpp.h}, add the
3596 #define TRACEPOINT_DEFINE
3597 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3601 . Compile the user library source file:
3606 gcc -I. -fpic -c emon.c
3610 . Build the user library shared object:
3615 gcc -shared -o libemon.so emon.o -ldl
3619 To build the application:
3621 . Compile the application source file:
3630 . Build the application:
3635 gcc -o app app.o -ldl -L. -lemon
3639 To run the application:
3641 * Start the application:
3651 The tracepoint provider package shared object is preloaded before the
3654 The application dynamically loads the instrumented user library.
3656 image::ust-sit+tp-so-preloaded+app-dlopens-lib+lib-instrumented.png[]
3659 include::../common/ust-sit-step-tp-so.txt[]
3661 To build the instrumented user library:
3663 . In path:{emon.c}, before including path:{tpp.h}, add the
3669 #define TRACEPOINT_DEFINE
3670 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3674 . Compile the user library source file:
3679 gcc -I. -fpic -c emon.c
3683 . Build the user library shared object:
3688 gcc -shared -o libemon.so emon.o -ldl
3692 To build the application:
3694 . Compile the application source file:
3703 . Build the application:
3708 gcc -o app app.o -L. -lemon
3712 To run the application with tracing support:
3714 * Preload the tracepoint provider package shared object and
3715 start the application:
3720 LD_PRELOAD=./libtpp.so ./app
3724 To run the application without tracing support:
3726 * Start the application:
3736 The application is statically linked with the tracepoint provider
3737 package object file.
3739 The application is linked with the instrumented user library.
3741 image::ust-sit+app-linked-with-tp-o+app-linked-with-lib+lib-instrumented.png[]
3744 include::../common/ust-sit-step-tp-o.txt[]
3746 To build the instrumented user library:
3748 . In path:{emon.c}, before including path:{tpp.h}, add the
3754 #define TRACEPOINT_DEFINE
3758 . Compile the user library source file:
3763 gcc -I. -fpic -c emon.c
3767 . Build the user library shared object:
3772 gcc -shared -o libemon.so emon.o
3776 To build the application:
3778 . Compile the application source file:
3787 . Build the application:
3792 gcc -o app app.o tpp.o -llttng-ust -ldl -L. -lemon
3796 To run the instrumented application:
3798 * Start the application:
3808 The application is statically linked with the tracepoint provider
3809 package object file.
3811 The application dynamically loads the instrumented user library.
3813 image::ust-sit+app-linked-with-tp-o+app-dlopens-lib+lib-instrumented.png[]
3816 include::../common/ust-sit-step-tp-o.txt[]
3818 To build the application:
3820 . In path:{app.c}, before including path:{tpp.h}, add the following line:
3825 #define TRACEPOINT_DEFINE
3829 . Compile the application source file:
3838 . Build the application:
3843 gcc -Wl,--export-dynamic -o app app.o tpp.o \
3848 The `--export-dynamic` option passed to the linker is necessary for the
3849 dynamically loaded library to ``see'' the tracepoint symbols defined in
3852 To build the instrumented user library:
3854 . Compile the user library source file:
3859 gcc -I. -fpic -c emon.c
3863 . Build the user library shared object:
3868 gcc -shared -o libemon.so emon.o
3872 To run the application:
3874 * Start the application:
3886 .Do not use man:dlclose(3) on a tracepoint provider package
3888 Never use man:dlclose(3) on any shared object which:
3890 * Is linked with, statically or dynamically, a tracepoint provider
3892 * Calls man:dlopen(3) itself to dynamically open a tracepoint provider
3893 package shared object.
3895 This is currently considered **unsafe** due to a lack of reference
3896 counting from LTTng-UST to the shared object.
3898 A known workaround (available since glibc 2.2) is to use the
3899 `RTLD_NODELETE` flag when calling man:dlopen(3) initially. This has the
3900 effect of not unloading the loaded shared object, even if man:dlclose(3)
3903 You can also preload the tracepoint provider package shared object with
3904 the env:LD_PRELOAD environment variable to overcome this limitation.
3908 [[using-lttng-ust-with-daemons]]
3909 ===== Use noch:{LTTng-UST} with daemons
3911 If your instrumented application calls man:fork(2), man:clone(2),
3912 or BSD's man:rfork(2), without a following man:exec(3)-family
3913 system call, you must preload the path:{liblttng-ust-fork.so} shared
3914 object when starting the application.
3918 LD_PRELOAD=liblttng-ust-fork.so ./my-app
3921 If your tracepoint provider package is
3922 a shared library which you also preload, you must put both
3923 shared objects in env:LD_PRELOAD:
3927 LD_PRELOAD=liblttng-ust-fork.so:/path/to/tp.so ./my-app
3931 [[lttng-ust-pkg-config]]
3932 ===== Use noch:{pkg-config}
3934 On some distributions, LTTng-UST ships with a
3935 https://www.freedesktop.org/wiki/Software/pkg-config/[pkg-config]
3936 metadata file. If this is your case, then you can use cmd:pkg-config to
3937 build an application on the command line:
3941 gcc -o my-app my-app.o tp.o $(pkg-config --cflags --libs lttng-ust)
3945 [[instrumenting-32-bit-app-on-64-bit-system]]
3946 ===== [[advanced-instrumenting-techniques]]Build a 32-bit instrumented application for a 64-bit target system
3948 In order to trace a 32-bit application running on a 64-bit system,
3949 LTTng must use a dedicated 32-bit
3950 <<lttng-consumerd,consumer daemon>>.
3952 The following steps show how to build and install a 32-bit consumer
3953 daemon, which is _not_ part of the default 64-bit LTTng build, how to
3954 build and install the 32-bit LTTng-UST libraries, and how to build and
3955 link an instrumented 32-bit application in that context.
3957 To build a 32-bit instrumented application for a 64-bit target system,
3958 assuming you have a fresh target system with no installed Userspace RCU
3961 . Download, build, and install a 32-bit version of Userspace RCU:
3967 wget http://lttng.org/files/urcu/userspace-rcu-latest-0.9.tar.bz2 &&
3968 tar -xf userspace-rcu-latest-0.9.tar.bz2 &&
3969 cd userspace-rcu-0.9.* &&
3970 ./configure --libdir=/usr/local/lib32 CFLAGS=-m32 &&
3972 sudo make install &&
3977 . Using your distribution's package manager, or from source, install
3978 the following 32-bit versions of the following dependencies of
3979 LTTng-tools and LTTng-UST:
3982 * https://sourceforge.net/projects/libuuid/[libuuid]
3983 * http://directory.fsf.org/wiki/Popt[popt]
3984 * http://www.xmlsoft.org/[libxml2]
3987 . Download, build, and install a 32-bit version of the latest
3988 LTTng-UST{nbsp}{revision}:
3994 wget http://lttng.org/files/lttng-ust/lttng-ust-latest-2.8.tar.bz2 &&
3995 tar -xf lttng-ust-latest-2.8.tar.bz2 &&
3996 cd lttng-ust-2.8.* &&
3997 ./configure --libdir=/usr/local/lib32 \
3998 CFLAGS=-m32 CXXFLAGS=-m32 \
3999 LDFLAGS='-L/usr/local/lib32 -L/usr/lib32' &&
4001 sudo make install &&
4008 Depending on your distribution,
4009 32-bit libraries could be installed at a different location than
4010 `/usr/lib32`. For example, Debian is known to install
4011 some 32-bit libraries in `/usr/lib/i386-linux-gnu`.
4013 In this case, make sure to set `LDFLAGS` to all the
4014 relevant 32-bit library paths, for example:
4018 LDFLAGS='-L/usr/lib/i386-linux-gnu -L/usr/lib32'
4022 . Download the latest LTTng-tools{nbsp}{revision}, build, and install
4023 the 32-bit consumer daemon:
4029 wget http://lttng.org/files/lttng-tools/lttng-tools-latest-2.8.tar.bz2 &&
4030 tar -xf lttng-tools-latest-2.8.tar.bz2 &&
4031 cd lttng-tools-2.8.* &&
4032 ./configure --libdir=/usr/local/lib32 CFLAGS=-m32 CXXFLAGS=-m32 \
4033 LDFLAGS='-L/usr/local/lib32 -L/usr/lib32' &&
4035 cd src/bin/lttng-consumerd &&
4036 sudo make install &&
4041 . From your distribution or from source,
4042 <<installing-lttng,install>> the 64-bit versions of
4043 LTTng-UST and Userspace RCU.
4044 . Download, build, and install the 64-bit version of the
4045 latest LTTng-tools{nbsp}{revision}:
4051 wget http://lttng.org/files/lttng-tools/lttng-tools-latest-2.8.tar.bz2 &&
4052 tar -xf lttng-tools-latest-2.8.tar.bz2 &&
4053 cd lttng-tools-2.8.* &&
4054 ./configure --with-consumerd32-libdir=/usr/local/lib32 \
4055 --with-consumerd32-bin=/usr/local/lib32/lttng/libexec/lttng-consumerd &&
4057 sudo make install &&
4062 . Pass the following options to man:gcc(1), man:g++(1), or man:clang(1)
4063 when linking your 32-bit application:
4066 -m32 -L/usr/lib32 -L/usr/local/lib32 \
4067 -Wl,-rpath,/usr/lib32,-rpath,/usr/local/lib32
4070 For example, let's rebuild the quick start example in
4071 <<tracing-your-own-user-application,Trace a user application>> as an
4072 instrumented 32-bit application:
4077 gcc -m32 -c -I. hello-tp.c
4079 gcc -m32 -o hello hello.o hello-tp.o \
4080 -L/usr/lib32 -L/usr/local/lib32 \
4081 -Wl,-rpath,/usr/lib32,-rpath,/usr/local/lib32 \
4086 No special action is required to execute the 32-bit application and
4087 to trace it: use the command-line man:lttng(1) tool as usual.
4094 man:tracef(3) is a small LTTng-UST API designed for quick,
4095 man:printf(3)-like instrumentation without the burden of
4096 <<tracepoint-provider,creating>> and
4097 <<building-tracepoint-providers-and-user-application,building>>
4098 a tracepoint provider package.
4100 To use `tracef()` in your application:
4102 . In the C or C++ source files where you need to use `tracef()`,
4103 include `<lttng/tracef.h>`:
4108 #include <lttng/tracef.h>
4112 . In the application's source code, use `tracef()` like you would use
4120 tracef("my message: %d (%s)", my_integer, my_string);
4126 . Link your application with `liblttng-ust`:
4131 gcc -o app app.c -llttng-ust
4135 To trace the events that `tracef()` calls emit:
4137 * <<enabling-disabling-events,Create an event rule>> which matches the
4138 `lttng_ust_tracef:*` event name:
4143 lttng enable-event --userspace 'lttng_ust_tracef:*'
4148 .Limitations of `tracef()`
4150 The `tracef()` utility function was developed to make user space tracing
4151 super simple, albeit with notable disadvantages compared to
4152 <<defining-tracepoints,user-defined tracepoints>>:
4154 * All the emitted events have the same tracepoint provider and
4155 tracepoint names, respectively `lttng_ust_tracef` and `event`.
4156 * There is no static type checking.
4157 * The only event record field you actually get, named `msg`, is a string
4158 potentially containing the values you passed to `tracef()`
4159 using your own format string. This also means that you cannot filter
4160 events with a custom expression at run time because there are no
4162 * Since `tracef()` uses the C standard library's man:vasprintf(3)
4163 function behind the scenes to format the strings at run time, its
4164 expected performance is lower than with user-defined tracepoints,
4165 which do not require a conversion to a string.
4167 Taking this into consideration, `tracef()` is useful for some quick
4168 prototyping and debugging, but you should not consider it for any
4169 permanent and serious applicative instrumentation.
4175 ==== Use `tracelog()`
4177 The man:tracelog(3) API is very similar to <<tracef,`tracef()`>>, with
4178 the difference that it accepts an additional log level parameter.
4180 The goal of `tracelog()` is to ease the migration from logging to
4183 To use `tracelog()` in your application:
4185 . In the C or C++ source files where you need to use `tracelog()`,
4186 include `<lttng/tracelog.h>`:
4191 #include <lttng/tracelog.h>
4195 . In the application's source code, use `tracelog()` like you would use
4196 man:printf(3), except for the first parameter which is the log
4204 tracelog(TRACE_WARNING, "my message: %d (%s)",
4205 my_integer, my_string);
4211 See man:lttng-ust(3) for a list of available log level names.
4213 . Link your application with `liblttng-ust`:
4218 gcc -o app app.c -llttng-ust
4222 To trace the events that `tracelog()` calls emit with a log level
4223 _as severe as_ a specific log level:
4225 * <<enabling-disabling-events,Create an event rule>> which matches the
4226 `lttng_ust_tracelog:*` event name and a minimum level
4232 lttng enable-event --userspace 'lttng_ust_tracelog:*'
4233 --loglevel=TRACE_WARNING
4237 To trace the events that `tracelog()` calls emit with a
4238 _specific log level_:
4240 * Create an event rule which matches the `lttng_ust_tracelog:*`
4241 event name and a specific log level:
4246 lttng enable-event --userspace 'lttng_ust_tracelog:*'
4247 --loglevel-only=TRACE_INFO
4252 [[prebuilt-ust-helpers]]
4253 === Prebuilt user space tracing helpers
4255 The LTTng-UST package provides a few helpers in the form or preloadable
4256 shared objects which automatically instrument system functions and
4259 The helper shared objects are normally found in dir:{/usr/lib}. If you
4260 built LTTng-UST <<building-from-source,from source>>, they are probably
4261 located in dir:{/usr/local/lib}.
4263 The installed user space tracing helpers in LTTng-UST{nbsp}{revision}
4266 path:{liblttng-ust-libc-wrapper.so}::
4267 path:{liblttng-ust-pthread-wrapper.so}::
4268 <<liblttng-ust-libc-pthread-wrapper,C{nbsp}standard library
4269 memory and POSIX threads function tracing>>.
4271 path:{liblttng-ust-cyg-profile.so}::
4272 path:{liblttng-ust-cyg-profile-fast.so}::
4273 <<liblttng-ust-cyg-profile,Function entry and exit tracing>>.
4275 path:{liblttng-ust-dl.so}::
4276 <<liblttng-ust-dl,Dynamic linker tracing>>.
4278 To use a user space tracing helper with any user application:
4280 * Preload the helper shared object when you start the application:
4285 LD_PRELOAD=liblttng-ust-libc-wrapper.so my-app
4289 You can preload more than one helper:
4294 LD_PRELOAD=liblttng-ust-libc-wrapper.so:liblttng-ust-dl.so my-app
4300 [[liblttng-ust-libc-pthread-wrapper]]
4301 ==== Instrument C standard library memory and POSIX threads functions
4303 The path:{liblttng-ust-libc-wrapper.so} and
4304 path:{liblttng-ust-pthread-wrapper.so} helpers
4305 add instrumentation to some C standard library and POSIX
4309 .Functions instrumented by preloading path:{liblttng-ust-libc-wrapper.so}.
4311 |TP provider name |TP name |Instrumented function
4313 .6+|`lttng_ust_libc` |`malloc` |man:malloc(3)
4314 |`calloc` |man:calloc(3)
4315 |`realloc` |man:realloc(3)
4316 |`free` |man:free(3)
4317 |`memalign` |man:memalign(3)
4318 |`posix_memalign` |man:posix_memalign(3)
4322 .Functions instrumented by preloading path:{liblttng-ust-pthread-wrapper.so}.
4324 |TP provider name |TP name |Instrumented function
4326 .4+|`lttng_ust_pthread` |`pthread_mutex_lock_req` |man:pthread_mutex_lock(3p) (request time)
4327 |`pthread_mutex_lock_acq` |man:pthread_mutex_lock(3p) (acquire time)
4328 |`pthread_mutex_trylock` |man:pthread_mutex_trylock(3p)
4329 |`pthread_mutex_unlock` |man:pthread_mutex_unlock(3p)
4332 When you preload the shared object, it replaces the functions listed
4333 in the previous tables by wrappers which contain tracepoints and call
4334 the replaced functions.
4337 [[liblttng-ust-cyg-profile]]
4338 ==== Instrument function entry and exit
4340 The path:{liblttng-ust-cyg-profile*.so} helpers can add instrumentation
4341 to the entry and exit points of functions.
4343 man:gcc(1) and man:clang(1) have an option named
4344 https://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html[`-finstrument-functions`]
4345 which generates instrumentation calls for entry and exit to functions.
4346 The LTTng-UST function tracing helpers,
4347 path:{liblttng-ust-cyg-profile.so} and
4348 path:{liblttng-ust-cyg-profile-fast.so}, take advantage of this feature
4349 to add tracepoints to the two generated functions (which contain
4350 `cyg_profile` in their names, hence the helper's name).
4352 To use the LTTng-UST function tracing helper, the source files to
4353 instrument must be built using the `-finstrument-functions` compiler
4356 There are two versions of the LTTng-UST function tracing helper:
4358 * **path:{liblttng-ust-cyg-profile-fast.so}** is a lightweight variant
4359 that you should only use when it can be _guaranteed_ that the
4360 complete event stream is recorded without any lost event record.
4361 Any kind of duplicate information is left out.
4363 Assuming no event record is lost, having only the function addresses on
4364 entry is enough to create a call graph, since an event record always
4365 contains the ID of the CPU that generated it.
4367 You can use a tool like man:addr2line(1) to convert function addresses
4368 back to source file names and line numbers.
4370 * **path:{liblttng-ust-cyg-profile.so}** is a more robust variant
4371 which also works in use cases where event records might get discarded or
4372 not recorded from application startup.
4373 In these cases, the trace analyzer needs more information to be
4374 able to reconstruct the program flow.
4376 See man:lttng-ust-cyg-profile(3) to learn more about the instrumentation
4377 points of this helper.
4379 All the tracepoints that this helper provides have the
4380 log level `TRACE_DEBUG_FUNCTION` (see man:lttng-ust(3)).
4382 TIP: It's sometimes a good idea to limit the number of source files that
4383 you compile with the `-finstrument-functions` option to prevent LTTng
4384 from writing an excessive amount of trace data at run time. When using
4385 man:gcc(1), you can use the
4386 `-finstrument-functions-exclude-function-list` option to avoid
4387 instrument entries and exits of specific function names.
4392 ==== Instrument the dynamic linker
4394 The path:{liblttng-ust-dl.so} helper adds instrumentation to the
4395 man:dlopen(3) and man:dlclose(3) function calls.
4397 See man:lttng-ust-dl(3) to learn more about the instrumentation points
4402 [[java-application]]
4403 === User space Java agent
4405 You can instrument any Java application which uses one of the following
4408 * The https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[**`java.util.logging`**]
4409 (JUL) core logging facilities.
4410 * http://logging.apache.org/log4j/1.2/[**Apache log4j 1.2**], since
4411 LTTng 2.6. Note that Apache Log4j{nbsp}2 is not supported.
4414 .LTTng-UST Java agent imported by a Java application.
4415 image::java-app.png[]
4417 Note that the methods described below are new in LTTng{nbsp}{revision}.
4418 Previous LTTng versions use another technique.
4420 NOTE: We use http://openjdk.java.net/[OpenJDK]{nbsp}8 for development
4421 and https://ci.lttng.org/[continuous integration], thus this version is
4422 directly supported. However, the LTTng-UST Java agent is also tested
4423 with OpenJDK{nbsp}7.
4428 ==== Use the LTTng-UST Java agent for `java.util.logging`
4430 To use the LTTng-UST Java agent in a Java application which uses
4431 `java.util.logging` (JUL):
4433 . In the Java application's source code, import the LTTng-UST
4434 log handler package for `java.util.logging`:
4439 import org.lttng.ust.agent.jul.LttngLogHandler;
4443 . Create an LTTng-UST JUL log handler:
4448 Handler lttngUstLogHandler = new LttngLogHandler();
4452 . Add this handler to the JUL loggers which should emit LTTng events:
4457 Logger myLogger = Logger.getLogger("some-logger");
4459 myLogger.addHandler(lttngUstLogHandler);
4463 . Use `java.util.logging` log statements and configuration as usual.
4464 The loggers with an attached LTTng-UST log handler can emit
4467 . Before exiting the application, remove the LTTng-UST log handler from
4468 the loggers attached to it and call its `close()` method:
4473 myLogger.removeHandler(lttngUstLogHandler);
4474 lttngUstLogHandler.close();
4478 This is not strictly necessary, but it is recommended for a clean
4479 disposal of the handler's resources.
4481 . Include the LTTng-UST Java agent's common and JUL-specific JAR files,
4482 path:{lttng-ust-agent-common.jar} and path:{lttng-ust-agent-jul.jar},
4484 https://docs.oracle.com/javase/tutorial/essential/environment/paths.html[class
4485 path] when you build the Java application.
4487 The JAR files are typically located in dir:{/usr/share/java}.
4489 IMPORTANT: The LTTng-UST Java agent must be
4490 <<installing-lttng,installed>> for the logging framework your
4493 .Use the LTTng-UST Java agent for `java.util.logging`.
4498 import java.io.IOException;
4499 import java.util.logging.Handler;
4500 import java.util.logging.Logger;
4501 import org.lttng.ust.agent.jul.LttngLogHandler;
4505 private static final int answer = 42;
4507 public static void main(String[] argv) throws Exception
4510 Logger logger = Logger.getLogger("jello");
4512 // Create an LTTng-UST log handler
4513 Handler lttngUstLogHandler = new LttngLogHandler();
4515 // Add the LTTng-UST log handler to our logger
4516 logger.addHandler(lttngUstLogHandler);
4519 logger.info("some info");
4520 logger.warning("some warning");
4522 logger.finer("finer information; the answer is " + answer);
4524 logger.severe("error!");
4526 // Not mandatory, but cleaner
4527 logger.removeHandler(lttngUstLogHandler);
4528 lttngUstLogHandler.close();
4537 javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar Test.java
4540 <<creating-destroying-tracing-sessions,Create a tracing session>>,
4541 <<enabling-disabling-events,create an event rule>> matching the
4542 `jello` JUL logger, and <<basic-tracing-session-control,start tracing>>:
4547 lttng enable-event --jul jello
4551 Run the compiled class:
4555 java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar:. Test
4558 <<basic-tracing-session-control,Stop tracing>> and inspect the
4568 You can use the opt:lttng-enable-event(1):--loglevel or
4569 opt:lttng-enable-event(1):--loglevel-only option of the
4570 man:lttng-enable-event(1) command to target a range of JUL log levels
4571 or a specific JUL log level.
4576 ==== Use the LTTng-UST Java agent for Apache log4j
4578 To use the LTTng-UST Java agent in a Java application which uses
4581 . In the Java application's source code, import the LTTng-UST
4582 log appender package for Apache log4j:
4587 import org.lttng.ust.agent.log4j.LttngLogAppender;
4591 . Create an LTTng-UST log4j log appender:
4596 Appender lttngUstLogAppender = new LttngLogAppender();
4600 . Add this appender to the log4j loggers which should emit LTTng events:
4605 Logger myLogger = Logger.getLogger("some-logger");
4607 myLogger.addAppender(lttngUstLogAppender);
4611 . Use Apache log4j log statements and configuration as usual. The
4612 loggers with an attached LTTng-UST log appender can emit LTTng events.
4614 . Before exiting the application, remove the LTTng-UST log appender from
4615 the loggers attached to it and call its `close()` method:
4620 myLogger.removeAppender(lttngUstLogAppender);
4621 lttngUstLogAppender.close();
4625 This is not strictly necessary, but it is recommended for a clean
4626 disposal of the appender's resources.
4628 . Include the LTTng-UST Java agent's common and log4j-specific JAR
4629 files, path:{lttng-ust-agent-common.jar} and
4630 path:{lttng-ust-agent-log4j.jar}, in the
4631 https://docs.oracle.com/javase/tutorial/essential/environment/paths.html[class
4632 path] when you build the Java application.
4634 The JAR files are typically located in dir:{/usr/share/java}.
4636 IMPORTANT: The LTTng-UST Java agent must be
4637 <<installing-lttng,installed>> for the logging framework your
4640 .Use the LTTng-UST Java agent for Apache log4j.
4645 import org.apache.log4j.Appender;
4646 import org.apache.log4j.Logger;
4647 import org.lttng.ust.agent.log4j.LttngLogAppender;
4651 private static final int answer = 42;
4653 public static void main(String[] argv) throws Exception
4656 Logger logger = Logger.getLogger("jello");
4658 // Create an LTTng-UST log appender
4659 Appender lttngUstLogAppender = new LttngLogAppender();
4661 // Add the LTTng-UST log appender to our logger
4662 logger.addAppender(lttngUstLogAppender);
4665 logger.info("some info");
4666 logger.warn("some warning");
4668 logger.debug("debug information; the answer is " + answer);
4670 logger.fatal("error!");
4672 // Not mandatory, but cleaner
4673 logger.removeAppender(lttngUstLogAppender);
4674 lttngUstLogAppender.close();
4680 Build this example (`$LOG4JPATH` is the path to the Apache log4j JAR
4685 javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-log4j.jar:$LOG4JPATH Test.java
4688 <<creating-destroying-tracing-sessions,Create a tracing session>>,
4689 <<enabling-disabling-events,create an event rule>> matching the
4690 `jello` log4j logger, and <<basic-tracing-session-control,start tracing>>:
4695 lttng enable-event --log4j jello
4699 Run the compiled class:
4703 java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-log4j.jar:$LOG4JPATH:. Test
4706 <<basic-tracing-session-control,Stop tracing>> and inspect the
4716 You can use the opt:lttng-enable-event(1):--loglevel or
4717 opt:lttng-enable-event(1):--loglevel-only option of the
4718 man:lttng-enable-event(1) command to target a range of Apache log4j log levels
4719 or a specific log4j log level.
4723 [[java-application-context]]
4724 ==== Provide application-specific context fields in a Java application
4726 A Java application-specific context field is a piece of state provided
4727 by the application which <<adding-context,you can add>>, using the
4728 man:lttng-add-context(1) command, to each <<event,event record>>
4729 produced by the log statements of this application.
4731 For example, a given object might have a current request ID variable.
4732 You can create a context information retriever for this object and
4733 assign a name to this current request ID. You can then, using the
4734 man:lttng-add-context(1) command, add this context field by name to
4735 the JUL or log4j <<channel,channel>>.
4737 To provide application-specific context fields in a Java application:
4739 . In the Java application's source code, import the LTTng-UST
4740 Java agent context classes and interfaces:
4745 import org.lttng.ust.agent.context.ContextInfoManager;
4746 import org.lttng.ust.agent.context.IContextInfoRetriever;
4750 . Create a context information retriever class, that is, a class which
4751 implements the `IContextInfoRetriever` interface:
4756 class MyContextInfoRetriever implements IContextInfoRetriever
4759 public Object retrieveContextInfo(String key)
4761 if (key.equals("intCtx")) {
4763 } else if (key.equals("strContext")) {
4764 return "context value!";
4773 This `retrieveContextInfo()` method is the only member of the
4774 `IContextInfoRetriever` interface. Its role is to return the current
4775 value of a state by name to create a context field. The names of the
4776 context fields and which state variables they return depends on your
4779 All primitive types and objects are supported as context fields.
4780 When `retrieveContextInfo()` returns an object, the context field
4781 serializer calls its `toString()` method to add a string field to
4782 event records. The method can also return `null`, which means that
4783 no context field is available for the required name.
4785 . Register an instance of your context information retriever class to
4786 the context information manager singleton:
4791 IContextInfoRetriever cir = new MyContextInfoRetriever();
4792 ContextInfoManager cim = ContextInfoManager.getInstance();
4793 cim.registerContextInfoRetriever("retrieverName", cir);
4797 . Before exiting the application, remove your context information
4798 retriever from the context information manager singleton:
4803 ContextInfoManager cim = ContextInfoManager.getInstance();
4804 cim.unregisterContextInfoRetriever("retrieverName");
4808 This is not strictly necessary, but it is recommended for a clean
4809 disposal of some manager's resources.
4811 . Build your Java application with LTTng-UST Java agent support as
4812 usual, following the procedure for either the <<jul,JUL>> or
4813 <<log4j,Apache log4j>> framework.
4816 .Provide application-specific context fields in a Java application.
4821 import java.util.logging.Handler;
4822 import java.util.logging.Logger;
4823 import org.lttng.ust.agent.jul.LttngLogHandler;
4824 import org.lttng.ust.agent.context.ContextInfoManager;
4825 import org.lttng.ust.agent.context.IContextInfoRetriever;
4829 // Our context information retriever class
4830 private static class MyContextInfoRetriever
4831 implements IContextInfoRetriever
4834 public Object retrieveContextInfo(String key) {
4835 if (key.equals("intCtx")) {
4837 } else if (key.equals("strContext")) {
4838 return "context value!";
4845 private static final int answer = 42;
4847 public static void main(String args[]) throws Exception
4849 // Get the context information manager instance
4850 ContextInfoManager cim = ContextInfoManager.getInstance();
4852 // Create and register our context information retriever
4853 IContextInfoRetriever cir = new MyContextInfoRetriever();
4854 cim.registerContextInfoRetriever("myRetriever", cir);
4857 Logger logger = Logger.getLogger("jello");
4859 // Create an LTTng-UST log handler
4860 Handler lttngUstLogHandler = new LttngLogHandler();
4862 // Add the LTTng-UST log handler to our logger
4863 logger.addHandler(lttngUstLogHandler);
4866 logger.info("some info");
4867 logger.warning("some warning");
4869 logger.finer("finer information; the answer is " + answer);
4871 logger.severe("error!");
4873 // Not mandatory, but cleaner
4874 logger.removeHandler(lttngUstLogHandler);
4875 lttngUstLogHandler.close();
4876 cim.unregisterContextInfoRetriever("myRetriever");
4885 javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar Test.java
4888 <<creating-destroying-tracing-sessions,Create a tracing session>>
4889 and <<enabling-disabling-events,create an event rule>> matching the
4895 lttng enable-event --jul jello
4898 <<adding-context,Add the application-specific context fields>> to the
4903 lttng add-context --jul --type='$app.myRetriever:intCtx'
4904 lttng add-context --jul --type='$app.myRetriever:strContext'
4907 <<basic-tracing-session-control,Start tracing>>:
4914 Run the compiled class:
4918 java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar:. Test
4921 <<basic-tracing-session-control,Stop tracing>> and inspect the
4933 [[python-application]]
4934 === User space Python agent
4936 You can instrument a Python 2 or Python 3 application which uses the
4937 standard https://docs.python.org/3/library/logging.html[`logging`]
4940 Each log statement emits an LTTng event once the
4941 application module imports the
4942 <<lttng-ust-agents,LTTng-UST Python agent>> package.
4945 .A Python application importing the LTTng-UST Python agent.
4946 image::python-app.png[]
4948 To use the LTTng-UST Python agent:
4950 . In the Python application's source code, import the LTTng-UST Python
4960 The LTTng-UST Python agent automatically adds its logging handler to the
4961 root logger at import time.
4963 Any log statement that the application executes before this import does
4964 not emit an LTTng event.
4966 IMPORTANT: The LTTng-UST Python agent must be
4967 <<installing-lttng,installed>>.
4969 . Use log statements and logging configuration as usual.
4970 Since the LTTng-UST Python agent adds a handler to the _root_
4971 logger, you can trace any log statement from any logger.
4973 .Use the LTTng-UST Python agent.
4984 logging.basicConfig()
4985 logger = logging.getLogger('my-logger')
4988 logger.debug('debug message')
4989 logger.info('info message')
4990 logger.warn('warn message')
4991 logger.error('error message')
4992 logger.critical('critical message')
4996 if __name__ == '__main__':
5000 NOTE: `logging.basicConfig()`, which adds to the root logger a basic
5001 logging handler which prints to the standard error stream, is not
5002 strictly required for LTTng-UST tracing to work, but in versions of
5003 Python preceding 3.2, you could see a warning message which indicates
5004 that no handler exists for the logger `my-logger`.
5006 <<creating-destroying-tracing-sessions,Create a tracing session>>,
5007 <<enabling-disabling-events,create an event rule>> matching the
5008 `my-logger` Python logger, and <<basic-tracing-session-control,start
5014 lttng enable-event --python my-logger
5018 Run the Python script:
5025 <<basic-tracing-session-control,Stop tracing>> and inspect the recorded
5035 You can use the opt:lttng-enable-event(1):--loglevel or
5036 opt:lttng-enable-event(1):--loglevel-only option of the
5037 man:lttng-enable-event(1) command to target a range of Python log levels
5038 or a specific Python log level.
5040 When an application imports the LTTng-UST Python agent, the agent tries
5041 to register to a <<lttng-sessiond,session daemon>>. Note that you must
5042 <<start-sessiond,start the session daemon>> _before_ you run the Python
5043 application. If a session daemon is found, the agent tries to register
5044 to it during 5{nbsp}seconds, after which the application continues
5045 without LTTng tracing support. You can override this timeout value with
5046 the env:LTTNG_UST_PYTHON_REGISTER_TIMEOUT environment variable
5049 If the session daemon stops while a Python application with an imported
5050 LTTng-UST Python agent runs, the agent retries to connect and to
5051 register to a session daemon every 3{nbsp}seconds. You can override this
5052 delay with the env:LTTNG_UST_PYTHON_REGISTER_RETRY_DELAY environment
5057 [[proc-lttng-logger-abi]]
5060 The `lttng-tracer` Linux kernel module, part of
5061 <<lttng-modules,LTTng-modules>>, creates the special LTTng logger file
5062 path:{/proc/lttng-logger} when it's loaded. Any application can write
5063 text data to this file to emit an LTTng event.
5066 .An application writes to the LTTng logger file to emit an LTTng event.
5067 image::lttng-logger.png[]
5069 The LTTng logger is the quickest method--not the most efficient,
5070 however--to add instrumentation to an application. It is designed
5071 mostly to instrument shell scripts:
5075 echo "Some message, some $variable" > /proc/lttng-logger
5078 Any event that the LTTng logger emits is named `lttng_logger` and
5079 belongs to the Linux kernel <<domain,tracing domain>>. However, unlike
5080 other instrumentation points in the kernel tracing domain, **any Unix
5081 user** can <<enabling-disabling-events,create an event rule>> which
5082 matches its event name, not only the root user or users in the tracing
5085 To use the LTTng logger:
5087 * From any application, write text data to the path:{/proc/lttng-logger}
5090 The `msg` field of `lttng_logger` event records contains the
5093 NOTE: The maximum message length of an LTTng logger event is
5094 1024{nbsp}bytes. Writing more than this makes the LTTng logger emit more
5095 than one event to contain the remaining data.
5097 You should not use the LTTng logger to trace a user application which
5098 can be instrumented in a more efficient way, namely:
5100 * <<c-application,C and $$C++$$ applications>>.
5101 * <<java-application,Java applications>>.
5102 * <<python-application,Python applications>>.
5104 .Use the LTTng logger.
5109 echo 'Hello, World!' > /proc/lttng-logger
5111 df --human-readable --print-type / > /proc/lttng-logger
5114 <<creating-destroying-tracing-sessions,Create a tracing session>>,
5115 <<enabling-disabling-events,create an event rule>> matching the
5116 `lttng_logger` Linux kernel tracepoint, and
5117 <<basic-tracing-session-control,start tracing>>:
5122 lttng enable-event --kernel lttng_logger
5126 Run the Bash script:
5133 <<basic-tracing-session-control,Stop tracing>> and inspect the recorded
5144 [[instrumenting-linux-kernel]]
5145 === LTTng kernel tracepoints
5147 NOTE: This section shows how to _add_ instrumentation points to the
5148 Linux kernel. The kernel's subsystems are already thoroughly
5149 instrumented at strategic places for LTTng when you
5150 <<installing-lttng,install>> the <<lttng-modules,LTTng-modules>>
5154 There are two methods to instrument the Linux kernel:
5156 . <<linux-add-lttng-layer,Add an LTTng layer>> over an existing ftrace
5157 tracepoint which uses the `TRACE_EVENT()` API.
5159 Choose this if you want to instrumentation a Linux kernel tree with an
5160 instrumentation point compatible with ftrace, perf, and SystemTap.
5162 . Use an <<linux-lttng-tracepoint-event,LTTng-only approach>> to
5163 instrument an out-of-tree kernel module.
5165 Choose this if you don't need ftrace, perf, or SystemTap support.
5169 [[linux-add-lttng-layer]]
5170 ==== [[instrumenting-linux-kernel-itself]][[mainline-trace-event]][[lttng-adaptation-layer]]Add an LTTng layer to an existing ftrace tracepoint
5172 This section shows how to add an LTTng layer to existing ftrace
5173 instrumentation using the `TRACE_EVENT()` API.
5175 This section does not document the `TRACE_EVENT()` macro. You can
5176 read the following articles to learn more about this API:
5178 * http://lwn.net/Articles/379903/[Using the TRACE_EVENT() macro (Part 1)]
5179 * http://lwn.net/Articles/381064/[Using the TRACE_EVENT() macro (Part 2)]
5180 * http://lwn.net/Articles/383362/[Using the TRACE_EVENT() macro (Part 3)]
5182 The following procedure assumes that your ftrace tracepoints are
5183 correctly defined in their own header and that they are created in
5184 one source file using the `CREATE_TRACE_POINTS` definition.
5186 To add an LTTng layer over an existing ftrace tracepoint:
5188 . Make sure the following kernel configuration options are
5194 * `CONFIG_HIGH_RES_TIMERS`
5195 * `CONFIG_TRACEPOINTS`
5198 . Build the Linux source tree with your custom ftrace tracepoints.
5199 . Boot the resulting Linux image on your target system.
5201 Confirm that the tracepoints exist by looking for their names in the
5202 dir:{/sys/kernel/debug/tracing/events/subsys} directory, where `subsys`
5203 is your subsystem's name.
5205 . Get a copy of the latest LTTng-modules{nbsp}{revision}:
5211 wget http://lttng.org/files/lttng-modules/lttng-modules-latest-2.8.tar.bz2 &&
5212 tar -xf lttng-modules-latest-2.8.tar.bz2 &&
5213 cd lttng-modules-2.8.*
5217 . In dir:{instrumentation/events/lttng-module}, relative to the root
5218 of the LTTng-modules source tree, create a header file named
5219 +__subsys__.h+ for your custom subsystem +__subsys__+ and write your
5220 LTTng-modules tracepoint definitions using the LTTng-modules
5223 Start with this template:
5227 .path:{instrumentation/events/lttng-module/my_subsys.h}
5230 #define TRACE_SYSTEM my_subsys
5232 #if !defined(_LTTNG_MY_SUBSYS_H) || defined(TRACE_HEADER_MULTI_READ)
5233 #define _LTTNG_MY_SUBSYS_H
5235 #include "../../../probes/lttng-tracepoint-event.h"
5236 #include <linux/tracepoint.h>
5238 LTTNG_TRACEPOINT_EVENT(
5240 * Format is identical to TRACE_EVENT()'s version for the three
5241 * following macro parameters:
5244 TP_PROTO(int my_int, const char *my_string),
5245 TP_ARGS(my_int, my_string),
5247 /* LTTng-modules specific macros */
5249 ctf_integer(int, my_int_field, my_int)
5250 ctf_string(my_bar_field, my_bar)
5254 #endif /* !defined(_LTTNG_MY_SUBSYS_H) || defined(TRACE_HEADER_MULTI_READ) */
5256 #include "../../../probes/define_trace.h"
5260 The entries in the `TP_FIELDS()` section are the list of fields for the
5261 LTTng tracepoint. This is similar to the `TP_STRUCT__entry()` part of
5262 ftrace's `TRACE_EVENT()` macro.
5264 See <<lttng-modules-tp-fields,Tracepoint fields macros>> for a
5265 complete description of the available `ctf_*()` macros.
5267 . Create the LTTng-modules probe's kernel module C source file,
5268 +probes/lttng-probe-__subsys__.c+, where +__subsys__+ is your
5273 .path:{probes/lttng-probe-my-subsys.c}
5275 #include <linux/module.h>
5276 #include "../lttng-tracer.h"
5279 * Build-time verification of mismatch between mainline
5280 * TRACE_EVENT() arguments and the LTTng-modules adaptation
5281 * layer LTTNG_TRACEPOINT_EVENT() arguments.
5283 #include <trace/events/my_subsys.h>
5285 /* Create LTTng tracepoint probes */
5286 #define LTTNG_PACKAGE_BUILD
5287 #define CREATE_TRACE_POINTS
5288 #define TRACE_INCLUDE_PATH ../instrumentation/events/lttng-module
5290 #include "../instrumentation/events/lttng-module/my_subsys.h"
5292 MODULE_LICENSE("GPL and additional rights");
5293 MODULE_AUTHOR("Your name <your-email>");
5294 MODULE_DESCRIPTION("LTTng my_subsys probes");
5295 MODULE_VERSION(__stringify(LTTNG_MODULES_MAJOR_VERSION) "."
5296 __stringify(LTTNG_MODULES_MINOR_VERSION) "."
5297 __stringify(LTTNG_MODULES_PATCHLEVEL_VERSION)
5298 LTTNG_MODULES_EXTRAVERSION);
5302 . Edit path:{probes/Makefile} and add your new kernel module object
5303 next to the existing ones:
5307 .path:{probes/Makefile}
5311 obj-m += lttng-probe-module.o
5312 obj-m += lttng-probe-power.o
5314 obj-m += lttng-probe-my-subsys.o
5320 . Build and install the LTTng kernel modules:
5325 make KERNELDIR=/path/to/linux
5326 sudo make modules_install
5330 Replace `/path/to/linux` with the path to the Linux source tree where
5331 you defined and used tracepoints with ftrace's `TRACE_EVENT()` macro.
5333 Note that you can also use the
5334 <<lttng-tracepoint-event-code,`LTTNG_TRACEPOINT_EVENT_CODE()` macro>>
5335 instead of `LTTNG_TRACEPOINT_EVENT()` to use custom local variables and
5336 C code that need to be executed before the event fields are recorded.
5338 The best way to learn how to use the previous LTTng-modules macros is to
5339 inspect the existing LTTng-modules tracepoint definitions in the
5340 dir:{instrumentation/events/lttng-module} header files. Compare them
5341 with the Linux kernel mainline versions in the
5342 dir:{include/trace/events} directory of the Linux source tree.
5346 [[lttng-tracepoint-event-code]]
5347 ===== Use custom C code to access the data for tracepoint fields
5349 Although we recommended to always use the
5350 <<lttng-adaptation-layer,`LTTNG_TRACEPOINT_EVENT()`>> macro to describe
5351 the arguments and fields of an LTTng-modules tracepoint when possible,
5352 sometimes you need a more complex process to access the data that the
5353 tracer records as event record fields. In other words, you need local
5354 variables and multiple C{nbsp}statements instead of simple
5355 argument-based expressions that you pass to the
5356 <<lttng-modules-tp-fields,`ctf_*()` macros of `TP_FIELDS()`>>.
5358 You can use the `LTTNG_TRACEPOINT_EVENT_CODE()` macro instead of
5359 `LTTNG_TRACEPOINT_EVENT()` to declare custom local variables and define
5360 a block of C{nbsp}code to be executed before LTTng records the fields.
5361 The structure of this macro is:
5364 .`LTTNG_TRACEPOINT_EVENT_CODE()` macro syntax.
5366 LTTNG_TRACEPOINT_EVENT_CODE(
5368 * Format identical to the LTTNG_TRACEPOINT_EVENT()
5369 * version for the following three macro parameters:
5372 TP_PROTO(int my_int, const char *my_string),
5373 TP_ARGS(my_int, my_string),
5375 /* Declarations of custom local variables */
5378 unsigned long b = 0;
5379 const char *name = "(undefined)";
5380 struct my_struct *my_struct;
5384 * Custom code which uses both tracepoint arguments
5385 * (in TP_ARGS()) and local variables (in TP_locvar()).
5387 * Local variables are actually members of a structure pointed
5388 * to by the special variable tp_locvar.
5392 tp_locvar->a = my_int + 17;
5393 tp_locvar->my_struct = get_my_struct_at(tp_locvar->a);
5394 tp_locvar->b = my_struct_compute_b(tp_locvar->my_struct);
5395 tp_locvar->name = my_struct_get_name(tp_locvar->my_struct);
5396 put_my_struct(tp_locvar->my_struct);
5405 * Format identical to the LTTNG_TRACEPOINT_EVENT()
5406 * version for this, except that tp_locvar members can be
5407 * used in the argument expression parameters of
5408 * the ctf_*() macros.
5411 ctf_integer(unsigned long, my_struct_b, tp_locvar->b)
5412 ctf_integer(int, my_struct_a, tp_locvar->a)
5413 ctf_string(my_string_field, my_string)
5414 ctf_string(my_struct_name, tp_locvar->name)
5419 IMPORTANT: The C code defined in `TP_code()` must not have any side
5420 effects when executed. In particular, the code must not allocate
5421 memory or get resources without deallocating this memory or putting
5422 those resources afterwards.
5425 [[instrumenting-linux-kernel-tracing]]
5426 ==== Load and unload a custom probe kernel module
5428 You must load a <<lttng-adaptation-layer,created LTTng-modules probe
5429 kernel module>> in the kernel before it can emit LTTng events.
5431 To load the default probe kernel modules and a custom probe kernel
5434 * Use the opt:lttng-sessiond(8):--extra-kmod-probes option to give extra
5435 probe modules to load when starting a root <<lttng-sessiond,session
5439 .Load the `my_subsys`, `usb`, and the default probe modules.
5443 sudo lttng-sessiond --extra-kmod-probes=my_subsys,usb
5448 You only need to pass the subsystem name, not the whole kernel module
5451 To load _only_ a given custom probe kernel module:
5453 * Use the opt:lttng-sessiond(8):--kmod-probes option to give the probe
5454 modules to load when starting a root session daemon:
5457 .Load only the `my_subsys` and `usb` probe modules.
5461 sudo lttng-sessiond --kmod-probes=my_subsys,usb
5466 To confirm that a probe module is loaded:
5473 lsmod | grep lttng_probe_usb
5477 To unload the loaded probe modules:
5479 * Kill the session daemon with `SIGTERM`:
5484 sudo pkill lttng-sessiond
5488 You can also use man:modprobe(8)'s `--remove` option if the session
5489 daemon terminates abnormally.
5492 [[controlling-tracing]]
5495 Once an application or a Linux kernel is
5496 <<instrumenting,instrumented>> for LTTng tracing,
5499 This section is divided in topics on how to use the various
5500 <<plumbing,components of LTTng>>, in particular the <<lttng-cli,cmd:lttng
5501 command-line tool>>, to _control_ the LTTng daemons and tracers.
5503 NOTE: In the following subsections, we refer to an man:lttng(1) command
5504 using its man page name. For example, instead of _Run the `create`
5505 command to..._, we use _Run the man:lttng-create(1) command to..._.
5509 === Start a session daemon
5511 In some situations, you need to run a <<lttng-sessiond,session daemon>>
5512 (man:lttng-sessiond(8)) _before_ you can use the man:lttng(1)
5515 You will see the following error when you run a command while no session
5519 Error: No session daemon is available
5522 The only command that automatically runs a session daemon is
5523 man:lttng-create(1), which you use to
5524 <<creating-destroying-tracing-sessions,create a tracing session>>. While
5525 this is most of the time the first operation that you do, sometimes it's
5526 not. Some examples are:
5528 * <<list-instrumentation-points,List the available instrumentation points>>.
5529 * <<saving-loading-tracing-session,Load a tracing session configuration>>.
5531 [[tracing-group]] Each Unix user must have its own running session
5532 daemon to trace user applications. The session daemon that the root user
5533 starts is the only one allowed to control the LTTng kernel tracer. Users
5534 that are part of the _tracing group_ can control the root session
5535 daemon. The default tracing group name is `tracing`; you can set it to
5536 something else with the opt:lttng-sessiond(8):--group option when you
5537 start the root session daemon.
5539 To start a user session daemon:
5541 * Run man:lttng-sessiond(8):
5546 lttng-sessiond --daemonize
5550 To start the root session daemon:
5552 * Run man:lttng-sessiond(8) as the root user:
5557 sudo lttng-sessiond --daemonize
5561 In both cases, remove the opt:lttng-sessiond(8):--daemonize option to
5562 start the session daemon in foreground.
5564 To stop a session daemon, use man:kill(1) on its process ID (standard
5567 Note that some Linux distributions could manage the LTTng session daemon
5568 as a service. In this case, you should use the service manager to
5569 start, restart, and stop session daemons.
5572 [[creating-destroying-tracing-sessions]]
5573 === Create and destroy a tracing session
5575 Almost all the LTTng control operations happen in the scope of
5576 a <<tracing-session,tracing session>>, which is the dialogue between the
5577 <<lttng-sessiond,session daemon>> and you.
5579 To create a tracing session with a generated name:
5581 * Use the man:lttng-create(1) command:
5590 The created tracing session's name is `auto` followed by the
5593 To create a tracing session with a specific name:
5595 * Use the optional argument of the man:lttng-create(1) command:
5600 lttng create my-session
5604 Replace `my-session` with the specific tracing session name.
5606 LTTng appends the creation date to the created tracing session's name.
5608 LTTng writes the traces of a tracing session in
5609 +$LTTNG_HOME/lttng-trace/__name__+ by default, where +__name__+ is the
5610 name of the tracing session. Note that the env:LTTNG_HOME environment
5611 variable defaults to `$HOME` if not set.
5613 To output LTTng traces to a non-default location:
5615 * Use the opt:lttng-create(1):--output option of the man:lttng-create(1) command:
5620 lttng create --output=/tmp/some-directory my-session
5624 You may create as many tracing sessions as you wish.
5626 To list all the existing tracing sessions for your Unix user:
5628 * Use the man:lttng-list(1) command:
5637 When you create a tracing session, it is set as the _current tracing
5638 session_. The following man:lttng(1) commands operate on the current
5639 tracing session when you don't specify one:
5641 [role="list-3-cols"]
5657 To change the current tracing session:
5659 * Use the man:lttng-set-session(1) command:
5664 lttng set-session new-session
5668 Replace `new-session` by the name of the new current tracing session.
5670 When you are done tracing in a given tracing session, you can destroy
5671 it. This operation frees the resources taken by the tracing session
5672 to destroy; it does not destroy the trace data that LTTng wrote for
5673 this tracing session.
5675 To destroy the current tracing session:
5677 * Use the man:lttng-destroy(1) command:
5687 [[list-instrumentation-points]]
5688 === List the available instrumentation points
5690 The <<lttng-sessiond,session daemon>> can query the running instrumented
5691 user applications and the Linux kernel to get a list of available
5692 instrumentation points. For the Linux kernel <<domain,tracing domain>>,
5693 they are tracepoints and system calls. For the user space tracing
5694 domain, they are tracepoints. For the other tracing domains, they are
5697 To list the available instrumentation points:
5699 * Use the man:lttng-list(1) command with the requested tracing domain's
5703 * opt:lttng-list(1):--kernel: Linux kernel tracepoints (your Unix user
5704 must be a root user, or it must be a member of the
5705 <<tracing-group,tracing group>>).
5706 * opt:lttng-list(1):--kernel with opt:lttng-list(1):--syscall: Linux
5707 kernel system calls (your Unix user must be a root user, or it must be
5708 a member of the tracing group).
5709 * opt:lttng-list(1):--userspace: user space tracepoints.
5710 * opt:lttng-list(1):--jul: `java.util.logging` loggers.
5711 * opt:lttng-list(1):--log4j: Apache log4j loggers.
5712 * opt:lttng-list(1):--python: Python loggers.
5715 .List the available user space tracepoints.
5719 lttng list --userspace
5723 .List the available Linux kernel system call tracepoints.
5727 lttng list --kernel --syscall
5732 [[enabling-disabling-events]]
5733 === Create and enable an event rule
5735 Once you <<creating-destroying-tracing-sessions,create a tracing
5736 session>>, you can create <<event,event rules>> with the
5737 man:lttng-enable-event(1) command.
5739 You specify each condition with a command-line option. The available
5740 condition options are shown in the following table.
5742 [role="growable",cols="asciidoc,asciidoc,default"]
5743 .Condition command-line options for the man:lttng-enable-event(1) command.
5745 |Option |Description |Applicable tracing domains
5751 . +--probe=__ADDR__+
5752 . +--function=__ADDR__+
5755 Instead of using the default _tracepoint_ instrumentation type, use:
5757 . A Linux system call.
5758 . A Linux https://lwn.net/Articles/132196/[KProbe] (symbol or address).
5759 . The entry and return points of a Linux function (symbol or address).
5763 |First positional argument.
5766 Tracepoint or system call name. In the case of a Linux KProbe or
5767 function, this is a custom name given to the event rule. With the
5768 JUL, log4j, and Python domains, this is a logger name.
5770 With a tracepoint, logger, or system call name, the last character
5771 can be `*` to match anything that remains.
5778 . +--loglevel=__LEVEL__+
5779 . +--loglevel-only=__LEVEL__+
5782 . Match only tracepoints or log statements with a logging level at
5783 least as severe as +__LEVEL__+.
5784 . Match only tracepoints or log statements with a logging level
5785 equal to +__LEVEL__+.
5787 See man:lttng-enable-event(1) for the list of available logging level
5790 |User space, JUL, log4j, and Python.
5792 |+--exclude=__EXCLUSIONS__+
5795 When you use a `*` character at the end of the tracepoint or logger
5796 name (first positional argument), exclude the specific names in the
5797 comma-delimited list +__EXCLUSIONS__+.
5800 User space, JUL, log4j, and Python.
5802 |+--filter=__EXPR__+
5805 Match only events which satisfy the expression +__EXPR__+.
5807 See man:lttng-enable-event(1) to learn more about the syntax of a
5814 You attach an event rule to a <<channel,channel>> on creation. If you do
5815 not specify the channel with the opt:lttng-enable-event(1):--channel
5816 option, and if the event rule to create is the first in its
5817 <<domain,tracing domain>> for a given tracing session, then LTTng
5818 creates a _default channel_ for you. This default channel is reused in
5819 subsequent invocations of the man:lttng-enable-event(1) command for the
5820 same tracing domain.
5822 An event rule is always enabled at creation time.
5824 The following examples show how you can combine the previous
5825 command-line options to create simple to more complex event rules.
5827 .Create an event rule targetting a Linux kernel tracepoint (default channel).
5831 lttng enable-event --kernel sched_switch
5835 .Create an event rule matching four Linux kernel system calls (default channel).
5839 lttng enable-event --kernel --syscall open,write,read,close
5843 .Create event rules matching tracepoints with filter expressions (default channel).
5847 lttng enable-event --kernel sched_switch --filter='prev_comm == "bash"'
5852 lttng enable-event --kernel --all \
5853 --filter='$ctx.tid == 1988 || $ctx.tid == 1534'
5858 lttng enable-event --jul my_logger \
5859 --filter='$app.retriever:cur_msg_id > 3'
5862 IMPORTANT: Make sure to always quote the filter string when you
5863 use man:lttng(1) from a shell.
5866 .Create an event rule matching any user space tracepoint of a given tracepoint provider with a log level range (default channel).
5870 lttng enable-event --userspace my_app:'*' --loglevel=TRACE_INFO
5873 IMPORTANT: Make sure to always quote the wildcard character when you
5874 use man:lttng(1) from a shell.
5877 .Create an event rule matching multiple Python loggers with a wildcard and with exclusions (default channel).
5881 lttng enable-event --python my-app.'*' \
5882 --exclude='my-app.module,my-app.hello'
5886 .Create an event rule matching any Apache log4j logger with a specific log level (default channel).
5890 lttng enable-event --log4j --all --loglevel-only=LOG4J_WARN
5894 .Create an event rule attached to a specific channel matching a specific user space tracepoint provider and tracepoint.
5898 lttng enable-event --userspace my_app:my_tracepoint --channel=my-channel
5902 The event rules of a given channel form a whitelist: as soon as an
5903 emitted event passes one of them, LTTng can record the event. For
5904 example, an event named `my_app:my_tracepoint` emitted from a user space
5905 tracepoint with a `TRACE_ERROR` log level passes both of the following
5910 lttng enable-event --userspace my_app:my_tracepoint
5911 lttng enable-event --userspace my_app:my_tracepoint \
5912 --loglevel=TRACE_INFO
5915 The second event rule is redundant: the first one includes
5919 [[disable-event-rule]]
5920 === Disable an event rule
5922 To disable an event rule that you <<enabling-disabling-events,created>>
5923 previously, use the man:lttng-disable-event(1) command. This command
5924 disables _all_ the event rules (of a given tracing domain and channel)
5925 which match an instrumentation point. The other conditions are not
5926 supported as of LTTng{nbsp}{revision}.
5928 The LTTng tracer does not record an emitted event which passes
5929 a _disabled_ event rule.
5931 .Disable an event rule matching a Python logger (default channel).
5935 lttng disable-event --python my-logger
5939 .Disable an event rule matching all `java.util.logging` loggers (default channel).
5943 lttng disable-event --jul '*'
5947 .Disable _all_ the event rules of the default channel.
5949 The opt:lttng-disable-event(1):--all-events option is not, like the
5950 opt:lttng-enable-event(1):--all option of man:lttng-enable-event(1), the
5951 equivalent of the event name `*` (wildcard): it disables _all_ the event
5952 rules of a given channel.
5956 lttng disable-event --jul --all-events
5960 NOTE: You cannot delete an event rule once you create it.
5964 === Get the status of a tracing session
5966 To get the status of the current tracing session, that is, its
5967 parameters, its channels, event rules, and their attributes:
5969 * Use the man:lttng-status(1) command:
5979 To get the status of any tracing session:
5981 * Use the man:lttng-list(1) command with the tracing session's name:
5986 lttng list my-session
5990 Replace `my-session` with the desired tracing session's name.
5993 [[basic-tracing-session-control]]
5994 === Start and stop a tracing session
5996 Once you <<creating-destroying-tracing-sessions,create a tracing
5998 <<enabling-disabling-events,create one or more event rules>>,
5999 you can start and stop the tracers for this tracing session.
6001 To start tracing in the current tracing session:
6003 * Use the man:lttng-start(1) command:
6012 LTTng is very flexible: you can launch user applications before
6013 or after the you start the tracers. The tracers only record the events
6014 if they pass enabled event rules and if they occur while the tracers are
6017 To stop tracing in the current tracing session:
6019 * Use the man:lttng-stop(1) command:
6028 If there were <<channel-overwrite-mode-vs-discard-mode,lost event
6029 records>> or lost sub-buffers since the last time you ran
6030 man:lttng-start(1), warnings are printed when you run the
6031 man:lttng-stop(1) command.
6034 [[enabling-disabling-channels]]
6035 === Create a channel
6037 Once you create a tracing session, you can create a <<channel,channel>>
6038 with the man:lttng-enable-channel(1) command.
6040 Note that LTTng automatically creates a default channel when, for a
6041 given <<domain,tracing domain>>, no channels exist and you
6042 <<enabling-disabling-events,create>> the first event rule. This default
6043 channel is named `channel0` and its attributes are set to reasonable
6044 values. Therefore, you only need to create a channel when you need
6045 non-default attributes.
6047 You specify each non-default channel attribute with a command-line
6048 option when you use the man:lttng-enable-channel(1) command. The
6049 available command-line options are:
6051 [role="growable",cols="asciidoc,asciidoc"]
6052 .Command-line options for the man:lttng-enable-channel(1) command.
6054 |Option |Description
6060 <<channel-overwrite-mode-vs-discard-mode,event loss mode>> instead of
6061 the default _discard_ mode.
6063 |`--buffers-pid` (user space tracing domain only)
6066 Use the per-process <<channel-buffering-schemes,buffering scheme>>
6067 instead of the default per-user buffering scheme.
6069 |+--subbuf-size=__SIZE__+
6072 Allocate sub-buffers of +__SIZE__+ bytes (power of two), for each CPU,
6073 either for each Unix user (default), or for each instrumented process.
6075 See <<channel-subbuf-size-vs-subbuf-count,Sub-buffer count and size>>.
6077 |+--num-subbuf=__COUNT__+
6080 Allocate +__COUNT__+ sub-buffers (power of two), for each CPU, either
6081 for each Unix user (default), or for each instrumented process.
6083 See <<channel-subbuf-size-vs-subbuf-count,Sub-buffer count and size>>.
6085 |+--tracefile-size=__SIZE__+
6088 Set the maximum size of each trace file that this channel writes within
6089 a stream to +__SIZE__+ bytes instead of no maximum.
6091 See <<tracefile-rotation,Trace file count and size>>.
6093 |+--tracefile-count=__COUNT__+
6096 Limit the number of trace files that this channel creates to
6097 +__COUNT__+ channels instead of no limit.
6099 See <<tracefile-rotation,Trace file count and size>>.
6101 |+--switch-timer=__PERIODUS__+
6104 Set the <<channel-switch-timer,switch timer period>>
6105 to +__PERIODUS__+{nbsp}µs.
6107 |+--read-timer=__PERIODUS__+
6110 Set the <<channel-read-timer,read timer period>>
6111 to +__PERIODUS__+{nbsp}µs.
6113 |+--output=__TYPE__+ (Linux kernel tracing domain only)
6116 Set the channel's output type to +__TYPE__+, either `mmap` or `splice`.
6120 You can only create a channel in the Linux kernel and user space
6121 <<domain,tracing domains>>: other tracing domains have their own channel
6122 created on the fly when <<enabling-disabling-events,creating event
6127 Because of a current LTTng limitation, you must create all channels
6128 _before_ you <<basic-tracing-session-control,start tracing>> in a given
6129 tracing session, that is, before the first time you run
6132 Since LTTng automatically creates a default channel when you use the
6133 man:lttng-enable-event(1) command with a specific tracing domain, you
6134 cannot, for example, create a Linux kernel event rule, start tracing,
6135 and then create a user space event rule, because no user space channel
6136 exists yet and it's too late to create one.
6138 For this reason, make sure to configure your channels properly
6139 before starting the tracers for the first time!
6142 The following examples show how you can combine the previous
6143 command-line options to create simple to more complex channels.
6145 .Create a Linux kernel channel with default attributes.
6149 lttng enable-channel --kernel my-channel
6153 .Create a user space channel with 4 sub-buffers or 1{nbsp}MiB each, per CPU, per instrumented process.
6157 lttng enable-channel --userspace --num-subbuf=4 --subbuf-size=1M \
6158 --buffers-pid my-channel
6162 .Create a Linux kernel channel which rotates 8 trace files of 4{nbsp}MiB each for each stream
6166 lttng enable-channel --kernel --tracefile-count=8 \
6167 --tracefile-size=4194304 my-channel
6171 .Create a user space channel in overwrite (or _flight recorder_) mode.
6175 lttng enable-channel --userspace --overwrite my-channel
6179 You can <<enabling-disabling-events,create>> the same event rule in
6180 two different channels:
6184 lttng enable-event --userspace --channel=my-channel app:tp
6185 lttng enable-event --userspace --channel=other-channel app:tp
6188 If both channels are enabled, when a tracepoint named `app:tp` is
6189 reached, LTTng records two events, one for each channel.
6193 === Disable a channel
6195 To disable a specific channel that you <<enabling-disabling-channels,created>>
6196 previously, use the man:lttng-disable-channel(1) command.
6198 .Disable a specific Linux kernel channel.
6202 lttng disable-channel --kernel my-channel
6206 The state of a channel precedes the individual states of event rules
6207 attached to it: event rules which belong to a disabled channel, even if
6208 they are enabled, are also considered disabled.
6212 === Add context fields to a channel
6214 Event record fields in trace files provide important information about
6215 events that occured previously, but sometimes some external context may
6216 help you solve a problem faster. Examples of context fields are:
6218 * The **process ID**, **thread ID**, **process name**, and
6219 **process priority** of the thread in which the event occurs.
6220 * The **hostname** of the system on which the event occurs.
6221 * The current values of many possible **performance counters** using
6223 ** CPU cycles, stalled cycles, idle cycles, and the other cycle types.
6225 ** Branch instructions, misses, and loads.
6227 * Any context defined at the application level (supported for the
6228 JUL and log4j <<domain,tracing domains>>).
6230 To get the full list of available context fields, see
6231 `lttng add-context --list`. Some context fields are reserved for a
6232 specific <<domain,tracing domain>> (Linux kernel or user space).
6234 You add context fields to <<channel,channels>>. All the events
6235 that a channel with added context fields records contain those fields.
6237 To add context fields to one or all the channels of a given tracing
6240 * Use the man:lttng-add-context(1) command.
6242 .Add context fields to all the channels of the current tracing session.
6244 The following command line adds the virtual process identifier and
6245 the per-thread CPU cycles count fields to all the user space channels
6246 of the current tracing session.
6250 lttng add-context --userspace --type=vpid --type=perf:thread:cpu-cycles
6254 .Add a context field to a specific channel.
6256 The following command line adds the thread identifier context field
6257 to the Linux kernel channel named `my-channel` in the current
6262 lttng add-context --kernel --channel=my-channel --type=tid
6266 .Add an application-specific context field to a specific channel.
6268 The following command line adds the `cur_msg_id` context field of the
6269 `retriever` context retriever for all the instrumented
6270 <<java-application,Java applications>> recording <<event,event records>>
6271 in the channel named `my-channel`:
6275 lttng add-context --kernel --channel=my-channel \
6276 --type='$app:retriever:cur_msg_id'
6279 IMPORTANT: Make sure to always quote the `$` character when you
6280 use man:lttng-add-context(1) from a shell.
6283 NOTE: You cannot remove context fields from a channel once you add it.
6288 === Track process IDs
6290 It's often useful to allow only specific process IDs (PIDs) to emit
6291 events. For example, you may wish to record all the system calls made by
6292 a given process (à la http://linux.die.net/man/1/strace[strace]).
6294 The man:lttng-track(1) and man:lttng-untrack(1) commands serve this
6295 purpose. Both commands operate on a whitelist of process IDs. You _add_
6296 entries to this whitelist with the man:lttng-track(1) command and remove
6297 entries with the man:lttng-untrack(1) command. Any process which has one
6298 of the PIDs in the whitelist is allowed to emit LTTng events which pass
6299 an enabled <<event,event rule>>.
6301 NOTE: The PID tracker tracks the _numeric process IDs_. Should a
6302 process with a given tracked ID exit and another process be given this
6303 ID, then the latter would also be allowed to emit events.
6305 .Track and untrack process IDs.
6307 For the sake of the following example, assume the target system has 16
6311 <<creating-destroying-tracing-sessions,create a tracing session>>,
6312 the whitelist contains all the possible PIDs:
6315 .All PIDs are tracked.
6316 image::track-all.png[]
6318 When the whitelist is full and you use the man:lttng-track(1) command to
6319 specify some PIDs to track, LTTng first clears the whitelist, then it
6320 tracks the specific PIDs. After:
6324 lttng track --pid=3,4,7,10,13
6330 .PIDs 3, 4, 7, 10, and 13 are tracked.
6331 image::track-3-4-7-10-13.png[]
6333 You can add more PIDs to the whitelist afterwards:
6337 lttng track --pid=1,15,16
6343 .PIDs 1, 15, and 16 are added to the whitelist.
6344 image::track-1-3-4-7-10-13-15-16.png[]
6346 The man:lttng-untrack(1) command removes entries from the PID tracker's
6347 whitelist. Given the previous example, the following command:
6351 lttng untrack --pid=3,7,10,13
6354 leads to this whitelist:
6357 .PIDs 3, 7, 10, and 13 are removed from the whitelist.
6358 image::track-1-4-15-16.png[]
6360 LTTng can track all possible PIDs again using the opt:track(1):--all
6365 lttng track --pid --all
6368 The result is, again:
6371 .All PIDs are tracked.
6372 image::track-all.png[]
6375 .Track only specific PIDs
6377 A very typical use case with PID tracking is to start with an empty
6378 whitelist, then <<basic-tracing-session-control,start the tracers>>, and
6379 then add PIDs manually while tracers are active. You can accomplish this
6380 by using the opt:lttng-untrack(1):--all option of the
6381 man:lttng-untrack(1) command to clear the whitelist after you
6382 <<creating-destroying-tracing-sessions,create a tracing session>>:
6386 lttng untrack --pid --all
6392 .No PIDs are tracked.
6393 image::untrack-all.png[]
6395 If you trace with this whitelist configuration, the tracer records no
6396 events for this <<domain,tracing domain>> because no processes are
6397 tracked. You can use the man:lttng-track(1) command as usual to track
6398 specific PIDs, for example:
6402 lttng track --pid=6,11
6408 .PIDs 6 and 11 are tracked.
6409 image::track-6-11.png[]
6414 [[saving-loading-tracing-session]]
6415 === Save and load tracing session configurations
6417 Configuring a <<tracing-session,tracing session>> can be long. Some of
6418 the tasks involved are:
6420 * <<enabling-disabling-channels,Create channels>> with
6421 specific attributes.
6422 * <<adding-context,Add context fields>> to specific channels.
6423 * <<enabling-disabling-events,Create event rules>> with specific log
6424 level and filter conditions.
6426 If you use LTTng to solve real world problems, chances are you have to
6427 record events using the same tracing session setup over and over,
6428 modifying a few variables each time in your instrumented program
6429 or environment. To avoid constant tracing session reconfiguration,
6430 the man:lttng(1) command-line tool can save and load tracing session
6431 configurations to/from XML files.
6433 To save a given tracing session configuration:
6435 * Use the man:lttng-save(1) command:
6440 lttng save my-session
6444 Replace `my-session` with the name of the tracing session to save.
6446 LTTng saves tracing session configurations to
6447 dir:{$LTTNG_HOME/.lttng/sessions} by default. Note that the
6448 env:LTTNG_HOME environment variable defaults to `$HOME` if not set. Use
6449 the opt:lttng-save(1):--output-path option to change this destination
6452 LTTng saves all configuration parameters, for example:
6454 * The tracing session name.
6455 * The trace data output path.
6456 * The channels with their state and all their attributes.
6457 * The context fields you added to channels.
6458 * The event rules with their state, log level and filter conditions.
6460 To load a tracing session:
6462 * Use the man:lttng-load(1) command:
6467 lttng load my-session
6471 Replace `my-session` with the name of the tracing session to load.
6473 When LTTng loads a configuration, it restores your saved tracing session
6474 as if you just configured it manually.
6476 See man:lttng(1) for the complete list of command-line options. You
6477 can also save and load all many sessions at a time, and decide in which
6478 directory to output the XML files.
6481 [[sending-trace-data-over-the-network]]
6482 === Send trace data over the network
6484 LTTng can send the recorded trace data to a remote system over the
6485 network instead of writing it to the local file system.
6487 To send the trace data over the network:
6489 . On the _remote_ system (which can also be the target system),
6490 start an LTTng <<lttng-relayd,relay daemon>> (man:lttng-relayd(8)):
6499 . On the _target_ system, create a tracing session configured to
6500 send trace data over the network:
6505 lttng create my-session --set-url=net://remote-system
6509 Replace `remote-system` by the host name or IP address of the
6510 remote system. See man:lttng-create(1) for the exact URL format.
6512 . On the target system, use the man:lttng(1) command-line tool as usual.
6513 When tracing is active, the target's consumer daemon sends sub-buffers
6514 to the relay daemon running on the remote system intead of flushing
6515 them to the local file system. The relay daemon writes the received
6516 packets to the local file system.
6518 The relay daemon writes trace files to
6519 +$LTTNG_HOME/lttng-traces/__hostname__/__session__+ by default, where
6520 +__hostname__+ is the host name of the target system and +__session__+
6521 is the tracing session name. Note that the env:LTTNG_HOME environment
6522 variable defaults to `$HOME` if not set. Use the
6523 opt:lttng-relayd(8):--output option of man:lttng-relayd(8) to write
6524 trace files to another base directory.
6529 === View events as LTTng emits them (noch:{LTTng} live)
6531 LTTng live is a network protocol implemented by the <<lttng-relayd,relay
6532 daemon>> (man:lttng-relayd(8)) to allow compatible trace viewers to
6533 display events as LTTng emits them on the target system while tracing is
6536 The relay daemon creates a _tee_: it forwards the trace data to both
6537 the local file system and to connected live viewers:
6540 .The relay daemon creates a _tee_, forwarding the trace data to both trace files and a connected live viewer.
6545 . On the _target system_, create a <<tracing-session,tracing session>>
6551 lttng create --live my-session
6555 This spawns a local relay daemon.
6557 . Start the live viewer and configure it to connect to the relay
6558 daemon. For example, with http://diamon.org/babeltrace[Babeltrace]:
6563 babeltrace --input-format=lttng-live net://localhost/host/hostname/my-session
6570 * `hostname` with the host name of the target system.
6571 * `my-session` with the name of the tracing session to view.
6574 . Configure the tracing session as usual with the man:lttng(1)
6575 command-line tool, and <<basic-tracing-session-control,start tracing>>.
6577 You can list the available live tracing sessions with Babeltrace:
6581 babeltrace --input-format=lttng-live net://localhost
6584 You can start the relay daemon on another system. In this case, you need
6585 to specify the relay daemon's URL when you create the tracing session
6586 with the opt:lttng-create(1):--set-url option. You also need to replace
6587 `localhost` in the procedure above with the host name of the system on
6588 which the relay daemon is running.
6590 See man:lttng-create(1) and man:lttng-relayd(8) for the complete list of
6591 command-line options.
6595 [[taking-a-snapshot]]
6596 === Take a snapshot of the current sub-buffers of a tracing session
6598 The normal behavior of LTTng is to append full sub-buffers to growing
6599 trace data files. This is ideal to keep a full history of the events
6600 that occurred on the target system, but it can
6601 represent too much data in some situations. For example, you may wish
6602 to trace your application continuously until some critical situation
6603 happens, in which case you only need the latest few recorded
6604 events to perform the desired analysis, not multi-gigabyte trace files.
6606 With the man:lttng-snapshot(1) command, you can take a snapshot of the
6607 current sub-buffers of a given <<tracing-session,tracing session>>.
6608 LTTng can write the snapshot to the local file system or send it over
6613 . Create a tracing session in _snapshot mode_:
6618 lttng create --snapshot my-session
6622 The <<channel-overwrite-mode-vs-discard-mode,event loss mode>> of
6623 <<channel,channels>> created in this mode is automatically set to
6624 _overwrite_ (flight recorder mode).
6626 . Configure the tracing session as usual with the man:lttng(1)
6627 command-line tool, and <<basic-tracing-session-control,start tracing>>.
6629 . **Optional**: When you need to take a snapshot,
6630 <<basic-tracing-session-control,stop tracing>>.
6632 You can take a snapshot when the tracers are active, but if you stop
6633 them first, you are sure that the data in the sub-buffers does not
6634 change before you actually take the snapshot.
6641 lttng snapshot record --name=my-first-snapshot
6645 LTTng writes the current sub-buffers of all the current tracing
6646 session's channels to trace files on the local file system. Those trace
6647 files have `my-first-snapshot` in their name.
6649 There is no difference between the format of a normal trace file and the
6650 format of a snapshot: viewers of LTTng traces also support LTTng
6653 By default, LTTng writes snapshot files to the path shown by
6654 `lttng snapshot list-output`. You can change this path or decide to send
6655 snapshots over the network using either:
6657 . An output path or URL that you specify when you create the
6659 . An snapshot output path or URL that you add using
6660 `lttng snapshot add-output`
6661 . An output path or URL that you provide directly to the
6662 `lttng snapshot record` command.
6664 Method 3 overrides method 2, which overrides method 1. When you
6665 specify a URL, a relay daemon must listen on a remote system (see
6666 <<sending-trace-data-over-the-network,Send trace data over the network>>).
6671 === Use the machine interface
6673 With any command of the man:lttng(1) command-line tool, you can set the
6674 opt:lttng(1):--mi option to `xml` (before the command name) to get an
6675 XML machine interface output, for example:
6679 lttng --mi=xml enable-event --kernel --syscall open
6682 A schema definition (XSD) is
6683 https://github.com/lttng/lttng-tools/blob/stable-2.8/src/common/mi-lttng-3.0.xsd[available]
6684 to ease the integration with external tools as much as possible.
6688 [[metadata-regenerate]]
6689 === Regenerate the metadata of an LTTng trace
6691 An LTTng trace, which is a http://diamon.org/ctf[CTF] trace, has both
6692 data stream files and a metadata file. This metadata file contains,
6693 amongst other things, information about the offset of the clock sources
6694 used to timestamp <<event,event records>> when tracing.
6696 If, once a <<tracing-session,tracing session>> is
6697 <<basic-tracing-session-control,started>>, a major
6698 https://en.wikipedia.org/wiki/Network_Time_Protocol[NTP] correction
6699 happens, the trace's clock offset also needs to be updated. You
6700 can use the man:lttng-metadata(1) command to do so.
6702 The main use case of this command is to allow a system to boot with
6703 an incorrect wall time and trace it with LTTng before its wall time
6704 is corrected. Once the system is known to be in a state where its
6705 wall time is correct, it can run `lttng metadata regenerate`.
6707 To regenerate the metadata of an LTTng trace:
6709 * Use the `regenerate` action of the man:lttng-metadata(1) command:
6714 lttng metadata regenerate
6720 `lttng metadata regenerate` has the following limitations:
6722 * Tracing session <<creating-destroying-tracing-sessions,created>>
6724 * User space <<channel,channels>>, if any, using
6725 <<channel-buffering-schemes,per-user buffering>>.
6730 [[persistent-memory-file-systems]]
6731 === Record trace data on persistent memory file systems
6733 https://en.wikipedia.org/wiki/Non-volatile_random-access_memory[Non-volatile random-access memory]
6734 (NVRAM) is random-access memory that retains its information when power
6735 is turned off (non-volatile). Systems with such memory can store data
6736 structures in RAM and retrieve them after a reboot, without flushing
6737 to typical _storage_.
6739 Linux supports NVRAM file systems thanks to either
6740 http://pramfs.sourceforge.net/[PRAMFS] or
6741 https://www.kernel.org/doc/Documentation/filesystems/dax.txt[DAX]{nbsp}+{nbsp}http://lkml.iu.edu/hypermail/linux/kernel/1504.1/03463.html[pmem]
6742 (requires Linux 4.1+).
6744 This section does not describe how to operate such file systems;
6745 we assume that you have a working persistent memory file system.
6747 When you create a <<tracing-session,tracing session>>, you can specify
6748 the path of the shared memory holding the sub-buffers. If you specify a
6749 location on an NVRAM file system, then you can retrieve the latest
6750 recorded trace data when the system reboots after a crash.
6752 To record trace data on a persistent memory file system and retrieve the
6753 trace data after a system crash:
6755 . Create a tracing session with a sub-buffer shared memory path located
6756 on an NVRAM file system:
6761 lttng create --shm-path=/path/to/shm
6765 . Configure the tracing session as usual with the man:lttng(1)
6766 command-line tool, and <<basic-tracing-session-control,start tracing>>.
6768 . After a system crash, use the man:lttng-crash(1) command-line tool to
6769 view the trace data recorded on the NVRAM file system:
6774 lttng-crash /path/to/shm
6778 The binary layout of the ring buffer files is not exactly the same as
6779 the trace files layout. This is why you need to use man:lttng-crash(1)
6780 instead of your preferred trace viewer directly.
6782 To convert the ring buffer files to LTTng trace files:
6784 * Use the opt:lttng-crash(1):--extract option of man:lttng-crash(1):
6789 lttng-crash --extract=/path/to/trace /path/to/shm
6797 [[lttng-modules-ref]]
6798 === noch:{LTTng-modules}
6801 [[lttng-modules-tp-fields]]
6802 ==== Tracepoint fields macros (for `TP_FIELDS()`)
6804 [[tp-fast-assign]][[tp-struct-entry]]The available macros to define
6805 tracepoint fields, which must be listed within `TP_FIELDS()` in
6806 `LTTNG_TRACEPOINT_EVENT()`, are:
6808 [role="func-desc growable",cols="asciidoc,asciidoc"]
6809 .Available macros to define LTTng-modules tracepoint fields
6811 |Macro |Description and parameters
6814 +ctf_integer(__t__, __n__, __e__)+
6816 +ctf_integer_nowrite(__t__, __n__, __e__)+
6818 +ctf_user_integer(__t__, __n__, __e__)+
6820 +ctf_user_integer_nowrite(__t__, __n__, __e__)+
6822 Standard integer, displayed in base 10.
6825 Integer C type (`int`, `long`, `size_t`, ...).
6831 Argument expression.
6834 +ctf_integer_hex(__t__, __n__, __e__)+
6836 +ctf_user_integer_hex(__t__, __n__, __e__)+
6838 Standard integer, displayed in base 16.
6847 Argument expression.
6849 |+ctf_integer_oct(__t__, __n__, __e__)+
6851 Standard integer, displayed in base 8.
6860 Argument expression.
6863 +ctf_integer_network(__t__, __n__, __e__)+
6865 +ctf_user_integer_network(__t__, __n__, __e__)+
6867 Integer in network byte order (big-endian), displayed in base 10.
6876 Argument expression.
6879 +ctf_integer_network_hex(__t__, __n__, __e__)+
6881 +ctf_user_integer_network_hex(__t__, __n__, __e__)+
6883 Integer in network byte order, displayed in base 16.
6892 Argument expression.
6895 +ctf_string(__n__, __e__)+
6897 +ctf_string_nowrite(__n__, __e__)+
6899 +ctf_user_string(__n__, __e__)+
6901 +ctf_user_string_nowrite(__n__, __e__)+
6903 Null-terminated string; undefined behavior if +__e__+ is `NULL`.
6909 Argument expression.
6912 +ctf_array(__t__, __n__, __e__, __s__)+
6914 +ctf_array_nowrite(__t__, __n__, __e__, __s__)+
6916 +ctf_user_array(__t__, __n__, __e__, __s__)+
6918 +ctf_user_array_nowrite(__t__, __n__, __e__, __s__)+
6920 Statically-sized array of integers.
6923 Array element C type.
6929 Argument expression.
6935 +ctf_array_bitfield(__t__, __n__, __e__, __s__)+
6937 +ctf_array_bitfield_nowrite(__t__, __n__, __e__, __s__)+
6939 +ctf_user_array_bitfield(__t__, __n__, __e__, __s__)+
6941 +ctf_user_array_bitfield_nowrite(__t__, __n__, __e__, __s__)+
6943 Statically-sized array of bits.
6945 The type of +__e__+ must be an integer type. +__s__+ is the number
6946 of elements of such type in +__e__+, not the number of bits.
6949 Array element C type.
6955 Argument expression.
6961 +ctf_array_text(__t__, __n__, __e__, __s__)+
6963 +ctf_array_text_nowrite(__t__, __n__, __e__, __s__)+
6965 +ctf_user_array_text(__t__, __n__, __e__, __s__)+
6967 +ctf_user_array_text_nowrite(__t__, __n__, __e__, __s__)+
6969 Statically-sized array, printed as text.
6971 The string does not need to be null-terminated.
6974 Array element C type (always `char`).
6980 Argument expression.
6986 +ctf_sequence(__t__, __n__, __e__, __T__, __E__)+
6988 +ctf_sequence_nowrite(__t__, __n__, __e__, __T__, __E__)+
6990 +ctf_user_sequence(__t__, __n__, __e__, __T__, __E__)+
6992 +ctf_user_sequence_nowrite(__t__, __n__, __e__, __T__, __E__)+
6994 Dynamically-sized array of integers.
6996 The type of +__E__+ must be unsigned.
6999 Array element C type.
7005 Argument expression.
7008 Length expression C type.
7014 +ctf_sequence_hex(__t__, __n__, __e__, __T__, __E__)+
7016 +ctf_user_sequence_hex(__t__, __n__, __e__, __T__, __E__)+
7018 Dynamically-sized array of integers, displayed in base 16.
7020 The type of +__E__+ must be unsigned.
7023 Array element C type.
7029 Argument expression.
7032 Length expression C type.
7037 |+ctf_sequence_network(__t__, __n__, __e__, __T__, __E__)+
7039 Dynamically-sized array of integers in network byte order (big-endian),
7040 displayed in base 10.
7042 The type of +__E__+ must be unsigned.
7045 Array element C type.
7051 Argument expression.
7054 Length expression C type.
7060 +ctf_sequence_bitfield(__t__, __n__, __e__, __T__, __E__)+
7062 +ctf_sequence_bitfield_nowrite(__t__, __n__, __e__, __T__, __E__)+
7064 +ctf_user_sequence_bitfield(__t__, __n__, __e__, __T__, __E__)+
7066 +ctf_user_sequence_bitfield_nowrite(__t__, __n__, __e__, __T__, __E__)+
7068 Dynamically-sized array of bits.
7070 The type of +__e__+ must be an integer type. +__s__+ is the number
7071 of elements of such type in +__e__+, not the number of bits.
7073 The type of +__E__+ must be unsigned.
7076 Array element C type.
7082 Argument expression.
7085 Length expression C type.
7091 +ctf_sequence_text(__t__, __n__, __e__, __T__, __E__)+
7093 +ctf_sequence_text_nowrite(__t__, __n__, __e__, __T__, __E__)+
7095 +ctf_user_sequence_text(__t__, __n__, __e__, __T__, __E__)+
7097 +ctf_user_sequence_text_nowrite(__t__, __n__, __e__, __T__, __E__)+
7099 Dynamically-sized array, displayed as text.
7101 The string does not need to be null-terminated.
7103 The type of +__E__+ must be unsigned.
7105 The behaviour is undefined if +__e__+ is `NULL`.
7108 Sequence element C type (always `char`).
7114 Argument expression.
7117 Length expression C type.
7123 Use the `_user` versions when the argument expression, `e`, is
7124 a user space address. In the cases of `ctf_user_integer*()` and
7125 `ctf_user_float*()`, `&e` must be a user space address, thus `e` must
7128 The `_nowrite` versions omit themselves from the session trace, but are
7129 otherwise identical. This means the `_nowrite` fields won't be written
7130 in the recorded trace. Their primary purpose is to make some
7131 of the event context available to the
7132 <<enabling-disabling-events,event filters>> without having to
7133 commit the data to sub-buffers.
7139 Terms related to LTTng and to tracing in general:
7142 The http://diamon.org/babeltrace[Babeltrace] project, which includes
7143 the cmd:babeltrace command, some libraries, and Python bindings.
7145 <<channel-buffering-schemes,buffering scheme>>::
7146 A layout of sub-buffers applied to a given channel.
7148 <<channel,channel>>::
7149 An entity which is responsible for a set of ring buffers.
7151 <<event,Event rules>> are always attached to a specific channel.
7154 A reference of time for a tracer.
7156 <<lttng-consumerd,consumer daemon>>::
7157 A process which is responsible for consuming the full sub-buffers
7158 and write them to a file system or send them over the network.
7160 <<channel-overwrite-mode-vs-discard-mode,discard mode>>:: The event loss
7161 mode in which the tracer _discards_ new event records when there's no
7162 sub-buffer space left to store them.
7165 The consequence of the execution of an instrumentation
7166 point, like a tracepoint that you manually place in some source code,
7167 or a Linux kernel KProbe.
7169 An event is said to _occur_ at a specific time. Different actions can
7170 be taken upon the occurance of an event, like record the event's payload
7173 <<channel-overwrite-mode-vs-discard-mode,event loss mode>>::
7174 The mechanism by which event records of a given channel are lost
7175 (not recorded) when there is no sub-buffer space left to store them.
7177 [[def-event-name]]event name::
7178 The name of an event, which is also the name of the event record.
7179 This is also called the _instrumentation point name_.
7182 A record, in a trace, of the payload of an event which occured.
7184 <<event,event rule>>::
7185 Set of conditions which must be satisfied for one or more occuring
7186 events to be recorded.
7188 `java.util.logging`::
7190 https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[core logging facilities].
7192 <<instrumenting,instrumentation>>::
7193 The use of LTTng probes to make a piece of software traceable.
7195 instrumentation point::
7196 A point in the execution path of a piece of software that, when
7197 reached by this execution, can emit an event.
7199 instrumentation point name::
7200 See _<<def-event-name,event name>>_.
7203 A http://logging.apache.org/log4j/1.2/[logging library] for Java
7204 developed by the Apache Software Foundation.
7207 Level of severity of a log statement or user space
7208 instrumentation point.
7211 The _Linux Trace Toolkit: next generation_ project.
7213 <<lttng-cli,cmd:lttng>>::
7214 A command-line tool provided by the LTTng-tools project which you
7215 can use to send and receive control messages to and from a
7219 The https://github.com/lttng/lttng-analyses[LTTng analyses] project,
7220 which is a set of analyzing programs that are used to obtain a
7221 higher level view of an LTTng trace.
7223 cmd:lttng-consumerd::
7224 The name of the consumer daemon program.
7227 A utility provided by the LTTng-tools project which can convert
7228 ring buffer files (usually
7229 <<persistent-memory-file-systems,saved on a persistent memory file system>>)
7232 LTTng Documentation::
7235 <<lttng-live,LTTng live>>::
7236 A communication protocol between the relay daemon and live viewers
7237 which makes it possible to see events "live", as they are received by
7240 <<lttng-modules,LTTng-modules>>::
7241 The https://github.com/lttng/lttng-modules[LTTng-modules] project,
7242 which contains the Linux kernel modules to make the Linux kernel
7243 instrumentation points available for LTTng tracing.
7246 The name of the relay daemon program.
7248 cmd:lttng-sessiond::
7249 The name of the session daemon program.
7252 The https://github.com/lttng/lttng-tools[LTTng-tools] project, which
7253 contains the various programs and libraries used to
7254 <<controlling-tracing,control tracing>>.
7256 <<lttng-ust,LTTng-UST>>::
7257 The https://github.com/lttng/lttng-ust[LTTng-UST] project, which
7258 contains libraries to instrument user applications.
7260 <<lttng-ust-agents,LTTng-UST Java agent>>::
7261 A Java package provided by the LTTng-UST project to allow the
7262 LTTng instrumentation of `java.util.logging` and Apache log4j 1.2
7265 <<lttng-ust-agents,LTTng-UST Python agent>>::
7266 A Python package provided by the LTTng-UST project to allow the
7267 LTTng instrumentation of Python logging statements.
7269 <<channel-overwrite-mode-vs-discard-mode,overwrite mode>>::
7270 The event loss mode in which new event records overwrite older
7271 event records when there's no sub-buffer space left to store them.
7273 <<channel-buffering-schemes,per-process buffering>>::
7274 A buffering scheme in which each instrumented process has its own
7275 sub-buffers for a given user space channel.
7277 <<channel-buffering-schemes,per-user buffering>>::
7278 A buffering scheme in which all the processes of a Unix user share the
7279 same sub-buffer for a given user space channel.
7281 <<lttng-relayd,relay daemon>>::
7282 A process which is responsible for receiving the trace data sent by
7283 a distant consumer daemon.
7286 A set of sub-buffers.
7288 <<lttng-sessiond,session daemon>>::
7289 A process which receives control commands from you and orchestrates
7290 the tracers and various LTTng daemons.
7292 <<taking-a-snapshot,snapshot>>::
7293 A copy of the current data of all the sub-buffers of a given tracing
7294 session, saved as trace files.
7297 One part of an LTTng ring buffer which contains event records.
7300 The time information attached to an event when it is emitted.
7303 A set of files which are the concatenations of one or more
7304 flushed sub-buffers.
7307 The action of recording the events emitted by an application
7308 or by a system, or to initiate such recording by controlling
7312 The http://tracecompass.org[Trace Compass] project and application.
7315 An instrumentation point using the tracepoint mechanism of the Linux
7316 kernel or of LTTng-UST.
7318 tracepoint definition::
7319 The definition of a single tracepoint.
7322 The name of a tracepoint.
7324 tracepoint provider::
7325 A set of functions providing tracepoints to an instrumented user
7328 Not to be confused with a _tracepoint provider package_: many tracepoint
7329 providers can exist within a tracepoint provider package.
7331 tracepoint provider package::
7332 One or more tracepoint providers compiled as an object file or as
7336 A software which records emitted events.
7338 <<domain,tracing domain>>::
7339 A namespace for event sources.
7342 The Unix group in which a Unix user can be to be allowed to trace the
7345 <<tracing-session,tracing session>>::
7346 A stateful dialogue between you and a <<lttng-sessiond,session
7350 An application running in user space, as opposed to a Linux kernel
7351 module, for example.