1 The LTTng Documentation
2 =======================
3 Philippe Proulx <pproulx@efficios.com>
7 include::../common/copyright.txt[]
10 include::../common/warning-not-maintained.txt[]
16 Welcome to the **LTTng Documentation**!
18 The _Linux Trace Toolkit: next generation_ is an open source software
19 toolkit which you can use to simultaneously trace the Linux kernel, user
20 applications, and user libraries.
24 * Kernel modules to trace the Linux kernel.
25 * Shared libraries to trace user applications written in C or C++.
26 * Java packages to trace Java applications which use
28 * A kernel module to trace shell scripts and other user applications
29 without a dedicated instrumentation mechanism.
30 * Daemons and a command-line tool, cmd:lttng, to control the
34 .Open source documentation
36 This is an **open documentation**: its source is available in a
37 https://github.com/lttng/lttng-docs[public Git repository].
39 Should you find any error in the content of this text, any grammatical
40 mistake, or any dead link, we would be very grateful if you would file a
41 GitHub issue for it or, even better, contribute a patch to this
42 documentation by creating a pull request.
46 include::../common/audience.txt[]
50 === Chapter descriptions
52 What follows is a list of brief descriptions of this documentation's
53 chapters. The latter are ordered in such a way as to make the reading
54 as linear as possible.
56 . <<nuts-and-bolts,Nuts and bolts>> explains the
57 rudiments of software tracing and the rationale behind the
59 . <<installing-lttng,Installing LTTng>> is divided into
60 sections describing the steps needed to get a working installation
61 of LTTng packages for common Linux distributions and from its
63 . <<getting-started,Getting started>> is a very concise guide to
64 get started quickly with LTTng kernel and user space tracing. This
65 chapter is recommended if you're new to LTTng or software tracing
67 . <<understanding-lttng,Understanding LTTng>> deals with some
68 core concepts and components of the LTTng suite. Understanding
69 those is important since the next chapter assumes you're familiar
71 . <<using-lttng,Using LTTng>> is a complete user guide of the
72 LTTng project. It shows in great details how to instrument user
73 applications and the Linux kernel, how to control tracing sessions
74 using the `lttng` command line tool and miscellaneous practical use
76 . <<reference,Reference>> contains references of LTTng components,
77 like links to online manpages and various APIs.
79 We recommend that you read the above chapters in this order, although
80 some of them may be skipped depending on your situation. You may skip
81 <<nuts-and-bolts,Nuts and bolts>> if you're familiar with tracing
82 and LTTng. Also, you may jump over <<installing-lttng,Installing LTTng>>
83 if LTTng is already properly installed on your target system.
86 include::../common/convention.txt[]
89 include::../common/acknowledgements.txt[]
93 == What's new in LTTng {revision}?
95 The **LTTng {revision}** toolchain introduces many interesting features,
96 some of them which have been requested by users many times.
99 <<saving-loading-tracing-session,save and restore tracing sessions>>.
100 Sessions are saved to and loaded from XML files located by default in a
101 subdirectory of the user's home directory. LTTng daemons are also
102 configurable by configuration files as of LTTng-tools {revision}. This version
103 also makes it possible to load user-defined kernel probes with the new
104 session daemon's `--kmod-probes` option (or using the
105 `LTTNG_KMOD_PROBES` environment variable).
107 <<tracef,`tracef()`>> is a new instrumentation facility in LTTng-UST {revision}
108 which makes it possible to insert `printf()`-like tracepoints in C/$$C++$$
109 code for quick debugging. LTTng-UST {revision} also adds support for perf PMU
110 counters in user space on the x86 architecture
111 (see <<adding-context,Adding some context to channels>>).
113 As of LTTng-modules {revision}, a new
114 <<proc-lttng-logger-abi,LTTng logger ABI>>
115 is made available, making tracing Bash scripts, for example, much more
116 easier (just `echo` whatever you need to record to path:{/proc/lttng-logger}
117 while tracing is active). On the kernel side, some tracepoints are
118 added: state dumps of block devices, file descriptors, and file modes,
119 as well as http://en.wikipedia.org/wiki/Video4Linux[V4L2] events. Linux
120 3.15 is now officially supported, and system call tracing is now
121 possible on the MIPS32 architecture.
123 To learn more about the new features of LTTng {revision}, see
124 http://lttng.org/blog/2014/08/04/lttng-toolchain-2-5-0-is-out/[this
125 release announcement].
131 What is LTTng? As its name suggests, the _Linux Trace Toolkit: next
132 generation_ is a modern toolkit for tracing Linux systems and
133 applications. So your first question might rather be: **what is
136 As the history of software engineering progressed and led to what
137 we now take for granted--complex, numerous and
138 interdependent software applications running in parallel on
139 sophisticated operating systems like Linux--the authors of such
140 components, or software developers, began feeling a natural
141 urge of having tools to ensure the robustness and good performance
142 of their masterpieces.
144 One major achievement in this field is, inarguably, the
145 https://www.gnu.org/software/gdb/[GNU debugger (GDB)], which is an
146 essential tool for developers to find and fix bugs. But even the best
147 debugger won't help make your software run faster, and nowadays, faster
148 software means either more work done by the same hardware, or cheaper
149 hardware for the same work.
151 A _profiler_ is often the tool of choice to identify performance
152 bottlenecks. Profiling is suitable to identify _where_ performance is
153 lost in a given software; the profiler outputs a profile, a statistical
154 summary of observed events, which you may use to know which functions
155 took the most time to execute. However, a profiler won't report _why_
156 some identified functions are the bottleneck. Also, bottlenecks might
157 only occur when specific conditions are met. For a thorough
158 investigation of software performance issues, a history of execution,
159 with historical values of chosen variables, is essential. This is where
160 tracing comes in handy.
162 _Tracing_ is a technique used to understand what goes on in a running
163 software system. The software used for tracing is called a _tracer_,
164 which is conceptually similar to a tape recorder. When recording,
165 specific points placed in the software source code generate events that
166 are saved on a giant tape: a _trace_ file. Both user applications and
167 the operating system may be traced at the same time, opening the
168 possibility of resolving a wide range of problems that are otherwise
169 extremely challenging.
171 Tracing is often compared to _logging_. However, tracers and loggers are
172 two different types of tools, serving two different purposes. Tracers
173 are designed to record much lower-level events that occur much more
174 frequently than log messages, often in the thousands per second range,
175 with very little execution overhead. Logging is more appropriate for
176 very high-level analysis of less frequent events: user accesses,
177 exceptional conditions (e.g., errors, warnings), database transactions,
178 instant messaging communications, etc. More formally, logging is one of
179 several use cases that can be accomplished with tracing.
181 The list of recorded events inside a trace file may be read manually
182 like a log file for the maximum level of detail, but it is generally
183 much more interesting to perform application-specific analyses to
184 produce reduced statistics and graphs that are useful to resolve a given
185 problem. Trace viewers and analysers are specialized tools which achieve
188 So, in the end, this is what LTTng is: a powerful, open source set of
189 tools to trace the Linux kernel and user applications. LTTng is composed
190 of several components actively maintained and developed by its
191 http://lttng.org/community/#where[community].
193 Excluding proprietary solutions, a few competing software tracers exist
195 https://www.kernel.org/doc/Documentation/trace/ftrace.txt[ftrace] is the
196 de facto function tracer of the Linux kernel.
197 http://linux.die.net/man/1/strace[strace] is able to record all system
198 calls made by a user process.
199 https://sourceware.org/systemtap/[SystemTap] is a Linux kernel and user
200 space tracer which uses custom user scripts to produce plain text
201 traces. http://www.sysdig.org/[sysdig] also uses scripts, written in
202 Lua, to trace and analyze the Linux kernel.
204 The main distinctive features of LTTng is that it produces correlated
205 kernel and user space traces, as well as doing so with the lowest
206 overhead amongst other solutions. It produces trace files in the
207 http://www.efficios.com/ctf[CTF] format, an optimized file format for
208 production and analyses of multi-gigabyte data. LTTng is the result of
209 close to 10 years of active development by a community of passionate
210 developers. It is currently available on some major desktop, server, and
211 embedded Linux distributions.
213 The main interface for tracing control is a single command line tool
214 named `lttng`. The latter can create several tracing sessions,
215 enable/disable events on the fly, filter them efficiently with custom
216 user expressions, start/stop tracing and do much more. Traces can be
217 recorded on disk or sent over the network, kept totally or partially,
218 and viewed once tracing is inactive or in real-time.
220 <<installing-lttng,Install LTTng now>> and start tracing!
226 include::../common/warning-no-installation.txt[]
228 **LTTng** is a set of software components which interact to allow
229 instrumenting the Linux kernel and user applications and controlling
230 tracing sessions (starting/stopping tracing, enabling/disabling events,
231 etc.). Those components are bundled into the following packages:
234 Libraries and command line interface to control tracing sessions.
237 Linux kernel modules allowing Linux to be traced using LTTng.
240 User space tracing library.
242 Most distributions mark the LTTng-modules and LTTng-UST packages as
243 optional. Note that LTTng-modules is only required if you intend to
244 trace the Linux kernel and LTTng-UST is only required if you intend to
245 trace user space applications.
249 == Getting started with LTTng
251 This is a small guide to get started quickly with LTTng kernel and user
252 space tracing. For intermediate to advanced use cases and a more
253 thorough understanding of LTTng, see <<using-lttng,Using LTTng>> and
254 <<understanding-lttng,Understanding LTTng>>.
256 Before reading this guide, make sure LTTng
257 <<installing-lttng,is installed>>. You will at least need LTTng-tools.
258 Also install LTTng-modules for
259 <<tracing-the-linux-kernel,tracing the Linux kernel>>
260 and LTTng-UST for <<tracing-your-own-user-application,tracing your own
261 user space applications>>. When your traces are finally written and
263 <<viewing-and-analyzing-your-traces,Viewing and analyzing your traces>>
264 section of this chapter will help you analyze your tracepoint
265 events to investigate.
268 [[tracing-the-linux-kernel]]
269 === Tracing the Linux kernel
271 Make sure LTTng-tools and LTTng-modules packages
272 <<installing-lttng,are installed>>.
274 Since you're about to trace the Linux kernel itself, let's look at the
275 available kernel events using the `lttng` tool, which has a
276 Git-like command line structure:
283 Before tracing, you need to create a session:
287 sudo lttng create my-session
290 TIP: You can avoid using `sudo` in the previous and following commands
291 if your user is a member of the <<lttng-sessiond,tracing group>>.
293 `my-session` is the tracing session name and could be anything you
294 like. `auto` will be used if omitted.
296 Let's now enable some events for this session:
300 sudo lttng enable-event --kernel sched_switch,sched_process_fork
303 or you might want to simply enable all available kernel events (beware
304 that trace files will grow rapidly when doing this):
308 sudo lttng enable-event --kernel --all
318 By default, traces are saved in
319 +\~/lttng-traces/__name__-__date__-__time__+,
320 where +__name__+ is the session name.
322 When you're done tracing:
330 Although `destroy` looks scary here, it doesn't actually destroy the
331 outputted trace files: it only destroys the tracing session.
333 What's next? Have a look at
334 <<viewing-and-analyzing-your-traces,Viewing and analyzing your traces>>
335 to view and analyze the trace you just recorded.
338 [[tracing-your-own-user-application]]
339 === Tracing your own user application
341 The previous section helped you create a trace out of Linux kernel
342 events. This section steps you through a simple example showing you how
343 to trace a _Hello world_ program written in C.
345 Make sure LTTng-tools and LTTng-UST packages
346 <<installing-lttng,are installed>>.
348 Tracing is just like having `printf()` calls at specific locations of
349 your source code, albeit LTTng is much faster and more flexible than
350 `printf()`. In the LTTng realm, **`tracepoint()`** is analogous to
353 Unlike `printf()`, though, `tracepoint()` does not use a format string to
354 know the types of its arguments: the formats of all tracepoints must be
355 defined before using them. So before even writing our _Hello world_ program,
356 we need to define the format of our tracepoint. This is done by writing a
357 **template file**, with a name usually ending
358 with the `.tp` extension (for **t**race**p**oint),
359 which the `lttng-gen-tp` tool (shipped with LTTng-UST) will use to generate
360 an object file (along with a `.c` file) and a header to be
361 included in our application source code.
363 Here's the whole flow:
366 .Build workflow for LTTng application tracing.
367 image::lttng-lttng-gen-tp.png[]
369 The template file format is a list of tracepoint definitions
370 and other optional definition entries which we will skip for
371 this quickstart. Each tracepoint is defined using the
372 `TRACEPOINT_EVENT()` macro. For each tracepoint, you must provide:
374 * a **provider name**, which is the "scope" of this tracepoint (this usually
375 includes the company and project names)
376 * a **tracepoint name**
377 * a **list of arguments** for the eventual `tracepoint()` call,
379 ** the argument C type
381 * a **list of fields**, which will be the actual fields of the recorded events
384 Here's a simple tracepoint definition example with two arguments: an integer
397 ctf_string(my_string_field, my_string_arg)
398 ctf_integer(int, my_integer_field, my_integer_arg)
403 The exact syntax is well explained in the
404 <<c-application,C application>> instrumenting guide of the
405 <<using-lttng,Using LTTng>> chapter, as well as in man:lttng-ust(3).
407 Save the above snippet as path:{hello-tp.tp} and run:
411 lttng-gen-tp hello-tp.tp
414 The following files will be created next to path:{hello-tp.tp}:
420 path:{hello-tp.o} is the compiled object file of path:{hello-tp.c}.
422 Now, by including path:{hello-tp.h} in your own application, you may use the
423 tracepoint defined above by properly refering to it when calling
429 #include "hello-tp.h"
431 int main(int argc, char* argv[])
435 puts("Hello, World!\nPress Enter to continue...");
437 /* The following getchar() call is only placed here for the purpose
438 * of this demonstration, for pausing the application in order for
439 * you to have time to list its events. It's not needed otherwise.
443 /* A tracepoint() call. Arguments, as defined in hello-tp.tp:
445 * 1st: provider name (always)
446 * 2nd: tracepoint name (always)
447 * 3rd: my_integer_arg (first user-defined argument)
448 * 4th: my_string_arg (second user-defined argument)
450 * Notice the provider and tracepoint names are NOT strings;
451 * they are in fact parts of variables created by macros in
454 tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");
456 for (x = 0; x < argc; ++x) {
457 tracepoint(hello_world, my_first_tracepoint, x, argv[x]);
460 puts("Quitting now!");
462 tracepoint(hello_world, my_first_tracepoint, x * x, "x^2");
468 Save this as path:{hello.c}, next to path:{hello-tp.tp}.
470 Notice path:{hello-tp.h}, the header file generated by path:{lttng-gen-tp} from
471 our template file path:{hello-tp.tp}, is included by path:{hello.c}.
473 You are now ready to compile the application with LTTng-UST support:
477 gcc -o hello hello.c hello-tp.o -llttng-ust -ldl
481 <<tracing-the-linux-kernel,Tracing the Linux kernel>> section, the
482 following steps will look familiar.
484 First, run the application with a few arguments:
488 ./hello world and beyond
495 Press Enter to continue...
498 Use the `lttng` tool to list all available user space events:
502 lttng list --userspace
505 You should see the `hello_world:my_first_tracepoint` tracepoint listed
506 under the `./hello` process.
508 Create a tracing session:
512 lttng create my-userspace-session
515 Enable the `hello_world:my_first_tracepoint` tracepoint:
519 lttng enable-event --userspace hello_world:my_first_tracepoint
529 Go back to the running path:{hello} application and press Enter. All
530 `tracepoint()` calls will be executed and the program will finally exit.
539 Done! You may use `lttng view` to list the recorded events. This command
541 http://www.efficios.com/babeltrace[`babeltrace`]
542 in the background, if it is installed:
549 should output something like:
552 [18:10:27.684304496] (+?.?????????) hostname hello_world:my_first_tracepoint: { cpu_id = 0 }, { my_string_field = "hi there!", my_integer_field = 23 }
553 [18:10:27.684338440] (+0.000033944) hostname hello_world:my_first_tracepoint: { cpu_id = 0 }, { my_string_field = "./hello", my_integer_field = 0 }
554 [18:10:27.684340692] (+0.000002252) hostname hello_world:my_first_tracepoint: { cpu_id = 0 }, { my_string_field = "world", my_integer_field = 1 }
555 [18:10:27.684342616] (+0.000001924) hostname hello_world:my_first_tracepoint: { cpu_id = 0 }, { my_string_field = "and", my_integer_field = 2 }
556 [18:10:27.684343518] (+0.000000902) hostname hello_world:my_first_tracepoint: { cpu_id = 0 }, { my_string_field = "beyond", my_integer_field = 3 }
557 [18:10:27.684357978] (+0.000014460) hostname hello_world:my_first_tracepoint: { cpu_id = 0 }, { my_string_field = "x^2", my_integer_field = 16 }
560 When you're done, you may destroy the tracing session, which does _not_
561 destroy the generated trace files, leaving them available for further
566 lttng destroy my-userspace-session
569 The next section presents other alternatives to view and analyze your
573 [[viewing-and-analyzing-your-traces]]
574 === Viewing and analyzing your traces
576 This section describes how to visualize the data gathered after tracing
577 the Linux kernel or a user space application.
579 Many ways exist to read your LTTng traces:
581 * **`babeltrace`** is a command line utility which converts trace formats;
582 it supports the format used by LTTng,
583 CTF, as well as a basic
584 text output which may be ++grep++ed. The `babeltrace` command is
585 part of the http://www.efficios.com/babeltrace[Babeltrace] project.
586 * Babeltrace also includes a **Python binding** so that you may
587 easily open and read an LTTng trace with your own script, benefiting
588 from the power of Python.
589 * **http://projects.eclipse.org/projects/tools.tracecompass[Trace Compass]**
590 is an Eclipse plugin used to visualize and analyze various types of
591 traces, including LTTng's. It also comes as a standalone application
592 and can be downloaded from
593 http://projects.eclipse.org/projects/tools.tracecompass/downloads[here].
595 LTTng trace files are usually recorded in the path:{~/lttng-traces} directory.
596 Let's now view the trace and perform a basic analysis using
599 The simplest way to list all the recorded events of a trace is to pass its
600 path to `babeltrace` with no options:
604 babeltrace ~/lttng-traces/my-session
607 `babeltrace` will find all traces within the given path recursively and
608 output all their events, merging them intelligently.
610 Listing all the system calls of a Linux kernel trace with their arguments is
611 easy with `babeltrace` and `grep`:
615 babeltrace ~/lttng-traces/my-kernel-session | grep sys_
618 Counting events is also straightforward:
622 babeltrace ~/lttng-traces/my-kernel-session | grep sys_read | wc --lines
625 The text output of `babeltrace` is useful for isolating events by simple
626 matching using `grep` and similar utilities. However, more elaborate filters
627 such as keeping only events with a field value falling within a specific range
628 are not trivial to write using a shell. Moreover, reductions and even the
629 most basic computations involving multiple events are virtually impossible
632 Fortunately, Babeltrace ships with a Python 3 binding which makes it
633 really easy to read the events of an LTTng trace sequentially and compute
634 the desired information.
636 Here's a simple example using the Babeltrace Python binding. The following
637 script accepts an LTTng Linux kernel trace path as its first argument and
638 outputs the short names of the top 5 running processes on CPU 0 during the
644 from collections import Counter
649 if len(sys.argv) != 2:
650 msg = 'Usage: python {} TRACEPATH'.format(sys.argv[0])
651 raise ValueError(msg)
653 # a trace collection holds one to many traces
654 col = babeltrace.TraceCollection()
656 # add the trace provided by the user
657 # (LTTng traces always have the 'ctf' format)
658 if col.add_trace(sys.argv[1], 'ctf') is None:
659 raise RuntimeError('Cannot add trace')
661 # this counter dict will hold execution times:
663 # task command name -> total execution time (ns)
664 exec_times = Counter()
666 # this holds the last `sched_switch` timestamp
670 for event in col.events:
671 # keep only `sched_switch` events
672 if event.name != 'sched_switch':
675 # keep only events which happened on CPU 0
676 if event['cpu_id'] != 0:
680 cur_ts = event.timestamp
686 # previous task command (short) name
687 prev_comm = event['prev_comm']
689 # initialize entry in our dict if not yet done
690 if prev_comm not in exec_times:
691 exec_times[prev_comm] = 0
693 # compute previous command execution time
694 diff = cur_ts - last_ts
696 # update execution time of this command
697 exec_times[prev_comm] += diff
699 # update last timestamp
703 for name, ns in exec_times.most_common(5):
705 print('{:20}{} s'.format(name, s))
708 if __name__ == '__main__':
712 Save this script as path:{top5proc.py} and run it with Python 3, providing the
713 path to an LTTng Linux kernel trace as the first argument:
717 python3 top5proc.py ~/lttng-sessions/my-session-.../kernel
720 Make sure the path you provide is the directory containing actual trace
721 files (path:{channel0_0}, path:{metadata}, etc.): the `babeltrace` utility
722 recurses directories, but the Python binding does not.
724 Here's an example of output:
727 swapper/0 48.607245889 s
728 chromium 7.192738188 s
729 pavucontrol 0.709894415 s
730 Compositor 0.660867933 s
731 Xorg.bin 0.616753786 s
734 Note that `swapper/0` is the "idle" process of CPU 0 on Linux; since we
735 weren't using the CPU that much when tracing, its first position in the list
739 [[understanding-lttng]]
740 == Understanding LTTng
742 If you're going to use LTTng in any serious way, it is fundamental that
743 you become familiar with its core concepts. Technical terms like
744 _tracing sessions_, _domains_, _channels_ and _events_ are used over
745 and over in the <<using-lttng,Using LTTng>> chapter,
746 and it is assumed that you understand what they mean when reading it.
748 LTTng, as you already know, is a _toolkit_. It would be wrong
749 to call it a simple _tool_ since it is composed of multiple interacting
750 components. This chapter also describes the latter, providing details
751 about their respective roles and how they connect together to form
752 the current LTTng ecosystem.
758 This section explains the various elementary concepts a user has to deal
759 with when using LTTng. They are:
761 * <<tracing-session,tracing session>>
763 * <<channel,channel>>
770 A _tracing session_ is--like any session--a container of
771 state. Anything that is done when tracing using LTTng happens in the
772 scope of a tracing session. In this regard, it is analogous to a bank
773 website's session: you can't interact online with your bank account
774 unless you are logged in a session, except for reading a few static
775 webpages (LTTng, too, can report some static information that does not
776 need a created tracing session).
778 A tracing session holds the following attributes and objects (some of
779 which are described in the following sections):
782 * the tracing state (tracing started or stopped)
783 * the trace data output path/URL (local path or sent over the network)
784 * a mode (normal, snapshot or live)
785 * the snapshot output paths/URLs (if applicable)
786 * for each <<domain,domain>>, a list of <<channel,channels>>
789 ** the channel state (enabled or disabled)
790 ** its parameters (event loss mode, sub-buffers size and count,
791 timer periods, output type, trace files size and count, etc.)
792 ** a list of added context information
793 ** a list of <<event,events>>
795 ** its state (enabled or disabled)
796 ** a list of instrumentation points (tracepoints, system calls,
797 dynamic probes, etc.)
798 ** associated log levels
799 ** a filter expression
801 All this information is completely isolated between tracing sessions.
803 Conceptually, a tracing session is a per-user object; the
804 <<plumbing,Plumbing>> section shows how this is actually
805 implemented. Any user may create as many concurrent tracing sessions
806 as desired. As you can see in the list above, even the tracing state
807 is a per-tracing session attribute, so that you may trace your target
808 system/application in a given tracing session with a specific
809 configuration while another one stays inactive.
811 The trace data generated in a tracing session may be either saved
812 to disk, sent over the network or not saved at all (in which case
813 snapshots may still be saved to disk or sent to a remote machine).
819 A tracing _domain_ is the official term the LTTng project uses to
820 designate a tracer category.
822 There are currently three known domains:
826 * `java.util.logging` (JUL)
828 Different tracers expose common features in their own interfaces, but,
829 from a user's perspective, you still need to target a specific type of
830 tracer to perform some actions. For example, since both kernel and user
831 space tracers support named tracepoints (probes manually inserted in
832 source code), you need to specify which one is concerned when enabling
833 an event because both domains could have existing events with the same
836 Some features are not available in all domains. Filtering enabled
837 events using custom expressions, for example, is currently not
838 supported in the kernel domain, but support could be added in the
845 A _channel_ is a set of events with specific parameters and potential
846 added context information. Channels have unique names per domain within
847 a tracing session. A given event is always registered to at least one
848 channel; having an enabled event in two channels will produce a trace
849 with this event recorded twice everytime it occurs.
851 Channels may be individually enabled or disabled. Occurring events of
852 a disabled channel will never make it to recorded events.
854 The fundamental role of a channel is to keep a shared ring buffer, where
855 events are eventually recorded by the tracer and consumed by a consumer
856 daemon. This internal ring buffer is divided into many sub-buffers of
859 Channels, when created, may be fine-tuned thanks to a few parameters,
860 many of them related to sub-buffers. The following subsections explain
861 what those parameters are and in which situations you should manually
865 [[channel-overwrite-mode-vs-discard-mode]]
866 ===== Overwrite and discard event loss modes
868 As previously mentioned, a channel's ring buffer is divided into many
869 equally sized sub-buffers.
871 As events occur, they are serialized as trace data into a specific
872 sub-buffer (yellow arc in the following animation) until it is full:
873 when this happens, the sub-buffer is marked as consumable (red) and
874 another, _empty_ (white) sub-buffer starts receiving the following
875 events. The marked sub-buffer will be consumed eventually by a consumer
876 daemon (returns to white).
879 [role="docsvg-channel-subbuf-anim"]
884 In an ideal world, sub-buffers are consumed faster than filled, like it
885 is the case above. In the real world, however, all sub-buffers could be
886 full at some point, leaving no space to record the following events. By
887 design, LTTng is a _non-blocking_ tracer: when no empty sub-buffer
888 exists, losing events is acceptable when the alternative would be to
889 cause substantial delays in the instrumented application's execution.
890 LTTng privileges performance over integrity, aiming at perturbing the
891 traced system as little as possible in order to make tracing of subtle
892 race conditions and rare interrupt cascades possible.
894 When it comes to losing events because no empty sub-buffer is available,
895 the channel's _event loss mode_ determines what to do amongst:
898 Drop the newest events until a sub-buffer is released.
901 Clear the sub-buffer containing the oldest recorded
902 events and start recording the newest events there. This mode is
903 sometimes called _flight recorder mode_ because it behaves like a
904 flight recorder: always keep a fixed amount of the latest data.
906 Which mechanism you should choose depends on your context: prioritize
907 the newest or the oldest events in the ring buffer?
909 Beware that, in overwrite mode, a whole sub-buffer is abandoned as soon
910 as a new event doesn't find an empty sub-buffer, whereas in discard
911 mode, only the event that doesn't fit is discarded.
913 Also note that a count of lost events will be incremented and saved in
914 the trace itself when an event is lost in discard mode, whereas no
915 information is kept when a sub-buffer gets overwritten before being
918 There are known ways to decrease your probability of losing events. The
919 next section shows how tuning the sub-buffers count and size can be
920 used to virtually stop losing events.
923 [[channel-subbuf-size-vs-subbuf-count]]
924 ===== Sub-buffers count and size
926 For each channel, an LTTng user may set its number of sub-buffers and
929 Note that there is a noticeable tracer's CPU overhead introduced when
930 switching sub-buffers (marking a full one as consumable and switching
931 to an empty one for the following events to be recorded). Knowing this,
932 the following list presents a few practical situations along with how
933 to configure sub-buffers for them:
935 High event throughput::
936 In general, prefer bigger sub-buffers to
937 lower the risk of losing events. Having bigger sub-buffers will
938 also ensure a lower sub-buffer switching frequency. The number of
939 sub-buffers is only meaningful if the channel is in overwrite mode:
940 in this case, if a sub-buffer overwrite happens, you will still have
941 the other sub-buffers left unaltered.
943 Low event throughput::
944 In general, prefer smaller sub-buffers
945 since the risk of losing events is already low. Since events
946 happen less frequently, the sub-buffer switching frequency should
947 remain low and thus the tracer's overhead should not be a problem.
950 If your target system has a low memory
951 limit, prefer fewer first, then smaller sub-buffers. Even if the
952 system is limited in memory, you want to keep the sub-buffers as
953 big as possible to avoid a high sub-buffer switching frequency.
955 You should know that LTTng uses CTF as its trace format, which means
956 event data is very compact. For example, the average LTTng Linux kernel
957 event weights about 32{nbsp}bytes. A sub-buffer size of 1{nbsp}MiB is
960 The previous situations highlight the major trade-off between a few big
961 sub-buffers and more, smaller sub-buffers: sub-buffer switching
962 frequency vs. how much data is lost in overwrite mode. Assuming a
963 constant event throughput and using the overwrite mode, the two
964 following configurations have the same ring buffer total size:
967 [role="docsvg-channel-subbuf-size-vs-count-anim"]
972 * **2 sub-buffers of 4 MiB each** lead to a very low sub-buffer
973 switching frequency, but if a sub-buffer overwrite happens, half of
974 the recorded events so far (4{nbsp}MiB) are definitely lost.
975 * **8 sub-buffers of 1 MiB each** lead to 4{nbsp}times the tracer's
976 overhead as the previous configuration, but if a sub-buffer
977 overwrite happens, only the eighth of events recorded so far are
980 In discard mode, the sub-buffers count parameter is pointless: use two
981 sub-buffers and set their size according to the requirements of your
985 [[channel-switch-timer]]
988 The _switch timer_ period is another important configurable feature of
989 channels to ensure periodic sub-buffer flushing.
991 When the _switch timer_ fires, a sub-buffer switch happens. This timer
992 may be used to ensure that event data is consumed and committed to
993 trace files periodically in case of a low event throughput:
996 [role="docsvg-channel-switch-timer"]
1001 It's also convenient when big sub-buffers are used to cope with
1002 sporadic high event throughput, even if the throughput is normally
1006 [[channel-buffering-schemes]]
1007 ===== Buffering schemes
1009 In the user space tracing domain, two **buffering schemes** are
1010 available when creating a channel:
1013 Keep one ring buffer per process.
1016 Keep one ring buffer for all processes of a single user.
1018 The per-PID buffering scheme will consume more memory than the per-UID
1019 option if more than one process is instrumented for LTTng-UST. However,
1020 per-PID buffering ensures that one process having a high event
1021 throughput won't fill all the shared sub-buffers, only its own.
1023 The Linux kernel tracing domain only has one available buffering scheme
1024 which is to use a single ring buffer for the whole system.
1030 An _event_, in LTTng's realm, is a term often used metonymically,
1031 having multiple definitions depending on the context:
1033 . When tracing, an event is a _point in space-time_. Space, in a
1034 tracing context, is the set of all executable positions of a
1035 compiled application by a logical processor. When a program is
1036 executed by a processor and some instrumentation point, or
1037 _probe_, is encountered, an event occurs. This event is accompanied
1038 by some contextual payload (values of specific variables at this
1039 point of execution) which may or may not be recorded.
1040 . In the context of a recorded trace file, the term _event_ implies
1042 . When configuring a tracing session, _enabled events_ refer to
1043 specific rules which could lead to the transfer of actual
1044 occurring events (1) to recorded events (2).
1046 The whole <<core-concepts,Core concepts>> section focuses on the
1047 third definition. An event is always registered to _one or more_
1048 channels and may be enabled or disabled at will per channel. A disabled
1049 event will never lead to a recorded event, even if its channel
1052 An event (3) is enabled with a few conditions that must _all_ be met
1053 when an event (1) happens in order to generate a recorded event (2):
1055 . A _probe_ or group of probes in the traced application must be
1057 . **Optionally**, the probe must have a log level matching a
1058 log level range specified when enabling the event.
1059 . **Optionally**, the occurring event must satisfy a custom
1060 expression, or _filter_, specified when enabling the event.
1062 The following illustration summarizes how tracing sessions, domains,
1063 channels and events are related:
1067 image::core-concepts.png[]
1069 This diagram also shows how events may be individually enabled/disabled
1070 (green/grey) and how a given event may be registered to more than one
1077 The previous section described the concepts at the heart of LTTng.
1078 This section summarizes LTTng's implementation: how those objects are
1079 managed by different applications and libraries working together to
1083 [[plumbing-overview]]
1086 As <<installing-lttng,mentioned previously>>, the whole LTTng suite
1087 is made of the following packages: LTTng-tools, LTTng-UST, and
1088 LTTng-modules. Together, they provide different daemons, libraries,
1089 kernel modules and command line interfaces. The following tree shows
1090 which usable component belongs to which package:
1093 ** session daemon (`lttng-sessiond`)
1094 ** consumer daemon (`lttng-consumerd`)
1095 ** relay daemon (`lttng-relayd`)
1096 ** tracing control library (`liblttng-ctl`)
1097 ** tracing control command line tool (`lttng`)
1099 ** user space tracing library (`liblttng-ust`) and its headers
1100 ** preloadable user space tracing helpers
1101 (`liblttng-ust-libc-wrapper`, `liblttng-ust-pthread-wrapper`,
1102 `liblttng-ust-cyg-profile`, `liblttng-ust-cyg-profile-fast`
1103 and `liblttng-ust-dl`)
1104 ** user space tracepoint code generator command line tool
1106 ** `java.util.logging` tracepoint provider (`liblttng-ust-jul-jni`)
1107 and JAR file (path:{liblttng-ust-jul.jar})
1108 * **LTTng-modules**:
1109 ** LTTng Linux kernel tracer module
1110 ** tracing ring buffer kernel modules
1111 ** many LTTng probe kernel modules
1113 The following diagram shows how the most important LTTng components
1114 interact. Plain black arrows represent trace data paths while dashed
1115 red arrows indicate control communications. The LTTng relay daemon is
1116 shown running on a remote system, although it could as well run on the
1117 target (monitored) system.
1121 image::plumbing.png[]
1123 Each component is described in the following subsections.
1129 At the heart of LTTng's plumbing is the _session daemon_, often called
1130 by its command name, `lttng-sessiond`.
1132 The session daemon is responsible for managing tracing sessions and
1133 what they logically contain (channel properties, enabled/disabled
1134 events, etc.). By communicating locally with instrumented applications
1135 (using LTTng-UST) and with the LTTng Linux kernel modules
1136 (LTTng-modules), it oversees all tracing activities.
1138 One of the many things that `lttng-sessiond` does is to keep
1139 track of the available event types. User space applications and
1140 libraries actively connect and register to the session daemon when they
1141 start. By contrast, `lttng-sessiond` seeks out and loads the appropriate
1142 LTTng kernel modules as part of its own initialization. Kernel event
1143 types are _pulled_ by `lttng-sessiond`, whereas user space event types
1144 are _pushed_ to it by the various user space tracepoint providers.
1146 Using a specific inter-process communication protocol with Linux kernel
1147 and user space tracers, the session daemon can send channel information
1148 so that they are initialized, enable/disable specific probes based on
1149 enabled/disabled events by the user, send event filters information to
1150 LTTng tracers so that filtering actually happens at the tracer site,
1151 start/stop tracing a specific application or the Linux kernel, etc.
1153 The session daemon is not useful without some user controlling it,
1154 because it's only a sophisticated control interchange and thus
1155 doesn't make any decision on its own. `lttng-sessiond` opens a local
1156 socket for controlling it, albeit the preferred way to control it is
1157 using `liblttng-ctl`, an installed C library hiding the communication
1158 protocol behind an easy-to-use API. The `lttng` tool makes use of
1159 `liblttng-ctl` to implement a user-friendly command line interface.
1161 `lttng-sessiond` does not receive any trace data from instrumented
1162 applications; the _consumer daemons_ are the programs responsible for
1163 collecting trace data using shared ring buffers. However, the session
1164 daemon is the one that must spawn a consumer daemon and establish
1165 a control communication with it.
1167 Session daemons run on a per-user basis. Knowing this, multiple
1168 instances of `lttng-sessiond` may run simultaneously, each belonging
1169 to a different user and each operating independently of the others.
1170 Only `root`'s session daemon, however, may control LTTng kernel modules
1171 (i.e. the kernel tracer). With that in mind, if a user has no root
1172 access on the target system, he cannot trace the system's kernel, but
1173 should still be able to trace its own instrumented applications.
1175 It has to be noted that, although only `root`'s session daemon may
1176 control the kernel tracer, the `lttng-sessiond` command has a `--group`
1177 option which may be used to specify the name of a special user group
1178 allowed to communicate with `root`'s session daemon and thus record
1179 kernel traces. By default, this group is named `tracing`.
1181 If not done yet, the `lttng` tool, by default, automatically starts a
1182 session daemon. `lttng-sessiond` may also be started manually:
1189 This will start the session daemon in foreground. Use
1193 lttng-sessiond --daemonize
1196 to start it as a true daemon.
1198 To kill the current user's session daemon, `pkill` may be used:
1202 pkill lttng-sessiond
1205 The default `SIGTERM` signal will terminate it cleanly.
1207 Several other options are available and described in
1208 man:lttng-sessiond(8) or by running `lttng-sessiond --help`.
1212 ==== Consumer daemon
1214 The _consumer daemon_, or `lttng-consumerd`, is a program sharing some
1215 ring buffers with user applications or the LTTng kernel modules to
1216 collect trace data and output it at some place (on disk or sent over
1217 the network to an LTTng relay daemon).
1219 Consumer daemons are created by a session daemon as soon as events are
1220 enabled within a tracing session, well before tracing is activated
1221 for the latter. Entirely managed by session daemons,
1222 consumer daemons survive session destruction to be reused later,
1223 should a new tracing session be created. Consumer daemons are always
1224 owned by the same user as their session daemon. When its owner session
1225 daemon is killed, the consumer daemon also exits. This is because
1226 the consumer daemon is always the child process of a session daemon.
1227 Consumer daemons should never be started manually. For this reason,
1228 they are not installed in one of the usual locations listed in the
1229 `PATH` environment variable. `lttng-sessiond` has, however, a
1230 bunch of options (see man:lttng-sessiond(8)) to
1231 specify custom consumer daemon paths if, for some reason, a consumer
1232 daemon other than the default installed one is needed.
1234 There are up to two running consumer daemons per user, whereas only one
1235 session daemon may run per user. This is because each process has
1236 independent bitness: if the target system runs a mixture of 32-bit and
1237 64-bit processes, it is more efficient to have separate corresponding
1238 32-bit and 64-bit consumer daemons. The `root` user is an exception: it
1239 may have up to _three_ running consumer daemons: 32-bit and 64-bit
1240 instances for its user space applications and one more reserved for
1241 collecting kernel trace data.
1243 As new tracing domains are added to LTTng, the development community's
1244 intent is to minimize the need for additionnal consumer daemon instances
1245 dedicated to them. For instance, the `java.util.logging` (JUL) domain
1246 events are in fact mapped to the user space domain, thus tracing this
1247 particular domain is handled by existing user space domain consumer
1254 When a tracing session is configured to send its trace data over the
1255 network, an LTTng _relay daemon_ must be used at the other end to
1256 receive trace packets and serialize them to trace files. This setup
1257 makes it possible to trace a target system without ever committing trace
1258 data to its local storage, a feature which is useful for embedded
1259 systems, amongst others. The command implementing the relay daemon
1262 The basic use case of `lttng-relayd` is to transfer trace data received
1263 over the network to trace files on the local file system. The relay
1264 daemon must listen on two TCP ports to achieve this: one control port,
1265 used by the target session daemon, and one data port, used by the
1266 target consumer daemon. The relay and session daemons agree on common
1267 default ports when custom ones are not specified.
1269 Since the communication transport protocol for both ports is standard
1270 TCP, the relay daemon may be started either remotely or locally (on the
1273 While two instances of consumer daemons (32-bit and 64-bit) may run
1274 concurrently for a given user, `lttng-relayd` needs only be of its
1275 host operating system's bitness.
1277 The other important feature of LTTng's relay daemon is the support of
1278 _LTTng live_. LTTng live is an application protocol to view events as
1279 they arrive. The relay daemon will still record events in trace files,
1280 but a _tee_ may be created to inspect incoming events. Using LTTng live
1281 locally thus requires to run a local relay daemon.
1284 [[liblttng-ctl-lttng]]
1285 ==== [[lttng-cli]]Control library and command line interface
1287 The LTTng control library, `liblttng-ctl`, can be used to communicate
1288 with the session daemon using a C API that hides the underlying
1289 protocol's details. `liblttng-ctl` is part of LTTng-tools.
1291 `liblttng-ctl` may be used by including its "master" header:
1295 #include <lttng/lttng.h>
1298 Some objects are referred by name (C string), such as tracing sessions,
1299 but most of them require creating a handle first using
1300 `lttng_create_handle()`. The best available developer documentation for
1301 `liblttng-ctl` is, for the moment, its installed header files as such.
1302 Every function/structure is thoroughly documented.
1304 The `lttng` program is the _de facto_ standard user interface to
1305 control LTTng tracing sessions. `lttng` uses `liblttng-ctl` to
1306 communicate with session daemons behind the scenes.
1307 Its man page, man:lttng(1), is exhaustive, as well as its command
1308 line help (+lttng _cmd_ --help+, where +_cmd_+ is the command name).
1310 The <<controlling-tracing,Controlling tracing>> section is a feature
1311 tour of the `lttng` tool.
1315 ==== User space tracing library
1317 The user space tracing part of LTTng is possible thanks to the user
1318 space tracing library, `liblttng-ust`, which is part of the LTTng-UST
1321 `liblttng-ust` provides header files containing macros used to define
1322 tracepoints and create tracepoint providers, as well as a shared object
1323 that must be linked to individual applications to connect to and
1324 communicate with a session daemon and a consumer daemon as soon as the
1327 The exact mechanism by which an application is registered to the
1328 session daemon is beyond the scope of this documentation. The only thing
1329 you need to know is that, since the library constructor does this job
1330 automatically, tracepoints may be safely inserted anywhere in the source
1331 code without prior manual initialization of `liblttng-ust`.
1333 The `liblttng-ust`-session daemon collaboration also provides an
1334 interesting feature: user space events may be enabled _before_
1335 applications actually start. By doing this and starting tracing before
1336 launching the instrumented application, you make sure that even the
1337 earliest occurring events can be recorded.
1339 The <<c-application,C application>> instrumenting guide of the
1340 <<using-lttng,Using LTTng>> chapter focuses on using `liblttng-ust`:
1341 instrumenting, building/linking and running a user application.
1345 ==== LTTng kernel modules
1347 The LTTng Linux kernel modules provide everything needed to trace the
1348 Linux kernel: various probes, a ring buffer implementation for a
1349 consumer daemon to read trace data and the tracer itself.
1351 Only in exceptional circumstances should you ever need to load the
1352 LTTng kernel modules manually: it is normally the responsability of
1353 `root`'s session daemon to do so. If you were to develop your own LTTng
1354 probe module, however--for tracing a custom kernel or some kernel
1355 module (this topic is covered in the
1356 <<instrumenting-linux-kernel,Linux kernel>> instrumenting guide of
1357 the <<using-lttng,Using LTTng>> chapter)--you should either
1358 load it manually, or use the `--kmod-probes` option of the session
1359 daemon to load a specific list of kernel probes (beware, however,
1360 that the `--kmod-probes` option specifies an _absolute_ list, which
1361 means you also have to specify the default probes you need). The
1362 session and consumer daemons of regular users do not interact with the
1363 LTTng kernel modules at all.
1365 LTTng kernel modules are installed, by default, in
1366 +/usr/lib/modules/_release_/extra+, where +_release_+ is the
1367 kernel release (see `uname --kernel-release`).
1373 Using LTTng involves two main activities: **instrumenting** and
1374 **controlling tracing**.
1376 _<<instrumenting,Instrumenting>>_ is the process of inserting probes
1377 into some source code. It can be done manually, by writing tracepoint
1378 calls at specific locations in the source code of the program to trace,
1379 or more automatically using dynamic probes (address in assembled code,
1380 symbol name, function entry/return, etc.).
1382 It has to be noted that, as an LTTng user, you may not have to worry
1383 about the instrumentation process. Indeed, you may want to trace a
1384 program already instrumented. As an example, the Linux kernel is
1385 thoroughly instrumented, which is why you can trace it without caring
1386 about adding probes.
1388 _<<controlling-tracing,Controlling tracing>>_ is everything
1389 that can be done by the LTTng session daemon, which is controlled using
1390 `liblttng-ctl` or its command line utility, `lttng`: creating tracing
1391 sessions, listing tracing sessions and events, enabling/disabling
1392 events, starting/stopping the tracers, taking snapshots, etc.
1394 This chapter is a complete user guide of both activities,
1395 with common use cases of LTTng exposed throughout the text. It is
1396 assumed that you are familiar with LTTng's concepts (events, channels,
1397 domains, tracing sessions) and that you understand the roles of its
1398 components (daemons, libraries, command line tools); if not, we invite
1399 you to read the <<understanding-lttng,Understanding LTTng>> chapter
1400 before you begin reading this one.
1402 If you're new to LTTng, we suggest that you rather start with the
1403 <<getting-started,Getting started>> small guide first, then come
1404 back here to broaden your knowledge.
1406 If you're only interested in tracing the Linux kernel with its current
1407 instrumentation, you may skip the
1408 <<instrumenting,Instrumenting>> section.
1414 There are many examples of tracing and monitoring in our everyday life.
1415 You have access to real-time and historical weather reports and forecasts
1416 thanks to weather stations installed around the country. You know your
1417 possibly hospitalized friends' and family's hearts are safe thanks to
1418 electrocardiography. You make sure not to drive your car too fast
1419 and have enough fuel to reach your destination thanks to gauges visible
1422 All the previous examples have something in common: they rely on
1423 **probes**. Without electrodes attached to the surface of a body's
1424 skin, cardiac monitoring would be futile.
1426 LTTng, as a tracer, is no different from the real life examples above.
1427 If you're about to trace a software system, i.e. record its history of
1428 execution, you better have probes in the subject you're
1429 tracing: the actual software. Various ways were developed to do this.
1430 The most straightforward one is to manually place probes, called
1431 _tracepoints_, in the software's source code. The Linux kernel tracing
1432 domain also allows probes added dynamically.
1434 If you're only interested in tracing the Linux kernel, it may very well
1435 be that your tracing needs are already appropriately covered by LTTng's
1436 built-in Linux kernel tracepoints and other probes. Or you may be in
1437 possession of a user space application which has already been
1438 instrumented. In such cases, the work will reside entirely in the design
1439 and execution of tracing sessions, allowing you to jump to
1440 <<controlling-tracing,Controlling tracing>> right now.
1442 This chapter focuses on the following use cases of instrumentation:
1444 * <<c-application,C>> and <<cxx-application,$$C++$$>> applications
1445 * <<prebuilt-ust-helpers,prebuilt user space tracing helpers>>
1446 * <<java-application,Java application>>
1447 * <<instrumenting-linux-kernel,Linux kernel>> module or the
1449 * the <<proc-lttng-logger-abi,path:{/proc/lttng-logger} ABI>>
1451 Some advanced techniques are also presented at the very end of this
1458 Instrumenting a C (or $$C++$$) application, be it an executable program or
1459 a library, implies using LTTng-UST, the
1460 user space tracing component of LTTng. For C/$$C++$$ applications, the
1461 LTTng-UST package includes a dynamically loaded library
1462 (`liblttng-ust`), C headers and the `lttng-gen-tp` command line utility.
1464 Since C and $$C++$$ are the base languages of virtually all other
1465 programming languages
1466 (Java virtual machine, Python, Perl, PHP and Node.js interpreters, etc.),
1467 implementing user space tracing for an unsupported language is just a
1468 matter of using the LTTng-UST C API at the right places.
1470 The usual work flow to instrument a user space C application with
1473 . Define tracepoints (actual probes)
1474 . Write tracepoint providers
1475 . Insert tracepoints into target source code
1476 . Package (build) tracepoint providers
1477 . Build user application and link it with tracepoint providers
1479 The steps above are discussed in greater detail in the following
1483 [[tracepoint-provider]]
1484 ===== Tracepoint provider
1486 Before jumping into defining tracepoints and inserting
1487 them into the application source code, you must understand what a
1488 _tracepoint provider_ is.
1490 For the sake of this guide, consider the following two files:
1495 #undef TRACEPOINT_PROVIDER
1496 #define TRACEPOINT_PROVIDER my_provider
1498 #undef TRACEPOINT_INCLUDE
1499 #define TRACEPOINT_INCLUDE "./tp.h"
1501 #if !defined(_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
1504 #include <lttng/tracepoint.h>
1508 my_first_tracepoint,
1510 int, my_integer_arg,
1511 char*, my_string_arg
1514 ctf_string(my_string_field, my_string_arg)
1515 ctf_integer(int, my_integer_field, my_integer_arg)
1521 my_other_tracepoint,
1526 ctf_integer(int, some_field, my_int)
1532 #include <lttng/tracepoint-event.h>
1538 #define TRACEPOINT_CREATE_PROBES
1543 The two files above are defining a _tracepoint provider_. A tracepoint
1544 provider is some sort of namespace for _tracepoint definitions_. Tracepoint
1545 definitions are written above with the `TRACEPOINT_EVENT()` macro, and allow
1546 eventual `tracepoint()` calls respecting their definitions to be inserted
1547 into the user application's C source code (we explore this in a
1550 Many tracepoint definitions may be part of the same tracepoint provider
1551 and many tracepoint providers may coexist in a user space application. A
1552 tracepoint provider is packaged either:
1554 * directly into an existing user application's C source file
1556 * as a static library
1557 * as a shared library
1559 The two files above, path:{tp.h} and path:{tp.c}, show a typical template for
1560 writing a tracepoint provider. LTTng-UST was designed so that two
1561 tracepoint providers should not be defined in the same header file.
1563 We will now go through the various parts of the above files and
1564 give them a meaning. As you may have noticed, the LTTng-UST API for
1565 C/$$C++$$ applications is some preprocessor sorcery. The LTTng-UST macros
1566 used in your application and those in the LTTng-UST headers are
1567 combined to produce actual source code needed to make tracing possible
1570 Let's start with the header file, path:{tp.h}. It begins with
1574 #undef TRACEPOINT_PROVIDER
1575 #define TRACEPOINT_PROVIDER my_provider
1578 `TRACEPOINT_PROVIDER` defines the name of the provider to which the
1579 following tracepoint definitions will belong. It is used internally by
1580 LTTng-UST headers and _must_ be defined. Since `TRACEPOINT_PROVIDER`
1581 could have been defined by another header file also included by the same
1582 C source file, the best practice is to undefine it first.
1584 NOTE: Names in LTTng-UST follow the C
1585 _identifier_ syntax (starting with a letter and containing either
1586 letters, numbers or underscores); they are _not_ C strings
1587 (not surrounded by double quotes). This is because LTTng-UST macros
1588 use those identifier-like strings to create symbols (named types and
1591 The tracepoint provider is a group of tracepoint definitions; its chosen
1592 name should reflect this. A hierarchy like Java packages is recommended,
1593 using underscores instead of dots, e.g., `org_company_project_component`.
1595 Next is `TRACEPOINT_INCLUDE`:
1599 #undef TRACEPOINT_INCLUDE
1600 #define TRACEPOINT_INCLUDE "./tp.h"
1603 This little bit of instrospection is needed by LTTng-UST to include
1604 your header at various predefined places.
1606 Include guard follows:
1610 #if !defined(_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
1614 Add these precompiler conditionals to ensure the tracepoint event
1615 generation can include this file more than once.
1617 The `TRACEPOINT_EVENT()` macro is defined in a LTTng-UST header file which
1622 #include <lttng/tracepoint.h>
1625 This will also allow the application to use the `tracepoint()` macro.
1627 Next is a list of `TRACEPOINT_EVENT()` macro calls which create the
1628 actual tracepoint definitions. We will skip this for the moment and
1629 come back to how to use `TRACEPOINT_EVENT()`
1630 <<defining-tracepoints,in a later section>>. Just pay attention to
1631 the first argument: it's always the name of the tracepoint provider
1632 being defined in this header file.
1634 End of include guard:
1641 Finally, include `<lttng/tracepoint-event.h>` to expand the macros:
1645 #include <lttng/tracepoint-event.h>
1648 That's it for path:{tp.h}. Of course, this is only a header file; it must be
1649 included in some C source file to actually use it. This is the job of
1654 #define TRACEPOINT_CREATE_PROBES
1659 When `TRACEPOINT_CREATE_PROBES` is defined, the macros used in path:{tp.h},
1660 which is included just after, will actually create the source code for
1661 LTTng-UST probes (global data structures and functions) out of your
1662 tracepoint definitions. How exactly this is done is out of this text's scope.
1663 `TRACEPOINT_CREATE_PROBES` is discussed further
1665 <<building-tracepoint-providers-and-user-application,Building/linking
1666 tracepoint providers and the user application>>.
1668 You could include other header files like path:{tp.h} here to create the probes
1669 of different tracepoint providers, e.g.:
1673 #define TRACEPOINT_CREATE_PROBES
1679 The rule is: probes of a given tracepoint provider
1680 must be created in exactly one source file. This source file could be one
1681 of your project's; it doesn't have to be on its own like
1682 path:{tp.c}, although
1683 <<building-tracepoint-providers-and-user-application,a later section>>
1684 shows that doing so allows packaging the tracepoint providers
1685 independently and keep them out of your application, also making it
1686 possible to reuse them between projects.
1688 The following sections explain how to define tracepoints, how to use the
1689 `tracepoint()` macro to instrument your user space C application and how
1690 to build/link tracepoint providers and your application with LTTng-UST
1695 ===== Using `lttng-gen-tp`
1697 LTTng-UST ships with `lttng-gen-tp`, a handy command line utility for
1698 generating most of the stuff discussed above. It takes a _template file_,
1699 with a name usually ending with the `.tp` extension, containing only
1700 tracepoint definitions, and outputs a tracepoint provider (either a C
1701 source file or a precompiled object file) with its header file.
1703 `lttng-gen-tp` should suffice in <<static-linking,static linking>>
1704 situations. When using it, write a template file containing a list of
1705 `TRACEPOINT_EVENT()` macro calls. The tool will find the provider names
1706 used and generate the appropriate files which are going to look a lot
1707 like path:{tp.h} and path:{tp.c} above.
1709 Just call `lttng-gen-tp` like this:
1713 lttng-gen-tp my-template.tp
1716 path:{my-template.c}, path:{my-template.o} and path:{my-template.h}
1717 will be created in the same directory.
1719 You may specify custom C flags passed to the compiler invoked by
1720 `lttng-gen-tp` using the `CFLAGS` environment variable:
1724 CFLAGS=-I/custom/include/path lttng-gen-tp my-template.tp
1727 For more information on `lttng-gen-tp`, see man:lttng-gen-tp(1).
1730 [[defining-tracepoints]]
1731 ===== Defining tracepoints
1733 As written in <<tracepoint-provider,Tracepoint provider>>,
1734 tracepoints are defined using the
1735 `TRACEPOINT_EVENT()` macro. Each tracepoint, when called using the
1736 `tracepoint()` macro in the actual application's source code, generates
1737 a specific event type with its own fields.
1739 Let's have another look at the example above, with a few added comments:
1744 /* tracepoint provider name */
1747 /* tracepoint/event name */
1748 my_first_tracepoint,
1750 /* list of tracepoint arguments */
1752 int, my_integer_arg,
1753 char*, my_string_arg
1756 /* list of fields of eventual event */
1758 ctf_string(my_string_field, my_string_arg)
1759 ctf_integer(int, my_integer_field, my_integer_arg)
1764 The tracepoint provider name must match the name of the tracepoint
1765 provider in which this tracepoint is defined
1766 (see <<tracepoint-provider,Tracepoint provider>>). In other words,
1767 always use the same string as the value of `TRACEPOINT_PROVIDER` above.
1769 The tracepoint name will become the event name once events are recorded
1770 by the LTTng-UST tracer. It must follow the tracepoint provider name
1771 syntax: start with a letter and contain either letters, numbers or
1772 underscores. Two tracepoints under the same provider cannot have the
1773 same name, i.e. you cannot overload a tracepoint like you would
1774 overload functions and methods in $$C++$$/Java.
1776 NOTE: The concatenation of the tracepoint
1777 provider name and the tracepoint name cannot exceed 254 characters. If
1778 it does, the instrumented application will compile and run, but LTTng
1779 will issue multiple warnings and you could experience serious problems.
1781 The list of tracepoint arguments gives this tracepoint its signature:
1782 see it like the declaration of a C function. The format of `TP_ARGS()`
1783 arguments is: C type, then argument name; repeat as needed, up to ten
1784 times. For example, if we were to replicate the signature of C standard
1785 library's `fseek()`, the `TP_ARGS()` part would look like:
1796 Of course, you will need to include appropriate header files before
1797 the `TRACEPOINT_EVENT()` macro calls if any argument has a complex type.
1799 `TP_ARGS()` may not be omitted, but may be empty. `TP_ARGS(void)` is
1802 The list of fields is where the fun really begins. The fields defined
1803 in this list will be the fields of the events generated by the execution
1804 of this tracepoint. Each tracepoint field definition has a C
1805 _argument expression_ which will be evaluated when the execution reaches
1806 the tracepoint. Tracepoint arguments _may be_ used freely in those
1807 argument expressions, but they _don't_ have to.
1809 There are several types of tracepoint fields available. The macros to
1810 define them are given and explained in the
1811 <<liblttng-ust-tp-fields,LTTng-UST library reference>> section.
1813 Field names must follow the standard C identifier syntax: letter, then
1814 optional sequence of letters, numbers or underscores. Each field must have
1817 Those `ctf_*()` macros are added to the `TP_FIELDS()` part of
1818 `TRACEPOINT_EVENT()`. Note that they are not delimited by commas.
1819 `TP_FIELDS()` may be empty, but the `TP_FIELDS(void)` form is _not_
1822 The following snippet shows how argument expressions may be used in
1823 tracepoint fields and how they may refer freely to tracepoint arguments.
1827 /* for struct stat */
1828 #include <sys/types.h>
1829 #include <sys/stat.h>
1841 /* simple integer field with constant value */
1843 int, /* field C type */
1844 my_constant_field, /* field name */
1845 23 + 17 /* argument expression */
1848 /* my_int_arg tracepoint argument */
1855 /* my_int_arg squared */
1859 my_int_arg * my_int_arg
1862 /* sum of first 4 characters of my_str_arg */
1866 my_str_arg[0] + my_str_arg[1] +
1867 my_str_arg[2] + my_str_arg[3]
1870 /* my_str_arg as string field */
1872 my_str_arg_field, /* field name */
1873 my_str_arg /* argument expression */
1876 /* st_size member of st tracepoint argument, hexadecimal */
1878 off_t, /* field C type */
1879 size_field, /* field name */
1880 st->st_size /* argument expression */
1883 /* st_size member of st tracepoint argument, as double */
1885 double, /* field C type */
1886 size_dbl_field, /* field name */
1887 (double) st->st_size /* argument expression */
1890 /* half of my_str_arg string as text sequence */
1892 char, /* element C type */
1893 half_my_str_arg_field, /* field name */
1894 my_str_arg, /* argument expression */
1895 size_t, /* length expression C type */
1896 strlen(my_str_arg) / 2 /* length expression */
1902 As you can see, having a custom argument expression for each field
1903 makes tracepoints very flexible for tracing a user space C application.
1904 This tracepoint definition is reused later in this guide, when
1905 actually using tracepoints in a user space application.
1908 [[using-tracepoint-classes]]
1909 ===== Using tracepoint classes
1911 In LTTng-UST, a _tracepoint class_ is a class of tracepoints sharing the
1912 same field types and names. A _tracepoint instance_ is one instance of
1913 such a declared tracepoint class, with its own event name and tracepoint
1916 What is documented in <<defining-tracepoints,Defining tracepoints>>
1917 is actually how to declare a _tracepoint class_ and define a
1918 _tracepoint instance_ at the same time. Without revealing the internals
1919 of LTTng-UST too much, it has to be noted that one serialization
1920 function is created for each tracepoint class. A serialization
1921 function is responsible for serializing the fields of a tracepoint
1922 into a sub-buffer when tracing. For various performance reasons, when
1923 your situation requires multiple tracepoints with different names, but
1924 with the same fields layout, the best practice is to manually create
1925 a tracepoint class and instantiate as many tracepoint instances as
1926 needed. One positive effect of such a design, amongst other advantages,
1927 is that all tracepoint instances of the same tracepoint class will
1928 reuse the same serialization function, thus reducing cache pollution.
1930 As an example, here are three tracepoint definitions as we know them:
1942 ctf_integer(int, userid, userid)
1943 ctf_integer(size_t, len, len)
1955 ctf_integer(int, userid, userid)
1956 ctf_integer(size_t, len, len)
1968 ctf_integer(int, userid, userid)
1969 ctf_integer(size_t, len, len)
1974 In this case, three tracepoint classes are created, with one tracepoint
1975 instance for each of them: `get_account`, `get_settings` and
1976 `get_transaction`. However, they all share the same field names and
1977 types. Declaring one tracepoint class and three tracepoint instances of
1978 the latter is a better design choice:
1982 /* the tracepoint class */
1983 TRACEPOINT_EVENT_CLASS(
1984 /* tracepoint provider name */
1987 /* tracepoint class name */
1998 ctf_integer(int, userid, userid)
1999 ctf_integer(size_t, len, len)
2003 /* the tracepoint instances */
2004 TRACEPOINT_EVENT_INSTANCE(
2005 /* tracepoint provider name */
2008 /* tracepoint class name */
2011 /* tracepoint/event name */
2020 TRACEPOINT_EVENT_INSTANCE(
2029 TRACEPOINT_EVENT_INSTANCE(
2040 Of course, all those names and `TP_ARGS()` invocations are redundant,
2041 but some C preprocessor magic can solve this:
2045 #define MY_TRACEPOINT_ARGS \
2051 TRACEPOINT_EVENT_CLASS(
2056 ctf_integer(int, userid, userid)
2057 ctf_integer(size_t, len, len)
2061 #define MY_APP_TRACEPOINT_INSTANCE(name) \
2062 TRACEPOINT_EVENT_INSTANCE( \
2066 MY_TRACEPOINT_ARGS \
2069 MY_APP_TRACEPOINT_INSTANCE(get_account)
2070 MY_APP_TRACEPOINT_INSTANCE(get_settings)
2071 MY_APP_TRACEPOINT_INSTANCE(get_transaction)
2075 [[assigning-log-levels]]
2076 ===== Assigning log levels to tracepoints
2078 Optionally, a log level can be assigned to a defined tracepoint.
2079 Assigning different levels of importance to tracepoints can be useful;
2080 when controlling tracing sessions,
2081 <<controlling-tracing,you can choose>> to only enable tracepoints
2082 falling into a specific log level range.
2084 Log levels are assigned to defined tracepoints using the
2085 `TRACEPOINT_LOGLEVEL()` macro. The latter must be used _after_ having
2086 used `TRACEPOINT_EVENT()` for a given tracepoint. The
2087 `TRACEPOINT_LOGLEVEL()` macro has the following construct:
2091 TRACEPOINT_LOGLEVEL(PROVIDER_NAME, TRACEPOINT_NAME, LOG_LEVEL)
2094 where the first two arguments are the same as the first two arguments
2095 of `TRACEPOINT_EVENT()` and `LOG_LEVEL` is one
2096 of the values given in the
2097 <<liblttng-ust-tracepoint-loglevel,LTTng-UST library reference>>
2100 As an example, let's assign a `TRACE_DEBUG_UNIT` log level to our
2101 previous tracepoint definition:
2105 TRACEPOINT_LOGLEVEL(my_provider, my_tracepoint, TRACE_DEBUG_UNIT)
2109 [[probing-the-application-source-code]]
2110 ===== Probing the application's source code
2112 Once tracepoints are properly defined within a tracepoint provider,
2113 they may be inserted into the user application to be instrumented
2114 using the `tracepoint()` macro. Its first argument is the tracepoint
2115 provider name and its second is the tracepoint name. The next, optional
2116 arguments are defined by the `TP_ARGS()` part of the definition of
2117 the tracepoint to use.
2119 As an example, let us again take the following tracepoint definition:
2124 /* tracepoint provider name */
2127 /* tracepoint/event name */
2128 my_first_tracepoint,
2130 /* list of tracepoint arguments */
2132 int, my_integer_arg,
2133 char*, my_string_arg
2136 /* list of fields of eventual event */
2138 ctf_string(my_string_field, my_string_arg)
2139 ctf_integer(int, my_integer_field, my_integer_arg)
2144 Assuming this is part of a file named path:{tp.h} which defines the tracepoint
2145 provider and which is included by path:{tp.c}, here's a complete C application
2146 calling this tracepoint (multiple times):
2150 #define TRACEPOINT_DEFINE
2153 int main(int argc, char* argv[])
2157 tracepoint(my_provider, my_first_tracepoint, 23, "Hello, World!");
2159 for (i = 0; i < argc; ++i) {
2160 tracepoint(my_provider, my_first_tracepoint, i, argv[i]);
2167 For each tracepoint provider, `TRACEPOINT_DEFINE` must be defined into
2168 exactly one translation unit (C source file) of the user application,
2169 before including the tracepoint provider header file. In other words,
2170 for a given tracepoint provider, you cannot define `TRACEPOINT_DEFINE`,
2171 and then include its header file in two separate C source files of
2172 the same application. `TRACEPOINT_DEFINE` is discussed further in
2173 <<building-tracepoint-providers-and-user-application,Building/linking
2174 tracepoint providers and the user application>>.
2176 As another example, remember this definition we wrote in a previous
2177 section (comments are stripped):
2181 /* for struct stat */
2182 #include <sys/types.h>
2183 #include <sys/stat.h>
2195 ctf_integer(int, my_constant_field, 23 + 17)
2196 ctf_integer(int, my_int_arg_field, my_int_arg)
2197 ctf_integer(int, my_int_arg_field2, my_int_arg * my_int_arg)
2198 ctf_integer(int, sum4_field, my_str_arg[0] + my_str_arg[1] +
2199 my_str_arg[2] + my_str_arg[3])
2200 ctf_string(my_str_arg_field, my_str_arg)
2201 ctf_integer_hex(off_t, size_field, st->st_size)
2202 ctf_float(double, size_dbl_field, (double) st->st_size)
2203 ctf_sequence_text(char, half_my_str_arg_field, my_str_arg,
2204 size_t, strlen(my_str_arg) / 2)
2209 Here's an example of calling it:
2213 #define TRACEPOINT_DEFINE
2220 stat("/etc/fstab", &s);
2222 tracepoint(my_provider, my_tracepoint, 23, "Hello, World!", &s);
2228 When viewing the trace, assuming the file size of path:{/etc/fstab} is
2229 301{nbsp}bytes, the event generated by the execution of this tracepoint
2230 should have the following fields, in this order:
2233 my_constant_field 40
2235 my_int_arg_field2 529
2237 my_str_arg_field "Hello, World!"
2239 size_dbl_field 301.0
2240 half_my_str_arg_field "Hello,"
2244 [[building-tracepoint-providers-and-user-application]]
2245 ===== Building/linking tracepoint providers and the user application
2247 The final step of using LTTng-UST for tracing a user space C application
2248 (beside running the application) is building and linking tracepoint
2249 providers and the application itself.
2251 As discussed above, the macros used by the user-written tracepoint provider
2252 header file are useless until actually used to create probes code
2253 (global data structures and functions) in a translation unit (C source file).
2254 This is accomplished by defining `TRACEPOINT_CREATE_PROBES` in a translation
2255 unit and then including the tracepoint provider header file.
2256 When `TRACEPOINT_CREATE_PROBES` is defined, macros used and included by
2257 the tracepoint provider header will output actual source code needed by any
2258 application using the defined tracepoints. Defining
2259 `TRACEPOINT_CREATE_PROBES` produces code used when registering
2260 tracepoint providers when the tracepoint provider package loads.
2262 The other important definition is `TRACEPOINT_DEFINE`. This one creates
2263 global, per-tracepoint structures referencing the tracepoint providers
2264 data. Those structures are required by the actual functions inserted
2265 where `tracepoint()` macros are placed and need to be defined by the
2266 instrumented application.
2268 Both `TRACEPOINT_CREATE_PROBES` and `TRACEPOINT_DEFINE` need to be defined
2269 at some places in order to trace a user space C application using LTTng.
2270 Although explaining their exact mechanism is beyond the scope of this
2271 document, the reason they both exist separately is to allow the trace
2272 providers to be packaged as a shared object (dynamically loaded library).
2274 There are two ways to compile and link the tracepoint providers
2275 with the application: _<<static-linking,statically>>_ or
2276 _<<dynamic-linking,dynamically>>_. Both methods are covered in the
2277 following subsections.
2281 ===== Static linking the tracepoint providers to the application
2283 With the static linking method, compiled tracepoint providers are copied
2284 into the target application. There are three ways to do this:
2286 . Use one of your **existing C source files** to create probes.
2287 . Create probes in a separate C source file and build it as an
2288 **object file** to be linked with the application (more decoupled).
2289 . Create probes in a separate C source file, build it as an
2290 object file and archive it to create a **static library**
2291 (more decoupled, more portable).
2293 The first approach is to define `TRACEPOINT_CREATE_PROBES` and include
2294 your tracepoint provider(s) header file(s) directly into an existing C
2295 source file. Here's an example:
2303 #define TRACEPOINT_CREATE_PROBES
2304 #define TRACEPOINT_DEFINE
2309 int my_func(int a, const char* b)
2313 tracepoint(my_provider, my_tracepoint, buf, sz, limit, &tt)
2321 Again, before including a given tracepoint provider header file,
2322 `TRACEPOINT_CREATE_PROBES` and `TRACEPOINT_DEFINE` must be defined in
2323 one, **and only one**, translation unit. Other C source files of the
2324 same application may include path:{tp.h} to use tracepoints with
2325 the `tracepoint()` macro, but must not define
2326 `TRACEPOINT_CREATE_PROBES`/`TRACEPOINT_DEFINE` again.
2328 This translation unit may be built as an object file by making sure to
2329 add `.` to the include path:
2336 The second approach is to isolate the tracepoint provider code into a
2337 separate object file by using a dedicated C source file to create probes:
2341 #define TRACEPOINT_CREATE_PROBES
2346 `TRACEPOINT_DEFINE` must be defined by a translation unit of the
2347 application. Since we're talking about static linking here, it could as
2348 well be defined directly in the file above, before `#include "tp.h"`:
2352 #define TRACEPOINT_CREATE_PROBES
2353 #define TRACEPOINT_DEFINE
2358 This is actually what <<lttng-gen-tp,`lttng-gen-tp`>> does, and is
2359 the recommended practice.
2361 Build the tracepoint provider:
2368 Finally, the resulting object file may be archived to create a
2369 more portable tracepoint provider static library:
2376 Using a static library does have the advantage of centralising the
2377 tracepoint providers objects so they can be shared between multiple
2378 applications. This way, when the tracepoint provider is modified, the
2379 source code changes don't have to be patched into each application's source
2380 code tree. The applications need to be relinked after each change, but need
2381 not to be otherwise recompiled (unless the tracepoint provider's API
2384 Regardless of which method you choose, you end up with an object file
2385 (potentially archived) containing the trace providers assembled code.
2386 To link this code with the rest of your application, you must also link
2387 with `liblttng-ust` and `libdl`:
2391 gcc -o app tp.o other.o files.o of.o your.o app.o -llttng-ust -ldl
2398 gcc -o app tp.a other.o files.o of.o your.o app.o -llttng-ust -ldl
2401 If you're using a BSD
2402 system, replace `-ldl` with `-lc`:
2406 gcc -o app tp.a other.o files.o of.o your.o app.o -llttng-ust -lc
2409 The application can be started as usual, e.g.:
2416 The `lttng` command line tool can be used to
2417 <<controlling-tracing,control tracing>>.
2421 ===== Dynamic linking the tracepoint providers to the application
2423 The second approach to package the tracepoint providers is to use
2424 dynamic linking: the library and its member functions are explicitly
2425 sought, loaded and unloaded at runtime using `libdl`.
2427 It has to be noted that, for a variety of reasons, the created shared
2428 library will be dynamically _loaded_, as opposed to dynamically
2429 _linked_. The tracepoint provider shared object is, however, linked
2430 with `liblttng-ust`, so that `liblttng-ust` is guaranteed to be loaded
2431 as soon as the tracepoint provider is. If the tracepoint provider is
2432 not loaded, since the application itself is not linked with
2433 `liblttng-ust`, the latter is not loaded at all and the tracepoint calls
2436 The process to create the tracepoint provider shared object is pretty
2437 much the same as the static library method, except that:
2439 * since the tracepoint provider is not part of the application
2440 anymore, `TRACEPOINT_DEFINE` _must_ be defined, for each tracepoint
2441 provider, in exactly one translation unit (C source file) of the
2443 * `TRACEPOINT_PROBE_DYNAMIC_LINKAGE` must be defined next to
2444 `TRACEPOINT_DEFINE`.
2446 Regarding `TRACEPOINT_DEFINE` and `TRACEPOINT_PROBE_DYNAMIC_LINKAGE`,
2447 the recommended practice is to use a separate C source file in your
2448 application to define them, and then include the tracepoint provider
2449 header files afterwards, e.g.:
2453 #define TRACEPOINT_DEFINE
2454 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
2456 /* include the header files of one or more tracepoint providers below */
2462 `TRACEPOINT_PROBE_DYNAMIC_LINKAGE` makes the macros included afterwards
2463 (by including the tracepoint provider header, which itself includes
2464 LTTng-UST headers) aware that the tracepoint provider is to be loaded
2465 dynamically and not part of the application's executable.
2467 The tracepoint provider object file used to create the shared library
2468 is built like it is using the static library method, only with the
2469 `-fpic` option added:
2473 gcc -c -fpic -I. tp.c
2476 It is then linked as a shared library like this:
2480 gcc -shared -Wl,--no-as-needed -o tp.so -llttng-ust tp.o
2483 As previously stated, this tracepoint provider shared object isn't
2484 linked with the user application: it will be loaded manually. This is
2485 why the application is built with no mention of this tracepoint
2486 provider, but still needs `libdl`:
2490 gcc -o app other.o files.o of.o your.o app.o -ldl
2493 Now, to make LTTng-UST tracing available to the application, the
2494 `LD_PRELOAD` environment variable is used to preload the tracepoint
2495 provider shared library _before_ the application actually starts:
2499 LD_PRELOAD=/path/to/tp.so ./app
2504 It is not safe to use
2505 `dlclose()` on a tracepoint provider shared object that
2506 is being actively used for tracing, due to a lack of reference
2507 counting from LTTng-UST to the shared object.
2509 For example, statically linking a tracepoint provider to a
2510 shared object which is to be dynamically loaded by an application
2511 (e.g., a plugin) is not safe: the shared object, which contains the
2512 tracepoint provider, could be dynamically closed
2513 (`dlclose()`) at any time by the application.
2515 To instrument a shared object, either:
2517 * Statically link the tracepoint provider to the _application_, or
2518 * Build the tracepoint provider as a shared object (following
2519 the procedure shown in this section), and preload it when
2520 tracing is needed using the `LD_PRELOAD`
2521 environment variable.
2524 Your application will still work without this preloading, albeit without
2525 LTTng-UST tracing support:
2533 [[using-lttng-ust-with-daemons]]
2534 ===== Using LTTng-UST with daemons
2536 Some extra care is needed when using `liblttng-ust` with daemon
2537 applications that call `fork()`, `clone()` or BSD's `rfork()` without
2538 a following `exec()` family system call. The `liblttng-ust-fork`
2539 library must be preloaded for the application.
2545 LD_PRELOAD=liblttng-ust-fork.so ./app
2548 Or, if you're using a tracepoint provider shared library:
2552 LD_PRELOAD="liblttng-ust-fork.so /path/to/tp.so" ./app
2556 [[lttng-ust-pkg-config]]
2557 ===== Using pkg-config
2559 On some distributions, LTTng-UST is shipped with a pkg-config metadata
2560 file, so that you may use the `pkg-config` tool:
2564 pkg-config --libs lttng-ust
2567 This will return `-llttng-ust -ldl` on Linux systems.
2569 You may also check the LTTng-UST version using `pkg-config`:
2573 pkg-config --modversion lttng-ust
2576 For more information about pkg-config, see
2577 http://linux.die.net/man/1/pkg-config[its manpage].
2581 ===== Using `tracef()`
2583 `tracef()` is a small LTTng-UST API to avoid defining your own
2584 tracepoints and tracepoint providers. The signature of `tracef()` is
2585 the same as `printf()`'s.
2587 The `tracef()` utility function was developed to make user space tracing
2588 super simple, albeit with notable disadvantages compared to custom,
2589 full-fledged tracepoint providers:
2591 * All generated events have the same provider/event names, respectively
2592 `lttng_ust_tracef` and `event`.
2593 * There's no static type checking.
2594 * The only event field you actually get, named `msg`, is a string
2595 potentially containing the values you passed to the function
2596 using your own format. This also means that you cannot use filtering
2597 using a custom expression at runtime because there are no isolated
2599 * Since `tracef()` uses C standard library's `vasprintf()` function
2600 in the background to format the strings at runtime, its
2601 expected performance is lower than using custom tracepoint providers
2602 with typed fields, which do not require a conversion to a string.
2604 Thus, `tracef()` is useful for quick prototyping and debugging, but
2605 should not be considered for any permanent/serious application
2608 To use `tracef()`, first include `<lttng/tracef.h>` in the C source file
2609 where you need to insert probes:
2613 #include <lttng/tracef.h>
2616 Use `tracef()` like you would use `printf()` in your source code, e.g.:
2622 tracef("my message, my integer: %d", my_integer);
2627 Link your application with `liblttng-ust`:
2631 gcc -o app app.c -llttng-ust
2634 Execute the application as usual:
2641 Voilà ! Use the `lttng` command line tool to
2642 <<controlling-tracing,control tracing>>. You can enable `tracef()`
2647 lttng enable-event --userspace 'lttng_ust_tracef:*'
2651 [[lttng-ust-environment-variables-compiler-flags]]
2652 ===== LTTng-UST environment variables and special compilation flags
2654 A few special environment variables and compile flags may affect the
2655 behavior of LTTng-UST.
2657 LTTng-UST's debugging can be activated by setting the environment
2658 variable `LTTNG_UST_DEBUG` to `1` when launching the application. It
2659 can also be enabled at compile time by defining `LTTNG_UST_DEBUG` when
2660 compiling LTTng-UST (using the `-DLTTNG_UST_DEBUG` compiler option).
2662 The environment variable `LTTNG_UST_REGISTER_TIMEOUT` can be used to
2663 specify how long the application should wait for the
2664 <<lttng-sessiond,session daemon>>'s _registration done_ command
2665 before proceeding to execute the main program. The timeout value is
2666 specified in milliseconds. 0 means _don't wait_. -1 means
2667 _wait forever_. Setting this environment variable to 0 is recommended
2668 for applications with time contraints on the process startup time.
2670 The default value of `LTTNG_UST_REGISTER_TIMEOUT` (when not defined)
2671 is **3000{nbsp}ms**.
2673 The compilation definition `LTTNG_UST_DEBUG_VALGRIND` should be enabled
2674 at build time (`-DLTTNG_UST_DEBUG_VALGRIND`) to allow `liblttng-ust`
2675 to be used with http://valgrind.org/[Valgrind].
2676 The side effect of defining `LTTNG_UST_DEBUG_VALGRIND` is that per-CPU
2677 buffering is disabled.
2681 ==== $$C++$$ application
2683 Because of $$C++$$'s cross-compatibility with the C language, $$C++$$
2684 applications can be readily instrumented with the LTTng-UST C API.
2686 Follow the <<c-application,C application>> user guide above. It
2687 should be noted that, in this case, tracepoint providers should have
2688 the typical `.cpp`, `.cxx` or `.cc` extension and be built with `g++`
2689 instead of `gcc`. This is the easiest way of avoiding linking errors
2690 due to symbol name mangling incompatibilities between both languages.
2693 [[prebuilt-ust-helpers]]
2694 ==== Prebuilt user space tracing helpers
2696 The LTTng-UST package provides a few helpers that one may find
2697 useful in some situations. They all work the same way: you must
2698 preload the appropriate shared object before running the user
2699 application (using the `LD_PRELOAD` environment variable).
2701 The shared objects are normally found in dir:{/usr/lib}.
2703 The current installed helpers are:
2705 path:{liblttng-ust-libc-wrapper.so} and path:{liblttng-ust-pthread-wrapper.so}::
2706 <<liblttng-ust-libc-pthread-wrapper,C{nbsp}standard library
2707 and POSIX threads tracing>>.
2709 path:{liblttng-ust-cyg-profile.so} and path:{liblttng-ust-cyg-profile-fast.so}::
2710 <<liblttng-ust-cyg-profile,Function tracing>>.
2712 path:{liblttng-ust-dl.so}::
2713 <<liblttng-ust-dl,Dynamic linker tracing>>.
2715 The following subsections document what helpers instrument exactly
2716 and how to use them.
2719 [[liblttng-ust-libc-pthread-wrapper]]
2720 ===== C standard library and POSIX threads tracing
2722 path:{liblttng-ust-libc-wrapper.so} and path:{liblttng-ust-pthread-wrapper.so}
2723 can add instrumentation to respectively some C standard library and
2724 POSIX threads functions.
2726 The following functions are traceable by path:{liblttng-ust-libc-wrapper.so}:
2729 .Functions instrumented by path:{liblttng-ust-libc-wrapper.so}
2731 |TP provider name |TP name |Instrumented function
2733 .6+|`ust_libc` |`malloc` |`malloc()`
2734 |`calloc` |`calloc()`
2735 |`realloc` |`realloc()`
2737 |`memalign` |`memalign()`
2738 |`posix_memalign` |`posix_memalign()`
2741 The following functions are traceable by
2742 path:{liblttng-ust-pthread-wrapper.so}:
2745 .Functions instrumented by path:{liblttng-ust-pthread-wrapper.so}
2747 |TP provider name |TP name |Instrumented function
2749 .4+|`ust_pthread` |`pthread_mutex_lock_req` |`pthread_mutex_lock()` (request time)
2750 |`pthread_mutex_lock_acq` |`pthread_mutex_lock()` (acquire time)
2751 |`pthread_mutex_trylock` |`pthread_mutex_trylock()`
2752 |`pthread_mutex_unlock` |`pthread_mutex_unlock()`
2755 All tracepoints have fields corresponding to the arguments of the
2756 function they instrument.
2758 To use one or the other with any user application, independently of
2759 how the latter is built, do:
2763 LD_PRELOAD=liblttng-ust-libc-wrapper.so my-app
2770 LD_PRELOAD=liblttng-ust-pthread-wrapper.so my-app
2777 LD_PRELOAD="liblttng-ust-libc-wrapper.so liblttng-ust-pthread-wrapper.so" my-app
2780 When the shared object is preloaded, it effectively replaces the
2781 functions listed in the above tables by wrappers which add tracepoints
2782 and call the replaced functions.
2784 Of course, like any other tracepoint, the ones above need to be enabled
2785 in order for LTTng-UST to generate events. This is done using the
2786 `lttng` command line tool
2787 (see <<controlling-tracing,Controlling tracing>>).
2790 [[liblttng-ust-cyg-profile]]
2791 ===== Function tracing
2793 Function tracing is the recording of which functions are entered and
2794 left during the execution of an application. Like with any LTTng event,
2795 the precise time at which this happens is also kept.
2797 GCC and clang have an option named
2798 https://gcc.gnu.org/onlinedocs/gcc-4.9.1/gcc/Code-Gen-Options.html[`-finstrument-functions`]
2799 which generates instrumentation calls for entry and exit to functions.
2800 The LTTng-UST function tracing helpers, path:{liblttng-ust-cyg-profile.so}
2801 and path:{liblttng-ust-cyg-profile-fast.so}, take advantage of this feature
2802 to add instrumentation to the two generated functions (which contain
2803 `cyg_profile` in their names, hence the shared object's name).
2805 In order to use LTTng-UST function tracing, the translation units to
2806 instrument must be built using the `-finstrument-functions` compiler
2809 LTTng-UST function tracing comes in two flavors, each providing
2810 different trade-offs: path:{liblttng-ust-cyg-profile-fast.so} and
2811 path:{liblttng-ust-cyg-profile.so}.
2813 **path:{liblttng-ust-cyg-profile-fast.so}** is a lightweight variant that
2814 should only be used where it can be _guaranteed_ that the complete event
2815 stream is recorded without any missing events. Any kind of duplicate
2816 information is left out. This version registers the following
2819 [role="growable",options="header,autowidth"]
2820 .Functions instrumented by path:{liblttng-ust-cyg-profile-fast.so}
2822 |TP provider name |TP name |Instrumented function
2824 .2+|`lttng_ust_cyg_profile_fast`
2830 Address of called function.
2836 Assuming no event is lost, having only the function addresses on entry
2837 is enough for creating a call graph (remember that a recorded event
2838 always contains the ID of the CPU that generated it). A tool like
2839 https://sourceware.org/binutils/docs/binutils/addr2line.html[`addr2line`]
2840 may be used to convert function addresses back to source files names
2844 **path:{liblttng-ust-cyg-profile.so}**,
2845 is a more robust variant which also works for use cases where
2846 events might get discarded or not recorded from application startup.
2847 In these cases, the trace analyzer needs extra information to be
2848 able to reconstruct the program flow. This version registers the
2849 following tracepoints:
2851 [role="growable",options="header,autowidth"]
2852 .Functions instrumented by path:{liblttng-ust-cyg-profile.so}
2854 |TP provider name |TP name |Instrumented function
2856 .2+|`lttng_ust_cyg_profile`
2862 Address of called function.
2871 Address of called function.
2877 To use one or the other variant with any user application, assuming at
2878 least one translation unit of the latter is compiled with the
2879 `-finstrument-functions` option, do:
2883 LD_PRELOAD=liblttng-ust-cyg-profile-fast.so my-app
2890 LD_PRELOAD=liblttng-ust-cyg-profile.so my-app
2893 It might be necessary to limit the number of source files where
2894 `-finstrument-functions` is used to prevent excessive amount of trace
2895 data to be generated at runtime.
2897 TIP: When using GCC, at least, you can use
2898 the `-finstrument-functions-exclude-function-list`
2899 option to avoid instrumenting entries and exits of specific
2902 All events generated from LTTng-UST function tracing are provided on
2903 log level `TRACE_DEBUG_FUNCTION`, which is useful to easily enable
2904 function tracing events in your tracing session using the
2905 `--loglevel-only` option of `lttng enable-event`
2906 (see <<controlling-tracing,Controlling tracing>>).
2910 ===== Dynamic linker tracing
2912 This LTTng-UST helper causes all calls to `dlopen()` and `dlclose()`
2913 in the target application to be traced with LTTng.
2915 The helper's shared object, path:{liblttng-ust-dl.so}, registers the
2916 following tracepoints when preloaded:
2918 [role="growable",options="header,autowidth"]
2919 .Functions instrumented by path:{liblttng-ust-dl.so}
2921 |TP provider name |TP name |Instrumented function
2929 Memory base address (where the dynamic linker placed the shared
2933 File system path to the loaded shared object.
2936 File size of the the loaded shared object.
2939 Last modification time (seconds since Epoch time) of the loaded shared
2946 Memory base address (where the dynamic linker placed the shared
2950 To use this LTTng-UST helper with any user application, independently of
2951 how the latter is built, do:
2955 LD_PRELOAD=liblttng-ust-dl.so my-app
2958 Of course, like any other tracepoint, the ones above need to be enabled
2959 in order for LTTng-UST to generate events. This is done using the
2960 `lttng` command line tool
2961 (see <<controlling-tracing,Controlling tracing>>).
2964 [[java-application]]
2965 ==== Java application
2967 LTTng-UST provides a _logging_ back-end for Java applications using
2968 http://docs.oracle.com/javase/7/docs/api/java/util/logging/Logger.html[`java.util.logging`]
2969 (JUL). This back-end is called the _LTTng-UST JUL agent_ and is
2970 responsible for communications with an LTTng session daemon.
2972 From the user's point of view, once the LTTng-UST JUL agent has been
2973 initialized, JUL loggers may be created and used as usual. The agent
2974 adds its own handler to the _root logger_, so that all loggers may
2975 generate LTTng events with no effort.
2977 Common JUL features are supported using the `lttng` tool
2978 (see <<controlling-tracing,Controlling tracing>>):
2980 * listing all logger names
2981 * enabling/disabling events per logger name
2988 import java.util.logging.Logger;
2989 import org.lttng.ust.jul.LTTngAgent;
2993 public static void main(String[] argv) throws Exception
2996 Logger logger = Logger.getLogger("jello");
2998 // call this as soon as possible (before logging)
2999 LTTngAgent lttngAgent = LTTngAgent.getLTTngAgent();
3002 logger.info("some info");
3003 logger.warning("some warning");
3005 logger.finer("finer information...");
3007 logger.severe("error!");
3009 // not mandatory, but cleaner
3010 lttngAgent.dispose();
3015 The LTTng-UST JUL agent Java classes are packaged in a JAR file named
3016 path:{liblttng-ust-jul.jar}. It is typically located in
3017 dir:{/usr/lib/lttng/java}. To compile the snippet above
3018 (saved as path:{Test.java}), do:
3022 javac -cp /usr/lib/lttng/java/liblttng-ust-jul.jar Test.java
3025 You can run the resulting compiled class:
3029 java -cp /usr/lib/lttng/java/liblttng-ust-jul.jar:. Test
3032 NOTE: http://openjdk.java.net/[OpenJDK] 7 is used for development and
3033 continuous integration, thus this version is directly supported.
3034 However, the LTTng-UST JUL agent has also been tested with OpenJDK 6.
3037 [[instrumenting-linux-kernel]]
3040 The Linux kernel can be instrumented for LTTng tracing, either its core
3041 source code or a kernel module. It has to be noted that Linux is
3042 readily traceable using LTTng since many parts of its source code are
3043 already instrumented: this is the job of the upstream
3044 http://git.lttng.org/?p=lttng-modules.git[LTTng-modules]
3045 package. This section presents how to add LTTng instrumentation where it
3046 does not currently exist and how to instrument custom kernel modules.
3048 All LTTng instrumentation in the Linux kernel is based on an existing
3049 infrastructure which bears the name of its main macro, `TRACE_EVENT()`.
3050 This macro is used to define tracepoints,
3051 each tracepoint having a name, usually with the
3052 +__subsys_____name__+ format,
3053 +_subsys_+ being the subsystem name and
3054 +_name_+ the specific event name.
3056 Tracepoints defined with `TRACE_EVENT()` may be inserted anywhere in
3057 the Linux kernel source code, after what callbacks, called _probes_,
3058 may be registered to execute some action when a tracepoint is
3059 executed. This mechanism is directly used by ftrace and perf,
3060 but cannot be used as is by LTTng: an adaptation layer is added to
3061 satisfy LTTng's specific needs.
3063 With that in mind, this documentation does not cover the `TRACE_EVENT()`
3064 format and how to use it, but it is mandatory to understand it and use
3065 it to instrument Linux for LTTng. A series of
3066 LWN articles explain
3067 `TRACE_EVENT()` in details:
3068 http://lwn.net/Articles/379903/[part 1],
3069 http://lwn.net/Articles/381064/[part 2], and
3070 http://lwn.net/Articles/383362/[part 3].
3071 Once you master `TRACE_EVENT()` enough for your use case, continue
3072 reading this section so that you can add the LTTng adaptation layer of
3075 This section first discusses the general method of instrumenting the
3076 Linux kernel for LTTng. This method is then reused for the specific
3077 case of instrumenting a kernel module.
3080 [[instrumenting-linux-kernel-itself]]
3081 ===== Instrumenting the Linux kernel for LTTng
3083 The following subsections explain strictly how to add custom LTTng
3084 instrumentation to the Linux kernel. They do not explain how the
3085 macros actually work and the internal mechanics of the tracer.
3087 You should have a Linux kernel source code tree to work with.
3088 Throughout this section, all file paths are relative to the root of
3089 this tree unless otherwise stated.
3091 You will need a copy of the LTTng-modules Git repository:
3095 git clone git://git.lttng.org/lttng-modules.git
3098 The steps to add custom LTTng instrumentation to a Linux kernel
3099 involves defining and using the mainline `TRACE_EVENT()` tracepoints
3100 first, then writing and using the LTTng adaptation layer.
3103 [[mainline-trace-event]]
3104 ===== Defining/using tracepoints with mainline `TRACE_EVENT()` infrastructure
3106 The first step is to define tracepoints using the mainline Linux
3107 `TRACE_EVENT()` macro and insert tracepoints where you want them.
3108 Your tracepoint definitions reside in a header file in
3109 dir:{include/trace/events}. If you're adding tracepoints to an existing
3110 subsystem, edit its appropriate header file.
3112 As an example, the following header file (let's call it
3113 path:{include/trace/events/hello.h}) defines one tracepoint using
3118 /* subsystem name is "hello" */
3120 #define TRACE_SYSTEM hello
3122 #if !defined(_TRACE_HELLO_H) || defined(TRACE_HEADER_MULTI_READ)
3123 #define _TRACE_HELLO_H
3125 #include <linux/tracepoint.h>
3128 /* "hello" is the subsystem name, "world" is the event name */
3131 /* tracepoint function prototype */
3132 TP_PROTO(int foo, const char* bar),
3134 /* arguments for this tracepoint */
3137 /* LTTng doesn't need those */
3145 /* this part must be outside protection */
3146 #include <trace/define_trace.h>
3149 Notice that we don't use any of the last three arguments: they
3150 are left empty here because LTTng doesn't need them. You would only fill
3151 `TP_STRUCT__entry()`, `TP_fast_assign()` and `TP_printk()` if you were
3152 to also use this tracepoint for ftrace/perf.
3154 Once this is done, you may place calls to `trace_hello_world()`
3155 wherever you want in the Linux source code. As an example, let us place
3156 such a tracepoint in the `usb_probe_device()` static function
3157 (path:{drivers/usb/core/driver.c}):
3161 /* called from driver core with dev locked */
3162 static int usb_probe_device(struct device *dev)
3164 struct usb_device_driver *udriver = to_usb_device_driver(dev->driver);
3165 struct usb_device *udev = to_usb_device(dev);
3168 trace_hello_world(udev->devnum, udev->product);
3174 This tracepoint should fire every time a USB device is plugged in.
3176 At the top of path:{driver.c}, we need to include our actual tracepoint
3177 definition and, in this case (one place per subsystem), define
3178 `CREATE_TRACE_POINTS`, which will create our tracepoint:
3186 #define CREATE_TRACE_POINTS
3187 #include <trace/events/hello.h>
3192 Build your custom Linux kernel. In order to use LTTng, make sure the
3193 following kernel configuration options are enabled:
3195 * `CONFIG_MODULES` (loadable module support)
3196 * `CONFIG_KALLSYMS` (load all symbols for debugging/kksymoops)
3197 * `CONFIG_HIGH_RES_TIMERS` (high resolution timer support)
3198 * `CONFIG_TRACEPOINTS` (kernel tracepoint instrumentation)
3200 Boot the custom kernel. The directory
3201 dir:{/sys/kernel/debug/tracing/events/hello} should exist if everything
3202 went right, with a dir:{hello_world} subdirectory.
3205 [[lttng-adaptation-layer]]
3206 ===== Adding the LTTng adaptation layer
3208 The steps to write the LTTng adaptation layer are, in your
3209 LTTng-modules copy's source code tree:
3211 . In dir:{instrumentation/events/lttng-module},
3212 add a header +__subsys__.h+ for your custom
3213 subsystem +__subsys__+ and write your
3214 tracepoint definitions using LTTng-modules macros in it.
3215 Those macros look like the mainline kernel equivalents,
3216 but they present subtle, yet important differences.
3217 . In dir:{probes}, create the C source file of the LTTng probe kernel
3218 module for your subsystem. It should be named
3219 +lttng-probe-__subsys__.c+.
3220 . Edit path:{probes/Makefile} so that the LTTng-modules project
3221 builds your custom LTTng probe kernel module.
3222 . Build and install LTTng kernel modules.
3224 Following our `hello_world` event example, here's the content of
3225 path:{instrumentation/events/lttng-module/hello.h}:
3230 #define TRACE_SYSTEM hello
3232 #if !defined(_TRACE_HELLO_H) || defined(TRACE_HEADER_MULTI_READ)
3233 #define _TRACE_HELLO_H
3235 #include <linux/tracepoint.h>
3237 LTTNG_TRACEPOINT_EVENT(
3238 /* format identical to mainline version for those */
3240 TP_PROTO(int foo, const char* bar),
3243 /* possible differences */
3245 __field(int, my_int)
3246 __field(char, char0)
3247 __field(char, char1)
3248 __string(product, bar)
3251 /* notice the use of tp_assign()/tp_strcpy() and no semicolons */
3253 tp_assign(my_int, foo)
3254 tp_assign(char0, bar[0])
3255 tp_assign(char1, bar[1])
3256 tp_strcpy(product, bar)
3259 /* This one is actually not used by LTTng either, but must be
3260 * present for the moment.
3264 /* no semicolon after this either */
3269 /* other difference: do NOT include <trace/define_trace.h> */
3270 #include "../../../probes/define_trace.h"
3273 Some possible entries for `TP_STRUCT__entry()` and `TP_fast_assign()`,
3274 in the case of LTTng-modules, are shown in the
3275 <<lttng-modules-ref,LTTng-modules reference>> section.
3277 The best way to learn how to use the above macros is to inspect
3278 existing LTTng tracepoint definitions in
3279 dir:{instrumentation/events/lttng-module} header files. Compare
3280 them with the Linux kernel mainline versions in
3281 dir:{include/trace/events}.
3283 The next step is writing the LTTng probe kernel module C source file.
3284 This one is named +lttng-probe-__subsys__.c+
3285 in dir:{probes}. You may always use the following template:
3289 #include <linux/module.h>
3290 #include "../lttng-tracer.h"
3292 /* Build time verification of mismatch between mainline TRACE_EVENT()
3293 * arguments and LTTng adaptation layer LTTNG_TRACEPOINT_EVENT() arguments.
3295 #include <trace/events/hello.h>
3297 /* create LTTng tracepoint probes */
3298 #define LTTNG_PACKAGE_BUILD
3299 #define CREATE_TRACE_POINTS
3300 #define TRACE_INCLUDE_PATH ../instrumentation/events/lttng-module
3302 #include "../instrumentation/events/lttng-module/hello.h"
3304 MODULE_LICENSE("GPL and additional rights");
3305 MODULE_AUTHOR("Your name <your-email>");
3306 MODULE_DESCRIPTION("LTTng hello probes");
3307 MODULE_VERSION(__stringify(LTTNG_MODULES_MAJOR_VERSION) "."
3308 __stringify(LTTNG_MODULES_MINOR_VERSION) "."
3309 __stringify(LTTNG_MODULES_PATCHLEVEL_VERSION)
3310 LTTNG_MODULES_EXTRAVERSION);
3313 Just replace `hello` with your subsystem name. In this example,
3314 `<trace/events/hello.h>`, which is the original mainline tracepoint
3315 definition header, is included for verification purposes: the
3316 LTTng-modules build system is able to emit an error at build time when
3317 the arguments of the mainline `TRACE_EVENT()` definitions do not match
3318 the ones of the LTTng-modules adaptation layer
3319 (`LTTNG_TRACEPOINT_EVENT()`).
3321 Edit path:{probes/Makefile} and add your new kernel module object
3322 next to existing ones:
3328 obj-m += lttng-probe-module.o
3329 obj-m += lttng-probe-power.o
3331 obj-m += lttng-probe-hello.o
3336 Time to build! Point to your custom Linux kernel source tree using
3337 the `KERNELDIR` variable:
3341 make KERNELDIR=/path/to/custom/linux
3344 Finally, install modules:
3348 sudo make modules_install
3352 [[instrumenting-linux-kernel-tracing]]
3355 The <<controlling-tracing,Controlling tracing>> section explains
3356 how to use the `lttng` tool to create and control tracing sessions.
3357 Although the `lttng` tool will load the appropriate _known_ LTTng kernel
3358 modules when needed (by launching `root`'s session daemon), it won't
3359 load your custom `lttng-probe-hello` module by default. You need to
3360 manually load the `lttng-probe-hello` module, and start an LTTng session
3365 sudo pkill -u root lttng-sessiond
3366 sudo modprobe lttng_probe_hello
3370 The first command makes sure any existing instance is killed. If
3371 you're not interested in using the default probes, or if you only
3372 want to use a few of them, you can use the `--kmod-probes` option
3373 of `lttng-sessiond` instead, which specifies an absolute list of
3374 probes to load (without the `lttng-probe-` prefix):
3378 sudo lttng-sessiond --kmod-probes=hello,ext4,net,block,signal,sched
3381 Confirm the custom probe module is loaded:
3385 lsmod | grep lttng_probe_hello
3388 The `hello_world` event should appear in the list when doing
3392 lttng list --kernel | grep hello
3395 You may now create an LTTng tracing session, enable the `hello_world`
3396 kernel event (and others if you wish) and start tracing:
3400 sudo lttng create my-session
3401 sudo lttng enable-event --kernel hello_world
3405 Plug a few USB devices, then stop tracing and inspect the trace (if
3406 http://diamon.org/babeltrace[Babeltrace]
3415 Here's a sample output:
3418 [15:30:34.835895035] (+?.?????????) hostname hello_world: { cpu_id = 1 }, { my_int = 8, char0 = 68, char1 = 97, product = "DataTraveler 2.0" }
3419 [15:30:42.262781421] (+7.426886386) hostname hello_world: { cpu_id = 1 }, { my_int = 9, char0 = 80, char1 = 97, product = "Patriot Memory" }
3420 [15:30:48.175621778] (+5.912840357) hostname hello_world: { cpu_id = 1 }, { my_int = 10, char0 = 68, char1 = 97, product = "DataTraveler 2.0" }
3423 Two USB flash drives were used for this test.
3425 You may change your LTTng custom probe, rebuild it and reload it at
3426 any time when not tracing. Make sure you remove the old module
3427 (either by killing the root LTTng session daemon which loaded the
3428 module in the first place (if you used `--kmod-probes`), or by
3429 using `modprobe --remove` directly) before loading the updated one.
3432 [[instrumenting-out-of-tree-linux-kernel]]
3433 ===== Advanced: Instrumenting an out-of-tree Linux kernel module for LTTng
3435 Instrumenting a custom Linux kernel module for LTTng follows the exact
3437 <<instrumenting-linux-kernel-itself,adding instrumentation
3438 to the Linux kernel itself>>,
3439 the only difference being that your mainline tracepoint definition
3440 header doesn't reside in the mainline source tree, but in your
3441 kernel module source tree.
3443 The only reference to this mainline header is in the LTTng custom
3444 probe's source code (path:{probes/lttng-probe-hello.c} in our example),
3445 for build time verification:
3451 /* Build time verification of mismatch between mainline TRACE_EVENT()
3452 * arguments and LTTng adaptation layer LTTNG_TRACEPOINT_EVENT() arguments.
3454 #include <trace/events/hello.h>
3459 The preferred, flexible way to include your module's mainline
3460 tracepoint definition header is to put it in a specific directory
3461 relative to your module's root, e.g., dir:{tracepoints}, and include it
3462 relative to your module's root directory in the LTTng custom probe's
3467 #include <tracepoints/hello.h>
3470 You may then build LTTng-modules by adding your module's root
3471 directory as an include path to the extra C flags:
3475 make ccflags-y=-I/path/to/kernel/module KERNELDIR=/path/to/custom/linux
3478 Using `ccflags-y` allows you to move your kernel module to another
3479 directory and rebuild the LTTng-modules project with no change to
3483 [[proc-lttng-logger-abi]]
3484 ==== LTTng logger ABI
3486 The `lttng-tracer` Linux kernel module, installed by the LTTng-modules
3487 package, creates a special LTTng logger ABI file path:{/proc/lttng-logger}
3488 when loaded. Writing text data to this file generates an LTTng kernel
3489 domain event named `lttng_logger`.
3491 Unlike other kernel domain events, `lttng_logger` may be enabled by
3492 any user, not only root users or members of the tracing group.
3494 To use the LTTng logger ABI, simply write a string to
3495 path:{/proc/lttng-logger}:
3499 echo -n 'Hello, World!' > /proc/lttng-logger
3502 The `msg` field of the `lttng_logger` event contains the recorded
3505 NOTE: Messages are split in chunks of 1024{nbsp}bytes.
3507 The LTTng logger ABI is a quick and easy way to trace some events from
3508 user space through the kernel tracer. However, it is much more basic
3509 than LTTng-UST: it's slower (involves system call round-trip to the
3510 kernel and only supports logging strings). The LTTng logger ABI is
3511 particularly useful for recording logs as LTTng traces from shell
3512 scripts, potentially combining them with other Linux kernel/user space
3516 [[instrumenting-32-bit-app-on-64-bit-system]]
3517 ==== Advanced: Instrumenting a 32-bit application on a 64-bit system
3519 [[advanced-instrumenting-techniques]]In order to trace a 32-bit
3520 application running on a 64-bit system,
3521 LTTng must use a dedicated 32-bit
3522 <<lttng-consumerd,consumer daemon>>. This section discusses how to
3523 build that daemon (which is _not_ part of the default 64-bit LTTng
3524 build) and the LTTng 32-bit tracing libraries, and how to instrument
3525 a 32-bit application in that context.
3527 Make sure you install all 32-bit versions of LTTng dependencies.
3528 Their names can be found in the path:{README.md} files of each LTTng package
3529 source. How to find and install them will vary depending on your target
3530 Linux distribution. `gcc-multilib` is a common package name for the
3531 multilib version of GCC, which you will also need.
3533 The following packages will be built for 32-bit support on a 64-bit
3534 system: http://urcu.so/[Userspace RCU],
3535 LTTng-UST and LTTng-tools.
3538 [[building-32-bit-userspace-rcu]]
3539 ===== Building 32-bit Userspace RCU
3545 git clone git://git.urcu.so/urcu.git
3548 ./configure --libdir=/usr/lib32 CFLAGS=-m32
3554 The `-m32` C compiler flag creates 32-bit object files and `--libdir`
3555 indicates where to install the resulting libraries.
3558 [[building-32-bit-lttng-ust]]
3559 ===== Building 32-bit LTTng-UST
3565 git clone http://git.lttng.org/lttng-ust.git
3568 ./configure --prefix=/usr \
3569 --libdir=/usr/lib32 \
3570 CFLAGS=-m32 CXXFLAGS=-m32 \
3571 LDFLAGS=-L/usr/lib32
3577 `-L/usr/lib32` is required for the build to find the 32-bit versions
3578 of Userspace RCU and other dependencies.
3582 Depending on your Linux distribution,
3583 32-bit libraries could be installed at a different location than
3584 dir:{/usr/lib32}. For example, Debian is known to install
3585 some 32-bit libraries in dir:{/usr/lib/i386-linux-gnu}.
3587 In this case, make sure to set `LDFLAGS` to all the
3588 relevant 32-bit library paths, e.g.,
3589 `LDFLAGS="-L/usr/lib32 -L/usr/lib/i386-linux-gnu"`.
3592 NOTE: You may add options to path:{./configure} if you need them, e.g., for
3593 Java and SystemTap support. Look at `./configure --help` for more
3597 [[building-32-bit-lttng-tools]]
3598 ===== Building 32-bit LTTng-tools
3600 Since the host is a 64-bit system, most 32-bit binaries and libraries of
3601 LTTng-tools are not needed; the host will use their 64-bit counterparts.
3602 The required step here is building and installing a 32-bit consumer
3609 git clone http://git.lttng.org/lttng-tools.git
3612 ./configure --prefix=/usr \
3613 --libdir=/usr/lib32 CFLAGS=-m32 CXXFLAGS=-m32 \
3614 LDFLAGS=-L/usr/lib32
3616 cd src/bin/lttng-consumerd
3621 The above commands build all the LTTng-tools project as 32-bit
3622 applications, but only installs the 32-bit consumer daemon.
3625 [[building-64-bit-lttng-tools]]
3626 ===== Building 64-bit LTTng-tools
3628 Finally, you need to build a 64-bit version of LTTng-tools which is
3629 aware of the 32-bit consumer daemon previously built and installed:
3635 ./configure --prefix=/usr \
3636 --with-consumerd32-libdir=/usr/lib32 \
3637 --with-consumerd32-bin=/usr/lib32/lttng/libexec/lttng-consumerd
3643 Henceforth, the 64-bit session daemon will automatically find the
3644 32-bit consumer daemon if required.
3647 [[building-instrumented-32-bit-c-application]]
3648 ===== Building an instrumented 32-bit C application
3650 Let us reuse the _Hello world_ example of
3651 <<tracing-your-own-user-application,Tracing your own user application>>
3652 (<<getting-started,Getting started>> chapter).
3654 The instrumentation process is unaltered.
3656 First, a typical 64-bit build (assuming you're running a 64-bit system):
3660 gcc -o hello64 -I. hello.c hello-tp.c -ldl -llttng-ust
3663 Now, a 32-bit build:
3667 gcc -o hello32 -I. -m32 hello.c hello-tp.c -L/usr/lib32 \
3668 -ldl -llttng-ust -Wl,-rpath,/usr/lib32
3671 The `-rpath` option, passed to the linker, will make the dynamic loader
3672 check for libraries in dir:{/usr/lib32} before looking in its default paths,
3673 where it should find the 32-bit version of `liblttng-ust`.
3676 [[running-32-bit-and-64-bit-c-applications]]
3677 ===== Running 32-bit and 64-bit versions of an instrumented C application
3679 Now, both 32-bit and 64-bit versions of the _Hello world_ example above
3680 can be traced in the same tracing session. Use the `lttng` tool as usual
3681 to create a tracing session and start tracing:
3685 lttng create session-3264
3686 lttng enable-event -u -a
3692 Use `lttng view` to verify both processes were
3693 successfully traced.
3696 [[controlling-tracing]]
3697 === Controlling tracing
3699 Once you're in possession of a software that is properly
3700 <<instrumenting,instrumented>> for LTTng tracing, be it thanks to
3701 the built-in LTTng probes for the Linux kernel, a custom user
3702 application or a custom Linux kernel, all that is left is actually
3703 tracing it. As a user, you control LTTng tracing using a single command
3704 line interface: the `lttng` tool. This tool uses `liblttng-ctl` behind
3705 the scene to connect to and communicate with session daemons. LTTng
3706 session daemons may either be started manually (`lttng-sessiond`) or
3707 automatically by the `lttng` command when needed. Trace data may
3708 be forwarded to the network and used elsewhere using an LTTng relay
3709 daemon (`lttng-relayd`).
3711 The manpages of `lttng`, `lttng-sessiond` and `lttng-relayd` are pretty
3712 complete, thus this section is not an online copy of the latter (we
3713 leave this contents for the
3714 <<online-lttng-manpages,Online LTTng manpages>> section).
3715 This section is rather a tour of LTTng
3716 features through practical examples and tips.
3718 If not already done, make sure you understand the core concepts
3719 and how LTTng components connect together by reading the
3720 <<understanding-lttng,Understanding LTTng>> chapter; this section
3721 assumes you are familiar with them.
3724 [[creating-destroying-tracing-sessions]]
3725 ==== Creating and destroying tracing sessions
3727 Whatever you want to do with `lttng`, it has to happen inside a
3728 **tracing session**, created beforehand. A session, in general, is a
3729 per-user container of state. A tracing session is no different; it
3730 keeps a specific state of stuff like:
3733 * enabled/disabled channels with associated parameters
3734 * enabled/disabled events with associated log levels and filters
3735 * context information added to channels
3736 * tracing activity (started or stopped)
3740 A single user may have many active tracing sessions. LTTng session
3741 daemons are the ultimate owners and managers of tracing sessions. For
3742 user space tracing, each user has its own session daemon. Since Linux
3743 kernel tracing requires root privileges, only `root`'s session daemon
3744 may enable and trace kernel events. However, `lttng` has a `--group`
3745 option (which is passed to `lttng-sessiond` when starting it) to
3746 specify the name of a _tracing group_ which selected users may be part
3747 of to be allowed to communicate with `root`'s session daemon. By
3748 default, the tracing group name is `tracing`.
3750 To create a tracing session, do:
3754 lttng create my-session
3757 This will create a new tracing session named `my-session` and make it
3758 the current one. If you don't specify any name (calling only
3759 `lttng create`), your tracing session will be named `auto`. Traces
3760 are written in +\~/lttng-traces/__session__-+ followed
3761 by the tracing session's creation date/time by default, where
3762 +__session__+ is the tracing session name. To save them
3763 at a different location, use the `--output` option:
3767 lttng create --output /tmp/some-directory my-session
3770 You may create as many tracing sessions as you wish:
3774 lttng create other-session
3775 lttng create yet-another-session
3778 You may view all existing tracing sessions using the `list` command:
3785 The state of a _current tracing session_ is kept in path:{~/.lttngrc}. Each
3786 invocation of `lttng` reads this file to set its current tracing
3787 session name so that you don't have to specify a session name for each
3788 command. You could edit this file manually, but the preferred way to
3789 set the current tracing session is to use the `set-session` command:
3793 lttng set-session other-session
3796 Most `lttng` commands accept a `--session` option to specify the name
3797 of the target tracing session.
3799 Any existing tracing session may be destroyed using the `destroy`
3804 lttng destroy my-session
3807 Providing no argument to `lttng destroy` will destroy the current
3808 tracing session. Destroying a tracing session will stop any tracing
3809 running within the latter. Destroying a tracing session frees resources
3810 acquired by the session daemon and tracer side, making sure to flush
3813 You can't do much with LTTng using only the `create`, `set-session`
3814 and `destroy` commands of `lttng`, but it is essential to know them in
3815 order to control LTTng tracing, which always happen within the scope of
3819 [[enabling-disabling-events]]
3820 ==== Enabling and disabling events
3822 Inside a tracing session, individual events may be enabled or disabled
3823 so that tracing them may or may not generate trace data.
3825 We sometimes use the term _event_ metonymically throughout this text to
3826 refer to a specific condition, or _rule_, that could lead, when
3827 satisfied, to an actual occurring event (a point at a specific position
3828 in source code/binary program, logical processor and time capturing
3829 some payload) being recorded as trace data. This specific condition is
3832 . A **domain** (kernel, user space or `java.util.logging`) (required).
3833 . One or many **instrumentation points** in source code or binary
3834 program (tracepoint name, address, symbol name, function name,
3835 logger name, etc.) to be executed (required).
3836 . A **log level** (each instrumentation point declares its own log
3837 level) or log level range to match (optional; only valid for user
3839 . A **custom user expression**, or **filter**, that must evaluate to
3840 _true_ when a tracepoint is executed (optional; only valid for user
3843 All conditions are specified using arguments passed to the
3844 `enable-event` command of the `lttng` tool.
3846 Condition 1 is specified using either `--kernel/-k` (kernel),
3847 `--userspace/-u` (user space) or `--jul/-j`
3848 (JUL). Exactly one of those
3849 three arguments must be specified.
3851 Condition 2 is specified using one of:
3857 Dynamic probe (address, symbol name or combination
3858 of both in binary program; only valid for kernel domain).
3861 function entry/exit (address, symbol name or
3862 combination of both in binary program; only valid for kernel domain).
3865 System call entry/exit (only valid for kernel domain).
3867 When none of the above is specified, `enable-event` defaults to
3868 using `--tracepoint`.
3870 Condition 3 is specified using one of:
3873 Log level range from the specified level to the most severe
3879 See `lttng enable-event --help` for the complete list of log level
3882 Condition 4 is specified using the `--filter` option. This filter is
3883 a C-like expression, potentially reading real-time values of event
3884 fields, that has to evaluate to _true_ for the condition to be satisfied.
3885 Event fields are read using plain identifiers while context fields
3886 must be prefixed with `$ctx.`. See `lttng enable-event --help` for
3889 The aforementioned arguments are combined to create and enable events.
3890 Each unique combination of arguments leads to a different
3891 _enabled event_. The log level and filter arguments are optional, their
3892 default values being respectively all log levels and a filter which
3893 always returns _true_.
3895 Here are a few examples (you must
3896 <<creating-destroying-tracing-sessions,create a tracing session>>
3901 lttng enable-event -u --tracepoint my_app:hello_world
3902 lttng enable-event -u --tracepoint my_app:hello_you --loglevel TRACE_WARNING
3903 lttng enable-event -u --tracepoint 'my_other_app:*'
3904 lttng enable-event -u --tracepoint my_app:foo_bar \
3905 --filter 'some_field <= 23 && !other_field'
3906 lttng enable-event -k --tracepoint sched_switch
3907 lttng enable-event -k --tracepoint gpio_value
3908 lttng enable-event -k --function usb_probe_device usb_probe_device
3909 lttng enable-event -k --syscall --all
3912 The wildcard symbol, `*`, matches _anything_ and may only be used at
3913 the end of the string when specifying a _tracepoint_. Make sure to
3914 use it between single quotes in your favorite shell to avoid
3915 undesired shell expansion.
3917 You can see a list of events (enabled or disabled) using
3921 lttng list some-session
3924 where `some-session` is the name of the desired tracing session.
3926 What you're actually doing when enabling events with specific conditions
3927 is creating a **whitelist** of traceable events for a given channel.
3928 Thus, the following case presents redundancy:
3932 lttng enable-event -u --tracepoint my_app:hello_you
3933 lttng enable-event -u --tracepoint my_app:hello_you --loglevel TRACE_DEBUG
3936 The second command, matching a log level range, is useless since the first
3937 command enables all tracepoints matching the same name,
3940 Disabling an event is simpler: you only need to provide the event
3941 name to the `disable-event` command:
3945 lttng disable-event --userspace my_app:hello_you
3948 This name has to match a name previously given to `enable-event` (it
3949 has to be listed in the output of `lttng list some-session`).
3950 The `*` wildcard is supported, as long as you also used it in a
3951 previous `enable-event` invocation.
3953 Disabling an event does not add it to some blacklist: it simply removes
3954 it from its channel's whitelist. This is why you cannot disable an event
3955 which wasn't previously enabled.
3957 A disabled event will not generate any trace data, even if all its
3958 specified conditions are met.
3960 Events may be enabled and disabled at will, either when LTTng tracers
3961 are active or not. Events may be enabled before a user space application
3965 [[basic-tracing-session-control]]
3966 ==== Basic tracing session control
3969 <<creating-destroying-tracing-sessions,created a tracing session>>
3970 and <<enabling-disabling-events,enabled one or more events>>,
3971 you may activate the LTTng tracers for the current tracing session at
3979 Subsequently, you may stop the tracers:
3986 LTTng is very flexible: user space applications may be launched before
3987 or after the tracers are started. Events will only be recorded if they
3988 are properly enabled and if they occur while tracers are started.
3990 A tracing session name may be passed to both the `start` and `stop`
3991 commands to start/stop tracing a session other than the current one.
3994 [[enabling-disabling-channels]]
3995 ==== Enabling and disabling channels
3997 <<event,As mentioned>> in the
3998 <<understanding-lttng,Understanding LTTng>> chapter, enabled
3999 events are contained in a specific channel, itself contained in a
4000 specific tracing session. A channel is a group of events with
4001 tunable parameters (event loss mode, sub-buffer size, number of
4002 sub-buffers, trace file sizes and count, etc.). A given channel may
4003 only be responsible for enabled events belonging to one domain: either
4004 kernel or user space.
4006 If you only used the `create`, `enable-event` and `start`/`stop`
4007 commands of the `lttng` tool so far, one or two channels were
4008 automatically created for you (one for the kernel domain and/or one
4009 for the user space domain). The default channels are both named
4010 `channel0`; channels from different domains may have the same name.
4012 The current channels of a given tracing session can be viewed with
4016 lttng list some-session
4019 where `some-session` is the name of the desired tracing session.
4021 To create and enable a channel, use the `enable-channel` command:
4025 lttng enable-channel --kernel my-channel
4028 This will create a kernel domain channel named `my-channel` with
4029 default parameters in the current tracing session.
4033 Because of a current limitation, all
4034 channels must be _created_ prior to beginning tracing in a
4035 given tracing session, i.e. before the first time you do
4038 Since a channel is automatically created by
4039 `enable-event` only for the specified domain, you cannot,
4040 for example, enable a kernel domain event, start tracing and then
4041 enable a user space domain event because no user space channel
4042 exists yet and it's too late to create one.
4044 For this reason, make sure to configure your channels properly
4045 before starting the tracers for the first time!
4048 Here's another example:
4052 lttng enable-channel --userspace --session other-session --overwrite \
4053 --tracefile-size 1048576 1mib-channel
4056 This will create a user space domain channel named `1mib-channel` in
4057 the tracing session named `other-session` that loses new events by
4058 overwriting previously recorded events (instead of the default mode of
4059 discarding newer ones) and saves trace files with a maximum size of
4062 Note that channels may also be created using the `--channel` option of
4063 the `enable-event` command when the provided channel name doesn't exist
4064 for the specified domain:
4068 lttng enable-event --kernel --channel some-channel sched_switch
4071 If no kernel domain channel named `some-channel` existed before calling
4072 the above command, it would be created with default parameters.
4074 You may enable the same event in two different channels:
4078 lttng enable-event --userspace --channel my-channel app:tp
4079 lttng enable-event --userspace --channel other-channel app:tp
4082 If both channels are enabled, the occurring `app:tp` event will
4083 generate two recorded events, one for each channel.
4085 Disabling a channel is done with the `disable-event` command:
4089 lttng disable-event --kernel some-channel
4092 The state of a channel precedes the individual states of events within
4093 it: events belonging to a disabled channel, even if they are
4094 enabled, won't be recorded.
4098 [[fine-tuning-channels]]
4099 ===== Fine-tuning channels
4101 There are various parameters that may be fine-tuned with the
4102 `enable-channel` command. The latter are well documented in
4103 man:lttng(1) and in the <<channel,Channel>> section of the
4104 <<understanding-lttng,Understanding LTTng>> chapter. For basic
4105 tracing needs, their default values should be just fine, but here are a
4106 few examples to break the ice.
4108 As the frequency of recorded events increases--either because the
4109 event throughput is actually higher or because you enabled more events
4110 than usual—__event loss__ might be experienced. Since LTTng never
4111 waits, by design, for sub-buffer space availability (non-blocking
4112 tracer), when a sub-buffer is full and no empty sub-buffers are left,
4113 there are two possible outcomes: either the new events that do not fit
4114 are rejected, or they start replacing the oldest recorded events.
4115 The choice of which algorithm to use is a per-channel parameter, the
4116 default being discarding the newest events until there is some space
4117 left. If your situation always needs the latest events at the expense
4118 of writing over the oldest ones, create a channel with the `--overwrite`
4123 lttng enable-channel --kernel --overwrite my-channel
4126 When an event is lost, it means no space was available in any
4127 sub-buffer to accommodate it. Thus, if you want to cope with sporadic
4128 high event throughput situations and avoid losing events, you need to
4129 allocate more room for storing them in memory. This can be done by
4130 either increasing the size of sub-buffers or by adding sub-buffers.
4131 The following example creates a user space domain channel with
4132 16{nbsp}sub-buffers of 512{nbsp}kiB each:
4136 lttng enable-channel --userspace --num-subbuf 16 --subbuf-size 512k big-channel
4139 Both values need to be powers of two, otherwise they are rounded up
4142 Two other interesting available parameters of `enable-channel` are
4143 `--tracefile-size` and `--tracefile-count`, which respectively limit
4144 the size of each trace file and the their count for a given channel.
4145 When the number of written trace files reaches its limit for a given
4146 channel-CPU pair, the next trace file will overwrite the very first
4147 one. The following example creates a kernel domain channel with a
4148 maximum of three trace files of 1{nbsp}MiB each:
4152 lttng enable-channel --kernel --tracefile-size 1M --tracefile-count 3 my-channel
4155 An efficient way to make sure lots of events are generated is enabling
4156 all kernel events in this channel and starting the tracer:
4160 lttng enable-event --kernel --all --channel my-channel
4164 After a few seconds, look at trace files in your tracing session
4165 output directory. For two CPUs, it should look like:
4168 my-channel_0_0 my-channel_1_0
4169 my-channel_0_1 my-channel_1_1
4170 my-channel_0_2 my-channel_1_2
4173 Amongst the files above, you might see one in each group with a size
4174 lower than 1{nbsp}MiB: they are the files currently being written.
4176 Since all those small files are valid LTTng trace files, LTTng trace
4177 viewers may read them. It is the viewer's responsibility to properly
4178 merge the streams so as to present an ordered list to the user.
4179 http://diamon.org/babeltrace[Babeltrace]
4180 merges LTTng trace files correctly and is fast at doing it.
4184 ==== Adding some context to channels
4186 If you read all the sections of
4187 <<controlling-tracing,Controlling tracing>> so far, you should be
4188 able to create tracing sessions, create and enable channels and events
4189 within them and start/stop the LTTng tracers. Event fields recorded in
4190 trace files provide important information about occurring events, but
4191 sometimes external context may help you solve a problem faster. This
4192 section discusses how to add context information to events of a
4193 specific channel using the `lttng` tool.
4195 There are various available context values which can accompany events
4196 recorded by LTTng, for example:
4198 * **process information**:
4202 ** scheduling priority (niceness)
4203 ** thread identifier (TID)
4204 * the **hostname** of the system on which the event occurred
4205 * plenty of **performance counters** using perf:
4206 ** CPU cycles, stalled cycles, idle cycles, etc.
4208 ** branch instructions, misses, loads, etc.
4212 The full list is available in the output of `lttng add-context --help`.
4213 Some of them are reserved for a specific domain (kernel or
4214 user space) while others are available for both.
4216 To add context information to one or all channels of a given tracing
4217 session, use the `add-context` command:
4221 lttng add-context --userspace --type vpid --type perf:thread:cpu-cycles
4224 The above example adds the virtual process identifier and per-thread
4225 CPU cycles count values to all recorded user space domain events of the
4226 current tracing session. Use the `--channel` option to select a specific
4231 lttng add-context --kernel --channel my-channel --type tid
4234 adds the thread identifier value to all recorded kernel domain events
4235 in the channel `my-channel` of the current tracing session.
4237 Beware that context information cannot be removed from channels once
4238 it's added for a given tracing session.
4241 [[saving-loading-tracing-session]]
4242 ==== Saving and loading tracing session configurations
4244 Configuring a tracing session may be long: creating and enabling
4245 channels with specific parameters, enabling kernel and user space
4246 domain events with specific log levels and filters, adding context
4247 to some channels, etc. If you're going to use LTTng to solve real
4248 world problems, chances are you're going to have to record events using
4249 the same tracing session setup over and over, modifying a few variables
4250 each time in your instrumented program or environment. To avoid
4251 constant tracing session reconfiguration, the `lttng` tool is able to
4252 save and load tracing session configurations to/from XML files.
4254 To save a given tracing session configuration, do:
4258 lttng save my-session
4261 where `my-session` is the name of the tracing session to save. Tracing
4262 session configurations are saved to dir:{~/.lttng/sessions} by default;
4263 use the `--output-path` option to change this destination directory.
4265 All configuration parameters are saved:
4267 * tracing session name
4268 * trace data output path
4269 * channels with their state and all their parameters
4270 * context information added to channels
4271 * events with their state, log level and filter
4272 * tracing activity (started or stopped)
4274 To load a tracing session, simply do:
4278 lttng load my-session
4281 or, if you used a custom path:
4285 lttng load --input-path /path/to/my-session.lttng
4288 Your saved tracing session will be restored as if you just configured
4292 [[sending-trace-data-over-the-network]]
4293 ==== Sending trace data over the network
4295 The possibility of sending trace data over the network comes as a
4296 built-in feature of LTTng-tools. For this to be possible, an LTTng
4297 _relay daemon_ must be executed and listening on the machine where
4298 trace data is to be received, and the user must create a tracing
4299 session using appropriate options to forward trace data to the remote
4302 The relay daemon listens on two different TCP ports: one for control
4303 information and the other for actual trace data.
4305 Starting the relay daemon on the remote machine is as easy as:
4312 This will make it listen to its default ports: 5342 for control and
4313 5343 for trace data. The `--control-port` and `--data-port` options may
4314 be used to specify different ports.
4316 Traces written by `lttng-relayd` are written to
4317 +\~/lttng-traces/__hostname__/__session__+ by
4318 default, where +__hostname__+ is the host name of the
4319 traced (monitored) system and +__session__+ is the
4320 tracing session name. Use the `--output` option to write trace data
4321 outside dir:{~/lttng-traces}.
4323 On the sending side, a tracing session must be created using the
4324 `lttng` tool with the `--set-url` option to connect to the distant
4329 lttng create my-session --set-url net://distant-host
4332 The URL format is described in the output of `lttng create --help`.
4333 The above example will use the default ports; the `--ctrl-url` and
4334 `--data-url` options may be used to set the control and data URLs
4337 Once this basic setup is completed and the connection is established,
4338 you may use the `lttng` tool on the target machine as usual; everything
4339 you do will be transparently forwarded to the remote machine if needed.
4340 For example, a parameter changing the maximum size of trace files will
4341 have an effect on the distant relay daemon actually writing the trace.
4345 ==== Viewing events as they arrive
4347 We have seen how trace files may be produced by LTTng out of generated
4348 application and Linux kernel events. We have seen that those trace files
4349 may be either recorded locally by consumer daemons or remotely using
4350 a relay daemon. And we have seen that the maximum size and count of
4351 trace files is configurable for each channel. With all those features,
4352 it's still not possible to read a trace file as it is being written
4353 because it could be incomplete and appear corrupted to the viewer.
4354 There is a way to view events as they arrive, however: using
4357 LTTng live is implemented, in LTTng, solely on the relay daemon side.
4358 As trace data is sent over the network to a relay daemon by a (possibly
4359 remote) consumer daemon, a _tee_ may be created: trace data will be
4360 recorded to trace files _as well as_ being transmitted to a
4361 connected live viewer:
4364 .LTTng live and the relay daemon.
4365 image::lttng-live-relayd.png[]
4367 In order to use this feature, a tracing session must created in live
4368 mode on the target system:
4375 An optional parameter may be passed to `--live` to set the interval
4376 of time (in microseconds) between flushes to the network
4377 (1{nbsp}second is the default):
4381 lttng create --live 100000
4384 will flush every 100{nbsp}ms.
4386 If no network output is specified to the `create` command, a local
4387 relay daemon will be spawned. In this very common case, viewing a live
4388 trace is easy: enable events and start tracing as usual, then use
4389 `lttng view` to start the default live viewer:
4396 The correct arguments will be passed to the live viewer so that it
4397 may connect to the local relay daemon and start reading live events.
4399 You may also wish to use a live viewer not running on the target
4400 system. In this case, you should specify a network output when using
4401 the `create` command (`--set-url` or `--ctrl-url`/`--data-url` options).
4402 A distant LTTng relay daemon should also be started to receive control
4403 and trace data. By default, `lttng-relayd` listens on 127.0.0.1:5344
4404 for an LTTng live connection. Otherwise, the desired URL may be
4405 specified using its `--live-port` option.
4408 http://diamon.org/babeltrace[`babeltrace`]
4409 viewer supports LTTng live as one of its input formats. `babeltrace` is
4410 the default viewer when using `lttng view`. To use it manually, first
4411 list active tracing sessions by doing the following (assuming the relay
4412 daemon to connect to runs on the same host):
4416 babeltrace --input-format lttng-live net://localhost
4419 Then, choose a tracing session and start viewing events as they arrive
4420 using LTTng live, e.g.:
4424 babeltrace --input-format lttng-live net://localhost/host/hostname/my-session
4428 [[taking-a-snapshot]]
4429 ==== Taking a snapshot
4431 The normal behavior of LTTng is to record trace data as trace files.
4432 This is ideal for keeping a long history of events that occurred on
4433 the target system and applications, but may be too much data in some
4434 situations. For example, you may wish to trace your application
4435 continuously until some critical situation happens, in which case you
4436 would only need the latest few recorded events to perform the desired
4437 analysis, not multi-gigabyte trace files.
4439 LTTng has an interesting feature called _snapshots_. When creating
4440 a tracing session in snapshot mode, no trace files are written; the
4441 tracers' sub-buffers are constantly overwriting the oldest recorded
4442 events with the newest. At any time, either when the tracers are started
4443 or stopped, you may take a snapshot of those sub-buffers.
4445 There is no difference between the format of a normal trace file and the
4446 format of a snapshot: viewers of LTTng traces will also support LTTng
4447 snapshots. By default, snapshots are written to disk, but they may also
4448 be sent over the network.
4450 To create a tracing session in snapshot mode, do:
4454 lttng create --snapshot my-snapshot-session
4457 Next, enable channels, events and add context to channels as usual.
4458 Once a tracing session is created in snapshot mode, channels will be
4460 <<channel-overwrite-mode-vs-discard-mode,overwrite>> mode
4461 (`--overwrite` option of the `enable-channel` command; also called
4462 _flight recorder mode_) and have an `mmap()` channel type
4465 Start tracing. When you're ready to take a snapshot, do:
4469 lttng snapshot record --name my-snapshot
4472 This will record a snapshot named `my-snapshot` of all channels of
4473 all domains of the current tracing session. By default, snapshots files
4474 are recorded in the path returned by `lttng snapshot list-output`. You
4475 may change this path or decide to send snapshots over the network
4478 . an output path/URL specified when creating the tracing session
4480 . an added snapshot output path/URL using
4481 `lttng snapshot add-output`
4482 . an output path/URL provided directly to the
4483 `lttng snapshot record` command
4485 Method 3 overrides method 2 which overrides method 1. When specifying
4486 a URL, a relay daemon must be listening on some machine (see
4487 <<sending-trace-data-over-the-network,Sending trace data over the network>>).
4489 If you need to make absolutely sure that the output file won't be
4490 larger than a certain limit, you can set a maximum snapshot size when
4491 taking it with the `--max-size` option:
4495 lttng snapshot record --name my-snapshot --max-size 2M
4498 Older recorded events will be discarded in order to respect this
4505 This chapter presents various references for LTTng packages such as links
4506 to online manpages, tables needed by the rest of the text, descriptions
4507 of library functions, etc.
4510 [[online-lttng-manpages]]
4511 === Online LTTng manpages
4513 LTTng packages currently install the following manpages, available
4514 online using the links below:
4518 ** man:lttng-sessiond(8)
4519 ** man:lttng-relayd(8)
4521 ** man:lttng-gen-tp(1)
4523 ** man:lttng-ust-cyg-profile(3)
4524 ** man:lttng-ust-dl(3)
4530 This section presents references of the LTTng-UST package.
4534 ==== LTTng-UST library (+liblttng‑ust+)
4536 The LTTng-UST library, or `liblttng-ust`, is the main shared object
4537 against which user applications are linked to make LTTng user space
4540 The <<c-application,C application>> guide shows the complete
4541 process to instrument, build and run a C/$$C++$$ application using
4542 LTTng-UST, while this section contains a few important tables.
4545 [[liblttng-ust-tp-fields]]
4546 ===== Tracepoint fields macros (for `TP_FIELDS()`)
4548 The available macros to define tracepoint fields, which should be listed
4549 within `TP_FIELDS()` in `TRACEPOINT_EVENT()`, are:
4551 [role="growable func-desc",cols="asciidoc,asciidoc"]
4552 .Available macros to define LTTng-UST tracepoint fields
4554 |Macro |Description and parameters
4557 +ctf_integer(__t__, __n__, __e__)+
4559 +ctf_integer_nowrite(__t__, __n__, __e__)+
4561 Standard integer, displayed in base 10.
4564 Integer C type (`int`, `long`, `size_t`, etc.).
4570 Argument expression.
4572 |+ctf_integer_hex(__t__, __n__, __e__)+
4574 Standard integer, displayed in base 16.
4583 Argument expression.
4585 |+ctf_integer_network(__t__, __n__, __e__)+
4587 Integer in network byte order (big endian), displayed in base 10.
4596 Argument expression.
4598 |+ctf_integer_network_hex(__t__, __n__, __e__)+
4600 Integer in network byte order, displayed in base 16.
4609 Argument expression.
4612 +ctf_float(__t__, __n__, __e__)+
4614 +ctf_float_nowrite(__t__, __n__, __e__)+
4616 Floating point number.
4619 Floating point number C type (`float` or `double`).
4625 Argument expression.
4628 +ctf_string(__n__, __e__)+
4630 +ctf_string_nowrite(__n__, __e__)+
4632 Null-terminated string; undefined behavior if +__e__+ is `NULL`.
4638 Argument expression.
4641 +ctf_array(__t__, __n__, __e__, __s__)+
4643 +ctf_array_nowrite(__t__, __n__, __e__, __s__)+
4645 Statically-sized array of integers
4648 Array element C type.
4654 Argument expression.
4660 +ctf_array_text(__t__, __n__, __e__, __s__)+
4662 +ctf_array_text_nowrite(__t__, __n__, __e__, __s__)+
4664 Statically-sized array, printed as text.
4666 The string does not need to be null-terminated.
4669 Array element C type (always `char`).
4675 Argument expression.
4681 +ctf_sequence(__t__, __n__, __e__, __T__, __E__)+
4683 +ctf_sequence_nowrite(__t__, __n__, __e__, __T__, __E__)+
4685 Dynamically-sized array of integers.
4687 The type of +__E__+ needs to be unsigned.
4690 Array element C type.
4696 Argument expression.
4699 Length expression C type.
4705 +ctf_sequence_text(__t__, __n__, __e__, __T__, __E__)+
4707 +ctf_sequence_text_nowrite(__t__, __n__, __e__, __T__, __E__)+
4709 Dynamically-sized array, displayed as text.
4711 The string does not need to be null-terminated.
4713 The type of +__E__+ needs to be unsigned.
4715 The behaviour is undefined if +__e__+ is `NULL`.
4718 Sequence element C type (always `char`).
4724 Argument expression.
4727 Length expression C type.
4733 The `_nowrite` versions omit themselves from the session trace, but are
4734 otherwise identical. This means the `_nowrite` fields won't be written
4735 in the recorded trace. Their primary purpose is to make some
4736 of the event context available to the
4737 <<enabling-disabling-events,event filters>> without having to
4738 commit the data to sub-buffers.
4741 [[liblttng-ust-tracepoint-loglevel]]
4742 ===== Tracepoint log levels (for `TRACEPOINT_LOGLEVEL()`)
4744 The following table shows the available log level values for the
4745 `TRACEPOINT_LOGLEVEL()` macro:
4751 Action must be taken immediately.
4754 Critical conditions.
4763 Normal, but significant, condition.
4766 Informational message.
4768 `TRACE_DEBUG_SYSTEM`::
4769 Debug information with system-level scope (set of programs).
4771 `TRACE_DEBUG_PROGRAM`::
4772 Debug information with program-level scope (set of processes).
4774 `TRACE_DEBUG_PROCESS`::
4775 Debug information with process-level scope (set of modules).
4777 `TRACE_DEBUG_MODULE`::
4778 Debug information with module (executable/library) scope (set of units).
4780 `TRACE_DEBUG_UNIT`::
4781 Debug information with compilation unit scope (set of functions).
4783 `TRACE_DEBUG_FUNCTION`::
4784 Debug information with function-level scope.
4786 `TRACE_DEBUG_LINE`::
4787 Debug information with line-level scope (TRACEPOINT_EVENT default).
4790 Debug-level message.
4792 Log levels `TRACE_EMERG` through `TRACE_INFO` and `TRACE_DEBUG` match
4793 http://man7.org/linux/man-pages/man3/syslog.3.html[syslog]
4794 level semantics. Log levels `TRACE_DEBUG_SYSTEM` through `TRACE_DEBUG`
4795 offer more fine-grained selection of debug information.
4798 [[lttng-modules-ref]]
4801 This section presents references of the LTTng-modules package.
4804 [[lttng-modules-tp-struct-entry]]
4805 ==== Tracepoint fields macros (for `TP_STRUCT__entry()`)
4807 This table describes possible entries for the `TP_STRUCT__entry()` part
4808 of `LTTNG_TRACEPOINT_EVENT()`:
4810 [role="growable func-desc",cols="asciidoc,asciidoc"]
4811 .Available entries for `TP_STRUCT__entry()` (in `LTTNG_TRACEPOINT_EVENT()`)
4813 |Macro |Description and parameters
4815 |+\__field(__t__, __n__)+
4817 Standard integer, displayed in base 10.
4820 Integer C type (`int`, `unsigned char`, `size_t`, etc.).
4825 |+\__field_hex(__t__, __n__)+
4827 Standard integer, displayed in base 16.
4835 |+\__field_oct(__t__, __n__)+
4837 Standard integer, displayed in base 8.
4845 |+\__field_network(__t__, __n__)+
4847 Integer in network byte order (big endian), displayed in base 10.
4855 |+\__field_network_hex(__t__, __n__)+
4857 Integer in network byte order (big endian), displayed in base 16.
4865 |+\__array(__t__, __n__, __s__)+
4867 Statically-sized array, elements displayed in base 10.
4870 Array element C type.
4878 |+\__array_hex(__t__, __n__, __s__)+
4880 Statically-sized array, elements displayed in base 16.
4883 array element C type.
4889 |+\__array_text(__t__, __n__, __s__)+
4891 Statically-sized array, displayed as text.
4894 Array element C type (always char).
4902 |+\__dynamic_array(__t__, __n__, __s__)+
4904 Dynamically-sized array, displayed in base 10.
4907 Array element C type.
4913 Length C expression.
4915 |+\__dynamic_array_hex(__t__, __n__, __s__)+
4917 Dynamically-sized array, displayed in base 16.
4920 Array element C type.
4926 Length C expression.
4928 |+\__dynamic_array_text(__t__, __n__, __s__)+
4930 Dynamically-sized array, displayed as text.
4933 Array element C type (always char).
4939 Length C expression.
4941 |+\__string(n, __s__)+
4943 Null-terminated string.
4945 The behaviour is undefined behavior if +__s__+ is `NULL`.
4951 String source (pointer).
4954 The above macros should cover the majority of cases. For advanced items,
4955 see path:{probes/lttng-events.h}.
4958 [[lttng-modules-tp-fast-assign]]
4959 ==== Tracepoint assignment macros (for `TP_fast_assign()`)
4961 This table describes possible entries for the `TP_fast_assign()` part
4962 of `LTTNG_TRACEPOINT_EVENT()`:
4964 .Available entries for `TP_fast_assign()` (in `LTTNG_TRACEPOINT_EVENT()`)
4965 [role="growable func-desc",cols="asciidoc,asciidoc"]
4967 |Macro |Description and parameters
4969 |+tp_assign(__d__, __s__)+
4971 Assignment of C expression +__s__+ to tracepoint field +__d__+.
4974 Name of destination tracepoint field.
4977 Source C expression (may refer to tracepoint arguments).
4979 |+tp_memcpy(__d__, __s__, __l__)+
4981 Memory copy of +__l__+ bytes from +__s__+ to tracepoint field
4982 +__d__+ (use with array fields).
4985 Name of destination tracepoint field.
4988 Source C expression (may refer to tracepoint arguments).
4991 Number of bytes to copy.
4993 |+tp_memcpy_from_user(__d__, __s__, __l__)+
4995 Memory copy of +__l__+ bytes from user space +__s__+ to tracepoint
4996 field +__d__+ (use with array fields).
4999 Name of destination tracepoint field.
5002 Source C expression (may refer to tracepoint arguments).
5005 Number of bytes to copy.
5007 |+tp_memcpy_dyn(__d__, __s__)+
5009 Memory copy of dynamically-sized array from +__s__+ to tracepoint field
5012 The number of bytes is known from the field's length expression
5013 (use with dynamically-sized array fields).
5016 Name of destination tracepoint field.
5019 Source C expression (may refer to tracepoint arguments).
5022 Number of bytes to copy.
5024 |+tp_strcpy(__d__, __s__)+
5026 String copy of +__s__+ to tracepoint field +__d__+ (use with string
5030 Name of destination tracepoint field.
5033 Source C expression (may refer to tracepoint arguments).