1 \input texinfo @c -*-texinfo-*-
4 @settitle LTTng Userspace Tracer (UST) Manual
8 This manual is for program, version version.
10 Copyright @copyright{} copyright-owner.
13 Permission is granted to ...
18 @title LTTng Userspace Tracer (UST) Manual
19 @c @subtitle subtitle-if-any
20 @c @subtitle second-subtitle
23 @c The following two commands
24 @c start the copyright page.
26 @c @vskip 0pt plus 1filll
32 @c So the toc is printed at the start.
37 @top LTTng Userspace Tracer
39 This manual is for UST 0.4.
46 * Instrumenting an application::
51 @c * Copying:: Your rights and freedoms.
60 * Supported platforms::
66 The LTTng Userspace Tracer (UST) is a library accompanied by a set of tools to
69 Code may be instrumented with either markers or tracepoints. A highly efficient
70 lockless tracer records these events to a trace buffers. These buffers are reaped
71 by a deamon which writes trace data to disk.
73 High performance is achieved by the use of lockless buffering algorithms, RCU and
74 per-cpu buffers. In addition, special care is taken to minize cache impact.
78 The LTTng Userspace Tracer is intended to be linkable to open source software
79 as well as to proprietary applications. This was accomplished by licensing
80 the code that needs to be linked to the traced program as @acronym{LGPL}.
82 Components licensed as LGPL v2.1:
89 Components licensed as GPL v2:
96 @node Supported platforms
97 @section Supported platforms
99 UST can currently trace applications running on Linux, on the x86-32, x86-64
100 and PowerPC 32 architectures.
103 @chapter Installation
105 The LTTng userspace tracer is a library and a set of userspace tools.
107 The following packages are required:
113 This contains the tracing library, the ustd daemon, trace control tools
114 and other helper tools.
116 Repository: http://git.dorsal.polymtl.ca
121 This is a library that contains a userspace port of some kernel APIs.
123 Repository: http://git.dorsal.polymtl.ca
128 This is the userspace read-copy update library by Mathieu Desnoyers.
130 Available in Debian as package liburcu-dev.
132 Home page: http://lttng.org/?q=node/18
137 LTTV is a graphical (and text) viewer for LTTng traces.
139 Home page: http://lttng.org
143 Libkcompat and liburcu should be installed first. UST may then be compiled
144 and installed. LTTV has no dependency on the other packages; it may therefore
145 be installed on a system which does not have UST installed.
147 Refer to the README in each of these packages for installation instructions.
155 First, instrument a program with a marker.
160 #include <ust/marker.h>
162 int main(int argc, char **argv)
167 /* ... set values of v and st ... */
170 trace_mark(ust, myevent, "firstarg %d secondarg %s", v, st);
172 /* a marker without arguments: */
173 trace_mark(ust, myotherevent, MARK_NOARGS);
181 Then compile it in the regular way, linking it with libust. For example:
184 gcc -o foo -lust foo.c
187 Run the program with @command{usttrace}. The @command{usttrace} output says where the trace
194 Finally, open the trace in LTTV.
197 lttv-gui -t /path/to/trace
200 The trace can also be dumped as text in the console:
203 lttv -m textDump -t /path/to/trace
206 @node Instrumenting an application
207 @chapter Instrumenting an application
209 In order to record a trace of events occurring in a application, the
210 application must be instrumented. Instrumentation points resemble function
211 calls. When the program reaches an instrumentation point, an event is
214 There are no limitations on the type of code that may be instrumented.
215 Multi-threaded programs may be instrumented without problem. Signal handlers
216 may be instrumented as well.
218 There are two APIs to instrument programs: markers and tracepoints. Markers are
219 quick to add and are usually used for temporary instrumentation. Tracepoints
220 provide a way to instrument code more cleanly and are suited for permanent
223 In addition to executable programs, shared libraries may also be instrumented
224 with the methods described in this chapter.
234 Adding a marker is simply a matter of inserting one line in the program.
238 #include <ust/marker.h>
240 int main(int argc, char **argv)
245 /* ... set values of v and st ... */
248 trace_mark(main, myevent, "firstarg %d secondarg %s", v, st);
250 /* another marker without arguments: */
251 trace_mark(main, myotherevent, MARK_NOARGS);
258 The invocation of the trace_mark() macro requires at least 3 arguments. The
259 first, here "main", is the name of the event category. It is also the name of
260 the channel the event will go in. The second, here "myevent" is the name of the
261 event. The third is a format string that announces the names and the types of
262 the event arguments. Its format resembles that of a printf() format string; it
263 is described thoroughly in Appendix x.
265 A given Marker may appear more than once in the same program. Other Markers may
266 have the same name and a different format string, although this might induce
267 some confusion at analysis time.
272 The Tracepoints API uses the Markers, but provides a higher-level abstraction.
273 Whereas the markers API provides limited type checking, the Tracepoints API
274 provides more thorough type checking and discharges from the need to insert
275 format strings directly in the code and to have format strings appear more than
276 once if a given marker is reused.
279 Although this example uses @emph{mychannel} as the channel, the
280 only channel name currently supported with early tracing is @strong{ust}. The
281 @command{usttrace} tool always uses the early tracing mode. When using manual
282 mode without early tracing, any channel name may be used.
285 A function instrumented with a tracepoint looks like this:
296 /* ... set values of v and st ... */
299 trace_mychannel_myevent(v, st);
304 Another file, here tp.h, contains declarations for the tracepoint.
308 #include <ust/tracepoint.h>
310 DECLARE_TRACE(mychannel_myevent, TP_PROTO(int v, char *st),
315 A third file, here tp.c, contains definitions for the tracepoint.
319 #include <ust/marker.h>
322 DEFINE_TRACE(mychannel_myevent);
324 void mychannel_myevent_probe(int v, char *st)
326 trace_mark(mychannel, myevent, "v %d st %s", v, st);
329 static void __attribute__((constructor)) init()
331 register_trace_mychannel_myevent(mychannel_myevent_probe);
336 Here, tp.h and tp.c could contain declarations and definitions for other
337 tracepoints. The constructor would contain other register_* calls.
339 @node Recording a trace
340 @chapter Recording a trace
343 * Using @command{usttrace}::
344 * Setting up the recording manually::
345 * Using early tracing::
347 * Tracing across @code{fork()} and @code{clone()}::
348 * Tracing programs and libraries that were not linked to libust::
351 @node Using @command{usttrace}
352 @section Using @command{usttrace}
354 The simplest way to record a trace is to use the @command{usttrace} script. An
355 example is given in the quickstart above.
357 The @command{usttrace} script automatically:
359 @item creates a daemon
360 @item enables all markers
361 @item runs the command specified on the command line
362 @item after the command ends, prints the location where the trace was saved
365 Each subdirectory of the save location contains the trace of one process that
366 was generated by the command. The name of a subdirectory consists in the the PID
367 of the process, followed by the timestamp of its creation.
369 The save location also contains logs of the tracing.
371 When using @command{usttrace}, the early tracing is always active, which means
372 that the tracing is guaranteed to be started by the time the process enters its
375 Several @command{usttrace}'s may be run simultaneously without risk of
376 conflict. This facilitates the use of the tracer by idependent users on a
377 system. Each instance of @command{usttrace} starts its own daemon which
378 collects the events of the processes it creates.
380 @node Setting up the recording manually
381 @section Setting up the recording manually
383 Instead of using @command{usttrace}, a trace may be recorded on an already
386 First the daemon must be started.
390 # Make sure the directory for the communication sockets exists.
391 $ mkdir /tmp/ustsocks
393 # Make sure the directory where ustd will write the trace exists.
399 # We assume the program we want to trace is already running and that
402 # List the available markers
403 $ ustctl --list-markers 1234
404 # A column indicates 0 for an inactive marker and 1 for an active marker.
407 $ ustctl --enable-marker ust/mymark 1234
410 $ ustctl --create-trace 1234
413 $ ustctl --start-trace 1234
418 $ ustctl --stop-trace 1234
421 $ ustctl --destroy-trace 1234
425 For more information about the manual mode, see the ustctl(1) man page.
427 @node Using early tracing
428 @section Using early tracing
430 Early tracing consists in starting the tracing as early as possible in the
431 program, so no events are lost between program start and the point where the
432 command to start the tracing is given. When using early tracing, it is
433 guaranteed that by the time the traced program enters its @code{main()}
434 function, the tracing will be started.
436 When using @command{usttrace}, the early tracing is always active.
438 When using the manual mode (@command{ustctl}), early tracing is enabled using
439 environment variables. Setting @env{UST_TRACE} to @code{1}, enables early
440 tracing, while setting @env{UST_AUTOPROBE} to @code{1} enables all markers
445 @section Crash recovery
447 When a process being traced crashes, the daemon is able to recover all the
448 events in its buffers that were successfully commited. This is possible because
449 the buffers are in a shared memory segment which remains available to the
450 daemon even after the termination of the traced process.
452 @node Tracing across @code{fork()} and @code{clone()}
453 @section Tracing across @code{fork()} and @code{clone()}
455 Tracing across @code{clone()} when the @code{CLONE_VM} flag is specified is
456 supported without any particular action.
458 When @code{clone()} is called without @code{CLONE_VM} or @code{fork()} is
459 called, a new address space is created and the tracer must be notified to
460 create new buffers for it.
462 This can be done automatically, by @env{LD_PRELOAD}'ing @file{libinterfork.so}.
463 This library intercepts calls to @code{fork()} and informs the tracer it is
464 being called. When using @command{usttrace}, this is accomplied by specifying
465 the @option{-f} command line argument.
467 Alternatively, the program can call @code{ust_before_fork()} before calling
468 @code{fork()} or @code{clone()} with @code{CLONE_VM}. After the call,
469 @code{ust_after_fork_parent()} must be called in the parent process and
470 @code{ust_after_fork_child()} must be called in the child process.
473 @node Tracing programs and libraries that were not linked to libust
474 @section Tracing programs and libraries that were not linked to libust
476 Some programs need to be traced even though they were not linked to libust
477 either because they were not instrumented or because it was not practical.
479 An executable that is not instrumented can still yield interesting traces when
480 at least one of its dynamic libraries is instrumented. It is also possible to
481 trace certain function calls by intercepting them with a specially crafted
482 library that is linked with @env{LD_PRELOAD} at program start.
484 In any case, a program that was not linked to libust at compile time must be
485 linked to it at run time with @env{LD_PRELOAD}. This can be accomplished with
486 @command{usttrace}'s @option{-l} option. It can also be done by setting the
487 @env{LD_PRELOAD} environment variable on the command line. For example:
491 # Run ls with usttrace, LD_PRELOAD'ing libust
492 # (assuming one of the libraries used by ls is instrumented).
495 # Run ls, manually adding the LD_PRELOAD.
496 $ LD_PRELOAD=/usr/local/lib/libust.so.0 ls
507 @chapter Viewing traces
509 Traces may be viewed with LTTV. An example of command for launching LTTV is
510 given in the quickstart.
513 * Viewing multiple traces::
514 * Combined kernel-userspace tracing::
517 @node Viewing multiple traces
518 @section Viewing multiple traces
520 When tracing multi-process applications or several applications simultaneously,
521 more than one trace will be obtained. LTTV can open and display all these
522 traces simultaneously.
524 @node Combined kernel-userspace tracing
525 @section Combined kernel-userspace tracing
527 In addition to multiple userspace traces, LTTV can open a kernel trace recorded
528 with the LTTng kernel tracer. This provides events that enable the rendering of
529 the Control Flow View and the Resource View.
531 When doing so, it is necessary to use the same time source for the kernel
532 tracer as well as the userspace tracer. Currently, the recommended method is to
533 use the timestamp counter for both. The TSC can however only be used on architectures
534 where it is synchronized across cores.
537 @chapter Resource Usage
539 The purpose of this section is to give an overview of the resource usage of libust. For
540 a developer, knowing this can be important: because libust is linked with applications, it
541 needs to share some resources with it. Some applications may make some assumptions that are in
542 conflict with libust's usage of resources.
544 In practice however, libust is designed to be transparent and is compatible
545 with the vast majority of applications. This means no changes are required in
546 the application (or library) being linked to libust.
548 Libust is initialized by a constructor, which by definition runs before the main() function
549 of the application starts. This constructor creates a thread called the @emph{listener thread}.
550 The listener thread initializes a named socket and waits for connections for ustd or ustctl.
552 Libust-specific code may:
554 @item use @code{malloc()} and @code{free()}
555 @item map shared memory segment in the process adress space
556 @item intercept some library calls, specifically @code{fork()} and @code{clone()}
557 @item do interprocess communication with the daemon or ustctl
558 @item create and open named sockets
564 @item handle any signal (all signals are blocked in the listener thread)
565 @item change any process-wide setting that could confuse the application