Commit | Line | Data |
---|---|---|
5cd1099a PMF |
1 | \input texinfo @c -*-texinfo-*- |
2 | @c %**start of header | |
3 | @setfilename ust.info | |
4 | @settitle LTTng Userspace Tracer (UST) Manual | |
5 | @c %**end of header | |
6 | ||
7 | @copying | |
8 | This manual is for program, version version. | |
9 | ||
10 | Copyright @copyright{} copyright-owner. | |
11 | ||
12 | @quotation | |
13 | Permission is granted to ... | |
14 | @end quotation | |
15 | @end copying | |
16 | ||
17 | @titlepage | |
18 | @title LTTng Userspace Tracer (UST) Manual | |
19 | @c @subtitle subtitle-if-any | |
20 | @c @subtitle second-subtitle | |
21 | @c @author author | |
22 | ||
23 | @c The following two commands | |
24 | @c start the copyright page. | |
25 | @c @page | |
26 | @c @vskip 0pt plus 1filll | |
27 | @c @insertcopying | |
28 | ||
29 | @c Published by ... | |
30 | @end titlepage | |
31 | ||
32 | @c So the toc is printed at the start. | |
33 | @contents | |
34 | ||
35 | @ifnottex | |
36 | @node Top | |
37 | @top LTTng Userspace Tracer | |
38 | ||
bb0496b2 | 39 | This manual is for UST 0.4. |
5cd1099a PMF |
40 | @end ifnottex |
41 | ||
42 | @menu | |
43 | * Overview:: | |
44 | * Installation:: | |
45 | * Quick start:: | |
46 | * Instrumenting an application:: | |
47 | * Recording a trace:: | |
48 | * Viewing traces:: | |
49 | * Performance:: | |
bb0496b2 | 50 | * Resource Usage:: |
5cd1099a PMF |
51 | @c * Copying:: Your rights and freedoms. |
52 | @end menu | |
53 | ||
54 | @node Overview | |
55 | @chapter Overview | |
56 | ||
57 | @menu | |
70e55dbe | 58 | * What is UST?:: |
79f9fac7 | 59 | * License:: |
5cd1099a PMF |
60 | * Supported platforms:: |
61 | @end menu | |
62 | ||
70e55dbe PMF |
63 | @node What is UST? |
64 | @section What is UST? | |
65 | ||
66 | The LTTng Userspace Tracer (UST) is a library accompanied by a set of tools to | |
67 | trace userspace code. | |
68 | ||
69 | Code may be instrumented with either markers or tracepoints. A highly efficient | |
70 | lockless tracer records these events to a trace buffers. These buffers are reaped | |
71 | by a deamon which writes trace data to disk. | |
72 | ||
73 | High performance is achieved by the use of lockless buffering algorithms, RCU and | |
74 | per-cpu buffers. In addition, special care is taken to minize cache impact. | |
75 | ||
79f9fac7 PMF |
76 | @node License |
77 | @section License | |
78 | The LTTng Userspace Tracer is intended to be linkable to open source software | |
79 | as well as to proprietary applications. This was accomplished by licensing | |
80 | the code that needs to be linked to the traced program as @acronym{LGPL}. | |
81 | ||
675ac4ab PMF |
82 | Components licensed as LGPL v2.1: |
83 | @itemize @bullet | |
84 | @item libust | |
85 | @item libinterfork | |
86 | @item libustcomm | |
87 | @end itemize | |
88 | ||
89 | Components licensed as GPL v2: | |
90 | @itemize @bullet | |
91 | @item ustctl | |
92 | @item libustcmd | |
93 | @item ustd | |
94 | @end itemize | |
79f9fac7 | 95 | |
5cd1099a PMF |
96 | @node Supported platforms |
97 | @section Supported platforms | |
98 | ||
bb0496b2 PMF |
99 | UST can currently trace applications running on Linux, on the x86-32, x86-64 |
100 | and PowerPC 32 architectures. | |
5cd1099a PMF |
101 | |
102 | @node Installation | |
103 | @chapter Installation | |
104 | ||
105 | The LTTng userspace tracer is a library and a set of userspace tools. | |
106 | ||
107 | The following packages are required: | |
108 | ||
109 | @itemize @bullet | |
110 | @item | |
111 | ust | |
112 | ||
113 | This contains the tracing library, the ustd daemon, trace control tools | |
114 | and other helper tools. | |
115 | ||
116 | Repository: http://git.dorsal.polymtl.ca | |
117 | ||
118 | @item | |
119 | libkcompat | |
120 | ||
121 | This is a library that contains a userspace port of some kernel APIs. | |
122 | ||
123 | Repository: http://git.dorsal.polymtl.ca | |
124 | ||
125 | @item | |
126 | liburcu | |
127 | ||
128 | This is the userspace read-copy update library by Mathieu Desnoyers. | |
129 | ||
130 | Available in Debian as package liburcu-dev. | |
131 | ||
132 | Home page: http://lttng.org/?q=node/18 | |
133 | ||
134 | @item | |
135 | LTTV | |
136 | ||
137 | LTTV is a graphical (and text) viewer for LTTng traces. | |
138 | ||
139 | Home page: http://lttng.org | |
140 | ||
141 | @end itemize | |
142 | ||
143 | Libkcompat and liburcu should be installed first. UST may then be compiled | |
144 | and installed. LTTV has no dependency on the other packages; it may therefore | |
145 | be installed on a system which does not have UST installed. | |
146 | ||
147 | Refer to the README in each of these packages for installation instructions. | |
148 | ||
149 | @c @menu | |
150 | @c @end menu | |
151 | ||
152 | @node Quick start | |
153 | @chapter Quick start | |
154 | ||
155 | First, instrument a program with a marker. | |
156 | ||
157 | @example | |
158 | @verbatim | |
159 | ||
160 | #include <ust/marker.h> | |
161 | ||
162 | int main(int argc, char **argv) | |
163 | { | |
164 | int v; | |
165 | char *st; | |
166 | ||
167 | /* ... set values of v and st ... */ | |
168 | ||
169 | /* a marker: */ | |
170 | trace_mark(ust, myevent, "firstarg %d secondarg %s", v, st); | |
171 | ||
172 | /* a marker without arguments: */ | |
173 | trace_mark(ust, myotherevent, MARK_NOARGS); | |
174 | ||
175 | return 0; | |
176 | } | |
177 | ||
178 | @end verbatim | |
179 | @end example | |
180 | ||
181 | Then compile it in the regular way, linking it with libust. For example: | |
182 | ||
183 | @example | |
184 | gcc -o foo -lust foo.c | |
185 | @end example | |
186 | ||
187 | Run the program with @command{usttrace}. The @command{usttrace} output says where the trace | |
188 | was written. | |
189 | ||
190 | @example | |
191 | usttrace ./foo | |
192 | @end example | |
193 | ||
194 | Finally, open the trace in LTTV. | |
195 | ||
196 | @example | |
197 | lttv-gui -t /path/to/trace | |
198 | @end example | |
199 | ||
200 | The trace can also be dumped as text in the console: | |
201 | ||
202 | @example | |
203 | lttv -m textDump -t /path/to/trace | |
204 | @end example | |
205 | ||
206 | @node Instrumenting an application | |
207 | @chapter Instrumenting an application | |
208 | ||
209 | In order to record a trace of events occurring in a application, the | |
210 | application must be instrumented. Instrumentation points resemble function | |
211 | calls. When the program reaches an instrumentation point, an event is | |
212 | generated. | |
213 | ||
214 | There are no limitations on the type of code that may be instrumented. | |
215 | Multi-threaded programs may be instrumented without problem. Signal handlers | |
216 | may be instrumented as well. | |
217 | ||
218 | There are two APIs to instrument programs: markers and tracepoints. Markers are | |
219 | quick to add and are usually used for temporary instrumentation. Tracepoints | |
220 | provide a way to instrument code more cleanly and are suited for permanent | |
221 | instrumentation. | |
222 | ||
223 | In addition to executable programs, shared libraries may also be instrumented | |
224 | with the methods described in this chapter. | |
225 | ||
226 | @menu | |
227 | * Markers:: | |
228 | * Tracepoints:: | |
229 | @end menu | |
230 | ||
231 | @node Markers | |
232 | @section Markers | |
233 | ||
70e55dbe | 234 | Adding a marker is simply a matter of inserting one line in the program. |
5cd1099a PMF |
235 | |
236 | @example | |
237 | @verbatim | |
238 | #include <ust/marker.h> | |
239 | ||
240 | int main(int argc, char **argv) | |
241 | { | |
242 | int v; | |
243 | char *st; | |
244 | ||
245 | /* ... set values of v and st ... */ | |
246 | ||
247 | /* a marker: */ | |
248 | trace_mark(main, myevent, "firstarg %d secondarg %s", v, st); | |
249 | ||
70e55dbe | 250 | /* another marker without arguments: */ |
5cd1099a PMF |
251 | trace_mark(main, myotherevent, MARK_NOARGS); |
252 | ||
253 | return 0; | |
254 | } | |
255 | @end verbatim | |
256 | @end example | |
257 | ||
258 | The invocation of the trace_mark() macro requires at least 3 arguments. The | |
259 | first, here "main", is the name of the event category. It is also the name of | |
260 | the channel the event will go in. The second, here "myevent" is the name of the | |
261 | event. The third is a format string that announces the names and the types of | |
262 | the event arguments. Its format resembles that of a printf() format string; it | |
263 | is described thoroughly in Appendix x. | |
264 | ||
265 | A given Marker may appear more than once in the same program. Other Markers may | |
266 | have the same name and a different format string, although this might induce | |
267 | some confusion at analysis time. | |
268 | ||
269 | @node Tracepoints | |
270 | @section Tracepoints | |
271 | ||
272 | The Tracepoints API uses the Markers, but provides a higher-level abstraction. | |
273 | Whereas the markers API provides limited type checking, the Tracepoints API | |
274 | provides more thorough type checking and discharges from the need to insert | |
275 | format strings directly in the code and to have format strings appear more than | |
276 | once if a given marker is reused. | |
277 | ||
bb0496b2 PMF |
278 | @quotation Note |
279 | Although this example uses @emph{mychannel} as the channel, the | |
5cd1099a PMF |
280 | only channel name currently supported with early tracing is @strong{ust}. The |
281 | @command{usttrace} tool always uses the early tracing mode. When using manual | |
bb0496b2 PMF |
282 | mode without early tracing, any channel name may be used. |
283 | @end quotation | |
5cd1099a PMF |
284 | |
285 | A function instrumented with a tracepoint looks like this: | |
286 | ||
287 | @example | |
288 | @verbatim | |
289 | #include "tp.h" | |
290 | ||
291 | void function() | |
292 | { | |
293 | int v; | |
294 | char *st; | |
295 | ||
296 | /* ... set values of v and st ... */ | |
297 | ||
298 | /* a tracepoint: */ | |
299 | trace_mychannel_myevent(v, st); | |
300 | } | |
301 | @end verbatim | |
302 | @end example | |
303 | ||
304 | Another file, here tp.h, contains declarations for the tracepoint. | |
305 | ||
306 | @example | |
307 | @verbatim | |
308 | #include <ust/tracepoint.h> | |
309 | ||
23e3d1f6 PMF |
310 | DECLARE_TRACE(mychannel_myevent, TP_PROTO(int v, char *st), |
311 | TP_ARGS(v, st)); | |
5cd1099a PMF |
312 | @end verbatim |
313 | @end example | |
314 | ||
315 | A third file, here tp.c, contains definitions for the tracepoint. | |
316 | ||
317 | @example | |
318 | @verbatim | |
319 | #include <ust/marker.h> | |
320 | #include "tp.h" | |
321 | ||
322 | DEFINE_TRACE(mychannel_myevent); | |
323 | ||
324 | void mychannel_myevent_probe(int v, char *st) | |
325 | { | |
326 | trace_mark(mychannel, myevent, "v %d st %s", v, st); | |
327 | } | |
328 | ||
329 | static void __attribute__((constructor)) init() | |
330 | { | |
331 | register_trace_mychannel_myevent(mychannel_myevent_probe); | |
332 | } | |
333 | @end verbatim | |
334 | @end example | |
335 | ||
336 | Here, tp.h and tp.c could contain declarations and definitions for other | |
337 | tracepoints. The constructor would contain other register_* calls. | |
338 | ||
339 | @node Recording a trace | |
340 | @chapter Recording a trace | |
341 | ||
342 | @menu | |
343 | * Using @command{usttrace}:: | |
344 | * Setting up the recording manually:: | |
345 | * Using early tracing:: | |
346 | * Crash recovery:: | |
347 | * Tracing across @code{fork()} and @code{clone()}:: | |
348 | * Tracing programs and libraries that were not linked to libust:: | |
349 | @end menu | |
350 | ||
351 | @node Using @command{usttrace} | |
352 | @section Using @command{usttrace} | |
353 | ||
354 | The simplest way to record a trace is to use the @command{usttrace} script. An | |
355 | example is given in the quickstart above. | |
356 | ||
357 | The @command{usttrace} script automatically: | |
358 | @itemize @bullet | |
359 | @item creates a daemon | |
360 | @item enables all markers | |
361 | @item runs the command specified on the command line | |
362 | @item after the command ends, prints the location where the trace was saved | |
363 | @end itemize | |
364 | ||
365 | Each subdirectory of the save location contains the trace of one process that | |
366 | was generated by the command. The name of a subdirectory consists in the the PID | |
367 | of the process, followed by the timestamp of its creation. | |
368 | ||
369 | The save location also contains logs of the tracing. | |
370 | ||
371 | When using @command{usttrace}, the early tracing is always active, which means | |
372 | that the tracing is guaranteed to be started by the time the process enters its | |
373 | main() function. | |
374 | ||
375 | Several @command{usttrace}'s may be run simultaneously without risk of | |
376 | conflict. This facilitates the use of the tracer by idependent users on a | |
377 | system. Each instance of @command{usttrace} starts its own daemon which | |
378 | collects the events of the processes it creates. | |
379 | ||
380 | @node Setting up the recording manually | |
381 | @section Setting up the recording manually | |
382 | ||
383 | Instead of using @command{usttrace}, a trace may be recorded on an already | |
384 | running process. | |
385 | ||
386 | First the daemon must be started. | |
387 | ||
388 | @example | |
389 | @verbatim | |
390 | # Make sure the directory for the communication sockets exists. | |
391 | $ mkdir /tmp/ustsocks | |
392 | ||
393 | # Make sure the directory where ustd will write the trace exists. | |
394 | $ mkdir /tmp/trace | |
395 | ||
396 | # Start the daemon | |
397 | $ ustd | |
398 | ||
399 | # We assume the program we want to trace is already running and that | |
400 | # it has pid 1234. | |
401 | ||
402 | # List the available markers | |
403 | $ ustctl --list-markers 1234 | |
404 | # A column indicates 0 for an inactive marker and 1 for an active marker. | |
405 | ||
406 | # Enable a marker | |
e61c1b29 | 407 | $ ustctl --enable-marker ust/mymark 1234 |
5cd1099a PMF |
408 | |
409 | # Create a trace | |
410 | $ ustctl --create-trace 1234 | |
411 | ||
412 | # Start tracing | |
413 | $ ustctl --start-trace 1234 | |
414 | ||
415 | # Do things... | |
416 | ||
417 | # Stop tracing | |
418 | $ ustctl --stop-trace 1234 | |
1af1f1db PMF |
419 | |
420 | # Destroy the trace | |
421 | $ ustctl --destroy-trace 1234 | |
5cd1099a PMF |
422 | @end verbatim |
423 | @end example | |
424 | ||
c3947c25 PMF |
425 | For more information about the manual mode, see the ustctl(1) man page. |
426 | ||
5cd1099a PMF |
427 | @node Using early tracing |
428 | @section Using early tracing | |
429 | ||
430 | Early tracing consists in starting the tracing as early as possible in the | |
431 | program, so no events are lost between program start and the point where the | |
432 | command to start the tracing is given. When using early tracing, it is | |
433 | guaranteed that by the time the traced program enters its @code{main()} | |
434 | function, the tracing will be started. | |
435 | ||
436 | When using @command{usttrace}, the early tracing is always active. | |
437 | ||
438 | When using the manual mode (@command{ustctl}), early tracing is enabled using | |
439 | environment variables. Setting @env{UST_TRACE} to @code{1}, enables early | |
440 | tracing, while setting @env{UST_AUTOPROBE} to @code{1} enables all markers | |
441 | automatically. | |
442 | ||
443 | ||
444 | @node Crash recovery | |
445 | @section Crash recovery | |
446 | ||
447 | When a process being traced crashes, the daemon is able to recover all the | |
448 | events in its buffers that were successfully commited. This is possible because | |
449 | the buffers are in a shared memory segment which remains available to the | |
450 | daemon even after the termination of the traced process. | |
451 | ||
452 | @node Tracing across @code{fork()} and @code{clone()} | |
453 | @section Tracing across @code{fork()} and @code{clone()} | |
454 | ||
2669948d PMF |
455 | Tracing across @code{clone()} when the @code{CLONE_VM} flag is specified is |
456 | supported without any particular action. | |
5cd1099a PMF |
457 | |
458 | When @code{clone()} is called without @code{CLONE_VM} or @code{fork()} is | |
459 | called, a new address space is created and the tracer must be notified to | |
2669948d PMF |
460 | create new buffers for it. |
461 | ||
462 | This can be done automatically, by @env{LD_PRELOAD}'ing @file{libinterfork.so}. | |
463 | This library intercepts calls to @code{fork()} and informs the tracer it is | |
464 | being called. When using @command{usttrace}, this is accomplied by specifying | |
465 | the @option{-f} command line argument. | |
466 | ||
467 | Alternatively, the program can call @code{ust_before_fork()} before calling | |
468 | @code{fork()} or @code{clone()} with @code{CLONE_VM}. After the call, | |
469 | @code{ust_after_fork_parent()} must be called in the parent process and | |
470 | @code{ust_after_fork_child()} must be called in the child process. | |
5cd1099a | 471 | |
5cd1099a PMF |
472 | |
473 | @node Tracing programs and libraries that were not linked to libust | |
474 | @section Tracing programs and libraries that were not linked to libust | |
475 | ||
2669948d PMF |
476 | Some programs need to be traced even though they were not linked to libust |
477 | either because they were not instrumented or because it was not practical. | |
478 | ||
479 | An executable that is not instrumented can still yield interesting traces when | |
480 | at least one of its dynamic libraries is instrumented. It is also possible to | |
481 | trace certain function calls by intercepting them with a specially crafted | |
482 | library that is linked with @env{LD_PRELOAD} at program start. | |
483 | ||
484 | In any case, a program that was not linked to libust at compile time must be | |
485 | linked to it at run time with @env{LD_PRELOAD}. This can be accomplished with | |
486 | @command{usttrace}'s @option{-l} option. It can also be done by setting the | |
487 | @env{LD_PRELOAD} environment variable on the command line. For example: | |
488 | ||
489 | @example | |
490 | @verbatim | |
491 | # Run ls with usttrace, LD_PRELOAD'ing libust | |
492 | # (assuming one of the libraries used by ls is instrumented). | |
493 | $ usttrace -l ls | |
494 | ||
495 | # Run ls, manually adding the LD_PRELOAD. | |
496 | $ LD_PRELOAD=/usr/local/lib/libust.so.0 ls | |
497 | @end verbatim | |
498 | @end example | |
499 | ||
5cd1099a PMF |
500 | |
501 | @node Performance | |
502 | @chapter Performance | |
503 | ||
504 | Todo. | |
505 | ||
506 | @node Viewing traces | |
507 | @chapter Viewing traces | |
508 | ||
2669948d PMF |
509 | Traces may be viewed with LTTV. An example of command for launching LTTV is |
510 | given in the quickstart. | |
511 | ||
512 | @menu | |
513 | * Viewing multiple traces:: | |
514 | * Combined kernel-userspace tracing:: | |
515 | @end menu | |
516 | ||
517 | @node Viewing multiple traces | |
518 | @section Viewing multiple traces | |
519 | ||
520 | When tracing multi-process applications or several applications simultaneously, | |
521 | more than one trace will be obtained. LTTV can open and display all these | |
522 | traces simultaneously. | |
523 | ||
524 | @node Combined kernel-userspace tracing | |
525 | @section Combined kernel-userspace tracing | |
526 | ||
527 | In addition to multiple userspace traces, LTTV can open a kernel trace recorded | |
528 | with the LTTng kernel tracer. This provides events that enable the rendering of | |
529 | the Control Flow View and the Resource View. | |
530 | ||
531 | When doing so, it is necessary to use the same time source for the kernel | |
532 | tracer as well as the userspace tracer. Currently, the recommended method is to | |
533 | use the timestamp counter for both. The TSC can however only be used on architectures | |
534 | where it is synchronized across cores. | |
535 | ||
bb0496b2 PMF |
536 | @node Resource Usage |
537 | @chapter Resource Usage | |
538 | ||
539 | The purpose of this section is to give an overview of the resource usage of libust. For | |
540 | a developer, knowing this can be important: because libust is linked with applications, it | |
541 | needs to share some resources with it. Some applications may make some assumptions that are in | |
542 | conflict with libust's usage of resources. | |
543 | ||
544 | In practice however, libust is designed to be transparent and is compatible | |
545 | with the vast majority of applications. This means no changes are required in | |
546 | the application (or library) being linked to libust. | |
547 | ||
548 | Libust is initialized by a constructor, which by definition runs before the main() function | |
549 | of the application starts. This constructor creates a thread called the @emph{listener thread}. | |
550 | The listener thread initializes a named socket and waits for connections for ustd or ustctl. | |
551 | ||
552 | Libust-specific code may: | |
553 | @itemize @bullet | |
554 | @item use @code{malloc()} and @code{free()} | |
555 | @item map shared memory segment in the process adress space | |
556 | @item intercept some library calls, specifically @code{fork()} and @code{clone()} | |
557 | @item do interprocess communication with the daemon or ustctl | |
558 | @item create and open named sockets | |
559 | ||
560 | @end itemize | |
561 | ||
562 | It will not: | |
563 | @itemize @bullet | |
564 | @item handle any signal (all signals are blocked in the listener thread) | |
565 | @item change any process-wide setting that could confuse the application | |
566 | @end itemize | |
567 | ||
568 | ||
5cd1099a | 569 | @bye |