| 1 | <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> |
| 2 | <html><head> |
| 3 | <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1"> |
| 4 | |
| 5 | |
| 6 | <title>LTTng User-space Tracing Design</title> |
| 7 | </head><body> |
| 8 | |
| 9 | <h1>LTTng User-space Tracing Design</h1> |
| 10 | |
| 11 | <p> |
| 12 | Earlier versions of LTTng contained simple, system call based, tracepoints and fast, |
| 13 | user-space buffered, tracepoints for user-space tracing. During the kernel inclusion phase |
| 14 | of LTTng, extensive rework and modularization of the kernel tracing portion was undertaken. |
| 15 | This phase is well under way and several portions have been included already in the mainline kernel. |
| 16 | The rework of the kernel tracing infrastructure will shortly thereafter be ported to fast user-space |
| 17 | tracing. |
| 18 | This fast user-space tracing scheme uses a direct function call to write events into buffers |
| 19 | mapped in user-space. This should be an order of magnitude faster than the |
| 20 | current Dtrace implementation (c.f. |
| 21 | <a href="http://ltt.polymtl.ca/tracingwiki/index.php/DTrace">Dtrace information |
| 22 | on the TracingWiki</a>) which uses a breakpoint to |
| 23 | perform both dynamic and static user-space tracing. Performance comparison of |
| 24 | a function call vs the int3 approach is available at |
| 25 | <a href="http://sourceware.org/ml/systemtap/2006-q3/msg00793.html">Markers vs |
| 26 | int3 performance comparison</a> (see "Conclusion"). |
| 27 | |
| 28 | </p><p> |
| 29 | Libmarkers will provide applications with user-space Markers and Tracepoints |
| 30 | declarations, such that programmers will be able to insert Markers and Tracepoints in their |
| 31 | libraries and applications. User-space Tracepoints and Markers, analogous to Kernel Tracepoints and Markers, |
| 32 | define program locations and arguments provided. Libmarkers will also |
| 33 | provide utility functions to enumerate, activate and deactivate tracepoints and markers in the |
| 34 | process, and to associate probes with any tracepoint or marker. |
| 35 | <ul><li> Linux Kernel Tracepoints documentation: <a href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=Documentation/tracepoints.txt" class="external text" title="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=Documentation/tracepoints.txt" rel="nofollow">Documentation/tracepoints.txt</a> |
| 36 | </li><li> Linux Kernel Tracepoints example: <a href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=samples/tracepoints" class="external text" title="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=samples/tracepoints" rel="nofollow">samples/tracepoints</a> |
| 37 | </li><li> Linux Kernel Markers documentation: <a href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=Documentation/markers.txt" class="external text" title="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=Documentation/markers.txt" rel="nofollow">Documentation/markers.txt</a> |
| 38 | </li><li> Linux Kernel Markers example: <a href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=samples/markers" class="external text" title="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=samples/markers" rel="nofollow">samples/markers</a> |
| 39 | </ul> |
| 40 | |
| 41 | |
| 42 | </p><p> |
| 43 | Libtracing will provide the infrastructure to allocate buffers, define event types, write event |
| 44 | metadata and data to the buffers, and get notification when buffers are full. The initial implementation |
| 45 | will simply use one set of buffers per process. Subsequent more optimized versions will allocate one |
| 46 | set of buffers per thread; one set of buffers per CPU would be desirable but user-space programs cannot check |
| 47 | or control CPU migration without resorting to more costly bus locking operations or system calls. The library |
| 48 | provides a generic probe for markers which, when connected, generates an event in the buffer each time the marker |
| 49 | is encountered. |
| 50 | |
| 51 | <UL> |
| 52 | </li><li> LTTng Linux Kernel Markers Efficicient API: <a href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=ltt/probes/ltt-type-serializer.h" class="external text" title="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=ltt/probes/ltt-type-serializer.h" rel="nofollow">ltt/probes/ltt-type-serializer.h</a> |
| 53 | </li><li> LTTng Linux Kernel Markers Efficicient API usage example: <a href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=ltt/probes/kernel-trace.c" class="external text" title="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=ltt/probes/kernel-trace.c" rel="nofollow">ltt/probes/kernel-trace.c</a> |
| 54 | </li> |
| 55 | </UL> |
| 56 | </p><p> |
| 57 | Finally, libtracingcontrol opens a connection allowing a separate process (e.g. LTTng daemon, Eclipse, GDB) to control the |
| 58 | tracing within the application. Through this connection, the remote process can: |
| 59 | |
| 60 | <UL> |
| 61 | <li> list the available Markers and Tracepoints; |
| 62 | </li><li> dynamically load a library (presumably containing probes to connect); |
| 63 | </li><li> connect a probe to a Tracepoint or Marker; |
| 64 | </li><li> activate a Tracepoint or Marker; |
| 65 | </li><li> initialize the tracing buffers; |
| 66 | </li><li> retrieve the content of the tracing buffers; |
| 67 | </li><li> flush the tracing buffers; |
| 68 | </li><li> finalize the tracing buffers; |
| 69 | </li> |
| 70 | </UL> |
| 71 | |
| 72 | In the first version, libtracingcontrol will simply contain the basic functionality |
| 73 | required to generate a trace of the Markers encountered. Subsequent versions will interact with |
| 74 | other probing mechanisms like GDB Tracepoints, user-space SystemTap and those defined by virtual |
| 75 | machines (e.g. Java Virtual Machine, Python, Errlang...). |
| 76 | </p><p> |
| 77 | Tracing of Java application is planned to be done through a JNI interface. |
| 78 | Linking standard low-level C tracing library to the application within a JNI |
| 79 | adaptation class will be required to trace Java events. This has been prototyped |
| 80 | in the past. The work is available |
| 81 | <a href="http://ltt.polymtl.ca/svn/trunk/obsolete/ltt-usertrace/java/">here</a> |
| 82 | for older LTTng versions. |
| 83 | <p></p> |
| 84 | The principle of operation of libtracingcontrol is that |
| 85 | when the instrumented application starts, a pipe is opened to allow external |
| 86 | tracing control. Asynchronous notification is requested when commands arrive in |
| 87 | the pipe, and a signal handler is installed for SIGIO (or a carefully chosen |
| 88 | chainable signal number). Every time such signal is received, the runtime |
| 89 | library checks for commands received on the external tracing control pipe. |
| 90 | The application may also spontaneously provide information to the remote control process |
| 91 | through the pipe: |
| 92 | |
| 93 | </p><ul></li><li> a buffer is full; |
| 94 | </li><li> new tracepoints or markers are appearing or disappearing, because modules are |
| 95 | dynamically loaded or unloaded; |
| 96 | </li></ul> |
| 97 | <p> |
| 98 | In addition, the tracing control application should be notified when the |
| 99 | application exits (to save the content of buffers if the application is |
| 100 | crashing) or forks (to trace the child as well if needed). |
| 101 | Such notification may be obtained through utrace. |
| 102 | |
| 103 | </p><p> |
| 104 | In summary, a user-space application linking with libmarkers may contain static |
| 105 | instrumentation, Tracepoints and Markers, just like the kernel with Kernel Markers and Tracepoints. |
| 106 | The application can exploit this instrumentation itself or link with libtracing and have tracing probes |
| 107 | connected to each Marker. Other instrumentation mechanisms, like the GCC instrument-function option, or hooks |
| 108 | inserted by a JIT compiler, can also use libtracing to define and write event to the trace buffers. |
| 109 | Finally, libtracingcontrol, analogous to GDB stubs, allows the remote control of the tracing by remote |
| 110 | monitoring or debugging frameworks. |
| 111 | |
| 112 | </p></body></html> |