[lttv.git] / doc / developer / ust.html

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html><head>
<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">


  <title>LTTng User-space Tracing Design</title>
</head><body>

<h1>LTTng User-space Tracing Design</h1>

<p>
Earlier versions of LTTng contained simple, system call based, tracepoints and fast,
user-space buffered, tracepoints for user-space tracing. During the kernel inclusion phase
of LTTng, extensive rework and modularization of the kernel tracing portion was undertaken.
This phase is well under way and several portions have been included already in the mainline kernel.
The rework of the  kernel tracing infrastructure will shortly thereafter be ported to fast user-space
tracing.
This fast user-space tracing scheme uses a direct function call to write events into buffers
mapped in user-space. This should be an order of magnitude faster than the
current Dtrace implementation (c.f.
<a href="http://ltt.polymtl.ca/tracingwiki/index.php/DTrace">Dtrace information
on the TracingWiki</a>) which uses a breakpoint to
perform both dynamic and static user-space tracing. Performance comparison of
a function call vs the int3 approach is available at
<a href="http://sourceware.org/ml/systemtap/2006-q3/msg00793.html">Markers vs
int3 performance comparison</a> (see "Conclusion").

</p><p>
Libmarkers will provide applications with user-space Markers and Tracepoints
declarations, such that programmers will be able to insert Markers and Tracepoints in their
libraries and applications. User-space Tracepoints and Markers, analogous to Kernel Tracepoints and Markers,
define program locations and arguments provided. Libmarkers will also
provide utility functions to enumerate, activate and deactivate tracepoints and markers in the
process, and to associate probes with any tracepoint or marker. 
<ul><li> Linux Kernel Tracepoints documentation: <a href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=Documentation/tracepoints.txt" class="external text" title="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=Documentation/tracepoints.txt" rel="nofollow">Documentation/tracepoints.txt</a>
</li><li> Linux Kernel Tracepoints example: <a href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=samples/tracepoints" class="external text" title="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=samples/tracepoints" rel="nofollow">samples/tracepoints</a>
</li><li> Linux Kernel Markers documentation: <a href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=Documentation/markers.txt" class="external text" title="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=Documentation/markers.txt" rel="nofollow">Documentation/markers.txt</a>
</li><li> Linux Kernel Markers example: <a href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=samples/markers" class="external text" title="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=samples/markers" rel="nofollow">samples/markers</a>
</ul>


</p><p>
Libtracing will provide the infrastructure to allocate buffers, define event types, write event
metadata and data to the buffers, and get notification when buffers are full. The initial implementation
will simply use one set of buffers per process. Subsequent more optimized versions will allocate one
set of buffers per thread; one set of buffers per CPU would be desirable but user-space programs cannot check
or control CPU migration without resorting to more costly bus locking operations or system calls. The library
provides a generic probe for markers which, when connected, generates an event in the buffer each time the marker
is encountered.

<UL>
</li><li> LTTng Linux Kernel Markers Efficicient API: <a href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=ltt/probes/ltt-type-serializer.h" class="external text" title="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=ltt/probes/ltt-type-serializer.h" rel="nofollow">ltt/probes/ltt-type-serializer.h</a>
</li><li> LTTng Linux Kernel Markers Efficicient API usage example: <a href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=ltt/probes/kernel-trace.c" class="external text" title="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=ltt/probes/kernel-trace.c" rel="nofollow">ltt/probes/kernel-trace.c</a>
</li>
</UL>
</p><p>
Finally, libtracingcontrol opens a connection allowing a separate process (e.g. LTTng daemon, Eclipse, GDB) to control the 
tracing within the application. Through this connection, the remote process can:

<UL>
<li> list the available Markers and Tracepoints;
</li><li> dynamically load a library (presumably containing probes to connect);
</li><li> connect a probe to a Tracepoint or Marker;
</li><li> activate a Tracepoint or Marker;
</li><li> initialize the tracing buffers;
</li><li> retrieve the content of the tracing buffers;
</li><li> flush the tracing buffers;
</li><li> finalize the tracing buffers;
</li>
</UL>

In the first version, libtracingcontrol will simply contain the basic functionality
required to generate a trace of the Markers encountered. Subsequent versions will interact with
other probing mechanisms like GDB Tracepoints, user-space SystemTap and those defined by virtual 
machines (e.g. Java Virtual Machine, Python, Errlang...).
</p><p>
Tracing of Java application is planned to be done through a JNI interface.
Linking standard low-level C tracing library to the application within a JNI
adaptation class will be required to trace Java events. This has been prototyped
in the past. The work is available
<a href="http://ltt.polymtl.ca/svn/trunk/obsolete/ltt-usertrace/java/">here</a>
for older LTTng versions.
<p></p>
The principle of operation of libtracingcontrol is that
when the instrumented application starts, a pipe is opened to allow external
tracing control. Asynchronous notification is requested when commands arrive in
the pipe, and a signal handler is installed for SIGIO (or a carefully chosen
chainable signal number). Every time such signal is received, the runtime
library checks for commands received on the external tracing control pipe.
The application may also spontaneously provide information to the remote control process
through the pipe:

</p><ul></li><li> a buffer is full;
</li><li> new tracepoints or markers are appearing or disappearing, because modules are
  dynamically loaded or unloaded;
</li></ul>
<p>
In addition, the tracing control application should be notified when the
application exits (to save the content of buffers if the application is
crashing) or forks (to trace the child as well if needed). 
Such notification may be obtained through utrace.

</p><p>
In summary, a user-space application linking with libmarkers may contain static
instrumentation, Tracepoints and Markers, just like the kernel with Kernel Markers and Tracepoints.
The application can exploit this instrumentation itself or link with libtracing and have tracing probes
connected to each Marker. Other instrumentation mechanisms, like the GCC instrument-function option, or hooks
inserted by a JIT compiler, can also use libtracing to define and write event to the trace buffers. 
Finally, libtracingcontrol, analogous to GDB stubs, allows the remote control of the tracing by remote
monitoring or debugging frameworks.

</p></body></html>
Commit	Line	Data
baec4c72	1	<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
	2	<html><head>
	3	<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
d4c85b6e	4
d4c85b6e	5
baec4c72	6	<title>LTTng User-space Tracing Design</title>
baec4c72	7	</head><body>
d4c85b6e	8
baec4c72	9	<h1>LTTng User-space Tracing Design</h1>
d4c85b6e	10
ca3fb06a	11	<p>
baec4c72	12	Earlier versions of LTTng contained simple, system call based, tracepoints and fast,
	13	user-space buffered, tracepoints for user-space tracing. During the kernel inclusion phase
	14	of LTTng, extensive rework and modularization of the kernel tracing portion was undertaken.
	15	This phase is well under way and several portions have been included already in the mainline kernel.
	16	The rework of the kernel tracing infrastructure will shortly thereafter be ported to fast user-space
	17	tracing.
	18	This fast user-space tracing scheme uses a direct function call to write events into buffers
14b9349e	19	mapped in user-space. This should be an order of magnitude faster than the
14b9349e	20	current Dtrace implementation (c.f.
9dd3d20d	21	<a href="http://ltt.polymtl.ca/tracingwiki/index.php/DTrace">Dtrace information
9dd3d20d	22	on the TracingWiki</a>) which uses a breakpoint to
14b9349e	23	perform both dynamic and static user-space tracing. Performance comparison of
14b9349e	24	a function call vs the int3 approach is available at
9dd3d20d	25	<a href="http://sourceware.org/ml/systemtap/2006-q3/msg00793.html">Markers vs
9dd3d20d	26	int3 performance comparison</a> (see "Conclusion").
e6f8d5e6	27
baec4c72	28	</p><p>
	29	Libmarkers will provide applications with user-space Markers and Tracepoints
	30	declarations, such that programmers will be able to insert Markers and Tracepoints in their
	31	libraries and applications. User-space Tracepoints and Markers, analogous to Kernel Tracepoints and Markers,
	32	define program locations and arguments provided. Libmarkers will also
	33	provide utility functions to enumerate, activate and deactivate tracepoints and markers in the
	34	process, and to associate probes with any tracepoint or marker.
	35	<ul><li> Linux Kernel Tracepoints documentation: <a href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=Documentation/tracepoints.txt" class="external text" title="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=Documentation/tracepoints.txt" rel="nofollow">Documentation/tracepoints.txt</a>
	36	</li><li> Linux Kernel Tracepoints example: <a href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=samples/tracepoints" class="external text" title="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=samples/tracepoints" rel="nofollow">samples/tracepoints</a>
	37	</li><li> Linux Kernel Markers documentation: <a href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=Documentation/markers.txt" class="external text" title="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=Documentation/markers.txt" rel="nofollow">Documentation/markers.txt</a>
	38	</li><li> Linux Kernel Markers example: <a href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=samples/markers" class="external text" title="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=samples/markers" rel="nofollow">samples/markers</a>
	39	</ul>
	40
	41
	42	</p><p>
	43	Libtracing will provide the infrastructure to allocate buffers, define event types, write event
	44	metadata and data to the buffers, and get notification when buffers are full. The initial implementation
	45	will simply use one set of buffers per process. Subsequent more optimized versions will allocate one
	46	set of buffers per thread; one set of buffers per CPU would be desirable but user-space programs cannot check
	47	or control CPU migration without resorting to more costly bus locking operations or system calls. The library
	48	provides a generic probe for markers which, when connected, generates an event in the buffer each time the marker
	49	is encountered.
	50
	51	<UL>
	52	</li><li> LTTng Linux Kernel Markers Efficicient API: <a href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=ltt/probes/ltt-type-serializer.h" class="external text" title="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=ltt/probes/ltt-type-serializer.h" rel="nofollow">ltt/probes/ltt-type-serializer.h</a>
	53	</li><li> LTTng Linux Kernel Markers Efficicient API usage example: <a href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=ltt/probes/kernel-trace.c" class="external text" title="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=ltt/probes/kernel-trace.c" rel="nofollow">ltt/probes/kernel-trace.c</a>
	54	</li>
	55	</UL>
	56	</p><p>
	57	Finally, libtracingcontrol opens a connection allowing a separate process (e.g. LTTng daemon, Eclipse, GDB) to control the
	58	tracing within the application. Through this connection, the remote process can:
	59
	60	<UL>
	61	<li> list the available Markers and Tracepoints;
	62	</li><li> dynamically load a library (presumably containing probes to connect);
	63	</li><li> connect a probe to a Tracepoint or Marker;
	64	</li><li> activate a Tracepoint or Marker;
	65	</li><li> initialize the tracing buffers;
	66	</li><li> retrieve the content of the tracing buffers;
	67	</li><li> flush the tracing buffers;
	68	</li><li> finalize the tracing buffers;
	69	</li>
	70	</UL>
	71
	72	In the first version, libtracingcontrol will simply contain the basic functionality
	73	required to generate a trace of the Markers encountered. Subsequent versions will interact with
	74	other probing mechanisms like GDB Tracepoints, user-space SystemTap and those defined by virtual
	75	machines (e.g. Java Virtual Machine, Python, Errlang...).
	76	</p><p>
e6f8d5e6	77	Tracing of Java application is planned to be done through a JNI interface.
	78	Linking standard low-level C tracing library to the application within a JNI
	79	adaptation class will be required to trace Java events. This has been prototyped
3ceafa01	80	in the past. The work is available
baec4c72	81	<a href="http://ltt.polymtl.ca/svn/trunk/obsolete/ltt-usertrace/java/">here</a>
3ceafa01	82	for older LTTng versions.
baec4c72	83	<p></p>
	84	The principle of operation of libtracingcontrol is that
	85	when the instrumented application starts, a pipe is opened to allow external
	86	tracing control. Asynchronous notification is requested when commands arrive in
	87	the pipe, and a signal handler is installed for SIGIO (or a carefully chosen
	88	chainable signal number). Every time such signal is received, the runtime
	89	library checks for commands received on the external tracing control pipe.
	90	The application may also spontaneously provide information to the remote control process
	91	through the pipe:
	92
	93	</p><ul></li><li> a buffer is full;
	94	</li><li> new tracepoints or markers are appearing or disappearing, because modules are
	95	dynamically loaded or unloaded;
	96	</li></ul>
	97	<p>
	98	In addition, the tracing control application should be notified when the
	99	application exits (to save the content of buffers if the application is
	100	crashing) or forks (to trace the child as well if needed).
	101	Such notification may be obtained through utrace.
	102
	103	</p><p>
	104	In summary, a user-space application linking with libmarkers may contain static
	105	instrumentation, Tracepoints and Markers, just like the kernel with Kernel Markers and Tracepoints.
	106	The application can exploit this instrumentation itself or link with libtracing and have tracing probes
	107	connected to each Marker. Other instrumentation mechanisms, like the GCC instrument-function option, or hooks
	108	inserted by a JIT compiler, can also use libtracing to define and write event to the trace buffers.
	109	Finally, libtracingcontrol, analogous to GDB stubs, allows the remote control of the tracing by remote
	110	monitoring or debugging frameworks.
e6f8d5e6	111
baec4c72	112	</p></body></html>