LTTng usertrace package Mathieu Desnoyers March 2006 This package contains all the user space headers and c files necessary to make your application and library trace through an active LTTng tracer. Here is a short quickstart guide of it. Here are the currently supported architectures : x86 (please add the ltt_trace_generic and ltt_register_generic system calls to other architectures as you need them : it will work magically) * Compile your kernel with the latest LTTng patch. Make sure the option "Allow tracing from userspace" is _active_! See the QUICKSTART guide at http://ltt.polymtl.ca/ for details about how to setup a working tracer and viewer. See the genevent installation step : it is required for method #2 below. * Extract the latest ltt-usertrace archive : su cd /usr/src wget http://ltt.polymtl.ca/packages/ltt-usertrace-x.x.tar.gz gzip -cd ltt-usertrace-x.x.tar.gz | tar xvof - * Build the sample programs and install the headers and librairies into your system : su cd /usr/src/ltt-usertrace make make install Feel free to look at the sample programs and the Makefile : they demonstrate very well the features of the usertrace package and how to use them. * There are three ways to trace information from your application. The choice will principally depend on the trace data rate. 1) Easy way, but slow (printf style) See sample-printf.c for code example. - Add the following statements to your program source (the define must come _before_ the includes!) : #define LTT_TRACE #define LTT_BLOCKING 1 #include #include Note the define of LTT_BLOCKING to 1 : if a trace buffer is full, your application will block. The default of this parameter is 0 (non blocking) : events are lost when trace buffer is full. The choice is up to you. - Add something like the following sample line in your code. Note that this is a very standard format string, this is only a suggested presentation. trace_user_generic_slow_printf("in: %s at: %s:%d: Counter value is: %u.", __FILE__, __func__, __LINE__, count); - Compile your application with at least these parameters to gcc (it is splitted on two lines, joined by a "\") : gcc -D LTT_SHOW_DEBUG -I /usr/src/usertrace-generic -o myapp myapp.c \ /usr/src/usertrace-generic/ltt-facility-loader-user_generic.c To see what the final result looks like : - Start tracing - Start your application ** You should see the following message when your program starts and the LTT_SHOW_DEBUG is defined : "LTT : ltt-facility-user_generic init in userspace" If you don't then you forgot to compile the facility loader in your application. If you find this output annoying, you can remove the "-D LTT_SHOW_DEBUG" gcc parameter, which will make the facility loader silent. - Stop tracing Then, to see only the user_generic events : lttv -m textDump -t /tmp/trace1 -e "event.facility=user_generic" It will show : user_generic.slow_printf: 35885.922829472 (/cpu_0), 15521, 7453, SYSCALL { "in: sample-printf.c at: main:18: Counter value is: 0." } user_generic.slow_printf: 35886.925685289 (/cpu_0), 15521, 7453, SYSCALL { "in: sample-printf.c at: main:18: Counter value is: 1." } ... 2) The second way to log events is still easy. The advantage is that it will make it easier to identify your data in the trace viewer afterward. Please read the comments in method 1) explained previously, as they are not repeated here. See sample.c and sample-thread-slow.c for code example. - Go to the ltt-usertrace directory su cd /usr/src/ltt-usertrace - Create your own facility (i.e. user_myfacility.xml). See the ones available in /usr/share/LinuxTraceToolkitViewer/facilities for examples. You facility _must_ be named following this standard : "user_*", where * is whatever you like. If it is not, it will be rejected by the kernel with a Operation not permitted (can be seen with the -D LTT_SHOW_DEBUG compilation parameter). user_myfacility.xml: Sample facility Sample event - AN IMPORTANT STEP FOLLOWS : *copy* the user_myfacility.xml file in your system : su cp user_myfacility.xml /usr/share/LinuxTraceToolkitViewer/facilities - Use genevent to create the c code and headers : su cd /tmp mkdir genevent cd genevent for a in /usr/share/LinuxTraceToolkitViewer/facilities/user_*.xml; do /usr/local/bin/genevent $a; done cd /usr/src/usertrace-generic cp /tmp/genevent/*load* . cd ltt cp /tmp/genevent/ltt-facility-id-user_myfacility.h . cp /tmp/genevent/ltt-facility-user_myfacility.h . cd .. make install - Add the following statements to your program source (the define must come _before_ the includes!) : #define LTT_TRACE #define LTT_BLOCKING 1 #include - Add a call following the trace_user_myfacility_myevent function found in /usr/include/ltt/ltt-facility-user_myfacility.h in your program. For instance : trace_user_myfacility_myevent(__FILE__, __func__, __LINE__, 1234, (void*)0xF0F0F0F0); - Compile your application with at least these parameters to gcc (it is splitted on two lines, joined by a "\") : gcc -I /usr/src/usertrace-generic -o myapp myapp.c \ /usr/src/usertrace-generic/ltt-facility-loader-user_myfacility.c To see what the final result looks like : - Start tracing - Start your application - Stop tracing Then, to see only the user_myfacility events : lttv -m textDump -t /tmp/trace1 -e "event.facility=user_myfacility" It will show, for example : user_myfacility.myevent: 39507.805584526 (/cpu_1), 15829, 15736, SYSCALL { "myapp.c", "main", 8, 1234, 0xf0f0f0f0 } 3) The third way to trace information from your application This method is cleary the _FASTEST_. It is principally I/O (disk and memory) bound. It will create a companion process for each of you program's thread which will dump the tracing information into /tmp/ltt-usertrace. See sample-highspeed.c and sample-thread-fast.c for code example. - Add the following statements to your program source (the define must come _before_ the includes!) : #define LTT_TRACE #define LTT_TRACE_FAST #include - Add a call following the trace_user_myfacility_myevent function found in /usr/include/ltt/ltt-facility-user_myfacility.h in your program. For instance : trace_user_myfacility_myevent(__FILE__, __func__, __LINE__, 1234, (void*)0xF0F0F0F0); - Compile your application with at least these parameters to gcc (it is splitted on two lines, joined by a "\") : gcc -lltt-usertrace-fast -I /usr/src/usertrace-generic -o myapp myapp.c \ /usr/src/usertrace-generic/ltt-facility-loader-user_myfacility.c It requires a supplementary operation when you take the trace : - Start tracing (with lttctl) - Start your application - Let your application run... - Stop tracing - Move or copy /tmp/ltt-usertrace info your trace. i.e., if your trace is in /tmp/trace1 : su mv /tmp/ltt-usertrace /tmp/trace1 Then, to see only the user_myfacility events : lttv -m textDump -t /tmp/trace1 -e "event.facility=user_myfacility" It will show, for example : user_myfacility.myevent: 39507.805584526 (/ltt-usertrace/process-26174.26174.39236180500380_0), 15829, 15736, USER_MODE { "myapp.c", "main", 8, 1234, 0xf0f0f0f0 } * Fun feature : function instrumentation Here is how to generate a full trace of you program function calls. See the sample-instrument-fct.c example program. - Compile your application with at least these parameters to gcc (it is splitted on two lines, joined by a "\") : gcc -g -finstrument-functions \ -lltt-instrument-functions -o myapp myapp.c To see what the final result looks like : - Start tracing - Start your application - Stop tracing Then, to see only the function_entry and function_exit events : lttv -m textDump -t /tmp/trace1 -e "event.facility=user_generic & (event.name=function_entry & event.name=function_exit)" It will show, for example : user_generic.function_entry: 59329.709939111 (/ltt-usertrace/process-26202.0.39949996866578_0), 19250, 18581, USER_MODE { 0x8048454, 0x80484c2 } user_generic.function_exit: 59329.709944613 (/ltt-usertrace/process-26202.0.39949996866578_0), 19250, 18581, USER_MODE { 0x8048454, 0x80484c2 } you can then use (from the binutils package) addr2line -e sample-instrument-fct -i -f 0x8048454 Which shows : test_function /usr/src/usertrace-generic/sample-instrument-fct.c:12 The lookup in LTTV through libbfd has not been implemented yet.