1 Linux Trace Toolkit Quickstart
2 ------------------------------
3 Author : Mathieu Desnoyers, September 2005
4 Last update : July 31, 2008
7 This document is made of four parts : the first one explains how to install
8 LTTng and LTTV from Debian and RPM binary packages, the second one explains how
9 to install LTTng and LTTV from sources and the third one describes the steps
10 to follow to trace a system and view it. The fourth and last part explains
11 briefly how to add a new trace point to the kernel and to user space
14 What you will typically want is to read sections 2 and 3 : install LTTng from
17 These operations are made for installing the LTTng 0.12 tracer on a linux 2.6.X
18 kernel. You will also find instructions for installation of LTTV 0.8.x : the
19 Linux Trace Toolkit Viewer.
21 To see the list of compatibilities between LTTng, ltt-control, LTTV, genevent
22 and ltt-usertrace, please refer to :
23 http://ltt.polymtl.ca > LTTng+LTTV versions compatibility
27 The following lttng patch is necessary to have the tracing hooks in the kernel.
28 The following ltt-control module controls the tracing.
30 Required programs and libraries are assumed to be automatically installed in an
31 installation with Debian or RPM packages. In the case of an installation from
32 sources, the dependencies are listed.
35 ** Current development status **
38 supported architectures :
39 Intel Pentium (UP/SMP) with TSC
40 PowerPC 32 and 64 bits
43 C2 Microsystems (variant of MIPS)
46 supported architectures :
49 PowerPC 32 and 64 bits
52 ***********************************************************
53 ** Section 1 * Installation from Debian or RPM packages **
54 ***********************************************************
56 ** NOTE : RPM and Debian packages are only made once a version has been
57 thoroughly tested. If they do not exist at the moment, please install from
58 sources (see section 2 below). To see the list of compatibilities between
59 LTTng, ltt-control, LTTV, genevent and lttng-modules, please refer to
60 http://ltt.polymtl.ca > LTTng+LTTV versions compatibility
63 * Install from RPM packages on Fedora Core 4 :
67 http://ltt.polymtl.ca/packages/fedora/RPMS
71 LTTng kernel and lttng-modules RPM are available for some architectures (i586,
72 i686). Feel free to help fix the spec files to have correct lttng-modules RPM
76 * Install from .deb packages on Debian :
78 You can use the ltt.polymtl.ca apt source to get LTTV for Debian :
80 Add the following two sources to your /etc/apt/sources.list :
82 deb http://ltt.polymtl.ca/packages/debian experimental main
83 deb-src http://ltt.polymtl.ca/packages/debian experimental main
86 * Install from precompiled binary packages (LTTV compiled only for i386, and
87 LTTng only for i686 smp), perform the following :
91 apt-get install lttv lttv-doc
92 apt-get install kernel-image-2.6.12-rc4-mm2-lttng-0.4.2
93 apt-get install lttng-modules-modules-2.6.12-rc4-mm2-lttng-0.4.2
94 * note : the packages are signed by myself. I am not considered a trusted
95 Debian source yet, so warnings are normal.
97 Then, follow the section "Editing the system wide configuration" in section 2.
99 * Create custom LTTV Debian packages
101 Binary packages are only available for i386. If you want to create your own LTTV
102 packages for other platforms, do :
108 dpkg-buildpackage -rfakeroot
110 You should then have your LTTV .deb files created for your architecture.
112 * Create custom LTTng packages
114 For building LTTng Debian packages :
117 apt-get install kernel-source-2.6.12-rc4-mm2-lttng-0.4.2
119 bzip2 -cd kernel-source-2.6.12-rc4-mm2-lttng-0.4.2.tar.bz2 | tar xvof -
120 cd kernel-source-2.6.12-rc4-mm2-lttng-0.4.2
121 make menuconfig (or xconfig or config) (customize your configuration)
122 make-kpkg kernel_image
124 You will then see your freshly created .deb in /usr/src. Install it with
125 dpkg -i /usr/src/(image-name).deb
127 You will also need to create a package for the lttng-modules :
131 apt-get source lttng-modules
132 cd kernel-source-2.6.12-rc4-mm2-lttng-0.4.2
133 make-kpkg --added_modules /usr/src/lttng-modules-0.3 modules_image
135 You will then see your freshly created .deb in /usr/src. Install it with
136 dpkg -i /usr/src/lttng-modules-modules-(your version).deb
139 Then, follow the section "Editing the system wide configuration" in section 2.
142 ***********************************************************
143 ** Section 2 * Installation from sources **
144 ***********************************************************
148 Tools needed to follow the package download steps :
155 You have to install the standard development libraries and programs necessary
156 to compile a kernel :
158 (from Documentation/Changes in the Linux kernel tree)
159 o Gnu C 2.95.3 # gcc --version
160 o Gnu make 3.79.1 # make --version
161 o binutils 2.12 # ld -v
162 o util-linux 2.10o # fdformat --version
163 o module-init-tools 0.9.10 # depmod -V
165 You might also want to have libncurses5 to have the text mode kernel
166 configuration menu, but there are alternatives.
168 Prerequisites for LTTV 0.x.x installation are :
171 gtk 2.4 or better development libraries
172 (Debian : libgtk2.0, libgtk2.0-dev)
173 (Fedora : gtk2, gtk2-devel)
174 note : For Fedora users : this might require at least core 3 from Fedora,
175 or you might have to compile your own GTK2 library.
176 glib 2.4 or better development libraries
177 (Debian : libglib2.0-0, libglib2.0-dev)
178 (Fedora : glib2, glib2-devel)
179 libpopt development libraries
180 (Debian : libpopt0, libpopt-dev)
182 libpango development libraries
183 (Debian : libpango1.0, libpango1.0-dev)
184 (Fedora : pango, pango-devel)
185 libc6 development librairies
186 (Debian : libc6, libc6-dev)
187 (Fedora : glibc, glibc)
190 * Getting the LTTng packages
195 (see http://ltt.polymtl.ca/lttng for package listing)
196 wget http://ltt.polymtl.ca/lttng/patch-2.6.X-lttng-0.x.xx.tar.bz2
197 bzip2 -cd patch-2.6.X-lttng-0.x.xx.tar.bz2 | tar xvof -
200 * Getting LTTng kernel sources
204 wget http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.X.tar.bz2
205 bzip2 -cd linux-2.6.X.tar.bz2 | tar xvof -
207 - For LTTng 0.9.4- cat /usr/src/lttng/patch*-2.6.X-lttng-0.x.xx* | patch -p1
208 - For LTTng 0.9.5+ apply the patches in the order specified in the series file,
211 mv linux-2.6.X linux-2.6.X-lttng-0.x.xx
214 * Installing a LTTng kernel
217 cd /usr/src/linux-2.6.X-lttng-0.x.xx
218 make menuconfig (or make xconfig or make config)
219 Select the < Help > button if you are not familiar with kernel
221 Items preceded by [*] means they has to be built into the kernel.
222 Items preceded by [M] means they has to be built as modules.
223 Items preceded by [ ] means they should be removed.
224 go to the "General setup" section
225 Select the following options :
226 [*] Activate tracepoints
228 [*] Activate userspace markers ABI
229 <*> Compile generic tracing probes
230 Linux Trace Toolkit --->
231 [LTTng fine-grained-timestamping]
232 [*] Linux Trace Toolkit Instrumentation Support
233 <M> or <*> Linux Trace Toolkit Relay+DebugFS Support
234 <M> or <*> Linux Trace Toolkit Serializer
235 <M> or <*> Linux Trace Toolkit Marker Control
236 <M> or <*> Linux Trace Toolkit Tracer
237 It makes no difference for the rest of the procedure whether the Tracer
238 is compiled built-in or as a module.
240 [*] Align Linux Trace Toolkit Traces
241 <M> Linux Trace Toolkit Netlink Controller
242 <M> Linux Trace Toolkit State Dump
243 your choice (see < Help >) :
244 [ ] Write heartbeat event to shrink traces
245 [ ] Support trace extraction from crash dump
251 (if necessary, create a initrd with mkinitrd or your preferate alternative)
252 (mkinitrd -o /boot/initrd.img-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx)
257 Select the Linux 2.6.17-lttng-0.x.xx kernel in your boot loader.
260 cp vmlinux.strip /boot/vmlinux-2.6.X-lttng-0.x.xx
261 cp System.map /boot/System.map-2.6.X-lttng-0.x.xx
262 cp .config /boot/config-2.6.X-lttng-0.x.xx
263 depmod -ae -F /boot/System.map-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx
264 mkinitrd /boot/initrd.img-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx
265 (edit /etc/yaboot.conf to add a new entry pointing to your kernel : the entry
266 that comes first is the default kernel)
268 select the right entry at the yaboot prompt (see choices : tab, select : type
269 the kernel name followed by enter)
270 Select the Linux 2.6.17-lttng-0.x.xx kernel in your boot loader.
275 * Editing the system wide configuration
277 You must activate debugfs and specify a mount point. This is typically done in
278 fstab such that it happens at boot time.
280 If you have never used DebugFS before, these operation would do this for you :
283 cp /etc/fstab /etc/fstab.lttng.bkp
284 echo "debugfs /mnt/debugfs debugfs rw 0 0" >> /etc/fstab
286 then, rebooting or issuing the following command will activate debugfs :
290 You need to load the LTT modules to be able to control tracing from user
291 space. This is done by issuing the following commands. Note however
292 these commands load all LTT modules. Depending on what options you chose to
293 compile statically, you may not need to issue all these commands.
296 modprobe ltt-marker-control
298 modprobe ltt-serialize
301 modprobe kernel-trace
306 If you want to have complete information about the kernel state (including all
307 the process names), you need to load the ltt-statedump module. This is done by
308 issuing the command :
310 modprobe ltt-statedump
312 You can automate at boot time loading the ltt-control module by :
314 cp /etc/modules /etc/modules.bkp
315 echo ltt-control >> /etc/modules
316 echo ltt-marker-control >> /etc/modules
317 echo ltt-tracer >> /etc/modules
318 echo ltt-serialize >> /etc/modules
319 echo ltt-relay >> /etc/modules
320 echo ipc-trace >> /etc/modules
321 echo kernel-trace >> /etc/modules
322 echo mm-trace >> /etc/modules
323 echo net-trace >> /etc/modules
324 echo fs-trace >> /etc/modules
327 * Getting and installing the ltt-control package (on the traced machine)
328 (note : the ltt-control package contains lttd and lttctl. Although it has the
329 same name as the ltt-control kernel module, they are *not* the same thing.)
332 wget http://ltt.polymtl.ca/lttng/ltt-control-0.x-xxxx2006.tar.gz
333 gzip -cd ltt-control-0.x-xxxx2006.tar.gz | tar xvof -
334 cd ltt-control-0.x-xxxx2006
335 (refer to README to see the development libraries that must be installed on you
341 * Getting and installing the markers-userspace package for user space tracing
342 See http://ltt.polymtl.ca/packages/markers-userspace-0.5.tar.bz2 or more recent.
345 * Getting and installing the LTTV package (on the visualisation machine, same or
346 different from the visualisation machine)
350 wget http://ltt.polymtl.ca/packages/LinuxTraceToolkitViewer-0.x.xx-xxxx2006.tar.gz
351 gzip -cd LinuxTraceToolkitViewer-0.x.xx-xxxx2006.tar.gz | tar xvof -
352 cd LinuxTraceToolkitViewer-0.x.xx-xxxx2006
353 (refer to README to see the development libraries that must be installed on your
362 ***********************************************************
363 ** Section 3 * Using LTTng and LTTV **
364 ***********************************************************
366 * Arm Linux Kernel Markers after each boot
370 * Use graphical LTTV to control tracing and analyse traces
372 lttv-gui (or /usr/local/bin/lttv-gui)
373 - Spot the "Tracing Control" icon : click on it
374 (it's a traffic light icon)
375 - enter the root password
379 * You should now see a trace
381 * Use text mode LTTng to control tracing
383 The tracing can be controlled from a terminal by using the lttctl command (as
388 lttctl -n trace -d -l /mnt/debugfs/ltt -t /tmp/trace
390 Stop tracing and destroy trace channels :
394 see lttctl --help for details.
396 (note : to see if the buffers has been filled, look at the dmesg output after
397 lttctl -R or after stopping tracing from the GUI, it will show an event lost
398 count. If it is the case, try using larger buffers. See lttctl --help to learn
403 Feel free to look in /usr/local/lib/lttv/plugins to see all the text and
404 graphical plugins available.
406 For example, a simple trace dump in text format is available with :
408 lttv -m textDump -t /tmp/trace
410 see lttv -m textDump --help for detailed command line options of textDump.
412 It is, in the current state of the project, very useful to use "grep" on the
413 text output to filter by specific event fields. You can later copy the timestamp
414 of the events to the clipboard and paste them in the GUI by clicking on the
415 bottom right label "Current time". Support for this type of filtering should
416 be added to the filter module soon.
420 Starting from LTTng 0.5.105 and ltt-control 0.20, a new mode can be used :
421 hybrid. It can be especially useful when studying big workloads on a long period
424 When using this mode, the most important, low rate control information will be
425 recorded during all the trace by lttd (i.e. process creation/exit). The high
426 rate information (i.e. interrupt/traps/syscall entry/exit) will be kept in a
427 flight recorder buffer (now named flight-channelname_X).
429 The following lttctl commands take an hybrid trace :
431 Create trace channel, start lttd on normal channels, start tracing:
432 lttctl -n tracename -d -l /mnt/relayfs/ltt -t /tmp/trace1 -m hybrid
434 Stop tracing, start lttd on flight recorder channels, destroy trace channels :
435 lttctl -n tracename -f -l /mnt/relayfs/ltt -t /tmp/trace1 -m hybrid
438 We will need to tweak what we consider "important" medium rate events. For
439 instance, thread branding events are actually considered a "high rate" event
440 when it should be considered "medium rate". The same should apply for the
441 state dump process enumeration.
443 * Flight recorder mode
445 The flight recorder mode writes data into overwritten buffers for all channels,
446 including control channels, except for the facilities tracefiles.
448 The following lttctl commands take a flight recorder trace :
450 lttctl -n trace -c -m flight
451 lttd -n -d -t /tmp/trace -c /sys/kernel/debug/ltt
455 lttd -f -d -t /tmp/trace -c /sys/kernel/debug/ltt
459 ***********************************************************
460 ** Section 4 * Adding new instrumentations with genevent **
461 ***********************************************************
463 * Getting and installing genevent
467 wget http://ltt.polymtl.ca/packages/genevent-0.xx.tar.gz
468 gzip -cd genevent-0.xx.tar.gz | tar xvof -
474 * Add new events to the kernel with genevent (deprecated in LTTng 0.9.x)
477 cd /usr/local/share/ltt-control/facilities
478 cp process.xml yourfacility.xml
479 * edit yourfacility.xml to fit your needs.
481 /usr/local/bin/genevent /usr/local/share/ltt-control/facilities/yourfacility.xml
482 cp ltt-facility-yourfacility.h ltt-facility-id-yourfacility.h \
483 /usr/src/linux-2.6.17-lttng-0.x.xx8/include/ltt
484 cp ltt-facility-loader-yourfacility.c ltt-facility-loader-yourfacility.h \
485 /usr/src/linux-2.6.17-lttng-0.x.xx/ltt/facilities
486 * edit the kernel file you want to instrument to add a marker to it. See
487 include/linux/marker.h.
488 * create a dynamically loadable probe. See ltt/probes for examples. The probe
489 will be connected to your marker and will typically call the logging
490 functions found in the header file you created with genevent.
492 * Add new kernel events
494 *Important* note : in its current state, LTTng and LTTV needs the programmer
495 to keep the marker/probe format string and the XML description of the
496 event data types in sync by hand. Failure to do so will result in errors in
499 See the markers documentation to see how to describe the marker. You will need
500 to clone probe modules found in ltt/probes to connect them to the markers so
501 that the information can be recorded in the trace.
503 * Add new events to userspace programs with genevent
504 See http://ltt.polymtl.ca/ > USERSPACE TRACING QUICKSTART
506 User-space tracing still uses genevent, which is subject to change in a near