c924c2c6 |
1 | <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> |
2 | <html> |
3 | <head> |
4 | <title>Linux Trace Toolkit Quickstart</title> |
5 | </head> |
6 | <body> |
7 | |
8 | <h1>Linux Trace Toolkit Quickstart</h1> |
9 | |
c924c2c6 |
10 | Author : Mathieu Desnoyers, September 2005<br> |
24070967 |
11 | Last update : January 21st, 2009<br> |
12 | |
13 | <h2>Table of Contents</h2> |
14 | <ul> |
15 | |
f9ff56f3 |
16 | <li><a href="#intro" name="TOCintro">Introduction</a></li> |
24070967 |
17 | <li><a href="#section1" name="TOCsection1">Installing LTTng and LTTV from |
18 | sources</a></li> |
19 | <li><a href="#section2" name="TOCsection2">Using LTTng and LTTV</a></li> |
20 | <li><a href="#section3" name="TOCsection3">Adding kernel and user-space |
21 | tracepoints</a></li> |
633bc4a3 |
22 | <li><a href="#section4" name="TOCsection4">Creating Debian and RPM packages |
23 | from LTTV</a> |
c924c2c6 |
24 | |
f9ff56f3 |
25 | </ul> |
26 | |
27 | <hr /> |
28 | |
b9e1fab1 |
29 | <h2><a href="#TOCintro" name="intro">Introduction</a></h2> |
c924c2c6 |
30 | <p> |
633bc4a3 |
31 | This document is made of four parts : the first one explains how |
24070967 |
32 | to install LTTng and LTTV from sources, the second one describes the steps |
633bc4a3 |
33 | to follow to trace a system and view it. The third part explains |
c924c2c6 |
34 | briefly how to add a new trace point to the kernel and to user space |
633bc4a3 |
35 | applications. The fourth and last part explains how to create Debian or RPM |
36 | packages from the LTTng and LTTV sources. |
c924c2c6 |
37 | |
38 | <p> |
24070967 |
39 | These operations are made for installing the LTTng 0.86 tracer on a linux 2.6.X |
c924c2c6 |
40 | kernel. You will also find instructions for installation of LTTV 0.12.x : the |
41 | Linux Trace Toolkit Viewer. |
24070967 |
42 | To see the list of compatibilities between LTTng, ltt-control, LTTV, please |
43 | refer to : |
c924c2c6 |
44 | <a |
45 | href="http://ltt.polymtl.ca/svn/trunk/lttv/doc/developer/lttng-lttv-compatibility.html">LTTng+LTTV versions compatibility</a> |
24070967 |
46 | The lttng patch is necessary to have the tracing hooks in the kernel. |
c924c2c6 |
47 | |
24070967 |
48 | <br> |
49 | <br> |
633bc4a3 |
50 | Supported architectures : |
24070967 |
51 | <br> |
c924c2c6 |
52 | LTTng :<br> |
c924c2c6 |
53 | <li> x86 32/64 bits |
54 | <li> PowerPC 32 and 64 bits |
55 | <li> ARM (with limited timestamping precision, e.g. 1HZ. Need |
56 | architecture-specific support for better precision) |
57 | <li> MIPS |
58 | <br> |
24070967 |
59 | <br> |
c924c2c6 |
60 | LTTV :<br> |
c924c2c6 |
61 | <li> Intel 32/64 bits |
62 | <li> PowerPC 32 and 64 bits |
63 | <li> Possibly others. Takes care of endianness and type size difference between |
64 | the LTTng traces and the LTTV analysis tool. |
65 | |
f9ff56f3 |
66 | <hr /> |
67 | |
c924c2c6 |
68 | |
633bc4a3 |
69 | <h2><a href="#TOCsection2" name="section2">Installation from sources</a></h2> |
70 | <p> |
c924c2c6 |
71 | |
633bc4a3 |
72 | <li>Prerequisites</li> |
73 | <ul> |
74 | <p> |
c924c2c6 |
75 | Tools needed to follow the package download steps : |
76 | |
633bc4a3 |
77 | <li>wget |
78 | <li>bzip2 |
79 | <li>gzip |
80 | <li>tar |
c924c2c6 |
81 | |
633bc4a3 |
82 | <p> |
c924c2c6 |
83 | You have to install the standard development libraries and programs necessary |
84 | to compile a kernel : |
85 | |
84cf5903 |
86 | <PRE> |
c924c2c6 |
87 | (from Documentation/Changes in the Linux kernel tree) |
84cf5903 |
88 | Gnu C 2.95.3 # gcc --version |
89 | Gnu make 3.79.1 # make --version |
90 | binutils 2.12 # ld -v |
91 | util-linux 2.10o # fdformat --version |
92 | module-init-tools 0.9.10 # depmod -V |
93 | </PRE> |
c924c2c6 |
94 | |
633bc4a3 |
95 | <p> |
c924c2c6 |
96 | You might also want to have libncurses5 to have the text mode kernel |
97 | configuration menu, but there are alternatives. |
98 | |
633bc4a3 |
99 | <p> |
c924c2c6 |
100 | Prerequisites for LTTV 0.x.x installation are : |
101 | |
84cf5903 |
102 | <PRE> |
103 | gcc 3.2 or better |
104 | gtk 2.4 or better development libraries |
c924c2c6 |
105 | (Debian : libgtk2.0, libgtk2.0-dev) |
106 | (Fedora : gtk2, gtk2-devel) |
107 | note : For Fedora users : this might require at least core 3 from Fedora, |
108 | or you might have to compile your own GTK2 library. |
84cf5903 |
109 | glib 2.4 or better development libraries |
c924c2c6 |
110 | (Debian : libglib2.0-0, libglib2.0-dev) |
111 | (Fedora : glib2, glib2-devel) |
84cf5903 |
112 | libpopt development libraries |
c924c2c6 |
113 | (Debian : libpopt0, libpopt-dev) |
114 | (Fedora : popt) |
84cf5903 |
115 | libpango development libraries |
c924c2c6 |
116 | (Debian : libpango1.0, libpango1.0-dev) |
117 | (Fedora : pango, pango-devel) |
84cf5903 |
118 | libc6 development librairies |
c924c2c6 |
119 | (Debian : libc6, libc6-dev) |
120 | (Fedora : glibc, glibc) |
84cf5903 |
121 | </PRE> |
633bc4a3 |
122 | </ul> |
c924c2c6 |
123 | |
c8997124 |
124 | <li>Reminder</li> |
c924c2c6 |
125 | |
c8997124 |
126 | <p> |
633bc4a3 |
127 | See the list of compatibilities between LTTng, ltt-control and LTTV at : |
128 | <a |
129 | href="http://ltt.polymtl.ca/svn/trunk/lttv/doc/developer/lttng-lttv-compatibility.html">LTTng+LTTV |
130 | versions compatibility</a>. |
c924c2c6 |
131 | |
132 | |
c8997124 |
133 | <li>Getting the LTTng packages</li> |
c924c2c6 |
134 | |
c8997124 |
135 | <PRE> |
c924c2c6 |
136 | su - |
137 | mkdir /usr/src/lttng |
138 | cd /usr/src/lttng |
139 | (see http://ltt.polymtl.ca/lttng for package listing) |
140 | wget http://ltt.polymtl.ca/lttng/patch-2.6.X-lttng-0.x.xx.tar.bz2 |
141 | bzip2 -cd patch-2.6.X-lttng-0.x.xx.tar.bz2 | tar xvof - |
c8997124 |
142 | </PRE> |
c924c2c6 |
143 | |
144 | |
c8997124 |
145 | <li>Getting LTTng kernel sources</li> |
c924c2c6 |
146 | |
c8997124 |
147 | <PRE> |
c924c2c6 |
148 | su - |
149 | cd /usr/src |
150 | wget http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.X.tar.bz2 |
151 | bzip2 -cd linux-2.6.X.tar.bz2 | tar xvof - |
152 | cd linux-2.6.X |
153 | - For LTTng 0.9.4- cat /usr/src/lttng/patch*-2.6.X-lttng-0.x.xx* | patch -p1 |
154 | - For LTTng 0.9.5+ apply the patches in the order specified in the series file, |
155 | or use quilt |
156 | cd .. |
157 | mv linux-2.6.X linux-2.6.X-lttng-0.x.xx |
c8997124 |
158 | </PRE> |
c924c2c6 |
159 | |
160 | |
c8997124 |
161 | <li>Installing a LTTng kernel</li> |
c924c2c6 |
162 | |
c8997124 |
163 | <PRE> |
c924c2c6 |
164 | su - |
165 | cd /usr/src/linux-2.6.X-lttng-0.x.xx |
166 | make menuconfig (or make xconfig or make config) |
167 | Select the < Help > button if you are not familiar with kernel |
168 | configuration. |
169 | Items preceded by [*] means they has to be built into the kernel. |
170 | Items preceded by [M] means they has to be built as modules. |
171 | Items preceded by [ ] means they should be removed. |
172 | go to the "General setup" section |
173 | Select the following options : |
174 | [*] Prompt for development and/or incomplete code/drivers |
175 | [*] Activate markers |
176 | [*] Activate userspace markers ABI (experimental, optional) |
177 | [*] Immediate value optimization (optional) |
178 | [*] Linux Trace Toolkit Next Generation (LTTng) ---> |
179 | <M> or <*> Compile lttng tracing probes |
180 | <M> or <*> Linux Trace Toolkit High-speed Lockless Data Relay |
181 | <M> or <*> Linux Trace Toolkit Lock-Protected Data Relay |
182 | <M> or <*> Linux Trace Toolkit Serializer |
183 | <M> or <*> Linux Trace Toolkit Marker Control |
184 | <M> or <*> Linux Trace Toolkit Tracer |
185 | [*] Align Linux Trace Toolkit Traces |
186 | <M> or <*> Support logging events from userspace |
187 | [*] Support trace extraction from crash dump |
188 | <M> or <*> Linux Trace Toolkit Trace Controller |
189 | <M> or <*> Linux Trace Toolkit State Dump |
190 | Select <Exit> |
191 | Select <Exit> |
192 | Select <Yes> |
193 | make |
194 | make modules_install |
195 | (if necessary, create a initrd with mkinitrd or your preferate alternative) |
196 | (mkinitrd -o /boot/initrd.img-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx) |
197 | |
198 | -- on X86, X86_64 |
199 | make install |
200 | reboot |
201 | Select the Linux 2.6.X-lttng-0.x.xx kernel in your boot loader. |
202 | |
203 | -- on PowerPC |
204 | cp vmlinux.strip /boot/vmlinux-2.6.X-lttng-0.x.xx |
205 | cp System.map /boot/System.map-2.6.X-lttng-0.x.xx |
206 | cp .config /boot/config-2.6.X-lttng-0.x.xx |
207 | depmod -ae -F /boot/System.map-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx |
208 | mkinitrd /boot/initrd.img-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx |
209 | (edit /etc/yaboot.conf to add a new entry pointing to your kernel : the entry |
210 | that comes first is the default kernel) |
211 | ybin |
212 | select the right entry at the yaboot prompt (see choices : tab, select : type |
213 | the kernel name followed by enter) |
214 | Select the Linux 2.6.X-lttng-0.x.xx kernel in your boot loader. |
215 | -- |
c8997124 |
216 | </PRE> |
c924c2c6 |
217 | |
c8997124 |
218 | <li>Editing the system wide configuration</li> |
c924c2c6 |
219 | |
c8997124 |
220 | <p> |
c924c2c6 |
221 | You must activate debugfs and specify a mount point. This is typically done in |
c8997124 |
222 | fstab such that it happens at boot time. If you have never used DebugFS before, |
223 | these operation would do this for you : |
c924c2c6 |
224 | |
c8997124 |
225 | <PRE> |
c924c2c6 |
226 | mkdir /mnt/debugfs |
227 | cp /etc/fstab /etc/fstab.lttng.bkp |
228 | echo "debugfs /mnt/debugfs debugfs rw 0 0" >> /etc/fstab |
c8997124 |
229 | </PRE> |
c924c2c6 |
230 | |
c8997124 |
231 | <p> |
c924c2c6 |
232 | then, rebooting or issuing the following command will activate debugfs : |
c8997124 |
233 | <PRE> |
c924c2c6 |
234 | mount /mnt/debugfs |
c8997124 |
235 | </PRE> |
c924c2c6 |
236 | |
c8997124 |
237 | <p> |
c924c2c6 |
238 | You need to load the LTT modules to be able to control tracing from user |
239 | space. This is done by issuing the following commands. Note however |
240 | these commands load all LTT modules. Depending on what options you chose to |
241 | compile statically, you may not need to issue all these commands. |
242 | |
c8997124 |
243 | <PRE> |
c924c2c6 |
244 | modprobe ltt-trace-control |
245 | modprobe ltt-marker-control |
246 | modprobe ltt-tracer |
247 | modprobe ltt-serialize |
248 | modprobe ltt-relay |
249 | modprobe ipc-trace |
250 | modprobe kernel-trace |
251 | modprobe mm-trace |
252 | modprobe net-trace |
253 | modprobe fs-trace |
254 | modprobe jbd2-trace |
255 | modprobe ext4-trace |
256 | modprobe syscall-trace |
257 | modprobe trap-trace |
258 | #if locking tracing is wanted, uncomment the following |
259 | #modprobe lockdep-trace |
c8997124 |
260 | </PRE> |
c924c2c6 |
261 | |
c8997124 |
262 | <p> |
c924c2c6 |
263 | If you want to have complete information about the kernel state (including all |
264 | the process names), you need to load the ltt-statedump module. This is done by |
265 | issuing the command : |
266 | |
c8997124 |
267 | <PRE> |
c924c2c6 |
268 | modprobe ltt-statedump |
c8997124 |
269 | </PRE> |
270 | <p> |
c924c2c6 |
271 | You can automate at boot time loading the ltt-control module by : |
272 | |
c8997124 |
273 | <PRE> |
c924c2c6 |
274 | cp /etc/modules /etc/modules.bkp |
275 | echo ltt-trace-control >> /etc/modules |
276 | echo ltt-marker-control >> /etc/modules |
277 | echo ltt-tracer >> /etc/modules |
278 | echo ltt-serialize >> /etc/modules |
279 | echo ltt-relay >> /etc/modules |
280 | echo ipc-trace >> /etc/modules |
281 | echo kernel-trace >> /etc/modules |
282 | echo mm-trace >> /etc/modules |
283 | echo net-trace >> /etc/modules |
284 | echo fs-trace >> /etc/modules |
285 | echo jbd2-trace >> /etc/modules |
286 | echo ext4-trace >> /etc/modules |
287 | echo syscall-trace >> /etc/modules |
288 | echo trap-trace >> /etc/modules |
289 | #if locking tracing is wanted, uncomment the following |
290 | #echo lockdep-trace >> /etc/modules |
c8997124 |
291 | </PRE> |
c924c2c6 |
292 | |
c8997124 |
293 | <li>Getting and installing the ltt-control package (on the traced machine)</li> |
294 | <p> |
c924c2c6 |
295 | (note : the ltt-control package contains lttd and lttctl. Although it has the |
296 | same name as the ltt-control kernel module, they are *not* the same thing.) |
c8997124 |
297 | |
298 | <PRE> |
c924c2c6 |
299 | su - |
300 | cd /usr/src |
301 | wget http://ltt.polymtl.ca/lttng/ltt-control-0.x-xxxx2006.tar.gz |
302 | gzip -cd ltt-control-0.x-xxxx2008.tar.gz | tar xvof - |
303 | cd ltt-control-0.x-xxxx2006 |
304 | (refer to README to see the development libraries that must be installed on you |
305 | system) |
306 | ./configure |
307 | make |
308 | make install |
c8997124 |
309 | </PRE> |
c924c2c6 |
310 | |
c8997124 |
311 | <li>Userspace tracing</li> |
c924c2c6 |
312 | |
c8997124 |
313 | <PRE> |
c924c2c6 |
314 | Make sure you selected the kernel menuconfig option : |
315 | <M> or <*> Support logging events from userspace |
316 | And that the ltt-userspace-event kernel module is loaded if selected as a |
317 | module. |
318 | |
319 | Simple userspace tracing is available through |
320 | echo "some text to record" > /mnt/debugfs/ltt/write_event |
321 | |
322 | It will appear in the trace under event : |
323 | channel : userspace |
324 | event name : event |
c8997124 |
325 | </PRE> |
c924c2c6 |
326 | |
c8997124 |
327 | <li>Getting and installing the LTTV package (on the visualisation machine, same |
328 | or different from the visualisation machine)</li> |
c924c2c6 |
329 | |
c8997124 |
330 | <PRE> |
c924c2c6 |
331 | su - |
332 | cd /usr/src |
333 | wget http://ltt.polymtl.ca/packages/lttv-0.x.xx-xxxx2008.tar.gz |
334 | gzip -cd lttv-0.x.xx-xxxx2008.tar.gz | tar xvof - |
335 | cd lttv-0.x.xx-xxxx2008 |
336 | (refer to README to see the development libraries that must be installed on your |
337 | system) |
338 | ./configure |
339 | make |
340 | make install |
341 | |
342 | |
c8997124 |
343 | <li>Getting and installing the markers-userspace package for user space |
344 | tracing (experimental)</li> |
c924c2c6 |
345 | |
c8997124 |
346 | See <a |
347 | href="http://ltt.polymtl.ca/packages/markers-userspace-0.5.tar.bz2">markers-userspace-0.5.tar.bz2</a> or more recent. |
c924c2c6 |
348 | |
c8997124 |
349 | <hr /> |
c924c2c6 |
350 | |
c924c2c6 |
351 | |
c8997124 |
352 | <h2><a href="#TOCsection3" name="section3">Using LTTng and LTTV</a></h2> |
c924c2c6 |
353 | |
c8997124 |
354 | <li>IMPORTANT : Arm Linux Kernel Markers after each boot</li> |
355 | |
356 | <PRE> |
c924c2c6 |
357 | ltt-armall |
c8997124 |
358 | </PRE> |
c924c2c6 |
359 | |
c8997124 |
360 | <li>Use graphical LTTV to control tracing and analyse traces</li> |
c924c2c6 |
361 | |
c8997124 |
362 | <PRE> |
c924c2c6 |
363 | lttv-gui (or /usr/local/bin/lttv-gui) |
364 | - Spot the "Tracing Control" icon : click on it |
365 | (it's a traffic light icon) |
366 | - enter the root password |
367 | - click "start" |
368 | - click "stop" |
369 | - Yes |
370 | * You should now see a trace |
c8997124 |
371 | </PRE> |
c924c2c6 |
372 | |
c8997124 |
373 | <li>Use text mode LTTng to control tracing</li> |
c924c2c6 |
374 | |
c8997124 |
375 | <PRE> |
c924c2c6 |
376 | The tracing can be controlled from a terminal by using the lttctl command (as |
377 | root). |
378 | |
379 | Start tracing : |
380 | |
381 | lttctl -C -w /tmp/trace1 trace1 |
382 | |
383 | Stop tracing and destroy trace channels : |
384 | |
385 | lttctl -D trace1 |
386 | |
387 | see lttctl --help for details. |
c8997124 |
388 | <PRE> |
c924c2c6 |
389 | |
c8997124 |
390 | <p> |
c924c2c6 |
391 | (note : to see if the buffers has been filled, look at the dmesg output after |
392 | lttctl -R or after stopping tracing from the GUI, it will show an event lost |
393 | count. If it is the case, try using larger buffers. See lttctl --help to learn |
394 | how. lttv now also shows event lost messages in the console when loading a trace |
395 | with missing events or lost subbuffers.) |
396 | |
c8997124 |
397 | <li>Use text mode LTTV</li> |
c8997124 |
398 | <p> |
c924c2c6 |
399 | Feel free to look in /usr/local/lib/lttv/plugins to see all the text and |
400 | graphical plugins available. |
c8997124 |
401 | <p> |
c924c2c6 |
402 | For example, a simple trace dump in text format is available with : |
403 | |
c8997124 |
404 | <PRE> |
c924c2c6 |
405 | lttv -m textDump -t /tmp/trace |
c8997124 |
406 | </PRE> |
c924c2c6 |
407 | |
c8997124 |
408 | <p> |
409 | See lttv -m textDump --help for detailed command line options of textDump. |
c924c2c6 |
410 | |
c8997124 |
411 | <p> |
c924c2c6 |
412 | It is, in the current state of the project, very useful to use "grep" on the |
413 | text output to filter by specific event fields. You can later copy the timestamp |
414 | of the events to the clipboard and paste them in the GUI by clicking on the |
415 | bottom right label "Current time". Support for this type of filtering should |
416 | be added to the filter module soon. |
417 | |
c8997124 |
418 | <li>Hybrid mode</li> |
c924c2c6 |
419 | |
c8997124 |
420 | <p> |
c924c2c6 |
421 | Starting from LTTng 0.5.105 and ltt-control 0.20, a new mode can be used : |
422 | hybrid. It can be especially useful when studying big workloads on a long period |
423 | of time. |
424 | |
c8997124 |
425 | <p> |
c924c2c6 |
426 | When using this mode, the most important, low rate control information will be |
427 | recorded during all the trace by lttd (i.e. process creation/exit). The high |
428 | rate information (i.e. interrupt/traps/syscall entry/exit) will be kept in a |
429 | flight recorder buffer (now named flight-channelname_X). |
430 | |
c8997124 |
431 | <p> |
c924c2c6 |
432 | The following lttctl commands take an hybrid trace : |
c8997124 |
433 | <p> |
c924c2c6 |
434 | Create trace channel, start lttd on normal channels, start tracing: |
c8997124 |
435 | <PRE> |
c924c2c6 |
436 | lttctl -C -w /tmp/trace2 -o channel.kernel.overwrite=1 trace2 |
c8997124 |
437 | </PRE> |
438 | <p> |
c924c2c6 |
439 | Stop tracing, start lttd on flight recorder channels, destroy trace channels : |
c8997124 |
440 | <PRE> |
c924c2c6 |
441 | lttctl -D -w /tmp/trace2 trace2 |
c8997124 |
442 | </PRE> |
443 | <p> |
c924c2c6 |
444 | Each "overwrite" channel is flight recorder channel. |
445 | |
c8997124 |
446 | <li>Flight recorder mode</li> |
c924c2c6 |
447 | |
c8997124 |
448 | <p> |
c924c2c6 |
449 | The flight recorder mode writes data into overwritten buffers for all channels, |
450 | including control channels, except for the facilities tracefiles. It consists of |
451 | setting all channels to "overwrite". |
452 | |
c8997124 |
453 | <p> |
c924c2c6 |
454 | The following lttctl commands take a flight recorder trace : |
455 | |
c8997124 |
456 | <PRE> |
c924c2c6 |
457 | lttctl -C -w /tmp/trace3 -o channel.all.overwrite=1 trace3 |
458 | ... |
459 | lttctl -D -w /tmp/trace3 trace3 |
c8997124 |
460 | </PRE> |
c924c2c6 |
461 | |
462 | ************************************************************** |
463 | ** Section 4 * Adding new instrumentations with the markers ** |
464 | ************************************************************** |
465 | |
466 | See Documentation/markers.txt and Documentation/tracepoints.txt in your kernel |
467 | tree. |
468 | |
469 | * Add new events to userspace programs with userspace markers |
470 | http://ltt.polymtl.ca/packages/ |
471 | |
472 | Get the latest markers-userspace-*.tar.bz2 and see the Makefile and examples. It |
473 | allows inserting markers in executables and libraries, currently only on x86_32 |
474 | and x86_64. |
475 | |
633bc4a3 |
476 | *********************************************************** |
477 | ** Section 5 * Creating Debian or RPM packages ** |
478 | *********************************************************** |
479 | |
480 | * Create custom LTTV Debian packages |
481 | |
482 | Use : dpkg-buildpackage -rfakeroot |
483 | |
484 | You should then have your LTTV .deb files created for your architecture. |
485 | |
486 | * Create custom LTTng packages |
487 | |
488 | For building LTTng Debian packages : |
489 | |
490 | Get the build tree with patches applies as explained in section 2. |
491 | |
492 | make menuconfig (or xconfig or config) (customize your configuration) |
493 | make-kpkg kernel_image |
494 | |
495 | You will then see your freshly created .deb in /usr/src. Install it with |
496 | dpkg -i /usr/src/(image-name).deb |
497 | |
498 | Then, follow the section "Editing the system wide configuration" in section 2. |
499 | |
500 | |
501 | |
c924c2c6 |
502 | </body> |
503 | </html> |