a469681d |
1 | Linux Trace Toolkit Quickstart |
2 | ------------------------------ |
3 | Author : Mathieu Desnoyers, September 2005 |
674a743c |
4 | Last update : January 9th, 2009 |
d9cd3a2e |
5 | |
d9cd3a2e |
6 | |
a469681d |
7 | This document is made of four parts : the first one explains how to install |
6c913994 |
8 | LTTng and LTTV from Debian and RPM binary packages, the second one explains how |
9 | to install LTTng and LTTV from sources and the third one describes the steps |
10 | to follow to trace a system and view it. The fourth and last part explains |
208a5623 |
11 | briefly how to add a new trace point to the kernel and to user space |
12 | applications. |
6c913994 |
13 | |
208a5623 |
14 | What you will typically want is to read sections 2 and 3 : install LTTng from |
15 | sources and use it. |
6c913994 |
16 | |
3ecb16e2 |
17 | These operations are made for installing the LTTng 0.74 tracer on a linux 2.6.X |
ce6a8d46 |
18 | kernel. You will also find instructions for installation of LTTV 0.12.x : the |
f70523d4 |
19 | Linux Trace Toolkit Viewer. |
3c3abcf1 |
20 | |
839a3ec6 |
21 | To see the list of compatibilities between LTTng, ltt-control, LTTV and |
22 | markers-userspace, please refer to : |
3c3abcf1 |
23 | http://ltt.polymtl.ca > LTTng+LTTV versions compatibility |
24 | |
25 | |
38e8e662 |
26 | |
27 | The following lttng patch is necessary to have the tracing hooks in the kernel. |
28 | The following ltt-control module controls the tracing. |
29 | |
a469681d |
30 | Required programs and libraries are assumed to be automatically installed in an |
38e8e662 |
31 | installation with Debian or RPM packages. In the case of an installation from |
32 | sources, the dependencies are listed. |
33 | |
d9cd3a2e |
34 | |
c96be0d0 |
35 | ** Current development status ** |
36 | |
37 | LTTng : |
38 | supported architectures : |
39 | Intel Pentium (UP/SMP) with TSC |
e3b182e5 |
40 | PowerPC 32 and 64 bits |
2a4c796f |
41 | ARM |
2b75fbd7 |
42 | x86_64 |
285468d4 |
43 | C2 Microsystems (variant of MIPS) |
c96be0d0 |
44 | |
45 | LTTV : |
46 | supported architectures : |
47 | Intel i386 and better |
d21695ed |
48 | Intel 64 bits |
e3b182e5 |
49 | PowerPC 32 and 64 bits |
c96be0d0 |
50 | |
51 | |
6c913994 |
52 | *********************************************************** |
53 | ** Section 1 * Installation from Debian or RPM packages ** |
54 | *********************************************************** |
55 | |
5694ce4d |
56 | * Create custom LTTV Debian packages |
57 | |
839a3ec6 |
58 | Use : dpkg-buildpackage -rfakeroot |
78d521e3 |
59 | |
5694ce4d |
60 | You should then have your LTTV .deb files created for your architecture. |
61 | |
62 | * Create custom LTTng packages |
63 | |
64 | For building LTTng Debian packages : |
65 | |
839a3ec6 |
66 | Get the build tree with patches applies as explained in section 2. |
67 | |
5694ce4d |
68 | make menuconfig (or xconfig or config) (customize your configuration) |
69 | make-kpkg kernel_image |
70 | |
71 | You will then see your freshly created .deb in /usr/src. Install it with |
72 | dpkg -i /usr/src/(image-name).deb |
73 | |
5694ce4d |
74 | Then, follow the section "Editing the system wide configuration" in section 2. |
6c913994 |
75 | |
76 | |
77 | *********************************************************** |
78 | ** Section 2 * Installation from sources ** |
79 | *********************************************************** |
80 | |
38e8e662 |
81 | * Prerequisites |
82 | |
83 | Tools needed to follow the package download steps : |
84 | |
85 | o wget |
86 | o bzip2 |
87 | o gzip |
88 | o tar |
89 | |
a469681d |
90 | You have to install the standard development libraries and programs necessary |
38e8e662 |
91 | to compile a kernel : |
92 | |
93 | (from Documentation/Changes in the Linux kernel tree) |
94 | o Gnu C 2.95.3 # gcc --version |
95 | o Gnu make 3.79.1 # make --version |
96 | o binutils 2.12 # ld -v |
97 | o util-linux 2.10o # fdformat --version |
98 | o module-init-tools 0.9.10 # depmod -V |
99 | |
100 | You might also want to have libncurses5 to have the text mode kernel |
101 | configuration menu, but there are alternatives. |
102 | |
3c3abcf1 |
103 | Prerequisites for LTTV 0.x.x installation are : |
38e8e662 |
104 | |
860c0a03 |
105 | gcc 3.2 or better |
106 | gtk 2.4 or better development libraries |
107 | (Debian : libgtk2.0, libgtk2.0-dev) |
108 | (Fedora : gtk2, gtk2-devel) |
109 | note : For Fedora users : this might require at least core 3 from Fedora, |
110 | or you might have to compile your own GTK2 library. |
111 | glib 2.4 or better development libraries |
112 | (Debian : libglib2.0-0, libglib2.0-dev) |
113 | (Fedora : glib2, glib2-devel) |
114 | libpopt development libraries |
115 | (Debian : libpopt0, libpopt-dev) |
116 | (Fedora : popt) |
117 | libpango development libraries |
118 | (Debian : libpango1.0, libpango1.0-dev) |
119 | (Fedora : pango, pango-devel) |
120 | libc6 development librairies |
121 | (Debian : libc6, libc6-dev) |
122 | (Fedora : glibc, glibc) |
38e8e662 |
123 | |
63dddd56 |
124 | * Reminder |
125 | |
126 | See the list of compatibilities between LTTng, ltt-control, LTTV and |
127 | markers-userspace at : |
128 | http://ltt.polymtl.ca > LTTng+LTTV versions compatibility |
129 | |
130 | |
38e8e662 |
131 | |
d9cd3a2e |
132 | * Getting the LTTng packages |
133 | |
134 | su - |
135 | mkdir /usr/src/lttng |
136 | cd /usr/src/lttng |
137 | (see http://ltt.polymtl.ca/lttng for package listing) |
208a5623 |
138 | wget http://ltt.polymtl.ca/lttng/patch-2.6.X-lttng-0.x.xx.tar.bz2 |
139 | bzip2 -cd patch-2.6.X-lttng-0.x.xx.tar.bz2 | tar xvof - |
d9cd3a2e |
140 | |
141 | |
142 | * Getting LTTng kernel sources |
143 | |
144 | su - |
145 | cd /usr/src |
52a20c2e |
146 | wget http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.X.tar.bz2 |
208a5623 |
147 | bzip2 -cd linux-2.6.X.tar.bz2 | tar xvof - |
148 | cd linux-2.6.X |
cee587c3 |
149 | - For LTTng 0.9.4- cat /usr/src/lttng/patch*-2.6.X-lttng-0.x.xx* | patch -p1 |
150 | - For LTTng 0.9.5+ apply the patches in the order specified in the series file, |
151 | or use quilt |
d9cd3a2e |
152 | cd .. |
208a5623 |
153 | mv linux-2.6.X linux-2.6.X-lttng-0.x.xx |
d9cd3a2e |
154 | |
155 | |
156 | * Installing a LTTng kernel |
157 | |
158 | su - |
208a5623 |
159 | cd /usr/src/linux-2.6.X-lttng-0.x.xx |
38e8e662 |
160 | make menuconfig (or make xconfig or make config) |
161 | Select the < Help > button if you are not familiar with kernel |
162 | configuration. |
163 | Items preceded by [*] means they has to be built into the kernel. |
164 | Items preceded by [M] means they has to be built as modules. |
165 | Items preceded by [ ] means they should be removed. |
fade4071 |
166 | go to the "General setup" section |
38e8e662 |
167 | Select the following options : |
674a743c |
168 | [*] Prompt for development and/or incomplete code/drivers |
fade4071 |
169 | [*] Activate markers |
674a743c |
170 | [*] Activate userspace markers ABI (experimental, optional) |
171 | [*] Immediate value optimization (optional) |
172 | [*] Linux Trace Toolkit Next Generation (LTTng) ---> |
173 | <M> or <*> Compile lttng tracing probes |
174 | <M> or <*> Linux Trace Toolkit High-speed Lockless Data Relay |
175 | <M> or <*> Linux Trace Toolkit Lock-Protected Data Relay |
fade4071 |
176 | <M> or <*> Linux Trace Toolkit Serializer |
674a743c |
177 | <M> or <*> Linux Trace Toolkit Marker Control |
178 | <M> or <*> Linux Trace Toolkit Tracer |
179 | [*] Align Linux Trace Toolkit Traces |
180 | <M> or <*> Support logging events from userspace |
181 | [*] Support trace extraction from crash dump |
182 | <M> or <*> Linux Trace Toolkit Trace Controller |
183 | <M> or <*> Linux Trace Toolkit State Dump |
b70ceef8 |
184 | Select <Exit> |
38e8e662 |
185 | Select <Exit> |
186 | Select <Yes> |
02bc6879 |
187 | make |
26d45a39 |
188 | make modules_install |
e55d936e |
189 | (if necessary, create a initrd with mkinitrd or your preferate alternative) |
190 | (mkinitrd -o /boot/initrd.img-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx) |
916ec268 |
191 | |
208a5623 |
192 | -- on X86, X86_64 |
193 | make install |
d9cd3a2e |
194 | reboot |
674a743c |
195 | Select the Linux 2.6.X-lttng-0.x.xx kernel in your boot loader. |
208a5623 |
196 | |
197 | -- on PowerPC |
3c8a1f17 |
198 | cp vmlinux.strip /boot/vmlinux-2.6.X-lttng-0.x.xx |
208a5623 |
199 | cp System.map /boot/System.map-2.6.X-lttng-0.x.xx |
200 | cp .config /boot/config-2.6.X-lttng-0.x.xx |
201 | depmod -ae -F /boot/System.map-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx |
202 | mkinitrd /boot/initrd.img-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx |
203 | (edit /etc/yaboot.conf to add a new entry pointing to your kernel : the entry |
204 | that comes first is the default kernel) |
205 | ybin |
206 | select the right entry at the yaboot prompt (see choices : tab, select : type |
207 | the kernel name followed by enter) |
674a743c |
208 | Select the Linux 2.6.X-lttng-0.x.xx kernel in your boot loader. |
208a5623 |
209 | -- |
d9cd3a2e |
210 | |
d9cd3a2e |
211 | |
5694ce4d |
212 | |
213 | * Editing the system wide configuration |
214 | |
9c86b517 |
215 | You must activate debugfs and specify a mount point. This is typically done in |
5694ce4d |
216 | fstab such that it happens at boot time. |
217 | |
9c86b517 |
218 | If you have never used DebugFS before, these operation would do this for you : |
5694ce4d |
219 | |
9c86b517 |
220 | mkdir /mnt/debugfs |
5694ce4d |
221 | cp /etc/fstab /etc/fstab.lttng.bkp |
9c86b517 |
222 | echo "debugfs /mnt/debugfs debugfs rw 0 0" >> /etc/fstab |
5694ce4d |
223 | |
9c86b517 |
224 | then, rebooting or issuing the following command will activate debugfs : |
5694ce4d |
225 | |
9c86b517 |
226 | mount /mnt/debugfs |
5694ce4d |
227 | |
b70ceef8 |
228 | You need to load the LTT modules to be able to control tracing from user |
a469681d |
229 | space. This is done by issuing the following commands. Note however |
230 | these commands load all LTT modules. Depending on what options you chose to |
231 | compile statically, you may not need to issue all these commands. |
8ede3ed9 |
232 | |
4ad053df |
233 | modprobe ltt-trace-control |
fade4071 |
234 | modprobe ltt-marker-control |
b70ceef8 |
235 | modprobe ltt-tracer |
fade4071 |
236 | modprobe ltt-serialize |
237 | modprobe ltt-relay |
238 | modprobe ipc-trace |
239 | modprobe kernel-trace |
240 | modprobe mm-trace |
241 | modprobe net-trace |
242 | modprobe fs-trace |
5c0db01b |
243 | modprobe jbd2-trace |
244 | modprobe ext4-trace |
45ebd119 |
245 | modprobe syscall-trace |
fbde640a |
246 | modprobe trap-trace |
b93e9a1e |
247 | #if locking tracing is wanted, uncomment the following |
248 | #modprobe lockdep-trace |
d9cd3a2e |
249 | |
9c78dc8f |
250 | If you want to have complete information about the kernel state (including all |
251 | the process names), you need to load the ltt-statedump module. This is done by |
252 | issuing the command : |
253 | |
254 | modprobe ltt-statedump |
255 | |
38e8e662 |
256 | You can automate at boot time loading the ltt-control module by : |
8ede3ed9 |
257 | |
b70ceef8 |
258 | cp /etc/modules /etc/modules.bkp |
4ad053df |
259 | echo ltt-trace-control >> /etc/modules |
fade4071 |
260 | echo ltt-marker-control >> /etc/modules |
b70ceef8 |
261 | echo ltt-tracer >> /etc/modules |
fade4071 |
262 | echo ltt-serialize >> /etc/modules |
263 | echo ltt-relay >> /etc/modules |
264 | echo ipc-trace >> /etc/modules |
265 | echo kernel-trace >> /etc/modules |
266 | echo mm-trace >> /etc/modules |
267 | echo net-trace >> /etc/modules |
268 | echo fs-trace >> /etc/modules |
5c0db01b |
269 | echo jbd2-trace >> /etc/modules |
270 | echo ext4-trace >> /etc/modules |
271 | echo syscall-trace >> /etc/modules |
272 | echo trap-trace >> /etc/modules |
b93e9a1e |
273 | #if locking tracing is wanted, uncomment the following |
274 | #echo lockdep-trace >> /etc/modules |
b70ceef8 |
275 | |
d9cd3a2e |
276 | |
daa38dd5 |
277 | * Getting and installing the ltt-control package (on the traced machine) |
3bef9a5f |
278 | (note : the ltt-control package contains lttd and lttctl. Although it has the |
279 | same name as the ltt-control kernel module, they are *not* the same thing.) |
d9cd3a2e |
280 | su - |
281 | cd /usr/src |
3c3abcf1 |
282 | wget http://ltt.polymtl.ca/lttng/ltt-control-0.x-xxxx2006.tar.gz |
674a743c |
283 | gzip -cd ltt-control-0.x-xxxx2008.tar.gz | tar xvof - |
3c3abcf1 |
284 | cd ltt-control-0.x-xxxx2006 |
daa38dd5 |
285 | (refer to README to see the development libraries that must be installed on you |
286 | system) |
287 | ./configure |
288 | make |
289 | make install |
290 | |
674a743c |
291 | * Userspace tracing |
292 | |
293 | Simple userspace tracing is available through |
294 | echo "some text to record" > /mnt/debugfs/ltt/write_event |
208a5623 |
295 | |
674a743c |
296 | It will appear in the trace under event : |
297 | channel : userspace |
298 | event name : event |
daa38dd5 |
299 | |
300 | * Getting and installing the LTTV package (on the visualisation machine, same or |
301 | different from the visualisation machine) |
302 | |
303 | su - |
304 | cd /usr/src |
839a3ec6 |
305 | wget http://ltt.polymtl.ca/packages/lttv-0.x.xx-xxxx2008.tar.gz |
306 | gzip -cd lttv-0.x.xx-xxxx2008.tar.gz | tar xvof - |
307 | cd lttv-0.x.xx-xxxx2008 |
a469681d |
308 | (refer to README to see the development libraries that must be installed on your |
36fcd0b7 |
309 | system) |
d9cd3a2e |
310 | ./configure |
311 | make |
312 | make install |
313 | |
6c913994 |
314 | |
674a743c |
315 | * Getting and installing the markers-userspace package for user space tracing |
316 | (experimental) |
317 | See http://ltt.polymtl.ca/packages/markers-userspace-0.5.tar.bz2 or more recent. |
318 | |
6c913994 |
319 | |
320 | |
321 | *********************************************************** |
d1f19ac5 |
322 | ** Section 3 * Using LTTng and LTTV ** |
6c913994 |
323 | *********************************************************** |
324 | |
839a3ec6 |
325 | * IMPORTANT : Arm Linux Kernel Markers after each boot |
867d1aae |
326 | |
327 | ltt-armall |
328 | |
d1f19ac5 |
329 | * Use graphical LTTV to control tracing and analyse traces |
d9cd3a2e |
330 | |
331 | lttv-gui (or /usr/local/bin/lttv-gui) |
e44b6048 |
332 | - Spot the "Tracing Control" icon : click on it |
333 | (it's a traffic light icon) |
d9cd3a2e |
334 | - enter the root password |
335 | - click "start" |
336 | - click "stop" |
337 | - Yes |
338 | * You should now see a trace |
339 | |
d1f19ac5 |
340 | * Use text mode LTTng to control tracing |
341 | |
5e5b1de1 |
342 | The tracing can be controlled from a terminal by using the lttctl command (as |
343 | root). |
d1f19ac5 |
344 | |
345 | Start tracing : |
346 | |
4ad053df |
347 | lttctl -C -w /tmp/trace1 trace1 |
d1f19ac5 |
348 | |
349 | Stop tracing and destroy trace channels : |
350 | |
4ad053df |
351 | lttctl -D trace1 |
d1f19ac5 |
352 | |
353 | see lttctl --help for details. |
354 | |
b70ceef8 |
355 | (note : to see if the buffers has been filled, look at the dmesg output after |
356 | lttctl -R or after stopping tracing from the GUI, it will show an event lost |
357 | count. If it is the case, try using larger buffers. See lttctl --help to learn |
4ad053df |
358 | how. lttv now also shows event lost messages in the console when loading a trace |
359 | with missing events or lost subbuffers.) |
d1f19ac5 |
360 | |
6c913994 |
361 | * Use text mode LTTV |
362 | |
e55d936e |
363 | Feel free to look in /usr/local/lib/lttv/plugins to see all the text and |
6c913994 |
364 | graphical plugins available. |
365 | |
366 | For example, a simple trace dump in text format is available with : |
367 | |
368 | lttv -m textDump -t /tmp/trace |
369 | |
370 | see lttv -m textDump --help for detailed command line options of textDump. |
371 | |
b70ceef8 |
372 | It is, in the current state of the project, very useful to use "grep" on the |
373 | text output to filter by specific event fields. You can later copy the timestamp |
374 | of the events to the clipboard and paste them in the GUI by clicking on the |
375 | bottom right label "Current time". Support for this type of filtering should |
376 | be added to the filter module soon. |
6c913994 |
377 | |
ee33438f |
378 | * Hybrid mode |
d0acb5ae |
379 | |
380 | Starting from LTTng 0.5.105 and ltt-control 0.20, a new mode can be used : |
381 | hybrid. It can be especially useful when studying big workloads on a long period |
382 | of time. |
383 | |
384 | When using this mode, the most important, low rate control information will be |
385 | recorded during all the trace by lttd (i.e. process creation/exit). The high |
386 | rate information (i.e. interrupt/traps/syscall entry/exit) will be kept in a |
387 | flight recorder buffer (now named flight-channelname_X). |
388 | |
389 | The following lttctl commands take an hybrid trace : |
390 | |
391 | Create trace channel, start lttd on normal channels, start tracing: |
4ad053df |
392 | lttctl -C -w /tmp/trace2 -o channel.kernel.overwrite=1 trace2 |
d0acb5ae |
393 | |
394 | Stop tracing, start lttd on flight recorder channels, destroy trace channels : |
4ad053df |
395 | lttctl -D -w /tmp/trace2 trace2 |
d0acb5ae |
396 | |
4ad053df |
397 | Each "overwrite" channel is flight recorder channel. |
d0acb5ae |
398 | |
ee33438f |
399 | * Flight recorder mode |
400 | |
401 | The flight recorder mode writes data into overwritten buffers for all channels, |
4ad053df |
402 | including control channels, except for the facilities tracefiles. It consists of |
403 | setting all channels to "overwrite". |
ee33438f |
404 | |
405 | The following lttctl commands take a flight recorder trace : |
406 | |
4ad053df |
407 | lttctl -C -w /tmp/trace3 -o channel.all.overwrite=1 trace3 |
408 | ... |
409 | lttctl -D -w /tmp/trace3 trace3 |
ee33438f |
410 | |
6c913994 |
411 | |
839a3ec6 |
412 | ************************************************************** |
413 | ** Section 4 * Adding new instrumentations with the markers ** |
414 | ************************************************************** |
d9cd3a2e |
415 | |
839a3ec6 |
416 | See Documentation/markers.txt and Documentation/tracepoints.txt in your kernel |
417 | tree. |
d9cd3a2e |
418 | |
839a3ec6 |
419 | * Add new events to userspace programs with userspace markers |
420 | http://ltt.polymtl.ca/packages/ |
d9cd3a2e |
421 | |
839a3ec6 |
422 | Get the latest markers-userspace-*.tar.bz2 and see the Makefile and examples. It |
423 | allows inserting markers in executables and libraries, currently only on x86_32 |
424 | and x86_64. |
d9cd3a2e |
425 | |