c924c2c6 |
1 | <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> |
2 | <html> |
3 | <head> |
bc58079d |
4 | <title>Linux Trace Toolkit Next Generation User Documentation</title> |
c924c2c6 |
5 | </head> |
6 | <body> |
7 | |
bc58079d |
8 | <h1>Linux Trace Toolkit Next Generation User Documentation</h1> |
c924c2c6 |
9 | |
c924c2c6 |
10 | Author : Mathieu Desnoyers, September 2005<br> |
24070967 |
11 | Last update : January 21st, 2009<br> |
bc58079d |
12 | (originally known as the LTTng QUICKSTART guide) |
24070967 |
13 | |
14 | <h2>Table of Contents</h2> |
24070967 |
15 | |
d58b406f |
16 | <ul> |
f9ff56f3 |
17 | <li><a href="#intro" name="TOCintro">Introduction</a></li> |
7e77d1a9 |
18 | <ul> |
19 | <li><a href="#arch" name="TOCarch">Supported architectures</a></li> |
20 | </ul> |
d58b406f |
21 | |
24070967 |
22 | <li><a href="#section1" name="TOCsection1">Installing LTTng and LTTV from |
6f88b01d |
23 | sources</a></li> |
38b04cd7 |
24 | <ul> |
2d540155 |
25 | <li><a href="#prerequisites" name="TOCprerequisites">Prerequisistes</li> |
26 | <li><a href="#getlttng" name="TOCgetlttng">Getting the LTTng packages</li> |
27 | <li><a href="#getlttngsrc" name="TOCgetlttngsrc">Getting the LTTng kernel sources</li> |
28 | <li><a href="#installlttng" name="TOCinstalllttng">Installing a LTTng kernel</li> |
6f88b01d |
29 | <li><a href="#editconfig" name="TOCeditconfig">Editing the system wide |
30 | configuration</a> |
31 | <li><a href="#getlttctl" name="TOCgetlttctl">Getting and installing the |
2d540155 |
32 | ltt-control package</li> |
33 | <li><a href="#userspacetracing" name="TOCuserspacetracing">Userspace Tracing</li> |
6f88b01d |
34 | <li><a href="#getlttv" name="TOCgetlttv">Getting and installing the LTTV package</ul> |
d58b406f |
35 | |
24070967 |
36 | <li><a href="#section2" name="TOCsection2">Using LTTng and LTTV</a></li> |
d58b406f |
37 | <ul> |
38 | <li><a href="#uselttvgui" name="TOCuselttvgui">Use graphical LTTV to control |
39 | tracing and analyse traces</a></li> |
40 | <li><a href="#uselttngtext" name="TOCuselttngtext">Use text mode LTTng to |
f7828b7c |
41 | control tracing</a></li> |
d58b406f |
42 | <li><a href="#uselttvtext" name="TOCuselttvtext">Use text mode LTTV</a></li> |
43 | <li><a href="#hybrid" name="TOChybrid">Tracing in "Hybrid" mode</a></li> |
44 | <li><a href="#flight" name="TOCflight">Tracing in flight recorder mode</a></li> |
d58b406f |
45 | </ul> |
46 | |
24070967 |
47 | <li><a href="#section3" name="TOCsection3">Adding kernel and user-space |
47e2b195 |
48 | instrumentation</a> |
49 | <ul> |
50 | <li><a href="#kerneltp" name="TOCkerneltp">Adding kernel instrumentation</a></li> |
51 | <li><a href="#usertp" name="TOCusertp">Adding userspace instrumentation</a></li> |
52 | </ul> |
d58b406f |
53 | |
633bc4a3 |
54 | <li><a href="#section4" name="TOCsection4">Creating Debian and RPM packages |
98dde887 |
55 | from LTTV</a></li> |
56 | <ul> |
57 | <li><a href="#pkgdebian" name="TOCpkgdebian">Create custom LTTV Debian |
58 | <li><a href="#pkglttng" name="TOCpkglttng">Create custom LTTng packages</a></li> |
98dde887 |
59 | </ul> |
c924c2c6 |
60 | |
70a3fc43 |
61 | <li><a href="#section5" name="TOCsection5">Examples of LTTng use in the |
62 | field</a></li> |
63 | |
f9ff56f3 |
64 | </ul> |
65 | |
66 | <hr /> |
67 | |
b9e1fab1 |
68 | <h2><a href="#TOCintro" name="intro">Introduction</a></h2> |
c924c2c6 |
69 | <p> |
70a3fc43 |
70 | This document is made of five parts : the first one explains how |
24070967 |
71 | to install LTTng and LTTV from sources, the second one describes the steps |
633bc4a3 |
72 | to follow to trace a system and view it. The third part explains |
c924c2c6 |
73 | briefly how to add a new trace point to the kernel and to user space |
70a3fc43 |
74 | applications. The fourth part explains how to create Debian or RPM |
75 | packages from the LTTng and LTTV sources. The fifth and last part describes use |
76 | of LTTng in the field. |
c924c2c6 |
77 | <p> |
24070967 |
78 | These operations are made for installing the LTTng 0.86 tracer on a linux 2.6.X |
c924c2c6 |
79 | kernel. You will also find instructions for installation of LTTV 0.12.x : the |
80 | Linux Trace Toolkit Viewer. |
24070967 |
81 | To see the list of compatibilities between LTTng, ltt-control, LTTV, please |
82 | refer to : |
c924c2c6 |
83 | <a |
84 | href="http://ltt.polymtl.ca/svn/trunk/lttv/doc/developer/lttng-lttv-compatibility.html">LTTng+LTTV versions compatibility</a> |
534f65eb |
85 | |
86 | The ongoing work had the Linux Kernel Markers integrated in the mainline Linux |
87 | kernel since Linux 2.6.24 and the Tracepoints since 2.6.28. In its current |
88 | state, the lttng patchset is necessary to have the trace clocksource, the |
89 | instrumentation and the LTTng high-speed data extraction mechanism added to the |
90 | kernel. |
c924c2c6 |
91 | |
24070967 |
92 | <br> |
93 | <br> |
7e77d1a9 |
94 | <h3><a href="#TOCarch" name="arch">Supported architectures</a></h3> |
70a3fc43 |
95 | <br> |
c924c2c6 |
96 | LTTng :<br> |
70a3fc43 |
97 | <br> |
c924c2c6 |
98 | <li> x86 32/64 bits |
99 | <li> PowerPC 32 and 64 bits |
100 | <li> ARM (with limited timestamping precision, e.g. 1HZ. Need |
101 | architecture-specific support for better precision) |
102 | <li> MIPS |
1aeeacdb |
103 | <li> sh (partial architecture-specific instrumentation) |
104 | <li> sparc64 (partial architecture-specific instrumentation) |
105 | <li> s390 (partial architecture-specific instrumentation) |
106 | <li> Other architectures supported without architecture-specific instrumentation |
107 | and with low-resolution timestamps.<br> |
c924c2c6 |
108 | <br> |
24070967 |
109 | <br> |
c924c2c6 |
110 | LTTV :<br> |
70a3fc43 |
111 | <br> |
c924c2c6 |
112 | <li> Intel 32/64 bits |
113 | <li> PowerPC 32 and 64 bits |
114 | <li> Possibly others. Takes care of endianness and type size difference between |
115 | the LTTng traces and the LTTV analysis tool. |
116 | |
f9ff56f3 |
117 | <hr /> |
118 | |
c924c2c6 |
119 | |
89aa576c |
120 | <h2><a href="#TOCsection1" name="section1">Installation from sources</a></h2> |
633bc4a3 |
121 | <p> |
c924c2c6 |
122 | |
6f88b01d |
123 | <h3><a href="#TOCprerequisites" name="prerequisites">Prerequisites</a></h3> |
633bc4a3 |
124 | <ul> |
125 | <p> |
c924c2c6 |
126 | Tools needed to follow the package download steps : |
127 | |
633bc4a3 |
128 | <li>wget |
129 | <li>bzip2 |
130 | <li>gzip |
131 | <li>tar |
c924c2c6 |
132 | |
633bc4a3 |
133 | <p> |
c924c2c6 |
134 | You have to install the standard development libraries and programs necessary |
135 | to compile a kernel : |
136 | |
84cf5903 |
137 | <PRE> |
c924c2c6 |
138 | (from Documentation/Changes in the Linux kernel tree) |
84cf5903 |
139 | Gnu C 2.95.3 # gcc --version |
140 | Gnu make 3.79.1 # make --version |
141 | binutils 2.12 # ld -v |
142 | util-linux 2.10o # fdformat --version |
143 | module-init-tools 0.9.10 # depmod -V |
144 | </PRE> |
c924c2c6 |
145 | |
633bc4a3 |
146 | <p> |
c924c2c6 |
147 | You might also want to have libncurses5 to have the text mode kernel |
148 | configuration menu, but there are alternatives. |
149 | |
633bc4a3 |
150 | <p> |
c924c2c6 |
151 | Prerequisites for LTTV 0.x.x installation are : |
152 | |
84cf5903 |
153 | <PRE> |
154 | gcc 3.2 or better |
155 | gtk 2.4 or better development libraries |
c924c2c6 |
156 | (Debian : libgtk2.0, libgtk2.0-dev) |
157 | (Fedora : gtk2, gtk2-devel) |
158 | note : For Fedora users : this might require at least core 3 from Fedora, |
159 | or you might have to compile your own GTK2 library. |
84cf5903 |
160 | glib 2.4 or better development libraries |
c924c2c6 |
161 | (Debian : libglib2.0-0, libglib2.0-dev) |
162 | (Fedora : glib2, glib2-devel) |
84cf5903 |
163 | libpopt development libraries |
c924c2c6 |
164 | (Debian : libpopt0, libpopt-dev) |
165 | (Fedora : popt) |
84cf5903 |
166 | libpango development libraries |
c924c2c6 |
167 | (Debian : libpango1.0, libpango1.0-dev) |
168 | (Fedora : pango, pango-devel) |
84cf5903 |
169 | libc6 development librairies |
c924c2c6 |
170 | (Debian : libc6, libc6-dev) |
171 | (Fedora : glibc, glibc) |
84cf5903 |
172 | </PRE> |
633bc4a3 |
173 | </ul> |
c924c2c6 |
174 | |
c8997124 |
175 | <li>Reminder</li> |
c924c2c6 |
176 | |
c8997124 |
177 | <p> |
633bc4a3 |
178 | See the list of compatibilities between LTTng, ltt-control and LTTV at : |
179 | <a |
180 | href="http://ltt.polymtl.ca/svn/trunk/lttv/doc/developer/lttng-lttv-compatibility.html">LTTng+LTTV |
181 | versions compatibility</a>. |
c924c2c6 |
182 | |
183 | |
6f88b01d |
184 | <h3><a href="#TOCgetlttng" name="getlttng">Getting the LTTng packages</a></h3> |
c924c2c6 |
185 | |
c8997124 |
186 | <PRE> |
c924c2c6 |
187 | su - |
188 | mkdir /usr/src/lttng |
189 | cd /usr/src/lttng |
190 | (see http://ltt.polymtl.ca/lttng for package listing) |
191 | wget http://ltt.polymtl.ca/lttng/patch-2.6.X-lttng-0.x.xx.tar.bz2 |
192 | bzip2 -cd patch-2.6.X-lttng-0.x.xx.tar.bz2 | tar xvof - |
c8997124 |
193 | </PRE> |
c924c2c6 |
194 | |
195 | |
6f88b01d |
196 | <h3><a href="#TOCgetlttngsrc" name="getlttngsrc">Getting LTTng kernel sources</a></h3> |
c924c2c6 |
197 | |
c8997124 |
198 | <PRE> |
c924c2c6 |
199 | su - |
200 | cd /usr/src |
201 | wget http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.X.tar.bz2 |
202 | bzip2 -cd linux-2.6.X.tar.bz2 | tar xvof - |
203 | cd linux-2.6.X |
204 | - For LTTng 0.9.4- cat /usr/src/lttng/patch*-2.6.X-lttng-0.x.xx* | patch -p1 |
205 | - For LTTng 0.9.5+ apply the patches in the order specified in the series file, |
206 | or use quilt |
207 | cd .. |
208 | mv linux-2.6.X linux-2.6.X-lttng-0.x.xx |
c8997124 |
209 | </PRE> |
c924c2c6 |
210 | |
211 | |
6f88b01d |
212 | <h3><a href="#TOCinstalllttng" name="installlttng">Installing a LTTng kernel</a></h3> |
c924c2c6 |
213 | |
c8997124 |
214 | <PRE> |
c924c2c6 |
215 | su - |
216 | cd /usr/src/linux-2.6.X-lttng-0.x.xx |
217 | make menuconfig (or make xconfig or make config) |
218 | Select the < Help > button if you are not familiar with kernel |
219 | configuration. |
220 | Items preceded by [*] means they has to be built into the kernel. |
221 | Items preceded by [M] means they has to be built as modules. |
222 | Items preceded by [ ] means they should be removed. |
223 | go to the "General setup" section |
224 | Select the following options : |
225 | [*] Prompt for development and/or incomplete code/drivers |
226 | [*] Activate markers |
227 | [*] Activate userspace markers ABI (experimental, optional) |
228 | [*] Immediate value optimization (optional) |
229 | [*] Linux Trace Toolkit Next Generation (LTTng) ---> |
230 | <M> or <*> Compile lttng tracing probes |
231 | <M> or <*> Linux Trace Toolkit High-speed Lockless Data Relay |
232 | <M> or <*> Linux Trace Toolkit Lock-Protected Data Relay |
233 | <M> or <*> Linux Trace Toolkit Serializer |
234 | <M> or <*> Linux Trace Toolkit Marker Control |
235 | <M> or <*> Linux Trace Toolkit Tracer |
236 | [*] Align Linux Trace Toolkit Traces |
237 | <M> or <*> Support logging events from userspace |
238 | [*] Support trace extraction from crash dump |
239 | <M> or <*> Linux Trace Toolkit Trace Controller |
240 | <M> or <*> Linux Trace Toolkit State Dump |
241 | Select <Exit> |
242 | Select <Exit> |
243 | Select <Yes> |
244 | make |
245 | make modules_install |
246 | (if necessary, create a initrd with mkinitrd or your preferate alternative) |
247 | (mkinitrd -o /boot/initrd.img-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx) |
248 | |
249 | -- on X86, X86_64 |
250 | make install |
251 | reboot |
252 | Select the Linux 2.6.X-lttng-0.x.xx kernel in your boot loader. |
253 | |
254 | -- on PowerPC |
255 | cp vmlinux.strip /boot/vmlinux-2.6.X-lttng-0.x.xx |
256 | cp System.map /boot/System.map-2.6.X-lttng-0.x.xx |
257 | cp .config /boot/config-2.6.X-lttng-0.x.xx |
258 | depmod -ae -F /boot/System.map-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx |
259 | mkinitrd /boot/initrd.img-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx |
260 | (edit /etc/yaboot.conf to add a new entry pointing to your kernel : the entry |
261 | that comes first is the default kernel) |
262 | ybin |
263 | select the right entry at the yaboot prompt (see choices : tab, select : type |
264 | the kernel name followed by enter) |
265 | Select the Linux 2.6.X-lttng-0.x.xx kernel in your boot loader. |
266 | -- |
c8997124 |
267 | </PRE> |
c924c2c6 |
268 | |
6f88b01d |
269 | <h3><a href="#TOCeditconfig" name="editconfig">Editing the system wide |
270 | configuration</a></h3> |
c924c2c6 |
271 | |
c8997124 |
272 | <p> |
c924c2c6 |
273 | You must activate debugfs and specify a mount point. This is typically done in |
c8997124 |
274 | fstab such that it happens at boot time. If you have never used DebugFS before, |
275 | these operation would do this for you : |
c924c2c6 |
276 | |
c8997124 |
277 | <PRE> |
c924c2c6 |
278 | mkdir /mnt/debugfs |
279 | cp /etc/fstab /etc/fstab.lttng.bkp |
280 | echo "debugfs /mnt/debugfs debugfs rw 0 0" >> /etc/fstab |
c8997124 |
281 | </PRE> |
c924c2c6 |
282 | |
c8997124 |
283 | <p> |
c924c2c6 |
284 | then, rebooting or issuing the following command will activate debugfs : |
c8997124 |
285 | <PRE> |
c924c2c6 |
286 | mount /mnt/debugfs |
c8997124 |
287 | </PRE> |
c924c2c6 |
288 | |
c8997124 |
289 | <p> |
c924c2c6 |
290 | You need to load the LTT modules to be able to control tracing from user |
291 | space. This is done by issuing the following commands. Note however |
292 | these commands load all LTT modules. Depending on what options you chose to |
293 | compile statically, you may not need to issue all these commands. |
294 | |
c8997124 |
295 | <PRE> |
c924c2c6 |
296 | modprobe ltt-trace-control |
297 | modprobe ltt-marker-control |
298 | modprobe ltt-tracer |
299 | modprobe ltt-serialize |
300 | modprobe ltt-relay |
301 | modprobe ipc-trace |
302 | modprobe kernel-trace |
303 | modprobe mm-trace |
304 | modprobe net-trace |
305 | modprobe fs-trace |
306 | modprobe jbd2-trace |
307 | modprobe ext4-trace |
308 | modprobe syscall-trace |
309 | modprobe trap-trace |
310 | #if locking tracing is wanted, uncomment the following |
311 | #modprobe lockdep-trace |
c8997124 |
312 | </PRE> |
c924c2c6 |
313 | |
c8997124 |
314 | <p> |
c924c2c6 |
315 | If you want to have complete information about the kernel state (including all |
316 | the process names), you need to load the ltt-statedump module. This is done by |
317 | issuing the command : |
318 | |
c8997124 |
319 | <PRE> |
c924c2c6 |
320 | modprobe ltt-statedump |
c8997124 |
321 | </PRE> |
322 | <p> |
c924c2c6 |
323 | You can automate at boot time loading the ltt-control module by : |
324 | |
c8997124 |
325 | <PRE> |
c924c2c6 |
326 | cp /etc/modules /etc/modules.bkp |
327 | echo ltt-trace-control >> /etc/modules |
328 | echo ltt-marker-control >> /etc/modules |
329 | echo ltt-tracer >> /etc/modules |
330 | echo ltt-serialize >> /etc/modules |
331 | echo ltt-relay >> /etc/modules |
332 | echo ipc-trace >> /etc/modules |
333 | echo kernel-trace >> /etc/modules |
334 | echo mm-trace >> /etc/modules |
335 | echo net-trace >> /etc/modules |
336 | echo fs-trace >> /etc/modules |
337 | echo jbd2-trace >> /etc/modules |
338 | echo ext4-trace >> /etc/modules |
339 | echo syscall-trace >> /etc/modules |
340 | echo trap-trace >> /etc/modules |
341 | #if locking tracing is wanted, uncomment the following |
342 | #echo lockdep-trace >> /etc/modules |
c8997124 |
343 | </PRE> |
c924c2c6 |
344 | |
6f88b01d |
345 | |
346 | <h3><a href="#TOCgetlttctl" name="getlttctl">Getting and installing the |
347 | ltt-control package (on the traced machine)</a></h3> |
c8997124 |
348 | <p> |
c924c2c6 |
349 | (note : the ltt-control package contains lttd and lttctl. Although it has the |
350 | same name as the ltt-control kernel module, they are *not* the same thing.) |
c8997124 |
351 | |
352 | <PRE> |
c924c2c6 |
353 | su - |
354 | cd /usr/src |
355 | wget http://ltt.polymtl.ca/lttng/ltt-control-0.x-xxxx2006.tar.gz |
356 | gzip -cd ltt-control-0.x-xxxx2008.tar.gz | tar xvof - |
357 | cd ltt-control-0.x-xxxx2006 |
358 | (refer to README to see the development libraries that must be installed on you |
359 | system) |
360 | ./configure |
361 | make |
362 | make install |
c8997124 |
363 | </PRE> |
c924c2c6 |
364 | |
38b04cd7 |
365 | <h3><a href="#TOCuserspacetracing" name="userspacetracing">Userspace tracing</a></h3> |
c924c2c6 |
366 | |
c8997124 |
367 | <PRE> |
c924c2c6 |
368 | Make sure you selected the kernel menuconfig option : |
369 | <M> or <*> Support logging events from userspace |
370 | And that the ltt-userspace-event kernel module is loaded if selected as a |
371 | module. |
372 | |
373 | Simple userspace tracing is available through |
374 | echo "some text to record" > /mnt/debugfs/ltt/write_event |
375 | |
376 | It will appear in the trace under event : |
377 | channel : userspace |
378 | event name : event |
c8997124 |
379 | </PRE> |
c924c2c6 |
380 | |
6f88b01d |
381 | <h3><a href="#TOCgetlttv" name="getlttv">Getting and installing the LTTV package |
382 | (on the visualisation machine, same |
383 | or different from the visualisation machine)</a></h3> |
c924c2c6 |
384 | |
c8997124 |
385 | <PRE> |
c924c2c6 |
386 | su - |
387 | cd /usr/src |
388 | wget http://ltt.polymtl.ca/packages/lttv-0.x.xx-xxxx2008.tar.gz |
389 | gzip -cd lttv-0.x.xx-xxxx2008.tar.gz | tar xvof - |
390 | cd lttv-0.x.xx-xxxx2008 |
391 | (refer to README to see the development libraries that must be installed on your |
392 | system) |
393 | ./configure |
394 | make |
395 | make install |
13d7a628 |
396 | </PRE> |
c924c2c6 |
397 | |
c8997124 |
398 | <hr /> |
c924c2c6 |
399 | |
c924c2c6 |
400 | |
89aa576c |
401 | <h2><a href="#TOCsection2" name="section2">Using LTTng and LTTV</a></h2> |
c924c2c6 |
402 | |
d58b406f |
403 | <li><b>IMPORTANT : Arm Linux Kernel Markers after each boot before tracing</b></li> |
c8997124 |
404 | <PRE> |
c924c2c6 |
405 | ltt-armall |
c8997124 |
406 | </PRE> |
c924c2c6 |
407 | |
d58b406f |
408 | <h3><a href="#TOCuselttvgui" name="uselttvgui">Use graphical LTTV to control |
409 | tracing and analyse traces</a></h3> |
c8997124 |
410 | <PRE> |
c924c2c6 |
411 | lttv-gui (or /usr/local/bin/lttv-gui) |
412 | - Spot the "Tracing Control" icon : click on it |
413 | (it's a traffic light icon) |
414 | - enter the root password |
415 | - click "start" |
416 | - click "stop" |
417 | - Yes |
418 | * You should now see a trace |
c8997124 |
419 | </PRE> |
c924c2c6 |
420 | |
d58b406f |
421 | <h3><a href="#TOCuselttngtext" name="uselttngtext">Use text mode LTTng to control tracing</a></h3> |
c8997124 |
422 | <PRE> |
c924c2c6 |
423 | The tracing can be controlled from a terminal by using the lttctl command (as |
424 | root). |
425 | |
426 | Start tracing : |
427 | |
428 | lttctl -C -w /tmp/trace1 trace1 |
429 | |
430 | Stop tracing and destroy trace channels : |
431 | |
432 | lttctl -D trace1 |
433 | |
434 | see lttctl --help for details. |
df7f63ab |
435 | </PRE> |
c8997124 |
436 | <p> |
c924c2c6 |
437 | (note : to see if the buffers has been filled, look at the dmesg output after |
438 | lttctl -R or after stopping tracing from the GUI, it will show an event lost |
439 | count. If it is the case, try using larger buffers. See lttctl --help to learn |
440 | how. lttv now also shows event lost messages in the console when loading a trace |
441 | with missing events or lost subbuffers.) |
442 | |
d58b406f |
443 | <h3><a href="#TOCuselttvtext" name="uselttvtext">Use text mode LTTV</a></h3> |
c8997124 |
444 | <p> |
c924c2c6 |
445 | Feel free to look in /usr/local/lib/lttv/plugins to see all the text and |
446 | graphical plugins available. |
c8997124 |
447 | <p> |
c924c2c6 |
448 | For example, a simple trace dump in text format is available with : |
c8997124 |
449 | <PRE> |
c924c2c6 |
450 | lttv -m textDump -t /tmp/trace |
c8997124 |
451 | </PRE> |
c8997124 |
452 | <p> |
453 | See lttv -m textDump --help for detailed command line options of textDump. |
c8997124 |
454 | <p> |
c924c2c6 |
455 | It is, in the current state of the project, very useful to use "grep" on the |
456 | text output to filter by specific event fields. You can later copy the timestamp |
457 | of the events to the clipboard and paste them in the GUI by clicking on the |
458 | bottom right label "Current time". Support for this type of filtering should |
459 | be added to the filter module soon. |
460 | |
d58b406f |
461 | <h3><a href="#TOChybrid" name="hybrid">Tracing in "Hybrid" mode</a></h3> |
c8997124 |
462 | <p> |
c924c2c6 |
463 | Starting from LTTng 0.5.105 and ltt-control 0.20, a new mode can be used : |
464 | hybrid. It can be especially useful when studying big workloads on a long period |
465 | of time. |
c8997124 |
466 | <p> |
c924c2c6 |
467 | When using this mode, the most important, low rate control information will be |
468 | recorded during all the trace by lttd (i.e. process creation/exit). The high |
469 | rate information (i.e. interrupt/traps/syscall entry/exit) will be kept in a |
470 | flight recorder buffer (now named flight-channelname_X). |
c8997124 |
471 | <p> |
c924c2c6 |
472 | The following lttctl commands take an hybrid trace : |
c8997124 |
473 | <p> |
c924c2c6 |
474 | Create trace channel, start lttd on normal channels, start tracing: |
c8997124 |
475 | <PRE> |
c924c2c6 |
476 | lttctl -C -w /tmp/trace2 -o channel.kernel.overwrite=1 trace2 |
c8997124 |
477 | </PRE> |
478 | <p> |
c924c2c6 |
479 | Stop tracing, start lttd on flight recorder channels, destroy trace channels : |
c8997124 |
480 | <PRE> |
c924c2c6 |
481 | lttctl -D -w /tmp/trace2 trace2 |
c8997124 |
482 | </PRE> |
483 | <p> |
c924c2c6 |
484 | Each "overwrite" channel is flight recorder channel. |
485 | |
d58b406f |
486 | |
487 | <h3><a href="#TOCflight" name="flight">Tracing in flight recorder mode</a></h3> |
c8997124 |
488 | <li>Flight recorder mode</li> |
c8997124 |
489 | <p> |
c924c2c6 |
490 | The flight recorder mode writes data into overwritten buffers for all channels, |
491 | including control channels, except for the facilities tracefiles. It consists of |
492 | setting all channels to "overwrite". |
c8997124 |
493 | <p> |
c924c2c6 |
494 | The following lttctl commands take a flight recorder trace : |
c8997124 |
495 | <PRE> |
c924c2c6 |
496 | lttctl -C -w /tmp/trace3 -o channel.all.overwrite=1 trace3 |
497 | ... |
498 | lttctl -D -w /tmp/trace3 trace3 |
c8997124 |
499 | </PRE> |
c924c2c6 |
500 | |
89aa576c |
501 | <hr /> |
502 | |
503 | |
504 | <h2><a href="#TOCsection3" name="section3">Adding new instrumentations with the |
505 | markers</a></h2> |
506 | <p> |
c924c2c6 |
507 | |
47e2b195 |
508 | <h3><a href="#TOCkerneltp" name="kerneltp">Adding kernel |
509 | instrumentation</a></h3> |
510 | |
89aa576c |
511 | <p> |
47e2b195 |
512 | See <a |
513 | href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=Documentation/markers.txt">Documentation/markers.txt</a> |
514 | and <a |
515 | href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=Documentation/tracepoints.txt">Documentation/tracepoints.txt</a> in your kernel |
c924c2c6 |
516 | tree. |
47e2b195 |
517 | <p> |
518 | Also see <a |
519 | href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=tree;f=ltt/probes">ltt/probes/</a> |
520 | for LTTng probe examples. |
521 | |
522 | <h3><a href="#TOCusertp" name="usertp">Adding userspace instrumentation</a></h3> |
c924c2c6 |
523 | |
305fd815 |
524 | Add new events to userspace programs with |
525 | <a href="http://ltt.polymtl.ca/packages/">userspace markers packages</a>. |
c924c2c6 |
526 | Get the latest markers-userspace-*.tar.bz2 and see the Makefile and examples. It |
527 | allows inserting markers in executables and libraries, currently only on x86_32 |
528 | and x86_64. |
e01a1ce1 |
529 | See <a |
530 | href="http://ltt.polymtl.ca/packages/markers-userspace-0.5.tar.bz2">markers-userspace-0.5.tar.bz2</a> or more recent. |
531 | |
38b04cd7 |
532 | <p> |
e01a1ce1 |
533 | Note that a new design document for a 3rd generation of tracepoint/marker-based |
534 | userspace tracing is available at <a |
535 | href="http://ltt.polymtl.ca/svn/trunk/lttv/doc/developer/ust.html">LTTng User-space Tracing |
98dde887 |
536 | Design</a>. This new infrastructure is not yet implemented. |
38b04cd7 |
537 | |
538 | <p> |
539 | The easy quick-and-dirty way to perform userspace tracing is currently to write |
540 | an string to /mnt/debugfs/ltt/write_event. See <a |
541 | href="#userspacetracing">Userspace tracing</a> in the |
542 | installation for sources section of this document. |
e01a1ce1 |
543 | |
89aa576c |
544 | <hr /> |
633bc4a3 |
545 | |
89aa576c |
546 | <h2><a href="#TOCsection4" name="section4">Creating Debian or RPM packages</a></h2> |
547 | <p> |
633bc4a3 |
548 | |
98dde887 |
549 | <h3><a href="#TOCpkgdebian" name="pkgdebian">Create custom LTTV Debian packages</a></h3> |
633bc4a3 |
550 | |
89aa576c |
551 | <PRE> |
552 | Use : dpkg-buildpackage -rfakeroot |
553 | </PRE> |
554 | <p> |
633bc4a3 |
555 | You should then have your LTTV .deb files created for your architecture. |
556 | |
98dde887 |
557 | <h3><a href="#TOCpkglttng" name="pkglttng">Create custom LTTng packages</a></h3> |
89aa576c |
558 | <p> |
633bc4a3 |
559 | For building LTTng Debian packages : |
89aa576c |
560 | get the build tree with patches applies as explained in section 2. |
633bc4a3 |
561 | |
89aa576c |
562 | <PRE> |
633bc4a3 |
563 | make menuconfig (or xconfig or config) (customize your configuration) |
564 | make-kpkg kernel_image |
89aa576c |
565 | </PRE> |
566 | <p> |
633bc4a3 |
567 | You will then see your freshly created .deb in /usr/src. Install it with |
89aa576c |
568 | <PRE> |
633bc4a3 |
569 | dpkg -i /usr/src/(image-name).deb |
89aa576c |
570 | </PRE> |
571 | <p> |
633bc4a3 |
572 | Then, follow the section "Editing the system wide configuration" in section 2. |
573 | |
70a3fc43 |
574 | <hr /> |
633bc4a3 |
575 | |
70a3fc43 |
576 | <h2><a href="#TOCsection5" name="section5">Examples of LTTng use in the field</a></h2> |
577 | <p> |
578 | A few examples of successful LTTng users : |
633bc4a3 |
579 | |
70a3fc43 |
580 | <ul> |
581 | <li> Google are deploying LTTng on their servers. They want to use it to |
582 | monitor their production servers (with flight recorder mode tracing) |
583 | and to help them solve hard to reproduce problems. They have had |
584 | success with such tracing approach to fix "rare disk delay" issues and |
585 | VM-related issues presented in this article : |
07e962a1 |
586 | <ul> |
83cebd38 |
587 | <li> <a href="http://ltt.polymtl.ca/papers/bligh-Reprint.pdf">Linux Kernel |
588 | Debugging on Google-sized clusters at Ottawa Linux |
589 | Symposium 2007</a> |
07e962a1 |
590 | </ul> |
70a3fc43 |
591 | <li> IBM Research have had problems with Commercial Scale-out applications, |
592 | which are being an increasing trend to split large server workloads. |
593 | They used LTTng successfully to solve a distributed filesystem-related |
594 | issue. It's presented in the same paper above. |
595 | |
596 | <li> Autodesk, in the development of their next-generation of Linux |
597 | audio/video edition applications, used LTTng extensively to solve |
598 | soft real-time issues they had. Also presented in the same paper. |
599 | |
600 | <li> Wind River included LTTng in their Linux distribution so their |
601 | clients, already familiar to Wind River own tracing solution in |
602 | VxWorks, car have the same kind of feature they have relied on for a |
603 | long time. |
604 | |
605 | <li> Montavista have integrated LTTng in their distribution for the same |
606 | reasons. It's used by Sony amongst others. |
607 | |
608 | <li> SuSE are currently integrating LTTng in their next SLES distribution, |
609 | because their clients asking for solutions which supports a kernel |
610 | closer to real-time need such tools to debug their problems. |
611 | |
612 | <li> A project between Ericsson, the Canadian Defense, NSERC and various |
613 | universities is just starting. It aims at monitoring and debugging |
614 | multi-core systems and provide automated and help user system behavior |
615 | analysis. |
616 | |
617 | <li> Siemens have been using LTTng internally for quite some time now. |
618 | </ul> |
c924c2c6 |
619 | </body> |
620 | </html> |