| | 1 | <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> |
| | 2 | <html> |
| | 3 | <head> |
| | 4 | <title>Linux Trace Toolkit Quickstart</title> |
| | 5 | </head> |
| | 6 | <body> |
| | 7 | |
| | 8 | <h1>Linux Trace Toolkit Quickstart</h1> |
| | 9 | |
| | 10 | Author : Mathieu Desnoyers, September 2005<br> |
| | 11 | Last update : January 21st, 2009<br> |
| | 12 | |
| | 13 | <h2>Table of Contents</h2> |
| | 14 | <ul> |
| | 15 | |
| | 16 | <li><a href="#intro" name="TOCintro">Introduction</a></li> |
| | 17 | <li><a href="#section1" name="TOCsection1">Installing LTTng and LTTV from |
| | 18 | sources</a></li> |
| | 19 | <li><a href="#section2" name="TOCsection2">Using LTTng and LTTV</a></li> |
| | 20 | <li><a href="#section3" name="TOCsection3">Adding kernel and user-space |
| | 21 | tracepoints</a></li> |
| | 22 | <li><a href="#section4" name="TOCsection4">Creating Debian and RPM packages |
| | 23 | from LTTV</a> |
| | 24 | |
| | 25 | </ul> |
| | 26 | |
| | 27 | <hr /> |
| | 28 | |
| | 29 | <h2><a href="#TOCintro" name="intro">Introduction</a></h2> |
| | 30 | <p> |
| | 31 | This document is made of four parts : the first one explains how |
| | 32 | to install LTTng and LTTV from sources, the second one describes the steps |
| | 33 | to follow to trace a system and view it. The third part explains |
| | 34 | briefly how to add a new trace point to the kernel and to user space |
| | 35 | applications. The fourth and last part explains how to create Debian or RPM |
| | 36 | packages from the LTTng and LTTV sources. |
| | 37 | |
| | 38 | <p> |
| | 39 | These operations are made for installing the LTTng 0.86 tracer on a linux 2.6.X |
| | 40 | kernel. You will also find instructions for installation of LTTV 0.12.x : the |
| | 41 | Linux Trace Toolkit Viewer. |
| | 42 | To see the list of compatibilities between LTTng, ltt-control, LTTV, please |
| | 43 | refer to : |
| | 44 | <a |
| | 45 | href="http://ltt.polymtl.ca/svn/trunk/lttv/doc/developer/lttng-lttv-compatibility.html">LTTng+LTTV versions compatibility</a> |
| | 46 | The lttng patch is necessary to have the tracing hooks in the kernel. |
| | 47 | |
| | 48 | <br> |
| | 49 | <br> |
| | 50 | Supported architectures : |
| | 51 | <br> |
| | 52 | LTTng :<br> |
| | 53 | <li> x86 32/64 bits |
| | 54 | <li> PowerPC 32 and 64 bits |
| | 55 | <li> ARM (with limited timestamping precision, e.g. 1HZ. Need |
| | 56 | architecture-specific support for better precision) |
| | 57 | <li> MIPS |
| | 58 | <br> |
| | 59 | <br> |
| | 60 | LTTV :<br> |
| | 61 | <li> Intel 32/64 bits |
| | 62 | <li> PowerPC 32 and 64 bits |
| | 63 | <li> Possibly others. Takes care of endianness and type size difference between |
| | 64 | the LTTng traces and the LTTV analysis tool. |
| | 65 | |
| | 66 | <hr /> |
| | 67 | |
| | 68 | |
| | 69 | <h2><a href="#TOCsection2" name="section2">Installation from sources</a></h2> |
| | 70 | <p> |
| | 71 | |
| | 72 | <li>Prerequisites</li> |
| | 73 | <ul> |
| | 74 | <p> |
| | 75 | Tools needed to follow the package download steps : |
| | 76 | |
| | 77 | <li>wget |
| | 78 | <li>bzip2 |
| | 79 | <li>gzip |
| | 80 | <li>tar |
| | 81 | |
| | 82 | <p> |
| | 83 | You have to install the standard development libraries and programs necessary |
| | 84 | to compile a kernel : |
| | 85 | |
| | 86 | <PRE> |
| | 87 | (from Documentation/Changes in the Linux kernel tree) |
| | 88 | Gnu C 2.95.3 # gcc --version |
| | 89 | Gnu make 3.79.1 # make --version |
| | 90 | binutils 2.12 # ld -v |
| | 91 | util-linux 2.10o # fdformat --version |
| | 92 | module-init-tools 0.9.10 # depmod -V |
| | 93 | </PRE> |
| | 94 | |
| | 95 | <p> |
| | 96 | You might also want to have libncurses5 to have the text mode kernel |
| | 97 | configuration menu, but there are alternatives. |
| | 98 | |
| | 99 | <p> |
| | 100 | Prerequisites for LTTV 0.x.x installation are : |
| | 101 | |
| | 102 | <PRE> |
| | 103 | gcc 3.2 or better |
| | 104 | gtk 2.4 or better development libraries |
| | 105 | (Debian : libgtk2.0, libgtk2.0-dev) |
| | 106 | (Fedora : gtk2, gtk2-devel) |
| | 107 | note : For Fedora users : this might require at least core 3 from Fedora, |
| | 108 | or you might have to compile your own GTK2 library. |
| | 109 | glib 2.4 or better development libraries |
| | 110 | (Debian : libglib2.0-0, libglib2.0-dev) |
| | 111 | (Fedora : glib2, glib2-devel) |
| | 112 | libpopt development libraries |
| | 113 | (Debian : libpopt0, libpopt-dev) |
| | 114 | (Fedora : popt) |
| | 115 | libpango development libraries |
| | 116 | (Debian : libpango1.0, libpango1.0-dev) |
| | 117 | (Fedora : pango, pango-devel) |
| | 118 | libc6 development librairies |
| | 119 | (Debian : libc6, libc6-dev) |
| | 120 | (Fedora : glibc, glibc) |
| | 121 | </PRE> |
| | 122 | </ul> |
| | 123 | |
| | 124 | <li>Reminder</li> |
| | 125 | |
| | 126 | <p> |
| | 127 | See the list of compatibilities between LTTng, ltt-control and LTTV at : |
| | 128 | <a |
| | 129 | href="http://ltt.polymtl.ca/svn/trunk/lttv/doc/developer/lttng-lttv-compatibility.html">LTTng+LTTV |
| | 130 | versions compatibility</a>. |
| | 131 | |
| | 132 | |
| | 133 | <li>Getting the LTTng packages</li> |
| | 134 | |
| | 135 | <PRE> |
| | 136 | su - |
| | 137 | mkdir /usr/src/lttng |
| | 138 | cd /usr/src/lttng |
| | 139 | (see http://ltt.polymtl.ca/lttng for package listing) |
| | 140 | wget http://ltt.polymtl.ca/lttng/patch-2.6.X-lttng-0.x.xx.tar.bz2 |
| | 141 | bzip2 -cd patch-2.6.X-lttng-0.x.xx.tar.bz2 | tar xvof - |
| | 142 | </PRE> |
| | 143 | |
| | 144 | |
| | 145 | <li>Getting LTTng kernel sources</li> |
| | 146 | |
| | 147 | <PRE> |
| | 148 | su - |
| | 149 | cd /usr/src |
| | 150 | wget http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.X.tar.bz2 |
| | 151 | bzip2 -cd linux-2.6.X.tar.bz2 | tar xvof - |
| | 152 | cd linux-2.6.X |
| | 153 | - For LTTng 0.9.4- cat /usr/src/lttng/patch*-2.6.X-lttng-0.x.xx* | patch -p1 |
| | 154 | - For LTTng 0.9.5+ apply the patches in the order specified in the series file, |
| | 155 | or use quilt |
| | 156 | cd .. |
| | 157 | mv linux-2.6.X linux-2.6.X-lttng-0.x.xx |
| | 158 | </PRE> |
| | 159 | |
| | 160 | |
| | 161 | <li>Installing a LTTng kernel</li> |
| | 162 | |
| | 163 | <PRE> |
| | 164 | su - |
| | 165 | cd /usr/src/linux-2.6.X-lttng-0.x.xx |
| | 166 | make menuconfig (or make xconfig or make config) |
| | 167 | Select the < Help > button if you are not familiar with kernel |
| | 168 | configuration. |
| | 169 | Items preceded by [*] means they has to be built into the kernel. |
| | 170 | Items preceded by [M] means they has to be built as modules. |
| | 171 | Items preceded by [ ] means they should be removed. |
| | 172 | go to the "General setup" section |
| | 173 | Select the following options : |
| | 174 | [*] Prompt for development and/or incomplete code/drivers |
| | 175 | [*] Activate markers |
| | 176 | [*] Activate userspace markers ABI (experimental, optional) |
| | 177 | [*] Immediate value optimization (optional) |
| | 178 | [*] Linux Trace Toolkit Next Generation (LTTng) ---> |
| | 179 | <M> or <*> Compile lttng tracing probes |
| | 180 | <M> or <*> Linux Trace Toolkit High-speed Lockless Data Relay |
| | 181 | <M> or <*> Linux Trace Toolkit Lock-Protected Data Relay |
| | 182 | <M> or <*> Linux Trace Toolkit Serializer |
| | 183 | <M> or <*> Linux Trace Toolkit Marker Control |
| | 184 | <M> or <*> Linux Trace Toolkit Tracer |
| | 185 | [*] Align Linux Trace Toolkit Traces |
| | 186 | <M> or <*> Support logging events from userspace |
| | 187 | [*] Support trace extraction from crash dump |
| | 188 | <M> or <*> Linux Trace Toolkit Trace Controller |
| | 189 | <M> or <*> Linux Trace Toolkit State Dump |
| | 190 | Select <Exit> |
| | 191 | Select <Exit> |
| | 192 | Select <Yes> |
| | 193 | make |
| | 194 | make modules_install |
| | 195 | (if necessary, create a initrd with mkinitrd or your preferate alternative) |
| | 196 | (mkinitrd -o /boot/initrd.img-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx) |
| | 197 | |
| | 198 | -- on X86, X86_64 |
| | 199 | make install |
| | 200 | reboot |
| | 201 | Select the Linux 2.6.X-lttng-0.x.xx kernel in your boot loader. |
| | 202 | |
| | 203 | -- on PowerPC |
| | 204 | cp vmlinux.strip /boot/vmlinux-2.6.X-lttng-0.x.xx |
| | 205 | cp System.map /boot/System.map-2.6.X-lttng-0.x.xx |
| | 206 | cp .config /boot/config-2.6.X-lttng-0.x.xx |
| | 207 | depmod -ae -F /boot/System.map-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx |
| | 208 | mkinitrd /boot/initrd.img-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx |
| | 209 | (edit /etc/yaboot.conf to add a new entry pointing to your kernel : the entry |
| | 210 | that comes first is the default kernel) |
| | 211 | ybin |
| | 212 | select the right entry at the yaboot prompt (see choices : tab, select : type |
| | 213 | the kernel name followed by enter) |
| | 214 | Select the Linux 2.6.X-lttng-0.x.xx kernel in your boot loader. |
| | 215 | -- |
| | 216 | </PRE> |
| | 217 | |
| | 218 | <li>Editing the system wide configuration</li> |
| | 219 | |
| | 220 | <p> |
| | 221 | You must activate debugfs and specify a mount point. This is typically done in |
| | 222 | fstab such that it happens at boot time. If you have never used DebugFS before, |
| | 223 | these operation would do this for you : |
| | 224 | |
| | 225 | <PRE> |
| | 226 | mkdir /mnt/debugfs |
| | 227 | cp /etc/fstab /etc/fstab.lttng.bkp |
| | 228 | echo "debugfs /mnt/debugfs debugfs rw 0 0" >> /etc/fstab |
| | 229 | </PRE> |
| | 230 | |
| | 231 | <p> |
| | 232 | then, rebooting or issuing the following command will activate debugfs : |
| | 233 | <PRE> |
| | 234 | mount /mnt/debugfs |
| | 235 | </PRE> |
| | 236 | |
| | 237 | <p> |
| | 238 | You need to load the LTT modules to be able to control tracing from user |
| | 239 | space. This is done by issuing the following commands. Note however |
| | 240 | these commands load all LTT modules. Depending on what options you chose to |
| | 241 | compile statically, you may not need to issue all these commands. |
| | 242 | |
| | 243 | <PRE> |
| | 244 | modprobe ltt-trace-control |
| | 245 | modprobe ltt-marker-control |
| | 246 | modprobe ltt-tracer |
| | 247 | modprobe ltt-serialize |
| | 248 | modprobe ltt-relay |
| | 249 | modprobe ipc-trace |
| | 250 | modprobe kernel-trace |
| | 251 | modprobe mm-trace |
| | 252 | modprobe net-trace |
| | 253 | modprobe fs-trace |
| | 254 | modprobe jbd2-trace |
| | 255 | modprobe ext4-trace |
| | 256 | modprobe syscall-trace |
| | 257 | modprobe trap-trace |
| | 258 | #if locking tracing is wanted, uncomment the following |
| | 259 | #modprobe lockdep-trace |
| | 260 | </PRE> |
| | 261 | |
| | 262 | <p> |
| | 263 | If you want to have complete information about the kernel state (including all |
| | 264 | the process names), you need to load the ltt-statedump module. This is done by |
| | 265 | issuing the command : |
| | 266 | |
| | 267 | <PRE> |
| | 268 | modprobe ltt-statedump |
| | 269 | </PRE> |
| | 270 | <p> |
| | 271 | You can automate at boot time loading the ltt-control module by : |
| | 272 | |
| | 273 | <PRE> |
| | 274 | cp /etc/modules /etc/modules.bkp |
| | 275 | echo ltt-trace-control >> /etc/modules |
| | 276 | echo ltt-marker-control >> /etc/modules |
| | 277 | echo ltt-tracer >> /etc/modules |
| | 278 | echo ltt-serialize >> /etc/modules |
| | 279 | echo ltt-relay >> /etc/modules |
| | 280 | echo ipc-trace >> /etc/modules |
| | 281 | echo kernel-trace >> /etc/modules |
| | 282 | echo mm-trace >> /etc/modules |
| | 283 | echo net-trace >> /etc/modules |
| | 284 | echo fs-trace >> /etc/modules |
| | 285 | echo jbd2-trace >> /etc/modules |
| | 286 | echo ext4-trace >> /etc/modules |
| | 287 | echo syscall-trace >> /etc/modules |
| | 288 | echo trap-trace >> /etc/modules |
| | 289 | #if locking tracing is wanted, uncomment the following |
| | 290 | #echo lockdep-trace >> /etc/modules |
| | 291 | </PRE> |
| | 292 | |
| | 293 | <li>Getting and installing the ltt-control package (on the traced machine)</li> |
| | 294 | <p> |
| | 295 | (note : the ltt-control package contains lttd and lttctl. Although it has the |
| | 296 | same name as the ltt-control kernel module, they are *not* the same thing.) |
| | 297 | |
| | 298 | <PRE> |
| | 299 | su - |
| | 300 | cd /usr/src |
| | 301 | wget http://ltt.polymtl.ca/lttng/ltt-control-0.x-xxxx2006.tar.gz |
| | 302 | gzip -cd ltt-control-0.x-xxxx2008.tar.gz | tar xvof - |
| | 303 | cd ltt-control-0.x-xxxx2006 |
| | 304 | (refer to README to see the development libraries that must be installed on you |
| | 305 | system) |
| | 306 | ./configure |
| | 307 | make |
| | 308 | make install |
| | 309 | </PRE> |
| | 310 | |
| | 311 | <li>Userspace tracing</li> |
| | 312 | |
| | 313 | <PRE> |
| | 314 | Make sure you selected the kernel menuconfig option : |
| | 315 | <M> or <*> Support logging events from userspace |
| | 316 | And that the ltt-userspace-event kernel module is loaded if selected as a |
| | 317 | module. |
| | 318 | |
| | 319 | Simple userspace tracing is available through |
| | 320 | echo "some text to record" > /mnt/debugfs/ltt/write_event |
| | 321 | |
| | 322 | It will appear in the trace under event : |
| | 323 | channel : userspace |
| | 324 | event name : event |
| | 325 | </PRE> |
| | 326 | |
| | 327 | <li>Getting and installing the LTTV package (on the visualisation machine, same |
| | 328 | or different from the visualisation machine)</li> |
| | 329 | |
| | 330 | <PRE> |
| | 331 | su - |
| | 332 | cd /usr/src |
| | 333 | wget http://ltt.polymtl.ca/packages/lttv-0.x.xx-xxxx2008.tar.gz |
| | 334 | gzip -cd lttv-0.x.xx-xxxx2008.tar.gz | tar xvof - |
| | 335 | cd lttv-0.x.xx-xxxx2008 |
| | 336 | (refer to README to see the development libraries that must be installed on your |
| | 337 | system) |
| | 338 | ./configure |
| | 339 | make |
| | 340 | make install |
| | 341 | |
| | 342 | |
| | 343 | <li>Getting and installing the markers-userspace package for user space |
| | 344 | tracing (experimental)</li> |
| | 345 | |
| | 346 | See <a |
| | 347 | href="http://ltt.polymtl.ca/packages/markers-userspace-0.5.tar.bz2">markers-userspace-0.5.tar.bz2</a> or more recent. |
| | 348 | |
| | 349 | <hr /> |
| | 350 | |
| | 351 | |
| | 352 | <h2><a href="#TOCsection3" name="section3">Using LTTng and LTTV</a></h2> |
| | 353 | |
| | 354 | <li>IMPORTANT : Arm Linux Kernel Markers after each boot</li> |
| | 355 | |
| | 356 | <PRE> |
| | 357 | ltt-armall |
| | 358 | </PRE> |
| | 359 | |
| | 360 | <li>Use graphical LTTV to control tracing and analyse traces</li> |
| | 361 | |
| | 362 | <PRE> |
| | 363 | lttv-gui (or /usr/local/bin/lttv-gui) |
| | 364 | - Spot the "Tracing Control" icon : click on it |
| | 365 | (it's a traffic light icon) |
| | 366 | - enter the root password |
| | 367 | - click "start" |
| | 368 | - click "stop" |
| | 369 | - Yes |
| | 370 | * You should now see a trace |
| | 371 | </PRE> |
| | 372 | |
| | 373 | <li>Use text mode LTTng to control tracing</li> |
| | 374 | <PRE> |
| | 375 | The tracing can be controlled from a terminal by using the lttctl command (as |
| | 376 | root). |
| | 377 | |
| | 378 | Start tracing : |
| | 379 | |
| | 380 | lttctl -C -w /tmp/trace1 trace1 |
| | 381 | |
| | 382 | Stop tracing and destroy trace channels : |
| | 383 | |
| | 384 | lttctl -D trace1 |
| | 385 | |
| | 386 | see lttctl --help for details. |
| | 387 | <PRE> |
| | 388 | <p> |
| | 389 | (note : to see if the buffers has been filled, look at the dmesg output after |
| | 390 | lttctl -R or after stopping tracing from the GUI, it will show an event lost |
| | 391 | count. If it is the case, try using larger buffers. See lttctl --help to learn |
| | 392 | how. lttv now also shows event lost messages in the console when loading a trace |
| | 393 | with missing events or lost subbuffers.) |
| | 394 | |
| | 395 | <li>Use text mode LTTV</li> |
| | 396 | <p> |
| | 397 | Feel free to look in /usr/local/lib/lttv/plugins to see all the text and |
| | 398 | graphical plugins available. |
| | 399 | <p> |
| | 400 | For example, a simple trace dump in text format is available with : |
| | 401 | |
| | 402 | <PRE> |
| | 403 | lttv -m textDump -t /tmp/trace |
| | 404 | </PRE> |
| | 405 | |
| | 406 | <p> |
| | 407 | See lttv -m textDump --help for detailed command line options of textDump. |
| | 408 | |
| | 409 | <p> |
| | 410 | It is, in the current state of the project, very useful to use "grep" on the |
| | 411 | text output to filter by specific event fields. You can later copy the timestamp |
| | 412 | of the events to the clipboard and paste them in the GUI by clicking on the |
| | 413 | bottom right label "Current time". Support for this type of filtering should |
| | 414 | be added to the filter module soon. |
| | 415 | |
| | 416 | <li>Hybrid mode</li> |
| | 417 | |
| | 418 | <p> |
| | 419 | Starting from LTTng 0.5.105 and ltt-control 0.20, a new mode can be used : |
| | 420 | hybrid. It can be especially useful when studying big workloads on a long period |
| | 421 | of time. |
| | 422 | |
| | 423 | <p> |
| | 424 | When using this mode, the most important, low rate control information will be |
| | 425 | recorded during all the trace by lttd (i.e. process creation/exit). The high |
| | 426 | rate information (i.e. interrupt/traps/syscall entry/exit) will be kept in a |
| | 427 | flight recorder buffer (now named flight-channelname_X). |
| | 428 | |
| | 429 | <p> |
| | 430 | The following lttctl commands take an hybrid trace : |
| | 431 | <p> |
| | 432 | Create trace channel, start lttd on normal channels, start tracing: |
| | 433 | <PRE> |
| | 434 | lttctl -C -w /tmp/trace2 -o channel.kernel.overwrite=1 trace2 |
| | 435 | </PRE> |
| | 436 | <p> |
| | 437 | Stop tracing, start lttd on flight recorder channels, destroy trace channels : |
| | 438 | <PRE> |
| | 439 | lttctl -D -w /tmp/trace2 trace2 |
| | 440 | </PRE> |
| | 441 | <p> |
| | 442 | Each "overwrite" channel is flight recorder channel. |
| | 443 | |
| | 444 | <li>Flight recorder mode</li> |
| | 445 | |
| | 446 | <p> |
| | 447 | The flight recorder mode writes data into overwritten buffers for all channels, |
| | 448 | including control channels, except for the facilities tracefiles. It consists of |
| | 449 | setting all channels to "overwrite". |
| | 450 | |
| | 451 | <p> |
| | 452 | The following lttctl commands take a flight recorder trace : |
| | 453 | |
| | 454 | <PRE> |
| | 455 | lttctl -C -w /tmp/trace3 -o channel.all.overwrite=1 trace3 |
| | 456 | ... |
| | 457 | lttctl -D -w /tmp/trace3 trace3 |
| | 458 | </PRE> |
| | 459 | |
| | 460 | ************************************************************** |
| | 461 | ** Section 4 * Adding new instrumentations with the markers ** |
| | 462 | ************************************************************** |
| | 463 | |
| | 464 | See Documentation/markers.txt and Documentation/tracepoints.txt in your kernel |
| | 465 | tree. |
| | 466 | |
| | 467 | * Add new events to userspace programs with userspace markers |
| | 468 | http://ltt.polymtl.ca/packages/ |
| | 469 | |
| | 470 | Get the latest markers-userspace-*.tar.bz2 and see the Makefile and examples. It |
| | 471 | allows inserting markers in executables and libraries, currently only on x86_32 |
| | 472 | and x86_64. |
| | 473 | |
| | 474 | *********************************************************** |
| | 475 | ** Section 5 * Creating Debian or RPM packages ** |
| | 476 | *********************************************************** |
| | 477 | |
| | 478 | * Create custom LTTV Debian packages |
| | 479 | |
| | 480 | Use : dpkg-buildpackage -rfakeroot |
| | 481 | |
| | 482 | You should then have your LTTV .deb files created for your architecture. |
| | 483 | |
| | 484 | * Create custom LTTng packages |
| | 485 | |
| | 486 | For building LTTng Debian packages : |
| | 487 | |
| | 488 | Get the build tree with patches applies as explained in section 2. |
| | 489 | |
| | 490 | make menuconfig (or xconfig or config) (customize your configuration) |
| | 491 | make-kpkg kernel_image |
| | 492 | |
| | 493 | You will then see your freshly created .deb in /usr/src. Install it with |
| | 494 | dpkg -i /usr/src/(image-name).deb |
| | 495 | |
| | 496 | Then, follow the section "Editing the system wide configuration" in section 2. |
| | 497 | |
| | 498 | |
| | 499 | |
| | 500 | </body> |
| | 501 | </html> |
projects / lttv.git / blame_incremental