| 1 | Linux Trace Toolkit Quickstart |
| 2 | ------------------------------ |
| 3 | Author : Mathieu Desnoyers, September 2005 |
| 4 | Last update : May 14, 2007 |
| 5 | |
| 6 | |
| 7 | This document is made of four parts : the first one explains how to install |
| 8 | LTTng and LTTV from Debian and RPM binary packages, the second one explains how |
| 9 | to install LTTng and LTTV from sources and the third one describes the steps |
| 10 | to follow to trace a system and view it. The fourth and last part explains |
| 11 | briefly how to add a new trace point to the kernel and to user space |
| 12 | applications. |
| 13 | |
| 14 | What you will typically want is to read sections 2 and 3 : install LTTng from |
| 15 | sources and use it. |
| 16 | |
| 17 | These operations are made for installing the LTTng 0.6.X tracer on a |
| 18 | linux 2.6.X kernel. You will also find instructions for installation of |
| 19 | LTTV 0.8.x : the Linux Trace Toolkit Viewer. |
| 20 | |
| 21 | To see the list of compatibilities between LTTng, ltt-control, LTTV, genevent |
| 22 | and ltt-usertrace, please refer to : |
| 23 | http://ltt.polymtl.ca > LTTng+LTTV versions compatibility |
| 24 | |
| 25 | |
| 26 | |
| 27 | The following lttng patch is necessary to have the tracing hooks in the kernel. |
| 28 | The following ltt-control module controls the tracing. |
| 29 | |
| 30 | Required programs and libraries are assumed to be automatically installed in an |
| 31 | installation with Debian or RPM packages. In the case of an installation from |
| 32 | sources, the dependencies are listed. |
| 33 | |
| 34 | |
| 35 | ** Current development status ** |
| 36 | |
| 37 | LTTng : |
| 38 | supported architectures : |
| 39 | Intel Pentium (UP/SMP) with TSC |
| 40 | PowerPC 32 and 64 bits |
| 41 | ARM |
| 42 | x86_64 |
| 43 | C2 Microsystems (variant of MIPS) |
| 44 | |
| 45 | LTTV : |
| 46 | supported architectures : |
| 47 | Intel i386 and better |
| 48 | Intel 64 bits |
| 49 | PowerPC 32 and 64 bits |
| 50 | |
| 51 | |
| 52 | *********************************************************** |
| 53 | ** Section 1 * Installation from Debian or RPM packages ** |
| 54 | *********************************************************** |
| 55 | |
| 56 | ** NOTE : RPM and Debian packages are only made once a version has been |
| 57 | thoroughly tested. If they do not exist at the moment, please install from |
| 58 | sources (see section 2 below). To see the list of compatibilities between |
| 59 | LTTng, ltt-control, LTTV, genevent and lttng-modules, please refer to |
| 60 | http://ltt.polymtl.ca > LTTng+LTTV versions compatibility |
| 61 | |
| 62 | |
| 63 | * Install from RPM packages on Fedora Core 4 : |
| 64 | |
| 65 | Get LTTV RPM from : |
| 66 | |
| 67 | http://ltt.polymtl.ca/packages/fedora/RPMS |
| 68 | |
| 69 | LTTV RPM are ready. |
| 70 | |
| 71 | LTTng kernel and lttng-modules RPM are available for some architectures (i586, |
| 72 | i686). Feel free to help fix the spec files to have correct lttng-modules RPM |
| 73 | package. |
| 74 | |
| 75 | |
| 76 | * Install from .deb packages on Debian : |
| 77 | |
| 78 | You can use the ltt.polymtl.ca apt source to get LTTV for Debian : |
| 79 | |
| 80 | Add the following two sources to your /etc/apt/sources.list : |
| 81 | |
| 82 | deb http://ltt.polymtl.ca/packages/debian experimental main |
| 83 | deb-src http://ltt.polymtl.ca/packages/debian experimental main |
| 84 | |
| 85 | |
| 86 | * Install from precompiled binary packages (LTTV compiled only for i386, and |
| 87 | LTTng only for i686 smp), perform the following : |
| 88 | |
| 89 | su - |
| 90 | apt-get update |
| 91 | apt-get install lttv lttv-doc |
| 92 | apt-get install kernel-image-2.6.12-rc4-mm2-lttng-0.4.2 |
| 93 | apt-get install lttng-modules-modules-2.6.12-rc4-mm2-lttng-0.4.2 |
| 94 | * note : the packages are signed by myself. I am not considered a trusted |
| 95 | Debian source yet, so warnings are normal. |
| 96 | |
| 97 | Then, follow the section "Editing the system wide configuration" in section 2. |
| 98 | |
| 99 | * Create custom LTTV Debian packages |
| 100 | |
| 101 | Binary packages are only available for i386. If you want to create your own LTTV |
| 102 | packages for other platforms, do : |
| 103 | |
| 104 | su - |
| 105 | cd /usr/src |
| 106 | apt-get source lttv |
| 107 | cd lttv-0.6.9 |
| 108 | dpkg-buildpackage -rfakeroot |
| 109 | |
| 110 | You should then have your LTTV .deb files created for your architecture. |
| 111 | |
| 112 | * Create custom LTTng packages |
| 113 | |
| 114 | For building LTTng Debian packages : |
| 115 | |
| 116 | su - |
| 117 | apt-get install kernel-source-2.6.12-rc4-mm2-lttng-0.4.2 |
| 118 | cd /usr/src |
| 119 | bzip2 -cd kernel-source-2.6.12-rc4-mm2-lttng-0.4.2.tar.bz2 | tar xvof - |
| 120 | cd kernel-source-2.6.12-rc4-mm2-lttng-0.4.2 |
| 121 | make menuconfig (or xconfig or config) (customize your configuration) |
| 122 | make-kpkg kernel_image |
| 123 | |
| 124 | You will then see your freshly created .deb in /usr/src. Install it with |
| 125 | dpkg -i /usr/src/(image-name).deb |
| 126 | |
| 127 | You will also need to create a package for the lttng-modules : |
| 128 | |
| 129 | su - |
| 130 | cd /usr/src |
| 131 | apt-get source lttng-modules |
| 132 | cd kernel-source-2.6.12-rc4-mm2-lttng-0.4.2 |
| 133 | make-kpkg --added_modules /usr/src/lttng-modules-0.3 modules_image |
| 134 | |
| 135 | You will then see your freshly created .deb in /usr/src. Install it with |
| 136 | dpkg -i /usr/src/lttng-modules-modules-(your version).deb |
| 137 | |
| 138 | |
| 139 | Then, follow the section "Editing the system wide configuration" in section 2. |
| 140 | |
| 141 | |
| 142 | *********************************************************** |
| 143 | ** Section 2 * Installation from sources ** |
| 144 | *********************************************************** |
| 145 | |
| 146 | * Prerequisites |
| 147 | |
| 148 | Tools needed to follow the package download steps : |
| 149 | |
| 150 | o wget |
| 151 | o bzip2 |
| 152 | o gzip |
| 153 | o tar |
| 154 | |
| 155 | You have to install the standard development libraries and programs necessary |
| 156 | to compile a kernel : |
| 157 | |
| 158 | (from Documentation/Changes in the Linux kernel tree) |
| 159 | o Gnu C 2.95.3 # gcc --version |
| 160 | o Gnu make 3.79.1 # make --version |
| 161 | o binutils 2.12 # ld -v |
| 162 | o util-linux 2.10o # fdformat --version |
| 163 | o module-init-tools 0.9.10 # depmod -V |
| 164 | |
| 165 | You might also want to have libncurses5 to have the text mode kernel |
| 166 | configuration menu, but there are alternatives. |
| 167 | |
| 168 | Prerequisites for LTTV 0.x.x installation are : |
| 169 | |
| 170 | gcc 3.2 or better |
| 171 | gtk 2.4 or better development libraries |
| 172 | (Debian : libgtk2.0, libgtk2.0-dev) |
| 173 | (Fedora : gtk2, gtk2-devel) |
| 174 | note : For Fedora users : this might require at least core 3 from Fedora, |
| 175 | or you might have to compile your own GTK2 library. |
| 176 | glib 2.4 or better development libraries |
| 177 | (Debian : libglib2.0-0, libglib2.0-dev) |
| 178 | (Fedora : glib2, glib2-devel) |
| 179 | libpopt development libraries |
| 180 | (Debian : libpopt0, libpopt-dev) |
| 181 | (Fedora : popt) |
| 182 | libpango development libraries |
| 183 | (Debian : libpango1.0, libpango1.0-dev) |
| 184 | (Fedora : pango, pango-devel) |
| 185 | libc6 development librairies |
| 186 | (Debian : libc6, libc6-dev) |
| 187 | (Fedora : glibc, glibc) |
| 188 | |
| 189 | |
| 190 | * Getting the LTTng packages |
| 191 | |
| 192 | su - |
| 193 | mkdir /usr/src/lttng |
| 194 | cd /usr/src/lttng |
| 195 | (see http://ltt.polymtl.ca/lttng for package listing) |
| 196 | wget http://ltt.polymtl.ca/lttng/patch-2.6.X-lttng-0.x.xx.tar.bz2 |
| 197 | bzip2 -cd patch-2.6.X-lttng-0.x.xx.tar.bz2 | tar xvof - |
| 198 | |
| 199 | |
| 200 | * Getting LTTng kernel sources |
| 201 | |
| 202 | su - |
| 203 | cd /usr/src |
| 204 | wget http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.X.tar.bz2 |
| 205 | bzip2 -cd linux-2.6.X.tar.bz2 | tar xvof - |
| 206 | cd linux-2.6.X |
| 207 | - For LTTng 0.9.4- cat /usr/src/lttng/patch*-2.6.X-lttng-0.x.xx* | patch -p1 |
| 208 | - For LTTng 0.9.5+ apply the patches in the order specified in the series file, |
| 209 | or use quilt |
| 210 | cd .. |
| 211 | mv linux-2.6.X linux-2.6.X-lttng-0.x.xx |
| 212 | |
| 213 | |
| 214 | * Installing a LTTng kernel |
| 215 | |
| 216 | su - |
| 217 | cd /usr/src/linux-2.6.X-lttng-0.x.xx |
| 218 | make menuconfig (or make xconfig or make config) |
| 219 | Select the < Help > button if you are not familiar with kernel |
| 220 | configuration. |
| 221 | Items preceded by [*] means they has to be built into the kernel. |
| 222 | Items preceded by [M] means they has to be built as modules. |
| 223 | Items preceded by [ ] means they should be removed. |
| 224 | go to the "Instrumentation Support" section |
| 225 | Select the following options : |
| 226 | [*] Linux Trace Toolkit Instrumentation Support |
| 227 | <M> or <*> Linux Trace Toolkit Tracer |
| 228 | <M> or <*> Linux Trace Toolkit Relay+DebugFS Support |
| 229 | It makes no difference for the rest of the procedure whether the Tracer |
| 230 | is compiled built-in or as a module. |
| 231 | activate : |
| 232 | [*] Align Linux Trace Toolkit Traces |
| 233 | [*] Allow tracing from userspace |
| 234 | <M> Linux Trace Toolkit Netlink Controller |
| 235 | <M> Linux Trace Toolkit State Dump |
| 236 | your choice (see < Help >) : |
| 237 | [ ] Activate Linux Trace Toolkit Heartbeat Timer |
| 238 | You may or may not decide to compile probes. Afterward, you will have to |
| 239 | load the probe modules to enable tracing of their events. The probes |
| 240 | automatically select the appropriate facilities. |
| 241 | Static instrumentation is a more invasive type of instrumentation that gives |
| 242 | the address taking a lock or doing a printk. |
| 243 | Select <Exit> |
| 244 | Select <Exit> |
| 245 | Select <Yes> |
| 246 | make |
| 247 | make modules_install |
| 248 | (if necessary, create a initrd with mkinitrd or your preferate alternative) |
| 249 | (mkinitrd -o /boot/initrd.img-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx) |
| 250 | |
| 251 | -- on X86, X86_64 |
| 252 | make install |
| 253 | reboot |
| 254 | Select the Linux 2.6.17-lttng-0.x.xx kernel in your boot loader. |
| 255 | |
| 256 | -- on PowerPC |
| 257 | cp vmlinux.strip /boot/vmlinux-2.6.X-lttng-0.x.xx |
| 258 | cp System.map /boot/System.map-2.6.X-lttng-0.x.xx |
| 259 | cp .config /boot/config-2.6.X-lttng-0.x.xx |
| 260 | depmod -ae -F /boot/System.map-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx |
| 261 | mkinitrd /boot/initrd.img-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx |
| 262 | (edit /etc/yaboot.conf to add a new entry pointing to your kernel : the entry |
| 263 | that comes first is the default kernel) |
| 264 | ybin |
| 265 | select the right entry at the yaboot prompt (see choices : tab, select : type |
| 266 | the kernel name followed by enter) |
| 267 | Select the Linux 2.6.17-lttng-0.x.xx kernel in your boot loader. |
| 268 | -- |
| 269 | |
| 270 | |
| 271 | |
| 272 | * Editing the system wide configuration |
| 273 | |
| 274 | You must activate debugfs and specify a mount point. This is typically done in |
| 275 | fstab such that it happens at boot time. |
| 276 | |
| 277 | If you have never used DebugFS before, these operation would do this for you : |
| 278 | |
| 279 | mkdir /mnt/debugfs |
| 280 | cp /etc/fstab /etc/fstab.lttng.bkp |
| 281 | echo "debugfs /mnt/debugfs debugfs rw 0 0" >> /etc/fstab |
| 282 | |
| 283 | then, rebooting or issuing the following command will activate debugfs : |
| 284 | |
| 285 | mount /mnt/debugfs |
| 286 | |
| 287 | You need to load the LTT modules to be able to control tracing from user |
| 288 | space. This is done by issuing the following commands. Note however |
| 289 | these commands load all LTT modules. Depending on what options you chose to |
| 290 | compile statically, you may not need to issue all these commands. |
| 291 | |
| 292 | modprobe ltt-control |
| 293 | modprobe ltt-core |
| 294 | modprobe ltt-relay |
| 295 | modprobe ltt-tracer |
| 296 | modprobe ltt-probe-mm |
| 297 | modprobe ltt-probe-kernel |
| 298 | modprobe ltt-probe-i386 (or x86_64, powerpc, ppc, arm, mips) |
| 299 | modprobe ltt-probe-net |
| 300 | modprobe ltt-probe-list |
| 301 | modprobe ltt-probe-ipc |
| 302 | modprobe ltt-probe-fs |
| 303 | |
| 304 | If you want to have complete information about the kernel state (including all |
| 305 | the process names), you need to load the ltt-statedump module. This is done by |
| 306 | issuing the command : |
| 307 | |
| 308 | modprobe ltt-statedump |
| 309 | |
| 310 | You can automate at boot time loading the ltt-control module by : |
| 311 | |
| 312 | cp /etc/modules /etc/modules.bkp |
| 313 | echo ltt-control >> /etc/modules |
| 314 | echo ltt-core >> /etc/modules |
| 315 | echo ltt-relay >> /etc/modules |
| 316 | echo ltt-tracer >> /etc/modules |
| 317 | echo ltt-probe-mm >> /etc/modules |
| 318 | echo ltt-probe-kernel >> /etc/modules |
| 319 | echo ltt-probe-i386 >> /etc/modules (or x86_64, powerpc, ppc, arm, mips) |
| 320 | echo ltt-probe-net >> /etc/modules |
| 321 | echo ltt-probe-list >> /etc/modules |
| 322 | echo ltt-probe-ipc >> /etc/modules |
| 323 | echo ltt-probe-fs >> /etc/modules |
| 324 | echo ltt-statedump >> /etc/modules |
| 325 | |
| 326 | (note : if you want to probe a marker which is within a module, make sure you |
| 327 | load the probe _after_ the module, otherwise the probe will not be able to |
| 328 | connect itself to the marker.) |
| 329 | |
| 330 | |
| 331 | * Getting and installing the ltt-control package (on the traced machine) |
| 332 | (note : the ltt-control package contains lttd and lttctl. Although it has the |
| 333 | same name as the ltt-control kernel module, they are *not* the same thing.) |
| 334 | su - |
| 335 | cd /usr/src |
| 336 | wget http://ltt.polymtl.ca/lttng/ltt-control-0.x-xxxx2006.tar.gz |
| 337 | gzip -cd ltt-control-0.x-xxxx2006.tar.gz | tar xvof - |
| 338 | cd ltt-control-0.x-xxxx2006 |
| 339 | (refer to README to see the development libraries that must be installed on you |
| 340 | system) |
| 341 | ./configure |
| 342 | make |
| 343 | make install |
| 344 | |
| 345 | * Getting and installing the ltt-usertrace package for user space tracing |
| 346 | See http://ltt.polymtl.ca/ > USERSPACE TRACING QUICKSTART |
| 347 | |
| 348 | |
| 349 | * Getting and installing the LTTV package (on the visualisation machine, same or |
| 350 | different from the visualisation machine) |
| 351 | |
| 352 | su - |
| 353 | cd /usr/src |
| 354 | wget http://ltt.polymtl.ca/packages/LinuxTraceToolkitViewer-0.x.xx-xxxx2006.tar.gz |
| 355 | gzip -cd LinuxTraceToolkitViewer-0.x.xx-xxxx2006.tar.gz | tar xvof - |
| 356 | cd LinuxTraceToolkitViewer-0.x.xx-xxxx2006 |
| 357 | (refer to README to see the development libraries that must be installed on your |
| 358 | system) |
| 359 | ./configure |
| 360 | make |
| 361 | make install |
| 362 | |
| 363 | |
| 364 | |
| 365 | |
| 366 | *********************************************************** |
| 367 | ** Section 3 * Using LTTng and LTTV ** |
| 368 | *********************************************************** |
| 369 | |
| 370 | * Arm Linux Kernel Markers after each boot |
| 371 | |
| 372 | ltt-armall |
| 373 | |
| 374 | * Use graphical LTTV to control tracing and analyse traces |
| 375 | |
| 376 | lttv-gui (or /usr/local/bin/lttv-gui) |
| 377 | - Spot the "Tracing Control" icon : click on it |
| 378 | (it's a traffic light icon) |
| 379 | - enter the root password |
| 380 | - click "start" |
| 381 | - click "stop" |
| 382 | - Yes |
| 383 | * You should now see a trace |
| 384 | |
| 385 | * Use text mode LTTng to control tracing |
| 386 | |
| 387 | The tracing can be controlled from a terminal by using the lttctl command (as |
| 388 | root). |
| 389 | |
| 390 | Start tracing : |
| 391 | |
| 392 | lttctl -n trace -d -l /mnt/debugfs/ltt -t /tmp/trace |
| 393 | |
| 394 | Stop tracing and destroy trace channels : |
| 395 | |
| 396 | lttctl -n trace -R |
| 397 | |
| 398 | see lttctl --help for details. |
| 399 | |
| 400 | (note : to see if the buffers has been filled, look at the dmesg output after |
| 401 | lttctl -R or after stopping tracing from the GUI, it will show an event lost |
| 402 | count. If it is the case, try using larger buffers. See lttctl --help to learn |
| 403 | how.) |
| 404 | |
| 405 | * Use text mode LTTV |
| 406 | |
| 407 | Feel free to look in /usr/local/lib/lttv/plugins to see all the text and |
| 408 | graphical plugins available. |
| 409 | |
| 410 | For example, a simple trace dump in text format is available with : |
| 411 | |
| 412 | lttv -m textDump -t /tmp/trace |
| 413 | |
| 414 | see lttv -m textDump --help for detailed command line options of textDump. |
| 415 | |
| 416 | It is, in the current state of the project, very useful to use "grep" on the |
| 417 | text output to filter by specific event fields. You can later copy the timestamp |
| 418 | of the events to the clipboard and paste them in the GUI by clicking on the |
| 419 | bottom right label "Current time". Support for this type of filtering should |
| 420 | be added to the filter module soon. |
| 421 | |
| 422 | |
| 423 | *********************************************************** |
| 424 | ** Section 4 * Adding new instrumentations with genevent ** |
| 425 | *********************************************************** |
| 426 | |
| 427 | * Getting and installing genevent |
| 428 | |
| 429 | su - |
| 430 | cd /usr/src |
| 431 | wget http://ltt.polymtl.ca/packages/genevent-0.xx.tar.gz |
| 432 | gzip -cd genevent-0.xx.tar.gz | tar xvof - |
| 433 | cd genevent-0.xx |
| 434 | make |
| 435 | make install |
| 436 | |
| 437 | |
| 438 | * Add new events to the kernel with genevent (deprecated in LTTng 0.9.x) |
| 439 | |
| 440 | su - |
| 441 | cd /usr/local/share/ltt-control/facilities |
| 442 | cp process.xml yourfacility.xml |
| 443 | * edit yourfacility.xml to fit your needs. |
| 444 | cd /tmp |
| 445 | /usr/local/bin/genevent /usr/local/share/ltt-control/facilities/yourfacility.xml |
| 446 | cp ltt-facility-yourfacility.h ltt-facility-id-yourfacility.h \ |
| 447 | /usr/src/linux-2.6.17-lttng-0.x.xx8/include/ltt |
| 448 | cp ltt-facility-loader-yourfacility.c ltt-facility-loader-yourfacility.h \ |
| 449 | /usr/src/linux-2.6.17-lttng-0.x.xx/ltt/facilities |
| 450 | * edit the kernel file you want to instrument to add a marker to it. See |
| 451 | include/linux/marker.h. |
| 452 | * create a dynamically loadable probe. See ltt/probes for examples. The probe |
| 453 | will be connected to your marker and will typically call the logging |
| 454 | functions found in the header file you created with genevent. |
| 455 | |
| 456 | * Add new kernel events |
| 457 | |
| 458 | *Important* note : in its current state, LTTng and LTTV needs the programmer |
| 459 | to keep the marker/probe format string and the XML description of the |
| 460 | event data types in sync by hand. Failure to do so will result in errors in |
| 461 | LTTV. |
| 462 | |
| 463 | See the markers documentation to see how to describe the marker. You will need |
| 464 | to clone probe modules found in ltt/probes to connect them to the markers so |
| 465 | that the information can be recorded in the trace. |
| 466 | |
| 467 | * Add new events to userspace programs with genevent |
| 468 | See http://ltt.polymtl.ca/ > USERSPACE TRACING QUICKSTART |
| 469 | |
| 470 | User-space tracing still uses genevent, which is subject to change in a near |
| 471 | future. |