From: Philippe Proulx Date: Wed, 28 May 2014 18:22:58 +0000 (-0400) Subject: Initial import X-Git-Url: https://git.lttng.org./?a=commitdiff_plain;h=5e0cbfb01373c18e521217342fd8a9159cc186b1;p=lttng-docs.git Initial import Signed-off-by: Philippe Proulx --- 5e0cbfb01373c18e521217342fd8a9159cc186b1 diff --git a/README.md b/README.md new file mode 100644 index 0000000..4286e5e --- /dev/null +++ b/README.md @@ -0,0 +1,13 @@ +LTTng Documentation +=================== + +This is the official repository of the +[online LTTng Documentation's](http://lttng.org/docs) source. + +To contribute: + + 1. Fork this repo + 2. Make your change, following the [contributor's guide](contrib-guide.md) + 3. Create a pull request + +Thanks for your contributions and fixes! diff --git a/contents/getting-started/intro.md b/contents/getting-started/intro.md new file mode 100644 index 0000000..82d31de --- /dev/null +++ b/contents/getting-started/intro.md @@ -0,0 +1,18 @@ +--- +id: getting-started +--- + +This is a small guide to get started quickly with LTTng kernel and user +space tracing. For intermediate to advanced use cases and a more +thorough understanding of LTTng, see [Using LTTng](#doc-using-lttng) and +[Understanding LTTng](#doc-understanding-lttng). + +Before reading this guide, make sure LTTng +[is installed](#doc-installing-lttng). You will at least need +LTTng-tools. Also install LTTng-modules for +[tracing the Linux kernel](#doc-tracing-the-linux-kernel) and LTTng-UST +for +[tracing your own user space applications](#doc-tracing-your-own-user-application). +When your traces are finally written and complete, the +[Viewing and analyzing your traces](#doc-viewing-and-analyzing-your-traces) +section of this chapter will help you do something useful with them. diff --git a/contents/getting-started/tracing-the-linux-kernel.md b/contents/getting-started/tracing-the-linux-kernel.md new file mode 100644 index 0000000..5f62d05 --- /dev/null +++ b/contents/getting-started/tracing-the-linux-kernel.md @@ -0,0 +1,68 @@ +--- +id: tracing-the-linux-kernel +--- + +Make sure LTTng-tools and LTTng-modules packages +[are installed](#doc-installing-lttng). + +Since you're about to trace the Linux kernel itself, let's look at the +available kernel events using the `lttng` tool, which has a +Git like command line structure: + +
+lttng list --kernel
+
+ +
+

+ Tip:You can avoid using sudo in + the previous and following commands if your user is part of the + tracing group. +

+
+ +Before tracing, you need to create a session: + +
+sudo lttng create my-session
+
+ +`my-session` is the tracing session name and could be anything you +like. `auto` will be used if omitted. + +Let's now enable some events for this session: + +
+sudo lttng enable-event --kernel sched_switch,sched_process_fork
+
+ +or you might want to simply enable all available kernel events (beware +that trace files will grow rapidly when doing this): + +
+sudo lttng enable-event --kernel --all
+
+ +Start tracing: + +
+sudo lttng start
+
+ +By default, traces are saved in +~/lttng-traces/name-date-time, +where name is the session name. + +When you're done tracing: + +
+sudo lttng stop
+sudo lttng destroy
+
+ +Although `destroy` looks scary here, it doesn't actually destroy the +outputted traces: it only destroys the tracing session. + +What's next? Have a look at +[Viewing and analyzing your traces](#doc-viewing-and-analyzing-your-traces) +to view and analyze the trace you just recorded. diff --git a/contents/getting-started/tracing-your-own-user-application.md b/contents/getting-started/tracing-your-own-user-application.md new file mode 100644 index 0000000..064e3c2 --- /dev/null +++ b/contents/getting-started/tracing-your-own-user-application.md @@ -0,0 +1,215 @@ +--- +id: tracing-your-own-user-application +--- + +The previous section helped you create a trace out of Linux kernel events. +This section steps you through a simple example showing you how to trace +a _Hello world_ program written in C. + +Make sure LTTng-tools and LTTng-UST packages +[are installed](#doc-installing-lttng). + +Tracing is just like having `printf()` calls at specific locations of +your source code, albeit LTTng is much more faster and flexible than +`printf()`. In the LTTng realm, **`tracepoint()`** is analogous to +`printf()`. + +Unlike `printf()`, though, `tracepoint()` does not use a format string to +know the types of its arguments: the formats of all tracepoints must be +defined before using them. So before even writing our _Hello world_ program, +we need to define the format of our tracepoint. This is done by writing a +**template file**, with a name usually ending with the `.tp` extension, +which the `lttng-gen-tp` tool (shipped with LTTng-UST) will use to generate +an object file and a header to be included in our application source code. + +Here's the whole flow: + +
+ + + +
+ +The template file format is a list of tracepoint definitions +and other optional definition entries which we will skip for +this quickstart. Each tracepoint is defined using the +`TRACEPOINT_EVENT()` macro. For each tracepoint, you must provide: + + * a **provider name**, which is the "scope" of this tracepoint (this usually + includes the company and project names) + * a **tracepoint name** + * a **list of arguments** for the eventual `tracepoint()` call, each item being: + * the argument C type + * the argument name + * a **list of fields**, which will be the actual fields of the recorded events + for this tracepoint + +Here's a simple tracepoint definition example with two arguments: an integer +and a string: + +~~~ c +TRACEPOINT_EVENT( + hello_world, + my_first_tracepoint, + TP_ARGS( + int, my_integer_arg, + char*, my_string_arg + ), + TP_FIELDS( + ctf_string(my_string_field, my_string_arg) + ctf_integer(int, my_integer_field, my_integer_arg) + ) +) +~~~ + +The exact syntax is well explained in the +[C application](#doc-c-application) instrumenting guide of the +[Using LTTng](#doc-using-lttng) chapter, as well as in the +LTTng-UST manpage. + +Save the above snippet as `hello-tp.tp` and run: + +
+lttng-gen-tp hello-tp.tp
+
+ +The following files will be created next to `hello-tp.tp`: + + * `hello-tp.c` + * `hello-tp.o` + * `hello-tp.h` + +`hello-tp.o` is the compiled object file of `hello-tp.c`. + +Now, by including `hello-tp.h` in your own application, you may use the +tracepoint defined above by properly refering to it when calling +`tracepoint()`: + +~~~ c +#include +#include "hello-tp.h" + +int main(int argc, char* argv[]) +{ + int x; + + puts("Hello, World!\nPress Enter to continue..."); + + /* The following getchar() call is only placed here for the purpose + * of this demonstration, for pausing the application in order for + * you to have time to list its events. It's not needed otherwise. + */ + getchar(); + + /* A tracepoint() call. Arguments, as defined in hello-tp.tp: + * + * 1st: provider name (always) + * 2nd: tracepoint name (always) + * 3rd: my_integer_arg (first user-defined argument) + * 4th: my_string_arg (second user-defined argument) + * + * Notice the provider and tracepoint names are NOT strings; + * they are in fact parts of variables created by macros in + * hello-tp.h. + */ + tracepoint(hello_world, my_first_tracepoint, 23, "hi there!"); + + for (x = 0; x < argc; ++x) { + tracepoint(hello_world, my_first_tracepoint, x, argv[x]); + } + + puts("Quitting now!"); + + tracepoint(hello_world, my_first_tracepoint, x * x, "x^2"); + + return 0; +} +~~~ + +Save this as `hello.c`, next to `hello-tp.tp`. + +Notice `hello-tp.h`, the header file generated by `lttng-gen-tp` from +our template file `hello-tp.tp`, is included by `hello.c`. + +You are now ready to compile the application with LTTng-UST support: + +
+gcc -o hello hello.c hello-tp.o -llttng-ust -ldl
+
+ +If you followed the +[Tracing the Linux kernel](#doc-tracing-the-linux-kernel) section, the +following steps will look familiar. + +First, run the application with a few arguments: + +
+./hello world and beyond
+
+ +You should see + +~~~ text +Hello, World! +Press Enter to continue... +~~~ + +Use the `lttng` tool to list all available user space events: + +
+lttng list --userspace
+
+ +You should see the `hello_world:my_first_tracepoint` tracepoint listed +under the `./hello` process. + +Create a tracing session: + +
+lttng create my-userspace-session
+
+ +Enable the `hello_world:my_first_tracepoint` tracepoint: + +
+lttng enable-event --userspace hello_world:my_first_tracepoint
+
+ +Start tracing: + +
+lttng start
+
+ +Go back to the running `hello` application and press Enter. All `tracepoint()` +calls will be executed and the program will finally exit. + +Stop tracing and destroy the tracing session: + +
+lttng stop
+lttng destroy my-userspace-session
+
+ +Done! You may use `lttng view` to list the recorded events. This command +starts +babeltrace +in the background, if it is installed: + +
+lttng view
+
+ +should output something like: + +~~~ text +[18:10:27.684304496] (+?.?????????) hostname hello_world:my_first_tracepoint: { cpu_id = 0 }, { my_string_field = "hi there!", my_integer_field = 23 } +[18:10:27.684338440] (+0.000033944) hostname hello_world:my_first_tracepoint: { cpu_id = 0 }, { my_string_field = "./hello", my_integer_field = 0 } +[18:10:27.684340692] (+0.000002252) hostname hello_world:my_first_tracepoint: { cpu_id = 0 }, { my_string_field = "world", my_integer_field = 1 } +[18:10:27.684342616] (+0.000001924) hostname hello_world:my_first_tracepoint: { cpu_id = 0 }, { my_string_field = "and", my_integer_field = 2 } +[18:10:27.684343518] (+0.000000902) hostname hello_world:my_first_tracepoint: { cpu_id = 0 }, { my_string_field = "beyond", my_integer_field = 3 } +[18:10:27.684357978] (+0.000014460) hostname hello_world:my_first_tracepoint: { cpu_id = 0 }, { my_string_field = "x^2", my_integer_field = 16 } +~~~ + +The next section presents other alternatives to view and analyze your +LTTng traces. diff --git a/contents/getting-started/viewing-and-analyzing.md b/contents/getting-started/viewing-and-analyzing.md new file mode 100644 index 0000000..501c77e --- /dev/null +++ b/contents/getting-started/viewing-and-analyzing.md @@ -0,0 +1,160 @@ +--- +id: viewing-and-analyzing-your-traces +--- + +This section describes how to visualize the data gathered after tracing +the Linux kernel or a user space application. + +Many ways exist to read your LTTng traces: + + * **`babeltrace`** is a command line utility which converts trace formats; + it supports the format used by LTTng, + CTF, as well as a basic + text output which may be `grep`ed. The `babeltrace` command is + part of the + Babeltrace project. + * Babeltrace also includes a **Python binding** so that you may + easily open and read an LTTng trace with your own script, benefiting + from the power of Python. + * The ** + Eclise IDE for C/C++ Developers** + includes the Tracing and Monitoring Framework (TMF) plugin which + supports LTTng traces, amongst others. + +LTTng trace files are usually recorded in the `~/lttng-traces` directory. +Let's now view the trace and perform a basic analysis using +`babeltrace`. + +The simplest way to list all the recorded events of a trace is to pass its +path to `babeltrace` with no options: + +
+babeltrace ~/lttng-traces/my-session
+
+ +`babeltrace` will find all traces within the given path recursively and +output all their events, merging them intelligently. + +Listing all the system calls of a Linux kernel trace with their arguments is +easy with `babeltrace` and `grep`: + +
+babeltrace ~/lttng-traces/my-kernel-session | grep sys_
+
+ +Counting events is also straightforward: + +
+babeltrace ~/lttng-traces/my-kernel-session | grep sys_read | wc -l
+
+ +The text output of `babeltrace` is useful for isolating events by simple +matching using `grep` and similar utilities. However, more elaborate filters +such as keeping only events with a field value falling within a specific range +are not trivial to write using a shell. Moreover, reductions and even the +most basic computations involving multiple events are virtually impossible +to implement. + +Fortunately, Babeltrace ships with a Python 3 binding which makes it +really easy to read the events of an LTTng trace sequentially and compute +the desired information. + +Here's a simple example using the Babeltrace Python binding. The following +script accepts an LTTng Linux kernel trace path as its first argument and +outputs the short names of the top 5 running processes on CPU 0 during the +whole trace: + +~~~ python +import sys +from collections import Counter +import babeltrace + + +def top5proc(): + if len(sys.argv) != 2: + msg = 'Usage: python {} TRACEPATH'.format(sys.argv[0]) + raise ValueError(msg) + + # a trace collection holds one to many traces + col = babeltrace.TraceCollection() + + # add the trace provided by the user + # (LTTng traces always have the 'ctf' format) + if col.add_trace(sys.argv[1], 'ctf') is None: + raise RuntimeError('Cannot add trace') + + # this counter dict will hold execution times: + # + # task command name -> total execution time (ns) + exec_times = Counter() + + # this holds the last `sched_switch` timestamp + last_ts = None + + # iterate events + for event in col.events: + # keep only `sched_switch` events + if event.name != 'sched_switch': + continue + + # keep only events which happened on CPU 0 + if event['cpu_id'] != 0: + continue + + # event timestamp + cur_ts = event.timestamp + + if last_ts is None: + # we start here + last_ts = cur_ts + + # previous task command (short) name + prev_comm = event['prev_comm'] + + # initialize entry in our dict if not yet done + if prev_comm not in exec_times: + exec_times[prev_comm] = 0 + + # compute previous command execution time + diff = cur_ts - last_ts + + # update execution time of this command + exec_times[prev_comm] += diff + + # update last timestamp + last_ts = cur_ts + + # display top 10 + for name, ns in exec_times.most_common()[:5]: + s = ns / 1000000000 + print('{:20}{} s'.format(name, s)) + + +if __name__ == '__main__': + top5proc() +~~~ + +Save this script as `top5proc.py` and run it with Python 3, providing the +path to an LTTng Linux kernel trace as the first argument: + +
+python3 top5proc.py ~/lttng-sessions/my-session-.../kernel
+
+ +Make sure the path you provide is the directory containing actual trace +files (`channel0_0`, `metadata`, etc.): the `babeltrace` utility recurses +directories, but the Python binding does not. + +Here's an example of output: + +~~~ text +swapper/0 48.607245889 s +chromium 7.192738188 s +pavucontrol 0.709894415 s +Compositor 0.660867933 s +Xorg.bin 0.616753786 s +~~~ + +Note that `swapper/0` is the "idle" process of CPU 0 on Linux; since we +weren't using the CPU that much when tracing, its first position in the list +makes sense. diff --git a/contents/installing-lttng/building-from-source.md b/contents/installing-lttng/building-from-source.md new file mode 100644 index 0000000..b5d3c9e --- /dev/null +++ b/contents/installing-lttng/building-from-source.md @@ -0,0 +1,24 @@ +--- +id: building-from-source +--- + +As [previously stated](#doc-installing-lttng), LTTng is shipped as +three packages: LTTng-tools, LTTng-modules and LTTng-UST. LTTng-tools +contains everything needed to control tracing sessions, while +LTTng-modules is only needed for Linux kernel tracing and LTTng-UST is +only needed for user space tracing. + +The tarballs are available in the +Download +section of the LTTng website. + +Please refer to the `README.md` files provided by each package to +properly build and install them. + +
+

+Tip:The aforementioned README.md files +are rendered as rich text when +viewed on GitHub. +

+
diff --git a/contents/installing-lttng/desktop-distributions/archlinux.md b/contents/installing-lttng/desktop-distributions/archlinux.md new file mode 100644 index 0000000..c5e811e --- /dev/null +++ b/contents/installing-lttng/desktop-distributions/archlinux.md @@ -0,0 +1,24 @@ +--- +id: archlinux +--- + +LTTng packages are available in the +AUR: +lttng-tools, +lttng-modules +and +lttng-ust. + +You can automate all this using +Yaourt. + +
+yaourt -Sy lttng-tools lttng-modules lttng-ust
+
+ +If you're living on the edge, the AUR also contains the latest Git masters +of those packages: +lttng-tools-git, +lttng-modules-git +and +lttng-ust-git. diff --git a/contents/installing-lttng/desktop-distributions/debian.md b/contents/installing-lttng/desktop-distributions/debian.md new file mode 100644 index 0000000..cd820b6 --- /dev/null +++ b/contents/installing-lttng/desktop-distributions/debian.md @@ -0,0 +1,13 @@ +--- +id: debian +--- + +Debian wheezy (stable) and previous versions are not supported; you will +need to build and install LTTng packages +[from source](#doc-building-from-source) for those. + +Debian jessie (testing) and sid (unstable) have everything you need: + +
+sudo apt-get install lttng-tools lttng-modules-dkms liblttng-ust-dev
+
diff --git a/contents/installing-lttng/desktop-distributions/enterprise.md b/contents/installing-lttng/desktop-distributions/enterprise.md new file mode 100644 index 0000000..a1a2021 --- /dev/null +++ b/contents/installing-lttng/desktop-distributions/enterprise.md @@ -0,0 +1,7 @@ +--- +id: enterprise-distributions +--- + +To install LTTng on enterprise distributions, please see +EfficiOS +Enterprise Packages. diff --git a/contents/installing-lttng/desktop-distributions/fedora.md b/contents/installing-lttng/desktop-distributions/fedora.md new file mode 100644 index 0000000..b77f8e3 --- /dev/null +++ b/contents/installing-lttng/desktop-distributions/fedora.md @@ -0,0 +1,14 @@ +--- +id: fedora +--- + +Starting from Fedora 17, LTTng-tools and LTTng-UST packages are officially +available using `yum`: + +
+sudo yum install lttng-tools lttng-ust
+
+ +LTTng-modules still needs to be built and installed +[from source](#doc-building-from-source). + diff --git a/contents/installing-lttng/desktop-distributions/intro.md b/contents/installing-lttng/desktop-distributions/intro.md new file mode 100644 index 0000000..7636e58 --- /dev/null +++ b/contents/installing-lttng/desktop-distributions/intro.md @@ -0,0 +1,15 @@ +--- +id: desktop-distributions +--- + +Official and unofficial LTTng packages are available for the major +Linux desktop distributions: [Ubuntu](#doc-ubuntu), +[Fedora](#doc-fedora), [Debian](#doc-debian), [openSUSE](#doc-opensuse) +(and other RPM-based distributions) and [Arch Linux](#doc-archlinux). +LTTng is regularly tested on those. Should any issue arise when +following the procedures below, please inform the +community about it. + +Support is also available for +[enterprise distributions](#doc-enterprise-distributions) such as +Red Hat Enterprise Linux (RHEL) and SUSE Linux Enterprise Server (SLES). diff --git a/contents/installing-lttng/desktop-distributions/opensuse.md b/contents/installing-lttng/desktop-distributions/opensuse.md new file mode 100644 index 0000000..eac0600 --- /dev/null +++ b/contents/installing-lttng/desktop-distributions/opensuse.md @@ -0,0 +1,27 @@ +--- +id: opensuse +--- + +openSUSE has LTTng packages since version 12.3. To install LTTng, you +first need to add an entry to your repositories. All LTTng repositories +are available +here. +For example, the following will add the LTTng repository for +openSUSE 13.1: + +
+sudo -i
+curl http://download.opensuse.org/repositories/devel:/tools:/lttng/openSUSE_13.1/devel:tools:lttng.repo > /etc/zypp/repos.d/lttng.repo
+
+ +Then, refresh the package database: + +
+sudo zypper refresh
+
+ +and install `lttng-tools`, `lttng-modules` and `lttng-ust-devel`: + +
+sudo zypper install lttng-tools lttng-modules lttng-ust-devel
+
diff --git a/contents/installing-lttng/desktop-distributions/ubuntu/intro.md b/contents/installing-lttng/desktop-distributions/ubuntu/intro.md new file mode 100644 index 0000000..cb3784a --- /dev/null +++ b/contents/installing-lttng/desktop-distributions/ubuntu/intro.md @@ -0,0 +1,11 @@ +--- +id: ubuntu +--- + +The following steps apply to Ubuntu ≥ 12.04. For +previous releases, you will need to build and install LTTng +[from source](#doc-building-from-source), as no Ubuntu packages were +available before version 12.04. + +Two sources of LTTng packages are available for Ubuntu: official +repositories and PPA. diff --git a/contents/installing-lttng/desktop-distributions/ubuntu/official-repositories.md b/contents/installing-lttng/desktop-distributions/ubuntu/official-repositories.md new file mode 100644 index 0000000..5f38712 --- /dev/null +++ b/contents/installing-lttng/desktop-distributions/ubuntu/official-repositories.md @@ -0,0 +1,13 @@ +--- +id: ubuntu-official-repositories +--- + +To install LTTng from the official Ubuntu repositories, simply use `apt-get`: + +
+sudo apt-get install lttng-tools lttng-modules-dkms liblttng-ust-dev
+
+ +No need to reboot! + + diff --git a/contents/installing-lttng/desktop-distributions/ubuntu/ppa.md b/contents/installing-lttng/desktop-distributions/ubuntu/ppa.md new file mode 100644 index 0000000..b820761 --- /dev/null +++ b/contents/installing-lttng/desktop-distributions/ubuntu/ppa.md @@ -0,0 +1,14 @@ +--- +id: ubuntu-ppa +--- + +The +LTTng PPA +offers the latest stable versions of LTTng packages. To get packages +from the PPA, follow these steps: + +
+sudo apt-add-repository ppa:lttng/ppa
+sudo apt-get update
+sudo apt-get install lttng-tools lttng-modules-dkms liblttng-ust-dev
+
diff --git a/contents/installing-lttng/embedded-distributions/buildroot.md b/contents/installing-lttng/embedded-distributions/buildroot.md new file mode 100644 index 0000000..5fe4776 --- /dev/null +++ b/contents/installing-lttng/embedded-distributions/buildroot.md @@ -0,0 +1,23 @@ +--- +id: buildroot +--- + +LTTng packages in Buildroot are `lttng-tools`, `lttng-modules` and +`lttng-libust`. + +To enable them, start the Buildroot configuration menu as usual: + +
+make menuconfig
+
+ +In: + + * _Kernel_: make sure _Linux kernel_ is enabled + * _Toolchain_: make sure the following options are enabled: + * _Enable large file (files > 2GB) support_ + * _Enable WCHAR support_ + +In _Target packages_/_Debugging, profiling and benchmark_, enable +_lttng-modules_ and _lttng-tools_. In +_Target packages_/_Libraries_/_Other_, enable _lttng-libust_. diff --git a/contents/installing-lttng/embedded-distributions/intro.md b/contents/installing-lttng/embedded-distributions/intro.md new file mode 100644 index 0000000..5ae7c3c --- /dev/null +++ b/contents/installing-lttng/embedded-distributions/intro.md @@ -0,0 +1,8 @@ +--- +id: embedded-distributions +--- + +Some developers may be interested in tracing the Linux kernel and user space +applications running on embedded systems. LTTng is packaged by two popular +embedded Linux distributions: [Buildroot](#doc-buildroot) and +[OpenEmbedded/Yocto](#doc-oe-yocto). diff --git a/contents/installing-lttng/embedded-distributions/oe-yocto.md b/contents/installing-lttng/embedded-distributions/oe-yocto.md new file mode 100644 index 0000000..e1b1cda --- /dev/null +++ b/contents/installing-lttng/embedded-distributions/oe-yocto.md @@ -0,0 +1,23 @@ +--- +id: oe-yocto +--- + +LTTng recipes are available in the `openembedded-core` layer of +OpenEmbedded: + + * `lttng-tools` + * `lttng-modules` + * `lttng-ust` + +Using BitBake, the simplest way to include LTTng recipes in your +target image is to add them to `IMAGE_INSTALL_append` in +`conf/local.conf`: + +~~~ text +IMAGE_INSTALL_append = " lttng-tools lttng-modules lttng-ust" +~~~ + +If you're using Hob, click _Edit image recipe_ once you have selected +a machine and an image recipe. Then, in the _All recipes_ tab, search +for `lttng` and you should find and be able to include the three LTTng +recipes. diff --git a/contents/installing-lttng/intro.md b/contents/installing-lttng/intro.md new file mode 100644 index 0000000..5d69a66 --- /dev/null +++ b/contents/installing-lttng/intro.md @@ -0,0 +1,26 @@ +--- +id: installing-lttng +--- + +**LTTng** is a set of software components which interact to allow +instrumenting the Linux kernel and user applications and controlling +tracing sessions (starting/stopping tracing, enabling/disabling events, +etc.). Those components are bundled into the following packages: + + * **LTTng-tools**: Libraries and command line interface to control + tracing sessions + * **LTTng-modules**: Linux kernel modules allowing Linux to be + traced using LTTng + * **LTTng-UST**: User space tracing library + +Most distributions mark the LTTng-modules and LTTng-UST packages as +optional. In the following sections, we always provide the steps to +install all three, but be aware that LTTng-modules is only required if +you intend to trace the Linux kernel and LTTng-UST is only required if +you intend to trace user space applications. + +This chapter shows how to install the above packages on a Linux +system. The easiest way is to use the package manager of the system's +distribution ([desktop](#doc-desktop-distributions) or +[embedded](#doc-embedded-distributions)). Otherwise, you can +[build the LTTng packages from source](#doc-building-from-source). diff --git a/contents/nuts-and-bolts/intro.md b/contents/nuts-and-bolts/intro.md new file mode 100644 index 0000000..15d5808 --- /dev/null +++ b/contents/nuts-and-bolts/intro.md @@ -0,0 +1,98 @@ +--- +id: nuts-and-bolts +--- + +What is LTTng? As its name suggests, the +_Linux Trace Toolkit: next generation_ is a modern toolkit for +tracing Linux systems and applications. So your first question might +rather be: **what is tracing?** + +As the history of software engineering progressed and led to what +we now take for granted—complex, numerous and +interdependent software applications running in parallel on +sophisticated operating systems like Linux—the authors of such +components, or software developers, began feeling a natural +urge of having tools to ensure the robustness and good performance +of their masterpieces. + +One major achievement in this field is, inarguably, the +GNU debugger +(GDB), which is an essential tool for developers to find and fix +bugs. But even the best debugger won't help make your software run +faster, and nowadays, faster softwares means either more work done by +the same hardware, or cheaper hardware for the same work. + +A _profiler_ is often the tool of choice to identify performance +bottleneck. Profiling is suitable to identify _where_ performance is +lost in a given software; the profiler outputs a profile, a +statistical summary of observed events, which you may use to know +which functions took the most time to execute. However, a profiler +won't report _why_ some identified functions are the bottleneck. +Also, bottlenecks might only occur when specific conditions are met. +For a thorough investigation of software performance issues, a history +of execution, with historical values of chosen variables, is +essential. This is where tracing comes in handy. + +_Tracing_ is a technique used to understand what goes on in a running +software system. The software used for tracing is called a _tracer_, +which is conceptually similar to a tape recorder. When recording, +specific points placed in the software source code generate events +that are saved on a giant tape: a _trace_ file. Both user applications +and the operating system may be traced at the same time, opening the +possibility of resolving a wide range of problems that are otherwise +extremely challenging. + +Tracing is often compared to _logging_. However, tracers and loggers +are two different tools, serving two different purposes. Tracers are +designed to record much lower-level events that occur much more +frequently than log messages, often in the thousands per second range, +with very little execution overhead. Logging is more appropriate for +very high-level analysis of less frequent events: user accesses, +exceptional conditions (e.g., errors, warnings), database +transactions, instant messaging communications, etc. More formally, +logging is one of several use cases that can be accomplished with +tracing. + +The list of recorded events inside a trace file may be read manually +like a log file for the maximum level of detail, but it is generally +much more interesting to perform application-specific analyses to +produce reduced statistics and graphs that are useful to resolve a +given problem. Trace viewers and analysers are specialized tools which +achieve this. + +So, in the end, this is what LTTng is: a powerful, open source set of +tools to trace the Linux kernel and user applications. LTTng is +composed of several components actively maintained and developed by +its community. + +Excluding proprietary solutions, a few competing software tracers +exist for Linux. +ftrace +is the de facto function tracer of the Linux kernel. +strace +is able to record all system calls made by a user process. +SystemTap +is a Linux kernel and user space tracer which uses custom user scripts +to produce plain text traces. +sysdig +also uses scripts, written in Lua, to trace and analyze the Linux +kernel. + +The main distinctive features of LTTng is that it produces correlated +kernel and user space traces, as well as doing so with the lowest +overhead amongst other solutions. It produces trace files in the +CTF +format, an optimized file format for production and analyses of +multi-gigabyte data. LTTng is the result of close to 10 years of +active development by a community of passionate developers. It is +currently available on all major desktop and embedded Linux +distributions. + +The main interface for tracing control is a single command line tool +named `lttng`. The latter can create several tracing sessions, +enable/disable events on the fly, filter them efficiently with custom +user expressions, start/stop tracing and do much more. Traces can be +recorded on disk or sent over the network, kept totally or partially, +and viewed once tracing is inactive or in real-time. + +[Install LTTng now](#doc-installing-lttng) and start tracing! diff --git a/contents/preface.md b/contents/preface.md new file mode 100644 index 0000000..050be73 --- /dev/null +++ b/contents/preface.md @@ -0,0 +1,142 @@ +--- +id: preface +--- + +
+

+ Copyright © 2014 The LTTng Project +

+ +

+ This work is licensed under a + Creative + Commons Attribution 4.0 International License. +

+
+ + +## Welcome! + +Welcome to the **LTTng Documentation**! + +The _Linux Trace Toolkit: next generation_ +is an open source system software package for correlated tracing of the +Linux kernel, user applications and libraries. LTTng consists of kernel +modules (for Linux kernel tracing) and dynamically loaded libraries (for +user application and library tracing). It is controlled by a session +daemon, which receives commands from a command line interface. + + +### Version + +This online LTTng Documentation is always up-to-date with the most +recent releases of LTTng packages. Older versions are always available +as tagged commits in the documentation's +official Git +repository. + + +### Convention + +Function and argument names, variable names, command names, +file system paths, file names and other precise strings are written +using a monospaced typeface in this document. An +italic word within such a block is a +placeholder, usually described in the following sentence. + +Practical tips and sidenotes are given throughout the document using a +blue background: + +
+

Tip:Make sure you read the tips.

+
+ +Terminal boxes are used to show command lines: + +
+echo This is a terminal box
+
+ +Typical command prompts, like `$` and `#`, are not shown in terminal +boxes to make copy/paste operations easier, especially for multiline +commands which may be copied and pasted as is in a user's terminal. +Commands to be executed as a root user begin with `sudo`. + + +### Target audience + +The material of this documentation is appropriate for intermediate to +advanced software developers working in a Linux environment who are +interested in efficient software tracing. LTTng may also be worth a +try for students interested in the inner mechanics of their systems. + +Readers who do not have a programming background may wish to skip +everything related to instrumentation, which requires, most of the +time, some C language skills. + +
+

Note to readers:This is an open +documentation: its source is available in a +public Git +repository. Should you find any error in the contents of this text, +any grammatical mistake or any dead link, we would be very grateful if +you would fill a GitHub issue for it or, even better, contribute a patch +to this documentation using a GitHub pull request.

+
+ +### Chapter descriptions + +What follows is a list of brief descriptions of this documentation's +chapters. The latter are ordered in such a way as to make the reading +as linear as possible. + + 1. [Nuts and bolts](#doc-nuts-and-bolts) explains the + rudiments of software tracing and the rationale behind the + LTTng project. + 2. [Installing LTTng](#doc-installing-lttng) is divided into + sections describing the steps needed to get a working installation + of LTTng packages for common Linux distributions and from its + source. + 3. [Getting started](#doc-getting-started) is a very concise guide to + get started quickly with LTTng kernel and user space tracing. This + chapter is recommended if you're new to LTTng or software tracing + in general. + 4. [Understanding LTTng](#doc-understanding-lttng) deals with some + core concepts and components of the LTTng suite. Understanding + those is important since the next chapter assumes you're familiar + with them. + 5. [Using LTTng](#doc-using-lttng) is a complete user guide of the + LTTng project. It shows in great details how to instrument user + applications and the Linux kernel, how to control tracing sessions + using the `lttng` command line tool and miscellaneous practical use + cases. + 6. [Reference](#doc-reference) contains references of LTTng components, + like links to online manpages and various APIs. + +We recommend that you read the above chapters in this order, although +some of them may be skipped depending on your situation. You may skip +[Nuts and bolts](#doc-nuts-and-bolts) if you're familiar with tracing +and LTTng. Also, you may jump over +[Installing LTTng](#doc-installing-lttng) if LTTng is already properly +installed on your target system. + + +### Acknowledgements + +A few people made the online LTTng Documentation possible. + +Philippe Proulx wrote and formatted most of the text. +Daniel U. Thibault, from the +DRDC, +wrote an open guide called LTTng: The Linux Trace Toolkit Next +Generation — A Comprehensive User's Guide (version 2.3 +edition) which was mostly used to complete parts of the +[Understanding LTTng](#doc-understanding-lttng) chapter and for a few +passages here and there. +The whole EfficiOS +team (Christian Babeux, Antoine Busque, Julien Desfossez, +Mathieu Desnoyers, Jérémie Galarneau and David Goulet) made essential +reviews of the whole document. + +We sincerely thank everyone who helped make this documentation what +it is. We hope you enjoy reading it as much as we did writing it. diff --git a/contents/reference/intro.md b/contents/reference/intro.md new file mode 100644 index 0000000..22d7901 --- /dev/null +++ b/contents/reference/intro.md @@ -0,0 +1,7 @@ +--- +id: reference +--- + +This chapter presents various references for LTTng packages such as links +to online manpages, tables needed by the rest of the text, descriptions +of library functions, etc. diff --git a/contents/reference/lttng-modules/intro.md b/contents/reference/lttng-modules/intro.md new file mode 100644 index 0000000..ff2269a --- /dev/null +++ b/contents/reference/lttng-modules/intro.md @@ -0,0 +1,5 @@ +--- +id: lttng-modules-ref +--- + +This section presents references of the LTTng-modules package. diff --git a/contents/reference/lttng-modules/lttng-modules-tp-fast-assign.md b/contents/reference/lttng-modules/lttng-modules-tp-fast-assign.md new file mode 100644 index 0000000..24e88a4 --- /dev/null +++ b/contents/reference/lttng-modules/lttng-modules-tp-fast-assign.md @@ -0,0 +1,150 @@ +--- +id: lttng-modules-tp-fast-assign +--- + +This table describes possible entries for the `TP_fast_assign()` part +of `LTTNG_TRACEPOINT_EVENT()`: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MacroDescription/arguments
+
    +
  • tp_assign(d, s)
    • +
+ 		+

+ Assignment of C expression s + to tracepoint field d +

+
    +
  • + d name of destination + tracepoint field +
    • +
  • + s source C expression + (may refer to tracepoint arguments) +
    • +
+
+
    +
  • tp_memcpy(d, s, l)
    • +
+ 		+

+ Memory copy of l bytes from + s to tracepoint field + d (use with array fields) +

+
    +
  • + d name of destination + tracepoint field +
    • +
  • + s source C expression + (may refer to tracepoint arguments) +
    • +
  • + l number of bytes to + copy +
    • +
+
+
    +
  • tp_memcpy_from_user(d, s, l)
    • +
+ 		+

+ Memory copy of l bytes from + user space s to tracepoint field + d (use with array fields) +

+
    +
  • + d name of destination + tracepoint field +
    • +
  • + s source C expression + (may refer to tracepoint arguments) +
    • +
  • + l number of bytes to + copy +
    • +
+
+
    +
  • tp_memcpy_dyn(d, s)
    • +
+ 		+

+ Memory copy of dynamically-sized array + from s to tracepoint field + d; number of bytes is + known from the field's length expression (use with + dynamically-sized array fields) +

+
    +
  • + d name of destination + tracepoint field +
    • +
  • + s source C expression + (may refer to tracepoint arguments) +
    • +
  • + l number of bytes to + copy +
    • +
+
+
    +
  • tp_strcpy(d, s)
    • +
+ 		+

+ String copy of s + to tracepoint field d + (use with string fields) +

+
    +
  • + d name of destination + tracepoint field +
    • +
  • + s source C expression + (may refer to tracepoint arguments) +
    • +
+
diff --git a/contents/reference/lttng-modules/lttng-modules-tp-struct-entry.md b/contents/reference/lttng-modules/lttng-modules-tp-struct-entry.md new file mode 100644 index 0000000..d6780d2 --- /dev/null +++ b/contents/reference/lttng-modules/lttng-modules-tp-struct-entry.md @@ -0,0 +1,227 @@ +--- +id: lttng-modules-tp-struct-entry +--- + +This table describes possible entries for the `TP_STRUCT__entry()` part +of `LTTNG_TRACEPOINT_EVENT()`: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MacroDescription/arguments
+
    +
  • __field(t, n)
    • +
+ 		+

Standard integer, displayed in base 10

+
    +
  • + t integer C type + (int, unsigned char, + size_t, etc.) +
    • +
  • n field name
    • +
+
+
    +
  • __field_hex(t, n)
    • +
+ 		+

Standard integer, displayed in base 16

+
    +
  • t integer C type
    • +
  • n field name
    • +
+
+
    +
  • __field_oct(t, n)
    • +
+ 		+

Standard integer, displayed in base 8

+
    +
  • + t integer C type +
    • +
  • n field name
    • +
+
+
    +
  • __field_network(t, n)
    • +
+ 		+

+ Integer in network byte order (big endian), + displayed in base 10 +

+
    +
  • + t integer C type +
    • +
  • n field name
    • +
+
+
    +
  • __field_network_hex(t, n)
    • +
+ 		+

+ Integer in network byte order (big endian), + displayed in base 16 +

+
    +
  • + t integer C type +
    • +
  • n field name
    • +
+
+
    +
  • __array(t, n, s)
    • +
+ 		+

Statically-sized array, elements displayed in base 10

+
    +
  • + t array element C type +
    • +
  • n field name
    • +
  • s number of elements
    • +
+
+
    +
  • __array_hex(t, n, s)
    • +
+ 		+

Statically-sized array, elements displayed in base 16

+
    +
  • + t array element C type +
    • +
  • n field name
    • +
  • s number of elements
    • +
+
+
    +
  • __array_text(t, n, s)
    • +
+ 		+

Statically-sized array, displayed as text

+
    +
  • + t array element C type + (always char) +
    • +
  • n field name
    • +
  • s number of elements
    • +
+
+
    +
  • __dynamic_array(t, n, s)
    • +
+ 		+

Dynamically-sized array, displayed in base 10

+
    +
  • + t array element C type +
    • +
  • n field name
    • +
  • s length C expression
    • +
+
+
    +
  • __dynamic_array_hex(t, n, s)
    • +
+ 		+

Dynamically-sized array, displayed in base 16

+
    +
  • + t array element C type +
    • +
  • n field name
    • +
  • s length C expression
    • +
+
+
    +
  • __dynamic_array_text(t, n, s)
    • +
+ 		+

Dynamically-sized array, displayed as text

+
    +
  • + t array element C type + (always char) +
    • +
  • n field name
    • +
  • s length C expression
    • +
+
+
    +
  • __string(n, s)
    • +
+ 		+

+ Null-terminated string; undefined behavior + if s is NULL +

+
    +
  • n field name
    • +
  • s string source (pointer)
    • +
+
+ +The above macros should cover the majority of cases. For advanced items, +see `probes/lttng-events.h`. diff --git a/contents/reference/lttng-ust/intro.md b/contents/reference/lttng-ust/intro.md new file mode 100644 index 0000000..e6581d5 --- /dev/null +++ b/contents/reference/lttng-ust/intro.md @@ -0,0 +1,5 @@ +--- +id: lttng-ust-ref +--- + +This section presents references of the LTTng-UST package. diff --git a/contents/reference/lttng-ust/liblttng-ust/intro.md b/contents/reference/lttng-ust/liblttng-ust/intro.md new file mode 100644 index 0000000..ebf03c5 --- /dev/null +++ b/contents/reference/lttng-ust/liblttng-ust/intro.md @@ -0,0 +1,11 @@ +--- +id: liblttng-ust +--- + +The LTTng-UST library, or `liblttng-ust`, is the main shared object +against which user applications are linked to make LTTng user space +tracing possible. + +The [C application](#doc-c-application) guide shows the complete +process to instrument, build and run a C/C++ application using +LTTng-UST, while this section contains a few important tables. diff --git a/contents/reference/lttng-ust/liblttng-ust/liblttng-ust-tp-fields.md b/contents/reference/lttng-ust/liblttng-ust/liblttng-ust-tp-fields.md new file mode 100644 index 0000000..a05c931 --- /dev/null +++ b/contents/reference/lttng-ust/liblttng-ust/liblttng-ust-tp-fields.md @@ -0,0 +1,199 @@ +--- +id: liblttng-ust-tp-fields +--- + +The available macros to define tracepoint fields, which should be listed +within `TP_FIELDS()` in `TRACEPOINT_EVENT()`, are: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MacroDescription/arguments
+
    +
  • ctf_integer(t, n, e)
    • +
  • ctf_integer_nowrite(t, n, e)
    • +
+ 		+

Standard integer, displayed in base 10

+
    +
  • + t integer C type + (int, long, + size_t, etc.) +
    • +
  • n field name
    • +
  • e argument expression
    • +
+
ctf_integer_hex(t, n, e) +

Standard integer, displayed in base 16

+
    +
  • t integer C type
    • +
  • n field name
    • +
  • e argument expression
    • +
+
ctf_integer_network(t, n, e) +

+ Integer in network byte order (big endian), + displayed in base 10 +

+
    +
  • t integer C type
    • +
  • n field name
    • +
  • e argument expression
    • +
+
ctf_integer_network_hex(t, n, e) +

+ Integer in network byte order, displayed + in base 16

+
    +
  • t integer C type
    • +
  • n field name
    • +
  • e argument expression
    • +
+
+
    +
  • ctf_float(t, n, e)
    • +
  • ctf_float_nowrite(t, n, e)
    • +
+ 		+

Floating point number

+
    +
  • + t floating point number + C type (float, double) +
    • +
  • n field name
    • +
  • e argument expression
    • +
+
+
    +
  • ctf_string(n, e)
    • +
  • ctf_string_nowrite(n, e)
    • +
+ 		+

+ Null-terminated string; undefined behavior if + e is NULL +

+
    +
  • n field name
    • +
  • e argument expression
    • +
+
+
    +
  • ctf_array(t, n, e, s)
    • +
  • ctf_array_nowrite(t, n, e, s)
    • +
+ 		+

Statically-sized array of integers

+
    +
  • t array element C type
    • +
  • n field name
    • +
  • e argument expression
    • +
  • s number of elements
    • +
+
+
    +
  • ctf_array_text(t, n, e, s)
    • +
  • ctf_array_nowrite_text(t, n, e, s)
    • +
+ 		+

+ Statically-sized array, printed as text; no need to be + null-terminated +

+
    +
  • t array element C type (always char)
    • +
  • n field name
    • +
  • e argument expression
    • +
  • s number of elements
    • +
+
+
    +
  • ctf_sequence(t, n, e, T, E)
    • +
  • ctf_sequence_nowrite(t, n, e, T, E)
    • +
+ 		+

+ Dynamically-sized array of integers; type of + E needs to be unsigned +

+
    +
  • t sequence element C type
    • +
  • n field name
    • +
  • e argument expression
    • +
  • T length expression C type
    • +
  • E length expression
    • +
+
+
    +
  • ctf_sequence_text(t, n, e, T, E)
    • +
  • ctf_sequence_text_nowrite(t, n, e, T, E)
    • +
+ 		+

+ Dynamically-sized array, displayed as text; no need to + be null-terminated; undefined behavior if + e is NULL

+
    +
  • t sequence element C type (always char)
    • +
  • n field name
    • +
  • e argument expression
    • +
  • T length expression C type
    • +
  • E length expression
    • +
+
+ +The `_nowrite` versions omit themselves from the session trace, but are +otherwise identical. This means the `_nowrite` fields won't be written +in the recorded trace. Their primary purpose is to make some +of the event context available to the +[event filters](#doc-enabling-disabling-events) without having to +commit the data to sub-buffers. diff --git a/contents/reference/lttng-ust/liblttng-ust/liblttng-ust-tracepoint-loglevel.md b/contents/reference/lttng-ust/liblttng-ust/liblttng-ust-tracepoint-loglevel.md new file mode 100644 index 0000000..c6f016d --- /dev/null +++ b/contents/reference/lttng-ust/liblttng-ust/liblttng-ust-tracepoint-loglevel.md @@ -0,0 +1,99 @@ +--- +id: liblttng-ust-tracepoint-loglevel +--- + +The following table shows the available log level values for the +`TRACEPOINT_LOGLEVEL()` macro: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Enum labelEnum valueDescription
TRACE_EMERG0System is unusable
TRACE_ALERT1Action must be taken immediately
TRACE_CRIT2Critical conditions
TRACE_ERR3Error conditions
TRACE_WARNING4Warning conditions
TRACE_NOTICE5Normal, but significant, condition
TRACE_INFO6Informational message
TRACE_DEBUG_SYSTEM7Debug information with system-level scope (set of programs)
TRACE_DEBUG_PROGRAM8Debug information with program-level scope (set of processes)
TRACE_DEBUG_PROCESS9Debug information with process-level scope (set of modules)
TRACE_DEBUG_MODULE10Debug information with module (executable/library) scope (set of units)
TRACE_DEBUG_UNIT11Debug information with compilation unit scope (set of functions)
TRACE_DEBUG_FUNCTION12Debug information with function-level scope
TRACE_DEBUG_LINE13Debug information with line-level scope (TRACEPOINT_EVENT default)
TRACE_DEBUG14Debug-level message
+ +Higher log level numbers imply the most verbosity (expect higher tracing +throughput). Log levels 0 through 6 and log level 14 match +syslog +level semantics. Log levels 7 through 13 offer more fine-grained +selection of debug information. diff --git a/contents/reference/online-manpages.md b/contents/reference/online-manpages.md new file mode 100644 index 0000000..7a502ca --- /dev/null +++ b/contents/reference/online-manpages.md @@ -0,0 +1,16 @@ +--- +id: online-lttng-manpages +--- + +LTTng packages currently install the following manpages, available +online using the links below: + + * **LTTng-tools** + * `lttng` + * `lttng-sessiond` + * `lttng-relayd` + * **LTTng-UST** + * `lttng-gen-tp` + * `lttng-ust` + * `lttng-ust-cyg-profile` + * `lttng-ust-cyg-dl` diff --git a/contents/understanding-lttng/core-concepts/channel/channel-buffering-schemes.md b/contents/understanding-lttng/core-concepts/channel/channel-buffering-schemes.md new file mode 100644 index 0000000..3b0c610 --- /dev/null +++ b/contents/understanding-lttng/core-concepts/channel/channel-buffering-schemes.md @@ -0,0 +1,18 @@ +--- +id: channel-buffering-schemes +--- + +In the user space tracing domain, two **buffering schemes** are +available when creating a channel: + + * **Per-PID buffering**: keep one ring buffer per process. + * **Per-UID buffering**: keep one ring buffer for all processes of + a single user. + +The per-PID buffering scheme will consume more memory than the per-UID +option if more than one process is instrumented for LTTng-UST. However, +per-PID buffering ensures that one process having a high event +throughput won't fill all the shared sub-buffers, only its own. + +The Linux kernel tracing domain only has one available buffering scheme +which is to use a single ring buffer for the whole system. diff --git a/contents/understanding-lttng/core-concepts/channel/channel-overwrite-mode-vs-discard-mode.md b/contents/understanding-lttng/core-concepts/channel/channel-overwrite-mode-vs-discard-mode.md new file mode 100644 index 0000000..c46e4c2 --- /dev/null +++ b/contents/understanding-lttng/core-concepts/channel/channel-overwrite-mode-vs-discard-mode.md @@ -0,0 +1,77 @@ +--- +id: channel-overwrite-mode-vs-discard-mode +--- + +As previously mentioned, a channel's ring buffer is divided into many +equally sized sub-buffers. + +As events occur, they are serialized as trace data into a specific +sub-buffer (yellow arc in the following animation) until it is full: +when this happens, the sub-buffer is marked as consumable (red) and +another, _empty_ (white) sub-buffer starts receiving the following +events. The marked sub-buffer will be consumed eventually by a consumer +daemon (returns to white). + + + + + +In an ideal world, sub-buffers are consumed faster than filled, like it +is the case above. In the real world, however, all sub-buffers could be +full at some point, leaving no space to record the following events. By +design, LTTng is a _non-blocking_ tracer: when no empty sub-buffer +exists, losing events is acceptable when the alternative would be to +cause substantial delays in the instrumented application's execution. +LTTng privileges performance over integrity, aiming at perturbing the +traced system as little as possible in order to make tracing of subtle +race conditions and rare interrupt cascades possible. + +When it comes to losing events because no empty sub-buffer is available, +the channel's _event loss mode_ determines what to do amongst: + + * **Discard**: drop the newest events until a sub-buffer is released. + * **Overwrite**: clear the sub-buffer containing the oldest recorded + events and start recording the newest events there. This mode is + sometimes called _flight recorder mode_ because it behaves like a + flight recorder: always keep a fixed amount of the latest data. + +Which mechanism you should choose depends on your context: prioritize +the newest or the oldest events in the ring buffer? + +Beware that, in overwrite mode, a whole sub-buffer is abandoned as soon +as a new event doesn't find an empty sub-buffer, whereas in discard +mode, only the event that doesn't fit is discarded. + +Also note that a count of lost events will be incremented and saved in +the trace itself when an event is lost in discard mode, whereas no +information is kept when a sub-buffer gets overwritten before being +committed. + +There are known ways to decrease your probability of losing events. The +next section shows how tuning the sub-buffers count and size can be +used to virtually stop losing events. diff --git a/contents/understanding-lttng/core-concepts/channel/channel-subbuf-size-vs-subbuf-count.md b/contents/understanding-lttng/core-concepts/channel/channel-subbuf-size-vs-subbuf-count.md new file mode 100644 index 0000000..f2f3b5b --- /dev/null +++ b/contents/understanding-lttng/core-concepts/channel/channel-subbuf-size-vs-subbuf-count.md @@ -0,0 +1,87 @@ +--- +id: channel-subbuf-size-vs-subbuf-count +--- + +For each channel, an LTTng user may set its number of sub-buffers and +their size. + +Note that there is a noticeable tracer's CPU overhead introduced when +switching sub-buffers (marking a full one as consumable and switching +to an empty one for the following events to be recorded). Knowing this, +the following list presents a few practical situations along with how +to configure sub-buffers for them: + + * **High event throughput**: in general, prefer bigger sub-buffers to + lower the risk of losing events. Having bigger sub-buffers will + also ensure a lower sub-buffer switching frequency. The number of + sub-buffers is only meaningful if the channel is in overwrite mode: + in this case, if a sub-buffer overwrite happens, you will still have + the other sub-buffers left unaltered. + * **Low event throughput**: in general, prefer smaller sub-buffers + since the risk of losing events is already low. Since events + happen less frequently, the sub-buffer switching frequency should + remain low and thus the tracer's overhead should not be a problem. + * **Low memory system**: if your target system has a low memory + limit, prefer fewer first, then smaller sub-buffers. Even if the + system is limited in memory, you want to keep the sub-buffers as + big as possible to avoid a high sub-buffer switching frequency. + +You should know that LTTng uses CTF as its trace format, which means +event data is very compact. For example, the average LTTng Linux kernel +event weights about 32 bytes. A sub-buffer size of 1 MiB is +thus considered big. + +The previous situations highlight the major trade-off between a few big +sub-buffers and more, smaller sub-buffers: sub-buffer switching +frequency vs. how much data is lost in overwrite mode. Assuming a +constant event throughput and using the overwrite mode, the two +following configurations have the same ring buffer total size: + + + + + + * **2 sub-buffers of 4 MiB each** lead to a very low sub-buffer + switching frequency, but if a sub-buffer overwrite happens, half of + the recorded events so far (4 MiB) are definitely lost. + * **8 sub-buffers of 1 MiB each** lead to 4 times the tracer's + overhead as the previous configuration, but if a sub-buffer + overwrite happens, only the eighth of events recorded so far are + definitely lost. + +In discard mode, the sub-buffers count parameter is pointless: use two +sub-buffers and set their size according to the requirements of your +situation. diff --git a/contents/understanding-lttng/core-concepts/channel/channel-timers.md b/contents/understanding-lttng/core-concepts/channel/channel-timers.md new file mode 100644 index 0000000..d98c0af --- /dev/null +++ b/contents/understanding-lttng/core-concepts/channel/channel-timers.md @@ -0,0 +1,83 @@ +--- +id: channel-switch-timer +--- + +The _switch timer_ period is another important configurable feature of +channels to ensure periodic sub-buffer flushing. + +When the _switch timer_ fires, a sub-buffer switch happens. This timer +may be used to ensure that event data is consumed and committed to +trace files periodically in case of a low event throughput: + + + + + +It's also convenient when big sub-buffers are used to cope with +sporadic high event throughput, even if the throughput is normally +lower. diff --git a/contents/understanding-lttng/core-concepts/channel/intro.md b/contents/understanding-lttng/core-concepts/channel/intro.md new file mode 100644 index 0000000..b7b51a7 --- /dev/null +++ b/contents/understanding-lttng/core-concepts/channel/intro.md @@ -0,0 +1,22 @@ +--- +id: channel +--- + +A _channel_ is a set of events with specific parameters and potential +added context information. Channels have unique names per domain within +a tracing session. A given event is always registered to at least one +channel; having an enabled event in two channels will produce a trace +with this event recorded twice everytime it occurs. + +Channels may be individually enabled or disabled. Occurring events of +a disabled channel will never make it to recorded events. + +The fundamental role of a channel is to keep a shared ring buffer, where +events are eventually recorded by the tracer and consumed by a consumer +daemon. This internal ring buffer is divided into many sub-buffers of +equal size. + +Channels, when created, may be fine-tuned thanks to a few parameters, +many of them related to sub-buffers. The following subsections explain +what those parameters are and in which situations you should manually +adjust them. diff --git a/contents/understanding-lttng/core-concepts/domain.md b/contents/understanding-lttng/core-concepts/domain.md new file mode 100644 index 0000000..724af91 --- /dev/null +++ b/contents/understanding-lttng/core-concepts/domain.md @@ -0,0 +1,25 @@ +--- +id: domain +--- + +A tracing _domain_ is the official term the LTTng project uses to +designate a tracer category. + +There are currently three known domains: + + * Linux kernel + * user space + * Java Util Logging (JUL) + +Different tracers expose common features in their own interfaces, but, +from a user's perspective, you still need to target a specific type of +tracer to perform some actions. For example, since both kernel and user +space tracers support named tracepoints (probes manually inserted in +source code), you need to specify which one is concerned when enabling +an event because both domains could have existing events with the same +name. + +Some features are not available in all domains. Filtering enabled +events using custom expressions, for example, is currently not +supported in the kernel domain, but support could be added in the +future. diff --git a/contents/understanding-lttng/core-concepts/event.md b/contents/understanding-lttng/core-concepts/event.md new file mode 100644 index 0000000..bca5fcc --- /dev/null +++ b/contents/understanding-lttng/core-concepts/event.md @@ -0,0 +1,48 @@ +--- +id: event +--- + +An _event_, in LTTng's realm, is a term often used metonymically, +having multiple definitions depending on the context: + + 1. When tracing, an event is a _point in space-time_. Space, in a + tracing context, is the set of all executable positions of a + compiled application by a logical processor. When a program is + executed by a processor and some instrumentation point, or + _probe_, is encountered, an event occurs. This event is accompanied + by some contextual payload (values of specific variables at this + point of execution) which may or may not be recorded. + 2. In the context of a recorded trace file, the term _event_ implies + a _recorded event_. + 3. When configuring a tracing session, _enabled events_ refer to + specific rules which could lead to the transfer of actual + occurring events (1) to recorded events (2). + +The whole [Core concepts](#doc-core-concepts) section focusses on the +third definition. An event is always registered to _one or more_ +channels and may be enabled or disabled at will per channel. A disabled +event will never lead to a recorded event, even if its channel +is enabled. + +An event (3) is enabled with a few conditions that must _all_ be met +when an event (1) happens in order to generate a recorded event (2): + + 1. A _probe_ or group of probes in the traced application must be + executed. + 2. **Optionally**, the probe must have a log level matching a + log level range specified when enabling the event. + 3. **Optionally**, the occurring event must satisfy a custom + expression, or _filter_, specified when enabling the event. + +The following illustration summarizes how tracing sessions, domains, +channels and events are related: + +
+ + + +
+ +This diagram also shows how events may be individually enabled/disabled +(green/grey) and how a given event may be registered to more than one +channel. diff --git a/contents/understanding-lttng/core-concepts/intro.md b/contents/understanding-lttng/core-concepts/intro.md new file mode 100644 index 0000000..70dd038 --- /dev/null +++ b/contents/understanding-lttng/core-concepts/intro.md @@ -0,0 +1,11 @@ +--- +id: core-concepts +--- + +This section explains the various elementary concepts a user has to deal +with when using LTTng. They are: + + * [tracing session](#doc-tracing-session) + * [domain](#doc-domain) + * [channel](#doc-channel) + * [event](#doc-event) diff --git a/contents/understanding-lttng/core-concepts/tracing-session.md b/contents/understanding-lttng/core-concepts/tracing-session.md new file mode 100644 index 0000000..3b0e40a --- /dev/null +++ b/contents/understanding-lttng/core-concepts/tracing-session.md @@ -0,0 +1,48 @@ +--- +id: tracing-session +--- + +A _tracing session_ is—like any session—a container of +state. Anything that is done when tracing using LTTng happens in the +scope of a tracing session. In this regard, it is analogous to a bank +website's session: you can't interact online with your bank account +unless you are logged in a session, except for reading a few static +webpages (LTTng, too, can report some static information that does not +need a created tracing session). + +A tracing session holds the following attributes and objects (some of +which are described in the following sections): + + * a name + * the tracing state (tracing started or stopped) + * the trace data output path/URL (local path or sent over the network) + * a mode (normal, snapshot or live) + * the snapshot output paths/URLs (if applicable) + * for each [domain](#doc-domain), a list of [channels](#doc-channel) + * for each channel: + * a name + * the channel state (enabled or disabled) + * its parameters (event loss mode, sub-buffers size and count, + timer periods, output type, trace files size and count, etc.) + * a list of added context information + * a list of [events](#doc-event) + * for each event: + * its state (enabled or disabled) + * a list of instrumentation points (tracepoints, system calls, + dynamic probes, etc.) + * associated log levels + * a filter expression + +All this information is completely isolated between tracing sessions. + +Conceptually, a tracing session is a per-user object; the +[Plumbing](#doc-plumbing) section shows how this is actually +implemented. Any user may create as many concurrent tracing sessions +as desired. As you can see in the list above, even the tracing state +is a per-tracing session attribute, so that you may trace your target +system/application in a given tracing session with a specific +configuration while another one stays inactive. + +The trace data generated in a tracing session may be either saved +to disk, sent over the network or not saved at all (in which case +snapshots may still be saved to disk or sent to a remote machine). diff --git a/contents/understanding-lttng/intro.md b/contents/understanding-lttng/intro.md new file mode 100644 index 0000000..9da52fd --- /dev/null +++ b/contents/understanding-lttng/intro.md @@ -0,0 +1,15 @@ +--- +id: understanding-lttng +--- + +If you're going to use LTTng in any serious way, it is fundamental that +you become familiar with its core concepts. Technical terms like +_tracing sessions_, _domains_, _channels_ and _events_ are used over +and over in the [Using LTTng](#doc-using-lttng) chapter, +and it is assumed that you understand what they mean when reading it. + +LTTng, as you already know, is a _toolkit_. It would be wrong +to call it a simple _tool_ since it is composed of multiple interacting +components. This chapter also describes the latter, providing details +about their respective roles and how they connect together to form +the current LTTng ecosystem. diff --git a/contents/understanding-lttng/plumbing/intro.md b/contents/understanding-lttng/plumbing/intro.md new file mode 100644 index 0000000..4aad2da --- /dev/null +++ b/contents/understanding-lttng/plumbing/intro.md @@ -0,0 +1,8 @@ +--- +id: plumbing +--- + +The previous section described the concepts at the heart of LTTng. +This section summarizes LTTng's implementation: how those objects are +managed by different applications and libraries working together to +form the toolkit. diff --git a/contents/understanding-lttng/plumbing/liblttng-ctl-lttng.md b/contents/understanding-lttng/plumbing/liblttng-ctl-lttng.md new file mode 100644 index 0000000..d880bd5 --- /dev/null +++ b/contents/understanding-lttng/plumbing/liblttng-ctl-lttng.md @@ -0,0 +1,29 @@ +--- +id: liblttng-ctl-lttng +--- + +The LTTng control library, `liblttng-ctl`, can be used to communicate +with the session daemon using a C API that hides the underlying +protocol's details. `liblttng-ctl` is part of LTTng-tools. + +`liblttng-ctl` may be used by including its "master" header: + +~~~ c +#include +~~~ + +Some objects are referred by name (C string), such as tracing sessions, +but most of them require creating a handle first using +`lttng_create_handle()`. The best available developer documentation for +`liblttng-ctl` is, for the moment, its installed header files as such. +Every function/structure is thoroughly documented. + +The `lttng` program is the _de facto_ standard user interface to +control LTTng tracing sessions. `lttng` uses `liblttng-ctl` to +communicate with session daemons behind the scenes. +Its manpage is exhaustive, as +well as its command line help (lttng cmd --help, +where cmd is the command name). + +The [Controlling tracing](#doc-controlling-tracing) section is a feature +tour of the `lttng` tool. diff --git a/contents/understanding-lttng/plumbing/lttng-consumerd.md b/contents/understanding-lttng/plumbing/lttng-consumerd.md new file mode 100644 index 0000000..3bd76fa --- /dev/null +++ b/contents/understanding-lttng/plumbing/lttng-consumerd.md @@ -0,0 +1,39 @@ +--- +id: lttng-consumerd +--- + +The _consumer daemon_, or `lttng-consumerd`, is a program sharing some +ring buffers with user applications or the LTTng kernel modules to +collect trace data and output it at some place (on disk or sent over +the network to an LTTng relay daemon). + +Consumer daemons are created by a session daemon as soon as events are +enabled within a tracing session, well before tracing is activated +for the latter. Entirely managed by session daemons, +consumer daemons survive session destruction to be reused later, +should a new tracing session be created. Consumer daemons are always +owned by the same user as their session daemon. When its owner session +daemon is killed, the consumer daemon also exits. This is because +the consumer daemon is always the child process of a session daemon. +Consumer daemons should never be started manually. For this reason, +they are not installed in one of the usual locations listed in the +`PATH` environment variable. `lttng-sessiond` has, however, a +bunch of options to +specify custom consumer daemon paths if, for some reason, a consumer +daemon other than the default installed one is needed. + +There are up to two running consumer daemons per user, whereas only one +session daemon may run per user. This is because each process has +independent bitness: if the target system runs a mixture of 32-bit and +64-bit processes, it is more efficient to have separate corresponding +32-bit and 64-bit consumer daemons. The `root` user is an exception: it +may have up to _three_ running consumer daemons: 32-bit and 64-bit +instances for its user space applications and one more reserved for +collecting kernel trace data. + +As new tracing domains are added to LTTng, the development community's +intent is to minimize the need for additionnal consumer daemon instances +dedicated to them. For instance, the Java Util Logging (JUL) domain +events are in fact mapped to the user space domain, thus tracing this +particular domain is handled by existing user space domain consumer +daemons. diff --git a/contents/understanding-lttng/plumbing/lttng-modules.md b/contents/understanding-lttng/plumbing/lttng-modules.md new file mode 100644 index 0000000..27075a5 --- /dev/null +++ b/contents/understanding-lttng/plumbing/lttng-modules.md @@ -0,0 +1,23 @@ +--- +id: lttng-modules +--- + +The LTTng Linux kernel modules provide everything needed to trace the +Linux kernel: various probes, a ring buffer implementation for a +consumer daemon to read trace data and the tracer itself. + +Only in exceptional circumstances should you ever need to load the +LTTng kernel modules manually: it is normally the responsability of +`root`'s session daemon to do so. Even if you were to develop your +own LTTng probe module—for tracing a custom kernel or some kernel +module (this topic is covered in the +[Linux kernel](#doc-instrumenting-linux-kernel) instrumenting guide of +the [Using LTTng](#doc-using-lttng) chapter)—you +should use the `--extra-kmod-probes` option of the session daemon to +append your probe to the default list. The session and consumer daemons +of regular users do not interact with the LTTng kernel modules at all. + +LTTng kernel modules are installed, by default, in +/usr/lib/modules/release/extra, where +release is the kernel release +(see `uname --kernel-release`). diff --git a/contents/understanding-lttng/plumbing/lttng-relayd.md b/contents/understanding-lttng/plumbing/lttng-relayd.md new file mode 100644 index 0000000..5ee4153 --- /dev/null +++ b/contents/understanding-lttng/plumbing/lttng-relayd.md @@ -0,0 +1,32 @@ +--- +id: lttng-relayd +--- + +When a tracing session is configured to send its trace data over the +network, an LTTng _relay daemon_ must be used at the other end to +receive trace packets and serialize them to trace files. This setup +makes it possible to trace a target system without ever committing trace +data to its local storage, a feature which is useful for embedded +systems, amongst others. The command implementing the relay daemon +is `lttng-relayd`. + +The basic use case of `lttng-relayd` is to transfer trace data received +over the network to trace files on the local file system. The relay +daemon must listen on two TCP ports to achieve this: one control port, +used by the target session daemon, and one data port, used by the +target consumer daemon. The relay and session daemons agree on common +default ports when custom ones are not specified. + +Since the communication transport protocol for both ports is standard +TCP, the relay daemon may be started either remotely or locally (on the +target system). + +While two instances of consumer daemons (32-bit and 64-bit) may run +concurrently for a given user, `lttng-relayd` needs only be of its +host operating system's bitness. + +The other important feature of LTTng's relay daemon is the support of +_LTTng live_. LTTng live is an application protocol to view events as +they arrive. The relay daemon will still record events in trace files, +but a _tee_ may be created to inspect incoming events. Using LTTng live +locally thus requires to run a local relay daemon. diff --git a/contents/understanding-lttng/plumbing/lttng-sessiond.md b/contents/understanding-lttng/plumbing/lttng-sessiond.md new file mode 100644 index 0000000..f964622 --- /dev/null +++ b/contents/understanding-lttng/plumbing/lttng-sessiond.md @@ -0,0 +1,82 @@ +--- +id: lttng-sessiond +--- + +At the heart of LTTng's plumbing is the _session daemon_, often called +by its command name, `lttng-sessiond`. + +The session daemon is responsible for managing tracing sessions and +what they logically contain (channel properties, enabled/disabled +events, etc.). By communicating locally with instrumented applications +(using LTTng-UST) and with the LTTng Linux kernel modules +(LTTng-modules), it oversees all tracing activities. + +One of the many things that `lttng-sessiond` does is to keep +track of the available event types. User space applications and +libraries actively connect and register to the session daemon when they +start. By contrast, `lttng-sessiond` seeks out and loads the appropriate +LTTng kernel modules as part of its own initialization. Kernel event +types are _pulled_ by `lttng-sessiond`, whereas user space event types +are _pushed_ to it by the various user space tracepoint providers. + +Using a specific inter-process communication protocol with Linux kernel +and user space tracers, the session daemon can send channel information +so that they are initialized, enable/disable specific probes based on +enabled/disabled events by the user, send event filters information to +LTTng tracers so that filtering actually happens at the tracer site, +start/stop tracing a specific application or the Linux kernel, etc. + +The session daemon is not useful without some user controlling it, +because it's only a sophisticated control interchange and thus +doesn't make any decision on its own. `lttng-sessiond` opens a local +socket for controlling it, albeit the preferred way to control it is +using `liblttng-ctl`, an installed C library hiding the communication +protocol behind an easy-to-use API. The `lttng` tool makes use of +`liblttng-ctl` to implement a user-friendly command line interface. + +`lttng-sessiond` does not receive any trace data from instrumented +applications; the _consumer daemons_ are the programs responsible for +collecting trace data using shared ring buffers. However, the session +daemon is the one that must spawn a consumer daemon and establish +a control communication with it. + +Session daemons run on a per-user basis. Knowing this, multiple +instances of `lttng-sessiond` may run simultaneously, each belonging +to a different user and each operating independently of the others. +Only `root`'s session daemon, however, may control LTTng kernel modules +(i.e. the kernel tracer). With that in mind, if a user has no root +access on the target system, he cannot trace the system's kernel, but +should still be able to trace its own instrumented applications. + +It has to be noted that, although only `root`'s session daemon may +control the kernel tracer, the `lttng-sessiond` command has a `--group` +option which may be used to specify the name of a special user group +allowed to communicate with `root`'s session daemon and thus record +kernel traces. By default, this group is named `tracing`. + +If not done yet, the `lttng` tool, by default, automatically starts a +session daemon. `lttng-sessiond` may also be started manually: + +
+lttng-sessiond
+
+ +This will start the session daemon in foreground. Use + +
+lttng-sessiond --daemonize
+
+ +to start it as a true daemon. + +To kill the current user's session daemon, `pkill` may be used: + +
+pkill lttng-sessiond
+
+ +The default `SIGTERM` signal will terminate it cleanly. + +Several other options are available and described in +lttng-sessiond's manpage +or by running `lttng-sessiond --help`. diff --git a/contents/understanding-lttng/plumbing/lttng-ust.md b/contents/understanding-lttng/plumbing/lttng-ust.md new file mode 100644 index 0000000..850a722 --- /dev/null +++ b/contents/understanding-lttng/plumbing/lttng-ust.md @@ -0,0 +1,29 @@ +--- +id: lttng-ust +--- + +The user space tracing part of LTTng is possible thanks to the user +space tracing library, `liblttng-ust`, which is part of the LTTng-UST +package. + +`liblttng-ust` provides header files containing macros used to define +tracepoints and create tracepoint providers, as well as a shared object +that must be linked to individual applications to connect to and +communicate with a session daemon and a consumer daemon as soon as the +application starts. + +The exact mechanism by which an application is registered to the +session daemon is beyond the scope of this documentation. The only thing +you need to know is that, since the library constructor does this job +automatically, tracepoints may be safely inserted anywhere in the source +code without prior manual initialization of `liblttng-ust`. + +The `liblttng-ust`-session daemon collaboration also provides an +interesting feature: user space events may be enabled _before_ +applications actually start. By doing this and starting tracing before +launching the instrumented application, you make sure that even the +earliest occurring events can be recorded. + +The [C application](#doc-c-application) instrumenting guide of the +[Using LTTng](#doc-using-lttng) chapter focuses on using `liblttng-ust`: +instrumenting, building/linking and running a user application. diff --git a/contents/understanding-lttng/plumbing/overview.md b/contents/understanding-lttng/plumbing/overview.md new file mode 100644 index 0000000..3a538f5 --- /dev/null +++ b/contents/understanding-lttng/plumbing/overview.md @@ -0,0 +1,44 @@ +--- +id: plumbing-overview +--- + +As [mentioned previously](#doc-installing-lttng), the whole LTTng suite +is made of the following packages: LTTng-tools, LTTng-UST, and +LTTng-modules. Together, they provide different daemons, libraries, +kernel modules and command line interfaces. The following tree shows +which usable component belongs to which package: + + * **LTTng-tools**: + * session daemon (`lttng-sessiond`) + * consumer daemon (`lttng-consumerd`) + * relay daemon (`lttng-relayd`) + * tracing control library (`liblttng-ctl`) + * tracing control command line tool (`lttng`) + * **LTTng-UST**: + * user space tracing library (`liblttng-ust`) and its headers + * preloadable user space tracing helpers + (`liblttng-ust-libc-wrapper`, `liblttng-ust-pthread-wrapper`, + `liblttng-ust-cyg-profile`, `liblttng-ust-cyg-profile-fast` + and `liblttng-ust-dl`) + * user space tracepoint code generator command line tool + (`lttng-gen-tp`) + * Java Util Logging tracepoint provider (`liblttng-ust-jul-jni`) + and JAR file (`liblttng-ust-jul.jar`) + * **LTTng-modules**: + * LTTng Linux kernel tracer module + * tracing ring buffer kernel modules + * many LTTng probe kernel modules + +The following diagram shows how the most important LTTng components +interact. Plain black arrows represent trace data paths while dashed +red arrows indicate control communications. The LTTng relay daemon is +shown running on a remote system, although it could as well run on the +target (monitored) system. + +
+ + + +
+ +Each component is described in the following subsections. diff --git a/contents/using-lttng/controlling-tracing/adding-context.md b/contents/using-lttng/controlling-tracing/adding-context.md new file mode 100644 index 0000000..c77f1d5 --- /dev/null +++ b/contents/using-lttng/controlling-tracing/adding-context.md @@ -0,0 +1,55 @@ +--- +id: adding-context +--- + +If you read all the sections of +[Controlling tracing](#doc-controlling-tracing) so far, you should be +able to create tracing sessions, create and enable channels and events +within them and start/stop the LTTng tracers. Event fields recorded in +trace files provide important information about occurring events, but +sometimes external context may help you solve a problem faster. This +section discusses how to add context information to events of a +specific channel using the `lttng` tool. + +There are various available context values which can accompany events +recorded by LTTng, for example: + + * **process information**: + * identifier (PID) + * name + * priority + * scheduling priority (niceness) + * thread identifier (TID) + * the **hostname** of the system on which the event occurred + * plenty of **performance counters** using perf: + * CPU cycles, stalled cycles, idle cycles, etc. + * cache misses + * branch instructions, misses, loads, etc. + * CPU faults + * etc. + +The full list is available in the output of `lttng add-context --help`. +Some of them are reserved for a specific domain (kernel or +user space) while others are available for both. + +To add context information to one or all channels of a given tracing +session, use the `add-context` command: + +
+lttng add-context --userspace --type vpid --type perf:thread:cpu-cycles
+
+ +The above example adds the virtual process identifier and per-thread +CPU cycles count values to all recorded user space domain events of the +current tracing session. Use the `--channel` option to select a specific +channel: + +
+lttng add-context --kernel --channel my-channel --type tid
+
+ +adds the thread identifier value to all recorded kernel domain events +in the channel `my-channel` of the current tracing session. + +Beware that context information cannot be removed from channels once +it's added for a given tracing session. diff --git a/contents/using-lttng/controlling-tracing/basic-tracing-session-control.md b/contents/using-lttng/controlling-tracing/basic-tracing-session-control.md new file mode 100644 index 0000000..d23c94d --- /dev/null +++ b/contents/using-lttng/controlling-tracing/basic-tracing-session-control.md @@ -0,0 +1,26 @@ +--- +id: basic-tracing-session-control +--- + +Once you have +[created a tracing session](#doc-creating-destroying-tracing-sessions) +and [enabled one or more events](#doc-enabling-disabling-events), +you may activate the LTTng tracers for the current tracing session at +any time: + +
+lttng start
+
+ +Subsequently, you may stop the tracers: + +
+lttng stop
+
+ +LTTng is very flexible: user space applications may be launched before +or after the tracers are started. Events will only be recorded if they +are properly enabled and if they occur while tracers are started. + +A tracing session name may be passed to both the `start` and `stop` +commands to start/stop tracing a session other than the current one. diff --git a/contents/using-lttng/controlling-tracing/creating-destroying-tracing-sessions.md b/contents/using-lttng/controlling-tracing/creating-destroying-tracing-sessions.md new file mode 100644 index 0000000..e7d173b --- /dev/null +++ b/contents/using-lttng/controlling-tracing/creating-destroying-tracing-sessions.md @@ -0,0 +1,88 @@ +--- +id: creating-destroying-tracing-sessions +--- + +Whatever you want to do with `lttng`, it has to happen inside a +**tracing session**, created beforehand. A session, in general, is a +per-user container of state. A tracing session is no different; it +keeps a specific state of stuff like: + + * session name + * enabled/disabled channels with associated parameters + * enabled/disabled events with associated log levels and filters + * context information added to channels + * tracing activity (started or stopped) + +and more. + +A single user may have many active tracing sessions. LTTng session +daemons are the ultimate owners and managers of tracing sessions. For +user space tracing, each user has its own session daemon. Since Linux +kernel tracing requires root privileges, only `root`'s session daemon +may enable and trace kernel events. However, `lttng` has a `--group` +option (which is passed to `lttng-sessiond` when starting it) to +specify the name of a _tracing group_ which selected users may be part +of to be allowed to communicate with `root`'s session daemon. By +default, the tracing group name is `tracing`. + +To create a tracing session, do: + +
+lttng create my-session
+
+ +This will create a new tracing session named `my-session` and make it +the current one. If you don't specify any name (calling only +`lttng create`), your tracing session will be named `auto`. Traces +are written in ~/lttng-traces/session- followed +by the tracing session's creation date/time by default, where +session is the tracing session name. To save them +at a different location, use the `--output` option: + +
+lttng create --output /tmp/some-directory my-session
+
+ +You may create as many tracing sessions as you wish: + +
+lttng create other-session
+lttng create yet-another-session
+
+ +You may view all existing tracing sessions using the `list` command: + +
+lttng list
+
+ +The state of a _current tracing session_ is kept in `~/.lttngrc`. Each +invocation of `lttng` reads this file to set its current tracing +session name so that you don't have to specify a session name for each +command. You could edit this file manually, but the preferred way to +set the current tracing session is to use the `set-session` command: + +
+lttng set-session other-session
+
+ +Most `lttng` commands accept a `--session` option to specify the name +of the target tracing session. + +Any existing tracing session may be destroyed using the `destroy` +command: + +
+lttng destroy my-session
+
+ +Providing no argument to `lttng destroy` will destroy the current +tracing session. Destroying a tracing session will stop any tracing +running within the latter. Destroying a tracing session frees resources +acquired by the session daemon and tracer side, making sure to flush +all trace data. + +You can't do much with LTTng using only the `create`, `set-session` +and `destroy` commands of `lttng`, but it is essential to know them in +order to control LTTng tracing, which always happen within the scope of +a tracing session. diff --git a/contents/using-lttng/controlling-tracing/enabling-disabling-channels/fine-tuning-channels.md b/contents/using-lttng/controlling-tracing/enabling-disabling-channels/fine-tuning-channels.md new file mode 100644 index 0000000..c0267cf --- /dev/null +++ b/contents/using-lttng/controlling-tracing/enabling-disabling-channels/fine-tuning-channels.md @@ -0,0 +1,81 @@ +--- +id: fine-tuning-channels +--- + +There are various parameters that may be fine-tuned with the +`enable-channel` command. The latter are well documented in +the manpage of `lttng` +and in the [Channel](#doc-channel) section of the +[Understanding LTTng](#doc-understanding-lttng) chapter. For basic +tracing needs, their default values should be just fine, but here are a +few examples to break the ice. + +As the frequency of recorded events increases—either because the +event throughput is actually higher or because you enabled more events +than usual—_event loss_ might be experienced. Since LTTng never +waits, by design, for sub-buffer space availability (non-blocking +tracer), when a sub-buffer is full and no empty sub-buffers are left, +there are two possible outcomes: either the new events that do not fit +are rejected, or they start replacing the oldest recorded events. +The choice of which algorithm to use is a per-channel parameter, the +default being discarding the newest events until there is some space +left. If your situation always needs the latest events at the expense +of writing over the oldest ones, create a channel with the `--overwrite` +option: + +
+lttng enable-channel --kernel --overwrite my-channel
+
+ +When an event is lost, it means no space was available in any +sub-buffer to accommodate it. Thus, if you want to cope with sporadic +high event throughput situations and avoid losing events, you need to +allocate more room for storing them in memory. This can be done by +either increasing the size of sub-buffers or by adding sub-buffers. +The following example creates a user space domain channel with +16 sub-buffers of 512 kiB each: + +
+lttng enable-channel --userspace --num-subbuf 16 --subbuf-size 512k big-channel
+
+ +Both values need to be powers of two, otherwise they are rounded up +to the next one. + +Two other interesting available parameters of `enable-channel` are +`--tracefile-size` and `--tracefile-count`, which respectively limit +the size of each trace file and the their count for a given channel. +When the number of written trace files reaches its limit for a given +channel-CPU pair, the next trace file will overwrite the very first +one. The following example creates a kernel domain channel with a +maximum of three trace files of 1 MiB each: + +
+lttng enable-channel --kernel --tracefile-size 1M --tracefile-count 3 my-channel
+
+ +An efficient way to make sure lots of events are generated is enabling +all kernel events in this channel and starting the tracer: + +
+lttng enable-event --kernel --all --channel my-channel
+lttng start
+
+ +After a few seconds, look at trace files in your tracing session +output directory. For two CPUs, it should look like: + +~~~ text +my-channel_0_0 my-channel_1_0 +my-channel_0_1 my-channel_1_1 +my-channel_0_2 my-channel_1_2 +~~~ + +Amongst the files above, you might see one in each group with a size +lower than 1 MiB: they are the files currently being written. + +Since all those small files are valid LTTng trace files, LTTng trace +viewers may read them. It is the viewer's responsibility to properly +merge the streams so as to present an ordered list to the user. +Babeltrace +merges LTTng trace files correctly and is fast at doing it. diff --git a/contents/using-lttng/controlling-tracing/enabling-disabling-channels/intro.md b/contents/using-lttng/controlling-tracing/enabling-disabling-channels/intro.md new file mode 100644 index 0000000..a6b884e --- /dev/null +++ b/contents/using-lttng/controlling-tracing/enabling-disabling-channels/intro.md @@ -0,0 +1,100 @@ +--- +id: enabling-disabling-channels +--- + +[As mentioned](#doc-event) in the +[Understanding LTTng](#doc-understanding-lttng) chapter, enabled +events are contained in a specific channel, itself contained in a +specific tracing session. A channel is a group of events with +tunable parameters (event loss mode, sub-buffer size, number of +sub-buffers, trace file sizes and count, etc.). A given channel may +only be responsible for enabled events belonging to one domain: either +kernel or user space. + +If you only used the `create`, `enable-event` and `start`/`stop` +commands of the `lttng` tool so far, one or two channels were +automatically created for you (one for the kernel domain and/or one +for the user space domain). The default channels are both named +`channel0`; channels from different domains may have the same name. + +The current channels of a given tracing session can be viewed with + +
+lttng list some-session
+
+ +where `some-session` is the name of the desired tracing session. + +To create and enable a channel, use the `enable-channel` command: + +
+lttng enable-channel --kernel my-channel
+
+ +This will create a kernel domain channel named `my-channel` with +default parameters in the current tracing session. + +
+

+ Note:Because of a current limitation, all + channels must be created prior to beginning tracing in a + given tracing session, i.e. before the first time you do + lttng start. +

+

+ Since a channel is automatically created by + enable-event only for the specified domain, you cannot, + for example, enable a kernel domain event, start tracing and then + enable a user space domain event because no user space channel + exists yet and it's too late to create one. +

+

+ For this reason, make sure to configure your channels properly + before starting the tracers for the first time! +

+
+ +Here's another example: + +
+lttng enable-channel --userspace --session other-session --overwrite \
+                     --tracefile-size 1048576 1mib-channel
+
+ +This will create a user space domain channel named `1mib-channel` in +the tracing session named `other-session` that loses new events by +overwriting previously recorded events (instead of the default mode of +discarding newer ones) and saves trace files with a maximum size of +1 MiB each. + +Note that channels may also be created using the `--channel` option of +the `enable-event` command when the provided channel name doesn't exist +for the specified domain: + +
+lttng enable-event --kernel --channel some-channel sched_switch
+
+ +If no kernel domain channel named `some-channel` existed before calling +the above command, it would be created with default parameters. + +You may enable the same event in two different channels: + +
+lttng enable-event --userspace --channel my-channel app:tp
+lttng enable-event --userspace --channel other-channel app:tp
+
+ +If both channels are enabled, the occurring `app:tp` event will +generate two recorded events, one for each channel. + +Disabling a channel is done with the `disable-event` command: + +
+lttng disable-event --kernel some-channel
+
+ +The state of a channel precedes the individual states of events within +it: events belonging to a disabled channel, even if they are +enabled, won't be recorded. + diff --git a/contents/using-lttng/controlling-tracing/enabling-disabling-events.md b/contents/using-lttng/controlling-tracing/enabling-disabling-events.md new file mode 100644 index 0000000..ac4d137 --- /dev/null +++ b/contents/using-lttng/controlling-tracing/enabling-disabling-events.md @@ -0,0 +1,131 @@ +--- +id: enabling-disabling-events +--- + +Inside a tracing session, individual events may be enabled or disabled +so that tracing them may or may not generate trace data. + +We sometimes use the term _event_ metonymically throughout this text to +refer to a specific condition, or _rule_, that could lead, when +satisfied, to an actual occurring event (a point at a specific position +in source code/binary program, logical processor and time capturing +some payload) being recorded as trace data. This specific condition is +composed of: + + 1. A **domain** (kernel, user space or Java Util Logging) (required). + 2. One or many **instrumentation points** in source code or binary + program (tracepoint name, address, symbol name, function name, + logger name, etc.) to be executed (required). + 3. A **log level** (each instrumentation point declares its own log + level) or log level range to match (optional; only valid for user + space domain). + 4. A **custom user expression**, or **filter**, that must evaluate to + _true_ when a tracepoint is executed (optional; only valid for user + space domain). + +All conditions are specified using arguments passed to the +`enable-event` command of the `lttng` tool. + +Condition 1 is specified using either `--kernel/-k` (kernel), +`--userspace/-u` (user space) or `--jul/-j` +(JUL). Exactly one of those +three arguments must be specified. + +Condition 2 is specified using one of: + + * `--tracepoint`: **tracepoint** + * `--probe`: **dynamic probe** (address, symbol name or combination + of both in binary program; only valid for kernel domain) + * `--function`: **function entry/exit** (address, symbol name or + combination of both in binary program; only valid for kernel domain) + * `--syscall`: **system call entry/exit** (only valid for kernel + domain) + +When none of the above is specified, `enable-event` defaults to +using `--tracepoint`. + +Condition 3 is specified using one of: + + * `--loglevel`: log level range from 0 to a specific log level + * `--loglevel-only`: specific log level + +See `lttng enable-event --help` for the complete list of log level +names. + +Condition 4 is specified using the `--filter` option. This filter is +a C-like expression, potentially reading real-time values of event +fields, that has to evaluate to _true_ for the condition to be satisfied. +Event fields are read using plain identifiers while context fields +must be prefixed with `$ctx.`. See `lttng enable-event --help` for +all usage details. + +The aforementioned arguments are combined to create and enable events. +Each unique combination of arguments leads to a different +_enabled event_. The log level and filter arguments are optional, their +default values being respectively all log levels and a filter which +always returns _true_. + +Here are a few examples (you must +[create a tracing session](#doc-creating-destroying-tracing-sessions) +first): + +
+lttng enable-event -u --tracepoint my_app:hello_world
+lttng enable-event -u --tracepoint my_app:hello_you --loglevel TRACE_WARNING
+lttng enable-event -u --tracepoint 'my_other_app:*'
+lttng enable-event -u --tracepoint my_app:foo_bar \
+                   --filter 'some_field <= 23 && !other_field'
+lttng enable-event -k --tracepoint sched_switch
+lttng enable-event -k --tracepoint gpio_value
+lttng enable-event -k --function usb_probe_device usb_probe_device
+lttng enable-event -k --syscall --all
+
+ +The wildcard symbol, `*`, matches _anything_ and may only be used at +the end of the string when specifying a _tracepoint_. Make sure to +use it between single quotes in your favorite shell to avoid +undesired shell expansion. + +You can see a list of events (enabled or disabled) using + +
+lttng list some-session
+
+ +where `some-session` is the name of the desired tracing session. + +What you're actually doing when enabling events with specific conditions +is creating a **whitelist** of traceable events for a given channel. +Thus, the following case presents redundancy: + +
+lttng enable-event -u --tracepoint my_app:hello_you
+lttng enable-event -u --tracepoint my_app:hello_you --loglevel TRACE_DEBUG
+
+ +The second command, matching a log level range, is useless since the first +command enables all tracepoints matching the same name, +`my_app:hello_you`. + +Disabling an event is simpler: you only need to provide the event +name to the `disable-event` command: + +
+lttng disable-event -u my_app:hello_you
+
+ +This name has to match a name previously given to `enable-event` (it +has to be listed in the output of `lttng list some-session`). +The `*` wildcard is supported, as long as you also used it in a +previous `enable-event` invocation. + +Disabling an event does not add it to some blacklist: it simply removes +it from its channel's whitelist. This is why you cannot disable an event +which wasn't previously enabled. + +A disabled event will not generate any trace data, even if all its +specified conditions are met. + +Events may be enabled and disabled at will, either when LTTng tracers +are active or not. Events may be enabled before a user space application +is even started. diff --git a/contents/using-lttng/controlling-tracing/intro.md b/contents/using-lttng/controlling-tracing/intro.md new file mode 100644 index 0000000..3b177c3 --- /dev/null +++ b/contents/using-lttng/controlling-tracing/intro.md @@ -0,0 +1,27 @@ +--- +id: controlling-tracing +--- + +Once you're in possession of a software that is properly +[instrumented](#doc-instrumenting) for LTTng tracing, be it thanks to +the built-in LTTng probes for the Linux kernel, a custom user +application or a custom Linux kernel, all that is left is actually +tracing it. As a user, you control LTTng tracing using a single command +line interface: the `lttng` tool. This tool uses `liblttng-ctl` behind +the scene to connect to and communicate with session daemons. LTTng +session daemons may either be started manually (`lttng-sessiond`) or +automatically by the `lttng` command when needed. Trace data may +be forwarded to the network and used elsewhere using an LTTng relay +daemon (`lttng-relayd`). + +The manpages of `lttng`, `lttng-sessiond` and `lttng-relayd` are pretty +complete, thus this section is not an online copy of the latter (we +leave this contents for the +[Online LTTng manpages](#doc-online-lttng-manpages) section). +This section is rather a tour of LTTng +features through practical examples and tips. + +If not already done, make sure you understand the core concepts +and how LTTng components connect together by reading the +[Understanding LTTng](#doc-understanding-lttng) chapter; this section +assumes you are familiar with them. diff --git a/contents/using-lttng/controlling-tracing/lttng-live.md b/contents/using-lttng/controlling-tracing/lttng-live.md new file mode 100644 index 0000000..9d64620 --- /dev/null +++ b/contents/using-lttng/controlling-tracing/lttng-live.md @@ -0,0 +1,80 @@ +--- +id: lttng-live +--- + +We have seen how trace files may be produced by LTTng out of generated +application and Linux kernel events. We have seen that those trace files +may be either recorded locally by consumer daemons or remotely using +a relay daemon. And we have seen that the maximum size and count of +trace files is configurable for each channel. With all those features, +it's still not possible to read a trace file as it is being written +because it could be incomplete and appear corrupted to the viewer. +There is a way to view events as they arrive, however: using +_LTTng live_. + +LTTng live is implemented, in LTTng, solely on the relay daemon side. +As trace data is sent over the network to a relay daemon by a (possibly +remote) consumer daemon, a _tee_ may be created: trace data will be +recorded to trace files _as well as_ being transmitted to a +connected live viewer: + +
+ + + +
+ +In order to use this feature, a tracing session must created in live +mode on the target system: + +
+lttng create --live
+
+ +An optional parameter may be passed to `--live` to set the interval +of time (in microseconds) between flushes to the network +(1 second is the default): + +
+lttng create --live 100000
+
+ +will flush every 100 ms. + +If no network output is specified to the `create` command, a local +relay daemon will be spawned. In this very common case, viewing a live +trace is easy: enable events and start tracing as usual, then use +`lttng view` to start the default live viewer: + +
+lttng view
+
+ +The correct arguments will be passed to the live viewer so that it +may connect to the local relay daemon and start reading live events. + +You may also wish to use a live viewer not running on the target +system. In this case, you should specify a network output when using +the `create` command (`--set-url` or `--ctrl-url`/`--data-url` options). +A distant LTTng relay daemon should also be started to receive control +and trace data. By default, `lttng-relayd` listens on 127.0.0.1:5344 +for an LTTng live connection. Otherwise, the desired URL may be +specified using its `--live-port` option. + +The +`babeltrace` +viewer supports LTTng live as one of its input formats. `babeltrace` is +the default viewer when using `lttng view`. To use it manually, first +list active tracing sessions by doing the following (assuming the relay +daemon to connect to runs on the same host): + +
+babeltrace -i lttng-live net://localhost
+
+ +Then, choose a tracing session and start viewing events as they arrive +using LTTng live, e.g.: + +
+babeltrace -i lttng-live net://localhost/host/hostname/my-session
+
diff --git a/contents/using-lttng/controlling-tracing/saving-loading-tracing-session.md b/contents/using-lttng/controlling-tracing/saving-loading-tracing-session.md new file mode 100644 index 0000000..5f8e928 --- /dev/null +++ b/contents/using-lttng/controlling-tracing/saving-loading-tracing-session.md @@ -0,0 +1,47 @@ +--- +id: saving-loading-tracing-session +--- + +Configuring a tracing session may be long: creating and enabling +channels with specific parameters, enabling kernel and user space +domain events with specific log levels and filters, adding context +to some channels, etc. If you're going to use LTTng to solve real +world problems, chances are you're going to have to record events using +the same tracing session setup over and over, modifying a few variables +each time in your instrumented program or environment. To avoid +constant tracing session reconfiguration, the `lttng` tool is able to +save and load tracing session configurations to/from XML files. + +To save a given tracing session configuration, do: + +
+lttng save my-session
+
+ +where `my-session` is the name of the tracing session to save. Tracing +session configurations are saved to `~/.lttng/sessions` by default; +use the `--output-path` option to change this destination directory. + +All configuration parameters are saved: + + * tracing session name + * trace data output path + * channels with their state and all their parameters + * context information added to channels + * events with their state, log level and filter + * tracing activity (started or stopped) + +To load a tracing session, simply do: + +
+lttng load my-session
+
+ +or, if you used a custom path: + +
+lttng load --input-path /path/to/my-session.lttng
+
+ +Your saved tracing session will be restored as if you just configured +it manually. diff --git a/contents/using-lttng/controlling-tracing/sending-trace-data-over-the-network.md b/contents/using-lttng/controlling-tracing/sending-trace-data-over-the-network.md new file mode 100644 index 0000000..1e1b2c2 --- /dev/null +++ b/contents/using-lttng/controlling-tracing/sending-trace-data-over-the-network.md @@ -0,0 +1,49 @@ +--- +id: sending-trace-data-over-the-network +--- + +The possibility of sending trace data over the network comes as a +built-in feature of LTTng-tools. For this to be possible, an LTTng +_relay daemon_ must be executed and listening on the machine where +trace data is to be received, and the user must create a tracing +session using appropriate options to forward trace data to the remote +relay daemon. + +The relay daemon listens on two different TCP ports: one for control +information and the other for actual trace data. + +Starting the relay daemon on the remote machine is as easy as: + +
+lttng-relayd
+
+ +This will make it listen to its default ports: 5342 for control and +5343 for trace data. The `--control-port` and `--data-port` options may +be used to specify different ports. + +Traces written by `lttng-relayd` are written to +~/lttng-traces/hostname/session by +default, where hostname is the host name of the +traced (monitored) system and session is the +tracing session name. Use the `--output` option to write trace data +outside `~/lttng-traces`. + +On the sending side, a tracing session must be created using the +`lttng` tool with the `--set-url` option to connect to the distant +relay daemon: + +
+lttng create my-session --set-url net://distant-host
+
+ +The URL format is described in the output of `lttng create --help`. +The above example will use the default ports; the `--ctrl-url` and +`--data-url` options may be used to set the control and data URLs +individually. + +Once this basic setup is completed and the connection is established, +you may use the `lttng` tool on the target machine as usual; everything +you do will be transparently forwarded to the remote machine if needed. +For example, a parameter changing the maximum size of trace files will +have an effect on the distant relay daemon actually writing the trace. diff --git a/contents/using-lttng/controlling-tracing/taking-a-snapshot.md b/contents/using-lttng/controlling-tracing/taking-a-snapshot.md new file mode 100644 index 0000000..fd6ef22 --- /dev/null +++ b/contents/using-lttng/controlling-tracing/taking-a-snapshot.md @@ -0,0 +1,68 @@ +--- +id: taking-a-snapshot +--- + +The normal behavior of LTTng is to record trace data as trace files. +This is ideal for keeping a long history of events that occurred on +the target system and applications, but may be too much data in some +situations. For example, you may wish to trace your application +continuously until some critical situation happens, in which case you +would only need the latest few recorded events to perform the desired +analysis, not multi-gigabyte trace files. + +LTTng has an interesting feature called _snapshots_. When creating +a tracing session in snapshot mode, no trace files are written; the +tracers' sub-buffers are constantly overwriting the oldest recorded +events with the newest. At any time, either when the tracers are started +or stopped, you may take a snapshot of those sub-buffers. + +There is no difference between the format of a normal trace file and the +format of a snapshot: viewers of LTTng traces will also support LTTng +snapshots. By default, snapshots are written to disk, but they may also +be sent over the network. + +To create a tracing session in snapshot mode, do: + +
+lttng create --snapshot my-snapshot-session
+
+ +Next, enable channels, events and add context to channels as usual. +Once a tracing session is created in snapshot mode, channels will be +forced to use the overwrite mode (`--overwrite` option of the +`enable-channel` command) and have an `mmap()` channel type +(`--output mmap`). + +Start tracing. When you're ready to take a snapshot, do: + +
+lttng snapshot record --name my-snapshot
+
+ +This will record a snapshot named `my-snapshot` of all channels of +all domains of the current tracing session. By default, snapshots files +are recorded in the path returned by `lttng snapshot list-output`. You +may change this path or decide to send snapshots over the network +using either: + + 1. an output path/URL specified when creating the tracing session + (`lttng create`) + 2. an added snapshot output path/URL using + `lttng snapshot add-output` + 3. an output path/URL provided directly to the + `lttng snapshot record` command + +Method 3 overrides method 2 which overrides method 1. When specifying +a URL, a relay daemon must be listening on some machine (see +[Sending trace data over the network](#doc-sending-trace-data-over-the-network)). + +If you need to make absolutely sure that the output file won't be +larger than a certain limit, you can set a maximum snapshot size when +taking it with the `--max-size` option: + +
+lttng snapshot record --name my-snapshot --max-size 2M
+
+ +Older recorded events will be discarded in order to respect this +maximum size. diff --git a/contents/using-lttng/instrumenting/advanced-techniques/32-bit-on-64-bit/building-32-bit-lttng-tools.md b/contents/using-lttng/instrumenting/advanced-techniques/32-bit-on-64-bit/building-32-bit-lttng-tools.md new file mode 100644 index 0000000..329e4e8 --- /dev/null +++ b/contents/using-lttng/instrumenting/advanced-techniques/32-bit-on-64-bit/building-32-bit-lttng-tools.md @@ -0,0 +1,26 @@ +--- +id: building-32-bit-lttng-tools +--- + +Since the host is a 64-bit system, most 32-bit binaries and libraries of +LTTng-tools are not needed; the host will use their 64-bit counterparts. +The required step here is building and installing a 32-bit consumer +daemon. + +Follow this: + +
+git clone http://git.lttng.org/lttng-tools.git
+cd lttng-ust
+./bootstrap
+./configure --prefix=/usr \
+            --libdir=/usr/lib32 CFLAGS=-m32 CXXFLAGS=-m32 \
+            LDFLAGS=-L/usr/lib32
+make
+cd src/bin/lttng-consumerd
+sudo make install
+sudo ldconfig
+
+ +The above commands build all the LTTng-tools project as 32-bit +applications, but only installs the 32-bit consumer daemon. diff --git a/contents/using-lttng/instrumenting/advanced-techniques/32-bit-on-64-bit/building-32-bit-lttng-ust.md b/contents/using-lttng/instrumenting/advanced-techniques/32-bit-on-64-bit/building-32-bit-lttng-ust.md new file mode 100644 index 0000000..17b67df --- /dev/null +++ b/contents/using-lttng/instrumenting/advanced-techniques/32-bit-on-64-bit/building-32-bit-lttng-ust.md @@ -0,0 +1,30 @@ +--- +id: building-32-bit-lttng-ust +--- + +Follow this: + +
+git clone http://git.lttng.org/lttng-ust.git
+cd lttng-ust
+./bootstrap
+./configure --prefix=/usr \
+            --libdir=/usr/lib32 \
+            CFLAGS=-m32 CXXFLAGS=-m32 \
+            LDFLAGS=-L/usr/lib32
+make
+sudo make install
+sudo ldconfig
+
+ +`-L/usr/lib32` is required for the build to find the 32-bit version +of Userspace RCU. + +
+

+ Note:You may add options to + ./configure if you need them, e.g., for + Java and SystemTap support. Look at + ./configure --help for more information. +

+
diff --git a/contents/using-lttng/instrumenting/advanced-techniques/32-bit-on-64-bit/building-32-bit-userspace-rcu.md b/contents/using-lttng/instrumenting/advanced-techniques/32-bit-on-64-bit/building-32-bit-userspace-rcu.md new file mode 100644 index 0000000..e990524 --- /dev/null +++ b/contents/using-lttng/instrumenting/advanced-techniques/32-bit-on-64-bit/building-32-bit-userspace-rcu.md @@ -0,0 +1,18 @@ +--- +id: building-32-bit-userspace-rcu +--- + +Follow this: + +
+git clone git://git.urcu.so/urcu.git
+cd urcu
+./bootstrap
+./configure --libdir=/usr/lib32 CFLAGS=-m32
+make
+sudo make install
+sudo ldconfig
+
+ +The `-m32` C compiler flag creates 32-bit object files and `--libdir` +indicates where to install the resulting libraries. diff --git a/contents/using-lttng/instrumenting/advanced-techniques/32-bit-on-64-bit/building-64-bit-lttng-tools.md b/contents/using-lttng/instrumenting/advanced-techniques/32-bit-on-64-bit/building-64-bit-lttng-tools.md new file mode 100644 index 0000000..c643c1a --- /dev/null +++ b/contents/using-lttng/instrumenting/advanced-techniques/32-bit-on-64-bit/building-64-bit-lttng-tools.md @@ -0,0 +1,20 @@ +--- +id: building-64-bit-lttng-tools +--- + +Finally, you need to build a 64-bit version of LTTng-tools which is +aware of the 32-bit consumer daemon previously built and installed: + +
+make clean
+./bootstrap
+./configure --prefix=/usr \
+            --with-consumerd32-libdir=/usr/lib32 \
+            --with-consumerd32-bin=/usr/lib32/lttng/libexec/lttng-consumerd
+make
+sudo make install
+sudo ldconfig
+
+ +Henceforth, the 64-bit session daemon will automatically find the +32-bit consumer daemon if required. diff --git a/contents/using-lttng/instrumenting/advanced-techniques/32-bit-on-64-bit/building-instrumented-32-bit-c-application.md b/contents/using-lttng/instrumenting/advanced-techniques/32-bit-on-64-bit/building-instrumented-32-bit-c-application.md new file mode 100644 index 0000000..aaed4c5 --- /dev/null +++ b/contents/using-lttng/instrumenting/advanced-techniques/32-bit-on-64-bit/building-instrumented-32-bit-c-application.md @@ -0,0 +1,26 @@ +--- +id: building-instrumented-32-bit-c-application +--- + +Let us reuse the _Hello world_ example of +[Tracing your own user application](#doc-tracing-your-own-user-application) +([Getting started](#doc-getting-started) chapter). + +The instrumentation process is unaltered. + +First, a typical 64-bit build (assuming you're running a 64-bit system): + +
+gcc -o hello64 -I. hello.c hello-tp.c -ldl -llttng-ust
+
+ +Now, a 32-bit build: + +
+gcc -o hello32 -I. -m32 hello.c hello-tp.c -L/usr/lib32 \
+    -ldl -llttng-ust -Wl,-rpath,/usr/lib32
+
+ +The `-rpath` option, passed to the linker, will make the dynamic loader +check for libraries in `/usr/lib32` before looking in its default paths, +where it should find the 32-bit version of `liblttng-ust`. diff --git a/contents/using-lttng/instrumenting/advanced-techniques/32-bit-on-64-bit/intro.md b/contents/using-lttng/instrumenting/advanced-techniques/32-bit-on-64-bit/intro.md new file mode 100644 index 0000000..3c9877c --- /dev/null +++ b/contents/using-lttng/instrumenting/advanced-techniques/32-bit-on-64-bit/intro.md @@ -0,0 +1,20 @@ +--- +id: instrumenting-32-bit-app-on-64-bit-system +--- + +In order to trace a 32-bit application running on a 64-bit system, +LTTng must use a dedicated 32-bit +[consumer daemon](#doc-lttng-consumerd). This section discusses how to +build that daemon (which is _not_ part of the default 64-bit LTTng +build) and the LTTng 32-bit tracing libraries, and how to instrument +a 32-bit application in that context. + +Make sure you install all 32-bit versions of LTTng dependencies. +Their names can be found in the `README.md` files of each LTTng package +source. How to find and install them will vary depending on your target +Linux distribution. `gcc-multilib` is a common package name for the +multilib version of GCC, which you will also need. + +The following packages will be built for 32-bit support on a 64-bit +system: Userspace RCU, +LTTng-UST and LTTng-tools. diff --git a/contents/using-lttng/instrumenting/advanced-techniques/32-bit-on-64-bit/running-32-bit-and-64-bit-c-applications.md b/contents/using-lttng/instrumenting/advanced-techniques/32-bit-on-64-bit/running-32-bit-and-64-bit-c-applications.md new file mode 100644 index 0000000..c768af4 --- /dev/null +++ b/contents/using-lttng/instrumenting/advanced-techniques/32-bit-on-64-bit/running-32-bit-and-64-bit-c-applications.md @@ -0,0 +1,18 @@ +--- +id: running-32-bit-and-64-bit-c-applications +--- + +Now, both 32-bit and 64-bit versions of the _Hello world_ example above +can be traced in the same tracing session. Use the `lttng` tool as usual +to create a tracing session and start tracing: + +
+lttng create session-3264
+lttng enable-event -u -a
+./hello32
+./hello64
+lttng stop
+
+ +Use `lttng view` to verify both processes were +successfully traced. diff --git a/contents/using-lttng/instrumenting/advanced-techniques/intro.md b/contents/using-lttng/instrumenting/advanced-techniques/intro.md new file mode 100644 index 0000000..99e75a7 --- /dev/null +++ b/contents/using-lttng/instrumenting/advanced-techniques/intro.md @@ -0,0 +1,6 @@ +--- +id: advanced-instrumenting-techniques +--- + +This section presents some advanced techniques related to +LTTng instrumenting. diff --git a/contents/using-lttng/instrumenting/c-application/assigning-log-levels.md b/contents/using-lttng/instrumenting/c-application/assigning-log-levels.md new file mode 100644 index 0000000..5565e2c --- /dev/null +++ b/contents/using-lttng/instrumenting/c-application/assigning-log-levels.md @@ -0,0 +1,31 @@ +--- +id: assigning-log-levels +--- + +Optionally, a log level can be assigned to a defined tracepoint. +Assigning different levels of importance to tracepoints can be useful; +when controlling tracing sessions, +[you can choose](#doc-controlling-tracing) to only enable tracepoints +falling into a specific log level range. + +Log levels are assigned to defined tracepoints using the +`TRACEPOINT_LOGLEVEL()` macro. The latter must be used _after_ having +used `TRACEPOINT_EVENT()` for a given tracepoint. The +`TRACEPOINT_LOGLEVEL()` macro has the following construct: + +~~~ c +TRACEPOINT_LOGLEVEL(, , ) +~~~ + +where the first two arguments are the same as the first two arguments +of `TRACEPOINT_EVENT()` and `` is one +of the values given in the +[LTTng-UST library reference](#doc-liblttng-ust-tracepoint-loglevel) +section. + +As an example, let's assign a `TRACE_DEBUG_UNIT` log level to our +previous tracepoint definition: + +~~~ c +TRACEPOINT_LOGLEVEL(my_provider, my_tracepoint, TRACE_DEBUG_UNIT) +~~~ diff --git a/contents/using-lttng/instrumenting/c-application/building-linking/dynamic-linking.md b/contents/using-lttng/instrumenting/c-application/building-linking/dynamic-linking.md new file mode 100644 index 0000000..10e13aa --- /dev/null +++ b/contents/using-lttng/instrumenting/c-application/building-linking/dynamic-linking.md @@ -0,0 +1,69 @@ +--- +id: dynamic-linking +--- + +The second approach to package the tracepoint providers is to use +dynamic linking: the library and its member functions are explicitly +sought, loaded and unloaded at runtime using `libdl`. + +It has to be noted that, for a variety of reasons, the created shared +library will be dynamically _loaded_, as opposed to dynamically +_linked_. The tracepoint provider shared object is, however, linked +with `liblttng-ust`, so that `liblttng-ust` is guaranteed to be loaded +as soon as the tracepoint provider is. If the tracepoint provider is +not loaded, since the application itself is not linked with +`liblttng-ust`, the latter is not loaded at all and the tracepoint calls +become inert. + +The process to create the tracepoint provider shared object is pretty +much the same as the static library method, except that: + + * since the tracepoint provider is not part of the application + anymore, `TRACEPOINT_DEFINE` _must_ be defined in one translation + unit (C source file) of the _application_; + * `TRACEPOINT_PROBE_DYNAMIC_LINKAGE` must be defined next to + `TRACEPOINT_DEFINE`. + +`TRACEPOINT_PROBE_DYNAMIC_LINKAGE` makes the macros included afterwards +(by including the tracepoint provider header, which itself includes +LTTng-UST headers) aware that the tracepoint provider is to be loaded +dynamically and not part of the application's executable. + +The tracepoint provider object file used to create the shared library +is built like it is using the static library method, only with the +`-fpic` option added: + +
+gcc -c -fpic -I. tp.c
+
+ +It is then linked as a shared library like this: + +
+gcc -shared -Wl,--no-as-needed -o tp.so -llttng-ust tp.o
+
+ +As previously stated, this tracepoint provider shared object isn't +linked with the user application: it will be loaded manually. This is +why the application is built with no mention of this tracepoint +provider, but still needs `libdl`: + +
+gcc -o app other.o files.o of.o your.o app.o -ldl
+
+ +Now, to make LTTng-UST tracing available to the application, +the `LD_PRELOAD` environment variable is used to preload the +tracepoint provider shared library _before_ the application actually +starts: + +
+LD_PRELOAD=/path/to/tp.so ./app
+
+ +You application will still work without this preloading, albeit without +LTTng-UST tracing support: + +
+./app
+
diff --git a/contents/using-lttng/instrumenting/c-application/building-linking/intro.md b/contents/using-lttng/instrumenting/c-application/building-linking/intro.md new file mode 100644 index 0000000..3f48212 --- /dev/null +++ b/contents/using-lttng/instrumenting/c-application/building-linking/intro.md @@ -0,0 +1,35 @@ +--- +id: building-tracepoint-providers-and-user-application +--- + +This section explains the final step of using LTTng-UST for tracing +a user space C application (beside running the application): building and +linking tracepoint providers and the application itself. + +As discussed above, the macros used by the user-written tracepoint provider +header file are useless until actually used to create probes code +(global data structures and functions) in a translation unit (C source file). +This is accomplished by defining `TRACEPOINT_CREATE_PROBES` in a translation +unit and then including the tracepoint provider header file. +When `TRACEPOINT_CREATE_PROBES` is defined, macros used and included by +the tracepoint provider header will output actual source code needed by any +application using the defined tracepoints. Defining +`TRACEPOINT_CREATE_PROBES` produces code used when registering +tracepoint providers when the tracepoint provider package loads. + +The other important definition is `TRACEPOINT_DEFINE`. This one creates +global, per-tracepoint structures referencing the tracepoint providers +data. Those structures are required by the actual functions inserted +where `tracepoint()` macros are placed and need to be defined by the +instrumented application. + +Both `TRACEPOINT_CREATE_PROBES` and `TRACEPOINT_DEFINE` need to be defined +at some places in order to trace a user space C application using LTTng. +Although explaining their exact mechanism is beyond the scope of this +document, the reason they both exist separately is to allow the trace +providers to be packaged as a shared object (dynamically loaded library). + +There are two ways to compile and link the tracepoint providers +with the application: _[statically](#doc-static-linking)_ or +_[dynamically](#doc-dynamic-linking)_. Both methods are covered in the +following subsections. diff --git a/contents/using-lttng/instrumenting/c-application/building-linking/lttng-ust-pkg-config.md b/contents/using-lttng/instrumenting/c-application/building-linking/lttng-ust-pkg-config.md new file mode 100644 index 0000000..285d7e6 --- /dev/null +++ b/contents/using-lttng/instrumenting/c-application/building-linking/lttng-ust-pkg-config.md @@ -0,0 +1,21 @@ +--- +id: lttng-ust-pkg-config +--- + +On some distributions, LTTng-UST is shipped with a pkg-config metadata +file, so that you may use the `pkg-config` tool: + +
+pkg-config --libs lttng-ust
+
+ +This will return `-llttng-ust -ldl` on Linux systems. + +You may also check the LTTng-UST version using `pkg-config`: + +
+pkg-config --modversion lttng-ust
+
+ +For more information about pkg-config, see +its manpage. diff --git a/contents/using-lttng/instrumenting/c-application/building-linking/static-linking.md b/contents/using-lttng/instrumenting/c-application/building-linking/static-linking.md new file mode 100644 index 0000000..4f53e2b --- /dev/null +++ b/contents/using-lttng/instrumenting/c-application/building-linking/static-linking.md @@ -0,0 +1,119 @@ +--- +id: static-linking +--- + +With the static linking method, compiled tracepoint providers are copied +into the target application. There are three ways to do this: + + 1. Use one of your **existing C source files** to create probes. + 2. Create probes in a separate C source file and build it as an + **object file** to be linked with the application (more decoupled). + 3. Create probes in a separate C source file, build it as an + object file and archive it to create a **static library** + (more decoupled, more portable). + +The first approach is to define `TRACEPOINT_CREATE_PROBES` and include +your tracepoint provider(s) header file(s) directly into an existing C +source file. Here's an example: + +~~~ c +#include +#include +/* ... */ + +#define TRACEPOINT_CREATE_PROBES +#define TRACEPOINT_DEFINE +#include "tp.h" + +/* ... */ + +int my_func(int a, const char* b) +{ + /* ... */ + + tracepoint(my_provider, my_tracepoint, buf, sz, limit, &tt) + + /* ... */ +} + +/* ... */ +~~~ + +Again, `TRACEPOINT_CREATE_PROBES` and `TRACEPOINT_DEFINE` must be +defined in one, **and only one**, translation unit. Other C source +files of the same application may include `tp.h` to use tracepoints +with `tracepoint()`, but must not define +`TRACEPOINT_CREATE_PROBES`/`TRACEPOINT_DEFINE` again. + +This translation unit may be built as an object file by making sure to +add `.` to the include path: + +
+gcc -c -I. file.c
+
+ +The second approach is to isolate the tracepoint provider code into a +separate object file by using a dedicated C source file to create probes: + +~~~ c +#define TRACEPOINT_CREATE_PROBES + +#include "tp.h" +~~~ + +`TRACEPOINT_DEFINE` must be defined by a translation unit of the +application. Since we're talking about static linking here, it could as +well be defined in the file above, before `#include "tp.h"`. This is +actually what [`lttng-gen-tp`](#doc-lttng-gen-tp) does. + +Build the tracepoint provider: + +
+gcc -c -I. tp.c
+
+ +Finally, the resulting object file may be archived to create a +more portable tracepoint provider static library: + +
+ar rc tp.a tp.o
+
+ +Using a static library does have the advantage of centralising the +tracepoint providers objects so they can be shared between multiple +applications. This way, when the tracepoint provider is modified, the +source code changes don't have to be patched into each application's source +code tree. The applications need to be relinked after each change, but need +not to be otherwise recompiled (unless the tracepoint provider's API +changes). + +Regardless of which method you choose, you end up with an object file +(potentially archived) containing the trace providers assembled code. +To link this code with the rest of your application, you must also link +with `liblttng-ust` and `libdl`: + +
+gcc -o app tp.o other.o files.o of.o your.o app.o -llttng-ust -ldl
+
+ +or + +
+gcc -o app tp.a other.o files.o of.o your.o app.o -llttng-ust -ldl
+
+ +If you're using a BSD +system, replace `-ldl` with `-lc`: + +
+gcc -o app tp.a other.o files.o of.o your.o app.o -llttng-ust -lc
+
+ +The application can be started as usual, e.g.: + +
+./app
+
+ +The `lttng` command line tool can be used to +[control tracing](#doc-controlling-tracing). diff --git a/contents/using-lttng/instrumenting/c-application/building-linking/using-lttng-ust-with-daemons.md b/contents/using-lttng/instrumenting/c-application/building-linking/using-lttng-ust-with-daemons.md new file mode 100644 index 0000000..d4a6cbb --- /dev/null +++ b/contents/using-lttng/instrumenting/c-application/building-linking/using-lttng-ust-with-daemons.md @@ -0,0 +1,20 @@ +--- +id: using-lttng-ust-with-daemons +--- + +Some extra care is needed when using `liblttng-ust` with daemon +applications that call `fork()`, `clone()` or BSD's `rfork()` without +a following `exec()` family system call. The `liblttng-ust-fork` +library must be preloaded for the application. + +Example: + +
+LD_PRELOAD=liblttng-ust-fork.so ./app
+
+ +Or, if you're using a tracepoint provider shared library: + +
+LD_PRELOAD="liblttng-ust-fork.so /path/to/tp.so" ./app
+
diff --git a/contents/using-lttng/instrumenting/c-application/defining-tracepoints.md b/contents/using-lttng/instrumenting/c-application/defining-tracepoints.md new file mode 100644 index 0000000..e9e007d --- /dev/null +++ b/contents/using-lttng/instrumenting/c-application/defining-tracepoints.md @@ -0,0 +1,176 @@ +--- +id: defining-tracepoints +--- + +As written in [Tracepoint provider](#doc-tracepoint-provider), +tracepoints are defined using the +`TRACEPOINT_EVENT()` macro. Each tracepoint, when called using the +`tracepoint()` macro in the actual application's source code, generates +a specific event type with its own fields. + +Let's have another look at the example above, with a few added comments: + +~~~ c +TRACEPOINT_EVENT( + /* tracepoint provider name */ + my_provider, + + /* tracepoint/event name */ + my_first_tracepoint, + + /* list of tracepoint arguments */ + TP_ARGS( + int, my_integer_arg, + char*, my_string_arg + ), + + /* list of fields of eventual event */ + TP_FIELDS( + ctf_string(my_string_field, my_string_arg) + ctf_integer(int, my_integer_field, my_integer_arg) + ) +) +~~~ + +The tracepoint provider name must match the name of the tracepoint +provider in which this tracepoint is defined +(see [Tracepoint provider](#doc-tracepoint-provider)). In other words, +always use the same string as the value of `TRACEPOINT_PROVIDER` above. + +The tracepoint name will become the event name once events are recorded +by the LTTng-UST tracer. It must follow the tracepoint provider name +syntax: start with a letter and contain either letters, numbers or +underscores. Two tracepoints under the same provider cannot have the +same name, i.e. you cannot overload a tracepoint like you would +overload functions and methods in C++/Java. + +
+

Note:The concatenation of the tracepoint +provider name and the tracepoint name cannot exceed 254 characters. If +it does, the instrumented application will compile and run, but LTTng +will issue multiple warnings and you could experience serious problems.

+
+ +The list of tracepoint arguments gives this tracepoint its signature: +see it like the declaration of a C function. The format of `TP_ARGS()` +arguments is: C type, then argument name; repeat as needed, up to ten +times. For example, if we were to replicate the signature of C standard +library's `fseek()`, the `TP_ARGS()` part would look like: + +~~~ c + TP_ARGS( + FILE*, stream, + long int, offset, + int, origin + ), +~~~ + +Of course, you will need to include appropriate header files before +the `TRACEPOINT_EVENT()` macro calls if any argument has a complex type. + +`TP_ARGS()` may not be omitted, but may be empty. `TP_ARGS(void)` is +also accepted. + +The list of fields is where the fun really begins. The fields defined +in this list will be the fields of the events generated by the execution +of this tracepoint. Each tracepoint field definition has a C +_argument expression_ which will be evaluated when the execution reaches +the tracepoint. Tracepoint arguments _may be_ used freely in those +argument expressions, but they _don't_ have to. + +There are several types of tracepoint fields available. The macros to +define them are given and explained in the +[LTTng-UST library reference](#doc-liblttng-ust-tp-fields) section. + +Field names must follow the standard C identifier syntax: letter, then +optional sequence of letters, numbers or underscores. Each field must have +a different name. + +Those `ctf_*()` macros are added to the `TP_FIELDS()` part of +`TRACEPOINT_EVENT()`. Note that they are not delimited by commas. +`TP_FIELDS()` may be empty, but the `TP_FIELDS(void)` form is _not_ +accepted. + +The following snippet shows how argument expressions may be used in +tracepoint fields and how they may refer freely to tracepoint arguments. + +~~~ c +/* for struct stat */ +#include +#include +#include + +TRACEPOINT_EVENT( + my_provider, + my_tracepoint, + TP_ARGS( + int, my_int_arg, + char*, my_str_arg, + struct stat*, st + ), + TP_FIELDS( + /* simple integer field with constant value */ + ctf_integer( + int, /* field C type */ + my_constant_field, /* field name */ + 23 + 17 /* argument expression */ + ) + + /* my_int_arg tracepoint argument */ + ctf_integer( + int, + my_int_arg_field, + my_int_arg + ) + + /* my_int_arg squared */ + ctf_integer( + int, + my_int_arg_field2, + my_int_arg * my_int_arg + ) + + /* sum of first 4 characters of my_str_arg */ + ctf_integer( + int, + sum4, + my_str_arg[0] + my_str_arg[1] + + my_str_arg[2] + my_str_arg[3] + ) + + /* my_str_arg as string field */ + ctf_string( + my_str_arg_field, /* field name */ + my_str_arg /* argument expression */ + ) + + /* st_size member of st tracepoint argument, hexadecimal */ + ctf_integer_hex( + off_t, /* field C type */ + size_field, /* field name */ + st->st_size /* argument expression */ + ) + + /* st_size member of st tracepoint argument, as double */ + ctf_float( + double, /* field C type */ + size_dbl_field, /* field name */ + (double) st->st_size /* argument expression */ + ) + + /* half of my_str_arg string as text sequence */ + ctf_sequence_text( + char, /* element C type */ + half_my_str_arg_field, /* field name */ + my_str_arg, /* argument expression */ + size_t, /* length expression C type */ + strlen(my_str_arg) / 2 /* length expression */ + ) + ) +) +~~~ + +As you can see, having a custom argument expression for each field +makes tracepoints very flexible for tracing a user space C application. +This tracepoint definition is reused later in this guide, when +actually using tracepoints in a user space application. diff --git a/contents/using-lttng/instrumenting/c-application/intro.md b/contents/using-lttng/instrumenting/c-application/intro.md new file mode 100644 index 0000000..ffafac9 --- /dev/null +++ b/contents/using-lttng/instrumenting/c-application/intro.md @@ -0,0 +1,27 @@ +--- +id: c-application +--- + +Instrumenting a C (or C++) application, be it an executable program or +a library, implies using LTTng-UST, the +user space tracing component of LTTng. For C/C++ applications, the +LTTng-UST package includes a dynamically loaded library +(`liblttng-ust`), C headers and the `lttng-gen-tp` command line utility. + +Since C and C++ are the base languages of virtually all other +programming languages +(Java virtual machine, Python, Perl, PHP and Node.js interpreters, etc.), +implementing user space tracing for an unsupported language is just a +matter of using the LTTng-UST C API at the right places. + +The usual work flow to instrument a user space C application with +LTTng-UST is: + + 1. Define tracepoints (actual probes) + 2. Write tracepoint providers + 3. Insert tracepoints into target source code + 4. Package (build) tracepoint providers + 5. Build user application and link it with tracepoint providers + +The steps above are discussed in greater detail in the following +subsections. diff --git a/contents/using-lttng/instrumenting/c-application/lttng-gen-tp.md b/contents/using-lttng/instrumenting/c-application/lttng-gen-tp.md new file mode 100644 index 0000000..78f4dfa --- /dev/null +++ b/contents/using-lttng/instrumenting/c-application/lttng-gen-tp.md @@ -0,0 +1,27 @@ +--- +id: lttng-gen-tp +--- + +LTTng-UST ships with `lttng-gen-tp`, a handy command line utility for +generating most of the stuff discussed above. It takes a _template file_, +with a name usually ending with the `.tp` extension, containing only +tracepoint definitions, and outputs a tracepoint provider (either a C +source file or a precompiled object file) with its header file. + +`lttng-gen-tp` should suffice in [static linking](#doc-static-linking) +situations. When using it, write a template file containing a list of +`TRACEPOINT_EVENT()` macro calls. The tool will find the provider names +used and generate the appropriate files which are going to look a lot +like `tp.h` and `tp.c` above. + +Just call `lttng-gen-tp` like this: + +
+lttng-gen-tp my-template.tp
+
+ +`my-template.c`, `my-template.o` and `my-template.h` will be created +in the same directory. + +For more information on `lttng-gen-tp`, see +its manpage. diff --git a/contents/using-lttng/instrumenting/c-application/lttng-ust-environment-variables-compiler-flags.md b/contents/using-lttng/instrumenting/c-application/lttng-ust-environment-variables-compiler-flags.md new file mode 100644 index 0000000..d64662d --- /dev/null +++ b/contents/using-lttng/instrumenting/c-application/lttng-ust-environment-variables-compiler-flags.md @@ -0,0 +1,28 @@ +--- +id: lttng-ust-environment-variables-compiler-flags +--- + +A few special environment variables and compile flags may affect the +behavior of LTTng-UST. + +LTTng-UST's debugging can be activated by setting the environment +variable `LTTNG_UST_DEBUG` to `1` when launching the application. It +can also be enabled at compile time by defining `LTTNG_UST_DEBUG` when +compiling LTTng-UST (using the `-DLTTNG_UST_DEBUG` compiler option). + +The environment variable `LTTNG_UST_REGISTER_TIMEOUT` can be used to +specify how long the application should wait for the +[session daemon](#doc-lttng-sessiond)'s _registration done_ command +before proceeding to execute the main program. The timeout value is +specified in milliseconds. 0 means _don't wait_. -1 means +_wait forever_. Setting this environment variable to 0 is recommended +for applications with time contraints on the process startup time. + +The default value of `LTTNG_UST_REGISTER_TIMEOUT` (when not defined) +is **3000 ms**. + +The compilation definition `LTTNG_UST_DEBUG_VALGRIND` should be enabled +at build time (`-DLTTNG_UST_DEBUG_VALGRIND`) to allow `liblttng-ust` +to be used with Valgrind. +The side effect of defining `LTTNG_UST_DEBUG_VALGRIND` is that per-CPU +buffering is disabled. diff --git a/contents/using-lttng/instrumenting/c-application/probing-the-application-source-code.md b/contents/using-lttng/instrumenting/c-application/probing-the-application-source-code.md new file mode 100644 index 0000000..89cf267 --- /dev/null +++ b/contents/using-lttng/instrumenting/c-application/probing-the-application-source-code.md @@ -0,0 +1,126 @@ +--- +id: probing-the-application-source-code +--- + +Once tracepoints are properly defined within a tracepoint provider, +they may be inserted into the user application to be instrumented +using the `tracepoint()` macro. Its first argument is the tracepoint +provider name and its second is the tracepoint name. The next, optional +arguments are defined by the `TP_ARGS()` part of the definition of +the tracepoint to use. + +As an example, let us again take the following tracepoint definition: + +~~~ c +TRACEPOINT_EVENT( + /* tracepoint provider name */ + my_provider, + + /* tracepoint/event name */ + my_first_tracepoint, + + /* list of tracepoint arguments */ + TP_ARGS( + int, my_integer_arg, + char*, my_string_arg + ), + + /* list of fields of eventual event */ + TP_FIELDS( + ctf_string(my_string_field, my_string_arg) + ctf_integer(int, my_integer_field, my_integer_arg) + ) +) +~~~ + +Assuming this is part of a file named `tp.h` which defines the tracepoint +provider and which is included by `tp.c`, here's a complete C application +calling this tracepoint (multiple times): + +~~~ c +#define TRACEPOINT_DEFINE +#include "tp.h" + +int main(int argc, char* argv[]) +{ + int i; + + tracepoint(my_provider, my_first_tracepoint, 23, "Hello, World!"); + + for (i = 0; i < argc; ++i) { + tracepoint(my_provider, my_first_tracepoint, i, argv[i]); + } + + return 0; +} +~~~ + +`TRACEPOINT_DEFINE` must be defined into exactly one translation unit (C +source file) of the user application, before including the tracepoint provider +header file. `TRACEPOINT_DEFINE` is discussed further in +[Building/linking tracepoint providers and the user application](#doc-building-tracepoint-providers-and-user-application). + +As another example, remember this definition we wrote in a previous +section (comments are stripped): + +~~~ c +/* for struct stat */ +#include +#include +#include + +TRACEPOINT_EVENT( + my_provider, + my_tracepoint, + TP_ARGS( + int, my_int_arg, + char*, my_str_arg, + struct stat*, st + ), + TP_FIELDS( + ctf_integer(int, my_constant_field, 23 + 17) + ctf_integer(int, my_int_arg_field, my_int_arg) + ctf_integer(int, my_int_arg_field2, my_int_arg * my_int_arg) + ctf_integer(int, sum4_field, my_str_arg[0] + my_str_arg[1] + + my_str_arg[2] + my_str_arg[3]) + ctf_string(my_str_arg_field, my_str_arg) + ctf_integer_hex(off_t, size_field, st->st_size) + ctf_float(double, size_dbl_field, (double) st->st_size) + ctf_sequence_text(char, half_my_str_arg_field, my_str_arg, + size_t, strlen(my_str_arg) / 2) + ) +) +~~~ + +Here's an example of calling it: + +~~~ c +#define TRACEPOINT_DEFINE +#include "tp.h" + +int main(void) +{ + struct stat s; + + stat("/etc/fstab", &s); + + tracepoint(my_provider, my_tracepoint, 23, "Hello, World!", &s); + + return 0; +} +~~~ + +When viewing the trace, assuming the file size of `/etc/fstab` is +301 bytes, the event generated by the execution of this tracepoint +should have the following fields, in this order: + +~~~ text +my_constant_field 40 +my_int_arg_field 23 +my_int_arg_field2 529 +sum4_field 389 +my_str_arg_field "Hello, World!" +size_field 0x12d +size_dbl_field 301.0 +half_my_str_arg_field "Hello," +~~~ diff --git a/contents/using-lttng/instrumenting/c-application/tracef.md b/contents/using-lttng/instrumenting/c-application/tracef.md new file mode 100644 index 0000000..b4a9428 --- /dev/null +++ b/contents/using-lttng/instrumenting/c-application/tracef.md @@ -0,0 +1,60 @@ +--- +id: tracef +--- + +`tracef()` is a small LTTng-UST API to avoid defining your own +tracepoints and tracepoint providers. The signature of `tracef()` is +the same as `printf()`'s. + +The `tracef()` utility function was developed to make user space tracing +super simple, albeit with notable disadvantages compared to custom, +full-fledged tracepoint providers: + + * All generated events have the same provider/event names, respectively + `lttng-ust-tracef` and `event`. + * There's no static type checking. + * The only event field you actually get, named `msg`, is a string + potentially containing the values you passed to the function + using your own format. This also means that you cannot use filtering + using a custom expression at runtime because there are no isolated + fields. + * Since `tracef()` uses C standard library's `vasprintf()` function + in the background to format the strings at runtime, its + expected performance is lower than using custom tracepoint providers + with typed fields, which do not require a conversion to a string. + +Thus, `tracef()` is useful for quick prototyping and debugging, but +should not be considered for any permanent/serious application +instrumentation. + +To use `tracef()`, first include `` in the C source file +where you need to insert probes: + +~~~ c +#include +~~~ + +Use `tracef()` like you would use `printf()` in your source code, e.g.: + +~~~ c + /* ... */ + + tracef("my message, my integer: %d", my_integer); + + /* ... */ +~~~ + +Link your application with `liblttng-ust`: + +
+gcc -o app app.c -llttng-ust
+
+ +Execute the application as usual: + +
+./app
+
+ +Voilà! Use the `lttng` command line tool to +[control tracing](#doc-controlling-tracing). diff --git a/contents/using-lttng/instrumenting/c-application/tracepoint-provider.md b/contents/using-lttng/instrumenting/c-application/tracepoint-provider.md new file mode 100644 index 0000000..3a68abe --- /dev/null +++ b/contents/using-lttng/instrumenting/c-application/tracepoint-provider.md @@ -0,0 +1,201 @@ +--- +id: tracepoint-provider +--- + +Before jumping into defining tracepoints and inserting +them into the application source code, you must understand what a +_tracepoint provider_ is. + +For the sake of this guide, consider the following two files: + +`tp.h`: + +~~~ c +#undef TRACEPOINT_PROVIDER +#define TRACEPOINT_PROVIDER my_provider + +#undef TRACEPOINT_INCLUDE +#define TRACEPOINT_INCLUDE "./tp.h" + +#if !defined(_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ) +#define _TP_H + +#include + +TRACEPOINT_EVENT( + my_provider, + my_first_tracepoint, + TP_ARGS( + int, my_integer_arg, + char*, my_string_arg + ), + TP_FIELDS( + ctf_string(my_string_field, my_string_arg) + ctf_integer(int, my_integer_field, my_integer_arg) + ) +) + +TRACEPOINT_EVENT( + my_provider, + my_other_tracepoint, + TP_ARGS( + int, my_int + ), + TP_FIELDS( + ctf_integer(int, some_field, my_int) + ) +) + +#endif /* _TP_H */ + +#include +~~~ + +`tp.c`: + +~~~ c +#define TRACEPOINT_CREATE_PROBES + +#include "tp.h" +~~~ + +The two files above are defining a _tracepoint provider_. A tracepoint +provider is some sort of namespace for _tracepoint definitions_. Tracepoint +definitions are written above with the `TRACEPOINT_EVENT()` macro, and allow +eventual `tracepoint()` calls respecting their definitions to be inserted +into the user application's C source code (we explore this in a +later section). + +Many tracepoint definitions may be part of the same tracepoint provider +and many tracepoint providers may coexist in a user space application. A +tracepoint provider is packaged either: + + * directly into an existing user application's C source file + * as an object file + * as a static library + * as a shared library + +The two files above, `tp.h` and `tp.c`, show a typical template for +writing a tracepoint provider. LTTng-UST was designed so that two +tracepoint providers should not be defined in the same header file. + +We will now go through the various parts of the above files and +give them a meaning. As you may have noticed, the LTTng-UST API for +C/C++ applications is some preprocessor sorcery. The LTTng-UST macros +used in your application and those in the LTTng-UST headers are +combined to produce actual source code needed to make tracing possible +using LTTng. + +Let's start with the header file, `tp.h`. It begins with + +~~~ c +#undef TRACEPOINT_PROVIDER +#define TRACEPOINT_PROVIDER my_provider +~~~ + +`TRACEPOINT_PROVIDER` defines the name of the provider to which the +following tracepoint definitions will belong. It is used internally by +LTTng-UST headers and _must_ be defined. Since `TRACEPOINT_PROVIDER` +could have been defined by another header file also included by the same +C source file, the best practice is to undefine it first. + +
+

Note:Names in LTTng-UST follow the C +identifier syntax (starting with a letter and containing either +letters, numbers or underscores); they are not C strings +(not surrounded by double quotes). This is because LTTng-UST macros +use those identifier-like strings to create symbols (named types and +variables).

+
+ +The tracepoint provider is a group of tracepoint definitions; its chosen +name should reflect this. A hierarchy like Java packages is recommended, +using underscores instead of dots, e.g., `org_company_project_component`. + +Next is `TRACEPOINT_INCLUDE`: + +~~~ c +#undef TRACEPOINT_INCLUDE +#define TRACEPOINT_INCLUDE "./tp.h" +~~~ + +This little bit of instrospection is needed by LTTng-UST to include +your header at various predefined places. + +Include guard follows: + +~~~ c +#if !defined(_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ) +#define _TP_H +~~~ + +Add this precompiler conditionals to ensure the tracepoint event generation +can include this file more than once. + +The `TRACEPOINT_EVENT()` macro is defined in a LTTng-UST header file which +must be included: + +~~~ c +#include +~~~ + +This will also allow the application to use the `tracepoint()` macro. + +Next is a list of `TRACEPOINT_EVENT()` macro calls which create the +actual tracepoint definitions. We will skip this for the moment and +come back to how to use `TRACEPOINT_EVENT()` +[in a later section](#doc-defining-tracepoints). Just pay attention to +the first argument: it's always the name of the tracepoint provider +being defined in this header file. + +End of include guard: + +~~~ c +#endif /* _TP_H */ +~~~ + +Finally, include `` to expand the macros: + +~~~ c +#include +~~~ + +That's it for `tp.h`. Of course, this is only a header file; it must be +included in some C source file to actually use it. This is the job of +`tp.c`: + +~~~ c +#define TRACEPOINT_CREATE_PROBES + +#include "tp.h" +~~~ + +When `TRACEPOINT_CREATE_PROBES` is defined, the macros used in `tp.h`, +which is included just after, will actually create the source code for +LTTng-UST probes (global data structures and functions) out of your +tracepoint definitions. How exactly this is done is out of this text's scope. +`TRACEPOINT_CREATE_PROBES` is discussed further +in [Building/linking tracepoint providers and the user application](#doc-building-tracepoint-providers-and-user-application). + +You could include other header files like `tp.h` here to create the probes +of different tracepoint providers, e.g.: + +~~~ c +#define TRACEPOINT_CREATE_PROBES + +#include "tp1.h" +#include "tp2.h" +~~~ + +The rule is: probes of a given tracepoint provider +must be created in exactly one source file. This source file could be one +of your project's; it doesn't have to be on its own like `tp.c`, although +[a later section](#doc-building-tracepoint-providers-and-user-application) +shows that doing so allows packaging the tracepoint providers +independently and keep them out of your application, also making it +possible to reuse them between projects. + +The following sections explain how to define tracepoints, how to use the +`tracepoint()` macro to instrument your user space C application and how +to build/link tracepoint providers and your application with LTTng-UST +support. diff --git a/contents/using-lttng/instrumenting/cxx-application.md b/contents/using-lttng/instrumenting/cxx-application.md new file mode 100644 index 0000000..786388c --- /dev/null +++ b/contents/using-lttng/instrumenting/cxx-application.md @@ -0,0 +1,12 @@ +--- +id: cxx-application +--- + +Because of C++'s cross-compatibility with the C language, C++ +applications can be readily instrumented with the LTTng-UST C API. + +Follow the [C application](#doc-c-application) user guide above. It +should be noted that, in this case, tracepoint providers should have +the typical `.cpp`, `.cxx` or `.cc` extension and be built with `g++` +instead of `gcc`. This is the easiest way of avoiding linking errors +due to symbol name mangling incompatibilities between both languages. diff --git a/contents/using-lttng/instrumenting/instrumenting-linux-kernel/instrumenting-linux-kernel-itself/instrumenting-linux-kernel-tracing.md b/contents/using-lttng/instrumenting/instrumenting-linux-kernel/instrumenting-linux-kernel-itself/instrumenting-linux-kernel-tracing.md new file mode 100644 index 0000000..9977cf9 --- /dev/null +++ b/contents/using-lttng/instrumenting/instrumenting-linux-kernel/instrumenting-linux-kernel-itself/instrumenting-linux-kernel-tracing.md @@ -0,0 +1,72 @@ +--- +id: instrumenting-linux-kernel-tracing +--- + +The [Controlling tracing](#doc-controlling-tracing) section explains +how to use the `lttng` tool to create and control tracing sessions. +Although the `lttng` tool will load the appropriate _known_ LTTng kernel +modules when needed (by launching `root`'s session daemon), it won't +load your custom `linux-probe-hello` module by default. You need to +manually start an LTTng session daemon as `root` and use the +`--extra-kmod-probes` option to append your custom probe module to the +default list: + +
+sudo pkill -u root lttng-sessiond
+sudo lttng-sessiond --extra-kmod-probes=hello
+
+ +The first command makes sure any existing instance is killed. If +you're not interested in using the default probes, or if you only +want to use a few of them, you could use `--kmod-probes` instead, +which specifies an absolute list: + +
+sudo lttng-sessiond --kmod-probes=ext4,sched,hello
+
+ +Confirm the custom probe module is loaded: + +
+lsmod | grep lttng_probe_hello
+
+ +The `hello_world` event should appear in the list when doing + +
+lttng list --kernel | grep hello
+
+ +You may now create an LTTng tracing session, enable the `hello_world` +kernel event (and others if you wish) and start tracing: + +
+sudo lttng create my-session
+sudo lttng enable-event --kernel hello_world
+sudo lttng start
+
+ +Plug a few USB devices, then stop tracing and inspect the trace (if +Babeltrace +is installed): + +
+sudo lttng stop
+sudo lttng view
+
+ +Here's a sample output: + +~~~ text +[15:30:34.835895035] (+?.?????????) hostname hello_world: { cpu_id = 1 }, { my_int = 8, char0 = 68, char1 = 97, product = "DataTraveler 2.0" } +[15:30:42.262781421] (+7.426886386) hostname hello_world: { cpu_id = 1 }, { my_int = 9, char0 = 80, char1 = 97, product = "Patriot Memory" } +[15:30:48.175621778] (+5.912840357) hostname hello_world: { cpu_id = 1 }, { my_int = 10, char0 = 68, char1 = 97, product = "DataTraveler 2.0" } +~~~ + +Two USB flash drives were used for this test. + +You may change your LTTng custom probe, rebuild it and reload it at +any time when not tracing. Make sure you remove the old module +(either by killing the root LTTng session daemon which loaded the +module in the first place, or by using `modprobe --remove` directly) +before loading the updated one. diff --git a/contents/using-lttng/instrumenting/instrumenting-linux-kernel/instrumenting-linux-kernel-itself/intro.md b/contents/using-lttng/instrumenting/instrumenting-linux-kernel/instrumenting-linux-kernel-itself/intro.md new file mode 100644 index 0000000..27b6b0c --- /dev/null +++ b/contents/using-lttng/instrumenting/instrumenting-linux-kernel/instrumenting-linux-kernel-itself/intro.md @@ -0,0 +1,21 @@ +--- +id: instrumenting-linux-kernel-itself +--- + +This section explains strictly how to add custom LTTng +instrumentation to the Linux kernel. It does not explain how the +macros actually work and the internal mechanics of the tracer. + +You should have a Linux kernel source code tree to work with. +Throughout this section, all file paths are relative to the root of +this tree unless otherwise stated. + +You will need a copy of the LTTng-modules Git repository: + +
+git clone git://git.lttng.org/lttng-modules.git
+
+ +The steps to add custom LTTng instrumentation to a Linux kernel +involves defining and using the mainline `TRACE_EVENT()` tracepoints +first, then writing and using the LTTng adaptation layer. diff --git a/contents/using-lttng/instrumenting/instrumenting-linux-kernel/instrumenting-linux-kernel-itself/lttng-adaptation-layer.md b/contents/using-lttng/instrumenting/instrumenting-linux-kernel/instrumenting-linux-kernel-itself/lttng-adaptation-layer.md new file mode 100644 index 0000000..1c117e0 --- /dev/null +++ b/contents/using-lttng/instrumenting/instrumenting-linux-kernel/instrumenting-linux-kernel-itself/lttng-adaptation-layer.md @@ -0,0 +1,140 @@ +--- +id: lttng-adaptation-layer +--- + +The steps to write the LTTng adaptation layer are, in your +LTTng-modules copy's source code tree: + + 1. In `instrumentation/events/lttng-module`, + add a header subsys.h for your custom + subsystem subsys and write your + tracepoint definitions using LTTng-modules macros in it. + Those macros look like the mainline kernel equivalents, + but they present subtle, yet important differences. + 2. In `probes`, create the C source file of the LTTng probe kernel + module for your subsystem. It should be named + lttng-probe-subsys.c. + 3. Edit `probes/Makefile` so that the LTTng-modules project + builds your custom LTTng probe kernel module. + 4. Build and install LTTng kernel modules. + +Following our `hello_world` event example, here's the content of +`instrumentation/events/lttng-module/hello.h`: + +~~~ c +#undef TRACE_SYSTEM +#define TRACE_SYSTEM hello + +#if !defined(_TRACE_HELLO_H) || defined(TRACE_HEADER_MULTI_READ) +#define _TRACE_HELLO_H + +#include + +LTTNG_TRACEPOINT_EVENT( + /* format identical to mainline version for those */ + hello_world, + TP_PROTO(int foo, const char* bar), + TP_ARGS(foo, bar), + + /* possible differences */ + TP_STRUCT__entry( + __field(int, my_int) + __field(char, char0) + __field(char, char1) + __string(product, bar) + ), + + /* notice the use of tp_assign()/tp_strcpy() and no semicolons */ + TP_fast_assign( + tp_assign(my_int, foo) + tp_assign(char0, bar[0]) + tp_assign(char1, bar[1]) + tp_strcpy(product, bar) + ), + + /* This one is actually not used by LTTng either, but must be + * present for the moment. + */ + TP_printk("", 0) + +/* no semicolon after this either */ +) + +#endif + +/* other difference: do NOT include */ +#include "../../../probes/define_trace.h" +~~~ + +Some possible entries for `TP_STRUCT__entry()` and `TP_fast_assign()`, +in the case of LTTng-modules, are shown in the +[LTTng-modules reference](#doc-lttng-modules-ref) section. + +The best way to learn how to use the above macros is to inspect +existing LTTng tracepoint definitions in `instrumentation/events/lttng-module` +header files. Compare them with the Linux kernel mainline versions +in `include/trace/events`. + +The next step is writing the LTTng probe kernel module C source file. +This one is named lttng-probe-subsys.c +in `probes`. You may always use the following template: + +~~~ c +#include +#include "../lttng-tracer.h" + +/* Build time verification of mismatch between mainline TRACE_EVENT() + * arguments and LTTng adaptation layer LTTNG_TRACEPOINT_EVENT() arguments. + */ +#include + +/* create LTTng tracepoint probes */ +#define LTTNG_PACKAGE_BUILD +#define CREATE_TRACE_POINTS +#define TRACE_INCLUDE_PATH ../instrumentation/events/lttng-module + +#include "../instrumentation/events/lttng-module/hello.h" + +MODULE_LICENSE("GPL and additional rights"); +MODULE_AUTHOR("Your name "); +MODULE_DESCRIPTION("LTTng hello probes"); +MODULE_VERSION(__stringify(LTTNG_MODULES_MAJOR_VERSION) "." + __stringify(LTTNG_MODULES_MINOR_VERSION) "." + __stringify(LTTNG_MODULES_PATCHLEVEL_VERSION) + LTTNG_MODULES_EXTRAVERSION); +~~~ + +Just replace `hello` with your subsystem name. In this example, +``, which is the original mainline tracepoint +definition header, is included for verification purposes: the +LTTng-modules build system is able to emit an error at build time when +the arguments of the mainline `TRACE_EVENT()` definitions do not match +the ones of the LTTng-modules adaptation layer +(`LTTNG_TRACEPOINT_EVENT()`). + +Edit `probes/Makefile` and add your new kernel module object +next to existing ones: + +~~~ makefile +# ... + +obj-m += lttng-probe-module.o +obj-m += lttng-probe-power.o + +obj-m += lttng-probe-hello.o + +# ... +~~~ + +Time to build! Point to your custom Linux kernel source tree using +the `KERNELDIR` variable: + +
+make KERNELDIR=/path/to/custom/linux
+
+ +Finally, install modules: + +
+sudo make modules_install
+
diff --git a/contents/using-lttng/instrumenting/instrumenting-linux-kernel/instrumenting-linux-kernel-itself/mainline-trace-event.md b/contents/using-lttng/instrumenting/instrumenting-linux-kernel/instrumenting-linux-kernel-itself/mainline-trace-event.md new file mode 100644 index 0000000..773c16d --- /dev/null +++ b/contents/using-lttng/instrumenting/instrumenting-linux-kernel/instrumenting-linux-kernel-itself/mainline-trace-event.md @@ -0,0 +1,98 @@ +--- +id: mainline-trace-event +--- + +The first step is to define tracepoints using the mainline Linux +`TRACE_EVENT()` macro and insert tracepoints where you want them. +Your tracepoint definitions reside in a header file in +`include/trace/events`. If you're adding tracepoints to an existing +subsystem, edit its appropriate header file. + +As an example, the following header file (let's call it +`include/trace/events/hello.h`) defines one tracepoint using +`TRACE_EVENT()`: + +~~~ c +/* subsystem name is "hello" */ +#undef TRACE_SYSTEM +#define TRACE_SYSTEM hello + +#if !defined(_TRACE_HELLO_H) || defined(TRACE_HEADER_MULTI_READ) +#define _TRACE_HELLO_H + +#include + +TRACE_EVENT( + /* "hello" is the subsystem name, "world" is the event name */ + hello_world, + + /* tracepoint function prototype */ + TP_PROTO(int foo, const char* bar), + + /* arguments for this tracepoint */ + TP_ARGS(foo, bar), + + /* LTTng doesn't need those */ + TP_STRUCT__entry(), + TP_fast_assign(), + TP_printk("", 0) +); + +#endif + +/* this part must be outside protection */ +#include +~~~ + +Notice that we don't use any of the last three arguments: they +are left empty here because LTTng doesn't need them. You would only fill +`TP_STRUCT__entry()`, `TP_fast_assign()` and `TP_printk()` if you were +to also use this tracepoint for ftrace/perf. + +Once this is done, you may place calls to `trace_hello_world()` +wherever you want in the Linux source code. As an example, let us place +such a tracepoint in the `usb_probe_device()` static function +(`drivers/usb/core/driver.c`): + +~~~ c +/* called from driver core with dev locked */ +static int usb_probe_device(struct device *dev) +{ + struct usb_device_driver *udriver = to_usb_device_driver(dev->driver); + struct usb_device *udev = to_usb_device(dev); + int error = 0; + + trace_hello_world(udev->devnum, udev->product); + + /* ... */ +} +~~~ + +This tracepoint should fire every time a USB device is plugged in. + +At the top of `driver.c`, we need to include our actual tracepoint +definition and, in this case (one place per subsystem), define +`CREATE_TRACE_POINTS`, which will create our tracepoint: + +~~~ c +/* ... */ + +#include "usb.h" + +#define CREATE_TRACE_POINTS +#include + +/* ... */ +~~~ + +Build your custom Linux kernel. In order to use LTTng, make sure the +following kernel configuration options are enabled: + + * `CONFIG_MODULES` (loadable module support) + * `CONFIG_KALLSYMS` (load all symbols for debugging/kksymoops) + * `CONFIG_HIGH_RES_TIMERS` (high resolution timer support) + * `CONFIG_TRACEPOINTS` (kernel tracepoint instrumentation) + +Boot the custom kernel. The directory +`/sys/kernel/debug/tracing/events/hello` should exist if everything +went right, with a `hello_world` subdirectory. diff --git a/contents/using-lttng/instrumenting/instrumenting-linux-kernel/instrumenting-out-of-tree-linux-kernel.md b/contents/using-lttng/instrumenting/instrumenting-linux-kernel/instrumenting-out-of-tree-linux-kernel.md new file mode 100644 index 0000000..ce1ae11 --- /dev/null +++ b/contents/using-lttng/instrumenting/instrumenting-linux-kernel/instrumenting-out-of-tree-linux-kernel.md @@ -0,0 +1,46 @@ +--- +id: instrumenting-out-of-tree-linux-kernel +--- + +Instrumenting a custom Linux kernel module for LTTng follows the exact +same steps as +[adding instrumentation to the Linux kernel itself](#doc-instrumenting-linux-kernel-itself), +the only difference being that your mainline tracepoint definition +header doesn't reside in the mainline source tree, but in your +kernel module source tree. + +The only reference to this mainline header is in the LTTng custom +probe's source code (`probes/lttng-probe-hello.c` in our example), for +build time verification: + +~~~ c +/* ... */ + +/* Build time verification of mismatch between mainline TRACE_EVENT() + * arguments and LTTng adaptation layer LTTNG_TRACEPOINT_EVENT() arguments. + */ +#include + +/* ... */ +~~~ + +The preferred, flexible way to include your module's mainline +tracepoint definition header is to put it in a specific directory +relative to your module's root, e.g., `tracepoints`, and include it +relative to your module's root directory in the LTTng custom probe's +source: + +~~~ c +#include +~~~ + +You may then build LTTng-modules by adding your module's root +directory as an include path to the extra C flags: + +
+make ccflags-y=-I/path/to/kernel/module KERNELDIR=/path/to/custom/linux
+
+ +Using `ccflags-y` allows you to move your kernel module to another +directory and rebuild the LTTng-modules project with no change to +source files. diff --git a/contents/using-lttng/instrumenting/instrumenting-linux-kernel/intro.md b/contents/using-lttng/instrumenting/instrumenting-linux-kernel/intro.md new file mode 100644 index 0000000..a6fbc6d --- /dev/null +++ b/contents/using-lttng/instrumenting/instrumenting-linux-kernel/intro.md @@ -0,0 +1,42 @@ +--- +id: instrumenting-linux-kernel +--- + +The Linux kernel can be instrumented for LTTng tracing, either its core +source code or a kernel module. It has to be noted that Linux is +readily traceable using LTTng since many parts of its source code are +already instrumented: this is the job of the upstream +LTTng-modules +package. This section presents how to add LTTng instrumentation where it +does not currently exist and how to instrument custom kernel modules. + +All LTTng instrumentation in the Linux kernel is based on an existing +infrastructure which bears the name of its main macro, `TRACE_EVENT()`. +This macro is used to define tracepoints, +each tracepoint having a name, usually with the +subsys_name format, +subsys being the subsystem name and +name the specific event name. + +Tracepoints defined with `TRACE_EVENT()` may be inserted anywhere in +the Linux kernel source code, after what callbacks, called _probes_, +may be registered to execute some action when a tracepoint is +executed. This mechanism is directly used by ftrace and perf, +but cannot be used as is by LTTng: an adaptation layer is added to +satisfy LTTng's specific needs. + +With that in mind, this documentation does not cover the `TRACE_EVENT()` +format and how to use it, but it is mandatory to understand it and use +it to instrument Linux for LTTng. A series of +LWN articles explain +`TRACE_EVENT()` in details: +part 1, +part 2, and +part 3. +Once you master `TRACE_EVENT()` enough for your use case, continue +reading this section so that you can add the LTTng adaptation layer of +instrumentation. + +This section first discusses the general method of instrumenting the +Linux kernel for LTTng. This method is then reused for the specific +case of instrumenting a kernel module. diff --git a/contents/using-lttng/instrumenting/intro.md b/contents/using-lttng/instrumenting/intro.md new file mode 100644 index 0000000..e34bdc5 --- /dev/null +++ b/contents/using-lttng/instrumenting/intro.md @@ -0,0 +1,43 @@ +--- +id: instrumenting +--- + +There are many examples of tracing and monitoring in our everyday life. +You have access to real-time and historical weather reports and forecasts +thanks to weather stations installed around the country. You know your +possibly hospitalized friends' and family's hearts are safe thanks to +electrocardiography. You make sure not to drive your car too fast +and have enough fuel to reach your destination thanks to gauges visible +on your dashboard. + +All the previous examples have something in common: they rely on +**probes**. Without electrodes attached to the surface of a body's +skin, cardiac monitoring would be futile. + +LTTng, as a tracer, is no different from the real life examples above. +If you're about to trace a software system, i.e. record its history of +execution, you better have probes in the subject you're +tracing: the actual software. Various ways were developed to do this. +The most straightforward one is to manually place probes, called +_tracepoints_, in the software's source code. The Linux kernel tracing +domain also allows probes added dynamically. + +If you're only interested in tracing the Linux kernel, it may very well +be that your tracing needs are already appropriately covered by LTTng's +built-in Linux kernel tracepoints and other probes. Or you may be in +possession of a user space application which has already been +instrumented. In such cases, the work will reside entirely in the design +and execution of tracing sessions, allowing you to jump to +[Controlling tracing](#doc-controlling-tracing) right now. + +This section focuses on the following use cases of instrumentation: + + * [C](#doc-c-application) and [C++](#doc-cxx-application) applications + * [prebuilt user space tracing helpers](#doc-prebuilt-ust-helpers) + * [Java application](#doc-java-application) + * [Linux kernel](#doc-instrumenting-linux-kernel) module or the + kernel itself + * the [`/proc/lttng-logger` ABI](#doc-proc-lttng-logger-abi) + +Some [advanced techniques](#doc-advanced-instrumenting-techniques) are +also presented at the very end. diff --git a/contents/using-lttng/instrumenting/java-application.md b/contents/using-lttng/instrumenting/java-application.md new file mode 100644 index 0000000..5180980 --- /dev/null +++ b/contents/using-lttng/instrumenting/java-application.md @@ -0,0 +1,73 @@ +--- +id: java-application +--- + +LTTng-UST provides a _logging_ back-end for Java applications using +Java Util Logging (JUL). This back-end is called the _LTTng-UST JUL agent_ and is +responsible for communications with an LTTng session daemon. + +From the user's point of view, once the LTTng-UST JUL agent has been +initialized, JUL loggers may be created and used as usual. The agent +adds its own handler to the _root logger_, so that all loggers may +generate LTTng events with no effort. + +Common JUL features are supported using the `lttng` tool +(see [Controlling tracing](#doc-controlling-tracing)): + + * listing all logger names + * enabling/disabling events per logger name + * JUL log levels + +Here's an example: + +~~~ java +import java.util.logging.Logger; +import org.lttng.ust.jul.LTTngAgent; + +public class Test +{ + public static void main(String[] argv) throws Exception + { + // create a logger + Logger logger = Logger.getLogger("jello"); + + // call this as soon as possible (before logging) + LTTngAgent lttngAgent = LTTngAgent.getLTTngAgent(); + + // log at will! + logger.info("some info"); + logger.warning("some warning"); + Thread.sleep(500); + logger.finer("finer information..."); + Thread.sleep(123); + logger.severe("error!"); + + // not mandatory, but cleaner + lttngAgent.dispose(); + } +} +~~~ + +The LTTng-UST JUL agent Java classes are packaged in a JAR file named +`liblttng-ust-jul.jar`. It is typically located in +`/usr/lib/lttng/java`. To compile the snippet above +(saved as `Test.java`), do: + +
+javac -cp /usr/lib/lttng/java/liblttng-ust-jul.jar Test.java
+
+ +You can run the resulting compiled class: + +
+java -cp /usr/lib/lttng/java/liblttng-ust-jul.jar:. Test
+
+ +
+

+ Note:OpenJDK 7 + is used for development and continuous integration, thus this + version is directly supported. However, the LTTng-UST JUL agent has + also been tested with OpenJDK 6. +

+
diff --git a/contents/using-lttng/instrumenting/prebuilt-ust-helpers/intro.md b/contents/using-lttng/instrumenting/prebuilt-ust-helpers/intro.md new file mode 100644 index 0000000..e9fcc28 --- /dev/null +++ b/contents/using-lttng/instrumenting/prebuilt-ust-helpers/intro.md @@ -0,0 +1,24 @@ +--- +id: prebuilt-ust-helpers +--- + +The LTTng-UST package provides a few helpers that one may find +useful in some situations. They all work the same way: you must +preload the appropriate shared object before running the user +application (using the `LD_PRELOAD` environment variable). + +The shared objects are normally found in `/usr/lib`. + +The current installed helpers are: + + * `liblttng-ust-libc-wrapper.so` and + `liblttng-ust-pthread-wrapper.so`: + [C standard library and POSIX threads tracing](#doc-liblttng‑ust‑libc‑pthread-wrapper) + * `liblttng-ust-cyg-profile.so` and + `liblttng-ust-cyg-profile-fast.so`: + [function tracing](#doc-liblttng‑ust‑cyg‑profile) + * `liblttng-ust-dl.so`: + [dynamic linker tracing](#doc-liblttng‑ust‑dl) + +The following subsections document what helpers instrument exactly +and how to use them. diff --git a/contents/using-lttng/instrumenting/prebuilt-ust-helpers/liblttng-ust-cyg-profile.md b/contents/using-lttng/instrumenting/prebuilt-ust-helpers/liblttng-ust-cyg-profile.md new file mode 100644 index 0000000..078eb16 --- /dev/null +++ b/contents/using-lttng/instrumenting/prebuilt-ust-helpers/liblttng-ust-cyg-profile.md @@ -0,0 +1,169 @@ +--- +id: liblttng‑ust‑cyg‑profile +--- + +Function tracing is the recording of which functions are entered and +left during the execution of an application. Like with any LTTng event, +the precise time at which this happens is also kept. + +GCC and clang have an option named +-finstrument-functions +which generates instrumentation calls for entry and exit to functions. +The LTTng-UST function tracing helpers, `liblttng-ust-cyg-profile.so` +and `liblttng-ust-cyg-profile-fast.so`, take advantage of this feature +to add instrumentation to the two generated functions (which contain +`cyg_profile` in their names, hence the shared object's name). + +In order to use LTTng-UST function tracing, the translation units to +instrument must be built using the `-finstrument-functions` compiler +flag. + +LTTng-UST function tracing comes in two flavors, each providing +different trade-offs: `liblttng-ust-cyg-profile-fast.so` and +`liblttng-ust-cyg-profile.so`. + +**`liblttng-ust-cyg-profile-fast.so`** is a lightweight variant that +should only be used where it can be _guaranteed_ that the complete event +stream is recorded without any missing events. Any kind of duplicate +information is left out. This version registers the following +tracepoints: + + + + + + + + + + + + + + + + + + + + +
TP provider nameTP nameDescription/fields
+ lttng_ust_cyg_profile_fast + + func_entry + +

Function entry

+ +
    +
  • + addr address of the + called function +
    • +
+
+ func_exit + +

Function exit

+
+ +Assuming no event is lost, having only the function addresses on entry +is enough for creating a call graph (remember that a recorded event +always contains the ID of the CPU that generated it). A tool like +addr2line +may be used to convert function addresses back to source files names +and line numbers. + +The other helper, +**`liblttng-ust-cyg-profile.so`**, +is a more robust variant which also works for use cases where +events might get discarded or not recorded from application startup. +In these cases, the trace analyzer needs extra information to be +able to reconstruct the program flow. This version registers the +following tracepoints: + + + + + + + + + + + + + + + + + + + + +
TP provider nameTP nameDescription/fields
+ lttng_ust_cyg_profile + + func_entry + +

Function entry

+ +
    +
  • + addr address of the + called function +
    • +
  • + call_site call site + address +
    • +
+
+ func_exit + +

Function exit

+ +
    +
  • + addr address of the + called function +
    • +
  • + call_site call site + address +
    • +
+
+ +To use one or the other variant with any user application, assuming at +least one translation unit of the latter is compiled with the +`-finstrument-functions` option, do: + +
+LD_PRELOAD=liblttng-ust-cyg-profile-fast.so my-app
+
+ +or + +
+LD_PRELOAD=liblttng-ust-cyg-profile.so my-app
+
+ +It might be necessary to limit the number of source files where +`-finstrument-functions` is used to prevent excessive amount of trace +data to be generated at runtime. + +
+

+ Tip: When using GCC, at least, you may use + the + -finstrument-functions-exclude-function-list + option to avoid instrumenting entries and exits of specific + symbol names. +

+
+ +All events generated from LTTng-UST function tracing are provided on +log level `TRACE_DEBUG_FUNCTION`, which is useful to easily enable +function tracing events in your tracing session using the +`--loglevel-only` option of `lttng enable-event` +(see [Controlling tracing](#doc-controlling-tracing)). diff --git a/contents/using-lttng/instrumenting/prebuilt-ust-helpers/liblttng-ust-dl.md b/contents/using-lttng/instrumenting/prebuilt-ust-helpers/liblttng-ust-dl.md new file mode 100644 index 0000000..2274119 --- /dev/null +++ b/contents/using-lttng/instrumenting/prebuilt-ust-helpers/liblttng-ust-dl.md @@ -0,0 +1,81 @@ +--- +id: liblttng‑ust‑dl +--- + +This LTTng-UST helper causes all calls to `dlopen()` and `dlclose()` +in the target application to be traced with LTTng. + +The helper's shared object, `liblttng-ust-dl.so`, registers the +following tracepoints when preloaded: + + + + + + + + + + + + + + + + + + + + +
TP provider nameTP nameDescription/fields
+ ust_baddr + + push + +

dlopen() call

+ +
    +
  • + baddr memory + base address + (where the dynamic linker placed the shared + object) +
    • +
  • + sopath file system + path to the loaded shared object +
    • +
  • + size file size + of the the loaded shared object +
    • +
  • + mtime last + modification time (seconds since Epoch time) + of the loaded shared object +
    • +
+
+ pop + +

dlclose() call

+ +
    +
  • + baddr memory + base address +
    • +
+
+ +To use this LTTng-UST helper with any user application, independently of +how the latter is built, do: + +
+LD_PRELOAD=liblttng-ust-dl.so my-app
+
+ +Of course, like any other tracepoint, the ones above need to be enabled +in order for LTTng-UST to generate events. This is done using the +`lttng` command line tool +(see [Controlling tracing](#doc-controlling-tracing)). diff --git a/contents/using-lttng/instrumenting/prebuilt-ust-helpers/liblttng-ust-libc-pthread-wrapper.md b/contents/using-lttng/instrumenting/prebuilt-ust-helpers/liblttng-ust-libc-pthread-wrapper.md new file mode 100644 index 0000000..a36f860 --- /dev/null +++ b/contents/using-lttng/instrumenting/prebuilt-ust-helpers/liblttng-ust-libc-pthread-wrapper.md @@ -0,0 +1,153 @@ +--- +id: liblttng‑ust‑libc‑pthread-wrapper +--- + +`liblttng-ust-libc-wrapper.so` and `liblttng-ust-pthread-wrapper.so` +can add instrumentation to respectively some C standard library and +POSIX threads functions. + +The following functions are traceable by `liblttng-ust-libc-wrapper.so`: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TP provider nameTP nameInstrumented function
+ ust_libc + + malloc + + malloc() +
+ calloc + + calloc() +
+ realloc + + realloc() +
+ free + + free() +
+ memalign + + memalign() +
+ posix_memalign + + posix_memalign() +
+ +The following functions are traceable by +`liblttng-ust-pthread-wrapper.so`: + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TP provider nameTP nameInstrumented function
+ ust_pthread + + pthread_mutex_lock_req + + pthread_mutex_lock() (request time) +
+ pthread_mutex_lock_acq + + pthread_mutex_lock() (acquire time) +
+ pthread_mutex_trylock + + pthread_mutex_trylock() +
+ pthread_mutex_unlock + + pthread_mutex_unlock() +
+ +All tracepoints have fields corresponding to the arguments of the +function they instrument. + +To use one or the other with any user application, independently of +how the latter is built, do: + +
+LD_PRELOAD=liblttng-ust-libc-wrapper.so my-app
+
+ +or + +
+LD_PRELOAD=liblttng-ust-pthread-wrapper.so my-app
+
+ +To use both, do: + +
+LD_PRELOAD="liblttng-ust-libc-wrapper.so liblttng-ust-pthread-wrapper.so" my-app
+
+ +When the shared object is preloaded, it effectively replaces the +functions listed in the above tables by wrappers which add tracepoints +and call the replaced functions. + +Of course, like any other tracepoint, the ones above need to be enabled +in order for LTTng-UST to generate events. This is done using the +`lttng` command line tool +(see [Controlling tracing](#doc-controlling-tracing)). diff --git a/contents/using-lttng/instrumenting/proc-lttng-logger-abi.md b/contents/using-lttng/instrumenting/proc-lttng-logger-abi.md new file mode 100644 index 0000000..2a7ba0f --- /dev/null +++ b/contents/using-lttng/instrumenting/proc-lttng-logger-abi.md @@ -0,0 +1,36 @@ +--- +id: proc-lttng-logger-abi +--- + +The `lttng-tracer` Linux kernel module, installed by the LTTng-modules +package, creates a special LTTng logger ABI file `/proc/lttng-logger` +when loaded. Writing text data to this file generates an LTTng kernel +domain event named `lttng_logger`. + +Unlike other kernel domain events, `lttng_logger` may be enabled by +any user, not only root users or members of the tracing group. + +To use the LTTng logger ABI, simply write a string to +`/proc/lttng-logger`: + +
+echo -n 'Hello, World!' > /proc/lttng-logger
+
+ +The `msg` field of the `lttng_logger` event contains the recorded +message. + +
+

+ Note:Messages are split in chunks of + 1024 bytes. +

+
+ +The LTTng logger ABI is a quick and easy way to trace some events from +user space through the kernel tracer. However, it is much more basic +than LTTng-UST: it's slower (involves system call round-trip to the +kernel and only supports logging strings). The LTTng logger ABI is +particularly useful for recording logs as LTTng traces from shell +scripts, potentially combining them with other Linux kernel/user space +events. diff --git a/contents/using-lttng/intro.md b/contents/using-lttng/intro.md new file mode 100644 index 0000000..0590913 --- /dev/null +++ b/contents/using-lttng/intro.md @@ -0,0 +1,40 @@ +--- +id: using-lttng +--- + +Using LTTng involves two main activities: **instrumenting** and +**controlling tracing**. + +_[Instrumenting](#doc-instrumenting)_ is the process of inserting probes +into some source code. It can be done manually, by writing tracepoint +calls at specific locations in the source code of the program to trace, +or more automatically using dynamic probes (address in assembled code, +symbol name, function entry/return, etc.). + +It has to be noted that, as an LTTng user, you may not have to worry +about the instrumentation process. Indeed, you may want to trace a +program already instrumented. As an example, the Linux kernel is +thoroughly instrumented, which is why you can trace it without caring +about adding probes. + +_[Controlling tracing](#doc-controlling-tracing)_ is everything +that can be done by the LTTng session daemon, which is controlled using +`liblttng-ctl` or its command line utility, `lttng`: creating tracing +sessions, listing tracing sessions and events, enabling/disabling +events, starting/stopping the tracers, taking snapshots, etc. + +This chapter is a complete user guide of both activities, +with common use cases of LTTng exposed throughout the text. It is +assumed that you are familiar with LTTng's concepts (events, channels, +domains, tracing sessions) and that you understand the roles of its +components (daemons, libraries, command line tools); if not, we invite +you to read the [Understanding LTTng](#doc-understanding-lttng) chapter +before you begin reading this one. + +If you're new to LTTng, we suggest that you rather start with the +[Getting started](#doc-getting-started) small guide first, then come +back here to broaden your knowledge. + +If you're only interested in tracing the Linux kernel with its current +instrumentation, you may skip the +[Instrumenting](#doc-instrumenting) section. diff --git a/contrib-guide.md b/contrib-guide.md new file mode 100644 index 0000000..1eed3df --- /dev/null +++ b/contrib-guide.md @@ -0,0 +1,224 @@ +Contributor's guide +=================== + +This guide presents the structure and conventions of the LTTng +Documentation's source. Make sure you read it thoroughly before +contributing a change. + + +structure +--------- + +`toc/docs.yml` is a YAML tree of all chapters, sections and subsections. +It indicates which unique ID is linked to which position in the +hierarchy and its true title. + +In the `contents` directory, the `preface.md` file is the preface contents. +Each chapter has its own directory (directory names are not significant). +Within those, `intro.md` files are partial introductions and then each +section has its own directory, and so on, unless a section has no +subsections, in which case all its contents is in a single Markdown file +named _more or less_ like its ID. + +Each Markdown file begins with a YAML front matter which only contains +the unique ID of this chapter/section: + +```yaml +--- +id: unique-id-goes-here +--- + +First paragraph goes here. +``` + +Editable image sources are placed in `images/src` and their rendered +equivalents are located in `images/export`. + +`tools/checkdocs.py` is a Python 3 script which may be used to find +typical errors in the whole documentation (dead internal links, +common grammar mistakes, etc.). It needs the +[`termcolor`](https://pypi.python.org/pypi/termcolor) Python package. +Run it from the repository's root: + + tools/checkdocs.py + +and it will potentially output a list of errors and warnings. + + +format of sources +----------------- + +The sources are made of a fusion of Markdown and HTML processed by +[kramdown](http://kramdown.gettalong.org/). Markdown is preferred, +HTML being only used for specific cases that need special formatting +not available using plain Markdown. The kramdown processor is clever +enough to support both languages in the same file, even in the same +paragraph! + + +### HTML specifics + +Here's a list of HTML blocks and inline code used throughout the +document. If you need to contribute, please use them when needed to +preserve the document's visual consistency. + + +#### tip/note block + +Tip/note block: + +```html +
+

+ Title goes here followed by colon:Text goes + here; plain HTML. +

+

+ Multiple paragraphs is allowed. +

+
+``` + +Title should be `Tip:` for a tip and `Note:` for a note. + + +#### external links + +Internal links should always use Markdown +(`[caption](#doc-section)`). External links, however, need a special +style and must use the `` tag with the `ext` CSS class: + +```html +The LTTng Documentation is +public. +``` + + +#### abbreviations + +Use `` for describing abbreviations. This should only be used +for the first use of the abbreviation: + +```html +The LTTng +project is an open source system software package [...] +``` + + +#### non-breaking spaces + +Sometimes, a non-breaking space HTML entity (` `) needs to be +explicitly written. + +Examples: + +```html +The size of this file is 1039 bytes. + +This integer is displayed in base 16. + +A check is performed every 3000 ms. +``` + + +#### placeholders in inline code + +You must use `` to emphasize a placeholder within a `` tag +because Markdown backticks (`) always render their +content literally: + +```html +Name your file something_sys.c, where +sys is your system name. +``` + + +#### terminal boxes + +A terminal box, where command lines are shown, is a simple `
`
+with the `term` class:
+
+```html
+
+echo This is a terminal box
+

+```
+
+Do not prefix command lines with prompts (`$`/`#`) since this makes
+copy/paste operations painful.
+
+You may use `` tags to emphasize a part of the command line:
+
+```html
+
+echo This is a terminal box
+

+```
+
+Results of commands, if needed, should be presented in a simple
+`text` kramdown code block:
+
+
+~~~ text
+[15:30:34.835895035] (+?.?????????) hostname hello_world: { cpu_id = 1 }, { my_int = 8, char0 = 68, char1 = 97, product = "DataTraveler 2.0" }
+[15:30:42.262781421] (+7.426886386) hostname hello_world: { cpu_id = 1 }, { my_int = 9, char0 = 80, char1 = 97, product = "Patriot Memory" }
+[15:30:48.175621778] (+5.912840357) hostname hello_world: { cpu_id = 1 }, { my_int = 10, char0 = 68, char1 = 97, product = "DataTraveler 2.0" }
+~~~
+

+
+
+#### images
+
+Use
+
+```html
+

+    Short description
+

+```
+
+to display an image. Change `img-70` to `img-` followed by the
+width percentage you wish.
+
+The SVG format is preferred. In this case, use the `` tag to
+render an interactive SVG, with an inner raster image fallback for
+basic browsers:
+
+```html
+

+  
+    Short description
+  
+

+```
+
+An interactive SVG object allows its text to be selected, amongst other
+features.
+
+
+convention
+----------
+
+A few rules to comply with in order to keep the text as
+consistent as possible:
+
+  * Use _user space_, not _userspace_ nor _user-space_.
+    (neither _user land_).
+  * Use _file system_, not _filesystem_.
+  * Use _use case_, not _use-case_ nor _usecase_.
+  * Use _the C standard library_, not _libc_.
+  * Use _log level_, not _loglevel_.
+  * Use complete LTTng project names: _LTTng-modules_, _LTTng-UST_ and
+    _LTTng-tools_, not _modules_, _UST_ and _tools_.
+  * All code snippets should use 4 spaces for indentation (even C)
+    so that they are not too large.
+  * Prefer emphasis (Markdown: `_something_`, HTML: `something`)
+    to strong (Markdown: `**something**`, HTML: `something`)
+    for emphasizing text.
+  * Try to stay behind the 72th column mark if possible, and behind
+    the 80th column otherwise.
+  * Do not end directory paths with a forward slash
+    (good: `include/trace/events`, bad: `include/trace/events/`).
+  * Keep the text as impersonal as possible (minimize the use of
+    _I_, _we_, _us_, etc.), except for user guides/tutorials where
+    _we_ have an ongoing example.
diff --git a/images/export/core-concepts.svg b/images/export/core-concepts.svg
new file mode 100644
index 0000000..610bac3
--- /dev/null
+++ b/images/export/core-concepts.svg
@@ -0,0 +1,424 @@
+
+
+
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      
+    
+  
+  
+    Tracing session
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      
+    
+  
+  
+    Linux kernel domain
+  
+  
+    Channel
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      
+    
+  
+  
+    Channel
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      Event A
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      Event B
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      Event C
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      Event E
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      Event B
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      Event C
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      Event D
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      
+    
+  
+  
+    User space domain
+  
+  
+    Channel
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      
+    
+  
+  
+    Channel
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      Event 2
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      Event 1
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      Event 2
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      Event 3
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      Event 4
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      Event 6
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      Event 5
+    
+  
+
diff --git a/images/export/lttng-live-relayd.svg b/images/export/lttng-live-relayd.svg
new file mode 100644
index 0000000..322e8cf
--- /dev/null
+++ b/images/export/lttng-live-relayd.svg
@@ -0,0 +1,112 @@
+
+
+
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      
+    
+  
+  
+    Relay daemon
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      Consumer
+      daemon
+    
+  
+  
+    
+    
+    
+  
+  
+    Events
+  
+  
+    
+    
+    
+  
+  
+    
+    
+    
+    
+      CTF
+      traces
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      Live viewer
+    
+  
+  
+    
+    
+    
+  
+  
+    
+    
+  
+  
+    
+    
+      Events over LTTng live
+      (TCP)
+    
+  
+  
+    (TCP)
+  
+  
+    CTF
+  
+  
+    (local FS)
+  
+
diff --git a/images/export/lttng-lttng-gen-tp.svg b/images/export/lttng-lttng-gen-tp.svg
new file mode 100644
index 0000000..9a2fb0a
--- /dev/null
+++ b/images/export/lttng-lttng-gen-tp.svg
@@ -0,0 +1,189 @@
+
+
+
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      hello-tp.tp
+    
+  
+  
+    
+    
+    
+      lttng-gen-tp
+    
+  
+  
+    
+    
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      hello-tp.o
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      hello-tp.h
+    
+  
+  
+    
+    
+    
+  
+  
+    
+    
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      hello.c
+    
+  
+  
+    
+    
+  
+  
+    
+    
+      #include
+    
+  
+  
+    
+    
+    
+      gcc/clang
+    
+  
+  
+    
+    
+    
+  
+  
+    
+    
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      hello
+    
+  
+  
+    
+    
+    
+  
+  
+    
+    
+      
+    
+  
+  
+    
+    
+    
+      
+    
+  
+  
+    
+    
+    
+      
+    
+  
+  
+    File written by user
+  
+  
+    Generated file
+  
+  
+    Tool
+  
+  
+    
+    
+    
+      
+    
+  
+
diff --git a/images/export/plumbing.svg b/images/export/plumbing.svg
new file mode 100644
index 0000000..f8cca32
--- /dev/null
+++ b/images/export/plumbing.svg
@@ -0,0 +1,455 @@
+
+
+
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      TP
+    
+  
+  
+    C/C++ application
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      liblttng-ust
+    
+  
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      Java application
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      liblttng-ust
+    
+  
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      liblttng-ust-jul-jni
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      liblttng-ust-jul.jar
+    
+  
+  
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      
+    
+  
+  
+    Linux kernel
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      LTTng modules
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      lttng-sessiond
+    
+  
+  
+    
+    
+    
+    
+    
+  
+  
+    
+    
+    
+    
+    
+  
+  
+    
+    
+    
+    
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      lttng-consumerd
+    
+  
+  
+    
+    
+    
+  
+  
+    
+    
+    
+  
+  
+    
+    
+    
+  
+  
+    
+    
+    
+    
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      lttng (CLI)
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      liblttng-ctl
+    
+  
+  
+  
+    
+    
+    
+    
+    
+  
+  
+    
+    
+    
+    
+    
+  
+  
+    Target (monitored) system
+  
+  
+    Remote system
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      lttng-relayd
+    
+  
+  
+    
+    
+    
+  
+  
+    
+    
+    
+    
+      CTF
+      traces
+    
+  
+  
+    
+    
+    
+  
+  
+    
+    
+    
+    
+      CTF
+      traces
+    
+  
+  
+    (may be local as well)
+  
+  
+    
+    
+    
+  
+  
+    
+    
+    
+    
+    
+  
+  
+    
+    
+    
+    
+    
+  
+  
+    
+    
+      TCP
+    
+  
+  
+    
+    
+      TCP
+    
+  
+  
+    Control
+  
+  
+    
+    
+    
+  
+  
+    Trace data
+  
+
diff --git a/images/src/core-concepts b/images/src/core-concepts
new file mode 100644
index 0000000..fa31b24
Binary files /dev/null and b/images/src/core-concepts differ
diff --git a/images/src/getting-started-lttng-gen-tp b/images/src/getting-started-lttng-gen-tp
new file mode 100644
index 0000000..d0e9f9e
Binary files /dev/null and b/images/src/getting-started-lttng-gen-tp differ
diff --git a/images/src/lttng-live-relayd b/images/src/lttng-live-relayd
new file mode 100644
index 0000000..b4d5532
Binary files /dev/null and b/images/src/lttng-live-relayd differ
diff --git a/images/src/plumbing b/images/src/plumbing
new file mode 100644
index 0000000..d693c9e
Binary files /dev/null and b/images/src/plumbing differ
diff --git a/toc/docs.yml b/toc/docs.yml
new file mode 100644
index 0000000..67a56e9
--- /dev/null
+++ b/toc/docs.yml
@@ -0,0 +1,212 @@
+title: The LTTng Documentation
+cats:
+  - id: nuts-and-bolts
+    title: Nuts and bolts
+  - id: installing-lttng
+    title: Installing LTTng
+    cats:
+      - id: desktop-distributions
+        title: Desktop distributions
+        cats:
+          - id: ubuntu
+            title: Ubuntu
+            cats:
+              - id: ubuntu-official-repositories
+                title: Official repositories
+              - id: ubuntu-ppa
+                title: PPA
+          - id: fedora
+            title: Fedora
+          - id: debian
+            title: Debian
+          - id: opensuse
+            title: openSUSE/RPM
+          - id: archlinux
+            title: Arch Linux
+          - id: enterprise-distributions
+            title: "Enterprise distributions (RHEL, SLES)"
+      - id: embedded-distributions
+        title: Embedded distributions
+        cats:
+          - id: buildroot
+            title: Buildroot
+          - id: oe-yocto
+            title: OpenEmbedded/Yocto
+      - id: building-from-source
+        title: Building from source
+  - id: getting-started
+    title: Getting started with LTTng
+    cats:
+      - id: tracing-the-linux-kernel
+        title: Tracing the Linux kernel
+      - id: tracing-your-own-user-application
+        title: Tracing your own user application
+      - id: viewing-and-analyzing-your-traces
+        title: Viewing and analyzing your traces
+  - id: understanding-lttng
+    title: Understanding LTTng
+    cats:
+      - id: core-concepts
+        title: Core concepts
+        cats:
+          - id: tracing-session
+            title: Tracing session
+          - id: domain
+            title: Domain
+          - id: channel
+            title: Channel
+            cats:
+              - id: channel-overwrite-mode-vs-discard-mode
+                title: Overwrite and discard event loss modes
+              - id: channel-subbuf-size-vs-subbuf-count
+                title: Sub-buffers count and size
+              - id: channel-switch-timer
+                title: Switch timer
+              - id: channel-buffering-schemes
+                title: Buffering schemes
+          - id: event
+            title: Event
+      - id: plumbing
+        title: Plumbing
+        cats:
+          - id: plumbing-overview
+            title: Overview
+          - id: lttng-sessiond
+            title: Session daemon
+          - id: lttng-consumerd
+            title: Consumer daemon
+          - id: lttng-relayd
+            title: Relay daemon
+          - id: liblttng-ctl-lttng
+            title: Control library and command line interface
+          - id: lttng-ust
+            title: User space tracing library
+          - id: lttng-modules
+            title: LTTng kernel modules
+  - id: using-lttng
+    title: Using LTTng
+    cats:
+      - id: instrumenting
+        title: Instrumenting
+        cats:
+          - id: c-application
+            title: C application
+            cats:
+              - id: tracepoint-provider
+                title: Tracepoint provider
+              - id: lttng-gen-tp
+                title: Using lttng-gen-tp
+              - id: defining-tracepoints
+                title: Defining tracepoints
+              - id: assigning-log-levels
+                title: Assigning log levels to tracepoints
+              - id: probing-the-application-source-code
+                title: Probing the application's source code
+              - id: building-tracepoint-providers-and-user-application
+                title: Building/linking tracepoint providers and the user application
+                cats:
+                  - id: static-linking
+                    title: Static linking
+                  - id: dynamic-linking
+                    title: Dynamic linking
+                  - id: using-lttng-ust-with-daemons
+                    title: Using LTTng-UST with daemons
+                  - id: lttng-ust-pkg-config
+                    title: pkg-config
+              - id: tracef
+                title: Using tracef()
+              - id: lttng-ust-environment-variables-compiler-flags
+                title: LTTng-UST environment variables and special compilation flags
+          - id: cxx-application
+            title: C++ application
+          - id: prebuilt-ust-helpers
+            title: Prebuilt user space tracing helpers
+            cats:
+              - id: liblttng‑ust‑libc‑pthread-wrapper
+                title: C standard library and POSIX threads tracing
+              - id: liblttng‑ust‑cyg‑profile
+                title: Function tracing
+              - id: liblttng‑ust‑dl
+                title: Dynamic linker tracing
+          - id: java-application
+            title: Java application
+          - id: instrumenting-linux-kernel
+            title: Linux kernel
+            cats:
+              - id: instrumenting-linux-kernel-itself
+                title: Instrumenting the Linux kernel for LTTng
+                cats:
+                  - id: mainline-trace-event
+                    title: Defining/using tracepoints with mainline TRACE_EVENT() infrastructure
+                  - id: lttng-adaptation-layer
+                    title: Adding the LTTng adaptation layer
+                  - id: instrumenting-linux-kernel-tracing
+                    title: Tracing
+              - id: instrumenting-out-of-tree-linux-kernel
+                title: Instrumenting an out-of-tree Linux kernel module for LTTng
+          - id: proc-lttng-logger-abi
+            title: LTTng logger ABI
+          - id: advanced-instrumenting-techniques
+            title: Advanced techniques
+            cats:
+              - id: instrumenting-32-bit-app-on-64-bit-system
+                title: Instrumenting a 32-bit application on a 64-bit system
+                cats:
+                  - id: building-32-bit-userspace-rcu
+                    title: Building 32-bit Userspace RCU
+                  - id: building-32-bit-lttng-ust
+                    title: Building 32-bit LTTng-UST
+                  - id: building-32-bit-lttng-tools
+                    title: Building 32-bit LTTng-tools
+                  - id: building-64-bit-lttng-tools
+                    title: Building 64-bit LTTng-tools
+                  - id: building-instrumented-32-bit-c-application
+                    title: Building an instrumented 32-bit C application
+                  - id: running-32-bit-and-64-bit-c-applications
+                    title: Running 32-bit and 64-bit versions of an instrumented C application
+      - id: controlling-tracing
+        title: Controlling tracing
+        cats:
+          - id: creating-destroying-tracing-sessions
+            title: Creating and destroying tracing sessions
+          - id: enabling-disabling-events
+            title: Enabling and disabling events
+          - id: basic-tracing-session-control
+            title: Basic tracing session control
+          - id: enabling-disabling-channels
+            title: Enabling and disabling channels
+            cats:
+              - id: fine-tuning-channels
+                title: Fine-tuning channels
+          - id: adding-context
+            title: Adding some context to channels
+          - id: saving-loading-tracing-session
+            title: Saving and loading tracing session configurations
+          - id: sending-trace-data-over-the-network
+            title: Sending trace data over the network
+          - id: lttng-live
+            title: Viewing events as they arrive
+          - id: taking-a-snapshot
+            title: Taking a snapshot
+  - id: reference
+    title: Reference
+    cats:
+      - id: online-lttng-manpages
+        title: Online LTTng manpages
+      - id: lttng-ust-ref
+        title: LTTng-UST
+        cats:
+          - id: liblttng-ust
+            title: LTTng-UST library (liblttng‑ust)
+            cats:
+              - id: liblttng-ust-tp-fields
+                title: Tracepoint fields macros (for TP_FIELDS())
+              - id: liblttng-ust-tracepoint-loglevel
+                title: Tracepoint log levels (for TRACEPOINT_LOGLEVEL())
+      - id: lttng-modules-ref
+        title: LTTng-modules
+        cats:
+          - id: lttng-modules-tp-struct-entry
+            title: Tracepoint fields macros (for TP_STRUCT__entry())
+          - id: lttng-modules-tp-fast-assign
+            title: Tracepoint assignment macros (for TP_fast_assign())
diff --git a/tools/checkdocs.py b/tools/checkdocs.py
new file mode 100755
index 0000000..bc53b4c
--- /dev/null
+++ b/tools/checkdocs.py
@@ -0,0 +1,144 @@
+#!/usr/bin/env python3
+
+# The MIT License (MIT)
+#
+# Copyright (c) 2014 Philippe Proulx 
+# Copyright (c) 2014 The LTTng Project 
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+
+import re
+import os
+import sys
+from termcolor import colored
+
+
+TOC_PATH = 'toc/docs.yml'
+CONTENTS_ROOT_PATH = 'contents'
+
+
+def _perror(filename, msg):
+    s = '{}  {} {}'.format(filename, colored('Error:', 'red'),
+                           colored(msg, 'red', attrs=['bold']))
+    print(s)
+
+
+def _pwarn(filename, msg):
+    s = '{}  {} {}'.format(filename, colored('Warning:', 'yellow'),
+                           colored(msg, 'yellow', attrs=['bold']))
+    print(s)
+
+
+def _get_files(root):
+    files = []
+
+    for dirpath, dirnames, filenames in os.walk(root):
+        for f in filenames:
+            files.append(os.path.join(dirpath, f))
+
+    return sorted(files)
+
+
+def _get_toc_ids(path):
+    p = re.compile(r'id\s*:\s*(.+)$', flags=re.M)
+
+    with open(path) as f:
+        orig_ids = p.findall(f.read())
+
+    ids = set(orig_ids)
+
+    if len(ids) != len(orig_ids):
+        _perror(path, 'Duplicate IDs')
+        return None
+
+    return ids
+
+
+def _check_file_links(toc_ids, path, c):
+    ilinkp = re.compile(r'\[[^\]]+\]\(([^)]+)\)', flags=re.M)
+    elinkp = re.compile(r'href="([^"]+)"')
+
+    ret = True
+
+    ilinks = ilinkp.findall(c)
+    elinks = elinkp.findall(c)
+
+    for link in ilinks:
+        if not link.startswith('#doc-'):
+            s = 'Internal link does not start with "#doc-": "{}"'.format(link)
+            _perror(path, s)
+            ret = False
+            continue
+
+        sid = link[5:]
+
+        if sid not in toc_ids:
+            _perror(path, 'Dead internal link: "{}"'.format(link))
+            ret = False
+
+    for link in elinks:
+        if link.startswith('#'):
+            _pwarn(path, 'External link starts with #: "{}"'.format(link))
+            ret = False
+
+    return ret
+
+
+def _check_contents(toc_ids, contents_files):
+    ret = True
+
+    for path in contents_files:
+        with open(path) as f:
+            c = f.read()
+
+        ret &= _check_file_links(toc_ids, path, c)
+
+    return ret
+
+
+def _check_non_md(files):
+    ret = True
+
+    for f in files:
+        if not f.endswith('.md'):
+            _perror(f, 'Wrong, non-Markdown file')
+            ret = False
+
+    return ret
+
+
+def checkdocs():
+    toc_ids = _get_toc_ids(TOC_PATH)
+
+    if toc_ids is None:
+        return False
+
+    contents_files = _get_files(CONTENTS_ROOT_PATH)
+
+    if not _check_non_md(contents_files):
+        return False
+
+    if not _check_contents(toc_ids, contents_files):
+        return False
+
+    return True
+
+
+if __name__ == '__main__':
+    sys.exit(0 if checkdocs() else 1)