| 1 | lttng-concepts(7) |
| 2 | ================= |
| 3 | :revdate: 14 June 2021 |
| 4 | :sect-event-rule: INSTRUMENTATION POINT, EVENT RULE, AND EVENT |
| 5 | :sect-session: RECORDING SESSION |
| 6 | :sect-domain: TRACING DOMAIN |
| 7 | :sect-channel: CHANNEL AND RING BUFFER |
| 8 | :sect-recording-event-rule: RECORDING EVENT RULE AND EVENT RECORD |
| 9 | |
| 10 | |
| 11 | NAME |
| 12 | ---- |
| 13 | lttng-concepts - LTTng concepts |
| 14 | |
| 15 | |
| 16 | DESCRIPTION |
| 17 | ----------- |
| 18 | This manual page documents the concepts of LTTng. |
| 19 | |
| 20 | Many other LTTng manual pages refer to this one so that you can |
| 21 | understand what are the various LTTng objects and how they relate to |
| 22 | each other. |
| 23 | |
| 24 | The concepts of LTTng{nbsp}{lttng_version} are: |
| 25 | |
| 26 | * Instrumentation point, event rule, and event |
| 27 | * Trigger |
| 28 | * Recording session |
| 29 | * Tracing domain |
| 30 | * Channel and ring buffer |
| 31 | * Recording event rule and event record |
| 32 | |
| 33 | |
| 34 | [[event-rule]] |
| 35 | {sect-event-rule} |
| 36 | ----------------- |
| 37 | An _instrumentation point_ is a point, within a piece of software, |
| 38 | which, when executed, creates an LTTng _event_. |
| 39 | |
| 40 | LTTng offers various types of instrumentation; see the |
| 41 | ``<<inst-point-types,Instrumentation point types>>'' section below to |
| 42 | learn about them. |
| 43 | |
| 44 | An _event rule_ is a set of conditions to match a set of events. |
| 45 | |
| 46 | When LTTng creates an event{nbsp}__E__, an event rule{nbsp}__ER__ is |
| 47 | said to __match__{nbsp}__E__ when{nbsp}__E__ satisfies *all* the |
| 48 | conditions of{nbsp}__ER__. This concept is similar to a regular |
| 49 | expression which matches a set of strings. |
| 50 | |
| 51 | When an event rule matches an event, LTTng _emits_ the event, therefore |
| 52 | attempting to execute one or more actions. |
| 53 | |
| 54 | [IMPORTANT] |
| 55 | ==== |
| 56 | The event creation and emission processes are documentation concepts to |
| 57 | help understand the journey from an instrumentation point to the |
| 58 | execution of actions. |
| 59 | |
| 60 | The actual creation of an event can be costly because LTTng needs to |
| 61 | evaluate the arguments of the instrumentation point. |
| 62 | |
| 63 | In practice, LTTng implements various optimizations for the Linux kernel |
| 64 | and user space tracing domains (see the ``<<domain,{sect-domain}>>'' |
| 65 | section below) to avoid actually creating an event when the tracer |
| 66 | knows, thanks to properties which are independent from the event payload |
| 67 | and current context, that it would never emit such an event. Those |
| 68 | properties are: |
| 69 | |
| 70 | * The instrumentation point type (see the |
| 71 | ``<<inst-point-types,Instrumentation point types>>'' section below). |
| 72 | |
| 73 | * The instrumentation point name. |
| 74 | |
| 75 | * The instrumentation point log level. |
| 76 | |
| 77 | * For a recording event rule (see the |
| 78 | ``<<recording-event-rule,{sect-recording-event-rule}>>'' section |
| 79 | below): |
| 80 | ** The status of the rule itself. |
| 81 | ** The status of the channel (see the ``<<channel,{sect-channel}>>'' |
| 82 | section below). |
| 83 | ** The activity of the recording session (started or stopped; see |
| 84 | the ``<<session,{sect-session}>>'' section below). |
| 85 | ** Whether or not the process for which LTTng would create the event is |
| 86 | allowed to record events (see man:lttng-track(1)). |
| 87 | |
| 88 | In other words: if, for a given instrumentation point{nbsp}__IP__, the |
| 89 | LTTng tracer knows that it would never emit an event, |
| 90 | executing{nbsp}__IP__ represents a simple boolean variable check and, |
| 91 | for a Linux kernel recording event rule, a few process attribute checks. |
| 92 | ==== |
| 93 | |
| 94 | As of LTTng{nbsp}{lttng_version}, there are two places where you can |
| 95 | find an event rule: |
| 96 | |
| 97 | Recording event rule:: |
| 98 | A specific type of event rule of which the action is to record the |
| 99 | matched event as an event record. |
| 100 | + |
| 101 | See the ``<<recording-event-rule,{sect-recording-event-rule}>>'' section |
| 102 | below. |
| 103 | + |
| 104 | Create or enable a recording event rule with the |
| 105 | man:lttng-enable-event(1) command. |
| 106 | + |
| 107 | List the recording event rules of a specific recording session |
| 108 | and/or channel with the man:lttng-list(1) and man:lttng-status(1) |
| 109 | commands. |
| 110 | |
| 111 | ``Event rule matches'' <<trigger,trigger>> condition (since LTTng{nbsp}2.13):: |
| 112 | When the event rule of the trigger condition matches an event, LTTng |
| 113 | can execute user-defined actions such as sending an LTTng |
| 114 | notification, starting a recording session, and more. |
| 115 | + |
| 116 | See man:lttng-add-trigger(1) and man:lttng-event-rule(7). |
| 117 | |
| 118 | For LTTng to emit an event{nbsp}__E__,{nbsp}__E__ must satisfy *all* the |
| 119 | basic conditions of an event rule{nbsp}__ER__, that is: |
| 120 | |
| 121 | * The instrumentation point from which LTTng creates{nbsp}__E__ has a |
| 122 | specific type. |
| 123 | + |
| 124 | See the ``<<inst-point-types,Instrumentation point types>>'' section |
| 125 | below. |
| 126 | |
| 127 | * A pattern matches the name of{nbsp}__E__ while another pattern |
| 128 | doesn't. |
| 129 | |
| 130 | * The log level of the instrumentation point from which LTTng |
| 131 | creates{nbsp}__E__ is at least as severe as some value, or is exactly |
| 132 | some value. |
| 133 | |
| 134 | * The fields of the payload of{nbsp}__E__ and the current context fields |
| 135 | satisfy a filter expression. |
| 136 | |
| 137 | A recording event rule has additional, implicit conditions to satisfy. |
| 138 | See the ``<<recording-event-rule,{sect-recording-event-rule}>>'' section |
| 139 | below to learn more. |
| 140 | |
| 141 | |
| 142 | [[inst-point-types]] |
| 143 | Instrumentation point types |
| 144 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 145 | As of LTTng{nbsp}{lttng_version}, the available instrumentation point |
| 146 | types are, depending on the tracing domain (see the |
| 147 | ``<<domain,{sect-domain}>>'' section below): |
| 148 | |
| 149 | Linux kernel:: |
| 150 | LTTng tracepoint::: |
| 151 | A statically defined point in the source code of the kernel |
| 152 | image or of a kernel module using the LTTng-modules macros. |
| 153 | + |
| 154 | List the available Linux kernel tracepoints with `lttng list --kernel`. |
| 155 | See man:lttng-list(1) to learn more. |
| 156 | |
| 157 | Linux kernel system call::: |
| 158 | Entry, exit, or both of a Linux kernel system call. |
| 159 | + |
| 160 | List the available Linux kernel system call instrumentation points with |
| 161 | `lttng list --kernel --syscall`. See man:lttng-list(1) to learn more. |
| 162 | |
| 163 | Linux kprobe::: |
| 164 | A single probe dynamically placed in the compiled kernel code. |
| 165 | + |
| 166 | When you create such an instrumentation point, you set its memory |
| 167 | address or symbol name. |
| 168 | |
| 169 | Linux user space probe::: |
| 170 | A single probe dynamically placed at the entry of a compiled |
| 171 | user space application/library function through the kernel. |
| 172 | + |
| 173 | When you create such an instrumentation point, you set: |
| 174 | + |
| 175 | -- |
| 176 | With the ELF method:: |
| 177 | Its application/library path and its symbol name. |
| 178 | |
| 179 | With the USDT method:: |
| 180 | Its application/library path, its provider name, and its probe name. |
| 181 | + |
| 182 | ``USDT'' stands for SystemTap User-level Statically Defined Tracing, |
| 183 | a DTrace-style marker. |
| 184 | -- |
| 185 | + |
| 186 | As of LTTng{nbsp}{lttng_version}, LTTng only supports USDT probes which |
| 187 | are :not: reference-counted. |
| 188 | |
| 189 | Linux kretprobe::: |
| 190 | Entry, exit, or both of a Linux kernel function. |
| 191 | + |
| 192 | When you create such an instrumentation point, you set the memory |
| 193 | address or symbol name of its function. |
| 194 | |
| 195 | User space:: |
| 196 | LTTng tracepoint::: |
| 197 | A statically defined point in the source code of a C/$$C++$$ |
| 198 | application/library using the LTTng-UST macros. |
| 199 | + |
| 200 | List the available Linux kernel tracepoints with |
| 201 | `lttng list --userspace`. See man:lttng-list(1) to learn more. |
| 202 | |
| 203 | `java.util.logging`, Apache log4j, and Python:: |
| 204 | Java or Python logging statement::: |
| 205 | A method call on a Java or Python logger attached to an |
| 206 | LTTng-UST handler. |
| 207 | + |
| 208 | List the available Java and Python loggers with `lttng list --jul`, |
| 209 | `lttng list --log4j`, and `lttng list --python`. See man:lttng-list(1) |
| 210 | to learn more. |
| 211 | |
| 212 | |
| 213 | [[trigger]] |
| 214 | TRIGGER |
| 215 | ------- |
| 216 | A _trigger_ associates a condition to one or more actions. |
| 217 | |
| 218 | When the condition of a trigger is satisfied, LTTng attempts to execute |
| 219 | its actions. |
| 220 | |
| 221 | As of LTTng{nbsp}{lttng_version}, the available trigger conditions and |
| 222 | actions are: |
| 223 | |
| 224 | Conditions:: |
| 225 | + |
| 226 | * The consumed buffer size of a given recording |
| 227 | session (see the ``<<session,{sect-session}>>'' section below) |
| 228 | becomes greater than some value. |
| 229 | |
| 230 | * The buffer usage of a given channel (see the |
| 231 | ``<<channel,{sect-channel}>>'' section below) becomes greater than |
| 232 | some value. |
| 233 | |
| 234 | * The buffer usage of a given channel becomes less than some value. |
| 235 | |
| 236 | * There's an ongoing recording session rotation (see the |
| 237 | ``<<rotation,Recording session rotation>>'' section below). |
| 238 | |
| 239 | * A recording session rotation becomes completed. |
| 240 | |
| 241 | * An event rule matches an event. |
| 242 | + |
| 243 | As of LTTng{nbsp}{lttng_version}, this is the only available condition |
| 244 | when you add a trigger with the man:lttng-add-trigger(1) command. The |
| 245 | other ones are available through the liblttng-ctl C{nbsp}API. |
| 246 | |
| 247 | Actions:: |
| 248 | + |
| 249 | * Send a notification to a user application. |
| 250 | * Start a given recording session, like man:lttng-start(1) would do. |
| 251 | * Stop a given recording session, like man:lttng-stop(1) would do. |
| 252 | * Archive the current trace chunk of a given recording session (rotate), |
| 253 | like man:lttng-rotate(1) would do. |
| 254 | * Take a snapshot of a given recording session, like |
| 255 | man:lttng-snapshot(1) would do. |
| 256 | |
| 257 | A trigger belongs to a session daemon (see man:lttng-sessiond(8)), not |
| 258 | to a specific recording session. For a given session daemon, each Unix |
| 259 | user has its own, private triggers. Note, however, that the `root` Unix |
| 260 | user may, for the root session daemon: |
| 261 | |
| 262 | * Add a trigger as another Unix user. |
| 263 | |
| 264 | * List all the triggers, regardless of their owner. |
| 265 | |
| 266 | * Remove a trigger which belongs to another Unix user. |
| 267 | |
| 268 | For a given session daemon and Unix user, a trigger has a unique name. |
| 269 | |
| 270 | Add a trigger to a session daemon with the man:lttng-add-trigger(1) |
| 271 | command. |
| 272 | |
| 273 | List the triggers of your Unix user (or of all users if your |
| 274 | Unix user is `root`) with the man:lttng-list-triggers(1) command. |
| 275 | |
| 276 | Remove a trigger with the man:lttng-remove-trigger(1) command. |
| 277 | |
| 278 | |
| 279 | [[session]] |
| 280 | {sect-session} |
| 281 | -------------- |
| 282 | A _recording session_ (named ``tracing session'' prior to |
| 283 | LTTng{nbsp}2.13) is a stateful dialogue between you and a session daemon |
| 284 | (see man:lttng-sessiond(8)) for everything related to event recording. |
| 285 | |
| 286 | Everything that you do when you control LTTng tracers to record events |
| 287 | happens within a recording session. In particular, a recording session: |
| 288 | |
| 289 | * Has its own name, unique for a given session daemon. |
| 290 | |
| 291 | * Has its own set of trace files, if any. |
| 292 | |
| 293 | * Has its own state of activity (started or stopped). |
| 294 | + |
| 295 | An active recording session is an implicit recording event rule |
| 296 | condition (see the |
| 297 | ``<<recording-event-rule,{sect-recording-event-rule}>>'' section below). |
| 298 | |
| 299 | * Has its own mode (local, network streaming, snapshot, or live). |
| 300 | + |
| 301 | See the ``<<session-modes,Recording session modes>>'' section below to |
| 302 | learn more. |
| 303 | |
| 304 | * Has its own channels (see the ``<<channel,{sect-channel}>>'' section |
| 305 | below) to which are attached their own recording event rules. |
| 306 | |
| 307 | * Has its own process attribute inclusion sets (see man:lttng-track(1)). |
| 308 | |
| 309 | Those attributes and objects are completely isolated between different |
| 310 | recording sessions. |
| 311 | |
| 312 | A recording session is like an ATM session: the operations you do on the |
| 313 | banking system through the ATM don't alter the data of other users of |
| 314 | the same system. In the case of the ATM, a session lasts as long as your |
| 315 | bank card is inside. In the case of LTTng, a recording session lasts from |
| 316 | the man:lttng-create(1) command to the man:lttng-destroy(1) command. |
| 317 | |
| 318 | A recording session belongs to a session daemon (see |
| 319 | man:lttng-sessiond(8)). For a given session daemon, each Unix user has |
| 320 | its own, private recording sessions. Note, however, that the `root` Unix |
| 321 | user may operate on or destroy another user's recording session. |
| 322 | |
| 323 | Create a recording session with the man:lttng-create(1) command. |
| 324 | |
| 325 | List the recording sessions of the connected session daemon with |
| 326 | the man:lttng-list(1) command. |
| 327 | |
| 328 | Start and stop a recording session with the man:lttng-start(1) and |
| 329 | man:lttng-stop(1) commands. |
| 330 | |
| 331 | Save and load a recording session with the man:lttng-save(1) and |
| 332 | man:lttng-load(1) commands. |
| 333 | |
| 334 | Archive the current trace chunk of (rotate) a recording session with the |
| 335 | man:lttng-rotate(1) command. |
| 336 | |
| 337 | Destroy a recording session with the man:lttng-destroy(1) command. |
| 338 | |
| 339 | |
| 340 | Current recording session |
| 341 | ~~~~~~~~~~~~~~~~~~~~~~~ |
| 342 | When you run the man:lttng-create(1) command, LTTng creates the |
| 343 | `$LTTNG_HOME/.lttngrc` file if it doesn't exist (`$LTTNG_HOME` defaults |
| 344 | to `$HOME`). |
| 345 | |
| 346 | `$LTTNG_HOME/.lttngrc` contains the name of the _current recording |
| 347 | session_. |
| 348 | |
| 349 | When you create a new recording session with the `create` command, LTTng |
| 350 | updates the current recording session. |
| 351 | |
| 352 | The following man:lttng(1) commands select the current recording session |
| 353 | if you don't specify one: |
| 354 | |
| 355 | * man:lttng-add-context(1) |
| 356 | * man:lttng-clear(1) |
| 357 | * man:lttng-destroy(1) |
| 358 | * man:lttng-disable-channel(1) |
| 359 | * man:lttng-disable-event(1) |
| 360 | * man:lttng-disable-rotation(1) |
| 361 | * man:lttng-enable-channel(1) |
| 362 | * man:lttng-enable-event(1) |
| 363 | * man:lttng-enable-rotation(1) |
| 364 | * man:lttng-regenerate(1) |
| 365 | * man:lttng-rotate(1) |
| 366 | * man:lttng-save(1) |
| 367 | * man:lttng-snapshot(1) |
| 368 | * man:lttng-start(1) |
| 369 | * man:lttng-status(1) |
| 370 | * man:lttng-stop(1) |
| 371 | * man:lttng-track(1) |
| 372 | * man:lttng-untrack(1) |
| 373 | * man:lttng-view(1) |
| 374 | |
| 375 | Set the current recording session manually with the |
| 376 | man:lttng-set-session(1) command, without having to edit the `.lttngrc` |
| 377 | file. |
| 378 | |
| 379 | |
| 380 | [[session-modes]] |
| 381 | Recording session modes |
| 382 | ~~~~~~~~~~~~~~~~~~~~~~~ |
| 383 | LTTng offers four recording session modes: |
| 384 | |
| 385 | Local mode:: |
| 386 | Write the trace data to the local file system. |
| 387 | |
| 388 | Network streaming mode:: |
| 389 | Send the trace data over the network to a listening relay daemon |
| 390 | (see man:lttng-relayd(8)). |
| 391 | |
| 392 | Snapshot mode:: |
| 393 | Only write the trace data to the local file system or send it to a |
| 394 | listening relay daemon (man:lttng-relayd(8)) when LTTng takes a |
| 395 | snapshot. |
| 396 | + |
| 397 | LTTng forces all the channels (see the ``<<channel,{sect-channel}>>'' |
| 398 | section below) to be created to be configured to be snapshot-ready. |
| 399 | + |
| 400 | LTTng takes a snapshot of such a recording session when: |
| 401 | + |
| 402 | -- |
| 403 | * You run the man:lttng-snapshot(1) command. |
| 404 | |
| 405 | * LTTng executes a `snapshot-session` trigger action (see the |
| 406 | ``<<trigger,TRIGGER>>'' section above). |
| 407 | -- |
| 408 | |
| 409 | Live mode:: |
| 410 | Send the trace data over the network to a listening relay daemon |
| 411 | (see man:lttng-relayd(8)) for live reading. |
| 412 | + |
| 413 | An LTTng live reader (for example, man:babeltrace2(1)) can connect to |
| 414 | the same relay daemon to receive trace data while the recording session is |
| 415 | active. |
| 416 | |
| 417 | |
| 418 | [[rotation]] |
| 419 | Recording session rotation |
| 420 | ~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 421 | A _recording session rotation_ is the action of archiving the current |
| 422 | trace chunk of the recording session to the file system. |
| 423 | |
| 424 | Once LTTng archives a trace chunk, it does :not: manage it anymore: you |
| 425 | can read it, modify it, move it, or remove it. |
| 426 | |
| 427 | An _archived trace chunk_ is a collection of metadata and data stream |
| 428 | files which form a self-contained LTTng trace. See the |
| 429 | ``<<trace-chunk-naming,Trace chunk naming>>'' section below to learn how |
| 430 | LTTng names a trace chunk archive directory. |
| 431 | |
| 432 | The _current trace chunk_ of a given recording session includes: |
| 433 | |
| 434 | * The stream files which LTTng already wrote to the file system, and |
| 435 | which are not part of a previously archived trace chunk, since the |
| 436 | most recent event amongst: |
| 437 | |
| 438 | ** The first time the recording session was started, either with the |
| 439 | man:lttng-start(1) command or with a `start-session` trigger action |
| 440 | (see the ``<<trigger,TRIGGER>>'' section above). |
| 441 | |
| 442 | ** The last rotation, performed with: |
| 443 | |
| 444 | *** An man:lttng-rotate(1) command. |
| 445 | |
| 446 | *** A rotation schedule previously set with |
| 447 | man:lttng-enable-rotation(1). |
| 448 | |
| 449 | *** An executed `rotate-session` trigger action (see the |
| 450 | ``<<trigger,TRIGGER>>'' section above). |
| 451 | |
| 452 | * The content of all the non-flushed sub-buffers of the channels of the |
| 453 | recording session. |
| 454 | |
| 455 | |
| 456 | [[trace-chunk-naming]] |
| 457 | Trace chunk archive naming |
| 458 | ~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 459 | A trace chunk archive is a subdirectory of the `archives` subdirectory |
| 460 | within the output directory of a recording session (see the |
| 461 | nloption:--output option of the man:lttng-create(1) command and |
| 462 | of man:lttng-relayd(8)). |
| 463 | |
| 464 | A trace chunk archive contains, through tracing domain and possibly |
| 465 | UID/PID subdirectories, metadata and data stream files. |
| 466 | |
| 467 | A trace chunk archive is, at the same time: |
| 468 | |
| 469 | * A self-contained LTTng trace. |
| 470 | |
| 471 | * A member of a set of trace chunk archives which form the complete |
| 472 | trace of a recording session. |
| 473 | |
| 474 | In other words, an LTTng trace reader can read both the recording |
| 475 | session output directory (all the trace chunk archives), or a |
| 476 | single trace chunk archive. |
| 477 | |
| 478 | When LTTng performs a recording session rotation, it names the resulting |
| 479 | trace chunk archive as such, relative to the output directory of the |
| 480 | recording session: |
| 481 | |
| 482 | [verse] |
| 483 | archives/__BEGIN__-__END__-__ID__ |
| 484 | |
| 485 | __BEGIN__:: |
| 486 | Date and time of the beginning of the trace chunk archive with |
| 487 | the ISO{nbsp}8601-compatible __YYYYmmddTHHMMSS±HHMM__ form, where |
| 488 | __YYYYmmdd__ is the date and __HHMMSS±HHMM__ is the time with the |
| 489 | time zone offset from UTC. |
| 490 | + |
| 491 | Example: `20171119T152407-0500` |
| 492 | |
| 493 | __END__:: |
| 494 | Date and time of the end of the trace chunk archive with |
| 495 | the ISO{nbsp}8601-compatible __YYYYmmddTHHMMSS±HHMM__ form, where |
| 496 | __YYYYmmdd__ is the date and __HHMMSS±HHMM__ is the time with the |
| 497 | time zone offset from UTC. |
| 498 | + |
| 499 | Example: `20180118T152407+0930` |
| 500 | |
| 501 | __ID__:: |
| 502 | Unique numeric identifier of the trace chunk within its recording |
| 503 | session. |
| 504 | |
| 505 | Trace chunk archive name example: |
| 506 | |
| 507 | ---- |
| 508 | archives/20171119T152407-0500-20171119T151422-0500-3 |
| 509 | ---- |
| 510 | |
| 511 | |
| 512 | [[domain]] |
| 513 | {sect-domain} |
| 514 | ------------- |
| 515 | A _tracing domain_ identifies a type of LTTng tracer. |
| 516 | |
| 517 | A tracing domain has its own properties and features. |
| 518 | |
| 519 | There are currently five available tracing domains: |
| 520 | |
| 521 | [options="header"] |
| 522 | |=== |
| 523 | |Tracing domain |``Event rule matches'' trigger condition option |Option for other CLI commands |
| 524 | |
| 525 | |Linux kernel |
| 526 | |nloption:--type option starts with `kernel:` |
| 527 | |nloption:--kernel |
| 528 | |
| 529 | |User space |
| 530 | |nloption:--type option starts with `user:` |
| 531 | |nloption:--userspace |
| 532 | |
| 533 | |`java.util.logging` (JUL) |
| 534 | |nloption:--type option starts with `jul:` |
| 535 | |nloption:--jul |
| 536 | |
| 537 | |Apache log4j |
| 538 | |nloption:--type option starts with `log4j:` |
| 539 | |nloption:--log4j |
| 540 | |
| 541 | |Python |
| 542 | |nloption:--type option starts with `python:` |
| 543 | |nloption:--python |
| 544 | |=== |
| 545 | |
| 546 | You must specify a tracing domain to target a type of LTTng tracer when |
| 547 | using some man:lttng(1) commands to avoid ambiguity. For example, |
| 548 | because the Linux kernel and user space tracing domains support named |
| 549 | tracepoints as instrumentation points (see the |
| 550 | ``<<"event-rule","{sect-event-rule}">>'' section above), you need to |
| 551 | specify a tracing domain when you create an event rule because both |
| 552 | tracing domains could have tracepoints sharing the same name. |
| 553 | |
| 554 | You can create channels (see the ``<<channel,{sect-channel}>>'' section |
| 555 | below) in the Linux kernel and user space tracing domains. The other |
| 556 | tracing domains have a single, default channel. |
| 557 | |
| 558 | |
| 559 | [[channel]] |
| 560 | {sect-channel} |
| 561 | -------------- |
| 562 | A _channel_ is an object which is responsible for a set of ring buffers. |
| 563 | |
| 564 | Each ring buffer is divided into multiple _sub-buffers_. When a |
| 565 | recording event rule (see the |
| 566 | ``<<recording-event-rule,{sect-recording-event-rule}>>'' section below) |
| 567 | matches an event, LTTng can record it to one or more sub-buffers of one |
| 568 | or more channels. |
| 569 | |
| 570 | When you create a channel with the man:lttng-enable-channel(1) command, |
| 571 | you set its final attributes, that is: |
| 572 | |
| 573 | * Its buffering scheme. |
| 574 | + |
| 575 | See the ``<<channel-buf-scheme,Buffering scheme>>'' section below. |
| 576 | |
| 577 | * What to do when there's no |
| 578 | space left for a new event record because all sub-buffers are full. |
| 579 | + |
| 580 | See the ``<<channel-er-loss-mode,Event record loss mode>>'' section |
| 581 | below. |
| 582 | |
| 583 | * The size of each ring buffer and how many sub-buffers a ring buffer |
| 584 | has. |
| 585 | + |
| 586 | See the ``<<channel-sub-buf-size-count,Sub-buffer size and count>>'' |
| 587 | section below. |
| 588 | |
| 589 | * The size of each trace file LTTng writes for this channel and the |
| 590 | maximum count of trace files. |
| 591 | + |
| 592 | See the ``<<channel-max-trace-file-size-count,Maximum trace file size |
| 593 | and count>>'' section below. |
| 594 | |
| 595 | * The periods of its read, switch, and monitor timers. |
| 596 | + |
| 597 | See the ``<<channel-timers,Timers>>'' section below. |
| 598 | |
| 599 | * For a Linux kernel channel: its output type (man:mmap(2) or |
| 600 | man:splice(2)). |
| 601 | + |
| 602 | See the nloption:--output option of the man:lttng-enable-channel(1) |
| 603 | command. |
| 604 | |
| 605 | * For a user space channel: the value of its blocking timeout. |
| 606 | + |
| 607 | See the nloption:--blocking-timeout option of the |
| 608 | man:lttng-enable-channel(1) command. |
| 609 | |
| 610 | Note that the man:lttng-enable-event(1) command can automatically create |
| 611 | a default channel with sane defaults when no channel exists for the |
| 612 | provided tracing domain. |
| 613 | |
| 614 | A channel is always associated to a tracing domain (see the |
| 615 | ``<<domain,{sect-domain}>>'' section below). The `java.util.logging` |
| 616 | (JUL), log4j, and Python tracing domains each have a default channel |
| 617 | which you can't configure. |
| 618 | |
| 619 | A channel owns recording event rules. |
| 620 | |
| 621 | List the channels of a given recording session with the |
| 622 | man:lttng-list(1) and man:lttng-status(1) commands. |
| 623 | |
| 624 | Disable an enabled channel with the man:lttng-disable-channel(1) |
| 625 | command. |
| 626 | |
| 627 | |
| 628 | [[channel-buf-scheme]] |
| 629 | Buffering scheme |
| 630 | ~~~~~~~~~~~~~~~~ |
| 631 | A channel has at least one ring buffer per CPU. LTTng always records an |
| 632 | event to the ring buffer dedicated to the CPU which emits it. |
| 633 | |
| 634 | The buffering scheme of a user space channel determines what has its own |
| 635 | set of per-CPU ring buffers: |
| 636 | |
| 637 | Per-user buffering (nloption:--buffers-uid option of the man:lttng-enable-channel(1) command):: |
| 638 | Allocate one set of ring buffers (one per CPU) shared by all the |
| 639 | instrumented processes of: |
| 640 | If your Unix user is `root`::: |
| 641 | Each Unix user. |
| 642 | Otherwise::: |
| 643 | Your Unix user. |
| 644 | |
| 645 | Per-process buffering (nloption:--buffers-pid option of the man:lttng-enable-channel(1) command):: |
| 646 | Allocate one set of ring buffers (one per CPU) for each instrumented |
| 647 | process of: |
| 648 | If your Unix user is `root`::: |
| 649 | All Unix users. |
| 650 | Otherwise::: |
| 651 | Your Unix user. |
| 652 | |
| 653 | The per-process buffering scheme tends to consume more memory than the |
| 654 | per-user option because systems generally have more instrumented |
| 655 | processes than Unix users running instrumented processes. However, the |
| 656 | per-process buffering scheme ensures that one process having a high |
| 657 | event throughput won't fill all the shared sub-buffers of the same Unix |
| 658 | user, only its own. |
| 659 | |
| 660 | The buffering scheme of a Linux kernel channel is always to allocate a |
| 661 | single set of ring buffers for the whole system. This scheme is similar |
| 662 | to the per-user option, but with a single, global user ``running'' the |
| 663 | kernel. |
| 664 | |
| 665 | |
| 666 | [[channel-er-loss-mode]] |
| 667 | Event record loss mode |
| 668 | ~~~~~~~~~~~~~~~~~~~~~~ |
| 669 | When LTTng emits an event, LTTng can record it to a specific, available |
| 670 | sub-buffer within the ring buffers of specific channels. When there's no |
| 671 | space left in a sub-buffer, the tracer marks it as consumable and |
| 672 | another, available sub-buffer starts receiving the following event |
| 673 | records. An LTTng consumer daemon eventually consumes the marked |
| 674 | sub-buffer, which returns to the available state. |
| 675 | |
| 676 | In an ideal world, sub-buffers are consumed faster than they are filled. |
| 677 | In the real world, however, all sub-buffers can be full at some point, |
| 678 | leaving no space to record the following events. |
| 679 | |
| 680 | By default, LTTng-modules and LTTng-UST are _non-blocking_ tracers: when |
| 681 | there's no available sub-buffer to record an event, it's acceptable to |
| 682 | lose event records when the alternative would be to cause substantial |
| 683 | delays in the execution of the instrumented application. LTTng |
| 684 | privileges performance over integrity; it aims at perturbing the |
| 685 | instrumented application as little as possible in order to make the |
| 686 | detection of subtle race conditions and rare interrupt cascades |
| 687 | possible. |
| 688 | |
| 689 | Since LTTng{nbsp}2.10, the LTTng user space tracer, LTTng-UST, supports |
| 690 | a _blocking mode_. See the nloption:--blocking-timeout of the |
| 691 | man:lttng-enable-channel(1) command to learn how to use the blocking |
| 692 | mode. |
| 693 | |
| 694 | When it comes to losing event records because there's no available |
| 695 | sub-buffer, or because the blocking timeout of the channel is |
| 696 | reached, the _event record loss mode_ of the channel determines what to |
| 697 | do. The available event record loss modes are: |
| 698 | |
| 699 | Discard mode:: |
| 700 | Drop the newest event records until a sub-buffer becomes available. |
| 701 | + |
| 702 | This is the only available mode when you specify a blocking timeout. |
| 703 | + |
| 704 | With this mode, LTTng increments a count of lost event records when an |
| 705 | event record is lost and saves this count to the trace. A trace reader |
| 706 | can use the saved discarded event record count of the trace to decide |
| 707 | whether or not to perform some analysis even if trace data is known to |
| 708 | be missing. |
| 709 | |
| 710 | Overwrite mode:: |
| 711 | Clear the sub-buffer containing the oldest event records and start |
| 712 | writing the newest event records there. |
| 713 | + |
| 714 | This mode is sometimes called _flight recorder mode_ because it's |
| 715 | similar to a https://en.wikipedia.org/wiki/Flight_recorder[flight |
| 716 | recorder]: always keep a fixed amount of the latest data. It's also |
| 717 | similar to the roll mode of an oscilloscope. |
| 718 | + |
| 719 | Since LTTng{nbsp}2.8, with this mode, LTTng writes to a given sub-buffer |
| 720 | its sequence number within its data stream. With a local, network |
| 721 | streaming, or live recording session (see the |
| 722 | ``<<session-modes,Recording session modes>>'' section above), a trace |
| 723 | reader can use such sequence numbers to report lost packets. A trace |
| 724 | reader can use the saved discarded sub-buffer (packet) count of the |
| 725 | trace to decide whether or not to perform some analysis even if trace |
| 726 | data is known to be missing. |
| 727 | + |
| 728 | With this mode, LTTng doesn't write to the trace the exact number of |
| 729 | lost event records in the lost sub-buffers. |
| 730 | |
| 731 | Which mechanism you should choose depends on your context: prioritize |
| 732 | the newest or the oldest event records in the ring buffer? |
| 733 | |
| 734 | Beware that, in overwrite mode, the tracer abandons a _whole sub-buffer_ |
| 735 | as soon as a there's no space left for a new event record, whereas in |
| 736 | discard mode, the tracer only discards the event record that doesn't |
| 737 | fit. |
| 738 | |
| 739 | Set the event record loss mode of a channel with the nloption:--discard |
| 740 | and nloption:--overwrite options of the man:lttng-enable-channel(1) |
| 741 | command. |
| 742 | |
| 743 | There are a few ways to decrease your probability of losing event |
| 744 | records. The ``<<channel-sub-buf-size-count,Sub-buffer size and |
| 745 | count>>'' section below shows how to fine-tune the sub-buffer size and |
| 746 | count of a channel to virtually stop losing event records, though at the |
| 747 | cost of greater memory usage. |
| 748 | |
| 749 | |
| 750 | [[channel-sub-buf-size-count]] |
| 751 | Sub-buffer size and count |
| 752 | ~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 753 | A channel has one or more ring buffer for each CPU of the target system. |
| 754 | |
| 755 | See the ``<<channel-buf-scheme,Buffering scheme>>'' section above to |
| 756 | learn how many ring buffers of a given channel are dedicated to each CPU |
| 757 | depending on its buffering scheme. |
| 758 | |
| 759 | Set the size of each sub-buffer the ring buffers of a channel contain |
| 760 | with the nloption:--subbuf-size option of the |
| 761 | man:lttng-enable-channel(1) command. |
| 762 | |
| 763 | Set the number of sub-buffers each ring buffer of a channel contains |
| 764 | with the nloption:--num-subbuf option of the man:lttng-enable-channel(1) |
| 765 | command. |
| 766 | |
| 767 | Note that LTTng switching the current sub-buffer of a ring buffer |
| 768 | (marking a full one as consumable and switching to an available one for |
| 769 | LTTng to record the next events) introduces noticeable CPU overhead. |
| 770 | Knowing this, the following list presents a few practical situations |
| 771 | along with how to configure the sub-buffer size and count for them: |
| 772 | |
| 773 | High event throughput:: |
| 774 | In general, prefer large sub-buffers to lower the risk of losing |
| 775 | event records. |
| 776 | + |
| 777 | Having larger sub-buffers also ensures a lower sub-buffer switching |
| 778 | frequency (see the ``<<channel-timers,Timers>>'' section below). |
| 779 | + |
| 780 | The sub-buffer count is only meaningful if you create the channel in |
| 781 | overwrite mode (see the ``<<channel-er-loss-mode,Event record loss |
| 782 | mode>>'' section above): in this case, if LTTng overwrites a sub-buffer, |
| 783 | then the other sub-buffers are left unaltered. |
| 784 | |
| 785 | Low event throughput:: |
| 786 | In general, prefer smaller sub-buffers since the risk of losing |
| 787 | event records is low. |
| 788 | + |
| 789 | Because LTTng emits events less frequently, the sub-buffer switching |
| 790 | frequency should remain low and therefore the overhead of the tracer |
| 791 | shouldn't be a problem. |
| 792 | |
| 793 | Low memory system:: |
| 794 | If your target system has a low memory limit, prefer fewer first, |
| 795 | then smaller sub-buffers. |
| 796 | + |
| 797 | Even if the system is limited in memory, you want to keep the |
| 798 | sub-buffers as large as possible to avoid a high sub-buffer switching |
| 799 | frequency. |
| 800 | |
| 801 | Note that LTTng uses https://diamon.org/ctf/[CTF] as its trace format, |
| 802 | which means event record data is very compact. For example, the average |
| 803 | LTTng kernel event record weights about 32{nbsp}bytes. Therefore, a |
| 804 | sub-buffer size of 1{nbsp}MiB is considered large. |
| 805 | |
| 806 | The previous scenarios highlight the major trade-off between a few large |
| 807 | sub-buffers and more, smaller sub-buffers: sub-buffer switching |
| 808 | frequency vs. how many event records are lost in overwrite mode. |
| 809 | Assuming a constant event throughput and using the overwrite mode, the |
| 810 | two following configurations have the same ring buffer total size: |
| 811 | |
| 812 | Two sub-buffers of 4{nbsp}MiB each:: |
| 813 | Expect a very low sub-buffer switching frequency, but if LTTng |
| 814 | ever needs to overwrite a sub-buffer, half of the event records so |
| 815 | far (4{nbsp}MiB) are definitely lost. |
| 816 | |
| 817 | Eight sub-buffers of 1{nbsp}MiB each:: |
| 818 | Expect four times the tracer overhead of the configuration above, |
| 819 | but if LTTng needs to overwrite a sub-buffer, only the eighth of |
| 820 | event records so far (1{nbsp}MiB) are definitely lost. |
| 821 | |
| 822 | In discard mode, the sub-buffer count parameter is pointless: use two |
| 823 | sub-buffers and set their size according to your requirements. |
| 824 | |
| 825 | |
| 826 | [[channel-max-trace-file-size-count]] |
| 827 | Maximum trace file size and count |
| 828 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 829 | By default, trace files can grow as large as needed. |
| 830 | |
| 831 | Set the maximum size of each trace file that LTTng writes of a given |
| 832 | channel with the nloption:--tracefile-size option of the man:lttng-enable-channel(1) |
| 833 | command. |
| 834 | |
| 835 | When the size of a trace file reaches the fixed maximum size of the |
| 836 | channel, LTTng creates another file to contain the next event records. |
| 837 | LTTng appends a file count to each trace file name in this case. |
| 838 | |
| 839 | If you set the trace file size attribute when you create a channel, the |
| 840 | maximum number of trace files that LTTng creates is _unlimited_ by |
| 841 | default. To limit them, use the nloption:--tracefile-count option of |
| 842 | man:lttng-enable-channel(1). When the number of trace files reaches the |
| 843 | fixed maximum count of the channel, LTTng overwrites the oldest trace |
| 844 | file. This mechanism is called _trace file rotation_. |
| 845 | |
| 846 | [IMPORTANT] |
| 847 | ==== |
| 848 | Even if you don't limit the trace file count, always assume that LTTng |
| 849 | manages all the trace files of the recording session. |
| 850 | |
| 851 | In other words, there's no safe way to know if LTTng still holds a given |
| 852 | trace file open with the trace file rotation feature. |
| 853 | |
| 854 | The only way to obtain an unmanaged, self-contained LTTng trace before |
| 855 | you destroy the recording session is with the recording session rotation |
| 856 | feature (see the ``<<rotation,Recording session rotation>>'' section |
| 857 | above), which is available since LTTng{nbsp}2.11. |
| 858 | ==== |
| 859 | |
| 860 | |
| 861 | [[channel-timers]] |
| 862 | Timers |
| 863 | ~~~~~~ |
| 864 | Each channel can have up to three optional timers: |
| 865 | |
| 866 | Switch timer:: |
| 867 | When this timer expires, a sub-buffer switch happens: for each ring |
| 868 | buffer of the channel, LTTng marks the current sub-buffer as |
| 869 | consumable and switches to an available one to record the next |
| 870 | events. |
| 871 | + |
| 872 | A switch timer is useful to ensure that LTTng consumes and commits trace |
| 873 | data to trace files or to a distant relay daemon (man:lttng-relayd(8)) |
| 874 | periodically in case of a low event throughput. |
| 875 | + |
| 876 | Such a timer is also convenient when you use large sub-buffers (see the |
| 877 | ``<<channel-sub-buf-size-count,Sub-buffer size and count>>'' section |
| 878 | above) to cope with a sporadic high event throughput, even if the |
| 879 | throughput is otherwise low. |
| 880 | + |
| 881 | Set the period of the switch timer of a channel, or disable the timer |
| 882 | altogether, with the nloption:--switch-timer option of the |
| 883 | man:lttng-enable-channel(1) command. |
| 884 | |
| 885 | Read timer:: |
| 886 | When this timer expires, LTTng checks for full, consumable |
| 887 | sub-buffers. |
| 888 | + |
| 889 | By default, the LTTng tracers use an asynchronous message mechanism to |
| 890 | signal a full sub-buffer so that a consumer daemon can consume it. |
| 891 | + |
| 892 | When such messages must be avoided, for example in real-time |
| 893 | applications, use this timer instead. |
| 894 | + |
| 895 | Set the period of the read timer of a channel, or disable the timer |
| 896 | altogether, with the nloption:--read-timer option of the |
| 897 | man:lttng-enable-channel(1) command. |
| 898 | |
| 899 | Monitor timer:: |
| 900 | When this timer expires, the consumer daemon samples some channel |
| 901 | statistics to evaluate the following trigger conditions: |
| 902 | + |
| 903 | -- |
| 904 | . The consumed buffer size of a given recording session becomes greater |
| 905 | than some value. |
| 906 | . The buffer usage of a given channel becomes greater than some value. |
| 907 | . The buffer usage of a given channel becomes less than some value. |
| 908 | -- |
| 909 | + |
| 910 | If you disable the monitor timer of a channel{nbsp}__C__: |
| 911 | + |
| 912 | -- |
| 913 | * The consumed buffer size value of the recording session of{nbsp}__C__ |
| 914 | could be wrong for trigger condition type{nbsp}1: the consumed buffer |
| 915 | size of{nbsp}__C__ won't be part of the grand total. |
| 916 | |
| 917 | * The buffer usage trigger conditions (types{nbsp}2 and{nbsp}3) |
| 918 | for{nbsp}__C__ will never be satisfied. |
| 919 | -- |
| 920 | + |
| 921 | See the ``<<trigger,TRIGGER>>'' section above to learn more about |
| 922 | triggers. |
| 923 | + |
| 924 | Set the period of the monitor timer of a channel, or disable the timer |
| 925 | altogether, with the nloption:--monitor-timer option of the |
| 926 | man:lttng-enable-channel(1) command. |
| 927 | |
| 928 | |
| 929 | [[recording-event-rule]] |
| 930 | {sect-recording-event-rule} |
| 931 | --------------------------- |
| 932 | A _recording event rule_ is a specific type of event rule (see the |
| 933 | ``<<"event-rule","{sect-event-rule}">>'' section above) of which the |
| 934 | action is to serialize and record the matched event as an _event |
| 935 | record_. |
| 936 | |
| 937 | Set the explicit conditions of a recording event rule when you create it |
| 938 | with the man:lttng-enable-event(1) command. A recording event rule also |
| 939 | has the following implicit conditions: |
| 940 | |
| 941 | * The recording event rule itself is enabled. |
| 942 | + |
| 943 | A recording event rule is enabled on creation. |
| 944 | |
| 945 | * The channel to which the recording event rule is attached is enabled. |
| 946 | + |
| 947 | A channel is enabled on creation. |
| 948 | + |
| 949 | See the ``<<channel,{sect-channel}>>'' section above. |
| 950 | |
| 951 | * The recording session of the recording event rule is active (started). |
| 952 | + |
| 953 | A recording session is inactive (stopped) on creation. |
| 954 | + |
| 955 | See the ``<<session,{sect-session}>>'' section above. |
| 956 | |
| 957 | * The process for which LTTng creates an event to match is allowed to |
| 958 | record events. |
| 959 | + |
| 960 | All processes are allowed to record events on recording session |
| 961 | creation. |
| 962 | + |
| 963 | Use the man:lttng-track(1) and man:lttng-untrack(1) commands to select |
| 964 | which processes are allowed to record events based on specific process |
| 965 | attributes. |
| 966 | |
| 967 | You always attach a recording event rule to a channel, which belongs to |
| 968 | a recording session, when you create it. |
| 969 | |
| 970 | When a recording event rule{nbsp}__ER__ matches an event{nbsp}__E__, |
| 971 | LTTng attempts to serialize and record{nbsp}__E__ to one of the |
| 972 | available sub-buffers of the channel to which{nbsp}__E__ is attached. |
| 973 | |
| 974 | When multiple matching recording event rules are attached to the same |
| 975 | channel, LTTng attempts to serialize and record the matched event |
| 976 | _once_. In the following example, the second recording event rule is |
| 977 | redundant when both are enabled: |
| 978 | |
| 979 | [role="term"] |
| 980 | ---- |
| 981 | $ lttng enable-event --userspace hello:world |
| 982 | $ lttng enable-event --userspace hello:world --loglevel=INFO |
| 983 | ---- |
| 984 | |
| 985 | List the recording event rules of a specific recording session |
| 986 | and/or channel with the man:lttng-list(1) and man:lttng-status(1) |
| 987 | commands. |
| 988 | |
| 989 | Disable a recording event rule with the man:lttng-disable-event(1) |
| 990 | command. |
| 991 | |
| 992 | As of LTTng{nbsp}{lttng_version}, you cannot remove a recording event |
| 993 | rule: it exists as long as its recording session exists. |
| 994 | |
| 995 | |
| 996 | include::common-footer.txt[] |
| 997 | |
| 998 | |
| 999 | SEE ALSO |
| 1000 | -------- |
| 1001 | man:lttng(1), |
| 1002 | man:lttng-relayd(8), |
| 1003 | man:lttng-sessiond(8) |