Commit | Line | Data |
---|---|---|
26f0c779 PP |
1 | lttng-concepts(7) |
2 | ================= | |
3 | :revdate: 3 May 2021 | |
4 | :sect-event-rule: INSTRUMENTATION POINT, EVENT RULE, AND EVENT | |
5 | :sect-session: TRACING SESSION | |
6 | :sect-domain: TRACING DOMAIN | |
7 | :sect-channel: CHANNEL AND RING BUFFER | |
8 | :sect-recording-event-rule: RECORDING EVENT RULE AND EVENT RECORD | |
9 | ||
10 | ||
11 | NAME | |
12 | ---- | |
13 | lttng-concepts - LTTng concepts | |
14 | ||
15 | ||
16 | DESCRIPTION | |
17 | ----------- | |
18 | This manual page documents the concepts of LTTng. | |
19 | ||
20 | Many other LTTng manual pages refer to this one so that you can | |
21 | understand what are the various LTTng objects and how they relate to | |
22 | each other. | |
23 | ||
24 | The concepts of LTTng{nbsp}{lttng_version} are: | |
25 | ||
26 | * Instrumentation point, event rule, and event | |
27 | * Trigger | |
28 | * Tracing session | |
29 | * Tracing domain | |
30 | * Channel and ring buffer | |
31 | * Recording event rule and event record | |
32 | ||
33 | ||
34 | [[event-rule]] | |
35 | {sect-event-rule} | |
36 | ----------------- | |
37 | An _instrumentation point_ is a point, within a piece of software, | |
38 | which, when executed, creates an LTTng _event_. | |
39 | ||
40 | LTTng offers various types of instrumentation; see the | |
41 | <<inst-point-types,Instrumentation point types>> section below to learn | |
42 | about them. | |
43 | ||
44 | An _event rule_ is a set of conditions to match a set of events. | |
45 | ||
46 | When LTTng creates an event{nbsp}__E__, an event rule{nbsp}__ER__ is | |
47 | said to __match__{nbsp}__E__ when{nbsp}__E__ satisfies *all* the | |
48 | conditions of{nbsp}__ER__. This concept is similar to a regular | |
49 | expression which matches a set of strings. | |
50 | ||
51 | When an event rule matches an event, LTTng _emits_ the event, therefore | |
52 | attempting to execute one or more actions. | |
53 | ||
54 | [IMPORTANT] | |
55 | ==== | |
56 | The event creation and emission processes are documentation concepts to | |
57 | help understand the journey from an instrumentation point to the | |
58 | execution of actions. | |
59 | ||
60 | The actual creation of an event can be costly because LTTng needs to | |
61 | evalute the arguments of the instrumentation point. | |
62 | ||
63 | In practice, LTTng implements various optimizations for the Linux kernel | |
64 | and user space tracing domains (see the <<domain,{sect-domain}>> section | |
65 | below) to avoid actually creating an event | |
66 | when the tracer knows, thanks to properties which are independent from | |
67 | the event payload and current context, that it would never emit such an | |
68 | event. Those properties are: | |
69 | ||
70 | * The instrumentation point type (see the | |
71 | <<inst-point-types,Instrumentation point types>> section below). | |
72 | ||
73 | * The instrumentation point name. | |
74 | ||
75 | * The instrumentation point log level. | |
76 | ||
77 | * For a recording event rule (see the | |
78 | <<recording-event-rule,{sect-recording-event-rule}>> section | |
79 | below): | |
80 | ** The status of the rule itself. | |
81 | ** The status of the channel (see the <<channel,{sect-channel}>> section | |
82 | below). | |
83 | ** The activity of the tracing session (started or stopped; see | |
84 | the <<session,{sect-session}>> section below). | |
85 | ** Whether or not the process for which LTTng would create the event is | |
86 | allowed to record events (see man:lttng-track(1)). | |
87 | ||
88 | In other words: if, for a given instrumentation point{nbsp}__IP__, the | |
89 | LTTng tracer knows that it would never emit an event, | |
90 | executing{nbsp}__IP__ represents a simple boolean variable check and, | |
91 | for a Linux kernel recording event rule, a few process attribute checks. | |
92 | ==== | |
93 | ||
94 | As of LTTng{nbsp}{lttng_version}, there are two places where you can | |
95 | find an event rule: | |
96 | ||
97 | Recording event rule:: | |
98 | A specific type of event rule of which the action is to record the | |
99 | matched event as an event record. | |
100 | + | |
101 | See the <<recording-event-rule,{sect-recording-event-rule}>> section | |
102 | below. | |
103 | + | |
104 | Create or enable a recording event rule with the | |
105 | man:lttng-enable-event(1) command. | |
106 | + | |
107 | List the recording event rules of a specific tracing session | |
108 | and/or channel with the man:lttng-list(1) and man:lttng-status(1) | |
109 | commands. | |
110 | ||
111 | ``Event rule matches'' <<trigger,trigger>> condition (since LTTng{nbsp}2.13):: | |
112 | When the event rule of the trigger condition matches an event, LTTng | |
113 | can execute user-defined actions such as sending an LTTng | |
114 | notification, starting a tracing session, and more. | |
115 | + | |
116 | See man:lttng-add-trigger(1) and man:lttng-event-rule(7). | |
117 | ||
118 | For LTTng to emit an event{nbsp}__E__,{nbsp}__E__ must satisfy *all* the | |
119 | basic conditions of an event rule{nbsp}__ER__, that is: | |
120 | ||
121 | * The instrumentation point from which LTTng creates{nbsp}__E__ has a | |
122 | specific type. | |
123 | + | |
124 | See the <<inst-point-types,Instrumentation point types>> section below. | |
125 | ||
126 | * A pattern matches the name of{nbsp}__E__ while another pattern | |
127 | doesn't. | |
128 | ||
129 | * The log level of the instrumentation point from which LTTng | |
130 | creates{nbsp}__E__ is at least as severe as some value, or is exactly | |
131 | some value. | |
132 | ||
133 | * The fields of the payload of{nbsp}__E__ and the current context fields | |
134 | satisfy a filter expression. | |
135 | ||
136 | A recording event rule has additional, implicit conditions to satisfy. | |
137 | See the <<recording-event-rule,{sect-recording-event-rule}>> section | |
138 | below to learn more. | |
139 | ||
140 | ||
141 | [[inst-point-types]] | |
142 | Instrumentation point types | |
143 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
144 | As of LTTng{nbsp}{lttng_version}, the available instrumentation point | |
145 | types are, depending on the tracing domain (see the | |
146 | <<domain,{sect-domain}>> section below): | |
147 | ||
148 | Linux kernel:: | |
149 | LTTng tracepoint::: | |
150 | A statically defined point in the source code of the kernel | |
151 | image or of a kernel module using the LTTng-modules macros. | |
152 | + | |
153 | List the available Linux kernel tracepoints with `lttng list --kernel`. | |
154 | See man:lttng-list(1) to learn more. | |
155 | ||
156 | Linux kernel system call::: | |
157 | Entry, exit, or both of a Linux kernel system call. | |
158 | + | |
159 | List the available Linux kernel system call instrumentation points with | |
160 | `lttng list --kernel --syscall`. See man:lttng-list(1) to learn more. | |
161 | ||
162 | Linux kprobe::: | |
163 | A single probe dynamically placed in the compiled kernel code. | |
164 | + | |
165 | When you create such an instrumentation point, you set its memory | |
166 | address or symbol name. | |
167 | ||
168 | Linux user space probe::: | |
169 | A single probe dynamically placed at the entry of a compiled | |
170 | user space application/library function through the kernel. | |
171 | + | |
172 | When you create such an instrumentation point, you set: | |
173 | + | |
174 | -- | |
175 | With the ELF method:: | |
176 | Its application/library path and its symbol name. | |
177 | ||
178 | With the USDT method:: | |
179 | Its application/library path, its provider name, and its probe name. | |
180 | + | |
181 | ``USDT'' stands for SystemTap User-level Statically Defined Tracing, | |
182 | a DTrace-style marker. | |
183 | -- | |
184 | + | |
185 | As of LTTng{nbsp}{lttng_version}, LTTng only supports USDT probes which | |
186 | are :not: reference-counted. | |
187 | ||
188 | Linux kretprobe::: | |
189 | Entry, exit, or both of a Linux kernel function. | |
190 | + | |
191 | When you create such an instrumentation point, you set the memory | |
192 | address or symbol name of its function. | |
193 | ||
194 | User space:: | |
195 | LTTng tracepoint::: | |
196 | A statically defined point in the source code of a C/$$C++$$ | |
197 | application/library using the LTTng-UST macros. | |
198 | + | |
199 | List the available Linux kernel tracepoints with | |
200 | `lttng list --userspace`. See man:lttng-list(1) to learn more. | |
201 | ||
202 | `java.util.logging`, Apache log4j, and Python:: | |
203 | Java or Python logging statement::: | |
204 | A method call on a Java or Python logger attached to an | |
205 | LTTng-UST handler. | |
206 | + | |
207 | List the available Java and Python loggers with `lttng list --jul`, | |
208 | `lttng list --log4j`, and `lttng list --python`. See man:lttng-list(1) | |
209 | to learn more. | |
210 | ||
211 | ||
212 | [[trigger]] | |
213 | TRIGGER | |
214 | ------- | |
215 | A _trigger_ associates a condition to one or more actions. | |
216 | ||
217 | When the condition of a trigger is satisfied, LTTng attempts to execute | |
218 | its actions. | |
219 | ||
220 | As of LTTng{nbsp}{lttng_version}, the available trigger conditions and | |
221 | actions are: | |
222 | ||
223 | Conditions:: | |
224 | + | |
225 | * The consumed buffer size of a given tracing | |
226 | session (see the <<session,{sect-session}>> section below) | |
227 | becomes greater than some value. | |
228 | ||
229 | * The buffer usage of a given channel (see the | |
230 | <<channel,{sect-channel}>> section below) becomes greater than some | |
231 | value. | |
232 | ||
233 | * The buffer usage of a given channel becomes less than some value. | |
234 | ||
235 | * There's an ongoing tracing session rotation (see the | |
236 | <<rotation,Tracing session rotation>> section below). | |
237 | ||
238 | * A tracing session rotation becomes completed. | |
239 | ||
240 | * An event rule matches an event. | |
241 | + | |
242 | As of LTTng{nbsp}{lttng_version}, this is the only available condition | |
243 | when you add a trigger with the man:lttng-add-trigger(1) command. The | |
244 | other ones are available through the liblttng-ctl C{nbsp}API. | |
245 | ||
246 | Actions:: | |
247 | + | |
248 | * Send a notification to a user application. | |
249 | * Start a given tracing session, like man:lttng-start(1) would do. | |
250 | * Stop a given tracing session, like man:lttng-stop(1) would do. | |
251 | * Archive the current trace chunk of a given tracing session (rotate), | |
252 | like man:lttng-rotate(1) would do. | |
253 | * Take a snapshot of a given tracing session, like man:lttng-snapshot(1) | |
254 | would do. | |
255 | ||
256 | A trigger belongs to a session daemon (see man:lttng-sessiond(8)), not | |
257 | to a specific tracing session. For a given session daemon, each Unix | |
258 | user has its own, private triggers. Note, however, that the `root` Unix | |
259 | user may, for the root session daemon: | |
260 | ||
261 | * Add a trigger as another Unix user. | |
262 | ||
263 | * List all the triggers, regardless of their owner. | |
264 | ||
265 | * Remove a trigger which belongs to another Unix user. | |
266 | ||
267 | For a given session daemon and Unix user, a trigger has a unique name. | |
268 | ||
269 | Add a trigger to a session daemon with the man:lttng-add-trigger(1) | |
270 | command. | |
271 | ||
272 | List the triggers of your Unix user (or of all users if your | |
273 | Unix user is `root`) with the man:lttng-list-triggers(1) command. | |
274 | ||
275 | Remove a trigger with the man:lttng-remove-trigger(1) command. | |
276 | ||
277 | ||
278 | [[session]] | |
279 | {sect-session} | |
280 | -------------- | |
281 | A _tracing session_ is a stateful dialogue between you and a session | |
282 | daemon (see man:lttng-sessiond(8)) for everything related to event | |
283 | recording. | |
284 | ||
285 | Everything that you do when you control LTTng tracers to record events | |
286 | happens within a tracing session. In particular, a tracing session: | |
287 | ||
288 | * Has its own name, unique for a given session daemon. | |
289 | ||
290 | * Has its own set of trace files, if any. | |
291 | ||
292 | * Has its own state of activity (started or stopped). | |
293 | + | |
294 | An active tracing session is an implicit recording event rule condition | |
295 | (see the <<recording-event-rule,{sect-recording-event-rule}>> section | |
296 | below). | |
297 | ||
298 | * Has its own mode (local, network streaming, snapshot, or live). | |
299 | + | |
300 | See the <<session-modes,Tracing session modes>> section below to learn | |
301 | more. | |
302 | ||
303 | * Has its own channels (see the <<channel,{sect-channel}>> section | |
304 | below) to which are attached their own recording event rules. | |
305 | ||
306 | * Has its own process attribute inclusion sets (see man:lttng-track(1)). | |
307 | ||
308 | Those attributes and objects are completely isolated between different | |
309 | tracing sessions. | |
310 | ||
311 | A tracing session is like an ATM session: the operations you do on the | |
312 | banking system through the ATM don't alter the data of other users of | |
313 | the same system. In the case of the ATM, a session lasts as long as your | |
314 | bank card is inside. In the case of LTTng, a tracing session lasts from | |
315 | the man:lttng-create(1) command to the man:lttng-destroy(1) command. | |
316 | ||
317 | A tracing session belongs to a session daemon (see | |
318 | man:lttng-sessiond(8)). For a given session daemon, each Unix user has | |
319 | its own, private tracing sessions. Note, however, that the `root` Unix | |
320 | user may operate on or destroy another user's tracing session. | |
321 | ||
322 | Create a tracing session with the man:lttng-create(1) command. | |
323 | ||
324 | List the tracing sessions of the connected session daemon with | |
325 | the man:lttng-list(1) command. | |
326 | ||
327 | Start and stop a tracing session with the man:lttng-start(1) and | |
328 | man:lttng-stop(1) commands. | |
329 | ||
330 | Save and load a tracing session with the man:lttng-save(1) and | |
331 | man:lttng-load(1) commands. | |
332 | ||
333 | Archive the current trace chunk of (rotate) a tracing session with the | |
334 | man:lttng-rotate(1) command. | |
335 | ||
336 | Destroy a tracing session with the man:lttng-destroy(1) command. | |
337 | ||
338 | ||
339 | Current tracing session | |
340 | ~~~~~~~~~~~~~~~~~~~~~~~ | |
341 | When you run the man:lttng-create(1) command, LTTng creates the | |
342 | `$LTTNG_HOME/.lttngrc` file if it doesn't exist (`$LTTNG_HOME` defaults | |
343 | to `$HOME`). | |
344 | ||
345 | `$LTTNG_HOME/.lttngrc` contains the name of the _current tracing | |
346 | session_. | |
347 | ||
348 | When you create a new tracing session with the `create` command, LTTng | |
349 | updates the current tracing session. | |
350 | ||
351 | The following man:lttng(1) commands select the current tracing session | |
352 | if you don't specify one: | |
353 | ||
354 | * man:lttng-add-context(1) | |
355 | * man:lttng-clear(1) | |
356 | * man:lttng-destroy(1) | |
357 | * man:lttng-disable-channel(1) | |
358 | * man:lttng-disable-event(1) | |
359 | * man:lttng-disable-rotation(1) | |
360 | * man:lttng-enable-channel(1) | |
361 | * man:lttng-enable-event(1) | |
362 | * man:lttng-enable-rotation(1) | |
363 | * man:lttng-regenerate(1) | |
364 | * man:lttng-rotate(1) | |
365 | * man:lttng-save(1) | |
366 | * man:lttng-snapshot(1) | |
367 | * man:lttng-start(1) | |
368 | * man:lttng-status(1) | |
369 | * man:lttng-stop(1) | |
370 | * man:lttng-track(1) | |
371 | * man:lttng-untrack(1) | |
372 | * man:lttng-view(1) | |
373 | ||
374 | Set the current tracing session manually with the | |
375 | man:lttng-set-session(1) command, without having to edit the `.lttngrc` | |
376 | file. | |
377 | ||
378 | ||
379 | [[session-modes]] | |
380 | Tracing session modes | |
381 | ~~~~~~~~~~~~~~~~~~~~~ | |
382 | LTTng offers four tracing session modes: | |
383 | ||
384 | Local mode:: | |
385 | Write the trace data to the local file system. | |
386 | ||
387 | Network streaming mode:: | |
388 | Send the trace data over the network to a listening relay daemon | |
389 | (see man:lttng-relayd(8)). | |
390 | ||
391 | Snapshot mode:: | |
392 | Only write the trace data to the local file system or send it to a | |
393 | listening relay daemon (man:lttng-relayd(8)) when LTTng takes a | |
394 | snapshot. | |
395 | + | |
396 | LTTng forces all the channels (see the <<channel,{sect-channel}>> | |
397 | section below) to be created to be configured to be snapshot-ready. | |
398 | + | |
399 | LTTng takes a snapshot of such a tracing session when: | |
400 | + | |
401 | -- | |
402 | * You run the man:lttng-snapshot(1) command. | |
403 | ||
404 | * LTTng executes a `snapshot-session` trigger action (see the | |
405 | <<trigger,TRIGGER>> section above). | |
406 | -- | |
407 | ||
408 | Live mode:: | |
409 | Send the trace data over the network to a listening relay daemon | |
410 | (see man:lttng-relayd(8)) for live reading. | |
411 | + | |
412 | An LTTng live reader (for example, man:babeltrace2(1)) can connect to | |
413 | the same relay daemon to receive trace data while the tracing session is | |
414 | active. | |
415 | ||
416 | ||
417 | [[rotation]] | |
418 | Tracing session rotation | |
419 | ~~~~~~~~~~~~~~~~~~~~~~~~ | |
420 | A _tracing session rotation_ is the action of archiving the current | |
421 | trace chunk of the tracing session to the file system. | |
422 | ||
423 | Once LTTng archives a trace chunk, it does :not: manage it anymore: you | |
424 | can read it, modify it, move it, or remove it. | |
425 | ||
426 | An _archived trace chunk_ is a collection of metadata and data stream | |
427 | files which form a self-contained LTTng trace. See the | |
428 | <<trace-chunk-naming,Trace chunk naming>> section below to learn how | |
429 | LTTng names a trace chunk archive directory. | |
430 | ||
431 | The _current trace chunk_ of a given tracing session includes: | |
432 | ||
433 | * The stream files which LTTng already wrote to the file system, and | |
434 | which are not part of a previously archived trace chunk, since the | |
435 | most recent event amongst: | |
436 | ||
437 | ** The first time the tracing session was started, either with the | |
438 | man:lttng-start(1) command or with a `start-session` trigger action | |
439 | (see the <<trigger,TRIGGER>> section above). | |
440 | ||
441 | ** The last rotation, performed with: | |
442 | ||
443 | *** An man:lttng-rotate(1) command. | |
444 | ||
445 | *** A rotation schedule previously set with | |
446 | man:lttng-enable-rotation(1). | |
447 | ||
448 | *** An executed `rotate-session` trigger action (see the | |
449 | <<trigger,TRIGGER>> section above). | |
450 | ||
451 | * The content of all the non-flushed sub-buffers of the channels of the | |
452 | tracing session. | |
453 | ||
454 | ||
455 | [[trace-chunk-naming]] | |
456 | Trace chunk archive naming | |
457 | ~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
458 | A trace chunk archive is a subdirectory of the `archives` subdirectory | |
459 | within the output directory of a tracing session (see the | |
460 | nloption:--output option of the man:lttng-create(1) command and | |
461 | of man:lttng-relayd(8)). | |
462 | ||
463 | A trace chunk archive contains, through tracing domain and possibly | |
464 | UID/PID subdirectories, metadata and data stream files. | |
465 | ||
466 | A trace chunk archive is, at the same time: | |
467 | ||
468 | * A self-contained LTTng trace. | |
469 | ||
470 | * A member of a set of trace chunk archives which form the complete | |
471 | trace of a tracing session. | |
472 | ||
473 | In other words, an LTTng trace reader can read both the tracing | |
474 | session output directory (all the trace chunk archives), or a | |
475 | single trace chunk archive. | |
476 | ||
477 | When LTTng performs a tracing session rotation, it names the resulting | |
478 | trace chunk archive as such, relative to the output directory of the | |
479 | tracing session: | |
480 | ||
481 | [verse] | |
482 | archives/__BEGIN__-__END__-__ID__ | |
483 | ||
484 | __BEGIN__:: | |
485 | Date and time of the beginning of the trace chunk archive with | |
486 | the ISO{nbsp}8601-compatible __YYYYmmddTHHMMSS±HHMM__ form, where | |
487 | __YYYYmmdd__ is the date and __HHMMSS±HHMM__ is the time with the | |
488 | time zone offset from UTC. | |
489 | + | |
490 | Example: `20171119T152407-0500` | |
491 | ||
492 | __END__:: | |
493 | Date and time of the end of the trace chunk archive with | |
494 | the ISO{nbsp}8601-compatible __YYYYmmddTHHMMSS±HHMM__ form, where | |
495 | __YYYYmmdd__ is the date and __HHMMSS±HHMM__ is the time with the | |
496 | time zone offset from UTC. | |
497 | + | |
498 | Example: `20180118T152407+0930` | |
499 | ||
500 | __ID__:: | |
501 | Unique numeric identifier of the trace chunk within its tracing | |
502 | session. | |
503 | ||
504 | Trace chunk archive name example: | |
505 | ||
506 | ---- | |
507 | archives/20171119T152407-0500-20171119T151422-0500-3 | |
508 | ---- | |
509 | ||
510 | ||
511 | [[domain]] | |
512 | {sect-domain} | |
513 | ------------- | |
514 | A _tracing domain_ identifies a type of LTTng tracer. | |
515 | ||
516 | A tracing domain has its own properties and features. | |
517 | ||
518 | There are currently five available tracing domains: | |
519 | ||
520 | [options="header"] | |
521 | |=== | |
522 | |Tracing domain |``Event rule matches'' trigger condition option |Option for other CLI commands | |
523 | ||
524 | |Linux kernel | |
525 | |nloption:--domain=++kernel++ | |
526 | |nloption:--kernel | |
527 | ||
528 | |User space | |
529 | |nloption:--domain=++user++ | |
530 | |nloption:--userspace | |
531 | ||
532 | |`java.util.logging` (JUL) | |
533 | |nloption:--domain=++jul++ | |
534 | |nloption:--jul | |
535 | ||
536 | |Apache log4j | |
537 | |nloption:--domain=++log4j++ | |
538 | |nloption:--log4j | |
539 | ||
540 | |Python | |
541 | |nloption:--domain=++python++ | |
542 | |nloption:--python | |
543 | |=== | |
544 | ||
545 | You must specify a tracing domain to target a type of LTTng tracer when | |
546 | using some man:lttng(1) to avoid ambiguity. For example, because the | |
547 | Linux kernel and user space tracing domains support named tracepoints as | |
548 | instrumentation points (see the <<"event-rule","{sect-event-rule}">> section | |
549 | above), you need to specify a tracing domain when you create an event | |
550 | rule because both tracing domains could have tracepoints sharing the | |
551 | same name. | |
552 | ||
553 | You can create channels (see the <<channel,{sect-channel}>> section | |
554 | below) in the Linux kernel and user space tracing domains. The other | |
555 | tracing domains have a single, default channel. | |
556 | ||
557 | ||
558 | [[channel]] | |
559 | {sect-channel} | |
560 | -------------- | |
561 | A _channel_ is an object which is responsible for a set of ring buffers. | |
562 | ||
563 | Each ring buffer is divided into multiple _sub-buffers_. When a | |
564 | recording event rule (see the | |
565 | <<recording-event-rule,{sect-recording-event-rule} section below) | |
566 | matches an event, LTTng can record it to one or more sub-buffers of one | |
567 | or more channels. | |
568 | ||
569 | When you create a channel with the man:lttng-enable-channel(1) command, | |
570 | you set its final attributes, that is: | |
571 | ||
572 | * Its buffering scheme. | |
573 | + | |
574 | See the <<channel-buf-scheme,Buffering scheme>> section below. | |
575 | ||
576 | * What to do when there's no | |
577 | space left for a new event record because all sub-buffers are full. | |
578 | + | |
579 | See the <<channel-er-loss-mode,Event record loss mode>> section below. | |
580 | ||
581 | * The size of each ring buffer and how many sub-buffers a ring buffer | |
582 | has. | |
583 | + | |
584 | See the <<channel-sub-buf-size-count,Sub-buffer size and count>> section | |
585 | below. | |
586 | ||
587 | * The size of each trace file LTTng writes for this channel and the | |
588 | maximum count of trace files. | |
589 | + | |
590 | See the <<channel-max-trace-file-size-count,Maximum trace file size and | |
591 | count>> section below. | |
592 | ||
593 | * The periods of its read, switch, and monitor timers. | |
594 | + | |
595 | See the <<channel-timers,Timers>> section below. | |
596 | ||
597 | * For a Linux kernel channel: its output type (man:mmap(2) or | |
598 | man:splice(2)). | |
599 | + | |
600 | See the nloption:--output option of the man:lttng-enable-channel(1) | |
601 | command. | |
602 | ||
603 | * For a user space channel: the value of its blocking timeout. | |
604 | + | |
605 | See the nloption:--blocking-timeout option of the | |
606 | man:lttng-enable-channel(1) command. | |
607 | ||
608 | Note that the man:lttng-enable-event(1) command can automatically create | |
609 | a default channel with sane defaults when no channel exists for the | |
610 | provided tracing domain. | |
611 | ||
612 | A channel is always associated to a tracing domain (see the | |
613 | <<domain,{sect-domain}>> section below). The `java.util.logging` (JUL), | |
614 | log4j, and Python tracing domains each have a default channel which you | |
615 | can't configure. | |
616 | ||
617 | A channel owns recording event rules. | |
618 | ||
619 | List the channels of a given tracing session with the | |
620 | man:lttng-list(1) and man:lttng-status(1) commands. | |
621 | ||
622 | Disable an enabled channel with the man:lttng-disable-channel(1) | |
623 | command. | |
624 | ||
625 | ||
626 | [[channel-buf-scheme]] | |
627 | Buffering scheme | |
628 | ~~~~~~~~~~~~~~~~ | |
629 | A channel has at least one ring buffer per CPU. LTTng always records an | |
630 | event to the ring buffer dedicated to the CPU which emits it. | |
631 | ||
632 | The buffering scheme of a user space channel determines what has its own | |
633 | set of per-CPU ring buffers: | |
634 | ||
635 | Per-user buffering (nloption:--buffers-uid option of the man:lttng-enable-channel(1) command):: | |
636 | Allocate one set of ring buffers (one per CPU) shared by all the | |
637 | instrumented processes of: | |
638 | If your Unix user is `root`::: | |
639 | Each Unix user. | |
640 | Otherwise::: | |
641 | Your Unix user. | |
642 | ||
643 | Per-process buffering (nloption:--buffers-pid option of the man:lttng-enable-channel(1) command):: | |
644 | Allocate one set of ring buffers (one per CPU) for each instrumented | |
645 | process of: | |
646 | If your Unix user is `root`::: | |
647 | All Unix users. | |
648 | Otherwise::: | |
649 | Your Unix user. | |
650 | ||
651 | The per-process buffering scheme tends to consume more memory than the | |
652 | per-user option because systems generally have more instrumented | |
653 | processes than Unix users running instrumented processes. However, the | |
654 | per-process buffering scheme ensures that one process having a high | |
655 | event throughput won't fill all the shared sub-buffers of the same Unix | |
656 | user, only its own. | |
657 | ||
658 | The buffering scheme of a Linux kernel channel is always to allocate a | |
659 | single set of ring buffers for the whole system. This scheme is similar | |
660 | to the per-user option, but with a single, global user ``running'' the | |
661 | kernel. | |
662 | ||
663 | ||
664 | [[channel-er-loss-mode]] | |
665 | Event record loss mode | |
666 | ~~~~~~~~~~~~~~~~~~~~~~ | |
667 | When LTTng emits an event, LTTng can record it to a specific, available | |
668 | sub-buffer within the ring buffers of specific channels. When there's no | |
669 | space left in a sub-buffer, the tracer marks it as consumable and | |
670 | another, available sub-buffer starts receiving the following event | |
671 | records. An LTTng consumer daemon eventually consumes the marked | |
672 | sub-buffer, which returns to the available state. | |
673 | ||
674 | In an ideal world, sub-buffers are consumed faster than they are filled. | |
675 | In the real world, however, all sub-buffers can be full at some point, | |
676 | leaving no space to record the following events. | |
677 | ||
678 | By default, LTTng-modules and LTTng-UST are _non-blocking_ tracers: when | |
679 | there's no available sub-buffer to record an event, it's acceptable to | |
680 | lose event records when the alternative would be to cause substantial | |
681 | delays in the execution of the instrumented application. LTTng | |
682 | privileges performance over integrity; it aims at perturbing the | |
683 | instrumented application as little as possible in order to make the | |
684 | detection of subtle race conditions and rare interrupt cascades | |
685 | possible. | |
686 | ||
687 | Since LTTng{nbsp}2.10, the LTTng user space tracer, LTTng-UST, supports | |
688 | a _blocking mode_. See the nloption:--blocking-timeout of the | |
689 | man:lttng-enable-channel(1) command to learn how to use the blocking | |
690 | mode. | |
691 | ||
692 | When it comes to losing event records because there's no available | |
693 | sub-buffer, or because the blocking timeout of the channel is | |
694 | reached, the _event record loss mode_ of the channel determines what to | |
695 | do. The available event record loss modes are: | |
696 | ||
697 | Discard mode:: | |
698 | Drop the newest event records until a sub-buffer becomes available. | |
699 | + | |
700 | This is the only available mode when you specify a blocking timeout. | |
701 | + | |
702 | With this mode, LTTng increments a count of lost event records when an | |
703 | event record is lost and saves this count to the trace. A trace reader | |
704 | can use the saved discarded event record count of the trace to decide | |
705 | whether or not to perform some analysis even if trace data is known to | |
706 | be missing. | |
707 | ||
708 | Overwrite mode:: | |
709 | Clear the sub-buffer containing the oldest event records and start | |
710 | writing the newest event records there. | |
711 | + | |
712 | This mode is sometimes called _flight recorder mode_ because it's | |
713 | similar to a https://en.wikipedia.org/wiki/Flight_recorder[flight | |
714 | recorder]: always keep a fixed amount of the latest data. It's also | |
715 | similar to the roll mode of an oscilloscope. | |
716 | + | |
717 | Since LTTng{nbsp}2.8, with this mode, LTTng writes to a given sub-buffer | |
718 | its sequence number within its data stream. With a local, network | |
719 | streaming, or live tracing session (see the <<session-modes,Tracing | |
720 | session modes>> section above), a trace reader can use such sequence | |
721 | numbers to report lost packets. A trace reader can use the saved | |
722 | discarded sub-buffer (packet) count of the trace to decide whether or | |
723 | not to perform some analysis even if trace data is known to be missing. | |
724 | + | |
725 | With this mode, LTTng doesn't write to the trace the exact number of | |
726 | lost event records in the lost sub-buffers. | |
727 | ||
728 | Which mechanism you should choose depends on your context: prioritize | |
729 | the newest or the oldest event records in the ring buffer? | |
730 | ||
731 | Beware that, in overwrite mode, the tracer abandons a _whole sub-buffer_ | |
732 | as soon as a there's no space left for a new event record, whereas in | |
733 | discard mode, the tracer only discards the event record that doesn't | |
734 | fit. | |
735 | ||
736 | Set the event record loss mode of a channel with the nloption:--discard | |
737 | and nloption:--overwrite options of the man:lttng-enable-channel(1) | |
738 | command. | |
739 | ||
740 | There are a few ways to decrease your probability of losing event | |
741 | records. The <<channel-sub-buf-size-count,Sub-buffer size and count>> | |
742 | section below shows how to fine-tune the sub-buffer size and count of a | |
743 | channel to virtually stop losing event records, though at the cost of | |
744 | greater memory usage. | |
745 | ||
746 | ||
747 | [[channel-sub-buf-size-count]] | |
748 | Sub-buffer size and count | |
749 | ~~~~~~~~~~~~~~~~~~~~~~~~~ | |
750 | A channel has one or more ring buffer for each CPU of the target system. | |
751 | ||
752 | See the <<channel-buf-scheme,Buffering scheme>> section above to learn | |
753 | how many ring buffers of a given channel are dedicated to each CPU | |
754 | depending on its buffering scheme. | |
755 | ||
756 | Set the size of each sub-buffer the ring buffers of a channel contain | |
757 | with the nloption:--subbuf-size option of the | |
758 | man:lttng-enable-channel(1) command. | |
759 | ||
760 | Set the number of sub-buffers each ring buffer of a channel contains | |
761 | with the nloption:--num-subbuf option of the man:lttng-enable-channel(1) | |
762 | command. | |
763 | ||
764 | Note that LTTng switching the current sub-buffer of a ring buffer | |
765 | (marking a full one as consumable and switching to an available one for | |
766 | LTTng to record the next events) introduces noticeable CPU overhead. | |
767 | Knowing this, the following list presents a few practical situations | |
768 | along with how to configure the sub-buffer size and count for them: | |
769 | ||
770 | High event throughput:: | |
771 | In general, prefer large sub-buffers to lower the risk of losing | |
772 | event records. | |
773 | + | |
774 | Having larger sub-buffers also ensures a lower sub-buffer switching | |
775 | frequency (see the <<channel-timers,Timers>> section below). | |
776 | + | |
777 | The sub-buffer count is only meaningful if you create the channel in | |
778 | overwrite mode (see the <<channel-er-loss-mode,Event record loss mode>> | |
779 | section above): in this case, if LTTng overwrites a sub-buffer, then the | |
780 | other sub-buffers are left unaltered. | |
781 | ||
782 | Low event throughput:: | |
783 | In general, prefer smaller sub-buffers since the risk of losing | |
784 | event records is low. | |
785 | + | |
786 | Because LTTng emits events less frequently, the sub-buffer switching | |
787 | frequency should remain low and therefore the overhead of the tracer | |
788 | shouldn't be a problem. | |
789 | ||
790 | Low memory system:: | |
791 | If your target system has a low memory limit, prefer fewer first, | |
792 | then smaller sub-buffers. | |
793 | + | |
794 | Even if the system is limited in memory, you want to keep the | |
795 | sub-buffers as large as possible to avoid a high sub-buffer switching | |
796 | frequency. | |
797 | ||
798 | Note that LTTng uses https://diamon.org/ctf/[CTF] as its trace format, | |
799 | which means event record data is very compact. For example, the average | |
800 | LTTng kernel event record weights about 32{nbsp}bytes. Therefore, a | |
801 | sub-buffer size of 1{nbsp}MiB is considered large. | |
802 | ||
803 | The previous scenarios highlight the major trade-off between a few large | |
804 | sub-buffers and more, smaller sub-buffers: sub-buffer switching | |
805 | frequency vs. how many event records are lost in overwrite mode. | |
806 | Assuming a constant event throughput and using the overwrite mode, the | |
807 | two following configurations have the same ring buffer total size: | |
808 | ||
809 | Two sub-buffers of 4{nbsp}MiB each:: | |
810 | Expect a very low sub-buffer switching frequency, but if LTTng | |
811 | ever needs to overwrite a sub-buffer, half of the event records so | |
812 | far (4{nbsp}MiB) are definitely lost. | |
813 | ||
814 | Eight sub-buffers of 1{nbsp}MiB each:: | |
815 | Expect four times the tracer overhead of the configuration above, | |
816 | but if LTTng needs to overwrite a sub-buffer, only the eighth of | |
817 | event records so far (1{nbsp}MiB) are definitely lost. | |
818 | ||
819 | In discard mode, the sub-buffer count parameter is pointless: use two | |
820 | sub-buffers and set their size according to your requirements. | |
821 | ||
822 | ||
823 | [[channel-max-trace-file-size-count]] | |
824 | Maximum trace file size and count | |
825 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
826 | By default, trace files can grow as large as needed. | |
827 | ||
828 | Set the maximum size of each trace file that LTTng writes of a given | |
829 | channel with the nloption:--tracefile-size option of the man:lttng-enable-channel(1) | |
830 | command. | |
831 | ||
832 | When the size of a trace file reaches the fixed maximum size of the | |
833 | channel, LTTng creates another file to contain the next event records. | |
834 | LTTng appends a file count to each trace file name in this case. | |
835 | ||
836 | If you set the trace file size attribute when you create a channel, the | |
837 | maximum number of trace files that LTTng creates is _unlimited_ by | |
838 | default. To limit them, use the nloption:--tracefile-count option of | |
839 | man:lttng-enable-channel(1). When the number of trace files reaches the | |
840 | fixed maximum count of the channel, LTTng overwrites the oldest trace | |
841 | file. This mechanism is called _trace file rotation_. | |
842 | ||
843 | [IMPORTANT] | |
844 | ==== | |
845 | Even if you don't limit the trace file count, always assume that LTTng | |
846 | manages all the trace files of the tracing session. | |
847 | ||
848 | In other words, there's no safe way to know if LTTng still holds a given | |
849 | trace file open with the trace file rotation feature. | |
850 | ||
851 | The only way to obtain an unmanaged, self-contained LTTng trace before | |
852 | you destroy the tracing session is with the tracing session rotation | |
853 | feature (see the <<rotation,Tracing session rotation>> section above), | |
854 | which is available since LTTng{nbsp}2.11. | |
855 | ==== | |
856 | ||
857 | ||
858 | [[channel-timers]] | |
859 | Timers | |
860 | ~~~~~~ | |
861 | Each channel can have up to three optional timers: | |
862 | ||
863 | Switch timer:: | |
864 | When this timer expires, a sub-buffer switch happens: for each ring | |
865 | buffer of the channel, LTTng marks the current sub-buffer as | |
866 | consumable and switches to an available one to record the next | |
867 | events. | |
868 | + | |
869 | A switch timer is useful to ensure that LTTng consumes and commits trace | |
870 | data to trace files or to a distant relay daemon (man:lttng-relayd(8)) | |
871 | periodically in case of a low event throughput. | |
872 | + | |
873 | Such a timer is also convenient when you use large sub-buffers (see the | |
874 | <<channel-sub-buf-size-count,Sub-buffer size and count>> section above) | |
875 | to cope with a sporadic high event throughput, even if the throughput is | |
876 | otherwise low. | |
877 | + | |
878 | Set the period of the switch timer of a channel, or disable the timer | |
879 | altogether, with the nloption:--switch-timer option of the | |
880 | man:lttng-enable-channel(1) command. | |
881 | ||
882 | Read timer:: | |
883 | When this timer expires, LTTng checks for full, consumable | |
884 | sub-buffers. | |
885 | + | |
886 | By default, the LTTng tracers use an asynchronous message mechanism to | |
887 | signal a full sub-buffer so that a consumer daemon can consume it. | |
888 | + | |
889 | When such messages must be avoided, for example in real-time | |
890 | applications, use this timer instead. | |
891 | + | |
892 | Set the period of the read timer of a channel, or disable the timer | |
893 | altogether, with the nloption:--read-timer option of the | |
894 | man:lttng-enable-channel(1) command. | |
895 | ||
896 | Monitor timer:: | |
897 | When this timer expires, the consumer daemon samples some channel | |
898 | (see the <<channel,{sect-channel}>> section above) | |
899 | statistics to evaluate the following trigger conditions: | |
900 | + | |
901 | -- | |
902 | . The consumed buffer size of a given tracing session becomes greater | |
903 | than some value. | |
904 | . The buffer usage of a given channel becomes greater than some value. | |
905 | . The buffer usage of a given channel becomes less than some value. | |
906 | -- | |
907 | + | |
908 | If you disable the monitor timer of a channel{nbsp}__C__: | |
909 | + | |
910 | -- | |
911 | * The consumed buffer size value of the tracing session of{nbsp}__C__ | |
912 | could be wrong for trigger condition type{nbsp}1: the consumed buffer | |
913 | size of{nbsp}__C__ won't be part of the grand total. | |
914 | ||
915 | * The buffer usage trigger conditions (types{nbsp}2 and{nbsp}3) | |
916 | for{nbsp}__C__ will never be satisfied. | |
917 | -- | |
918 | + | |
919 | See the <<trigger,TRIGGER>> section above to learn more about triggers. | |
920 | + | |
921 | Set the period of the monitor timer of a channel, or disable the timer | |
922 | altogether, with the nloption:--monitor-timer option of the | |
923 | man:lttng-enable-channel(1) command. | |
924 | ||
925 | ||
926 | [[recording-event-rule]] | |
927 | {sect-recording-event-rule} | |
928 | --------------------------- | |
929 | A _recording event rule_ is a specific type of event rule (see the | |
930 | <<"event-rule","{sect-event-rule}">> section above) of which the action is | |
931 | to serialize and record the matched event as an _event record_. | |
932 | ||
933 | Set the explicit conditions of a recording event rule when you create it | |
934 | with the man:lttng-enable-event(1) command. A recording event rule also | |
935 | has the following implicit conditions: | |
936 | ||
937 | * The recording event rule itself is enabled. | |
938 | + | |
939 | A recording event rule is enabled on creation. | |
940 | ||
941 | * The channel to which the recording event rule is attached is enabled. | |
942 | + | |
943 | A channel is enabled on creation. | |
944 | + | |
945 | See the <<channel,{sect-channel}>> section above. | |
946 | ||
947 | * The tracing session of the recording event rule is active (started). | |
948 | + | |
949 | A tracing session is inactive (stopped) on creation. | |
950 | + | |
951 | See the <<session,{sect-session}>> section above. | |
952 | ||
953 | * The process for which LTTng creates an event to match is allowed to | |
954 | record events. | |
955 | + | |
956 | All processes are allowed to record events on tracing session | |
957 | creation. | |
958 | + | |
959 | Use the man:lttng-track(1) and man:lttng-untrack(1) commands to select | |
960 | which processes are allowed to record events based on specific process | |
961 | attributes. | |
962 | ||
963 | You always attach a recording event rule to a channel, which belongs to | |
964 | a tracing session, when you create it. | |
965 | ||
966 | When a recording event rule{nbsp}__ER__ matches an event{nbsp}__E__, | |
967 | LTTng attempts to serialize and record{nbsp}__E__ to one of the | |
968 | available sub-buffers of the channel to which{nbsp}__E__ is attached. | |
969 | ||
970 | When multiple matching recording event rules are attached to the same | |
971 | channel, LTTng attempts to serialize and record the matched event | |
972 | _once_. In the following example, the second recording event rule is | |
973 | redundant when both are enabled: | |
974 | ||
975 | [role="term"] | |
976 | ---- | |
977 | $ lttng enable-event --userspace hello:world | |
978 | $ lttng enable-event --userspace hello:world --loglevel=TRACE_INFO | |
979 | ---- | |
980 | ||
981 | List the recording event rules of a specific tracing session | |
982 | and/or channel with the man:lttng-list(1) and man:lttng-status(1) | |
983 | commands. | |
984 | ||
985 | Disable a recording event rule with the man:lttng-disable-event(1) | |
986 | command. | |
987 | ||
988 | As of LTTng{nbsp}{lttng_version}, you cannot remove a recording event | |
989 | rule: it exists as long as its tracing session exists. | |
990 | ||
991 | ||
992 | include::common-footer.txt[] | |
993 | ||
994 | ||
995 | SEE ALSO | |
996 | -------- | |
997 | man:lttng(1), | |
998 | man:lttng-relayd(8), | |
999 | man:lttng-sessiond(8) |