| 1 | RFC - LTTng session daemon architecture |
| 2 | |
| 3 | WARNING: |
| 4 | Parts of the proposal are obselete but we keep this version for historical |
| 5 | purposes. |
| 6 | |
| 7 | Author: David Goulet <david.goulet@polymtl.ca> |
| 8 | |
| 9 | Contributors: |
| 10 | * Mathieu Desnoyers <mathieu.desnoyers@efficios.com> |
| 11 | * Yannick Brosseau <yannick.brosseau@polymtl.ca> |
| 12 | * Nils Carlson <nils.carlson@ericsson.com> |
| 13 | * Michel Dagenais <michel.dagenais@polymtl.ca> |
| 14 | * Stefan Hajnoczi <stefanha@gmail.com> |
| 15 | |
| 16 | Version: |
| 17 | - v0.1: 17/01/2011 |
| 18 | * Initial proposal |
| 19 | |
| 20 | - v0.2: 19/01/2011 |
| 21 | After multiple reply from all the contributors above, here is the list |
| 22 | of what has changed: |
| 23 | * Change/Add Terminology elements from the initial model |
| 24 | * New figures for four new scenarios |
| 25 | * Add inprocess library section |
| 26 | * LTTng kernel tracer support proposition |
| 27 | * More details for the Model and Components |
| 28 | * Improve the basic model. Quite different from the last one |
| 29 | |
| 30 | - v0.3: 28/01/2011 |
| 31 | In response from Michel Dagenais and Nils Carlson comments: |
| 32 | * Add scaling to reasons of this re-engineering |
| 33 | * Purpose of the session ID |
| 34 | * Explain why ltt-sessiond creates the tracing buffers |
| 35 | * ust-consumerd interaction schema |
| 36 | * Clarify inprocess library behavior |
| 37 | |
| 38 | - v0.4: 01/02/2011 |
| 39 | After Mathieu Desnoyers and Michel Dagenais comments: |
| 40 | * Add section Introduction |
| 41 | * Define the global and per-user ltt-sessiond |
| 42 | * Add details for ltt-sessiond in the inprocess lib section |
| 43 | * Session ID are now UUID |
| 44 | * Add buffer snapshot schema for ust-consumerd |
| 45 | * ltt-sessiond validate inprocess lib version |
| 46 | * ltt-sessiond socket validation by the inprocess lib. |
| 47 | * Add lttng definition |
| 48 | * Add consumer to the Model section |
| 49 | - v0.5: 23/07/2012 |
| 50 | * Please refer to the thesis of David Goulet for the complete |
| 51 | and up to date specifications of the architecture and algorithms used. |
| 52 | (http://publications.polymtl.ca/842/1/2012_DavidGoulet.pdf) |
| 53 | |
| 54 | Terminology |
| 55 | ----------------- |
| 56 | |
| 57 | ltt-sessiond - Main daemon for trace session registry for UST and LTTng |
| 58 | NOTE: Changed to lttng-sessiond in the git tree |
| 59 | |
| 60 | ust-consumerd - Daemon that consume UST buffers for a specific application |
| 61 | |
| 62 | ltt-consumerd - Daemon that consume LTTng buffers |
| 63 | |
| 64 | tracing session - A trace linked to a set of specific tracepoints and to a set |
| 65 | of tracing buffers |
| 66 | |
| 67 | tracing buffers - Buffers containing tracing data |
| 68 | |
| 69 | tracing data - Data created by tracing an application |
| 70 | |
| 71 | inprocess library - UST library linked with the application |
| 72 | |
| 73 | shared memory - system V shared memory |
| 74 | |
| 75 | application common named pipe - Global named pipe that triggers application |
| 76 | registration, on pipe event, to ltt-sessiond |
| 77 | |
| 78 | lttng - New command line tool for LTTng and UST tracing control |
| 79 | |
| 80 | Introduction |
| 81 | ----------------- |
| 82 | |
| 83 | This RFC propose a brand new UST and LTTng daemon model. This re-engineering |
| 84 | was mostly driven by the need of: |
| 85 | |
| 86 | * Better security in terms of access rights on tracing data |
| 87 | * Manage tracing session |
| 88 | * Scaling in terms of thread/processes needed to perform tracing |
| 89 | * LTTng and UST integration in terms of merging traces and session control |
| 90 | * Networking such as streaming and remote control over different traces |
| 91 | |
| 92 | The new model follows the basic principles of having a session registry |
| 93 | (ltt-sessiond) and consumers for each tracing session (ust-consumerd and |
| 94 | ltt-consumerd). |
| 95 | |
| 96 | With this proposal, LTTng and UST will share the same tracing session, be |
| 97 | managed by the same tool and bring a complete integration between these two |
| 98 | powerful tools. |
| 99 | |
| 100 | NOTE: This proposal does NOT makes UST dependent on LTTng and vice versa. |
| 101 | |
| 102 | Model |
| 103 | ----------------- |
| 104 | |
| 105 | A global and/or per-user registry keeps track of all tracing sessions. Any user |
| 106 | that wants to manage either a kernel trace using LTTng or an application trace |
| 107 | with UST must interact with that registry for any possible actions. |
| 108 | |
| 109 | The model address multiple tracing use cases based on the fact that we |
| 110 | introduce a tracing Unix group (tracing group). Only users in that group or |
| 111 | root can use the global registry. Other users will create a local registry |
| 112 | (per-user registry) that will be completely independent from the global one. |
| 113 | |
| 114 | Two cases: |
| 115 | |
| 116 | 1) Users in the tracing group, it's tracing session can consume all tracing |
| 117 | buffers from all applications and the kernel. |
| 118 | |
| 119 | 2) Users NOT in the tracing group, it's tracing session can only consume |
| 120 | data from its own applications' buffers hence tracing his applications. |
| 121 | |
| 122 | A session stored by the registry consist of: |
| 123 | |
| 124 | * Session name (given by the user or automatically assigned) |
| 125 | * List of traces (LTTng or UST) |
| 126 | * Tracepoints/markers associated to a trace of that session |
| 127 | * UUID |
| 128 | * Associated user (UID) |
| 129 | |
| 130 | Then, consumers are used to extract data from tracing buffers. These consumers |
| 131 | are daemon consuming either UST or/and LTTng buffers. For a single session, |
| 132 | only one UST consumer and one LTTng consumer is necessary. The daemon CAN |
| 133 | handle multiple tracing buffers for network streaming by example or for quick |
| 134 | snapshot. These consumers are told by the inprocess library or the kernel to |
| 135 | start getting out data on disk or network. |
| 136 | |
| 137 | For the next subsections, every components of this new proposal is explained |
| 138 | from the global and per-user registry perspective. |
| 139 | |
| 140 | LTT-SESSIOND: |
| 141 | |
| 142 | The ltt-sessiond daemon acts as a session registry i.e. by keeping reference to |
| 143 | all active session and, by active, it means a session in any state other than |
| 144 | destroyed. Each entity we are keeping track of, here session, will have a |
| 145 | universal unique identifier (UUID) assigned to it. The purpose of this UUID is |
| 146 | to track a session in order to apply any kind of actions (Ex: Attach, Destroy). |
| 147 | A human readable version SHOULD be consider in order to facilitate the session |
| 148 | identification when listed by lttng. |
| 149 | |
| 150 | The daemon creates two local Unix sockets (AF_UNIX). The first one is for what |
| 151 | we call client communication i.e. interaction with lttng (or any other |
| 152 | compatible tools). That socket is set with the ltt-sessiond credentials with |
| 153 | read-write mode for both user and group. The second one is a global socket for |
| 154 | application registration for the UST case (see inprocess lib subsection below). |
| 155 | |
| 156 | This daemon is also responsible for tracing buffers creation. Two main reasons |
| 157 | motivate this design: |
| 158 | |
| 159 | * The ltt-sessiond needs to keep track of all the shared memory segments in |
| 160 | order to be able to give reference to any other possible consumer. |
| 161 | |
| 162 | * For the case of sharing tracing buffers between all userspace |
| 163 | applications, having the registry allocating them will allow that but, if |
| 164 | the inprocess library was allocating them, we will need to redesign the |
| 165 | whole model. |
| 166 | |
| 167 | For all tracing actions either to interact with a session or a specific trace, |
| 168 | the lttng client MUST go through ltt-sessiond. The daemon will take care of |
| 169 | routing the command to the write inprocess library or the kernel. |
| 170 | |
| 171 | Global registry: |
| 172 | |
| 173 | A global registry SHOULD be started, idealy at boot, with credentials UID root |
| 174 | and GID of the tracing group. Only user within the tracing group will be able |
| 175 | to interact with that registry. All applications will try to register to that |
| 176 | registry using the global socket (second one discuss above). |
| 177 | |
| 178 | Per-user registry: |
| 179 | |
| 180 | This type of registry address two use cases. The first one is when a session |
| 181 | creation is requested from lttng but no global ltt-sessiond exist. So, a |
| 182 | ltt-sessiond will be spawned in order to manage the tracing of that user. The |
| 183 | second use case is when a user is not in the tracing group thus he cannot |
| 184 | communication with the global registry. |
| 185 | |
| 186 | However, care MUST be put in order to manage the socket's daemon. They are not |
| 187 | global anymore so they should be created in the home directory of the user |
| 188 | requesting tracing. |
| 189 | |
| 190 | In both cases, for global and per-user registry, all applications MUST try to |
| 191 | register to both ltt-sessiond. (see inprocess library subsection for details) |
| 192 | |
| 193 | The trace roles of ltt-sessiond: |
| 194 | |
| 195 | Trace interaction - Create, Destroy, Pause, Stop, Start, Set options |
| 196 | |
| 197 | Registry - keep track of trace's information: |
| 198 | * shared memory location (only the keyid) |
| 199 | * application PID (UST) |
| 200 | * type (kernel or UST) |
| 201 | * session name |
| 202 | * UID |
| 203 | |
| 204 | Buffers creation - creates shared memory for the tracing buffers. |
| 205 | |
| 206 | UST-CONSUMERD: |
| 207 | |
| 208 | The purpose of this daemon is to consume the UST trace buffers for only a |
| 209 | specific session. The session MAY have several traces for example two different |
| 210 | applications. The client tool, lttng has to create the ust-consumerd if NONE |
| 211 | is available for that session. It is very important to understand that for a |
| 212 | tracing session, there is only one ust-consumerd for all the traced |
| 213 | applications. |
| 214 | |
| 215 | This daemon basically empty the tracing buffers when asked for and writes that |
| 216 | data to disk for future analysis using LTTv or/and TMF (Tracing Monitoring |
| 217 | Frameworks). The inprocess library is the one that tells the ust-consumerd |
| 218 | daemon that the buffers are ready for consumption. |
| 219 | |
| 220 | Here is a flow of action to illustrate the ust-consumerd life span: |
| 221 | |
| 222 | 1) |
| 223 | +-----------+ ops +--------------+ |
| 224 | | lttng A |<---------->| ltt-sessiond | |
| 225 | +-----------+ +--------------+ |
| 226 | |
| 227 | lttng ask for tracing an application using the PID and the session UUID. The |
| 228 | shared memory reference is given to lttng and the ust-consumerd communication |
| 229 | socket if ust-consumerd already exist. |
| 230 | |
| 231 | 2a) If ust-consumerd EXIST |
| 232 | |
| 233 | +-----------+ |
| 234 | | lttng A | |
| 235 | +-----------+ |
| 236 | | mem ref. |
| 237 | | +---------------+ read +------------+ |
| 238 | +-->| ust-consumerd |--------->| shared mem | |
| 239 | +---------------+ +------------+ |
| 240 | |
| 241 | In that case, lttng only ask ust-consumerd to consume the buffers using |
| 242 | the reference it previously got from ltt-sessiond. |
| 243 | |
| 244 | 2b) If ust-consumerd DOES NOT EXIST |
| 245 | |
| 246 | +-----------+ +--------------+ |
| 247 | | lttng A | +---->| ltt-sessiond | |
| 248 | +-----------+ | +--------------+ |
| 249 | | ID | |
| 250 | | mem ref. | register |
| 251 | | +---------------+ |
| 252 | +-->| ust-consumerd | |
| 253 | +---------------+ |
| 254 | |
| 255 | lttng spawns the ust-consumerd for the session using the session UUID in |
| 256 | order for the daemon to register as a consumer to ltt-sessiond for that |
| 257 | session. |
| 258 | |
| 259 | Quick buffer snapshot: |
| 260 | |
| 261 | 1) Here, lttng will request a buffer snapshot for an already running session. |
| 262 | |
| 263 | +-----------+ +--------------+ |
| 264 | | lttng A |-------- ops ------->| ltt-sessiond | |
| 265 | +-----------+ +--------------+ |
| 266 | | | command |
| 267 | | +-----------------+ +-------+<--+ |
| 268 | | | ust-consumerd 1 |<----| app_1 |-+ |
| 269 | | +-----------------+ +-------+ | write |
| 270 | | 1 | v |
| 271 | | | +-------------+ |
| 272 | | +--- read ----->| shared mem. | |
| 273 | | +-------------+ |
| 274 | | ^ |
| 275 | | +-----------------+ | |
| 276 | +->| ust-consumerd 2 |----------+ |
| 277 | +-----------------+ snapshot |
| 278 | | write |
| 279 | | |
| 280 | +---> disk/network |
| 281 | |
| 282 | The first ust-consumerd (1) was already consuming buffers for the current |
| 283 | session. So, lttng ask for a live snapshot. A new ust-consumerd (2) is |
| 284 | spawned, snapshot the buffers using the shared memory reference from |
| 285 | ltt-sessiond, writes date to disk and die after all. |
| 286 | |
| 287 | On the security side, the ust-consumerd gets UID/GID from the lttng |
| 288 | credentials since it was spawned by lttng and so the files containing the |
| 289 | tracing data will also be set to UID/GID of the lttng client. No setuid or |
| 290 | setgid is used, we only use the credentials of the user. |
| 291 | |
| 292 | The roles of ust-consumerd: |
| 293 | |
| 294 | Register to ltt-sessiond - Using a session UUID and credentials (UID/GID) |
| 295 | |
| 296 | Consume buffers - Write data to a file descriptor (on disk, network, ...) |
| 297 | |
| 298 | Buffer consumption is triggered by the inprocess library which tells |
| 299 | ust-consumerd when to consume. |
| 300 | |
| 301 | LTT-CONSUMERD: |
| 302 | |
| 303 | The purpose of this daemon is to consume the LTTng trace buffers for only a |
| 304 | specific session. |
| 305 | |
| 306 | For that kernel consumer, ltt-sessiond will pass different anonymous file |
| 307 | descriptors to the ltt-consumerd using a Unix socket. From these file |
| 308 | desriptors, it will be able to get the data from a special function export by |
| 309 | the LTTng kernel. |
| 310 | |
| 311 | ltt-consumerd will be managed by the exact same way as ust-consumerd. However, |
| 312 | in order to trace the kernel, you are either root (UID=0) or in the tracing |
| 313 | group. |
| 314 | |
| 315 | The roles of ltt-consumerd: |
| 316 | |
| 317 | Register to ltt-sessiond - Using a session UUID and credentials (UID/GID) |
| 318 | |
| 319 | Consume buffers - Write data to a file descriptor (on disk, network, ...) |
| 320 | |
| 321 | Kernel triggers ltt-consumerd for buffer consumption. |
| 322 | |
| 323 | UST INPROCESS LIBRARY: |
| 324 | |
| 325 | When the application starts, this library will check for the global named pipe |
| 326 | of ltt-sessiond. If present, it MUST validate that root is the owner. This |
| 327 | check is very important to prevent ltt-sessiond spoofing. If the pipe is root, |
| 328 | we are certain that it's the privileged user that operates tracing. Then, using |
| 329 | it's UID, the application will try to register to the per-user ltt-sessiond |
| 330 | again verifying before the owner ship of the named pipe that should match the |
| 331 | UID. |
| 332 | |
| 333 | Before registration, the inprocess library MUST validate with the ltt-sessiond |
| 334 | the library version for compatibility reason. This is mechanism is useful for |
| 335 | library compatibility but also to see if ltt-sessiond socket is valid (means |
| 336 | that an actual ltt-sessiond is listening on the other side). Having no response |
| 337 | for over 10 seconds, the application will cut communication on that socket and |
| 338 | fallback to the application common named pipe (explain below). |
| 339 | |
| 340 | If the socket is valid, it will register as a traceable application using the |
| 341 | apps credentials and will open a local Unix socket, passed to ltt-sessiond, in |
| 342 | order to receive an eventual shared memory reference. It will then wait on it |
| 343 | if any other command are given by the lttng client. This socket becomes the |
| 344 | only channel of communication between the registry and the application. |
| 345 | |
| 346 | If no ltt-sessiond is present at registration, the application tries to open |
| 347 | the application common named pipe or create it if it does not exist and wait on |
| 348 | it (using poll or epoll Linux API). Having any type of event on that pipe, the |
| 349 | inprocess library will then try to register to the global and per-user |
| 350 | ltt-sessiond. If it fails again, it goes back again to wait on that pipe. |
| 351 | |
| 352 | SHARED MEMORY |
| 353 | |
| 354 | For UST, this is the memory area where the tracing buffers will be held and |
| 355 | given access in read-write mode for the inprocess library of the application. |
| 356 | |
| 357 | On the LTTng side (for ltt-consumerd), these buffers are in the kernel space |
| 358 | and given access by opening a file in the debugfs file system. With an |
| 359 | anonymous file desriptor, this consumer will be able to extract the data. |
| 360 | |
| 361 | This memory is ONLY used for the tracing data. No communication between |
| 362 | components is done using that memory. |
| 363 | |
| 364 | A shared memory segment for tracing MUST be set with the tracing group GID for |
| 365 | the UST buffers. This is the job of ltt-sessiond. |
| 366 | |
| 367 | PREREQUISITES: |
| 368 | |
| 369 | The global ltt-sessiond daemon MUST always be running as "root" or an |
| 370 | equivalent user having the same privilege as root (UID = 0). |
| 371 | |
| 372 | The ltt-sessiond daemon SHOULD be up and running at all time in order to trace |
| 373 | a tracable application. |
| 374 | |
| 375 | The new lttng library API MUST be used to interact with the |
| 376 | ltt-sessiond registry daemon for every trace action needed by the user. |
| 377 | |
| 378 | A tracing group MUST be created. Whoever is in that group is able to access the |
| 379 | tracing data of any buffers and is able to trace any application or the kernel. |
| 380 | |
| 381 | WARNING: The tracing group name COULD interfere with other already existing |
| 382 | groups. Care should be put at install time for that (from source and packages) |
| 383 | |
| 384 | The next section illustrates different use cases using that new model. |
| 385 | |
| 386 | Use Cases |
| 387 | ----------------- |
| 388 | |
| 389 | Each case considers these : |
| 390 | |
| 391 | * user A - UID: A; GID: A, tracing |
| 392 | * user B - UID: B; GID: B, tracing |
| 393 | |
| 394 | Scenario 1 - Single user tracing app_1 |
| 395 | ------ |
| 396 | |
| 397 | This first scenario shows how user A will start a trace for application app_1 |
| 398 | that is not running. |
| 399 | |
| 400 | 1) lttng ask ltt-sessiond for a new session through a Unix socket. If |
| 401 | allowed, ltt-sessiond returns a session UUID to the client. |
| 402 | (Ex: ops --> new session) |
| 403 | |
| 404 | +-----------+ ops +--------------+ |
| 405 | | lttng A |<---------->| ltt-sessiond | |
| 406 | +-----------+ +--------------+ |
| 407 | |
| 408 | 2) The app_1 is spawned by lttng having the user A credentials. Then, app_1 |
| 409 | automatically register to ltt-sessiond has a "tracable apps" through the global |
| 410 | named pipe of ltt-sessiond using the UID/GID and session UUID. |
| 411 | |
| 412 | The shared memory is created with the app_1 UID (rw-) and tracing group GID |
| 413 | (r--) and a reference is given back to app_1 |
| 414 | |
| 415 | +-----------+ +--------------+ |
| 416 | | lttng A | | ltt-sessiond | |
| 417 | +-----------+ +--------------+ |
| 418 | | ^ | |
| 419 | | +-------+ | | +-------------+ |
| 420 | +-->| app_1 |<--------+ +-->| shared mem. | |
| 421 | +-------+ +-------------+ |
| 422 | |
| 423 | 3) app_1 connect to the shared memory and ust-consumerd is spawned with the |
| 424 | session UUID and lttng credentials (user A). It then register to ltt-sessiond |
| 425 | for a valid session to consume using the previous session UUID and credentials. |
| 426 | |
| 427 | +-----------+ +--------------+ |
| 428 | | lttng A | +-->| ltt-sessiond |----------+ |
| 429 | +-----------+ | +--------------+ | |
| 430 | | | | |
| 431 | | +---------------+ read | commands |
| 432 | +-->| ust-consumerd |---------+ | and |
| 433 | +---------------+ v | options |
| 434 | ^ | +-------------+ | |
| 435 | | v +------>| shared mem. | | |
| 436 | +-------+ | +-------------+ | |
| 437 | | app_1 |-------- | |
| 438 | +-------+ write | |
| 439 | ^ | |
| 440 | +--------------------------------------- |
| 441 | |
| 442 | Scenario 2 - Single user tracing already running app_1 |
| 443 | ------ |
| 444 | |
| 445 | 1) lttng ask ltt-sessiond for a new session through a Unix socket. If allowed |
| 446 | (able to write on socket), ltt-sessiond returns a session UUID to the client. |
| 447 | |
| 448 | +-----------+ ops +--------------+ |
| 449 | | lttng A |<---------->| ltt-sessiond | |
| 450 | +-----------+ +--------------+ |
| 451 | ^ |
| 452 | +-------+ read | |
| 453 | | app_1 |----------+ |
| 454 | +-------+ |
| 455 | |
| 456 | NOTE: At this stage, since app_1 is already running, the registration of app_1 |
| 457 | to ltt-sessiond has already been done. However, the shared memory segment is |
| 458 | not allocated yet until a trace session is initiated. Having no shared memory, |
| 459 | the inprocess library of app_1 will wait on the local Unix socket connected to |
| 460 | ltt-sessiond for the reference. |
| 461 | |
| 462 | +-----------+ +--------------+ |
| 463 | | lttng A | | ltt-sessiond | |
| 464 | +-----------+ +--------------+ |
| 465 | ^ | |
| 466 | +-------+ | | +-------------+ |
| 467 | | app_1 |<--------+ +-->| shared mem. | |
| 468 | +-------+ commands +-------------+ |
| 469 | | ^ |
| 470 | +---------- write ----------+ |
| 471 | |
| 472 | 2) lttng spawns a ust-consumerd for the session. We get the same figure as |
| 473 | step 3 in the first scenario. |
| 474 | |
| 475 | There is a small difference though. The application MAY NOT be using the same |
| 476 | credentials as user A (lttng). However, the shared memory is always GID of |
| 477 | the tracing group. So, in order for user A to trace app_1, is MUST be in the |
| 478 | tracing group otherwise, if the application is not set with the user |
| 479 | credentials, user A will not be able to trace app_1 |
| 480 | |
| 481 | Scenario 3 - Multiple users tracing the same running application |
| 482 | ------ |
| 483 | |
| 484 | 1) Session are created for the two users. Using the same exact mechanism as |
| 485 | before, the shared memory and consumers are created. Two users, two sessions, |
| 486 | two consumers and two shared memories for the same application. |
| 487 | |
| 488 | +-----------+ +--------------+ |
| 489 | | lttng A |-------- ops ------->| ltt-sessiond | |
| 490 | +-----------+ ^ +--------------+ |
| 491 | | ^ commands |
| 492 | +-----------+ | +-------+<--+ |
| 493 | | lttng B |------+ +--->| app_1 |------- write -----+ |
| 494 | +-----------+ | +-------+ | |
| 495 | | | |
| 496 | +-----------------+ | +-------------+ | |
| 497 | | ust-consumerd A |--O--- read ----->| shared mem. |<-+ |
| 498 | +-----------------+ | +-------------+ | |
| 499 | | | |
| 500 | +-----------------+ v +-------------+ | |
| 501 | | ust-consumerd B |--+--- read ----->| shared mem. |<-+ |
| 502 | +-----------------+ +-------------+ |
| 503 | |
| 504 | ust-consumerd A - UID: user A (rw-), GID: tracing (r--) |
| 505 | ust-consumerd B - UID: user B (rw-), GID: tracing (r--) |
| 506 | |
| 507 | Scenario 4 - User not in the tracing group |
| 508 | ------ |
| 509 | |
| 510 | For this particular case, it's all goes back to the first scenario. The user |
| 511 | MUST start the application using his credentials. The session will be created |
| 512 | by the per-user ltt-sessiond but he will not be able to trace anything that the |
| 513 | user does not owned. |