| 1 | RFC - New processes model for UST and LTTng |
| 2 | |
| 3 | Author: David Goulet <david.goulet@polymtl.ca> |
| 4 | |
| 5 | Contributors: |
| 6 | * Mathieu Desnoyers <mathieu.desnoyers@efficios.com> |
| 7 | * Yannick Brosseau <yannick.brosseau@polymtl.ca> |
| 8 | * Nils Carlson <nils.carlson@ericsson.com> |
| 9 | * Michel Dagenais <michel.dagenais@polymtl.ca> |
| 10 | * Stefan Hajnoczi <stefanha@gmail.com> |
| 11 | |
| 12 | Version: |
| 13 | - v0.1: 17/01/2011 |
| 14 | * Initial proposal |
| 15 | |
| 16 | - v0.2: 19/01/2011 |
| 17 | After multiple reply from all the contributors above, here is the list |
| 18 | of what has changed: |
| 19 | * Change/Add Terminology elements from the initial model |
| 20 | * New figures for four new scenarios |
| 21 | * Add inprocess library section |
| 22 | * LTTng kernel tracer support proposition |
| 23 | * More details for the Model and Components |
| 24 | * Improve the basic model. Quite different from the last one |
| 25 | |
| 26 | - v0.3: 28/01/2011 |
| 27 | In response from Michel Dagenais and Nils Carlson comments: |
| 28 | * Add scaling to reasons of this re-engineering |
| 29 | * Purpose of the session ID |
| 30 | * Explain why ltt-sessiond creates the tracing buffers |
| 31 | * ust-consumerd interaction schema |
| 32 | * Clarify inprocess library behavior |
| 33 | |
| 34 | - v0.4: 01/02/2011 |
| 35 | After Mathieu Desnoyers and Michel Dagenais comments: |
| 36 | * Add section Introduction |
| 37 | * Define the global and per-user ltt-sessiond |
| 38 | * Add details for ltt-sessiond in the inprocess lib section |
| 39 | * Session ID are now UUID |
| 40 | * Add buffer snapshot schema for ust-consumerd |
| 41 | * ltt-sessiond validate inprocess lib version |
| 42 | * ltt-sessiond socket validation by the inprocess lib. |
| 43 | * Add lttng definition |
| 44 | * Add consumer to the Model section |
| 45 | |
| 46 | Terminology |
| 47 | ----------------- |
| 48 | |
| 49 | ltt-sessiond - Main daemon for trace session registry for UST and LTTng |
| 50 | |
| 51 | ust-consumerd - Daemon that consume UST buffers for a specific application |
| 52 | |
| 53 | ltt-consumerd - Daemon that consume LTTng buffers |
| 54 | |
| 55 | tracing session - A trace linked to a set of specific tracepoints and to a set |
| 56 | of tracing buffers |
| 57 | |
| 58 | tracing buffers - Buffers containing tracing data |
| 59 | |
| 60 | tracing data - Data created by tracing an application |
| 61 | |
| 62 | inprocess library - UST library linked with the application |
| 63 | |
| 64 | shared memory - system V shared memory |
| 65 | |
| 66 | application common named pipe - Global named pipe that triggers application |
| 67 | registration, on pipe event, to ltt-sessiond |
| 68 | |
| 69 | lttng - New command line tool for LTTng and UST tracing control |
| 70 | |
| 71 | Introduction |
| 72 | ----------------- |
| 73 | |
| 74 | This RFC propose a brand new UST and LTTng daemon model. This re-engineering |
| 75 | was mostly driven by the need of: |
| 76 | |
| 77 | * Better security in terms of access rights on tracing data |
| 78 | * Manage tracing session |
| 79 | * Scaling in terms of thread/processes needed to perform tracing |
| 80 | * LTTng and UST integration in terms of merging traces and session control |
| 81 | * Networking such as streaming and remote control over different traces |
| 82 | |
| 83 | The new model follows the basic principles of having a session registry |
| 84 | (ltt-sessiond) and consumers for each tracing session (ust-consumerd and |
| 85 | ltt-consumerd). |
| 86 | |
| 87 | With this proposal, LTTng and UST will share the same tracing session, be |
| 88 | managed by the same tool and bring a complete integration between these two |
| 89 | powerful tools. |
| 90 | |
| 91 | NOTE: This proposal does NOT makes UST dependent on LTTng and vice versa. |
| 92 | |
| 93 | Model |
| 94 | ----------------- |
| 95 | |
| 96 | A global and/or per-user registry keeps track of all tracing sessions. Any user |
| 97 | that wants to manage either a kernel trace using LTTng or an application trace |
| 98 | with UST must interact with that registry for any possible actions. |
| 99 | |
| 100 | The model address multiple tracing use cases based on the fact that we |
| 101 | introduce a tracing Unix group (tracing group). Only users in that group or |
| 102 | root can use the global registry. Other users will create a local registry |
| 103 | (per-user registry) that will be completely independent from the global one. |
| 104 | |
| 105 | Two cases: |
| 106 | |
| 107 | 1) Users in the tracing group, it's tracing session can consume all tracing |
| 108 | buffers from all applications and the kernel. |
| 109 | |
| 110 | 2) Users NOT in the tracing group, it's tracing session can only consume |
| 111 | data from its own applications' buffers hence tracing his applications. |
| 112 | |
| 113 | A session stored by the registry consist of: |
| 114 | |
| 115 | * Session name (given by the user or automatically assigned) |
| 116 | * List of traces (LTTng or UST) |
| 117 | * Tracepoints/markers associated to a trace of that session |
| 118 | * UUID |
| 119 | * Associated user (UID) |
| 120 | |
| 121 | Then, consumers are used to extract data from tracing buffers. These consumers |
| 122 | are daemon consuming either UST or/and LTTng buffers. For a single session, |
| 123 | only one UST consumer and one LTTng consumer is necessary. The daemon CAN |
| 124 | handle multiple tracing buffers for network streaming by example or for quick |
| 125 | snapshot. These consumers are told by the inprocess library or the kernel to |
| 126 | start getting out data on disk or network. |
| 127 | |
| 128 | For the next subsections, every components of this new proposal is explained |
| 129 | from the global and per-user registry perspective. |
| 130 | |
| 131 | LTT-SESSIOND: |
| 132 | |
| 133 | The ltt-sessiond daemon acts as a session registry i.e. by keeping reference to |
| 134 | all active session and, by active, it means a session in any state other than |
| 135 | destroyed. Each entity we are keeping track of, here session, will have a |
| 136 | universal unique identifier (UUID) assigned to it. The purpose of this UUID is |
| 137 | to track a session in order to apply any kind of actions (Ex: Attach, Destroy). |
| 138 | A human readable version SHOULD be consider in order to facilitate the session |
| 139 | identification when listed by lttng. |
| 140 | |
| 141 | The daemon creates two local Unix sockets (AF_UNIX). The first one is for what |
| 142 | we call client communication i.e. interaction with lttng (or any other |
| 143 | compatible tools). That socket is set with the ltt-sessiond credentials with |
| 144 | read-write mode for both user and group. The second one is a global socket for |
| 145 | application registration for the UST case (see inprocess lib subsection below). |
| 146 | |
| 147 | This daemon is also responsible for tracing buffers creation. Two main reasons |
| 148 | motivate this design: |
| 149 | |
| 150 | * The ltt-sessiond needs to keep track of all the shared memory segments in |
| 151 | order to be able to give reference to any other possible consumer. |
| 152 | |
| 153 | * For the case of sharing tracing buffers between all userspace |
| 154 | applications, having the registry allocating them will allow that but, if |
| 155 | the inprocess library was allocating them, we will need to redesign the |
| 156 | whole model. |
| 157 | |
| 158 | For all tracing actions either to interact with a session or a specific trace, |
| 159 | the lttng client MUST go through ltt-sessiond. The daemon will take care of |
| 160 | routing the command to the write inprocess library or the kernel. |
| 161 | |
| 162 | Global registry: |
| 163 | |
| 164 | A global registry SHOULD be started, idealy at boot, with credentials UID root |
| 165 | and GID of the tracing group. Only user within the tracing group will be able |
| 166 | to interact with that registry. All applications will try to register to that |
| 167 | registry using the global socket (second one discuss above). |
| 168 | |
| 169 | Per-user registry: |
| 170 | |
| 171 | This type of registry address two use cases. The first one is when a session |
| 172 | creation is requested from lttng but no global ltt-sessiond exist. So, a |
| 173 | ltt-sessiond will be spawned in order to manage the tracing of that user. The |
| 174 | second use case is when a user is not in the tracing group thus he cannot |
| 175 | communication with the global registry. |
| 176 | |
| 177 | However, care MUST be put in order to manage the socket's daemon. They are not |
| 178 | global anymore so they should be created in the home directory of the user |
| 179 | requesting tracing. |
| 180 | |
| 181 | In both cases, for global and per-user registry, all applications MUST try to |
| 182 | register to both ltt-sessiond. (see inprocess library subsection for details) |
| 183 | |
| 184 | The trace roles of ltt-sessiond: |
| 185 | |
| 186 | Trace interaction - Create, Destroy, Pause, Stop, Start, Set options |
| 187 | |
| 188 | Registry - keep track of trace's information: |
| 189 | * shared memory location (only the keyid) |
| 190 | * application PID (UST) |
| 191 | * type (kernel or UST) |
| 192 | * session name |
| 193 | * UID |
| 194 | |
| 195 | Buffers creation - creates shared memory for the tracing buffers. |
| 196 | |
| 197 | UST-CONSUMERD: |
| 198 | |
| 199 | The purpose of this daemon is to consume the UST trace buffers for only a |
| 200 | specific session. The session MAY have several traces for example two different |
| 201 | applications. The client tool, lttng has to create the ust-consumerd if NONE |
| 202 | is available for that session. It is very important to understand that for a |
| 203 | tracing session, there is only one ust-consumerd for all the traced |
| 204 | applications. |
| 205 | |
| 206 | This daemon basically empty the tracing buffers when asked for and writes that |
| 207 | data to disk for future analysis using LTTv or/and TMF (Tracing Monitoring |
| 208 | Frameworks). The inprocess library is the one that tells the ust-consumerd |
| 209 | daemon that the buffers are ready for consumption. |
| 210 | |
| 211 | Here is a flow of action to illustrate the ust-consumerd life span: |
| 212 | |
| 213 | 1) |
| 214 | +-----------+ ops +--------------+ |
| 215 | | lttng A |<---------->| ltt-sessiond | |
| 216 | +-----------+ +--------------+ |
| 217 | |
| 218 | lttng ask for tracing an application using the PID and the session UUID. The |
| 219 | shared memory reference is given to lttng and the ust-consumerd communication |
| 220 | socket if ust-consumerd already exist. |
| 221 | |
| 222 | 2a) If ust-consumerd EXIST |
| 223 | |
| 224 | +-----------+ |
| 225 | | lttng A | |
| 226 | +-----------+ |
| 227 | | mem ref. |
| 228 | | +---------------+ read +------------+ |
| 229 | +-->| ust-consumerd |--------->| shared mem | |
| 230 | +---------------+ +------------+ |
| 231 | |
| 232 | In that case, lttng only ask ust-consumerd to consume the buffers using |
| 233 | the reference it previously got from ltt-sessiond. |
| 234 | |
| 235 | 2b) If ust-consumerd DOES NOT EXIST |
| 236 | |
| 237 | +-----------+ +--------------+ |
| 238 | | lttng A | +---->| ltt-sessiond | |
| 239 | +-----------+ | +--------------+ |
| 240 | | ID | |
| 241 | | mem ref. | register |
| 242 | | +---------------+ |
| 243 | +-->| ust-consumerd | |
| 244 | +---------------+ |
| 245 | |
| 246 | lttng spawns the ust-consumerd for the session using the session UUID in |
| 247 | order for the daemon to register as a consumer to ltt-sessiond for that |
| 248 | session. |
| 249 | |
| 250 | Quick buffer snapshot: |
| 251 | |
| 252 | 1) Here, lttng will request a buffer snapshot for an already running session. |
| 253 | |
| 254 | +-----------+ +--------------+ |
| 255 | | lttng A |-------- ops ------->| ltt-sessiond | |
| 256 | +-----------+ +--------------+ |
| 257 | | | command |
| 258 | | +-----------------+ +-------+<--+ |
| 259 | | | ust-consumerd 1 |<----| app_1 |-+ |
| 260 | | +-----------------+ +-------+ | write |
| 261 | | 1 | v |
| 262 | | | +-------------+ |
| 263 | | +--- read ----->| shared mem. | |
| 264 | | +-------------+ |
| 265 | | ^ |
| 266 | | +-----------------+ | |
| 267 | +->| ust-consumerd 2 |----------+ |
| 268 | +-----------------+ snapshot |
| 269 | | write |
| 270 | | |
| 271 | +---> disk/network |
| 272 | |
| 273 | The first ust-consumerd (1) was already consuming buffers for the current |
| 274 | session. So, lttng ask for a live snapshot. A new ust-consumerd (2) is |
| 275 | spawned, snapshot the buffers using the shared memory reference from |
| 276 | ltt-sessiond, writes date to disk and die after all. |
| 277 | |
| 278 | On the security side, the ust-consumerd gets UID/GID from the lttng |
| 279 | credentials since it was spawned by lttng and so the files containing the |
| 280 | tracing data will also be set to UID/GID of the lttng client. No setuid or |
| 281 | setgid is used, we only use the credentials of the user. |
| 282 | |
| 283 | The roles of ust-consumerd: |
| 284 | |
| 285 | Register to ltt-sessiond - Using a session UUID and credentials (UID/GID) |
| 286 | |
| 287 | Consume buffers - Write data to a file descriptor (on disk, network, ...) |
| 288 | |
| 289 | Buffer consumption is triggered by the inprocess library which tells |
| 290 | ust-consumerd when to consume. |
| 291 | |
| 292 | LTT-CONSUMERD: |
| 293 | |
| 294 | The purpose of this daemon is to consume the LTTng trace buffers for only a |
| 295 | specific session. |
| 296 | |
| 297 | For that kernel consumer, ltt-sessiond will pass different anonymous file |
| 298 | descriptors to the ltt-consumerd using a Unix socket. From these file |
| 299 | desriptors, it will be able to get the data from a special function export by |
| 300 | the LTTng kernel. |
| 301 | |
| 302 | ltt-consumerd will be managed by the exact same way as ust-consumerd. However, |
| 303 | in order to trace the kernel, you are either root (UID=0) or in the tracing |
| 304 | group. |
| 305 | |
| 306 | The roles of ltt-consumerd: |
| 307 | |
| 308 | Register to ltt-sessiond - Using a session UUID and credentials (UID/GID) |
| 309 | |
| 310 | Consume buffers - Write data to a file descriptor (on disk, network, ...) |
| 311 | |
| 312 | Kernel triggers ltt-consumerd for buffer consumption. |
| 313 | |
| 314 | UST INPROCESS LIBRARY: |
| 315 | |
| 316 | When the application starts, this library will check for the global named pipe |
| 317 | of ltt-sessiond. If present, it MUST validate that root is the owner. This |
| 318 | check is very important to prevent ltt-sessiond spoofing. If the pipe is root, |
| 319 | we are certain that it's the privileged user that operates tracing. Then, using |
| 320 | it's UID, the application will try to register to the per-user ltt-sessiond |
| 321 | again verifying before the owner ship of the named pipe that should match the |
| 322 | UID. |
| 323 | |
| 324 | Before registration, the inprocess library MUST validate with the ltt-sessiond |
| 325 | the library version for compatibility reason. This is mechanism is useful for |
| 326 | library compatibility but also to see if ltt-sessiond socket is valid (means |
| 327 | that an actual ltt-sessiond is listening on the other side). Having no response |
| 328 | for over 10 seconds, the application will cut communication on that socket and |
| 329 | fallback to the application common named pipe (explain below). |
| 330 | |
| 331 | If the socket is valid, it will register as a traceable application using the |
| 332 | apps credentials and will open a local Unix socket, passed to ltt-sessiond, in |
| 333 | order to receive an eventual shared memory reference. It will then wait on it |
| 334 | if any other command are given by the lttng client. This socket becomes the |
| 335 | only channel of communication between the registry and the application. |
| 336 | |
| 337 | If no ltt-sessiond is present at registration, the application tries to open |
| 338 | the application common named pipe or create it if it does not exist and wait on |
| 339 | it (using poll or epoll Linux API). Having any type of event on that pipe, the |
| 340 | inprocess library will then try to register to the global and per-user |
| 341 | ltt-sessiond. If it fails again, it goes back again to wait on that pipe. |
| 342 | |
| 343 | SHARED MEMORY |
| 344 | |
| 345 | For UST, this is the memory area where the tracing buffers will be held and |
| 346 | given access in read-write mode for the inprocess library of the application. |
| 347 | |
| 348 | On the LTTng side (for ltt-consumerd), these buffers are in the kernel space |
| 349 | and given access by opening a file in the debugfs file system. With an |
| 350 | anonymous file desriptor, this consumer will be able to extract the data. |
| 351 | |
| 352 | This memory is ONLY used for the tracing data. No communication between |
| 353 | components is done using that memory. |
| 354 | |
| 355 | A shared memory segment for tracing MUST be set with the tracing group GID for |
| 356 | the UST buffers. This is the job of ltt-sessiond. |
| 357 | |
| 358 | PREREQUISITES: |
| 359 | |
| 360 | The global ltt-sessiond daemon MUST always be running as "root" or an |
| 361 | equivalent user having the same privilege as root (UID = 0). |
| 362 | |
| 363 | The ltt-sessiond daemon SHOULD be up and running at all time in order to trace |
| 364 | a tracable application. |
| 365 | |
| 366 | The new lttng library API MUST be used to interact with the |
| 367 | ltt-sessiond registry daemon for every trace action needed by the user. |
| 368 | |
| 369 | A tracing group MUST be created. Whoever is in that group is able to access the |
| 370 | tracing data of any buffers and is able to trace any application or the kernel. |
| 371 | |
| 372 | WARNING: The tracing group name COULD interfere with other already existing |
| 373 | groups. Care should be put at install time for that (from source and packages) |
| 374 | |
| 375 | The next section illustrates different use cases using that new model. |
| 376 | |
| 377 | Use Cases |
| 378 | ----------------- |
| 379 | |
| 380 | Each case considers these : |
| 381 | |
| 382 | * user A - UID: A; GID: A, tracing |
| 383 | * user B - UID: B; GID: B, tracing |
| 384 | |
| 385 | Scenario 1 - Single user tracing app_1 |
| 386 | ------ |
| 387 | |
| 388 | This first scenario shows how user A will start a trace for application app_1 |
| 389 | that is not running. |
| 390 | |
| 391 | 1) lttng ask ltt-sessiond for a new session through a Unix socket. If |
| 392 | allowed, ltt-sessiond returns a session UUID to the client. |
| 393 | (Ex: ops --> new session) |
| 394 | |
| 395 | +-----------+ ops +--------------+ |
| 396 | | lttng A |<---------->| ltt-sessiond | |
| 397 | +-----------+ +--------------+ |
| 398 | |
| 399 | 2) The app_1 is spawned by lttng having the user A credentials. Then, app_1 |
| 400 | automatically register to ltt-sessiond has a "tracable apps" through the global |
| 401 | named pipe of ltt-sessiond using the UID/GID and session UUID. |
| 402 | |
| 403 | The shared memory is created with the app_1 UID (rw-) and tracing group GID |
| 404 | (r--) and a reference is given back to app_1 |
| 405 | |
| 406 | +-----------+ +--------------+ |
| 407 | | lttng A | | ltt-sessiond | |
| 408 | +-----------+ +--------------+ |
| 409 | | ^ | |
| 410 | | +-------+ | | +-------------+ |
| 411 | +-->| app_1 |<--------+ +-->| shared mem. | |
| 412 | +-------+ +-------------+ |
| 413 | |
| 414 | 3) app_1 connect to the shared memory and ust-consumerd is spawned with the |
| 415 | session UUID and lttng credentials (user A). It then register to ltt-sessiond |
| 416 | for a valid session to consume using the previous session UUID and credentials. |
| 417 | |
| 418 | +-----------+ +--------------+ |
| 419 | | lttng A | +-->| ltt-sessiond |----------+ |
| 420 | +-----------+ | +--------------+ | |
| 421 | | | | |
| 422 | | +---------------+ read | commands |
| 423 | +-->| ust-consumerd |---------+ | and |
| 424 | +---------------+ v | options |
| 425 | ^ | +-------------+ | |
| 426 | | v +------>| shared mem. | | |
| 427 | +-------+ | +-------------+ | |
| 428 | | app_1 |-------- | |
| 429 | +-------+ write | |
| 430 | ^ | |
| 431 | +--------------------------------------- |
| 432 | |
| 433 | Scenario 2 - Single user tracing already running app_1 |
| 434 | ------ |
| 435 | |
| 436 | 1) lttng ask ltt-sessiond for a new session through a Unix socket. If allowed |
| 437 | (able to write on socket), ltt-sessiond returns a session UUID to the client. |
| 438 | |
| 439 | +-----------+ ops +--------------+ |
| 440 | | lttng A |<---------->| ltt-sessiond | |
| 441 | +-----------+ +--------------+ |
| 442 | ^ |
| 443 | +-------+ read | |
| 444 | | app_1 |----------+ |
| 445 | +-------+ |
| 446 | |
| 447 | NOTE: At this stage, since app_1 is already running, the registration of app_1 |
| 448 | to ltt-sessiond has already been done. However, the shared memory segment is |
| 449 | not allocated yet until a trace session is initiated. Having no shared memory, |
| 450 | the inprocess library of app_1 will wait on the local Unix socket connected to |
| 451 | ltt-sessiond for the reference. |
| 452 | |
| 453 | +-----------+ +--------------+ |
| 454 | | lttng A | | ltt-sessiond | |
| 455 | +-----------+ +--------------+ |
| 456 | ^ | |
| 457 | +-------+ | | +-------------+ |
| 458 | | app_1 |<--------+ +-->| shared mem. | |
| 459 | +-------+ commands +-------------+ |
| 460 | | ^ |
| 461 | +---------- write ----------+ |
| 462 | |
| 463 | 2) lttng spawns a ust-consumerd for the session. We get the same figure as |
| 464 | step 3 in the first scenario. |
| 465 | |
| 466 | There is a small difference though. The application MAY NOT be using the same |
| 467 | credentials as user A (lttng). However, the shared memory is always GID of |
| 468 | the tracing group. So, in order for user A to trace app_1, is MUST be in the |
| 469 | tracing group otherwise, if the application is not set with the user |
| 470 | credentials, user A will not be able to trace app_1 |
| 471 | |
| 472 | Scenario 3 - Multiple users tracing the same running application |
| 473 | ------ |
| 474 | |
| 475 | 1) Session are created for the two users. Using the same exact mechanism as |
| 476 | before, the shared memory and consumers are created. Two users, two sessions, |
| 477 | two consumers and two shared memories for the same application. |
| 478 | |
| 479 | +-----------+ +--------------+ |
| 480 | | lttng A |-------- ops ------->| ltt-sessiond | |
| 481 | +-----------+ ^ +--------------+ |
| 482 | | ^ commands |
| 483 | +-----------+ | +-------+<--+ |
| 484 | | lttng B |------+ +--->| app_1 |------- write -----+ |
| 485 | +-----------+ | +-------+ | |
| 486 | | | |
| 487 | +-----------------+ | +-------------+ | |
| 488 | | ust-consumerd A |--O--- read ----->| shared mem. |<-+ |
| 489 | +-----------------+ | +-------------+ | |
| 490 | | | |
| 491 | +-----------------+ v +-------------+ | |
| 492 | | ust-consumerd B |--+--- read ----->| shared mem. |<-+ |
| 493 | +-----------------+ +-------------+ |
| 494 | |
| 495 | ust-consumerd A - UID: user A (rw-), GID: tracing (r--) |
| 496 | ust-consumerd B - UID: user B (rw-), GID: tracing (r--) |
| 497 | |
| 498 | Scenario 4 - User not in the tracing group |
| 499 | ------ |
| 500 | |
| 501 | For this particular case, it's all goes back to the first scenario. The user |
| 502 | MUST start the application using his credentials. The session will be created |
| 503 | by the per-user ltt-sessiond but he will not be able to trace anything that the |
| 504 | user does not owned. |