Commit | Line | Data |
---|---|---|
7591bab1 MD |
1 | LTTng Relay Daemon Architecture |
2 | Mathieu Desnoyers, August 2015 | |
3 | ||
4 | This document describes the object model and architecture of the relay | |
5 | daemon, after the refactoring done within the commit "Fix: Relay daemon | |
6 | ownership and reference counting". | |
7 | ||
8 | We have the following object composition hierarchy: | |
9 | ||
1bd87328 | 10 | relay connection (main.cpp, for sessiond/consumer) |
7591bab1 | 11 | | |
1bd87328 | 12 | \-> 0 or 1 per-consumer relay session |
7591bab1 MD |
13 | | |
14 | \-> 0 or many ctf-trace | |
15 | | | |
16 | \-> 0 or many stream | |
17 | | | | |
18 | | \-> 0 or many index | |
19 | | | |
20 | \-------> 0 or 1 viewer stream | |
21 | ||
1bd87328 | 22 | live connection (live.cpp, for client) |
7591bab1 MD |
23 | | |
24 | \-> 1 viewer session | |
98b82dfa KS |
25 | | | |
26 | | \-> (transient ref) 0 or many unannounced relay streams | |
7591bab1 | 27 | | |
1bd87328 KS |
28 | \-> 0 or many per-consumer relay sessions (actually a reference to a |
29 | | per-consumer relay session as created by the relay connection) | |
7591bab1 MD |
30 | | |
31 | \-> ..... (ctf-trace, stream, index, viewer stream) | |
32 | ||
1bd87328 | 33 | There are global tables declared in lttng-relayd.h for per-consumer relay sessions |
7591bab1 | 34 | (sessions_ht, indexed by session id), streams (relay_streams_ht, indexed |
98b82dfa KS |
35 | by stream handle), viewer sessions (viewer_sessions_ht, indexed by |
36 | connection sock fd), and viewer streams (viewer_streams_ht, indexed | |
37 | by stream handle). The purpose of those tables is to allow fast lookup of | |
7591bab1 MD |
38 | those objects using the IDs received in the communication protocols. |
39 | ||
40 | There is also one connection hash table per worker thread. There is one | |
41 | worker thread to receive data (main.c), and one worker thread to | |
42 | interact with viewer clients (live.c). Those tables are indexed by | |
43 | socket file descriptor. | |
44 | ||
45 | A RCU lookup+refcounting scheme has been introduced for all objects | |
46 | (except viewer session which is still an exception at the moment). This | |
47 | scheme allows looking up the objects or doing a traversal on the RCU | |
48 | linked list or hash table in combination with a getter on the object. | |
49 | This getter validates that there is still at least one reference to the | |
ce4d4083 | 50 | object, else the lookup acts just as if the object does not exist. |
7591bab1 MD |
51 | |
52 | The relay_connection (connection between the sessiond/consumer and the | |
53 | relayd) is the outermost object of its hierarchy. | |
54 | ||
55 | The live connection (connection between a live client and the relayd) | |
56 | is the outermost object of its hierarchy. | |
57 | ||
58 | There is also a "lock" mutex in each object. Those are used to | |
59 | synchronize between threads (currently the main.c relay thread and | |
60 | live.c client thread) when objects are shared. Locks can be nested from | |
61 | the outermost object to the innermost object. IOW, the ctf-trace lock can | |
1bd87328 KS |
62 | nest within the per-consumer relay session lock. The unannounced stream list |
63 | lock in viewer sessions is an exception to the default locking order: it may | |
64 | be nested inside the following locks (in order): relay session, ctf_trace, | |
65 | and relay stream. | |
7591bab1 | 66 | |
7591bab1 MD |
67 | RCU linked lists are used to iterate using RCU, and are protected by |
68 | their own mutex for modifications. Iterations should be confirmed using | |
69 | the object "getter" to ensure its refcount is not 0 (except in cases | |
70 | where the caller actually owns the objects and therefore can assume its | |
71 | refcount is not 0). | |
72 | ||
73 | RCU hash tables are used to iterate using RCU. Iteration should be | |
74 | confirmed using the object "getter" to ensure its refcount is not 0 | |
75 | (except again if we have ownership and can assume the object refcount is | |
76 | not 0). | |
77 | ||
78 | Object creation has a refcount of 1. Each getter increments the | |
79 | refcount, and needs to be paired with a "put" to decrement it. A final | |
80 | put on "self" (ownership) will allow refcount to reach 0, therefore | |
81 | triggering release, and thus free through call_rcu. | |
82 | ||
83 | In the composition scheme, we find back references from each composite | |
84 | to its container. Therefore, each composite holds a reference (refcount) | |
85 | on its container. This allows following pointers from e.g. viewer stream | |
1bd87328 KS |
86 | to stream to ctf-trace to per-consumer relay session without performing |
87 | any validation, due to transitive refcounting of those back-references. | |
7591bab1 MD |
88 | |
89 | In addition to those back references, there are a few key ownership | |
90 | references held. The connection in the relay worker thread (main.c) | |
1bd87328 KS |
91 | holds ownership on the per-consumer relay session, and on each stream it |
92 | contains. The connection in the live worker thread (live.c) holds ownership on | |
93 | each viewer stream it creates. The rest is ensured by back references from | |
7591bab1 MD |
94 | composite to container objects. When a connection is closed, it puts all |
95 | the ownership references it is holding. This will then eventually | |
1bd87328 KS |
96 | trigger destruction of the per-consumer relay session, streams, and |
97 | viewer streams associated with the connection when all the back references | |
98 | reach 0. | |
7591bab1 MD |
99 | |
100 | RCU read-side locks are now only held during iteration on RCU lists and | |
101 | hash tables, and within the internals of the get (lookup) and put | |
102 | functions. Those functions then use refcounting to ensure existence of | |
103 | the object when returned to their caller. | |
1bd87328 KS |
104 | |
105 | In the current implementation, while a live viewer may attach to | |
106 | multiple per-consumer relay sessions, they are meant to belong to | |
107 | only a single lttng-sessiond session. |