From fa8f9c825308ce02d5409979895c104cf9b6364c Mon Sep 17 00:00:00 2001 From: David Goulet Date: Mon, 23 Jul 2012 09:42:13 -0400 Subject: [PATCH] Fix and update content of doc/ directory Add lttng-relayd.8 to the Makefile.am Create the proposals/ directory which contains pas and current proposals (RFC) for any features/changes in lttng-tools code tree. Add the streaming-howto.txt to the dist data. Rename and update old RFCs and move them to proposals/ directory. Incidently, remove the dev/ directory. Signed-off-by: David Goulet --- doc/Makefile.am | 4 +- doc/man/Makefile.am | 2 +- .../0001-session-daemon-architecture.txt} | 68 +++--- .../0002-lttng-command-line-UI.txt} | 8 +- doc/proposals/0003-network.consumer.txt | 202 ++++++++++++++++++ 5 files changed, 247 insertions(+), 37 deletions(-) rename doc/{session-daemon-model.txt => proposals/0001-session-daemon-architecture.txt} (94%) rename doc/{dev/lttng-cli.txt => proposals/0002-lttng-command-line-UI.txt} (97%) create mode 100644 doc/proposals/0003-network.consumer.txt diff --git a/doc/Makefile.am b/doc/Makefile.am index 23d93ed70..954c67ab3 100644 --- a/doc/Makefile.am +++ b/doc/Makefile.am @@ -1,4 +1,4 @@ SUBDIRS = man -EXTRA_DIST = quickstart.txt session-daemon-model.txt +EXTRA_DIST = quickstart.txt streaming-howto.txt -dist_doc_DATA = quickstart.txt +dist_doc_DATA = quickstart.txt streaming-howto.txt diff --git a/doc/man/Makefile.am b/doc/man/Makefile.am index 83a115dc0..d91a91a88 100644 --- a/doc/man/Makefile.am +++ b/doc/man/Makefile.am @@ -1,2 +1,2 @@ dist_man1_MANS = lttng.1 -dist_man8_MANS = lttng-sessiond.8 +dist_man8_MANS = lttng-sessiond.8 lttng-relayd.8 diff --git a/doc/session-daemon-model.txt b/doc/proposals/0001-session-daemon-architecture.txt similarity index 94% rename from doc/session-daemon-model.txt rename to doc/proposals/0001-session-daemon-architecture.txt index da42e276d..411d148d8 100644 --- a/doc/session-daemon-model.txt +++ b/doc/proposals/0001-session-daemon-architecture.txt @@ -1,4 +1,8 @@ -RFC - New processes model for UST and LTTng +RFC - LTTng session daemon architecture + +WARNING: + Parts of the proposal are obselete but we keep this version for historical + purposes. Author: David Goulet @@ -13,7 +17,7 @@ Version: - v0.1: 17/01/2011 * Initial proposal - - v0.2: 19/01/2011 + - v0.2: 19/01/2011 After multiple reply from all the contributors above, here is the list of what has changed: * Change/Add Terminology elements from the initial model @@ -22,7 +26,7 @@ Version: * LTTng kernel tracer support proposition * More details for the Model and Components * Improve the basic model. Quite different from the last one - + - v0.3: 28/01/2011 In response from Michel Dagenais and Nils Carlson comments: * Add scaling to reasons of this re-engineering @@ -34,7 +38,7 @@ Version: - v0.4: 01/02/2011 After Mathieu Desnoyers and Michel Dagenais comments: * Add section Introduction - * Define the global and per-user ltt-sessiond + * Define the global and per-user ltt-sessiond * Add details for ltt-sessiond in the inprocess lib section * Session ID are now UUID * Add buffer snapshot schema for ust-consumerd @@ -42,6 +46,10 @@ Version: * ltt-sessiond socket validation by the inprocess lib. * Add lttng definition * Add consumer to the Model section + - v0.5: 23/07/2012 + * Please refer to the thesis of David Goulet for the complete + and up to date specifications of the architecture and algorithms used. + (http://publications.polymtl.ca/842/1/2012_DavidGoulet.pdf) Terminology ----------------- @@ -211,9 +219,9 @@ daemon that the buffers are ready for consumption. Here is a flow of action to illustrate the ust-consumerd life span: -1) +1) +-----------+ ops +--------------+ -| lttng A |<---------->| ltt-sessiond | +| lttng A |<---------->| ltt-sessiond | +-----------+ +--------------+ lttng ask for tracing an application using the PID and the session UUID. The @@ -222,11 +230,11 @@ socket if ust-consumerd already exist. 2a) If ust-consumerd EXIST -+-----------+ -| lttng A | -+-----------+ - | mem ref. - | +---------------+ read +------------+ ++-----------+ +| lttng A | ++-----------+ + | mem ref. + | +---------------+ read +------------+ +-->| ust-consumerd |--------->| shared mem | +---------------+ +------------+ @@ -236,7 +244,7 @@ the reference it previously got from ltt-sessiond. 2b) If ust-consumerd DOES NOT EXIST +-----------+ +--------------+ -| lttng A | +---->| ltt-sessiond | +| lttng A | +---->| ltt-sessiond | +-----------+ | +--------------+ | ID | | mem ref. | register @@ -252,18 +260,18 @@ Quick buffer snapshot: 1) Here, lttng will request a buffer snapshot for an already running session. -+-----------+ +--------------+ -| lttng A |-------- ops ------->| ltt-sessiond | -+-----------+ +--------------+ ++-----------+ +--------------+ +| lttng A |-------- ops ------->| ltt-sessiond | ++-----------+ +--------------+ | | command | +-----------------+ +-------+<--+ | | ust-consumerd 1 |<----| app_1 |-+ | +-----------------+ +-------+ | write - | 1 | v + | 1 | v | | +-------------+ | +--- read ----->| shared mem. | - | +-------------+ - | ^ + | +-------------+ + | ^ | +-----------------+ | +->| ust-consumerd 2 |----------+ +-----------------+ snapshot @@ -282,7 +290,7 @@ tracing data will also be set to UID/GID of the lttng client. No setuid or setgid is used, we only use the credentials of the user. The roles of ust-consumerd: - + Register to ltt-sessiond - Using a session UUID and credentials (UID/GID) Consume buffers - Write data to a file descriptor (on disk, network, ...) @@ -293,7 +301,7 @@ ust-consumerd when to consume. LTT-CONSUMERD: The purpose of this daemon is to consume the LTTng trace buffers for only a -specific session. +specific session. For that kernel consumer, ltt-sessiond will pass different anonymous file descriptors to the ltt-consumerd using a Unix socket. From these file @@ -365,7 +373,7 @@ The ltt-sessiond daemon SHOULD be up and running at all time in order to trace a tracable application. The new lttng library API MUST be used to interact with the -ltt-sessiond registry daemon for every trace action needed by the user. +ltt-sessiond registry daemon for every trace action needed by the user. A tracing group MUST be created. Whoever is in that group is able to access the tracing data of any buffers and is able to trace any application or the kernel. @@ -440,7 +448,7 @@ Scenario 2 - Single user tracing already running app_1 +-----------+ ops +--------------+ | lttng A |<---------->| ltt-sessiond | +-----------+ +--------------+ - ^ + ^ +-------+ read | | app_1 |----------+ +-------+ @@ -477,21 +485,21 @@ Scenario 3 - Multiple users tracing the same running application before, the shared memory and consumers are created. Two users, two sessions, two consumers and two shared memories for the same application. -+-----------+ +--------------+ -| lttng A |-------- ops ------->| ltt-sessiond | -+-----------+ ^ +--------------+ ++-----------+ +--------------+ +| lttng A |-------- ops ------->| ltt-sessiond | ++-----------+ ^ +--------------+ | ^ commands +-----------+ | +-------+<--+ | lttng B |------+ +--->| app_1 |------- write -----+ -+-----------+ | +-------+ | - | | ++-----------+ | +-------+ | + | | +-----------------+ | +-------------+ | | ust-consumerd A |--O--- read ----->| shared mem. |<-+ - +-----------------+ | +-------------+ | - | | + +-----------------+ | +-------------+ | + | | +-----------------+ v +-------------+ | | ust-consumerd B |--+--- read ----->| shared mem. |<-+ - +-----------------+ +-------------+ + +-----------------+ +-------------+ ust-consumerd A - UID: user A (rw-), GID: tracing (r--) ust-consumerd B - UID: user B (rw-), GID: tracing (r--) diff --git a/doc/dev/lttng-cli.txt b/doc/proposals/0002-lttng-command-line-UI.txt similarity index 97% rename from doc/dev/lttng-cli.txt rename to doc/proposals/0002-lttng-command-line-UI.txt index f8fa17263..2c83b769a 100644 --- a/doc/dev/lttng-cli.txt +++ b/doc/proposals/0002-lttng-command-line-UI.txt @@ -1,7 +1,7 @@ -******** DEPRECATED ********** -* Kept for historic purposes * -******** DEPRECATED ********** +******** DEPRECATED ************ +* Kept for historical purposes * +******** DEPRECATED ************ lttng-tools command line interface @@ -52,7 +52,7 @@ lttng version command_name # show lttng versions for app by command * Tracing session control lttng create [--session] name # create a tracing session (default name provided) - [--output path] # optionally specify the output path + [--output path] # optionally specify the output path ---> This command prints "Working directory of created session is /path/to/name. Change your" diff --git a/doc/proposals/0003-network.consumer.txt b/doc/proposals/0003-network.consumer.txt new file mode 100644 index 000000000..a832d18db --- /dev/null +++ b/doc/proposals/0003-network.consumer.txt @@ -0,0 +1,202 @@ +RFC - Network consumer + +Author: David Goulet + +Contributors: + * Mathieu Desnoyers + * Julien Desfossez + +Version: + - v0.1: 16/04/2012 + * Initial proposal + - v0.2: 04/05/2012 + * Add snapshot description + * Propose a new API and lttng cli command + - v0.3: 23/07/2012 + * Remove snapshot and focus only on network consumer for straming. + +Introduction +----------------- + +This RFC proposes a way for the lttng 2.0 session daemon to handle network +session for streaming purposes and eventually remote control. + +The next sections introduce the concept of network session and how it is +envisioned in the lttng 2.0 toolchain. + +Please note that this RFC is neither final nor complete without the community +feedbacks. The text below is a proposal. + +Network consumer +----------------- + +For version 2.1 of lttngt-tools, we propose to add a network consumer which will +be used for streaming. + +We allow to pass a full URI to the enable consumer command to override the +current consumer or define a new one. + +We should at least support the following protocols: + +* net://HOST[:PORT_CTRL[:PORT_DATA]] +* tcp://HOST:PORT +* tcp6://HOST:PORT +* udp://HOST:PORT +* udp6://HOST:PORT +* file:// + +The net:// URI scheme makes the control and data stream use the default +transport protocol which is TCP. The same remote host is also +used for both. The ports can be specified and if not the defaults are used which +are 5342 for control and 5343 for data. + +If URI not recognized, we use the arguments as a file name (same behavior as +using file:///). + +The control and data stream are two separate arguments of the API since we allow +the user to control the protocol and path (address). However, for a transfer to +succeed, the lttng-sessiond and the remote end must establish a session for the +control _and_ data path. If one fails to do so, the procedure is aborted. Thus, +a different address for the control path from the data path is allowed but the +user has to make sure that both communication channels end up at the same +physical destination. + +Note that the control path is a crucial and high priority channel of +communication so for now we only allow it to use the TCP protocol. + +Session with Network Transport +----------------- + +In order to tell the session daemon where to send the data for streaming, a +tracing session has to be aware of some information of the remote target. + + * Remote end network address (Ex: IP or Hostname) + * Destination control port + * Destination data port + +Streaming can be initiated by telling the session daemon that a specific session +is set for network streaming. This will make the session daemon establish a +connection with the remote end. Once tracing starts, the local consumer will be +made aware of this information and will start sending data following a strict +protocol defined in the streaming RFC written by Julien Desfossez. + +Finally, a trace received by a network consumer will have a new "namespace" +prepended to the trace output directory hierarchy: the hostname from _where_ the +trace is coming from. + +host01 +\-- my_session1 + \-- ust + \-- my_app1[...] + \-- trace data... + \-- kernel + \-- trace data... + +Client API integration +----------------- + +Adding an API call to set attributes such as network information to a session. +Since lttng_create_session only takes a name and a path, a new call is required +to pass this information. The naming convention is NOT final and can be +improved. + +struct lttng_handle handle; + +enum lttng_dst_type { + LTTNG_DST_IPV4, + LTTNG_DST_IPV6, + LTTNG_DST_HOST, + LTTNG_DST_PATH, +}; + +enum lttng_uri_type { + LTTNG_URI_HOP, + LTTNG_URI_DST, +}; + +enum lttng_stream_type { + LTTNG_STREAM_CONTROL, + LTTNG_STREAM_DATA +}; + +enum lttng_proto_type { + LTTNG_UDP, + LTTNG_TCP, +}; + +#define LTTNG_NETWORK_PADDING1_LEN 32 +#define LTTNG_NETWORK_PADDING2_LEN 128 +struct lttng_uri { + enum lttng_dst_type dtype; + enum lttng_uri_type utype; + enum lttng_stream_type stype; + enum lttng_proto proto; + in_port_t port; + char padding[LTTNG_NETWORK_PADDING1_LEN]; + char subdir[PATH_MAX]; + union { + char ipv4[INET_ADDRSTRLEN]; + char ipv6[INET6_ADDRSTRLEN]; + char path[PATH_NAME]; + char padding[LTTNG_NETWORK_PADDING2_LEN]; + } dst; +}; + +/* Set URI in the consumer template*/ +lttng_set_consumer_uri(handle, struct lttng_uri *u); + + +/* + * Enable consumer template for the session. Once enabled, no more URI setting + * are possible for that specific consumer. + */ +lttng_enable_consumer(handle); + +/* + * Disable the consumer means that the consumer will stop consuming but will + * still be exist. Executing the enable_consumer call again will simply re + * enable it. + */ +lttng_disable_consumer(handle); + +If lttng_set_consumer_uri is executed on a session which already has a network +consumer attached to it, the present consumer is freed and a new template is +added. + +We propose to add two commands to the lttng command line actions: + +i) lttng enable-consumer [URI] [OPTIONS] + -s SESSION_NAME + -C, --control-uri=[HOP1,]URI + -D, --data-uri=[HOP1,]URI + +ii) lttng disable-consumer + -s SESSION_NAME + +Each option defining URI(s) can contains a list of hops preceeding the final +destination. However, the proxy feature is still not supported but we prefer to +inform the community of is future existence. + +So, the regular chain of command to enable a network consumer would be: + +# lttng create session1 +// The command sets the destination but uses the default protocols and ports. +# lttng enable-consumer net://192.168.1.10 +# lttng enable-event -a -k +# lttng start +(tracing...) +# lttng stop + +(This example considers that there is a lttng-relayd on the remote end.) + +Session daemon integration +----------------- + +As mentioned earlier, the session daemon will be in charge of establishing a +streaming session with the target over the network i.e. creating the control and +data path bidirectional socket. Once done, a network consumer is spawned and +those sockets are passed over. + +From there, the session daemon can interact with the consumer by stopping the +network streaming or re-establishing a local trace collection with a non network +consumer. -- 2.34.1