+++ /dev/null
-
-Some thoughts about userspace tracing
-
-Mathieu Desnoyers January 2006
-
-
-
-* Goals
-
-Fast and secure user space tracing.
-
-Fast :
-
-- 5000ns for a system call is too long. Writing an event directly to memory
- takes 220ns.
-- Still, we can afford a system call for buffer switch, which occurs less often.
-- No locking, no signal disabling. Disabling signals require 2 system calls.
- Mutexes are implemented with a short spin lock, followed by a yield. Yet
- another system call. In addition, we have no way to know on which CPU we are
- running when in user mode. We can be preempted anywhere.
-- No contention.
-- No interrupt disabling : it doesn't exist in user mode.
-
-Secure :
-
-- A process shouldn't be able to corrupt the system's trace or another
- process'trace. It should be limited to its own memory space.
-
-
-
-* Solution
-
-- Signal handler concurrency
-
-Using atomic space reservation in the buffer(s) will remove the requirement for
-locking. This is the fast and safe way to deal with concurrency coming from
-signal handlers.
-
-- Start/stop tracing
-
-Two possible solutions :
-
-Either we export a read-only memory page from kernel to user space. That would
-be somehow seen as a hack, as I have never even seen such interface anywhere
-else. It may lead to problems related to exported types. The proper, but slow,
-way to do it would be to have a system call that would return the tracing
-status.
-
-My suggestion is to go for a system call, but only call it :
-
-- when the thread starts
-- when receiving a SIGRTMIN+3 (multithread ?)
-
-Note : save the thread ID (process ID) in the logging function and the update
-handler. Use it as a comparison to check if we are a forked child thread.
-Start a brand new buffer list in that case.
-
-
-Two possibilities :
-
-- one system call per information to get/one system call to get all information.
-- one signal per information to get/one signal for "update" tracing info.
-
-I would tend to adopt :
-
-- One signal for "general tracing update"
- One signal handler would clearly be enough, more would be unnecessary
- overhead/pollution.
-- One system call for all updates.
- We will need to have multiple parameters though. We have up to 6 parameters.
-
-syscall get_tracing_info
-
-parameter 1 : trace buffer map address. (id)
-
-parameter 2 : active ? (int)
-
-
-Concurrency
-
-We must have per thread buffers. Then, no memory can be written by two threads
-at once. It removes the need for locks (ok, atomic reservation was already doing
-that) and removes false sharing.
-
-
-Multiple traces
-
-By having the number of active traces, we can allocate as much buffers as we
-need. Allocation is done in the kernel with relay_open. User space mapping is
-done when receiving the signal/starting the process and getting the number of
-traces actives.
-
-It means that we must make sure to only update the data structures used by
-tracing functions once the buffers are created.
-
-We could have a syscall "get_next_buffer" that would basically mmap the next
-unmmapped buffer, or return NULL is all buffers are mapped.
-
-If we remove a trace, the kernel should stop the tracing, and then get the last
-buffer for this trace. What is important is to make sure no writers are still
-trying to write in a memory region that get desallocated.
-
-For that, we will keep an atomic variable "tracing_level", which tells how many
-times we are nested in tracing code (program code/signal handlers) for a
-specific trace.
-
-We could do that trace removal in two operations :
-
-- Send an update tracing signal to the process
- - the sig handler get the new tracing status, which tells that tracing is
- disabled for the specific trace. It writes this status in the tracing
- control structure of the process.
- - If tracing_level is 0, well, it's fine : there are no potential writers in
- the removed trace. It's up to us to buffer switch the removed trace, and,
- after the control returns to us, set_tracing_info this page to NULL and
- delete this memory area.
- - Else (tracing_level > 0), flag the removed trace for later switch/delete.
-
- It then returns control to the process.
-
-- If the tracing_level was > 0, there was one or more writers potentially
- accessing this memory area. When the control comes back to the writer, at the
- end of the write in a trace, if the trace is marked for switch/delete and the
- tracing_level is 0 (after the decrement of the writer itself), then the
- writer must buffer switch, and then delete the memory area.
-
-
-Filter
-
-The update tracing info signal will make the thread get the new filter
-information. Getting this information will also happen upon process creation.
-
-parameter 3 for the get tracing info : a integer containing the 32 bits mask.
-
-
-Buffer switch
-
-There could be a tracing_buffer_switch system call, that would give the page
-start address as parameter. The job of the kernel is to steal this page,
-possibly replacing it with a zeroed page (we don't care about the content of the
-page after the syscall).
-
-Process dying
-
-The kernel should be aware of the current pages used for tracing in each thread.
-If a thread dies unexpectedly, we want the kernel to get the last bits of
-information before the thread crashes.
-
-Memory protection
-
-If a process corrupt its own mmaped buffers, the rest of the trace will be
-readable, and each process have its own memory space.
-
-Two possibilities :
-
-Either we create one channel per process, or we have per cpu tracefiles for all
-the processes, with the specification that data is written in a monotically
-increasing time order and that no process share a 4k page with another process.
-
-The problem with having only one tracefile per cpu is that we cannot safely
-steal a process'buffer upon a schedule change because it may be currently
-writing to it.
-
-It leaves the one tracefile per thread as the only solution.
-
-Another argument in favor of this solution is the possibility to have mixed
-32-64 bits processes on the same machine. Dealing with types will be easier.
-
-
-Corrupted trace
-
-A corrupted tracefile will only affect one thread. The rest of the trace will
-still be readable.
-
-
-Facilities
-
-Upon process creation or when receiving the signal of trace info update, when a
-new trace appears, the thread should write the facility information into it. It
-must then have a list of registered facilities, all done at the thread level.
-
-We must decide if we allow a facility channel for each thread. The advantage is
-that we have a readable channel in flight recorder mode, while the disadvantage
-is to duplicate the number of channels, which may become quite high. To follow
-the general design of a high throughput channel and a low throughput channel for
-vital information, I suggest to have a separate channel for facilities, per
-trace, per process.
-
-
-
-API :
-
-syscall 1 :
-
-in :
-buffer : NULL means get new traces
- non NULL means to get the information for the specified buffer
-out :
-buffer : returns the address of the trace buffer
-active : is the trace active ?
-filter : 32 bits filter mask
-
-return : 0 on success, 1 on error.
-
-int ltt_update(void **buffer, int *active, int *filter);
-
-syscall 2 :
-
-in :
-buffer : Switch the specified buffer.
-return : 0 on success, 1 on error.
-
-int ltt_switch(void *buffer);
-
-
-Signal :
-
-SIGRTMIN+3
-(like hardware fault and expiring timer : to the thread, see p. 413 of Advances
-prog. in the UNIX env.)
-
-Signal is sent on tracing create/destroy, start/stop and filter change.
-
-Will update for itself only : it will remove unnecessary concurrency.
-
-
-
-Notes :
-
-It doesn't matter "when" the process receives the update signal after a trace
-start : it will receive it in priority, before executing anything else when it
-will be scheduled in.
-
-
-
-Major enhancement :
-
-* Buffer pool *
-
-The problem with the design, up to now, is if an heavily threaded application
-launches many threads that has a short lifetime : it will allocate memory for
-each traced thread, consuming time and it will create an incredibly high
-number of files in the trace (or per thread).
-
-(thanks to Matthew Khouzam)
-The solution to this sits in the use of a buffer poll : We typically create a
-buffer pool of a specified size (say, 10 buffers by default, alterable by the
-user), each 8k in size (4k for normal trace, 4k for facility channel), for a
-total of 80kB of memory. It has to be tweaked to the maximum number of
-expected threads running at once, or it will have to grow dynamically (thus
-impacting on the trace).
-
-A typical approach to dynamic growth is to double the number of allocated
-buffers each time a threashold near the limit is reached.
-
-Each channel would be found as :
-
-trace_name/user/facilities_0
-trace_name/user/cpu_0
-trace_name/user/facilities_1
-trace_name/user/cpu_1
-...
-
-When a thread asks for being traced, it gets a buffer from free buffers pool. If
-the number of available buffers falls under a threshold, the pool is marked for
-expansion and the thread gets its buffer quickly. The expansion will be executed
-a little bit later by a worker thread. If however, the number of available
-buffer is 0, then an "emergency" reservation will be done, allocating only one
-buffer. The goal of this is to modify the thread fork time as less as possible.
-
-When a thread releases a buffer (the thread terminates), a buffer switch is
-performed, so the data can be flushed to disk and no other thread will mess
-with it or render the buffer unreadable.
-
-Upon trace creation, the pre-allocated pool is allocated. Upon trace
-destruction, the threads are first informed of the trace destruction, any
-pending worker thread (for pool allocation) is cancelled and then the pool is
-released. Buffers used by threads at this moment but not mapped for reading
-will be simply destroyed (as their refcount will fall to 0). It means that
-between the "trace stop" and "trace destroy", there should be enough time to let
-the lttd daemon open the newly created channels or they will be lost.
-
-Upon buffer switch, the reader can read directly from the buffer. Note that when
-the reader finish reading a buffer, if the associated thread writer has
-exited, it must fill the buffer with zeroes and put it back into the free pool.
-In the case where the trace is destroyed, it must just derement its refcount (as
-it would do otherwise) and the buffer will be destroyed.
-
-This pool will reduce the number of trace files created to the order of the
-number of threads present in the system at a given time.
-
-A worse cast scenario is 32768 processes traced at the same time, for a total
-amount of 256MB of buffers. If a machine has so many threads, it probably have
-enough memory to handle this.
-
-In flight recorder mode, it would be interesting to use a LRU algorithm to
-choose which buffer from the pool we must take for a newly forked thread. A
-simple queue would do it.
-
-SMP : per cpu pools ? -> no, L1 and L2 caches are typically too small to be
-impacted by the fact that a reused buffer is on a different or the same CPU.
-
-
-
-
-
-
-
-
-
-
-
-
-
--- /dev/null
+
+Some thoughts about userspace tracing
+
+Mathieu Desnoyers January 2006
+
+
+
+* Goals
+
+Fast and secure user space tracing.
+
+Fast :
+
+- 5000ns for a system call is too long. Writing an event directly to memory
+ takes 220ns.
+- Still, we can afford a system call for buffer switch, which occurs less often.
+- No locking, no signal disabling. Disabling signals require 2 system calls.
+ Mutexes are implemented with a short spin lock, followed by a yield. Yet
+ another system call. In addition, we have no way to know on which CPU we are
+ running when in user mode. We can be preempted anywhere.
+- No contention.
+- No interrupt disabling : it doesn't exist in user mode.
+
+Secure :
+
+- A process shouldn't be able to corrupt the system's trace or another
+ process'trace. It should be limited to its own memory space.
+
+
+
+* Solution
+
+- Signal handler concurrency
+
+Using atomic space reservation in the buffer(s) will remove the requirement for
+locking. This is the fast and safe way to deal with concurrency coming from
+signal handlers.
+
+- Start/stop tracing
+
+Two possible solutions :
+
+Either we export a read-only memory page from kernel to user space. That would
+be somehow seen as a hack, as I have never even seen such interface anywhere
+else. It may lead to problems related to exported types. The proper, but slow,
+way to do it would be to have a system call that would return the tracing
+status.
+
+My suggestion is to go for a system call, but only call it :
+
+- when the thread starts
+- when receiving a SIGRTMIN+3 (multithread ?)
+
+Note : save the thread ID (process ID) in the logging function and the update
+handler. Use it as a comparison to check if we are a forked child thread.
+Start a brand new buffer list in that case.
+
+
+Two possibilities :
+
+- one system call per information to get/one system call to get all information.
+- one signal per information to get/one signal for "update" tracing info.
+
+I would tend to adopt :
+
+- One signal for "general tracing update"
+ One signal handler would clearly be enough, more would be unnecessary
+ overhead/pollution.
+- One system call for all updates.
+ We will need to have multiple parameters though. We have up to 6 parameters.
+
+syscall get_tracing_info
+
+parameter 1 : trace buffer map address. (id)
+
+parameter 2 : active ? (int)
+
+
+Concurrency
+
+We must have per thread buffers. Then, no memory can be written by two threads
+at once. It removes the need for locks (ok, atomic reservation was already doing
+that) and removes false sharing.
+
+
+Multiple traces
+
+By having the number of active traces, we can allocate as much buffers as we
+need. Allocation is done in the kernel with relay_open. User space mapping is
+done when receiving the signal/starting the process and getting the number of
+traces actives.
+
+It means that we must make sure to only update the data structures used by
+tracing functions once the buffers are created.
+
+We could have a syscall "get_next_buffer" that would basically mmap the next
+unmmapped buffer, or return NULL is all buffers are mapped.
+
+If we remove a trace, the kernel should stop the tracing, and then get the last
+buffer for this trace. What is important is to make sure no writers are still
+trying to write in a memory region that get desallocated.
+
+For that, we will keep an atomic variable "tracing_level", which tells how many
+times we are nested in tracing code (program code/signal handlers) for a
+specific trace.
+
+We could do that trace removal in two operations :
+
+- Send an update tracing signal to the process
+ - the sig handler get the new tracing status, which tells that tracing is
+ disabled for the specific trace. It writes this status in the tracing
+ control structure of the process.
+ - If tracing_level is 0, well, it's fine : there are no potential writers in
+ the removed trace. It's up to us to buffer switch the removed trace, and,
+ after the control returns to us, set_tracing_info this page to NULL and
+ delete this memory area.
+ - Else (tracing_level > 0), flag the removed trace for later switch/delete.
+
+ It then returns control to the process.
+
+- If the tracing_level was > 0, there was one or more writers potentially
+ accessing this memory area. When the control comes back to the writer, at the
+ end of the write in a trace, if the trace is marked for switch/delete and the
+ tracing_level is 0 (after the decrement of the writer itself), then the
+ writer must buffer switch, and then delete the memory area.
+
+
+Filter
+
+The update tracing info signal will make the thread get the new filter
+information. Getting this information will also happen upon process creation.
+
+parameter 3 for the get tracing info : a integer containing the 32 bits mask.
+
+
+Buffer switch
+
+There could be a tracing_buffer_switch system call, that would give the page
+start address as parameter. The job of the kernel is to steal this page,
+possibly replacing it with a zeroed page (we don't care about the content of the
+page after the syscall).
+
+Process dying
+
+The kernel should be aware of the current pages used for tracing in each thread.
+If a thread dies unexpectedly, we want the kernel to get the last bits of
+information before the thread crashes.
+
+Memory protection
+
+If a process corrupt its own mmaped buffers, the rest of the trace will be
+readable, and each process have its own memory space.
+
+Two possibilities :
+
+Either we create one channel per process, or we have per cpu tracefiles for all
+the processes, with the specification that data is written in a monotically
+increasing time order and that no process share a 4k page with another process.
+
+The problem with having only one tracefile per cpu is that we cannot safely
+steal a process'buffer upon a schedule change because it may be currently
+writing to it.
+
+It leaves the one tracefile per thread as the only solution.
+
+Another argument in favor of this solution is the possibility to have mixed
+32-64 bits processes on the same machine. Dealing with types will be easier.
+
+
+Corrupted trace
+
+A corrupted tracefile will only affect one thread. The rest of the trace will
+still be readable.
+
+
+Facilities
+
+Upon process creation or when receiving the signal of trace info update, when a
+new trace appears, the thread should write the facility information into it. It
+must then have a list of registered facilities, all done at the thread level.
+
+We must decide if we allow a facility channel for each thread. The advantage is
+that we have a readable channel in flight recorder mode, while the disadvantage
+is to duplicate the number of channels, which may become quite high. To follow
+the general design of a high throughput channel and a low throughput channel for
+vital information, I suggest to have a separate channel for facilities, per
+trace, per process.
+
+
+
+API :
+
+syscall 1 :
+
+in :
+buffer : NULL means get new traces
+ non NULL means to get the information for the specified buffer
+out :
+buffer : returns the address of the trace buffer
+active : is the trace active ?
+filter : 32 bits filter mask
+
+return : 0 on success, 1 on error.
+
+int ltt_update(void **buffer, int *active, int *filter);
+
+syscall 2 :
+
+in :
+buffer : Switch the specified buffer.
+return : 0 on success, 1 on error.
+
+int ltt_switch(void *buffer);
+
+
+Signal :
+
+SIGRTMIN+3
+(like hardware fault and expiring timer : to the thread, see p. 413 of Advances
+prog. in the UNIX env.)
+
+Signal is sent on tracing create/destroy, start/stop and filter change.
+
+Will update for itself only : it will remove unnecessary concurrency.
+
+
+
+Notes :
+
+It doesn't matter "when" the process receives the update signal after a trace
+start : it will receive it in priority, before executing anything else when it
+will be scheduled in.
+
+
+
+Major enhancement :
+
+* Buffer pool *
+
+The problem with the design, up to now, is if an heavily threaded application
+launches many threads that has a short lifetime : it will allocate memory for
+each traced thread, consuming time and it will create an incredibly high
+number of files in the trace (or per thread).
+
+(thanks to Matthew Khouzam)
+The solution to this sits in the use of a buffer poll : We typically create a
+buffer pool of a specified size (say, 10 buffers by default, alterable by the
+user), each 8k in size (4k for normal trace, 4k for facility channel), for a
+total of 80kB of memory. It has to be tweaked to the maximum number of
+expected threads running at once, or it will have to grow dynamically (thus
+impacting on the trace).
+
+A typical approach to dynamic growth is to double the number of allocated
+buffers each time a threashold near the limit is reached.
+
+Each channel would be found as :
+
+trace_name/user/facilities_0
+trace_name/user/cpu_0
+trace_name/user/facilities_1
+trace_name/user/cpu_1
+...
+
+When a thread asks for being traced, it gets a buffer from free buffers pool. If
+the number of available buffers falls under a threshold, the pool is marked for
+expansion and the thread gets its buffer quickly. The expansion will be executed
+a little bit later by a worker thread. If however, the number of available
+buffer is 0, then an "emergency" reservation will be done, allocating only one
+buffer. The goal of this is to modify the thread fork time as less as possible.
+
+When a thread releases a buffer (the thread terminates), a buffer switch is
+performed, so the data can be flushed to disk and no other thread will mess
+with it or render the buffer unreadable.
+
+Upon trace creation, the pre-allocated pool is allocated. Upon trace
+destruction, the threads are first informed of the trace destruction, any
+pending worker thread (for pool allocation) is cancelled and then the pool is
+released. Buffers used by threads at this moment but not mapped for reading
+will be simply destroyed (as their refcount will fall to 0). It means that
+between the "trace stop" and "trace destroy", there should be enough time to let
+the lttd daemon open the newly created channels or they will be lost.
+
+Upon buffer switch, the reader can read directly from the buffer. Note that when
+the reader finish reading a buffer, if the associated thread writer has
+exited, it must fill the buffer with zeroes and put it back into the free pool.
+In the case where the trace is destroyed, it must just derement its refcount (as
+it would do otherwise) and the buffer will be destroyed.
+
+This pool will reduce the number of trace files created to the order of the
+number of threads present in the system at a given time.
+
+A worse cast scenario is 32768 processes traced at the same time, for a total
+amount of 256MB of buffers. If a machine has so many threads, it probably have
+enough memory to handle this.
+
+In flight recorder mode, it would be interesting to use a LRU algorithm to
+choose which buffer from the pool we must take for a newly forked thread. A
+simple queue would do it.
+
+SMP : per cpu pools ? -> no, L1 and L2 caches are typically too small to be
+impacted by the fact that a reused buffer is on a different or the same CPU.
+
+
+
+
+
+
+
+
+
+
+
+
+
projects / lttv.git / commitdiff
