for (i = 0; i < 5; i++) {
tracepoint(sample_component, message, "Hello World");
usleep(1);
}
printf("Run `lttng regenerate statedump. Press enter \n");
getchar();
dlclose(handle_dog);
printf("Run `lttng regenerate statedump. Press enter \n");
getchar();
dlclose(handle_cat);
return 0;
}
On lttng side:
lttng create
lttng enable-event -u -a
lttng start
valgrind sample
Issue `lttng regenerate statedump` as the app suggest.
The second `lttng regenerate statedump` results in:
==934747== Invalid read of size 8
==934747== at 0x48BA90F: iter_end (lttng-ust-statedump.c:439)
==934747== by 0x48BAD73: lttng_ust_dl_update (lttng-ust-statedump.c:586)
==934747== by 0x48BADC0: do_baddr_statedump (lttng-ust-statedump.c:599)
==934747== by 0x48BAE62: do_lttng_ust_statedump (lttng-ust-statedump.c:633)
==934747== by 0x489F820: lttng_handle_pending_statedump (lttng-events.c:969)
==934747== by 0x488C000: handle_pending_statedump (lttng-ust-comm.c:717)
==934747== by 0x488DCF7: handle_message (lttng-ust-comm.c:1110)
==934747== by 0x48905EA: ust_listener_thread (lttng-ust-comm.c:1756)
==934747== by 0x4B62608: start_thread (pthread_create.c:477)
==934747== by 0x4A4D162: clone (clone.S:95)
==934747== Address 0x4c4ea88 is 4,152 bytes inside a block of size 4,176 free'd
==934747== at 0x483CA3F: free (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==934747== by 0x48B9588: free_dl_node (lttng-ust-statedump.c:123)
==934747== by 0x48BA90A: iter_end (lttng-ust-statedump.c:450)
==934747== by 0x48BAD73: lttng_ust_dl_update (lttng-ust-statedump.c:586)
==934747== by 0x48BADC0: do_baddr_statedump (lttng-ust-statedump.c:599)
==934747== by 0x48BAE62: do_lttng_ust_statedump (lttng-ust-statedump.c:633)
==934747== by 0x489F820: lttng_handle_pending_statedump (lttng-events.c:969)
==934747== by 0x488C000: handle_pending_statedump (lttng-ust-comm.c:717)
==934747== by 0x488DCF7: handle_message (lttng-ust-comm.c:1110)
==934747== by 0x48905EA: ust_listener_thread (lttng-ust-comm.c:1756)
==934747== by 0x4B62608: start_thread (pthread_create.c:477)
==934747== by 0x4A4D162: clone (clone.S:95)
==934747== Block was alloc'd at
==934747== at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==934747== by 0x48B936A: zmalloc (helper.h:27)
==934747== by 0x48B936A: alloc_dl_node (lttng-ust-statedump.c:85)
==934747== by 0x48B98F7: find_or_create_dl_node (lttng-ust-statedump.c:184)
==934747== by 0x48BA205: extract_baddr (lttng-ust-statedump.c:339)
==934747== by 0x48BABC6: extract_bin_info_events (lttng-ust-statedump.c:528)
==934747== by 0x4A8D2F4: dl_iterate_phdr (dl-iteratephdr.c:75)
==934747== by 0x48BAD4C: lttng_ust_dl_update (lttng-ust-statedump.c:583)
==934747== by 0x48BADC0: do_baddr_statedump (lttng-ust-statedump.c:599)
==934747== by 0x48BAE62: do_lttng_ust_statedump (lttng-ust-statedump.c:633)
==934747== by 0x489F820: lttng_handle_pending_statedump (lttng-events.c:969)
==934747== by 0x488C000: handle_pending_statedump (lttng-ust-comm.c:717)
==934747== by 0x488DCF7: handle_message (lttng-ust-comm.c:1110)
==934747==
Cause
=========
Nodes can be removed during the `cds_hlist_for_each_entry_2` iteration which
is not meant to be used when items are removed within the traversal.
Solution
=========
Use `cds_hlist_for_each_entry_safe_2`.
Change-Id: Ibf3d94a4d6f7abac19ed9740eeacfbcb1bdf1f4f Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Fix: bytecode interpreter context_get_index() leaves byte order uninitialized
Observed Issue
==============
With lttng-ust 2.13, when using the event notification capture feature
to capture a context field, e.g. '$ctx.cpu_id', the captured value is
often observed in reverse byte order.
This issue is not visible in lttng-ust 2.12 because it does not
implement the event notification capture feature. However, it would
become observable if a lttng-tools emits a filter bytecode
BYTECODE_OP_GET_SYMBOL instruction to load the context value. For
compatibility purposes, lttng-tools only uses
BYTECODE_OP_GET_CONTEXT_REF to load the filter context fields,
but nothing prevents a future lttng-tools version from using
BYTECODE_OP_GET_SYMBOL instead.
Cause
=====
Within the bytecode interpreter, context_get_index() leaves the "rev_bo"
field uninitialized in the top of stack.
Solution
========
Initialize the rev_bo field based on the context field type
reserve_byte_order field.
If exec(2) is executed by the application concurrently with LTTng-UST
listener threads between the creation of a file descriptor with
socket(2), recvmsg(2), or pipe(2) and call to fcntl(3) FD_CLOEXEC, those
file descriptors will stay open after the exec, which is not intended.
As a consequence, shared memory files for ring buffers can stay present
on the file system for long-running traced processes.
Use:
- pipe2(2) O_CLOEXEC (supported since Linux 2.6.27, and by FreeBSD),
- socket(2) SOCK_CLOEXEC (supported since Linux 2.6.27, and by FreeBSD),
- recvmsg(2) MSG_CMSG_CLOEXEC (supported since Linux 2.6.23 and by FreeBSD),
rather than fcntl(2) FD_CLOEXEC to make sure the file descriptors are
closed on exec immediately upon their creation.
Michael Jeanson [Thu, 10 Feb 2022 15:25:02 +0000 (15:25 +0000)]
Add 'domain' parameter to the Log4j 2.x agent
The initial Log4j 2.x agent commit only implemented a compatibility mode
to be used with the existing LOG4J domain in lttng-tools.
In this mode the agent converts the new Log4j 2.x loglevel values to
their corresponding Log4j 1.x values in the same way the upstream
compatibility bridge does.
This is great when doing in-place migration using the upstream
compatibility bridge but doesn't cover the usecase of an application
that natively uses Log4j 2.x.
This commit adds a new mandatory 'domain' parameter to the Log4j2 agent
which currently only implements the 'LOG4J' compatibility domain in
preparation to adding a 'LOG4J2' domain.
The configuration for a single appender in Log4j 1.x compat mode will
now look like this:
Michael Jeanson [Wed, 2 Feb 2022 19:04:50 +0000 (19:04 +0000)]
fix: Convert custom loglevels in Log4j 2.x agent
The loglevel integer representation has changed between log4j 1.x and
2.x, we currently convert the standard loglevels but passthrough the
custom ones.
This can be problematic when using severity ranges as custom loglevels
won't be properly filtered.
Use the same strategy as the upstream Log4j 2.x compatibility layer by
converting the custom loglevels to their equivalent standard loglevel
value.
Change-Id: I8cbd4706cb774e334380050cf0b407e19d7bc7c4 Signed-off-by: Michael Jeanson <mjeanson@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
This backport differs from the master branch for the
'--enable-java-agent-all' option won't select this new agent since we
wanted to avoid introducing a new dependency in existing configurations.
The name of the new agent jar file is "lttng-ust-agent-log4j2.jar".
It will be installed in the arch-agnostic "$prefix/share/java" path
e.g: "/usr/share/java".
It uses the same jni library "liblttng-ust-log4j-jni.so" as the Log4j 1.x agent.
The agent was designed as a mostly drop-in replacement for applications
upgrading from Log4j 1.x to 2.x. It requires no modification to the
tracing configuration as it uses the same domain "-l / LOG4J" and the
loglevels integer representations are converted to the Log4j 1.x values
(excluding custom loglevels).
The recommended way to use this agent with Log4j 2.x is to add an
"Lttng" Appender with an arbiraty name and associate it with one or more
Logger using an AppenderRef.
For example, here is a basic log4j2 xml configuration that would send
all logging statements exlusively to an lttng appender:
More examples can be found in the 'doc/examples' directory.
The implementation of the appender is based on this[1] great guide by
Keith D. Gregory which is so much more detailed than the official
documentation, my thanks to him.
The pthread cancelstate disable performed to ensure threads are not
cancelled while holding mutexes which are used in library destructors
does not currently support that those mutexes may be nested. It
generates error messages when using the fork and fd helpers when running
with LTTNG_UST_DEBUG=1. The effect of this is that the pthread
cancelstate can be re-enabled too soon when the first unlock is
performed (in a nested lock scenario), thus allowing the thread to be
cancelled while still holding a lock, and causing a deadlock on
application exit.
Fix: abort on decrement_sem_count during concurrent tracing start and teardown
Observed issue
==============
The following backtrace has been reported:
#0 __GI_raise (sig=sig@entry=6)
at /usr/src/debug/glibc/2.31/git/sysdeps/unix/sysv/linux/raise.c:50
#1 0x0000007f90b3fdd4 in __GI_abort () at /usr/src/debug/glibc/2.31/git/stdlib/abort.c:79
#2 0x0000007f90b4bf50 in __assert_fail_base (fmt=0x7f90c3da98 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
assertion=assertion@entry=0x7f9112cb90 "uatomic_read(&sem_count) >= count",
file=file@entry=0x7f9112cb30 "/usr/src/debug/lttng-ust/2_2.13.0-r0/lttng-ust-2.13.0/src/lib/lttng-ust/lttng-ust-comm.c",
line=line@entry=664, function=function@entry=0x7f911317e8 <__PRETTY_FUNCTION__.10404> "decrement_sem_count")
at /usr/src/debug/glibc/2.31/git/assert/assert.c:92
#3 0x0000007f90b4bfb4 in __GI___assert_fail (assertion=assertion@entry=0x7f9112cb90 "uatomic_read(&sem_count) >= count",
file=file@entry=0x7f9112cb30 "/usr/src/debug/lttng-ust/2_2.13.0-r0/lttng-ust-2.13.0/src/lib/lttng-ust/lttng-ust-comm.c",
line=line@entry=664, function=function@entry=0x7f911317e8 <__PRETTY_FUNCTION__.10404> "decrement_sem_count")
at /usr/src/debug/glibc/2.31/git/assert/assert.c:101
#4 0x0000007f910e3830 in decrement_sem_count (count=<optimized out>)
at /usr/src/debug/lttng-ust/2_2.13.0-r0/lttng-ust-2.13.0/src/lib/lttng-ust/lttng-ust-comm.c:664
#5 0x0000007f910e5d28 in handle_pending_statedump (sock_info=0x7f9115c608 <global_apps>)
at /usr/src/debug/lttng-ust/2_2.13.0-r0/lttng-ust-2.13.0/src/lib/lttng-ust/lttng-ust-comm.c:737
#6 handle_message (lum=0x7f8dde46d8, sock=3, sock_info=0x7f9115c608 <global_apps>)
at /usr/src/debug/lttng-ust/2_2.13.0-r0/lttng-ust-2.13.0/src/lib/lttng-ust/lttng-ust-comm.c:1410
#7 ust_listener_thread (arg=0x7f9115c608 <global_apps>)
at /usr/src/debug/lttng-ust/2_2.13.0-r0/lttng-ust-2.13.0/src/lib/lttng-ust/lttng-ust-comm.c:2055
#8 0x0000007f90af73e0 in start_thread (arg=0x7fc27a82f6)
at /usr/src/debug/glibc/2.31/git/nptl/pthread_create.c:477
#9 0x0000007f90bead5c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
It turns out that the main thread is at that point iterating over the
libraries destructors:
Thread 3 (LWP 1983):
#0 0x0000007f92a68a0c in _dl_fixup (l=0x7f9054e510, reloc_arg=432)
at /usr/src/debug/glibc/2.31/git/elf/dl-runtime.c:69
#1 0x0000007f92a6ea3c in _dl_runtime_resolve () at ../sysdeps/aarch64/dl-trampoline.S:100
#2 0x0000007f905170f8 in __do_global_dtors_aux () from <....>/crash/work/rootfs/usr/lib/libbsd.so.0
#3 0x0000007f92a697f8 in _dl_fini () at /usr/src/debug/glibc/2.31/git/elf/dl-fini.c:138
#4 0x0000007f90b54864 in __run_exit_handlers (status=0, listp=0x7f90c65648 <__exit_funcs>,
run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true)
at /usr/src/debug/glibc/2.31/git/stdlib/exit.c:108
#5 0x0000007f90b549f4 in __GI_exit (status=<optimized out>)
at /usr/src/debug/glibc/2.31/git/stdlib/exit.c:139
#6 0x0000000000404c98 in a_function_name (....) at main.c:152
#7 0x0000000000404a98 in main (argc=3, argv=0x7fc27a8858, env=0x7fc27a8878) at main.c:97
Cause
=====
An enable command is processed at the same time that the lttng-ust
destructor is run. At the end of the command handling,
`handle_pending_statedump` is called. Multiple variables from the
`sock_info` struct are checked outside the UST lock at that point.
lttng-ust-comm.c +1406:
/*
* Performed delayed statedump operations outside of the UST
* lock. We need to take the dynamic loader lock before we take
* the UST lock internally within handle_pending_statedump().
*/
handle_pending_statedump(sock_info);
`statedump_pending` is set during the enable command
(`lttng_session_statedump`, lttng-events.c +631) in the same thread.
As for `registration_done` and `initial_statedump_done` they are invariant
from the registration of the app until `lttng_ust_cleanup` is called.
`cleanup_sock_info` called by `lttng_ust_cleanup`, itself called by
`lttng_ust_exit` resets the `registration_done` and
`initial_statedump_done` fields. Note that these operations are done
outside of the listener thread.
Note that by that point `lttng_ust_exit` expects all "getters" on
`sock_info` to fail while trying to acquire the UST lock due to
`lttng_ust_comm_should_quit` set to 1. Note that the listener threads
can still exist because we do not join them, we only execute
pthread_cancel which is async.
Clearly we are missing mutual exclusion provided by locking
when accessing `registration_done` and `initial_statedump_done`.
Solution
========
Here we can do better and simply not require any mutual exclusion based on locking.
`registration_done` and `initial_statedump_done` only need to be reset
to zero when we are not actually exiting (`lttng_ust_after_fork_child`).
In this case, no concurrent listener thread exists at that point
that could access those fields during the reset. Hence we can move the
reset to only the non-exiting code path and alleviate the current
situation.
Known drawbacks
===============
None.
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I45ba3eaee20c49a3988837a87fa680ce0a6ed953
Michael Jeanson [Tue, 9 Mar 2021 17:38:06 +0000 (12:38 -0500)]
fix: liblttng-ust-fd async-signal-safe close()
"close(2)" is documented as async-signal-safe (see signal-safety(7)) and
as such our override function should also be. This means we shouldn't do
lazy initialization of the function pointer to the original libc close
symbol in the override function.
If we move the initialization of the function pointer in the library
constructor we risk breaking other constructors that may run before ours
and call close().
A compromise is to explicitly do the initialization in the constructor
but keep a lazy init scheme if close() is called before it is executed.
The dlsym call is now done only once, if it fails the wrappers will
return -1 and set errno to ENOSYS.
Change-Id: I05c66d2f5d51b2022c6803ca215340fb9c00f5a8 Signed-off-by: Michael Jeanson <mjeanson@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Michael Jeanson [Wed, 10 Nov 2021 20:00:34 +0000 (15:00 -0500)]
fix: link benchmark with pthread
In commit aefa2e417b0d0a3c43cb4d078fa4a2d4cfbf429c we removed '-lpthread'
from the global link flags. Some toolchains don't require to explicitly
link with pthread but some do, add it to the benchmark specific link
flags.
Signed-off-by: Michael Jeanson <mjeanson@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I1a52d1f5363a86e17f82bcf56bd2273179c4328e
Fix: liblttng-ust-ctl have dependencies on liburcu
Observed issue
==============
On a yocto build the liblttng-ust-ctl shared object have dependencies on
liburcu.
(The 'not found's are irrelevant here)
./liblttng-ust-ctl.so: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ./liblttng-ust-ctl.so)
linux-vdso.so.1 (0x00007ffc07910000)
libpseudo.so => /root/oe-core/build/tmp-glibc/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so (0x00007f8fa3e0d000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f8fa3c05000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f8fa3a01000)
liburcu-bp.so.6 => not found
liburcu-cds.so.6 => not found
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f8fa37e2000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8fa33f1000)
/lib64/ld-linux-x86-64.so.2 (0x00007f8fa4069000)
The default behavior for AC_CHECK_LIB, when the `action-if-found` is NOT
defined, is to prepend the library to LIBS. [1]
"
If action-if-found is not specified, the default action prepends
-llibrary to LIBS and defines ‘HAVE_LIBlibrary’ (in all capitals).
"
It is important to note that the LIBS variable is used for ALL linking.
This is normally not a problem for most distributions since they force
the use of `--as-needed` at the toolchain level (gcc specs) (for example
debian [2]). One could also pass the `--as-needed` flag manually but
libtool reorganize flags in the case of shared object creation [3].
In our case, we normally explicitly state the dependencies via the *_LIBADD
automake clause. We do not rely on the LIBS variable.
For the current tree, when the configure script is run, the LIBS
variable has the following value:
The current configure.ac does define what seems to be `empty but
defined` clause for the `action-if-found` as "[]". This is not a valid
"empty but defined" `action-if-find` clause and end up generating the
default behavior as defined in [1]. "[ ]" would be but is supposedly not
the case for all autoconf version.
This leads to unnecessary dependencies for most of the shared objects, at
link time, generated when using a distro that do not enforce the
`--as-needed` flag on linking.
Solution
========
Define an actual no-op shell operation `:` for the `action-if-found`
parameter.
For libnuma define HAVE_LIBNUMA manually. This prevent the prepending to
LIBS while providing the HAVE_LIB* default behavior.
Testing with a fixed number of loops per-thread only works if the
workload is distributed perfectly across CPUs. For instance, if a lock
is held in the workload (e.g. internally by open() and close()), those
may cause starvation of some threads, and therefore cause the benchmark
to be wrong because it will wait for the slowest thread to complete its
loops.
It is also not good for testing overcommit of threads vs cpus.
Change the test to report the number of loops performed in a given wall
time, and use this to report the average and std.dev. of tracing
overhead per event on each active CPU.
Change the benchmark workload to be only CPU-bound and not generate
system calls to minimize the inherent non-scalability of the workload
(e.g. locks held within the kernel).
Kill session daemon after script, conditionally drop caches if root
(else it is forbidden to do so), handle cleanup on trap, use snapshot
(flight recorder) tracing to benchmark ring buffer, output the result in
nanoseconds, removing trailing numbers after dot.
Before this, the test was pretty much always resulting in an output of
"0s" extra overhead due to truncation of significant numbers in the
output.
Fix: bytecode linker: validate event and field array/sequence encoding
The bytecode linker should only allow linking filter expressions loading
fields which are string-encoded arrays and sequence for comparison
against a string, and reject arrays and sequences without encoding, so
the filter interpreter does not attempt to load non-NULL terminated
arrays/sequences as if they were strings.
Fix: uninitialized variable in lib_ring_buffer_channel_switch_timer_start
Found by Coverity:
** CID 1447027: Uninitialized variables (UNINIT)
/libringbuffer/ring_buffer_frontend.c: 810 in lib_ring_buffer_channel_switch_timer_start()
>>> CID 1447027: Uninitialized variables (UNINIT)
>>> Using uninitialized value "sev". Field "sev._sigev_un" is uninitialized when calling "timer_create".
Fix: Use unix socket peercred for pid, uid, gid credentials
Currently, the session daemon trust the pid, ppid, uid, and gid values
passed by the application, but should really validate the uid using unix
socket peercred. This fix uses the peercred values rather than the
values provided by the application on registration for:
- pid, uid and gid on Linux,
- uid and gid on FreeBSD.
This should improve how the session daemon deals with containerized
applications on Linux as well. Applications are required to be either in
the same pid namespace, or in a pid namespace nested within the pid
namespace of the lttng-sessiond, so the session daemon can map the
application pid to something meaningful within its own pid namespace.
Applications in a unrelated (disjoint) pid namespace will be refused by
the session daemon.
About the uid and gid with user namespaces on Linux, those will provide
meaningful IDs if the application user namespace is either the same as
the user namespace of the session daemon, or a nested user namespace.
Otherwise, the IDs will be that of /proc/sys/kernel/overflowuid and
/proc/sys/kernel/overflowgid, which typically maps to nobody.nogroup on
current distributions.
Given that fetching the parent pid (ppid) of the application would
require to use /proc/<pid>/status (which is racy wrt pid reuse), expose
the ppid provided by the application on registration instead, but only
in situations where the application sits in the same pid namespace as
the session daemon (on Linux), which is detected by checking if the pid
provided by the application matches the pid obtained using unix socket
credentials. The ppid is only used for logging/debugging purposes in the
session daemon anyway, so it is OK to use the value provided by the
application for that purpose.
If the command callback takes ownership of a pointer or file descriptor,
it sets them to NULL or -1. Therefore, the caller can always try to free
the pointer, or close it if it is greater or equal to 0.
If the command callback takes ownership of a pointer or file descriptor,
it sets them to NULL or -1. Therefore, the caller can always try to free
the pointer, or close it if it is greater or equal to 0.
Fix: Use default visibility for tracepoint provider symbol
When building a probe provider `someprobe` with -fvisibility=hidden into
a shared library, the `__tracepoint_provider_someprobe` symbol is hidden,
which does not allow tracepoint instrumentation to link to it.
Fix this by using the "default" visibility attribute.
For a shared library built with -fvisibility=hidden, this changes the
output of nm for that symbol from:
The newly-released autoconf 2.70 introduces a number of breaking
changes [1] and is being rolled-out by some distros.
Amongst those changes, the AC_PROG_CC_STDC macro is marked as obsolete
and was merged into AC_PROG_CC, which we already use. On 2.70, this
results in a warning which we handle as an error.
A version check is added to invoke the AC_PROG_CC_STDC macro only when
running a pre-2.70 version of autoconf, fixing the issue.
[1] https://lwn.net/Articles/839395/
[ Backport by Mathieu Desnoyers: removed ax_pthread.m4 bits which are
not present in stable-2.12 and prior. ]
Fix: ustctl_release_object: eliminate double-close/free on error
When ustctl_release_object returns an error, it is unclear to the caller
what close/free side effects were effectively performed and which were
not. So the only courses of action are to either leak file descriptors
or memory, or call ustctl_release_object again which can trigger double
close or double free.
Fix this by setting the file descriptors to -1 after successful close,
and pointers to NULL after successful free.
Michael Jeanson [Mon, 13 Jul 2020 19:22:35 +0000 (15:22 -0400)]
tests: return the proper TAP exit code
The C TAP library provides the 'exit_status()' function that will return
the proper exit code according to the number of tests that succeeded or
failed.
Signed-off-by: Michael Jeanson <mjeanson@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I786527dfa9cfe2d1a7c8bc80086d54186f60b4d9
Fix: libc-wrapper: undef temporary token rather than value
The lttng-ust malloc wrapper defines pthread_mutex_lock/unlock
preprocessor tokens to ust_malloc_spin_lock/unlock around the definition
of a TLS variable, which uses pthread mutexes when relying on the
pthread key fallback. Undefining those tokens should be done on the
preprocessor token, rather than its value.
Michael Jeanson [Fri, 26 Jun 2020 19:44:20 +0000 (15:44 -0400)]
Fix: support compile units including 'sys/sdt.h' without defining SDT_USE_VARIADIC
Instead of using SDT_USE_VARIADIC from 'sys/sdt.h', use our own namespaced
macros since the instrumented application might already have included
'sys/sdt.h' without variadic support.
Signed-off-by: Michael Jeanson <mjeanson@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Idfece1a65a5296a90d33370bc8d73ea554c14b0f
Background
==========
When userspace events with exclusions are enabled by the CLI user, the
session daemon enables the events in a 3-steps process using the
following ustctl commands:
1. `LTTNG_UST_EVENT` to create an event and get an handle on it,
2. `LTTNG_UST_EXCLUSION` to attach exclusions, and
3. `LTTNG_UST_ENABLE` to activate the tracing of this event.
Also, the session daemon uses the `LTTNG_UST_SESSION_START` to start the
tracing of events. In various use cases, this can happen before OR after
the 3-steps process above.
Scenario
========
If the`LTTNG_UST_SESSION_START` is done before the 3-steps process the
tracer will end up not considering the exclusions and trace all events
matching the event name glob pattern provided at step #1.
This is due to the fact that when we do step #1, we end up calling
`lttng_session_lazy_sync_enablers()`. This function will sync the event
enablers if the session is active (which it is in our scenario because
of the _SESSION_START).
Syncing the enablers will then match event name glob patterns provided
by the user to the event descriptors of the tracepoints and attach
probes in their corresponding callsite on matches.
All of this is done before we even received the exclusions in step #2.
Problem
=======
The problem is that we attach probes to tracepoints before being sure we
received the entire description of the events.
Step #2 may reduced the set of tracepoints to trace so we must wait
until exclusions are all received to attached probes to tracepoints.
Solution
========
Only match event names and exclusions (and ultimately, register probes)
on enabled enablers to ensure that no other modifications will be
applied to the event.
Event enablers are enabled when at step #3.
Note
====
Filters are not causing problems here because adding a filter won't
change what tracepoints are to be traced.
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Id984f266d976f346b001db81cd8a2b74965b5ef2
Michael Jeanson [Wed, 22 Apr 2020 20:32:51 +0000 (16:32 -0400)]
fix: Java examples CLASSPATH override
Variables provided to make as arguments, called 'command variables' can't
be reassigned a new value in the Makefile. Since the CLASSPATH is now
passed this way in the glue between automake and the example makefiles,
use different names for the internal variables.
Change-Id: Id6289273f211f544a66d933a96f06df75243415f Signed-off-by: Michael Jeanson <mjeanson@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
When reading from a TCP socket, there is no guarantee that the
read will return all the requested data. We need to loop and continue
reading until we gather all the expected data.
The namespace contexts introduced in lttng-ust 2.12 require to
initialize TLS variable to nonzero values. However, in !CONFIG_RCU_TLS
(compatibility mode using pthread setspecific), this initialization
does not build.
Use the new DEFINE_URCU_TLS_INIT from liburcu when building
!CONFIG_RCU_TLS to fix this issue. Since this requires a dependency on
a new liburcu version, only !CONFIG_RCU_TLS adds this dependency in the
fix. A followup cleanup patch will use DEFINE_URCU_TLS_INIT as we add
a strict version dependency.
The stream shm FDs are allocated by the consumer process, and then
passed to the applications over unix sockets. When opening those
file descriptors on reception, the FD_CLOEXEC flag is not set.
In a fork + exec scenario, parent process streams shm FDs and channel
wake FDs are present in the resulting child process.
Set FD_CLOEXEC on reception (ustcomm_recv_fds_unix_sock) to
prevent such scenario.
Change-Id: Id58077b272be9c1ab239846639ffd8103b3d50f1 Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Fix: tracepoint.h: Disable address sanitizer on pointer array section variables
The tracepoint header declares pointer global variables meant to be
placed contiguously within the __tracepoints_ptrs section, and then used
as an array of pointers when loading an executable or shared object.
Clang Address Sanitizer adds redzones around each variable, thus leading to
detection of a global buffer overflow.
Those redzones should not be placed within this section, because it
defeats its purpose. Therefore, teach asan not to add redzones
around those variables with an attribute.
Note that there does not appear to be any issue with gcc (tested with
gcc-8 with address sanitization enabled), and gcc ignores the
no_sanitize_address attribute when applied to a global variable.
jhash.h implements "special" code for valgrind because it reads memory
out-of-bound (and then applies a mask) when reading strings.
Considering that lttng-ust does not use jhash.h in a fast-path, remove
this "optimization" and use the verifiable VALGRIND code instead. This
fixes an ASan splat.
Fix: lttng-ust-comm.c: return number of fd rather size of array
There are two conflicting comments for this function. One says it
returns the size of the received data and the other says it returns the
number of fd received.
It's more useful to receive the number of fd.
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I74084b461d396c3e623fa55100e6dd7e59dbea83
Michael Jeanson [Thu, 16 Jan 2020 15:59:14 +0000 (10:59 -0500)]
Fix: build with -fno-common
GCC 10 will default to building with -fno-common, this inhibits the
linker from merging multiple tentative definitions of a symbol in an
archive. Keep only the declaration in the libustsnprintf.la convenience
library.
Signed-off-by: Michael Jeanson <mjeanson@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I8fb7c72811ce7e62f10342f55fcabeeabfdd4c67
I inadvertently inserted a character that looks like a space, but that
is not a space. make tries to interpret it as a target name, which
obviously fails.
Replace it with a proper space.
Signed-off-by: Simon Marchi <simon.marchi@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Simon Marchi [Mon, 18 Nov 2019 17:07:51 +0000 (12:07 -0500)]
doc: reformat long lines in doc/examples/Makefile.am
Format the long lines in the all-local target a bit like the "cmake"
target is formatted already. I think it helps readability to have one
argument per line instead of very long lines.
At the same time, I removed the "cd .." at the end of parentheses. The
parentheses start a new subshell, so it's unnecessary to do "cd .."
before the subshell exits.
Signed-off-by: Simon Marchi <simon.marchi@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Simon Marchi [Mon, 18 Nov 2019 17:07:50 +0000 (12:07 -0500)]
doc: pass AR when building examples
As reported here [1], when cross-compiling lttng-ust, the
"hello-static-lib" example uses the ar tool made for the --build machine
instead of the prefixed one, for the --host machine.
The Makefiles in the subdirectories of doc/examples are written by hand,
so that they can be easily copied and modified by users. They are
therefore not integrated in the automake build system, and any value
detected by configure must be passed explicitly when invoking it.
For example, the CC value is already explicitly passed, so that the
compiler value found by configure is passed down. We just need to do
the same for AR.
This patch adds AM_PROG_AR in configure.ac, so that configure finds the
prefixed version of ar, if cross-compiling.
It then sets the AR variable in doc/examples/Makefile.am, when invoking
sub-Makefiles. I don't think we really need it in the cmake case, but
it doesn't hurt to have it there.
Jonathan Rajotte [Fri, 28 Jun 2019 21:28:15 +0000 (17:28 -0400)]
Add procname to lttng_ust_statedump information
Adding the process procname to the statedump allows users to disable
procname context in scenario for which the data and serialization overhead
of the procname process is problematic. Users can stitch information in
post-processing based on other contexts (pid).
Users can skip this statedump via the
LTTNG_UST_WITHOUT_PROCNAME_STATEDUMP env variable.
Note that the procname statedump value is the procname seen at
application start (lttng_ust_init). Subsequent calls to pthread_setname_np
or equivalent have no effect on this value.
Since we cannot use the current thread name due to the
lttng_pthread_setname_np call in ust_listener_thread, we store the
process name inside the sock_info struct before the call to
lttng_pthread_setname_np. This data structure is already present as the
"owner" object in the statedump mechanism. During the statedump, we fetch
the procname from the "owner" object.
Use LTTNG_HIDDEN to reduce visibility of lttng_ust_sockinfo_get_procname.
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Fix: uninitialized variable in lib_ring_buffer_reserve_committed
This internal function implemented in libringbuffer is not used within
lttng-ust actually, but uses an uninitialized variable:
As reported by clang:
./frontend_internal.h:263:75: warning: variable 'idx' is uninitialized when used here [-Wuninitialized]
struct commit_counters_hot *cc_hot = shmp_index(handle, buf->commit_hot, idx);
^~~
./shm.h:74:86: note: expanded from macro 'shmp_index'
____ptr_ret = (__typeof__(____ptr_ret)) _shmp_offset((handle)->table, &(ref)._ref, index, sizeof(*____ptr_ret)); \
^~~~~
./frontend_internal.h:262:27: note: initialize the variable 'idx' to silence this warning
unsigned long offset, idx, commit_count;
^
= 0
In file included from ring_buffer_backend.c:29:
In file included from ./backend.h:33:
./frontend_internal.h:263:75: warning: variable 'idx' is uninitialized when used here [-Wuninitialized]
struct commit_counters_hot *cc_hot = shmp_index(handle, buf->commit_hot, idx);
^~~
./shm.h:74:86: note: expanded from macro 'shmp_index'
____ptr_ret = (__typeof__(____ptr_ret)) _shmp_offset((handle)->table, &(ref)._ref, index, sizeof(*____ptr_ret)); \
^~~~~
./frontend_internal.h:262:27: note: initialize the variable 'idx' to silence this warning
unsigned long offset, idx, commit_count;
^
= 0
Michael Jeanson [Tue, 12 Feb 2019 15:38:25 +0000 (10:38 -0500)]
Add userspace namespace contexts
Add a context for each available kernel namespace which currently are :
cgroup, ipc, mnt, net, pid, user and uts. The id chosen to identify the
namespaces is the inode number of the file representing each one of them
in the proc filesystem.
This was introduced in kernel v3.8.0, if any of these context are
enabled on a system running an older kernel, zero will be returned.
Signed-off-by: Michael Jeanson <mjeanson@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Introduce a new lttng_perf_lock to protect the lttng perf context
data structures from concurrent modifications and from fork. This
lock can be nested within the ust_lock, but never the opposite.
This removes the circular locking dependency involving urcu bp.
Fix: fd tracker: do not allow signal handlers to close lttng-ust FDs
Split the thread_fd_tracking state from the ust_fd_mutex_nest used to
track whether a signal handler is nested over a fd tracker lock.
lttng-ust listener threads need to invoke
lttng_ust_fd_tracker_register_thread() so the fd tracker can
distinguish them from application threads.
Otherwise, using ust_fd_mutex_nest to try to distinguish between
ust and application threads makes it possible for signal handlers
to appear as if they are ust listener threads, and thus attempt to
close UST file descriptors.
Fix: fd tracker: provide async-signal-safety for close wrapper
close(3) is part of the async-signal-safe functions. Therefore, it is
expected that the close wrapper provided by liblttng-ust-fd-tracker
behaves in a async-signal-safe way.
Use a similar strategy as ust_lock() does: disable signals when taking
and releasing the lock, and keep track of nesting with a TLS variable.
This ensures signals are restored to their original state when close(3)
ends up being invoked.
If fork() is performed while other threads are holding the fd tracker
lock, it will stay in locked state in the child process and eventually
cause a deadlock.
One way to solve this is to hold the fd tracker lock across fork(), in
the same way we do for the ust_lock. This ensures no other threads are
holding that lock in the parent, and therefore provides a consistent
lock state in the child.
Fix: wait for initial statedump before proceeding to the main program
In the case of short lived applications, the application may exit before
the initial statedump has completed.
Higher-level trace analysis features such as translating addresses to
symbols rely on statedump. That information is required for those
analyses to work on such short-lived applications.
Force the statedump to occur before handing the control to the
application.
Jonathan Rajotte [Mon, 29 Jul 2019 18:49:59 +0000 (14:49 -0400)]
Use MAP_POPULATE to reduce pagefault when available
Any ring buffer configuration bigger than PAGE_SIZE would result
in an increased latency for the first tracepoint hit (1200ns) landing on a
new PAGE_SIZE sized chunk of the mapped memory. This happens at least
for the first ring buffer traversal.
To alleviate this we can use MAP_POPULATE that will "prefault" the page
tables.
A similar flag seems to exist on freebsd (MAP_PREFAULT_READ) but I do
not have access to a system to test it and ensure it does indeed results
in the same effect. It mostly indicates that it prefaults for the
read case so I doubt it is the case.
Default to using MAP_POPULATE on Linux only for now. Support of
prefaulting on other platforms will be added as needed.
Commit 973eac638e4fd introduces an uninitialised value that may prevent
shared memory from being allocated. The compiler didn't give any warning
because the pointer to the value is sent to a function that don't do anything
with it. We simply pass NULL to that function.
-Waddress-of-packed-member, enabled by default, warns about an
unaligned pointer value from the address of a packed member of a
struct or union.
The warning is triggered in some place in LTTng-UST in cases where we
pass a pointer to get a result. Rather than passing the pointer directly
from the struct member, we get the result into a local storage, then
write into in the struct.
Michael Jeanson [Mon, 3 Jun 2019 19:25:32 +0000 (15:25 -0400)]
Fix: namespace our gettid wrapper
Since glibc 2.30, a gettid wrapper was added that conflicts with our
static declaration. Namespace our wrapper so there is no conflict,
we'll add support for the glibc provided wrapper in a further commit.
Signed-off-by: Michael Jeanson <mjeanson@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
projects / lttng-ust.git / log
