Jérémie Galarneau [Tue, 4 Apr 2023 15:03:03 +0000 (11:03 -0400)] 
Fix: sessiond: size-based notification occasionally not triggered
Issue observed
==============
When tracing to multiple sessions with scheduled size-based rotations
that occur simultaneously (typically because they trace the same events
and use the same rotation schedule configuration), the start of some
rotations seems to be delayed indefinitely.
Cause
=====
The size-based rotations are implemented by piggy-backing onto the
channel monitoring facilities.
Essentially, a per-channel timer samples a number of statistics on the
consumer daemon end, which transmits them to the session daemon.
The session daemon's notification subsystem evaluates the statistics
against the various registered triggers bound to the channels being
monitored when a statistics sample is received.
To implement size-based rotations, internal triggers are registered with
the "consumed size" condition set to a given threshold. A session
rotation management thread (which also performs other tasks) uses a
notification channel to wait for sessions to reach their target size,
starts rotations as needed, and sets a new threshold according to the
sessions' configured rotation schedule.
The rotation thread uses liblttng-ctl's API to consume notifications
from a notification channel.
At any time, a notification channel may have multiple notifications
queued-up internally in its buffers. This is because a notification
channel multiplexes command replies and notifications over the same UNIX
socket. The current protocol specifies that multiple notifications can
be received before the reply to a command.
In such cases, the notification channel client implementation internally
queues them and provides them on the next calls to
lttng_notification_channel_get_next_notification().
This is correct with respect to the public API, which is intended to be
used in "blocking mode". However, this internal user uses the
notification channel's raw file descriptor to wake-up when a
notification is available.
This is problematic because notifications may be queued by the
notification channel (and thus removed from the socket) while waiting
for command replies (subscribing and unsubscribing from notification
conditions). In such a case, a notification is available but the
rotation thread does not wake-up to consume it as nothing is available
in the socket's buffer.
When this happens, a session that is supposed to rotate automatically
appears to grow indefinitely. It will typically eventually rotate as new
notifications become available and cause the rotation thread to wake-up.
However, a "lag" builds up as the notification that caused the wake-up
is not consumed. Instead, the last buffered notification is provided to
the rotation thread.
Solution
========
Use an event_fd to wake-up the rotation thread whenever a command
completes on the notification channel. This ensures that any
notification that was queued while waiting for a reply to the command is
eventually consumed.
Known drawbacks
===============
None.
Note
====
The use of C++ features is kept to a minimum in this patch in order to
make it easier to backport to the stable releases. A clean-up patch
follows and makes the code conform to the coding standards.
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I8974b10124704d1e66e8da32d495fee738e3d43f
Olivier Dion [Wed, 22 Feb 2023 20:19:14 +0000 (15:19 -0500)] 
Fix: adding a user space probe fails on thumb functions
On some architectures, calling convention details are embedded in the
symbol addresses. Uprobe requires a "clean" symbol offset (or at least,
an address where an instruction boundary would be legal) to add
instrumentation. sanitize_uprobe_offset implements that sanitization
logic on a per-architecture basis.
The least significant bit is used when branching to switch to thumb ISA.
However, it's an invalid address for us; mask the least significant bit.
We were not masking the thumb bit, thus using the wrong address offset
by one.
Change-Id: Iaff8ccea3a319f9d9ad80501f1beccd74d1ef56d
Signed-off-by: Olivier Dion <odion@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Olivier Dion [Thu, 16 Feb 2023 20:34:26 +0000 (15:34 -0500)]
Fix: Tests that assume CPU 0 is present
Add util_event_generator.sh to shellcheck test while at it.
Change-Id: I261452496827a5b9fc08b39a1132b13a39d0a7f5
Signed-off-by: Olivier Dion <odion@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Olivier Dion [Tue, 14 Feb 2023 19:35:34 +0000 (14:35 -0500)]
Fix: Wrong assumption about possible CPUs
cpuset is not necessary contiguous. Thus, we need to parse
/sys/devices/system/cpu/posssible correctly.
Also, the `get_exposed_cpus_list' utility functions is required for
taskset-ting on the correct CPUs available to the test environment.
Change-Id: I062ce8d311ff0e8c4b757fe6f36387e3007cfa27
Signed-off-by: Olivier Dion <odion@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Olivier Dion [Fri, 10 Mar 2023 18:17:46 +0000 (13:17 -0500)]
Tests: fix: parse-callback reports missing addr2line
addr2line from binutils is required for this script to work correctly.
However, it silently fails. Fix this by using `subprocess.run' with
`check=True' instead of `subprocess.getoutput'. That way, an exception
is raised if an error occurs.
Fix the shebang by not assuming where python is installed while at it.
Change-Id: I5157b3dbccf6bfbe08a6b6840b38f5db9010fe96
Signed-off-by: Olivier Dion <odion@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Olivier Dion [Wed, 22 Feb 2023 21:45:27 +0000 (16:45 -0500)]
Fix: lttng remove-trigger -h fails
The error was that `argv' was incremented before passing it to
`argpar_iter_create'. However, the macro `SHOW_HELP' implicitly
deferences `argv[0]' to determine the command name.
Fix this by introducing a new variable `args' leaving `argv' untouched.
Change-Id: Id50fa2424550280a1e5e207429d643f5e8e00396
Signed-off-by: Olivier Dion <odion@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Jérémie Galarneau [Mon, 13 Feb 2023 20:43:18 +0000 (15:43 -0500)]
Tests: snapshot tests complain that nothing is output
The user space snapshot tests invoke a common script that runs the
actual tests. In doing so, the parent script's tap generator isn't
involved in the production of the tests' tap output and outputs an error
message as diagnostic.
ok 134 - Wait after kill session daemon
# Looks like your test died before it could output anything.
`exec`-ing the common script sidesteps the problem by replacing the
shell entirely.
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I53783fa783eb5581bf41fc1a5504dbd4e14fa595
Olivier Dion [Wed, 1 Feb 2023 22:11:02 +0000 (17:11 -0500)]
Tests: Test snapshot maximum size correctly
The minimum size for a channel is determined by:
sub-buffer-size * sub-buffer-count * possible-cpus
where sub-buffer-size is the system page size and sub-buffer-count is 2.
We set a snapshot with a maximum size of the minimum size. From there,
we need to spam that amount of events, assuming each event to be one
byte, on every online CPUs. We can then ensure that the total snapshot's
size is equal to the minimum size for a channel. However, there's a
little bias if the number of possible cores is greater than the number
of online cores. In that case, the bias is one sub-buffer for each extra
ring buffer.
Change-Id: I4718e134684463789b4f7be9b12c9bf3d6cfec20
Signed-off-by: Olivier Dion <odion@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Olivier Dion [Wed, 1 Feb 2023 21:04:14 +0000 (16:04 -0500)]
Tests: Add get_possible_cpus_count utility
lttng-ust uses the possible number of CPUs to allocate its ring buffers.
Certain tests have to take that into consideration in their calculation
instead of relying on online processors.
Thus, add the same logic for calculating the possible CPUs on the
system.
Change-Id: I9f14afba3e4adad9547cbd9386f8e1b1b55a3253
Signed-off-by: Olivier Dion <odion@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Jérémie Galarneau [Mon, 16 Jan 2023 19:19:20 +0000 (14:19 -0500)]
Fix: ini parser: truncation of value name
clang 14 reports the following:
ini-config/ini.cpp:88:16: warning: 'char* strncpy(char*, const char*, size_t)' output may be truncated copying 49 bytes from a string of length 199 [-Wstringop-truncation]
88 | strncpy(dest, src, size - 1);
| ~~~~~~~^~~~~~~~~~~~~~~~~~~~~
Indeed, a silent truncation of `name` occurs whenever it is longer than
prev_name (49 characters, excluding the terminator).
Report an error when this condition occurs.
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I973bd27185e0130d8e4a452525d9277de45ba200
Michael Jeanson [Thu, 23 Mar 2023 16:45:18 +0000 (12:45 -0400)] 
Fix: truncated len in lttng_event_rule_user_tracepoint_serialize()
Observed issue
==============
On 64-bit big-endian platforms, the serialization/deserialization tests
of tracepoint event rules fail since the length of individual exclusions
is truncated in lttng_event_rule_user_tracepoint_serialize and appear as
"0" on the receiving end.
Cause
=====
The length of the exclusion name string is stored in a variable of type
`size_t`. However, since the protocol expects it to be expressed as a
uint32_t, the value is added to the payload by copying the first 4 bytes
of the value.
On a 32-bit system this would be fine since `sizeof(size_t) == 4`. Even
worse, it would work most of the time (assuming an exclusion name string
< 4GiB) on a little-endian 64-bit system as the least significant bits
would be copied and correctly express the length of the string.
On a big-endian 64-bit platform, the most-significant 4 bytes are copied
to the payload buffer thus making the string length appear as "0".
Solution
========
A temporary variable is used to hold the "casted" value and make it safe
to copy to the payload buffer regardless of the platform's endianness.
Known drawbacks
===============
None.
Change-Id: I64c03345fff7ffea2f8fcb84692a085da31c421b
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Michael Jeanson [Wed, 22 Mar 2023 19:08:10 +0000 (15:08 -0400)]
Tests: remove leftover debug printing in test_add_trigger_cli
Change-Id: Ie5d0cc18ae35ff5848008fae5942d93eac5f2f70
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Jérémie Galarneau [Fri, 6 Jan 2023 03:13:30 +0000 (22:13 -0500)]
Update version to v2.13.9
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Michael Jeanson [Mon, 31 Oct 2022 19:52:46 +0000 (15:52 -0400)]
Fix: lttng: poptGetArg doesn't provide string ownership
The string returned by poptGetArg() is 'const' because it is owned by
the popt librairy and is free'd by it when poptFreeContext() is called.
Copy those strings when we need to alter them to maintain proper
ownership.
The latest release of popt (v1.19) introduced a breaking
change (changing the ownership of left-over command line arguments) that
can cause double free()-s.
This is ultimately due to this upstream commit in popt 1.19:
https://github.com/rpm-software-management/popt/commit/
7182e4618ad5a0186145fc2aa4a98c2229afdfa8
which is derived from a package patch:
https://src.fedoraproject.org/rpms/babeltrace/c/
d48452beff87b145c038f070e7182358db04336c?branch=rawhide
Change-Id: Id2535d1534c0e47cc0747968d6dd60a587f0b810
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Jérémie Galarneau [Fri, 2 Dec 2022 22:11:00 +0000 (17:11 -0500)]
Fix: relayd: missing space in trace creation logging statement
A missing space results in hard to read logging statements when a
ctf_trace is created:
[...] Created ctf_trace 15of session "hello" [...]
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I3881ddfc17556f2300f2140939c45d49e3b18d2b
Mathieu Desnoyers [Mon, 26 Sep 2022 17:36:38 +0000 (13:36 -0400)] 
Fix: event field value: assertion fails on empty string
When converting msgpack objects to their event_field_value equivalent,
the following assertion fails: LTTNG_ASSERT(val);
#4 0x00007f1f65349486 in __assert_fail () from /usr/lib/libc.so.6
#5 0x00007f1f65584da7 in lttng_event_field_value_string_create_with_size (val=0x0, size=0) at event-field-value.cpp:186
#6 0x00007f1f65576a1a in event_field_value_from_obj (obj=0x557f597ccdb8, field_val=0x7ffcc9675dd0)
at conditions/event-rule-matches.cpp:1120
#7 0x00007f1f65577176 in event_field_value_from_capture_payload (condition=0x557f597c8520,
capture_payload=0x557f597c825b "\221\240", capture_payload_size=2) at conditions/event-rule-matches.cpp:1340
#8 0x00007f1f655772ea in lttng_evaluation_event_rule_matches_create (condition=0x557f597c8520,
capture_payload=0x557f597c825b "\221\240", capture_payload_size=2, decode_capture_payload=true)
at conditions/event-rule-matches.cpp:1398
#9 0x00007f1f655765fc in lttng_evaluation_event_rule_matches_create_from_payload (condition=0x557f597c8520,
view=0x7ffcc9675ff0, _evaluation=0x7ffcc9676080) at conditions/event-rule-matches.cpp:990
#10 0x00007f1f6557f273 in lttng_evaluation_create_from_payload (condition=0x557f597c8520, src_view=0x7ffcc9676100,
evaluation=0x7ffcc9676080) at evaluation.cpp:120
#11 0x00007f1f6559ba36 in lttng_notification_create_from_payload (src_view=0x7ffcc9676190, notification=0x7ffcc9676180)
at notification.cpp:123
#12 0x00007f1f65552577 in create_notification_from_current_message (channel=0x557f597c8ee0) at channel.cpp:124
#13 0x00007f1f6555298c in lttng_notification_channel_get_next_notification (channel=0x557f597c8ee0, _notification=0x7ffcc9676280)
at channel.cpp:292
The msgpack API represents string as p-style while the implementation of
event_field_value relies on null-terminated strings. When an empty
string is captured by a tracer, it is decoded as a msgpack_object with
`str = {size = 0, ptr = 0x0}`.
lttng_event_field_value_string_create_with_size does not require a
null-terminated string since it also receives the length. Hence, this
fix causes lttng_event_field_value_string_create_with_size to accept
null strings when their length is zero. A copy of an empty string is
made to accomodate the null-termination convention used by the rest of
that API.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I7c3a839dbbeeb95a1b3bf6ddc3205a2f6b4538e3
Jérémie Galarneau [Mon, 24 Oct 2022 14:53:40 +0000 (10:53 -0400)]
Build fix: rpath of test libraries results in non-reproducible build
Yocto has trouble making LTTng-tools builds reproducibile since
abs_builddir is used as the rpath of the test libraries under
tests/regression/ust/ust-dl/.
From their commit message:
Specifing abs_builddir as an RPATH is plain wrong when cross
compiling. Sadly, removing the rpath makes libtool/automake do weird
things and breaks the build as shared libs are no longer generated.
We already try and delete the RPATH at do_install with chrpath however
that does leave the path in the string table so it doesn't help us
with reproducibility.
Their fix consists in hardcoding /usr/lib as the rpath of those
libraries. As mentionned, the rpath doesn't matter; it's used to
workaround libtool's behaviour.
This fix uses `$libdir`, which takes the user's specified prefix into
account to generate an rpath that is consistent with the one inserted in
the other artifacts.
Note that the change in the notification tests is a bit more involved
since we have to bypass libtool to instrument a test application with
uprobes.
Fixes #1361
Change-Id: I7739956f8bc8571ef90802c3b37a4e1f21352385
Reported-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Michael Jeanson [Tue, 18 Oct 2022 15:10:46 +0000 (11:10 -0400)]
Fix: never use 'no' in a command variable
Command variable may be used in the Makefiles and tests, when not found,
set them to an empty string instead of trying to execute the command
'no'.
Change-Id: I5429bb96599e3b7f166e52545269cd99eed758ae
Reported-by: Heng Guo <heng.guo@windriver.com>
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Jérémie Galarneau [Thu, 13 Oct 2022 13:28:59 +0000 (14:28 +0100)]
Fix: sessiond: abort called on undefined client command
Issue observed
--------------
When running in verbose mode, the session daemon calls abort()
when it receives an unknown client command:
#1 0x00007f66ffd69958 in raise () from /usr/lib/libc.so.6
#2 0x00007f66ffd5353d in abort () from /usr/lib/libc.so.6
#3 0x000055a671a6f6bd in lttcomm_sessiond_command_str (cmd=
1633771873)
at ../../../src/common/sessiond-comm/sessiond-comm.hpp:199
#4 0x000055a671a73897 in process_client_msg (cmd_ctx=0x7f66f5ff6d10,
sock=0x7f66f5ff6c34, sock_error=0x7f66f5ff6c38) at client.cpp:1006
#5 0x000055a671a777fc in thread_manage_clients (data=0x55a673956100)
at client.cpp:2622
#6 0x000055a671a6d290 in launch_thread (data=0x55a673956170) at thread.cpp:68
Cause
-----
process_client_msg() logs the client command on entry. While it
previously logged the numerical value, it now provides the string-ified
version of the command id (since
19f912db8).
The lttcomm_sessiond_command_str() function aborts when it encounters an
unknown command id. This is reasonable (in so far that it is how we
handle these situations, typically). However, the validity of the
command must be checked beforehand as it comes from an
external (untrusted) source.
Solution
--------
Add lttcomm_sessiond_command_is_valid and tombstone command IDs to
lttcomm_sessiond_command to ensure only valid command ids are passed
to lttcomm_sessiond_command_str when logging.
Drawbacks
---------
None
Reported-by: Michael Jeanson <mjeanson@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: Ibd36f1e69da984c7f27b55ec68e5e3fe06d7ac91
Jonathan Rajotte [Wed, 1 Jun 2022 19:20:55 +0000 (15:20 -0400)] 
Fix: consumer: snapshot: assertion on subsequent snapshot
Observed issue
==============
While a snapshot is being taken, the containing folder can disappear
unexpectedly. This can lead to the following errors, which are expected
and mostly handled fine:
PERROR - 14:47:32.
002564464 [
2922498/
2922507]: Failed to open file relative to trace chunk file_path = "channel0_0", flags = 577, mode = 432: No such file or directory (in _lttng_trace_chunk_open_fs_handle_locked() at trace-chunk.cpp:1411)
Error: Failed to open stream file "channel0_0"
Error: Snapshot channel failed
The problem happens on the subsequent snapshot for the session:
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007fbbdadb3859 in __GI_abort () at abort.c:79
#2 0x00007fbbdadb3729 in __assert_fail_base (fmt=0x7fbbdaf49588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x55c4212cfbb5 "!stream->trace_chunk", file=0x55c4212cf820 "kernel-co
#3 0x00007fbbdadc5006 in __GI___assert_fail (assertion=0x55c4212cfbb5 "!stream->trace_chunk", file=0x55c4212cf820 "kernel-consumer/kernel-consumer.cpp", line=188, function=0x55c4212cfb00 "
#4 0x000055c421268cc6 in lttng_kconsumer_snapshot_channel (channel=0x7fbbc4000b60, key=1, path=0x7fbbd37f8fd4 "", relayd_id=
18446744073709551615, nb_packets_per_stream=0) at kernel-consume
#5 0x000055c42126b39d in lttng_kconsumer_recv_cmd (ctx=0x55c421b80a90, sock=31, consumer_sockpoll=0x7fbbd37fd280) at kernel-consumer/kernel-consumer.cpp:986
#6 0x000055c4212546d1 in lttng_consumer_recv_cmd (ctx=0x55c421b80a90, sock=31, consumer_sockpoll=0x7fbbd37fd280) at consumer/consumer.cpp:2090
#7 0x000055c421259963 in consumer_thread_sessiond_poll (data=0x55c421b80a90) at consumer/consumer.cpp:3281
#8 0x00007fbbdaf8b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#9 0x00007fbbdaeb0163 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
How to reproduce:
1. Setting a breakpoint on snapshot_channel() inside
src/common/ust-consumer/ust-consumer.cpp
2. When the breakpoint hits, remove the the complete lttng directory
containing the session data.
3. Continue the lttng_consumerd process from gdb.
4. In that case you see a negative return value -1 from
consumer_stream_create_output_files() inside snapshot_channel().
5. Take another snapshot and lttng_consumerd crashes because
of the `assert(!stream->trace_chunk)` in snapshot_channel().
This last action does not require any breakpoint intervention.
Cause
=====
During the snapshot, the stream is assigned the channel current chunk.
It is expected that the stream does not have a chunk at this point.
The error handling is faulty here, the stream chunk must be
invalidated/reset on error to allow its reuse later on.
The problem exists for both consumer domains (user/kernel).
Solution
========
For the ust consumer, we can directly use the `error_close_stream`
label.
For the kernel consumer, the code path is slightly different since it
does not uses `consumer_stream_close`. Note that `consumer_stream_close`
cannot be used as is for the kernel consumer. The current implementation
partially resembles `consumer_stream_close` at the end of the iteration.
It is extracted to its own function for easier reuse from the new
`error_finalize_stream` label.
Known drawbacks
=========
None.
Fixes: #1352
Signed-off-by: Marcel Hamer <marcel.hamer@windriver.com>
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I9fc81917b19aa436ed8e8679672648f2d5baf41a
Jérémie Galarneau [Tue, 3 Jan 2023 23:41:23 +0000 (18:41 -0500)]
Fix: sessiond: instance uuid is not sufficiently unique
Observed issue
==============
Tracing a cluster of machines -- all launched simultaneously -- to the
same relay daemon occasionally produces corrupted traces.
The size of packets received (as seen from the relay daemon logs) and
that of those present in the on-disk stream occasionally didn't match.
The traces were observed to all relate to the same trace UUID, but
present packet begin timestamps that were not monotonic for a given
stream.
This causes both Babeltrace 1.x and 2.x to fail to open the traces with
different error messages related to clocks.
Cause
=====
On start, the session daemon generates a UUID to uniquely identify the
sessiond instance. Since the UUID generation utils use time() to seed
the random number generator, two session daemons launched within the
same second can end up with the same instance UUID.
Since the relay daemon relies on this UUID to uniquely identify a
session daemon accross its various connections, identifier clashes can
cause streams from the same `uid` or `pid` to become scrambled resulting
in corrupted traces.
Solution
========
The UUID utils now initializes its random seed using the getrandom() API
in non-blocking mode. If that fails -- most likely because the random
pool is depleted or the syscall is not available on the platform -- it
falls back to using a hash of two time readings (with nanosecond
precision), of the hostname, and the PID.
Known drawbacks
===============
This fix implements many fallbacks, each with their own caveats and we
don't have full test coverage for all of those for the moment.
This article presents the different drawbacks of using /dev/urandom vs
getrandom().
https://lwn.net/Articles/884875/
As for the pseudo-random time and configuration based fallback, it is
meant as a last resort for platforms or configurations where both
getrandom() (old kernels or non-Linux platforms) and /dev/urandom (e.g.
locked-down container) are not be available. I haven't done a formal
analysis of the entropy of this home-grown method. The practical
use-case we want to enable is launching multiple virtual machines (or
containers) at roughly the same time and ensure that they don't end up
using the same sessiond UUID.
In that respect, having a different host name and minute timing changes
seem enough to prevent a UUID clash.
Using the PID as part of the hash also helps when launching multiple
session daemons simultaneously for different users.
Change-Id: I320fff7bc52752ff504643569e49fa3c02472ec2
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Jérémie Galarneau [Mon, 22 Aug 2022 18:45:38 +0000 (14:45 -0400)]
Update version to v2.13.8
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Mathieu Desnoyers [Thu, 23 Jun 2022 20:27:41 +0000 (16:27 -0400)]
Fix: waiter: futex wait: handle spurious futex wakeups
Observed issue
==============
The waiter lttng_waiter_wait() implements a futex wait/wakeup
scheme similar to the liburcu workqueue code, which has an issue with
spurious wakeups.
A spurious wakeup on lttng_waiter_wait can cause
lttng_waiter_wait to reach label skip_futex_wait with a
waiter->state state of WAITER_WAITING, which is unexpected. It would
cause busy-waiting on WAITER_TEARDOWN state to start early. The
wait-teardown stage is done with WAIT_ATTEMPTS active attempts,
following by attempts spaced by 10ms sleeps. I do not expect that these
spurious wakeups will cause user-observable effects other than being
slightly less efficient that it should be.
This issue will cause spurious unexpected high CPU use, but will not
lead to data corruption.
Cause
=====
From futex(5):
FUTEX_WAIT
Returns 0 if the caller was woken up. Note that a wake-up can
also be caused by common futex usage patterns in unrelated code
that happened to have previously used the futex word's memory
location (e.g., typical futex-based implementations of Pthreads
mutexes can cause this under some conditions). Therefore, call‐
ers should always conservatively assume that a return value of 0
can mean a spurious wake-up, and use the futex word's value
(i.e., the user-space synchronization scheme) to decide whether
to continue to block or not.
Solution
========
We therefore need to validate whether the value differs from
WAITER_WAITING in user-space after the call to FUTEX_WAIT returns 0.
Known drawbacks
===============
None.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: Ida9905d1f0b5d9543c8b85ecbd7d748a6f7c1c97
Mathieu Desnoyers [Thu, 23 Jun 2022 20:15:51 +0000 (16:15 -0400)]
Fix: futex wait: handle spurious futex wakeups
Observed issue
==============
The futex futex_nto1_wait() implements a futex wait/wakeup scheme
identical to the liburcu workqueue code, which has an issue with
spurious wakeups.
A spurious wakeup on futex_nto1_wait can cause futex_nto1_wait to return
with a futex state of -1, which is unexpected.
futex_nto1_wait is used by the relayd live dispatcher thread, by the
relayd main dispatcher thread, as well as by the sessiond dispatcher
thread.
Given that following a futex_nto1_wait returning due to a spurious
wakeup futex_nto1_prepare will set the futex value to -1, things go
back to normal for the following futex_nto1_wait calls.
Therefore, the only impact of this issue is to spuriously use slightly
more CPU time than strictly required.
The effect is even shorter-lasting that in the liburcu counterparts
because futex_nto1_prepare explicitly sets the futex state to -1 rather
than use an atomic decrement, which immediately sets to state back to
a consistent state.
Cause
=====
From futex(5):
FUTEX_WAIT
Returns 0 if the caller was woken up. Note that a wake-up can
also be caused by common futex usage patterns in unrelated code
that happened to have previously used the futex word's memory
location (e.g., typical futex-based implementations of Pthreads
mutexes can cause this under some conditions). Therefore, call‐
ers should always conservatively assume that a return value of 0
can mean a spurious wake-up, and use the futex word's value
(i.e., the user-space synchronization scheme) to decide whether
to continue to block or not.
Solution
========
We therefore need to validate whether the value differs from -1 in
user-space after the call to FUTEX_WAIT returns 0.
Known drawbacks
===============
None.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I8a1b6aaf77b6a2f941fd4f89b61bed71cf17906b
Jérémie Galarneau [Mon, 22 Aug 2022 15:25:21 +0000 (11:25 -0400)]
Fix: utils: unhandled close return value
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I3f932ac53e6b39c86babe726d5ec8e9ba999e561
Jonathan Rajotte [Thu, 21 Jul 2022 13:30:27 +0000 (09:30 -0400)]
Fix: agent port file is o+w when launching as root
Observed issue
==============
When starting as root, the following permissions are observed:
[-rw-rw-rw-] agent.port
[-rw-r--r--] lttng-sessiond.pid
When starting as user:
[-rw-rw----] agent.port
[-rw-rw-r--] lttng-sessiond.pid
Note that despite being created by the same function,
`utils_create_pid_file`, the permissions are not the same.
Cause
=====
`get_wait_shm` manipulates the umask and does not restore it, thus
influencing the outcome of following file creations that don't enforce
specific permissions (using chmod).
Also `fopen` defaults to mode `0666 & ~umask`, thus resulting in
unnecessarily lax permissions when the session daemon is started as a
non-privileged user (umask = 0002, most of the time).
Solution
========
Mimic other call sites of umask(), modify then revert the umask.
Open the pid and agent port files as 0644 letting the umask to do its
job as necessary for those files.
Remove unnecessary umask() usage when chmod is directly used.
Known drawbacks
===============
Use of umask in a multi-threaded process is not recommended. Still our
current usage is limited and mostly happens during the initialization
phase. The usage of umask() is required for the `wait_shm` since on
FreeBSD it is not possible to chmod an shm file descriptor. The default
umask would interfere here.
Discussion
==========
The usage in run-as is valid even when in no-clone mode (valgrind) since
it is the sole user of umask() following the initialization phase. When
spawned as a separate process the clearing of umask is totally valid
even if it is not ideal since we are ignoring any umask set by the user.
It seems like the current usage is the lesser evil here.
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: Ie224d254714fff05f4bced471ebfa8f19eede26a
Michael Jeanson [Tue, 9 Aug 2022 15:38:16 +0000 (15:38 +0000)]
Fix: tests: don't assume sequential cpuids
On Linux CPU ids aren't sequential if a CPU is offlined or unplugged.
Get the list of currently available CPU ids from sysfs and pick a random
one, if sysfs is not available use the previous behavior.
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: Ibdb63c7d036389104ac2f629827a6dce59e06983
Jérémie Galarneau [Wed, 6 Jul 2022 16:09:41 +0000 (12:09 -0400)]
Fix: sessiond: handle empty scheduled rotations
A number of error codes were added to cmd_rotate_session since the
implementation of size-based rotations. The rotation thread doesn't
expect LTTNG_ERR_ROTATION_MULTIPLE_AFTER_STOP and
LTTNG_ERR_ROTATION_AFTER_STOP_CLEAR which are not fatal failures.
These rotations would simply result in an empty trace archive and are,
therefore, not produced. In both cases, it is safe to wait for the next
size cycle.
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: Ib90526b586de17c0b14d16970c862d9f981ed464
Jérémie Galarneau [Thu, 30 Jun 2022 16:03:23 +0000 (12:03 -0400)]
Fix: sessiond: report client list allocation failure as a fatal error
Report the failure to allocate a notification client list as a fatal
notification thread error.
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I8f2654020d0d890cc9275d445fdeccde940d2ae0
Jérémie Galarneau [Tue, 21 Jun 2022 20:21:17 +0000 (16:21 -0400)]
Fix: ust-consumerd: set `hangup_flush_done` in a locked context
hangup_flush_done is updated after releasing the stream lock. This
doesn't appear to be a problem right now since this attribute is
apparently always accessed by the same thread, but it is conceptually
sus.
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I191cb01b02c3d96e19034e0d0e80cb7f8dff2140
Jérémie Galarneau [Fri, 17 Jun 2022 20:53:53 +0000 (16:53 -0400)]
Fix: sessiond: size-based rotations never trigger
Issue observed
==============
Size-based scheduled rotations have no effect.
Cause
=====
Since
c08136a3f, the rotation thread's handle_condition() checks that
the notification received matches the trigger that was registered.
As part of the equality check, the triggers' credentials are compared.
This checks fails systematically since the group id of a trigger's
credentials is not transported by the serialize/create_from functions.
The trigger that is received through the notification thus has an unset
group id, while the rotation trigger of the `ltt_session` has a group id
set; it was not stripped by the communication layer.
The check also fails since the trigger registered for the size-based
rotation is "hidden". This internal attribute is not propagated through
the communication layer, which causes the comparison to fail.
Solution
========
Since triggers only use the 'uid' part of lttng_credentials, we ensure
that lttng_trigger_set_credentials only sets this part of the structure.
Also, the `is_hidden` attribute of a trigger is now propagated through
the communication layer. This has no effect for external applications
since this attribute is not exposed through the API. However, it is
useful for internal triggers which use the same communication
facilities.
This allows the equality check in rotation-thread.cpp to go through as
expected.
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I216f5cb9297ecd1a867dc292c10b8da595efce34
Jérémie Galarneau [Tue, 17 May 2022 17:41:49 +0000 (13:41 -0400)]
Fix: lttng-snapshot: use after free of max size argument
gcc 12.1.0 reports:
commands/snapshot.cpp: In function ‘int cmd_snapshot(int, const char**)’:
../../../src/common/error.hpp:139:32: error: pointer ‘max_size_arg’ may be used after ‘void free(void*)’ [-Werror=use-after-free]
free max_size_arg on both paths.
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I3775e835e10b364f32f4797afb9c090ac4dc133c
Jérémie Galarneau [Fri, 15 Apr 2022 06:09:53 +0000 (02:09 -0400)]
Fix: lttng: snapshot: add-output: leak of max size parameter
==
1920281==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 6 byte(s) in 1 object(s) allocated from:
#0 0x7fa95633add9 in __interceptor_malloc /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:145
#1 0x7fa955e90c09 (/usr/lib/libpopt.so.0+0x3c09)
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I87ce90a77d9624add0cab5d3090a7e83734da7f4
Jérémie Galarneau [Fri, 15 Apr 2022 05:55:45 +0000 (01:55 -0400)]
Tests: fix: lttng-create: leaked command parameter
==
1853705==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 3 byte(s) in 1 object(s) allocated from:
#0 0x7fb67ee0edd9 in __interceptor_malloc /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:145
#1 0x7fb67e964c09 (/usr/lib/libpopt.so.0+0x3c09)
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I2f3346157cb26de6712c6e6ebd5fafa6b51fac08
Jérémie Galarneau [Fri, 15 Apr 2022 05:30:50 +0000 (01:30 -0400)]
Fix: sessiond: rotation trigger leak
==
1801304==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 224 byte(s) in 2 object(s) allocated from:
#0 0x7fe0f4e73fb9 in __interceptor_calloc /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:154
#1 0x559fbeb64175 in zmalloc_internal ../../src/common/macros.hpp:60
#2 0x559fbeb6a291 in lttng_trigger* zmalloc<lttng_trigger>() ../../src/common/macros.hpp:89
#3 0x559fbeb64aa6 in lttng_trigger_create /home/jgalar/EfficiOS/src/lttng-tools/src/common/trigger.cpp:58
#4 0x559fbe9dc417 in subscribe_session_consumed_size_rotation(ltt_session*, unsigned long, notification_thread_handle*) /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/rotate.cpp:87
#5 0x559fbe995d6f in cmd_rotation_set_schedule(ltt_session*, bool, lttng_rotation_schedule_type, unsigned long, notification_thread_handle*) /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/cmd.cpp:5993
#6 0x559fbe9fe559 in process_client_msg /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/client.cpp:2246
#7 0x559fbea01378 in thread_manage_clients /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/client.cpp:2624
#8 0x559fbe9ea642 in launch_thread /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/thread.cpp:68
#9 0x7fe0f44935c1 in start_thread (/usr/lib/libc.so.6+0x8d5c1)
Indirect leak of 208 byte(s) in 2 object(s) allocated from:
#0 0x7fe0f4e73fb9 in __interceptor_calloc /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:154
#1 0x559fbeb16e21 in zmalloc_internal ../../src/common/macros.hpp:60
#2 0x559fbeb16e31 in lttng_action_notify* zmalloc<lttng_action_notify>() ../../src/common/macros.hpp:89
#3 0x559fbeb168a0 in lttng_action_notify_create actions/notify.cpp:135
#4 0x559fbe9dc34b in subscribe_session_consumed_size_rotation(ltt_session*, unsigned long, notification_thread_handle*) /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/rotate.cpp:80
#5 0x559fbe995d6f in cmd_rotation_set_schedule(ltt_session*, bool, lttng_rotation_schedule_type, unsigned long, notification_thread_handle*) /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/cmd.cpp:5993
#6 0x559fbe9fe559 in process_client_msg /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/client.cpp:2246
#7 0x559fbea01378 in thread_manage_clients /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/client.cpp:2624
#8 0x559fbe9ea642 in launch_thread /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/thread.cpp:68
#9 0x7fe0f44935c1 in start_thread (/usr/lib/libc.so.6+0x8d5c1)
Indirect leak of 160 byte(s) in 2 object(s) allocated from:
#0 0x7fe0f4e73fb9 in __interceptor_calloc /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:154
#1 0x559fbeb3d7a1 in zmalloc_internal ../../src/common/macros.hpp:60
#2 0x559fbeb3fa35 in lttng_condition_session_consumed_size* zmalloc<lttng_condition_session_consumed_size>() ../../src/common/macros.hpp:89
#3 0x559fbeb3e6fd in lttng_condition_session_consumed_size_create conditions/session-consumed-size.cpp:206
#4 0x559fbe9dc0f1 in subscribe_session_consumed_size_rotation(ltt_session*, unsigned long, notification_thread_handle*) /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/rotate.cpp:54
#5 0x559fbe995d6f in cmd_rotation_set_schedule(ltt_session*, bool, lttng_rotation_schedule_type, unsigned long, notification_thread_handle*) /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/cmd.cpp:5993
#6 0x559fbe9fe559 in process_client_msg /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/client.cpp:2246
#7 0x559fbea01378 in thread_manage_clients /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/client.cpp:2624
#8 0x559fbe9ea642 in launch_thread /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/thread.cpp:68
#9 0x7fe0f44935c1 in start_thread (/usr/lib/libc.so.6+0x8d5c1)
Indirect leak of 112 byte(s) in 2 object(s) allocated from:
#0 0x7fe0f4e73fb9 in __interceptor_calloc /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:154
#1 0x559fbeb242ad in zmalloc_internal ../../src/common/macros.hpp:60
#2 0x559fbeb27062 in zmalloc<(anonymous namespace)::lttng_rate_policy_every_n> ../../src/common/macros.hpp:89
#3 0x559fbeb25e9f in lttng_rate_policy_every_n_create actions/rate-policy.cpp:492
#4 0x559fbeb168b9 in lttng_action_notify_create actions/notify.cpp:141
#5 0x559fbe9dc34b in subscribe_session_consumed_size_rotation(ltt_session*, unsigned long, notification_thread_handle*) /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/rotate.cpp:80
#6 0x559fbe995d6f in cmd_rotation_set_schedule(ltt_session*, bool, lttng_rotation_schedule_type, unsigned long, notification_thread_handle*) /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/cmd.cpp:5993
#7 0x559fbe9fe559 in process_client_msg /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/client.cpp:2246
#8 0x559fbea01378 in thread_manage_clients /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/client.cpp:2624
#9 0x559fbe9ea642 in launch_thread /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/thread.cpp:68
#10 0x7fe0f44935c1 in start_thread (/usr/lib/libc.so.6+0x8d5c1)
Indirect leak of 34 byte(s) in 2 object(s) allocated from:
#0 0x7fe0f4e19319 in __interceptor_strdup /usr/src/debug/gcc/libsanitizer/asan/asan_interceptors.cpp:454
#1 0x559fbeb3f603 in lttng_condition_session_consumed_size_set_session_name conditions/session-consumed-size.cpp:442
#2 0x559fbe9dc2c4 in subscribe_session_consumed_size_rotation(ltt_session*, unsigned long, notification_thread_handle*) /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/rotate.cpp:71
#3 0x559fbe995d6f in cmd_rotation_set_schedule(ltt_session*, bool, lttng_rotation_schedule_type, unsigned long, notification_thread_handle*) /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/cmd.cpp:5993
#4 0x559fbe9fe559 in process_client_msg /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/client.cpp:2246
#5 0x559fbea01378 in thread_manage_clients /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/client.cpp:2624
#6 0x559fbe9ea642 in launch_thread /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/thread.cpp:68
#7 0x7fe0f44935c1 in start_thread (/usr/lib/libc.so.6+0x8d5c1)
The rotation trigger of a session (used for size-based rotations) is
never cleaned-up. It is now cleaned up every time its condition is
hit and whenever the session is destroyed.
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I5a89341535f87b7851b548ded9838c18bd1ccb95
Jérémie Galarneau [Fri, 15 Apr 2022 05:34:54 +0000 (01:34 -0400)]
Tests: fix: schedule api: leak of rotation schedule list
==
1769573==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 24 byte(s) in 1 object(s) allocated from:
#0 0x7fef37a29fb9 in __interceptor_calloc /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:154
#1 0x7fef37792f2f in zmalloc_internal ../../../src/common/macros.hpp:60
#2 0x7fef3779573a in lttng_rotation_schedules* zmalloc<lttng_rotation_schedules>() ../../../src/common/macros.hpp:89
#3 0x7fef377947cc in lttng_rotation_schedules_create /home/jgalar/EfficiOS/src/lttng-tools/src/lib/lttng-ctl/rotate.cpp:353
#4 0x7fef37794aa0 in get_schedules /home/jgalar/EfficiOS/src/lttng-tools/src/lib/lttng-ctl/rotate.cpp:392
#5 0x7fef377956dc in lttng_session_list_rotation_schedules /home/jgalar/EfficiOS/src/lttng-tools/src/lib/lttng-ctl/rotate.cpp:665
#6 0x5646131713f2 in test_add_list_remove_schedule /home/jgalar/EfficiOS/src/lttng-tools/tests/regression/tools/rotation/schedule_api.c:252
#7 0x56461317157b in test_add_list_remove_size_schedule /home/jgalar/EfficiOS/src/lttng-tools/tests/regression/tools/rotation/schedule_api.c:270
#8 0x564613171680 in main /home/jgalar/EfficiOS/src/lttng-tools/tests/regression/tools/rotation/schedule_api.c:307
#9 0x7fef373ae30f in __libc_start_call_main (/usr/lib/libc.so.6+0x2d30f)
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I9b7eb537d158791db76f9a7676ffeb5d4a1f2203
Jérémie Galarneau [Fri, 15 Apr 2022 05:29:46 +0000 (01:29 -0400)]
Fix: lttng: enable-rotation: leak of command parameter
==
1759491==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 6 byte(s) in 1 object(s) allocated from:
#0 0x7fdbdc94add9 in __interceptor_malloc /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:145
#1 0x7fdbdc4a0c09 (/usr/lib/libpopt.so.0+0x3c09)
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I29cc6ec4390e71829107f309f162247b9be2868c
Jérémie Galarneau [Fri, 15 Apr 2022 04:35:35 +0000 (00:35 -0400)]
Fix: lttng: track: leaked command parameter
==
1676099==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 5 byte(s) in 1 object(s) allocated from:
#0 0x7f19429d9dd9 in __interceptor_malloc /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:145
#1 0x7f19425342ad in poptGetNextOpt (/usr/lib/libpopt.so.0+0x82ad)
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: Ibafcaf42ad4f842b3fa74cf91dc5ecc8acb3487d
Jérémie Galarneau [Fri, 15 Apr 2022 03:43:10 +0000 (23:43 -0400)]
Fix: lttng: add-trigger: leak of parser context on capture
==
1501334==ERROR: LeakSanitizer: detected memory leaks
Indirect leak of 16386 byte(s) in 1 object(s) allocated from:
#0 0x7f95efc3cdd9 in __interceptor_malloc /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:145
#1 0x55acb0681ed3 in lttng_filter_yyalloc(unsigned long, void*) filter/filter-lexer.cpp:2511
#2 0x55acb067f2f2 in lttng_filter_yy_create_buffer(_IO_FILE*, int, void*) filter/filter-lexer.cpp:1895
#3 0x55acb067ea44 in yyrestart(_IO_FILE*, void*) filter/filter-lexer.cpp:1824
#4 0x55acb0649a43 in filter_parser_ctx_alloc(_IO_FILE*) filter/filter-parser.ypp:271
#5 0x55acb0649e7f in filter_parser_ctx_create_from_filter_expression(char const*, filter_parser_ctx**) filter/filter-parser.ypp:332
#6 0x55acb058ee89 in parse_event_rule commands/add_trigger.cpp:783
#7 0x55acb05920c0 in handle_condition_event commands/add_trigger.cpp:1361
#8 0x55acb0592739 in parse_condition commands/add_trigger.cpp:1457
#9 0x55acb0596b56 in cmd_add_trigger(int, char const**) commands/add_trigger.cpp:2304
#10 0x55acb05a5b80 in handle_command /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng/lttng.cpp:238
#11 0x55acb05a6643 in parse_args /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng/lttng.cpp:427
#12 0x55acb05a694a in main /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng/lttng.cpp:476
#13 0x7f95ef28730f in __libc_start_call_main (/usr/lib/libc.so.6+0x2d30f)
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I6fa21e7d066e0cf48afc3f91ceefbfd19c6b86fd
Jérémie Galarneau [Fri, 15 Apr 2022 03:26:12 +0000 (23:26 -0400)]
Tests: fix: leak of trigger in trigger listing tests
==
1480456==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 112 byte(s) in 1 object(s) allocated from:
#0 0x7fdb9260cfb9 in __interceptor_calloc /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:154
#1 0x7fdb9242348d in zmalloc_internal ../../src/common/macros.hpp:60
#2 0x7fdb924295a9 in lttng_trigger* zmalloc<lttng_trigger>() ../../src/common/macros.hpp:89
#3 0x7fdb92423dbe in lttng_trigger_create /home/jgalar/EfficiOS/src/lttng-tools/src/common/trigger.cpp:58
#4 0x56304832331f in register_trigger /home/jgalar/EfficiOS/src/lttng-tools/tests/regression/tools/trigger/utils/register-some-triggers.cpp:24
#5 0x5630483233f1 in register_trigger_action_list_notify /home/jgalar/EfficiOS/src/lttng-tools/tests/regression/tools/trigger/utils/register-some-triggers.cpp:46
#6 0x5630483239a0 in test_session_rotation_conditions /home/jgalar/EfficiOS/src/lttng-tools/tests/regression/tools/trigger/utils/register-some-triggers.cpp:246
#7 0x563048323d4d in main /home/jgalar/EfficiOS/src/lttng-tools/tests/regression/tools/trigger/utils/register-some-triggers.cpp:309
#8 0x7fdb91c6630f in __libc_start_call_main (/usr/lib/libc.so.6+0x2d30f)
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: Ie163989a70f65f9c2c4e93c36cc9fc6ba6bdeeb5
Jérémie Galarneau [Fri, 15 Apr 2022 03:21:27 +0000 (23:21 -0400)]
Fix: action error query: leak of action path
==
1429021==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 8 byte(s) in 1 object(s) allocated from:
#0 0x7fe305f031b2 in __interceptor_realloc /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:164
#1 0x559f1b022238 in lttng_dynamic_buffer_set_capacity(lttng_dynamic_buffer*, unsigned long) /home/jgalar/EfficiOS/src/lttng-tools/src/common/dynamic-buffer.cpp:159
#2 0x559f1b021d9f in lttng_dynamic_buffer_append(lttng_dynamic_buffer*, void const*, unsigned long) /home/jgalar/EfficiOS/src/lttng-tools/src/common/dynamic-buffer.cpp:52
#3 0x559f1b02144a in lttng_dynamic_array_add_element(lttng_dynamic_array*, void const*) /home/jgalar/EfficiOS/src/lttng-tools/src/common/dynamic-array.cpp:58
#4 0x559f1b07d07b in lttng_action_path_copy(lttng_action_path const*, lttng_action_path*) actions/path.cpp:116
#5 0x559f1b02383f in lttng_error_query_action_create /home/jgalar/EfficiOS/src/lttng-tools/src/common/error-query.cpp:232
#6 0x559f1b02760e in lttng_error_query_create_from_payload(lttng_payload_view*, lttng_error_query**) /home/jgalar/EfficiOS/src/lttng-tools/src/common/error-query.cpp:911
#7 0x559f1af5c361 in receive_lttng_error_query /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/client.cpp:740
#8 0x559f1af64eba in process_client_msg /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/client.cpp:2336
#9 0x559f1af67378 in thread_manage_clients /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/client.cpp:2624
#10 0x559f1af50642 in launch_thread /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/thread.cpp:68
#11 0x7fe3055225c1 in start_thread (/usr/lib/libc.so.6+0x8d5c1)
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I7a6f7d2a9746124581eebf30877466f16db67a6b
Jérémie Galarneau [Fri, 15 Apr 2022 00:22:03 +0000 (20:22 -0400)]
Fix: lttng: enable-channel: leak of popt arguments
==
1245463==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 5 byte(s) in 1 object(s) allocated from:
#0 0x7fe7c494fdd9 in __interceptor_malloc /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:145
#1 0x7fe7c44a5c09 (/usr/lib/libpopt.so.0+0x3c09)
Arguments obtained with poptGetOptArg() must be free'd.
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I5a65ca6fbaa18f7717ea918a5bc7f42daeb1009a
Jérémie Galarneau [Fri, 15 Apr 2022 00:09:58 +0000 (20:09 -0400)]
Tests: clean-up: rate policy: remove stale comment
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: Idd030c765b0a4afa2d13ff015a17bd52493204a6
Jérémie Galarneau [Fri, 15 Apr 2022 00:09:24 +0000 (20:09 -0400)]
Tests: fix: leak of rate policy in rate policy unit tests
==
1198508==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 56 byte(s) in 1 object(s) allocated from:
#0 0x7f8b62634fb9 in __interceptor_calloc /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:154
#1 0x557871869adb in zmalloc_internal ../../src/common/macros.hpp:60
#2 0x55787186c8a0 in zmalloc<(anonymous namespace)::lttng_rate_policy_once_after_n> ../../src/common/macros.hpp:89
#3 0x55787186c173 in lttng_rate_policy_once_after_n_create actions/rate-policy.cpp:707
#4 0x55787186a368 in lttng_rate_policy_once_after_n_create_from_payload actions/rate-policy.cpp:183
#5 0x55787186ad02 in lttng_rate_policy_create_from_payload(lttng_payload_view*, lttng_rate_policy**) actions/rate-policy.cpp:287
#6 0x557871865b5b in test_rate_policy_once_after_n /home/jgalar/EfficiOS/src/lttng-tools/tests/unit/test_rate_policy.cpp:231
#7 0x557871865dc9 in main /home/jgalar/EfficiOS/src/lttng-tools/tests/unit/test_rate_policy.cpp:250
#8 0x7f8b61c7130f in __libc_start_call_main (/usr/lib/libc.so.6+0x2d30f)
Direct leak of 56 byte(s) in 1 object(s) allocated from:
#0 0x7f8b62634fb9 in __interceptor_calloc /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:154
#1 0x557871869adb in zmalloc_internal ../../src/common/macros.hpp:60
#2 0x55787186c890 in zmalloc<(anonymous namespace)::lttng_rate_policy_every_n> ../../src/common/macros.hpp:89
#3 0x55787186b6cd in lttng_rate_policy_every_n_create actions/rate-policy.cpp:492
#4 0x55787186a699 in lttng_rate_policy_every_n_create_from_payload actions/rate-policy.cpp:220
#5 0x55787186ad02 in lttng_rate_policy_create_from_payload(lttng_payload_view*, lttng_rate_policy**) actions/rate-policy.cpp:287
#6 0x557871864cae in test_rate_policy_every_n /home/jgalar/EfficiOS/src/lttng-tools/tests/unit/test_rate_policy.cpp:122
#7 0x557871865dc4 in main /home/jgalar/EfficiOS/src/lttng-tools/tests/unit/test_rate_policy.cpp:249
#8 0x7f8b61c7130f in __libc_start_call_main (/usr/lib/libc.so.6+0x2d30f)
SUMMARY: AddressSanitizer: 112 byte(s) leaked in 2 allocation(s).
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I3a9b4d99e93f355ddb8623a289f8397907486ab0
Jérémie Galarneau [Fri, 15 Apr 2022 00:06:19 +0000 (20:06 -0400)]
Tests: fix: leak of payload in serdes test of log level rule
==
1190137==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 8 byte(s) in 1 object(s) allocated from:
#0 0x7f40a9d4c1b2 in __interceptor_realloc /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:164
#1 0x55ab716e1def in lttng_dynamic_buffer_set_capacity(lttng_dynamic_buffer*, unsigned long) /home/jgalar/EfficiOS/src/lttng-tools/src/common/dynamic-buffer.cpp:159
#2 0x55ab716e1956 in lttng_dynamic_buffer_append(lttng_dynamic_buffer*, void const*, unsigned long) /home/jgalar/EfficiOS/src/lttng-tools/src/common/dynamic-buffer.cpp:52
#3 0x55ab716ca64e in lttng_log_level_rule_serialize(lttng_log_level_rule const*, lttng_payload*) /home/jgalar/EfficiOS/src/lttng-tools/src/common/log-level-rule.cpp:177
#4 0x55ab716c760f in test_log_level_rule_serialize_deserialize /home/jgalar/EfficiOS/src/lttng-tools/tests/unit/test_log_level_rule.cpp:60
#5 0x55ab716c8457 in test_log_level_rule_at_least_as_severe_as /home/jgalar/EfficiOS/src/lttng-tools/tests/unit/test_log_level_rule.cpp:177
#6 0x55ab716c84d3 in main /home/jgalar/EfficiOS/src/lttng-tools/tests/unit/test_log_level_rule.cpp:185
#7 0x7f40a938830f in __libc_start_call_main (/usr/lib/libc.so.6+0x2d30f)
Direct leak of 8 byte(s) in 1 object(s) allocated from:
#0 0x7f40a9d4c1b2 in __interceptor_realloc /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:164
#1 0x55ab716e1def in lttng_dynamic_buffer_set_capacity(lttng_dynamic_buffer*, unsigned long) /home/jgalar/EfficiOS/src/lttng-tools/src/common/dynamic-buffer.cpp:159
#2 0x55ab716e1956 in lttng_dynamic_buffer_append(lttng_dynamic_buffer*, void const*, unsigned long) /home/jgalar/EfficiOS/src/lttng-tools/src/common/dynamic-buffer.cpp:52
#3 0x55ab716ca64e in lttng_log_level_rule_serialize(lttng_log_level_rule const*, lttng_payload*) /home/jgalar/EfficiOS/src/lttng-tools/src/common/log-level-rule.cpp:177
#4 0x55ab716c760f in test_log_level_rule_serialize_deserialize /home/jgalar/EfficiOS/src/lttng-tools/tests/unit/test_log_level_rule.cpp:60
#5 0x55ab716c8135 in test_log_level_rule_exactly /home/jgalar/EfficiOS/src/lttng-tools/tests/unit/test_log_level_rule.cpp:154
#6 0x55ab716c84ce in main /home/jgalar/EfficiOS/src/lttng-tools/tests/unit/test_log_level_rule.cpp:184
#7 0x7f40a938830f in __libc_start_call_main (/usr/lib/libc.so.6+0x2d30f)
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I2d1eafabbd5c101c188bad8a2137615b29c0ef68
Jérémie Galarneau [Fri, 15 Apr 2022 00:02:18 +0000 (20:02 -0400)]
Tests: fix: leak of some attributes of ltt_ust_session
==
1175545==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 8696 byte(s) in 1 object(s) allocated from:
#0 0x7efed0f39fb9 in __interceptor_calloc /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:154
#1 0x55707ddc6004 in zmalloc_internal ../../../src/common/macros.hpp:60
#2 0x55707ddceb17 in ltt_ust_session* zmalloc<ltt_ust_session>() ../../../src/common/macros.hpp:89
#3 0x55707ddc81e7 in trace_ust_create_session(unsigned long) /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/trace-ust.cpp:274
#4 0x55707ddc2bea in test_create_one_ust_session /home/jgalar/EfficiOS/src/lttng-tools/tests/unit/test_ust_data.cpp:63
#5 0x55707ddc4941 in main /home/jgalar/EfficiOS/src/lttng-tools/tests/unit/test_ust_data.cpp:283
#6 0x7efed04f930f in __libc_start_call_main (/usr/lib/libc.so.6+0x2d30f)
Indirect leak of 24672 byte(s) in 1 object(s) allocated from:
#0 0x7efed0f39fb9 in __interceptor_calloc /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:154
#1 0x55707dee4ec1 in zmalloc_internal ../../../src/common/macros.hpp:60
#2 0x55707def774e in consumer_output* zmalloc<consumer_output>() ../../../src/common/macros.hpp:89
#3 0x55707dee90df in consumer_create_output(consumer_dst_type) /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/consumer.cpp:523
#4 0x55707ddc8821 in trace_ust_create_session(unsigned long) /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/trace-ust.cpp:321
#5 0x55707ddc2bea in test_create_one_ust_session /home/jgalar/EfficiOS/src/lttng-tools/tests/unit/test_ust_data.cpp:63
#6 0x55707ddc4941 in main /home/jgalar/EfficiOS/src/lttng-tools/tests/unit/test_ust_data.cpp:283
#7 0x7efed04f930f in __libc_start_call_main (/usr/lib/libc.so.6+0x2d30f)
Indirect leak of 1024 byte(s) in 1 object(s) allocated from:
#0 0x7efed0f39fb9 in __interceptor_calloc /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:154
#1 0x7efed0bf985f in alloc_split_items_count /home/jgalar/EfficiOS/src/userspace-rcu/src/rculfhash.c:688
#2 0x7efed0bf985f in _cds_lfht_new /home/jgalar/EfficiOS/src/userspace-rcu/src/rculfhash.c:1642
Indirect leak of 656 byte(s) in 1 object(s) allocated from:
#0 0x7efed0f39fb9 in __interceptor_calloc /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:154
#1 0x7efed0bfac68 in __default_alloc_cds_lfht ../src/rculfhash-internal.h:172
#2 0x7efed0bfac68 in alloc_cds_lfht /home/jgalar/EfficiOS/src/userspace-rcu/src/rculfhash-mm-order.c:81
Indirect leak of 48 byte(s) in 2 object(s) allocated from:
#0 0x7efed0f39fb9 in __interceptor_calloc /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:154
#1 0x7efed0bfabd4 in cds_lfht_alloc_bucket_table /home/jgalar/EfficiOS/src/userspace-rcu/src/rculfhash-mm-order.c:35
#2 0x7efed0bfabd4 in cds_lfht_alloc_bucket_table /home/jgalar/EfficiOS/src/userspace-rcu/src/rculfhash-mm-order.c:28
Indirect leak of 24 byte(s) in 1 object(s) allocated from:
#0 0x7efed0f39fb9 in __interceptor_calloc /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:154
#1 0x55707de3a9af in zmalloc_internal ../../src/common/macros.hpp:60
#2 0x55707de3a9bf in lttng_ht* zmalloc<lttng_ht>() ../../src/common/macros.hpp:89
#3 0x55707de38461 in lttng_ht_new(unsigned long, lttng_ht_type) hashtable/hashtable.cpp:113
#4 0x55707dee9340 in consumer_create_output(consumer_dst_type) /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/consumer.cpp:535
#5 0x55707ddc8821 in trace_ust_create_session(unsigned long) /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng-sessiond/trace-ust.cpp:321
#6 0x55707ddc2bea in test_create_one_ust_session /home/jgalar/EfficiOS/src/lttng-tools/tests/unit/test_ust_data.cpp:63
#7 0x55707ddc4941 in main /home/jgalar/EfficiOS/src/lttng-tools/tests/unit/test_ust_data.cpp:283
#8 0x7efed04f930f in __libc_start_call_main (/usr/lib/libc.so.6+0x2d30f)
Indirect leak of 16 byte(s) in 1 object(s) allocated from:
#0 0x7efed0f39fb9 in __interceptor_calloc /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:154
#1 0x7efed0bfac15 in cds_lfht_alloc_bucket_table /home/jgalar/EfficiOS/src/userspace-rcu/src/rculfhash-mm-order.c:31
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: Ib2ad82a197f2a4ccb86ae5799c1d93ff059888e3
Jérémie Galarneau [Thu, 14 Apr 2022 23:45:28 +0000 (19:45 -0400)]
Fix: liblttng-ctl: leak of payload on field listing
LeakSanitizer reports the following leak:
==974957==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 32 byte(s) in 1 object(s) allocated from:
#0 0x7fdb86fcd1b2 in __interceptor_realloc /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:164
#1 0x7fdb86d7c296 in lttng_dynamic_buffer_set_capacity(lttng_dynamic_buffer*, unsigned long) /home/jgalar/EfficiOS/src/lttng-tools/src/common/dynamic-buffer.cpp:159
#2 0x7fdb86d7c060 in lttng_dynamic_buffer_set_size(lttng_dynamic_buffer*, unsigned long) /home/jgalar/EfficiOS/src/lttng-tools/src/common/dynamic-buffer.cpp:112
#3 0x7fdb86d2589a in recv_payload_sessiond /home/jgalar/EfficiOS/src/lttng-tools/src/lib/lttng-ctl/lttng-ctl.cpp:230
#4 0x7fdb86d26fa5 in lttng_ctl_ask_sessiond_payload(lttng_payload_view*, lttng_payload*) /home/jgalar/EfficiOS/src/lttng-tools/src/lib/lttng-ctl/lttng-ctl.cpp:662
#5 0x7fdb86d2cd8d in lttng_list_tracepoint_fields /home/jgalar/EfficiOS/src/lttng-tools/src/lib/lttng-ctl/lttng-ctl.cpp:1767
#6 0x56481623cb4c in list_ust_event_fields commands/list.cpp:850
#7 0x5648162448d9 in cmd_list(int, char const**) commands/list.cpp:2394
#8 0x56481628fb3e in handle_command /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng/lttng.cpp:238
#9 0x564816290601 in parse_args /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng/lttng.cpp:427
#10 0x564816290908 in main /home/jgalar/EfficiOS/src/lttng-tools/src/bin/lttng/lttng.cpp:476
#11 0x7fdb8661730f in __libc_start_call_main (/usr/lib/libc.so.6+0x2d30f)
SUMMARY: AddressSanitizer: 32 byte(s) leaked in 1 allocation(s).
The session daemon's reply is indeed never released in
lttng_list_tracepoint_fields.
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: Idd244b52a69f3b74e5c131c1c36c6ee6d76f4285
Jérémie Galarneau [Thu, 14 Apr 2022 21:36:54 +0000 (17:36 -0400)]
Fix: liblttng-ctl: non-packed structure used for tracker serialization
Using unpacked structures in liblttng-ctl's protocol can cause issues
when mixing sessiond and client of different bitness. In this specific
case I doubt it causes a problem, but it could rightfully do on some
architectures.
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: Ie17096a55a4b7508c604e184cae877b83df6e451
Jonathan Rajotte [Fri, 25 Mar 2022 18:26:38 +0000 (14:26 -0400)]
Fix: test: lttng kernel modules still loaded after running test_clock_override
Observed issue
==============
After running test_clock_override, some lttng modules are still loaded.
$ lsmod | ag lttng
lttng_test 32768 0
lttng_tracer
2326528 1 lttng_test
lttng_statedump 749568 1 lttng_tracer
lttng_wrapper 16384 2 lttng_statedump,lttng_tracer
lttng_uprobes 16384 1 lttng_tracer
lttng_kprobes 16384 1 lttng_tracer
lttng_lib_ring_buffer 61440 1 lttng_tracer
lttng_kretprobes 16384 1 lttng_tracer
lttng_clock_plugin_test 16384 1
lttng_clock 16384 2 lttng_tracer,lttng_clock_plugin_test
Cause
=====
The order in which the modules are removed is important.
In `test_clock_override_timestamp` the last `modprobe --remove order` is
modprobe --remove lttng-clock-plugin-test lttng-clock lttng-test
While other callsites order is:
modprobe --remove lttng-test lttng-clock-plugin-test lttng-clock
Solution
========
Use
modprobe --remove lttng-test lttng-clock-plugin-test lttng-clock
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I528df2e3e90664433337a547a74cdbe476d4ee62
Jérémie Galarneau [Wed, 13 Apr 2022 18:27:57 +0000 (14:27 -0400)]
Update version to v2.13.7
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Jérémie Galarneau [Fri, 8 Apr 2022 23:34:04 +0000 (19:34 -0400)]
Fix: sessiond: assert on empty payload when handling client out event
Observed issue
==============
When servicing a large number of tracer notifications and sending
notifications to clients, the session daemon occasionally hits
an assertion:
#4 0x00007fb224d7d116 in __assert_fail () from /usr/lib/libc.so.6
#5 0x000056038b2fe4d7 in client_flush_outgoing_queue (client=0x7fb21400c3b0) at notification-thread-events.cpp:3586
#6 0x000056038b2ff819 in handle_notification_thread_client_out (state=0x7fb221974090, socket=77) at notification-thread-events.cpp:4104
#7 0x000056038b2f3d77 in thread_notification (data=0x56038cc7fe90) at notification-thread.cpp:763
#8 0x000056038b30ca7d in launch_thread (data=0x56038cc7e220) at thread.cpp:66
#9 0x00007fb224dcf5c2 in start_thread () from /usr/lib/libc.so.6
#10 0x00007fb224e54584 in clone () from /usr/lib/libc.so.6
Cause
=====
A client "out" event can be received when no payload is left
to send under some circumstances.
Many threads can flush a client's outgoing queue and, if they
had to queue their message (socket was full), will use the
"communication update" command to signal the (e)poll thread
to monitor for space being made available in the socket.
Commands are sent over an internal pipe serviced by the same
thread as the client sockets.
When space is made available in the socket, there is a race
between the (e)poll thread and the other threads that may
wish to use the client's socket to flush its outgoing queue.
A non-(e)poll thread may attempt (and succeed) in flushing
the queue before the (e)poll thread gets a chance to service
the client's "out" event.
In this situation, the (e)poll thread processing the client
out event will see an empty payload: there is nothing to do.
Solution
========
The (e)poll thread can simply ignore the "client out" event
when an empty payload is seen.
There is also no need to update the transmission status as
the other thread has already enqueued a "communication
update" command to do so.
Known drawbacks
===============
None.
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I8a181bea1e37e8e14cc67b624b76d139b488eded
Jonathan Rajotte [Wed, 6 Apr 2022 13:17:38 +0000 (09:17 -0400)]
Fix: Revert of
814b4934e2604a419bcb8eec57c0450dbb47e2c3
Observed issue
==============
During high throughput event notification generation scenarios the
following deadlock happens:
Thread 14 (Thread 0x7f74b4ff9700 (LWP 76062)):
#0 __lll_lock_wait (futex=futex@entry=0x56408765dde8, private=0) at lowlevellock.c:52
#1 0x00007f74c941a0a3 in __GI___pthread_mutex_lock (mutex=0x56408765dde8) at ../nptl/pthread_mutex_lock.c:80
#2 0x000056408704b207 in run_command_wait (handle=0x56408765ddd0, cmd=0x7f74b4ff7f70) at notification-thread-commands.cpp:31
#3 0x000056408704bcef in notification_thread_command_remove_tracer_event_source (handle=0x56408765ddd0, tracer_event_source_fd=54) at notification-thread-commands.cpp:319
#4 0x000056408708a0c1 in delete_ust_app (app=0x7f749c000bf0) at ust-app.cpp:1059
#5 0x000056408708a511 in delete_ust_app_rcu (head=0x7f749c000ca0) at ust-app.cpp:1122
#6 0x00007f74c988b4a7 in call_rcu_thread (arg=0x7f74b8004a80) at ../src/urcu-call-rcu-impl.h:369
#7 0x00007f74c9417609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#8 0x00007f74c933a163 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 13 (Thread 0x7f74b57fa700 (LWP 76047)):
#0 0x00007f74c933a49e in epoll_wait (epfd=48, events=0x7f74a4000b60, maxevents=2, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1 0x00005640870eafa6 in compat_epoll_wait (events=0x7f74b57f9240, timeout=-1, interruptible=false) at compat/poll.cpp:280
#2 0x00005640870abb65 in thread_agent_management (data=0x56408765f0b0) at agent-thread.cpp:424
#3 0x0000564087062b1a in launch_thread (data=0x56408765f150) at thread.cpp:66
#4 0x00007f74c9417609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#5 0x00007f74c933a163 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 12 (Thread 0x7f74b5ffb700 (LWP 76046)): [630/709]
#0 0x00007f74c933a49e in epoll_wait (epfd=47, events=0x7f74a0000b60, maxevents=2, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1 0x00005640870eafa6 in compat_epoll_wait (events=0x7f74b5ffa170, timeout=-1, interruptible=false) at compat/poll.cpp:280
#2 0x00005640870a4095 in thread_application_notification (data=0x56408765ee40) at notify-apps.cpp:78
#3 0x0000564087062b1a in launch_thread (data=0x56408765eed0) at thread.cpp:66
#4 0x00007f74c9417609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#5 0x00007f74c933a163 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 11 (Thread 0x7f74b67fc700 (LWP 76045)):
#0 0x00007f74c933a49e in epoll_wait (epfd=44, events=0x7f74ac000b60, maxevents=2, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1 0x00005640870eafa6 in compat_epoll_wait (events=0x7f74b67fb170, timeout=-1, interruptible=false) at compat/poll.cpp:280
#2 0x00005640870723db in thread_application_management (data=0x56408765ebd0) at manage-apps.cpp:93
#3 0x0000564087062b1a in launch_thread (data=0x56408765ec60) at thread.cpp:66
#4 0x00007f74c9417609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#5 0x00007f74c933a163 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 10 (Thread 0x7f74b6ffd700 (LWP 76044)):
#0 0x00007f74c933a49e in epoll_wait (epfd=39, events=0x7f74a8000b60, maxevents=2, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1 0x00005640870eafa6 in compat_epoll_wait (events=0x7f74b6ffc130, timeout=-1, interruptible=false) at compat/poll.cpp:280
#2 0x0000564087070a27 in thread_application_registration (data=0x56408765e940) at register.cpp:214
#3 0x0000564087062b1a in launch_thread (data=0x56408765e9f0) at thread.cpp:66
#4 0x00007f74c9417609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#5 0x00007f74c933a163 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 9 (Thread 0x7f74b77fe700 (LWP 76043)): [654/709]
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00005640870c8e25 in futex (uaddr=0x5640871e2800 <ust_cmd_queue>, op=0, val=-1, timeout=0x0, uaddr2=0x0, val3=0) at /home/joraj/lttng/master/install/include/urcu/futex.h:72
#2 0x00005640870c8e6d in futex_async (uaddr=0x5640871e2800 <ust_cmd_queue>, op=0, val=-1, timeout=0x0, uaddr2=0x0, val3=0) at /home/joraj/lttng/master/install/include/urcu/futex.h:104
#3 0x00005640870c939a in futex_nto1_wait (futex=0x5640871e2800 <ust_cmd_queue>) at futex.cpp:77
#4 0x000056408706f2af in thread_dispatch_ust_registration (data=0x56408765e740) at dispatch.cpp:453
#5 0x0000564087062b1a in launch_thread (data=0x56408765e760) at thread.cpp:66
#6 0x00007f74c9417609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7 0x00007f74c933a163 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 8 (Thread 0x7f74b7fff700 (LWP 76042)):
#0 0x00007f74c933a49e in epoll_wait (epfd=33, events=0x7f74b0000b60, maxevents=2, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1 0x00005640870eafa6 in compat_epoll_wait (events=0x7f74b7ffad40, timeout=-1, interruptible=false) at compat/poll.cpp:280
#2 0x000056408706c424 in thread_manage_clients (data=0x56408765e4f0) at client.cpp:2528
#3 0x0000564087062b1a in launch_thread (data=0x56408765e560) at thread.cpp:66
#4 0x00007f74c9417609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#5 0x00007f74c933a163 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 7 (Thread 0x7f74c4b8f700 (LWP 76041)): [672/709]
#0 0x00007f74c933a49e in epoll_wait (epfd=31, events=0x7f74bc000b60, maxevents=3, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1 0x00005640870eafa6 in compat_epoll_wait (events=0x7f74c4b8e240, timeout=-1, interruptible=false) at compat/poll.cpp:280
#2 0x000056408705f2b6 in thread_rotation (data=0x56408765e280) at rotation-thread.cpp:804
#3 0x0000564087062b1a in launch_thread (data=0x56408765e310) at thread.cpp:66
#4 0x00007f74c9417609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#5 0x00007f74c933a163 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 6 (Thread 0x7f74c5390700 (LWP 76040)):
#0 0x00007f74c925f1d2 in __GI___sigtimedwait (set=0x7f74c538f090, info=0x7f74c538f110, timeout=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:29
#1 0x000056408706138a in thread_timer (data=0x7ffc1fcbe3f0) at timer.cpp:359
#2 0x0000564087062b1a in launch_thread (data=0x56408765e0a0) at thread.cpp:66
#3 0x00007f74c9417609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#4 0x00007f74c933a163 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 5 (Thread 0x7f74c5b91700 (LWP 76039)):
#0 __libc_write (nbytes=8, buf=0x7f74c5b8fc88, fd=24) at ../sysdeps/unix/sysv/linux/write.c:26
#1 __libc_write (fd=24, buf=0x7f74c5b8fc88, nbytes=8) at ../sysdeps/unix/sysv/linux/write.c:24
#2 0x00005640870eeb4f in lttng_write (fd=24, buf=0x7f74c5b8fc88, count=8) at readwrite.cpp:77
#3 0x000056408704b535 in run_command_no_wait (handle=0x56408765ddd0, in_cmd=0x7f74c5b8fdf0) at notification-thread-commands.cpp:92
#4 0x000056408704bf49 in notification_thread_client_communication_update (handle=0x56408765ddd0, id=2, transmission_status=CLIENT_TRANSMISSION_STATUS_QUEUED) at notification-thread-command
#5 0x000056408707bc62 in client_handle_transmission_status (client=0x7f74b80050d0, status=CLIENT_TRANSMISSION_STATUS_QUEUED, user_data=0x7f74b8004410) at action-executor.cpp:258
#6 0x0000564087057525 in notification_client_list_send_evaluation (client_list=0x7f74b8004df0, trigger=0x7f74b0001030, evaluation=0x7f74b815d1d0, source_object_creds=0x0, client_report=0x5
#7 0x000056408707bce9 in action_executor_notify_handler (executor=0x7f74b8004410, work_item=0x7f74b815d430, item=0x7f74b80e48e0) at action-executor.cpp:269
#8 0x000056408707dd6d in action_executor_generic_handler (executor=0x7f74b8004410, work_item=0x7f74b815d430, item=0x7f74b80e48e0) at action-executor.cpp:670
#9 0x000056408707df01 in action_work_item_execute (executor=0x7f74b8004410, work_item=0x7f74b815d430) at action-executor.cpp:689
#10 0x000056408707e525 in action_executor_thread (_data=0x7f74b8004410) at action-executor.cpp:771 [698/709]
#11 0x0000564087062b1a in launch_thread (data=0x7f74b80044b0) at thread.cpp:66
#12 0x00007f74c9417609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#13 0x00007f74c933a163 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 4 (Thread 0x7f74c6392700 (LWP 76038)):
#0 __lll_lock_wait (futex=futex@entry=0x56408765dde8, private=0) at lowlevellock.c:52
#1 0x00007f74c941a0a3 in __GI___pthread_mutex_lock (mutex=0x56408765dde8) at ../nptl/pthread_mutex_lock.c:80
#2 0x0000564087053c89 in handle_notification_thread_command (handle=0x56408765ddd0, state=0x7f74c63911b0) at notification-thread-events.cpp:3142
#3 0x000056408704ac81 in thread_notification (data=0x56408765ddd0) at notification-thread.cpp:715
#4 0x0000564087062b1a in launch_thread (data=0x56408765dec0) at thread.cpp:66
#5 0x00007f74c9417609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#run_command_no_wait6 0x00007f74c933a163 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 3 (Thread 0x7f74c6b93700 (LWP 76037)):
#0 0x00007f74c933a49e in epoll_wait (epfd=21, events=0x7f74c0000b60, maxevents=2, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1 0x00005640870eafa6 in compat_epoll_wait (events=0x7f74c6b92170, timeout=-1, interruptible=false) at compat/poll.cpp:280
#2 0x000056408706400a in thread_manage_health (data=0x56408765db50) at health.cpp:140
#3 0x0000564087062b1a in launch_thread (data=0x56408765dbf0) at thread.cpp:66
#4 0x00007f74c9417609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#5 0x00007f74c933a163 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 2 (Thread 0x7f74c7394700 (LWP 76036)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007f74c987d238 in futex (uaddr=0x564087659b10, op=0, val=-1, timeout=0x0, uaddr2=0x0, val3=0) at ../include/urcu/futex.h:72
#2 futex_async (uaddr=0x564087659b10, op=0, val=-1, timeout=0x0, uaddr2=0x0, val3=0) at ../include/urcu/futex.h:104
#3 futex_wait (futex=0x564087659b10) at workqueue.c:136
#4 0x00007f74c987ced2 in workqueue_thread (arg=0x564087659ad0) at workqueue.c:237
#5 0x00007f74c9417609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6 0x00007f74c933a163 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 1 (Thread 0x7f74c73cd300 (LWP 76034)):
#0 0x00007f74c933a49e in epoll_wait (epfd=50, events=0x564087666880, maxevents=1, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1 0x00005640870eafa6 in compat_epoll_wait (events=0x7ffc1fcbe280, timeout=-1, interruptible=false) at compat/poll.cpp:280
#2 0x0000564087062244 in sessiond_wait_for_quit_pipe (timeout_ms=-1) at thread-utils.cpp:83
#3 0x00005640870127dc in main (argc=1, argv=0x7ffc1fcbe668) at main.cpp:1921
Cause
=====
The event_pipe used to notify the notification poll loop is full and the
lttng_write call blocks with the locks for both the client and the
cmd_queue held.
Solution
========
Go back to using eventfd but without the use of EFD_SEMAPHORE (linux
2.6.30) to continue supporting kernel between 2.6.27 and 2.6.29.
The EFD_SEMAPHORE is emulated with a read, decrement, write as explained
by the initial committer of EFD_SEMAPHORE [1].
Known drawbacks
=========
This does not solve the actual block+lock problem but simply push it
back further. The lttng_write on the eventfd can block when reaching
UINT64_MAX. This would represent, at 1 command queued per ns (which is
ridiculous), ~584 years of queueing without a dequeue operation.
Reference
=======
[1] https://lwn.net/Articles/318151/
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: Ie749c4169708f57463fe3cfab2366f1015bae4e0
Jérémie Galarneau [Wed, 6 Apr 2022 17:37:16 +0000 (13:37 -0400)]
Fix: sessiond: inverted condition checking for empty hash table
I inverted a condition while reformating
2a6ebf6bd. This reverts the
condition to the intention of the original author. Mea culpa.
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I8c39a89f430dbb4a0f1e385b3b8e4788f188a468
Jonathan Rajotte [Fri, 1 Apr 2022 12:41:17 +0000 (08:41 -0400)]
Fix: notification: kernel: consumption of event notification stalls
Observed issue
==============
Using:
lttng add-trigger --condition event-rule-matches --type kernel:tracepoint --name "sched_waking" --capture comm --action notify
The sessiond receives multiple event notifications from the kernel event
source then stop receiving despite the kernel event source buffer
being full.
Cause
=====
It turns out that the kernel event source, when reaching near the end of
its buffer capacity, raises the POLLPRI [1] flag and not the POLLIN
flag.
Solution
========
lttng-modules stretches a bit the usage of POLLPRI as defined by the man
page (man 2 poll):
There is some exceptional condition on the file descriptor. Possibilities
include:
* There is out-of-band data on a TCP socket (see tcp(7)).
* A pseudoterminal master in packet mode has seen a state change on the
slave (see ioctl_tty(2)).
* A cgroup.events file has been modified (see cgroups(7)).
Still, even if lttng-modules changes how it does things, lttng-tools
needs to support other lttng-modules versions.
Thus, add LPOLLPRI (EPOLLPRI/POLLPRI) to the event mask when dealing
with notification event sources.
Note
=====
In the future, during the poll loop we could also prioritize
event sources in POLLPRI 'state'.
Known drawbacks
=========
None.
References
==========
[1] https://github.com/lttng/lttng-modules/blob/
c312bda00d2dc10ce5f6c1189acbefee5c6c8c6c/src/lttng-abi.c#L1169
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: Ieb428ef1037c8eb197b489a38a1ae5216ac63d4b
Jonathan Rajotte [Thu, 31 Mar 2022 13:46:15 +0000 (09:46 -0400)]
Fix: notification: assert on len > 0 for dropped notification message
Observed issue
==============
Using the notification client from
doc/examples/trigger-condition-event-matches/notification-client.cpp, an
assert is hit when the notification subsystem is under load.
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007f69eab58859 in __GI_abort () at abort.c:79
#2 0x00007f69eab58729 in __assert_fail_base (fmt=0x7f69eacee588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x7f69eae1d5dd "len > 0", file=0x7f69eae1d5cb "unix.cpp", line=179, function=<optimized out>) at assert.c:92
#3 0x00007f69eab6a006 in __GI___assert_fail (assertion=0x7f69eae1d5dd "len > 0", file=0x7f69eae1d5cb "unix.cpp", line=179, function=0x7f69eae1d598 "ssize_t lttcomm_recv_unix_sock(int, void*, size_t)") at assert.c:101
#4 0x00007f69eadd5fe6 in lttcomm_recv_unix_sock (sock=3, buf=0x55da9ecd5f89, len=0) at unix.cpp:179
#5 0x00007f69ead7df3f in receive_message (channel=0x55da9ecd6ee0) at channel.cpp:64
#6 0x00007f69ead7e478 in lttng_notification_channel_get_next_notification (channel=0x55da9ecd6ee0, _notification=0x7ffdefed2570) at channel.cpp:279
#7 0x000055da9e0e742f in main (argc=2, argv=0x7ffdefed2698) at notification-client.cpp:506
(gdb) frame
#5 0x00007f69ead7df3f in receive_message (channel=0x55da9ecd6ee0) at channel.cpp:64
64 ret = lttcomm_recv_unix_sock(channel->socket,
(gdb) print msg
$2 = {type = 5 '\005', size = 0, fds = 0, payload = 0x7ffdefed24a8 ""}
The msg type 5 is
`LTTNG_NOTIFICATION_CHANNEL_MESSAGE_TYPE_NOTIFICATION_DROPPED`
Cause
=====
The msg portion of a
`LTTNG_NOTIFICATION_CHANNEL_MESSAGE_TYPE_NOTIFICATION_DROPPED` is indeed
zero. There is no extra payload.
Solution
========
When the msg size is zero, skip the 'payload' reception phase.
Known drawbacks
=========
None.
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: Ibabb922d0e410c9902414a5eabbe04738861d772
Jonathan Rajotte [Thu, 31 Mar 2022 13:44:24 +0000 (09:44 -0400)]
Fix: example: print_notification is called on status all returned status
The notification should only be printed for
`LTTNG_NOTIFICATION_CHANNEL_STATUS_OK`.
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I5534406d8fbd5c0fff7013fda6335d54bef071a2
Jonathan Rajotte [Mon, 28 Mar 2022 20:49:17 +0000 (16:49 -0400)]
Fix: sessiond: assertion hit in ltt_sessions_ht_empty
Observed issue
==============
Scenario:
gdb lttng-sessiond
set non-stop
break rotation-thread.cpp:584
^ simulates a slow rotation thread or not scheduled thread.
lttng create test1
lttng enable-event -u -a
lttng start test1
lttng create test2
lttng enable-event -u -a
lttng start test2
lttng destroy test1
This will hang on rotation pending checks on the CLI side.
In another shell:
lttng destroy test2
This will hang on rotation pending checks on the CLI side.
Back to gdb
thread 7
continue
Results in:
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007ffff786c859 in __GI_abort () at abort.c:79
#2 0x00007ffff786c729 in __assert_fail_base (fmt=0x7ffff7a02588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x5555556bb148 "count == lttng_ht_get_count(ltt_sessions_ht_by_name)", file=0x5555556bae9f "session.cpp", line=395, function=<optimized out>) at assert.c:92
#3 0x00007ffff787e006 in __GI___assert_fail (assertion=0x5555556bb148 "count == lttng_ht_get_count(ltt_sessions_ht_by_name)", file=0x5555556bae9f "session.cpp", line=395, function=0x5555556bb129 "int ltt_sessions_ht_empty()") at assert.c:101
#4 0x0000555555586d59 in ltt_sessions_ht_empty () at session.cpp:395
#5 0x0000555555586e53 in del_session_ht (ls=0x7fffdc000c30) at session.cpp:418
#6 0x0000555555588a95 in session_release (ref=0x7fffdc001e50) at session.cpp:999
#7 0x000055555558620f in urcu_ref_put (ref=0x7fffdc001e50, release=0x5555555886eb <session_release(urcu_ref*)>) at /home/joraj/lttng/master/install/include/urcu/ref.h:68
#8 0x0000555555588c8f in session_put (session=0x7fffdc000c30) at session.cpp:1048
#9 0x00005555555bf995 in handle_job_queue (handle=0x55555575d260, state=0x7fffeeffc240, queue=0x555555758960) at rotation-thread.cpp:612
#10 0x00005555555c05da in thread_rotation (data=0x55555575d260) at rotation-thread.cpp:847
#11 0x00005555555c3b1c in launch_thread (data=0x55555575d2f0) at thread.cpp:66
#12 0x00007ffff7a46609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#13 0x00007ffff7969163 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Other scenarios can lead to a similar backtrace when using the
`--no-wait` lttng destroy option.
Cause
=====
Since
ed41e5709047ef545aa28082416e641e003b45e0 [1], hash table removal
for the session object for the `ltt_sessions_ht_by_name` and
`ltt_sessions_ht_by_name` are "decoupled". Removal from
`ltt_sessions_ht_by_name` is done early in `session_destroy()` while
removal from `ltt_sessions_ht_by_id` is done during `session_release` when
the last reference of a session object is released.
This can leads to `imbalances` between the size of the two hash tables
when multiple sessions are at play.
Solution
========
Rework `ltt_sessions_ht_empty()` to exit early when
`ltt_sessions_ht_by_id` is not empty. Perform a sanity check on
`ltt_sessions_ht_by_name` only when `ltt_sessions_ht_by_id` is empty.
Note
========
Ideally both hash tables' lifetime would be managed separately but it
seems easier in term of initialization to bundle them together for now
considering the limited scope of the `ltt_sessions_ht_by_name` hash
table.
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I66c459f80298f929add703ac977cccd1da6dd556
Jonathan Rajotte [Thu, 31 Mar 2022 15:20:01 +0000 (11:20 -0400)]
Fix: tests: missing _GNU_SOURCE for F_GETPIPE_SZ
Per man 2 fcntl:
F_GETOWN_EX, F_SETOWN_EX, F_SETPIPE_SZ, F_GETPIPE_SZ, F_GETSIG, F_SETSIG,
F_NOTIFY, F_GETLEASE, and F_SETLEASE are Linux-specific. (Define the
_GNU_SOURCE macro to obtain these definitions.)
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I2b61dfb79ffa384dc2bab56cd3510ddc6ae21e85
Jonathan Rajotte [Tue, 29 Mar 2022 20:31:44 +0000 (16:31 -0400)]
Fix: compat: 'LTTNG_UST_ABI_PROCNAME_LEN' is undeclared
Observed issue
==============
On old systems, the `lttng_pthread_setname_np` function fallsback to
using the compat prctl version. In that context,
`LTTNG_UST_ABI_PROCNAME_LEN` is indeed not declared.
Solution
========
Use `LTTNG_PTHREAD_NAMELEN`. This mimics what is done in other versions
of `lttng_pthread_setname_np`.
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I73956cacd7b1e9400881b17b1cd89db2530d3a00
Jérémie Galarneau [Fri, 25 Mar 2022 19:34:47 +0000 (15:34 -0400)]
Fix: sessiond: cmd_enable_channel: negative error code used
A negative `lttng_error_code` value is returned (as an integer)
when a channel copy fails. Return a positive error code.
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I340f739cc33858a06832bb75a7a6d5e18459551f
Jérémie Galarneau [Tue, 29 Mar 2022 01:52:30 +0000 (21:52 -0400)]
Update version to v2.13.6
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Michael Jeanson [Mon, 28 Mar 2022 22:25:14 +0000 (18:25 -0400)]
Fix: hide symbols erroneously exported in 2.13.5
Change-Id: I508ac4791749240e511ef832a97e92878dd68df2
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Jérémie Galarneau [Mon, 28 Mar 2022 16:11:03 +0000 (12:11 -0400)]
Update version to v2.13.5
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Jonathan Rajotte [Thu, 24 Mar 2022 20:05:20 +0000 (16:05 -0400)]
Fix: doc: action: wrong function documented for action_list destroy
The lttng_action_list_destroy function is internal.
API users must use `lttng_action_destroy()` to destroy the returned
object of `lttng_action_list_create()`.
Change-Id: Ic910efd07dd071f7e38e48d34a5e000b3f805729
Reported-by: Michael Jeason <mjeanson@efficios.com>
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Jérémie Galarneau [Fri, 18 Mar 2022 14:58:03 +0000 (10:58 -0400)]
Fix: lttng-elf: wrong error label used by error path
1486805 Resource leak
The system resource will not be reclaimed and reused, reducing the future availability of the resource.
In lttng_elf_get_symbol_offset: Leak of memory or pointers to system resources (CWE-404)
Reported-by: Coverity Scan
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I78b868773b389c2eaf3f1d45151fe9416b1fe447
Michael Jeanson [Wed, 16 Mar 2022 15:40:52 +0000 (11:40 -0400)]
Fix: use the correct endian compat macros
Document which variant of the endian macros our compat header guarantees
across all platforms and fix incorrect uses.
This was discovered with -Wundef on macOS.
Change-Id: Iaf442fe5887063661273ac2a00c9fa4015e83d5c
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Jérémie Galarneau [Wed, 16 Mar 2022 21:39:07 +0000 (17:39 -0400)]
Fix: tests: uninitialized lttng_payload
1474980 Uninitialized pointer read
Incorrect values could be read from, or even written to, an arbitrary memory location, causing incorrect computations.
In test_event_rule_userspace_probe(): Reads an uninitialized pointer or its target (CWE-457)
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I7645278f18e4a4678fb5ede9523d0cfa8d3aa106
Jérémie Galarneau [Wed, 16 Mar 2022 21:35:43 +0000 (17:35 -0400)]
Fix: sessiond: ust-app: uninitialized name logged on stream copy failure
1466302 Uninitialized scalar variable
The variable will contain an arbitrary value left from earlier computations.
In send_channel_uid_to_ust(buffer_reg_channel *, ust_app *, ust_app_session *, ust_app_channel *): Use of an uninitialized variable (CWE-457)
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: Icd38de3b67dab783fa26a721c68c48ebfbb59785
Jérémie Galarneau [Wed, 16 Mar 2022 21:29:11 +0000 (17:29 -0400)]
Fix: lttng-elf: untrusted entry size divisor
1405557 Untrusted divisor
The divisor could be controlled by an attacker, who could cause a division by zero.
In lttng_elf_get_symbol_offset: An unscrutinized value from an untrusted source used as a divisor (CWE-369)
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I029708a0df4f62fe0031e374d50839c26f4f3f4b
Jérémie Galarneau [Wed, 16 Mar 2022 15:56:21 +0000 (11:56 -0400)]
Fix: event: erroneous bound check on perf counter name size
The wrong size if used when initializing a perf counter name from a
payload. The destination size must be used to prevent out of bound
writes.
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I8ea41d30815bd2c02bb2ad8b01e8cecd2d6549a8
Jérémie Galarneau [Wed, 16 Mar 2022 15:55:08 +0000 (11:55 -0400)]
Fix: sessiond: event name length check is too strict
A truncation check when initializing an event from an event rule limits
the name to one less character than is supposed to be allowed.
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I172c5098597923d14508c150c7b3577f759bae72
Jérémie Galarneau [Wed, 16 Mar 2022 14:48:59 +0000 (10:48 -0400)]
Fix: relayd: missing session unlock on error path
1475890 Missing unlock May result in deadlock if there is another
attempt to acquire the lock.
In viewer_get_new_streams(relay_connection *): Missing a release of a
lock on a path (CWE-667)
Reported-by: Coverity Scan
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I66de344f5f39ac85bf8db93cf39a07d0c6cf7694
Michael Jeanson [Mon, 7 Mar 2022 16:37:49 +0000 (11:37 -0500)]
fix: warning '-Wstringop-truncation' with GCC 11.2
Building with GCC 11.2 results in the following warning :
In file included from ../../src/common/tracker.h:18,
from ../../src/bin/lttng-sessiond/trace-ust.h:17,
from test_ust_data.cpp:19:
../../src/common/sessiond-comm/sessiond-comm.h:569:14: note: while referencing ‘lttng_event_exclusion::names’
569 | char names[0][LTTNG_SYMBOL_NAME_LEN];
| ^~~~~
test_ust_data.cpp:209:16: warning: ‘char* strncpy(char*, const char*, size_t)’ specified bound 256 equals destination size [-Wstringop-truncation]
209 | strncpy(LTTNG_EVENT_EXCLUSION_NAME_AT(exclusion, 0),
| ~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
210 | get_random_string(), LTTNG_SYMBOL_NAME_LEN);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
test_ust_data.cpp:211:16: warning: ‘char* strncpy(char*, const char*, size_t)’ specified bound 256 equals destination size [-Wstringop-truncation]
211 | strncpy(LTTNG_EVENT_EXCLUSION_NAME_AT(exclusion, 1),
| ~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
212 | get_random_string(), LTTNG_SYMBOL_NAME_LEN);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Change-Id: I78eea760b4684227ee457c3368c6397d0a767af5
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Michael Jeanson [Fri, 11 Feb 2022 15:26:02 +0000 (15:26 +0000)]
Add Log4j 2.x agent tests for the 'log4j' domain
Add integration tests for the new Log4j 2.x agent in Log4j 1.x compat
mode using the current 'log4j' domain, use the new configure switch
'--enable-test-java-agent-log4j2' to enable it.
To run only this new test, use this command :
cd tests/regression && make check TESTS="ust/java-log4j2/test_agent_log4j2_domain_log4j"
Change-Id: Id780c9ee13913c91c18548f58b14cc600e77e9fa
Depends-on: lttng-ust: I7fd5f79ad58c77175714bd4198d8ff5db2e6b846
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Jérémie Galarneau [Thu, 10 Mar 2022 22:46:31 +0000 (17:46 -0500)]
Fix: relayd: session id is ignored by 2.11+ create session command
The id of the session used by the sessiond is not returned by
cmd_create_session_2_11 and its caller sets the value in the
relay_session to an uninitialized value.
Up until recently this didn't have much effect as this uninitialized
value was stored and used to perform look-ups in the trace chunk
registry, which would work.
However, the recent multi-consumer rotation fixes make this problem more
significant as this 'id' is used as a key to join relay sessions
originating from the same session daemon.
This was discovered by enabling the '-Wunused-parameter' warning.
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I7e33f5f93dc46bb630c431408c9472be3a75c030
Jérémie Galarneau [Wed, 9 Mar 2022 12:56:47 +0000 (07:56 -0500)]
Build fix: consumer: aggregate initialization isn't supported
Aggregate initialization is not supported in C. Change the syntax to use
a compound literal.
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I630ca41b13ea730e699e53ae9a23bdd78acbed57
Jérémie Galarneau [Thu, 3 Mar 2022 00:27:31 +0000 (19:27 -0500)]
Fix: consumerd: use-after-free of metadata bucket
Observed issue
==============
When consumer_stream_destroy() is called from, for example, the error
path in setup_metadata(), consumer_stream_free() can end up being called
twice on the same stream. Since the stream->metadata_bucket is not set
to NULL after being destroyed, it leads to a use-after-free:
ERROR: AddressSanitizer: heap-use-after-free on address 0x604000000318
READ of size 8 at 0x604000000318 thread T7
#0 in metadata_bucket_destroy
#1 in consumer_stream_free
#2 in consumer_stream_destroy
#3 in setup_metadata
#4 in lttng_ustconsumer_recv_cmd
#5 in lttng_consumer_recv_cmd
#6 in consumer_thread_sessiond_poll
#7 in start_thread nptl/pthread_create.c:481
#8 in clone (/lib/x86_64-linux-gnu/libc.so.6+0xfcbde)
0x604000000318 is located 8 bytes inside of 48-byte region [0x604000000310,0x604000000340)
freed by thread T7 here:
#0 in __interceptor_free
#1 in metadata_bucket_destroy
#2 in consumer_stream_free
#3 in consumer_stream_destroy
#4 in clean_channel_stream_list
#5 in consumer_del_channel
#6 in consumer_stream_destroy
#7 in setup_metadata
#8 in lttng_ustconsumer_recv_cmd
#9 in lttng_consumer_recv_cmd
#10 in consumer_thread_sessiond_poll
#11 in start_thread nptl/pthread_create.c:481
previously allocated by thread T7 here:
#0 in __interceptor_calloc
#1 in zmalloc
#2 in metadata_bucket_create
#3 in consumer_stream_enable_metadata_bucketization
#4 in lttng_ustconsumer_set_stream_ops
#5 in lttng_ustconsumer_on_recv_stream
#6 in lttng_consumer_on_recv_stream
#7 in create_ust_streams
#8 in ask_channel
#9 in lttng_ustconsumer_recv_cmd
#10 in lttng_consumer_recv_cmd
#11 in consumer_thread_sessiond_poll
#12 in start_thread nptl/pthread_create.c:481
Thread T7 created by T0 here:
#0 in __interceptor_pthread_create
#1 in main
#2 in __libc_start_main ../csu/libc-start.c:332
SUMMARY: AddressSanitizer: heap-use-after-free in metadata_bucket_destroy
This can be easily reproduced by forcing a failure during the setup
of the metadata reproducible using the following change:
diff --git a/src/common/ust-consumer/ust-consumer.c b/src/common/ust-consumer/ust-consumer.c
index
fa1c71299..
97ed59632 100644
--- a/src/common/ust-consumer/ust-consumer.c
+++ b/src/common/ust-consumer/ust-consumer.c
@@ -908,8 +908,7 @@ static int setup_metadata(struct lttng_consumer_local_data *ctx, uint64_t key)
/* Send metadata stream to relayd if needed. */
if (metadata->metadata_stream->net_seq_idx != (uint64_t) -1ULL) {
- ret = consumer_send_relayd_stream(metadata->metadata_stream,
- metadata->pathname);
+ ret = -1;
if (ret < 0) {
ret = LTTCOMM_CONSUMERD_ERROR_METADATA;
goto error;
Cause
=====
Channels have a list of streams that are being "setup" and are not
yet monitored for consumption. During this setup phase, the streams are
owned by the channel. On destruction of the channel, any stream in that
list will thus be cleaned-up.
When destroying a consumer stream, a reference to its channel is 'put'.
This can result in the destruction of the channel.
In the situation described above, the release of the channel's reference
is done before the stream is removed from the channel's stream list.
This causes the channel's clean-up to invoke (again) the current
stream's clean-up, resulting in the double-free of the metadata bucket.
This problem is present in a number of error paths.
Solution
========
Some error paths already manually removed the consumer stream from it's
channel's stream list before invoking consumer_stream_destroy(). The
various error paths that have to deal with this possible situation are
changed to simply invoke consumer_stream_destroy().
consumer_stream_destroy() is modified to always remove the stream from
its channel's list before performing the rest of the clean-up. This
ensures that those double clean-ups can't occur.
Drawbacks
=========
None.
Reported-by: Vincent Whitchurch <vincent.whitchurch@axis.com>
Tested-by: Vincent Whitchurch <vincent.whitchurch@axis.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: Ibeca9b675b86fc46be3f57826f7158de4da43df8
Jérémie Galarneau [Thu, 3 Mar 2022 22:52:33 +0000 (17:52 -0500)]
Fix: ust-consumerd: leak of stream control structure
The following leak is reported by LeakSanitizer when
setup_metadata() fails to send the metadata stream to the relay
daemon:
==
3050181==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 240 byte(s) in 5 object(s) allocated from:
#0 0x7f5fce02cfb9 in __interceptor_calloc /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:154
#1 0x7f5fcdd95a7a in zmalloc ../../../src/common/macros.h:23
#2 0x7f5fcdd95a7a in lttng_ust_ctl_create_stream /home/jgalar/EfficiOS/src/lttng-ust/src/lib/lttng-ust-ctl/ustctl.c:1649
A consumer stream can have an allocated
`struct lttng_ust_ctl_consumer_stream *` (ustream) even if it is
not globally visible at the time of its teardown.
In the case of the user space consumer, the only site that creates
consumer stream instances ensures that the allocation of the
lttng_ust_ctl_consumer_stream succeeded, ensuring that the
consumer stream's 'ustream' is always set.
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: Ia4be7969e85bd8812ae13b042e1e100812a63c1d
Jérémie Galarneau [Fri, 4 Mar 2022 20:29:12 +0000 (15:29 -0500)]
Fix: liblttng-ctl: erroneous flat size computation
compute_flattened_size() erroneously computes (over-estimates) the size
of the allocation required to hold the flat array of struct lttng_event
returned to the user by lttng_list_{events, syscalls, tracepoints}.
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I0a80ef0fa66428b7df34303804a024e80b635c69
Jonathan Rajotte [Thu, 3 Mar 2022 23:26:35 +0000 (18:26 -0500)]
Fix: test: babeltrace1 python bindings exposes `op_enum` as a string
Note that `babeltrace2` is the "default" reader listed in the dependency
section of the readme but `babeltrace` is the actual reader used by the
test suite.
Change-Id: I5b47ba1e37a2671560f51ac866a7a35095be2338
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Jérémie Galarneau [Mon, 7 Mar 2022 20:17:53 +0000 (15:17 -0500)]
Build fix: relayd: mismatching function signatures
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I4ba2e7cacbe4357ad0cdbe1dc2c85894f8bedc27
Jérémie Galarneau [Mon, 7 Mar 2022 20:12:35 +0000 (15:12 -0500)]
Build fix: use of mutable keyword in C file
4045de280 is a backport of a fix against a C++ file in which `mutable`
is used to allow an ASSERT_LOCKED check. Remove the use of mutable and
make session_has_ongoing_rotation() non-const.
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: Id6317ad5c271f08701c9545827f75010f5e2e55d
Jérémie Galarneau [Wed, 2 Mar 2022 17:59:17 +0000 (12:59 -0500)]
Fix: relayd: connection abruptly closed on viewer stream creation failure
Commit
fe88e5175 explains (and fixes) an issue that could cause the
creation of viewer streams to fail. Currently, the error path causes the
relay daemon to abruptly close the connection to its live viewer peer.
This behaviour makes it impossible for the viewer to determine if an
error occurred or if the network connection simply failed.
Returning an `LTTNG_VIEWER_NEW_STREAMS_ERR` status code allows the
viewer to report a precise error. The viewer connection is closed since
the internal error is unlikely to be recoverable.
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I557a8b66c1bd1c0bf361cfbabe962d8a6808f4f4
Jérémie Galarneau [Wed, 2 Mar 2022 17:37:39 +0000 (12:37 -0500)]
Fix: relayd: live client fails on clear of multi-domain session
Observed issue
==============
Two test cases of the clear/test_ust test suite occasionally fail in the
integration jobs testing cross-bitness (32/64) LTTng deployments.
Babeltrace fails with the following error when a clear occurs while a
live client consumes a trace:
02-28 16:55:03.262 32362 32362 E PLUGIN/SRC.CTF.LTTNG-LIVE/VIEWER lttng_live_recv@viewer-connection.c:198 [lttng-live] Remote side has closed connection
02-28 16:55:03.262 32362 32362 E PLUGIN/SRC.CTF.LTTNG-LIVE/VIEWER lttng_live_session_get_new_streams@viewer-connection.c:1706 [lttng-live] Error receiving get new streams reply
02-28 16:55:03.262 32362 32362 E PLUGIN/SRC.CTF.LTTNG-LIVE lttng_live_msg_iter_next@lttng-live.c:1665 [lttng-live] Error preparing the next batch of messages: live-iter-status=LTTNG_LIVE_ITERATOR_STATUS_ERROR
02-28 16:55:03.262 32362 32362 W LIB/MSG-ITER bt_message_iterator_next@iterator.c:864 Component input port message iterator's "next" method failed: iter-addr=0x55eab7eb1170, iter-upstream-comp-name="lttng-live", iter-upstream-comp-log-level=WARNING, iter-upstream-comp-class-type=SOURCE, iter-upstream-comp-class-name="lttng-live", iter-upstream-comp-class-partial-descr="Connect to an LTTng relay daemon", iter-upstream-port-type=OUTPUT, iter-upstream-port-name="out", status=ERROR
02-28 16:55:03.262 32362 32362 E PLUGIN/FLT.UTILS.MUXER muxer_upstream_msg_iter_next@muxer.c:454 [muxer] Upstream iterator's next method returned an error: status=ERROR
02-28 16:55:03.262 32362 32362 E PLUGIN/FLT.UTILS.MUXER validate_muxer_upstream_msg_iters@muxer.c:991 [muxer] Cannot validate muxer's upstream message iterator wrapper: muxer-msg-iter-addr=0x55eab7eb1120, muxer-upstream-msg-iter-wrap-addr=0x55eab7eb3a70
02-28 16:55:03.262 32362 32362 E PLUGIN/FLT.UTILS.MUXER muxer_msg_iter_next@muxer.c:1415 [muxer] Cannot get next message: comp-addr=0x55eab7eb0470, muxer-comp-addr=0x55eab7eb0510, muxer-msg-iter-addr=0x55eab7eb1120, msg-iter-addr=0x55eab7eb0fb0, status=ERROR
02-28 16:55:03.262 32362 32362 W LIB/MSG-ITER bt_message_iterator_next@iterator.c:864 Component input port message iterator's "next" method failed: iter-addr=0x55eab7eb0fb0, iter-upstream-comp-name="muxer", iter-upstream-comp-log-level=WARNING, iter-upstream-comp-class-type=FILTER, iter-upstream-comp-class-name="muxer", iter-upstream-comp-class-partial-descr="Sort messages from multiple inpu", iter-upstream-port-type=OUTPUT, iter-upstream-port-name="out", status=ERROR
02-28 16:55:03.262 32362 32362 W LIB/GRAPH consume_graph_sink@graph.c:473 Component's "consume" method failed: status=ERROR, comp-addr=0x55eab7eb0760, comp-name="pretty", comp-log-level=WARNING, comp-class-type=SINK, comp-class-name="pretty", comp-class-partial-descr="Pretty-print messages (`text` fo", comp-class-is-frozen=1, comp-class-so-handle-addr=0x55eab7ebd910, comp-class-so-handle-path="/root/workspace/joraj_integration_base_job/deps-64/build/lib/babeltrace2/plugins/babeltrace-plugin-text.so", comp-input-port-count=1, comp-output-port-count=0
02-28 16:55:03.262 32362 32362 E CLI cmd_run@babeltrace2.c:2548 Graph failed to complete successfully
ERROR: [Babeltrace CLI] (babeltrace2.c:2548)
Graph failed to complete successfully
CAUSED BY [libbabeltrace2] (graph.c:473)
Component's "consume" method failed: status=ERROR, comp-addr=0x55eab7eb0760,
comp-name="pretty", comp-log-level=WARNING, comp-class-type=SINK,
comp-class-name="pretty", comp-class-partial-descr="Pretty-print messages
(`text` fo", comp-class-is-frozen=1, comp-class-so-handle-addr=0x55eab7ebd910,
comp-class-so-handle-path="/root/workspace/joraj_integration_base_job/deps-64/build/lib/babeltrace2/plugins/babeltrace-plugin-text.so",
comp-input-port-count=1, comp-output-port-count=0
CAUSED BY [libbabeltrace2] (iterator.c:864)
Component input port message iterator's "next" method failed:
iter-addr=0x55eab7eb0fb0, iter-upstream-comp-name="muxer",
iter-upstream-comp-log-level=WARNING, iter-upstream-comp-class-type=FILTER,
iter-upstream-comp-class-name="muxer",
iter-upstream-comp-class-partial-descr="Sort messages from multiple inpu",
iter-upstream-port-type=OUTPUT, iter-upstream-port-name="out", status=ERROR
CAUSED BY [muxer: 'filter.utils.muxer'] (muxer.c:991)
Cannot validate muxer's upstream message iterator wrapper:
muxer-msg-iter-addr=0x55eab7eb1120,
muxer-upstream-msg-iter-wrap-addr=0x55eab7eb3a70
CAUSED BY [muxer: 'filter.utils.muxer'] (muxer.c:454)
Upstream iterator's next method returned an error: status=ERROR
CAUSED BY [libbabeltrace2] (iterator.c:864)
Component input port message iterator's "next" method failed:
iter-addr=0x55eab7eb1170, iter-upstream-comp-name="lttng-live",
iter-upstream-comp-log-level=WARNING, iter-upstream-comp-class-type=SOURCE,
iter-upstream-comp-class-name="lttng-live",
iter-upstream-comp-class-partial-descr="Connect to an LTTng relay daemon",
iter-upstream-port-type=OUTPUT, iter-upstream-port-name="out", status=ERROR
CAUSED BY [lttng-live: 'source.ctf.lttng-live'] (lttng-live.c:1665)
Error preparing the next batch of messages:
live-iter-status=LTTNG_LIVE_ITERATOR_STATUS_ERROR
CAUSED BY [lttng-live: 'source.ctf.lttng-live'] (viewer-connection.c:1706)
Error receiving get new streams reply
CAUSED BY [lttng-live: 'source.ctf.lttng-live'] (viewer-connection.c:198)
Remote side has closed connection
Looking at the relay daemon logs, we see the following error:
DBG1 - 16:55:03.
262106718 [32139/32146]: Adding new file "ust/pid/gen-ust-events-32373-
20220228-165503/chan_0" to trace chunk "(unnamed)" (in lttng_trace_chunk_add_file() at trace-chunk.cpp:1310)
PERROR - 16:55:03.
262133333 [32139/32146]: Failed to open fs handle to ust/pid/gen-ust-events-32373-
20220228-165503/chan_0, open() returned: No such file or directory (in fd_tracker_open_fs_handle() at fd-tracker/fd-tracker.cpp:548)
Cause
=====
Adding more debugging logging allows us to see the following situation
takes place:
- relay thread: Create trace chunk on session 1.
- live thread: get new streams against session 1, returns NO_NEW_STREAMS
since the session has an 'ongoing_rotation'.
- live thread: get new streams against session 2, sees no rotation
ongoing and attempts to open `chan_0` when creating a viewer stream
The "ongoing rotation" check was introduced in
a7ceb342d and, in a
nutshell, prevents live viewers from creating new viewer streams during
a rotation.
The "ongoing rotation" state is entered when a CREATE_NEW_TRACE_CHUNK
command is issued against a session.
However, this presumes that a relay_session maps 1:1 to a session on the
session daemon's end. This isn't the case as, in multi-domain
scenarios (tracing 32-bit, 64-bit, and kernel events), a single session
daemon session can map to multiple relay_session objects. This is
because the consumer daemons maintain independant connections to the
relay daemon.
To synchronize rotations accross related relay_session instances, the
relay daemon uses the same trace chunk instances accross relay_session
instances. This means that while a trace chunk is created against a
specific relay session, it can be used by other relay_session instances.
To manage shared trace chunks between relay_sessions, the relay daemon
makes use of the trace_chunk_registry. This registry allows
relay_sessions to share trace chunk instances using a unique key tuple:
- session daemon instance uuid,
- session daemon session id,
- trace chunk id.
There is no equivalent mechanism to track the "ongoing_rotation" state
accross relay_sessions originating from the same sessiond session.
In the current scenario, this causes the live client to correctly see
that no new streams are available for session 1 (say, the 32-bit user
space session). Unfortunately, this state is not entered for other
sessions (64-bit and kernel relay sessions). Hence, the viewer succeds
in acquiring new streams from session 2, exposing the race the 'ongoing
rotation' state aims to protect against.
Solution
========
Like the trace chunk instances, the "ongoing rotation" state must be
shared accross relay sessions that originate from the same session
daemon session.
To "emulate" this shared state, session_has_ongoing_rotation() checks
if any relay session originating from the same sessiond session
have an ongoing rotation. If it is the case, we temporarily prevent
live viewers from acquiring new streams.
Known drawbacks
===============
session_has_ongoing_rotation() iterates over all sessions, acquiring
their lock in the process, which is certainly undesirable from a
performance standpoint.
Optimizing this is not a great challenge, but is beyond the scope
of this immediate fix.
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I457a32fa497d151ca933c25433c80665268a7c1c
Michael Jeanson [Thu, 5 Aug 2021 20:49:26 +0000 (16:49 -0400)]
fix: msgpack requires limits.h for UINT_MAX
Building with '-Wundef' reveals this issue :
unpack.c: In function ‘template_callback_array’:
unpack.c:197:17: warning: "UINT_MAX" is not defined, evaluates to 0 [-Wundef]
197 | #if SIZE_MAX == UINT_MAX
| ^~~~~~~~
unpack.c: In function ‘template_callback_map’:
unpack.c:241:17: warning: "UINT_MAX" is not defined, evaluates to 0 [-Wundef]
241 | #if SIZE_MAX == UINT_MAX
| ^~~~~~~~
Change-Id: I7dadd9f7013d613509f66e67ff1beb8ae593d2bf
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Jérémie Galarneau [Thu, 3 Mar 2022 21:43:08 +0000 (16:43 -0500)]
Fix: liblttng-ctl: hide symbols introduced by packing fixes
abidiff reports the following when comparing the ABIs of
liblttng-ctl of v2.13.0 to the stable-2.13 branch.
12 Added functions:
[A] 'function ssize_t lttng_event_context_create_from_payload(lttng_payload_view*, lttng_event_context**)' {lttng_event_context_create_from_payload}
[A] 'function void lttng_event_context_destroy(lttng_event_context*)' {lttng_event_context_destroy}
[A] 'function int lttng_event_context_serialize(lttng_event_context*, lttng_payload*)' {lttng_event_context_serialize}
[A] 'function ssize_t lttng_event_create_from_payload(lttng_payload_view*, lttng_event**, lttng_event_exclusion**, char**, lttng_bytecode**)' {lttng_event_create_from_payload}
[A] 'function ssize_t lttng_event_field_create_from_payload(lttng_payload_view*, lttng_event_field**)' {lttng_event_field_create_from_payload}
[A] 'function int lttng_event_field_serialize(const lttng_event_field*, lttng_payload*)' {lttng_event_field_serialize}
[A] 'function lttng_error_code lttng_event_fields_create_and_flatten_from_payload(lttng_payload_view*, unsigned int, lttng_event_field**)' {lttng_event_fields_create_and_flatten_from_payload}
[A] 'function int lttng_event_serialize(const lttng_event*, unsigned int, char**, char*, size_t, lttng_bytecode*, lttng_payload*)' {lttng_event_serialize}
[A] 'function lttng_error_code lttng_events_create_and_flatten_from_payload(lttng_payload_view*, unsigned int, lttng_event**)' {lttng_events_create_and_flatten_from_payload}
[A] 'function lttng_rate_policy_status lttng_rate_policy_every_n_get_interval(const lttng_rate_policy*, uint64_t*)' {lttng_rate_policy_every_n_get_interval}
[A] 'function lttng_rate_policy_status lttng_rate_policy_once_after_n_get_threshold(const lttng_rate_policy*, uint64_t*)' {lttng_rate_policy_once_after_n_get_threshold}
[A] 'function lttng_session_descriptor_status lttng_session_descriptor_get_session_name(const lttng_session_descriptor*, const char**)' {lttng_session_descriptor_get_session_name}
The packing fix patch series exposes a number of internal symbols
through liblttng-ctl. Mark those functions as hidden.
Note that the three last functions of this report were correctly
exposed by fixes following the 2.13.0 release.
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: Idf98f9d22388f945913c801a3da966167e7c3f3f
Jonathan Rajotte [Mon, 14 Feb 2022 16:23:28 +0000 (11:23 -0500)]
Fix: rotation: hang on destroy when using scheduled rotation based on timer
Observed issue
==============
The following scenario results in a hang for `lttng destroy`:
lttng create test
lttng enable-event -u -a
lttng enable-rotation --timer 100000
lttng start
lttng stop
lttng start
lttng destroy
Cause
=====
There is an imbalance in how many times we start the rotation timer.
The rotation timer is only removed on `lttng destroy` or when disabling
a time-based-rotation. On the other hand, the timer is "started"
on `lttng start` and when enabling a time based rotation.
The imbalance emerging from a start/stop/start sequence would prevent the
teardown of the session object since each time the timer is started a
reference to the session is held.
Solution
========
Do not start the rotation schedule timer if it was already launched.
Known drawbacks
=========
None.
Change-Id: Ic5b8938166358fe7629187bebdf02a09e90846c0
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Jonathan Rajotte [Wed, 9 Feb 2022 19:23:18 +0000 (14:23 -0500)]
Fix: consumerd: fd still open after `lttng snapshot record` returns
Observed issue
=====
Using a snapshot output located on a pramfs mount:
lttng snapshot record
rm -rf /my_mount/my_trace_output
`rm` fails with ENOTEMPTY on rmdir for /my_mount/my_trace_output.
At that point, the lttng-consumerd daemon have an open fd on:
/my_mount/my_trace_output/ust
Note that a sleep between both command "fixes" the issue.
Cause
=====
The reclaim for the in-registry trace chunks can happen after the LTTng
CLI returns since we use `call_rcu`.
```
static
void lttng_trace_chunk_release(struct urcu_ref *ref)
....
if (chunk->in_registry_element) {
struct lttng_trace_chunk_registry_element *element;
element = container_of(chunk, typeof(*element), chunk);
if (element->registry) {
rcu_read_lock();
cds_lfht_del(element->registry->ht, &element->trace_chunk_registry_ht_node);
rcu_read_unlock();
-> call_rcu(&element->rcu_node, free_lttng_trace_chunk_registry_element);
} else {
```
The delayed reclaim of the `lttng_trace_chunk_registry_element` can
result in lttng-consumerd holding an open fd for the "chunk directory"
of the chunk since the close() is only done during the "*fini" phase of
the chunk (`lttng_trace_chunk_fini`).
Solution
========
Considering that the rcu lookup+refcount access scheme is used for the
trace chunk object and that at that point the refcount for the trace
chunk object is effectively zero, we can move the
`lttng_trace_chunk_fini` safely outside of the
`free_lttng_trace_chunk_registry_element` call_rcu call.
Known drawbacks
=========
Even if this solves the current situation, it is important to note that
the actual object holding the reference is itself refcounted and only
close the fd on release. This means that we are still exposed to this
problem if at some point the directory handle is shared and outlives the
trace chunk for some reason in the future.
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I6da3948824bf8b092fc8248b1bb0263fdd5887be
Jonathan Rajotte [Wed, 2 Mar 2022 15:50:16 +0000 (10:50 -0500)]
Fix: event: unchecked return value for lttng_strncpy
Reported by Coverity:
CID
1475806: Unchecked return value (CHECKED_RETURN)
Calling lttng_strncpy without checking return value.
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: Ib2b16bef61bcc24fea381c18fd8f914e9a84a6e7
Jonathan Rajotte [Wed, 2 Mar 2022 15:36:11 +0000 (10:36 -0500)]
Fix: event: format specifier for ssize_t
Observed issue
==============
In file included from event.cpp:15:
event.cpp: In function ‘ssize_t lttng_event_create_from_payload(lttng_payload_view*, lttng_event**, lttng_event_exclusion**, char**, lttng_bytecode**)’:
../../src/common/error.h:191:28: warning: format ‘%lu’ expects argument of type ‘long unsigned int’, but argument 4 has type ‘ssize_t’ {aka ‘int’} [-Wformat=]
191 | __lttng_print(PRINT_WARN, "Warning: " fmt "\n", ## args)
| ^~~~~~~~~~~~~~~~~~~~
../../src/common/error.h:139:51: note: in definition of macro ‘__lttng_print’
139 | fprintf((type) == PRINT_MSG ? stdout : stderr, fmt, ## args); \
| ^~~
event.cpp:624:4: note: in expansion of macro ‘WARN’
624 | WARN("Userspace probe location from the received buffer is not the advertised length: header length = %" PRIu32 ", payload length = %lu", event_comm->userspace_probe_location_len, ret);
| ^~~~
Solution
========
Albeit there is no "canonical" way of printing ssize_t, use '%zd' since
we already make use of it elsewhere.
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: Id41e6ccf07bd580813f169b65d281a4fa305fb48
Jonathan Rajotte [Wed, 2 Mar 2022 15:24:29 +0000 (10:24 -0500)]
Fix: sessiond: use after free
Introduced by
88b02a8e7cbdbc728cc681cd07bbe0c738964717
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I9b1c3eeed7e51976af1a4982fc522066b3b29461
Jonathan Rajotte [Tue, 1 Mar 2022 15:35:44 +0000 (10:35 -0500)]
Fix: sessiond: `break` used instead of `goto`
On error path, use `goto`.
Reported by Coverity:
CID
1475805: Unused value (UNUSED_VALUE)
Assigning value LTTNG_ERR_UNK to ret_code here, but that stored value is overwritten before it can be used.
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: Ife3b5486b8463242dfecb61a3a9718e8eb7149b4
Jonathan Rajotte [Tue, 1 Mar 2022 15:31:33 +0000 (10:31 -0500)]
Fix: sessiond: lttng_channel object is not reclaimed
The reclaim is also missing on the error path.
Reported by Coverity:
CID
1475807: Resource leak (RESOURCE_LEAK)
Variable channel going out of scope leaks the storage it points to.
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I9ae59900f6bc33282b8b8a65b834a718428a7f50
Jonathan Rajotte [Tue, 1 Mar 2022 15:27:37 +0000 (10:27 -0500)]
Fix: common: local_attr might leak
Reported by Coverity:
CID
1475808: Resource leak (RESOURCE_LEAK)
Variable local_attr going out of scope leaks the storage it points to.
CID
1475810: Resource leak (RESOURCE_LEAK)
Variable local_attr going out of scope leaks the storage it points to.
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: Ibe2ee6516618827b6f52676d9a9f7d510cadf1bc
Jonathan Rajotte [Tue, 1 Mar 2022 15:23:24 +0000 (10:23 -0500)]
Fix: session: end goto label used for error path
Reported by Coverity:
CID
1475801: Unused value (UNUSED_VALUE)
Assigning value LTTNG_ERR_FATAL to ret_code here, but that stored value is overwritten before it can be used
CID
1475802: Unused value (UNUSED_VALUE)
Assigning value LTTNG_ERR_NOMEM to ret_code here, but that stored value is overwritten before it can be used.
CID
1475814: Unused value (UNUSED_VALUE)
Assigning value LTTNG_ERR_UST_CHAN_NOT_FOUND to ret_code here, but that stored value is overwritten before it can be used.
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I79fb5d65ea0f4f3dfbbef2a42ec3dcf0542043c5
Jonathan Rajotte [Tue, 1 Mar 2022 15:18:46 +0000 (10:18 -0500)]
Fix: sessiond: event, filter expression, bytecode, exclusions might leak
Reported by Coverity:
CID
1475803: Resource leak (RESOURCE_LEAK)
Variable filter_expression going out of scope leaks the storage it points to.
CID
1475816: Resource leak (RESOURCE_LEAK)
Variable exclusions going out of scope leaks the storage it points to.
CID
1475817: Resource leak (RESOURCE_LEAK)
Variable bytecode going out of scope leaks the storage it points to.
CID
1475820: Resource leak (RESOURCE_LEAK)
Variable event going out of scope leaks the storage it points to.
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I705e3cc1a1894af52d6ddeee94798a887dc82bba
Jonathan Rajotte [Tue, 1 Mar 2022 14:52:21 +0000 (09:52 -0500)]
Fix: lttng-sessiond: event_context might leak
If the parsing completed but is not the expected size, the
*out_event_context object leaks.
Reported by Coverity:
CID
1475821 (#1 of 1): Resource leak (RESOURCE_LEAK)
Variable event_context going out of scope leaks the storage it points to.
Change-Id: I8198f7f3f1e1af3dcb377d89674402ec49cb788b
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Jonathan Rajotte [Tue, 1 Mar 2022 14:09:04 +0000 (09:09 -0500)]
Fix: lttng-ctl: uninitialized pointer read
Reported by Coverity:
CID
1475822: Uninitialized pointer read (UNINIT)
Using uninitialized value reply._fd_handles.array.size when calling lttng_payload_reset.
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: Ied7f5de61d736e8d8f9c869a36322262a4d79cd5
Jonathan Rajotte [Tue, 1 Mar 2022 14:50:06 +0000 (09:50 -0500)]
Fix: ser/des: missing null terminator on payload append
Reported by the lttng-ust-java-tests_master_build CI job.
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I47b048c65a9855347c9b02fe44d9fdbae7e06dbf
Jonathan Rajotte [Tue, 1 Mar 2022 17:33:53 +0000 (12:33 -0500)]
Fix: test: test plan/skip must come after TAP initialization
TAP is initialized during the sourcing of `utils.sh`. Calling it before
yield nothing.
Move the platform validation after the `PLAN_TEST` call.
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: Ib7b4434440ed7590b39b0580ea294045ace40495
This page took 0.061618 seconds and 4 git commands to generate.
projects / lttng-tools.git / log
