Fix: ambiguous ownership of kernel context by multiple channels
A kernel context, when added to multiple channels, must be copied
before being added to individual channels. The current code
adds the same ltt_kernel_context structure to multiple kernel
channels which introduces a conceptual ambiguity in the ownership
of the context object.
Concretely, creating multiple kernel channels and adding a context
to all of them (by not specifying a channel name) causes the context
to be added to each channels' list of contexts, overwritting the
context's list node, and causing the channel context lists to become
corrupted. This results in crashes being observed during the
destruction of the session.
Fix: lost packet accounting always lost on snapshot
Because of the continue when we fail to get a subbuff, the lost_packet
count is always reset to 0 before we can account it in the channel. Now
we account it directly before the continue.
Jonathan Rajotte [Mon, 24 Jul 2017 20:07:00 +0000 (16:07 -0400)]
Fix live-comm: merge TCP socket write-write sequence in a single write
The live protocol implementation is often sending content
on TCP sockets in two separate writes. One to send a command header,
and the second one sending the command's payload. This was presumably
done under the assumption that it would not result in two separate
TCP packets being sent on the network (or that it would not matter).
Delayed ACK-induced delays were observed [1] on the second write of the
"write header, write payload" sequence and result in problematic
latency build-ups for live clients connected to moderately/highly
active sessions.
Fundamentaly, this problem arises due to the combination of Nagle's
algorithm and the delayed ACK mechanism which make write-write-read
sequences on TCP sockets problematic as near-constant latency is
expected when clients can keep-up with the event production rate.
In such a write-write-read sequence, the second write is held up until
the first write is acknowledged (TCP ACK). The solution implemented
by this patch bundles the writes into a single one [2].
[1] https://github.com/tbricks/wireshark-lttng-plugin
Basic Wireshark dissector for lttng-live by Anto Smyk from Itiviti
[2] https://lists.freebsd.org/pipermail/freebsd-net/2006-January/009527.html
Reported-by: Anton Smyk <anton.smyk@itiviti.com> Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com> Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Detaching the timer thread has the unfortunate side-effect of letting
the health management data structures be freed by main() while the timer
thread may still be using them (if, e.g., main() exits quickly).
Overcome this situation by tearing down and joining the timer thread.
For per-pid buffers, we need to sum the counters for each application.
For per-uid buffers, if no application has launched yet, it should not
be considered as an error (which stops iteration on all other channels),
but rather as values of 0.
A session teardown can be initiated by a dying application. Hence, a
session object can exist without a valid registry. As a result,
get_session_registry can return null. To prevent this, the UST
application session lock should be held, when possible, when looking up
the registry to ensure synchronization. Otherwise the presence of a
registry is not guaranteed. In such case, handling a null return value
from look-up registry function is necessary.
Core dumps, triggered by the "assert(registry)" statement found in
reply_ust_register_channel, were observed when killing instrumented
applications. In this occurrence, obtaining the UST application lock
result in a deadlock since the lock is already held during
ust_app_global_create. Handling the null value is simpler and
corresponds with the handling of previous look-up done during the
function.
Handling of null value is also applied to:
add_event_ust_registry
add_enum_ust_registry
ust_app_snapshot_record
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com> Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
When the flush empty ioctl is available, use it to produce an empty
packet at the end of the snapshot, which ensures the stream intersection
feature works.
If this specific ioctl is not available, fallback on the "flush" ioctl,
which does not produce empty packets.
In that situation, there were two prior behaviors possible for
lttng-modules: earlier versions implement a "snapshot" command which
does not perform an implicit "flush_empty". In that case, the stream
intersection feature may not be reliable. In more recent lttng-modules
versions (included stable branch) which did not implement the
flush_empty ioctl, the snapshot ioctl implicitly performed a
flush_empty, which makes the stream intersection feature work, but has
side-effects on the snapshot ioctl performed by the live timer (produces
a stream of empty packets in live mode).
[ Please apply to master, 2.10, 2.9, 2.8 branches. ]
Fix: lttng-consumerd: cpu hotplug: send "streams_sent" command
When creating a new channel, the streams being sent to the relayd are
kept invisible to the live client until the "streams_sent" command is
received. This ensures the client does not see a partial stream set.
This "streams_sent" command needs to be sent on CPU hotplug too,
otherwise the live client handling within relayd is not aware of those
streams (they are never published).
Fix: lttng-sessiond: cpu hotplug: send channel to consumer only once
On CPU hotplug, we currently send a duplicate of the channel key, which
allocates its own object (duplicated) within the consumerd. We want the
newly added stream to map to the pre-existing channel key, so don't send
the channel duplicate.
Fix: lttng-sessiond: cpu hotplug stream number mismatch
The counter should be always increasing (kept in the channel), rather
than local to the function. This causes cpu hotplug handling to
disregard further streams that should be added to the consumer output
on CPU hotplug.
Fix: COMPAT_EPOLL_PROC_PATH is available from Linux 2.6.28
v2: Typo in commit message "per see" -> "per se"
Failing on opening [1] is not an error per se. [1] was
introduced in Linux 2.6.28 but epoll is available since
2.5.44. Hence, goto end and set a default value without
setting error return value.
[1] /proc/sys/fs/epoll/max_user_watches
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com> Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Jonathan Rajotte [Mon, 24 Apr 2017 19:32:15 +0000 (15:32 -0400)]
Fix: fail on relayd lookup when finding a relayd is expected
An actual relayd lookup error leads to using the code path of a local
handling. Since stream->index_file is NULL when expecting a relayd, using
the code path for local handling results in an invalid access.
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com> Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Jonathan Rajotte [Fri, 13 Jan 2017 22:04:42 +0000 (17:04 -0500)]
Man: move [SESSION] before options
The previous synopses for the live mode can cause confusion to users
since it can lead to an error while trying one of the simplest create
command for live session that the synopsis is proposing:
lttng create --live test.
Other synopsis are modified for symmetry.
Fixes #1081
Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com> Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Session daemon should not send streams to consumer daemon
repeatedly when CPU hotplug is performed while doing kernel
tracing.
This causes the consumer daemon to have multiple file descriptors
on the same stream, and thus try to perform operations like reading
a sub-buffer and checking for data pending concurrently. This triggers
safety-net warnings in the kernel tracer.
The failure/exit of any of the consumerd, relayd or applications
(in per-PID buffer mode) will cause the metadata closed flag to
be set.
While pushing new metadata updates to the consumerd (and relayd
in streaming/live scenarios) will fail, those conditions should
be handled in-place.
Applications are _expected_ to exit during the course of a per-PID
session. However, they will typically have pushed their metadata
to the metadata cache before doing so. The session daemon must
flush the unconsumed metadata to the consumerd in this case.
Failure to answer to the metadata request originating from the
consumerd can cause it to keep the stream lock held and, thus,
prevent the channel poll thread from cleaning up on channel
close.
Fix: consumerd: order of metadata cache vs stream lock
The locking order comment in consumer.h is incorrect. First, its
description of locking order is not in sync with the comment found in
consumer-metadata-cache.h. The comment in struct consumer_metadata_cache
only states that the metadata cache lock nests inside the consumer_data
lock, and does not mention the stream lock, which implies that the
metadata cache lock does NOT nest inside the stream lock. But let's
investigate further to confirm:
* lttng_consumer_read_subbuffer() acquires the stream lock, and then
calls lttng_ustconsumer_read_subbuffer() with stream lock held,
and then invokes commin_one_metadata_packet(), which acquires the
metadata cache lock.
* lttng_ustconsumer_sync_metadata() acquires the metadata stream lock,
and calls commit_one_metadata_packet(), which takes the metadata cache
lock.
Therefore, update the comment in consumer.h to state that the metadata
cache lock nests INSIDE the stream lock, and update
consumer_del_metadata_stream() accordingly.
This should take care of fixing the locking order reversal found by
Coverity.
CID 1368314 (#1 of 1): Thread deadlock (ORDER_REVERSAL)
CID 1368319: Program hangs (ORDER_REVERSAL)
Fixes: 5feafd4130 "Fix: protect the channel's metadata stream using the metadata cache lock" Fixes: 1ea6cc572b "Fix: lock nesting order reversed" Fixes: fb549e7ac2 "Fix: reverse channel and metadata cache lock nesting order" Reported-by: Coverity Scan Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Fix: add missing rcu_barrier before daemon teardown
When performing the "cleanup" of sessiond, consumerd, and relayd, we
destroy data structures that may still be concurrently accessed by
call_rcu worker thread.
Ensure no more work is present in the call_rcu worker thread by issuing
a rcu_barrier barrier. Note that this expects call_rcu handlers don't
chain work to other call_rcu handlers.
Fix: support for older versions of Babeltrace in test script
A new context field was introduced in version LTTng 2.8 that is printed
by Babeltrace prior to v1.2.5. This regex thus fails to match the
output. Since the context fields are not used by the script, we create a
non-capturing group for these fields that matches on both old and new
Babeltrace.
This is causing problems on Ubuntu 14.04 Trusty when building
lttng-tools from source and using the Babeltrace package from the
official repository (v1.2.1) to run the test suite.
Also, this patch removes commented and used code in the function but
keeps the names of non-capturing groups for readability.
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com> CC: Philippe Proulx <pproulx@efficios.com> Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Fix: only lock the metadata_cache in userspace consumers
The kernel consumer, which re-uses the consumer_del_metadata_stream
function, has no metadata cache. Therefore, it can't be used to
protect the metadata stream (see 5feafd41).
However, only the userspace consumers invoke
consumer_metadata_cache_write() which the previous fix seeked to
protect against. It is therefore safe to omit this lock in the
kernel consumer case.
Fix: protect the channel's metadata stream using the metadata cache lock
The consumer_thread_data_poll and consumer_thread_metadata_poll
both access the channel's metadata stream.
During a session destruction, consumer_thread_metadata_poll will
destroy all metadata streams. However, the consumer_thread_data_poll
may still invoke a consumer_metadata_cache_write() triggered
by a "ready" subbuffer. Hence, the metadata stream must be protected
from this action by the metadata cache lock.
relay and consumerd 2.7 and 2.8 are expected to negociate compatibility
with the lowest common minor version.
If a consumer daemon 2.8 interacts with a relayd 2.7, it needs to send
the index fields for ctf index 1.0. Same if a relayd 2.8 interacts with
a consumer daemon 2.7: relayd should expect ctf index 1.0 fields, and
generate a ctf index 1.0 index file layout.
If both relayd and consumerd versions are 2.8+, then we can send the ctf
index 1.1 fields over the protocol, and store them in the index files.
Whenever the relayd live viewer server opens and reads an index file,
it needs to use the file's header to figure out the index "element"
size.
[ Should be applied to master, stable-2.9, stable-2.8. ]
Liguang Li [Mon, 28 Nov 2016 08:37:47 +0000 (16:37 +0800)]
Fix: truncate the metadata file in shm-path
In the shm-path mode, the metadata will be backuped to a metadata
file, when run the lttng command "lttng metadata regenerate" to
resample the wall time following a major NTP correction, the metadata
file will not be truncated and regenerated.
Add the function clear_metadata_file() to truncate and regenerate the
metadata file.
Signed-off-by: Liguang Li <liguang.li@windriver.com> Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Philippe Proulx [Fri, 28 Oct 2016 22:33:19 +0000 (18:33 -0400)]
doc/man: only require asciidoc-attrs.conf when building the man pages
Situations:
* If you want to and can build the man pages:
* If it's a tarball tree:
* Make the man page destinations depend on asciidoc-attrs.conf.
Since it's a generated file, its date is greater than the
date of the prebuilt man pages, therefore the man pages are
built again, which is a good thing because they include the
default values of this build.
* If it's a Git tree:
* Always build the man pages anyway (no prebuilt man pages here).
* If you want to, but cannot build the man pages:
* If it's a tarball tree:
* Make the man page destinations NOT depend on asciidoc-attrs.conf,
because its recent date would ask said destinations to be rebuilt
and this is not possible because we don't have the tools.
However, warn the user at configure time that the prebuilt man
pages will be installed, which means that they will contain the
project's default values, not this build's default values.
* If it's a Git tree:
* Not valid: error at configure time as usual.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com> Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Fix: stop lttng-relayd threads on health thread error
The lttng-relayd health thread may fail to initialize for
a variety of reason (notably, a too long unix domain socket
address), which will cause it to never notify that it is
ready.
In such circumstances, the lttng-relayd command, in background or
daemonize mode, will never return as the daemon's "readyness"
will never be signaled.
Issuing fprintf() to stderr (thus write() to the standard error file
descriptor) within the SIGPIPE signal handler is bad: it can trigger
SIGPIPE repeatedly if the listening end has closed its end of the pipe.
Set the SIGPIPE action to SIG_IGN in relayd, sessiond, and consumerd.
This was affecting sessiond and relayd. The consumerd did not print
anything to stderr.
Fix: handle backward compatibility with lttng-modules 2.7
There is no major version bump between lttng-module 2.7 and 2.8 ABI.
Even though we do not guarantee compatibility, do a best effort to
maintain it when possible.
Tests: tap.sh spams tests' output when no plan is set
Some tests are implemented in C (using tap.h) or in Python
and don't use tap.sh's facilities. However, it is sourced
by utils.sh and prints an error message during its clean-up
because a plan was never set.
Use -M parameter instead of --manpath when invoking man(1)
Older versions of man (and the implementation used in FreeBSD) do
not support the long version of the --manpath/-M option. Use
'-M' in the interest of portability.
Fix: Mark ASCIIDOC_ATTRS_CONF as a dependency of man page targets
ASCIIDOC_ATTRS_CONF contains the various paths set by autoconf,
such as datadir, syscondif and prefix, and it may be changed
by the user by invoking ./configure with different options. In
such a case, the man pages should be regenerated to take the new
paths into account.
projects / lttng-tools.git / log
