Fix: notification-thread: handling event from a removed tracer event src
authorFrancis Deslauriers <francis.deslauriers@efficios.com>
Mon, 27 Sep 2021 13:42:54 +0000 (09:42 -0400)
committerJérémie Galarneau <jeremie.galarneau@efficios.com>
Mon, 18 Oct 2021 18:58:29 +0000 (14:58 -0400)
Issue
=====
The issue is caused by a race condition where the `lttng_poll_wait()`
returns a _REMOVE_TRACER_EVENT_SOURCE event followed by an actual
notification event on the removed event source fd.

This causes the notification thread to remove the fd from the potential
notification sources list and later fail to find that same fd in the
next iteration.

This race condition can lead to the notification thread to hang
indefinitely or to failed assertions within the `fini_thread_state()`
function.

Fix
===
When removing an tracer event source, force the notification thread
`lttng_poll_wait()` loop to restart to ignore events from the removed
fd.

Use the `restart_poll` for that purpose (see note below).

Reproducer
==========
It's easy to reproduce this issue by adding a `usleep(5000)` just before
the `lttng_poll_wait()` call in the notification thread.

Note
====
It's the second time that I fix this issue.

It was first fixed by this commit by adding the `restart_poll` flag:
  commit 8b5240601e4ddf6127e4291b7194dd5179cb35b5
  Author: Francis Deslauriers <francis.deslauriers@efficios.com>
  Date:   Thu Dec 10 15:41:29 2020 -0500

    notification-thread: drain all tracer notification on removal

and later, that other commit refactored that code but accidently removed
the use of the `restart_poll`:
  commit 34bf4f69e49d8a69331a6aa6826ef1f155e20ede
  Author: Francis Deslauriers <francis.deslauriers@efficios.com>
  Date:   Wed May 26 16:05:16 2021 -0400

    notification-thread: remove fd from pollset on LPOLLHUP and friends

Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I6da0ed4374b612934adc72fb88d5c142505c5d53

src/bin/lttng-sessiond/notification-thread-events.c

index 747740233094477c00fb59065cce2dd3f9861013..1388b3ce34a0f743e3475af910e692502a4592ab 100644 (file)
@@ -2153,6 +2153,12 @@ int remove_tracer_event_source_from_pollset(
 
        source_element->is_fd_in_poll_set = false;
 
+       /*
+        * Force the notification thread to restart the poll() loop to ensure
+        * that any events from the removed fd are removed.
+        */
+       state->restart_poll = true;
+
        ret = drain_event_notifier_notification_pipe(state, source_element->fd,
                        source_element->domain);
        if (ret) {
This page took 0.027833 seconds and 4 git commands to generate.