Fix: liblttng-ust-fork deadlock

Fix: liblttng-ust-fork deadlock

* Mathieu Desnoyers (mathieu.desnoyers@efficios.com) wrote:
> * Burton, Michael (mburton@ciena.com) wrote:
> > Mathieu,
> >
> > I think there is a deadlock scenario in UST, which has been causing my problem.
>
> Good catch !
>
> >
> > sessiond is started as root:
> >     - creates global sockets ONLY
> >     - DOES NOT CREATE shm in $HOME/lttng-ust-wait-<uid>
> >
> > application linked against ust is run as root:
> >   - in lttng_ust_init constructor
> >   - ust_listener_thread  (local_apps)
> >     - fails to connect to local_apps in $HOME/.lttng (as expected)
> >     - prev_connect_failed=1
> >     - ust_unlock()
> >     - restart
> >     - wait_for_sessiond()
> >  -->  - ust_lock()
> >  |      - get_map_shm()
> >  |        - get_wait_shm()
> > DEADLOCK    - shm_open() FAILS (not created by sessiond when run by root)
> >  |            - fork() (trying to create shared memory itself)
> >  |              - ust_before_fork()
> >  ------------>    - ust_lock()
> >
> >
> > You should be able to create this with an empty main, with no
> > tracepoints.  As long as sessiond is started as root so
> > $HOME/lttng-ust-wait-<uid> is not created.  You can also make the
> > lttng-ust constructor (lttng_ust_init) wait forever and then you'll be
> > able to see the deadlock in gdb without even leaving the
> > lttng_ust_init constructor.
>
> Ah, I see. This deadlock is caused by the interaction between
> [ liblttng-ust-fork ] and liblttng-ust (the fork override is
> performed by [ liblttng-ust-fork ]).

This can be reproduced easily with the in-tree tests: by removing the
lttng-ust-apps-wait* files belonging to the user in /dev/shm, running
the "tests/fork" test (with ./run) hangs. If we run "hello" first, and
then the fork test, it works fine.

Fixing this by keeping a nesting counter around the fork() call, so we
return immediately from the pre/post fork handlers if they are
overridden by liblttng-ust-fork.

Reported-by: Michael Burton <mburton@ciena.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>