Tests: Add test for byte/short atomics on addresses which are not word-aligned
Add a unit test to catch architectures which do not allow byte and short
atomic operations on addresses which are not word aligned.
If an architecture supports byte and short atomic operations, it should
be valid to issue those operations on variables which are not
word-aligned, otherwise the architecture should not define
UATOMIC_HAS_ATOMIC_BYTE nor UATOMIC_HAS_ATOMIC_SHORT.
This should help identify architectures which mistakenly define
UATOMIC_HAS_ATOMIC_BYTE and UATOMIC_HAS_ATOMIC_SHORT.
This commit completes removal of the urcu-signal flavor.
Users can migrate to liburcu-memb with a kernel implementing the
membarrier(2) system call to have similar read-side performance without
requiring use of a reserved signal, and with improved grace period
performance.
Fix: Add missing cmm_smp_mb() in deprecated urcu-signal
commit 97d13221f8a1 ("Phase 1 of deprecating liburcu-signal") miss a
cmm_smp_mb() at the beginning of the read-side critical sections, which
causes spurious failures in the CI tests.
Olivier Dion [Mon, 14 Aug 2023 20:40:30 +0000 (16:40 -0400)]
Phase 1 of deprecating liburcu-signal
The first phase of liburcu-signal deprecation consists of implementing
it in term of liburcu-mb. In other words, liburcu-signal is identical to
liburcu-mb at the exception of the function symbols and public header
files.
This is done by:
1) Removing the RCU_SIGNAL specific code in urcu.c
2) Making the RCU_MB specific code also specific to RCU_SIGNAL in
urcu.c
3) Rewriting _urcu_signal_read_unlock_update_and_wakeup to use a
atomic store with CMM_SEQ_CST instead of a store CMM_RELAXED with
cmm_barrier() around it. We could keep the explicit barriers, but that
would require to add some cmm_annotate annotations. Therefore, to be
less intrusive in a public header file, simply use the CMM_SEQ_CST
like for the mb flavor.
Change-Id: Ie406f7df2f47da0a9f464df94b968ad9204821f3 Signed-off-by: Olivier Dion <odion@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
abort(3) was explicitly declared external to avoid including
<stdlib.h>. However, this emit a redundant declaration warning if it was
already declared before including <urcu/uatomic.h>.
Fix this by including <stdlib.h> and not declaring abort().
Change-Id: If9557814c311e2b531e85fec8c41788462338fe4 Signed-off-by: Olivier Dion <odion@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Olivier Dion [Mon, 29 May 2023 15:21:11 +0000 (11:21 -0400)]
Add cmm_emit_legacy_smp_mb()
Some public APIs stipulate implicit memory barriers on operations. These
were coherent with the memory model used at that time. However, with the
migration to a memory model closer to the C11 memory model, these memory
barriers are not strictly emitted by the atomic operations in the new
memory model.
Therefore, introducing the `--disable-legacy-mb' configuration
option. By default, liburcu is configured to emit these legacy memory
barriers, thus keeping backward compatibility at the expense of slower
performances. However, users can opt-out by disabling the legacy memory
barriers.
This options is publicly exported in the system configuration header
file and can be overrode manually on a compilation unit basis by
defining `CONFIG_RCU_EMIT_LEGACY_MB' before including any liburcu files.
The usage of this macro requires to re-write atomic operations in term
of the CMM memory model. This is done for the queue and stack APIs.
Change-Id: Ia5ce3b3d8cd1955556ce96fa4408a63aa098a1a6 Signed-off-by: Olivier Dion <odion@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Olivier Dion [Fri, 31 Mar 2023 17:47:17 +0000 (13:47 -0400)]
urcu/annotate: Add CMM annotation
The CMM annotation is highly experimental and not meant to be used by
user for now, even though it is exposed in the public API since some
parts of the liburcu public API require those annotations.
The main primitive is the cmm_annotate_t which denotes a group of memory
operations associated with a memory barrier. A group follows a state
machine, starting from the `CMM_ANNOTATE_VOID' state. The following are
the only valid transitions:
The macro `cmm_annotate_define(name)' can be used to create an
annotation object on the stack. The rest of the `cmm_annotate_*' macros
can be used to change the state of the group after validating that the
transition is allowed. Some of these macros also inject TSAN annotations
to help it understand the flow of events in the program since it does
not currently support thread fence.
Sometime, a single memory access does not need to be associated with a
group. In the case, the acquire/release macros variant without the
`group' infix can be used to annotate memory accesses.
Note that TSAN can not be used on the liburcu-signal flavor. This is
because TSAN hijacks calls to sigaction(3) and places its own handler
that will deliver the signal to the application at a synchronization
point.
Thus, the usage of TSAN on the signal flavor is undefined
behavior. However, there's at least one known behavior which is a
deadlock between readers that want to unregister them-self by locking
the `rcu_registry_lock' while a synchronize RCU is made on the writer
side which has already locked that mutex until all the registered
readers execute a memory barrier in a signal handler defined by
liburcu-signal. However, TSAN will not call the registered handler while
waiting on the mutex. Therefore, the writer spin infinitely on
pthread_kill(3p) because the reader simply never complete the handshake.
Olivier Dion [Fri, 31 Mar 2023 14:53:43 +0000 (10:53 -0400)]
benchmark: Use uatomic for accessing global states
Global states accesses were protected via memory barriers. Use the
uatomic API with the CMM memory model so that TSAN can understand the
ordering imposed by the synchronization flags.
Olivier Dion [Wed, 29 Mar 2023 19:22:13 +0000 (15:22 -0400)]
tests: Use uatomic for accessing global states
Global states accesses were protected via memory barriers. Use the
uatomic API with the CMM memory model so that TSAN does not warn about
non-atomic concurrent accesses.
Also, the thread id map mutex must be unlocked after setting the new
created thread id in the map. Otherwise, the new thread could observe an
unset id.
Olivier Dion [Thu, 30 Mar 2023 20:09:50 +0000 (16:09 -0400)]
urcu-wait: Fix wait state load/store
The state of a wait node must be accessed atomically. Also, the action
of busy loading until the teardown state is seen must follow a
CMM_ACQUIRE semantic while storing the teardown must follow a
CMM_RELEASE semantic.
The CMM memory model reflects the C11 memory model with an additional
CMM_SEQ_CST_FENCE memory order. The memory order can be selected through
the enum cmm_memorder.
* With Atomic Builtins
If configured with atomic builtins, the correspondence between the CMM
memory model and the C11 memory model is a one to one at the exception
of the CMM_SEQ_CST_FENCE memory order which implies the memory order
CMM_SEQ_CST and a thread fence after the operation.
* Without Atomic Builtins
However, if not configured with atomic builtins, the following stipulate
the memory model.
For load operations with uatomic_load(), the memory orders CMM_RELAXED,
CMM_CONSUME, CMM_ACQUIRE, CMM_SEQ_CST and CMM_SEQ_CST_FENCE are
allowed. A barrier may be inserted before and after the load from memory
depending on the memory order:
- CMM_RELAXED: No barrier
- CMM_CONSUME: Memory barrier after read
- CMM_ACQUIRE: Memory barrier after read
- CMM_SEQ_CST: Memory barriers before and after read
- CMM_SEQ_CST_FENCE: Memory barriers before and after read
For store operations with uatomic_store(), the memory orders
CMM_RELAXED, CMM_RELEASE, CMM_SEQ_CST and CMM_SEQ_CST_FENCE are
allowed. A barrier may be inserted before and after the store to memory
depending on the memory order:
- CMM_RELAXED: No barrier
- CMM_RELEASE: Memory barrier before operation
- CMM_SEQ_CST: Memory barriers before and after operation
- CMM_SEQ_CST_FENCE: Memory barriers before and after operation
For load/store operations with uatomic_and_mo(), uatomic_or_mo(),
uatomic_add_mo(), uatomic_sub_mo(), uatomic_inc_mo(), uatomic_dec_mo(),
uatomic_add_return_mo() and uatomic_sub_return_mo(), all memory orders
are allowed. A barrier may be inserted before and after the operation
depending on the memory order:
- CMM_RELAXED: No barrier
- CMM_ACQUIRE: Memory barrier after operation
- CMM_CONSUME: Memory barrier after operation
- CMM_RELEASE: Memory barrier before operation
- CMM_ACQ_REL: Memory barriers before and after operation
- CMM_SEQ_CST: Memory barriers before and after operation
- CMM_SEQ_CST_FENCE: Memory barriers before and after operation
For the exchange operation uatomic_xchg_mo(), any memory order is
valid. A barrier may be inserted before and after the exchange to memory
depending on the memory order:
- CMM_RELAXED: No barrier
- CMM_ACQUIRE: Memory barrier after operation
- CMM_CONSUME: Memory barrier after operation
- CMM_RELEASE: Memory barrier before operation
- CMM_ACQ_REL: Memory barriers before and after operation
- CMM_SEQ_CST: Memory barriers before and after operation
- CMM_SEQ_CST_FENCE: Memory barriers before and after operation
For the compare exchange operation uatomic_cmpxchg_mo(), the success
memory order can be anything while the failure memory order cannot be
CMM_RELEASE nor CMM_ACQ_REL and cannot be stronger than the success
memory order. A barrier may be inserted before and after the store to
memory depending on the memory orders:
Success memory order:
- CMM_RELAXED: No barrier
- CMM_ACQUIRE: Memory barrier after operation
- CMM_CONSUME: Memory barrier after operation
- CMM_RELEASE: Memory barrier before operation
- CMM_ACQ_REL: Memory barriers before and after operation
- CMM_SEQ_CST: Memory barriers before and after operation
- CMM_SEQ_CST_FENCE: Memory barriers before and after operation
Barriers after the operations are only emitted if the compare exchange
succeed.
Failure memory order:
- CMM_RELAXED: No barrier
- CMM_ACQUIRE: Memory barrier after operation
- CMM_CONSUME: Memory barrier after operation
- CMM_SEQ_CST: Memory barriers before and after operation
- CMM_SEQ_CST_FENCE: Memory barriers before and after operation
Barriers after the operations are only emitted if the compare exchange
failed. Barriers before the operation are never emitted by this
memory order.
If the toolchain supports atomic builtins and the user ask for atomic
builtins, use them for the uatomic API. This requires that the
toolchains used to compile the library and the user application supports
such builtins.
The advantage of using these builtins is that they are well known
synchronization primitives by several tools such as TSAN.
However, they may introduce redundant memory barriers, mainly on
strongly ordered architectures.
Change-Id: Ia8e97112681f744f17816dbc4cbbec805a483331 Co-authored-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Signed-off-by: Olivier Dion <odion@efficios.com> Signed-off-by: Michael Jeanson <mjeanson@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Olivier Dion [Fri, 14 Jul 2023 16:49:24 +0000 (12:49 -0400)]
tests/regression/rcutorture: Use urcu-wait
pthread_cond_wait(3) can have spurious wakeups on some OS. To detect
such spurious wakeup, a global variable is shared between the waiter and
the waker.
We can use urcu-wait instead.
Change-Id: I6a2d2f3c9104ea23df16a7c8ba3557bb5d58306c Signed-off-by: Olivier Dion <odion@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Michael Jeanson [Thu, 6 Jul 2023 15:35:11 +0000 (11:35 -0400)]
Complete REUSE support
The SPDX identifiers [1] are a legally binding shorthand, which can be
used instead of the full boiler plate text. This is the final step
towards implementing the full REUSE spec [2] to help with copyright and
licensing audits and compliance.
This will reduce a lot a manual work required for the licensing audit
required in Debian on each update.
Michael Jeanson [Wed, 5 Jul 2023 18:20:10 +0000 (14:20 -0400)]
extras/abi: license data files under CC-1.0
The SPDX identifiers [1] are a legally binding shorthand, which can be
used instead of the full boiler plate text. This is another step towards
implementing the full REUSE spec [2] to help with copyright and
licensing audits and compliance.
This will reduce a lot a manual work required for the licensing audit
required in Debian on each update.
These are generated files, use the CC-1.0 license to make their
licensing clear.
Michael Jeanson [Wed, 5 Jul 2023 15:17:24 +0000 (11:17 -0400)]
examples: use SPDX identifiers
The SPDX identifiers [1] are a legally binding shorthand, which can be
used instead of the full boiler plate text. This is another step towards
implementing the full REUSE spec [2] to help with copyright and
licensing audits and compliance.
This will reduce a lot a manual work required for the licensing audit
required in Debian on each update.
Relicense all examples from 'Boehm-GC' to the more well-known and
functionnaly identical 'MIT' license. This is possible since all the
examples were written by Mathieu Desnoyers and only a few trivial fixes
from external contributors were applied over the years.
Michael Jeanson [Tue, 4 Jul 2023 20:53:30 +0000 (16:53 -0400)]
tests: use SPDX identifiers
The SPDX identifiers [1] are a legally binding shorthand, which can be
used instead of the full boiler plate text. This is another step towards
implementing the full REUSE spec [2] to help with copyright and
licensing audits and compliance.
This will reduce a lot a manual work required for the licensing audit
required in Debian on each update.
For files that lacked copyright and licensing information, I used the
following guidelines. Use the author from the git history and the test
scripts license as stated in LICENSE, 'GPL-2.0-only'.
Michael Jeanson [Tue, 4 Jul 2023 20:53:07 +0000 (16:53 -0400)]
src: use SPDX identifiers
The SPDX identifiers [1] are a legally binding shorthand, which can be
used instead of the full boiler plate text. This is another step towards
implementing the full REUSE spec [2] to help with copyright and
licensing audits and compliance.
This will reduce a lot a manual work required for the licensing audit
required in Debian on each update.
Michael Jeanson [Tue, 4 Jul 2023 20:52:00 +0000 (16:52 -0400)]
Public headers: use SPDX identifiers
The SPDX identifiers [1] are a legally binding shorthand, which can be
used instead of the full boiler plate text. This is another step towards
implementing the full REUSE spec [2] to help with copyright and
licensing audits and compliance.
This will reduce a lot a manual work required for the licensing audit
required in Debian on each update.
For files that lacked copyright and licensing information, I used the
following guidelines. Use the author from the git history and the main
project license 'LGPL-2.1-or-later'.
Michael Jeanson [Tue, 4 Jul 2023 20:47:07 +0000 (16:47 -0400)]
Build system: use SPDX identifiers
The SPDX identifiers [1] are a legally binding shorthand, which can be
used instead of the full boiler plate text. This is the first step
towards implementing the full REUSE spec [2] to help with copyright and
licensing audits and compliance.
This will reduce a lot a manual work required for the licensing audit
required in Debian on each update.
For files that lacked copyright and licensing information, I used the
following guidelines. If a clear author could be determined from the git
history use it, otherwise use 'EfficiOS Inc.'. For build system files,
use 'MIT', for documentation 'CC-BY-4.0' and for data files 'CC-1.0'.
The approach taken by caa_unqual_scalar_typeof requires use of _Generic
which requires full C11 support. Currently liburcu supports C99.
Therefore, this approach is not appropriate for now.
Instead, introduce caa_container_of_check_null which returns NULL if the
ptr is NULL before offsetting by the member offset.
Avoid calling caa_container_of on NULL pointer in cds_lfht macros
The cds_lfht_for_each_entry and cds_lfht_for_each_entry_duplicate macros
would call caa_container_of() macro on NULL pointer. This is not a
problem under normal circumstances as the check in the for loop fails
and the loop-statement is not called with invalid (pos) value.
However AddressSanitizer doesn't like that and complains about this:
Move the cds_lfht_iter_get_node(iter) != NULL from the cond-expression
of the for loop into both init-clause and iteration-expression as
conditional operator and check for (pos) value in the cond-expression
instead. Introduce the cds_lfht_entry() macro to eliminate code
duplication.
Michael Jeanson [Thu, 23 Mar 2023 18:23:55 +0000 (14:23 -0400)]
fix: warning 'noreturn' function does return on ppc
On a ppc64 system with gcc 9.5.0 I get the following error when building
with -O0 :
/usr/include/urcu/uatomic/generic.h: In function 'void _uatomic_link_error()':
/usr/include/urcu/uatomic/generic.h:53:1: warning: 'noreturn' function does return
53 | }
| ^
Split the inline function in 2 variants and apply the noreturn attribute
only on the builtin_trap one.
Change-Id: I5ae8e764c4cc27af0463924a653b9eaa9f698c34 Signed-off-by: Michael Jeanson <mjeanson@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Ondřej Surý [Fri, 17 Mar 2023 15:44:10 +0000 (16:44 +0100)]
Fix: use __noreturn__ for C11-compatibility
The noreturn convenience macro provided by stdnoreturn.h might get
included before urcu headers, use __noreturn__ for better compatibility
with code using <stdnoreturn.h> header.
Brad Smith [Sat, 25 Feb 2023 05:53:06 +0000 (00:53 -0500)]
Adjust shell scripts to allow Bash in other locations
Linux-based OS for the most part provide Bash and being located in /bin,
but on other OS's the shell would be in another location. Utilize env(1)
and allow it to be located elsewhere.
Signed-off-by: Brad Smith <brad@comstyle.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I9d4d4a3feaf993754c64b740ea91e42b336ba2b4
Brad Smith [Sat, 25 Feb 2023 02:17:16 +0000 (21:17 -0500)]
Add support for OpenBSD
- Add OpenBSD to syscall compatibility header as appropriate.
- Add function for retrieving the thread id in urcu_get_thread_id().
- Rely on pthread cond variables for futex compatibility.
It builds on all of our archs and fully run time tested on amd64.
Signed-off-by: Brad Smith <brad@comstyle.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I5cca5962ba3dc3113c9bd12e544b6e6f77dfdb61
Fix: call_rcu: teardown default call_rcu worker on application exit
Teardown the default call_rcu worker thread if there are no queued
callbacks on process exit. This prevents leaking memory.
Here is how an application can ensure graceful teardown of this
worker thread:
- An application queuing call_rcu callbacks should invoke
rcu_barrier() before it exits.
- When chaining call_rcu callbacks, the number of calls to
rcu_barrier() on application exit must match at least the maximum
number of chained callbacks.
- If an application chains callbacks endlessly, it would have to be
modified to stop chaining callbacks when it detects an application
exit (e.g. with a flag), and wait for quiescence with rcu_barrier()
after setting that flag.
- The statements above apply to a library which queues call_rcu
callbacks, only it needs to invoke rcu_barrier in its library
destructor.
Fix a deadlock for auto-resize hash tables when cds_lfht_destroy
is called with RCU read-side lock held.
Example stack track of a hang:
Thread 2 (Thread 0x7f21ba876700 (LWP 26114)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007f21beba7aa0 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=-1, op=0, uaddr=0x7f21bedac308 <urcu_memb_gp+8>) at ../include/urcu/futex.h:81
#2 futex_noasync (timeout=0x0, uaddr2=0x0, val3=0, val=-1, op=0, uaddr=0x7f21bedac308 <urcu_memb_gp+8>) at ../include/urcu/futex.h:90
#3 wait_gp () at urcu.c:265
#4 wait_for_readers (input_readers=input_readers@entry=0x7f21ba8751b0, cur_snap_readers=cur_snap_readers@entry=0x0,
qsreaders=qsreaders@entry=0x7f21ba8751c0) at urcu.c:357
#5 0x00007f21beba8339 in urcu_memb_synchronize_rcu () at urcu.c:498
#6 0x00007f21be99f93f in fini_table (last_order=<optimized out>, first_order=13, ht=0x5651cec75400) at rculfhash.c:1489
#7 _do_cds_lfht_shrink (new_size=<optimized out>, old_size=<optimized out>, ht=0x5651cec75400) at rculfhash.c:2001
#8 _do_cds_lfht_resize (ht=ht@entry=0x5651cec75400) at rculfhash.c:2023
#9 0x00007f21be99fa26 in do_resize_cb (work=0x5651e20621a0) at rculfhash.c:2063
#10 0x00007f21be99dbfd in workqueue_thread (arg=0x5651cec74a00) at workqueue.c:234
#11 0x00007f21bd7c06db in start_thread (arg=0x7f21ba876700) at pthread_create.c:463
#12 0x00007f21bd4e961f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 1 (Thread 0x7f21bf285300 (LWP 26098)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007f21be99d8b7 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=-1, op=0, uaddr=0x5651d8b38584) at ../include/urcu/futex.h:81
#2 futex_async (timeout=0x0, uaddr2=0x0, val3=0, val=-1, op=0, uaddr=0x5651d8b38584) at ../include/urcu/futex.h:113
#3 futex_wait (futex=futex@entry=0x5651d8b38584) at workqueue.c:135
#4 0x00007f21be99e2c8 in urcu_workqueue_wait_completion (completion=completion@entry=0x5651d8b38580) at workqueue.c:423
#5 0x00007f21be99e3f9 in urcu_workqueue_flush_queued_work (workqueue=0x5651cec74a00) at workqueue.c:452
#6 0x00007f21be9a0c83 in cds_lfht_destroy (ht=0x5651d8b2fcf0, attr=attr@entry=0x0) at rculfhash.c:1906
This deadlock is easy to reproduce when rapidly adding a large number of
entries in the cds_lfht, removing them, and calling cds_lfht_destroy().
The deadlock will occur if the call to cds_lfht_destroy() takes place
while a resize of the hash table is ongoing.
Fix this by moving the teardown of the lfht worker thread to libcds
library destructor, so it does not have to wait on synchronize_rcu from
a resize callback from within a read-side critical section. As a
consequence, the atfork callbacks are left registered within each urcu
flavor for which a resizeable hash table is created until the end of the
executable lifetime.
The other part of the fix is to move the hash table destruction to the
worker thread for auto-resize hash tables. This prevents having to wait
for resize callbacks from RCU read-side critical section. This is
guaranteed by the fact that the worker thread serializes previously
queued resize callbacks before the destroy callback.
Christopher Ng [Fri, 3 Feb 2023 12:16:06 +0000 (12:16 +0000)]
Fix building on MSYS2
Update cygwin libtool config in `configure.ac` to match MSYS2 build
environments as well. MSYS2 is also a Windows build environment that
produces DLLs.
Signed-off-by: Christopher Ng <facboy@gmail.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I48ca648123fd40b8003c72c0447c70a8b4bde6d6
Wu Yongwei [Tue, 1 Nov 2022 13:48:24 +0000 (21:48 +0800)]
Fix Markdown issues
`_`, `<`, and `>` are special characters in Markdown, and need to be
escaped except in code blocks. So backticks or backslahes are used to
fix the apparent issues, which caused wrong rendering.
Eric Wong [Sun, 2 Oct 2022 16:13:43 +0000 (12:13 -0400)]
Fix: Always check pthread_create for failures
pthread_create may fail with EAGAIN (which is no fault of the
programmer), so don't allow the check to be compiled out.
Signed-off-by: Eric Wong <normalperson@yhbt.net> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ia2695ea6953b589ac8ab8b444fb668daee06a614
Applications using signalfd depend on signals being blocked in all
threads of the process, otherwise threads with unblocked signals
can receive them and starve the signalfd.
While some threads in URCU do block signals (e.g. workqueue
worker for rculfhash), the call_rcu, defer_rcu, and rculfhash
partition_resize_helper threads do not.
Always block all signals before creating threads, and only unblock
SIGRCU when registering a urcu-signal thread. Restore the SIGRCU
signal to its pre-registration blocked state on unregistration.
For rculfhash, cds_lfht_worker_init can be removed, because its only
effect is to block all signals except SIGRCU. Blocking all signals is
already done by the workqueue code, and unbloking SIGRCU is now done by
the urcu signal flavor thread regisration.
Co-developed-by: Eric Wong <normalperson@yhbt.net> Signed-off-by: Eric Wong <normalperson@yhbt.net> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: If78346b15bdc287417b992a8963098c6ea0dc7d2
Simon Marchi [Wed, 17 Aug 2022 15:24:25 +0000 (11:24 -0400)]
Fix: change method used by _rcu_dereference to strip type constness
Commit 1e41ec3b07e4 ("Make temporary variable in _rcu_dereference
non-const") used the trick to add 0 to the pointer passed as a parameter
to the macro to get rid of its constness, should it be const (with the
end goal of avoiding compiler warnings). This is problematic (as shown
in [1]) if it is a pointer to an opaque type though, as the compiler
cannot perform pointer arithmetic on such a pointer (even though it
wouldn't really need to here, as we add 0).
Change it to use another trick to strip away the constness, that
shouldn't hit this problem. It was found in the same stackoverflow post
as the original trick [2]. It consists of using a statement expression
like so:
__typeof__(({ const int foo; foo; }))
The statement expression yields a value of type `int`. Statement
expressions are extensions to the C language, but we already use them
here.
The test_build* binaries now need to be linked against the urcu library,
otherwise they would be missing the rcu_dereference_sym symbol.
Simon Marchi [Wed, 17 Aug 2022 17:11:21 +0000 (13:11 -0400)]
Fix: remove type constness in URCU_FORCE_CAST's C++ version
The test added by the following patch wouldn't compile, when built
without _LGPL_SOURCE:
CXX test_build_dynlink_cxx-test_build_cxx.o
In file included from ../../include/urcu/arch.h:25,
from /home/simark/src/urcu/tests/unit/test_build.c:28,
from /home/simark/src/urcu/tests/unit/test_build_cxx.cpp:3:
/home/simark/src/urcu/tests/unit/test_build.c: In function ‘void test_build_rcu_dereference()’:
/home/simark/src/urcu/include/urcu/compiler.h:85:42: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
85 | #define URCU_FORCE_CAST(type, arg) (reinterpret_cast<type>(arg))
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/simark/src/urcu/include/urcu/pointer.h:71:49: note: in expansion of macro ‘URCU_FORCE_CAST’
71 | __typeof__(p) _________p1 = URCU_FORCE_CAST(__typeof__(p), \
| ^~~~~~~~~~~~~~~
/home/simark/src/urcu/tests/unit/test_build.c:133:9: note: in expansion of macro ‘rcu_dereference’
133 | rcu_dereference(opaque_const);
| ^~~~~~~~~~~~~~~
The compiler complains that we do a cast to a const type, equivalent to:
reinterpret_cast<const int>(arg)
... and that the const is meaningless in this context.
Use std::remove_cv to strip away any const or volatile qualifiers from
the type (using a volatile type would result in the same warning).
Change-Id: I94e79fcccfc2108021752f65977e1548084c646a Signed-off-by: Simon Marchi <simon.marchi@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Simon Marchi [Wed, 17 Aug 2022 16:49:50 +0000 (12:49 -0400)]
Move extern "C" down in include/urcu/urcu-bp.h
A following patch adds a <type_traits> include in
urcu/compiler.h. However, compiler.h gets included by urcu/pointer.h,
which gets included by urcu/urcu-bp.h inside an extern "C" scope.
Including the C++ header file <type_traits> inside an extern "C" scope
doesn't work:
In file included from /home/simark/src/urcu/include/urcu/compiler.h:25,
from /home/simark/src/urcu/include/urcu/pointer.h:29,
from /home/simark/src/urcu/include/urcu/urcu-bp.h:58,
from /home/simark/src/urcu/include/urcu-bp.h:2,
from /home/simark/src/urcu/tests/unit/test_urcu_multiflavor-bp.c:28,
from /home/simark/src/urcu/tests/unit/test_urcu_multiflavor-bp_cxx.cpp:3:
/usr/include/c++/12.1.1/type_traits:44:3: error: template with C linkage
44 | template<typename _Tp>
| ^~~~~~~~
/home/simark/src/urcu/include/urcu/urcu-bp.h:41:1: note: ‘extern "C"’ linkage started here
41 | extern "C" {
| ^~~~~~~~~~
Move the extern "C" in urcu-bp.h down, so that the includes are not
inside it. Each header file is responsible to use extern "C" where
relevant, and we should avoid including files inside such a scope.
Change-Id: I42bdfa6ab445e8c40f5bcac1c1ae0786d443626c Signed-off-by: Simon Marchi <simon.marchi@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Michael Jeanson [Mon, 15 Aug 2022 15:11:54 +0000 (11:11 -0400)]
fix: ifdef linux specific cpu count compat
Expand the '#ifdef __linux__' block in src/compat-cpu.h to all static
inline functions related to sysfs since they are only useful on Linux
and fail to build on some non-Linux platforms. This issue was reported
on QNX.
The corresponding unit tests have to be skipped on non-Linux platforms.
Thanks to Elad Lahav <e2lahav@gmail.com> for reporting this issue.
Change-Id: I17c88a9a2fb5b9be6cf5325234a18ff40788cd09 Signed-off-by: Michael Jeanson <mjeanson@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Michael Jeanson [Wed, 27 Jul 2022 14:44:00 +0000 (10:44 -0400)]
fix: sysconf(_SC_NPROCESSORS_CONF) can be less than max cpu id
We rely on sysconf(_SC_NPROCESSORS_CONF) to get the maximum possible
number of CPUs that can be attached to the system for the lifetime of an
application.
As such we expect that the highest possible CPU id would be one less
than the number returned by sysconf(_SC_NPROCESSORS_CONF) which is
unfortunatly not always the case and can vary across libc
implementations and versions.
Glibc up to 2.35 will count the number of "cpuX" directories in
"/sys/devices/system/cpu" which doesn't include CPUS that were
hot-unplugged.
This information is however provided by the Linux kernel in
"/sys/devices/system/cpu/possible" in the form of a mask listing all the
CPUs that could possibly be hot-plugged in the system.
This patch replaces sysconf(_SC_NPROCESSORS_CONF) with an internal
function that first tries parsing the possible CPU mask to extract the
highest possible value and if this fails fallback to the previous
behavior.
Change-Id: I68dfed42ebbab02728a02eeefd4a395a22bb1bea Signed-off-by: Michael Jeanson <mjeanson@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Shao-Tse Hung [Tue, 2 Aug 2022 17:44:00 +0000 (01:44 +0800)]
Fix: revise obsolete command in README.md
The obsolete command `make bench` was replaced by `make short_bench` and
`make long_bench` in 2015. However, this command wasn't revised in
README, so I follow the previous commit and rewrite it.
The urcu-qsbr wait_gp() implements a futex wait/wakeup scheme identical to
the workqueue code, which has an issue with spurious wakeups.
A spurious wakeup on wait_gp can cause wait_gp to return with a
urcu_qsbr_gp.futex state of -1, which is unexpected. It would cause the
following loops in wait_for_readers() to decrement the
urcu_qsbr_gp.futex to values below -1, thus actively using CPU as values
will be decremented to very low negative values until it reaches 0
through underflow, or until the input_readers list is found to be empty.
The state is restored to 0 when the input_readers list is found to be
empty, which restores the futex state to a correct state for the
following calls to wait_for_readers().
This issue will cause spurious unexpected high CPU use, but will not
lead to data corruption.
Cause
=====
From futex(5):
FUTEX_WAIT
Returns 0 if the caller was woken up. Note that a wake-up can
also be caused by common futex usage patterns in unrelated code
that happened to have previously used the futex word's memory
location (e.g., typical futex-based implementations of Pthreads
mutexes can cause this under some conditions). Therefore, call‐
ers should always conservatively assume that a return value of 0
can mean a spurious wake-up, and use the futex word's value
(i.e., the user-space synchronization scheme) to decide whether
to continue to block or not.
Solution
========
We therefore need to validate whether the value differs from -1 in
user-space after the call to FUTEX_WAIT returns 0.
The urcu wait_gp() implements a futex wait/wakeup scheme identical to
the workqueue code, which has an issue with spurious wakeups.
A spurious wakeup on wait_gp can cause wait_gp to return with a
rcu_gp.futex state of -1, which is unexpected. It would cause the
following loops in wait_for_readers() to decrement the
rcu_gp.futex to values below -1, thus actively using CPU as values
will be decremented to very low negative values until it reaches 0
through underflow, or until the input_readers list is found to be empty.
The state is restored to 0 when the input_readers list is found to be
empty, which restores the futex state to a correct state for the
following calls to wait_for_readers().
This issue will cause spurious unexpected high CPU use, but will not
lead to data corruption.
Cause
=====
From futex(5):
FUTEX_WAIT
Returns 0 if the caller was woken up. Note that a wake-up can
also be caused by common futex usage patterns in unrelated code
that happened to have previously used the futex word's memory
location (e.g., typical futex-based implementations of Pthreads
mutexes can cause this under some conditions). Therefore, call‐
ers should always conservatively assume that a return value of 0
can mean a spurious wake-up, and use the futex word's value
(i.e., the user-space synchronization scheme) to decide whether
to continue to block or not.
Solution
========
We therefore need to validate whether the value differs from -1 in
user-space after the call to FUTEX_WAIT returns 0.
The urcu-wait urcu_adaptative_busy_wait() implements a futex wait/wakeup
scheme similar to the workqueue code, which has an issue with spurious
wakeups.
A spurious wakeup on urcu_adaptative_busy_wait can cause
urcu_adaptative_busy_wait to reach label skip_futex_wait with a
wait->state state of URCU_WAIT_WAITING, which is unexpected. It would
cause busy-waiting on URCU_WAIT_TEARDOWN state to start early. The
wait-teardown stage is done with URCU_WAIT_ATTEMPTS active attempts,
following by attempts spaced by 10ms sleeps. I do not expect that these
spurious wakeups will cause user-observable effects other than being
slightly less efficient that it should be.
urcu-wait is used by all urcu flavor's synchronize_rcu() to implement
the grace period batching scheme.
This issue will cause spurious unexpected high CPU use, but will not
lead to data corruption.
Cause
=====
From futex(5):
FUTEX_WAIT
Returns 0 if the caller was woken up. Note that a wake-up can
also be caused by common futex usage patterns in unrelated code
that happened to have previously used the futex word's memory
location (e.g., typical futex-based implementations of Pthreads
mutexes can cause this under some conditions). Therefore, call‐
ers should always conservatively assume that a return value of 0
can mean a spurious wake-up, and use the futex word's value
(i.e., the user-space synchronization scheme) to decide whether
to continue to block or not.
Solution
========
We therefore need to validate whether the value differs from
URCU_WAIT_WAITING in user-space after the call to FUTEX_WAIT returns 0.
The urcu-defer wait_defer() implements a futex wait/wakeup scheme identical to
the workqueue code, which has an issue with spurious wakeups.
A spurious wakeup on wait_defer can cause wait_defer to return with a
defer_thread_futex state of -1, which is unexpected. It would cause the
following loops in thr_defer() to decrement the defer_thread_futex to
values below -1, thus actively using CPU as values will be decremented
to very low negative values until it reaches 0 through underflow, or
until callbacks are eventually queued. The state is restored to 0 when
callbacks are found, which restores the futex state to a correct state
for the following calls to wait_defer().
This issue will cause spurious unexpected high CPU use, but will not
lead to data corruption.
Cause
=====
From futex(5):
FUTEX_WAIT
Returns 0 if the caller was woken up. Note that a wake-up can
also be caused by common futex usage patterns in unrelated code
that happened to have previously used the futex word's memory
location (e.g., typical futex-based implementations of Pthreads
mutexes can cause this under some conditions). Therefore, call‐
ers should always conservatively assume that a return value of 0
can mean a spurious wake-up, and use the futex word's value
(i.e., the user-space synchronization scheme) to decide whether
to continue to block or not.
Solution
========
We therefore need to validate whether the value differs from -1 in
user-space after the call to FUTEX_WAIT returns 0.
The urcu call_rcu() and rcu_barrier() each implement a futex wait/wakeup
scheme identical to the workqueue code, which has an issue with spurious
wakeups.
* call_rcu
A spurious wakeup on call_rcu_wait can cause call_rcu_wait to return
with a crdp->futex state of -1, which is unexpected. It would cause the
following loops in call_rcu_thread() to decrement the crdp->futex to
values below -1, thus actively using CPU time as values will be
decremented to very low negative values until the futex value underflows
back to 0. The state is *not* restored to 0 when the callback list is
found to be non-empty, so this unexpected state will persist until the
crdp->futex state underflows back to 0, or until the call_rcu_thread is
stopped. What prevents this from having too much user-observable effects
is that the call rcu thread has a 10ms sleep between loops, to favor
batching of callbacks. Therefore, rather than being a purely 100% active
busy-wait, this scenario leads to a busy-wait which is paced by 10ms
sleeps.
Therefore the observed issue will be that the call_rcu_thread will
unexpectedly wake up the CPU each 10ms after this spurious wakeup
happens.
* rcu_barrier
A spurious wakeup on call_rcu_completion_wait can cause
call_rcu_completion_wait to return with a completion->futex state of -1,
which is unexpected. It would cause the following loops in rcu_barrier()
to decrement the completion->futex to values below -1, thus actively
using CPU time as values will be decremented to very low negative values
until either the barrier count reaches 0 or until the futex value
underflows to 0.
Therefore the observed issue will be that rcu_barrier() will
unexpectedly use a lot of CPU time when this spurious wakeup happens.
These issues will cause spurious unexpected high CPU use, but will not
lead to data corruption.
Cause
=====
From futex(5):
FUTEX_WAIT
Returns 0 if the caller was woken up. Note that a wake-up can
also be caused by common futex usage patterns in unrelated code
that happened to have previously used the futex word's memory
location (e.g., typical futex-based implementations of Pthreads
mutexes can cause this under some conditions). Therefore, call‐
ers should always conservatively assume that a return value of 0
can mean a spurious wake-up, and use the futex word's value
(i.e., the user-space synchronization scheme) to decide whether
to continue to block or not.
Solution
========
We therefore need to validate whether the value differs from -1 in
user-space after the call to FUTEX_WAIT returns 0.
The workqueue thread futex_wait() returns with a workqueue->futex state
of -1, which is unexpected. In this situation, the workqueue thread is
observed to use 99% of CPU as workqueue->futex values are decremented to
very low negative values while the workqueue is empty.
This issue will cause spurious unexpected high CPU use, but will not
lead to data corruption.
Cause
=====
From futex(5):
FUTEX_WAIT
Returns 0 if the caller was woken up. Note that a wake-up can
also be caused by common futex usage patterns in unrelated code
that happened to have previously used the futex word's memory
location (e.g., typical futex-based implementations of Pthreads
mutexes can cause this under some conditions). Therefore, call‐
ers should always conservatively assume that a return value of 0
can mean a spurious wake-up, and use the futex word's value
(i.e., the user-space synchronization scheme) to decide whether
to continue to block or not.
Solution
========
We therefore need to validate whether the value differs from -1 in
user-space after the call to FUTEX_WAIT returns 0.
Michael Jeanson [Mon, 13 Sep 2021 20:13:44 +0000 (16:13 -0400)]
Improved test framework
This is based on the babeltrace / librseq test framework with the
objective of standardising across projects.
Regroup all the configure detected values relevant to the test suite in
a single generated file. This file will be automatically sourced by the
test suite in most scenarios but can also be sourced in the shell of a
user.
* All user overridable variables start with 'URCU_TESTS_'.
* The priority for variables is :
Environment -> env.sh -> utils.sh (defaults).
* A user can source 'env.sh', override some of the values and manually
run test scripts.
* The test suite can run without an 'env.sh' file present.
Change-Id: Id94f7085ed1ea0e30207856cf1594ca30585536c Signed-off-by: Michael Jeanson <mjeanson@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Allow initializing lfht node to "removed" state to allow querying
whether the node is published in a hash table before it is added to the
hash table and after it has been removed from the hash table.
projects / userspace-rcu.git / log
