Fix urcu-bp: don't move registry

It is not correct to move the registry address range, since there are
external references from reader threads. This will trigger on workloads
with many threads.

Typically, on Linux, mremap can expand the existing range, which is OK.
However, if there is not enough space around the existing range, it may
try to map it at a different address, which is incorrect.

It is more likely that this bug will be observed on operating systems
where urcu uses the mmap/munmap fallback instead of mremap.

Moreover, prior to commit:

  "Fix: urcu-bp: Bulletproof RCU arena resize bug"

this issue was hidden by the fact that each thread ended up with their
own memory mapping (leaked), on Linux at least.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Fix: compat futex duplicated lock and completion

compat_futex.c has one instance included in each urcu shared object, as
well as within some of the test applications. However, it is expected
that an entire program interact with the same lock and completion
variables. Therefore, define them as globally visible, but weak, so the
entire program agree on which object should be used.

Reported-by: Vladimir Nikulichev <nvs@tbricks.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Fix: i386 compat code duplicated mutex instances

compat_arch_x86.c is linked into many .so and even into test programs.
The basic problem with this is that it contains a statically defined
mutex, which will fail to protect concurrent use of this compat code by
different shared objects.

Fix this by defining both the mutex (now called __urcu_x86_compat_mutex)
and __rcu_cas_avail as weak symbols. Therefore, the first symbol that
gets loaded in a program will by used by everyone.

Reported-by: Vladimir Nikulichev <nvs@tbricks.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Fix: urcu-bp: Bulletproof RCU arena resize bug

> From: "Milosz Tanski" <milosz@adfin.com>

> While trying to use the BP flavor of RCU I ran into random crashes. I
> tracked it down to issues with resizing of the BP RCU memory pool.
>
> The problem is in the urcu-bp.c file in the resize_arena() function.
> On successful allocation / remapping the len member of the
> registry_arena struct is never set anywhere function. On the second
> resize of the arena the code in resize_arena() still thinks the
> previous size is equal to the original mapping size. I've fixed this
> issue locally by just adding the following code at the bottom of
> resize_arena().

Good catch !!

However, I think your fix misses one case: if we happen to re-use the
same region, we want to update the length too.

Reported-by: Milosz Tanski <milosz@adfin.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Fix: test_mutex.c uninitialized mutex

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Version 0.8.0

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Fix: hash table growth (for small tables) should be limited

Buckets with many entries encountered in a hash table could cause it to
grow to a large size, beyond the scope for which this mechanism is
expected to play a role when node accounting is available. Indeed, when
the hash table grows to larger size, split-counter node accounting is
expected to deal with resize/shrink rather than relying on an heuristic
based on the largest bucket size.

This is fixing an issue where we see hash tables sometimes reaching 65k
entries index (65536*8 = 524288 bytes) for a workload limited to adding
1000 entries and then removing all of them, done in a loop (random
keys).

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Fix: doc/examples cross-build

We need to pass automake's CPPFLAGS and LDFLAGS to examples. Add CFLAGS
too for completeness.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Introduce URCU_INLINE_SMALL_FUNCTIONS

Reviewed-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Add missing tests/common/Makefile.am

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

README: document make check/regtest/bench

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

tests: split in check, regtest and bench targets

Allow make check to run without requiring build machines to use hours of
CPU time.

make check: short unit tests
make regtest: long regression tests
make bench: long benchmarks

Fixes #611

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Cleanup: doc/examples makefile

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Fix: doc/examples VPATH build

Fixes #568

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: Move the LIBS after the OBJECTS in the Makefile

Fixes #569.

Signed-off-by: Alexandre Montplaisir <alexmonthy@voxpopuli.im>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Document build work-around on MacOS X

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Fix tests: use of uninitialized variables

Were working fine by luck, since they were allocated at the start of
newly spawned test programs. Identified by Coverity.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

test_urcu_hash*: initialize TLS seeds

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: cds_lfht_for_each_entry_duplicate

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: cds_lfht_lookup

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: cds_lfht_destroy

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: cds_lfht_add_replace

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: cds_lfht_add_unique

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: cds_lfht_add/cds_lfht_del

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: add rculfqueue example

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: add synchronize_rcu()

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: add bp flavor

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: add dist toplevel makefile

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: add membarrier flavor

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: document call_rcu()

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: update qsbr example

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

urcu signal: remove assertion on exit

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: signal flavor

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: add mb flavor

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: update qsbr

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: introduce urcu-flavors examples directory

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: enhance rcu-flavor-qsbr example

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: rename qsbr-minimal to rcu-flavor-qsbr

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: automake stop on error

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: hlist

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

hlist/rcuhlist update

introduce:

- cds_hlist_for_each()
- cds_hlist_for_each_safe()
- CDS_HLIST_HEAD()
- CDS_HLIST_HEAD_INIT()
- cds_hlist_for_each_entry_2() (takes 3 argument, like the Linux kernel
  API),
- cds_hlist_for_each_entry_safe_2() (takes 4 arguments, like the Linux
  kernel API),
- cds_hlist_for_each_rcu()
- cds_hlist_for_each_entry_rcu_2() (takes 3 arguments, like the Linux
  kernel API).

Left cds_hlist_for_each_entry(), cds_hlist_for_each_entry_safe() and
cds_hlist_for_each_entry_rcu() as-is (different from the ones found in
the Linux kernel) because those APIs were already exposed by Userspace
RCU.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: fix typo in list example

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

rcuhlist: make pointer stores atomic

Use rcu_assign_pointer() to store into head->next in
cds_hlist_add_head_rcu(). This includes the write barrier needed before
publishing the new node.

Use CMM_STORE_SHARED() to store into elem->prev->next in
cds_hlist_del_rcu().

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

hlist, rcuhlist: cleanup coding style

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: lfstack

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: update cds_wfs_pop_all_blocking

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: cds_wfs_pop_all_blocking

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: cds_wfs_pop

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: add missing Makefile

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: cds_wfs_push

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/wfcqueue: cds_wfcq_splice

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: add cds_wfcq_dequeue

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: wfcq needs to link against urcu-common

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: update queue comment

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: fix make clean

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

gitignore: add qsbr-minimal

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: cds_wfcq_enqueue

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: Move LIBS to each makefile

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: cds_list_for_each_rcu

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: cds_list_for_each_entry_rcu

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: cds_list_replace_rcu

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: cds_list_add_tail_rcu

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: cds_list_del_rcu

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

doc/examples: cds_list_add_rcu

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

rculist: ensure atomic updates of next pointers

in cds_list_add_rcu, use rcu_assign_pointer to update head->next
atomically and provide the memory barrier before publishing head->next.
Notice that we don't need the wmb() prior to store to prev, because RCU
traversals only go forward, and thus only use "next".

in cds_list_del_rcu, use CMM_STORE_SHARED() to store to elem->prev->next
atomically.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

rculist: implement cds_list_add_tail_rcu

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

rculist.h and list.h style cleanup

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

example makefile: add missing cd ..

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Update gitignore

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Fix: examples Makefile on FreeBSD

Support BSD-style make.

Fixes #565

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

hash table test: don't redefine CACHE_LINE_SIZE

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

tests: use thread-id.h wrapper

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Implement thread-id.h wrapper

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

tests: add missing unsigned long casts to pthread_self()

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Fix: don't build examples in static builds

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Add QSBR minimal example

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

compiler.h: implement CAA_ARRAY_SIZE()

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

document rcu barrier

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

rcu barrier: handle OOM die urcu_die

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Implement rcu_barrier()

Awaits for all in-flight call_rcu handlers to complete execution before
returning.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

rculfhash: document destroy context limitations

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Add MIPS to README

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Update README

Detail i386/amd64 as tested for FreeBSD.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Update README testing info about FreeBSD

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

test: fix api.h missing if brackets

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

tests: fix incorrect counter

- not initialized to 0 in 2 tests,
- within inner loop in 3rd test, thus always 0.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Fix: membarrier fallback symbol conflict

* Lai Jiangshan (laijs@cn.fujitsu.com) wrote:
> Hi, Mathieu,
>
> There is a big compatible problem in URCU which should be fix in next round.
>
> LB: liburcu built on the system which has sys_membarrier().
> LU: liburcu built on the system which does NOT have sys_membarrier().
>
> LBM: liburcu-mb ....
> LUM: liburcu-mb ...
>
> AB: application(-lliburcu) built on the system which has sys_membarrier().
> AU: application(-lliburcu) built on the system which does NOT have
> sys_membarrier().
>
> ABM application(-lliburcu-mb) ...
> AUM application(-lliburcu-mb) ...
>
> AB/AU + LB/LU: 4 combinations
> ABM/AUM + LBM/LUM: 4 combinations
>
> I remember some of the 8 combinations can't works due to symbols are
> miss match.  only LU+AB and LB+AU ?
>
> could you check it?
>
> How to fix it: In LU and AU, keep all the symbol name/ABI as LA and
> AB, but only the behaviors falls back to URCU_MB.

Define membarrier() as -ENOSYS when SYS_membarrier is not found in the
system headers. Check dynamically for membarrier availability to ensure
ABI compatibility between applications and librairies.

Reported-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Fix: Use a filled signal mask to disable all signals

Changelog from David Pelton's original patch:

While using lttng-ust with an application that was calling fork()
with pending signals, I found that all signals were getting unmasked
shortly before the underlying call to fork().  After some
investigation, I found that the rcu_bp_before_fork() function was
unmasking all signals.  Based on the comments for this function, it
should be masking all signals.  Inspection of the rest of the code
in urcu-bp.c revealed the same pattern in two other functions.

This patch changes the code to use a filled signal mask to disable
all signals.  The change to rcu_bp_before_fork() addressed the
problem I was seeing while using lttng-ust.  The changes to the
other two functions appear to fix other instances of the same
problem.

Updates by Mathieu Desnoyers:

- Use SIG_BLOCK instead of SIG_SETMASK when setting a filled mask. This
  has the same behavior in this case (since we're blocking all signals),
  but is semantically neater: if we ever some signals from that mask,
  we'd like to to a union with the signal mask already blocked by the
  application.
- Also fix incorrect signal masking in compat_arch_x86.c.

Reported-by: David Pelton <dpelton@ciena.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

urcu-bp: introduce struct urcu_gp

Make urcu-bp similar to urcu-qsbr and other urcu flavors.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Fix: struct urcu_gp broke multiflavor

Add mapping to namespace urcu_gp.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Cleanup test usage printout

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

wfstack tests: use pop "last" state info

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

wfstack: return whether pop is popping the last element

Newly introduced "with_state" pop API members return stack state
atomically sampled with the pop operation.

Allow testing behavior of pop with respect to number of push-to-empty
and pop-all-from-non-empty.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

wfcqueue tests: use dequeue empty state

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

wfcqueue: return whether dequeue is dequeuing last element

Newly introduced "with_state" dequeue API members return queue state
atomically sampled with the dequeue operation.

Allow testing behavior of dequeue with respect to number of
enqueue-to-empty and splice-from-non-empty.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

urcu: avoid false sharing for rcu_gp_ctr

@rcu_gp_ctr and @registry share the same cache line, it causes
false sharing and slowdown both of the read site and update site.

Fix: Use different cache line for them.

Although rcu_gp_futex is updated less than rcu_gp_ctr, but
they always be accessed at almost the same time, so we also move rcu_gp_futex
to the cacheline of rcu_gp_ctr to reduce the cacheline-usage or cache-missing
of read site.

test: (4X6=24 CPUs)

Before patch:

[root@localhost userspace-rcu]# ./tests/test_urcu_mb 20 1 20
SUMMARY ./tests/test_urcu_mb      testdur   20 nr_readers  20 rdur      0 wdur      0 nr_writers   1 wdelay      0 nr_reads   2100285330 nr_writes      3390219 nr_ops   2103675549
[root@localhost userspace-rcu]# ./tests/test_urcu_mb 20 1 20
SUMMARY ./tests/test_urcu_mb      testdur   20 nr_readers  20 rdur      0 wdur      0 nr_writers   1 wdelay      0 nr_reads   1619868562 nr_writes      3529478 nr_ops   1623398040
[root@localhost userspace-rcu]# ./tests/test_urcu_mb 20 1 20
SUMMARY ./tests/test_urcu_mb      testdur   20 nr_readers  20 rdur      0 wdur      0 nr_writers   1 wdelay      0 nr_reads   1949067038 nr_writes      3469334 nr_ops   1952536372

after patch:

[root@localhost userspace-rcu]# ./tests/test_urcu_mb 20 1 20
SUMMARY ./tests/test_urcu_mb      testdur   20 nr_readers  20 rdur      0 wdur      0 nr_writers   1 wdelay      0 nr_reads   3380191848 nr_writes      4903248 nr_ops   3385095096
[root@localhost userspace-rcu]# ./tests/test_urcu_mb 20 1 20
SUMMARY ./tests/test_urcu_mb      testdur   20 nr_readers  20 rdur      0 wdur      0 nr_writers   1 wdelay      0 nr_reads   3397637486 nr_writes      4129809 nr_ops   3401767295

Singed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

urcu: make the code of urcu-qsbr as normal urcu

urcu-qsbr's read site's quiescence is much longer than normal urcu ==>
synchronize_rcu() is much slower ==>
rcu_gp_ctr is updated much less ==>
the whole urcu-qsbr will not be slowed down by false sharing of rcu_gp_ctr.

But this patch makes sense to keep the code of urcu-qsbr like normal urcu,
better readability and maintenance.

Test: (4*6 CPUs)
Before patch:
[root@localhost userspace-rcu]# ./tests/test_urcu_qsbr 20 1 20
SUMMARY ./tests/test_urcu_qsbr    testdur   20 nr_readers  20 rdur      0 wdur      0 nr_writers   1 wdelay      0 nr_reads  65498297587 nr_writes      2000665 nr_ops  65500298252
[root@localhost userspace-rcu]# ./tests/test_urcu_qsbr 20 1 20
SUMMARY ./tests/test_urcu_qsbr    testdur   20 nr_readers  20 rdur      0 wdur      0 nr_writers   1 wdelay      0 nr_reads  67218079467 nr_writes      1981593 nr_ops  67220061060

After patch
./tests/test_urcu_qsbr 20 1 20
SUMMARY ./tests/test_urcu_qsbr    testdur   20 nr_readers  20 rdur      0 wdur      0 nr_writers   1 wdelay      0 nr_reads  67473798999 nr_writes      1999151 nr_ops  67475798150
[root@localhost userspace-rcu]# ./tests/test_urcu_qsbr 20 1 20
SUMMARY ./tests/test_urcu_qsbr    testdur   20 nr_readers  20 rdur      0 wdur      0 nr_writers   1 wdelay      0 nr_reads  67065521397 nr_writes      1993956 nr_ops  67067515353

Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

rculfhash: detect if resize/destroy are called within RCU read-side C.S.

Report errors.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Documentation: rculfhash: cds_lfht_resize not within read-side C.S.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

fix: rculfhash don't change qsbr online state

resize and destroy should not change the QSBR online state. Use the new
rcu_read_ongoing() API for this.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>