wfqueue: provide API for caller-locked dequeue

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

LGPL-ize wfqueue

Provide the appropriate wrappers to ensure that non-LGPL applications can use
the half-blocking wait-free queue.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

WF queue cleanup

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Wait-free stack: rcu-less implementation

Don't have to hold a rcu read lock anymore. pop side is blocking and holds a
mutex.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Wait-free queue: wait-free enqueue, blocking dequeue without RCU dependency

Implementation better suited for supporting call_rcu().

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

RCU lf queue/stack: update comments

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

ifdef C, wait-free stack adaptative pop wait

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

urcu static coding style fix (whitespace)

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Coding style fix

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

RCU wait-free stack: add _blocking in pop() primitive name

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

RCU lock-free queue comment updates

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

RCU wait-free enqueue/blocking dequeue queue

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Add ifdef cond around headers

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

RCU lock-free queue: don't hold RCU read lock across retry

Be nicer to grace periods.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

RCU lock-free stack: don't hold read lock across retry

Be nicer to grace-periods.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Wait-free push/Blocking pop stack: Update following Paul's review

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Add lock-free queue comments from Paul's review

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Add wait-free push/blocking pop stack

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Lock-free stack: add comments

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Add lock-free RCU queue and stack

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Update version to 0.4.6

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Update README for ARM support

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Update version to 0.4.5

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Update email address from @polymtl.ca to @efficios.com

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Force build failure on unknown architectures

Create urcu/arch_unknown.h and urcu/uatomic_arch_unknown.h, which
contain only #error statements and explanatory comments.  This forces
build failures on unrecognized architectures in preference to trying
to guess at what operations might be safe on such architectures.

One other semi-feasible alternative is to use hashed arrays of locks
that are acquired with signals disabled.  However, this seems a bit
too ornate, especially for architectures for which the gcc __sync_
primitives work correctly.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Add native ARM port for armv7l

Add native support for armv7l.  Other variants of ARM will likely require
separate ports.  The gcc __sync_ approach works back to at least 2.6.15
kernels.

As suggested by Paolo Bonzini, defer the definition of CACHE_LINE_SIZE
to arch_generic.h.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Add ia64 architecture based on gcc primitives

This of course assumes that ia64 gcc correctly implements the various
__sync_ primitives.  ;-)

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Add header files supporting gcc __sync_ primitives

Add a urcu/arch_gcc.h and urcu/uatomic_arch_gcc.h whose primitives are
based on the gcc __sync_ primitives.  This should be usable for all
systems that have correctly implemented __sync_ primitives, which sadly
does not include all combinations of systems and compilers.  In addition,
specific systems may gain higher performance with hand-coded primitives.
Nevertheless, this is nice for getting a new architecture up and running
quickly.

As suggested by Paolo Bonzini, defer the definition of mb() to the common
arch_generic.h file, and also defer the uatomic*() primitives to
uatomic_generic.h.

[Mathieu : fixed arch_gcc.h header which contained incorrect arch_unknown.h in
           header.]

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Add test cycles per loop

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Add uatomic test to make check

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Remove unused sync_core() definitions

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Update version to 0.4.4

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Fix powerpc uatomic 4-byte cmpxchg (cmpd -> cmpw)

Necessary to make it work on 32-bit only powerpc architectures. Was using the
64-bit comparison rather than the 32-bit comparison for 4-byte cmpxchg.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

urcu-qsbr: use same ongoing helper for 32 and 64 bits

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

"make check" runs the test suite

a trivial patch to enable the Makefile "check" rule.

Signed-off-by: Giuseppe Scrivano <gscrivano@gnu.org>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Simplify qsbr online condition

Alan Stern wrote:

>       return v && (v - rcu_gp_ctr > ULONG_MAX / 2);
>
> Wouldn't it be better to write:
>
>       return v != 0 && v != rcu_gp_ctr;
>
> or something equivalent?  The point is, it's foolish to return 0 when
> rcu_gp_ctr has wrapped around to below v.  This can happen only when
> rcu_thread_online() is delayed (as by preemption), in which case we
> know that there's a good chance the algorithm will fail.  My version of
> the test eliminates the possibility of this failure, although it
> doesn't eliminate the possibility that rcu_gp_ctr has wrapped around so
> far that it is equal to v.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Cleanup: head -> node field name

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Update urcu and urcu-qsbr, make compiler barriers explicit

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

urcu: Add extra "engineering safety factor" memory barrier in update_counter_and_wait()

Even though the memory barriers within update_counter_and_wait() are proven not
to be needed in urcu-mb/signal/qsbr implementations, we leave them in place as
an engineering safety factor. Basically, we've proven they are not required
(formally for urcu-mb and urcu-signal by model checking, less formally for
urcu-qsbr by looking at the execution order of concurrent synchronize_rcu() and
RCU read-sides with out-of-order load/stores). However, given that on the
overall performance impact of synchronize_rcu(), these memory barriers do not
add a significant overhead, let's leave them in place with a comment stating
that they are not required.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

urcu-qsbr: document compiler barrier

This compiler barrier is, at the implementation level, already provided by the
volatile accesses for STORE/LOAD_SHARED. However, given that these primitives
does not provide this guarantee at the high-level (and might use lighter gcc
primitives eventually), we need to add an explicit barrier().

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

update version to 0.4.3

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Fix typo in urcu/uatomic_generic.h

Signed-off-by: Jon Bernard <jbernard@debian.org>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Update version to 0.4.2

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Update README

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

urcu-bp: reset ctr to 0 upon garbage collection

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

compiler.h: add () in ACCESS_ONCE()

Protect x in ACCESS_ONCE macro.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

urcu-defer: remove "ratelimit" leftover

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Update readme: ia64 and alpha support

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Install missing arch_generic.h

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

urcu: fix multiple register/unregister assertion failure

Caused by the fact that we check for the whole gp count (which includes the last
grace period phase bit) rather than just checking if the nesting count mask is
zero.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

test cases: add multiple register/unregister tests

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

centralize definition of BITS_PER_LONG

Use __SIZEOF_LONG__, defined by GCC 4.3 or later, or _LP64, defined
by GCC 3.4 or later.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

avoid multiple evaluation of STORE_SHARED argument

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

support compiling on unknown architectures

The new defaults mean that, for example, ia64 support comes for free.

[edit by Mathieu Desnoyers]

defaults -> generic

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

add Alpha support

And you already know what needs to be special-cased for Alpha...

[edit by Mathieu Desnoyers]

defaults -> generic

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

move whether atomic byte/short exists to uatomic_arch_*.h

And add more generic implementations to uatomic_defaults.h.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

use uatomic_generic.h for common fallback implementations

And now, really remove the code for S390/SPARC, using the fallback
implementation of uatomic_generic.h

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

add uatomic_generic.h, use it for common definitions

uatomic_generic.h can be included by uatomic_arch_*.h to provide useful
default definitions.  uatomic_arch_*.h can define whatever builtins
it wants to override, then uatomic_generic.h will provide what is not
already defined, as follows:

- uatomic_cmpxchg will use __sync_val_compare_and_swap builtins;

- uatomic_add_return will use __sync_fetch_and_add if uatomic_arch_*.h
did not provide a definition of uatomic_cmpxchg.  If it did, we assume
__sync builtins are buggy or otherwise undesirable on this platform,
so uatomic_generic.h will implement uatomic_add_return in terms of
uatomic_cmpxchg too.

- uatomic_xchg is like uatomic_add_return.  However, since GCC does
not provide an adequate builtin, it needs to use a compare-and-swap
loop using __sync_bool_compare_and_swap if uatomic_cmpxchg is not
provided.

- uatomic_sub_return/uatomic_add/uatomic_sub will be implemented
in terms of uatomic_add_return;

- uatomic_inc/uatomic_dec will be implemented in terms of uatomic_add.

After this patch, uatomic_generic.h is already used for the latter two
categories.  The next patch will use uatomic_generic.h whenever there is
no assembly code involved, or otherwise it makes no difference to use
uatomic_generic.h.  We keep custom per-arch macros to provide support for
compilers back to early GCC 3.x versions; however future ports may
not have this limitation.

The hunk in tests/test_uatomic.c is only needed for bisectability
and will be removed later.

[edit by Mathieu Desnoyers]

Fixed Makefile.am conflict.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

remove compat_uatomic_cmpxchg #define from non-x86

This is unused everywhere except on x86.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

define sync_core for x86 PIC

Pushing/popping the reserved ebx register is surely less expensive
than a memory barrier.

Note that since ebx is a callee-save register, this is even safe for
signals (i.e. it would be safe even if we needed the value that cpuid
puts in %%ebx).

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

add urcu/arch_generic.h

Most of the memory barrier definitions are shared between all
architectures, especially smp_* and mc/rmc/wmc.  Put them in
a common file.

[edit by Mathieu Desnoyers]

+ * arch_defaults.h: common definitions for multiple architectures.
 becomes
+ * arch_generic.h: common definitions for multiple architectures.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

use autoconf symbolic linking

Instead of copying.  This is less error-prone while developing and
lets Automake handle more stuff.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

use kernel style makefile output

A breath of fresh air.  Requires Automake 1.11, but the
patch degrades gracefully.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

update README for urcu-signal

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

list: new -> _new for c++ compatibility

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

rcuhlist: add head name fix

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

RCU list: add rcuhlist.h, add list_replace_rcu

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

tests: update writer cs len script for 64-core ppc

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

tests: update summary

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

urcu tests: add writer duration parameter

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Update to version 0.4.1

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

urcu-static.h: use SYS_membarrier rather than __NR_membarrier

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

s390: fix arch_s390.h include of urcu/compiler.h

gcc -DHAVE_CONFIG_H -I. -I.. -I../urcu    -DTORTURE_URCU -I.. -I.. -g -O2 -c -o rcutorture_urcu-urcutorture.o `test -f 'urcutorture.c' || echo './'`urcutorture.c
/bin/sh ../libtool --tag=CC   --mode=link gcc -DTORTURE_URCU -I.. -I.. -g -O2 -lpthread  -o rcutorture_urcu rcutorture_urcu-urcutorture.o ../urcu.c ../urcu-pointer.c
gcc -DTORTURE_URCU -I.. -I.. -g -O2 -o rcutorture_urcu rcutorture_urcu-urcutorture.o ../urcu.c ../urcu-pointer.c  -lpthread
In file included from ../urcu-static.h:38,
                 from ../urcu.c:36:
../urcu/arch.h:31:22: error: compiler.h: No such file or directory
In file included from ../urcu/system.h:22,
                 from ../urcu/uatomic_arch.h:33,
                 from ../urcu-pointer.c:27:
../urcu/arch.h:31:22: error: compiler.h: No such file or directory
make[3]: *** [rcutorture_urcu] Error 1
make[3]: Leaving directory `/build/buildd-liburcu_0.4.0-1-s390-vwO9c1/liburcu-0.4.0/tests'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/build/buildd-liburcu_0.4.0-1-s390-vwO9c1/liburcu-0.4.0'
make[1]: *** [all] Error 2
dh_auto_build: make -j1 returned exit code 2
make: *** [build] Error 2
dpkg-buildpackage: error: debian/rules build gave error exit status 2
make[1]: Leaving directory `/build/buildd-liburcu_0.4.0-1-s390-vwO9c1/liburcu-0.4.0'

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

README: discuss mutex vs RCU interaction

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

standardize rcu_gp_ongoing

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

rename rcu_old_gp_ongoing to rcu_gp_ongoing

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

update LOAD_SHARED/STORE_SHARED for need_mb.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

urcu: use unsigned long instead of long

As it is currently designed, urcu scheme never overflows and we do not use the
upper "sign" bit. But, nevertheless, just in case we choose to do it differently
in the future, use an unsigned long type to allow:

- overflow
- defined used of 0x80000000

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

update version to 0.4.0

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

URCU: rename light/heavy barriers to slave/master

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

QSBR URCU: remove unused reader_barrier() declaration

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Update comments

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Internal cleanup

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

rename internal_rcu_lock() into mutex_lock/unlock(&rcu_gp_lock)

Cleanup.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Add SIGURCU backward compatibility

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Update support for MEMBARRIER_QUERY

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Support sys_membarrier v6 ABI

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Add sys_membarrier() dynamic detection, old liburcu.so -> liburcu-signal.so

See updated README for library renaming details.

liburcu is now dynamically detecting if sys_membarrier is available, and using
urcu-mb as fall back.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

urcu signal-based renames

reader_barrier renamed to smp_mb_light
force_mb_all_readers renamed to smp_mb_heavy

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

signal-based urcu: rename force_mb_all_threads -> force_mb_all_readers

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Cleanup urcu-defer static header

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

API prefix standardization: urcu_ -> rcu_

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

version 0.3.4

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Support earlier glibc sched_setaffinity

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>
Signed-off-by: Pedro Alves <pedro@codesourcery.com>

Remove bogus ; from defines

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

version 0.3.3

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Consider __NO_LWSYNC__

This is set by the gcc when the target does not support the lwsync
opcode. This is the case for e500 core based CPUs.

I haven't touch tests/api_ppc.h because I don't see any users.

Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

version 0.3.2

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

s390 uatomic: fix new compiler support

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Add extern "C" to support linking userspace RCU library with C++ applications

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>