generic urcu deferral (call_rcu())

Found out a way to encode the queues so the standard scenario is to use a single
pointer per call_rcu().

Uses more space for:
- unaligned functions pointers.
- unaligned data pointers.
- function/data value : -2L. (this is arbitrary)

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Rename liburcu-reclaim to liburcu-defer

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

urcu-reclaim cleanup

Rename "reader" to "reclaimer", because urcu-reclaim keeps track of "reclaimer"
threads (rcu writers) and has its own reclaiming thread periodically performing
the batch reclamation.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Update header heading

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Add rcu-reclaim.so library

Add "automated" RCU memory reclamation.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

add static declarations

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

qsbr: Add write+read thread support to 32-bit QSBR

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

QSBR: Implement 2-phase grace period for 32-bit arch

Ensures we never run in overflow on 32-bit arch.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Permit both 32 and 64-bit builds

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Default to architecture size, add Makefile32

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Update makefile to compiler for 32-bit architectures on x86_64

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Rename define

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Fix urcu.c comment

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

qsbr: use defines to clarify code.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

qsbr: portability fixes: use unsigned long for the gp counters.

unsigned wraps properly in C (modulo ULONG_MAX + 1), IOW ULONG_MAX + 2
_is_ defined and is 1.

Rewrite rcu_gp_ongoing test to work in the unsigned case:

    a - b < 0

becomes

    a - b > ULONG_MAX / 2

This test actually means a - b > 0x7f....f which is exactly what we want
to test for in the first place, but in a portable way.

Signed-off-by: Pierre Habouzit <madcoder@debian.org>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

qsbr: micro optimization of the gp use.

Follow-up on 1a80f3: since now `0' for a gp counter means "offline", just
be sure that the global GP counter is never zero. To achieve that, let its
first value ever be one, and increment it 2 by 2 like previously done.
Note that it internaly uses the fact that signed integers wraps (which the
previous code already assumed anyways) but this is undefined behaviour in
C. Using unsigned longs would probably be more portable.

This way, setting the cpu online, marking a quiescent state is just a
matter of copying the global gp counter instead of incrementing it by one.

This saves one CPU instruction, and supposedly even register use for
architectures providing memory to memory copy operands.

Signed-off-by: Pierre Habouzit <madcoder@debian.org>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

qsbr: save two full smp_mb() when the writer is also a reader.

Now that you've put the barriers before taking lock, and after releasing
it, one can expand the _rcu_thread_{on,off}line call, and optimize two
barrier calls away. (this is patch 1/2).

Signed-off-by: Pierre Habouzit <madcoder@debian.org>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

rename QSBR force_mb_all_threads to smp_mb

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Move mb() outside of the synchronize C.S.

Will be useful for future Q.S. merging. Does not change the RCU g.p. guarantee.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Ensure LOAD_SHARED/STORE_SHARED semantic is used in QSBR RCU

Ensures all volatile accesses are indeed volatile.
On architectures without coherent caches, will perform the proper cache flushes
when needed.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

QSBR offline thread micro-optimization

> That's an interesting paper for sure. I had one micro-optimization in
> mind when I read the paper for QSBR, you can decide that making a CPU
> offline can be done by writing 0 to the per thread counter, instead of
> reading the global counter. It saves a load from a shared variable,
> which probably helps in its tiny way.
>

In wait_for_quiescent_state, the test for Q.S. is :

                while (rcu_gp_ongoing(index->rcu_reader_qs_gp) &&
                       (*index->rcu_reader_qs_gp - urcu_gp_ctr < 0))
                        cpu_relax();

where :

static inline int rcu_gp_ongoing(long *value)
{
        if (value == NULL)
                return 0;

        return LOAD_SHARED(*value) & 1;
}

Your proposal would work for the rcu_gp_ongoing test, as it only checks
for the parity. Given this test is enough to guarantee that we skip the
reader thread, then yes, it seems to work just fine.

Proposed-by: Pierre Habouzit <madcoder@debian.org>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Expose nop QSBR read lock to non-GPL/LGPL programs

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Update README (again)

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

update README

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Rename CONFIG_URCU_AVOID_SIGNALS to URCU_MB, and README updates

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Revert "qsbr urcu: make it safe to call rcu_sychronize from a registered thread."

This reverts commit d1238495fdaf1c85447690231bf5d1838b83bf88.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

qsbr urcu: make it safe to call rcu_sychronize from a registered thread

Actually, the proper way to deal with threads registered as readers but also
calling synchronize_rcu() is to mark them as offline while the synchronize is in
progress.

Else, since calling rcu_sychronize will increment urcu_gp_ctr, we will be
stupidly waiting for ourselves and deadlock.

Original-patch-from: Pierre Habouzit <madcoder@debian.org>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

qsbr urcu: make it safe to call rcu_sychronize from a registered thread.

Else, since calling rcu_sychronize will increment urcu_gp_ctr, we will be
stupidly waiting for ourselves and deadlock.

No need to use _rcu_quiescent_state() since we are currently owning the
internal URCU lock, so the memory barriers and ACCESS_ONCE() are overkill.

Signed-off-by: Pierre Habouzit <madcoder@debian.org>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

qsbr urcu: simplifications and fixes.

need_mb is unused in qsbr, get rid of it.
add missing static's

Signed-off-by: Pierre Habouzit <madcoder@debian.org>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Build urcu-qsbr.so library, update README

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

urcu qsbr: add DEBUG_RCU self check

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

update README

- Add comment about new threads and signals
- document DEBUG_RCU

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

urcu qsbr: move thread online after add reader

For a matter of symmetry with the unregister, make thread online only after it
has been added as reader.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

qsbr rcu: rc_register_thread also make the thread online.

Else, all its RCU reader critical sections aren't safe until its first
_rcu_quiescent_state() call.

Signed-off-by: Pierre Habouzit <madcoder@debian.org>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

update rcu_dereference comment

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Document access ordering of rcu_dereference wrt VSS

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

add global include Makefile

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Move test programs to tests/ subdir

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

urcu.c: declare noop urcu_init() function non-static

The RCU library is sometimes used in constructor context. In these cases,
because it is impossible to predict whether the liburcu constructor has already
been called, an explicit call to the urcu_init function must be made.
Declaring the noop version of the urcu_init() function as static breaks code
that calls urcu_init() explicitly when CONFIG_AVOID_SIGNALS is true. To prevent
this, this patch makes the declaration non-static.

From: Pierre-Marc Fournier <pierre-marc.fournier@polymtl.ca>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Move DEBUG_FULL_MB to properly named CONFIG_URCU_AVOID_SIGNALS

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Add SA_RESTART flag to signal handler

With this flag, the kernel restarts automatically some system calls
that were interrupted by the signal. Without this flag, these system
calls would fail with errno = EINTR.

Even with this flag, the kernel cannot restart some system calls, and
developers should verify whether these system calls failed with EINTR
and restart them manually in that case. For the list of system calls
that cannot be restarted automatically, see signal(7).

Because of this, we cannot completely eliminate the side-effects of
liburcu on the application, but I think we should try our best and
set SA_RESTART.

From: Pierre-Marc Fournier <pierre-marc.fournier@polymtl.ca>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

update license

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Update license files and readme

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Re-license arch_atomic_s390.h and arch_s390.h

Mathieu Desnoyers asked for re-license the low-level architecture specific
bits under MIT style license.

Signed-off-by: Jan Blunck <jblunck@suse.de>

s390: Fix some compiler warnings

Fix warning about _atomic_exchange being defined but unused and explicitly
cast the 2nd argument.

Signed-off-by: Jan Blunck <jblunck@suse.de>

Add S390 support

This patch adds support for the S390 architecture.

Signed-off-by: Jan Blunck <jblunck@suse.de>

URCU's api_*.h

On Tue, Jun 16, 2009 at 07:09:11PM +0200, Jan Blunck wrote:
> I already did most of that for libkcompat. That was actually why I was
> asking if I should port that over.

Nevertheless, I offer the following patch to userspace-rcu to provide
a gcc-library version of the api_*.h files and to clean up the build a
bit.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

api.h is only needed by urcutorture

Signed-off-by: Jan Blunck <jblunck@suse.de>

Add S390 architecture specific headers

This implements atomic exchange and some other trivial operations like
barriers for the S390 architecture. All of this is based on the
"z/Architecture Principles of Operation" as released by IBM.

Note that api_s390.h is missing so urcutortures applications don't build.

Signed-off-by: Jan Blunck <jblunck@suse.de>

Add runpaul-phase6.sh

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Fix phase5 test log collection

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Add local reclaim tests

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Update phase3 batch size

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Add phase 5 test, educated guess for batch size 32768

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Update default batch size to 32768, after removal of mutex

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Remote mutex from RCU write-side

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Protect shared affinity table with mutex

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Deal with kernel affinity bug for tests

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Test qsbr debug

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Add verbose debug to test_qsbr

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

urcu: add quick start guide to README

Signed-off-by: Pierre-Marc Fournier <pierre-marc.fournier@polymtl.ca>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Add test phase 4 for striding

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Add verif results

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Add nosched model results (for signal readers)

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Cleanup verif directory.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Add phase 3 : scalability run

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Put back 4096 for runall.sh batch size

Basically, when we get to 16384, it's the error due to non-handled removals that
makes this value "look" faster.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

update runtests.sh

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Update runall.sh default batch size

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Make batch test shorter

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Add 2 phases test

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Add "runpaul.sh" test script

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Fix makefile

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Add more batch sizes to tests, just in case

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Update default value for batch size

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Fix gc tests

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Update summary

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Update makefile64

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Update makefile

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Add GC tests

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Merge branch 'master' of lttng.org:/home/git/userspace-rcu

Fix typo in license file

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Add cpu time

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Add output to runall.log

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Fix test_looplen, use double-precition arithmetic

Here you go!!!

        [root@perfsqh userspace-rcu]# ./test_looplen
        CALIBRATION : 0 cycles per loop

Hmmm...  Probably not what you had in mind.  Breaking out the
floating-point arithmetic:

        [root@perfsqh userspace-rcu]# ./test_looplen
        CALIBRATION : 0.386172 cycles per loop
        time_tot = 4049312, LOOPS = 1048576, TESTS = 10

More helpful?  ;-)

                                                Thanx, Paul

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Other timing tests, update cache line size

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Deal with POWER5+ 256B L3 cachefalse sharing for per thread lock

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Add looplen cycle test

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Fix affinity in test script

You need to change the script!!!  See patch below.  All prior results
with more than 32 CPUs are bogus because the -a arguments confined the
threads to only 32 CPUs (the even-numbered ones from zero to 62).

Allow 64 CPUs.  Really needs to do the calculation based on the
stride and the number of CPUs, but a quick hack for now.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

Added a few points to the readercs test

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Update update fraction test

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Update readcslen test to use NUM_CPUS readers

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Fix perthreadlock test

segmentation fault when 0 readers due to incorrect mutex array indexing (int vs
long types).

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Allow use on high-end Power systems.

Power systems self-identify as ppc64 rather than powerpc.  Add the
corresponding logic to the Makefile.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Update fraction update

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Update update-fraction test

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Update test

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Add runall.sh test

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Fix alignment of test names

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

Align the summary result

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>