Commit | Line | Data |
---|---|---|
d589a916 PP |
1 | Userspace RCU Implementation |
2 | ============================ | |
3 | ||
4 | by Mathieu Desnoyers and Paul E. McKenney | |
5 | ||
6 | ||
7 | Building | |
8 | -------- | |
9 | ||
10 | ./bootstrap # skip if using tarball | |
11 | ./configure | |
12 | make | |
13 | make install | |
14 | ldconfig | |
15 | ||
16 | Hints: | |
17 | ||
18 | - Forcing 32-bit build: | |
19 | ||
20 | CFLAGS="-m32 -g -O2" ./configure | |
21 | ||
22 | - Forcing 64-bit build: | |
23 | ||
24 | CFLAGS="-m64 -g -O2" ./configure | |
25 | ||
26 | - Forcing a 32-bit build with 386 backward compatibility: | |
27 | ||
28 | CFLAGS="-m32 -g -O2" ./configure --host=i386-pc-linux-gnu | |
29 | ||
30 | - Forcing a 32-bit build for Sparcv9 (typical for Sparc v9) | |
31 | ||
32 | CFLAGS="-m32 -Wa,-Av9a -g -O2" ./configure | |
33 | ||
34 | ||
35 | Architectures supported | |
36 | ----------------------- | |
37 | ||
38 | Currently, the following architectures are supported: | |
39 | ||
f328865f | 40 | - x86 (i386, i486, i586, i686) |
72886af7 | 41 | - amd64 / x86\_64 |
d589a916 PP |
42 | - PowerPC 32/64 |
43 | - S390, S390x | |
44 | - ARM 32/64 | |
45 | - MIPS | |
859050b3 | 46 | - NIOS2 |
d589a916 PP |
47 | - Alpha |
48 | - ia64 | |
49 | - Sparcv9 32/64 | |
50 | - Tilera | |
51 | - hppa/PA-RISC | |
f328865f MD |
52 | - m68k |
53 | - RISC-V | |
7ec50625 | 54 | - LoongArch |
f328865f MD |
55 | |
56 | Tested on: | |
57 | ||
58 | - Linux all architectures | |
59 | - FreeBSD 8.2/8.3/9.0/9.1/10.0 i386/amd64 | |
60 | - Solaris 10/11 i386 | |
61 | - Cygwin i386/amd64 | |
0af4b40c | 62 | - MacOS amd64/arm64 |
d589a916 | 63 | |
d589a916 PP |
64 | Should also work on: |
65 | ||
66 | - Android | |
67 | - NetBSD 5 | |
68 | - OpenBSD | |
d589a916 PP |
69 | |
70 | (more testing needed before claiming support for these OS). | |
71 | ||
72 | Linux ARM depends on running a Linux kernel 2.6.15 or better, GCC 4.4 or | |
73 | better. | |
74 | ||
4d1f67b9 MD |
75 | The C compiler used needs to support at least C99. The C++ compiler used |
76 | needs to support at least C++11. | |
77 | ||
d589a916 PP |
78 | The GCC compiler versions 3.3, 3.4, 4.0, 4.1, 4.2, 4.3, 4.4 and 4.5 are |
79 | supported, with the following exceptions: | |
80 | ||
81 | - GCC 3.3 and 3.4 have a bug that prevents them from generating volatile | |
82 | accesses to offsets in a TLS structure on 32-bit x86. These versions are | |
83 | therefore not compatible with `liburcu` on x86 32-bit | |
84 | (i386, i486, i586, i686). | |
85 | The problem has been reported to the GCC community: | |
efa4515d | 86 | <http://www.mail-archive.com/gcc-bugs@gcc.gnu.org/msg281255.html> |
d589a916 | 87 | - GCC 3.3 cannot match the "xchg" instruction on 32-bit x86 build. |
efa4515d | 88 | See <http://kerneltrap.org/node/7507> |
d589a916 PP |
89 | - Alpha, ia64 and ARM architectures depend on GCC 4.x with atomic builtins |
90 | support. For ARM this was introduced with GCC 4.4: | |
efa4515d | 91 | <http://gcc.gnu.org/gcc-4.4/changes.html>. |
ddec79fd MD |
92 | - Linux aarch64 depends on GCC 5.1 or better because prior versions |
93 | perform unsafe access to deallocated stack. | |
d589a916 PP |
94 | |
95 | Clang version 3.0 (based on LLVM 3.0) is supported. | |
96 | ||
ef6da886 MJ |
97 | Glibc >= 2.4 should work but the older version we test against is |
98 | currently 2.17. | |
99 | ||
d589a916 PP |
100 | For developers using the Git tree: |
101 | ||
102 | This source tree is based on the autotools suite from GNU to simplify | |
103 | portability. Here are some things you should have on your system in order to | |
104 | compile the git repository tree : | |
105 | ||
afb6113f | 106 | - GNU autotools (automake >=1.12, autoconf >=2.69) |
d589a916 PP |
107 | (make sure your system wide `automake` points to a recent version!) |
108 | - GNU Libtool >=2.2 | |
efa4515d | 109 | (for more information, go to <http://www.gnu.org/software/autoconf/>) |
d589a916 PP |
110 | |
111 | If you get the tree from the repository, you will need to use the `bootstrap` | |
112 | script in the root of the tree. It calls all the GNU tools needed to prepare | |
113 | the tree configuration. | |
114 | ||
115 | Test scripts provided in the `tests/` directory of the source tree depend | |
116 | on `bash` and the `seq` program. | |
117 | ||
118 | ||
119 | API | |
120 | --- | |
121 | ||
122 | See the relevant API documentation files in `doc/`. The APIs provided by | |
123 | Userspace RCU are, by prefix: | |
124 | ||
dcb9c05a | 125 | - `rcu_`: Read-Copy Update (see [`doc/rcu-api.md`](doc/rcu-api.md)) |
d589a916 PP |
126 | - `cmm_`: Concurrent Memory Model |
127 | - `caa_`: Concurrent Architecture Abstraction | |
128 | - `cds_`: Concurrent Data Structures | |
dcb9c05a | 129 | (see [`doc/cds-api.md`](doc/cds-api.md)) |
d589a916 | 130 | - `uatomic_`: Userspace Atomic |
dcb9c05a | 131 | (see [`doc/uatomic-api.md`](doc/uatomic-api.md)) |
d589a916 PP |
132 | |
133 | ||
134 | Quick start guide | |
135 | ----------------- | |
136 | ||
137 | ### Usage of all urcu libraries: | |
138 | ||
139 | - Define `_LGPL_SOURCE` (only) if your code is LGPL or GPL compatible | |
140 | before including the `urcu.h` or `urcu-qsbr.h` header. If your application | |
141 | is distributed under another license, function calls will be generated | |
142 | instead of inlines, so your application can link with the library. | |
143 | - Linking with one of the libraries below is always necessary even for | |
144 | LGPL and GPL applications. | |
145 | - Define `URCU_INLINE_SMALL_FUNCTIONS` before including Userspace RCU | |
146 | headers if you want Userspace RCU to inline small functions (10 | |
147 | lines or less) into the application. It can be used by applications | |
148 | distributed under any kind of license, and does *not* make the | |
149 | application a derived work of Userspace RCU. | |
150 | ||
151 | Those small inlined functions are guaranteed to match the library | |
152 | content as long as the library major version is unchanged. | |
153 | Therefore, the application *must* be compiled with headers matching | |
154 | the library major version number. Applications using | |
155 | `URCU_INLINE_SMALL_FUNCTIONS` may be unable to use debugging | |
156 | features of Userspace RCU without being recompiled. | |
157 | ||
f328865f MD |
158 | There are multiple flavors of liburcu available: |
159 | ||
160 | - `memb`, | |
161 | - `qsbr`, | |
162 | - `mb`, | |
163 | - `signal`, | |
164 | - `bp`. | |
165 | ||
72886af7 WY |
166 | The API members start with the prefix `urcu_<flavor>_`, where |
167 | `<flavor>` is the chosen flavor name. | |
f328865f | 168 | |
d589a916 | 169 | |
f328865f | 170 | ### Usage of `liburcu-memb` |
d589a916 | 171 | |
f328865f MD |
172 | 1. `#include <urcu/urcu-memb.h>` |
173 | 2. Link the application with `-lurcu-memb` | |
d589a916 PP |
174 | |
175 | This is the preferred version of the library, in terms of | |
176 | grace-period detection speed, read-side speed and flexibility. | |
177 | Dynamically detects kernel support for `sys_membarrier()`. Falls back | |
178 | on `urcu-mb` scheme if support is not present, which has slower | |
cef5f31d | 179 | read-side. Use the `--disable-sys-membarrier-fallback` configure option |
d8d9a340 MD |
180 | to disable the fall back, thus requiring `sys_membarrier()` to be |
181 | available. This gives a small speedup when `sys_membarrier()` is | |
182 | supported by the kernel, and aborts in the library constructor if not | |
183 | supported. | |
d589a916 PP |
184 | |
185 | ||
186 | ### Usage of `liburcu-qsbr` | |
187 | ||
f328865f | 188 | 1. `#include <urcu/urcu-qsbr.h>` |
d589a916 PP |
189 | 2. Link with `-lurcu-qsbr` |
190 | ||
191 | The QSBR flavor of RCU needs to have each reader thread executing | |
192 | `rcu_quiescent_state()` periodically to progress. `rcu_thread_online()` | |
193 | and `rcu_thread_offline()` can be used to mark long periods for which | |
194 | the threads are not active. It provides the fastest read-side at the | |
195 | expense of more intrusiveness in the application code. | |
196 | ||
197 | ||
198 | ### Usage of `liburcu-mb` | |
199 | ||
f328865f MD |
200 | 1. `#include <urcu/urcu-mb.h>` |
201 | 2. Link with `-lurcu-mb` | |
d589a916 PP |
202 | |
203 | This version of the urcu library uses memory barriers on the writer | |
204 | and reader sides. This results in faster grace-period detection, but | |
205 | results in slower reads. | |
206 | ||
207 | ||
208 | ### Usage of `liburcu-signal` | |
209 | ||
f328865f MD |
210 | 1. `#include <urcu/urcu-signal.h>` |
211 | 2. Link the application with `-lurcu-signal` | |
d589a916 PP |
212 | |
213 | Version of the library that requires a signal, typically `SIGUSR1`. Can | |
214 | be overridden with `-DSIGRCU` by modifying `Makefile.build.inc`. | |
215 | ||
216 | ||
217 | ### Usage of `liburcu-bp` | |
218 | ||
f328865f | 219 | 1. `#include <urcu/urcu-bp.h>` |
d589a916 PP |
220 | 2. Link with `-lurcu-bp` |
221 | ||
222 | The BP library flavor stands for "bulletproof". It is specifically | |
223 | designed to help tracing library to hook on applications without | |
5b46e39d MD |
224 | requiring to modify these applications. `urcu_bp_init()`, and |
225 | `urcu_bp_unregister_thread()` all become nops, whereas calling | |
226 | `urcu_bp_register_thread()` becomes optional. The state is dealt with by | |
227 | the library internally at the expense of read-side and write-side | |
228 | performance. | |
d589a916 PP |
229 | |
230 | ||
231 | ### Initialization | |
232 | ||
233 | Each thread that has reader critical sections (that uses | |
f328865f MD |
234 | `urcu_<flavor>_read_lock()`/`urcu_<flavor>_read_unlock()` must first |
235 | register to the URCU library. This is done by calling | |
236 | `urcu_<flavor>_register_thread()`. Unregistration must be performed | |
237 | before exiting the thread by using `urcu_<flavor>_unregister_thread()`. | |
d589a916 PP |
238 | |
239 | ||
240 | ### Reading | |
241 | ||
242 | Reader critical sections must be protected by locating them between | |
f328865f MD |
243 | calls to `urcu_<flavor>_read_lock()` and `urcu_<flavor>_read_unlock()`. |
244 | Inside that lock, `rcu_dereference()` may be called to read an RCU | |
245 | protected pointer. | |
d589a916 PP |
246 | |
247 | ||
248 | ### Writing | |
249 | ||
250 | `rcu_assign_pointer()` and `rcu_xchg_pointer()` may be called anywhere. | |
f328865f MD |
251 | After, `urcu_<flavor>_synchronize_rcu()` must be called. When it |
252 | returns, the old values are not in usage anymore. | |
d589a916 | 253 | |
111bda8f MD |
254 | As an alternative to `urcu_<flavor>_synchronize_rcu()`, |
255 | it is also possible to use the urcu polling mechanism to wait for a | |
256 | grace period to elapse. This can be done by using | |
257 | `urcu_<flavor>_start_poll_synchronize_rcu()` | |
258 | to start the grace period polling, and then invoke | |
259 | `urcu_<flavor>_poll_state_synchronize_rcu()`, which returns true if | |
260 | the grace period has completed, false otherwise. | |
261 | ||
d589a916 PP |
262 | |
263 | ### Usage of `liburcu-defer` | |
264 | ||
f328865f | 265 | - Follow instructions for either `liburcu-memb`, `liburcu-qsbr`, |
d589a916 PP |
266 | `liburcu-mb`, `liburcu-signal`, or `liburcu-bp` above. |
267 | The `liburcu-defer` functionality is pulled into each of | |
268 | those library modules. | |
f328865f MD |
269 | - Provides `urcu_<flavor>_defer_rcu()` primitive to enqueue delayed |
270 | callbacks. Queued callbacks are executed in batch periodically after | |
271 | a grace period. Do _not_ use `urcu_<flavor>_defer_rcu()` within a | |
272 | read-side critical section, because it may call | |
273 | `urcu_<flavor>_synchronize_rcu()` if the thread queue is full. This | |
274 | can lead to deadlock or worse. | |
275 | - Requires that `urcu_<flavor>_defer_barrier()` must be called in | |
276 | library destructor if a library queues callbacks and is expected to | |
277 | be unloaded with `dlclose()`. | |
d589a916 PP |
278 | |
279 | Its API is currently experimental. It may change in future library releases. | |
280 | ||
281 | ||
282 | ### Usage of `urcu-call-rcu` | |
283 | ||
f328865f | 284 | - Follow instructions for either `liburcu-memb`, `liburcu-qsbr`, |
d589a916 PP |
285 | `liburcu-mb`, `liburcu-signal`, or `liburcu-bp` above. |
286 | The `urcu-call-rcu` functionality is pulled into each of | |
287 | those library modules. | |
f328865f MD |
288 | - Provides the `urcu_<flavor>_call_rcu()` primitive to enqueue delayed |
289 | callbacks in a manner similar to `urcu_<flavor>_defer_rcu()`, but | |
290 | without ever delaying for a grace period. On the other hand, | |
291 | `urcu_<flavor>_call_rcu()`'s best-case overhead is not quite as good | |
292 | as that of `urcu_<flavor>_defer_rcu()`. | |
293 | - Provides `urcu_<flavor>_call_rcu()` to allow asynchronous handling | |
294 | of RCU grace periods. A number of additional functions are provided | |
295 | to manage the helper threads used by `urcu_<flavor>_call_rcu()`, but | |
296 | reasonable defaults are used if these additional functions are not | |
297 | invoked. See [`doc/rcu-api.md`](doc/rcu-api.md) in userspace-rcu | |
298 | documentation for more details. | |
d589a916 PP |
299 | |
300 | ||
301 | ### Being careful with signals | |
302 | ||
f328865f | 303 | The `liburcu-signal` library uses signals internally. The signal handler is |
d589a916 PP |
304 | registered with the `SA_RESTART` flag. However, these signals may cause |
305 | some non-restartable system calls to fail with `errno = EINTR`. Care | |
306 | should be taken to restart system calls manually if they fail with this | |
307 | error. A list of non-restartable system calls may be found in | |
f328865f | 308 | `signal(7)`. |
d589a916 PP |
309 | |
310 | Read-side critical sections are allowed in a signal handler, | |
f328865f | 311 | except those setup with `sigaltstack(2)`, with `liburcu-memb` and |
d589a916 | 312 | `liburcu-mb`. Be careful, however, to disable these signals |
f328865f MD |
313 | between thread creation and calls to `urcu_<flavor>_register_thread()`, |
314 | because a signal handler nesting on an unregistered thread would not be | |
315 | allowed to call `urcu_<flavor>_read_lock()`. | |
d589a916 PP |
316 | |
317 | Read-side critical sections are _not_ allowed in a signal handler with | |
318 | `liburcu-qsbr`, unless signals are disabled explicitly around each | |
f328865f MD |
319 | `urcu_qsbr_quiescent_state()` calls, when threads are put offline and around |
320 | calls to `urcu_qsbr_synchronize_rcu()`. Even then, we do not recommend it. | |
d589a916 PP |
321 | |
322 | ||
323 | ### Interaction with mutexes | |
324 | ||
325 | One must be careful to do not cause deadlocks due to interaction of | |
f328865f MD |
326 | `urcu_<flavor>_synchronize_rcu()` and RCU read-side with mutexes. If |
327 | `urcu_<flavor>_synchronize_rcu()` is called with a mutex held, this | |
328 | mutex (or any mutex which has this mutex in its dependency chain) should | |
329 | not be acquired from within a RCU read-side critical section. | |
d589a916 PP |
330 | |
331 | This is especially important to understand in the context of the | |
332 | QSBR flavor: a registered reader thread being "online" by | |
333 | default should be considered as within a RCU read-side critical | |
334 | section unless explicitly put "offline". Therefore, if | |
f328865f MD |
335 | `urcu_qsbr_synchronize_rcu()` is called with a mutex held, this mutex, |
336 | as well as any mutex which has this mutex in its dependency chain should | |
337 | only be taken when the RCU reader thread is "offline" (this can be | |
338 | performed by calling `urcu_qsbr_thread_offline()`). | |
d589a916 PP |
339 | |
340 | ||
341 | ### Interaction with `fork()` | |
342 | ||
343 | Special care must be taken for applications performing `fork()` without | |
344 | any following `exec()`. This is caused by the fact that Linux only clones | |
345 | the thread calling `fork()`, and thus never replicates any of the other | |
346 | parent thread into the child process. Most `liburcu` implementations | |
347 | require that all registrations (as reader, `defer_rcu` and `call_rcu` | |
348 | threads) should be released before a `fork()` is performed, except for the | |
349 | rather common scenario where `fork()` is immediately followed by `exec()` in | |
350 | the child process. The only implementation not subject to that rule is | |
351 | `liburcu-bp`, which is designed to handle `fork()` by calling | |
f328865f MD |
352 | `urcu_bp_before_fork`, `urcu_bp_after_fork_parent` and |
353 | `urcu_bp_after_fork_child`. | |
354 | ||
355 | Applications that use `urcu_<flavor>_call_rcu()` and that `fork()` | |
356 | without doing an immediate `exec()` must take special action. The | |
357 | parent must invoke `urcu_<flavor>_call_rcu_before_fork()` before the | |
358 | `fork()` and `urcu_<flavor>_call_rcu_after_fork_parent()` after the | |
359 | `fork()`. The child process must invoke | |
360 | `urcu_<flavor>_call_rcu_after_fork_child()`. Even though these three | |
361 | APIs are suitable for passing to `pthread_atfork()`, use of | |
362 | `pthread_atfork()` is **STRONGLY DISCOURAGED** for programs calling the | |
363 | glibc memory allocator (`malloc()`, `calloc()`, `free()`, ...) within | |
364 | `urcu_<flavor>_call_rcu` callbacks. This is due to limitations in the | |
365 | way glibc memory allocator handles calls to the memory allocator from | |
366 | concurrent threads while the `pthread_atfork()` handlers are executing. | |
d589a916 PP |
367 | |
368 | Combining e.g.: | |
369 | ||
f328865f MD |
370 | - call to `free()` from callbacks executed within |
371 | `urcu_<flavor>_call_rcu` worker threads, | |
372 | - executing `urcu_<flavor>_call_rcu` atfork handlers within the glibc | |
373 | pthread atfork mechanism, | |
d589a916 PP |
374 | |
375 | will sometimes trigger interesting process hangs. This usually | |
376 | hangs on a memory allocator lock within glibc. | |
377 | ||
378 | ||
379 | ### Thread Local Storage (TLS) | |
380 | ||
381 | Userspace RCU can fall back on `pthread_getspecific()` to emulate | |
382 | TLS variables on systems where it is not available. This behavior | |
383 | can be forced by specifying `--disable-compiler-tls` as configure | |
384 | argument. | |
385 | ||
386 | ||
d4e640c0 | 387 | ### Usage of `DEBUG_RCU` & `--enable-rcu-debug` |
d589a916 | 388 | |
d4e640c0 JR |
389 | By default the library is configured with internal debugging |
390 | self-checks disabled. | |
391 | ||
392 | For always-on debugging self-checks: | |
af9254ab WY |
393 | |
394 | ./configure --enable-rcu-debug | |
d4e640c0 JR |
395 | |
396 | For fine grained enabling of debugging self-checks, build | |
72886af7 WY |
397 | userspace-rcu with `DEBUG_RCU` defined and compile dependent |
398 | applications with `DEBUG_RCU` defined when necessary. | |
d4e640c0 JR |
399 | |
400 | Warning: Enabling this feature result in a performance penalty. | |
d589a916 PP |
401 | |
402 | ||
403 | ### Usage of `DEBUG_YIELD` | |
404 | ||
405 | `DEBUG_YIELD` is used to add random delays in the code for testing | |
406 | purposes. | |
407 | ||
408 | ||
409 | ### SMP support | |
410 | ||
411 | By default the library is configured to use synchronization primitives | |
412 | adequate for SMP systems. On uniprocessor systems, support for SMP | |
413 | systems can be disabled with: | |
414 | ||
415 | ./configure --disable-smp-support | |
416 | ||
417 | theoretically yielding slightly better performance. | |
418 | ||
419 | ||
d7c76f85 MD |
420 | ### Usage of `--enable-cds-lfht-iter-debug` |
421 | ||
422 | By default the library is configured with extra debugging checks for | |
423 | lock-free hash table iterator traversal disabled. | |
424 | ||
cef5f31d | 425 | Building liburcu with `--enable-cds-lfht-iter-debug` and rebuilding |
d7c76f85 MD |
426 | application to match the ABI change allows finding cases where the hash |
427 | table iterator is re-purposed to be used on a different hash table while | |
428 | still being used to iterate on a hash table. | |
429 | ||
430 | This option alters the rculfhash ABI. Make sure to compile both library | |
431 | and application with matching configuration. | |
432 | ||
433 | ||
d589a916 PP |
434 | Make targets |
435 | ------------ | |
436 | ||
437 | In addition to the usual `make check` target, Userspace RCU features | |
ff59d427 | 438 | `make regtest`, `make short_bench` and `make long_bench` targets: |
d589a916 PP |
439 | |
440 | - `make check`: short tests, meant to be run when rebuilding or | |
441 | porting Userspace RCU. | |
442 | - `make regtest`: long (many hours) test, meant to be run when | |
443 | modifying Userspace RCU or porting it to a new architecture or | |
444 | operating system. | |
ff59d427 STH |
445 | - `make short_bench`: short benchmarks, 3 seconds per test. |
446 | - `make long_bench`: long (many hours) benchmarks, 30 seconds per test. | |
d589a916 PP |
447 | |
448 | ||
43f53c96 MD |
449 | Known issues |
450 | ------------ | |
451 | ||
452 | There is an application vs library compatibility issue between | |
453 | applications built using Userspace RCU 0.10 headers linked against | |
454 | Userspace RCU 0.11 or 0.12 shared objects. The problem occurs as | |
455 | follows: | |
456 | ||
72886af7 | 457 | - An application executable is built with `_LGPL_SOURCE` defined, includes |
43f53c96 | 458 | any of the Userspace RCU 0.10 urcu flavor headers, and is built |
cef5f31d | 459 | without the `-fpic` compiler option. |
43f53c96 MD |
460 | |
461 | - The Userspace RCU 0.10 library shared objects are updated to 0.11 | |
462 | or 0.12 without rebuilding the application. | |
463 | ||
464 | - The application will hang, typically when RCU grace period | |
465 | (synchronize_rcu) is invoked. | |
466 | ||
467 | Some possible work-arounds for this are: | |
468 | ||
469 | - Rebuild the application against Userspace RCU 0.11+. | |
470 | ||
cef5f31d | 471 | - Rebuild the application with `-fpic`. |
43f53c96 MD |
472 | |
473 | - Upgrade Userspace RCU to 0.13+ without installing 0.11 nor 0.12. | |
474 | ||
475 | ||
d589a916 PP |
476 | Contacts |
477 | -------- | |
478 | ||
479 | You can contact the maintainers on the following mailing list: | |
dcb9c05a | 480 | `lttng-dev@lists.lttng.org`. |