rmlock(9) two additions and one change/fix:

 - add rm_try_rlock().
 - add RM_SLEEPABLE to use sx(9) as the back-end lock in order to sleep while
   holding the write lock.
 - change rm_noreadtoken to a cpu bitmask to indicate which CPUs need to go
   through the lock/unlock in order to synchronize.  As a side effect, this
   also avoids IPI to CPUs without any readers during rm_wlock.

Discussed with:		ups@, rwatson@ on arch@
Sponsored by:		Isilon Systems, Inc.

3 files changed, 37 insertions, 3 deletions

diff --git a/share/man/man9/Makefile b/share/man/man9/Makefile
index b438b90..e6d8881 100644
--- a/share/man/man9/Makefile
+++ b/share/man/man9/Makefile
@@ -986,6 +986,7 @@ MLINKS+=rman.9 rman_activate_resource.9 \
 MLINKS+=rmlock.9 rm_destroy.9 \
 	rmlock.9 rm_init.9 \
 	rmlock.9 rm_rlock.9 \
+	rmlock.9 rm_try_rlock.9 \
 	rmlock.9 rm_runlock.9 \
 	rmlock.9 RM_SYSINIT.9 \
 	rmlock.9 rm_wlock.9 \
diff --git a/share/man/man9/locking.9 b/share/man/man9/locking.9
index bd8a233..f2ff906 100644
--- a/share/man/man9/locking.9
+++ b/share/man/man9/locking.9
@@ -301,7 +301,7 @@ one of the synchronization primitives discussed:
 .It mutex     Ta \&ok Ta \&ok-1 Ta \&no Ta \&ok Ta \&ok Ta \&no-3
 .It sx        Ta \&ok Ta \&ok Ta \&ok-2 Ta \&ok Ta \&ok Ta \&ok-4
 .It rwlock    Ta \&ok Ta \&ok Ta \&no Ta \&ok-2 Ta \&ok Ta \&no-3
-.It rmlock    Ta \&ok Ta \&ok Ta \&no Ta \&ok Ta \&ok-2 Ta \&no
+.It rmlock    Ta \&ok Ta \&ok Ta \&ok-5 Ta \&ok Ta \&ok-2 Ta \&ok-5
 .El
 .Pp
 .Em *1
@@ -326,6 +326,13 @@ Though one can sleep holding an sx lock, one can also use
 .Fn sx_sleep
 which will atomically release this primitive when going to sleep and
 reacquire it on wakeup.
+.Pp
+.Em *5
+.Em Read-mostly
+locks can be initialized to support sleeping while holding a write lock.
+See
+.Xr rmlock 9
+for details.
 .Ss Context mode table
 The next table shows what can be used in different contexts.
 At this time this is a rather easy to remember table.
diff --git a/share/man/man9/rmlock.9 b/share/man/man9/rmlock.9
index e99661d..2e68b07 100644
--- a/share/man/man9/rmlock.9
+++ b/share/man/man9/rmlock.9
@@ -35,6 +35,7 @@
 .Nm rm_init_flags ,
 .Nm rm_destroy ,
 .Nm rm_rlock ,
+.Nm rm_try_rlock ,
 .Nm rm_wlock ,
 .Nm rm_runlock ,
 .Nm rm_wunlock ,
@@ -53,6 +54,8 @@
 .Fn rm_destroy "struct rmlock *rm"
 .Ft void
 .Fn rm_rlock "struct rmlock *rm"  "struct rm_priotracker* tracker"
+.Ft int
+.Fn rm_try_rlock "struct rmlock *rm"  "struct rm_priotracker* tracker"
 .Ft void
 .Fn rm_wlock "struct rmlock *rm"
 .Ft void
@@ -84,14 +87,16 @@ Although reader/writer locks look very similar to
 locks, their usage pattern is different.
 Reader/writer locks can be treated as mutexes (see
 .Xr mutex 9 )
-with shared/exclusive semantics.
+with shared/exclusive semantics unless initialized with
+.Dv RM_SLEEPABLE .
 Unlike
 .Xr sx 9 ,
 an
 .Nm
 can be locked while holding a non-spin mutex, and an
 .Nm
-cannot be held while sleeping.
+cannot be held while sleeping, again unless initialized with
+.Dv RM_SLEEPABLE .
 The
 .Nm
 locks have full priority propagation like mutexes.
@@ -135,6 +140,13 @@ to ignore this lock.
 .It Dv RM_RECURSE
 Allow threads to recursively acquire exclusive locks for
 .Fa rm .
+.It Dv RM_SLEEPABLE
+Allow writers to sleep while holding the lock.
+Readers must not sleep while holding the lock and can avoid to sleep on
+taking the lock by using
+.Fn rm_try_rlock
+instead of
+.Fn rm_rlock .
 .El
 .It Fn rm_rlock "struct rmlock *rm" "struct rm_priotracker* tracker"
 Lock
@@ -161,6 +173,16 @@ access on
 .Fa rm .
 This is called
 .Dq "recursing on a lock" .
+.It Fn rm_try_rlock "struct rmlock *rm" "struct rm_priotracker* tracker"
+Try to lock
+.Fa rm
+as a reader.
+.Fn rm_try_rlock
+will return 0 if the lock cannot be acquired immediately;
+otherwise the lock will be acquired and a non-zero value will be returned.
+Note that
+.Fn rm_try_rlock
+may fail even while the lock is not currently held by a writer.
 .It Fn rm_wlock "struct rmlock *rm"
 Lock
 .Fa rm
@@ -217,6 +239,10 @@ The
 .Nm
 implementation is currently not optimized for single processor systems.
 .Pp
+.Fn rm_try_rlock
+can fail transiently even when there is no writer, while another reader
+updates the state on the local CPU.
+.Pp
 The
 .Nm
 implementation uses a single per CPU list shared by all