Optimize sx locks to use simple atomic operations for the common cases of

obtaining and releasing shared and exclusive locks.  The algorithms for
manipulating the lock cookie are very similar to that rwlocks.  This patch
also adds support for exclusive locks using the same algorithm as mutexes.

A new sx_init_flags() function has been added so that optional flags can be
specified to alter a given locks behavior.  The flags include SX_DUPOK,
SX_NOWITNESS, SX_NOPROFILE, and SX_QUITE which are all identical in nature
to the similar flags for mutexes.

Adaptive spinning on select locks may be enabled by enabling the
ADAPTIVE_SX kernel option.  Only locks initialized with the SX_ADAPTIVESPIN
flag via sx_init_flags() will adaptively spin.

The common cases for sx_slock(), sx_sunlock(), sx_xlock(), and sx_xunlock()
are now performed inline in non-debug kernels.  As a result, <sys/sx.h> now
requires <sys/lock.h> to be included prior to <sys/sx.h>.

The new kernel option SX_NOINLINE can be used to disable the aforementioned
inlining in non-debug kernels.

The size of struct sx has changed, so the kernel ABI is probably greatly
disturbed.

MFC after:	1 month
Submitted by:	attilio
Tested by:	kris, pjd

1 files changed, 58 insertions, 4 deletions

diff --git a/share/man/man9/sx.9 b/share/man/man9/sx.9
index 447e48c..c0037d1 100644
--- a/share/man/man9/sx.9
+++ b/share/man/man9/sx.9
@@ -32,6 +32,7 @@
 .Sh NAME
 .Nm sx ,
 .Nm sx_init ,
+.Nm sx_init_flags ,
 .Nm sx_destroy ,
 .Nm sx_slock ,
 .Nm sx_xlock ,
@@ -54,6 +55,8 @@
 .Ft void
 .Fn sx_init "struct sx *sx" "const char *description"
 .Ft void
+.Fn sx_init_flags "struct sx *sx" "const char *description" "int opts"
+.Ft void
 .Fn sx_destroy "struct sx *sx"
 .Ft void
 .Fn sx_slock "struct sx *sx"
@@ -87,11 +90,14 @@
 .Sh DESCRIPTION
 Shared/exclusive locks are used to protect data that are read far more often
 than they are written.
-Mutexes are inherently more efficient than shared/exclusive locks, so
+Shared/exclusive locks do not implement priority propagation like mutexes and
+reader/writer locks to prevent priority inversions, so
 shared/exclusive locks should be used prudently.
 .Pp
-Shared/exclusive locks are created with
-.Fn sx_init ,
+Shared/exclusive locks are created with either
+.Fn sx_init
+or
+.Fn sx_init_flags
 where
 .Fa sx
 is a pointer to space for a
@@ -100,8 +106,40 @@ and
 .Fa description
 is a pointer to a null-terminated character string that describes the
 shared/exclusive lock.
+The
+.Fa opts
+argument to
+.Fn sx_init_flags
+specifies a set of optional flags to alter the behavior of
+.Fa sx .
+It contains one or more of the following flags:
+.Bl -tag -width SX_ADAPTIVESPIN
+.It Dv SX_ADAPTIVESPIN
+If the kernel is compiled with
+.Cd "options ADAPTIVE_SX" ,
+then lock operations for
+.Fa sx
+will spin instead of sleeping while an exclusive lock holder is executing on
+another CPU.
+.It Dv SX_DUPOK
+Witness should not log messages about duplicate locks being acquired.
+.It Dv SX_NOWITNESS
+Instruct
+.Xr witness 4
+to ignore this lock.
+.It Dv SX_NOPROFILE
+Do not profile this lock.
+.It Dv SX_QUIET
+Do not log any operations for this lock via
+.Xr ktr 4 .
+.El
+.Pp
 Shared/exclusive locks are destroyed with
 .Fn sx_destroy .
+The lock
+.Fa sx
+must not be locked by any thread when it is destroyed.
+.Pp
 Threads acquire and release a shared lock by calling
 .Fn sx_slock
 or
@@ -155,7 +193,7 @@ function tests
 for the assertions specified in
 .Fa what ,
 and panics if they are not met.
-The following assertions are supported:
+One of the following assertions must be specified:
 .Bl -tag -width ".Dv SX_UNLOCKED"
 .It Dv SX_LOCKED
 Assert that the current thread has either a shared or an exclusive lock on the
@@ -178,6 +216,22 @@ lock pointed to
 by the first argument.
 .El
 .Pp
+In addition, one of the following optional assertions may be included with
+either an
+.Dv SX_LOCKED ,
+.Dv SX_SLOCKED ,
+or
+.Dv SX_XLOCKED
+assertion:
+.Bl -tag -width ".Dv SX_NOTRECURSED"
+.It Dv SX_RECURSED
+Assert that the current thread has a recursed lock on
+.Fa sx .
+.It Dv SX_NOTRECURSED
+Assert that the current thread does not have a recursed lock on
+.Fa sx .
+.El
+.Pp
 .Fn sx_xlocked
 will return non-zero if the current thread holds the exclusive lock;
 otherwise, it will return zero.