Add description of bounded sleep vs unbounded sleep (aka blocking). Move

rules into their own section.

1 files changed, 61 insertions, 80 deletions

diff --git a/share/man/man9/locking.9 b/share/man/man9/locking.9
index 45d7ecf..cbf7470 100644
--- a/share/man/man9/locking.9
+++ b/share/man/man9/locking.9
@@ -73,38 +73,15 @@ code.
 Mutexes are the most commonly used synchronization primitive in the kernel.
 Thread acquires (locks) a mutex before accessing data shared with other
 threads (including interrupt threads), and releases (unlocks) it afterwards.
-If the mutex cannot be acquired, the thread requesting it will block.
-.Pp
-Sleeping while holding mutex is generally prohibited.
-You may only call the
-.Xr sleep 9
-call via
-.Fn msleep
-or the new
-.Fn mtx_sleep
-variant.
-These will atomically drop the mutex and reacquire it
-as part of waking up.
-This is often however a
-.Em BAD
-idea because it generally relies on you having
-such a good knowledge of all the call graph above you
-and what assumptions it is making that there are a lot
-of ways to make hard-to-find mistakes.
-For example you MUST re-test all the assumptions you made before,
-all the way up the call graph to where you got the lock.
-You can not just assume that mtx_sleep can be inserted anywhere.
-If any caller above you has any mutex or
-rwlock, your sleep, will cause a panic.
-If the sleep only happens rarely it may be years before the 
-bad code path is found.
+If the mutex cannot be acquired, the thread requesting it will sleep.
+Mutexes fully support priority propagation.
 .Pp
 See the
 .Xr mutex 9
 page for more information.
 .Ss Spin mutexes
 Spin mutexes are variation of basic mutexes; the main difference between
-the two is that spin mutexes never block - instead, they spin, waiting
+the two is that spin mutexes never sleep - instead, they spin, waiting
 for the thread holding the lock, which runs on another CPU, to release it.
 Differently from ordinary mutex, spin mutexes disable interrupts when acquired.
 Since disabling interrupts is expensive, they are also generally slower.
@@ -136,23 +113,12 @@ A thread with exclusive access is known as a
 .Em writer
 since it may modify protected data.
 .Pp
-Although reader/writer locks look very similar to
-.Xr sx 9
-(see below) locks, their usage pattern is different.
 Reader/writer locks can be treated as mutexes (see above and
 .Xr mutex 9 )
 with shared/exclusive semantics.
 More specifically, regular mutexes can be 
 considered to be equivalent to a write-lock on an
 .Em rw_lock.
-In the future this may in fact
-become literally the fact.
-An
-.Em rw_lock
-can be locked while holding a regular mutex, but 
-can
-.Em not
-be held while sleeping.
 The
 .Em rw_lock
 locks have priority propagation like mutexes, but priority
@@ -164,19 +130,15 @@ Another important property is that shared holders of
 can recurse, but exclusive locks are not allowed to recurse.
 This ability should not be used lightly and 
 .Em may go away.
-Users of recursion in any locks should be prepared to 
-defend their decision against vigorous criticism.
 .Pp
 See the
 .Xr rwlock 9
 page for more information.
 .Ss Read-mostly locks
 Mostly reader locks are similar to
-.Em Reader/write
-locks but optimized for very infrequent 
-.Em writer
-locking.
-.Em rm_lock
+.Em reader/writer
+locks but optimized for very infrequent write locking.
+.Em Read-mostly
 locks implement full priority propagation by tracking shared owners
 using a lock user supplied
 .Em tracker
@@ -186,27 +148,12 @@ See the
 .Xr rmlock 9
 page for more information.
 .Ss Shared/exclusive locks
-Shared/exclusive locks are used to protect data that are read far more often
-than they are written.
-Mutexes are inherently more efficient than shared/exclusive locks, so
-shared/exclusive locks should be used prudently.
-The main reason for using an
-.Em sx_lock
-is that a thread may hold a shared or exclusive lock on an
-.Em sx_lock
-lock while sleeping.
-As a consequence of this however, an
-.Em sx_lock
-lock may not be acquired while holding a mutex.
-The reason for this is that, if one thread slept while holding an
-.Em sx_lock
-lock while another thread blocked on the same
-.Em sx_lock
-lock after acquiring a mutex, then the second thread would effectively
-end up sleeping while holding a mutex, which is not allowed.
-The
-.Em sx_lock
-should be considered to be closely related to
+Shared/exclusive locks are similar to reader/writer locks; the main difference
+between them is that shared/exclusive locks may be held during unbounded sleep
+(and may thus perform an unbounded sleep).
+They are inherently less efficient than mutexes, reader/writer locks
+and read-mostly locks.  They don't support priority propagation.
+They should be considered to be closely related to
 .Xr sleep 9 .
 In fact it could in some cases be 
 considered a conditional sleep.
@@ -239,8 +186,7 @@ See the
 .Xr condvar 9
 page for more information.
 .Ss Giant
-Giant is a special instance of a sleep lock.
-It has several special characteristics.
+Giant is an instance of a mutex, with some special characteristics:
 .Bl -enum
 .It
 It is recursive.
@@ -255,7 +201,7 @@ Giant must be locked first before other locks.
 There are places in the kernel that drop Giant and pick it back up
 again.
 Sleep locks will do this before sleeping.
-Parts of the Network or VM code may do this as well, depending on the
+Parts of the network or VM code may do this as well, depending on the
 setting of a sysctl.
 This means that you cannot count on Giant keeping other code from
 running if your code sleeps, even if you want it to.
@@ -321,7 +267,7 @@ See the
 page for more information.
 .Pp
 .Ss Lockmanager locks
-Shared/exclusive sleep locks, used mostly in
+Shared/exclusive locks, used mostly in
 .Xr VFS 9 ,
 in particular as a
 .Xr vnode 9
@@ -334,19 +280,54 @@ See the
 .Xr lock 9
 page for more information.
 .Sh INTERACTIONS
-.Ss Interaction table.
+.Ss Bounded vs. unbounded sleep
+The following primitives perform bounded sleep: mutexes, pool mutexes,
+reader/writer locks and read-mostly locks.
+.Pp
+The following primitives block (perform unbounded sleep): shared/exclusive locks,
+counting semaphores, condition variables, sleep/wakeup and lockmanager locks.
+.Pp
+It is an error to do any operation that could result in any kind of sleep while
+holding spin mutex.
+.Pp
+As a general rule, it is an error to do any operation that could result
+in unbounded sleep while holding any primitive from the 'bounded sleep' group.
+For example, it is an error to try to acquire shared/exclusive lock while
+holding mutex, or to try to allocate memory with M_WAITOK while holding
+read-write lock.
+.Pp
+As a special case, it is possible to call
+.Fn sleep 9
+or
+.Fn mtx_sleep 9
+while holding a mutex.
+It will atomically drop the mutex and reacquire it
+as part of waking up.
+This is often however a bad
+idea because it generally relies on you having
+such a good knowledge of all the call graph above you
+and what assumptions it is making that there are a lot
+of ways to make hard-to-find mistakes.
+For example you must re-test all the assumptions you made before,
+all the way up the call graph to where you got the lock.
+You can not just assume that mtx_sleep can be inserted anywhere.
+If any caller above you has any mutex or
+rwlock, your sleep, will cause a panic.
+If the sleep only happens rarely it may be years before the 
+bad code path is found.
+.Ss Interaction table
 The following table shows what you can and can not do if you hold
 one of the synchronization primitives discussed here:
 (someone who knows what they are talking about should write this table)
-.Bl -column ".Ic xxxxxxxxxxxxxxxxxxxx" ".Xr XXXXXXXXX" ".Xr XXXXXXX" ".Xr XXXXXXX" ".Xr XXXXXXX" ".Xr XXXXX" -offset indent
+.Bl -column ".Ic xxxxxxxxxxxxxxxxxxx" ".Xr XXXXXXXXX" ".Xr XXXXXXX" ".Xr XXXXXXX" ".Xr XXXXXXX" ".Xr XXXXXX" -offset indent
 .It Xo
-.Em "You have: You want:" Ta Spin_mtx Ta Slp_mtx Ta sx_lock Ta rw_lock Ta rm_lock Ta sleep
+.Em "You have: You want:" Ta spin mtx Ta mutex Ta sx Ta rwlock Ta rmlock Ta sleep
 .Xc
-.It Ic SPIN mutex  Ta \&ok-1 Ta \&no Ta \&no Ta \&no Ta \&no Ta \&no-3
-.It Ic Sleep mutex Ta \&ok Ta \&ok-1 Ta \&no Ta \&ok Ta \&ok Ta \&no-3
-.It Ic sx_lock     Ta \&ok Ta \&ok Ta \&ok-2 Ta \&ok Ta \&ok Ta \&ok-4
-.It Ic rw_lock     Ta \&ok Ta \&ok Ta \&no Ta \&ok-2 Ta \&ok Ta \&no-3
-.It Ic rm_lock     Ta \&ok Ta \&ok Ta \&no Ta \&ok Ta \&ok-2 Ta \&no
+.It Ic spin mtx  Ta \&ok-1 Ta \&no Ta \&no Ta \&no Ta \&no Ta \&no-3
+.It Ic mutex     Ta \&ok Ta \&ok-1 Ta \&no Ta \&ok Ta \&ok Ta \&no-3
+.It Ic sxlock    Ta \&ok Ta \&ok Ta \&ok-2 Ta \&ok Ta \&ok Ta \&ok-4
+.It Ic rwlock    Ta \&ok Ta \&ok Ta \&no Ta \&ok-2 Ta \&ok Ta \&no-3
+.It Ic rmlock    Ta \&ok Ta \&ok Ta \&no Ta \&ok Ta \&ok-2 Ta \&no
 .El
 .Pp
 .Em *1
@@ -371,12 +352,12 @@ Though one can sleep holding an sx lock, one can also use
 .Fn sx_sleep
 which atomically release this primitive when going to sleep and
 reacquire it on wakeup.
-.Ss Context mode table.
+.Ss Context mode table
 The next table shows what can be used in different contexts.
 At this time this is a rather easy to remember table.
-.Bl -column ".Ic Xxxxxxxxxxxxxxxxxxxx" ".Xr XXXXXXXXX" ".Xr XXXXXXX" ".Xr XXXXXXX" ".Xr XXXXXXX" ".Xr XXXXX" -offset indent
+.Bl -column ".Ic Xxxxxxxxxxxxxxxxxxx" ".Xr XXXXXXXXX" ".Xr XXXXXXX" ".Xr XXXXXXX" ".Xr XXXXXXX" ".Xr XXXXXX" -offset indent
 .It Xo
-.Em "Context:" Ta Spin_mtx Ta Slp_mtx Ta sx_lock Ta rw_lock Ta rm_lock Ta sleep
+.Em "Context:" Ta spin mtx Ta mutex Ta sx Ta rwlock Ta rmlock Ta sleep
 .Xc
 .It interrupt:  Ta \&ok Ta \&no Ta \&no Ta \&no Ta \&no Ta \&no 
 .It idle:  Ta \&ok Ta \&no Ta \&no Ta \&no Ta \&no Ta \&no