Add implementation of robust mutexes, hopefully close enough to the

intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013.

A robust mutex is guaranteed to be cleared by the system upon either
thread or process owner termination while the mutex is held.  The next
mutex locker is then notified about inconsistent mutex state and can
execute (or abandon) corrective actions.

The patch mostly consists of small changes here and there, adding
neccessary checks for the inconsistent and abandoned conditions into
existing paths.  Additionally, the thread exit handler was extended to
iterate over the userspace-maintained list of owned robust mutexes,
unlocking and marking as terminated each of them.

The list of owned robust mutexes cannot be maintained atomically
synchronous with the mutex lock state (it is possible in kernel, but
is too expensive).  Instead, for the duration of lock or unlock
operation, the current mutex is remembered in a special slot that is
also checked by the kernel at thread termination.

Kernel must be aware about the per-thread location of the heads of
robust mutex lists and the current active mutex slot.  When a thread
touches a robust mutex for the first time, a new umtx op syscall is
issued which informs about location of lists heads.

The umtx sleep queues for PP and PI mutexes are split between
non-robust and robust.

Somewhat unrelated changes in the patch:
1. Style.
2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared
   pi mutexes.
3. Removal of the userspace struct pthread_mutex m_owner field.
4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls
   the lifetime of the shared mutex associated with a vnode' page.

Reviewed by:	jilles (previous version, supposedly the objection was fixed)
Discussed with:	brooks, Martin Simmons <martin@lispworks.com> (some aspects)
Tested by:	pho
Sponsored by:	The FreeBSD Foundation

8 files changed, 199 insertions, 11 deletions

diff --git a/share/man/man3/Makefile b/share/man/man3/Makefile
index 2a3c12a..72e4c54 100644
--- a/share/man/man3/Makefile
+++ b/share/man/man3/Makefile
@@ -238,6 +238,7 @@ PTHREAD_MAN=	pthread.3 \
 		pthread_multi_np.3 \
 		pthread_mutexattr.3 \
 		pthread_mutexattr_getkind_np.3 \
+		pthread_mutex_consistent.3 \
 		pthread_mutex_destroy.3 \
 		pthread_mutex_init.3 \
 		pthread_mutex_lock.3 \
@@ -312,10 +313,12 @@ PTHREAD_MLINKS+=pthread_multi_np.3 pthread_single_np.3
 PTHREAD_MLINKS+=pthread_mutexattr.3 pthread_mutexattr_destroy.3 \
 		pthread_mutexattr.3 pthread_mutexattr_getprioceiling.3 \
 		pthread_mutexattr.3 pthread_mutexattr_getprotocol.3 \
+		pthread_mutexattr.3 pthread_mutexattr_getrobust.3 \
 		pthread_mutexattr.3 pthread_mutexattr_gettype.3 \
 		pthread_mutexattr.3 pthread_mutexattr_init.3 \
 		pthread_mutexattr.3 pthread_mutexattr_setprioceiling.3 \
 		pthread_mutexattr.3 pthread_mutexattr_setprotocol.3 \
+		pthread_mutexattr.3 pthread_mutexattr_setrobust.3 \
 		pthread_mutexattr.3 pthread_mutexattr_settype.3
 PTHREAD_MLINKS+=pthread_mutexattr_getkind_np.3 pthread_mutexattr_setkind_np.3
 PTHREAD_MLINKS+=pthread_rwlock_rdlock.3 pthread_rwlock_tryrdlock.3
diff --git a/share/man/man3/pthread_cond_wait.3 b/share/man/man3/pthread_cond_wait.3
index c057098..427c347 100644
--- a/share/man/man3/pthread_cond_wait.3
+++ b/share/man/man3/pthread_cond_wait.3
@@ -27,7 +27,7 @@
 .\"
 .\" $FreeBSD$
 .\"
-.Dd February 16, 2006
+.Dd April 29, 2016
 .Dt PTHREAD_COND_WAIT 3
 .Os
 .Sh NAME
@@ -75,13 +75,25 @@ is invalid.
 The specified
 .Fa mutex
 was not locked by the calling thread.
+.It Bq Er EOWNERDEAD
+The argument
+.Fa mutex
+points to a robust mutex and the previous owning thread terminated
+while holding the mutex lock.
+The lock was granted to the caller and it is up to the new owner
+to make the state consistent.
+.It Bq Er ENOTRECOVERABLE
+The state protected by the
+.Fa mutex
+is not recoverable.
 .El
 .Sh SEE ALSO
 .Xr pthread_cond_broadcast 3 ,
 .Xr pthread_cond_destroy 3 ,
 .Xr pthread_cond_init 3 ,
 .Xr pthread_cond_signal 3 ,
-.Xr pthread_cond_timedwait 3
+.Xr pthread_cond_timedwait 3 ,
+.Xr pthread_mutex_consistent 3
 .Sh STANDARDS
 The
 .Fn pthread_cond_wait
diff --git a/share/man/man3/pthread_mutex_consistent.3 b/share/man/man3/pthread_mutex_consistent.3
new file mode 100644
index 0000000..67bfbcb
--- /dev/null
+++ b/share/man/man3/pthread_mutex_consistent.3
@@ -0,0 +1,94 @@
+.\" Copyright (c) 2016 The FreeBSD Foundation, Inc.
+.\" All rights reserved.
+.\"
+.\" This documentation was written by
+.\" Konstantin Belousov <kib@FreeBSD.org> under sponsorship
+.\" from the FreeBSD Foundation.
+.\"
+.\" Redistribution and use in source and binary forms, with or without
+.\" modification, are permitted provided that the following conditions
+.\" are met:
+.\" 1. Redistributions of source code must retain the above copyright
+.\"    notice, this list of conditions and the following disclaimer.
+.\" 2. Redistributions in binary form must reproduce the above copyright
+.\"    notice, this list of conditions and the following disclaimer in the
+.\"    documentation and/or other materials provided with the distribution.
+.\"
+.\" THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND
+.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+.\" ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE
+.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+.\" SUCH DAMAGE.
+.\"
+.\" $FreeBSD$
+.\"
+.Dd May 8, 2016
+.Dt PTHREAD_MUTEX_CONSISTENT 3
+.Os
+.Sh NAME
+.Nm pthread_mutex_consistent
+.Nd mark state protected by robust mutex as consistent
+.Sh LIBRARY
+.Lb libpthread
+.Sh SYNOPSIS
+.In pthread.h
+.Ft int
+.Fn pthread_mutex_consistent "pthread_mutex_t *mutex"
+.Sh DESCRIPTION
+If the thread owning a robust mutex terminates while holding the
+mutex, the mutex becomes inconsistent and the next thread that
+acquires the mutex lock is notified of the state by the return value
+.Er EOWNERDEAD .
+In this case, the mutex does not become normally usable again until
+the state is marked consistent.
+.Pp
+The
+.Fn pthread_mutex_consistent ,
+when called with the
+.Fa mutex
+argument, which points to the initialized robust mutex in an
+inconsistent state, marks the by mutex as consistent again.
+The consequent unlock of the mutex, by either
+.Fn pthread_mutex_unlock
+or other methods, allows other contenders to lock the mutex.
+.Pp
+If the mutex in the inconsistent state is not marked consistent
+by the call to
+.Fn pthread_mutex_consistent
+before unlock,
+further attempts to lock the
+.Fa mutex
+result in the
+.Er ENOTRECOVERABLE
+condition reported by the locking functions.
+.Sh RETURN VALUES
+If successful,
+.Fn pthread_mutex_consistent
+will return zero, otherwise an error number will be returned to
+indicate the error.
+.Sh ERRORS
+The
+.Fn pthread_mutex_lock
+function will fail if:
+.Bl -tag -width Er
+.It Bq Er EINVAL
+The mutex pointed to by the
+.Fa mutex
+argument is not robust, or is not in the inconsistent state.
+.El
+.Sh SEE ALSO
+.Xr pthread_mutexattr_setrobust 3 ,
+.Xr pthread_mutex_init 3 ,
+.Xr pthread_mutex_lock 3 ,
+.Xr pthread_mutex_unlock 3
+.Sh STANDARDS
+The
+.Fn pthread_mutex_lock
+function conforms to
+.St -susv4 .
diff --git a/share/man/man3/pthread_mutex_lock.3 b/share/man/man3/pthread_mutex_lock.3
index 8479a69..bd94380 100644
--- a/share/man/man3/pthread_mutex_lock.3
+++ b/share/man/man3/pthread_mutex_lock.3
@@ -27,7 +27,7 @@
 .\"
 .\" $FreeBSD$
 .\"
-.Dd January 31, 2006
+.Dd April 29, 2016
 .Dt PTHREAD_MUTEX_LOCK 3
 .Os
 .Sh NAME
@@ -55,7 +55,7 @@ indicate the error.
 The
 .Fn pthread_mutex_lock
 function will fail if:
-.Bl -tag -width Er
+.Bl -tag -width "Er ENOTRECOVERABLE"
 .It Bq Er EINVAL
 The value specified by
 .Fa mutex
@@ -63,8 +63,20 @@ is invalid.
 .It Bq Er EDEADLK
 A deadlock would occur if the thread blocked waiting for
 .Fa mutex .
+.It Bq Er EOWNERDEAD
+The argument
+.Fa mutex
+points to a robust mutex and the previous owning thread terminated
+while holding the mutex lock.
+The lock was granted to the caller and it is up to the new owner
+to make the state consistent.
+.It Bq Er ENOTRECOVERABLE
+The state protected by the
+.Fa mutex
+is not recoverable.
 .El
 .Sh SEE ALSO
+.Xr pthread_mutex_consistent 3 ,
 .Xr pthread_mutex_destroy 3 ,
 .Xr pthread_mutex_init 3 ,
 .Xr pthread_mutex_trylock 3 ,
diff --git a/share/man/man3/pthread_mutex_timedlock.3 b/share/man/man3/pthread_mutex_timedlock.3
index abc7e2a..384ee9d 100644
--- a/share/man/man3/pthread_mutex_timedlock.3
+++ b/share/man/man3/pthread_mutex_timedlock.3
@@ -24,7 +24,7 @@
 .\"
 .\" $FreeBSD$
 .\"
-.Dd December 30, 2003
+.Dd April 29, 2016
 .Dt PTHREAD_MUTEX_TIMEDLOCK 3
 .Os
 .Sh NAME
@@ -59,7 +59,7 @@ The
 .Fn pthread_mutex_timedlock
 function will fail if:
 .Bl -tag -width Er
-.It Bq Er EINVAL
+.It Bq "Er ENOTRECOVERABLE"
 The
 .Fa mutex
 was created with the protocol attribute having the
@@ -89,8 +89,20 @@ has been exceeded.
 .It Bq Er EDEADLK
 The current thread already owns the
 .Fa mutex .
+.It Bq Er EOWNERDEAD
+The argument
+.Fa mutex
+points to a robust mutex and the previous owning thread terminated
+while holding the mutex lock.
+The lock was granted to the caller and it is up to the new owner
+to make the state consistent.
+.It Bq Er ENOTRECOVERABLE
+The state protected by the
+.Fa mutex
+is not recoverable.
 .El
 .Sh SEE ALSO
+.Xr pthread_mutex_consistent 3 ,
 .Xr pthread_mutex_destroy 3 ,
 .Xr pthread_mutex_init 3 ,
 .Xr pthread_mutex_lock 3 ,
diff --git a/share/man/man3/pthread_mutex_trylock.3 b/share/man/man3/pthread_mutex_trylock.3
index 049006f..2837704 100644
--- a/share/man/man3/pthread_mutex_trylock.3
+++ b/share/man/man3/pthread_mutex_trylock.3
@@ -27,7 +27,7 @@
 .\"
 .\" $FreeBSD$
 .\"
-.Dd July 30, 1998
+.Dd April 29, 2016
 .Dt PTHREAD_MUTEX_TRYLOCK 3
 .Os
 .Sh NAME
@@ -56,7 +56,7 @@ indicate the error.
 The
 .Fn pthread_mutex_trylock
 function will fail if:
-.Bl -tag -width Er
+.Bl -tag -width "Er ENOTRECOVERABLE"
 .It Bq Er EINVAL
 The value specified by
 .Fa mutex
@@ -64,8 +64,20 @@ is invalid.
 .It Bq Er EBUSY
 .Fa Mutex
 is already locked.
+.It Bq Er EOWNERDEAD
+The argument
+.Fa mutex
+points to a robust mutex and the previous owning thread terminated
+while holding the mutex lock.
+The lock was granted to the caller and it is up to the new owner
+to make the state consistent.
+.It Bq Er ENOTRECOVERABLE
+The state protected by the
+.Fa mutex
+is not recoverable.
 .El
 .Sh SEE ALSO
+.Xr pthread_mutex_consistent 3 ,
 .Xr pthread_mutex_destroy 3 ,
 .Xr pthread_mutex_init 3 ,
 .Xr pthread_mutex_lock 3 ,
diff --git a/share/man/man3/pthread_mutex_unlock.3 b/share/man/man3/pthread_mutex_unlock.3
index 77784e1..4d3badd 100644
--- a/share/man/man3/pthread_mutex_unlock.3
+++ b/share/man/man3/pthread_mutex_unlock.3
@@ -27,7 +27,7 @@
 .\"
 .\" $FreeBSD$
 .\"
-.Dd July 30, 1998
+.Dd April 29, 2016
 .Dt PTHREAD_MUTEX_UNLOCK 3
 .Os
 .Sh NAME
@@ -46,6 +46,17 @@ then the
 .Fn pthread_mutex_unlock
 function unlocks
 .Fa mutex .
+.Pp
+If the argument pointed by the
+.Fa mutex
+is a robust mutex in the inconsistent state, and the call to
+.Fn pthread_mutex_consistent
+function was not done prior to unlocking, further locking attempts on
+the mutex
+.Fa mutex
+are denied and locking functions return
+.Er ENOTRECOVERABLE
+error.
 .Sh RETURN VALUES
 If successful,
 .Fn pthread_mutex_unlock
diff --git a/share/man/man3/pthread_mutexattr.3 b/share/man/man3/pthread_mutexattr.3
index f976026..e7042d5 100644
--- a/share/man/man3/pthread_mutexattr.3
+++ b/share/man/man3/pthread_mutexattr.3
@@ -26,7 +26,7 @@
 .\" EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 .\"
 .\" $FreeBSD$
-.Dd May 1, 2000
+.Dd April 29, 2016
 .Dt PTHREAD_MUTEXATTR 3
 .Os
 .Sh NAME
@@ -36,6 +36,8 @@
 .Nm pthread_mutexattr_getprioceiling ,
 .Nm pthread_mutexattr_setprotocol ,
 .Nm pthread_mutexattr_getprotocol ,
+.Nm pthread_mutexattr_setrobust ,
+.Nm pthread_mutexattr_getrobust ,
 .Nm pthread_mutexattr_settype ,
 .Nm pthread_mutexattr_gettype
 .Nd mutex attribute operations
@@ -56,6 +58,10 @@
 .Ft int
 .Fn pthread_mutexattr_getprotocol "pthread_mutexattr_t *attr" "int *protocol"
 .Ft int
+.Fn pthread_mutexattr_setrobust "pthread_mutexattr_t *attr" "int robust"
+.Ft int
+.Fn pthread_mutexattr_getrobust "pthread_mutexattr_t *attr" "int *robust"
+.Ft int
 .Fn pthread_mutexattr_settype "pthread_mutexattr_t *attr" "int type"
 .Ft int
 .Fn pthread_mutexattr_gettype "pthread_mutexattr_t *attr" "int *type"
@@ -165,6 +171,26 @@ function will fail if:
 Invalid value for
 .Fa attr .
 .El
+.Pp
+The
+.Fn pthread_mutexattr_setrobust
+function will fail if:
+.Bl -tag -width Er
+.It Bq Er EINVAL
+Invalid value for
+.Fa attr ,
+or invalid value for
+.Fa robust .
+.El
+.Pp
+The
+.Fn pthread_mutexattr_getrobust
+function will fail if:
+.Bl -tag -width Er
+.It Bq Er EINVAL
+Invalid value for
+.Fa attr .
+.El
 .Sh SEE ALSO
 .Xr pthread_mutex_init 3
 .Sh STANDARDS
@@ -184,4 +210,10 @@ The
 and
 .Fn pthread_mutexattr_gettype
 functions conform to
-.St -susv2
+.St -susv2 .
+The
+.Fn pthread_mutexattr_setrobust
+and
+.Fn pthread_mutexattr_getrobust
+functions conform to
+.St -susv4 .