In the current world order, each socket has two mutexes: a mutex

that protects socket and receive socket buffer state, and a second
mutex to protect send socket buffer state.  In some places, the
mutex shared between the socket and receive socket buffer will be
acquired twice, once by each layer, resulting in some
inconsistency, but providing the abstraction benefit of being able
to more easily separate the two mutexes in the future if desired.

When transitioning a socket to the SS_ISDISCONNECTING or
SS_ISDISCONNECTED states, grab the socket/receive socket buffer lock
once rather than grabbing it as the socket lock, modifying socket
state, then grabbing a second time as the receive lock in order to
modify the socket buffer state to indicate no further data can be
read.  This change is believed to close a race between the change in
socket state and the change in socket buffer state, which for a
remotely initiated close on a UNIX domain socket, resulted in
soreceive() returning ENOTCONN rather than an EOF condition.

A similar race still exists in the case of send, however, and is
harder to fix as the socket and send socket buffer mutexes are not
the same, and we would like to avoid holding combinations of socket
mutexes over sb_upcall until we've finished clarifying the locking
protocol for upcalls.

This change has the side affect of reducing the number of mutex
operations to initiate disconnect or perform disconnect on a
socket by two.

PR:		78824
Rerported by:	Marc Olzheim <marcolz@stack.nl>
MFC after:	2 weeks

1 files changed, 6 insertions, 13 deletions

diff --git a/sys/kern/uipc_sockbuf.c b/sys/kern/uipc_sockbuf.c
index 0db83df..cb1a24c 100644
--- a/sys/kern/uipc_sockbuf.c
+++ b/sys/kern/uipc_sockbuf.c
@@ -159,15 +159,12 @@ soisdisconnecting(so)
 {
 
 	/*
-	 * XXXRW: This code separately acquires SOCK_LOCK(so) and
-	 * SOCKBUF_LOCK(&so->so_rcv) even though they are the same mutex to
-	 * avoid introducing the assumption  that they are the same.
+	 * XXXRW: This code assumes that SOCK_LOCK(so) and
+	 * SOCKBUF_LOCK(&so->so_rcv) are the same.
 	 */
-	SOCK_LOCK(so);
+	SOCKBUF_LOCK(&so->so_rcv);
 	so->so_state &= ~SS_ISCONNECTING;
 	so->so_state |= SS_ISDISCONNECTING;
-	SOCK_UNLOCK(so);
-	SOCKBUF_LOCK(&so->so_rcv);
 	so->so_rcv.sb_state |= SBS_CANTRCVMORE;
 	sorwakeup_locked(so);
 	SOCKBUF_LOCK(&so->so_snd);
@@ -182,16 +179,12 @@ soisdisconnected(so)
 {
 
 	/*
-	 * XXXRW: This code separately acquires SOCK_LOCK(so) and
-	 * SOCKBUF_LOCK(&so->so_rcv) even though they are the same mutex to
-	 * avoid introducing the assumption  that they are the same.
+	 * XXXRW: This code assumes that SOCK_LOCK(so) and
+	 * SOCKBUF_LOCK(&so->so_rcv) are the same.
 	 */
-	/* XXXRW: so_state locking? */
-	SOCK_LOCK(so);
+	SOCKBUF_LOCK(&so->so_rcv);
 	so->so_state &= ~(SS_ISCONNECTING|SS_ISCONNECTED|SS_ISDISCONNECTING);
 	so->so_state |= SS_ISDISCONNECTED;
-	SOCK_UNLOCK(so);
-	SOCKBUF_LOCK(&so->so_rcv);
 	so->so_rcv.sb_state |= SBS_CANTRCVMORE;
 	sorwakeup_locked(so);
 	SOCKBUF_LOCK(&so->so_snd);