Reduce the number of unnecessary unlock-relocks on socket buffer mutexes

associated with performing a wakeup on the socket buffer:

- When performing an sbappend*() followed by a so[rw]wakeup(), explicitly
  acquire the socket buffer lock and use the _locked() variants of both
  calls.  Note that the _locked() sowakeup() versions unlock the mutex on
  return.  This is done in uipc_send(), divert_packet(), mroute
  socket_send(), raw_append(), tcp_reass(), tcp_input(), and udp_append().

- When the socket buffer lock is dropped before a sowakeup(), remove the
  explicit unlock and use the _locked() sowakeup() variant.  This is done
  in soisdisconnecting(), soisdisconnected() when setting the can't send/
  receive flags and dropping data, and in uipc_rcvd() which adjusting
  back-pressure on the sockets.

For UNIX domain sockets running mpsafe with a contention-intensive SMP
mysql benchmark, this results in a 1.6% query rate improvement due to
reduce mutex costs.

1 files changed, 6 insertions, 4 deletions

diff --git a/sys/netinet/ip_divert.c b/sys/netinet/ip_divert.c
index ef4bb85..f31a7f8 100644
--- a/sys/netinet/ip_divert.c
+++ b/sys/netinet/ip_divert.c
@@ -228,12 +228,14 @@ divert_packet(struct mbuf *m, int incoming)
 		/* XXX why does only one socket match? */
 		if (inp->inp_lport == nport) {
 			sa = inp->inp_socket;
-			if (sbappendaddr(&sa->so_rcv,
+			SOCKBUF_LOCK(&sa->so_rcv);
+			if (sbappendaddr_locked(&sa->so_rcv,
 			    (struct sockaddr *)&divsrc, m,
-			    (struct mbuf *)0) == 0)
+			    (struct mbuf *)0) == 0) {
 				sa = NULL;	/* force mbuf reclaim below */
-			else
-				sorwakeup(sa);
+				SOCKBUF_UNLOCK(&sa->so_rcv);
+			} else
+				sorwakeup_locked(sa);
 			INP_UNLOCK(inp);
 			break;
 		}