When we read data from socket buffer using soreceive() the socket layer

does not clear m_nextpkt for us. The mbufs are sent into netgraph and
then, if they contain a TCP packet delivered locally, they will enter
socket code again. They can pass the first assert in sbappendstream()
because m_nextpkt may be set not in the first mbuf, but deeper in the
chain. So the problem will trigger much later, when local program
reads the data from socket, and an mbuf with m_nextpkt becomes a
first one.

This bug was demasked by revision 1.54, when I made upcall queueable.
Before revision 1.54 there was a very small probability to have 2
mbufs in GRE socket buffer, because ng_ksocket_incoming2() dequeued
the first one immediately.

 - in ng_ksocket_incoming2() clear m_nextpkt on all mbufs
   read from socket.
 - restore rev. 1.54 change in ng_ksocket_incoming().

PR:			kern/84952
PR:			kern/82413
In collaboration with:	rwatson

1 files changed, 13 insertions, 4 deletions

diff --git a/sys/netgraph/ng_ksocket.c b/sys/netgraph/ng_ksocket.c
index a4fbdfe..e2f466c 100644
--- a/sys/netgraph/ng_ksocket.c
+++ b/sys/netgraph/ng_ksocket.c
@@ -1001,6 +1001,8 @@ ng_ksocket_disconnect(hook_p hook)
  * the request has at least been done, but the 'so' may not be so lucky.
  * handle this by checking the validity of the node in the target function
  * before dereferencing the socket pointer.
+ *
+ * To decouple stack, we use queue version of ng_send_fn().
  */
 
 static void
@@ -1011,7 +1013,7 @@ ng_ksocket_incoming(struct socket *so, void *arg, int waitflag)
 
 	wait = (waitflag & M_WAITOK) ? NG_WAITOK : 0;
 	ng_send_fn1(node, NULL, &ng_ksocket_incoming2, so, waitflag,
-	    wait);
+	    wait | NG_QUEUE);
 }
 
 
@@ -1111,10 +1113,17 @@ ng_ksocket_incoming2(node_p node, hook_p hook, void *arg1, int waitflag)
 			break;
 		}
 
-		/* Don't trust the various socket layers to get the
-		   packet header and length correct (eg. kern/15175) */
-		for (n = m, m->m_pkthdr.len = 0; n != NULL; n = n->m_next)
+		/*
+		 * Don't trust the various socket layers to get the
+		 * packet header and length correct (e.g. kern/15175).
+		 *
+		 * Also, do not trust that soreceive() will clear m_nextpkt
+		 * for us (e.g. kern/84952, kern/82413).
+		 */
+		for (n = m, m->m_pkthdr.len = 0; n != NULL; n = n->m_next) {
 			m->m_pkthdr.len += n->m_len;
+			n->m_nextpkt = NULL;
+		}
 
 		/* Put peer's socket address (if any) into a tag */
 		if (sa != NULL) {