MFC Alexander Motin's GEOM direct dispatch work:

r256603:
Introduce new function devstat_end_transaction_bio_bt(), adding new argument
to specify present time.  Use this function to move binuptime() out of lock,
substantially reducing lock congestion when slow timecounter is used.

r256606:
Move g_io_deliver() out of the lock, as required for direct dispatch.
Move g_destroy_bio() out too to reduce lock scope even more.

r256607:
Fix passing uninitialized bio_resid argument to g_trace().

r256610:
Add unmapped I/O support to GEOM RAID.

r256830:
Restore BIO_UNMAPPED and BIO_TRANSIENT_MAPPING in biodonne() when unmapping
temporary mapped buffer.  That fixes double unmap if biodone() called twice
for the same BIO (but with different done methods).

r256880:
Merge GEOM direct dispatch changes from the projects/camlock branch.

When safety requirements are met, it allows to avoid passing I/O requests
to GEOM g_up/g_down thread, executing them directly in the caller context.
That allows to avoid CPU bottlenecks in g_up/g_down threads, plus avoid
several context switches per I/O.

r259247:
Fix bug introduced at r256607.  We have to recalculate bp_resid here since
sizes of original and completed requests may differ due to end of media.

Testing of the stable/10 merge was done by Netflix, but all of the credit
goes to Alexander and iX Systems.

Submitted by:   mav
Sponsored by:   iX Systems

1 files changed, 15 insertions, 18 deletions

diff --git a/sys/geom/geom_vfs.c b/sys/geom/geom_vfs.c
index 92f1ad2..eda4b75 100644
--- a/sys/geom/geom_vfs.c
+++ b/sys/geom/geom_vfs.c
@@ -102,14 +102,10 @@ g_vfs_done(struct bio *bip)
 	/*
 	 * Collect statistics on synchronous and asynchronous read
 	 * and write counts for disks that have associated filesystems.
-	 * Since this run by the g_up thread it is single threaded and
-	 * we do not need to use atomic increments on the counters.
 	 */
 	bp = bip->bio_caller2;
 	vp = bp->b_vp;
-	if (vp == NULL) {
-		mp = NULL;
-	} else {
+	if (vp != NULL) {
 		/*
 		 * If not a disk vnode, use its associated mount point
 		 * otherwise use the mountpoint associated with the disk.
@@ -122,20 +118,20 @@ g_vfs_done(struct bio *bip)
 			mp = vp->v_mount;
 		else
 			mp = cdevp->si_mountpt;
-		VI_UNLOCK(vp);
-	}
-	if (mp != NULL) {
-		if (bp->b_iocmd == BIO_WRITE) {
-			if (LK_HOLDER(bp->b_lock.lk_lock) == LK_KERNPROC)
-				mp->mnt_stat.f_asyncwrites++;
-			else
-				mp->mnt_stat.f_syncwrites++;
-		} else {
-			if (LK_HOLDER(bp->b_lock.lk_lock) == LK_KERNPROC)
-				mp->mnt_stat.f_asyncreads++;
-			else
-				mp->mnt_stat.f_syncreads++;
+		if (mp != NULL) {
+			if (bp->b_iocmd == BIO_READ) {
+				if (LK_HOLDER(bp->b_lock.lk_lock) == LK_KERNPROC)
+					mp->mnt_stat.f_asyncreads++;
+				else
+					mp->mnt_stat.f_syncreads++;
+			} else if (bp->b_iocmd == BIO_WRITE) {
+				if (LK_HOLDER(bp->b_lock.lk_lock) == LK_KERNPROC)
+					mp->mnt_stat.f_asyncwrites++;
+				else
+					mp->mnt_stat.f_syncwrites++;
+			}
 		}
+		VI_UNLOCK(vp);
 	}
 
 	cp = bip->bio_from;
@@ -260,6 +256,7 @@ g_vfs_open(struct vnode *vp, struct g_consumer **cpp, const char *fsname, int wr
 	vnode_create_vobject(vp, pp->mediasize, curthread);
 	*cpp = cp;
 	cp->private = vp;
+	cp->flags |= G_CF_DIRECT_SEND | G_CF_DIRECT_RECEIVE;
 	bo->bo_ops = g_vfs_bufops;
 	bo->bo_private = cp;
 	bo->bo_bsize = pp->sectorsize;