Merge CAM locking changes from the projects/camlock branch to radically

reduce lock congestion and improve SMP scalability of the SCSI/ATA stack,
preparing the ground for the coming next GEOM direct dispatch support.

Replace big per-SIM locks with bunch of smaller ones:
 - per-LUN locks to protect device and peripheral drivers state;
 - per-target locks to protect list of LUNs on target;
 - per-bus locks to protect reference counting;
 - per-send queue locks to protect queue of CCBs to be sent;
 - per-done queue locks to protect queue of completed CCBs;
 - remaining per-SIM locks now protect only HBA driver internals.

While holding LUN lock it is allowed (while not recommended for performance
reasons) to take SIM lock.  The opposite acquisition order is forbidden.
All the other locks are leaf locks, that can be taken anywhere, but should
not be cascaded.  Many functions, such as: xpt_action(), xpt_done(),
xpt_async(), xpt_create_path(), etc. are no longer require (but allow) SIM
lock to be held.

To keep compatibility and solve cases where SIM lock can't be dropped, all
xpt_async() calls in addition to xpt_done() calls are queued to completion
threads for async processing in clean environment without SIM lock held.

Instead of single CAM SWI thread, used for commands completion processing
before, use multiple (depending on number of CPUs) threads.  Load balanced
between them using "hash" of the device B:T:L address.

HBA drivers that can drop SIM lock during completion processing and have
sufficient number of completion threads to efficiently scale to multiple
CPUs can use new function xpt_done_direct() to avoid extra context switch.
Make ahci(4) driver to use this mechanism depending on hardware setup.

Sponsored by:	iXsystems, Inc.
MFC after:	2 months

1 files changed, 14 insertions, 14 deletions

diff --git a/sys/cam/cam_sim.c b/sys/cam/cam_sim.c
index 530e160..9a8c666 100644
--- a/sys/cam/cam_sim.c
+++ b/sys/cam/cam_sim.c
@@ -87,7 +87,6 @@ cam_sim_alloc(sim_action_func sim_action, sim_poll_func sim_poll,
 	sim->flags = 0;
 	sim->refcount = 1;
 	sim->devq = queue;
-	sim->max_ccbs = 8;	/* Reserve for management purposes. */
 	sim->mtx = mtx;
 	if (mtx == &Giant) {
 		sim->flags |= 0;
@@ -96,17 +95,12 @@ cam_sim_alloc(sim_action_func sim_action, sim_poll_func sim_poll,
 		sim->flags |= CAM_SIM_MPSAFE;
 		callout_init(&sim->callout, 1);
 	}
-
-	SLIST_INIT(&sim->ccb_freeq);
-	TAILQ_INIT(&sim->sim_doneq);
-
 	return (sim);
 }
 
 void
 cam_sim_free(struct cam_sim *sim, int free_devq)
 {
-	union ccb *ccb;
 	int error;
 
 	mtx_assert(sim->mtx, MA_OWNED);
@@ -118,10 +112,6 @@ cam_sim_free(struct cam_sim *sim, int free_devq)
 
 	KASSERT(sim->refcount == 0, ("sim->refcount == 0"));
 
-	while ((ccb = (union ccb *)SLIST_FIRST(&sim->ccb_freeq)) != NULL) {
-		SLIST_REMOVE_HEAD(&sim->ccb_freeq, xpt_links.sle);
-		xpt_free_ccb(ccb);
-	}
 	if (free_devq)
 		cam_simq_free(sim->devq);
 	free(sim, M_CAMSIM);
@@ -130,21 +120,31 @@ cam_sim_free(struct cam_sim *sim, int free_devq)
 void
 cam_sim_release(struct cam_sim *sim)
 {
-	KASSERT(sim->refcount >= 1, ("sim->refcount >= 1"));
-	mtx_assert(sim->mtx, MA_OWNED);
+	int lock;
 
+	lock = (mtx_owned(sim->mtx) == 0);
+	if (lock)
+		CAM_SIM_LOCK(sim);
+	KASSERT(sim->refcount >= 1, ("sim->refcount >= 1"));
 	sim->refcount--;
 	if (sim->refcount == 0)
 		wakeup(sim);
+	if (lock)
+		CAM_SIM_UNLOCK(sim);
 }
 
 void
 cam_sim_hold(struct cam_sim *sim)
 {
-	KASSERT(sim->refcount >= 1, ("sim->refcount >= 1"));
-	mtx_assert(sim->mtx, MA_OWNED);
+	int lock;
 
+	lock = (mtx_owned(sim->mtx) == 0);
+	if (lock)
+		CAM_SIM_LOCK(sim);
+	KASSERT(sim->refcount >= 1, ("sim->refcount >= 1"));
 	sim->refcount++;
+	if (lock)
+		CAM_SIM_UNLOCK(sim);
 }
 
 void