MFC Alexander Motin's direct dispatch, multi-queue, and finer-grained

locking support for CAM

r256826:
Fix several target mode SIMs to not blindly clear ccb_h.flags field of
ATIO CCBs.  Not all CCB flags there belong to them.

r256836:
Remove hard limit on number of BIOs handled with one ATA TRIM request.

r256843:
Merge CAM locking changes from the projects/camlock branch to radically
reduce lock congestion and improve SMP scalability of the SCSI/ATA stack,
preparing the ground for the coming next GEOM direct dispatch support.

r256888:
Unconditionally acquire periph reference on CCB allocation failure.

r256895:
Fix memory and references leak due to unfreed path.

r256960:
Move CAM_UNQUEUED_INDEX setting to the last moment and under the periph lock.
This fixes race condition with cam_periph_ccbwait(), causing use-after-free.

r256975:
Minor (mostly cosmetical) addition to r256960.

r257054:
Some microoptimizations for da and ada drivers:
 - Replace ordered_tag_count counter with single flag;
 - From da remove outstanding_cmds counter, duplicating pending_ccbs list;
 - From da_softc remove unused links field.

r257482:
Fix lock recursion, triggered by `smartctl -a /dev/adaX`.

r257501:
Make getenv_*() functions and respectively TUNABLE_*_FETCH() macros not
allocate memory and so not require sleepable environment.  getenv() has
already used on-stack temporary storage, so just use it more rationally.
getenv_string() receives buffer as argument, so don't need another one.

r257914:
Some CAM locks polishing:
 - Fix LOR and possible lock recursion when handling high-power commands.
Introduce new lock to protect left power quota and list of frozen devices.
 - Correct locking around xpt periph creation.
 - Remove seems never used XPT_FLAG_OPEN xpt periph flag.

Again, Netflix assisted with testing the merge, but all of the credit goes
to Alexander and iX Systems.

Submitted by:	mav
Sponsored by:	iX Systems

1 files changed, 7 insertions, 4 deletions

diff --git a/sys/cam/cam_queue.h b/sys/cam/cam_queue.h
index 0bb4491..0f74e82 100644
--- a/sys/cam/cam_queue.h
+++ b/sys/cam/cam_queue.h
@@ -33,6 +33,8 @@
 
 #ifdef _KERNEL
 
+#include <sys/lock.h>
+#include <sys/mutex.h>
 #include <sys/queue.h>
 #include <cam/cam.h>
 
@@ -59,8 +61,8 @@ struct cam_ccbq {
 	struct	camq queue;
 	struct ccb_hdr_tailq	queue_extra_head;
 	int	queue_extra_entries;
+	int	total_openings;
 	int	devq_openings;
-	int	devq_allocating;
 	int	dev_openings;
 	int	dev_active;
 	int	held;
@@ -69,9 +71,10 @@ struct cam_ccbq {
 struct cam_ed;
 
 struct cam_devq {
-	struct	camq send_queue;
-	int	send_openings;
-	int	send_active;
+	struct mtx	 send_mtx;
+	struct camq	 send_queue;
+	int		 send_openings;
+	int		 send_active;
 };