Add the CAM Target Layer (CTL).

CTL is a disk and processor device emulation subsystem originally written
for Copan Systems under Linux starting in 2003.  It has been shipping in
Copan (now SGI) products since 2005.

It was ported to FreeBSD in 2008, and thanks to an agreement between SGI
(who acquired Copan's assets in 2010) and Spectra Logic in 2010, CTL is
available under a BSD-style license.  The intent behind the agreement was
that Spectra would work to get CTL into the FreeBSD tree.

Some CTL features:

 - Disk and processor device emulation.
 - Tagged queueing
 - SCSI task attribute support (ordered, head of queue, simple tags)
 - SCSI implicit command ordering support.  (e.g. if a read follows a mode
   select, the read will be blocked until the mode select completes.)
 - Full task management support (abort, LUN reset, target reset, etc.)
 - Support for multiple ports
 - Support for multiple simultaneous initiators
 - Support for multiple simultaneous backing stores
 - Persistent reservation support
 - Mode sense/select support
 - Error injection support
 - High Availability support (1)
 - All I/O handled in-kernel, no userland context switch overhead.

(1) HA Support is just an API stub, and needs much more to be fully
    functional.

ctl.c:			The core of CTL.  Command handlers and processing,
			character driver, and HA support are here.

ctl.h:			Basic function declarations and data structures.

ctl_backend.c,
ctl_backend.h:		The basic CTL backend API.

ctl_backend_block.c,
ctl_backend_block.h:	The block and file backend.  This allows for using
			a disk or a file as the backing store for a LUN.
			Multiple threads are started to do I/O to the
			backing device, primarily because the VFS API
			requires that to get any concurrency.

ctl_backend_ramdisk.c:	A "fake" ramdisk backend.  It only allocates a
			small amount of memory to act as a source and sink
			for reads and writes from an initiator.  Therefore
			it cannot be used for any real data, but it can be
			used to test for throughput.  It can also be used
			to test initiators' support for extremely large LUNs.

ctl_cmd_table.c:	This is a table with all 256 possible SCSI opcodes,
			and command handler functions defined for supported
			opcodes.

ctl_debug.h:		Debugging support.

ctl_error.c,
ctl_error.h:		CTL-specific wrappers around the CAM sense building
			functions.

ctl_frontend.c,
ctl_frontend.h:		These files define the basic CTL frontend port API.

ctl_frontend_cam_sim.c:	This is a CTL frontend port that is also a CAM SIM.
			This frontend allows for using CTL without any
			target-capable hardware.  So any LUNs you create in
			CTL are visible in CAM via this port.

ctl_frontend_internal.c,
ctl_frontend_internal.h:
			This is a frontend port written for Copan to do
			some system-specific tasks that required sending
			commands into CTL from inside the kernel.  This
			isn't entirely relevant to FreeBSD in general,
			but can perhaps be repurposed.

ctl_ha.h:		This is a stubbed-out High Availability API.  Much
			more is needed for full HA support.  See the
			comments in the header and the description of what
			is needed in the README.ctl.txt file for more
			details.

ctl_io.h:		This defines most of the core CTL I/O structures.
			union ctl_io is conceptually very similar to CAM's
			union ccb.

ctl_ioctl.h:		This defines all ioctls available through the CTL
			character device, and the data structures needed
			for those ioctls.

ctl_mem_pool.c,
ctl_mem_pool.h:		Generic memory pool implementation used by the
			internal frontend.

ctl_private.h:		Private data structres (e.g. CTL softc) and
			function prototypes.  This also includes the SCSI
			vendor and product names used by CTL.

ctl_scsi_all.c,
ctl_scsi_all.h:		CTL wrappers around CAM sense printing functions.

ctl_ser_table.c:	Command serialization table.  This defines what
			happens when one type of command is followed by
			another type of command.

ctl_util.c,
ctl_util.h:		CTL utility functions, primarily designed to be
			used from userland.  See ctladm for the primary
			consumer of these functions.  These include CDB
			building functions.

scsi_ctl.c:		CAM target peripheral driver and CTL frontend port.
			This is the path into CTL for commands from
			target-capable hardware/SIMs.

README.ctl.txt:		CTL code features, roadmap, to-do list.

usr.sbin/Makefile:	Add ctladm.

ctladm/Makefile,
ctladm/ctladm.8,
ctladm/ctladm.c,
ctladm/ctladm.h,
ctladm/util.c:		ctladm(8) is the CTL management utility.
			It fills a role similar to camcontrol(8).
			It allow configuring LUNs, issuing commands,
			injecting errors and various other control
			functions.

usr.bin/Makefile:	Add ctlstat.

ctlstat/Makefile
ctlstat/ctlstat.8,
ctlstat/ctlstat.c:	ctlstat(8) fills a role similar to iostat(8).
			It reports I/O statistics for CTL.

sys/conf/files:		Add CTL files.

sys/conf/NOTES:		Add device ctl.

sys/cam/scsi_all.h:	To conform to more recent specs, the inquiry CDB
			length field is now 2 bytes long.

			Add several mode page definitions for CTL.

sys/cam/scsi_all.c:	Handle the new 2 byte inquiry length.

sys/dev/ciss/ciss.c,
sys/dev/ata/atapi-cam.c,
sys/cam/scsi/scsi_targ_bh.c,
scsi_target/scsi_cmds.c,
mlxcontrol/interface.c:	Update for 2 byte inquiry length field.

scsi_da.h:		Add versions of the format and rigid disk pages
			that are in a more reasonable format for CTL.

amd64/conf/GENERIC,
i386/conf/GENERIC,
ia64/conf/GENERIC,
sparc64/conf/GENERIC:	Add device ctl.

i386/conf/PAE:		The CTL frontend SIM at least does not compile
			cleanly on PAE.

Sponsored by:	Copan Systems, SGI and Spectra Logic
MFC after:	1 month

5 files changed, 5195 insertions, 0 deletions

diff --git a/usr.sbin/ctladm/Makefile b/usr.sbin/ctladm/Makefile
new file mode 100644
index 0000000..88e82f4
--- /dev/null
+++ b/usr.sbin/ctladm/Makefile
@@ -0,0 +1,21 @@
+# $FreeBSD$
+
+PROG=		ctladm
+SRCS=		ctladm.c util.c ctl_util.c ctl_scsi_all.c
+.PATH:		${.CURDIR}/../../sys/cam/ctl 
+SDIR=		${.CURDIR}/../../sys
+CFLAGS+=	-I${SDIR}
+# This is necessary because of these warnings:
+# warning: cast increases required alignment of target type
+# The solution is to either upgrade the compiler (preferred), or do void 
+# pointer gymnastics to get around the warning.  For now, disable the
+# warning instead of doing the void pointer workaround.
+.if ${MACHINE_CPUARCH} == "arm"
+WARNS?= 3
+.endif
+
+DPADD=		${LIBCAM} ${LIBSBUF}
+LDADD=		-lcam -lsbuf -lbsdxml
+MAN=		ctladm.8
+
+.include <bsd.prog.mk>
diff --git a/usr.sbin/ctladm/ctladm.8 b/usr.sbin/ctladm/ctladm.8
new file mode 100644
index 0000000..027d89e
--- /dev/null
+++ b/usr.sbin/ctladm/ctladm.8
@@ -0,0 +1,963 @@
+.\" 
+.\" Copyright (c) 2003 Silicon Graphics International Corp.
+.\" All rights reserved.
+.\" 
+.\" Redistribution and use in source and binary forms, with or without
+.\" modification, are permitted provided that the following conditions
+.\" are met:
+.\" 1. Redistributions of source code must retain the above copyright
+.\"    notice, this list of conditions, and the following disclaimer,
+.\"    without modification.
+.\" 2. Redistributions in binary form must reproduce at minimum a disclaimer
+.\"    substantially similar to the "NO WARRANTY" disclaimer below
+.\"    ("Disclaimer") and any redistribution must be conditioned upon
+.\"    including a substantially similar Disclaimer requirement for further
+.\"    binary redistribution.
+.\" 
+.\" NO WARRANTY
+.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTIBILITY AND FITNESS FOR
+.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+.\" HOLDERS OR CONTRIBUTORS BE LIABLE FOR SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+.\" STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
+.\" IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+.\" POSSIBILITY OF SUCH DAMAGES.
+.\" 
+.\" ctladm utility man page.
+.\"
+.\" Author: Ken Merry <ken@FreeBSD.org>
+.\"
+.\" $Id: //depot/users/kenm/FreeBSD-test2/usr.sbin/ctladm/ctladm.8#3 $
+.\" $FreeBSD$
+.\"
+.Dd July 8, 2011
+.Dt CTLADM 8
+.Os
+.Sh NAME
+.Nm ctladm
+.Nd CAM Target Layer control utility
+.Sh SYNOPSIS
+.Nm
+.Aq Ar command
+.Op target:lun
+.Op generic args
+.Op command args
+.Nm
+.Ic tur
+.Aq target:lun
+.Op general options
+.Nm
+.Ic inquiry
+.Aq target:lun
+.Op general options
+.Nm
+.Ic reqsense
+.Aq target:lun
+.Op general options
+.Nm
+.Ic reportluns
+.Aq target:lun
+.Op general options
+.Nm
+.Ic read
+.Aq target:lun
+.Op general options
+.Aq Fl l Ar lba
+.Aq Fl d Ar datalen
+.Aq Fl f Ar file|-
+.Aq Fl b Ar blocksize_bytes
+.Op Fl c Ar cdbsize
+.Op Fl N
+.Nm
+.Ic write
+.Aq target:lun
+.Op general options
+.Aq Fl l Ar lba
+.Aq Fl d Ar datalen
+.Aq Fl f Ar file|-
+.Aq Fl b Ar blocksize_bytes
+.Op Fl c Ar cdbsize
+.Op Fl N
+.Nm
+.Ic bbrread
+.Aq target:lun
+.Op general options
+.Aq Fl -l Ar lba
+.Aq Fl -d Ar datalen
+.Nm
+.Ic readcap
+.Aq target:lun
+.Op general options
+.Op Fl c Ar cdbsize
+.Nm
+.Ic modesense
+.Aq target:lun
+.Aq Fl m Ar page | Fl l
+.Op Fl P Ar pc
+.Op Fl d
+.Op Fl S Ar subpage
+.Op Fl c Ar size
+.Nm
+.Ic start
+.Aq target:lun
+.Op general options
+.Op Fl i
+.Op Fl o
+.Nm
+.Ic stop
+.Aq target:lun
+.Op general options
+.Op Fl i
+.Op Fl o
+.Nm
+.Ic synccache
+.Aq target:lun
+.Op general options
+.Op Fl l Ar lba
+.Op Fl b Ar blockcount
+.Op Fl r
+.Op Fl i
+.Op Fl c Ar cdbsize
+.Nm
+.Ic shutdown
+.Op general options
+.Nm
+.Ic startup
+.Op general options
+.Nm
+.Ic hardstop
+.Nm
+.Ic hardstart
+.Nm
+.Ic lunlist
+.Nm
+.Ic delay
+.Aq target:lun
+.Aq Fl l Ar datamove|done
+.Aq Fl t Ar secs
+.Op Fl T Ar oneshot|cont
+.Nm
+.Ic realsync Aq on|off|query
+.Nm
+.Ic setsync interval
+.Aq target:lun
+.Aq Fl i Ar interval
+.Nm
+.Ic getsync
+.Aq target:lun
+.Nm
+.Ic inject
+.Aq Fl i Ar action
+.Aq Fl p Ar pattern
+.Op Fl r Ar lba,len
+.Op Fl s Ar len fmt Op Ar args
+.Op Fl c
+.Op Fl d Ar delete_id
+.Nm
+.Ic create
+.Aq Fl b Ar backend
+.Op Fl B Ar blocksize
+.Op Fl d Ar device_id
+.Op Fl l Ar lun_id
+.Op Fl o Ar name=value
+.Op Fl s Ar size_bytes
+.Op Fl S Ar serial_num
+.Op Fl t Ar device_type
+.Nm
+.Ic remove
+.Aq Fl b Ar backend
+.Aq Fl l Ar lun_id
+.Op Fl o Ar name=value
+.Nm
+.Ic devlist
+.Op Fl b Ar backend
+.Op Fl v
+.Op Fl x
+.Nm
+.Ic port
+.Op Fl l
+.Op Fl o Ar on|off
+.Op Fl w Ar wwpn
+.Op Fl W Ar wwnn
+.Op Fl p Ar targ_port
+.Op Fl t Ar fe_type
+.Op Fl q
+.Op Fl x
+.Nm
+.Ic dumpooa
+.Nm
+.Ic dumpstructs
+.Nm
+.Ic help
+.Sh DESCRIPTION
+The
+.Nm
+utility is designed to provide a way to access and control the CAM Target
+Layer (CTL).
+It provides a way to send
+.Tn SCSI
+commands to the CTL layer, and also provides
+some meta-commands that utilize
+.Tn SCSI
+commands.
+(For instance, the
+.Ic lunlist
+command is implemented using the
+.Tn SCSI
+REPORT LUNS and INQUIRY commands.)
+.Pp
+The
+.Nm
+utility has a number of primary functions, many of which require a device
+identifier.
+The device identifier takes the following form:
+.Bl -tag -width 14n
+.It target:lun
+Specify the target (almost always 0) and LUN number to operate on.
+.El
+Many of the primary functions of the
+.Nm
+utility take the following optional arguments:
+.Pp
+.Bl -tag -width 10n
+.It Fl C Ar retries
+Specify the number of times to retry a command in the event of failure.
+.It Fl D Ar device
+Specify the device to open.  This allows opening a device other than the
+default device,
+.Pa /dev/cam/ctl ,
+to be opened for sending commands.
+.It Fl I Ar id
+Specify the initiator number to use.
+By default,
+.Nm
+will use 7 as the initiator number.
+.El
+.Pp
+Primary commands:
+.Bl -tag -width 11n
+.It Ic tur
+Send the
+.Tn SCSI
+TEST UNIT READY command to the device and report whether or not it is
+ready.
+.It Ic inquiry
+Send the
+.Tn SCSI
+INQUIRY command to the device and display some of the returned inquiry
+data.
+.It Ic reqsense
+Send the
+.Tn SCSI
+REQUEST SENSE command to the device and display the returned sense
+information.
+.It Ic reportluns
+Send the
+.Tn SCSI
+REPORT LUNS command to the device and display supported LUNs.
+.It Ic read
+Send a
+.Tn SCSI
+READ command to the device, and write the requested data to a file or
+stdout.
+.Bl -tag -width 12n
+.It Fl l Ar lba
+Specify the starting Logical Block Address for the READ.  This can be
+specified in decimal, octal (starting with 0), hexadecimal (starting with
+0x) or any other base supported by
+.Xr strtoull 3 .
+.It Fl d Ar datalen
+Specify the length, in 512 byte blocks, of the READ request.
+.It Fl f Ar file
+Specify the destination for the data read by the READ command.  Either a
+filename or
+.Sq -
+for stdout may be specified.
+.It Fl c Ar cdbsize
+Specify the minimum
+.Tn SCSI
+CDB (Command Data Block) size to be used for the READ request.  Allowable
+values are 6, 10, 12 and 16.  Depending upon the LBA and amount of data
+requested, a larger CDB size may be used to satisfy the request.  (e.g.,
+for LBAs above 0xffffffff, READ(16) must be used to satisfy the request.)
+.It Fl b Ar blocksize
+Specify the blocksize of the underlying
+.Tn SCSI
+device, so the transfer length
+can be calculated accurately.  The blocksize can be obtained via the
+.Tn SCSI
+READ CAPACITY command.
+.It Fl N 
+Do not copy data to
+.Nm
+from the kernel when doing a read, just execute the command without copying
+data.
+This is to be used for performance testing.
+.El
+.It Ic write
+Read data from a file or stdin, and write the data to the device using the
+.Tn SCSI
+WRITE command.
+.Bl -tag -width 12n
+.It Fl l Ar lba
+Specify the starting Logical Block Address for the WRITE.  This can be
+specified in decimal, octal (starting with 0), hexadecimal (starting with
+0x) or any other base supported by
+.Xr strtoull 3 .
+.It Fl d Ar atalen
+Specify the length, in 512 byte blocks, of the WRITE request.
+.It Fl f Ar file
+Specify the source for the data to be written by the WRITE command.  Either a
+filename or
+.Sq -
+for stdin may be specified.
+.It Fl c Ar cdbsize
+Specify the minimum
+.Tn SCSI
+CDB (Command Data Block) size to be used for the READ request.  Allowable
+values are 6, 10, 12 and 16.  Depending upon the LBA and amount of data
+requested, a larger CDB size may be used to satisfy the request.  (e.g.,
+for LBAs above 0xffffffff, READ(16) must be used to satisfy the request.)
+.It Fl b Ar blocksize
+Specify the blocksize of the underlying
+.Tn SCSI
+device, so the transfer length
+can be calculated accurately.  The blocksize can be obtained via the
+.Tn SCSI
+READ CAPACITY command.
+.It Fl N
+Do not copy data to
+.Nm
+to the kernel when doing a write, just execute the command without copying
+data.
+This is to be used for performance testing.
+.El
+.It Ic bbrread
+Issue a SCSI READ command to the logical device to potentially force a bad
+block on a disk in the RAID set to be reconstructed from the other disks in
+the array.  This command should only be used on an array that is in the
+normal state.  If used on a critical array, it could cause the array to go
+offline if the bad block to be remapped is on one of the disks that is
+still active in the array.
+.Pp
+The data for this particular command will be discarded, and not returned to
+the user.
+.Pp
+In order to determine which LUN to read from, the user should first
+determine which LUN the disk with a bad block belongs to.  Then he should
+map the bad disk block back to the logical block address for the array in
+order to determine which LBA to pass in to the
+.Ic bbrread
+command.
+.Pp
+This command is primarily intended for testing.  In practice, bad block
+remapping will generally be triggered by the in-kernel Disk Aerobics and
+Disk Scrubbing code.
+.Bl -tag -width 10n
+.It Fl l Ar lba
+Specify the starting Logical Block Address.
+.It Fl d Ar datalen
+Specify the amount of data in bytes to read from the LUN.  This must be a
+multiple of the LUN blocksize.
+.El
+.It Ic readcap
+Send the
+.Tn SCSI
+READ CAPACITY command to the device and display the device size and device
+block size.  By default, READ CAPACITY(10) is
+used.  If the device returns a maximum LBA of 0xffffffff, however,
+.Nm
+will automatically issue a READ CAPACITY(16), which is implemented as a
+service action of the SERVICE ACTION IN(16) opcode.  The user can specify
+the minimum CDB size with the
+.Fl c
+argument.  Valid values for the
+.Fl c
+option are 10 and 16.  If a 10 byte CDB is specified, the request will be
+automatically reissued with a 16 byte CDB if the maximum LBA returned is
+0xffffffff.
+.It Ic modesense
+Send a
+.Tn SCSI
+MODE SENSE command to the device, and display the requested mode page(s) or
+page list.
+.Bl -tag -width 10n
+.It Fl m Ar page
+Specify the mode page to display.  This option and the
+.Fl l
+option are mutually exclusive.  One of the two must be specified, though.
+Mode page numbers may be specified in decimal or hexadecimal.
+.It Fl l
+Request that the list of mode pages supported by the device be returned.
+This option and the
+.Fl m
+option are mutually exclusive.  One of the two must be specified, though.
+.It Fl P Ar pc
+Specify the mode page page control value.  Possible values are:
+.Bl -tag -width 2n -compact
+.It 0
+Current values.
+.It 1
+Changeable value bitmask.
+.It 2
+Default values.
+.It 3
+Saved values.
+.El
+.It Fl d
+Disable block descriptors when sending the mode sense request.
+.It Fl S Ar subpage
+Specify the subpage used with the mode sense request.
+.It Fl c Ar cdbsize
+Specify the CDB size used for the mode sense request.  Supported values are
+6 and 10.
+.El
+.It Ic start
+Send the
+.Tn SCSI
+START STOP UNIT command to the specified LUN with the start
+bit set.
+.Bl -tag -width 4n
+.It Fl i
+Set the immediate bit in the CDB.  Note that CTL does not support the
+immediate bit, so this is primarily useful for making sure that CTL returns
+the proper error.
+.It Fl o
+Set the Copan proprietary on/offline bit in the CDB.  When this flag is
+used, the LUN will be marked online again (see the description of the
+.Ic shutdown
+and
+.Ic startup
+commands).  When this flag is used with a
+start command, the LUN will NOT be spun up.  You need to use a start
+command without the
+.Fl o
+flag to spin up the disks in the LUN.
+.El
+.It Ic stop
+Send the
+.Tn SCSI
+START STOP UNIT command to the specified LUN with the start
+bit cleared.  We use an ordered tag to stop the LUN, so we can guarantee
+that all pending I/O executes before it is stopped.  (CTL guarantees this
+anyway, but
+.Nm
+sends an ordered tag for completeness.)
+.Bl -tag -width 4n
+.It Fl i
+Set the immediate bit in the CDB.  Note that CTL does not support the
+immediate bit, so this is primarily useful for making sure that CTL returns
+the proper error.
+.It Fl o
+Set the Copan proprietary on/offline bit in the CDB.  When this flag is
+used, the LUN will be spun down and taken offline ("Logical unit not ready,
+manual intervention required").  See the description of the 
+.Ic shutdown
+and
+.Ic startup
+options.
+.El
+.It Ic synccache
+Send the
+.Tn SCSI 
+SYNCHRONIZE CACHE command to the device.  By default, SYNCHRONIZE
+CACHE(10) is used.  If the specified starting LBA is greater than
+0xffffffff or the length is greater than 0xffff, though,
+SYNCHRONIZE CACHE(16) will be used.  The 16 byte command will also be used
+if the user specifies a 16 byte CDB with the
+.Fl c
+argument.
+.Bl -tag -width 14n
+.It Fl l Ar lba
+Specify the starting LBA of the cache region to synchronize.  This option is a
+no-op for CTL.  If you send a SYNCHRONIZE CACHE command, it will sync the
+cache for the entire LUN.
+.It Fl b Ar blockcount
+Specify the length of the cache region to synchronize.  This option is a
+no-op for CTL.  If you send a SYNCHRONIZE CACHE command, it will sync the
+cache for the entire LUN.
+.It Fl r
+Specify relative addressing for the starting LBA.  CTL does not support
+relative addressing, since it only works for linked commands, and CTL
+doesn't support linked commands.
+.It Fl i
+Tell the target to return status immediately after issuing the SYHCHRONIZE CACHE
+command rather than waiting for the cache to finish syncing.  CTL does not
+support this bit.
+.It Fl c Ar cdbsize
+Specify the minimum CDB size.  Valid values are 10 and 16 bytes.
+.El
+.It Ic shutdown
+Issue a
+.Tn SCSI
+START STOP UNIT command with the start bit cleared and the on/offline bit
+set to all direct access LUNs.  This will spin down all direct access LUNs,
+and mark them offline ("Logical unit not ready, manual intervention
+required").  Once marked offline, the state can only be cleared by sending
+a START STOP UNIT command with the start bit set and the on/offline bit
+set.  The
+.Nm
+commands
+.Ic startup
+and
+.Ic start
+will accomplish this.  Note that the
+on/offline bit is a non-standard Copan extension to the
+.Tn SCSI
+START STOP UNIT command, so merely sending a normal start command from an
+initiator will not clear the condition.  (This is by design.)
+.It Ic startup
+Issue a
+.Tn SCSI
+START STOP UNIT command with the start bit set and the on/offline bit set
+to all direct access LUNs.  This will mark all direct access LUNs "online"
+again.  It will not cause any LUNs to start up.  A separate start command
+without the on/offline bit set is necessary for that.
+.It Ic hardstop
+Use the kernel facility for stopping all direct access LUNs and setting the
+offline bit.  Unlike the
+.Ic shutdown
+command above, this command allows shutting down LUNs with I/O active.  It
+will also issue a LUN reset to any reserved LUNs to break the reservation
+so that the LUN can be stopped.
+.Ic shutdown
+command instead.
+.It Ic hardstart
+This command is functionally identical to the
+.Ic startup
+command described above.  The primary difference is that the LUNs are
+enumerated and commands sent by the in-kernel Front End Target Driver
+instead of by
+.Nm .
+.It Ic lunlist
+List all LUNs registered with CTL.
+Because this command uses the ioctl port, it will only work when the FETDs
+(Front End Target Drivers) are enabled.
+This command is the equivalent of doing a REPORT LUNS on one LUN and then
+and then an INQUIRY on each LUN in the system.
+.It Ic delay
+Delay commands at the given location.  There are two places where commands
+may be delayed currently: before data is transferred
+.Pq Dq datamove
+and just prior to sending status to the host
+.Pq Dq done .
+One of the two must be supplied as an argument to the
+.Fl l 
+option.  The
+.Fl t
+option must also be specified.
+.Bl -tag -width 12n
+.It Fl l Ar delayloc
+Delay command(s) at the specified location.
+This can either be at the data movement stage (datamove) or prior to
+command completion (done).
+.It Fl t Ar delaytime
+Delay command(s) for the specified number of seconds.  This must be
+specified.  If set to 0, it will clear out any previously set delay for
+this particular location (datamove or done).
+.It Fl T Ar delaytype
+Specify the delay type.
+By default, the
+.Ic delay
+option will delay the next command sent to the given LUN.
+With the
+.Fl T Ar cont
+option, every command will be delayed by the specified period of time.
+With the
+.Fl T Ar oneshot
+the next command sent to the given LUN will be delayed and all subsequent
+commands will be completed normally.
+This is the default.
+.El
+.It Ic realsync
+Query and control CTL's SYNCHRONIZE CACHE behavior.  The
+.Sq query
+argument
+will show whether SYNCHRONIZE CACHE commands are being sent to the backend
+or not.
+The default is to send SYNCHRONIZE CACHE commands to the backend.
+The 
+.Sq on
+argument will cause all SYNCHRONIZE CACHE commands sent to all LUNs to be
+sent to the backend.
+The
+.Sq off
+argument will cause all SYNCHRONIZE CACHE commands sent to all LUNs to be
+immediately returned to the initiator with successful status.
+.It Ic setsync
+For a given lun, only actually service every Nth SYNCHRONIZE CACHE command
+that is sent.  This can be used for debugging the optimal time period for
+sending SYNCHRONIZE cache commands.  An interval of 0 means that the cache
+will be flushed for this LUN every time a SYNCHRONIZE CACHE command is
+received.
+.Pp
+You must specify the target and LUN you want to modify.
+.It Ic getsync
+Get the interval at which we actually service the SYNCHRONIZE CACHE
+command, as set by the
+.Ic setsync
+command above.
+The reported number means that we will actually flush the cache on every
+Nth SYNCHRONIZE CACHE command.  A value of 0 means that we will flush the
+cache every time.
+.Pp
+You must specify the target and LUN you want to query.
+.It Ic inject
+Inject the specified type of error for the LUN specified, when a command
+that matches the given pattern is seen.
+The sense data returned is in either fixed or descriptor format, depending
+upon the status of the D_SENSE bit in the control mode page (page 0xa) for
+the LUN.
+.Pp
+Errors are only injected for commands that have not already failed for
+other reasons.
+By default, only the first command matching the pattern specified is
+returned with the supplied error.
+.Pp
+If the
+.Fl c
+flag is specified, all commands matching the pattern will be returned with
+the specified error until the error injection command is deleted with
+.Fl d 
+flag.
+.Bl -tag -width 17n
+.It Fl i Ar action
+Specify the error to return:
+.Bl -tag -width 10n
+.It aborted
+Return the next matching command on the specified LUN with the sense key
+ABORTED COMMAND (0x0b), and the ASC/ASCQ 0x45,0x00 ("Select or reselect
+failure").
+.It mediumerr
+Return the next matching command on the specified LUN with the sense key
+MEDIUM ERROR (0x03) and the ASC/ASCQ 0x11,0x00 ("Unrecovered read error") for
+reads, or ASC/ASCQ 0x0c,0x02 ("Write error - auto reallocation failed")
+for write errors.
+.It ua
+Return the next matching command on the specified LUN with the sense key
+UNIT ATTENTION (0x06) and the ASC/ASCQ 0x29,0x00 ("POWER ON, RESET, OR BUS
+DEVICE RESET OCCURRED").
+.It custom
+Return the next matching command on the specified LUN with the supplied
+sense data.
+The
+.Fl s
+argument must be specified.
+.El
+.It Fl p Ar pattern
+Specify which commands should be returned with the given error.
+.Bl -tag -width 10n
+.It read
+The error should apply to READ(6), READ(10), READ(12), READ(16), etc.
+.It write
+The error should apply to WRITE(6), WRITE(10), WRITE(12), WRITE(16), WRITE
+AND VERIFY(10), etc.
+.It rw
+The error should apply to both read and write type commands.
+.It readcap
+The error should apply to READ CAPACITY(10) and READ CAPACITY(16) commands.
+.It tur
+The error should apply to TEST UNIT READY commands.
+.It any
+The error should apply to any command.
+.El
+.It Fl r Ar lba,len
+Specify the starting lba and length of the range of LBAs which should
+trigger an error.
+This option is only applies when read and/or write patterns are specified.
+If used with other command types, the error will never be triggered.
+.It Fl s Ar len fmt Op Ar args
+Specify the sense data that is to be returned for custom actions.
+If the format is
+.Sq - ,
+len bytes of sense data will be read from standard input and written to the
+sense buffer.
+If len is longer than 252 bytes (the maximum allowable
+.Tn SCSI
+sense data length), it will be truncated to that length.
+The sense data format is described in
+.Xr cam_cdparse 3 .
+.It Fl c
+The error injection should be persistent, instead of happening once.
+Persistent errors must be deleted with the
+.Fl d
+argument.
+.It Fl d Ar delete_id
+Delete the specified error injection serial number.
+The serial number is returned when the error is injected.
+.El
+.It Ic port
+Perform one of several CTL frontend port operations.
+Either get a list of frontend ports
+.Pq Fl l ,
+turn one or more frontends on
+or off
+.Pq Fl o Ar on|off ,
+or set the World Wide Node Name
+.Pq Fl w Ar wwnn
+or World Wide Port Name
+.Pq Fl W Ar wwpn
+for a given port.
+One of
+.Fl l ,
+.Fl o ,
+or 
+.Fl w
+or
+.Fl W
+must be specified.
+The WWNN and WWPN may both be specified at the same time, but cannot be
+combined with enabling/disabling or listing ports.
+.Bl -tag -width 12n
+.It Fl l
+List all CTL frontend ports or a specific port type or number.
+.It Fl o Ar on|off
+Turn the specified CTL frontend ports off or on.
+If no port number or port type is specified, all ports are turned on or
+off.
+.It Fl p Ar targ_port 
+Specify the frontend port number.
+The port numbers can be found in the frontend port list.
+.It Fl q
+Omit the header in the port list output.
+.It Fl t Ar fe_type
+Specify the frontend type.
+Currently defined port types are
+.Dq fc
+(Fibre Channel),
+.Dq scsi
+(Parallel SCSI),
+.Dq ioctl
+(CTL ioctl interface),
+and
+.Dq internal
+(CTL CAM SIM).
+.It Fl w Ar wwnn
+Set the World Wide Node Name for the given port.
+The
+.Fl n 
+argument must be specified, since this is only possible to implement on a
+single port.
+As a general rule, the WWNN should be the same across all ports on the
+system.
+.It Fl W Ar wwpn
+Set the World Wide Node Name for the given port.
+The
+.Fl n
+argument must be specified, since this is only possible to implement on a
+single port.
+As a general rule, the WWPN must be different for every port in the system.
+.It Fl x
+Output the port list in XML format.
+.El
+.It Ic dumpooa
+Dump the OOA (Order Of Arrival) queue for each LUN registered with CTL.
+.It Ic dumpstructs
+Dump the CTL structures to the console.
+.It Ic create
+Create a new LUN.
+The backend must be specified, and depending upon the backend requested,
+some of the other options may be required.
+If the LUN is created successfully, the LUN configuration will be
+displayed.
+If LUN creation fails, a message will be displayed describing the failure.
+.Bl -tag -width 14n
+.It Fl b Ar backend
+The
+.Fl b
+flag is required.
+This specifies the name backend to use when creating the LUN.
+Examples are
+.Dq ramdisk
+and
+.Dq block .
+.It Fl B Ar blocksize
+Specify the blocksize of the backend in bytes.
+.It Fl d Ar device_id
+Specify the LUN-associated string to use in the
+.Tn SCSI
+INQUIRY VPD page 0x83 data.
+.It Fl l Ar lun_id
+Request that a particular LUN number be assigned.
+If the requested LUN number is not available, the request will fail.
+.It Fl o Ar name=value
+Specify a backend-specific name/value pair.
+Multiple
+.Fl o
+arguments may be specified.
+Refer to the backend documentation for arguments that may be used.
+.It Fl s Ar size_bytes
+Specify the size of the LUN in bytes.
+Some backends may allow setting the size (e.g. the ramdisk backend) and for
+others the size may be implicit (e.g. the block backend).
+.It Fl S Ar serial_num
+Specify the serial number to be used in the
+.Tn SCSI
+INQUIRY VPD page 0x80 data.
+.It Fl t Ar device_type
+Specify the numeric SCSI device type to use when creating the LUN.
+For example, the Direct Access type is 0.
+If this flag is not used, the type of LUN created is backend-specific.
+Not all LUN types are supported.
+Currently CTL only supports Direct Access (type 0) and Processor (type 3)
+LUNs.
+The backend requested may or may not support all of the LUN types that CTL
+supports.
+.El
+.It Ic remove
+Remove a LUN.
+The backend must be specified, and the LUN number must also be specified.
+Backend-specific options may also be specified with the
+.Fl o
+flag.
+.Bl -tag -width 14n
+.It Fl b Ar backend
+Specify the backend that owns the LUN to be removed.
+Examples are
+.Dq ramdisk
+and
+.Dq block .
+.It Fl l Ar lun_id
+Specify the LUN number to remove.
+.It Fl o Ar name=value
+Specify a backend-specific name/value pair.
+Multiple
+.Fl o
+arguments may be specified.
+Refer to the backend documentation for arguments that may be used.
+.El
+.It Ic devlist
+Get a list of all configured LUNs.
+This also includes the LUN size and blocksize, serial number and device ID.
+.Bl -tag -width 11n
+.It Fl b Ar backend
+Specify the backend.
+This restricts the LUN list to the named backend.
+Examples are
+.Dq ramdisk
+and
+.Dq block .
+.It Fl v
+Be verbose.
+This will also display any backend-specific LUN attributes in addition to
+the standard per-LUN information.
+.It Fl x
+Dump the raw XML.
+The LUN list information from the kernel comes in XML format, and this
+option allows the display of the raw XML data.
+This option and the
+.Fl v
+and
+.Fl b
+options are mutually exclusive.
+If you specify
+.Fl x ,
+the entire LUN database is displayed in XML format.
+.El
+.It Ic help
+Display
+.Nm
+usage information.
+.El
+.Sh EXAMPLES
+.Dl ctladm tur 0:1
+.Pp
+Send a
+.Tn SCSI
+TEST UNIT READY command to LUN 1.
+.Pp
+.Dl ctladm modesense 0:1 -l
+.Pp
+Display the list of mode pages supported by LUN 1.
+.Pp
+.Dl ctladm modesense 0:0 -m 10 -P 3 -d -c 10
+.Pp
+Display the saved version of the Control mode page (page 10) on LUN 0.
+Disable fetching block descriptors, and use a 10 byte MODE SENSE command
+instead of the default 6 byte command.
+.Pp
+.Bd -literal
+ctladm read 0:2 -l 0 -d 1 -b 512 -f - > foo
+.Ed
+.Pp
+Read the first 512 byte block from LUN 2 and dump it to the file
+.Pa foo .
+.Bd -literal
+ctladm write 0:3 -l 0xff432140 -d 20 -b 512 -f /tmp/bar
+.Ed
+.Pp
+Read 10240 bytes from the file
+.Pa /tmp/bar
+and write it to target 0, LUN 3.
+starting at LBA 0xff432140.
+.Pp
+.Dl ctladm create -b ramdisk -s 10485760000000000
+.Pp
+Create a LUN with the
+.Dq fake
+ramdisk as a backing store.
+The LUN will claim to have a size of approximately 10 terabytes.
+.Pp
+.Dl ctladm create -b block -o file=src/usr.sbin/ctladm/ctladm.8
+.Pp
+Create a LUN using the block backend, and specify the file
+.Pa src/usr.sbin/ctladm/ctladm.8
+as the backing store.
+The size of the LUN will be derived from the size of the file.
+.Pp
+.Dl ctladm create -b block -o file=src/usr.sbin/ctladm/ctladm.8 -S MYSERIAL321 -d MYDEVID123
+.Pp
+Create a LUN using the block backend, specify the file
+.Pa src/usr.sbin/ctladm/ctladm.8
+as the backing store, and specify the
+.Tn SCSI
+VPD page 0x80 and 0x83 serial number (
+.Fl S)
+and device ID (
+.Fl d).
+.Pp
+.Dl ctladm remove -b block -l 12
+.Pp
+Remove LUN 12, which is handled by the block backend, from the system.
+.Pp
+.Dl ctladm devlist
+.Pp
+List configured LUNs in the system, along with their backend and serial
+number.
+This works when the Front End Target Drivers are enabled or disabled.
+.Pp
+.Dl ctladm lunlist
+.Pp
+List all LUNs in the system, along with their inquiry data and device type.
+This only works when the FETDs are enabled, since the commands go through the 
+ioctl port.
+.Pp
+.Dl ctladm inject 0:6 -i mediumerr -p read -r 0,512 -c
+.Pp
+Inject a medium error on LUN 6 for every read that covers the first 512
+blocks of the LUN.
+.Pp
+.Bd -literal -offset indent
+ctladm inject 0:6 -i custom -p tur -s 18 "f0 0 02 s12 04 02"
+.Ed
+.Pp
+Inject a custom error on LUN 6 for the next TEST UNIT READY command only.
+This will result in a sense key of NOT READY (0x02), and an ASC/ASCQ of
+0x04,0x02 ("Logical unit not ready, initializing command required").
+.Sh SEE ALSO
+.Xr cam 3 ,
+.Xr cam_cdbparse 3 ,
+.Xr cam 4 ,
+.Xr xpt 4 ,
+.Xr camcontrol 8
+.Sh HISTORY
+The
+.Nm
+utility was originally written during the Winter/Spring of 2003 as an
+interface to CTL.
+.Sh AUTHORS
+.An Ken Merry Aq ken@FreeBSD.org
diff --git a/usr.sbin/ctladm/ctladm.c b/usr.sbin/ctladm/ctladm.c
new file mode 100644
index 0000000..77e7b02
--- /dev/null
+++ b/usr.sbin/ctladm/ctladm.c
@@ -0,0 +1,4005 @@
+/*-
+ * Copyright (c) 2003, 2004 Silicon Graphics International Corp.
+ * Copyright (c) 1997-2007 Kenneth D. Merry
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions, and the following disclaimer,
+ *    without modification.
+ * 2. Redistributions in binary form must reproduce at minimum a disclaimer
+ *    substantially similar to the "NO WARRANTY" disclaimer below
+ *    ("Disclaimer") and any redistribution must be conditioned upon
+ *    including a substantially similar Disclaimer requirement for further
+ *    binary redistribution.
+ *
+ * NO WARRANTY
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTIBILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * HOLDERS OR CONTRIBUTORS BE LIABLE FOR SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
+ * IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGES.
+ *
+ * $Id: //depot/users/kenm/FreeBSD-test2/usr.sbin/ctladm/ctladm.c#4 $
+ */
+/*
+ * CAM Target Layer exercise program.
+ *
+ * Author: Ken Merry <ken@FreeBSD.org>
+ */
+
+#include <sys/cdefs.h>
+__FBSDID("$FreeBSD$");
+
+#include <sys/ioctl.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <sys/param.h>
+#include <sys/queue.h>
+#include <sys/callout.h>
+#include <sys/sbuf.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <fcntl.h>
+#include <getopt.h>
+#include <string.h>
+#include <errno.h>
+#include <err.h>
+#include <ctype.h>
+#include <bsdxml.h>
+#include <cam/scsi/scsi_all.h>
+#include <cam/scsi/scsi_message.h>
+#include <cam/ctl/ctl.h>
+#include <cam/ctl/ctl_io.h>
+#include <cam/ctl/ctl_frontend_internal.h>
+#include <cam/ctl/ctl_backend.h>
+#include <cam/ctl/ctl_ioctl.h>
+#include <cam/ctl/ctl_backend_block.h>
+#include <cam/ctl/ctl_util.h>
+#include <cam/ctl/ctl_scsi_all.h>
+#include <camlib.h>
+#include "ctladm.h"
+
+#ifdef min
+#undef min
+#endif
+#define min(x,y) (x < y) ? x : y
+
+typedef enum {
+	CTLADM_CMD_TUR,
+	CTLADM_CMD_INQUIRY,
+	CTLADM_CMD_REQ_SENSE,
+	CTLADM_CMD_ARRAYLIST,
+	CTLADM_CMD_REPORT_LUNS,
+	CTLADM_CMD_HELP,
+	CTLADM_CMD_DEVLIST,
+	CTLADM_CMD_ADDDEV,
+	CTLADM_CMD_RM,
+	CTLADM_CMD_CREATE,
+	CTLADM_CMD_READ,
+	CTLADM_CMD_WRITE,
+	CTLADM_CMD_PORT,
+	CTLADM_CMD_READCAPACITY,
+	CTLADM_CMD_MODESENSE,
+	CTLADM_CMD_DUMPOOA,
+	CTLADM_CMD_DUMPSTRUCTS,
+	CTLADM_CMD_START,
+	CTLADM_CMD_STOP,
+	CTLADM_CMD_SYNC_CACHE,
+	CTLADM_CMD_SHUTDOWN,
+	CTLADM_CMD_STARTUP,
+	CTLADM_CMD_LUNLIST,
+	CTLADM_CMD_HARDSTOP,
+	CTLADM_CMD_HARDSTART,
+	CTLADM_CMD_DELAY,
+	CTLADM_CMD_REALSYNC,
+	CTLADM_CMD_SETSYNC,
+	CTLADM_CMD_GETSYNC,
+	CTLADM_CMD_ERR_INJECT,
+	CTLADM_CMD_BBRREAD,
+	CTLADM_CMD_PRES_IN,
+	CTLADM_CMD_PRES_OUT,
+	CTLADM_CMD_INQ_VPD_DEVID,
+	CTLADM_CMD_RTPG
+} ctladm_cmdfunction;
+
+typedef enum {
+	CTLADM_ARG_NONE		= 0x0000000,
+	CTLADM_ARG_AUTOSENSE	= 0x0000001,
+	CTLADM_ARG_DEVICE	= 0x0000002,
+	CTLADM_ARG_ARRAYSIZE	= 0x0000004,
+	CTLADM_ARG_BACKEND	= 0x0000008,
+	CTLADM_ARG_CDBSIZE	= 0x0000010,
+	CTLADM_ARG_DATALEN	= 0x0000020,
+	CTLADM_ARG_FILENAME	= 0x0000040,
+	CTLADM_ARG_LBA		= 0x0000080,
+	CTLADM_ARG_PC		= 0x0000100,
+	CTLADM_ARG_PAGE_CODE	= 0x0000200,
+	CTLADM_ARG_PAGE_LIST	= 0x0000400,
+	CTLADM_ARG_SUBPAGE	= 0x0000800,
+	CTLADM_ARG_PAGELIST	= 0x0001000,
+	CTLADM_ARG_DBD		= 0x0002000,
+	CTLADM_ARG_TARG_LUN	= 0x0004000,
+	CTLADM_ARG_BLOCKSIZE	= 0x0008000,
+	CTLADM_ARG_IMMED	= 0x0010000,
+	CTLADM_ARG_RELADR	= 0x0020000,
+	CTLADM_ARG_RETRIES	= 0x0040000,
+	CTLADM_ARG_ONOFFLINE	= 0x0080000,
+	CTLADM_ARG_ONESHOT	= 0x0100000,
+	CTLADM_ARG_TIMEOUT	= 0x0200000,
+	CTLADM_ARG_INITIATOR 	= 0x0400000,
+	CTLADM_ARG_NOCOPY	= 0x0800000,
+	CTLADM_ARG_NEED_TL	= 0x1000000
+} ctladm_cmdargs;
+
+struct ctladm_opts {
+	const char	*optname;
+	uint32_t	cmdnum;
+	ctladm_cmdargs	argnum;
+	const char	*subopt;
+};
+
+typedef enum {
+	CC_OR_NOT_FOUND,
+	CC_OR_AMBIGUOUS,
+	CC_OR_FOUND
+} ctladm_optret;
+
+static const char rw_opts[] = "Nb:c:d:f:l:";
+static const char startstop_opts[] = "io";
+
+struct ctladm_opts option_table[] = {
+	{"adddev", CTLADM_CMD_ADDDEV, CTLADM_ARG_NONE, NULL},
+	{"bbrread", CTLADM_CMD_BBRREAD, CTLADM_ARG_NEED_TL, "d:l:"},
+	{"create", CTLADM_CMD_CREATE, CTLADM_ARG_NONE, "b:B:d:l:o:s:S:t:"},
+	{"delay", CTLADM_CMD_DELAY, CTLADM_ARG_NEED_TL, "T:l:t:"},
+	{"devid", CTLADM_CMD_INQ_VPD_DEVID, CTLADM_ARG_NEED_TL, NULL},
+	{"devlist", CTLADM_CMD_DEVLIST, CTLADM_ARG_NONE, "b:vx"},
+	{"dumpooa", CTLADM_CMD_DUMPOOA, CTLADM_ARG_NONE, NULL},
+	{"dumpstructs", CTLADM_CMD_DUMPSTRUCTS, CTLADM_ARG_NONE, NULL},
+	{"getsync", CTLADM_CMD_GETSYNC, CTLADM_ARG_NEED_TL, NULL},
+	{"hardstart", CTLADM_CMD_HARDSTART, CTLADM_ARG_NONE, NULL},
+	{"hardstop", CTLADM_CMD_HARDSTOP, CTLADM_ARG_NONE, NULL},
+	{"help", CTLADM_CMD_HELP, CTLADM_ARG_NONE, NULL},
+	{"inject", CTLADM_CMD_ERR_INJECT, CTLADM_ARG_NEED_TL, "cd:i:p:r:s:"},
+	{"inquiry", CTLADM_CMD_INQUIRY, CTLADM_ARG_NEED_TL, NULL},
+	{"lunlist", CTLADM_CMD_LUNLIST, CTLADM_ARG_NONE, NULL},
+	{"modesense", CTLADM_CMD_MODESENSE, CTLADM_ARG_NEED_TL, "P:S:dlm:c:"},
+	{"port", CTLADM_CMD_PORT, CTLADM_ARG_NONE, "lo:p:qt:w:W:x"},
+	{"prin", CTLADM_CMD_PRES_IN, CTLADM_ARG_NEED_TL, "a:"},
+	{"prout", CTLADM_CMD_PRES_OUT, CTLADM_ARG_NEED_TL, "a:k:r:s:"},
+	{"read", CTLADM_CMD_READ, CTLADM_ARG_NEED_TL, rw_opts},
+	{"readcapacity", CTLADM_CMD_READCAPACITY, CTLADM_ARG_NEED_TL, "c:"},
+	{"realsync", CTLADM_CMD_REALSYNC, CTLADM_ARG_NONE, NULL},
+	{"remove", CTLADM_CMD_RM, CTLADM_ARG_NONE, "b:l:o:"},
+	{"reportluns", CTLADM_CMD_REPORT_LUNS, CTLADM_ARG_NEED_TL, NULL},
+	{"reqsense", CTLADM_CMD_REQ_SENSE, CTLADM_ARG_NEED_TL, NULL},
+	{"rtpg", CTLADM_CMD_RTPG, CTLADM_ARG_NEED_TL, NULL},
+	{"setsync", CTLADM_CMD_SETSYNC, CTLADM_ARG_NEED_TL, "i:"},
+	{"shutdown", CTLADM_CMD_SHUTDOWN, CTLADM_ARG_NONE, NULL},
+	{"start", CTLADM_CMD_START, CTLADM_ARG_NEED_TL, startstop_opts},
+	{"startup", CTLADM_CMD_STARTUP, CTLADM_ARG_NONE, NULL},
+	{"stop", CTLADM_CMD_STOP, CTLADM_ARG_NEED_TL, startstop_opts},
+	{"synccache", CTLADM_CMD_SYNC_CACHE, CTLADM_ARG_NEED_TL, "b:c:il:r"},
+	{"tur", CTLADM_CMD_TUR, CTLADM_ARG_NEED_TL, NULL},
+	{"write", CTLADM_CMD_WRITE, CTLADM_ARG_NEED_TL, rw_opts},
+	{"-?", CTLADM_CMD_HELP, CTLADM_ARG_NONE, NULL},
+	{"-h", CTLADM_CMD_HELP, CTLADM_ARG_NONE, NULL},
+	{NULL, 0, 0, NULL}
+};
+
+
+ctladm_optret getoption(struct ctladm_opts *table, char *arg, uint32_t *cmdnum,
+			ctladm_cmdargs *argnum, const char **subopt);
+static int cctl_parse_tl(char *str, int *target, int *lun);
+static int cctl_dump_ooa(int fd, int argc, char **argv);
+static int cctl_port_dump(int fd, int quiet, int xml, int32_t fe_num,
+			  ctl_port_type port_type);
+static int cctl_port(int fd, int argc, char **argv, char *combinedopt);
+static int cctl_do_io(int fd, int retries, union ctl_io *io, const char *func);
+static int cctl_delay(int fd, int target, int lun, int argc, char **argv,
+		      char *combinedopt);
+static int cctl_lunlist(int fd);
+static void cctl_cfi_mt_statusstr(cfi_mt_status status, char *str, int str_len);
+static void cctl_cfi_bbr_statusstr(cfi_bbrread_status, char *str, int str_len);
+static int cctl_hardstopstart(int fd, ctladm_cmdfunction command);
+static int cctl_bbrread(int fd, int target, int lun, int iid, int argc,
+			char **argv, char *combinedopt);
+static int cctl_startup_shutdown(int fd, int target, int lun, int iid,
+				 ctladm_cmdfunction command);
+static int cctl_sync_cache(int fd, int target, int lun, int iid, int retries,
+			   int argc, char **argv, char *combinedopt);
+static int cctl_start_stop(int fd, int target, int lun, int iid, int retries,
+			   int start, int argc, char **argv, char *combinedopt);
+static int cctl_mode_sense(int fd, int target, int lun, int iid, int retries, 
+			   int argc, char **argv, char *combinedopt);
+static int cctl_read_capacity(int fd, int target, int lun, int iid,
+			      int retries, int argc, char **argv,
+			      char *combinedopt);
+static int cctl_read_write(int fd, int target, int lun, int iid, int retries,
+			   int argc, char **argv, char *combinedopt,
+			   ctladm_cmdfunction command);
+static int cctl_get_luns(int fd, int target, int lun, int iid, int retries,
+			 struct scsi_report_luns_data **lun_data,
+			 uint32_t *num_luns);
+static int cctl_report_luns(int fd, int target, int lun, int iid, int retries);
+static int cctl_tur(int fd, int target, int lun, int iid, int retries);
+static int cctl_get_inquiry(int fd, int target, int lun, int iid, int retries,
+			    char *path_str, int path_len,
+			    struct scsi_inquiry_data *inq_data);
+static int cctl_inquiry(int fd, int target, int lun, int iid, int retries);
+static int cctl_req_sense(int fd, int target, int lun, int iid, int retries);
+static int cctl_persistent_reserve_in(int fd, int target, int lun,
+				      int initiator, int argc, char **argv,
+				      char *combinedopt, int retry_count);
+static int cctl_persistent_reserve_out(int fd, int target, int lun, 
+				       int initiator, int argc, char **argv,
+				       char *combinedopt, int retry_count);
+static int cctl_create_lun(int fd, int argc, char **argv, char *combinedopt);
+static int cctl_inquiry_vpd_devid(int fd, int target, int lun, int initiator);
+static int cctl_report_target_port_group(int fd, int target, int lun,
+					 int initiator);
+
+ctladm_optret
+getoption(struct ctladm_opts *table, char *arg, uint32_t *cmdnum,
+	  ctladm_cmdargs *argnum, const char **subopt)
+{
+	struct ctladm_opts *opts;
+	int num_matches = 0;
+
+	for (opts = table; (opts != NULL) && (opts->optname != NULL);
+	     opts++) {
+		if (strncmp(opts->optname, arg, strlen(arg)) == 0) {
+			*cmdnum = opts->cmdnum;
+			*argnum = opts->argnum;
+			*subopt = opts->subopt;
+
+			if (strcmp(opts->optname, arg) == 0)
+				return (CC_OR_FOUND);
+
+			if (++num_matches > 1)
+				return(CC_OR_AMBIGUOUS);
+		}
+	}
+
+	if (num_matches > 0)
+		return(CC_OR_FOUND);
+	else
+		return(CC_OR_NOT_FOUND);
+}
+
+
+static int
+cctl_parse_tl(char *str, int *target, int *lun)
+{
+	char *tmpstr;
+	int retval;
+
+	retval = 0;
+
+	while (isspace(*str) && (*str != '\0'))
+		str++;
+
+	tmpstr = (char *)strtok(str, ":");
+	if ((tmpstr != NULL) && (*tmpstr != '\0')) {
+		*target = strtol(tmpstr, NULL, 0);
+		tmpstr = (char *)strtok(NULL, ":");
+		if ((tmpstr != NULL) && (*tmpstr != '\0')) {
+			*lun = strtol(tmpstr, NULL, 0);
+		} else
+			retval = -1;
+	} else
+		retval = -1;
+
+	return (retval);
+}
+
+static int
+cctl_dump_ooa(int fd, int argc, char **argv)
+{
+	struct ctl_ooa ooa;
+	long double cmd_latency;
+	int num_entries, len;
+	int target = -1, lun = -1;
+	int retval;
+	unsigned int i;
+
+	num_entries = 104;
+
+	if ((argc > 2)
+	 && (isdigit(argv[2][0]))) {
+		retval = cctl_parse_tl(argv[2], &target, &lun);
+		if (retval != 0)
+			warnx("invalid target:lun argument %s", argv[2]);
+	}
+retry:
+
+	len = num_entries * sizeof(struct ctl_ooa_entry);
+
+	bzero(&ooa, sizeof(ooa));
+
+	ooa.entries = malloc(len);
+
+	if (ooa.entries == NULL) {
+		warn("%s: error mallocing %d bytes", __func__, len);
+		return (1);
+	}
+
+	if (argc > 2) {
+		ooa.lun_num = lun;
+	} else
+		ooa.flags |= CTL_OOA_FLAG_ALL_LUNS;
+
+	ooa.alloc_len = len;
+	ooa.alloc_num = num_entries;
+	if (ioctl(fd, CTL_GET_OOA, &ooa) == -1) {
+		warn("%s: CTL_GET_OOA ioctl failed", __func__);
+		retval = 1;
+		goto bailout;
+	}
+
+	if (ooa.status == CTL_OOA_NEED_MORE_SPACE) {
+		num_entries = num_entries * 2;
+		free(ooa.entries);
+		ooa.entries = NULL;
+		goto retry;
+	}
+
+	if (ooa.status != CTL_OOA_OK) {
+		warnx("%s: CTL_GET_OOA ioctl returned error %d", __func__,
+		      ooa.status);
+		retval = 1;
+		goto bailout;
+	}
+
+	fprintf(stdout, "Dumping OOA queues\n");
+	for (i = 0; i < ooa.fill_num; i++) {
+		struct ctl_ooa_entry *entry;
+		char cdb_str[(SCSI_MAX_CDBLEN * 3) +1];
+		struct bintime delta_bt;
+		struct timespec ts;
+
+		entry = &ooa.entries[i];
+
+		delta_bt = ooa.cur_bt;
+		bintime_sub(&delta_bt, &entry->start_bt);
+		bintime2timespec(&delta_bt, &ts);
+		cmd_latency = ts.tv_sec * 1000;
+		if (ts.tv_nsec > 0)
+			cmd_latency += ts.tv_nsec / 1000000;
+		
+		fprintf(stdout, "LUN %jd tag 0x%04x%s%s%s%s%s: %s. CDB: %s "
+			"(%0.0Lf ms)\n",
+			(intmax_t)entry->lun_num, entry->tag_num,
+			(entry->cmd_flags & CTL_OOACMD_FLAG_BLOCKED) ?
+			 " BLOCKED" : "",
+			(entry->cmd_flags & CTL_OOACMD_FLAG_DMA) ? " DMA" : "",
+			(entry->cmd_flags & CTL_OOACMD_FLAG_DMA_QUEUED) ?
+			 " DMAQUEUED" : "",
+			(entry->cmd_flags & CTL_OOACMD_FLAG_ABORT) ?
+			 " ABORT" : "",
+			(entry->cmd_flags & CTL_OOACMD_FLAG_RTR) ? " RTR" :"",
+			scsi_op_desc(entry->cdb[0], NULL),
+			scsi_cdb_string(entry->cdb, cdb_str, sizeof(cdb_str)),
+			cmd_latency);
+	}
+	fprintf(stdout, "OOA queues dump done\n");
+#if 0
+	if (ioctl(fd, CTL_DUMP_OOA) == -1) {
+		warn("%s: CTL_DUMP_OOA ioctl failed", __func__);
+		return (1);
+	}
+#endif
+
+bailout:
+	free(ooa.entries);
+
+	return (0);
+}
+
+static int
+cctl_dump_structs(int fd, ctladm_cmdargs cmdargs __unused)
+{
+	if (ioctl(fd, CTL_DUMP_STRUCTS) == -1) {
+		warn(__func__);
+		return (1);
+	}
+	return (0);
+}
+
+static int
+cctl_port_dump(int fd, int quiet, int xml, int32_t targ_port,
+	       ctl_port_type port_type)
+{
+	struct ctl_port_list port_list;
+	struct ctl_port_entry *entries;
+	struct sbuf *sb = NULL;
+	int num_entries;
+	int did_print = 0;
+	unsigned int i;
+
+	num_entries = 16;
+
+retry:
+
+	entries = malloc(sizeof(*entries) * num_entries);
+	bzero(&port_list, sizeof(port_list));
+	port_list.entries = entries;
+	port_list.alloc_num = num_entries;
+	port_list.alloc_len = num_entries * sizeof(*entries);
+	if (ioctl(fd, CTL_GET_PORT_LIST, &port_list) != 0) {
+		warn("%s: CTL_GET_PORT_LIST ioctl failed", __func__);
+		return (1);
+	}
+	if (port_list.status == CTL_PORT_LIST_NEED_MORE_SPACE) {
+		printf("%s: allocated %d, need %d, retrying\n", __func__,
+		       num_entries, port_list.fill_num + port_list.dropped_num);
+		free(entries);
+		num_entries = port_list.fill_num + port_list.dropped_num;
+		goto retry;
+	}
+
+	if ((quiet == 0)
+	 && (xml == 0))
+		printf("Port Online Type     Name         pp vp %-18s %-18s\n",
+		       "WWNN", "WWPN");
+
+	if (xml != 0) {
+		sb = sbuf_new_auto();
+		sbuf_printf(sb, "<ctlfelist>\n");
+	}
+	for (i = 0; i < port_list.fill_num; i++) {
+		struct ctl_port_entry *entry;
+		const char *type;
+
+		entry = &entries[i];
+
+		switch (entry->port_type) {
+		case CTL_PORT_FC:
+			type = "FC";
+			break;
+		case CTL_PORT_SCSI:
+			type = "SCSI";
+			break;
+		case CTL_PORT_IOCTL:
+			type = "IOCTL";
+			break;
+		case CTL_PORT_INTERNAL:
+			type = "INTERNAL";
+			break;
+		case CTL_PORT_ISC:
+			type = "ISC";
+			break;
+		default:
+			type = "UNKNOWN";
+			break;
+		}
+
+		/*
+		 * If the user specified a frontend number or a particular
+		 * frontend type, only print out that particular frontend
+		 * or frontend type.
+		 */
+		if ((targ_port != -1)
+		 && (targ_port != entry->targ_port))
+			continue;
+		else if ((port_type != CTL_PORT_NONE)
+		      && ((port_type & entry->port_type) == 0))
+			continue;
+
+		did_print = 1;
+
+#if 0
+		printf("Num: %ju Type: %s (%#x) Name: %s Physical Port: %d "
+		       "Virtual Port: %d\n", (uintmax_t)entry->fe_num, type,
+		       entry->port_type, entry->fe_name, entry->physical_port,
+		       entry->virtual_port);
+		printf("WWNN %#jx WWPN %#jx Online: %s\n",
+		       (uintmax_t)entry->wwnn, (uintmax_t)entry->wwpn,
+		       (entry->online) ? "YES" : "NO" );
+#endif
+		if (xml == 0) {
+			printf("%-4d %-6s %-8s %-12s %-2d %-2d %#-18jx "
+			       "%#-18jx\n",
+			       entry->targ_port, (entry->online) ? "YES" : "NO",
+			       type, entry->port_name, entry->physical_port,
+			       entry->virtual_port, (uintmax_t)entry->wwnn,
+			       (uintmax_t)entry->wwpn);
+		} else {
+			sbuf_printf(sb, "<targ_port id=\"%d\">\n",
+				    entry->targ_port);
+			sbuf_printf(sb, "<online>%s</online>\n",
+				    (entry->online) ? "YES" : "NO");
+			sbuf_printf(sb, "<port_type>%s</port_type>\n", type);
+			sbuf_printf(sb, "<port_name>%s</port_name>\n",
+				    entry->port_name);
+			sbuf_printf(sb, "<physical_port>%d</physical_port>\n",
+				    entry->physical_port);
+			sbuf_printf(sb, "<virtual_port>%d</virtual_port>\n",
+				    entry->virtual_port);
+			sbuf_printf(sb, "<wwnn>%#jx</wwnn>\n",
+				    (uintmax_t)entry->wwnn);
+			sbuf_printf(sb, "<wwpn>%#jx</wwpn>\n",
+				    (uintmax_t)entry->wwpn);
+			sbuf_printf(sb, "</targ_port>\n");
+		}
+
+	}
+	if (xml != 0) {
+		sbuf_printf(sb, "</ctlfelist>\n");
+		sbuf_finish(sb);
+		printf("%s", sbuf_data(sb));
+		sbuf_delete(sb);
+	}
+
+	/*
+	 * Give some indication that we didn't find the frontend or
+	 * frontend type requested by the user.  We could print something
+	 * out, but it would probably be better to hide that behind a
+	 * verbose flag.
+	 */
+	if ((did_print == 0)
+	 && ((targ_port != -1)
+	  || (port_type != CTL_PORT_NONE)))
+		return (1);
+	else
+		return (0);
+}
+
+typedef enum {
+	CCTL_PORT_MODE_NONE,
+	CCTL_PORT_MODE_LIST,
+	CCTL_PORT_MODE_SET,
+	CCTL_PORT_MODE_ON,
+	CCTL_PORT_MODE_OFF
+} cctl_port_mode;
+
+struct ctladm_opts cctl_fe_table[] = {
+	{"fc", CTL_PORT_FC, CTLADM_ARG_NONE, NULL},
+	{"scsi", CTL_PORT_SCSI, CTLADM_ARG_NONE, NULL},
+	{"internal", CTL_PORT_INTERNAL, CTLADM_ARG_NONE, NULL},
+	{"all", CTL_PORT_ALL, CTLADM_ARG_NONE, NULL},
+	{NULL, 0, 0, NULL}
+};
+
+static int
+cctl_port(int fd, int argc, char **argv, char *combinedopt)
+{
+	int c;
+	int32_t targ_port = -1;
+	int retval = 0;
+	int wwnn_set = 0, wwpn_set = 0;
+	uint64_t wwnn = 0, wwpn = 0;
+	cctl_port_mode port_mode = CCTL_PORT_MODE_NONE;
+	struct ctl_port_entry entry;
+	ctl_port_type port_type = CTL_PORT_NONE;
+	int quiet = 0, xml = 0;
+
+	while ((c = getopt(argc, argv, combinedopt)) != -1) {
+		switch (c) {
+		case 'l':
+			if (port_mode != CCTL_PORT_MODE_NONE)
+				goto bailout_badarg;
+
+			port_mode = CCTL_PORT_MODE_LIST;
+			break;
+		case 'o':
+			if (port_mode != CCTL_PORT_MODE_NONE)
+				goto bailout_badarg;
+			
+			if (strcasecmp(optarg, "on") == 0)
+				port_mode = CCTL_PORT_MODE_ON;
+			else if (strcasecmp(optarg, "off") == 0)
+				port_mode = CCTL_PORT_MODE_OFF;
+			else {
+				warnx("Invalid -o argument %s, \"on\" or "
+				      "\"off\" are the only valid args",
+				      optarg);
+				retval = 1;
+				goto bailout;
+			}
+			break;
+		case 'p':
+			targ_port = strtol(optarg, NULL, 0);
+			break;
+		case 'q':
+			quiet = 1;
+			break;
+		case 't': {
+			ctladm_optret optret;
+			ctladm_cmdargs argnum;
+			const char *subopt;
+			ctl_port_type tmp_port_type;
+
+			optret = getoption(cctl_fe_table, optarg, &tmp_port_type,
+					   &argnum, &subopt);
+			if (optret == CC_OR_AMBIGUOUS) {
+				warnx("%s: ambiguous frontend type %s",
+				      __func__, optarg);
+				retval = 1;
+				goto bailout;
+			} else if (optret == CC_OR_NOT_FOUND) {
+				warnx("%s: invalid frontend type %s",
+				      __func__, optarg);
+				retval = 1;
+				goto bailout;
+			}
+
+			port_type |= tmp_port_type;
+			break;
+		}
+		case 'w':
+			if ((port_mode != CCTL_PORT_MODE_NONE)
+			 && (port_mode != CCTL_PORT_MODE_SET))
+				goto bailout_badarg;
+
+			port_mode = CCTL_PORT_MODE_SET;
+
+			wwnn = strtoull(optarg, NULL, 0);
+			wwnn_set = 1;
+			break;
+		case 'W':
+			if ((port_mode != CCTL_PORT_MODE_NONE)
+			 && (port_mode != CCTL_PORT_MODE_SET))
+				goto bailout_badarg;
+
+			port_mode = CCTL_PORT_MODE_SET;
+
+			wwpn = strtoull(optarg, NULL, 0);
+			wwpn_set = 1;
+			break;
+		case 'x':
+			xml = 1;
+			break;
+		}
+	}
+
+	/*
+	 * The user can specify either one or more frontend types (-t), or
+	 * a specific frontend, but not both.
+	 *
+	 * If the user didn't specify a frontend type or number, set it to
+	 * all.  This is primarily needed for the enable/disable ioctls.
+	 * This will be a no-op for the listing code.  For the set ioctl,
+	 * we'll throw an error, since that only works on one port at a time.
+	 */
+	if ((port_type != CTL_PORT_NONE) && (targ_port != -1)) {
+		warnx("%s: can only specify one of -t or -n", __func__);
+		retval = 1;
+		goto bailout;
+	} else if ((targ_port == -1) && (port_type == CTL_PORT_NONE))
+		port_type = CTL_PORT_ALL;
+
+	bzero(&entry, sizeof(&entry));
+
+	/*
+	 * These are needed for all but list/dump mode.
+	 */
+	entry.port_type = port_type;
+	entry.targ_port = targ_port;
+
+	switch (port_mode) {
+	case CCTL_PORT_MODE_LIST:
+		cctl_port_dump(fd, quiet, xml, targ_port, port_type);
+		break;
+	case CCTL_PORT_MODE_SET:
+		if (targ_port == -1) {
+			warnx("%s: -w and -W require -n", __func__);
+			retval = 1;
+			goto bailout;
+		}
+
+		if (wwnn_set) {
+			entry.flags |= CTL_PORT_WWNN_VALID;
+			entry.wwnn = wwnn;
+		}
+		if (wwpn_set) {
+			entry.flags |= CTL_PORT_WWPN_VALID;
+			entry.wwpn = wwpn;
+		}
+
+		if (ioctl(fd, CTL_SET_PORT_WWNS, &entry) == -1) {
+			warn("%s: CTL_SET_PORT_WWNS ioctl failed", __func__);
+			retval = 1;
+			goto bailout;
+		}
+		break;
+	case CCTL_PORT_MODE_ON:
+		if (ioctl(fd, CTL_ENABLE_PORT, &entry) == -1) {
+			warn("%s: CTL_ENABLE_PORT ioctl failed", __func__);
+			retval = 1;
+			goto bailout;
+		}
+		fprintf(stdout, "Front End Ports enabled\n");
+		break;
+	case CCTL_PORT_MODE_OFF:
+		if (ioctl(fd, CTL_DISABLE_PORT, &entry) == -1) {
+			warn("%s: CTL_DISABLE_PORT ioctl failed", __func__);
+			retval = 1;
+			goto bailout;
+		}
+		fprintf(stdout, "Front End Ports disabled\n");
+		break;
+	default:
+		warnx("%s: one of -l, -o or -w/-W must be specified", __func__);
+		retval = 1;
+		goto bailout;
+		break;
+	}
+
+bailout:
+
+	return (retval);
+
+bailout_badarg:
+	warnx("%s: only one of -l, -o or -w/-W may be specified", __func__);
+	return (1);
+}
+
+static int
+cctl_do_io(int fd, int retries, union ctl_io *io, const char *func)
+{
+	do {
+		if (ioctl(fd, CTL_IO, io) == -1) {
+			warn("%s: error sending CTL_IO ioctl", func);
+			return (-1);
+		}
+	} while (((io->io_hdr.status & CTL_STATUS_MASK) != CTL_SUCCESS)
+	      && (retries-- > 0));
+
+	return (0);
+}
+
+static int
+cctl_delay(int fd, int target, int lun, int argc, char **argv,
+	   char *combinedopt)
+{
+	int datamove_delay;
+	struct ctl_io_delay_info delay_info;
+	char *delayloc = NULL;
+	char *delaytype = NULL;
+	int delaytime = -1;
+	int retval;
+	int c;
+
+	retval = 0;
+	datamove_delay = 0;
+
+	memset(&delay_info, 0, sizeof(delay_info));
+
+	while ((c = getopt(argc, argv, combinedopt)) != -1) {
+		switch (c) {
+		case 'T':
+			delaytype = strdup(optarg);
+			break;
+		case 'l':
+			delayloc = strdup(optarg);
+			break;
+		case 't':
+			delaytime = strtoul(optarg, NULL, 0);
+			break;
+		}
+	}
+
+	if (delaytime == -1) {
+		warnx("%s: you must specify the delaytime with -t", __func__);
+		retval = 1;
+		goto bailout;
+	}
+
+	if (strcasecmp(delayloc, "datamove") == 0)
+		delay_info.delay_loc = CTL_DELAY_LOC_DATAMOVE;
+	else if (strcasecmp(delayloc, "done") == 0)
+		delay_info.delay_loc = CTL_DELAY_LOC_DONE;
+	else {
+		warnx("%s: invalid delay location %s", __func__, delayloc);
+		retval = 1;
+		goto bailout;
+	}
+
+	if ((delaytype == NULL)
+	 || (strcmp(delaytype, "oneshot") == 0))
+		delay_info.delay_type = CTL_DELAY_TYPE_ONESHOT;
+	else if (strcmp(delaytype, "cont") == 0)
+		delay_info.delay_type = CTL_DELAY_TYPE_CONT;
+	else {
+		warnx("%s: invalid delay type %s", __func__, delaytype);
+		retval = 1;
+		goto bailout;
+	}
+
+	delay_info.target_id = target;
+	delay_info.lun_id = lun;
+	delay_info.delay_secs = delaytime;
+
+	if (ioctl(fd, CTL_DELAY_IO, &delay_info) == -1) {
+		warn("%s: CTL_DELAY_IO ioctl failed", __func__);
+		retval = 1;
+		goto bailout;
+	}
+	switch (delay_info.status) {
+	case CTL_DELAY_STATUS_NONE:
+		warnx("%s: no delay status??", __func__);
+		retval = 1;
+		break;
+	case CTL_DELAY_STATUS_OK:
+		break;
+	case CTL_DELAY_STATUS_INVALID_LUN:
+		warnx("%s: invalid lun %d", __func__, lun);
+		retval = 1;
+		break;
+	case CTL_DELAY_STATUS_INVALID_TYPE:
+		warnx("%s: invalid delay type %d", __func__,
+		      delay_info.delay_type);
+		retval = 1;
+		break;
+	case CTL_DELAY_STATUS_INVALID_LOC:
+		warnx("%s: delay location %s not implemented?", __func__,
+		      delayloc);
+		retval = 1;
+		break;
+	case CTL_DELAY_STATUS_NOT_IMPLEMENTED:
+		warnx("%s: delay not implemented in the kernel", __func__);
+		warnx("%s: recompile with the CTL_IO_DELAY flag set", __func__);
+		retval = 1;
+		break;
+	default:
+		warnx("%s: unknown delay return status %d", __func__,
+		      delay_info.status);
+		retval = 1;
+		break;
+	}
+bailout:
+
+	/* delayloc should never be NULL, but just in case...*/
+	if (delayloc != NULL)
+		free(delayloc);
+
+	return (retval);
+}
+
+static int
+cctl_realsync(int fd, int argc, char **argv)
+{
+	int syncstate;
+	int retval;
+	char *syncarg;
+
+	retval = 0;
+
+	if (argc != 3) {
+		warnx("%s %s takes exactly one argument", argv[0], argv[1]);
+		retval = 1;
+		goto bailout;
+	}
+
+	syncarg = argv[2];
+
+	if (strncasecmp(syncarg, "query", min(strlen(syncarg),
+			strlen("query"))) == 0) {
+		if (ioctl(fd, CTL_REALSYNC_GET, &syncstate) == -1) {
+			warn("%s: CTL_REALSYNC_GET ioctl failed", __func__);
+			retval = 1;
+			goto bailout;
+		}
+		fprintf(stdout, "SYNCHRONIZE CACHE support is: ");
+		switch (syncstate) {
+		case 0:
+			fprintf(stdout, "OFF\n");
+			break;
+		case 1:
+			fprintf(stdout, "ON\n");
+			break;
+		default:
+			fprintf(stdout, "unknown (%d)\n", syncstate);
+			break;
+		}
+		goto bailout;
+	} else if (strcasecmp(syncarg, "on") == 0) {
+		syncstate = 1;
+	} else if (strcasecmp(syncarg, "off") == 0) {
+		syncstate = 0;
+	} else {
+		warnx("%s: invalid realsync argument %s", __func__, syncarg);
+		retval = 1;
+		goto bailout;
+	}
+
+	if (ioctl(fd, CTL_REALSYNC_SET, &syncstate) == -1) {
+		warn("%s: CTL_REALSYNC_SET ioctl failed", __func__);
+		retval = 1;
+		goto bailout;
+	}
+bailout:
+	return (retval);
+}
+
+static int
+cctl_getsetsync(int fd, int target, int lun, ctladm_cmdfunction command,
+		int argc, char **argv, char *combinedopt)
+{
+	struct ctl_sync_info sync_info;
+	uint32_t ioctl_cmd;
+	int sync_interval = -1;
+	int retval;
+	int c;
+
+	retval = 0;
+
+	memset(&sync_info, 0, sizeof(sync_info));
+	sync_info.target_id = target;
+	sync_info.lun_id = lun;
+
+	while ((c = getopt(argc, argv, combinedopt)) != -1) {
+		switch (c) {
+		case 'i':
+			sync_interval = strtoul(optarg, NULL, 0);
+			break;
+		default:
+			break;
+		}
+	}
+
+	if (command == CTLADM_CMD_SETSYNC) {
+		if (sync_interval == -1) {
+			warnx("%s: you must specify the sync interval with -i",
+			      __func__);
+			retval = 1;
+			goto bailout;
+		}
+		sync_info.sync_interval = sync_interval;
+		ioctl_cmd = CTL_SETSYNC;
+	} else {
+		ioctl_cmd = CTL_GETSYNC;
+	}
+
+	if (ioctl(fd, ioctl_cmd, &sync_info) == -1) {
+		warn("%s: CTL_%sSYNC ioctl failed", __func__,
+		     (command == CTLADM_CMD_SETSYNC) ? "SET" : "GET");
+		retval = 1;
+		goto bailout;
+	}
+
+	switch (sync_info.status) {
+	case CTL_GS_SYNC_OK:
+		if (command == CTLADM_CMD_GETSYNC) {
+			fprintf(stdout, "%d:%d: sync interval: %d\n",
+				target, lun, sync_info.sync_interval);
+		}
+		break;
+	case CTL_GS_SYNC_NO_LUN:
+		warnx("%s: unknown target:LUN %d:%d", __func__, target, lun);
+		retval = 1;
+		break;
+	case CTL_GS_SYNC_NONE:
+	default:
+		warnx("%s: unknown CTL_%sSYNC status %d", __func__,
+		      (command == CTLADM_CMD_SETSYNC) ? "SET" : "GET",
+		      sync_info.status);
+		retval = 1;
+		break;
+	}
+bailout:
+	return (retval);
+}
+
+struct ctladm_opts cctl_err_types[] = {
+	{"aborted", CTL_LUN_INJ_ABORTED, CTLADM_ARG_NONE, NULL},
+	{"mediumerr", CTL_LUN_INJ_MEDIUM_ERR, CTLADM_ARG_NONE, NULL},
+	{"ua", CTL_LUN_INJ_UA, CTLADM_ARG_NONE, NULL},
+	{"custom", CTL_LUN_INJ_CUSTOM, CTLADM_ARG_NONE, NULL},
+	{NULL, 0, 0, NULL}
+
+};
+
+struct ctladm_opts cctl_err_patterns[] = {
+	{"read", CTL_LUN_PAT_READ, CTLADM_ARG_NONE, NULL},
+	{"write", CTL_LUN_PAT_WRITE, CTLADM_ARG_NONE, NULL},
+	{"rw", CTL_LUN_PAT_READWRITE, CTLADM_ARG_NONE, NULL},
+	{"readwrite", CTL_LUN_PAT_READWRITE, CTLADM_ARG_NONE, NULL},
+	{"readcap", CTL_LUN_PAT_READCAP, CTLADM_ARG_NONE, NULL},
+	{"tur", CTL_LUN_PAT_TUR, CTLADM_ARG_NONE, NULL},
+	{"any", CTL_LUN_PAT_ANY, CTLADM_ARG_NONE, NULL},
+#if 0
+	{"cmd", CTL_LUN_PAT_CMD,  CTLADM_ARG_NONE, NULL},
+#endif
+	{NULL, 0, 0, NULL}
+};
+
+static int
+cctl_error_inject(int fd, uint32_t target, uint32_t lun, int argc, char **argv, 
+		  char *combinedopt)
+{
+	int retval;
+	struct ctl_error_desc err_desc;
+	uint64_t lba = 0;
+	uint32_t len = 0;
+	uint64_t delete_id = 0;
+	int delete_id_set = 0;
+	int continuous = 0;
+	int sense_len = 0;
+	int fd_sense = 0;
+	int c;
+
+	bzero(&err_desc, sizeof(err_desc));
+	err_desc.target_id = target;
+	err_desc.lun_id = lun;
+
+	while ((c = getopt(argc, argv, combinedopt)) != -1) {
+		switch (c) {
+		case 'c':
+			continuous = 1;
+			break;
+		case 'd':
+			delete_id = strtoull(optarg, NULL, 0);
+			delete_id_set = 1;
+			break;
+		case 'i':
+		case 'p': {
+			ctladm_optret optret;
+			ctladm_cmdargs argnum;
+			const char *subopt;
+
+			if (c == 'i') {
+				ctl_lun_error err_type;
+
+				if (err_desc.lun_error != CTL_LUN_INJ_NONE) {
+					warnx("%s: can't specify multiple -i "
+					      "arguments", __func__);
+					retval = 1;
+					goto bailout;
+				}
+				optret = getoption(cctl_err_types, optarg,
+						   &err_type, &argnum, &subopt);
+				err_desc.lun_error = err_type;
+			} else {
+				ctl_lun_error_pattern pattern;
+
+				optret = getoption(cctl_err_patterns, optarg,
+						   &pattern, &argnum, &subopt);
+				err_desc.error_pattern |= pattern;
+			}
+
+			if (optret == CC_OR_AMBIGUOUS) {
+				warnx("%s: ambiguous argument %s", __func__,
+				      optarg);
+				retval = 1;
+				goto bailout;
+			} else if (optret == CC_OR_NOT_FOUND) {
+				warnx("%s: argument %s not found", __func__,
+				      optarg);
+				retval = 1;
+				goto bailout;
+			}
+			break;
+		}
+		case 'r': {
+			char *tmpstr, *tmpstr2;
+
+			tmpstr = strdup(optarg);
+			if (tmpstr == NULL) {
+				warn("%s: error duplicating string %s",
+				     __func__, optarg);
+				retval = 1;
+				goto bailout;
+			}
+
+			tmpstr2 = strsep(&tmpstr, ",");
+			if (tmpstr2 == NULL) {
+				warnx("%s: invalid -r argument %s", __func__,
+				      optarg);
+				retval = 1;
+				free(tmpstr);
+				goto bailout;
+			}
+			lba = strtoull(tmpstr2, NULL, 0);
+			tmpstr2 = strsep(&tmpstr, ",");
+			if (tmpstr2 == NULL) {
+				warnx("%s: no len argument for -r lba,len, got"
+				      " %s", __func__, optarg);
+				retval = 1;
+				free(tmpstr);
+				goto bailout;
+			}
+			len = strtoul(tmpstr2, NULL, 0);
+			free(tmpstr);
+			break;
+		}
+		case 's': {
+			struct get_hook hook;
+			char *sensestr;
+
+			sense_len = strtol(optarg, NULL, 0);
+			if (sense_len <= 0) {
+				warnx("invalid number of sense bytes %d",
+				      sense_len);
+				retval = 1;
+				goto bailout;
+			}
+
+			sense_len = MIN(sense_len, SSD_FULL_SIZE);
+
+			hook.argc = argc - optind;
+			hook.argv = argv + optind;
+			hook.got = 0;
+
+			sensestr = cget(&hook, NULL);
+			if ((sensestr != NULL)
+			 && (sensestr[0] == '-')) {
+				fd_sense = 1;
+			} else {
+				buff_encode_visit(
+				    (uint8_t *)&err_desc.custom_sense,
+				    sense_len, sensestr, iget, &hook);
+			}
+			optind += hook.got;
+			break;
+		}
+		default:
+			break;
+		}
+	}
+
+	if (delete_id_set != 0) {
+		err_desc.serial = delete_id;
+		if (ioctl(fd, CTL_ERROR_INJECT_DELETE, &err_desc) == -1) {
+			warn("%s: error issuing CTL_ERROR_INJECT_DELETE ioctl",
+			     __func__);
+			retval = 1;
+		}
+		goto bailout;
+	}
+
+	if (err_desc.lun_error == CTL_LUN_INJ_NONE) {
+		warnx("%s: error injection command (-i) needed",
+		      __func__);
+		retval = 1;
+		goto bailout;
+	} else if ((err_desc.lun_error == CTL_LUN_INJ_CUSTOM)
+		&& (sense_len == 0)) {
+		warnx("%s: custom error requires -s", __func__);
+		retval = 1;
+		goto bailout;
+	}
+
+	if (continuous != 0)
+		err_desc.lun_error |= CTL_LUN_INJ_CONTINUOUS;
+
+	/*
+	 * If fd_sense is set, we need to read the sense data the user
+	 * wants returned from stdin.
+	 */
+        if (fd_sense == 1) {
+		ssize_t amt_read;
+		int amt_to_read = sense_len;
+ 		u_int8_t *buf_ptr = (uint8_t *)&err_desc.custom_sense;
+
+		for (amt_read = 0; amt_to_read > 0;
+		     amt_read = read(STDIN_FILENO, buf_ptr, amt_to_read)) {
+			if (amt_read == -1) {
+         			warn("error reading sense data from stdin");
+				retval = 1;
+				goto bailout;
+			}
+			amt_to_read -= amt_read;
+			buf_ptr += amt_read;
+		}
+	}
+
+	if (err_desc.error_pattern == CTL_LUN_PAT_NONE) {
+		warnx("%s: command pattern (-p) needed", __func__);
+		retval = 1;
+		goto bailout;
+	}
+
+	if (len != 0) {
+		err_desc.error_pattern |= CTL_LUN_PAT_RANGE;
+		/*
+		 * We could check here to see whether it's a read/write
+		 * command, but that will be pointless once we allow
+		 * custom patterns.  At that point, the user could specify
+		 * a READ(6) CDB type, and we wouldn't have an easy way here
+		 * to verify whether range checking is possible there.  The
+		 * user will just figure it out when his error never gets
+		 * executed.
+		 */
+#if 0
+		if ((err_desc.pattern & CTL_LUN_PAT_READWRITE) == 0) {
+			warnx("%s: need read and/or write pattern if range "
+			      "is specified", __func__);
+			retval = 1;
+			goto bailout;
+		}
+#endif
+		err_desc.lba_range.lba = lba;
+		err_desc.lba_range.len = len;
+	}
+
+	if (ioctl(fd, CTL_ERROR_INJECT, &err_desc) == -1) {
+		warn("%s: error issuing CTL_ERROR_INJECT ioctl", __func__);
+		retval = 1;
+	} else {
+		printf("Error injection succeeded, serial number is %ju\n",
+		       (uintmax_t)err_desc.serial);
+	}
+bailout:
+
+	return (retval);
+}
+
+static int
+cctl_lunlist(int fd)
+{
+	struct scsi_report_luns_data *lun_data;
+	struct scsi_inquiry_data *inq_data;
+	uint32_t num_luns;
+	int target;
+	int initid;
+	unsigned int i;
+	int retval;
+
+	retval = 0;
+	inq_data = NULL;
+
+	target = 6;
+	initid = 7;
+
+	/*
+	 * XXX KDM assuming LUN 0 is fine, but we may need to change this
+	 * if we ever acquire the ability to have multiple targets.
+	 */
+	if ((retval = cctl_get_luns(fd, target, /*lun*/ 0, initid,
+				    /*retries*/ 2, &lun_data, &num_luns)) != 0)
+		goto bailout;
+
+	inq_data = malloc(sizeof(*inq_data));
+	if (inq_data == NULL) {
+		warn("%s: couldn't allocate memory for inquiry data\n",
+		     __func__);
+		retval = 1;
+		goto bailout;
+	}
+	for (i = 0; i < num_luns; i++) {
+		char scsi_path[40];
+		int lun_val;
+
+		switch (lun_data->luns[i].lundata[0] & RPL_LUNDATA_ATYP_MASK) {
+		case RPL_LUNDATA_ATYP_PERIPH:
+			lun_val = lun_data->luns[i].lundata[1];
+			break;
+		case RPL_LUNDATA_ATYP_FLAT:
+			lun_val = (lun_data->luns[i].lundata[0] &
+				RPL_LUNDATA_FLAT_LUN_MASK) |
+				(lun_data->luns[i].lundata[1] <<
+				RPL_LUNDATA_FLAT_LUN_BITS);
+			break;
+		case RPL_LUNDATA_ATYP_LUN:
+		case RPL_LUNDATA_ATYP_EXTLUN:
+		default:
+			fprintf(stdout, "Unsupported LUN format %d\n",
+				lun_data->luns[i].lundata[0] &
+				RPL_LUNDATA_ATYP_MASK);
+			lun_val = -1;
+			break;
+		}
+		if (lun_val == -1)
+			continue;
+
+		if ((retval = cctl_get_inquiry(fd, target, lun_val, initid,
+					       /*retries*/ 2, scsi_path,
+					       sizeof(scsi_path),
+					       inq_data)) != 0) {
+			goto bailout;
+		}
+		printf("%s", scsi_path);
+		scsi_print_inquiry(inq_data);
+	}
+bailout:
+
+	if (lun_data != NULL)
+		free(lun_data);
+
+	if (inq_data != NULL)
+		free(inq_data);
+
+	return (retval);
+}
+
+static void
+cctl_cfi_mt_statusstr(cfi_mt_status status, char *str, int str_len)
+{
+	switch (status) {
+	case CFI_MT_PORT_OFFLINE:
+		snprintf(str, str_len, "Port Offline");
+		break;
+	case CFI_MT_ERROR:
+		snprintf(str, str_len, "Error");
+		break;
+	case CFI_MT_SUCCESS:
+		snprintf(str, str_len, "Success");
+		break;
+	case CFI_MT_NONE:
+		snprintf(str, str_len, "None??");
+		break;
+	default:
+		snprintf(str, str_len, "Unknown status: %d", status);
+		break;
+	}
+}
+
+static void
+cctl_cfi_bbr_statusstr(cfi_bbrread_status status, char *str, int str_len)
+{
+	switch (status) {
+	case CFI_BBR_SUCCESS:
+		snprintf(str, str_len, "Success");
+		break;
+	case CFI_BBR_LUN_UNCONFIG:
+		snprintf(str, str_len, "LUN not configured");
+		break;
+	case CFI_BBR_NO_LUN:
+		snprintf(str, str_len, "LUN does not exist");
+		break;
+	case CFI_BBR_NO_MEM:
+		snprintf(str, str_len, "Memory allocation error");
+		break;
+	case CFI_BBR_BAD_LEN:
+		snprintf(str, str_len, "Length is not a multiple of blocksize");
+		break;
+	case CFI_BBR_RESERV_CONFLICT:
+		snprintf(str, str_len, "Reservation conflict");
+		break;
+	case CFI_BBR_LUN_STOPPED:
+		snprintf(str, str_len, "LUN is powered off");
+		break;
+	case CFI_BBR_LUN_OFFLINE_CTL:
+		snprintf(str, str_len, "LUN is offline");
+		break;
+	case CFI_BBR_LUN_OFFLINE_RC:
+		snprintf(str, str_len, "RAIDCore array is offline (double "
+			 "failure?)");
+		break;
+	case CFI_BBR_SCSI_ERROR:
+		snprintf(str, str_len, "SCSI Error");
+		break;
+	case CFI_BBR_ERROR:
+		snprintf(str, str_len, "Error");
+		break;
+	default:
+		snprintf(str, str_len, "Unknown status: %d", status);
+		break;
+	}
+}
+
+static int
+cctl_hardstopstart(int fd, ctladm_cmdfunction command)
+{
+	struct ctl_hard_startstop_info hs_info;
+	char error_str[256];
+	int do_start;
+	int retval;
+
+	retval = 0;
+
+	if (command == CTLADM_CMD_HARDSTART)
+		do_start = 1;
+	else
+		do_start = 0;
+
+	if (ioctl(fd, (do_start == 1) ? CTL_HARD_START : CTL_HARD_STOP,
+		  &hs_info) == -1) {
+		warn("%s: CTL_HARD_%s ioctl failed", __func__,
+		     (do_start == 1) ? "START" : "STOP");
+		retval = 1;
+		goto bailout;
+	}
+
+	fprintf(stdout, "Hard %s Status: ", (command == CTLADM_CMD_HARDSTOP) ?
+		"Stop" : "Start");
+	cctl_cfi_mt_statusstr(hs_info.status, error_str, sizeof(error_str));
+	fprintf(stdout, "%s\n", error_str);
+	fprintf(stdout, "Total LUNs: %d\n", hs_info.total_luns);
+	fprintf(stdout, "LUNs complete: %d\n", hs_info.luns_complete);
+	fprintf(stdout, "LUNs failed: %d\n", hs_info.luns_failed);
+
+bailout:
+	return (retval);
+}
+
+static int
+cctl_bbrread(int fd, int target __unused, int lun, int iid __unused,
+	     int argc, char **argv, char *combinedopt)
+{
+	struct ctl_bbrread_info bbr_info;
+	char error_str[256];
+	int datalen = -1;
+	uint64_t lba = 0;
+	int lba_set = 0;
+	int retval;
+	int c;
+
+	retval = 0;
+
+	while ((c = getopt(argc, argv, combinedopt)) != -1) {
+		switch (c) {
+		case 'd':
+			datalen = strtoul(optarg, NULL, 0);
+			break;
+		case 'l':
+			lba = strtoull(optarg, NULL, 0);
+			lba_set = 1;
+			break;
+		default:
+			break;
+		}
+	}
+
+	if (lba_set == 0) {
+		warnx("%s: you must specify an LBA with -l", __func__);
+		retval = 1;
+		goto bailout;
+	}
+
+	if (datalen == -1) {
+		warnx("%s: you must specify a length with -d", __func__);
+		retval = 1;
+		goto bailout;
+	}
+
+	bbr_info.lun_num = lun;
+	bbr_info.lba = lba;
+	/*
+	 * XXX KDM get the blocksize first??
+	 */
+	if ((datalen % 512) != 0) {
+		warnx("%s: data length %d is not a multiple of 512 bytes",
+		     __func__, datalen);
+		retval = 1;
+		goto bailout;
+	}
+	bbr_info.len = datalen;
+
+	if (ioctl(fd, CTL_BBRREAD, &bbr_info) == -1) {
+		warn("%s: CTL_BBRREAD ioctl failed", __func__);
+		retval = 1;
+		goto bailout;
+	}
+	cctl_cfi_mt_statusstr(bbr_info.status, error_str, sizeof(error_str));
+	fprintf(stdout, "BBR Read Overall Status: %s\n", error_str);
+	cctl_cfi_bbr_statusstr(bbr_info.bbr_status, error_str,
+			       sizeof(error_str));
+	fprintf(stdout, "BBR Read Status: %s\n", error_str);
+	/*
+	 * XXX KDM should we bother printing out SCSI status if we get
+	 * CFI_BBR_SCSI_ERROR back?
+	 *
+	 * Return non-zero if this fails?
+	 */
+bailout:
+	return (retval);
+}
+
+static int
+cctl_startup_shutdown(int fd, int target, int lun, int iid,
+		      ctladm_cmdfunction command)
+{
+	union ctl_io *io;
+	struct ctl_id id;
+	struct scsi_report_luns_data *lun_data;
+	struct scsi_inquiry_data *inq_data;
+	uint32_t num_luns;
+	unsigned int i;
+	int retval;
+
+	retval = 0;
+	inq_data = NULL;
+
+	/*
+	 * - report luns
+	 * - step through each lun, do an inquiry
+	 * - check OOA queue on direct access luns
+	 * - send stop with offline bit to each direct access device with a
+	 *   clear OOA queue
+	 *   - if we get a reservation conflict, reset the LUN to clear it
+	 *     and reissue the stop with the offline bit set
+	 */
+
+	id.id = iid;
+
+	io = ctl_scsi_alloc_io(id);
+	if (io == NULL) {
+		warnx("%s: can't allocate memory", __func__);
+		return (1);
+	}
+
+	if ((retval = cctl_get_luns(fd, target, lun, iid, /*retries*/ 2,
+				    &lun_data, &num_luns)) != 0)
+		goto bailout;
+
+	inq_data = malloc(sizeof(*inq_data));
+	if (inq_data == NULL) {
+		warn("%s: couldn't allocate memory for inquiry data\n",
+		     __func__);
+		retval = 1;
+		goto bailout;
+	}
+	for (i = 0; i < num_luns; i++) {
+		char scsi_path[40];
+		int lun_val;
+
+		/*
+		 * XXX KDM figure out a way to share this code with
+		 * cctl_lunlist()?
+		 */
+		switch (lun_data->luns[i].lundata[0] & RPL_LUNDATA_ATYP_MASK) {
+		case RPL_LUNDATA_ATYP_PERIPH:
+			lun_val = lun_data->luns[i].lundata[1];
+			break;
+		case RPL_LUNDATA_ATYP_FLAT:
+			lun_val = (lun_data->luns[i].lundata[0] &
+				RPL_LUNDATA_FLAT_LUN_MASK) |
+				(lun_data->luns[i].lundata[1] <<
+				RPL_LUNDATA_FLAT_LUN_BITS);
+			break;
+		case RPL_LUNDATA_ATYP_LUN:
+		case RPL_LUNDATA_ATYP_EXTLUN:
+		default:
+			fprintf(stdout, "Unsupported LUN format %d\n",
+				lun_data->luns[i].lundata[0] &
+				RPL_LUNDATA_ATYP_MASK);
+			lun_val = -1;
+			break;
+		}
+		if (lun_val == -1)
+			continue;
+
+		if ((retval = cctl_get_inquiry(fd, target, lun_val, iid,
+					       /*retries*/ 2, scsi_path,
+					       sizeof(scsi_path),
+					       inq_data)) != 0) {
+			goto bailout;
+		}
+		printf("%s", scsi_path);
+		scsi_print_inquiry(inq_data);
+		/*
+		 * We only want to shutdown direct access devices.
+		 */
+		if (SID_TYPE(inq_data) != T_DIRECT) {
+			printf("%s LUN is not direct access, skipped\n",
+			       scsi_path);
+			continue;
+		}
+
+		if (command == CTLADM_CMD_SHUTDOWN) {
+			struct ctl_ooa_info ooa_info;
+
+			ooa_info.target_id = target;
+			ooa_info.lun_id = lun_val;
+
+			if (ioctl(fd, CTL_CHECK_OOA, &ooa_info) == -1) {
+				printf("%s CTL_CHECK_OOA ioctl failed\n",
+				       scsi_path);
+				continue;
+			}
+
+			if (ooa_info.status != CTL_OOA_SUCCESS) {
+				printf("%s CTL_CHECK_OOA returned status %d\n", 
+				       scsi_path, ooa_info.status);
+				continue;
+			}
+			if (ooa_info.num_entries != 0) {
+				printf("%s %d entr%s in the OOA queue, "
+				       "skipping shutdown\n", scsi_path,
+				       ooa_info.num_entries,
+				       (ooa_info.num_entries > 1)?"ies" : "y" );
+				continue;
+			}
+		}
+		
+		ctl_scsi_start_stop(/*io*/ io,
+				    /*start*/(command == CTLADM_CMD_STARTUP) ?
+					      1 : 0,
+				    /*load_eject*/ 0,
+				    /*immediate*/ 0,
+				    /*power_conditions*/ SSS_PC_START_VALID,
+				    /*onoffline*/ 1,
+				    /*ctl_tag_type*/
+				    (command == CTLADM_CMD_STARTUP) ?
+				    CTL_TAG_SIMPLE :CTL_TAG_ORDERED,
+				    /*control*/ 0);
+
+		io->io_hdr.nexus.targ_target.id = target;
+		io->io_hdr.nexus.targ_lun = lun_val;
+		io->io_hdr.nexus.initid = id;
+
+		if (cctl_do_io(fd, /*retries*/ 3, io, __func__) != 0) {
+			retval = 1;
+			goto bailout;
+		}
+
+		if ((io->io_hdr.status & CTL_STATUS_MASK) != CTL_SUCCESS)
+			ctl_io_error_print(io, inq_data, stderr);
+		else {
+			printf("%s LUN is now %s\n", scsi_path,
+			       (command == CTLADM_CMD_STARTUP) ? "online" :
+			       "offline");
+		}
+	}
+bailout:
+	if (lun_data != NULL)
+		free(lun_data);
+
+	if (inq_data != NULL)
+		free(inq_data);
+
+	if (io != NULL)
+		ctl_scsi_free_io(io);
+
+	return (retval);
+}
+
+static int
+cctl_sync_cache(int fd, int target, int lun, int iid, int retries,
+		int argc, char **argv, char *combinedopt)
+{
+	union ctl_io *io;
+	struct ctl_id id;
+	int cdb_size = -1;
+	int retval;
+	uint64_t our_lba = 0;
+	uint32_t our_block_count = 0;
+	int reladr = 0, immed = 0; 
+	int c;
+
+	id.id = iid;
+	retval = 0;
+
+	io = ctl_scsi_alloc_io(id);
+	if (io == NULL) {
+		warnx("%s: can't allocate memory", __func__);
+		return (1);
+	}
+
+	while ((c = getopt(argc, argv, combinedopt)) != -1) {
+		switch (c) {
+		case 'b':
+			our_block_count = strtoul(optarg, NULL, 0);
+			break;
+		case 'c':
+			cdb_size = strtol(optarg, NULL, 0);
+			break;
+		case 'i':
+			immed = 1;
+			break;
+		case 'l':
+			our_lba = strtoull(optarg, NULL, 0);
+			break;
+		case 'r':
+			reladr = 1;
+			break;
+		default:
+			break;
+		}
+	}
+
+	if (cdb_size != -1) {
+		switch (cdb_size) {
+		case 10:
+		case 16:
+			break;
+		default:
+			warnx("%s: invalid cdbsize %d, valid sizes are 10 "
+			      "and 16", __func__, cdb_size);
+			retval = 1;
+			goto bailout;
+			break; /* NOTREACHED */
+		}
+	} else
+		cdb_size = 10;
+
+	ctl_scsi_sync_cache(/*io*/ io,
+			    /*immed*/ immed,
+			    /*reladr*/ reladr,
+			    /*minimum_cdb_size*/ cdb_size,
+			    /*starting_lba*/ our_lba,
+			    /*block_count*/ our_block_count,
+			    /*tag_type*/ CTL_TAG_SIMPLE,
+			    /*control*/ 0);
+
+	io->io_hdr.nexus.targ_target.id = target;
+	io->io_hdr.nexus.targ_lun = lun;
+	io->io_hdr.nexus.initid = id;
+
+	if (cctl_do_io(fd, retries, io, __func__) != 0) {
+		retval = 1;
+		goto bailout;
+	}
+
+	if ((io->io_hdr.status & CTL_STATUS_MASK) == CTL_SUCCESS) {
+		fprintf(stdout, "Cache synchronized successfully\n");
+	} else
+		ctl_io_error_print(io, NULL, stderr);
+bailout:
+	ctl_scsi_free_io(io);
+
+	return (retval);
+}
+
+static int
+cctl_start_stop(int fd, int target, int lun, int iid, int retries, int start,
+		int argc, char **argv, char *combinedopt)
+{
+	union ctl_io *io;
+	struct ctl_id id;
+	char scsi_path[40];
+	int immed = 0, onoffline = 0;
+	int retval, c;
+
+	id.id = iid;
+	retval = 0;
+
+	io = ctl_scsi_alloc_io(id);
+	if (io == NULL) {
+		warnx("%s: can't allocate memory", __func__);
+		return (1);
+	}
+
+	while ((c = getopt(argc, argv, combinedopt)) != -1) {
+		switch (c) {
+		case 'i':
+			immed = 1;
+			break;
+		case 'o':
+			onoffline = 1;
+			break;
+		default:
+			break;
+		}
+	}
+	/*
+	 * Use an ordered tag for the stop command, to guarantee that any
+	 * pending I/O will finish before the stop command executes.  This
+	 * would normally be the case anyway, since CTL will basically
+	 * treat the start/stop command as an ordered command with respect
+	 * to any other command except an INQUIRY.  (See ctl_ser_table.c.)
+	 */
+	ctl_scsi_start_stop(/*io*/ io,
+			    /*start*/ start,
+			    /*load_eject*/ 0,
+			    /*immediate*/ immed,
+			    /*power_conditions*/ SSS_PC_START_VALID,
+			    /*onoffline*/ onoffline,
+			    /*ctl_tag_type*/ start ? CTL_TAG_SIMPLE :
+						     CTL_TAG_ORDERED,
+			    /*control*/ 0);
+
+	io->io_hdr.nexus.targ_target.id = target;
+	io->io_hdr.nexus.targ_lun = lun;
+	io->io_hdr.nexus.initid = id;
+
+	if (cctl_do_io(fd, retries, io, __func__) != 0) {
+		retval = 1;
+		goto bailout;
+	}
+
+	ctl_scsi_path_string(io, scsi_path, sizeof(scsi_path));
+	if ((io->io_hdr.status & CTL_STATUS_MASK) == CTL_SUCCESS) {
+		fprintf(stdout, "%s LUN %s successfully\n", scsi_path,
+			(start) ?  "started" : "stopped");
+	} else
+		ctl_io_error_print(io, NULL, stderr);
+
+bailout:
+	ctl_scsi_free_io(io);
+
+	return (retval);
+}
+
+static int
+cctl_mode_sense(int fd, int target, int lun, int iid, int retries, 
+		int argc, char **argv, char *combinedopt)
+{
+	union ctl_io *io;
+	struct ctl_id id;
+	uint32_t datalen;
+	uint8_t *dataptr;
+	int pc = -1, cdbsize, retval, dbd = 0, subpage = -1;
+	int list = 0;
+	int page_code = -1;
+	int c;
+
+	id.id = iid;
+	cdbsize = 0;
+	retval = 0;
+	dataptr = NULL;
+
+	io = ctl_scsi_alloc_io(id);
+	if (io == NULL) {
+		warn("%s: can't allocate memory", __func__);
+		return (1);
+	}
+
+	while ((c = getopt(argc, argv, combinedopt)) != -1) {
+		switch (c) {
+		case 'P':
+			pc = strtoul(optarg, NULL, 0);
+			break;
+		case 'S':
+			subpage = strtoul(optarg, NULL, 0);
+			break;
+		case 'd':
+			dbd = 1;
+			break;
+		case 'l':
+			list = 1;
+			break;
+		case 'm':
+			page_code = strtoul(optarg, NULL, 0);
+			break;
+		case 'c':
+			cdbsize = strtol(optarg, NULL, 0);
+			break;
+		default:
+			break;
+		}
+	}
+
+	if (((list == 0) && (page_code == -1))
+	 || ((list != 0) && (page_code != -1))) {
+		warnx("%s: you must specify either a page code (-m) or -l",
+		      __func__);
+		retval = 1;
+		goto bailout;
+	}
+
+	if ((page_code != -1)
+	 && ((page_code > SMS_ALL_PAGES_PAGE)
+	  || (page_code < 0))) {
+		warnx("%s: page code %d is out of range", __func__,
+		      page_code);
+		retval = 1;
+		goto bailout;
+	}
+
+	if (list == 1) {
+		page_code = SMS_ALL_PAGES_PAGE;
+		if (pc != -1) {
+			warnx("%s: arg -P makes no sense with -l",
+			      __func__);
+			retval = 1;
+			goto bailout;
+		}
+		if (subpage != -1) {
+			warnx("%s: arg -S makes no sense with -l", __func__);
+			retval = 1;
+			goto bailout;
+		}
+	}
+
+	if (pc == -1)
+		pc = SMS_PAGE_CTRL_CURRENT;
+	else {
+		if ((pc > 3)
+		 || (pc < 0)) {
+			warnx("%s: page control value %d is out of range: 0-3",
+			      __func__, pc);
+			retval = 1;
+			goto bailout;
+		}
+	}
+
+
+	if ((subpage != -1)
+	 && ((subpage > 255)
+	  || (subpage < 0))) {
+		warnx("%s: subpage code %d is out of range: 0-255", __func__,
+		      subpage);
+		retval = 1;
+		goto bailout;
+	}
+	if (cdbsize != 0) {
+		switch (cdbsize) {
+		case 6:
+		case 10:
+			break;
+		default:
+			warnx("%s: invalid cdbsize %d, valid sizes are 6 "
+			      "and 10", __func__, cdbsize);
+			retval = 1;
+			goto bailout;
+			break;
+		}
+	} else
+		cdbsize = 6;
+
+	if (subpage == -1)
+		subpage = 0;
+
+	if (cdbsize == 6)
+		datalen = 255;
+	else
+		datalen = 65535;
+
+	dataptr = (uint8_t *)malloc(datalen);
+	if (dataptr == NULL) {
+		warn("%s: can't allocate %d bytes", __func__, datalen);
+		retval = 1;
+		goto bailout;
+	}
+
+	memset(dataptr, 0, datalen);
+
+	ctl_scsi_mode_sense(io,
+			    /*data_ptr*/ dataptr,
+			    /*data_len*/ datalen,
+			    /*dbd*/ dbd,
+			    /*llbaa*/ 0,
+			    /*page_code*/ page_code,
+			    /*pc*/ pc << 6,
+			    /*subpage*/ subpage,
+			    /*minimum_cdb_size*/ cdbsize,
+			    /*tag_type*/ CTL_TAG_SIMPLE,
+			    /*control*/ 0);
+
+	io->io_hdr.nexus.targ_target.id = target;
+	io->io_hdr.nexus.targ_lun = lun;
+	io->io_hdr.nexus.initid = id;
+
+	if (cctl_do_io(fd, retries, io, __func__) != 0) {
+		retval = 1;
+		goto bailout;
+	}
+
+	if ((io->io_hdr.status & CTL_STATUS_MASK) == CTL_SUCCESS) {
+		int pages_len, used_len;
+		uint32_t returned_len;
+		uint8_t *ndataptr;
+
+		if (io->scsiio.cdb[0] == MODE_SENSE_6) {
+			struct scsi_mode_hdr_6 *hdr6;
+			int bdlen;
+
+			hdr6 = (struct scsi_mode_hdr_6 *)dataptr;
+
+			returned_len = hdr6->datalen + 1;
+			bdlen = hdr6->block_descr_len;
+
+			ndataptr = (uint8_t *)((uint8_t *)&hdr6[1] + bdlen);
+		} else {
+			struct scsi_mode_hdr_10 *hdr10;
+			int bdlen;
+
+			hdr10 = (struct scsi_mode_hdr_10 *)dataptr;
+
+			returned_len = scsi_2btoul(hdr10->datalen) + 2;
+			bdlen = scsi_2btoul(hdr10->block_descr_len);
+
+			ndataptr = (uint8_t *)((uint8_t *)&hdr10[1] + bdlen);
+		}
+		/* just in case they can give us more than we allocated for */
+		returned_len = min(returned_len, datalen);
+		pages_len = returned_len - (ndataptr - dataptr);
+#if 0
+		fprintf(stdout, "returned_len = %d, pages_len = %d\n",
+			returned_len, pages_len);
+#endif
+		if (list == 1) {
+			fprintf(stdout, "Supported mode pages:\n");
+			for (used_len = 0; used_len < pages_len;) {
+				struct scsi_mode_page_header *header;
+
+				header = (struct scsi_mode_page_header *)
+					&ndataptr[used_len];
+				fprintf(stdout, "%d\n", header->page_code);
+				used_len += header->page_length + 2;
+			}
+		} else {
+			for (used_len = 0; used_len < pages_len; used_len++) {
+				fprintf(stdout, "0x%x ", ndataptr[used_len]);
+				if (((used_len+1) % 16) == 0)
+					fprintf(stdout, "\n");
+			}
+			fprintf(stdout, "\n");
+		}
+	} else
+		ctl_io_error_print(io, NULL, stderr);
+bailout:
+
+	ctl_scsi_free_io(io);
+
+	if (dataptr != NULL)
+		free(dataptr);
+
+	return (retval);
+}
+
+static int
+cctl_read_capacity(int fd, int target, int lun, int iid, int retries, 
+           	   int argc, char **argv, char *combinedopt)
+{
+	union ctl_io *io;
+	struct ctl_id id;
+	struct scsi_read_capacity_data *data;
+	struct scsi_read_capacity_data_long *longdata;
+	int cdbsize = -1, retval;
+	uint8_t *dataptr;
+	int c;
+
+	cdbsize = 10;
+	dataptr = NULL;
+	retval = 0;
+	id.id = iid;
+
+	io = ctl_scsi_alloc_io(id);
+	if (io == NULL) {
+		warn("%s: can't allocate memory\n", __func__);
+		return (1);
+	}
+
+	while ((c = getopt(argc, argv, combinedopt)) != -1) {
+		switch (c) {
+		case 'c':
+			cdbsize = strtol(optarg, NULL, 0);
+			break;
+		default:
+			break;
+		}
+	}
+	if (cdbsize != -1) {
+		switch (cdbsize) {
+		case 10:
+		case 16:
+			break;
+		default:
+			warnx("%s: invalid cdbsize %d, valid sizes are 10 "
+			      "and 16", __func__, cdbsize);
+			retval = 1;
+			goto bailout;
+			break; /* NOTREACHED */
+		}
+	} else
+		cdbsize = 10;
+
+	dataptr = (uint8_t *)malloc(sizeof(*longdata));
+	if (dataptr == NULL) {
+		warn("%s: can't allocate %zd bytes\n", __func__,
+		     sizeof(*longdata));
+		retval = 1;
+		goto bailout;
+	}
+	memset(dataptr, 0, sizeof(*longdata));
+
+retry:
+
+	switch (cdbsize) {
+	case 10:
+		ctl_scsi_read_capacity(io,
+				       /*data_ptr*/ dataptr,
+				       /*data_len*/ sizeof(*longdata),
+				       /*addr*/ 0,
+				       /*reladr*/ 0,
+				       /*pmi*/ 0,
+				       /*tag_type*/ CTL_TAG_SIMPLE,
+				       /*control*/ 0);
+		break;
+	case 16:
+		ctl_scsi_read_capacity_16(io,
+					  /*data_ptr*/ dataptr,
+					  /*data_len*/ sizeof(*longdata),
+					  /*addr*/ 0,
+					  /*reladr*/ 0,
+					  /*pmi*/ 0,
+					  /*tag_type*/ CTL_TAG_SIMPLE,
+					  /*control*/ 0);
+		break;
+	}
+
+	io->io_hdr.nexus.initid = id;
+	io->io_hdr.nexus.targ_target.id = target;
+	io->io_hdr.nexus.targ_lun = lun;
+
+	if (cctl_do_io(fd, retries, io, __func__) != 0) {
+		retval = 1;
+		goto bailout;
+	}
+
+	if ((io->io_hdr.status & CTL_STATUS_MASK) == CTL_SUCCESS) {
+		uint64_t maxlba;
+		uint32_t blocksize;
+
+		if (cdbsize == 10) {
+
+			data = (struct scsi_read_capacity_data *)dataptr;
+
+			maxlba = scsi_4btoul(data->addr);
+			blocksize = scsi_4btoul(data->length);
+
+			if (maxlba == 0xffffffff) {
+				cdbsize = 16;
+				goto retry;
+			}
+		} else {
+			longdata=(struct scsi_read_capacity_data_long *)dataptr;
+
+			maxlba = scsi_8btou64(longdata->addr);
+			blocksize = scsi_4btoul(longdata->length);
+		}
+
+		fprintf(stdout, "Disk Capacity: %ju, Blocksize: %d\n",
+			(uintmax_t)maxlba, blocksize);
+	} else {
+		ctl_io_error_print(io, NULL, stderr);
+	}
+bailout:
+	ctl_scsi_free_io(io);
+
+	if (dataptr != NULL)
+		free(dataptr);
+
+	return (retval);
+}
+
+static int
+cctl_read_write(int fd, int target, int lun, int iid, int retries,
+		int argc, char **argv, char *combinedopt,
+		ctladm_cmdfunction command)
+{
+	union ctl_io *io;
+	struct ctl_id id;
+	int file_fd, do_stdio;
+	int cdbsize = -1, databytes;
+	uint8_t *dataptr;
+	char *filename = NULL;
+	int datalen = -1, blocksize = -1;
+	uint64_t lba = 0;
+	int lba_set = 0;
+	int retval;
+	int c;
+
+	retval = 0;
+	do_stdio = 0;
+	dataptr = NULL;
+	file_fd = -1;
+	id.id = iid;
+
+	io = ctl_scsi_alloc_io(id);
+	if (io == NULL) {
+		warn("%s: can't allocate memory\n", __func__);
+		return (1);
+	}
+
+	while ((c = getopt(argc, argv, combinedopt)) != -1) {
+		switch (c) {
+		case 'N':
+			io->io_hdr.flags |= CTL_FLAG_NO_DATAMOVE;
+			break;
+		case 'b':
+			blocksize = strtoul(optarg, NULL, 0);
+			break;
+		case 'c':
+			cdbsize = strtoul(optarg, NULL, 0);
+			break;
+		case 'd':
+			datalen = strtoul(optarg, NULL, 0);
+			break;
+		case 'f':
+			filename = strdup(optarg);
+			break;
+		case 'l':
+			lba = strtoull(optarg, NULL, 0);
+			lba_set = 1;
+			break;
+		default:
+			break;
+		}
+	}
+	if (filename == NULL) {
+		warnx("%s: you must supply a filename using -f", __func__);
+		retval = 1;
+		goto bailout;
+	}
+
+	if (datalen == -1) {
+		warnx("%s: you must specify the data length with -d", __func__);
+		retval = 1;
+		goto bailout;
+	}
+
+	if (lba_set == 0) {
+		warnx("%s: you must specify the LBA with -l", __func__);
+		retval = 1;
+		goto bailout;
+	}
+
+	if (blocksize == -1) {
+		warnx("%s: you must specify the blocksize with -b", __func__);
+		retval = 1;
+		goto bailout;
+	}
+
+	if (cdbsize != -1) {
+		switch (cdbsize) {
+		case 6:
+		case 10:
+		case 12:
+		case 16:
+			break;
+		default:
+			warnx("%s: invalid cdbsize %d, valid sizes are 6, "
+			      "10, 12 or 16", __func__, cdbsize);
+			retval = 1;
+			goto bailout;
+			break; /* NOTREACHED */
+		}
+	} else
+		cdbsize = 6;
+
+	databytes = datalen * blocksize;
+	dataptr = (uint8_t *)malloc(databytes);
+
+	if (dataptr == NULL) {
+		warn("%s: can't allocate %d bytes\n", __func__, databytes);
+		retval = 1;
+		goto bailout;
+	}
+	if (strcmp(filename, "-") == 0) {
+		if (command == CTLADM_CMD_READ)
+			file_fd = STDOUT_FILENO;
+		else
+			file_fd = STDIN_FILENO;
+		do_stdio = 1;
+	} else {
+		file_fd = open(filename, O_RDWR | O_CREAT, S_IRUSR | S_IWUSR);
+		if (file_fd == -1) {
+			warn("%s: can't open file %s", __func__, filename);
+			retval = 1;
+			goto bailout;
+		}
+	}
+
+	memset(dataptr, 0, databytes);
+
+	if (command == CTLADM_CMD_WRITE) {
+		int bytes_read;
+
+		bytes_read = read(file_fd, dataptr, databytes);
+		if (bytes_read == -1) {
+			warn("%s: error reading file %s", __func__, filename);
+			retval = 1;
+			goto bailout;
+		}
+		if (bytes_read != databytes) {
+			warnx("%s: only read %d bytes from file %s",
+			      __func__, bytes_read, filename);
+			retval = 1;
+			goto bailout;
+		}
+	}
+	ctl_scsi_read_write(io,
+			    /*data_ptr*/ dataptr,
+			    /*data_len*/ databytes,
+			    /*read_op*/ (command == CTLADM_CMD_READ) ? 1 : 0,
+			    /*byte2*/ 0,
+			    /*minimum_cdb_size*/ cdbsize,
+			    /*lba*/ lba,
+			    /*num_blocks*/ datalen,
+			    /*tag_type*/ CTL_TAG_SIMPLE,
+			    /*control*/ 0);
+
+	io->io_hdr.nexus.targ_target.id = target;
+	io->io_hdr.nexus.targ_lun = lun;
+	io->io_hdr.nexus.initid = id;
+
+	if (cctl_do_io(fd, retries, io, __func__) != 0) {
+		retval = 1;
+		goto bailout;
+	}
+
+	if (((io->io_hdr.status & CTL_STATUS_MASK) == CTL_SUCCESS)
+	 && (command == CTLADM_CMD_READ)) {
+		int bytes_written;
+
+		bytes_written = write(file_fd, dataptr, databytes);
+		if (bytes_written == -1) {
+			warn("%s: can't write to %s", __func__, filename);
+			goto bailout;
+		}
+	} else if ((io->io_hdr.status & CTL_STATUS_MASK) != CTL_SUCCESS)
+		ctl_io_error_print(io, NULL, stderr);
+
+
+bailout:
+
+	ctl_scsi_free_io(io);
+
+	if (dataptr != NULL)
+		free(dataptr);
+
+	if ((do_stdio == 0)
+	 && (file_fd != -1))
+		close(file_fd);
+
+	return (retval);
+}
+
+static int
+cctl_get_luns(int fd, int target, int lun, int iid, int retries, struct
+	      scsi_report_luns_data **lun_data, uint32_t *num_luns)
+{
+	union ctl_io *io;
+	struct ctl_id id;
+	uint32_t nluns;
+	int lun_datalen;
+	int retval;
+
+	retval = 0;
+	id.id = iid;
+
+	io = ctl_scsi_alloc_io(id);
+	if (io == NULL) {
+		warnx("%s: can't allocate memory", __func__);
+		return (1);
+	}
+
+	/*
+	 * lun_data includes space for 1 lun, allocate space for 4 initially.
+	 * If that isn't enough, we'll allocate more.
+	 */
+	nluns = 4;
+retry:
+	lun_datalen = sizeof(*lun_data) +
+		(nluns * sizeof(struct scsi_report_luns_lundata));
+	*lun_data = malloc(lun_datalen);
+
+	if (*lun_data == NULL) {
+		warnx("%s: can't allocate memory", __func__);
+		ctl_scsi_free_io(io);
+		return (1);
+	}
+
+	ctl_scsi_report_luns(io,
+			     /*data_ptr*/ (uint8_t *)*lun_data,
+			     /*data_len*/ lun_datalen,
+			     /*select_report*/ RPL_REPORT_ALL,
+			     /*tag_type*/ CTL_TAG_SIMPLE,
+			     /*control*/ 0);
+
+	io->io_hdr.nexus.initid = id;
+	io->io_hdr.nexus.targ_target.id = target;
+	io->io_hdr.nexus.targ_lun = lun;
+
+	if (cctl_do_io(fd, retries, io, __func__) != 0) {
+		retval = 1;
+		goto bailout;
+	}
+
+	if ((io->io_hdr.status & CTL_STATUS_MASK) == CTL_SUCCESS) {
+		uint32_t returned_len, returned_luns;
+
+		returned_len = scsi_4btoul((*lun_data)->length);
+		returned_luns = returned_len / 8;
+		if (returned_luns > nluns) {
+			nluns = returned_luns;
+			free(*lun_data);
+			goto retry;
+		}
+		/* These should be the same */
+		*num_luns = MIN(returned_luns, nluns);
+	} else {
+		ctl_io_error_print(io, NULL, stderr);
+		retval = 1;
+	}
+bailout:
+	ctl_scsi_free_io(io);
+
+	return (retval);
+}
+
+static int
+cctl_report_luns(int fd, int target, int lun, int iid, int retries)
+{
+	struct scsi_report_luns_data *lun_data;
+	uint32_t num_luns, i;
+	int retval;
+
+	lun_data = NULL;
+
+	if ((retval = cctl_get_luns(fd, target, lun, iid, retries, &lun_data,
+				   &num_luns)) != 0)
+		goto bailout;
+
+	fprintf(stdout, "%u LUNs returned\n", num_luns);
+	for (i = 0; i < num_luns; i++) {
+		int lun_val;
+
+		/*
+		 * XXX KDM figure out a way to share this code with
+		 * cctl_lunlist()?
+		 */
+		switch (lun_data->luns[i].lundata[0] & RPL_LUNDATA_ATYP_MASK) {
+		case RPL_LUNDATA_ATYP_PERIPH:
+			lun_val = lun_data->luns[i].lundata[1];
+			break;
+		case RPL_LUNDATA_ATYP_FLAT:
+			lun_val = (lun_data->luns[i].lundata[0] &
+				RPL_LUNDATA_FLAT_LUN_MASK) |
+				(lun_data->luns[i].lundata[1] <<
+				RPL_LUNDATA_FLAT_LUN_BITS);
+			break;
+		case RPL_LUNDATA_ATYP_LUN:
+		case RPL_LUNDATA_ATYP_EXTLUN:
+		default:
+			fprintf(stdout, "Unsupported LUN format %d\n",
+				lun_data->luns[i].lundata[0] &
+				RPL_LUNDATA_ATYP_MASK);
+			lun_val = -1;
+			break;
+		}
+		if (lun_val == -1)
+			continue;
+
+		fprintf(stdout, "%d\n", lun_val);
+	}
+
+bailout:
+	if (lun_data != NULL)
+		free(lun_data);
+
+	return (retval);
+}
+
+static int
+cctl_tur(int fd, int target, int lun, int iid, int retries)
+{
+	union ctl_io *io;
+	struct ctl_id id;
+
+	id.id = iid;
+
+	io = ctl_scsi_alloc_io(id);
+	if (io == NULL) {
+		fprintf(stderr, "can't allocate memory\n");
+		return (1);
+	}
+
+	ctl_scsi_tur(io,
+		     /* tag_type */ CTL_TAG_SIMPLE,
+		     /* control */ 0);
+
+	io->io_hdr.nexus.targ_target.id = target;
+	io->io_hdr.nexus.targ_lun = lun;
+	io->io_hdr.nexus.initid = id;
+
+	if (cctl_do_io(fd, retries, io, __func__) != 0) {
+		ctl_scsi_free_io(io);
+		return (1);
+	}
+
+	if ((io->io_hdr.status & CTL_STATUS_MASK) == CTL_SUCCESS)
+		fprintf(stdout, "Unit is ready\n");
+	else
+		ctl_io_error_print(io, NULL, stderr);
+
+	return (0);
+}
+
+static int
+cctl_get_inquiry(int fd, int target, int lun, int iid, int retries, 
+		 char *path_str, int path_len,
+		 struct scsi_inquiry_data *inq_data)
+{
+	union ctl_io *io;
+	struct ctl_id id;
+	int retval;
+
+	retval = 0;
+
+	id.id = iid;
+
+	io = ctl_scsi_alloc_io(id);
+	if (io == NULL) {
+		warnx("cctl_inquiry: can't allocate memory\n");
+		return (1);
+	}
+
+	ctl_scsi_inquiry(/*io*/ io,
+			 /*data_ptr*/ (uint8_t *)inq_data,
+			 /*data_len*/ sizeof(*inq_data),
+			 /*byte2*/ 0,
+			 /*page_code*/ 0,
+			 /*tag_type*/ CTL_TAG_SIMPLE,
+			 /*control*/ 0);
+
+	io->io_hdr.nexus.targ_target.id = target;
+	io->io_hdr.nexus.targ_lun = lun;
+	io->io_hdr.nexus.initid = id;
+
+	if (cctl_do_io(fd, retries, io, __func__) != 0) {
+		retval = 1;
+		goto bailout;
+	}
+
+	if ((io->io_hdr.status & CTL_STATUS_MASK) != CTL_SUCCESS) {
+		retval = 1;
+		ctl_io_error_print(io, NULL, stderr);
+	} else if (path_str != NULL)
+		ctl_scsi_path_string(io, path_str, path_len);
+
+bailout:
+	ctl_scsi_free_io(io);
+
+	return (retval);
+}
+
+static int
+cctl_inquiry(int fd, int target, int lun, int iid, int retries)
+{
+	struct scsi_inquiry_data *inq_data;
+	char scsi_path[40];
+	int retval;
+
+	retval = 0;
+
+	inq_data = malloc(sizeof(*inq_data));
+	if (inq_data == NULL) {
+		warnx("%s: can't allocate inquiry data", __func__);
+		retval = 1;
+		goto bailout;
+	}
+
+	if ((retval = cctl_get_inquiry(fd, target, lun, iid, retries, scsi_path,
+				       sizeof(scsi_path), inq_data)) != 0)
+		goto bailout;
+
+	printf("%s", scsi_path);
+	scsi_print_inquiry(inq_data);
+
+bailout:
+	if (inq_data != NULL)
+		free(inq_data);
+
+	return (retval);
+}
+
+static int
+cctl_req_sense(int fd, int target, int lun, int iid, int retries)
+{
+	union ctl_io *io;
+	struct scsi_sense_data *sense_data;
+	struct ctl_id id;
+	int retval;
+
+	retval = 0;
+
+	id.id = iid;
+
+	io = ctl_scsi_alloc_io(id);
+	if (io == NULL) {
+		warnx("cctl_req_sense: can't allocate memory\n");
+		return (1);
+	}
+	sense_data = malloc(sizeof(*sense_data));
+	memset(sense_data, 0, sizeof(*sense_data));
+
+	ctl_scsi_request_sense(/*io*/ io,
+			       /*data_ptr*/ (uint8_t *)sense_data,
+			       /*data_len*/ sizeof(*sense_data),
+			       /*byte2*/ 0,
+			       /*tag_type*/ CTL_TAG_SIMPLE,
+			       /*control*/ 0);
+
+	io->io_hdr.nexus.targ_target.id = target;
+	io->io_hdr.nexus.targ_lun = lun;
+	io->io_hdr.nexus.initid = id;
+
+	if (cctl_do_io(fd, retries, io, __func__) != 0) {
+		retval = 1;
+		goto bailout;
+	}
+
+	if ((io->io_hdr.status & CTL_STATUS_MASK) == CTL_SUCCESS) {
+		bcopy(sense_data, &io->scsiio.sense_data, sizeof(*sense_data));
+		io->scsiio.sense_len = sizeof(*sense_data);
+		ctl_scsi_sense_print(&io->scsiio, NULL, stdout);
+	} else
+		ctl_io_error_print(io, NULL, stderr);
+
+bailout:
+
+	ctl_scsi_free_io(io);
+	free(sense_data);
+
+	return (retval);
+}
+
+static int
+cctl_report_target_port_group(int fd, int target, int lun, int initiator)
+{
+	union ctl_io *io;
+	struct ctl_id id;
+	uint32_t datalen;
+	uint8_t *dataptr;
+	int retval;
+
+	id.id = initiator;
+	dataptr = NULL;
+	retval = 0;
+
+	io = ctl_scsi_alloc_io(id);
+	if (io == NULL) {
+		warn("%s: can't allocate memory", __func__);
+		return (1);
+	}
+
+	datalen = 64;
+	dataptr = (uint8_t *)malloc(datalen);
+	if (dataptr == NULL) {
+		warn("%s: can't allocate %d bytes", __func__, datalen);
+	    	retval = 1;
+		goto bailout;
+	}
+
+	memset(dataptr, 0, datalen);
+
+	ctl_scsi_maintenance_in(/*io*/ io,
+				/*data_ptr*/ dataptr,
+				/*data_len*/ datalen,
+				/*action*/ SA_RPRT_TRGT_GRP,
+				/*tag_type*/ CTL_TAG_SIMPLE,
+				/*control*/ 0);
+
+	io->io_hdr.nexus.targ_target.id = target;
+	io->io_hdr.nexus.targ_lun = lun;
+	io->io_hdr.nexus.initid = id;
+
+	if (cctl_do_io(fd, 0, io, __func__) != 0) {
+		retval = 1;
+		goto bailout;
+	}
+
+	if ((io->io_hdr.status & CTL_STATUS_MASK) == CTL_SUCCESS) {
+		int returned_len, used_len;
+
+		returned_len = scsi_4btoul(&dataptr[0]) + 4;
+
+		for (used_len = 0; used_len < returned_len; used_len++) {
+			fprintf(stdout, "0x%02x ", dataptr[used_len]);
+			if (((used_len+1) % 8) == 0)
+				fprintf(stdout, "\n");
+		}
+		fprintf(stdout, "\n");
+	} else
+		ctl_io_error_print(io, NULL, stderr);
+
+bailout:
+	ctl_scsi_free_io(io);
+
+	if (dataptr != NULL)
+		free(dataptr);
+
+	return (retval);
+}
+
+static int
+cctl_inquiry_vpd_devid(int fd, int target, int lun, int initiator)
+{
+	union ctl_io *io;
+	struct ctl_id id;
+	uint32_t datalen;
+	uint8_t *dataptr;
+	int retval;
+
+	id.id = initiator;
+	retval = 0;
+	dataptr = NULL;
+
+	io = ctl_scsi_alloc_io(id);
+	if (io == NULL) {
+		warn("%s: can't allocate memory", __func__);
+		return (1);
+	}
+
+	datalen = 256;
+	dataptr = (uint8_t *)malloc(datalen);
+	if (dataptr == NULL) {
+		warn("%s: can't allocate %d bytes", __func__, datalen);
+	    	retval = 1;
+		goto bailout;
+	}
+
+	memset(dataptr, 0, datalen);
+
+	ctl_scsi_inquiry(/*io*/        io,
+			 /*data_ptr*/  dataptr,
+			 /*data_len*/  datalen,
+			 /*byte2*/     SI_EVPD,
+			 /*page_code*/ SVPD_DEVICE_ID,
+			 /*tag_type*/  CTL_TAG_SIMPLE,
+			 /*control*/   0);
+
+	io->io_hdr.nexus.targ_target.id = target;
+	io->io_hdr.nexus.targ_lun = lun;
+	io->io_hdr.nexus.initid = id;
+
+	if (cctl_do_io(fd, 0, io, __func__) != 0) {
+		retval = 1;
+		goto bailout;
+	}
+
+	if ((io->io_hdr.status & CTL_STATUS_MASK) == CTL_SUCCESS) {
+		int returned_len, used_len;
+
+		returned_len = scsi_2btoul(&dataptr[2]) + 4;
+
+		for (used_len = 0; used_len < returned_len; used_len++) {
+			fprintf(stdout, "0x%02x ", dataptr[used_len]);
+			if (((used_len+1) % 8) == 0)
+				fprintf(stdout, "\n");
+		}
+		fprintf(stdout, "\n");
+	} else
+		ctl_io_error_print(io, NULL, stderr);
+
+bailout:
+	ctl_scsi_free_io(io);
+
+	if (dataptr != NULL)
+		free(dataptr);
+
+	return (retval);
+}
+
+static int
+cctl_persistent_reserve_in(int fd, int target, int lun, int initiator, 
+                           int argc, char **argv, char *combinedopt,
+			   int retry_count)
+{
+	union ctl_io *io;
+	struct ctl_id id;
+	uint32_t datalen;
+	uint8_t *dataptr;
+	int action = -1;
+	int retval;
+	int c;
+
+	id.id = initiator;
+	retval = 0;
+	dataptr = NULL;
+
+	io = ctl_scsi_alloc_io(id);
+	if (io == NULL) {
+		warn("%s: can't allocate memory", __func__);
+		return (1);
+	}
+
+	while ((c = getopt(argc, argv, combinedopt)) != -1) {
+		switch (c) {
+		case 'a':
+			action = strtol(optarg, NULL, 0);
+			break;
+		default:
+			break;
+		}
+	}
+
+	if (action < 0 || action > 2) {
+		warn("action must be specified and in the range: 0-2");
+		retval = 1;
+		goto bailout;
+	}
+
+
+	datalen = 256;
+	dataptr = (uint8_t *)malloc(datalen);
+	if (dataptr == NULL) {
+		warn("%s: can't allocate %d bytes", __func__, datalen);
+	    	retval = 1;
+		goto bailout;
+	}
+
+	memset(dataptr, 0, datalen);
+
+	ctl_scsi_persistent_res_in(io,
+				   /*data_ptr*/ dataptr,
+				   /*data_len*/ datalen,
+				   /*action*/   action,
+				   /*tag_type*/ CTL_TAG_SIMPLE,
+				   /*control*/  0);
+
+	io->io_hdr.nexus.targ_target.id = target;
+	io->io_hdr.nexus.targ_lun = lun;
+	io->io_hdr.nexus.initid = id;
+
+	if (cctl_do_io(fd, retry_count, io, __func__) != 0) {
+		retval = 1;
+		goto bailout;
+	}
+
+	if ((io->io_hdr.status & CTL_STATUS_MASK) == CTL_SUCCESS) {
+		int returned_len, used_len;
+
+		returned_len = 0;
+
+		switch (action) {
+		case 0:
+			returned_len = scsi_4btoul(&dataptr[4]) + 8;
+			returned_len = min(returned_len, 256);
+			break;
+		case 1:
+			returned_len = scsi_4btoul(&dataptr[4]) + 8;
+			break;
+		case 2:
+			returned_len = 8;
+			break;
+		default:
+			warnx("%s: invalid action %d", __func__, action);
+			goto bailout;
+			break; /* NOTREACHED */
+		}
+
+		for (used_len = 0; used_len < returned_len; used_len++) {
+			fprintf(stdout, "0x%02x ", dataptr[used_len]);
+			if (((used_len+1) % 8) == 0)
+				fprintf(stdout, "\n");
+		}
+		fprintf(stdout, "\n");
+	} else
+		ctl_io_error_print(io, NULL, stderr);
+
+bailout:
+	ctl_scsi_free_io(io);
+
+	if (dataptr != NULL)
+		free(dataptr);
+
+	return (retval);
+}
+
+static int
+cctl_persistent_reserve_out(int fd, int target, int lun, int initiator, 
+			    int argc, char **argv, char *combinedopt, 
+			    int retry_count)
+{
+	union ctl_io *io;
+	struct ctl_id id;
+	uint32_t datalen;
+	uint64_t key = 0, sa_key = 0;
+	int action = -1, restype = -1;
+	uint8_t *dataptr;
+	int retval;
+	int c;
+
+	id.id = initiator;
+	retval = 0;
+	dataptr = NULL;
+
+	io = ctl_scsi_alloc_io(id);
+	if (io == NULL) {
+		warn("%s: can't allocate memory", __func__);
+		return (1);
+	}
+
+	while ((c = getopt(argc, argv, combinedopt)) != -1) {
+		switch (c) {
+		case 'a':
+			action = strtol(optarg, NULL, 0);
+			break;
+		case 'k':
+			key = strtoull(optarg, NULL, 0);
+			break;
+		case 'r':
+			restype = strtol(optarg, NULL, 0);
+			break;
+		case 's':
+			sa_key = strtoull(optarg, NULL, 0);
+			break;
+		default:
+			break;
+		}
+	}
+	if (action < 0 || action > 5) {
+		warn("action must be specified and in the range: 0-5");
+		retval = 1;
+		goto bailout;
+	}
+
+	if (restype < 0 || restype > 5) {
+		if (action != 0 && action != 5 && action != 3) {
+			warn("'restype' must specified and in the range: 0-5");
+			retval = 1;
+			goto bailout;
+		}
+	}
+
+	datalen = 24;
+	dataptr = (uint8_t *)malloc(datalen);
+	if (dataptr == NULL) {
+		warn("%s: can't allocate %d bytes", __func__, datalen);
+		retval = 1;
+		goto bailout;
+	}
+
+	memset(dataptr, 0, datalen);
+
+	ctl_scsi_persistent_res_out(io,
+				    /*data_ptr*/ dataptr,
+				    /*data_len*/ datalen,
+				    /*action*/   action,
+				    /*type*/     restype,
+				    /*key*/      key,
+				    /*sa key*/   sa_key,
+				    /*tag_type*/ CTL_TAG_SIMPLE,
+				    /*control*/  0);
+
+	io->io_hdr.nexus.targ_target.id = target;
+	io->io_hdr.nexus.targ_lun = lun;
+	io->io_hdr.nexus.initid = id;
+
+	if (cctl_do_io(fd, retry_count, io, __func__) != 0) {
+		retval = 1;
+		goto bailout;
+	}
+	if ((io->io_hdr.status & CTL_STATUS_MASK) == CTL_SUCCESS) {
+		char scsi_path[40];
+		ctl_scsi_path_string(io, scsi_path, sizeof(scsi_path));
+		fprintf( stdout, "%sPERSISTENT RESERVE OUT executed "
+			"successfully\n", scsi_path);
+	} else
+		ctl_io_error_print(io, NULL, stderr);
+
+bailout:
+	ctl_scsi_free_io(io);
+
+	if (dataptr != NULL)
+		free(dataptr);
+
+	return (retval);
+}
+
+struct cctl_req_option {
+	char			     *name;
+	int			      namelen;
+	char			     *value;
+	int			      vallen;
+	STAILQ_ENTRY(cctl_req_option) links;
+};
+
+static int
+cctl_create_lun(int fd, int argc, char **argv, char *combinedopt)
+{
+	struct ctl_lun_req req;
+	int device_type = -1;
+	uint64_t lun_size = 0;
+	uint32_t blocksize = 0, req_lun_id = 0;
+	char *serial_num = NULL;
+	char *device_id = NULL;
+	int lun_size_set = 0, blocksize_set = 0, lun_id_set = 0;
+	char *backend_name = NULL;
+	STAILQ_HEAD(, cctl_req_option) option_list;
+	int num_options = 0;
+	int retval = 0, c;
+
+	STAILQ_INIT(&option_list);
+
+	while ((c = getopt(argc, argv, combinedopt)) != -1) {
+		switch (c) {
+		case 'b':
+			backend_name = strdup(optarg);
+			break;
+		case 'B':
+			blocksize = strtoul(optarg, NULL, 0);
+			blocksize_set = 1;
+			break;
+		case 'd':
+			device_id = strdup(optarg);
+			break;
+		case 'l':
+			req_lun_id = strtoul(optarg, NULL, 0);
+			lun_id_set = 1;
+			break;
+		case 'o': {
+			struct cctl_req_option *option;
+			char *tmpstr;
+			char *name, *value;
+
+			tmpstr = strdup(optarg);
+			name = strsep(&tmpstr, "=");
+			if (name == NULL) {
+				warnx("%s: option -o takes \"name=value\""
+				      "argument", __func__);
+				retval = 1;
+				goto bailout;
+			}
+			value = strsep(&tmpstr, "=");
+			if (value == NULL) {
+				warnx("%s: option -o takes \"name=value\""
+				      "argument", __func__);
+				retval = 1;
+				goto bailout;
+			}
+			option = malloc(sizeof(*option));
+			if (option == NULL) {
+				warn("%s: error allocating %zd bytes",
+				     __func__, sizeof(*option));
+				retval = 1;
+				goto bailout;
+			}
+			option->name = strdup(name);
+			option->namelen = strlen(name) + 1;
+			option->value = strdup(value);
+			option->vallen = strlen(value) + 1;
+			free(tmpstr);
+
+			STAILQ_INSERT_TAIL(&option_list, option, links);
+			num_options++;
+			break;
+		}
+		case 's':
+			lun_size = strtoull(optarg, NULL, 0);
+			lun_size_set = 1;
+			break;
+		case 'S':
+			serial_num = strdup(optarg);
+			break;
+		case 't':
+			device_type = strtoul(optarg, NULL, 0);
+			break;
+		default:
+			break;
+		}
+	}
+
+	if (backend_name == NULL) {
+		warnx("%s: backend name (-b) must be specified", __func__);
+		retval = 1;
+		goto bailout;
+	}
+
+	bzero(&req, sizeof(req));
+
+	strlcpy(req.backend, backend_name, sizeof(req.backend));
+	req.reqtype = CTL_LUNREQ_CREATE;
+
+	if (blocksize_set != 0)
+		req.reqdata.create.blocksize_bytes = blocksize;
+
+	if (lun_size_set != 0)
+		req.reqdata.create.lun_size_bytes = lun_size;
+
+	if (lun_id_set != 0) {
+		req.reqdata.create.flags |= CTL_LUN_FLAG_ID_REQ;
+		req.reqdata.create.req_lun_id = req_lun_id;
+	}
+
+	req.reqdata.create.flags |= CTL_LUN_FLAG_DEV_TYPE;
+
+	if (device_type != -1)
+		req.reqdata.create.device_type = device_type;
+	else
+		req.reqdata.create.device_type = T_DIRECT;
+
+	if (serial_num != NULL) {
+		strlcpy(req.reqdata.create.serial_num, serial_num,
+			sizeof(req.reqdata.create.serial_num));
+		req.reqdata.create.flags |= CTL_LUN_FLAG_SERIAL_NUM;
+	}
+
+	if (device_id != NULL) {
+		strlcpy(req.reqdata.create.device_id, device_id,
+			sizeof(req.reqdata.create.device_id));
+		req.reqdata.create.flags |= CTL_LUN_FLAG_DEVID;
+	}
+
+	req.num_be_args = num_options;
+	if (num_options > 0) {
+		struct cctl_req_option *option, *next_option;
+		int i;
+
+		req.be_args = malloc(num_options * sizeof(*req.be_args));
+		if (req.be_args == NULL) {
+			warn("%s: error allocating %zd bytes", __func__,
+			     num_options * sizeof(*req.be_args));
+			retval = 1;
+			goto bailout;
+		}
+
+		for (i = 0, option = STAILQ_FIRST(&option_list);
+		     i < num_options; i++, option = next_option) {
+			next_option = STAILQ_NEXT(option, links);
+
+			req.be_args[i].namelen = option->namelen;
+			req.be_args[i].name = strdup(option->name);
+			req.be_args[i].vallen = option->vallen;
+			req.be_args[i].value = strdup(option->value);
+			/*
+			 * XXX KDM do we want a way to specify a writeable
+			 * flag of some sort?  Do we want a way to specify
+			 * binary data?
+			 */
+			req.be_args[i].flags = CTL_BEARG_ASCII | CTL_BEARG_RD;
+
+			STAILQ_REMOVE(&option_list, option, cctl_req_option,
+				      links);
+			free(option->name);
+			free(option->value);
+			free(option);
+		}
+	}
+
+	if (ioctl(fd, CTL_LUN_REQ, &req) == -1) {
+		warn("%s: error issuing CTL_LUN_REQ ioctl", __func__);
+		retval = 1;
+		goto bailout;
+	}
+
+	if (req.status == CTL_LUN_ERROR) {
+		warnx("%s: error returned from LUN creation request:\n%s",
+		      __func__, req.error_str);
+		retval = 1;
+		goto bailout;
+	} else if (req.status != CTL_LUN_OK) {
+		warnx("%s: unknown LUN creation request status %d",
+		      __func__, req.status);
+		retval = 1;
+		goto bailout;
+	}
+
+	fprintf(stdout, "LUN created successfully\n");
+	fprintf(stdout, "backend:       %s\n", req.backend);
+	fprintf(stdout, "device type:   %d\n",req.reqdata.create.device_type);
+	fprintf(stdout, "LUN size:      %ju bytes\n",
+		(uintmax_t)req.reqdata.create.lun_size_bytes);
+	fprintf(stdout, "blocksize      %u bytes\n",
+		req.reqdata.create.blocksize_bytes);
+	fprintf(stdout, "LUN ID:        %d\n", req.reqdata.create.req_lun_id);
+	fprintf(stdout, "Serial Number: %s\n", req.reqdata.create.serial_num);
+	fprintf(stdout, "Device ID;     %s\n", req.reqdata.create.device_id);
+
+bailout:
+	return (retval);
+}
+
+static int
+cctl_rm_lun(int fd, int argc, char **argv, char *combinedopt)
+{
+	struct ctl_lun_req req;
+	uint32_t lun_id = 0;
+	int lun_id_set = 0;
+	char *backend_name = NULL;
+	STAILQ_HEAD(, cctl_req_option) option_list;
+	int num_options = 0;
+	int retval = 0, c;
+
+	STAILQ_INIT(&option_list);
+
+	while ((c = getopt(argc, argv, combinedopt)) != -1) {
+		switch (c) {
+		case 'b':
+			backend_name = strdup(optarg);
+			break;
+		case 'l':
+			lun_id = strtoul(optarg, NULL, 0);
+			lun_id_set = 1;
+			break;
+		case 'o': {
+			struct cctl_req_option *option;
+			char *tmpstr;
+			char *name, *value;
+
+			tmpstr = strdup(optarg);
+			name = strsep(&tmpstr, "=");
+			if (name == NULL) {
+				warnx("%s: option -o takes \"name=value\""
+				      "argument", __func__);
+				retval = 1;
+				goto bailout;
+			}
+			value = strsep(&tmpstr, "=");
+			if (value == NULL) {
+				warnx("%s: option -o takes \"name=value\""
+				      "argument", __func__);
+				retval = 1;
+				goto bailout;
+			}
+			option = malloc(sizeof(*option));
+			if (option == NULL) {
+				warn("%s: error allocating %zd bytes",
+				     __func__, sizeof(*option));
+				retval = 1;
+				goto bailout;
+			}
+			option->name = strdup(name);
+			option->namelen = strlen(name) + 1;
+			option->value = strdup(value);
+			option->vallen = strlen(value) + 1;
+			free(tmpstr);
+
+			STAILQ_INSERT_TAIL(&option_list, option, links);
+			num_options++;
+			break;
+		}
+		default:
+			break;
+		}
+	}
+
+	if (backend_name == NULL)
+		errx(1, "%s: backend name (-b) must be specified", __func__);
+
+	if (lun_id_set == 0)
+		errx(1, "%s: LUN id (-l) must be specified", __func__);
+
+	bzero(&req, sizeof(req));
+
+	strlcpy(req.backend, backend_name, sizeof(req.backend));
+	req.reqtype = CTL_LUNREQ_RM;
+
+	req.reqdata.rm.lun_id = lun_id;
+
+	req.num_be_args = num_options;
+	if (num_options > 0) {
+		struct cctl_req_option *option, *next_option;
+		int i;
+
+		req.be_args = malloc(num_options * sizeof(*req.be_args));
+		if (req.be_args == NULL) {
+			warn("%s: error allocating %zd bytes", __func__,
+			     num_options * sizeof(*req.be_args));
+			retval = 1;
+			goto bailout;
+		}
+
+		for (i = 0, option = STAILQ_FIRST(&option_list);
+		     i < num_options; i++, option = next_option) {
+			next_option = STAILQ_NEXT(option, links);
+
+			req.be_args[i].namelen = option->namelen;
+			req.be_args[i].name = strdup(option->name);
+			req.be_args[i].vallen = option->vallen;
+			req.be_args[i].value = strdup(option->value);
+			/*
+			 * XXX KDM do we want a way to specify a writeable
+			 * flag of some sort?  Do we want a way to specify
+			 * binary data?
+			 */
+			req.be_args[i].flags = CTL_BEARG_ASCII | CTL_BEARG_RD;
+
+			STAILQ_REMOVE(&option_list, option, cctl_req_option,
+				      links);
+			free(option->name);
+			free(option->value);
+			free(option);
+		}
+	}
+
+	if (ioctl(fd, CTL_LUN_REQ, &req) == -1) {
+		warn("%s: error issuing CTL_LUN_REQ ioctl", __func__);
+		retval = 1;
+		goto bailout;
+	}
+
+	if (req.status == CTL_LUN_ERROR) {
+		warnx("%s: error returned from LUN creation request:\n%s",
+		      __func__, req.error_str);
+		retval = 1;
+		goto bailout;
+	} else if (req.status != CTL_LUN_OK) {
+		warnx("%s: unknown LUN creation request status %d",
+		      __func__, req.status);
+		retval = 1;
+		goto bailout;
+	}
+
+	printf("LUN %d deleted successfully\n", lun_id);
+
+bailout:
+	return (retval);
+}
+
+/*
+ * Name/value pair used for per-LUN attributes.
+ */
+struct cctl_lun_nv {
+	char *name;
+	char *value;
+	STAILQ_ENTRY(cctl_lun_nv) links;
+};
+
+/*
+ * Backend LUN information.  
+ */
+struct cctl_lun {
+	uint64_t lun_id;
+	char *backend_type;
+	uint64_t size_blocks;
+	uint32_t blocksize;
+	char *serial_number;
+	char *device_id;
+	STAILQ_HEAD(,cctl_lun_nv) attr_list;
+	STAILQ_ENTRY(cctl_lun) links;
+};
+
+struct cctl_devlist_data {
+	int num_luns;
+	STAILQ_HEAD(,cctl_lun) lun_list;
+	struct cctl_lun *cur_lun;
+	int level;
+	struct sbuf *cur_sb[32];
+};
+
+static void
+cctl_start_element(void *user_data, const char *name, const char **attr)
+{
+	int i;
+	struct cctl_devlist_data *devlist;
+	struct cctl_lun *cur_lun;
+
+	devlist = (struct cctl_devlist_data *)user_data;
+	cur_lun = devlist->cur_lun;
+	devlist->level++;
+	if ((u_int)devlist->level > (sizeof(devlist->cur_sb) /
+	    sizeof(devlist->cur_sb[0])))
+		errx(1, "%s: too many nesting levels, %zd max", __func__,
+		     sizeof(devlist->cur_sb) / sizeof(devlist->cur_sb[0]));
+
+	devlist->cur_sb[devlist->level] = sbuf_new_auto();
+	if (devlist->cur_sb[devlist->level] == NULL)
+		err(1, "%s: Unable to allocate sbuf", __func__);
+
+	if (strcmp(name, "lun") == 0) {
+		if (cur_lun != NULL)
+			errx(1, "%s: improper lun element nesting", __func__);
+
+		cur_lun = calloc(1, sizeof(*cur_lun));
+		if (cur_lun == NULL)
+			err(1, "%s: cannot allocate %zd bytes", __func__,
+			    sizeof(*cur_lun));
+
+		devlist->num_luns++;
+		devlist->cur_lun = cur_lun;
+
+		STAILQ_INIT(&cur_lun->attr_list);
+		STAILQ_INSERT_TAIL(&devlist->lun_list, cur_lun, links);
+
+		for (i = 0; attr[i] != NULL; i += 2) {
+			if (strcmp(attr[i], "id") == 0) {
+				cur_lun->lun_id = strtoull(attr[i+1], NULL, 0);
+			} else {
+				errx(1, "%s: invalid LUN attribute %s = %s",
+				     __func__, attr[i], attr[i+1]);
+			}
+		}
+	}
+}
+
+static void
+cctl_end_element(void *user_data, const char *name)
+{
+	struct cctl_devlist_data *devlist;
+	struct cctl_lun *cur_lun;
+	char *str;
+
+	devlist = (struct cctl_devlist_data *)user_data;
+	cur_lun = devlist->cur_lun;
+
+	if ((cur_lun == NULL)
+	 && (strcmp(name, "ctllunlist") != 0))
+		errx(1, "%s: cur_lun == NULL! (name = %s)", __func__, name);
+
+	if (devlist->cur_sb[devlist->level] == NULL)
+		errx(1, "%s: no valid sbuf at level %d (name %s)", __func__,
+		     devlist->level, name);
+
+	sbuf_finish(devlist->cur_sb[devlist->level]);
+	str = strdup(sbuf_data(devlist->cur_sb[devlist->level]));
+	if (str == NULL)
+		err(1, "%s can't allocate %zd bytes for string", __func__,
+		    sbuf_len(devlist->cur_sb[devlist->level]));
+
+	if (strlen(str) == 0) {
+		free(str);
+		str = NULL;
+	}
+
+	sbuf_delete(devlist->cur_sb[devlist->level]);
+	devlist->cur_sb[devlist->level] = NULL;
+	devlist->level--;
+
+	if (strcmp(name, "backend_type") == 0) {
+		cur_lun->backend_type = str;
+		str = NULL;
+	} else if (strcmp(name, "size") == 0) {
+		cur_lun->size_blocks = strtoull(str, NULL, 0);
+	} else if (strcmp(name, "blocksize") == 0) {
+		cur_lun->blocksize = strtoul(str, NULL, 0);
+	} else if (strcmp(name, "serial_number") == 0) {
+		cur_lun->serial_number = str;
+		str = NULL;
+	} else if (strcmp(name, "device_id") == 0) {
+		cur_lun->device_id = str;
+		str = NULL;
+	} else if (strcmp(name, "lun") == 0) {
+		devlist->cur_lun = NULL;
+	} else if (strcmp(name, "ctllunlist") == 0) {
+		
+	} else {
+		struct cctl_lun_nv *nv;
+
+		nv = calloc(1, sizeof(*nv));
+		if (nv == NULL)
+			err(1, "%s: can't allocate %zd bytes for nv pair",
+			    __func__, sizeof(*nv));
+
+		nv->name = strdup(name);
+		if (nv->name == NULL)
+			err(1, "%s: can't allocated %zd bytes for string",
+			    __func__, strlen(name));
+
+		nv->value = str;
+		str = NULL;
+		STAILQ_INSERT_TAIL(&cur_lun->attr_list, nv, links);
+	}
+
+	free(str);
+}
+
+static void
+cctl_char_handler(void *user_data, const XML_Char *str, int len)
+{
+	struct cctl_devlist_data *devlist;
+
+	devlist = (struct cctl_devlist_data *)user_data;
+
+	sbuf_bcat(devlist->cur_sb[devlist->level], str, len);
+}
+
+static int
+cctl_devlist(int fd, int argc, char **argv, char *combinedopt)
+{
+	struct ctl_lun_list list;
+	struct cctl_devlist_data devlist;
+	struct cctl_lun *lun;
+	XML_Parser parser;
+	char *lun_str;
+	int lun_len;
+	int dump_xml = 0;
+	int retval, c;
+	char *backend = NULL;
+	int verbose = 0;
+
+	retval = 0;
+	lun_len = 4096;
+
+	bzero(&devlist, sizeof(devlist));
+	STAILQ_INIT(&devlist.lun_list);
+
+	while ((c = getopt(argc, argv, combinedopt)) != -1) {
+		switch (c) {
+		case 'b':
+			backend = strdup(optarg);
+			break;
+		case 'v':
+			verbose++;
+			break;
+		case 'x':
+			dump_xml = 1;
+			break;
+		default:
+			break;
+		}
+	}
+
+retry:
+	lun_str = malloc(lun_len);
+
+	bzero(&list, sizeof(list));
+	list.alloc_len = lun_len;
+	list.status = CTL_LUN_LIST_NONE;
+	list.lun_xml = lun_str;
+
+	if (ioctl(fd, CTL_LUN_LIST, &list) == -1) {
+		warn("%s: error issuing CTL_LUN_LIST ioctl", __func__);
+		retval = 1;
+		goto bailout;
+	}
+
+	if (list.status == CTL_LUN_LIST_ERROR) {
+		warnx("%s: error returned from CTL_LUN_LIST ioctl:\n%s",
+		      __func__, list.error_str);
+	} else if (list.status == CTL_LUN_LIST_NEED_MORE_SPACE) {
+		lun_len = lun_len << 1;
+		goto retry;
+	}
+
+	if (dump_xml != 0) {
+		printf("%s", lun_str);
+		goto bailout;
+	}
+
+	parser = XML_ParserCreate(NULL);
+	if (parser == NULL) {
+		warn("%s: Unable to create XML parser", __func__);
+		retval = 1;
+		goto bailout;
+	}
+
+	XML_SetUserData(parser, &devlist);
+	XML_SetElementHandler(parser, cctl_start_element, cctl_end_element);
+	XML_SetCharacterDataHandler(parser, cctl_char_handler);
+
+	retval = XML_Parse(parser, lun_str, strlen(lun_str), 1);
+	XML_ParserFree(parser);
+	if (retval != 1) {
+		retval = 1;
+		goto bailout;
+	}
+
+	printf("LUN Backend  %18s %4s %-16s %-16s\n", "Size (Blocks)", "BS",
+	       "Serial Number", "Device ID");
+	STAILQ_FOREACH(lun, &devlist.lun_list, links) {
+		struct cctl_lun_nv *nv;
+
+		if ((backend != NULL)
+		 && (strcmp(lun->backend_type, backend) != 0))
+			continue;
+
+		printf("%3ju %-8s %18ju %4u %-16s %-16s\n",
+		       (uintmax_t)lun->lun_id,
+		       lun->backend_type, (uintmax_t)lun->size_blocks,
+		       lun->blocksize, lun->serial_number, lun->device_id);
+
+		if (verbose == 0)
+			continue;
+
+		STAILQ_FOREACH(nv, &lun->attr_list, links) {
+			printf("      %s=%s\n", nv->name, nv->value);
+		}
+	}
+bailout:
+	free(lun_str);
+
+	return (retval);
+}
+
+void
+usage(int error)
+{
+	fprintf(error ? stderr : stdout,
+"Usage:\n"
+"Primary commands:\n"
+"         ctladm tur         [dev_id][general options]\n"
+"         ctladm inquiry     [dev_id][general options]\n"
+"         ctladm devid       [dev_id][general options]\n"
+"         ctladm reqsense    [dev_id][general options]\n"
+"         ctladm reportluns  [dev_id][general options]\n"
+"         ctladm read        [dev_id][general options] <-l lba> <-d len>\n"
+"                            <-f file|-> <-b blocksize> [-c cdbsize][-N]\n"
+"         ctladm write       [dev_id][general options] <-l lba> <-d len>\n"
+"                            <-f file|-> <-b blocksize> [-c cdbsize][-N]\n"
+"         ctladm readcap     [dev_id][general options] [-c cdbsize]\n"
+"         ctladm modesense   [dev_id][general options] <-m page|-l> [-P pc]\n"
+"                            [-d] [-S subpage] [-c cdbsize]\n"
+"         ctladm prin        [dev_id][general options] <-a action>\n"
+"         ctladm prout       [dev_id][general options] <-a action>\n"
+"                            <-r restype] [-k key] [-s sa_key]\n"
+"         ctladm rtpg        [dev_id][general options]\n"
+"         ctladm start       [dev_id][general options] [-i] [-o]\n"
+"         ctladm stop        [dev_id][general options] [-i] [-o]\n"
+"         ctladm synccache   [dev_id][general options] [-l lba]\n"
+"                            [-b blockcount] [-r] [-i] [-c cdbsize]\n"
+"         ctladm create      <-b backend> [-B blocksize] [-d device_id]\n"
+"                            [-l lun_id] [-o name=value] [-s size_bytes]\n"
+"                            [-S serial_num] [-t dev_type]\n"
+"         ctladm remove      <-b backend> <-l lun_id> [-o name=value]\n"
+"         ctladm devlist     [-b][-v][-x]\n"
+"         ctladm shutdown\n"
+"         ctladm startup\n"
+"         ctladm hardstop\n"
+"         ctladm hardstart\n"
+"         ctladm lunlist\n"
+"         ctladm bbrread     [dev_id] <-l lba> <-d datalen>\n"
+"         ctladm delay       [dev_id] <-l datamove|done> [-T oneshot|cont]\n"
+"                            [-t secs]\n"
+"         ctladm realsync    <on|off|query>\n"
+"         ctladm setsync     [dev_id] <-i interval>\n"
+"         ctladm getsync     [dev_id]\n"
+"         ctladm inject      [dev_id] <-i action> <-p pattern> [-r lba,len]\n"
+"                            [-s len fmt [args]] [-c] [-d delete_id]\n"
+"         ctladm port        <-l | -o <on|off> | [-w wwnn][-W wwpn]>\n"
+"                            [-p targ_port] [-t port_type] [-q] [-x]\n"
+"         ctladm dumpooa\n"
+"         ctladm dumpstructs\n"
+"         ctladm help\n"
+"General Options:\n"
+"-I intiator_id           : defaults to 7, used to change the initiator id\n"
+"-C retries               : specify the number of times to retry this command\n"
+"-D devicename            : specify the device to operate on\n"
+"                         : (default is %s)\n"
+"read/write options:\n"
+"-l lba                   : logical block address\n"
+"-d len                   : read/write length, in blocks\n"
+"-f file|-                : write/read data to/from file or stdout/stdin\n"
+"-b blocksize             : block size, in bytes\n"
+"-c cdbsize               : specify minimum cdb size: 6, 10, 12 or 16\n"
+"-N                       : do not copy data to/from userland\n"
+"readcapacity options:\n"
+"-c cdbsize               : specify minimum cdb size: 10 or 16\n"
+"modesense options:\n"
+"-m page                  : specify the mode page to view\n"
+"-l                       : request a list of supported pages\n"
+"-P pc                    : specify the page control value: 0-3 (current,\n"
+"                           changeable, default, saved, respectively)\n"
+"-d                       : disable block descriptors for mode sense\n"
+"-S subpage               : specify a subpage\n"
+"-c cdbsize               : specify minimum cdb size: 6 or 10\n"
+"persistent reserve in options:\n"
+"-a action                : specify the action value: 0-2 (read key, read\n"
+"                           reservation, read capabilities, respectively)\n"
+"persistent reserve out options:\n"
+"-a action                : specify the action value: 0-5 (register, reserve,\n"
+"                           release, clear, preempt, register and ignore)\n"
+"-k key                   : key value\n"
+"-s sa_key                : service action value\n"
+"-r restype               : specify the reservation type: 0-5(wr ex, ex ac,\n"
+"                           wr ex ro, ex ac ro, wr ex ar, ex ac ar)\n"
+"start/stop options:\n"
+"-i                       : set the immediate bit (CTL does not support this)\n"
+"-o                       : set the on/offline bit\n"
+"synccache options:\n"
+"-l lba                   : set the starting LBA\n"
+"-b blockcount            : set the length to sync in blocks\n"
+"-r                       : set the relative addressing bit\n"
+"-i                       : set the immediate bit\n"
+"-c cdbsize               : specify minimum cdb size: 10 or 16\n"
+"create options:\n"
+"-b backend               : backend name (\"block\", \"ramdisk\", etc.)\n"
+"-B blocksize             : LUN blocksize in bytes (some backends)\n"
+"-d device_id             : SCSI VPD page 0x83 ID\n"
+"-l lun_id                : requested LUN number\n"
+"-o name=value            : backend-specific options, multiple allowed\n"
+"-s size_bytes            : LUN size in bytes (some backends)\n"
+"-S serial_num            : SCSI VPD page 0x80 serial number\n"
+"-t dev_type              : SCSI device type (0=disk, 3=processor)\n"
+"remove options:\n"
+"-b backend               : backend name (\"block\", \"ramdisk\", etc.)\n"
+"-l lun_id                : LUN number to delete\n"
+"-o name=value            : backend-specific options, multiple allowed\n"
+"devlist options:\n"
+"-b backend               : list devices from specified backend only\n"
+"-v                       : be verbose, show backend attributes\n"
+"-x                       : dump raw XML\n"
+"delay options:\n"
+"-l datamove|done         : delay command at datamove or done phase\n"
+"-T oneshot               : delay one command, then resume normal completion\n"
+"-T cont                  : delay all commands\n"
+"-t secs                  : number of seconds to delay\n"
+"inject options:\n"
+"-i error_action          : action to perform\n"
+"-p pattern               : command pattern to look for\n"
+"-r lba,len               : LBA range for pattern\n"
+"-s len fmt [args]        : sense data for custom sense action\n"
+"-c                       : continuous operation\n"
+"-d delete_id             : error id to delete\n"
+"port options:\n"
+"-l                       : list frontend ports\n"
+"-o on|off                : turn frontend ports on or off\n"
+"-w wwnn                  : set WWNN for one frontend\n"
+"-W wwpn                  : set WWPN for one frontend\n"
+"-t port_type             : specify fc, scsi, ioctl, internal frontend type\n"
+"-p targ_port             : specify target port number\n"
+"-q                       : omit header in list output\n"
+"-x                       : output port list in XML format\n"
+"bbrread options:\n"
+"-l lba                   : starting LBA\n"
+"-d datalen               : length, in bytes, to read\n",
+CTL_DEFAULT_DEV);
+}
+
+int
+main(int argc, char **argv)
+{
+	int option_index, c;
+	ctladm_cmdfunction command;
+	ctladm_cmdargs cmdargs;
+	ctladm_optret optreturn;
+	char *device;
+	const char *mainopt = "C:D:I:";
+	const char *subopt = NULL;
+	char combinedopt[256];
+	int target, lun;
+	int optstart = 2;
+	int retval, fd;
+	int retries, timeout;
+	int initid;
+
+	option_index = 0;
+	retval = 0;
+	cmdargs = CTLADM_ARG_NONE;
+	command = CTLADM_CMD_HELP;
+	device = NULL;
+	fd = -1;
+	retries = 0;
+	target = 0;
+	lun = 0;
+	timeout = 0;
+	initid = 7;
+
+	if (argc < 2) {
+		usage(1);
+		retval = 1;
+		goto bailout;
+	}
+
+	/*
+	 * Get the base option.
+	 */
+	optreturn = getoption(option_table,argv[1], &command, &cmdargs,&subopt);
+
+	if (optreturn == CC_OR_AMBIGUOUS) {
+		warnx("ambiguous option %s", argv[1]);
+		usage(0);
+		exit(1);
+	} else if (optreturn == CC_OR_NOT_FOUND) {
+		warnx("option %s not found", argv[1]);
+		usage(0);
+		exit(1);
+	}
+
+	if (cmdargs & CTLADM_ARG_NEED_TL) {
+	 	if ((argc < 3)
+		 || (!isdigit(argv[2][0]))) {
+			warnx("option %s requires a target:lun argument",
+			      argv[1]);
+			usage(0);
+			exit(1);
+		}
+		retval = cctl_parse_tl(argv[2], &target, &lun);
+		if (retval != 0)
+			errx(1, "invalid target:lun argument %s", argv[2]);
+
+		cmdargs |= CTLADM_ARG_TARG_LUN;
+		optstart++;
+	}
+
+	/*
+	 * Ahh, getopt(3) is a pain.
+	 *
+	 * This is a gross hack.  There really aren't many other good
+	 * options (excuse the pun) for parsing options in a situation like
+	 * this.  getopt is kinda braindead, so you end up having to run
+	 * through the options twice, and give each invocation of getopt
+	 * the option string for the other invocation.
+	 *
+	 * You would think that you could just have two groups of options.
+	 * The first group would get parsed by the first invocation of
+	 * getopt, and the second group would get parsed by the second
+	 * invocation of getopt.  It doesn't quite work out that way.  When
+	 * the first invocation of getopt finishes, it leaves optind pointing
+	 * to the argument _after_ the first argument in the second group.
+	 * So when the second invocation of getopt comes around, it doesn't
+	 * recognize the first argument it gets and then bails out.
+	 *
+	 * A nice alternative would be to have a flag for getopt that says
+	 * "just keep parsing arguments even when you encounter an unknown
+	 * argument", but there isn't one.  So there's no real clean way to
+	 * easily parse two sets of arguments without having one invocation
+	 * of getopt know about the other.
+	 *
+	 * Without this hack, the first invocation of getopt would work as
+	 * long as the generic arguments are first, but the second invocation
+	 * (in the subfunction) would fail in one of two ways.  In the case
+	 * where you don't set optreset, it would fail because optind may be
+	 * pointing to the argument after the one it should be pointing at.
+	 * In the case where you do set optreset, and reset optind, it would
+	 * fail because getopt would run into the first set of options, which
+	 * it doesn't understand.
+	 *
+	 * All of this would "sort of" work if you could somehow figure out
+	 * whether optind had been incremented one option too far.  The
+	 * mechanics of that, however, are more daunting than just giving
+	 * both invocations all of the expect options for either invocation.
+	 *
+	 * Needless to say, I wouldn't mind if someone invented a better
+	 * (non-GPL!) command line parsing interface than getopt.  I
+	 * wouldn't mind if someone added more knobs to getopt to make it
+	 * work better.  Who knows, I may talk myself into doing it someday,
+	 * if the standards weenies let me.  As it is, it just leads to
+	 * hackery like this and causes people to avoid it in some cases.
+	 *
+	 * KDM, September 8th, 1998
+	 */
+	if (subopt != NULL)
+		sprintf(combinedopt, "%s%s", mainopt, subopt);
+	else
+		sprintf(combinedopt, "%s", mainopt);
+
+	/*
+	 * Start getopt processing at argv[2/3], since we've already
+	 * accepted argv[1..2] as the command name, and as a possible
+	 * device name.
+	 */
+	optind = optstart;
+
+	/*
+	 * Now we run through the argument list looking for generic
+	 * options, and ignoring options that possibly belong to
+	 * subfunctions.
+	 */
+	while ((c = getopt(argc, argv, combinedopt))!= -1){
+		switch (c) {
+		case 'C':
+			cmdargs |= CTLADM_ARG_RETRIES;
+			retries = strtol(optarg, NULL, 0);
+			break;
+		case 'D':
+			device = strdup(optarg);
+			cmdargs |= CTLADM_ARG_DEVICE;
+			break;
+		case 'I':
+			cmdargs |= CTLADM_ARG_INITIATOR;
+			initid = strtol(optarg, NULL, 0);
+			break;
+		default:
+			break;
+		}
+	}
+
+	if ((cmdargs & CTLADM_ARG_INITIATOR) == 0)
+		initid = 7;
+
+	optind = optstart;
+	optreset = 1;
+
+	/*
+	 * Default to opening the CTL device for now.
+	 */
+	if (((cmdargs & CTLADM_ARG_DEVICE) == 0)
+	 && (command != CTLADM_CMD_HELP)) {
+		device = strdup(CTL_DEFAULT_DEV);
+		cmdargs |= CTLADM_ARG_DEVICE;
+	}
+
+	if ((cmdargs & CTLADM_ARG_DEVICE)
+	 && (command != CTLADM_CMD_HELP)) {
+		fd = open(device, O_RDWR);
+		if (fd == -1) {
+			fprintf(stderr, "%s: error opening %s: %s\n",
+				argv[0], device, strerror(errno));
+			retval = 1;
+			goto bailout;
+		}
+	} else if ((command != CTLADM_CMD_HELP)
+		&& ((cmdargs & CTLADM_ARG_DEVICE) == 0)) {
+		fprintf(stderr, "%s: you must specify a device with the "
+			"--device argument for this command\n", argv[0]);
+		command = CTLADM_CMD_HELP;
+		retval = 1;
+	}
+
+	switch (command) {
+	case CTLADM_CMD_TUR:
+		retval = cctl_tur(fd, target, lun, initid, retries);
+		break;
+	case CTLADM_CMD_INQUIRY:
+		retval = cctl_inquiry(fd, target, lun, initid, retries);
+		break;
+	case CTLADM_CMD_REQ_SENSE:
+		retval = cctl_req_sense(fd, target, lun, initid, retries);
+		break;
+	case CTLADM_CMD_REPORT_LUNS:
+		retval = cctl_report_luns(fd, target, lun, initid, retries);
+		break;
+	case CTLADM_CMD_CREATE:
+		retval = cctl_create_lun(fd, argc, argv, combinedopt);
+		break;
+	case CTLADM_CMD_RM:
+		retval = cctl_rm_lun(fd, argc, argv, combinedopt);
+		break;
+	case CTLADM_CMD_DEVLIST:
+		retval = cctl_devlist(fd, argc, argv, combinedopt);
+		break;
+	case CTLADM_CMD_READ:
+	case CTLADM_CMD_WRITE:
+		retval = cctl_read_write(fd, target, lun, initid, retries,
+					 argc, argv, combinedopt, command);
+		break;
+	case CTLADM_CMD_PORT:
+		retval = cctl_port(fd, argc, argv, combinedopt);
+		break;
+	case CTLADM_CMD_READCAPACITY:
+		retval = cctl_read_capacity(fd, target, lun, initid, retries,
+					    argc, argv, combinedopt);
+		break;
+	case CTLADM_CMD_MODESENSE:
+		retval = cctl_mode_sense(fd, target, lun, initid, retries,
+					 argc, argv, combinedopt);
+		break;
+	case CTLADM_CMD_START:
+	case CTLADM_CMD_STOP:
+		retval = cctl_start_stop(fd, target, lun, initid, retries,
+					 (command == CTLADM_CMD_START) ? 1 : 0,
+					 argc, argv, combinedopt);
+		break;
+	case CTLADM_CMD_SYNC_CACHE:
+		retval = cctl_sync_cache(fd, target, lun, initid, retries,
+					 argc, argv, combinedopt);
+		break;
+	case CTLADM_CMD_SHUTDOWN:
+	case CTLADM_CMD_STARTUP:
+		retval = cctl_startup_shutdown(fd, target, lun, initid,
+					       command);
+		break;
+	case CTLADM_CMD_HARDSTOP:
+	case CTLADM_CMD_HARDSTART:
+		retval = cctl_hardstopstart(fd, command);
+		break;
+	case CTLADM_CMD_BBRREAD:
+		retval = cctl_bbrread(fd, target, lun, initid, argc, argv,
+				      combinedopt);
+		break;
+	case CTLADM_CMD_LUNLIST:
+		retval = cctl_lunlist(fd);
+		break;
+	case CTLADM_CMD_DELAY:
+		retval = cctl_delay(fd, target, lun, argc, argv, combinedopt);
+		break;
+	case CTLADM_CMD_REALSYNC:
+		retval = cctl_realsync(fd, argc, argv);
+		break;
+	case CTLADM_CMD_SETSYNC:
+	case CTLADM_CMD_GETSYNC:
+		retval = cctl_getsetsync(fd, target, lun, command,
+					 argc, argv, combinedopt);
+		break;
+	case CTLADM_CMD_ERR_INJECT:
+		retval = cctl_error_inject(fd, target, lun, argc, argv,
+					   combinedopt);
+		break;
+	case CTLADM_CMD_DUMPOOA:
+		retval = cctl_dump_ooa(fd, argc, argv);
+		break;
+	case CTLADM_CMD_DUMPSTRUCTS:
+		retval = cctl_dump_structs(fd, cmdargs);
+		break;
+	case CTLADM_CMD_PRES_IN:
+		retval = cctl_persistent_reserve_in(fd, target, lun, initid, 
+		                                    argc, argv, combinedopt,
+						    retries);
+		break;
+	case CTLADM_CMD_PRES_OUT:
+		retval = cctl_persistent_reserve_out(fd, target, lun, initid, 
+						     argc, argv, combinedopt,
+						     retries);
+		break;
+	case CTLADM_CMD_INQ_VPD_DEVID:
+	        retval = cctl_inquiry_vpd_devid(fd, target, lun, initid);
+		break;
+	case CTLADM_CMD_RTPG:
+	        retval = cctl_report_target_port_group(fd, target, lun, initid);
+		break;
+	case CTLADM_CMD_HELP:
+	default:
+		usage(retval);
+		break;
+	}
+bailout:
+
+	if (fd != -1)
+		close(fd);
+
+	exit (retval);
+}
+
+/*
+ * vim: ts=8
+ */
diff --git a/usr.sbin/ctladm/ctladm.h b/usr.sbin/ctladm/ctladm.h
new file mode 100644
index 0000000..b2e9f12
--- /dev/null
+++ b/usr.sbin/ctladm/ctladm.h
@@ -0,0 +1,50 @@
+/*
+ * Copyright (c) 1998 Kenneth D. Merry.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ * 3. The name of the author may not be used to endorse or promote products
+ *    derived from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * $FreeBSD$
+ * $Id: //depot/users/kenm/FreeBSD-test2/usr.sbin/ctladm/ctladm.h#1 $
+ */
+
+#ifndef _CTLADM_H
+#define _CTLADM_H
+
+/*
+ * get_hook: Structure for evaluating args in a callback.
+ */
+struct get_hook
+{
+	int argc;
+	char **argv;
+	int got;
+};
+
+char *cget(void *hook, char *name);
+int iget(void *hook, char *name);
+void arg_put(void *hook, int letter, void *arg, int count, char *name);
+void usage(int error);
+
+#endif /* _CTLADM_H */
diff --git a/usr.sbin/ctladm/util.c b/usr.sbin/ctladm/util.c
new file mode 100644
index 0000000..df13357
--- /dev/null
+++ b/usr.sbin/ctladm/util.c
@@ -0,0 +1,156 @@
+/*
+ * Written By Julian ELischer
+ * Copyright julian Elischer 1993.
+ * Permission is granted to use or redistribute this file in any way as long
+ * as this notice remains. Julian Elischer does not guarantee that this file
+ * is totally correct for any given task and users of this file must
+ * accept responsibility for any damage that occurs from the application of this
+ * file.
+ *
+ * (julian@tfs.com julian@dialix.oz.au)
+ *
+ * User SCSI hooks added by Peter Dufault:
+ *
+ * Copyright (c) 1994 HD Associates
+ * (contact: dufault@hda.com)
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ * 3. The name of HD Associates
+ *    may not be used to endorse or promote products derived from this software
+ *    without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY HD ASSOCIATES ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL HD ASSOCIATES BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+/*
+ * Taken from the original scsi(8) program.
+ * from: scsi.c,v 1.17 1998/01/12 07:57:57 charnier Exp $";
+ */
+#include <sys/cdefs.h>
+__FBSDID("$FreeBSD$");
+
+#include <sys/stdint.h>
+#include <sys/types.h>
+
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+
+#include <camlib.h>
+#include "ctladm.h"
+
+int verbose;
+
+/* iget: Integer argument callback
+ */
+int
+iget(void *hook, char *name)
+{
+	struct get_hook *h = (struct get_hook *)hook;
+	int arg;
+
+	if (h->got >= h->argc)
+	{
+		fprintf(stderr, "Expecting an integer argument.\n");
+		usage(0);
+		exit(1);
+	}
+	arg = strtol(h->argv[h->got], 0, 0);
+	h->got++;
+
+	if (verbose && name && *name)
+		printf("%s: %d\n", name, arg);
+
+	return arg;
+}
+
+/* cget: char * argument callback
+ */
+char *
+cget(void *hook, char *name)
+{
+	struct get_hook *h = (struct get_hook *)hook;
+	char *arg;
+
+	if (h->got >= h->argc)
+	{
+		fprintf(stderr, "Expecting a character pointer argument.\n");
+		usage(0);
+		exit(1);
+	}
+	arg = h->argv[h->got];
+	h->got++;
+
+	if (verbose && name)
+		printf("cget: %s: %s", name, arg);
+
+	return arg;
+}
+
+/* arg_put: "put argument" callback
+ */
+void
+arg_put(void *hook __unused, int letter, void *arg, int count, char *name)
+{
+	if (verbose && name && *name)
+		printf("%s:  ", name);
+
+	switch(letter)
+	{
+		case 'i':
+		case 'b':
+		printf("%jd ", (intmax_t)(intptr_t)arg);
+		break;
+
+		case 'c':
+		case 'z':
+		{
+			char *p;
+
+			p = malloc(count + 1);
+			if (p == NULL) {
+				fprintf(stderr, "can't malloc memory for p\n");
+				exit(1);
+			}
+
+			bzero(p, count +1);
+			strncpy(p, (char *)arg, count);
+			if (letter == 'z')
+			{
+				int i;
+				for (i = count - 1; i >= 0; i--)
+					if (p[i] == ' ')
+						p[i] = 0;
+					else
+						break;
+			}
+			printf("%s ", p);
+
+			free(p);
+		}
+
+		break;
+
+		default:
+		printf("Unknown format letter: '%c'\n", letter);
+	}
+	if (verbose)
+		putchar('\n');
+}