Import the gvinum work that have been done during and after Summer of Code 2007.

The work have been under testing and fixing since then, and it is mature enough
to be put into HEAD for further testing.

A lot have changed in this time, and here are the most important:
- Gvinum now uses one single workerthread instead of one thread for each
  volume and each plex. The reason for this is that the previous scheme was
  very complex, and was the cause of many of the bugs discovered in gvinum.
  Instead, gvinum now uses one worker thread with an event queue, quite
  similar to what used in gmirror.
- The rebuild/grow/initialize/parity check routines no longer runs in
  separate threads, but are run as regular I/O requests with special flags.
  This made it easier to support mounted growing and parity rebuild.
- Support for growing striped and raid5-plexes, meaning that one can extend the
  volumes for these plex types in addition to the concat type. Also works while
  the volume is mounted.
- Implementation of many of the missing commands from the old vinum:
  attach/detach, start (was partially implemented), stop (was partially
  implemented), concat, mirror, stripe, raid5 (shortcuts for creating volumes
  with one plex of these organizations).
- The parity check and rebuild no longer goes between userland/kernel, meaning
  that the gvinum command will not stay and wait forever for the rebuild to
  finish. You can instead watch the status with the list command.
- Many problems with gvinum have been reported since 5.x, and some has been hard
  to fix due to the complicated architecture. Hopefully, it should be more
  stable and better handle edge cases that previously made gvinum crash.
- Failed drives no longer disappears entirely, but now leave behind a dummy
  drive that makes sure the original state is not forgotten in case the system
  is rebooted between drive failures/swaps.
- Update manpage to reflect new commands and extend it with some examples.

Sponsored by:   Google Summer of Code 2007
Mentored by:    le
Tested by:      Rick C. Petty <rick-freebsd2008 -at- kiwi-computer.com>

2 files changed, 605 insertions, 99 deletions

diff --git a/sbin/gvinum/gvinum.8 b/sbin/gvinum/gvinum.8
index 83b6876..d2b9616 100644
--- a/sbin/gvinum/gvinum.8
+++ b/sbin/gvinum/gvinum.8
@@ -40,6 +40,13 @@
 .Op Fl options
 .Sh COMMANDS
 .Bl -tag -width indent
+.It Ic attach Ar plex volume Op Cm rename
+.It Ic attach Ar subdisk plex Oo Ar offset Oc Op Cm rename
+Attach a plex to a volume, or a subdisk to a plex. 
+If offset is specified, the subdisk will be attached to the given offset within
+the plex.
+If rename is specified, the subdisk or plex will change name according to the
+object it attaches to.
 .It Ic checkparity Oo Fl f Oc Ar plex
 Check the parity blocks of a RAID-5 plex.
 The parity check will start at the
@@ -49,7 +56,10 @@ flag is specified, or otherwise at the location of the parity check pointer,
 the first location at which plex's parity is incorrect.
 All subdisks in the
 plex must be up for a parity check.
-.It Ic create Op Ar description-file
+.It Ic concat Oo Fl fv Oc Oo Fl n Ar name Oc Ar drives
+Create a concatenated volume from the specified drives.
+If no name is specified, a unique name will be set by gvinum.
+.It Ic create Oo Fl f Oc Op Ar description-file
 Create a volume as described in
 .Ar description-file .
 If no
@@ -57,6 +67,18 @@ If no
 provided, opens an editor and provides the current
 .Nm
 configuration for editing.
+The
+.Fl f
+flag will make gvinum ignore any errors regarding creating objects that already
+exists.
+However, in contrast to vinum, objects that are not properly named in the
+.Ar description-file
+will not be created when the
+.Fl f
+flag is given.
+.It Ic detach Oo Fl f Oc Op Ar plex | subdisk
+Detach a plex or subdisk from the volume or plex to which it is
+attached.
 .It Ic help
 Provides a synopsis of
 .Nm
@@ -76,6 +98,14 @@ The
 and
 .Fl V
 flags provide progressively more detailed output.
+.It Ic mirror Oo Fl fsv Oc Oo Fl n Ar name Oc Ar drives
+Create a mirrored volume from the specified drives. 
+It requires at least a multiple of 2 drives.
+If no name is specified, a unique name will be set by gvinum.
+If the 
+.Fl s
+flag is specified, a striped mirror will be created, and thus requires a
+multiple of 4 drives.
 .It Ic move | mv Fl f Ar drive subdisk Op Ar ...
 Move the subdisk(s) to the specified drive.
 The
@@ -85,12 +115,19 @@ part of the move.
 This can currently only be done when the subdisk is
 not being accessed.
 .Pp
-If the subdisk(s) form part of a RAID-5 plex, the disk(s) will need to be set
-to the
+If a single subdisk is moved, and it forms a part of a RAID-5 plex, the moved
+subdisks will need to be set to the
+.Dq stale
+state, and the plex will require a
+.Ic start
+command.
+If multiple subdisk(s) is moved, and form part of a RAID-5 plex, the
+moved disk(s) will need to be set to the
 .Dq up
 state and the plex will require a
 .Ic rebuildparity
-command; if the subdisk(s) form part of a plex that is mirrored with other
+command.
+If the subdisk(s) form part of a plex that is mirrored with other
 plexes, the plex will require restarting and will sync once restarted.
 Moving
 more than one subdisk in a RAID-5 plex or subdisks from both sides of a
@@ -105,6 +142,11 @@ Exit
 when running in interactive mode.
 Normally this would be done by entering the
 EOF character.
+.It Ic raid5 Oo Fl fv Oc Oo Fl s Ar stripesize Oc Oo Fl n Ar name Oc Ar drives
+Create a RAID-5 volume from the specified drives.
+If no name is specified,a unique name will be set by
+.Ic gvinum. 
+This organization requires at least three drives.
 .It Ic rename Oo Fl r Oc Ar drive | subdisk | plex | volume newname
 Change the name of the specified object.
 The
@@ -143,9 +185,21 @@ flag forces state changes regardless of whether they are legal.
 Read configuration from all vinum drives.
 .It Ic start Oo Fl S Ar size Oc Ar volume | plex | subdisk
 Allow the system to access the objects.
+If necessary, plexes will be synced and rebuilt.
+If a subdisk was added to a running RAID-5 or striped plex, gvinum will
+expand into this subdisk and grow the whole RAID-5 array.
+This can be done without unmounting your filesystem.
 The
 .Fl S
 flag is currently ignored.
+.It Ic stop Oo Fl f Oc Op Ar volume | plex | subdisk
+Terminate access to the objects, or stop
+.Nm
+if no parameters are specified.
+.It Ic stripe Oo Fl fv Oc Oo Fl n Ar name Oc Ar drives
+Create a striped volume from the specified drives. If no name is specified,
+a unique name will be set by Ic gvinum. This organization requires at least two
+drives.
 .El
 .Sh DESCRIPTION
 The
@@ -217,15 +271,90 @@ is invoked.
 directory with device nodes for
 .Nm
 objects
-.It Pa /dev/gvinum/plex
-directory containing device nodes for
-.Nm
-plexes
-.It Pa /dev/gvinum/sd
-directory containing device nodes for
-.Nm
-subdisks
 .El
+.Sh EXAMPLES
+To create a mirror on disks /dev/ad1 and /dev/ad2, create a filesystem, mount,
+unmount and then stop Ic gvinum:
+.Pp
+.Dl "gvinum mirror /dev/ad1 /dev/ad2"
+.Dl "newfs /dev/gvinum/gvinumvolume0"
+.Dl "mount /dev/gvinum/gvinumvolume0 /mnt" 
+.Dl "..."
+.Dl "unmount /mnt"
+.Dl "gvinum stop"
+.Pp
+To create a striped mirror on disks /dev/ad1 /dev/ad2 /dev/ad3 and /dev/ad4
+named "data" and create a filesystem:
+.Pp
+.Dl "gvinum mirror -s -n data /dev/ad1 /dev/ad2 /dev/ad3 /dev/ad4"
+.Dl "newfs /dev/gvinum/data"
+.Pp
+To create a raid5 array on disks /dev/ad1 /dev/ad2 and /dev/ad3, with stripesize
+493k you can use the raid5 command:
+.Pp
+.Dl "gvinum raid5 -s 493k /dev/ad1 /dev/ad2 /dev/ad3"
+.Pp
+Then the volume will be created automatically.
+Afterwards, you have to initialize the volume:
+.Pp
+.Dl "gvinum start myraid5vol"
+.Pp
+The initialization will start, and the states will be updated when it's
+finished.
+The list command will give you information about its progress.
+.Pp
+Imagine that one of the drives fails, and the output of 'printconfig' looks
+something like this:
+.Pp
+.Dl "drive gvinumdrive1 device /dev/ad2"
+.Dl "drive gvinumdrive2 device /dev/???"
+.Dl "drive gvinumdrive0 device /dev/ad1"
+.Dl "volume myraid5vol"
+.Dl "plex name myraid5vol.p0 org raid5 986s vol myraid5vol"
+.Dl "sd name myraid5vol.p0.s2 drive gvinumdrive2 len 32538s driveoffset 265s"
+.Dl "plex myraid5vol.p0 plexoffset 1972s"
+.Dl "sd name myraid5vol.p0.s1 drive gvinumdrive1 len 32538s driveoffset 265s"
+.Dl "plex myraid5vol.p0 plexoffset 986s"
+.Dl "sd name myraid5vol.p0.s0 drive gvinumdrive0 len 32538s driveoffset 265s"
+.Dl "plex myraid5vol.p0 plexoffset 0s"
+.Pp
+Create a new drive with this configuration:
+.Pp
+.Dl "drive gdrive4 device /dev/ad4"
+.Pp
+Then move the stale subdisk to the new drive:
+.Pp
+.Dl "gvinum move gdrive4 myraid5vol.p0.s2"
+.Pp
+Then, initiate the rebuild:
+.Pp
+.Dl "gvinum start myraid5vol.p0"
+.Pp
+The plex will go up form degraded mode after the rebuild is finished.
+The plex can still be used while the rebuild is in progress, although requests
+might be delayed.
+For a more advanced usage and detailed explanation of gvinum, the
+handbook is recommended.
+.Pp
+Given the configuration as in the previous example, growing a RAID-5 or STRIPED
+array is accomplished by adding a new subdisk to the plex with a
+.Ar description-file
+similar to this:
+.Pp
+.Dl "drive newdrive device /dev/ad4"
+.Dl "sd drive newdrive plex myraid5vol.p0"
+.Pp
+If everything went ok, the plex state should now be set to growable.
+You can then start the growing with the
+.Ic start
+command:
+.Pp
+.Dl "gvinum start myraid5vol.p0"
+.Pp
+As with rebuilding, you can watch the progress using the
+.Ic list
+command.
+.Pp
 .Sh SEE ALSO
 .Xr geom 4 ,
 .Xr geom 8
@@ -255,9 +384,13 @@ documentation were added by
 .An "Chris Jones"
 through the 2005 Google Summer
 of Code program.
+.Ic a partial rewrite of gvinum was done by "Lukas Ertl" and "Ulf Lilleengen"
+through the 2007 Google Summer of Code program.
+The documentation have been updated to reflect the new functionality.
 .Sh AUTHORS
 .An Lukas Ertl Aq le@FreeBSD.org
 .An Chris Jones Aq soc-cjones@FreeBSD.org
+.An Ulf Lilleengen Aq lulf@FreeBSD.org
 .Sh BUGS
 Currently,
 .Nm
@@ -271,10 +404,6 @@ initsize flag to
 .Ic start
 is ignored.
 .Pp
-The
-.Ic stop
-command does not work.
-.Pp
 Moving subdisks that are not part of a mirrored or RAID-5 volume will
 destroy data.
 It is perhaps a bug to permit this.
@@ -291,18 +420,10 @@ Specifically, the following commands from
 .Xr vinum 4
 are not supported:
 .Bl -tag -width indent
-.It Ic attach Ar plex volume Op Cm rename
-.It Ic attach Ar subdisk plex Oo Ar offset Oc Op Cm rename
-Attach a plex to a volume, or a subdisk to a plex.
-.It Ic concat Oo Fl fv Oc Oo Fl n Ar name Oc Ar drives
-Create a concatenated volume from the specified drives.
 .It Ic debug
 Cause the volume manager to enter the kernel debugger.
 .It Ic debug Ar flags
 Set debugging flags.
-.It Ic detach Oo Fl f Oc Op Ar plex | subdisk
-Detach a plex or subdisk from the volume or plex to which it is
-attached.
 .It Ic dumpconfig Op Ar drive ...
 List the configuration information stored on the specified drives, or all
 drives in the system if no drive names are specified.
@@ -310,17 +431,9 @@ drives in the system if no drive names are specified.
 List information about volume manager state.
 .It Ic label Ar volume
 Create a volume label.
-.It Ic mirror Oo Fl fsv Oc Oo Fl n Ar name Oc Ar drives
-Create a mirrored volume from the specified drives.
 .It Ic resetstats Oo Fl r Oc Op Ar volume | plex | subdisk
 Reset statistics counters for the specified objects, or for all objects if none
 are specified.
 .It Ic setdaemon Op Ar value
 Set daemon configuration.
-.It Ic stop Oo Fl f Oc Op Ar volume | plex | subdisk
-Terminate access to the objects, or stop
-.Nm
-if no parameters are specified.
-.It Ic stripe Oo Fl fv Oc Oo Fl n Ar name Oc Ar drives
-Create a striped volume from the specified drives.
 .El
diff --git a/sbin/gvinum/gvinum.c b/sbin/gvinum/gvinum.c
index ea33b75..bea4f7c 100644
--- a/sbin/gvinum/gvinum.c
+++ b/sbin/gvinum/gvinum.c
@@ -1,5 +1,7 @@
 /*
- *  Copyright (c) 2004 Lukas Ertl, 2005 Chris Jones
+ *  Copyright (c) 2004 Lukas Ertl
+ *  Copyright (c) 2005 Chris Jones
+ *  Copyright (c) 2007 Ulf Lilleengen
  *  All rights reserved.
  *
  * Portions of this software were developed for the FreeBSD Project
@@ -43,6 +45,7 @@
 
 #include <ctype.h>
 #include <err.h>
+#include <errno.h>
 #include <libgeom.h>
 #include <stdint.h>
 #include <stdio.h>
@@ -54,12 +57,17 @@
 
 #include "gvinum.h"
 
+void	gvinum_attach(int, char **);
+void	gvinum_concat(int, char **);
 void	gvinum_create(int, char **);
+void	gvinum_detach(int, char **);
 void	gvinum_help(void);
 void	gvinum_list(int, char **);
 void	gvinum_move(int, char **);
+void	gvinum_mirror(int, char **);
 void	gvinum_parityop(int, char **, int);
 void	gvinum_printconfig(int, char **);
+void	gvinum_raid5(int, char **);
 void	gvinum_rename(int, char **);
 void	gvinum_resetconfig(void);
 void	gvinum_rm(int, char **);
@@ -67,9 +75,15 @@ void	gvinum_saveconfig(void);
 void	gvinum_setstate(int, char **);
 void	gvinum_start(int, char **);
 void	gvinum_stop(int, char **);
+void	gvinum_stripe(int, char **);
 void	parseline(int, char **);
 void	printconfig(FILE *, char *);
 
+char	*create_drive(char *);
+void	 create_volume(int, char **, char *);
+char	*find_name(const char *, int, int);
+char	*find_pattern(char *, char *);
+
 int
 main(int argc, char **argv)
 {
@@ -111,6 +125,44 @@ main(int argc, char **argv)
 	exit(0);
 }
 
+/* Attach a plex to a volume or a subdisk to a plex. */
+void
+gvinum_attach(int argc, char **argv)
+{
+	struct gctl_req *req;
+	const char *errstr;
+	int rename;
+	off_t offset;
+
+	rename = 0;
+	offset = -1;
+	if (argc < 3) {
+		warnx("usage:\tattach <subdisk> <plex> [rename] "
+		    "[<plexoffset>]\n"
+		    "\tattach <plex> <volume> [rename]");
+		return;
+	}
+	if (argc > 3) {
+		if (!strcmp(argv[3], "rename")) {
+			rename = 1;
+			if (argc == 5)
+				offset = strtol(argv[4], NULL, 0);
+		} else
+			offset = strtol(argv[3], NULL, 0);
+	}
+	req = gctl_get_handle();
+	gctl_ro_param(req, "class", -1, "VINUM");
+	gctl_ro_param(req, "verb", -1, "attach");
+	gctl_ro_param(req, "child", -1, argv[1]);
+	gctl_ro_param(req, "parent", -1, argv[2]);
+	gctl_ro_param(req, "offset", sizeof(off_t), &offset);
+	gctl_ro_param(req, "rename", sizeof(int), &rename);
+	errstr = gctl_issue(req);
+	if (errstr != NULL)
+		warnx("attach failed: %s", errstr);
+	gctl_free(req);
+}
+
 void
 gvinum_create(int argc, char **argv)
 {
@@ -120,19 +172,30 @@ gvinum_create(int argc, char **argv)
 	struct gv_sd *s;
 	struct gv_volume *v;
 	FILE *tmp;
-	int drives, errors, fd, line, plexes, plex_in_volume;
-	int sd_in_plex, status, subdisks, tokens, volumes;
+	int drives, errors, fd, flags, i, line, plexes, plex_in_volume;
+	int sd_in_plex, status, subdisks, tokens, undeffd, volumes;
 	const char *errstr;
-	char buf[BUFSIZ], buf1[BUFSIZ], commandline[BUFSIZ], *ed;
+	char buf[BUFSIZ], buf1[BUFSIZ], commandline[BUFSIZ], *ed, *sdname;
 	char original[BUFSIZ], tmpfile[20], *token[GV_MAXARGS];
 	char plex[GV_MAXPLEXNAME], volume[GV_MAXVOLNAME];
 
-	if (argc == 2) {
-		if ((tmp = fopen(argv[1], "r")) == NULL) {
-			warn("can't open '%s' for reading", argv[1]);
-			return;
-		}
-	} else {
+	tmp = NULL;
+	flags = 0;
+	for (i = 1; i < argc; i++) {
+		/* Force flag used to ignore already created drives. */
+		if (!strcmp(argv[i], "-f")) {
+			flags |= GV_FLAG_F;
+		/* Else it must be a file. */
+		} else {
+			if ((tmp = fopen(argv[1], "r")) == NULL) {
+				warn("can't open '%s' for reading", argv[1]);
+				return;
+			}
+		}	
+	}
+
+	/* We didn't get a file. */
+	if (tmp == NULL) {
 		snprintf(tmpfile, sizeof(tmpfile), "/tmp/gvinum.XXXXXX");
 		
 		if ((fd = mkstemp(tmpfile)) == -1) {
@@ -167,9 +230,11 @@ gvinum_create(int argc, char **argv)
 	req = gctl_get_handle();
 	gctl_ro_param(req, "class", -1, "VINUM");
 	gctl_ro_param(req, "verb", -1, "create");
+	gctl_ro_param(req, "flags", sizeof(int), &flags);
 
 	drives = volumes = plexes = subdisks = 0;
-	plex_in_volume = sd_in_plex = 0;
+	plex_in_volume = sd_in_plex = undeffd = 0;
+	plex[0] = '\0';
 	errors = 0;
 	line = 1;
 	while ((fgets(buf, BUFSIZ, tmp)) != NULL) {
@@ -187,7 +252,7 @@ gvinum_create(int argc, char **argv)
 		 * Copy the original input line in case we need it for error
 		 * output.
 		 */
-		strncpy(original, buf, sizeof(buf));
+		strlcpy(original, buf, sizeof(original));
 
 		tokens = gv_tokenize(buf, token, GV_MAXARGS);
 		if (tokens <= 0) {
@@ -214,7 +279,7 @@ gvinum_create(int argc, char **argv)
 			 * Set default volume name for following plex
 			 * definitions.
 			 */
-			strncpy(volume, v->name, sizeof(volume));
+			strlcpy(volume, v->name, sizeof(volume));
 
 			snprintf(buf1, sizeof(buf1), "volume%d", volumes);
 			gctl_ro_param(req, buf1, sizeof(*v), v);
@@ -236,13 +301,13 @@ gvinum_create(int argc, char **argv)
 
 			/* Default name. */
 			if (strlen(p->name) == 0) {
-				snprintf(p->name, GV_MAXPLEXNAME, "%s.p%d",
+				snprintf(p->name, sizeof(p->name), "%s.p%d",
 				    volume, plex_in_volume++);
 			}
 
 			/* Default volume. */
 			if (strlen(p->volume) == 0) {
-				snprintf(p->volume, GV_MAXVOLNAME, "%s",
+				snprintf(p->volume, sizeof(p->volume), "%s",
 				    volume);
 			}
 
@@ -250,7 +315,7 @@ gvinum_create(int argc, char **argv)
 			 * Set default plex name for following subdisk
 			 * definitions.
 			 */
-			strncpy(plex, p->name, GV_MAXPLEXNAME);
+			strlcpy(plex, p->name, sizeof(plex));
 
 			snprintf(buf1, sizeof(buf1), "plex%d", plexes);
 			gctl_ro_param(req, buf1, sizeof(*p), p);
@@ -270,13 +335,21 @@ gvinum_create(int argc, char **argv)
 
 			/* Default name. */
 			if (strlen(s->name) == 0) {
-				snprintf(s->name, GV_MAXSDNAME, "%s.s%d",
-				    plex, sd_in_plex++);
+				if (strlen(plex) == 0) {
+					sdname = find_name("gvinumsubdisk.p",
+					    GV_TYPE_SD, GV_MAXSDNAME);
+					snprintf(s->name, sizeof(s->name),
+					    "%s.s%d", sdname, undeffd++);
+					free(sdname);
+				} else {
+					snprintf(s->name, sizeof(s->name),
+					    "%s.s%d",plex, sd_in_plex++);
+				}
 			}
 
 			/* Default plex. */
 			if (strlen(s->plex) == 0)
-				snprintf(s->plex, GV_MAXPLEXNAME, "%s", plex);
+				snprintf(s->plex, sizeof(s->plex), "%s", plex);
 
 			snprintf(buf1, sizeof(buf1), "sd%d", subdisks);
 			gctl_ro_param(req, buf1, sizeof(*s), s);
@@ -320,7 +393,279 @@ gvinum_create(int argc, char **argv)
 			warnx("create failed: %s", errstr);
 	}
 	gctl_free(req);
-	gvinum_list(0, NULL);
+}
+
+/* Create a concatenated volume. */
+void
+gvinum_concat(int argc, char **argv)
+{
+
+	if (argc < 2) {
+		warnx("usage:\tconcat [-fv] [-n name] drives\n");
+		return;
+	}
+	create_volume(argc, argv, "concat");
+}
+
+
+/* Create a drive quick and dirty. */
+char *
+create_drive(char *device)
+{
+	struct gv_drive *d;
+	struct gctl_req *req;
+	const char *errstr;
+	char *drivename, *dname;
+	int drives, i, flags, volumes, subdisks, plexes;
+
+	flags = plexes = subdisks = volumes = 0;
+	drives = 1;
+	dname = NULL;
+
+	/* Strip away eventual /dev/ in front. */
+	if (strncmp(device, "/dev/", 5) == 0)
+		device += 5;
+
+	drivename = find_name("gvinumdrive", GV_TYPE_DRIVE, GV_MAXDRIVENAME);
+	if (drivename == NULL)
+		return (NULL);
+
+	req = gctl_get_handle();
+	gctl_ro_param(req, "class", -1, "VINUM");
+	gctl_ro_param(req, "verb", -1, "create");
+	d = malloc(sizeof(struct gv_drive));
+	if (d == NULL)
+		err(1, "unable to allocate for gv_drive object");
+	memset(d, 0, sizeof(struct gv_drive));
+
+	strlcpy(d->name, drivename, sizeof(d->name));
+	strlcpy(d->device, device, sizeof(d->device));
+	gctl_ro_param(req, "drive0", sizeof(*d), d);
+	gctl_ro_param(req, "flags", sizeof(int), &flags);
+	gctl_ro_param(req, "drives", sizeof(int), &drives);
+	gctl_ro_param(req, "volumes", sizeof(int), &volumes);
+	gctl_ro_param(req, "plexes", sizeof(int), &plexes);
+	gctl_ro_param(req, "subdisks", sizeof(int), &subdisks);
+	errstr = gctl_issue(req);
+	if (errstr != NULL) {
+		warnx("error creating drive: %s", errstr);
+		gctl_free(req);
+		return (NULL);
+	} else {
+		gctl_free(req);
+		/* XXX: This is needed because we have to make sure the drives
+		 * are created before we return. */
+		/* Loop until it's in the config. */
+		for (i = 0; i < 100000; i++) {
+			dname = find_name("gvinumdrive", GV_TYPE_DRIVE,
+			    GV_MAXDRIVENAME);
+			/* If we got a different name, quit. */
+			if (dname == NULL)
+				continue;
+			if (strcmp(dname, drivename)) {
+				free(dname);
+				return (drivename);
+			}
+			free(dname);
+			dname = NULL;
+			usleep(100000); /* Sleep for 0.1s */
+		}
+	}
+	gctl_free(req);
+	return (drivename);
+}
+
+/* 
+ * General routine for creating a volume. Mainly for use by concat, mirror,
+ * raid5 and stripe commands.
+ */
+void
+create_volume(int argc, char **argv, char *verb)
+{
+	struct gctl_req *req;
+	const char *errstr;
+	char buf[BUFSIZ], *drivename, *volname;
+	int drives, flags, i;
+	off_t stripesize;
+
+	flags = 0;
+	drives = 0;
+	volname = NULL;
+	stripesize = 262144;
+
+	/* XXX: Should we check for argument length? */
+
+	req = gctl_get_handle();
+	gctl_ro_param(req, "class", -1, "VINUM");
+
+	for (i = 1; i < argc; i++) {
+		if (!strcmp(argv[i], "-f")) {
+			flags |= GV_FLAG_F;
+		} else if (!strcmp(argv[i], "-n")) {
+			volname = argv[++i];
+		} else if (!strcmp(argv[i], "-v")) {
+			flags |= GV_FLAG_V;
+		} else if (!strcmp(argv[i], "-s")) {
+			flags |= GV_FLAG_S;
+			if (!strcmp(verb, "raid5"))
+				stripesize = gv_sizespec(argv[++i]);
+		} else {
+			/* Assume it's a drive. */
+			snprintf(buf, sizeof(buf), "drive%d", drives++);
+
+			/* First we create the drive. */
+			drivename = create_drive(argv[i]); 
+			if (drivename == NULL)
+				goto bad;
+			/* Then we add it to the request. */
+			gctl_ro_param(req, buf, -1, drivename);
+		}
+	}
+
+	gctl_ro_param(req, "stripesize", sizeof(off_t), &stripesize);
+
+	/* Find a free volume name. */
+	if (volname == NULL)
+		volname = find_name("gvinumvolume", GV_TYPE_VOL, GV_MAXVOLNAME);
+
+	/* Then we send a request to actually create the volumes. */
+	gctl_ro_param(req, "verb", -1, verb);
+	gctl_ro_param(req, "flags", sizeof(int), &flags); 
+	gctl_ro_param(req, "drives", sizeof(int), &drives);
+	gctl_ro_param(req, "name", -1, volname);
+	errstr = gctl_issue(req);
+	if (errstr != NULL)
+		warnx("creating %s volume failed: %s", verb, errstr);
+bad:
+	gctl_free(req);
+}
+
+/* Parse a line of the config, return the word after <pattern>. */
+char *
+find_pattern(char *line, char *pattern)
+{
+	char *ptr;
+
+	ptr = strsep(&line, " ");
+	while (ptr != NULL) {
+		if (!strcmp(ptr, pattern)) {
+			/* Return the next. */
+			ptr = strsep(&line, " ");
+			return (ptr);
+		}
+		ptr = strsep(&line, " ");
+	}
+	return (NULL);
+}
+
+/* Find a free name for an object given a a prefix. */
+char *
+find_name(const char *prefix, int type, int namelen)
+{
+	struct gctl_req *req;
+	char comment[1], buf[GV_CFG_LEN - 1], *name, *sname, *ptr;
+	const char *errstr;
+	int i, n, begin, len, conflict;
+	char line[1024];
+
+	comment[0] = '\0';
+
+	/* Find a name. Fetch out configuration first. */
+	req = gctl_get_handle();
+	gctl_ro_param(req, "class", -1, "VINUM");
+	gctl_ro_param(req, "verb", -1, "getconfig");
+	gctl_ro_param(req, "comment", -1, comment);
+	gctl_rw_param(req, "config", sizeof(buf), buf);
+	errstr = gctl_issue(req);
+	if (errstr != NULL) {
+		warnx("can't get configuration: %s", errstr);
+		return (NULL);
+	}
+	gctl_free(req);
+
+	begin = 0;
+	len = strlen(buf);
+	i = 0;
+	sname = malloc(namelen + 1);
+
+	/* XXX: Max object setting? */
+	for (n = 0; n < 10000; n++) {
+		snprintf(sname, namelen, "%s%d", prefix, n);
+		conflict = 0;
+		begin = 0;
+		/* Loop through the configuration line by line. */
+		for (i = 0; i < len; i++) {
+			if (buf[i] == '\n' || buf[i] == '\0') {
+				ptr = buf + begin;
+				strlcpy(line, ptr, (i - begin) + 1);
+				begin = i + 1;
+				switch (type) {
+				case GV_TYPE_DRIVE:
+					name = find_pattern(line, "drive");
+					break;
+				case GV_TYPE_VOL:
+					name = find_pattern(line, "volume");
+					break;
+				case GV_TYPE_PLEX:
+				case GV_TYPE_SD:
+					name = find_pattern(line, "name");
+					break;
+				default:
+					printf("Invalid type given\n");
+					continue;
+				}
+				if (name == NULL)
+					continue;
+				if (!strcmp(sname, name)) {
+					conflict = 1;
+					/* XXX: Could quit the loop earlier. */
+				}
+			}
+		}
+		if (!conflict)
+			return (sname);
+	}
+	free(sname);
+	return (NULL);
+}
+
+/* Detach a plex or subdisk from its parent. */
+void
+gvinum_detach(int argc, char **argv)
+{
+	const char *errstr;
+	struct gctl_req *req;
+	int flags, i;
+
+	optreset = 1;
+	optind = 1;
+	while ((i = getopt(argc, argv, "f")) != -1) {
+		switch(i) {
+		case 'f':
+			flags |= GV_FLAG_F;
+			break;
+		default:
+			warn("invalid flag: %c", i);
+			return;
+		}
+	}
+	argc -= optind;
+	argv += optind;
+	if (argc != 1) {
+		warnx("usage: detach [-f] <subdisk> | <plex>");
+		return;
+	}
+
+	req = gctl_get_handle();
+	gctl_ro_param(req, "class", -1, "VINUM");
+	gctl_ro_param(req, "verb", -1, "detach");
+	gctl_ro_param(req, "object", -1, argv[0]);
+	gctl_ro_param(req, "flags", sizeof(int), &flags);
+
+	errstr = gctl_issue(req);
+	if (errstr != NULL)
+		warnx("detach failed: %s", errstr);
+	gctl_free(req);
 }
 
 void
@@ -329,8 +674,16 @@ gvinum_help(void)
 	printf("COMMANDS\n"
 	    "checkparity [-f] plex\n"
 	    "        Check the parity blocks of a RAID-5 plex.\n"
-	    "create description-file\n"
+	    "create [-f] description-file\n"
 	    "        Create as per description-file or open editor.\n"
+	    "attach plex volume [rename]\n"
+	    "attach subdisk plex [offset] [rename]\n"
+	    "        Attach a plex to a volume, or a subdisk to a plex\n"
+	    "concat [-fv] [-n name] drives\n"
+	    "        Create a concatenated volume from the specified drives.\n"
+	    "detach [-f] [plex | subdisk]\n"
+	    "        Detach a plex or a subdisk from the volume or plex to\n"
+	    "        which it is attached.\n"
 	    "l | list [-r] [-v] [-V] [volume | plex | subdisk]\n"
 	    "        List information about specified objects.\n"
 	    "ld [-r] [-v] [-V] [volume]\n"
@@ -341,18 +694,22 @@ gvinum_help(void)
 	    "        List information about plexes.\n"
 	    "lv [-r] [-v] [-V] [volume]\n"
 	    "        List information about volumes.\n"
+	    "mirror [-fsv] [-n name] drives\n"
+	    "        Create a mirrored volume from the specified drives.\n"
 	    "move | mv -f drive object ...\n"
 	    "        Move the object(s) to the specified drive.\n"
 	    "quit    Exit the vinum program when running in interactive mode."
 	    "  Nor-\n"
 	    "        mally this would be done by entering the EOF character.\n"
+	    "raid5 [-fv] [-s stripesize] [-n name] drives\n"
+	    "        Create a RAID-5 volume from the specified drives.\n"
 	    "rename [-r] [drive | subdisk | plex | volume] newname\n"
 	    "        Change the name of the specified object.\n"
 	    "rebuildparity plex [-f]\n"
 	    "        Rebuild the parity blocks of a RAID-5 plex.\n"
 	    "resetconfig\n"
 	    "        Reset the complete gvinum configuration\n"
-	    "rm [-r] volume | plex | subdisk | drive\n"
+	    "rm [-r] [-f] volume | plex | subdisk | drive\n"
 	    "        Remove an object.\n"
 	    "saveconfig\n"
 	    "        Save vinum configuration to disk after configuration"
@@ -363,6 +720,8 @@ gvinum_help(void)
 	    "        poses only.\n"
 	    "start [-S size] volume | plex | subdisk\n"
 	    "        Allow the system to access the objects.\n"
+	    "stripe [-fv] [-n name] drives\n"
+	    "        Create a striped volume from the specified drives.\n"
 	);
 
 	return;
@@ -488,6 +847,18 @@ gvinum_list(int argc, char **argv)
 	return;
 }
 
+/* Create a mirrored volume. */
+void
+gvinum_mirror(int argc, char **argv)
+{
+
+	if (argc < 2) {
+		warnx("usage\tmirror [-fsv] [-n name] drives\n");
+		return;
+	}
+	create_volume(argc, argv, "mirror");
+}
+
 /* Note that move is currently of form '[-r] target object [...]' */
 void
 gvinum_move(int argc, char **argv)
@@ -553,8 +924,7 @@ void
 gvinum_parityop(int argc, char **argv, int rebuild)
 {
 	struct gctl_req *req;
-	int flags, i, rv;
-	off_t offset;
+	int flags, i;
 	const char *errstr;
 	char *op, *msg;
 
@@ -591,47 +961,32 @@ gvinum_parityop(int argc, char **argv, int rebuild)
 		return;
 	}
 
-	do {
-		rv = 0;
-		req = gctl_get_handle();
-		gctl_ro_param(req, "class", -1, "VINUM");
-		gctl_ro_param(req, "verb", -1, "parityop");
-		gctl_ro_param(req, "flags", sizeof(int), &flags);
-		gctl_ro_param(req, "rebuild", sizeof(int), &rebuild);
-		gctl_rw_param(req, "rv", sizeof(int), &rv);
-		gctl_rw_param(req, "offset", sizeof(off_t), &offset);
-		gctl_ro_param(req, "plex", -1, argv[0]);
-		errstr = gctl_issue(req);
-		if (errstr) {
-			warnx("%s\n", errstr);
-			gctl_free(req);
-			break;
-		}
-		gctl_free(req);
-		if (flags & GV_FLAG_V) {
-			printf("\r%s at %s ... ", msg,
-			    gv_roughlength(offset, 1));
-		}
-		if (rv == 1) {
-			printf("Parity incorrect at offset 0x%jx\n",
-			    (intmax_t)offset);
-			if (!rebuild)
-				break;
-		}
-		fflush(stdout);
+	req = gctl_get_handle();
+	gctl_ro_param(req, "class", -1, "VINUM");
+	gctl_ro_param(req, "verb", -1, op);
+	gctl_ro_param(req, "rebuild", sizeof(int), &rebuild);
+	gctl_ro_param(req, "flags", sizeof(int), &flags);
+	gctl_ro_param(req, "plex", -1, argv[0]);
+
+	errstr = gctl_issue(req);
+	if (errstr)
+		warnx("%s\n", errstr);
+	gctl_free(req);
+}
 
-		/* Clear the -f flag. */
-		flags &= ~GV_FLAG_F;
-	} while (rv >= 0);
+/* Create a RAID-5 volume. */
+void
+gvinum_raid5(int argc, char **argv)
+{
 
-	if ((rv == 2) && (flags & GV_FLAG_V)) {
-		if (rebuild)
-			printf("Rebuilt parity on %s\n", argv[0]);
-		else
-			printf("%s has correct parity\n", argv[0]);
+	if (argc < 2) {
+		warnx("usage:\traid5 [-fv] [-s stripesize] [-n name] drives\n");
+		return;
 	}
+	create_volume(argc, argv, "raid5");
 }
 
+
 void
 gvinum_rename(int argc, char **argv)
 {
@@ -697,8 +1052,11 @@ gvinum_rm(int argc, char **argv)
 	flags = 0;
 	optreset = 1;
 	optind = 1;
-	while ((j = getopt(argc, argv, "r")) != -1) {
+	while ((j = getopt(argc, argv, "rf")) != -1) {
 		switch (j) {
+		case 'f':
+			flags |= GV_FLAG_F;
+			break;
 		case 'r':
 			flags |= GV_FLAG_R;
 			break;
@@ -728,7 +1086,6 @@ gvinum_rm(int argc, char **argv)
 		return;
 	}
 	gctl_free(req);
-	gvinum_list(0, NULL);
 }
 
 void
@@ -763,7 +1120,6 @@ gvinum_resetconfig(void)
 		return;
 	}
 	gctl_free(req);
-	gvinum_list(0, NULL);
 	printf("gvinum configuration obliterated\n");
 }
 
@@ -833,28 +1189,53 @@ gvinum_start(int argc, char **argv)
 	}
 
 	gctl_free(req);
-	gvinum_list(0, NULL);
 }
 
 void
 gvinum_stop(int argc, char **argv)
 {
-	int fileid;
+	int err, fileid;
 
 	fileid = kldfind(GVINUMMOD);
 	if (fileid == -1) {
 		warn("cannot find " GVINUMMOD);
 		return;
 	}
-	if (kldunload(fileid) != 0) {
+
+	/*
+	 * This little hack prevents that we end up in an infinite loop in
+	 * g_unload_class().  gv_unload() will return EAGAIN so that the GEOM
+	 * event thread will be free for the g_wither_geom() call from
+	 * gv_unload().  It's silly, but it works.
+	 */
+	printf("unloading " GVINUMMOD " kernel module... ");
+	fflush(stdout);
+	if ((err = kldunload(fileid)) != 0 && (errno == EAGAIN)) {
+		sleep(1);
+		err = kldunload(fileid);
+	}
+	if (err != 0) {
+		printf(" failed!\n");
 		warn("cannot unload " GVINUMMOD);
 		return;
 	}
 
-	warnx(GVINUMMOD " unloaded");
+	printf("done\n");
 	exit(0);
 }
 
+/* Create a striped volume. */
+void
+gvinum_stripe(int argc, char **argv)
+{
+
+	if (argc < 2) {
+		warnx("usage:\tstripe [-fv] [-n name] drives\n");
+		return;
+	}
+	create_volume(argc, argv, "stripe");
+}
+
 void
 parseline(int argc, char **argv)
 {
@@ -865,6 +1246,12 @@ parseline(int argc, char **argv)
 		gvinum_create(argc, argv);
 	else if (!strcmp(argv[0], "exit") || !strcmp(argv[0], "quit"))
 		exit(0);
+	else if (!strcmp(argv[0], "attach"))
+		gvinum_attach(argc, argv);
+	else if (!strcmp(argv[0], "detach"))
+		gvinum_detach(argc, argv);
+	else if (!strcmp(argv[0], "concat"))
+		gvinum_concat(argc, argv);
 	else if (!strcmp(argv[0], "help"))
 		gvinum_help();
 	else if (!strcmp(argv[0], "list") || !strcmp(argv[0], "l"))
@@ -877,12 +1264,16 @@ parseline(int argc, char **argv)
 		gvinum_list(argc, argv);
 	else if (!strcmp(argv[0], "lv"))
 		gvinum_list(argc, argv);
+	else if (!strcmp(argv[0], "mirror"))
+		gvinum_mirror(argc, argv);
 	else if (!strcmp(argv[0], "move"))
 		gvinum_move(argc, argv);
 	else if (!strcmp(argv[0], "mv"))
 		gvinum_move(argc, argv);
 	else if (!strcmp(argv[0], "printconfig"))
 		gvinum_printconfig(argc, argv);
+	else if (!strcmp(argv[0], "raid5"))
+		gvinum_raid5(argc, argv);
 	else if (!strcmp(argv[0], "rename"))
 		gvinum_rename(argc, argv);
 	else if (!strcmp(argv[0], "resetconfig"))
@@ -897,6 +1288,8 @@ parseline(int argc, char **argv)
 		gvinum_start(argc, argv);
 	else if (!strcmp(argv[0], "stop"))
 		gvinum_stop(argc, argv);
+	else if (!strcmp(argv[0], "stripe"))
+		gvinum_stripe(argc, argv);
 	else if (!strcmp(argv[0], "checkparity"))
 		gvinum_parityop(argc, argv, 0);
 	else if (!strcmp(argv[0], "rebuildparity"))