Introduce "feature flags" for ZFS pools (bump SPA version to 5000).

Add first feature "com.delphix:async_destroy" (asynchronous destroy
of ZFS datasets).
Implement features support in ZFS boot code.

Illumos revisions merged:
13700:2889e2596bd6
13701:1949b688d5fb
2619 asynchronous destruction of ZFS file systems
2747 SPA versioning with zfs feature flags

References:
https://www.illumos.org/issues/2619
https://www.illumos.org/issues/2747

Obtained from:	illumos (issue #2619, #2747)
MFC after:	1 month

3 files changed, 403 insertions, 37 deletions

diff --git a/cddl/contrib/opensolaris/cmd/zpool/zpool-features.5 b/cddl/contrib/opensolaris/cmd/zpool/zpool-features.5
new file mode 100644
index 0000000..d590fdc
--- /dev/null
+++ b/cddl/contrib/opensolaris/cmd/zpool/zpool-features.5
@@ -0,0 +1,174 @@
+'\" te
+.\" Copyright (c) 2012, Martin Matuska <mm@FreeBSD.org>.
+.\" All Rights Reserved.
+.\"
+.\" The contents of this file are subject to the terms of the
+.\" Common Development and Distribution License (the "License").
+.\" You may not use this file except in compliance with the License.
+.\"
+.\" You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
+.\" or http://www.opensolaris.org/os/licensing.
+.\" See the License for the specific language governing permissions
+.\" and limitations under the License.
+.\"
+.\" When distributing Covered Code, include this CDDL HEADER in each
+.\" file and include the License file at usr/src/OPENSOLARIS.LICENSE.
+.\" If applicable, add the following below this CDDL HEADER, with the
+.\" fields enclosed by brackets "[]" replaced with your own identifying
+.\" information: Portions Copyright [yyyy] [name of copyright owner]
+.\"
+.\" Copyright (c) 2012 by Delphix. All rights reserved.
+.\"
+.\" $FreeBSD$
+.\"
+.Dd May 28, 2012
+.Dt ZPOOL-FEATURES 8
+.Os
+.Sh NAME
+.Nm zpool-features
+.Nd ZFS pool feature descriptions
+.Sh DESCRIPTION
+ZFS pool on\-disk format versions are specified via "features" which replace
+the old on\-disk format numbers (the last supported on\-disk format number is
+28).
+To enable a feature on a pool use the
+.Xr zpool 8
+command to set the
+.Sy feature@feature_name
+property to
+.Ar enabled .
+.Pp
+The pool format does not affect file system version compatibility or the ability
+to send file systems between pools.
+.Pp
+Since most features can be enabled independently of each other the on\-disk
+format of the pool is specified by the set of all features marked as
+.Sy active
+on the pool. If the pool was created by another software version this set may
+include unsupported features.
+.Ss Identifying features
+Every feature has a guid of the form
+.Sy com.example:feature_name .
+The reverse DNS name ensures that the feature's guid is unique across all ZFS
+implementations. When unsupported features are encountered on a pool they will
+be identified by their guids.
+Refer to the documentation for the ZFS implementation that created the pool
+for information about those features.
+.Pp
+Each supported feature also has a short name.
+By convention a feature's short name is the portion of its guid which follows
+the ':' (e.g.
+.Sy com.example:feature_name
+would have the short name
+.Sy feature_name ),
+however a feature's short name may differ across ZFS implementations if
+following the convention would result in name conflicts.
+.Ss Feature states
+Features can be in one of three states:
+.Bl -tag
+.It Sy active
+This feature's on\-disk format changes are in effect on the pool.
+Support for this feature is required to import the pool in read\-write mode.
+If this feature is not read-only compatible, support is also required to
+import the pool in read\-only mode (see "Read\-only compatibility").
+.It Sy enabled
+An administrator has marked this feature as enabled on the pool, but the
+feature's on\-disk format changes have not been made yet.
+The pool can still be imported by software that does not support this feature,
+but changes may be made to the on\-disk format at any time which will move
+the feature to the
+.Sy active
+state.
+Some features may support returning to the
+.Sy enabled
+state after becoming
+.Sy active .
+See feature\-specific documentation for details.
+.It Sy disabled
+This feature's on\-disk format changes have not been made and will not be made
+unless an administrator moves the feature to the
+.Sy enabled
+state.
+Features cannot be disabled once they have been enabled.
+.El
+The state of supported features is exposed through pool properties of the form
+.Sy feature@short_name .
+.Ss Read\-only compatibility
+Some features may make on\-disk format changes that do not interfere with other
+software's ability to read from the pool.
+These features are referred to as "read\-only compatible".
+If all unsupported features on a pool are read\-only compatible, the pool can
+be imported in read\-only mode by setting the
+.Sy readonly
+property during import (see
+.Xr zpool 8
+for details on importing pools).
+.Ss Unsupported features
+For each unsupported feature enabled on an imported pool a pool property
+named
+.Sy unsupported@feature_guid
+will indicate why the import was allowed despite the unsupported feature.
+Possible values for this property are:
+.Bl -tag
+.It Sy inactive
+The feature is in the
+.Sy enabled
+state and therefore the pool's on\-disk format is still compatible with
+software that does not support this feature.
+.It Sy readonly
+The feature is read\-only compatible and the pool has been imported in
+read\-only mode.
+.El
+.Ss Feature dependencies
+Some features depend on other features being enabled in order to function
+properly.
+Enabling a feature will automatically enable any features it depends on.
+.Sh FEATURES
+The following features are supported on this system:
+.Bl -tag
+.It Sy async_destroy
+.Bl -column "READ\-ONLY COMPATIBLE" "com.delphix:async_destroy"
+.It GUID Ta com.delphix:async_destroy
+.It READ\-ONLY COMPATIBLE Ta yes
+.It DEPENDENCIES Ta none
+.El
+.Pp
+Destroying a file system requires traversing all of its data in order to
+return its used space to the pool.
+Without
+.Sy async_destroy
+the file system is not fully removed until all space has been reclaimed.
+If the destroy operation is interrupted by a reboot or power outage the next
+attempt to open the pool will need to complete the destroy operation
+synchronously.
+.Pp
+When
+.Sy async_destroy
+is enabled the file system's data will be reclaimed by a background process,
+allowing the destroy operation to complete without traversing the entire file
+system.
+The background process is able to resume interrupted destroys after the pool
+has been opened, eliminating the need to finish interrupted destroys as part
+of the open operation.
+The amount of space remaining to be reclaimed by the background process is
+available through the
+.Sy freeing
+property.
+.Sh SEE ALSO
+.Xr zpool 8
+.Sh AUTHORS
+This manual page is a
+.Xr mdoc 7
+reimplementation of the
+.Tn illumos
+manual page
+.Em zpool-features(5) ,
+modified and customized for
+.Fx
+and licensed under the Common Development and Distribution License
+.Pq Tn CDDL .
+.Pp
+The
+.Xr mdoc 7
+implementation of this manual page was initially written by
+.An Martin Matuska Aq mm@FreeBSD.org .
diff --git a/cddl/contrib/opensolaris/cmd/zpool/zpool.8 b/cddl/contrib/opensolaris/cmd/zpool/zpool.8
index 07e03bd..89333c7 100644
--- a/cddl/contrib/opensolaris/cmd/zpool/zpool.8
+++ b/cddl/contrib/opensolaris/cmd/zpool/zpool.8
@@ -1,5 +1,5 @@
 '\" te
-.\" Copyright (c) 2011, Martin Matuska <mm@FreeBSD.org>.
+.\" Copyright (c) 2012, Martin Matuska <mm@FreeBSD.org>.
 .\" All Rights Reserved.
 .\"
 .\" The contents of this file are subject to the terms of the
@@ -20,6 +20,7 @@
 .\" Copyright (c) 2010, Sun Microsystems, Inc. All Rights Reserved.
 .\" Copyright 2011, Nexenta Systems, Inc. All Rights Reserved.
 .\" Copyright (c) 2011, Justin T. Gibbs <gibbs@FreeBSD.org>
+.\" Copyright (c) 2012 by Delphix. All Rights Reserved.
 .\"
 .\" $FreeBSD$
 .\"
@@ -47,7 +48,7 @@
 .Op Ar device
 .Nm
 .Cm create
-.Op Fl fn
+.Op Fl fnd
 .Op Fl o Ar property Ns = Ns Ar value
 .Ar ...
 .Op Fl O Ar file-system-property Ns = Ns Ar value
@@ -537,6 +538,16 @@ value of 1.76 indicates that 1.76 units of data were stored but only 1 unit of d
 for a description of the deduplication feature.
 .It Sy free
 Number of blocks within the pool that are not allocated.
+.It Sy freeing
+After a file system or snapshot is destroyed, the space it was using is
+returned to the pool asynchronously.
+.Sy freeing
+is the amount of space remaining to be reclaimed.
+Over time
+.Sy freeing
+will decrease while
+.Sy free
+increases.
 .It Sy expandsize
 This property has currently no value on FreeBSD.
 .It Sy guid
@@ -552,11 +563,16 @@ or
 .Qq Sy UNAVAIL .
 .It Sy size
 Total size of the storage pool.
+.It Sy unsupported@ Ns Ar feature_guid
+Information about unsupported features that are enabled on the pool.
+See
+.Xr zpool-features 5
+for details.
 .It Sy used
 Amount of storage space used within the pool.
 .El
 .Pp
-These space usage properties report actual physical space available to the
+The space usage properties report actual physical space available to the
 storage pool. The physical space can be different from the total amount of
 space that any contained datasets can actually use. The amount of space used in
 a
@@ -653,6 +669,11 @@ Setting it to the special value
 creates a temporary pool that is never cached, and the special value
 .Cm ''
 (empty string) uses the default location.
+.It Sy comment Ns = Ns Ar text
+A text string consisting of printable ASCII characters that will be stored
+such that it is available even if the pool becomes faulted.
+An administrator can provide additional information about a pool using this
+property.
 .It Sy dedupditto Ns = Ns Ar number
 Threshold for the number of block ditto copies. If the reference count for a
 deduplicated block increases above this number, a new ditto copy of this block
@@ -686,6 +707,17 @@ requests that have yet to be committed to disk would be blocked.
 .It Sy panic
 Prints out a message to the console and generates a system crash dump.
 .El
+.It Sy feature@ Ns Ar feature_name Ns = Ns Sy enabled
+The value of this property is the current state of
+.Ar feature_name .
+The only valid value when setting this property is
+.Sy enabled
+which moves
+.Ar feature_name
+to the enabled state.
+See
+.Xr zpool-features 5
+for details on feature states.
 .It Sy listsnaps Ns = Ns Cm on No | Cm off
 Controls whether information about snapshots associated with this pool is
 output when
@@ -699,9 +731,9 @@ The current on-disk version of the pool. This can be increased, but never
 decreased. The preferred method of updating pools is with the
 .Qq Nm Cm upgrade
 command, though this property can be used when a specific version is needed
-for backwards compatibility. This property can be any number between 1 and the
-current version reported by
-.Qo Ic zpool upgrade -v Qc .
+for backwards compatibility. 
+Once feature flags is enabled on a pool this property will no longer have a
+value.
 .El
 .Sh SUBCOMMANDS
 All subcommands that modify state are logged persistently to the pool in their
@@ -810,7 +842,7 @@ do not actually discard any transactions.
 .It Xo
 .Nm
 .Cm create
-.Op Fl fn
+.Op Fl fnd
 .Op Fl o Ar property Ns = Ns Ar value
 .Ar ...
 .Op Fl O Ar file-system-property Ns = Ns Ar value
@@ -859,6 +891,10 @@ The mount point must not exist or must be empty, or else the
 root dataset cannot be mounted. This can be overridden with the
 .Fl m
 option.
+.Pp
+By default all supported features are enabled on the new pool unless the
+.Fl d
+option is specified.
 .Bl -tag -width indent
 .It Fl f
 Forces use of
@@ -869,6 +905,17 @@ Not all devices can be overridden in this manner.
 Displays the configuration that would be used without actually creating the
 pool. The actual pool creation can still fail due to insufficient privileges or
 device sharing.
+.It Fl d
+Do not enable any features on the new pool.
+Individual features can be enabled by setting their corresponding properties
+to
+.Sy enabled
+with the
+.Fl o
+option.
+See
+.Xr zpool-features 5
+for details about feature properties.
 .It Xo
 .Fl o Ar property Ns = Ns Ar value
 .Op Fl o Ar property Ns = Ns Ar value
@@ -1873,6 +1920,7 @@ Invalid command line options were specified.
 .El
 .Sh SEE ALSO
 .Xr zfs 8
+.Xr zpool-features 5
 .Sh AUTHORS
 This manual page is a
 .Xr mdoc 7
diff --git a/cddl/contrib/opensolaris/cmd/zpool/zpool_main.c b/cddl/contrib/opensolaris/cmd/zpool/zpool_main.c
index 3904e68..54a7470 100644
--- a/cddl/contrib/opensolaris/cmd/zpool/zpool_main.c
+++ b/cddl/contrib/opensolaris/cmd/zpool/zpool_main.c
@@ -54,6 +54,7 @@
 
 #include "zpool_util.h"
 #include "zfs_comutil.h"
+#include "zfeature_common.h"
 
 #include "statcommon.h"
 
@@ -207,7 +208,7 @@ get_usage(zpool_help_t idx) {
 	case HELP_CLEAR:
 		return (gettext("\tclear [-nF] <pool> [device]\n"));
 	case HELP_CREATE:
-		return (gettext("\tcreate [-fn] [-o property=value] ... \n"
+		return (gettext("\tcreate [-fnd] [-o property=value] ... \n"
 		    "\t    [-O file-system-property=value] ... \n"
 		    "\t    [-m mountpoint] [-R root] <pool> <vdev> ...\n"));
 	case HELP_DESTROY:
@@ -339,6 +340,12 @@ usage(boolean_t requested)
 		/* Iterate over all properties */
 		(void) zprop_iter(print_prop_cb, fp, B_FALSE, B_TRUE,
 		    ZFS_TYPE_POOL);
+
+		(void) fprintf(fp, "\t%-15s   ", "feature@...");
+		(void) fprintf(fp, "YES   disabled | enabled | active\n");
+
+		(void) fprintf(fp, gettext("\nThe feature@ properties must be "
+		    "appended with a feature name.\nSee zpool-features(5).\n"));
 	}
 
 	/*
@@ -405,12 +412,16 @@ add_prop_list(const char *propname, char *propval, nvlist_t **props,
 	proplist = *props;
 
 	if (poolprop) {
-		if ((prop = zpool_name_to_prop(propname)) == ZPROP_INVAL) {
+		if ((prop = zpool_name_to_prop(propname)) == ZPROP_INVAL &&
+		    !zpool_prop_feature(propname)) {
 			(void) fprintf(stderr, gettext("property '%s' is "
 			    "not a valid pool property\n"), propname);
 			return (2);
 		}
-		normnm = zpool_prop_to_name(prop);
+		if (zpool_prop_feature(propname))
+			normnm = propname;
+		else
+			normnm = zpool_prop_to_name(prop);
 	} else {
 		if ((fprop = zfs_name_to_prop(propname)) != ZPROP_INVAL) {
 			normnm = zfs_prop_to_name(fprop);
@@ -701,7 +712,7 @@ errout:
 }
 
 /*
- * zpool create [-fn] [-o property=value] ...
+ * zpool create [-fnd] [-o property=value] ...
  *		[-O file-system-property=value] ...
  *		[-R root] [-m mountpoint] <pool> <dev> ...
  *
@@ -710,8 +721,10 @@ errout:
  *		were to be created.
  *      -R	Create a pool under an alternate root
  *      -m	Set default mountpoint for the root dataset.  By default it's
- *      	'/<pool>'
+ *		'/<pool>'
  *	-o	Set property=value.
+ *	-d	Don't automatically enable all supported pool features
+ *		(individual features can be enabled with -o).
  *	-O	Set fsproperty=value in the pool's root file system
  *
  * Creates the named pool according to the given vdev specification.  The
@@ -724,6 +737,7 @@ zpool_do_create(int argc, char **argv)
 {
 	boolean_t force = B_FALSE;
 	boolean_t dryrun = B_FALSE;
+	boolean_t enable_all_pool_feat = B_TRUE;
 	int c;
 	nvlist_t *nvroot = NULL;
 	char *poolname;
@@ -735,7 +749,7 @@ zpool_do_create(int argc, char **argv)
 	char *propval;
 
 	/* check options */
-	while ((c = getopt(argc, argv, ":fnR:m:o:O:")) != -1) {
+	while ((c = getopt(argc, argv, ":fndR:m:o:O:")) != -1) {
 		switch (c) {
 		case 'f':
 			force = B_TRUE;
@@ -743,6 +757,9 @@ zpool_do_create(int argc, char **argv)
 		case 'n':
 			dryrun = B_TRUE;
 			break;
+		case 'd':
+			enable_all_pool_feat = B_FALSE;
+			break;
 		case 'R':
 			altroot = optarg;
 			if (add_prop_list(zpool_prop_to_name(
@@ -770,6 +787,21 @@ zpool_do_create(int argc, char **argv)
 
 			if (add_prop_list(optarg, propval, &props, B_TRUE))
 				goto errout;
+
+			/*
+			 * If the user is creating a pool that doesn't support
+			 * feature flags, don't enable any features.
+			 */
+			if (zpool_name_to_prop(optarg) == ZPOOL_PROP_VERSION) {
+				char *end;
+				u_longlong_t ver;
+
+				ver = strtoull(propval, &end, 10);
+				if (*end == '\0' &&
+				    ver < SPA_VERSION_FEATURES) {
+					enable_all_pool_feat = B_FALSE;
+				}
+			}
 			break;
 		case 'O':
 			if ((propval = strchr(optarg, '=')) == NULL) {
@@ -835,7 +867,6 @@ zpool_do_create(int argc, char **argv)
 		goto errout;
 	}
 
-
 	if (altroot != NULL && altroot[0] != '/') {
 		(void) fprintf(stderr, gettext("invalid alternate root '%s': "
 		    "must be an absolute path\n"), altroot);
@@ -917,6 +948,27 @@ zpool_do_create(int argc, char **argv)
 		/*
 		 * Hand off to libzfs.
 		 */
+		if (enable_all_pool_feat) {
+			int i;
+			for (i = 0; i < SPA_FEATURES; i++) {
+				char propname[MAXPATHLEN];
+				zfeature_info_t *feat = &spa_feature_table[i];
+
+				(void) snprintf(propname, sizeof (propname),
+				    "feature@%s", feat->fi_uname);
+
+				/*
+				 * Skip feature if user specified it manually
+				 * on the command line.
+				 */
+				if (nvlist_exists(props, propname))
+					continue;
+
+				if (add_prop_list(propname, ZFS_FEATURE_ENABLED,
+				    &props, B_TRUE) != 0)
+					goto errout;
+			}
+		}
 		if (zpool_create(g_zfs, poolname,
 		    nvroot, props, fsprops) == 0) {
 			zfs_handle_t *pool = zfs_open(g_zfs, poolname,
@@ -1249,6 +1301,10 @@ print_status_config(zpool_handle_t *zhp, const char *name, nvlist_t *nv,
 			(void) printf(gettext("newer version"));
 			break;
 
+		case VDEV_AUX_UNSUP_FEAT:
+			(void) printf(gettext("unsupported feature(s)"));
+			break;
+
 		case VDEV_AUX_SPARED:
 			verify(nvlist_lookup_uint64(nv, ZPOOL_CONFIG_GUID,
 			    &cb.cb_guid) == 0);
@@ -1366,6 +1422,10 @@ print_import_config(const char *name, nvlist_t *nv, int namewidth, int depth)
 			(void) printf(gettext("newer version"));
 			break;
 
+		case VDEV_AUX_UNSUP_FEAT:
+			(void) printf(gettext("unsupported feature(s)"));
+			break;
+
 		case VDEV_AUX_ERR_EXCEEDED:
 			(void) printf(gettext("too many errors"));
 			break;
@@ -1532,6 +1592,20 @@ show_import(nvlist_t *config)
 		    "incompatible version.\n"));
 		break;
 
+	case ZPOOL_STATUS_UNSUP_FEAT_READ:
+		(void) printf(gettext("status: The pool uses the following "
+		    "feature(s) not supported on this sytem:\n"));
+		zpool_print_unsup_feat(config);
+		break;
+
+	case ZPOOL_STATUS_UNSUP_FEAT_WRITE:
+		(void) printf(gettext("status: The pool can only be accessed "
+		    "in read-only mode on this system. It\n\tcannot be "
+		    "accessed in read-write mode because it uses the "
+		    "following\n\tfeature(s) not supported on this system:\n"));
+		zpool_print_unsup_feat(config);
+		break;
+
 	case ZPOOL_STATUS_HOSTID_MISMATCH:
 		(void) printf(gettext(" status: The pool was last accessed by "
 		    "another system.\n"));
@@ -1589,6 +1663,20 @@ show_import(nvlist_t *config)
 			    "newer\n\tsoftware, or recreate the pool from "
 			    "backup.\n"));
 			break;
+		case ZPOOL_STATUS_UNSUP_FEAT_READ:
+			(void) printf(gettext("action: The pool cannot be "
+			    "imported. Access the pool on a system that "
+			    "supports\n\tthe required feature(s), or recreate "
+			    "the pool from backup.\n"));
+			break;
+		case ZPOOL_STATUS_UNSUP_FEAT_WRITE:
+			(void) printf(gettext("action: The pool cannot be "
+			    "imported in read-write mode. Import the pool "
+			    "with\n"
+			    "\t\"-o readonly=on\", access the pool on a system "
+			    "that supports the\n\trequired feature(s), or "
+			    "recreate the pool from backup.\n"));
+			break;
 		case ZPOOL_STATUS_MISSING_DEV_R:
 		case ZPOOL_STATUS_MISSING_DEV_NR:
 		case ZPOOL_STATUS_BAD_GUID_SUM:
@@ -1664,9 +1752,9 @@ do_import(nvlist_t *config, const char *newname, const char *mntopts,
 	    ZPOOL_CONFIG_POOL_STATE, &state) == 0);
 	verify(nvlist_lookup_uint64(config,
 	    ZPOOL_CONFIG_VERSION, &version) == 0);
-	if (version > SPA_VERSION) {
+	if (!SPA_VERSION_IS_SUPPORTED(version)) {
 		(void) fprintf(stderr, gettext("cannot import '%s': pool "
-		    "is formatted using a newer ZFS version\n"), name);
+		    "is formatted using an unsupported ZFS version\n"), name);
 		return (1);
 	} else if (state != POOL_STATE_EXPORTED &&
 	    !(flags & ZFS_IMPORT_ANY_HOST)) {
@@ -2601,15 +2689,13 @@ static void
 print_header(list_cbdata_t *cb)
 {
 	zprop_list_t *pl = cb->cb_proplist;
+	char headerbuf[ZPOOL_MAXPROPLEN];
 	const char *header;
 	boolean_t first = B_TRUE;
 	boolean_t right_justify;
 	size_t width = 0;
 
 	for (; pl != NULL; pl = pl->pl_next) {
-		if (pl->pl_prop == ZPROP_INVAL)
-			continue;
-
 		width = pl->pl_width;
 		if (first && cb->cb_verbose) {
 			/*
@@ -2624,8 +2710,18 @@ print_header(list_cbdata_t *cb)
 		else
 			first = B_FALSE;
 
-		header = zpool_prop_column_name(pl->pl_prop);
-		right_justify = zpool_prop_align_right(pl->pl_prop);
+		right_justify = B_FALSE;
+		if (pl->pl_prop != ZPROP_INVAL) {
+			header = zpool_prop_column_name(pl->pl_prop);
+			right_justify = zpool_prop_align_right(pl->pl_prop);
+		} else {
+			int i;
+
+			for (i = 0; pl->pl_user_prop[i] != '\0'; i++)
+				headerbuf[i] = toupper(pl->pl_user_prop[i]);
+			headerbuf[i] = '\0';
+			header = headerbuf;
+		}
 
 		if (pl->pl_next == NULL && !right_justify)
 			(void) printf("%s", header);
@@ -2685,6 +2781,11 @@ print_pool(zpool_handle_t *zhp, list_cbdata_t *cb)
 				propstr = property;
 
 			right_justify = zpool_prop_align_right(pl->pl_prop);
+		} else if ((zpool_prop_feature(pl->pl_user_prop) ||
+		    zpool_prop_unsupported(pl->pl_user_prop)) &&
+		    zpool_prop_get_feature(zhp, pl->pl_user_prop, property,
+		    sizeof (property)) == 0) {
+			propstr = property;
 		} else {
 			propstr = "-";
 		}
@@ -4024,6 +4125,31 @@ status_callback(zpool_handle_t *zhp, void *data)
 		    "backup.\n"));
 		break;
 
+	case ZPOOL_STATUS_UNSUP_FEAT_READ:
+		(void) printf(gettext("status: The pool cannot be accessed on "
+		    "this system because it uses the\n\tfollowing feature(s) "
+		    "not supported on this system:\n"));
+		zpool_print_unsup_feat(config);
+		(void) printf("\n");
+		(void) printf(gettext("action: Access the pool from a system "
+		    "that supports the required feature(s),\n\tor restore the "
+		    "pool from backup.\n"));
+		break;
+
+	case ZPOOL_STATUS_UNSUP_FEAT_WRITE:
+		(void) printf(gettext("status: The pool can only be accessed "
+		    "in read-only mode on this system. It\n\tcannot be "
+		    "accessed in read-write mode because it uses the "
+		    "following\n\tfeature(s) not supported on this system:\n"));
+		zpool_print_unsup_feat(config);
+		(void) printf("\n");
+		(void) printf(gettext("action: The pool cannot be accessed in "
+		    "read-write mode. Import the pool with\n"
+		    "\t\"-o readonly=on\", access the pool from a system that "
+		    "supports the\n\trequired feature(s), or restore the "
+		    "pool from backup.\n"));
+		break;
+
 	case ZPOOL_STATUS_FAULTED_DEV_R:
 		(void) printf(gettext("status: One or more devices are "
 		    "faulted in response to persistent errors.\n\tSufficient "
@@ -4274,7 +4400,8 @@ upgrade_cb(zpool_handle_t *zhp, void *arg)
 	verify(nvlist_lookup_uint64(config, ZPOOL_CONFIG_VERSION,
 	    &version) == 0);
 
-	if (!cbp->cb_newer && version < SPA_VERSION) {
+	if (!cbp->cb_newer && SPA_VERSION_IS_SUPPORTED(version) &&
+	    version != SPA_VERSION) {
 		if (!cbp->cb_all) {
 			if (cbp->cb_first) {
 				(void) printf(gettext("The following pools are "
@@ -4303,13 +4430,14 @@ upgrade_cb(zpool_handle_t *zhp, void *arg)
 				}
 			}
 		}
-	} else if (cbp->cb_newer && version > SPA_VERSION) {
+	} else if (cbp->cb_newer && !SPA_VERSION_IS_SUPPORTED(version)) {
 		assert(!cbp->cb_all);
 
 		if (cbp->cb_first) {
 			(void) printf(gettext("The following pools are "
-			    "formatted using a newer software version and\n"
-			    "cannot be accessed on the current system.\n\n"));
+			    "formatted using an unsupported software version "
+			    "and\ncannot be accessed on the current "
+			    "system.\n\n"));
 			(void) printf(gettext("VER  POOL\n"));
 			(void) printf(gettext("---  ------------\n"));
 			cbp->cb_first = B_FALSE;
@@ -4397,8 +4525,8 @@ zpool_do_upgrade(int argc, char **argv)
 			break;
 		case 'V':
 			cb.cb_version = strtoll(optarg, &end, 10);
-			if (*end != '\0' || cb.cb_version > SPA_VERSION ||
-			    cb.cb_version < SPA_VERSION_1) {
+			if (*end != '\0' ||
+			    !SPA_VERSION_IS_SUPPORTED(cb.cb_version)) {
 				(void) fprintf(stderr,
 				    gettext("invalid version '%s'\n"), optarg);
 				usage(B_FALSE);
@@ -4443,8 +4571,8 @@ zpool_do_upgrade(int argc, char **argv)
 		}
 	}
 
-	(void) printf(gettext("This system is currently running "
-	    "ZFS pool version %llu.\n\n"), SPA_VERSION);
+	(void) printf(gettext("This system supports ZFS pool feature "
+	    "flags.\n\n"));
 	cb.cb_first = B_TRUE;
 	if (showversions) {
 		(void) printf(gettext("The following versions are "
@@ -4705,13 +4833,26 @@ get_callback(zpool_handle_t *zhp, void *data)
 		    pl == cbp->cb_proplist)
 			continue;
 
-		if (zpool_get_prop(zhp, pl->pl_prop,
-		    value, sizeof (value), &srctype) != 0)
-			continue;
+		if (pl->pl_prop == ZPROP_INVAL &&
+		    (zpool_prop_feature(pl->pl_user_prop) ||
+		    zpool_prop_unsupported(pl->pl_user_prop))) {
+			srctype = ZPROP_SRC_LOCAL;
+
+			if (zpool_prop_get_feature(zhp, pl->pl_user_prop,
+			    value, sizeof (value)) == 0) {
+				zprop_print_one_property(zpool_get_name(zhp),
+				    cbp, pl->pl_user_prop, value, srctype,
+				    NULL, NULL);
+			}
+		} else {
+			if (zpool_get_prop(zhp, pl->pl_prop, value,
+			    sizeof (value), &srctype) != 0)
+				continue;
 
-		zprop_print_one_property(zpool_get_name(zhp), cbp,
-		    zpool_prop_to_name(pl->pl_prop), value, srctype, NULL,
-		    NULL);
+			zprop_print_one_property(zpool_get_name(zhp), cbp,
+			    zpool_prop_to_name(pl->pl_prop), value, srctype,
+			    NULL, NULL);
+		}
 	}
 	return (0);
 }
@@ -4723,8 +4864,11 @@ zpool_do_get(int argc, char **argv)
 	zprop_list_t fake_name = { 0 };
 	int ret;
 
-	if (argc < 3)
+	if (argc < 2) {
+		(void) fprintf(stderr, gettext("missing property "
+		    "argument\n"));
 		usage(B_FALSE);
+	}
 
 	cb.cb_first = B_TRUE;
 	cb.cb_sources = ZPROP_SRC_ALL;
@@ -4734,7 +4878,7 @@ zpool_do_get(int argc, char **argv)
 	cb.cb_columns[3] = GET_COL_SOURCE;
 	cb.cb_type = ZFS_TYPE_POOL;
 
-	if (zprop_get_list(g_zfs, argv[1],  &cb.cb_proplist,
+	if (zprop_get_list(g_zfs, argv[1], &cb.cb_proplist,
 	    ZFS_TYPE_POOL) != 0)
 		usage(B_FALSE);