3 files changed, 111 insertions, 13 deletions
diff --git a/cddl/contrib/opensolaris/cmd/zfs/zfs.8 b/cddl/contrib/opensolaris/cmd/zfs/zfs.8
index 11bf80b..1ab9a54 100644
--- a/cddl/contrib/opensolaris/cmd/zfs/zfs.8
+++ b/cddl/contrib/opensolaris/cmd/zfs/zfs.8
@@ -116,6 +116,9 @@
 .Oo Fl S Ar property Oc Ns ...
 .Ar filesystem Ns | Ns Ar volume Ns | Ns Ar snapshot | Ns Ar bookmark Ns ...
 .Nm
+.Cm remap
+.Ar filesystem Ns | Ns Ar volume
+.Nm
 .Cm set
 .Ar property Ns = Ns Ar value Oo Ar property Ns = Ns Ar value Oc Ns ...
 .Ar filesystem Ns | Ns Ar volume Ns | Ns Ar snapshot Ns ...
@@ -2268,6 +2271,17 @@ option was not specified.
 .El
 .It Xo
 .Nm
+.Cm remap
+.Ar filesystem Ns | Ns Ar volume
+.Xc
+.Pp
+Remap the indirect blocks in the given fileystem or volume so that they no
+longer reference blocks on previously removed vdevs and we can eventually
+shrink the size of the indirect mapping objects for the previously removed
+vdevs. Note that remapping all blocks might not be possible and that
+references from snapshots will still exist and cannot be remapped.
+.It Xo
+.Nm
 .Cm upgrade
 .Op Fl v
 .Xc
diff --git a/cddl/contrib/opensolaris/cmd/zpool/zpool-features.7 b/cddl/contrib/opensolaris/cmd/zpool/zpool-features.7
index 18e2421..cc4e7c3 100644
--- a/cddl/contrib/opensolaris/cmd/zpool/zpool-features.7
+++ b/cddl/contrib/opensolaris/cmd/zpool/zpool-features.7
@@ -448,6 +448,40 @@ when the "zpool checkpoint" command is used to checkpoint the pool.
 The feature will only return back to being
 .Sy enabled
 when the pool is rewound or the checkpoint has been discarded.
+.It Sy device_removal
+.Bl -column "READ\-ONLY COMPATIBLE" "com.delphix:device_removal"
+.It GUID Ta com.delphix:device_removal
+.It READ\-ONLY COMPATIBLE Ta no
+.It DEPENDENCIES Ta none
+.El
+.Pp
+This feature enables the "zpool remove" subcommand to remove top-level
+vdevs, evacuating them to reduce the total size of the pool.
+.Pp
+This feature becomes
+.Sy active
+when the "zpool remove" command is used
+on a top-level vdev, and will never return to being 
+.Sy enabled .
+.It Sy obsolete_counts
+.Bl -column "READ\-ONLY COMPATIBLE" "com.delphix:obsolete_counts"
+.It GUID Ta com.delphix:obsolete_counts
+.It READ\-ONLY COMPATIBLE Ta yes
+.It DEPENDENCIES Ta device_removal
+.El
+.Pp
+This feature is an enhancement of device_removal, which will over time
+reduce the memory used to track removed devices.  When indirect blocks
+are freed or remapped, we note that their part of the indirect mapping
+is "obsolete", i.e. no longer needed.  See also the "zfs remap"
+subcommand in
+.Xr zfs 8 .
+
+This feature becomes
+.Sy active
+when the "zpool remove" command is
+used on a top-level vdev, and will never return to being
+.Sy enabled .
 .It Sy large_blocks
 .Bl -column "READ\-ONLY COMPATIBLE" "org.open-zfs:large_block"
 .It GUID Ta org.open-zfs:large_block
diff --git a/cddl/contrib/opensolaris/cmd/zpool/zpool.8 b/cddl/contrib/opensolaris/cmd/zpool/zpool.8
index 3e9cd13..168b936 100644
--- a/cddl/contrib/opensolaris/cmd/zpool/zpool.8
+++ b/cddl/contrib/opensolaris/cmd/zpool/zpool.8
@@ -21,7 +21,7 @@
 .\" Copyright (c) 2010, Sun Microsystems, Inc. All Rights Reserved.
 .\" Copyright (c) 2011, Justin T. Gibbs <gibbs@FreeBSD.org>
 .\" Copyright (c) 2012, Glen Barber <gjb@FreeBSD.org>
-.\" Copyright (c) 2013 by Delphix. All Rights Reserved.
+.\" Copyright (c) 2012, 2017 by Delphix. All Rights Reserved.
 .\" Copyright 2017 Nexenta Systems, Inc.
 .\" Copyright (c) 2017 Datto Inc.
 .\"
@@ -149,8 +149,13 @@
 .Ar pool
 .Nm
 .Cm remove
+.Op Fl np
 .Ar pool device ...
 .Nm
+.Cm remove
+.Fl s
+.Ar pool
+.Nm
 .Cm reopen
 .Ar pool
 .Nm
@@ -508,8 +513,8 @@ Multiple log devices can also be specified, and they can be mirrored. See the
 section for an example of mirroring multiple log devices.
 .Pp
 Log devices can be added, replaced, attached, detached, imported and exported
-as part of the larger pool. Mirrored log devices can be removed by specifying
-the top-level mirror for the log.
+as part of the larger pool.
+Mirrored devices can be removed by specifying the top-level mirror vdev.
 .Ss Cache devices
 Devices can be added to a storage pool as "cache devices." These devices
 provide an additional layer of caching between main memory and disk. For
@@ -1574,17 +1579,53 @@ devices in this pool are online and healthy before performing this action.
 .It Xo
 .Nm
 .Cm remove
+.Op Fl np
 .Ar pool device ...
 .Xc
 .Pp
-Removes the specified device from the pool. This command currently only
-supports removing hot spares, cache, and log devices. A mirrored log device can
-be removed by specifying the top-level mirror for the log. Non-log devices that
-are part of a mirrored configuration can be removed using the
+Removes the specified device from the pool.
+This command currently only supports removing hot spares, cache, log
+devices and mirrored top-level vdevs (mirror of leaf devices); but not raidz.
+.sp
+Removing a top-level vdev reduces the total amount of space in the storage pool.
+The specified device will be evacuated by copying all allocated space from it to
+the other devices in the pool.
+In this case, the
+.Nm zpool Cm remove
+command initiates the removal and returns, while the evacuation continues in
+the background.
+The removal progress can be monitored with
+.Nm zpool Cm status.
+This feature must be enabled to be used, see
+.Xr zpool-features 5
+.Pp
+A mirrored top-level device (log or data) can be removed by specifying the
+top-level mirror for the same.
+Non-log devices or data devices that are part of a mirrored configuration can
+be removed using the
 .Qq Nm Cm detach
-command. Non-redundant and
-.No raidz
-devices cannot be removed from a pool.
+command.
+.Bl -tag -width Ds
+.It Fl n
+Do not actually perform the removal ("no-op").
+Instead, print the estimated amount of memory that will be used by the
+mapping table after the removal completes.
+This is nonzero only for top-level vdevs.
+.El
+.Bl -tag -width Ds
+.It Fl p
+Used in conjunction with the
+.Fl n
+flag, displays numbers as parsable (exact) values.
+.El
+.It Xo
+.Nm
+.Cm remove
+.Fl s
+.Ar pool
+.Xc
+.Pp
+Stops and cancels an in-progress removal of a top-level vdev.
 .It Xo
 .Nm
 .Cm reopen
@@ -2068,11 +2109,13 @@ data      23.9G  14.6G  9.30G    48%         -    61%  1.00x  ONLINE  -
 .Ed
 .It Xo
 .Sy Example 16
-Removing a Mirrored Log Device
+Removing a Mirrored top-level (Log or Data) Device
 .Xc
 .Pp
-The following command removes the mirrored log device
-.Em mirror-2 .
+The following commands remove the mirrored log device
+.Sy mirror-2
+and mirrored top-level data device
+.Sy mirror-1 .
 .Pp
 Given this configuration:
 .Bd -literal -offset 2n
@@ -2101,6 +2144,13 @@ is:
 .Bd -literal -offset 2n
 .Li # Ic zpool remove tank mirror-2
 .Ed
+.Pp
+The command to remove the mirrored data
+.Em mirror-1
+is:
+.Bd -literal -offset 2n
+.Li # Ic zpool remove tank mirror-1
+.Ed
 .It Xo
 .Sy Example 17
 Recovering a Faulted