Add snapshots to the fast filesystem. Most of the changes support

the gating of system calls that cause modifications to the underlying
filesystem. The gating can be enabled by any filesystem that needs
to consistently suspend operations by adding the vop_stdgetwritemount
to their set of vnops. Once gating is enabled, the function
vfs_write_suspend stops all new write operations to a filesystem,
allows any filesystem modifying system calls already in progress
to complete, then sync's the filesystem to disk and returns. The
function vfs_write_resume allows the suspended write operations to
begin again. Gating is not added by default for all filesystems as
for SMP systems it adds two extra locks to such critical kernel
paths as the write system call. Thus, gating should only be added
as needed.

Details on the use and current status of snapshots in FFS can be
found in /sys/ufs/ffs/README.snapshot so for brevity and timelyness
is not included here. Unless and until you create a snapshot file,
these changes should have no effect on your system (famous last words).

15 files changed, 1365 insertions, 184 deletions

diff --git a/sys/ufs/ffs/README.snapshot b/sys/ufs/ffs/README.snapshot
new file mode 100644
index 0000000..f3177c3
--- /dev/null
+++ b/sys/ufs/ffs/README.snapshot
@@ -0,0 +1,112 @@
+$FreeBSD$
+
+Soft Updates Status
+
+As is detailed in the operational information below, snapshots
+are definitely alpha-test code and are NOT yet ready for production
+use. Much remains to be done to make them really useful, but I
+wanted to let folks get a chance to try it out and start reporting
+bugs and other shortcomings. Such reports should be sent to
+Kirk McKusick <mckusick@mckusick.com>.
+
+
+Snapshot Copyright Restrictions
+
+Snapshots have been introduced to FreeBSD with a `Berkeley-style'
+copyright. The file implementing snapshots resides in the sys/ufs/ffs
+directory and is compiled into the generic kernel by default.
+
+
+Using Snapshots
+
+To create a snapshot of your /var filesystem, run the command:
+
+	mount -u -o snapshot /var/snapshot/snap1 /var
+
+This command will take a snapshot of your /var filesystem and
+leave it in the file /var/snapshot/snap1. Note that snapshot
+files must be created in the filesystem that is being snapshotted.
+I use the convention of putting a `snapshot' directory at the
+root of each filesystem into which I can place snapshots.
+You may create up to 20 snapshots per filesystem. Active snapshots
+are recorded in the superblock, so they persist across unmount
+and remount operations and across system reboots. When your
+are done with a snapshot, it can be removed with the `rm'
+command. Snapshots may be removed in any order, however you
+may not get back all the space contained in the snapshot as
+another snapshot may claim some of the blocks that it is releasing. 
+Note that the `schg' flag is set on snapshots to ensure that
+not even the root user can write to them. The unlink command
+makes an exception for snapshot files in that it allows them
+to be removed even though they have the `schg' flag set, so it
+is not necessary to clear the `schg' flag before removing a
+snapshot file.
+
+Once you have taken a snapshot, there are three interesting
+things that you can do with it:
+
+1) Run fsck on the snapshot file. Assuming that the filesystem
+   was clean when it was mounted, you should always get a clean
+   (and unchanging) result from running fsck on the snapshot.
+   If you are running with soft updates and rebooted after a
+   crash without cleaning up the filesystem, then fsck of the
+   snapshot may find missing blocks and inodes or inodes with
+   link counts that are too high. I have not yet added the
+   system calls to allow fsck to add these missing resources
+   back to the filesystem - that will be added once the basic
+   snapshot code is working properly. So, view those reports
+   as informational for now.
+
+2) Run dump on the snapshot. You will get a dump that is
+   consistent with the filesystem as of the timestamp of the
+   snapshot. Note that I have not yet changed dump to set the
+   dumpdates file correctly, so do not use this feature in
+   production until that fix is made.
+
+3) Mount the snapshot as a frozen image of the filesystem.
+   To mount the snapshot /var/snapshot/snap1:
+
+	vnconfig -c vn0c /var/snapshot/snap1
+	mount -r /dev/vn0c /mnt
+
+   You can now cruise around your frozen /var filesystem
+   at /mnt. Everything will be in the same state that it
+   was at the time the snapshot was taken. The one exception
+   is that any earlier snapshots will appear as zero length
+   files. When you are done with the mounted snapshot:
+
+	umount /mnt
+	vnconfig -u vn0c
+
+   Note that under some circumstances, the process accessing
+   the frozen filesystem may deadlock. I am aware of this
+   problem, but the solution is not simple. It requires
+   using buffer read locks rather than exclusive locks when
+   traversing the inode indirect blocks. Until this problem
+   is fixed, you should avoid putting mounted snapshots into
+   production.
+
+
+Performance
+
+It takes about 30 seconds to create a snapshot of an 8Gb filesystem.
+Of that time 25 seconds is spent in preparation; filesystem activity
+is only suspended for the final 5 seconds of that period. Snapshot
+removal of an 8Gb filesystem takes about two minutes. Filesystem
+activity is never suspended during snapshot removal.
+
+The suspend time may be expanded by several minutes if a process
+is in the midst of removing many files as all the soft updates
+backlog must be cleared. Generally snapshots do not slow the system
+down appreciably except when removing many small files (i.e., any
+file less than 96Kb whose last block is a fragment) that are claimed
+by a snapshot. Here, the snapshot code must make a copy of every
+released fragment which slows the rate of file removal to about
+twenty files per second once the soft updates backlog limit is
+reached.
+
+
+How Snapshots Work
+
+For more general information on snapshots, please see:
+	http://www.mckusick.com/softdep/
diff --git a/sys/ufs/ffs/ffs_alloc.c b/sys/ufs/ffs/ffs_alloc.c
index 1f24b2b..5efe0e7 100644
--- a/sys/ufs/ffs/ffs_alloc.c
+++ b/sys/ufs/ffs/ffs_alloc.c
@@ -186,6 +186,8 @@ ffs_realloccg(ip, lbprev, bpref, osize, nsize, cred, bpp)
 	*bpp = 0;
 	fs = ip->i_fs;
 #ifdef DIAGNOSTIC
+	if (ITOV(ip)->v_mount->mnt_kern_flag & MNTK_SUSPENDED)
+		panic("ffs_realloccg: allocation on suspended filesystem");
 	if ((u_int)osize > fs->fs_bsize || fragoff(fs, osize) != 0 ||
 	    (u_int)nsize > fs->fs_bsize || fragoff(fs, nsize) != 0) {
 		printf(
@@ -763,6 +765,10 @@ ffs_hashalloc(ip, cg, pref, size, allocator)
 	long result;	/* XXX why not same type as we return? */
 	int i, icg = cg;
 
+#ifdef DIAGNOSTIC
+	if (ITOV(ip)->v_mount->mnt_kern_flag & MNTK_SUSPENDED)
+		panic("ffs_hashalloc: allocation on suspended filesystem");
+#endif
 	fs = ip->i_fs;
 	/*
 	 * 1: preferred cylinder group
@@ -1311,9 +1317,13 @@ ffs_blkfree(ip, bno, size)
 	ufs_daddr_t blkno;
 	int i, error, cg, blk, frags, bbase;
 	u_int8_t *blksfree;
+	struct vnode *vp;
 
 	fs = ip->i_fs;
-	VOP_FREEBLKS(ip->i_devvp, fsbtodb(fs, bno), size);
+#ifdef DIAGNOSTIC
+	if ((vp = ITOV(ip)) != NULL && vp->v_mount != NULL &&
+	    (vp->v_mount->mnt_kern_flag & MNTK_SUSPENDED))
+		panic("ffs_blkfree: deallocation on suspended filesystem");
 	if ((u_int)size > fs->fs_bsize || fragoff(fs, size) != 0 ||
 	    fragnum(fs, bno) + numfrags(fs, size) > fs->fs_frag) {
 		printf("dev=%s, bno = %ld, bsize = %ld, size = %ld, fs = %s\n",
@@ -1321,6 +1331,11 @@ ffs_blkfree(ip, bno, size)
 		    fs->fs_fsmnt);
 		panic("ffs_blkfree: bad size");
 	}
+#endif
+	if ((ip->i_devvp->v_flag & VCOPYONWRITE) &&
+	    ffs_snapblkfree(ip, bno, size))
+		return;
+	VOP_FREEBLKS(ip->i_devvp, fsbtodb(fs, bno), size);
 	cg = dtog(fs, bno);
 	if ((u_int)bno >= fs->fs_size) {
 		printf("bad block %ld, ino %lu\n",
diff --git a/sys/ufs/ffs/ffs_balloc.c b/sys/ufs/ffs/ffs_balloc.c
index 28cc1ed..92fe379 100644
--- a/sys/ufs/ffs/ffs_balloc.c
+++ b/sys/ufs/ffs/ffs_balloc.c
@@ -125,6 +125,8 @@ ffs_balloc(ap)
 	 * The first NDADDR blocks are direct blocks
 	 */
 	if (lbn < NDADDR) {
+		if (flags & B_METAONLY)
+			panic("ffs_balloc: B_METAONLY for direct block");
 		nb = ip->i_db[lbn];
 		if (nb != 0 && ip->i_size >= smalllblktosize(fs, lbn + 1)) {
 			error = bread(vp, lbn, fs->fs_bsize, NOCRED, &bp);
@@ -289,6 +291,13 @@ ffs_balloc(ap)
 		}
 	}
 	/*
+	 * If asked only for the indirect block, then return it.
+	 */
+	if (flags & B_METAONLY) {
+		*ap->a_bpp = bp;
+		return (0);
+	}
+	/*
 	 * Get the data block, allocating if necessary.
 	 */
 	if (nb == 0) {
diff --git a/sys/ufs/ffs/ffs_extern.h b/sys/ufs/ffs/ffs_extern.h
index fe7391b..8e011bb 100644
--- a/sys/ufs/ffs/ffs_extern.h
+++ b/sys/ufs/ffs/ffs_extern.h
@@ -67,6 +67,7 @@ struct vop_balloc_args;
 struct vop_bmap_args;
 struct vop_fsync_args;
 struct vop_reallocblks_args;
+struct vop_copyonwrite_args;
 
 int	ffs_alloc __P((struct inode *,
 	    ufs_daddr_t, ufs_daddr_t, int, struct ucred *, ufs_daddr_t *));
@@ -76,6 +77,7 @@ void	ffs_blkfree __P((struct inode *, ufs_daddr_t, long));
 ufs_daddr_t ffs_blkpref __P((struct inode *, ufs_daddr_t, int, ufs_daddr_t *));
 int	ffs_bmap __P((struct vop_bmap_args *));
 void	ffs_clrblock __P((struct fs *, u_char *, ufs_daddr_t));
+int	ffs_copyonwrite __P((struct vop_copyonwrite_args *ap));
 int	ffs_fhtovp __P((struct mount *, struct fid *, struct vnode **));
 int	ffs_flushfiles __P((struct mount *, int, struct proc *));
 void	ffs_fragacct __P((struct fs *, int, int32_t [], int));
@@ -89,6 +91,10 @@ int	ffs_reallocblks __P((struct vop_reallocblks_args *));
 int	ffs_realloccg __P((struct inode *,
 	    ufs_daddr_t, ufs_daddr_t, int, int, struct ucred *, struct buf **));
 void	ffs_setblock __P((struct fs *, u_char *, ufs_daddr_t));
+int	ffs_snapblkfree __P((struct inode *freeip, ufs_daddr_t bno, long size));
+int	ffs_snapshot __P((struct mount *mp, char *snapfile));
+void	ffs_snapshot_mount __P((struct mount *mp));
+void	ffs_snapshot_unmount __P((struct mount *mp));
 int	ffs_statfs __P((struct mount *, struct statfs *, struct proc *));
 int	ffs_sync __P((struct mount *, int, struct ucred *, struct proc *));
 int	ffs_truncate __P((struct vnode *, off_t, int, struct ucred *, struct proc *));
diff --git a/sys/ufs/ffs/ffs_snapshot.c b/sys/ufs/ffs/ffs_snapshot.c
new file mode 100644
index 0000000..73da537
--- /dev/null
+++ b/sys/ufs/ffs/ffs_snapshot.c
@@ -0,0 +1,1028 @@
+/*
+ * Copyright 2000 Marshall Kirk McKusick. All Rights Reserved.
+ *
+ * Further information about snapshots can be obtained from:
+ *
+ *	Marshall Kirk McKusick		http://www.mckusick.com/softdep/
+ *	1614 Oxford Street		mckusick@mckusick.com
+ *	Berkeley, CA 94709-1608		+1-510-843-9542
+ *	USA
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY MARSHALL KIRK MCKUSICK ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+ * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+ * DISCLAIMED.  IN NO EVENT SHALL MARSHALL KIRK MCKUSICK BE LIABLE FOR
+ * ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ *	@(#)ffs_snapshot.c	8.10 (McKusick) 7/11/00
+ * $FreeBSD$
+ */
+
+#include <sys/param.h>
+#include <sys/systm.h>
+#include <sys/bio.h>
+#include <sys/buf.h>
+#include <sys/proc.h>
+#include <sys/namei.h>
+#include <sys/stat.h>
+#include <sys/malloc.h>
+#include <sys/mount.h>
+#include <sys/resource.h>
+#include <sys/resourcevar.h>
+#include <sys/vnode.h>
+
+#include <ufs/ufs/extattr.h>
+#include <ufs/ufs/quota.h>
+#include <ufs/ufs/ufsmount.h>
+#include <ufs/ufs/inode.h>
+#include <ufs/ufs/ufs_extern.h>
+
+#include <ufs/ffs/fs.h>
+#include <ufs/ffs/ffs_extern.h>
+
+#define KERNCRED proc0.p_ucred
+#define CURPROC curproc
+#define DEBUG
+
+static int indiracct __P((struct vnode *, struct vnode *, int, ufs_daddr_t,
+	int, int, int, int));
+static int snapacct __P((struct vnode *, ufs_daddr_t *, ufs_daddr_t *));
+static int readblock __P((struct buf *, daddr_t));
+
+#ifdef DEBUG
+#include <sys/sysctl.h>
+int snapdebug = 0;
+SYSCTL_INT(_debug, OID_AUTO, snapdebug, CTLFLAG_RW, &snapdebug, 0, "");
+#endif /* DEBUG */
+
+/*
+ * Create a snapshot file and initialize it for the filesystem.
+ */
+int
+ffs_snapshot(mp, snapfile)
+	struct mount *mp;
+	char *snapfile;
+{
+	ufs_daddr_t rlbn;
+	ufs_daddr_t lbn, blkno, copyblkno, inoblks[FSMAXSNAP];
+	int error, cg, snaploc, indiroff, numblks;
+	int i, size, base, len, loc, inoblkcnt;
+	int blksperindir, flag = mp->mnt_flag;
+	struct fs *fs = VFSTOUFS(mp)->um_fs;
+	struct proc *p = CURPROC;
+	struct inode *devip, *ip, *xp;
+	struct buf *bp, *nbp, *ibp;
+	struct vnode *vp, *devvp;
+	struct nameidata nd;
+	struct mount *wrtmp;
+	struct dinode *dip;
+	struct vattr vat;
+	struct cg *cgp;
+
+	/*
+	 * Need to serialize access to snapshot code per filesystem.
+	 */
+	/*
+	 * Assign a snapshot slot in the superblock.
+	 */
+	for (snaploc = 0; snaploc < FSMAXSNAP; snaploc++)
+		if (fs->fs_snapinum[snaploc] == 0)
+			break;
+	if (snaploc == FSMAXSNAP)
+		return (ENOSPC);
+	/*
+	 * Create the snapshot file.
+	 */
+restart:
+	NDINIT(&nd, CREATE, LOCKPARENT | LOCKLEAF, UIO_USERSPACE, snapfile, p);
+	if ((error = namei(&nd)) != 0)
+		return (error);
+	if (nd.ni_vp != NULL) {
+		vput(nd.ni_vp);
+		error = EEXIST;
+	}
+	if (nd.ni_dvp->v_mount != mp)
+		error = EXDEV;
+	if (error) {
+		NDFREE(&nd, NDF_ONLY_PNBUF);
+		if (nd.ni_dvp == nd.ni_vp)
+			vrele(nd.ni_dvp);
+		else
+			vput(nd.ni_dvp);
+		return (error);
+	}
+	VATTR_NULL(&vat);
+	vat.va_type = VREG;
+	vat.va_mode = S_IRUSR;
+	vat.va_vaflags |= VA_EXCLUSIVE;
+	if (VOP_GETWRITEMOUNT(nd.ni_dvp, &wrtmp))
+		wrtmp = NULL;
+	if (wrtmp != mp)
+		panic("ffs_snapshot: mount mismatch");
+	if (vn_start_write(wrtmp, V_NOWAIT) != 0) {
+		NDFREE(&nd, NDF_ONLY_PNBUF);
+		vput(nd.ni_dvp);
+		if ((error = vn_start_write(wrtmp, V_XSLEEP | PCATCH)) != 0)
+			return (error);
+		goto restart;
+	}
+	VOP_LEASE(nd.ni_dvp, p, KERNCRED, LEASE_WRITE);
+	error = VOP_CREATE(nd.ni_dvp, &nd.ni_vp, &nd.ni_cnd, &vat);
+	vput(nd.ni_dvp);
+	if (error) {
+		NDFREE(&nd, NDF_ONLY_PNBUF);
+		vn_finished_write(wrtmp);
+		return (error);
+	}
+	vp = nd.ni_vp;
+	ip = VTOI(vp);
+	devvp = ip->i_devvp;
+	devip = VTOI(devvp);
+	/*
+	 * Allocate and copy the last block contents so as to be able
+	 * to set size to that of the filesystem.
+	 */
+	numblks = howmany(fs->fs_size, fs->fs_frag);
+	error = VOP_BALLOC(vp, lblktosize(fs, (off_t)(numblks - 1)),
+	    fs->fs_bsize, KERNCRED, B_CLRBUF, &bp);
+	if (error)
+		goto out;
+	ip->i_size = lblktosize(fs, (off_t)numblks);
+	ip->i_flag |= IN_CHANGE | IN_UPDATE;
+	if ((error = readblock(bp, numblks - 1)) != 0)
+		goto out;
+	bawrite(bp);
+	/*
+	 * Preallocate critical data structures so that we can copy
+	 * them in without further allocation after we suspend all
+	 * operations on the filesystem. We would like to just release
+	 * the allocated buffers without writing them since they will
+	 * be filled in below once we are ready to go, but this upsets
+	 * the soft update code, so we go ahead and write the new buffers.
+	 *
+	 * Allocate all indirect blocks. Also allocate shadow copies
+	 * for each of the indirect blocks.
+	 */
+	for (blkno = NDADDR; blkno < numblks; blkno += NINDIR(fs)) {
+		error = VOP_BALLOC(vp, lblktosize(fs, (off_t)blkno),
+		    fs->fs_bsize, p->p_ucred, B_METAONLY, &ibp);
+		if (error)
+			goto out;
+		copyblkno = fragstoblks(fs, dbtofsb(fs, ibp->b_blkno));
+		bdwrite(ibp);
+		error = VOP_BALLOC(vp, lblktosize(fs, (off_t)copyblkno),
+		    fs->fs_bsize, p->p_ucred, 0, &nbp);
+		if (error)
+			goto out;
+		bawrite(nbp);
+	}
+	/*
+	 * Allocate shadow blocks to copy all of the other snapshot inodes
+	 * so that we will be able to expunge them from this snapshot.
+	 */
+	for (loc = 0, inoblkcnt = 0; loc < snaploc; loc++) {
+		blkno = fragstoblks(fs, ino_to_fsba(fs, fs->fs_snapinum[loc]));
+		for (i = 0; i < inoblkcnt; i++)
+			if (inoblks[i] == blkno)
+				break;
+		if (i == inoblkcnt) {
+			inoblks[inoblkcnt++] = blkno;
+			error = VOP_BALLOC(vp, lblktosize(fs, (off_t)blkno),
+			    fs->fs_bsize, KERNCRED, 0, &nbp);
+			if (error)
+				goto out;
+			bawrite(nbp);
+		}
+	}
+	/*
+	 * Allocate all cylinder group blocks.
+	 */
+	for (cg = 0; cg < fs->fs_ncg; cg++) {
+		error = VOP_BALLOC(vp, (off_t)(cgtod(fs, cg)) << fs->fs_fshift,
+		    fs->fs_bsize, KERNCRED, 0, &nbp);
+		if (error)
+			goto out;
+		bawrite(nbp);
+	}
+	/*
+	 * Allocate copies for the superblock and its summary information.
+	 */
+	error = VOP_BALLOC(vp, (off_t)(SBOFF), fs->fs_bsize, KERNCRED,
+	    0, &nbp);
+	if (error)
+		goto out;
+	bawrite(nbp);
+	blkno = fragstoblks(fs, fs->fs_csaddr);
+	len = howmany(fs->fs_cssize, fs->fs_bsize);
+	for (loc = 0; loc < len; loc++) {
+		error = VOP_BALLOC(vp, lblktosize(fs, (off_t)(blkno + loc)),
+		    fs->fs_bsize, KERNCRED, 0, &nbp);
+		if (error)
+			goto out;
+		bawrite(nbp);
+	}
+	/*
+	 * Change inode to snapshot type file.
+	 */
+	ip->i_flags |= SF_IMMUTABLE | SF_SNAPSHOT;
+	ip->i_flag |= IN_CHANGE | IN_UPDATE;
+	/*
+	 * Ensure that the snapshot is completely on disk.
+	 */
+	if ((error = VOP_FSYNC(vp, KERNCRED, MNT_WAIT, p)) != 0)
+		goto out;
+	/*
+	 * All allocations are done, so we can now snapshot the system.
+	 *
+	 * Suspend operation on filesystem.
+	 */
+	for (;;) {
+		vn_finished_write(wrtmp);
+		vfs_write_suspend(vp->v_mount);
+		if (mp->mnt_kern_flag & MNTK_SUSPENDED)
+			break;
+		vn_start_write(wrtmp, V_WAIT);
+	}
+	/*
+	 * First, copy all the cylinder group maps. All the unallocated
+	 * blocks are marked BLK_NOCOPY so that the snapshot knows that
+	 * it need not copy them if they are later written.
+	 */
+	len = howmany(fs->fs_fpg, fs->fs_frag);
+	for (cg = 0; cg < fs->fs_ncg; cg++) {
+		error = bread(ip->i_devvp, fsbtodb(fs, cgtod(fs, cg)),
+			(int)fs->fs_cgsize, KERNCRED, &bp);
+		if (error) {
+			brelse(bp);
+			goto out1;
+		}
+		cgp = (struct cg *)bp->b_data;
+		if (!cg_chkmagic(cgp)) {
+			brelse(bp);
+			error = EIO;
+			goto out1;
+		}
+		error = bread(vp, fragstoblks(fs, cgtod(fs, cg)), fs->fs_bsize,
+			KERNCRED, &nbp);
+		if (error) {
+			brelse(bp);
+			brelse(nbp);
+			goto out1;
+		}
+		bcopy(bp->b_data, nbp->b_data, fs->fs_cgsize);
+		if (fs->fs_cgsize < fs->fs_bsize)
+			bzero(&nbp->b_data[fs->fs_cgsize],
+			    fs->fs_bsize - fs->fs_cgsize);
+		bawrite(nbp);
+		base = cg * fs->fs_fpg / fs->fs_frag;
+		if (base + len > numblks)
+			len = numblks - base;
+		loc = 0;
+		if (base < NDADDR) {
+			for ( ; loc < NDADDR; loc++) {
+				if (!ffs_isblock(fs, cg_blksfree(cgp), loc))
+					continue;
+				ip->i_db[loc] = BLK_NOCOPY;
+			}
+		}
+		error = VOP_BALLOC(vp, lblktosize(fs, (off_t)(base + loc)),
+		    fs->fs_bsize, KERNCRED, B_METAONLY, &ibp);
+		if (error) {
+			brelse(bp);
+			goto out1;
+		}
+		indiroff = (base + loc - NDADDR) % NINDIR(fs);
+		for ( ; loc < len; loc++, indiroff++) {
+			if (indiroff >= NINDIR(fs)) {
+				bawrite(ibp);
+				error = VOP_BALLOC(vp,
+				    lblktosize(fs, (off_t)(base + loc)),
+				    fs->fs_bsize, KERNCRED, B_METAONLY, &ibp);
+				if (error) {
+					brelse(bp);
+					goto out1;
+				}
+				indiroff = 0;
+			}
+			if (!ffs_isblock(fs, cg_blksfree(cgp), loc))
+				continue;
+			((ufs_daddr_t *)(ibp->b_data))[indiroff] = BLK_NOCOPY;
+		}
+		brelse(bp);
+		bdwrite(ibp);
+	}
+	/*
+	 * Snapshot the superblock and its summary information.
+	 */
+	error = VOP_BALLOC(vp, (off_t)(SBOFF), fs->fs_bsize, KERNCRED,
+	    0, &nbp);
+	if (error)
+		goto out1;
+	bcopy(fs, nbp->b_data, fs->fs_sbsize);
+	((struct fs *)(nbp->b_data))->fs_clean = 1;
+	if (fs->fs_sbsize < fs->fs_bsize)
+		bzero(&nbp->b_data[fs->fs_sbsize],
+		    fs->fs_bsize - fs->fs_sbsize);
+	bawrite(nbp);
+	blkno = fragstoblks(fs, fs->fs_csaddr);
+	len = howmany(fs->fs_cssize, fs->fs_bsize) - 1;
+	size = fs->fs_bsize;
+	for (loc = 0; loc <= len; loc++) {
+		error = VOP_BALLOC(vp, lblktosize(fs, (off_t)(blkno + loc)),
+		    fs->fs_bsize, KERNCRED, 0, &nbp);
+		if (error)
+			goto out1;
+		if (loc == len) {
+			readblock(nbp, blkno + loc);
+			size = fs->fs_cssize % fs->fs_bsize;
+		}
+		bcopy(fs->fs_csp[loc], nbp->b_data, size);
+		bawrite(nbp);
+	}
+	/*
+	 * Copy the shadow blocks for the snapshot inodes so that
+	 * the copies can can be expunged.
+	 */
+	for (loc = 0; loc < inoblkcnt; loc++) {
+		error = VOP_BALLOC(vp, lblktosize(fs, (off_t)inoblks[loc]),
+		    fs->fs_bsize, KERNCRED, 0, &nbp);
+		if (error)
+			goto out1;
+		readblock(nbp, inoblks[loc]);
+		bdwrite(nbp);
+	}
+	/*
+	 * Copy allocation information from other snapshots and then
+	 * expunge them from the view of the current snapshot.
+	 */
+	for (xp = devip->i_copyonwrite; xp; xp = xp->i_copyonwrite) {
+		/*
+		 * Before expunging a snapshot inode, note all the
+		 * blocks that it claims with BLK_SNAP so that fsck will
+		 * be able to account for those blocks properly and so
+		 * that this snapshot knows that it need not copy them
+		 * if the other snapshot holding them is freed.
+		 */
+		if ((error = snapacct(vp, &xp->i_db[0], &xp->i_ib[NIADDR])) !=0)
+			goto out1;
+		blksperindir = 1;
+		lbn = -NDADDR;
+		len = numblks - NDADDR;
+		rlbn = NDADDR;
+		for (i = 0; len > 0 && i < NIADDR; i++) {
+			error = indiracct(vp, ITOV(xp), i, xp->i_ib[i], lbn,
+			    rlbn, len, blksperindir);
+			if (error)
+				goto out1;
+			blksperindir *= NINDIR(fs);
+			lbn -= blksperindir + 1;
+			len -= blksperindir;
+			rlbn += blksperindir;
+		}
+		/*
+		 * Set copied snapshot inode to be a zero length file.
+		 */
+		blkno = fragstoblks(fs, ino_to_fsba(fs, xp->i_number));
+		error = VOP_BALLOC(vp, lblktosize(fs, (off_t)blkno),
+		    fs->fs_bsize, KERNCRED, 0, &nbp);
+		if (error)
+			goto out1;
+		dip = (struct dinode *)nbp->b_data +
+		    ino_to_fsbo(fs, xp->i_number);
+		dip->di_size = 0;
+		dip->di_blocks = 0;
+		dip->di_flags &= ~(SF_IMMUTABLE | SF_SNAPSHOT);
+		bzero(&dip->di_db[0], (NDADDR + NIADDR) * sizeof(ufs_daddr_t));
+		bdwrite(nbp);
+	}
+	/*
+	 * Copy all indirect blocks to their shadows (allocated above)
+	 * to avoid deadlock in ffs_copyonwrite.
+	 */
+	for (blkno = NDADDR; blkno < numblks; blkno += NINDIR(fs)) {
+		error = VOP_BALLOC(vp, lblktosize(fs, (off_t)blkno),
+		    fs->fs_bsize, p->p_ucred, B_METAONLY, &ibp);
+		if (error)
+			goto out1;
+		copyblkno = fragstoblks(fs, dbtofsb(fs, ibp->b_blkno));
+		brelse(ibp);
+		error = VOP_BALLOC(vp, lblktosize(fs, (off_t)copyblkno),
+		    fs->fs_bsize, p->p_ucred, 0, &nbp);
+		if (error)
+			goto out1;
+		error = VOP_BALLOC(vp, lblktosize(fs, (off_t)blkno),
+		    fs->fs_bsize, p->p_ucred, B_METAONLY, &ibp);
+		if (error) {
+			brelse(nbp);
+			goto out1;
+		}
+		bcopy(ibp->b_data, nbp->b_data, fs->fs_bsize);
+		brelse(ibp);
+		bawrite(nbp);
+	}
+	/*
+	 * Record snapshot inode. Since this is the newest snapshot,
+	 * it must be placed at the end of the list.
+	 */
+	fs->fs_snapinum[snaploc] = ip->i_number;
+	if (ip->i_copyonwrite != 0)
+		panic("ffs_snapshot: %d already on list", ip->i_number);
+	if (devip->i_copyonwrite == 0) {
+		devvp->v_flag |= VCOPYONWRITE;
+		devip->i_copyonwrite = ip;
+	} else {
+		for (xp = devip->i_copyonwrite; xp->i_copyonwrite != 0; )
+			xp = xp->i_copyonwrite;
+		xp->i_copyonwrite = ip;
+	}
+	vp->v_flag |= VSYSTEM;
+	/*
+	 * Resume operation on filesystem.
+	 */
+out1:
+	vfs_write_resume(vp->v_mount);
+	vn_start_write(wrtmp, V_WAIT);
+out:
+	mp->mnt_flag = flag;
+	(void) VOP_FSYNC(vp, KERNCRED, MNT_WAIT, p);
+	if (error)
+		vput(vp);
+	else
+		VOP_UNLOCK(vp, 0, p);
+	vn_finished_write(wrtmp);
+	return (error);
+}
+
+/*
+ * Descend an indirect block chain for vnode cancelvp accounting for all
+ * its indirect blocks in snapvp.
+ */ 
+static int
+indiracct(snapvp, cancelvp, level, blkno, lbn, rlbn, remblks, blksperindir)
+	struct vnode *snapvp;
+	struct vnode *cancelvp;
+	int level;
+	ufs_daddr_t blkno;
+	int lbn;
+	int rlbn;
+	int remblks;
+	int blksperindir;
+{
+	int subblksperindir, error, last, num, i;
+	struct indir indirs[NIADDR + 2];
+	ufs_daddr_t *bap;
+	struct buf *bp;
+	struct fs *fs;
+
+	if ((error = ufs_getlbns(cancelvp, rlbn, indirs, &num)) != 0)
+		return (error);
+	if (lbn != indirs[num - 1 - level].in_lbn || blkno == 0 || num < 2)
+		panic("indiracct: botched params");
+	/*
+	 * We have to expand bread here since it will deadlock looking
+	 * up the block number for any blocks that are not in the cache.
+	 */
+	fs = VTOI(cancelvp)->i_fs;
+	bp = getblk(cancelvp, lbn, fs->fs_bsize, 0, 0);
+	bp->b_blkno = fsbtodb(fs, blkno);
+	if ((bp->b_flags & (B_DONE | B_DELWRI)) == 0 &&
+	    (error = readblock(bp, fragstoblks(fs, blkno)))) {
+		brelse(bp);
+		return (error);
+	}
+	/*
+	 * Account for the block pointers in this indirect block.
+	 */
+	last = howmany(remblks, blksperindir);
+	if (last > NINDIR(fs))
+		last = NINDIR(fs);
+	if (snapvp != cancelvp) {
+		bap = (ufs_daddr_t *)bp->b_data;
+	} else {
+		MALLOC(bap, ufs_daddr_t *, fs->fs_bsize, M_DEVBUF, M_WAITOK);
+		bcopy(bp->b_data, (caddr_t)bap, fs->fs_bsize);
+		brelse(bp);
+	}
+	error = snapacct(snapvp, &bap[0], &bap[last]);
+	if (error || level == 0)
+		goto out;
+	/*
+	 * Account for the block pointers in each of the indirect blocks
+	 * in the levels below us.
+	 */
+	subblksperindir = blksperindir / NINDIR(fs);
+	for (lbn++, level--, i = 0; i < last; i++) {
+		error = indiracct(snapvp, cancelvp, level, bap[i], lbn,
+		    rlbn, remblks, subblksperindir);
+		if (error)
+			goto out;
+		rlbn += blksperindir;
+		lbn -= blksperindir;
+		remblks -= blksperindir;
+	}
+out:
+	if (snapvp != cancelvp)
+		brelse(bp);
+	else
+		FREE(bap, M_DEVBUF);
+	return (error);
+}
+
+/*
+ * Account for a set of blocks allocated in a snapshot inode.
+ */
+static int
+snapacct(vp, oldblkp, lastblkp)
+	struct vnode *vp;
+	ufs_daddr_t *oldblkp, *lastblkp;
+{
+	struct inode *ip = VTOI(vp);
+	struct fs *fs = ip->i_fs;
+	ufs_daddr_t lbn, blkno, *blkp;
+	struct buf *ibp;
+	int error;
+
+	for ( ; oldblkp < lastblkp; oldblkp++) {
+		blkno = *oldblkp;
+		if (blkno == 0 || blkno == BLK_NOCOPY || blkno == BLK_SNAP)
+			continue;
+		lbn = fragstoblks(fs, blkno);
+		if (lbn < NDADDR) {
+			blkp = &ip->i_db[lbn];
+			ip->i_flag |= IN_CHANGE | IN_UPDATE;
+		} else {
+			error = VOP_BALLOC(vp, lblktosize(fs, (off_t)lbn),
+			    fs->fs_bsize, KERNCRED, B_METAONLY, &ibp);
+			if (error)
+				return (error);
+			blkp = &((ufs_daddr_t *)(ibp->b_data))
+			    [(lbn - NDADDR) % NINDIR(fs)];
+		}
+		if (*blkp != 0)
+			panic("snapacct: bad block");
+		*blkp = BLK_SNAP;
+		if (lbn >= NDADDR)
+			bdwrite(ibp);
+	}
+	return (0);
+}
+
+/*
+ * Prepare a snapshot file for being removed.
+ */
+void
+ffs_snapremove(vp)
+	struct vnode *vp;
+{
+	struct inode *ip, *xp;
+	struct vnode *devvp;
+	struct buf *ibp;
+	struct fs *fs;
+	ufs_daddr_t blkno, dblk;
+	int error, snaploc, loc, last;
+
+	ip = VTOI(vp);
+	fs = ip->i_fs;
+	/*
+	 * Delete snapshot inode from superblock. Keep list dense.
+	 */
+	for (snaploc = 0; snaploc < FSMAXSNAP; snaploc++)
+		if (fs->fs_snapinum[snaploc] == ip->i_number)
+			break;
+	if (snaploc < FSMAXSNAP) {
+		for (snaploc++; snaploc < FSMAXSNAP; snaploc++) {
+			if (fs->fs_snapinum[snaploc] == 0)
+				break;
+			fs->fs_snapinum[snaploc - 1] = fs->fs_snapinum[snaploc];
+		}
+		fs->fs_snapinum[snaploc - 1] = 0;
+	}
+	/*
+	 * Delete from incore list.
+	 * Clear copy-on-write flag if last snapshot.
+	 */
+	devvp = ip->i_devvp;
+	for (xp = VTOI(devvp); xp; xp = xp->i_copyonwrite) {
+		if (xp->i_copyonwrite != ip)
+			continue;
+		xp->i_copyonwrite = ip->i_copyonwrite;
+		ip->i_copyonwrite = 0;
+		break;
+	}
+	if (xp == 0) {
+		printf("ffs_snapremove: lost snapshot vnode %d\n",
+		    ip->i_number);
+		vref(vp);
+	}
+	if (VTOI(devvp)->i_copyonwrite == 0)
+		devvp->v_flag &= ~VCOPYONWRITE;
+	/*
+	 * Clear all BLK_NOCOPY fields. Pass any block claims to other
+	 * snapshots that want them (see ffs_snapblkfree below).
+	 */
+	for (blkno = 1; blkno < NDADDR; blkno++) {
+		dblk = ip->i_db[blkno];
+		if (dblk == BLK_NOCOPY || dblk == BLK_SNAP ||
+		    (dblk == blkstofrags(fs, blkno) &&
+		     ffs_snapblkfree(ip, dblk, fs->fs_bsize)))
+			ip->i_db[blkno] = 0;
+	}
+	for (blkno = NDADDR; blkno < fs->fs_size; blkno += NINDIR(fs)) {
+		error = VOP_BALLOC(vp, lblktosize(fs, (off_t)blkno),
+		    fs->fs_bsize, KERNCRED, B_METAONLY, &ibp);
+		if (error)
+			continue;
+		if ((last = fs->fs_size - blkno) > NINDIR(fs))
+			last = NINDIR(fs);
+		for (loc = 0; loc < last; loc++) {
+			dblk = ((ufs_daddr_t *)(ibp->b_data))[loc];
+			if (dblk == BLK_NOCOPY || dblk == BLK_SNAP ||
+			    (dblk == blkstofrags(fs, blkno) &&
+			     ffs_snapblkfree(ip, dblk, fs->fs_bsize)))
+				((ufs_daddr_t *)(ibp->b_data))[loc] = 0;
+		}
+		bawrite(ibp);
+	}
+	/*
+	 * Clear snapshot flag and drop reference.
+	 */
+	ip->i_flags &= ~(SF_IMMUTABLE | SF_SNAPSHOT);
+	ip->i_flag |= IN_CHANGE | IN_UPDATE;
+	vrele(vp);
+}
+
+/*
+ * Notification that a block is being freed. Return zero if the free
+ * should be allowed to proceed. Return non-zero if the snapshot file
+ * wants to claim the block. The block will be claimed if it is an
+ * uncopied part of one of the snapshots. It will be freed if it is
+ * either a BLK_NOCOPY or has already been copied in all of the snapshots.
+ * If a fragment is being freed, then all snapshots that care about
+ * it must make a copy since a snapshot file can only claim full sized
+ * blocks. Note that if more than one snapshot file maps the block,
+ * we can pick one at random to claim it. Since none of the snapshots
+ * can change, we are assurred that they will all see the same unmodified
+ * image. When deleting a snapshot file (see ffs_snapremove above), we
+ * must push any of these claimed blocks to one of the other snapshots
+ * that maps it. These claimed blocks are easily identified as they will
+ * have a block number equal to their logical block number within the
+ * snapshot. A copied block can never have this property because they
+ * must always have been allocated from a BLK_NOCOPY location.
+ */
+int
+ffs_snapblkfree(freeip, bno, size)
+	struct inode *freeip;
+	ufs_daddr_t bno;
+	long size;
+{
+	struct buf *ibp, *cbp, *savedcbp = 0;
+	struct fs *fs = freeip->i_fs;
+	struct proc *p = CURPROC;
+	struct inode *ip;
+	struct vnode *vp;
+	ufs_daddr_t lbn, blkno;
+	int indiroff = 0, error = 0, claimedblk = 0;
+
+	lbn = fragstoblks(fs, bno);
+	for (ip = VTOI(freeip->i_devvp)->i_copyonwrite; ip;
+	     ip = ip->i_copyonwrite) {
+		vp = ITOV(ip);
+		/*
+		 * Lookup block being written.
+		 */
+		if (lbn < NDADDR) {
+			blkno = ip->i_db[lbn];
+		} else {
+			vn_lock(vp, LK_EXCLUSIVE | LK_RETRY, p);
+			p->p_flag |= P_COWINPROGRESS;
+			error = VOP_BALLOC(vp, lblktosize(fs, (off_t)lbn),
+			    fs->fs_bsize, KERNCRED, B_METAONLY, &ibp);
+			p->p_flag &= ~P_COWINPROGRESS;
+			VOP_UNLOCK(vp, 0, p);
+			if (error)
+				break;
+			indiroff = (lbn - NDADDR) % NINDIR(fs);
+			blkno = ((ufs_daddr_t *)(ibp->b_data))[indiroff];
+		}
+		/*
+		 * Check to see if block needs to be copied.
+		 */
+		switch (blkno) {
+		/*
+		 * If the snapshot has already copied the block (default),
+		 * or does not care about the block, it is not needed.
+		 */
+		default:
+		case BLK_NOCOPY:
+			if (lbn >= NDADDR)
+				brelse(ibp);
+			continue;
+		/*
+		 * No previous snapshot claimed the block, so it will be
+		 * freed and become a BLK_NOCOPY (don't care) for us.
+		 */
+		case BLK_SNAP:
+			if (claimedblk)
+				panic("snapblkfree: inconsistent block type");
+			vn_lock(vp, LK_EXCLUSIVE | LK_RETRY, p);
+			if (lbn < NDADDR) {
+				ip->i_db[lbn] = BLK_NOCOPY;
+				ip->i_flag |= IN_CHANGE | IN_UPDATE;
+			} else {
+				((ufs_daddr_t *)(ibp->b_data))[indiroff] =
+				    BLK_NOCOPY;
+				bdwrite(ibp);
+			}
+			VOP_UNLOCK(vp, 0, p);
+			continue;
+		/*
+		 * A block that we map is being freed. If it has not been
+		 * claimed yet, we will claim or copy it (below).
+		 */
+		case 0:
+			claimedblk = 1;
+			break;
+		}
+		/*
+		 * If this is a full size block, we will just grab it
+		 * and assign it to the snapshot inode. Otherwise we
+		 * will proceed to copy it. See explanation for this
+		 * routine as to why only a single snapshot needs to
+		 * claim this block.
+		 */
+		if (size == fs->fs_bsize) {
+#ifdef DEBUG
+			if (snapdebug)
+				printf("%s %d lbn %d from inum %d\n",
+				    "Grabonremove: snapino", ip->i_number, lbn,
+				    freeip->i_number);
+#endif
+			vn_lock(vp, LK_EXCLUSIVE | LK_RETRY, p);
+			if (lbn < NDADDR) {
+				ip->i_db[lbn] = bno;
+			} else {
+				((ufs_daddr_t *)(ibp->b_data))[indiroff] = bno;
+				bdwrite(ibp);
+			}
+			ip->i_blocks += btodb(size);
+			ip->i_flag |= IN_CHANGE | IN_UPDATE;
+			VOP_UNLOCK(vp, 0, p);
+			return (1);
+		}
+		if (lbn >= NDADDR)
+			brelse(ibp);
+		/*
+		 * Allocate the block into which to do the copy. Note that this
+		 * allocation will never require any additional allocations for
+		 * the snapshot inode.
+		 */
+		vn_lock(vp, LK_EXCLUSIVE | LK_RETRY, p);
+		p->p_flag |= P_COWINPROGRESS;
+		error = VOP_BALLOC(vp, lblktosize(fs, (off_t)lbn),
+		    fs->fs_bsize, KERNCRED, 0, &cbp);
+		p->p_flag &= ~P_COWINPROGRESS;
+		VOP_UNLOCK(vp, 0, p);
+		if (error)
+			break;
+#ifdef DEBUG
+		if (snapdebug)
+			printf("%s%d lbn %d for inum %d size %ld to blkno %d\n",
+			    "Copyonremove: snapino ", ip->i_number, lbn,
+			    freeip->i_number, size, cbp->b_blkno);
+#endif
+		/*
+		 * If we have already read the old block contents, then
+		 * simply copy them to the new block.
+		 */
+		if (savedcbp != 0) {
+			bcopy(savedcbp->b_data, cbp->b_data, fs->fs_bsize);
+			bawrite(cbp);
+			continue;
+		}
+		/*
+		 * Otherwise, read the old block contents into the buffer.
+		 */
+		if ((error = readblock(cbp, lbn)) != 0)
+			break;
+		savedcbp = cbp;
+	}
+	if (savedcbp)
+		bawrite(savedcbp);
+	/*
+	 * If we have been unable to allocate a block in which to do
+	 * the copy, then return non-zero so that the fragment will
+	 * not be freed. Although space will be lost, the snapshot
+	 * will stay consistent.
+	 */
+	return (error);
+}
+
+/*
+ * Associate snapshot files when mounting.
+ */
+void
+ffs_snapshot_mount(mp)
+	struct mount *mp;
+{
+	struct ufsmount *ump = VFSTOUFS(mp);
+	struct fs *fs = ump->um_fs;
+	struct proc *p = CURPROC;
+	struct inode *ip, **listtailp;
+	struct vnode *vp;
+	int error, snaploc, loc;
+
+	listtailp = &VTOI(ump->um_devvp)->i_copyonwrite;
+	for (snaploc = 0; snaploc < FSMAXSNAP; snaploc++) {
+		if (fs->fs_snapinum[snaploc] == 0)
+			return;
+		if ((error = VFS_VGET(mp, fs->fs_snapinum[snaploc], &vp)) != 0){
+			printf("ffs_snapshot_mount: vget failed %d\n", error);
+			continue;
+		}
+		ip = VTOI(vp);
+		if ((ip->i_flags & SF_SNAPSHOT) == 0) {
+			printf("ffs_snapshot_mount: non-snapshot inode %d\n",
+			    fs->fs_snapinum[snaploc]);
+			vput(vp);
+			for (loc = snaploc + 1; loc < FSMAXSNAP; loc++) {
+				if (fs->fs_snapinum[loc] == 0)
+					break;
+				fs->fs_snapinum[loc - 1] = fs->fs_snapinum[loc];
+			}
+			fs->fs_snapinum[loc - 1] = 0;
+			snaploc--;
+			continue;
+		}
+		if (ip->i_copyonwrite != 0)
+			panic("ffs_snapshot_mount: %d already on list",
+			    ip->i_number);
+		*listtailp = ip;
+		listtailp = &ip->i_copyonwrite;
+		vp->v_flag |= VSYSTEM;
+		VOP_UNLOCK(vp, 0, p);
+		ump->um_devvp->v_flag |= VCOPYONWRITE;
+	}
+}
+
+/*
+ * Disassociate snapshot files when unmounting.
+ */
+void
+ffs_snapshot_unmount(mp)
+	struct mount *mp;
+{
+	struct ufsmount *ump = VFSTOUFS(mp);
+	struct inode *devip = VTOI(ump->um_devvp);
+	struct inode *xp;
+
+	while ((xp = devip->i_copyonwrite) != 0) {
+		devip->i_copyonwrite = xp->i_copyonwrite;
+		xp->i_copyonwrite = 0;
+		vrele(ITOV(xp));
+	}
+	ump->um_devvp->v_flag &= ~VCOPYONWRITE;
+}
+
+/*
+ * Check for need to copy block that is about to be written,
+ * copying the block if necessary.
+ */
+int
+ffs_copyonwrite(ap)
+	struct vop_copyonwrite_args /* {
+		struct vnode *a_vp;
+		struct buf *a_bp;
+	} */ *ap;
+{
+	struct buf *ibp, *cbp, *savedcbp = 0, *bp = ap->a_bp;
+	struct fs *fs = VTOI(bp->b_vp)->i_fs;
+	struct proc *p = CURPROC;
+	struct inode *ip;
+	struct vnode *vp;
+	ufs_daddr_t lbn, blkno;
+	int indiroff, error = 0;
+
+	lbn = fragstoblks(fs, dbtofsb(fs, bp->b_blkno));
+	if (p->p_flag & P_COWINPROGRESS)
+		panic("ffs_copyonwrite: recursive call");
+	for (ip = VTOI(ap->a_vp)->i_copyonwrite; ip; ip = ip->i_copyonwrite) {
+		vp = ITOV(ip);
+		/*
+		 * We ensure that everything of our own that needs to be
+		 * copied will be done at the time that ffs_snapshot is
+		 * called. Thus we can skip the check here which can
+		 * deadlock in doing the lookup in VOP_BALLOC.
+		 */
+		if (bp->b_vp == vp)
+			continue;
+		/*
+		 * Check to see if block needs to be copied.
+		 */
+		if (lbn < NDADDR) {
+			blkno = ip->i_db[lbn];
+		} else {
+			vn_lock(vp, LK_EXCLUSIVE | LK_RETRY, p);
+			p->p_flag |= P_COWINPROGRESS;
+			error = VOP_BALLOC(vp, lblktosize(fs, (off_t)lbn),
+			    fs->fs_bsize, KERNCRED, B_METAONLY, &ibp);
+			p->p_flag &= ~P_COWINPROGRESS;
+			VOP_UNLOCK(vp, 0, p);
+			if (error)
+				break;
+			indiroff = (lbn - NDADDR) % NINDIR(fs);
+			blkno = ((ufs_daddr_t *)(ibp->b_data))[indiroff];
+			brelse(ibp);
+		}
+#ifdef DIAGNOSTIC
+		if (blkno == BLK_SNAP && bp->b_lblkno >= 0)
+			panic("ffs_copyonwrite: bad copy block");
+#endif
+		if (blkno != 0)
+			continue;
+		/*
+		 * Allocate the block into which to do the copy. Note that this
+		 * allocation will never require any additional allocations for
+		 * the snapshot inode.
+		 */
+		vn_lock(vp, LK_EXCLUSIVE | LK_RETRY, p);
+		p->p_flag |= P_COWINPROGRESS;
+		error = VOP_BALLOC(vp, lblktosize(fs, (off_t)lbn),
+		    fs->fs_bsize, KERNCRED, 0, &cbp);
+		p->p_flag &= ~P_COWINPROGRESS;
+		VOP_UNLOCK(vp, 0, p);
+#ifdef DEBUG
+		if (snapdebug) {
+			printf("Copyonwrite: snapino %d lbn %d for ",
+			    ip->i_number, lbn);
+			if (bp->b_vp == ap->a_vp)
+				printf("fs metadata");
+			else
+				printf("inum %d", VTOI(bp->b_vp)->i_number);
+			printf(" lblkno %d to blkno %d\n", bp->b_lblkno,
+			    cbp->b_blkno);
+		}
+#endif
+		if (error)
+			break;
+		/*
+		 * If we have already read the old block contents, then
+		 * simply copy them to the new block.
+		 */
+		if (savedcbp != 0) {
+			bcopy(savedcbp->b_data, cbp->b_data, fs->fs_bsize);
+			bawrite(cbp);
+			continue;
+		}
+		/*
+		 * Otherwise, read the old block contents into the buffer.
+		 */
+		if ((error = readblock(cbp, lbn)) != 0)
+			break;
+		savedcbp = cbp;
+	}
+	if (savedcbp)
+		bawrite(savedcbp);
+	return (error);
+}
+
+/*
+ * Read the specified block into the given buffer.
+ * Much of this boiler-plate comes from bwrite().
+ */
+static int
+readblock(bp, lbn)
+	struct buf *bp;
+	daddr_t lbn;
+{
+	struct uio auio;
+	struct iovec aiov;
+	struct proc *p = CURPROC;
+	struct inode *ip = VTOI(bp->b_vp);
+
+	aiov.iov_base = bp->b_data;
+	aiov.iov_len = bp->b_bcount;
+	auio.uio_iov = &aiov;
+	auio.uio_iovcnt = 1;
+	auio.uio_offset = dbtob(fsbtodb(ip->i_fs, blkstofrags(ip->i_fs, lbn)));
+	auio.uio_resid = bp->b_bcount;
+	auio.uio_rw = UIO_READ;
+	auio.uio_segflg = UIO_SYSSPACE;
+	auio.uio_procp = p;
+	return (physio(ip->i_devvp->v_rdev, &auio, 0));
+}
diff --git a/sys/ufs/ffs/ffs_softdep.c b/sys/ufs/ffs/ffs_softdep.c
index 40e9669..d9e6414 100644
--- a/sys/ufs/ffs/ffs_softdep.c
+++ b/sys/ufs/ffs/ffs_softdep.c
@@ -508,7 +508,7 @@ softdep_process_worklist(matchmnt)
 {
 	struct proc *p = CURPROC;
 	struct worklist *wk;
-	struct fs *matchfs;
+	struct mount *mp;
 	int matchcnt, loopcount;
 
 	/*
@@ -517,9 +517,6 @@ softdep_process_worklist(matchmnt)
 	 */
 	filesys_syncer = p;
 	matchcnt = 0;
-	matchfs = NULL;
-	if (matchmnt != NULL)
-		matchfs = VFSTOUFS(matchmnt)->um_fs;
 	/*
 	 * There is no danger of having multiple processes run this
 	 * code. It is single threaded solely so that softdep_flushfiles
@@ -550,30 +547,42 @@ softdep_process_worklist(matchmnt)
 
 		case D_DIRREM:
 			/* removal of a directory entry */
-			if (WK_DIRREM(wk)->dm_mnt == matchmnt)
+			mp = WK_DIRREM(wk)->dm_mnt;
+			if (mp == matchmnt)
 				matchcnt += 1;
+			vn_start_write(NULL, &mp, V_WAIT);
 			handle_workitem_remove(WK_DIRREM(wk));
+			vn_finished_write(mp);
 			break;
 
 		case D_FREEBLKS:
 			/* releasing blocks and/or fragments from a file */
-			if (WK_FREEBLKS(wk)->fb_fs == matchfs)
+			mp = WK_FREEBLKS(wk)->fb_mnt;
+			if (mp == matchmnt)
 				matchcnt += 1;
+			vn_start_write(NULL, &mp, V_WAIT);
 			handle_workitem_freeblocks(WK_FREEBLKS(wk));
+			vn_finished_write(mp);
 			break;
 
 		case D_FREEFRAG:
 			/* releasing a fragment when replaced as a file grows */
-			if (WK_FREEFRAG(wk)->ff_fs == matchfs)
+			mp = WK_FREEFRAG(wk)->ff_mnt;
+			if (mp == matchmnt)
 				matchcnt += 1;
+			vn_start_write(NULL, &mp, V_WAIT);
 			handle_workitem_freefrag(WK_FREEFRAG(wk));
+			vn_finished_write(mp);
 			break;
 
 		case D_FREEFILE:
 			/* releasing an inode when its link count drops to 0 */
-			if (WK_FREEFILE(wk)->fx_fs == matchfs)
+			mp = WK_FREEFILE(wk)->fx_mnt;
+			if (mp == matchmnt)
 				matchcnt += 1;
+			vn_start_write(NULL, &mp, V_WAIT);
 			handle_workitem_freefile(WK_FREEFILE(wk));
+			vn_finished_write(mp);
 			break;
 
 		default:
@@ -1316,7 +1325,7 @@ newfreefrag(ip, blkno, size)
 	freefrag->ff_list.wk_type = D_FREEFRAG;
 	freefrag->ff_state = ip->i_uid & ~ONWORKLIST;	/* XXX - used below */
 	freefrag->ff_inum = ip->i_number;
-	freefrag->ff_fs = fs;
+	freefrag->ff_mnt = ITOV(ip)->v_mount;
 	freefrag->ff_devvp = ip->i_devvp;
 	freefrag->ff_blkno = blkno;
 	freefrag->ff_fragsize = size;
@@ -1333,7 +1342,8 @@ handle_workitem_freefrag(freefrag)
 {
 	struct inode tip;
 
-	tip.i_fs = freefrag->ff_fs;
+	tip.i_vnode = NULL;
+	tip.i_fs = VFSTOUFS(freefrag->ff_mnt)->um_fs;
 	tip.i_devvp = freefrag->ff_devvp;
 	tip.i_dev = freefrag->ff_devvp->v_rdev;
 	tip.i_number = freefrag->ff_inum;
@@ -1601,7 +1611,7 @@ softdep_setup_freeblocks(ip, length)
 	freeblks->fb_uid = ip->i_uid;
 	freeblks->fb_previousinum = ip->i_number;
 	freeblks->fb_devvp = ip->i_devvp;
-	freeblks->fb_fs = fs;
+	freeblks->fb_mnt = ITOV(ip)->v_mount;
 	freeblks->fb_oldsize = ip->i_size;
 	freeblks->fb_newsize = length;
 	freeblks->fb_chkcnt = ip->i_blocks;
@@ -1845,7 +1855,7 @@ softdep_freefile(pvp, ino, mode)
 	freefile->fx_mode = mode;
 	freefile->fx_oldinum = ino;
 	freefile->fx_devvp = ip->i_devvp;
-	freefile->fx_fs = ip->i_fs;
+	freefile->fx_mnt = ITOV(ip)->v_mount;
 
 	/*
 	 * If the inodedep does not exist, then the zero'ed inode has
@@ -1949,13 +1959,13 @@ handle_workitem_freeblocks(freeblks)
 	int error, allerror = 0;
 	ufs_lbn_t baselbns[NIADDR], tmpval;
 
+	tip.i_fs = fs = VFSTOUFS(freeblks->fb_mnt)->um_fs;
 	tip.i_number = freeblks->fb_previousinum;
 	tip.i_devvp = freeblks->fb_devvp;
 	tip.i_dev = freeblks->fb_devvp->v_rdev;
-	tip.i_fs = freeblks->fb_fs;
 	tip.i_size = freeblks->fb_oldsize;
 	tip.i_uid = freeblks->fb_uid;
-	fs = freeblks->fb_fs;
+	tip.i_vnode = NULL;
 	tmpval = 1;
 	baselbns[0] = NDADDR;
 	for (i = 1; i < NIADDR; i++) {
@@ -2715,20 +2725,23 @@ static void
 handle_workitem_freefile(freefile)
 	struct freefile *freefile;
 {
+	struct fs *fs;
 	struct vnode vp;
 	struct inode tip;
 	struct inodedep *idp;
 	int error;
 
+	fs = VFSTOUFS(freefile->fx_mnt)->um_fs;
 #ifdef DEBUG
 	ACQUIRE_LOCK(&lk);
-	if (inodedep_lookup(freefile->fx_fs, freefile->fx_oldinum, 0, &idp))
+	if (inodedep_lookup(fs, freefile->fx_oldinum, 0, &idp))
 		panic("handle_workitem_freefile: inodedep survived");
 	FREE_LOCK(&lk);
 #endif
 	tip.i_devvp = freefile->fx_devvp;
 	tip.i_dev = freefile->fx_devvp->v_rdev;
-	tip.i_fs = freefile->fx_fs;
+	tip.i_fs = fs;
+	tip.i_vnode = &vp;
 	vp.v_data = &tip;
 	if ((error = ffs_freefile(&vp, freefile->fx_oldinum, freefile->fx_mode)) != 0)
 		softdep_error("handle_workitem_freefile", error);
@@ -4419,14 +4432,18 @@ clear_remove(p)
 			mp = pagedep->pd_mnt;
 			ino = pagedep->pd_ino;
 			FREE_LOCK(&lk);
+			if (vn_start_write(NULL, &mp, V_WAIT | PCATCH) != 0)
+				return;
 			if ((error = VFS_VGET(mp, ino, &vp)) != 0) {
 				softdep_error("clear_remove: vget", error);
+				vn_finished_write(mp);
 				return;
 			}
 			if ((error = VOP_FSYNC(vp, p->p_ucred, MNT_NOWAIT, p)))
 				softdep_error("clear_remove: fsync", error);
 			drain_output(vp, 0);
 			vput(vp);
+			vn_finished_write(mp);
 			return;
 		}
 	}
@@ -4486,8 +4503,11 @@ clear_inodedeps(p)
 		if (inodedep_lookup(fs, ino, 0, &inodedep) == 0)
 			continue;
 		FREE_LOCK(&lk);
+		if (vn_start_write(NULL, &mp, V_WAIT | PCATCH) != 0)
+			return;
 		if ((error = VFS_VGET(mp, ino, &vp)) != 0) {
 			softdep_error("clear_inodedeps: vget", error);
+			vn_finished_write(mp);
 			return;
 		}
 		if (ino == lastino) {
@@ -4499,6 +4519,7 @@ clear_inodedeps(p)
 			drain_output(vp, 0);
 		}
 		vput(vp);
+		vn_finished_write(mp);
 		ACQUIRE_LOCK(&lk);
 	}
 	FREE_LOCK(&lk);
diff --git a/sys/ufs/ffs/ffs_vfsops.c b/sys/ufs/ffs/ffs_vfsops.c
index 89ff6d3..5280181 100644
--- a/sys/ufs/ffs/ffs_vfsops.c
+++ b/sys/ufs/ffs/ffs_vfsops.c
@@ -133,7 +133,7 @@ VFS_SET(ufs_vfsops, ufs, 0);
  *		namei() if it is a genuine NULL from the user.
  */
 static int
-ffs_mount( mp, path, data, ndp, p)
+ffs_mount(mp, path, data, ndp, p)
         struct mount		*mp;	/* mount struct pointer*/
         char			*path;	/* path to mount point*/
         caddr_t			data;	/* arguments to FS specific mount*/
@@ -141,49 +141,34 @@ ffs_mount( mp, path, data, ndp, p)
         struct proc		*p;	/* process requesting mount*/
 {
 	size_t		size;
-	int		err = 0;
 	struct vnode	*devvp;
-
 	struct ufs_args args;
 	struct ufsmount *ump = 0;
 	register struct fs *fs;
-	int error, flags, ronly = 0;
+	int error, flags;
 	mode_t accessmode;
 
 	/*
-	 * Use NULL path to flag a root mount
+	 * Use NULL path to indicate we are mounting the root file system.
 	 */
-	if( path == NULL) {
-		/*
-		 ***
-		 * Mounting root file system
-		 ***
-		 */
-	
-		if ((err = bdevvp(rootdev, &rootvp))) {
+	if (path == NULL) {
+		if ((error = bdevvp(rootdev, &rootvp))) {
 			printf("ffs_mountroot: can't find rootvp\n");
-			return (err);
-		}
-
-		if( ( err = ffs_mountfs(rootvp, mp, p, M_FFSNODE)) != 0) {
-			/* fs specific cleanup (if any)*/
-			goto error_1;
+			return (error);
 		}
 
-		goto dostatfs;		/* success*/
+		if ((error = ffs_mountfs(rootvp, mp, p, M_FFSNODE)) != 0)
+			return (error);
 
+		(void)VFS_STATFS(mp, &mp->mnt_stat, p);
+		return (0);
 	}
 
 	/*
-	 ***
 	 * Mounting non-root file system or updating a file system
-	 ***
 	 */
-
-	/* copy in user arguments*/
-	err = copyin(data, (caddr_t)&args, sizeof (struct ufs_args));
-	if (err)
-		goto error_1;		/* can't get arguments*/
+	if ((error = copyin(data, (caddr_t)&args, sizeof(struct ufs_args)))!= 0)
+		return (error);
 
 	/*
 	 * If updating, check whether changing from read-only to
@@ -193,25 +178,36 @@ ffs_mount( mp, path, data, ndp, p)
 		ump = VFSTOUFS(mp);
 		fs = ump->um_fs;
 		devvp = ump->um_devvp;
-		err = 0;
-		ronly = fs->fs_ronly;	/* MNT_RELOAD might change this */
-		if (ronly == 0 && (mp->mnt_flag & MNT_RDONLY)) {
+		if (fs->fs_ronly == 0 && (mp->mnt_flag & MNT_RDONLY)) {
+			if ((error = vn_start_write(NULL, &mp, V_WAIT)) != 0)
+				return (error);
 			flags = WRITECLOSE;
 			if (mp->mnt_flag & MNT_FORCE)
 				flags |= FORCECLOSE;
 			if (mp->mnt_flag & MNT_SOFTDEP) {
-				err = softdep_flushfiles(mp, flags, p);
+				error = softdep_flushfiles(mp, flags, p);
 			} else {
-				err = ffs_flushfiles(mp, flags, p);
+				error = ffs_flushfiles(mp, flags, p);
 			}
-			ronly = 1;
-		}
-		if (!err && (mp->mnt_flag & MNT_RELOAD))
-			err = ffs_reload(mp, ndp->ni_cnd.cn_cred, p);
-		if (err) {
-			goto error_1;
+			if (error) {
+				vn_finished_write(mp);
+				return (error);
+			}
+			fs->fs_ronly = 1;
+			if ((fs->fs_flags & FS_UNCLEAN) == 0)
+				fs->fs_clean = 1;
+			if ((error = ffs_sbupdate(ump, MNT_WAIT)) != 0) {
+				fs->fs_ronly = 0;
+				fs->fs_clean = 0;
+				vn_finished_write(mp);
+				return (error);
+			}
+			vn_finished_write(mp);
 		}
-		if (ronly && (mp->mnt_kern_flag & MNTK_WANTRDWR)) {
+		if ((mp->mnt_flag & MNT_RELOAD) &&
+		    (error = ffs_reload(mp, ndp->ni_cnd.cn_cred, p)) != 0)
+			return (error);
+		if (fs->fs_ronly && (mp->mnt_kern_flag & MNTK_WANTRDWR)) {
 			/*
 			 * If upgrade to read-write by non-root, then verify
 			 * that user has necessary permissions on the device.
@@ -225,31 +221,36 @@ ffs_mount( mp, path, data, ndp, p)
 				}
 				VOP_UNLOCK(devvp, 0, p);
 			}
-
 			fs->fs_flags &= ~FS_UNCLEAN;
 			if (fs->fs_clean == 0) {
 				fs->fs_flags |= FS_UNCLEAN;
 				if (mp->mnt_flag & MNT_FORCE) {
-					printf(
-"WARNING: %s was not properly dismounted\n",
-					    fs->fs_fsmnt);
+					printf("WARNING: %s was not %s\n",
+					   fs->fs_fsmnt, "properly dismounted");
 				} else {
 					printf(
 "WARNING: R/W mount of %s denied.  Filesystem is not clean - run fsck\n",
 					    fs->fs_fsmnt);
-					err = EPERM;
-					goto error_1;
+					return (EPERM);
 				}
 			}
-
+			if ((error = vn_start_write(NULL, &mp, V_WAIT)) != 0)
+				return (error);
+			fs->fs_ronly = 0;
+			fs->fs_clean = 0;
+			if ((error = ffs_sbupdate(ump, MNT_WAIT)) != 0) {
+				vn_finished_write(mp);
+				return (error);
+			}
 			/* check to see if we need to start softdep */
-			if (fs->fs_flags & FS_DOSOFTDEP) {
-				err = softdep_mount(devvp, mp, fs, p->p_ucred);
-				if (err)
-					goto error_1;
+			if ((fs->fs_flags & FS_DOSOFTDEP) &&
+			    (error = softdep_mount(devvp, mp, fs, p->p_ucred))){
+				vn_finished_write(mp);
+				return (error);
 			}
-
-			ronly = 0;
+			if (fs->fs_snapinum[0] != 0)
+				ffs_snapshot_mount(mp);
+			vn_finished_write(mp);
 		}
 		/*
 		 * Soft updates is incompatible with "async",
@@ -258,18 +259,18 @@ ffs_mount( mp, path, data, ndp, p)
 		 * Softdep_mount() clears it in an initial mount 
 		 * or ro->rw remount.
 		 */
-		if (mp->mnt_flag & MNT_SOFTDEP) {
+		if (mp->mnt_flag & MNT_SOFTDEP)
 			mp->mnt_flag &= ~MNT_ASYNC;
-		}
-		/* if not updating name...*/
-		if (args.fspec == 0) {
-			/*
-			 * Process export requests.  Jumping to "success"
-			 * will return the vfs_export() error code.
-			 */
-			err = vfs_export(mp, &ump->um_export, &args.export);
-			goto success;
-		}
+		/*
+		 * If not updating name, process export requests.
+		 */
+		if (args.fspec == 0)
+			return (vfs_export(mp, &ump->um_export, &args.export));
+		/*
+		 * If this is a snapshot request, take the snapshot.
+		 */
+		if (mp->mnt_flag & MNT_SNAPSHOT)
+			return (ffs_snapshot(mp, args.fspec));
 	}
 
 	/*
@@ -277,17 +278,14 @@ ffs_mount( mp, path, data, ndp, p)
 	 * and verify that it refers to a sensible block device.
 	 */
 	NDINIT(ndp, LOOKUP, FOLLOW, UIO_USERSPACE, args.fspec, p);
-	err = namei(ndp);
-	if (err) {
-		/* can't get devvp!*/
-		goto error_1;
-	}
-
+	if ((error = namei(ndp)) != 0)
+		return (error);
 	NDFREE(ndp, NDF_ONLY_PNBUF);
 	devvp = ndp->ni_vp;
-
-	if (!vn_isdisk(devvp, &err))
-		goto error_2;
+	if (!vn_isdisk(devvp, &error)) {
+		vrele(devvp);
+		return (error);
+	}
 
 	/*
 	 * If mount by non-root, then verify that user has necessary
@@ -298,7 +296,7 @@ ffs_mount( mp, path, data, ndp, p)
 		if ((mp->mnt_flag & MNT_RDONLY) == 0)
 			accessmode |= VWRITE;
 		vn_lock(devvp, LK_EXCLUSIVE | LK_RETRY, p);
-		if ((error = VOP_ACCESS(devvp, accessmode, p->p_ucred, p)) != 0) {
+		if ((error = VOP_ACCESS(devvp, accessmode, p->p_ucred, p))!= 0){
 			vput(devvp);
 			return (error);
 		}
@@ -307,96 +305,43 @@ ffs_mount( mp, path, data, ndp, p)
 
 	if (mp->mnt_flag & MNT_UPDATE) {
 		/*
-		 ********************
-		 * UPDATE
+		 * Update only
+		 *
 		 * If it's not the same vnode, or at least the same device
 		 * then it's not correct.
-		 ********************
 		 */
 
-		if (devvp != ump->um_devvp) {
-			if ( devvp->v_rdev == ump->um_devvp->v_rdev) {
-				vrele(devvp);
-			} else {
-				err = EINVAL;	/* needs translation */
-			}
-		} else
-			vrele(devvp);
-		/*
-		 * Update device name only on success
-		 */
-		if( !err) {
-			/* Save "mounted from" info for mount point (NULL pad)*/
-			copyinstr(	args.fspec,
-					mp->mnt_stat.f_mntfromname,
-					MNAMELEN - 1,
-					&size);
-			bzero( mp->mnt_stat.f_mntfromname + size, MNAMELEN - size);
-		}
+		if (devvp != ump->um_devvp &&
+		    devvp->v_rdev != ump->um_devvp->v_rdev)
+			error = EINVAL;	/* needs translation */
+		vrele(devvp);
+		if (error)
+			return (error);
 	} else {
 		/*
-		 ********************
-		 * NEW MOUNT
-		 ********************
+		 * New mount
+		 *
+		 * We need the name for the mount point (also used for
+		 * "last mounted on") copied in. If an error occurs,
+		 * the mount point is discarded by the upper level code.
 		 */
-
-		/*
-		 * Since this is a new mount, we want the names for
-		 * the device and the mount point copied in.  If an
-		 * error occurs,  the mountpoint is discarded by the
-		 * upper level code.
-		 */
-		/* Save "last mounted on" info for mount point (NULL pad)*/
-		copyinstr(	path,				/* mount point*/
-				mp->mnt_stat.f_mntonname,	/* save area*/
-				MNAMELEN - 1,			/* max size*/
-				&size);				/* real size*/
+		copyinstr(path, mp->mnt_stat.f_mntonname, MNAMELEN - 1, &size);
 		bzero( mp->mnt_stat.f_mntonname + size, MNAMELEN - size);
-
-		/* Save "mounted from" info for mount point (NULL pad)*/
-		copyinstr(	args.fspec,			/* device name*/
-				mp->mnt_stat.f_mntfromname,	/* save area*/
-				MNAMELEN - 1,			/* max size*/
-				&size);				/* real size*/
-		bzero( mp->mnt_stat.f_mntfromname + size, MNAMELEN - size);
-
-		err = ffs_mountfs(devvp, mp, p, M_FFSNODE);
-	}
-	if (err) {
-		goto error_2;
+		if ((error = ffs_mountfs(devvp, mp, p, M_FFSNODE)) != 0) {
+			vrele(devvp);
+			return (error);
+		}
 	}
-
-dostatfs:
 	/*
-	 * Initialize FS stat information in mount struct; uses both
-	 * mp->mnt_stat.f_mntonname and mp->mnt_stat.f_mntfromname
-	 *
-	 * This code is common to root and non-root mounts
+	 * Save "mounted from" device name info for mount point (NULL pad).
+	 */
+	copyinstr(args.fspec, mp->mnt_stat.f_mntfromname, MNAMELEN - 1, &size);
+	bzero( mp->mnt_stat.f_mntfromname + size, MNAMELEN - size);
+	/*
+	 * Initialize filesystem stat information in mount struct.
 	 */
 	(void)VFS_STATFS(mp, &mp->mnt_stat, p);
-
-	goto success;
-
-
-error_2:	/* error with devvp held*/
-
-	/* release devvp before failing*/
-	vrele(devvp);
-
-error_1:	/* no state to back out*/
-
-success:
-	if (!err && path && (mp->mnt_flag & MNT_UPDATE)) {
-		/* Update clean flag after changing read-onlyness. */
-		fs = ump->um_fs;
-		if (ronly != fs->fs_ronly) {
-			fs->fs_ronly = ronly;
-			fs->fs_clean = ronly &&
-			    (fs->fs_flags & FS_UNCLEAN) == 0 ? 1 : 0;
-			ffs_sbupdate(ump, MNT_WAIT);
-		}
-	}
-	return (err);
+	return (0);
 }
 
 /*
@@ -478,7 +423,7 @@ ffs_reload(mp, cred, p)
 	newfs->fs_maxcluster = fs->fs_maxcluster;
 	bcopy(newfs, fs, (u_int)fs->fs_sbsize);
 	if (fs->fs_sbsize < SBSIZE)
-		bp->b_flags |= B_INVAL;
+		bp->b_flags |= B_INVAL | B_NOCACHE;
 	brelse(bp);
 	mp->mnt_maxsymlinklen = fs->fs_maxsymlinklen;
 	ffs_oldfscompat(fs);
@@ -670,7 +615,7 @@ ffs_mountfs(devvp, mp, p, malloctype)
 	ump->um_vfree = ffs_vfree;
 	bcopy(bp->b_data, ump->um_fs, (u_int)fs->fs_sbsize);
 	if (fs->fs_sbsize < SBSIZE)
-		bp->b_flags |= B_INVAL;
+		bp->b_flags |= B_INVAL | B_NOCACHE;
 	brelse(bp);
 	bp = NULL;
 	fs = ump->um_fs;
@@ -750,6 +695,8 @@ ffs_mountfs(devvp, mp, p, malloctype)
 			free(base, M_UFSMNT);
 			goto out;
 		}
+		if (fs->fs_snapinum[0] != 0)
+			ffs_snapshot_mount(mp);
 		fs->fs_fmod = 1;
 		fs->fs_clean = 0;
 		(void) ffs_sbupdate(ump, MNT_WAIT);
@@ -886,6 +833,15 @@ ffs_flushfiles(mp, flags, p)
 		 */
 	}
 #endif
+	if (ump->um_devvp->v_flag & VCOPYONWRITE) {
+		if ((error = vflush(mp, NULL, SKIPSYSTEM | flags)) != 0)
+			return (error);
+		ffs_snapshot_unmount(mp);
+		/*
+		 * Here we fall through to vflush again to ensure
+		 * that we have gotten rid of all the system vnodes.
+		 */
+	}
         /*
 	 * Flush all the files.
 	 */
diff --git a/sys/ufs/ffs/ffs_vnops.c b/sys/ufs/ffs/ffs_vnops.c
index 539f302..eb6d621 100644
--- a/sys/ufs/ffs/ffs_vnops.c
+++ b/sys/ufs/ffs/ffs_vnops.c
@@ -95,6 +95,7 @@ vop_t **ffs_specop_p;
 static struct vnodeopv_entry_desc ffs_specop_entries[] = {
 	{ &vop_default_desc,		(vop_t *) ufs_vnoperatespec },
 	{ &vop_fsync_desc,		(vop_t *) ffs_fsync },
+	{ &vop_copyonwrite_desc,	(vop_t *) ffs_copyonwrite },
 	{ NULL, NULL }
 };
 static struct vnodeopv_desc ffs_specop_opv_desc =
@@ -129,11 +130,20 @@ ffs_fsync(ap)
 	} */ *ap;
 {
 	struct vnode *vp = ap->a_vp;
+	struct inode *ip = VTOI(vp);
 	struct buf *bp;
 	struct buf *nbp;
 	int s, error, wait, passes, skipmeta;
 	daddr_t lbn;
 
+	/*
+	 * Snapshots have to be unlocked so they do not deadlock
+	 * checking whether they need to copy their written buffers.
+	 * We always hold a reference, so they cannot be removed
+	 * out from underneath us.
+	 */
+	if (ip->i_flags & SF_SNAPSHOT)
+		VOP_UNLOCK(vp, 0, ap->a_p);
 	wait = (ap->a_waitfor == MNT_WAIT);
 	if (vn_isdisk(vp, NULL)) {
 		lbn = INT_MAX;
@@ -141,8 +151,6 @@ ffs_fsync(ap)
 		    (vp->v_specmountpoint->mnt_flag & MNT_SOFTDEP))
 			softdep_fsync_mountdev(vp);
 	} else {
-		struct inode *ip;
-		ip = VTOI(vp);
 		lbn = lblkno(ip->i_fs, (ip->i_size + ip->i_fs->fs_bsize - 1));
 	}
 
@@ -279,5 +287,7 @@ loop:
 	}
 	splx(s);
 	error = UFS_UPDATE(vp, wait);
+	if (ip->i_flags & SF_SNAPSHOT)
+		vn_lock(vp, LK_EXCLUSIVE | LK_RETRY, ap->a_p);
 	return (error);
 }
diff --git a/sys/ufs/ffs/softdep.h b/sys/ufs/ffs/softdep.h
index 1908a3e..cf9cac8 100644
--- a/sys/ufs/ffs/softdep.h
+++ b/sys/ufs/ffs/softdep.h
@@ -382,7 +382,7 @@ struct freefrag {
 	struct	worklist ff_list;	/* id_inowait or delayed worklist */
 #	define	ff_state ff_list.wk_state /* owning user; should be uid_t */
 	struct	vnode *ff_devvp;	/* filesystem device vnode */
-	struct	fs *ff_fs;		/* addr of superblock */
+	struct	mount *ff_mnt;		/* associated mount point */
 	ufs_daddr_t ff_blkno;		/* fragment physical block number */
 	long	ff_fragsize;		/* size of fragment being deleted */
 	ino_t	ff_inum;		/* owning inode number */
@@ -398,7 +398,7 @@ struct freeblks {
 	struct	worklist fb_list;	/* id_inowait or delayed worklist */
 	ino_t	fb_previousinum;	/* inode of previous owner of blocks */
 	struct	vnode *fb_devvp;	/* filesystem device vnode */
-	struct	fs *fb_fs;		/* addr of superblock */
+	struct	mount *fb_mnt;		/* associated mount point */
 	off_t	fb_oldsize;		/* previous file size */
 	off_t	fb_newsize;		/* new file size */
 	int	fb_chkcnt;		/* used to check cnt of blks released */
@@ -418,7 +418,7 @@ struct freefile {
 	mode_t	fx_mode;		/* mode of inode */
 	ino_t	fx_oldinum;		/* inum of the unlinked file */
 	struct	vnode *fx_devvp;	/* filesystem device vnode */
-	struct	fs *fx_fs;		/* addr of superblock */
+	struct	mount *fx_mnt;		/* associated mount point */
 };
 
 /*
diff --git a/sys/ufs/ufs/inode.h b/sys/ufs/ufs/inode.h
index 83960b0..6417a10 100644
--- a/sys/ufs/ufs/inode.h
+++ b/sys/ufs/ufs/inode.h
@@ -84,6 +84,7 @@ struct inode {
 	struct	 dquot *i_dquot[MAXQUOTAS]; /* Dquot structures. */
 	u_quad_t i_modrev;	/* Revision level for NFS lease. */
 	struct	 lockf *i_lockf;/* Head of byte-level lock list. */
+	struct	 inode *i_copyonwrite; /* copy-on-write list */
 	/*
 	 * Side effects; used during directory lookup.
 	 */
diff --git a/sys/ufs/ufs/ufs_bmap.c b/sys/ufs/ufs/ufs_bmap.c
index 9056340..ab4ac52 100644
--- a/sys/ufs/ufs/ufs_bmap.c
+++ b/sys/ufs/ufs/ufs_bmap.c
@@ -47,6 +47,7 @@
 #include <sys/vnode.h>
 #include <sys/mount.h>
 #include <sys/resourcevar.h>
+#include <sys/stat.h>
 
 #include <ufs/ufs/extattr.h>
 #include <ufs/ufs/quota.h>
@@ -115,7 +116,7 @@ ufs_bmaparray(vp, bn, bnp, ap, nump, runp, runb)
 	struct indir a[NIADDR+1], *xap;
 	ufs_daddr_t daddr;
 	long metalbn;
-	int error, maxrun, num;
+	int error, num, maxrun = 0;
 
 	ip = VTOI(vp);
 	mp = vp->v_mount;
@@ -127,6 +128,7 @@ ufs_bmaparray(vp, bn, bnp, ap, nump, runp, runb)
 #endif
 
 	if (runp) {
+		maxrun = mp->mnt_iosize_max / mp->mnt_stat.f_iosize - 1;
 		*runp = 0;
 	}
 
@@ -134,7 +136,6 @@ ufs_bmaparray(vp, bn, bnp, ap, nump, runp, runb)
 		*runb = 0;
 	}
 
-	maxrun = mp->mnt_iosize_max / mp->mnt_stat.f_iosize - 1;
 
 	xap = ap == NULL ? a : ap;
 	if (!nump)
@@ -146,9 +147,12 @@ ufs_bmaparray(vp, bn, bnp, ap, nump, runp, runb)
 	num = *nump;
 	if (num == 0) {
 		*bnp = blkptrtodb(ump, ip->i_db[bn]);
-		if (*bnp == 0)
-			*bnp = -1;
-		else if (runp) {
+		if (*bnp == 0) {
+			if (ip->i_flags & SF_SNAPSHOT)
+				*bnp = blkptrtodb(ump, bn * ump->um_seqinc);
+			else
+				*bnp = -1;
+		} else if (runp) {
 			daddr_t bnb = bn;
 			for (++bn; bn < NDADDR && *runp < maxrun &&
 			    is_sequential(ump, ip->i_db[bn - 1], ip->i_db[bn]);
@@ -226,8 +230,13 @@ ufs_bmaparray(vp, bn, bnp, ap, nump, runp, runb)
 	if (bp)
 		bqrelse(bp);
 
-	daddr = blkptrtodb(ump, daddr);
-	*bnp = daddr == 0 ? -1 : daddr;
+	*bnp = blkptrtodb(ump, daddr);
+	if (*bnp == 0) {
+		if (ip->i_flags & SF_SNAPSHOT)
+			*bnp = blkptrtodb(ump, bn * ump->um_seqinc);
+		else
+			*bnp = -1;
+	}
 	return (0);
 }
 
diff --git a/sys/ufs/ufs/ufs_extern.h b/sys/ufs/ufs/ufs_extern.h
index d576be9..b740792 100644
--- a/sys/ufs/ufs/ufs_extern.h
+++ b/sys/ufs/ufs/ufs_extern.h
@@ -87,6 +87,7 @@ int	 ufs_init __P((struct vfsconf *));
 void	 ufs_itimes __P((struct vnode *vp));
 int	 ufs_lookup __P((struct vop_cachedlookup_args *));
 int	 ufs_reclaim __P((struct vop_reclaim_args *));
+void	 ffs_snapremove __P((struct vnode *vp));
 int	 ufs_root __P((struct mount *, struct vnode **));
 int	 ufs_start __P((struct mount *, int, struct proc *));
 int	 ufs_vinit __P((struct mount *, vop_t **, vop_t **, struct vnode **));
diff --git a/sys/ufs/ufs/ufs_inode.c b/sys/ufs/ufs/ufs_inode.c
index 507e716..485a6d2 100644
--- a/sys/ufs/ufs/ufs_inode.c
+++ b/sys/ufs/ufs/ufs_inode.c
@@ -77,6 +77,7 @@ ufs_inactive(ap)
 	if (ip->i_mode == 0)
 		goto out;
 	if (ip->i_nlink <= 0 && (vp->v_mount->mnt_flag & MNT_RDONLY) == 0) {
+		(void) vn_write_suspend_wait(vp, V_WAIT);
 #ifdef QUOTA
 		if (!getinoquota(ip))
 			(void)chkiq(ip, -1, NOCRED, 0);
@@ -91,8 +92,15 @@ ufs_inactive(ap)
 		ip->i_flag |= IN_CHANGE | IN_UPDATE;
 		UFS_VFREE(vp, ip->i_number, mode);
 	}
-	if (ip->i_flag & (IN_ACCESS | IN_CHANGE | IN_MODIFIED | IN_UPDATE))
-		UFS_UPDATE(vp, 0);
+	if (ip->i_flag & (IN_ACCESS | IN_CHANGE | IN_MODIFIED | IN_UPDATE)) {
+		if ((ip->i_flag & (IN_CHANGE | IN_UPDATE | IN_MODIFIED)) == 0 &&
+		    vn_write_suspend_wait(vp, V_NOWAIT)) {
+			ip->i_flag &= ~IN_ACCESS;
+		} else {
+			(void) vn_write_suspend_wait(vp, V_WAIT);
+			UFS_UPDATE(vp, 0);
+		}
+	}
 out:
 	VOP_UNLOCK(vp, 0, p);
 	/*
diff --git a/sys/ufs/ufs/ufs_quota.c b/sys/ufs/ufs/ufs_quota.c
index 574a330..6396f67 100644
--- a/sys/ufs/ufs/ufs_quota.c
+++ b/sys/ufs/ufs/ufs_quota.c
@@ -889,6 +889,7 @@ dqsync(vp, dq)
 	struct vnode *dqvp;
 	struct iovec aiov;
 	struct uio auio;
+	struct mount *mp;
 	int error;
 
 	if (dq == NODQUOT)
@@ -897,6 +898,7 @@ dqsync(vp, dq)
 		return (0);
 	if ((dqvp = dq->dq_ump->um_quotas[dq->dq_type]) == NULLVP)
 		panic("dqsync: file");
+	(void) vn_write_suspend_wait(dqvp, V_WAIT);
 	if (vp != dqvp)
 		vn_lock(dqvp, LK_EXCLUSIVE | LK_RETRY, p);
 	while (dq->dq_flags & DQ_LOCK) {
diff --git a/sys/ufs/ufs/ufs_vnops.c b/sys/ufs/ufs/ufs_vnops.c
index e3b6e29..d97568c 100644
--- a/sys/ufs/ufs/ufs_vnops.c
+++ b/sys/ufs/ufs/ufs_vnops.c
@@ -702,6 +702,8 @@ ufs_remove(ap)
 	int error;
 
 	ip = VTOI(vp);
+	if ((ip->i_flags & SF_SNAPSHOT) != 0)
+		ffs_snapremove(vp);
 	if ((ip->i_flags & (NOUNLINK | IMMUTABLE | APPEND)) ||
 	    (VTOI(dvp)->i_flags & APPEND)) {
 		error = EPERM;
@@ -2215,6 +2217,7 @@ static struct vnodeopv_entry_desc ufs_vnodeop_entries[] = {
 	{ &vop_open_desc,		(vop_t *) ufs_open },
 	{ &vop_pathconf_desc,		(vop_t *) ufs_pathconf },
 	{ &vop_poll_desc,		(vop_t *) vop_stdpoll },
+	{ &vop_getwritemount_desc, 	(vop_t *) vop_stdgetwritemount },
 	{ &vop_print_desc,		(vop_t *) ufs_print },
 	{ &vop_readdir_desc,		(vop_t *) ufs_readdir },
 	{ &vop_readlink_desc,		(vop_t *) ufs_readlink },