Release the evil twin of nanobsd.sh: sysbuild.sh

quoth the README:

I have been running -current on my laptop since before FreeBSD 2.0 was
released and along the way developed this little trick to making the
task easier.

sysbuild.sh is a way to build a new FreeBSD system on a computer from
a specification, while leaving the current installation intact.

sysbuild.sh assume you have two partitions that can hold your rootfs
and can be booted, and roughly speaking, all it does is build a new
system into the one you don't use, from the one you do use.

A partition named /freebsd is assumed to be part of your layout, and
that is where the sources and ports will be found.

If you know how nanobsd works, you will find a lot of similarity.

2 files changed, 681 insertions, 0 deletions

diff --git a/tools/tools/sysbuild/README b/tools/tools/sysbuild/README
new file mode 100644
index 0000000..2d35304
--- /dev/null
+++ b/tools/tools/sysbuild/README
@@ -0,0 +1,153 @@
+$FreeBSD$
+
+About sysbuild.sh
+=================
+
+I have been running -current on my laptop since before FreeBSD 2.0 was
+released and along the way developed this little trick to making the
+task easier.
+
+sysbuild.sh is a way to build a new FreeBSD system on a computer from
+a specification, while leaving the current installation intact.
+
+sysbuild.sh assume you have two partitions that can hold your rootfs
+and can be booted, and roughly speaking, all it does is build a new
+system into the one you don't use, from the one you do use.
+
+A partition named /freebsd is assumed to be part of your layout, and
+that is where the sources and ports will be found.
+
+If you know how nanobsd works, you will find a lot of similarity.
+
+HOWTO
+=====
+
+In all likelyhood, it is easier if we imagine you start with a blank
+computer.
+
+Grab a FreeBSD install ISO and boot it.
+
+Create four disk slices:
+
+	ad0s1 = 5GB
+	ad0s2 = 5GB
+	ad0s3 = 5GB
+	ad0s4 = the rest
+
+Create a root filesystem in s1a filling the entire ad0s1 slice.
+
+Create a swap partition, if you want one, in ad0s4b.
+
+Install the boot0 bootmanager.
+
+Install the "Minimal" FreeBSD system into ad0s1a.
+
+Reboot from the newly installed system.
+
+Run these commands to set up the other partitions sysbuild.sh cares about:
+
+	# /freebsd filesystem
+	newfs -b 4096 -f 512 -O2 -U /dev/ad0s3
+	echo "/dev/ad0s3 /freebsd ufs rw 2 2" >> /etc/fstab
+	mkdir /freebsd
+	mount /freebsd
+
+	# deputy rootfilesystem
+	bsdlabel -B -w /dev/ad0s2
+	newfs -O2 -U /dev/ad0s2a
+
+Next, install ports and sources:
+
+	cd /usr
+	rm -rf ports src
+	ln -s /freebsd/src
+	ln -s /freebsd/ports
+	cd /freebsd
+	mkdir ports src packages
+
+	# Or use svn if you prefer
+	csup -h cvsup.???.freebsd.org /usr/share/examples/cvsup/ports-supfile
+	csup -h cvsup.???.freebsd.org /usr/share/examples/cvsup/stable-supfile
+
+And we should be ready to try a shot:
+
+	cd /root
+	cp /usr/src/tools/tools/sysbuild/sysbuild.sh
+	sh sysbuild.sh |& tee _.sb
+
+If it succeeds, you should be able to:
+
+	boot0cfg -s 2 -v /dev/ad0
+	reboot
+
+And come up with your newly built system.
+
+	Next time you want a new system, you just run sysbuild.sh again
+	and boot slice 1 when it's done.
+
+TWEAKS
+======
+
+The sysbuild.sh script takes various parameters:
+
+	-c specfile	# configure stuff, see below.
+	-w		# skip buildworld, assume it was done earlier.
+	-k		# skip buildkernel, ---//---
+	-b		# skip both buildworld & buildkernel
+	-p		# install cached packacges if found.
+
+The specfile is a shellscript where you can override or set a number of
+shell variables and functions.
+
+A partial example:
+
+	# use a kernel different from GENERIC
+	KERNCONF=SMP
+
+	# Cache built packages, so we can use -p
+	PKG_DIR=/freebsd/packages
+
+	# Mount ports distfiles from another machine
+	REMOTEDISTFILES=fs:/rdonly/distfiles
+
+	# Fetch distfiles through a proxy
+	FTP_PROXY=http://127.0.0.1:3128/
+	HTTP_PROXY=http://127.0.0.1:3128/
+	export FTP_PROXY HTTP_PROXY
+
+	# We want these ports
+	PORTS_WE_WANT='
+		/usr/ports/archivers/unzip
+		/usr/ports/archivers/zip
+		/usr/ports/cad/linux-eagle
+		/usr/ports/comms/lrzsz
+		/usr/ports/databases/rrdtool 
+		/usr/ports/devel/subversion-freebsd
+	'
+
+	# Files to move over
+	CONFIGFILES='
+		/root/.ssh
+		/etc/X11/xorg.conf
+		/etc/ssh/ssh_host*
+		/etc/rc.conf
+		/etc/rc.local
+	'
+
+	# Shell functions to tweak things
+	# (This makes commits to /etc mostly painless)
+	final_chroot() (
+		chpass -p "\$1\$IgMjWs2L\$Nu12OCsjfiwHHj0I7TmUN1" root
+
+		pw useradd phk -u 488 -d /home/phk -c "Poul-Henning Kamp" \
+		    -G "wheel,operator,dialer" -s /bin/csh -w none
+
+		chpass -p "\$1\$VcM.9Ow8\$IcXHs0h9jsk27b8N64lOm/" phk
+
+		sed -i "" -e 's/^DS/DSorigo.freebsd.dk/' /etc/mail/sendmail.cf
+		sed -i "" -e '/console/s/^/#/' /etc/syslog.conf
+		echo "beastie_disable=YES" >> /boot/loader.conf
+		touch /root/.hushlogin
+	)
+
+
diff --git a/tools/tools/sysbuild/sysbuild.sh b/tools/tools/sysbuild/sysbuild.sh
new file mode 100644
index 0000000..3f084da
--- /dev/null
+++ b/tools/tools/sysbuild/sysbuild.sh
@@ -0,0 +1,528 @@
+#!/bin/sh
+#
+# Copyright (c) 1994-2009 Poul-Henning Kamp.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+#
+# $FreeBSD$
+#
+
+set -e
+
+exec < /dev/null
+
+if [ `uname -m` = "i386" ] ; then
+	TARGET_PART=`df / | sed '
+	1d
+	s/[    ].*//
+	s,/dev/,,
+	s,s1a,s3a,
+	s,s2a,s1a,
+	s,s3a,s2a,
+	'`
+
+	# Where our build-bits are to be found
+	FREEBSD_PART=`echo $TARGET_PART | sed 's/s[12]a/s3/'`
+else
+	TARGET_PART=unknown
+	FREEBSD_PART=unknown
+fi
+
+# Relative to /freebsd
+PORTS_PATH=ports
+SRC_PATH=src
+# OBJ_PATH=obj
+
+# Name of kernel
+KERNCONF=GENERIC
+
+# srcconf
+#SRCCONF="SRCCONF=/usr/src/src.conf"
+
+# -j arg to make(1)
+
+ncpu=`sysctl -n kern.smp.cpus`
+if [ $ncpu -gt 1 ] ; then
+	JARG="-j $ncpu"
+fi
+
+# serial console ?
+SERCONS=false
+
+# Remotely mounted distfiles
+# REMOTEDISTFILES=fs:/rdonly/distfiles
+
+# Proxy
+#FTP_PROXY=http://127.0.0.1:3128/
+#HTTP_PROXY=http://127.0.0.1:3128/
+#export FTP_PROXY HTTP_PROXY
+
+PORTS_WE_WANT='
+'
+
+PORTS_OPTS="BATCH=YES MAKE_IDEA=YES A4=yes"
+
+CONFIGFILES='
+'
+
+cleanup() (
+)
+
+before_ports() (
+)
+
+before_ports_chroot() (
+)
+
+final_root() (
+)
+
+final_chroot() (
+)
+
+#######################################################################
+#######################################################################
+
+usage () {
+	(
+        echo "Usage: $0 [-b/-k/-w] [-c config_file]"
+        echo "  -b      suppress builds (both kernel and world)"
+        echo "  -k      suppress buildkernel"
+        echo "  -w      suppress buildworld"
+        echo "  -p      used cached packages"
+        echo "  -c      specify config file"
+        ) 1>&2
+        exit 2
+}
+
+#######################################################################
+#######################################################################
+
+if [ ! -f $0 ] ; then
+	echo "Must be able to access self ($0)" 1>&2
+	exit 1
+fi
+
+if grep -q 'Magic String: 0`0nQT40W%l,CX&' $0 ; then
+	true
+else
+	echo "self ($0) does not contain magic string" 1>&2
+	exit 1
+fi
+
+#######################################################################
+
+set -e
+
+log_it() (
+	set +x
+	a="$*"
+	set `cat /tmp/_sb_log`
+	TX=`date +%s`
+	echo "$1 $TX" > /tmp/_sb_log
+	DT=`expr $TX - $1 || true`
+	DL=`expr $TX - $2 || true`
+	echo -n "### `date +%H:%M:%S`"
+	printf " ### %5d ### %5d ### %s\n" $DT $DL "$a"
+)
+
+#######################################################################
+
+
+ports_recurse() (
+	set +x
+	for d
+	do
+		if [ ! -d $d ] ; then
+			echo "Missing port $d" 1>&2
+			exit 2
+		fi
+		if grep -q "^$d\$" /tmp/_.plist ; then
+			true
+		else
+			(
+			cd $d
+			ports_recurse `make -V _DEPEND_DIRS`
+			)
+			echo $d >> /tmp/_.plist
+		fi
+	done
+)
+
+ports_build() (
+	set +x
+
+	true > /tmp/_.plist
+	ports_recurse $PORTS_WE_WANT 
+
+	# Now build & install them
+	for p in `cat /tmp/_.plist`
+	do
+		t=`echo $p | sed 's,/usr/ports/,,'`
+		pn=`cd $p && make package-name`
+		if [ "x${PKG_DIR}" != "x" -a -f ${PKG_DIR}/$pn.tbz ] ; then
+			if [ "x$use_pkg" = "x-p" ] ; then
+				log_it "install $p from ${PKG_DIR}/$pn.tbz"
+				pkg_add ${PKG_DIR}/$pn.tbz
+			fi
+		fi
+		i=`pkg_info -qO $t`
+		if [ -z "$i" ] ; then
+			log_it "build $p"
+			b=`echo $p | tr / _`
+			(
+				set -x
+				cd /usr/ports
+				cd $p
+				set +e
+				make clean
+				if make install ${PORTS_OPTS} ; then
+					if [ "x${PKG_DIR}" != "x" ] ; then
+						make package ${PORTS_OPTS}
+						mv *.tbz ${PKG_DIR}
+					fi
+				else
+					log_it FAIL build $p
+				fi
+				make clean
+			) > _.$b 2>&1 < /dev/null
+			date
+		fi
+	done
+)
+
+ports_prefetch() (
+	(
+	set +x
+	ports_recurse $PORTS_WE_WANT
+
+	# Now checksump/fetch them
+	for p in `cat /tmp/_.plist`
+	do
+		b=`echo $p | tr / _`
+		(
+			cd $p
+			if make checksum $PORTS_OPTS ; then
+				true
+			else
+				make distclean
+				make checksum $PORTS_OPTS || true
+			fi
+		) > /mnt/_.prefetch.$b 2>&1
+	done
+	) 
+)
+
+#######################################################################
+
+do_world=true
+do_kernel=true
+use_pkg=""
+c_arg=""
+
+set +e
+args=`getopt bc:hkpw $*`
+if [ $? -ne 0 ] ; then
+	usage
+fi
+set -e
+
+set -- $args
+for i
+do
+	case "$i"
+	in
+	-b)
+		shift;
+		do_world=false
+		do_kernel=false
+		;;
+	-c)
+		c_arg=$2
+		if [ ! -f "$c_arg" ] ; then
+			echo "Cannot read $c_arg" 1>&2
+			usage
+		fi
+		. "$2"
+		shift
+		shift
+		;;
+	-h)
+		usage
+		;;
+	-k)
+		shift;
+		do_kernel=false
+		;;
+	-p)
+		shift;
+		use_pkg="-p"
+		;;
+	-w)
+		shift;
+		do_world=false
+		;;
+	--)
+		shift
+		break;
+		;;
+	esac
+done
+
+#######################################################################
+
+if [ "x$1" = "xchroot_script" ] ; then
+	set +x
+	set -e
+
+	shift
+
+	before_ports_chroot
+
+	ports_build
+
+	exit 0
+fi
+
+if [ "x$1" = "xfinal_chroot" ] ; then
+	final_chroot
+	exit 0
+fi
+
+if [ $# -gt 0 ] ; then
+        echo "$0: Extraneous arguments supplied"
+        usage
+fi
+
+#######################################################################
+
+T0=`date +%s`
+echo $T0 $T0 > /tmp/_sb_log
+
+log_it Unmount everything
+(
+	( cleanup )
+	umount /freebsd/distfiles || true
+	umount /mnt/freebsd/distfiles || true
+	umount /dev/${FREEBSD_PART} || true
+	umount /mnt/freebsd || true
+	umount /mnt/dev || true
+	umount /mnt || true
+	umount /dev/${TARGET_PART} || true
+) # > /dev/null 2>&1
+
+log_it Prepare running image
+mkdir -p /freebsd
+mount /dev/${FREEBSD_PART} /freebsd
+
+#######################################################################
+
+if [ ! -d /freebsd/${PORTS_PATH} ] ;  then
+	echo PORTS_PATH does not exist 1>&2
+	exit 1
+fi
+
+if [ ! -d /freebsd/${SRC_PATH} ] ;  then
+	echo SRC_PATH does not exist 1>&2
+	exit 1
+fi
+
+log_it TARGET_PART $TARGET_PART
+sleep 5
+
+rm -rf /usr/ports
+ln -s /freebsd/${PORTS_PATH} /usr/ports
+
+rm -rf /usr/src
+ln -s /freebsd/${SRC_PATH} /usr/src
+
+if $do_world ; then
+	if [ "x${OBJ_PATH}" != "x" ] ; then
+		rm -rf /usr/obj
+		mkdir -p /freebsd/${OBJ_PATH}
+		ln -s /freebsd/${OBJ_PATH} /usr/obj
+	else
+		rm -rf /usr/obj
+		mkdir -p /usr/obj
+	fi
+fi
+
+#######################################################################
+
+for i in ${PORTS_WE_WANT}
+do
+	if [ ! -d $i ]  ; then
+		echo "Port $i not found" 1>&2
+		exit 2
+	fi
+done
+
+export PORTS_WE_WANT
+export PORTS_OPTS
+
+#######################################################################
+
+log_it Prepare destination partition
+newfs -O2 -U /dev/${TARGET_PART} > /dev/null
+mount /dev/${TARGET_PART} /mnt
+mkdir -p /mnt/dev
+mount -t devfs devfs /mnt/dev
+
+if [ "x${REMOTEDISTFILES}" != "x" ] ; then
+	rm -rf /freebsd/${PORTS_PATH}/distfiles
+	ln -s /freebsd/distfiles /freebsd/${PORTS_PATH}/distfiles
+	mkdir -p /freebsd/distfiles
+	mount  ${REMOTEDISTFILES} /freebsd/distfiles
+fi
+
+log_it "Start prefetch of ports distfiles"
+ports_prefetch &
+
+if $do_world ; then
+	(
+	cd /usr/src
+	log_it "Buildworld"
+	make ${JARG} -s buildworld ${SRCCONF} > /mnt/_.bw 2>&1
+	)
+fi
+
+if $do_kernel ; then
+	(
+	cd /usr/src
+	log_it "Buildkernel"
+	make ${JARG} -s buildkernel KERNCONF=$KERNCONF > /mnt/_.bk 2>&1
+	)
+fi
+
+
+log_it Installworld
+(cd /usr/src && make ${JARG} installworld DESTDIR=/mnt ${SRCCONF} ) \
+	> /mnt/_.iw 2>&1
+
+log_it distribution
+(cd /usr/src/etc && make distribution DESTDIR=/mnt ${SRCCONF} ) \
+	> /mnt/_.dist 2>&1
+
+log_it Installkernel
+(cd /usr/src && make ${JARG} installkernel DESTDIR=/mnt KERNCONF=$KERNCONF ) \
+	> /mnt/_.ik 2>&1
+
+if [ "x${OBJ_PATH}" != "x" ] ; then
+	rmdir /mnt/usr/obj
+	ln -s /freebsd/${OBJ_PATH} /mnt/usr/obj
+fi
+
+log_it Wait for ports prefetch
+wait
+
+log_it Move filesystems
+
+if [ "x${REMOTEDISTFILES}" != "x" ] ; then
+	umount /freebsd/distfiles
+fi
+umount /dev/${FREEBSD_PART} || true
+mkdir -p /mnt/freebsd
+mount /dev/${FREEBSD_PART} /mnt/freebsd
+if [ "x${REMOTEDISTFILES}" != "x" ] ; then
+	mount  ${REMOTEDISTFILES} /mnt/freebsd/distfiles
+fi
+
+rm -rf /mnt/usr/ports || true
+ln -s /freebsd/${PORTS_PATH} /mnt/usr/ports
+
+rm -rf /mnt/usr/src || true
+ln -s /freebsd/${SRC_PATH} /mnt/usr/src
+
+log_it Build and install ports
+
+# Make sure fetching will work in the chroot
+if [ -f /etc/resolv.conf ] ; then
+	log_it copy resolv.conf
+	cp /etc/resolv.conf /mnt/etc
+	chflags schg /mnt/etc/resolv.conf
+fi
+
+if [ -f /etc/localtime ] ; then
+	log_it copy localtime
+	cp /etc/localtime /mnt/etc
+fi
+
+log_it copy ports config files
+(cd / ; find var/db/ports -print | cpio -dumpv /mnt )
+
+log_it ldconfig in chroot
+chroot /mnt sh /etc/rc.d/ldconfig start
+
+log_it before_ports
+( 
+	before_ports 
+)
+
+log_it build ports
+pwd
+cp $0 /mnt/root
+cp /tmp/_sb_log /mnt/tmp
+b=`basename $0`
+if [ "x$c_arg" != "x" ] ; then
+	cp $c_arg /mnt/root
+	chroot /mnt sh /root/$0 -c /root/`basename $c_arg` $use_pkg chroot_script 
+else
+	chroot /mnt sh /root/$0 $use_pkg chroot_script
+fi
+cp /mnt/tmp/_sb_log /tmp
+
+log_it fixing fstab
+sed "/[ 	]\/[ 	]/s;^[^ 	]*[ 	];/dev/${TARGET_PART}	;" \
+	/etc/fstab > /mnt/etc/fstab
+
+log_it create all mountpoints
+grep -v '^[ 	]*#' /mnt/etc/fstab | 
+while read a b c
+do
+	mkdir -p /mnt/$b
+done
+
+if [ "x$SERCONS" != "xfalse" ] ; then
+	log_it serial console
+	echo " -h" > /mnt/boot.config
+	sed -i "" -e /ttyd0/s/off/on/ /mnt/etc/ttys
+	sed -i "" -e /ttyu0/s/off/on/ /mnt/etc/ttys
+	sed -i "" -e '/^ttyv[0-8]/s/	on/	off/' /mnt/etc/ttys
+fi
+
+log_it move config files
+(cd / && find ${CONFIGFILES} -print | cpio -dumpv /mnt)
+
+log_it final_root
+( final_root )
+log_it final_chroot
+cp /tmp/_sb_log /mnt/tmp
+if [ "x$c_arg" != "x" ] ; then
+	chroot /mnt sh /root/$0 -c /root/`basename $c_arg` final_chroot
+else
+	chroot /mnt sh /root/$0 final_chroot
+fi
+cp /mnt/tmp/_sb_log /tmp
+log_it "Check these messages (if any):"
+grep '^Stop' /mnt/_* || true
+log_it DONE