Add a script to perform simple analysis of a crash dump (either a full

dump or minidump).  When the script is run, it generates a text file
containing the output of several commands run againt the core dump such
as kgdb (stack trace), ps, netstat, vmstat, iostat, dmesg, and fstat.

Obtained from:	Yahoo!
MFC after:	2 weeks

4 files changed, 422 insertions, 0 deletions

diff --git a/usr.sbin/Makefile b/usr.sbin/Makefile
index a3190da..9a73744 100644
--- a/usr.sbin/Makefile
+++ b/usr.sbin/Makefile
@@ -33,6 +33,7 @@ SUBDIR=	ac \
 	ckdist \
 	clear_locks \
 	config \
+	crashinfo \
 	cron \
 	crunch \
 	ctm \
diff --git a/usr.sbin/crashinfo/Makefile b/usr.sbin/crashinfo/Makefile
new file mode 100644
index 0000000..fa2a19c
--- /dev/null
+++ b/usr.sbin/crashinfo/Makefile
@@ -0,0 +1,6 @@
+# $FreeBSD$
+
+SCRIPTS= crashinfo.sh
+MAN=	crashinfo.8
+
+.include <bsd.prog.mk>
diff --git a/usr.sbin/crashinfo/crashinfo.8 b/usr.sbin/crashinfo/crashinfo.8
new file mode 100644
index 0000000..e4ae67b
--- /dev/null
+++ b/usr.sbin/crashinfo/crashinfo.8
@@ -0,0 +1,109 @@
+.\" Copyright (c) 2008 Yahoo!, Inc.
+.\" All rights reserved.
+.\"
+.\" Redistribution and use in source and binary forms, with or without
+.\" modification, are permitted provided that the following conditions
+.\" are met:
+.\" 1. Redistributions of source code must retain the above copyright
+.\"    notice, this list of conditions and the following disclaimer.
+.\" 2. Redistributions in binary form must reproduce the above copyright
+.\"    notice, this list of conditions and the following disclaimer in the
+.\"    documentation and/or other materials provided with the distribution.
+.\" 3. Neither the name of the author nor the names of any co-contributors
+.\"    may be used to endorse or promote products derived from this software
+.\"    without specific prior written permission.
+.\"
+.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+.\" ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+.\" SUCH DAMAGE.
+.\"
+.\" $FreeBSD$
+.\"
+.Dd June 28, 2008
+.Dt CRASHINFO 8
+.Os
+.Sh NAME
+.Nm crashinfo
+.Nd "analyze a core dump of the operating system"
+.Sh SYNOPSIS
+.Nm
+.Op Fl d Ar crashdir
+.Op Fl n Ar dumpnr
+.Op Fl k Ar kernel
+.Op Ar core
+.Sh DESCRIPTION
+The
+.Nm
+utility analyzes a core dump saved by
+.Xr savecore 8 .
+It generates a text file containing the analysis in the same directory as
+the core dump.
+For a given core dump file named
+.Pa vmcore.XX
+the generated text file will be named
+.Pa core.txt.XX .
+.Pp
+By default,
+.Nm
+analyzes the most recent core dump in the core dump directory.
+A specific core dump may be specified via either the
+.Ar core
+or
+.Ar dumpnr
+arguments.
+Once
+.Nm
+has located a core dump,
+it analyzes the core dump to determine the exact version of the kernel
+that generated the core.
+It then looks for a matching kernel file under each of the subdirectories in
+.Pa /boot .
+The location of the kernel file can also be explicitly provided via the
+.Ar kernel
+argument.
+.Pp
+Once
+.Nm
+has located a core dump and kernel,
+it uses several utilities to analyze the core including
+.Xr dmesg 8 ,
+.Xr fstat 1 ,
+.Xr iostat 8 ,
+.Xr ipcs 1 ,
+.Xr kgdb 1 ,
+.Xr netstat 1 ,
+.Xr nfsstat 1 ,
+.Xr ps 1 ,
+.Xr pstat 8 ,
+and
+.Xr vmstat 8 .
+.Pp
+The options are as follows:
+.Bl -tag -width indent
+.It Fl d Ar crashdir
+Specify an alternate core dump directory.
+The default crash dump directory is 
+.Pa /var/crash .
+.It Fl n Ar dumpnr
+Use the core dump saved in
+.Pa vmcore. Ns Ar dumpnr
+instead of the latest core in the core dump directory. 
+.It Fl k Ar kernel
+Specify an explicit kernel file.
+.El
+.Sh SEE ALSO
+.Xr savecore 8 ,
+.Xr textdump 4
+.Sh HISTORY
+The
+.Nm
+utility appeared in
+.Fx 8.0 .
diff --git a/usr.sbin/crashinfo/crashinfo.sh b/usr.sbin/crashinfo/crashinfo.sh
new file mode 100755
index 0000000..cd41009
--- /dev/null
+++ b/usr.sbin/crashinfo/crashinfo.sh
@@ -0,0 +1,306 @@
+#!/bin/sh
+#
+# Copyright (c) 2008 Yahoo!, Inc.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the distribution.
+# 3. Neither the name of the author nor the names of any co-contributors
+#    may be used to endorse or promote products derived from this software
+#    without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+#
+# $FreeBSD$
+
+usage()
+{
+	echo "usage: crashinfo [-d crashdir] [-n dumpnr] [-k kernel] [core]"
+	exit 1
+}
+
+find_kernel()
+{
+	local ivers k kvers
+
+	ivers=$(awk '
+	/Version String/ {
+		print
+		nextline=1
+		next
+	}
+	// {
+		if (nextline) {
+			print
+			nextline=0
+		}
+	}' $INFO)
+
+	# Look for a matching kernel version.
+	for k in /boot/kernel/kernel $(ls -t /boot/*/kernel); do
+		kvers=$(echo 'printf "  Version String: %s", version' | \
+		    gdb -x /dev/stdin -batch $k 2>/dev/null)
+		if [ "$ivers" = "$kvers" ]; then
+			KERNEL=$k
+			break
+		fi
+	done
+}
+
+CRASHDIR=/var/crash
+DUMPNR=
+KERNEL=
+
+while getopts "d:n:k:" opt; do
+	case "$opt" in
+	d)
+		CRASHDIR=$OPTARG
+		;;
+	n)
+		DUMPNR=$OPTARG
+		;;
+	k)
+		KERNEL=$OPTARG
+		;;
+	\?)
+		usage
+		;;
+	esac
+done
+
+shift $((OPTIND - 1))
+
+if [ $# -eq 1 ]; then
+	if [ -n "$DUMPNR" ]; then
+		echo "-n and an explicit vmcore are mutually exclusive"
+		usage
+	fi
+
+	# Figure out the crash directory and number from the vmcore name.
+	CRASHDIR=`dirname $1`
+	DUMPNR=$(expr $(basename $1) : 'vmcore\.\([0-9]*\)$')
+	if [ -z "$DUMPNR" ]; then
+		echo "Unable to determine dump number from vmcore file $1."
+		exit 1
+	fi
+elif [ $# -gt 1 ]; then
+	usage
+else
+	# If we don't have an explicit dump number, operate on the most
+	# recent dump.
+	if [ -z "$DUMPNR" ]; then
+		if ! [ -r $CRASHDIR/bounds ]; then
+			echo "No crash dumps in $CRASHDIR."
+			exit 1
+		fi			
+		next=`cat $CRASHDIR/bounds`
+		if [ -z "$next" ] || [ "$next" -eq 0 ]; then
+			echo "No crash dumps in $CRASHDIR."
+			exit 1
+		fi
+		DUMPNR=$(($next - 1))
+	fi
+fi
+
+VMCORE=$CRASHDIR/vmcore.$DUMPNR
+INFO=$CRASHDIR/info.$DUMPNR
+FILE=$CRASHDIR/core.txt.$DUMPNR
+HOSTNAME=`hostname`
+
+if [ ! -e $VMCORE ]; then
+	echo "$VMCORE not found"
+	exit 1
+fi
+
+if [ ! -e $INFO ]; then
+	echo "$INFO not found"
+	exit 1
+fi
+
+# If the user didn't specify a kernel, then try to find one.
+if [ -z "$KERNEL" ]; then
+	find_kernel
+	if [ -z "$KERNEL" ]; then
+		echo "Unable to find matching kernel for $VMCORE"
+		exit 1
+	fi
+elif [ ! -e $KERNEL ]; then
+	echo "$KERNEL not found"
+	exit 1
+fi
+
+echo "Writing crash summary to $FILE."
+
+# Simulate uname
+ostype=$(echo -e printf '"%s", ostype' | gdb -x /dev/stdin -batch $KERNEL)
+osrelease=$(echo -e printf '"%s", osrelease' | gdb -x /dev/stdin -batch $KERNEL)
+version=$(echo -e printf '"%s", version' | gdb -x /dev/stdin -batch $KERNEL | \
+    tr '\t\n' '  ')
+machine=$(echo -e printf '"%s", machine' | gdb -x /dev/stdin -batch $KERNEL)
+
+exec > $FILE 2>&1
+
+echo "$HOSTNAME dumped core - see $VMCORE"
+echo
+date
+echo
+echo "$ostype $HOSTNAME $osrelease $version $machine"
+echo
+sed -ne '/^  Panic String: /{s//panic: /;p;}' $INFO
+echo
+
+# XXX: /bin/sh on 7.0+ is broken so we can't simply pipe the commands to
+# kgdb via stdin and have to use a temporary file instead.
+file=`mktemp /tmp/crashinfo.XXXXXX`
+if [ $? -eq 0 ]; then
+	echo "bt" >> $file
+	echo "quit" >> $file
+	kgdb $KERNEL $VMCORE < $file
+	rm -f $file
+	echo
+fi
+echo
+
+echo "------------------------------------------------------------------------"
+echo "ps -axl"
+echo
+ps -M $VMCORE -N $KERNEL -axl
+echo
+
+echo "------------------------------------------------------------------------"
+echo "vmstat -s"
+echo
+vmstat -M $VMCORE -N $KERNEL -s
+echo
+
+echo "------------------------------------------------------------------------"
+echo "vmstat -m"
+echo
+vmstat -M $VMCORE -N $KERNEL -m
+echo
+
+echo "------------------------------------------------------------------------"
+echo "vmstat -z"
+echo
+vmstat -M $VMCORE -N $KERNEL -z
+echo
+
+echo "------------------------------------------------------------------------"
+echo "vmstat -i"
+echo
+vmstat -M $VMCORE -N $KERNEL -i
+echo
+
+echo "------------------------------------------------------------------------"
+echo "pstat -T"
+echo
+pstat -M $VMCORE -N $KERNEL -T
+echo
+
+echo "------------------------------------------------------------------------"
+echo "pstat -s"
+echo
+pstat -M $VMCORE -N $KERNEL -s
+echo
+
+echo "------------------------------------------------------------------------"
+echo "iostat"
+echo
+iostat -M $VMCORE -N $KERNEL
+echo
+
+echo "------------------------------------------------------------------------"
+echo "ipcs -a"
+echo
+ipcs -C $VMCORE -N $KERNEL -a
+echo
+
+echo "------------------------------------------------------------------------"
+echo "ipcs -T"
+echo
+ipcs -C $VMCORE -N $KERNEL -T
+echo
+
+# XXX: This doesn't actually work in 5.x+
+if false; then
+echo "------------------------------------------------------------------------"
+echo "w -dn"
+echo
+w -M $VMCORE -N $KERNEL -dn
+echo
+fi
+
+echo "------------------------------------------------------------------------"
+echo "nfsstat"
+echo
+nfsstat -M $VMCORE -N $KERNEL
+echo
+
+echo "------------------------------------------------------------------------"
+echo "netstat -s"
+echo
+netstat -M $VMCORE -N $KERNEL -s
+echo
+
+echo "------------------------------------------------------------------------"
+echo "netstat -m"
+echo
+netstat -M $VMCORE -N $KERNEL -m
+echo
+
+echo "------------------------------------------------------------------------"
+echo "netstat -id"
+echo
+netstat -M $VMCORE -N $KERNEL -id
+echo
+
+echo "------------------------------------------------------------------------"
+echo "netstat -anr"
+echo
+netstat -M $VMCORE -N $KERNEL -anr
+echo
+
+echo "------------------------------------------------------------------------"
+echo "netstat -anA"
+echo
+netstat -M $VMCORE -N $KERNEL -anA
+echo
+
+echo "------------------------------------------------------------------------"
+echo "netstat -aL"
+echo
+netstat -M $VMCORE -N $KERNEL -aL
+echo
+
+echo "------------------------------------------------------------------------"
+echo "fstat"
+echo
+fstat -M $VMCORE -N $KERNEL
+echo
+
+echo "------------------------------------------------------------------------"
+echo "dmesg"
+echo
+dmesg -a -M $VMCORE -N $KERNEL
+echo
+
+echo "------------------------------------------------------------------------"
+echo "kernel config"
+echo
+config -x $KERNEL