Switch over to a different, more flexible test output protocol that's

understood by Perl's Test::Harness module and prove(1) commands.

Update README to describe the new protocol.  The work's broken down into
two main sets of changes.

First, update the existing test programs (shell scripts and C programs)
to produce output in the ok/not ok format, and to, where possible, also
produce a header describing the number of tests that are expected to be
run.

Second, provide the .t files that actually run the tests.  In some cases
these are copies of, or very similar too, scripts that already existed.
I've kept the old scripts around so that it's possible to verify that
behaviour under this new system (in terms of whether or not a test fails)
is identical to the behaviour under the old system.

Add a TODO file.

1 files changed, 52 insertions, 5 deletions

diff --git a/tools/regression/README b/tools/regression/README
index 0c98abb..3bc4c8e 100644
--- a/tools/regression/README
+++ b/tools/regression/README
@@ -5,16 +5,63 @@ This directory is for regression test programs.
 A regression test program is one that will exercise a particular bit of the
 system to check that we have not reintroduced an old bug.
 
+Tests should be implemented in files with a .t extension.  Each .t file
+can contain more than one test, and can be implemented in any scripting
+language -- /bin/sh, Perl...
+
+The test protocol is quite simple.  At its most basic, each .t file should, 
+when run, print a line in this format:
+
+   1..m
+
+where m is the number of tests that will be run.
+
+Each test should produce a single line of output.  This line should start
+with one of
+
+   ok n
+   not ok n
+
+to indicate whether or not the test succeeded.  'n' is the test's number.
+Anything after this on the line (up to the first '#' if present) is 
+considered to be the name of the test.  Naming tests is optional, but 
+encouraged.
+
+A test may be written which is conditional, and may need to be skipped.
+For example, the netatalk tests require 'options NETATALK' in the kernel.
+A test may be skipped by printing '# skip Reason for skipping' after the
+test name.  For example,
+
+    ok 1 - netatalk # skip 'options NETATALK' not compiled in
+
+A test may be flagged as 'todo'.  This indicates that you expect the test
+to fail (perhaps because the necessary functionality hasn't been written
+yet).  'todo' tests are expected to fail, so when they start working the
+test framework can alert you to this happy occurence.  Flag these tests 
+with a '# TODO' comment after the test name
+
+    not ok 1 - infiniteloop # TODO write test for an infinite loop
+
+This is modelled on the protocol followed by the Test::Harness Perl
+module (and therefore much of the automated testing carried out by the 
+Perl community).  More documentation can be found at:
+
+    http://search.cpan.org/~petdance/Test-Harness-2.42/lib/Test/Harness.pm
+
+To run the tests and parse their output install the devel/p5-Test-Harness
+port.  This includes the prove(1) command which is used to run the tests
+and collate the output.
+
+    prove geom_concat		# run all the tests in geom_concat
+    prove -r lib		# run all tests in lib/, and subdirectories
+    prove -r -v lib		# as above, with verbose output
+    prove -r			# run *all* the tests
+
 Tests that are for parts of the base system should go into a directory here
 which is the same as their path relative to src/, for example the uuencode(1)
 utility resides in src/usr.bin/uuencode so its regression test resides in
 src/tools/regression/usr.bin/uuencode.
 
-Base system regression tests should prefix their test results with 'PASS' or
-'FAIL' accordingly, to make checking for failed tests easier.  For now the
-tests are very free-form, and it's up to you to do that.  Eventually, it
-should be a part of the regression test framework.
-
 Please make a subdir per other regression test, and add a brief description to
 this file.