diff options
Diffstat (limited to 'contrib/cvs/TESTS')
| -rw-r--r-- | contrib/cvs/TESTS | 240 |
1 files changed, 0 insertions, 240 deletions
diff --git a/contrib/cvs/TESTS b/contrib/cvs/TESTS deleted file mode 100644 index dc904d1..0000000 --- a/contrib/cvs/TESTS +++ /dev/null @@ -1,240 +0,0 @@ -To run the tests: - - $ make check - -Note that if your /bin/sh doesn't support shell functions, you'll -have to try something like this, where "/bin/sh5" is replaced by the -pathname of a shell which handles normal shell functions: - - $ make SHELL=/bin/sh5 check - -Also note that you must be logged in as a regular user, not root. - -WARNING: This test can take quite a while to run, esp. if your -disks are slow or over-loaded. - -The tests work in /tmp/cvs-sanity (which the tests create) by default. -If for some reason you want them to work in a different directory, you -can set the TESTDIR environment variable to the desired location -before running them. - -The tests use a number of tools (awk, expr, id, tr, etc.) that are not -required for running CVS itself. In most cases, the standard vendor- -supplied versions of these tools work just fine, but there are some -exceptions -- expr in particular is heavily used and many vendor -versions are deficient in one way or another. Note that some vendors -provide multiple versions of tools (typically an ancient, traditional -version and a new, standards-conforming version), so you may already -have a usable version even if the default version isn't. If you don't -have a suitable tool, you can probably get one from the GNU Project (see -http://www.gnu.org). At this writting, expr and id are both part of the -GNU shellutils package, tr is part of the GNU textutils package, and awk -is part of the GNU gawk package. The test script tries to verify that -the tools exist and are usable; if not, it tries to find the GNU -versions and use them instead. If it can't find the GNU versions -either, it will print an error message and, depending on the severity of -the deficiency, it may exit. There are environment variables you can -set to use a particular version of a tool -- see the test script -(src/sanity.sh) for details. - -Some of the tests use fairly long command lines -- this usually isn't a -problem, but if you have a very short command line length limit (or a -lot of environment variables), you may run into trouble. Also, some of -the tests expect your local timezone to be an integral number of hours -from UTC -- if you usually use a fractional timezone, use a different -(integral) timezone when running the tests to avoid spurious failures. - -If running the tests produces the output "FAIL:" followed by the name -of the test that failed, then the details on the failure are in the -file check.log. If it says "exit status is " followed by a number, -then the exit status of the command under test was not what the test -expected. If it says "** expected:" followed by a regular expression -followed by "** got:" followed by some text, then the regular -expression is the output which the test expected, and the text is the -output which the command under test actually produced. In some cases -you'll have to look closely to see how they differ. - -If output from "make remotecheck" is out of order compared to what is -expected (for example, - - a - b - cvs foo: this is a demo - -is expected and - - a - cvs foo: this is a demo - b - -is output), this is probably a well-known bug in the CVS server -(search for "out-of-order" in src/server.c for a comment explaining -the cause). It is a real pain in running the testsuite, but if you -are lucky and/or your machine is fast and/or lightly loaded, you won't -run into it. Running the tests again might succeed if the first run -failed in this manner. - -For more information on what goes in check.log, and how the tests are -run in general, you'll have to read sanity.sh. Depending on just what -you are looking for, and how familiar you are with the Bourne shell -and regular expressions, it will range from relatively straightforward -to obscure. - -If you choose to submit a bug report based on tests failing, be -aware that, as with all bug reports, you may or may not get a -response, and your odds might be better if you include enough -information to reproduce the bug, an analysis of what is going -wrong (if you have the time to provide one), etc. The check.log -file is the first place to look. - -ABOUT STDOUT AND STDERR -*********************** - -The sanity.sh test framework combines stdout and stderr and for tests -to pass requires that output appear in the given order. Some people -suggest that ordering between stdout and stderr should not be -required, or to put it another way, that the out-of-order bug referred -to above, and similar behaviors, should be considered features, or at -least tolerable. The reasoning behind the current behavior is that -having the output appear in a certain order is the correct behavior -for users using CVS interactively--that users get confused if the -order is unpredictable. - -ABOUT TEST FRAMEWORKS -********************* - -People periodically suggest using dejagnu or some other test -framework. A quick look at sanity.sh should make it clear that there -are indeed reasons to be dissatisfied with the status quo. Ideally a -replacement framework would achieve the following: - -1. Widely portable, including to a wide variety of unices, NT, Win95, -OS/2, VMS, probably DOS and Win3, etc. - -2. Nicely match extended regular expressions of unlimited length. - -3. Be freely redistributable, and if possible already the kind of -thing people might have already installed. The harder it is to get -and install the framework, the less people will run the tests. - -The various contenders are: - -* Bourne shell and GNU expr (the status quo). Falls short on #1 -(we've only tried unix and NT, although MKS might help with other DOS -mutants). #3 is pretty good (the main dependency is GNU expr which is -fairly widely available). - -* Bourne shell with a new regexp matcher we would distribute with -CVS. This means maintaining a regexp matcher and the makefiles which -go with it. Not clearly a win over Bourne shell and GNU expr. - -* Bourne shell, and use sed to remove variable portions of output, and -thus produce a form that can be compared with cmp or diff (this -sidesteps the need for a full regular expression matcher as mentioned -in #2 above). The C News tests are said to work this way. This would -appear to rely on variable portions of output having a certain syntax -and might spuriously recognize them out of context (this issue needs -more investigation; it isn't clear how big a problem it is in -practice). Same portability issues as the other choices based on the -Bourne shell. - -* Dejagnu. This is overkill; most of dejagnu is either unnecessary -(e.g. libraries for communicating with target boards) or undesirable -(e.g. the code which stats every file in sight to find the tests). On -the plus side, dejagnu is probably closer than any of the other -choices to having everything which is needed already there. - -* Write our own small framework directly in tcl and distribute with -CVS. The tests would look much like dejagnu tests, but we'd avoid the -unnecessary baggage. The only dependency would be on tcl (that is, -wish). - -* perl or python or <any other serious contenders here?> - -It is worth thinking about how to: - -a. include spaces in arguments which we pass to the program under -test (sanity.sh dotest cannot do this; see test rcs-9 for a -workaround). - -b. pass stdin to the program under test (sanity.sh, again, handles -this by bypassing dotest). - -c. have a send-expect type dialog with the program under test - (e.g. see server-7 or pserver-4 which want to talk the CVS - protocol, or the many tests which need to answer the prompt of "cvs - release", e.g. deep-5). - -ABOUT ADDING YOUR OWN TESTS -*************************** - -As stated in the HACKING file, patches are not accepted without documentation -and tests. Many people seem to be scared off by the large size of the -sanity.sh script, but it is not really very complicated. - -You can probably ignore most of the begining of the script. This section -just sets some environment variables and finds the tools the script needs to -run. - -There is one main loop you can find by grepping for "The big loop". This loop -repeatedly calls a case statement where the individual cases are of the form: - - testname) - ... - ;; - -If you add a complete new test be sure to add it into the default list of tests -(grep for 'tests=' near the begining of the script) as well as the case -statement. During debugging, be aware that the sanity.sh usage allows for a '-f -testname' option to continue through the default list "from" a particular test -as well as interpreting everything in argv past the required options as test -names to run individual tests. - -Within each major test section, individual tests usually look like: - - dotest testname-subtestname "shell command" "optionally multiline regexp" - -Tests should always start in $testdir and create a subdirectory to operate in -and remove their cruft and end back in $testdir. The dotest functions output -failure messages and exit if the shell command exits with the wrong exit code or -its stdin/stderr output doesn't match the regexp. There are a few dotest -variations, most notably dotest_fail for expected non-zero exit codes. - -Other than that the script is mostly vanilla Bourne shell. There are a few -constructs used for versatility and portability. You can grep for the ones I -miss, but here are a few important ones. I'm leaving off long explanations -after the first few since it probably gives you the idea and the data is in -sanity.sh. - -Note that the boolean variables contain shell commands which return true or -false when executed and are intended to be used like, -"if $remote; then ... ; else ... ; fi" - - - * $testdir = the directory this test is taking place in - (CVSROOT=$testdir/cvsroot or - CVSROOT=:fork:$testdir/cvsroot) - * $testcvs = full path to the cvs executable we are testing - * $PLUS = expr dependant uninterpreted '+' since this can vary - * $DOTSTAR = expr dependant _interpreted_ .* since some exprs don't - match EOL - * $username = the username of the user running the tests - * $username8 = the first 8 characters of $username, output by some - system and CVS commands - * $anyusername = regexp to match any valid system or CVS username - * $hostname = regexp to match a hostname - * $PROG = regexp to match progname in CVS error messages - * $remote = ':' (true) or 'false', depending on whether the script is - running with a remote CVSROOT - * $keep = ':' (true) or 'false'. When set, the first test run will - leave any files and directories it created in $testdir and - exit when complete. - -And, of course, some characters like '.' in regexps need to be '\' escaped when -you mean them literally. Some characters may be interpreted by the shell, -e.g. backquotes and '$', are usually either escaped or replaced with '.'. -dotest adds the final '$' anchor to the regexp itself and all the expr -implementations I know of implicitly supply the start anchor ('^'). - -If you only make a few mistakes, the work is, of course, still usable, though we -may send the patch back to you for repair. :) |
