diff options
Diffstat (limited to 'contrib/atf/doc')
-rw-r--r-- | contrib/atf/doc/atf-formats.5 | 231 | ||||
-rw-r--r-- | contrib/atf/doc/atf-test-case.4 | 319 | ||||
-rw-r--r-- | contrib/atf/doc/atf-test-program.1 | 103 | ||||
-rw-r--r-- | contrib/atf/doc/atf.7.in | 192 |
4 files changed, 845 insertions, 0 deletions
diff --git a/contrib/atf/doc/atf-formats.5 b/contrib/atf/doc/atf-formats.5 new file mode 100644 index 0000000..bb919f4 --- /dev/null +++ b/contrib/atf/doc/atf-formats.5 @@ -0,0 +1,231 @@ +.\" +.\" Automated Testing Framework (atf) +.\" +.\" Copyright (c) 2007 The NetBSD Foundation, Inc. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" 2. Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND +.\" CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, +.\" INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +.\" MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. +.\" IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS BE LIABLE FOR ANY +.\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE +.\" GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER +.\" IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR +.\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN +.\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.\" +.Dd December 20, 2011 +.Dt ATF-FORMATS 5 +.Os +.Sh NAME +.Nm atf-formats +.Nd machine-parseable data formats used by ATF +.Sh DESCRIPTION +This manual page describes the multiple data formats used in ATF. +These formats affect configuration files, control files and any data that +is externalized or internalized by the tools. +.Pp +Data files are always organized as follows: +.Bd -literal -offset indent +Header1: Value1 \\ + ... | head +HeaderN: ValueN / + mandatory blank line +Free-form text contents \\ + ... | body + ... / +.Ed +.Pp +A file must always contain a +.Sq Content-Type +header and must always separate that header from the body with a blank +line, even if the body is empty. +.Pp +The +.Sq Content-Type +is always of the form: +.Bd -literal -offset indent +Content-Type: application/X-atf-<subtype>; version="<version>" +.Ed +.Pp +where +.Sq subtype +indicates the specific file format and +.Sq version +its format version. +This header must be the first one of the file. +.Pp +The main purpose of the +.Sq Content-Type +header, aside from determining the format used in the file, is to allow +future changes to a given format. +Whenever an incompatible change is made, the version is bumped by one. +By keeping the header in the first line, future versions may even remove +the need for such a header -- e.g. if some format was replaced by XML +files, which have their own mandatory header. +.Pp +The rest of this document details the different format types. +.Ss Format: application/X-atf-atffile, version: 1 +Atffiles are logically divided into three sections: +.Bl -bullet +.It +Test programs: the list of test programs that define the test suite +described by the Atffile. +.It +Meta-data properties: these define some constant values applicable to +all the test programs defined in the file. +In some sense they define the properties that describe the test suite. +.It +Configuration variables: defaults for configuration variables that +can be overridden through configuration files or the command line. +.El +.Pp +The grammar for Atffiles is the following: +.Bd -literal -offset indent +DATA ::= ( ( CONF | PROP | TP )? COMMENT? NEWLINE )* EOF +CONF ::= 'conf:' WORD EQUAL STRING +PROP ::= 'prop:' WORD EQUAL STRING +TP ::= TPFILE | TPGLOB +TPFILE ::= 'tp: ' STRING +TPGLOB ::= 'tp-glob: ' STRING +STRING ::= WORD | '"' WORD* '"' +.Ed +.Pp +The meaning of the constructions above is: +.Bl -tag -width TPGLOBXX +.It CONF +Definition of a configuration variable. +.It PROP +Definition of a meta-data property. +.It TPFILE +Addition of a test program into the test suite. +The string is taken literally as the program's name, and this program +must exist. +.It TPGLOB +Addition of multiple test programs into the test suite. +The string is taken as a glob pattern, which may have or not have any +matches in the current directory. +.El +.Pp +An example: +.Bd -literal -offset indent +prop: test-suite = utilities + +conf: unprivileged-user = nobody + +tp: t_cp +tp: t_mv +tp: t_df +tp-glob: t_dir_* +.Ed +.Ss Format: application/X-atf-config, version: 1 +Configuration files are very simple: they only contain a list of variable +name/variable value pairs. +Their grammar is: +.Bd -literal -offset indent +DATA ::= ( VAR? COMMENT? NEWLINE )* EOF +VAR ::= WORD EQUAL STRING +COMMENT ::= HASH WORD* +STRING ::= WORD | '"' WORD* '"' +.Ed +.Pp +An example: +.Bd -literal -offset indent +# This is the system-wide configuration file for ATF. +# The above and this line are comments placed on their own line. + +var1 = this is a variable value +var2 = this is another one # Optional comment at the end. +.Ed +.Ss Format: application/X-atf-tps, version: 3 +The +.Sq application/X-atf-tps +format multiplexes the standard output, standard error and results output +streams from multiple test programs into a single data file. +This format is used by +.Xr atf-run 1 +to report the execution of several test programs and is later parsed by +.Xr atf-report 1 +to inform the user of this process. +It has the following grammar: +.Bd -literal -offset indent +DATA ::= INFO* TPS-COUNT TP-STANZA* INFO* EOF +INFO ::= 'info' COLON STRING COMMA STRING NEWLINE +TPS-COUNT ::= 'tps-count' COLON POSITIVE-NUMBER NEWLINE +TP-STANZA ::= TP-START TC-STANZA* TC-END +TP-START ::= 'tp-start' COLON TIMESTAMP COMMA STRING COMMA + POSITIVE-NUMBER NEWLINE +TP-END ::= 'tc-end' COLON TIMESTAMP COMMA STRING (COMMA STRING)? +TC-STANZA ::= TC-START (TC-SO | TC-SE)* TC-END +TC-START ::= 'tc-start' COLON TIMESTAMP COMMA STRING NEWLINE +TC-SO ::= 'tc-so' COLON STRING NEWLINE +TC-SE ::= 'tc-se' COLON STRING NEWLINE +TC-END ::= 'tc-end' COLON TIMESTAMP COMMA STRING COMMA TCR NEWLINE +TCR ::= 'passed' | ('failed' | 'skipped') COMMA STRING +TIMESTAMP ::= [0-9]+.[0-9]+ +.Ed +.Pp +The meaning of the constructions above is: +.Bl -tag -width TPSXCOUNTXX +.It TPS-COUNT +Indicates the number of test programs that will be executed. +There will be this exact amount of +.Sq TP-STANZA +constructions following it. +.It TP-START +Indicates the beginning of a test program. +This includes the program's name and the amount of test cases that +will follow. +.It TP-END +Indicates the completion of a test program. +This is followed by the program's name and, if the program ended +prematurely, an error message indicating the reason of its failure. +A successful execution of a test program (regardless of the status of its +test cases) must not be accompanied by any reason. +.It TC-START +Indicates the beginning of a test case. +This is accompanied by the test case's name. +.It TC-SO +Contains a text line sent to the standard output stream during the +execution of the test case. +Leading and trailing space is preserved. +.It TC-SE +Contains a text line sent to the standard error stream during the +execution of the test case. +Leading and trailing space is preserved. +.It TC-END +Indicates the completion of a test case. +This is accompanied by the test case's name, its result and the reason +associated with this result (if applicable). +.El +.Pp +An example: +.Bd -literal -offset indent +tps-count: 2 +tp-start: calculator, 1324318951.838923, 2 +tc-start: add, 1324318951.839101 +tc-end: add, 1324318951.839500, passed +tc-start: subtract, 1324318951.840001 +tc-so: 3-2 expected to return 1 but got 0 +tc-end: subtract, 1324318952.000123, failed, Calculated an unexpected value +tp-end: calculator, 1324318952.002301 +tp-start: files, 1, 1324318952.502348 +tc-start: copy, 1324318952.508291 +tc-se: could not find the cp(1) utility +tc-end: copy, 1324318953.203145, skipped +tp-end: files, 1324318953.203800 +.Ed +.Sh SEE ALSO +.Xr atf 7 diff --git a/contrib/atf/doc/atf-test-case.4 b/contrib/atf/doc/atf-test-case.4 new file mode 100644 index 0000000..e1b2bf4 --- /dev/null +++ b/contrib/atf/doc/atf-test-case.4 @@ -0,0 +1,319 @@ +.\" +.\" Automated Testing Framework (atf) +.\" +.\" Copyright (c) 2007 The NetBSD Foundation, Inc. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" 2. Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND +.\" CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, +.\" INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +.\" MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. +.\" IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS BE LIABLE FOR ANY +.\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE +.\" GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER +.\" IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR +.\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN +.\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.\" +.Dd January 13, 2011 +.Dt ATF-TEST-CASE 4 +.Os +.Sh NAME +.Nm atf-test-case +.Nd generic description of test cases +.Sh DESCRIPTION +A +.Em test case +is a piece of code that stress-tests a specific feature of the software. +This feature is typically self-contained enough, either in the amount of +code that implements it or in the general idea that describes it, to +warrant its independent testing. +Given this, test cases are very fine-grained, but they attempt to group +similar smaller tests which are semantically related. +.Pp +A test case is defined by three components regardless of the language it is +implemented in: a header, a body and a cleanup routine. +The +.Em header +is, basically, a declarative piece of code that defines several +properties to describe what the test case does and how it behaves. +In other words: it defines the test case's +.Em meta-data , +further described in the +.Sx Meta-data +section. +The +.Em body +is the test case itself. +It executes all actions needed to reproduce the test, and checks for +failures. +This body is only executed if the abstract conditions specified by the +header are met. +The +.Em cleanup +routine is a piece of code always executed after the body, regardless of +the exit status of the test case. +It can be used to undo side-effects of the test case. +Note that almost all side-effects of a test case are automatically cleaned +up by the library; this is explained in more detail in the rest of this +document. +.Pp +It is extremely important to keep the separation between a test case's +header and body well-defined, because the header is +.Em always +parsed, whereas the body is only executed when the conditions defined in +the header are met and when the user specifies that test case. +.Pp +At last, test cases are always contained into test programs. +The test programs act as a front-end to them, providing a consistent +interface to the user and several APIs to ease their implementation. +.Ss Results +Upon termination, a test case reports a status and, optionally, a textual +reason describing why the test reported such status. +The caller must ensure that the test case really performed the task that its +status describes, as the test program may be bogus and therefore providing a +misleading result (e.g. providing a result that indicates success but the +error code of the program says otherwise). +.Pp +The possible exit status of a test case are one of the following: +.Bl -tag -width expectedXfailureXX +.It expected_death +The test case expects to terminate abruptly. +.It expected_exit +The test case expects to exit cleanly. +.It expected_failure +The test case expects to exit with a controller fatal/non-fatal failure. +If this happens, the test program exits with a success error code. +.It expected_signal +The test case expects to receive a signal that makes it terminate. +.It expected_timeout +The test case expects to execute for longer than its timeout. +.It passed +The test case was executed successfully. +The test program exits with a success error code. +.It skipped +The test case could not be executed because some preconditions were not +met. +This is not a failure because it can typically be resolved by adjusting +the system to meet the necessary conditions. +This is always accompanied by a +.Em reason , +a message describing why the test was skipped. +The test program exits with a success error code. +.It failed +An error appeared during the execution of the test case. +This is always accompanied by a +.Em reason , +a message describing why the test failed. +The test program exits with a failure error code. +.El +.Pp +The usefulness of the +.Sq expected_* +results comes when writing test cases that verify known failures caused, +in general, due to programming errors (aka bugs). +Whenever the faulty condition that the expectation is trying to convery is +fixed, then the test case will be reported as +.Sq failed +and the developer will have to adjust it to match its new condition. +.Pp +It is important to note that all +.Sq expected_* +results are only provided as a +.Em hint +to the caller; the caller must verify that the test case did actually terminate +as the expected condition says. +.Ss Input/output +Test cases are free to print whatever they want to their +.Xr stdout 4 +and +.Xr stderr 4 +file descriptors. +They are, in fact, encouraged to print status information as they execute +to keep the user informed of their actions. +This is specially important for long test cases. +.Pp +Test cases will log their results to an auxiliary file, which is then +collected by the test program they are contained in. +The developer need not care about this as long as he uses the correct +APIs to implement the test cases. +.Pp +The standard input of the test cases is unconditionally connected to +.Sq /dev/zero . +.Ss Meta-data +The following list describes all meta-data properties interpreted +internally by ATF. +You are free to define new properties in your test cases and use them as +you wish, but non-standard properties must be prefixed by +.Sq X- . +.Bl -tag -width requireXmachineXX +.It descr +Type: textual. +Required. +.Pp +A brief textual description of the test case's purpose. +Will be shown to the user in reports. +Also good for documentation purposes. +.It has.cleanup +Type: boolean. +Optional. +.Pp +If set to true, specifies that the test case has a cleanup routine that has +to be executed by +.Xr atf-run 1 +during the cleanup phase of the execution. +This property is automatically set by the framework when defining a test case +with a cleanup routine, so it should never be set by hand. +.It ident +Type: textual. +Required. +.Pp +The test case's identifier. +Must be unique inside the test program and should be short but descriptive. +.It require.arch +Type: textual. +Optional. +.Pp +A whitespace separated list of architectures that the test case can be run +under without causing errors due to an architecture mismatch. +.It require.config +Type: textual. +Optional. +.Pp +A whitespace separated list of configuration variables that must be defined +to execute the test case. +If any of the required variables is not defined, the test case is +.Em skipped . +.It require.files +Type: textual. +Optional. +.Pp +A whitespace separated list of files that must be present to execute the +test case. +The names of these files must be absolute paths. +If any of the required files is not found, the test case is +.Em skipped . +.It require.machine +Type: textual. +Optional. +.Pp +A whitespace separated list of machine types that the test case can be run +under without causing errors due to a machine type mismatch. +.It require.memory +Type: integer. +Optional. +Specifies the minimum amount of physical memory needed by the test. +The value can have a size suffix such as +.Sq K , +.Sq M , +.Sq G +or +.Sq T +to make the amount of bytes easier to type and read. +.It require.progs +Type: textual. +Optional. +.Pp +A whitespace separated list of programs that must be present to execute +the test case. +These can be given as plain names, in which case they are looked in the +user's +.Ev PATH , +or as absolute paths. +If any of the required programs is not found, the test case is +.Em skipped . +.It require.user +Type: textual. +Optional. +.Pp +The required privileges to execute the test case. +Can be one of +.Sq root +or +.Sq unprivileged . +.Pp +If the test case is running as a regular user and this property is +.Sq root , +the test case is +.Em skipped . +.Pp +If the test case is running as root and this property is +.Sq unprivileged , +.Xr atf-run 1 +will automatically drop the privileges if the +.Sq unprivileged-user +configuration property is set; otherwise the test case is +.Em skipped . +.It timeout +Type: integral. +Optional; defaults to +.Sq 300 . +.Pp +Specifies the maximum amount of time the test case can run. +This is particularly useful because some tests can stall either because they +are incorrectly coded or because they trigger an anomalous behavior of the +program. +It is not acceptable for these tests to stall the whole execution of the +test program. +.Pp +Can optionally be set to zero, in which case the test case has no run-time +limit. +This is discouraged. +.El +.Ss Environment +Every time a test case is executed, several environment variables are +cleared or reseted to sane values to ensure they do not make the test fail +due to unexpected conditions. +These variables are: +.Bl -tag -width LCXMESSAGESXX +.It Ev HOME +Set to the work directory's path. +.It Ev LANG +Undefined. +.It Ev LC_ALL +Undefined. +.It Ev LC_COLLATE +Undefined. +.It Ev LC_CTYPE +Undefined. +.It Ev LC_MESSAGES +Undefined. +.It Ev LC_MONETARY +Undefined. +.It Ev LC_NUMERIC +Undefined. +.It Ev LC_TIME +Undefined. +.It Ev TZ +Hardcoded to +.Sq UTC . +.El +.Ss Work directories +The test program always creates a temporary directory +and switches to it before running the test case's body. +This way the test case is free to modify its current directory as it +wishes, and the runtime engine will be able to clean it up later on in a +safe way, removing any traces of its execution from the system. +To do so, the runtime engine will perform a recursive removal of the work +directory without crossing mount points; if a mount point is found, the +file system will be unmounted (if possible). +.Ss File creation mode mask (umask) +Test cases are always executed with a file creation mode mask (umask) of +.Sq 0022 . +The test case's code is free to change this during execution. +.Sh SEE ALSO +.Xr atf-run 1 , +.Xr atf-test-program 1 , +.Xr atf-formats 5 , +.Xr atf 7 diff --git a/contrib/atf/doc/atf-test-program.1 b/contrib/atf/doc/atf-test-program.1 new file mode 100644 index 0000000..49ee12d --- /dev/null +++ b/contrib/atf/doc/atf-test-program.1 @@ -0,0 +1,103 @@ +.\" +.\" Automated Testing Framework (atf) +.\" +.\" Copyright (c) 2007 The NetBSD Foundation, Inc. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" 2. Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND +.\" CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, +.\" INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +.\" MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. +.\" IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS BE LIABLE FOR ANY +.\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE +.\" GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER +.\" IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR +.\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN +.\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.\" +.Dd February 6, 2011 +.Dt ATF-TEST-PROGRAM 1 +.Os +.Sh NAME +.Nm atf-test-program +.Nd common interface to ATF test programs +.Sh SYNOPSIS +.Nm +.Op Fl r Ar resfile +.Op Fl s Ar srcdir +.Op Fl v Ar var1=value1 Op .. Fl v Ar varN=valueN +.Ar test_case +.Nm +.Fl l +.Sh DESCRIPTION +Test programs written using the ATF libraries all share a common user +interface, which is what this manual page describes. +.Em NOTE: There is no binary known as +.Nm ; +.Em what is described in this manual page is the command-line interface +.Em exposed by the atf-c, atf-c++ and atf-sh bindings . +.Pp +In the first synopsis form, the test program will execute the provided +test case and print its results to the standard output, unless otherwise +stated by the +.Fl r +flag. +Optionally, the test case name can be suffixed by +.Sq :cleanup , +in which case the cleanup routine of the test case will be executed +instead of the test case body; see +.Xr atf-test-case 4 . +Note that the test case is +.Em executed without isolation , +so it can and probably will create and modify files in the current directory. +To execute test cases in a controller manner, refer to +.Xr atf-run 1 , +which is the preferred way to run test cases. +You should only execute test cases by hand for debugging purposes. +.Pp +In the second synopsis form, the test program will list all available +test cases alongside their meta-data properties in a format that is +machine parseable. +This list is processed by +.Xr atf-run 1 +to know how to execute the test cases of a given test program. +.Pp +The following options are available: +.Bl -tag -width XvXvarXvalueXX +.It Fl l +Lists available test cases alongside a brief description for each of them. +.It Fl r Ar resfile +Specifies the file that will receive the test case result. +If not specified, the test case prints its results to stdout. +If the result of a test case needs to be parsed by another program, you must +use this option to redirect the result to a file and then read the resulting +file from the other program. +Note: +.Em do not try to process the stdout of the test case +because your program may break in the future. +.It Fl s Ar srcdir +The path to the directory where the test program is located. +This is needed in all cases, except when the test program is being executed +from the current directory. +The test program will use this path to locate any helper data files or +utilities. +.It Fl v Ar var=value +Sets the configuration variable +.Ar var +to the value +.Ar value . +.El +.Sh SEE ALSO +.Xr atf-run 1 , +.Xr atf 7 diff --git a/contrib/atf/doc/atf.7.in b/contrib/atf/doc/atf.7.in new file mode 100644 index 0000000..b11054c --- /dev/null +++ b/contrib/atf/doc/atf.7.in @@ -0,0 +1,192 @@ +.\" +.\" Automated Testing Framework (atf) +.\" +.\" Copyright (c) 2007 The NetBSD Foundation, Inc. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" 2. Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND +.\" CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, +.\" INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +.\" MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. +.\" IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS BE LIABLE FOR ANY +.\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE +.\" GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER +.\" IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR +.\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN +.\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.\" +.Dd August 28, 2010 +.Dt ATF 7 +.Os +.Sh NAME +.Nm ATF +.Nd introduction to the Automated Testing Framework +.Sh DESCRIPTION +.Em IMPORTANT: If you are here because you want to know how to run the tests in +.Pa __TESTSDIR__ , +.Em you most likely want to read the +.Xr tests 7 +.Em manual page instead. +.Pp +The Automated Testing Framework +.Pf ( Nm ) +is a collection of libraries and utilities designed to ease unattended +application testing in the hands of developers and end users of a specific +piece of software. +.Pp +As regards developers, +.Nm +provides the necessary means to easily create +test suites composed of multiple test programs, which in turn are a +collection of test cases. +It also attempts to simplify the debugging of problems when these test +cases detect an error by providing as much information as possible +about the failure. +.Pp +As regards users, it simplifies the process of running the test suites and, +in special, encourages end users to run them often: they do not need to +have source trees around nor any other development tools installed to be +able to certify that a given piece of software works on their machine as +advertised. +.Pp +If your operating systems distributes +.Nm , +it is possible that it provides an introductory +.Xr tests 7 +manual page. +You are encouraged to read it now. +.Ss License +.Nm +is distributed under the terms of the TNF License, a 2-clause BSD license. +For more details please see: +.Bd -literal -offset indent +.Pa __DOCDIR__/COPYING +.Ed +.Ss Components +.Nm +is a highly modular piece of software. +It provides a couple of libraries to ease the implementation of test +programs: one for the C and C++ languages and another one for shell +scripts. +It also includes multiple small utilities that follow the principle of +doing a single thing but doing it right. +This section outlines which these components are. +.Pp +Public utilities: +.Bl -tag -width atfXtestXprogramXXXXX +.It Xr atf-check 1 +Executes a command and checks that its exit code, its standard output +and its standard error output match pre-specified expected values. +.It Xr atf-config 1 +Queries static configuration information. +.It Xr atf-report 1 +Converts the output of +.Nm atf-run +to user-friendly and/or machine-parseable reports. +.It Xr atf-run 1 +Automates the execution of a series of test programs and collects their +results in a unified report. +.It Xr atf-sh 1 +Shell interpreter for shell-based test programs. +.El +.Pp +Programming interfaces: +.Bl -tag -width atfXtestXprogramXXXXX +.It Xr atf-c-api 3 +C programming interface for test programs. +.It Xr atf-c++-api 3 +C++ programming interface for test programs. +.It Xr atf-sh-api 3 +POSIX shell programming interface for test programs. +.El +.Pp +Other: +.Bl -tag -width atfXtestXprogramXXXXX +.It Xr atf-formats 5 +Description of the machine-parseable data formats used by the tools. +.It Xr atf-test-case 4 +Generic description of test cases, independent of the language they are +implemented in. +.It Xr atf-test-program 1 +Common interface provided by the test programs written using the +.Nm +libraries. +.El +.Ss Recommended reading order +For end users wishing to run tests: +.Bl -enum -compact +.It +.Xr tests 7 +(only if provided by your operating system). +.It +.Xr atf-test-program 1 +.It +.Xr atf-run 1 +.It +.Xr atf-report 1 +.It +.Xr atf-config 1 +.El +.Pp +For developers wanting to write their own tests: +.Bl -enum -compact +.It +Everything recommended to users. +.It +.Xr atf-test-case 4 +.It +.Xr atf-c-api 3 +.It +.Xr atf-c++-api 3 +.It +.Xr atf-sh-api 3 +.It +.Xr atf-sh 1 +.It +.Xr atf-check 1 +.El +.Pp +For those interested in +.Nm +internals: +.Bl -enum -compact +.It +Everything recommended to users. +.It +Everything recommended to developers. +.It +.Xr atf-formats 5 +.El +.Sh SEE ALSO +.Xr tests 7 +.Sh HISTORY +.Nm +started as a Google Summer of Code 2007 project mentored by The NetBSD +Foundation. +Its original goal was to provide a testing framework for The NetBSD +Operating System, but it grew as an independent project because the +framework itself did not need to be tied to a specific operating system. +.Pp +For more details on this subject, please see: +.Bd -literal -offset indent +.Pa __DOCDIR__/NEWS +.Pa __DOCDIR__/ROADMAP +.Ed +.Sh AUTHORS +For more details on the people that made +.Nm +possible, please see: +.Bd -literal -offset indent +.Pa __DOCDIR__/AUTHORS +.Ed |