Add some guidelines how to write regression tests to the bsd.regress.mk(5)

man page.  The goal is that all tests behave alike to make automatic
testing easy.
input and OK schwarze@

1 files changed, 73 insertions, 2 deletions

diff --git a/share/man/man5/bsd.regress.mk.5 b/share/man/man5/bsd.regress.mk.5
index 51a03ff887a..d7a93af0b29 100644
--- a/share/man/man5/bsd.regress.mk.5
+++ b/share/man/man5/bsd.regress.mk.5
@@ -1,4 +1,4 @@
-.\" $OpenBSD: bsd.regress.mk.5,v 1.9 2015/09/23 01:11:13 schwarze Exp $
+.\" $OpenBSD: bsd.regress.mk.5,v 1.10 2017/01/15 22:41:50 bluhm Exp $
 .\"
 .\" Copyright (c) 2002 Anil Madhavapeddy
 .\" Copyright (c) 2000 Marc Espie
@@ -25,7 +25,7 @@
 .\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
 .\" THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 .\"
-.Dd $Mdocdate: September 23 2015 $
+.Dd $Mdocdate: January 15 2017 $
 .Dt BSD.REGRESS.MK 5
 .Os
 .Sh NAME
@@ -52,6 +52,13 @@ Build any dependencies required to carry out the current set
 of regression tests.
 .It Cm regress
 Executes all of the regression targets defined in the Makefile.
+If one of the tests fails, the line
+.Qq FAILED
+is printed to the standard output.
+By default, execution continues with the next test, and after running
+all the tests,
+.Sy make Cm regress
+exits with an exit status of 0 even if some of the tests failed.
 .It Cm run-regress-*
 Runs an individual regression test.
 If the exit status of the program indicates an error or timeout,
@@ -65,6 +72,10 @@ If this variable is set to anything but
 the
 .Cm regress
 target will abort as soon as a test fails.
+This variable is intended to be set at runtime in the environment
+or in
+.Pa /etc/mk.conf ,
+but not in the test's Makefile itself.
 .It Ev REGRESS_LOG
 Points to the fully-qualified path of a file to which regression
 results are appended.
@@ -101,6 +112,66 @@ test targets which require it.
 See
 .Xr doas 1 .
 .El
+.Sh GUIDELINES
+If an indivual test passes,
+.Sy make Ar testname
+should return with an exit status of 0.
+If it fails, it should return with a non-zero exit status.
+.Pp
+If a test cannot be executed because a package is not installed or
+some environment variable is not set,
+.Sy make Ar testname
+should print
+.Qq SKIPPED
+to stdout and exit with status 0.
+To skip everything, implement the
+.Cm regress
+target with a command that prints
+.Qq SKIPPED .
+.Pp
+Tests should not fail because an intended feature has not been
+implemented yet.
+To avoid such false failures, a test should show the reason, print
+.Qq DISABLED
+to stdout and exit with status 0.
+.Pp
+Tests must be able to run with an
+.Pa obj
+directory.
+In case the test is not linked to the build or the tester forgot
+to run
+.Sy make Cm obj
+before, this directory or symlink may not exist.
+Then the test should run anyway.
+.Pp
+Test are executed with
+.Sy make Cm regress ,
+but running
+.Sy make Cm all
+or
+.Sy make
+should have the same effect.
+.Pp
+Tests should use the binaries installed and the kernel running on
+the local system.
+They may use environment variables to test alternative binaries or
+remote kernels running on other machines.
+In some cases a test may need binaries or libraries or object files
+to be present in
+.Pa /usr/obj/
+that exist only after
+.Sy make Cm build
+was run in
+.Pa /usr/src/ .
+The test must not assume that they have already been built, but
+should run
+.Sy make
+in the appropriate source directory as a dependency.
+The
+.Pa /usr/src/
+tree can be found with a relative path or with the
+.Ev BSDSRCDIR
+variable.
 .Sh SEE ALSO
 .Xr bsd.port.mk 5
 .Sh HISTORY