1 The Automake test suite
15 By default, verbose output of a test 't/foo.sh' or 't/foo.tap' is retained
16 in the log file 't/foo.log'. Also, a summary log is created in the file
17 'test-suite.log' (in the top-level directory).
19 You can use '-jN' for faster completion (it even helps on a uniprocessor
20 system, due to unavoidable sleep delays, as noted below):
24 To rerun only failed tests:
28 To run only tests that are newer than their last results:
30 make -k check RECHECK_LOGS=
32 To run only selected tests:
34 make -k check TESTS="t/foo.sh t/bar.tap" (GNU make)
35 env TESTS="t/foo.sh t/bar.tap" make -e -k check (non-GNU make)
37 To run the tests in cross-compilation mode, you should first configure
38 the automake source tree to a cross-compilation setup. For example, to
39 run with a Linux-to-MinGW cross compiler, you will need something like
42 ./configure --host i586-mingw32msvc --build i686-pc-linux-gnu
44 To avoid possible spurious error, you really have to *explicitly* specify
45 '--build' in addition to '--host'; the 'lib/config.guess' script can help
46 determine the correct value to pass to '--build'.
47 Then you can just run the testsuite in the usual way, and the test cases
48 using a compiler should automatically use a cross-compilation setup.
56 XFAIL - expected failure
60 XPASS - unexpected success
63 SKIP - skipped tests (third party tools not available)
64 ERROR - some unexpected error condition
70 There are two kinds of tests in the Automake testsuite (both implemented
71 as shell scripts). The scripts with the '.sh' suffix are "simple"
72 tests, their outcome completely determined by their exit status. Those
73 with the '.tap' suffix use the TAP protocol.
75 If you want to run a test by hand, you should be able to do so using the
76 'runtest' script provided in the Automake distribution:
79 ./runtest t/add-missing.tap
81 This will run the test using the correct shell, and should also work in
82 VPATH builds. Note that, to run the TAP tests this way, you'll need to
83 have the prove(1) utility available in $PATH.
89 By default, the tests are run by a proper shell detected at configure
90 time. Here is how you can run the tests with a different shell, say
93 # Running through the makefile test driver.
94 make check AM_TEST_RUNNER_SHELL=/bin/my-sh (GNU make)
95 AM_TEST_RUNNER_SHELL=/bin/my-sh make -e check (non-GNU make)
97 # Run a test directly from the command line.
98 AM_TEST_RUNNER_SHELL=/bin/my-sh ./runtest t/foo.sh
100 The test scripts are written with portability in mind, and should run
101 with any decent POSIX shell. However, it is worth nothing that older
102 versions of Zsh (pre-4.3) exhibited several bugs and incompatibilities
103 with our uses, and are thus not supported for running Automake's test
110 Send verbose output, i.e., the contents of test-suite.log, of failing
111 tests to <bug-automake@gnu.org>, along with the usual version numbers
112 (which Automake, which Autoconf, which operating system, which make
113 version, which shell, etc.)
120 * If you plan to fix a bug, write the test case first. This way you'll
121 make sure the test catches the bug, and that it succeeds once you have
124 * Add a copyright/license paragraph.
126 * Explain what the test does, i.e., which features it checks, which
127 invariants it verifies, or what bugs/issues it guard against.
129 * Cite the PR number (if any), and the original reporter (if any), so
130 we can find or ask for information if needed.
132 * If a test checks examples or idioms given in the documentation, make
133 sure the documentation reference them appropriately in comments, as
136 @c Keep in sync with autodist-config-headers.sh
141 * Use "required=..." for required tools. Do not explicitly require
142 tools which can be taken for granted because they're listed in the
143 GNU Coding Standards (for example, 'gzip').
145 * Include 'test-init.sh' in every test script (see existing tests for
146 examples of how to do this).
148 * Use the 'skip_' function to skip tests, with a meaningful message if
149 possible. Where convenient, use the 'warn_' function to print generic
150 warnings, the 'fail_' function for test failures, and the 'fatal_'
151 function for hard errors. In case a hard error is due to a failed
152 set-up of a test scenario, you can use the 'framework_fail_' function
155 * For those tests checking the Automake-provided test harnesses that
156 are expected to work also when the 'serial-tests' Automake option
157 is used (thus causing the serial testsuite harness to be used in the
158 generated Makefile), place a line containing "try-with-serial-tests"
159 somewhere in the file (usually in a comment).
160 That will ensure that the 'gen-testsuite-part' script generates a
161 sibling of that test which uses the serial harness instead of the
162 parallel one. For those tests that are *not* meant to work with the
163 parallel testsuite harness at all (these should be very very few),
164 set the shell variable 'am_serial_tests' to "yes" before including
167 * Some tests in the Automake testsuite are auto-generated; those tests
168 might have custom extensions, but their basename (that is, with such
169 extension stripped) is expected to end with "-w" string, optionally
170 followed by decimal digits. For example, the name of a valid
171 auto-generated test can be 'color-w.sh' or 'tap-signal-w09.tap'.
172 Please don't name hand-written tests in a way that could cause them
173 to be confused with auto-generated tests; for example, 'u-v-w.sh'
174 or 'option-w0.tap' are *not* valid name for hand-written tests.
176 * test-init.sh brings in some commonly required files, and sets a skeleton
177 configure.ac. If possible, append to this file. In some cases you'll
178 have to overwrite it, but this should be the exception. Note that
179 configure.ac registers Makefile.in but do not output anything by
180 default. If you need ./configure to create Makefile, append AC_OUTPUT
181 to configure.ac. In case you don't want your test directory to be
182 pre-populate by test-init.sh (this should be a rare occurrence), set
183 the 'am_create_testdir' shell variable to "empty" before sourcing
186 * By default, the testcases are run with the errexit shell flag on,
187 to make it easier to catch failures you might not have thought of.
188 If this is undesirable in some testcase, you can use "set +e" to
189 disable the errexit flag (but please do so only if you have a very
192 * End the test script with a ':' command. Otherwise, when somebody
193 changes the test by adding a failing command after the last command,
194 the test will spuriously fail because '$?' is nonzero at the end.
195 Note that this is relevant even if the errexit shell flag is on, in
196 case the test contains commands like "grep ... Makefile.in && exit 1"
197 (and there are indeed a lot of such tests).
199 * Use $ACLOCAL, $AUTOMAKE, $AUTOCONF, $AUTOUPDATE, $AUTOHEADER,
200 $PERL, $MAKE, $EGREP, and $FGREP, instead of the corresponding
203 * When you want to redirect the output from a make invocation, use the
204 'run_make' function rather than calling $MAKE directly. Not only is
205 this more idiomatic, but it also avoid possible spurious racy failures
206 when the make invocations in the testsuite are run in parallel mode
207 (as with "make check AM_TESTSUITE_MAKE='make -j4"').
209 * Do not override Makefile variables using make arguments, as in e.g.:
211 $MAKE prefix=/opt install # BAD
213 This is not portable for recursive targets (with non-GNU make,
214 targets that call a sub-make may not pass "prefix=/opt" along).
215 Instead, use the 'run_make' function, which automatically uses
216 the AM_MAKEFLAGS to propagate the variable definitions along to
219 run_make prefix=/opt install # GOOD
221 * Use '$sleep' when you have to make sure that some file is newer
224 * Use cat or grep or similar commands to display (part of) files that
225 may be interesting for debugging, so that when a user send a verbose
226 output we don't have to ask him for more details. Display stderr
227 output on the stderr file descriptor. If some redirected command is
228 likely to fail, display its output even in the failure case, before
231 * Use '$PATH_SEPARATOR', not hard-coded ':', as the separator of
234 * It's more important to make sure that a feature works, than make
235 sure that Automake's output looks correct. It might look correct
236 and still fail to work. In other words, prefer running 'make' over
237 grepping Makefile.in (or do both).
239 * If you run $ACLOCAL, $AUTOMAKE or $AUTOCONF several times in the
240 same test and change configure.ac by the meantime, do
242 rm -rf autom4te*.cache
244 before the following runs. On fast machines the new configure.ac
245 could otherwise have the same timestamp as the old autom4te.cache.
247 * Use filenames with two consecutive spaces when testing that some
248 code preserves filenames with spaces. This will catch errors like
249 `echo $filename | ...`.
251 * Make sure your test script can be used to faithfully check an
252 installed version of automake (as with "make installcheck"). For
253 example, if you need to copy or grep an automake-provided script,
254 do not assume that they can be found in the '$top_srcdir/lib'
255 directory, but use '$am_scriptdir' instead. The complete list of
256 such "$am_...dir" variables can be found in the 't/ax/test-defs.in'
259 * When writing input for lex, include the following in the definitions
263 #define YY_NO_UNISTD_H 1
266 to accommodate non-ANSI systems, since GNU flex generates code that
267 includes unistd.h otherwise. Also add:
269 int isatty (int fd) { return 0; }
271 to the definitions section if the generated code is to be compiled
272 by a C++ compiler, for similar reasons (i.e., the isatty(3) function
273 from that same unistd.h header would be required otherwise).
275 * Add any new test to handwritten_TESTS in 't/list-of-tests.mk', and
276 to XFAIL_TESTS in addition if needed (that is, if the test is expected
279 * In test scripts, prefer using POSIX constructs over their old
280 Bourne-only equivalents:
282 - use $(...), not `...`, for command substitution;
283 - use $((...)), not `expr ...`, for arithmetic processing;
284 - liberally use '!' to invert the exit status of a command, e.g.,
285 in idioms like "if ! CMD; then ...", instead of relying on clumsy
286 paraphrases like "if CMD; then :; else ...".
287 - prefer use of ${param%pattern} and ${param#pattern} parameter
288 expansions over processing by 'sed' or 'expr'.
290 * Note however that, when writing Makefile recipes or shell code in a
291 configure.ac, you should still use `...` instead, because the Autoconf
292 generated configure scripts do not ensure they will find a truly POSIX
293 shell (even though they will prefer and use it *if* it's found).
295 * Do not test an Automake error with "$AUTOMAKE && exit 1", or in three
296 years we'll discover that this test failed for some other bogus reason.
297 This happened many times. Better use something like
300 grep 'expected diagnostic' stderr
302 Note this doesn't prevent the test from failing for another reason,
303 but at least it makes sure the original error is still here.