4 Sparse has a number of test cases in its validation directory. The test-suite
5 script aims at making automated checking of these tests possible. It works by
6 embedding tags in C comments in the test cases.
11 ``check-name:`` *name*
13 Name of the test. This is the only mandatory tag.
15 ``check-description:`` *description ...*
17 A description of what the test checks.
19 ``check-command:`` *command arg ...*
21 There are different kinds of tests. Some can validate the sparse
22 preprocessor, while others will use sparse, cgcc, or even other backends
23 of the library. check-command allows you to give a custom command to
25 The ``$file`` string is special. It will be expanded to the file name at
27 It defaults to ``sparse $file``.
29 ``check-arch-ignore:`` *arch[|...]*
31 ``check-arch-only:`` *arch[|...]*
33 Ignore the test if the current architecture (as returned by ``uname -m``)
34 matches or not one of the archs given in the pattern.
36 ``check-assert:`` *condition*
38 Ignore the test if the given condition is false when evaluated as a
39 static assertion (``_Static_assert``).
41 ``check-cpp-if:`` *condition*
43 Ignore the test if the given condition is false when evaluated
44 by sparse's pre-processor.
46 ``check-exit-value:`` *value*
48 The expected exit value of check-command. It defaults to 0.
50 ``check-timeout:`` *timeout*
52 The maximum expected duration of check-command, in seconds.
55 ``check-output-start`` / ``check-output-end``
57 The expected output (stdout and stderr) of check-command lies between
58 those two tags. It defaults to no output.
60 ``check-output-ignore`` / ``check-error-ignore``
62 Don't check the expected output (stdout or stderr) of check-command
63 (useful when this output is not comparable or if you're only interested
64 in the exit value). By default this check is done.
66 ``check-known-to-fail``
68 Mark the test as being known to fail.
70 ``check-output-contains:`` *pattern*
72 Check that the output (stdout) contains the given pattern.
73 Several such tags can be given, in which case the output
74 must contains all the patterns.
76 ``check-output-excludes:`` *pattern*
78 Similar than the above one, but with opposite logic.
79 Check that the output (stdout) doesn't contain the given pattern.
80 Several such tags can be given, in which case the output
81 must contains none of the patterns.
83 ``check-output-pattern(``\ *nbr*\ ``):`` *pattern*
85 ``check-output-pattern(``\ *min*\ ``,``\ *max*\ ``):`` *pattern*
87 Similar to the contains/excludes above, but with full control
88 of the number of times the pattern should occur in the output.
89 If *min* or *max* is ``-`` the corresponding check is ignored.
91 ``check-output-match(``\ *start*\ ``):`` *pattern*
93 Check that in the output (stdout) all lines starting with the
94 first pattern also contains the second pattern. This should be
95 reserved for matching IR instructions since the '.$size' suffix
96 is ignored in the first pattern but is expected to be followed
99 ``check-output-returns:`` *value*
101 Check that in the output (stdout) all IR return instructions
102 have the given value.
107 The test-suite script is called through the check target of the Makefile. It
108 will try to check every test case it finds (``find validation -name '*.c'``).
109 It can be called to check a single test with::
112 $ ./test-suite single preprocessor/preprocessor1.c
113 TEST Preprocessor #1 (preprocessor/preprocessor1.c)
114 preprocessor/preprocessor1.c passed !
120 The test-suite comes with a format command to make a test easier to write::
122 test-suite format [-a] [-l] [-f] file [name [cmd]]
124 `name:` check-name value
125 If no name is provided, it defaults to the file name.
127 `cmd:` check-command value
128 If no cmd is provided, it defaults to ``sparse $file``.
130 The output of the test-suite format command can be redirected into the
131 test case to create a test-suite formatted file.::
133 $ ./test-suite format bad-assignment.c Assignment >> bad-assignment.c
137 * check-name: bad assignment
139 * check-command: sparse $file
140 * check-exit-value: 1
143 bad-assignment.c:3:6: error: Expected ; at end of statement
144 bad-assignment.c:3:6: error: got \
148 The same effect without the redirection can be achieved by using the ``-a``
151 You can define the check-command you want to use for the test.::
153 $ ./test-suite format -a validation/preprocessor2.c "Preprocessor #2" \
156 cat validation/preprocessor2.c
158 * This one we happen to get right.
160 * It should result in a simple
164 * for a proper preprocessor.
168 #define UNARY(x) BINARY(x)
169 #define BINARY(x, y) x + y
173 * check-name: Preprocessor #2
175 * check-command: sparse -E $file
176 * check-exit-value: 0