| blob | 2cde00dd2be8562423bd91053b64d5ef0f483095 |
1 .\"
2 .\" Copyright (c) 2011
3 .\" The DragonFly Project. All rights reserved.
4 .\"
5 .\" Redistribution and use in source and binary forms, with or without
6 .\" modification, are permitted provided that the following conditions
7 .\" are met:
8 .\"
9 .\" 1. Redistributions of source code must retain the above copyright
10 .\" notice, this list of conditions and the following disclaimer.
11 .\" 2. Redistributions in binary form must reproduce the above copyright
12 .\" notice, this list of conditions and the following disclaimer in
13 .\" the documentation and/or other materials provided with the
14 .\" distribution.
15 .\" 3. Neither the name of The DragonFly Project nor the names of its
16 .\" contributors may be used to endorse or promote products derived
17 .\" from this software without specific, prior written permission.
18 .\"
19 .\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
20 .\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
21 .\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
22 .\" FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
23 .\" COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
24 .\" INCIDENTAL, SPECIAL, EXEMPLARY OR CONSEQUENTIAL DAMAGES (INCLUDING,
25 .\" BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
26 .\" LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
27 .\" AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
28 .\" OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
29 .\" OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
30 .\" SUCH DAMAGE.
31 .\"
32 .Dd November 27, 2020
33 .Dt DFREGRESS 8
34 .Os
35 .Sh NAME
36 .Nm dfregress
37 .Nd an automation test driver and framework
38 .Sh SYNOPSIS
39 .Nm
40 .Op Fl o Ar output_plist
41 .Op Fl t Ar testcase_dir
42 .Op Fl p Ar prepost_dir
43 .Ar runlist_file
44 .Nm
45 .Fl h
46 .Sh DESCRIPTION
47 .Nm
48 is a regression test framework and automation driver.
49 It executes a series of testcases as specified in the
50 .Ar runlist_file
51 and collects the results.
52 .Pp
53 The path to the testcase collection is specified via the
54 .Ar testcase_dir
55 argument.
56 If this argument is not specified, the default is assumed to be the
57 same directory as that of the runlist.
58 For example if the used runlist is
59 .Pa /usr/src/test/testcases/sample.run
60 the default testcase directory, unless otherwise specified, will be
61 .Pa /usr/src/test/testcases .
62 .Pp
63 Similarly the path to the pre- and post commands is
64 specified via
65 .Ar prepost_dir .
66 The
67 .Ar prepost_dir
68 only needs to be specified if the runlist contains custom pre or
69 post commands.
70 .Pp
71 The output is saved in plist format to the
72 .Ar output_plist
73 file, or if none is specified, to a file with the same base name as
74 the runlist, but in the current working directory and with a
75 .Pa .plist
76 extension.
77 For example if the used runlist is
78 .Pa /usr/src/test/testcases/sample.run
79 the default output, unless otherwise specified, will be
80 .Pa ./sample.plist .
81 Other tools, known as frontends, can parse the plist output into
82 an easier to read form.
83 .Pp
84 The following is a summary of the optional parameters:
85 .Bl -tag -width indent
86 .It Fl o Ar output_plist
87 Specifies the file to which to write the results.
88 The resulting file,
89 .Ar output_plist ,
90 will be in plist format.
91 .It Fl t Ar testcase_dir
92 Specifies the directory in which to find the testcases specified in the runlist.
93 .It Fl p Ar prepost_dir
94 Specifies the directory in which to find the pre- and post commands used
95 in the runlist.
96 This argument is only required when the runlist uses custom pre- or post
97 commands.
98 .It Fl h
99 Prints a short synopsis.
100 .El
101 .Sh RUNLIST SYNTAX
102 Testcases are specified one testcase per line, with whitespace separated
103 values.
104 Empty lines and lines beginning with a
105 .Dq #
106 are ignored.
107 .Pp
108 Runlist lines are of the following format:
109 .Bd -literal -offset indent
110 .Ic testcase type options Cm arguments ...
111 .Ed
112 .Pp
113 The
114 .Ic testcase
115 part needs to be a relative path from the testcase base directory specified
116 by the
117 .Fl t
118 argument to the resulting (after building the testcase) testcase executable.
119 The testcase will be executed with the
120 .Cm arguments
121 passed as command line arguments to it.
122 .Pp
123 Valid types are
124 .Ic userland ,
125 .Ic kernel
126 and
127 .Ic buildonly :
128 .Bl -tag -width indent -offset indent
129 .It Ic userland
130 A userland testcase is a normal userland executable that returns a non-zero
131 exit value on test failure.
132 .It Ic kernel
133 A kernel testcase is run with the kernel test bridge inside the kernel.
134 .It Ic buildonly
135 A buildonly test passes when the build for the given testcase succeeds.
136 .El
137 .Pp
138 Valid options are
139 .Ic defaults ,
140 .Ic interpreter ,
141 .Ic make ,
142 .Ic rc ,
143 .Ic timeout ,
144 .Ic nobuild ,
145 .Ic runas ,
146 .Ic intpre ,
147 .Ic intpost ,
148 .Ic pre ,
149 and
150 .Ic post :
151 .Bl -tag -width indent -offset indent
152 .It Ic defaults
153 This option does nothing.
154 All default options are applied.
155 .It Ic interpreter Ar command
156 Uses
157 .Ar command
158 for running the test case instead of calling it directly.
159 This is useful when you have tests that you want to call with a specific shell
160 like
161 .Xr sh 1 ,
162 and there is no shebang in the testcase file, there are no
163 execution permissions and you don't want or cannot modify the testcase.
164 .It Ic make Ar make_command
165 Uses
166 .Ar make_command
167 instead of
168 .Xr make 1
169 to build the testcase.
170 .It Ic rc Ar code
171 Sets the return code that will be considered successful.
172 By default 0 is considered success.
173 .It Ic timeout Ar timeout
174 Sets the timeout for the testcase after which the testcase will be aborted to
175 .Ar timeout
176 seconds.
177 .It Ic nobuild
178 If this option is set, the build stage and cleanup stage of the test case
179 are not run.
180 .It Ic runas Ar uid
181 Runs the testcase as the user identified by the given
182 .Ar uid .
183 .It Ic intpre
184 Executes the testcase command with the argument
185 .Dq pre
186 during the pre-run command stage.
187 .It Ic intpost
188 Executes the testcase command with the argument
189 .Dq post
190 during the post-run command stage.
191 .It Ic pre Ar precmd
192 Executes the command specified by
193 .Ar precmd
194 during the pre-run command stage.
195 .It Ic pre Ar postcmd
196 Executes the command specified by
197 .Ar postcmd
198 during the post-run command stage.
199 .El
200 .Sh TESTCASE EXECUTION
201 .Bl -enum -width 3n -offset indent
202 .It
203 .Tn "CHDIR"
204 to the testcase directory.
205 If it fails, the result will be set to
206 .Dv RESULT_PREFAIL
207 and the
208 .Ar sysbuf
209 buffer will provide further details.
210 .It
211 .Tn "BUILD"
212 the testcase using the command specified by the
213 .Ic make
214 option or, if not specified,
215 .Xr make 1 ,
216 unless the
217 .Ic nobuild
218 option was specified.
219 If there is an internal driver error, the result will be set to
220 .Dv RESULT_PREFAIL ,
221 the next step to be performed will be
222 .Tn "CLEANUP"
223 and the
224 .Ar sysbuf
225 buffer will provide further details.
226 If no internal error occurs, the
227 .Ar buildbuf
228 will contain the build log.
229 .It
230 .Tn "RUN PRE COMMAND"
231 if
232 .Ic intpre
233 or
234 .Ic pre
235 are set.
236 If there is an internal driver error, the result will be set to
237 .Dv RESULT_PREFAIL ,
238 the next step to be performed will be
239 .Tn "CLEANUP"
240 and the
241 .Ar sysbuf
242 buffer will provide further details.
243 If the pre command has a non-zero exit value, the result will be set to
244 .Dv RESULT_PREFAIL
245 and the
246 next step to be performed will be
247 .Tn "CLEANUP" .
248 In this case and in the case where the command succeeds, the
249 .Ar precmd_buf
250 will contain the execution log.
251 .It
252 .Tn "RUN TESTCASE"
253 depending on type:
254 .Bl -tag -width 2n -compact
255 .It "buildonly"
256 testcases will get their result set to
257 .Dv RESULT_PASS
258 at this point, since the build must have succeeded already.
259 .It "userland"
260 testcases will get executed in a separate child process, possibly as a
261 different user, depending on whether the
262 .Ic runas
263 option was specified.
264 The result will be set to
265 .Dv RESULT_TIMEOUT
266 if the timeout is reached before the testcase finishes,
267 .Dv RESULT_SIGNALLED
268 if the testcase dies because of an unhandled signal (often due to crashing),
269 .Dv RESULT_NOTRUN
270 if the testcase could not be executed,
271 .Dv RESULT_FAIL
272 if the testcase completes but returns failure or
273 .Dv RESULT_PASS
274 if the testcase completes and returns success.
275 .It "kernel"
276 testcases will be executed in kernel space by loading a given module and
277 running the testcase entry point function.
278 The result will be set to
279 .Dv RESULT_NOTRUN
280 if the testcase could not be executed.
281 Otherwise the result will be set according to the kernel test case to
282 one of
283 .Dv RESULT_TIMEOUT ,
284 .Dv RESULT_FAIL ,
285 or
286 .Dv RESULT_PASS .
287 .El
288 The output will be logged separately for stdout and stderr to the
289 .Ar stdout_buf
290 and
291 .Ar stderr_buf
292 respectively.
293 If the result was
294 .Dv RESULT_NOTRUN
295 the
296 .Ar sysbuf
297 will contain more information.
298 .It
299 .Tn "RUN POST COMMAND"
300 if
301 .Ic intpost
302 or
303 .Ic post
304 are set.
305 If there is an internal driver error, the result will be set to
306 .Dv RESULT_POSTFAIL ,
307 the next step to be performed will be
308 .Tn "CLEANUP"
309 and the
310 .Ar sysbuf
311 buffer will provide further details.
312 If the post command has a non-zero exit value, the result will be set to
313 .Dv RESULT_POSTFAIL
314 and the
315 next step to be performed will be
316 .Tn "CLEANUP" .
317 In this case and in the case where the command succeeds, the
318 .Ar postcmd_buf
319 will contain the execution log.
320 .It
321 .Tn "CLEANUP"
322 the testcase execution using the command specified by the
323 .Ic make
324 option or, if not specified,
325 .Xr make 1
326 with the parameter
327 .Dq clean ,
328 unless the
329 .Ic nobuild
330 option was specified.
331 If there is an internal driver error the
332 .Ar sysbuf
333 buffer will contain more information.
334 If no internal error occurs, the
335 .Ar cleanu_pbuf
336 will contain the cleanup log.
337 .El
338 .Sh FRONTENDS
339 The output of
340 .Nm
341 is in the Apple plist serialized object format.
342 This format can be easily parsed by using
343 .Xr proplib 3
344 when using C.
345 Ruby and Python also have parsers for the plist format.
346 .Pp
347 A frontend for
348 .Nm
349 parses the intermediate output plist into a more easily readable format
350 such as plain text or websites.
351 .Pp
352 By default
353 .Nm
354 ships only with the
355 .Xr dfr2text 8
356 text-based frontend.
357 .Sh HOW TO WRITE A TESTCASE
358 A userland testcase is a simple program that prints some relevant
359 information to stdout and stderr, both of which are captured by the test
360 driver, and returns an exit value of 0 if the test passed, or any other
361 non-zero exit value to signal a failure.
362 The exact exit value is recorded by
363 .Nm .
364 All signals/exceptions not explicitly caught by the testcase will abort
365 the execution of the testcase and the result will be set appropriately and
366 the signal number will be recorded.
367 .Pp
368 A kernel testcase is a simple kernel module that defines two methods:
369 a
370 .Fn run
371 method as well as an optional
372 .Fn abort
373 method.
374 The
375 .Fn run
376 method will be run from a separate kernel thread.
377 The testcase will need to call
378 .Xr tbridge 9
379 functions to record output and to notify of testcase completion.
380 Refer to the
381 .Xr tbridge 9
382 manual page for more details.
383 .Pp
384 For all testcase types the build stage is common.
385 Every testcase should either have the
386 .Ic nobuild
387 option set, or have a Makefile or similar in its directory.
388 By default
389 .Nm
390 assumes it is a standard
391 .Xr make 1
392 Makefile.
393 If that is not the case, the
394 .Ic build
395 option needs to be adjusted accordingly.
396 .Sh GENERAL ADVICE ON WRITING TESTCASES
397 Test only one thing at a time, it is not good practice to test multiple
398 things in the same testcase as it makes it less obvious what's going on.
399 .Pp
400 Keep it short, simple and well documented on what the requirements are,
401 what the preconditions need to be, what exactly is being tested, ideally
402 with a reference to a particular bug if that exists, and most importantly
403 what the expected outcomes are.
404 .Pp
405 Make sure your testcase doesn't depend on any other being run previously
406 as well as that it won't hinder any other testcase from running.
407 This effectively means that your testcase should make no assumptions as
408 to what has been run previously unless it has a registered pre-command
409 and that the system should be left as found before your testcase.
410 .Sh EXAMPLES
411 An example runlist can be found under
412 .Pa test/testcases/sample.run .
413 .Pp
414 Several example testcases for both userland and kernel are under
415 .Pa test/testcases/sample .
416 .Sh SEE ALSO
417 .Xr dfr2text 8 ,
418 .Xr tbridge 9
419 .Sh HISTORY
420 The
421 .Nm
422 utility appeared in
423 .Dx 2.13 .
424 .Sh AUTHORS
425 .An Alex Hornung