Updated the documentation for the floating point extensions.
[lisp-unit.git] / documentation / lisp-unit.html
blob12a5eb3e1cfafd3a622e8b435d91dd00b19e9007
1 <?xml version="1.0" encoding="utf-8"?>
2 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
3 "http://www.w3.org/TR/html4/loose.dtd">
5 <html xmlns="http://www.w3.org/1999/xhtml">
7 <head>
8 <meta http-equiv="content-type" content="text/html; charset=UTF-8"></meta>
9 <title>LISP-UNIT</title>
10 <link href="lisp-unit.css" rel="stylesheet" type="text/css"></link>
11 </head>
13 <body>
15 <div id="banner">LISP-UNIT</div>
17 <p>
18 <strong>lisp-unit</strong> is a Common Lisp library that supports
19 unit testing. It is an extension of the
20 <a href="http://www.cs.northwestern.edu/academics/courses/325/readings/lisp-unit.html">
21 library written by Chris Riesbeck </a>. There is a long history of
22 testing packages in Lisp, usually called "regression" testers. More
23 recent packages in Lisp and other languages have been inspired by
24 JUnit for Java. For more information on both unit testing and JUnit,
25 visit <a href="http://www.junit.org/" target="_blank">www.junit.org</a>.
26 </p>
28 <p>This page has three parts:</p>
30 <ul>
31 <li><a href="#overview">An overview with examples</a></li>
32 <li><a href="#reference">A reference section with the basic forms and functions</a></li>
33 <li><a href="#extension">A reference section with the extended forms and functions</a></li>
34 </ul>
36 <a name="overview"></a>
37 <h1>Overview</h1>
39 <p>
40 The main goal for <strong>lisp-unit</strong> was to make it simple
41 to use. The advantages of <strong>lisp-unit</strong> are:
42 </p>
44 <ul>
45 <li>Written in portable Common Lisp.</li>
46 <li>Loadable with ASDF.</li>
47 <li>Dead-simple to define and run tests. See <a href="#example">example</a>.</li>
48 <li>Supports redefining functions and even macros without reloading tests.</li>
49 <li>Supports <a href="http://www.extremeprogramming.org/rules/testfirst.html">
50 test-first programming</a>.</li>
51 <li>Supports testing return values, printed output, macro
52 expansions, and error conditions.</li>
53 <li>Produces short readable output with a reasonable level of detail.</li>
54 <li>Groups tests by package for modularity.</li>
55 </ul>
57 <h2>How to Use lisp-unit</h2>
59 <ol>
60 <li>Load (or compile and load) <code>(asdf:operate 'asdf:load-op :lisp-unit)</code>.</li>
61 <li>Evaluate <code>(in-package :lisp-unit)</code>.</li>
62 <li>Load a file of tests. See below for how to define tests.</li>
63 <li>Run the tests with <code>run-tests</code>.</li>
64 </ol>
66 <p>
67 Any test failures will be printed, along with a summary of how many
68 tests were run, how many passed, and how many failed.
69 </p>
71 <p>You define a test with <code>define-test</code>:</p>
73 <pre class="code-syntax">(define-test <em>name exp<sub>1</sub> exp<sub>2</sub></em> ...)</pre>
75 <p>
76 This defines a test called <em>name</em>. The expressions can be
77 anything, but typically most will be <a href="#assertions">
78 assertion forms</a>.
79 </p>
81 <p>
82 Tests can be defined before the code they test, even if they're
83 testing macros. This is to support
84 <a href="http://www.extremeprogramming.org/rules/testfirst.html">
85 test-first programming</a>.
86 </p>
88 <p>
89 After defining your tests and the code they test, run the tests with
90 </p>
92 <pre>(run-tests)</pre>
94 <p>
95 This runs every test defined in the current package. To run just
96 certain specific tests, use
97 </p>
99 <pre class="code-syntax">(run-tests <em>name<sub>1</sub> name<sub>2</sub></em> ...)</pre>
101 <p>e.g., <code>(run-tests greater summit)</code>.</p>
103 <a name="example"></a>
104 <p>The following example</p>
105 <ul>
106 <li>defines some tests to see if <code>my-max</code> returns the
107 larger of two arguments</li>
108 <li>defines a deliberately broken version of <code>my-max</code></li>
109 <li>runs the tests</li>
110 </ul>
112 <p>First, we define some tests.</p>
114 <pre>
115 &gt; <b>(in-package :example)</b>
116 #&lt;PACKAGE EXAMPLE&gt;
117 &gt; <b>(define-test test-my-max
118 (assert-equal 5 (my-max 2 5))
119 (assert-equal 5 (my-max 5 2))
120 (assert-equal 10 (my-max 10 10))
121 (assert-equal 0 (my-max -5 0)))</b>
122 TEST-MY-MAX
123 </pre>
126 Following good test-first programming practice, we run these tests
127 <b>before</b> writing any code.
128 </p>
130 <pre>
131 &gt; <b>(run-tests test-my-max)</b>
132 TEST-MY-MAX: Undefined function MY-MAX called with arguments (2 5).
133 </pre>
136 This shows that we need to do some work. So we define our broken
137 version of <code>my-max</code>.
138 </p>
140 <pre>
141 &gt; <b>(defun my-max (x y) x)</b> ;; <em>deliberately wrong</em>
142 MY-MAX
143 </pre>
145 <p>Now we run the tests again:</p>
147 <pre>
148 &gt; <b>(run-tests my-max)</b>
149 MY-MAX: (MY-MAX 2 5) failed: Expected 5 but saw 2
150 MY-MAX: (MY-MAX -5 0) failed: Expected 0 but saw -5
151 MY-MAX: 2 assertions passed, 2 failed.
152 </pre>
155 This shows two failures. In both cases, the equality test returned
156 NIL. In the first case it was because <code>(my-max 2 5)</code>
157 returned 2 when 5 was expected, and in the second case, it was
158 because <code>(my-max -5 0)</code> returned -5 when 0 was expected.
159 </p>
161 <a name="assertions"></a>
162 <h2>Assertion Forms</h2>
164 <p>The most commonly used assertion form is </p>
166 <pre class="code-syntax">(assert-equal <em>value form</em>)</pre>
169 This tallies a failure if <em>form</em> returns a value not
170 <code>equal</code> to <em>value</em>. Both <em>value</em> and
171 <em>test</em> are evaluated in the local lexical environment. This
172 means that you can use local variables in tests. In particular, you
173 can write loops that run many tests at once:
174 </p>
176 <pre>
177 &gt; <b>(define-test my-sqrt
178 (dotimes (i 5)
179 (assert-equal i (my-sqrt (* i i)))))</b>
180 MY-SQRT
182 &gt; <b>(defun my-sqrt (n) (/ n 2))</b> <em>;; wrong!!</em>
184 &gt; <b>(run-tests my-sqrt)</b>
185 MY-SQRT: (MY-SQRT (* I I)) failed: Expected 1 but saw 1/2
186 MY-SQRT: (MY-SQRT (* I I)) failed: Expected 3 but saw 9/2
187 MY-SQRT: (MY-SQRT (* I I)) failed: Expected 4 but saw 8
188 MY-SQRT: 2 assertions passed, 3 failed.
189 </pre>
192 However, the above output doesn't tell us for which values of
193 <code>i</code> the code failed. Fortunately, you can fix this by
194 adding expressions at the end of the <code>assert-equal</code>.
195 These expression and their values will be printed on failure.
196 </p>
198 <pre>
199 &gt; <b>(define-test my-sqrt
200 (dotimes (i 5)
201 (assert-equal i (my-sqrt (* i i)) i)))</b> <em>;; added i at the end</em>
202 MY-SQRT
203 &gt; <b>(run-tests my-sqrt)</b>
204 MY-SQRT: (MY-SQRT (* I I)) failed: Expected 1 but saw 1/2
205 I =&gt; 1
206 MY-SQRT: (MY-SQRT (* I I)) failed: Expected 3 but saw 9/2
207 I =&gt; 3
208 MY-SQRT: (MY-SQRT (* I I)) failed: Expected 4 but saw 8
209 I =&gt; 4
210 MY-SQRT: 2 assertions passed, 3 failed.
211 </pre>
213 <p>The next most useful assertion form is</p>
215 <pre class="code-syntax">(assert-true <em>test</em>)</pre>
218 This tallies a failure if <em>test</em> returns false. Again, if you
219 need to print out extra information, just add expressions after
220 <em>test</em>.
221 </p>
224 There are also assertion forms to test what code prints, what errors
225 code returns, or what a macro expands into. A complete list of
226 assertion forms is in the <a href="#ref-assertion">reference</a> and
227 <a href="#extension">extensions</a> sections.
228 </p>
230 <p class="quote">
231 Do not confuse <code>assert-true</code> with Common Lisp's
232 <code>assert</code> macro. <code>assert</code> is used in code to
233 guarantee that some condition is true. If it isn't, the code
234 halts. <code>assert</code> has options you can use to let a user fix
235 what's wrong and resume execution. A similar collision of names
236 exists in JUnit and Java.
237 </p>
239 <h2>How to Organize Tests with Packages</h2>
242 Tests are grouped internally by the current package, so that a set
243 of tests can be defined for one package of code without interfering
244 with tests for other packages.
245 </p>
248 If your code is being defined in <code>cl-user</code>, which is
249 common when learning Common Lisp, but not for production-level code,
250 then you should define your tests in <code>cl-user</code> as well.
251 </p>
254 If your code is being defined in its own package, you should define
255 your tests either in that same package, or in another package for
256 test code. The latter approach has the advantage of making sure that
257 your tests have access to only the exported symbols of your code
258 package.
259 </p>
262 For example, if you were defining a date package, your
263 <code>date.lisp</code> file would look like this:
264 </p>
266 <blockquote><pre>
267 (defpackage :date
268 (:use :common-lisp)
269 (:export :date-&gt;string :string-&gt;date))
271 (in-package :date)
273 (defun date-&gt;string (date) ...)
274 (defun string-&gt;date (string) ...)
275 </pre></blockquote>
277 <p>Your <code>date-tests.lisp</code> file would look like this:</p>
279 <blockquote><pre>
280 (defpackage :date-tests
281 (:use :common-lisp :lisp-unit :date))
283 (in-package :date-tests)
285 (define-test date-&gt;string
286 (assert-true (string= ... (date-&gt;string ...)))
287 ...)
289 </pre></blockquote>
291 <p>You could then run all your date tests in the test package:</p>
293 <blockquote><pre>
294 (in-package :date-tests)
296 (run-tests)
297 </pre></blockquote>
300 Alternately, you could run all your date tests from any package
301 with:
302 </p>
304 <blockquote><pre>(lisp-unit:run-all-tests :date-tests)</pre></blockquote>
306 <a name="reference"></a>
307 <h1>Reference Section</h1>
310 The list of the basic functions and macros exported by
311 <strong>lisp-unit</strong> are presented in this section.
312 </p>
314 <a name="ref-tests"></a>
315 <h2>Functions for managing tests</h2>
317 <dl>
318 <dt class="code-syntax">
319 (define-test <em>name exp<sub>1</sub> exp<sub>2</sub></em> ...)
320 </dt>
321 <dd>
322 This macro defines a test called <em>name</em> with the
323 expressions specified, in the package specified by the value of
324 <code>*package*</code> in effect when <code>define-test</code> is
325 <u>executed</u>. The expresssions are assembled into runnable
326 code whenever needed by <code>run-tests</code>. Hence you can
327 define or redefine macros without reloading tests using those
328 macros.
329 </dd>
331 <dt class="code-syntax">
332 (get-tests [<em>package</em>])
333 </dt>
334 <dd>
335 This function returns the names of all the tests that have been
336 defined for the <em>package</em>. If no package is given, the
337 value of <code>*package*</code> is used.
338 </dd>
340 <dt class="code-syntax">
341 (get-test-code <em>name</em> [<em>package</em>])
342 </dt>
343 <dd>
344 This function returns the body of the code stored for the test
345 <em>name</em> under <em>package</em>. If no package is given, the
346 value of <code>*package*</code> is used.
347 </dd>
349 <dt class="code-syntax">
350 (remove-tests <em>names</em> [<em>package</em>])
351 </dt>
352 <dd>
353 This function removes the tests named for the given package. If no
354 package is given, the value of <code>*package*</code> is used.
355 </dd>
357 <dt class="code-syntax">
358 (remove-all-tests [<em>package</em>])
359 </dt>
360 <dd>
361 This function removes the tests for the given package. If no
362 package is given, it removes all tests for the current package.
363 If <code>nil</code> is given, it removes all tests for all
364 packages.
365 </dd>
367 <dt class="code-syntax">
368 (run-all-tests <em>package</em>)
369 </dt>
370 <dd>
371 This macro runs all the tests defined in the specified package and
372 reports the results.
373 </dd>
375 <dt class="code-syntax">
376 (run-tests <em>name<sub>1</sub> name<sub>2</sub></em> ...)
377 </dt>
378 <dd>
379 This macro runs the tests named and reports the results. The
380 package used is the value of <code>*package*</code> in effect when
381 the macro is <u>expanded</u>. If no names are given, all tests for
382 that package are run.
383 </dd>
385 <dt class="code-syntax">
386 (use-debugger [<em>flag</em>])
387 </dt>
388 <dd>
389 By default, errors that occur while running tests are simply
390 counted and ignored. You can change this behavior by calling
391 <tt>use-debugger</tt> with one of three possible flag values:
392 <tt>t</tt> (the default) means your Lisp's normal error handling
393 routines will be invoked when errors occur; <tt>:ask</tt> means
394 you will be asked what to do when an error occurs, and
395 <tt>nil</tt> means errors are counted and ignored, i.e., the
396 standard behavior.
397 </dd>
398 </dl>
400 <a name="ref-assertion"></a>
401 <h2>Forms for assertions</h2>
404 All of the assertion forms are macros. They tally a failure if the
405 associated predication returns false. Assertions can be made about
406 return values, printed output, macro expansions, and even expected
407 errors. Assertion form arguments are evaluated in the local lexical
408 environment.
409 </p>
412 All assertion forms allow you to include additional expressions at
413 the end of the form. These expressions and their values will be
414 printed only when the test fails.
415 </p>
417 <p>Return values are unspecified for all assertion forms.</p>
419 <dl>
420 <dt class="code-syntax">
421 (assert-eq <em>value</em> <em>form</em> [<em>form</em><sub>1</sub> <em>form</em><sub>2</sub> ...])
422 </dt>
423 <dt class="code-syntax">
424 (assert-eql <em>value</em> <em>form</em> [<em>form</em><sub>1</sub> <em>form</em><sub>2</sub> ...])
425 </dt>
426 <dt class="code-syntax">
427 (assert-equal <em>value</em> <em>form</em> [<em>form</em><sub>1</sub> <em>form</em><sub>2</sub> ...])
428 </dt>
429 <dt class="code-syntax">
430 (assert-equalp <em>value</em> <em>form</em> [<em>form</em><sub>1</sub> <em>form</em><sub>2</sub> ...])
431 </dt>
432 <dt class="code-syntax">
433 (assert-equality <em>predicate</em> <em>value</em> <em>form</em> [<em>form</em><sub>1</sub> <em>form</em><sub>2</sub> ...])
434 </dt>
435 <dd>
436 These macros tally a failure if <em>value</em> is not equal to the
437 result returned by <em>form</em>, using the specified equality
438 predicate. In general, <code>assert-equal</code> is used for most
439 tests. Example use of <code>assert-equality</code>:
440 <pre style="margin-left: 2%;">
441 (assert-equality #'set-equal '(a b c) (unique-atoms '((b c) a ((b a) c))))
442 </pre>
443 </dd>
445 <dt class="code-syntax">
446 (assert-true <em>test</em> [<em>form</em><sub>1</sub> <em>form</em><sub>2</sub> ...])
447 </dt>
448 <dt class="code-syntax">
449 (assert-false <em>test</em> [<em>form</em><sub>1</sub> <em>form</em><sub>2</sub> ...])
450 </dt>
451 <dd>
452 <code>assert-true</code> tallies a failure if <em>test</em>
453 returns false. <code>assert-false</code> tallies a failure if
454 <em>test</em> returns true.
455 </dd>
457 <dt class="code-syntax">
458 (assert-prints "<em>output</em>" <em>form</em> [<em>form</em><sub>1</sub> <em>form</em><sub>2</sub> ...])
459 </dt>
460 <dd>
461 This macro tallies a failure if form does not print to standard
462 output stream output equal to the given string, ignoring
463 differences in beginning and ending newlines.
464 </dd>
466 <dt class="code-syntax">
467 (assert-expands <em>expansion</em> <em>form</em> [<em>form</em><sub>1</sub> <em>form</em><sub>2</sub> ...])
468 </dt>
469 <dd>
470 This macro tallies a failure if <code>(macroexpand-1
471 <em>form</em>)</code> does not produce a value equal to
472 <em>expansion</em>.
473 </dd>
475 <dt class="code-syntax">
476 (assert-error <em>condition-type</em> <em>form</em> [<em>form</em><sub>1</sub> <em>form</em><sub>2</sub> ...])
477 </dt>
478 <dd>
479 This macro tallies a failure if <em>form</em> does not signal an
480 error that is equal to or a subtype of
481 <em>condition-type</em>. Use <code>error</code> to refer to any
482 kind of error. See <a
483 href="http://www.lispworks.com/documentation/HyperSpec/Body/09_aa.htm">condition
484 types in the Common Lisp Hyperspec</a> for other possible
485 names. For example,
487 <pre>(assert-error 'arithmetic-error (foo 0))</pre>
489 would assert that <code>foo</code> is supposed to signal an
490 arithmetic error when passed zero.
491 </dd>
492 </dl>
494 <a name="ref-predicates"></a>
495 <h2>Utility predicates</h2>
498 Several predicate functions are exported that are often useful in
499 writing tests with <code>assert-equality</code>.
500 </p>
502 <dl>
503 <dt class="code-syntax">
504 (logically-equal <em>value</em><sub>1</sub> <em>value</em><sub>2</sub>)
505 </dt>
506 <dd>
507 This predicate returns true of the two values are either both
508 true, i.e., non-NIL, or both false.
509 </dd>
511 <dt class="code-syntax">
512 (set-equal <em>list</em><sub>1</sub> <em>list</em><sub>2</sub> [:test])
513 </dt>
514 <dd>
515 This predicate returns true the first list is a subset of the
516 second and vice versa. <code>:test</code> can be used to specify
517 an equality predicate. The default is <code>eql</code>.
518 </dd>
519 </dl>
521 <a name="extension"></a>
522 <h1>Extensions Section</h1>
525 The extensions to <strong>lisp-unit</strong> are presented in this
526 section. The original <strong>lisp-unit</strong> has been extended
527 with predicate functions and assertions to support numerical
528 testing. The predicates can be used with
529 <code>assert-equality</code> or with the corresponding
530 assertions. All extensions are implemented using generic functions
531 and consequently can be specialized for user classes.
532 </p>
534 <h2>Floating point predicates and assertions</h2>
537 The internal default value of <code>epsilon</code> is is twice the
538 appropriate float epsilon (i.e. <code>2*single-float-epsilon</code>
539 or <code>2*double-float-epsilon</code>). Epsilon can be controlled
540 at a lexical level using the package variable
541 <code>*epsilon*</code>. If <code>*epsilon*</code> is set to
542 <code>nil</code>, the internal epsilon values are used. This is the
543 default value of <code>epsilon</code>.
544 </p>
546 <dl>
547 <dt class="code-syntax">
548 (float-equal <em>data</em><sub>1</sub> <em>data</em><sub>2</sub> [epsilon])
549 </dt>
550 <dt class="code-syntax">
551 (assert-float-equal <em>value</em> <em>form</em>
552 [<em>form</em><sub>1</sub> <em>form</em><sub>2</sub> ...])
553 </dt>
554 <dd>
555 Return true if the relative error between
556 <code>data<sub>1</sub></code> and <code>data<sub>2</sub></code>
557 is less than <code>epsilon</code>. The assertion tallies the
558 failure if <code>value</code> is not equal to the result returned
559 from <code>form</code>, using <code>float-equal</code>.
560 </dd>
562 <dt class="code-syntax">
563 (sigfig-equal <em>float</em><sub>1</sub> <em>float</em><sub>2</sub>
564 [significant-figures])
565 </dt>
566 <dt class="code-syntax">
567 (assert-sigfig-equal <em>value</em> <em>form</em>
568 [<em>form</em><sub>1</sub> <em>form</em><sub>2</sub> ...])
569 </dt>
570 <dd>
571 Return true if <code>float<sub>1</sub></code> and
572 <code>float<sub>2</sub></code> are equal to the specified
573 <code>significant-figures</code>. The default value of significant
574 figures is 4, set by the global variable
575 <code>*significant-figures</code>. The test tallies the failure
576 if <code>value</code> is not equal to the result returned from
577 <code>form</code>, using <code>sigfig-equal</code>.
578 </dd>
580 <dt class="code-syntax">
581 (norm-equal <em>data</em><sub>1</sub> <em>data</em><sub>2</sub>
582 [epsilon] [measure])
583 </dt>
584 <dt class="code-syntax">
585 (assert-norm-equal <em>value</em> <em>form</em>
586 [<em>form</em><sub>1</sub> <em>form</em><sub>2</sub> ...])
587 </dt>
588 <dd>
589 Return true if the relative error norm between
590 <code>data<sub>1</sub></code> and <code>data<sub>2</sub></code> is
591 less than <code>epsilon</code>.
592 </dd>
593 </dl>
595 <h2>GSLL Specific Predicates and Assertions</h2>
596 <dl>
597 <dt class="code-syntax">
598 (number-equal <em>number</em><sub>1</sub> <em>number</em><sub>2</sub>
599 [epsilon] [type-eq-p])
600 </dt>
601 <dt class="code-syntax">
602 (assert-number-equal <em>value</em> <em>form</em>
603 [<em>form</em><sub>1</sub> <em>form</em><sub>2</sub> ...])
604 </dt>
605 <dd>
606 Return true if the error between <code>number<sub>1</sub></code>
607 and <code>number<sub>2</sub></code> is less than
608 <code>epsilon</code>. The numbers must be the same type unless
609 <code>type-eq-p</code> is <code>t</code>. For the comparison, both
610 numbers are coerced to <code>(complex double-float)</code> and
611 passed to <code>float-equal</code>. The test tallies the failure
612 if <code>value</code> is not equal to the result returned from
613 <code>form</code>, using <code>number-equal</code>.
614 </dd>
616 <dt class="code-syntax">
617 (numerical-equal <em>result</em><sub>1</sub> <em>result</em><sub>2</sub> [:test])
618 </dt>
619 <dt class="code-syntax">
620 (assert-numerical-equal <em>value</em> <em>form</em>
621 [<em>form</em><sub>1</sub> <em>form</em><sub>2</sub> ...])
622 </dt>
623 <dd>
624 Return true if the numerical <code>result<sub>1</sub></code> is
625 equal to <code>result<sub>2</sub></code> as defined by
626 <code>:test</code>. The results can be numbers, sequences, nested
627 sequences and arrays. The test tallies the failure if
628 <code>value</code> is not equal to the result returned from
629 <code>form</code>, using <code>numerical-equal</code>. In general,
630 <code>test</code> must be a function that accepts 2 arguments and
631 returns <em>true</em> or <em>false</em>.
632 </dd>
633 </dl>
635 <h2>Floating point functions</h2>
637 The floating point functions can be specialized for user data types
638 and aid in writing user specific predicates.
639 </p>
641 <dl>
642 <dt class="code-syntax">
643 (default-epsilon <em>value</em>)
644 </dt>
645 <dd>
646 Return the default epsilon for <code>value</code>.
647 </dd>
648 <dt class="code-syntax">
649 (relative-error <em>exact</em> <em>approximate</em>)
650 </dt>
651 <dd>
652 Return the relative error.
653 </dd>
654 <dt class="code-syntax">
655 (sumsq <em>data</em>)
656 </dt>
657 <dd>
658 Return the scaling parameter and the sum of the squares of the
659 data.
660 </dd>
661 <dt class="code-syntax">
662 (sump <em>data</em> <em>p</em>)
663 </dt>
664 <dd>
665 Return the scaling parameter and the sum of the powers of p of the
666 data.
667 </dd>
668 <dt class="code-syntax">
669 (norm <em>data</em> [<em>measure</em>])
670 </dt>
671 <dd>
672 Return the element-wise norm of the data.
673 </dd>
674 <dt class="code-syntax">
675 (relative-error-norm <em>exact</em> <em>approximate</em> [<em>measure</em>])
676 </dt>
677 <dd>
678 Return the relative error norm.
679 </dd>
680 </dl>
682 <hr></hr>
684 <p id="closing">
685 <a href="mailto:tmh.public@gmail.com">Comments or suggestions?</a>
686 </p>
688 </body>
689 </html>