documentation/lisp-unit.html

   1 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
   2 <html><head>
   3 <meta http-equiv="content-type" content="text/html; charset=UTF-8">
   4    <title>LISP-UNIT</title>
   5 <link href="lisp-unit.css" rel="stylesheet" type="text/css">
   6 </head><body>
   7
   8 <div id="banner">LISP-UNIT</div>
   9
  10 <p><strong>lisp-unit</strong> is a Common Lisp library that supports
  11 unit testing. It is an extension of the
  12 <a href="http://www.cs.northwestern.edu/academics/courses/325/readings/lisp-unit.html">
  13 library written by Chris Riesbeck</a>. There is a long history of testing
  14 packages in Lisp, usually called "regression" testers. More recent
  15 packages in Lisp and other languages have been inspired by JUnit for
  16 Java. For more information on both unit testing and JUnit, visit <a
  17 href="http://www.junit.org/" target="_blank">www.junit.org</a>.</p>
  18
  19 <p>This page has two parts:</p>
  20
  21 <ul>
  22 <li><a href="#overview">An overview with examples</a></li>
  23 <li><a href="#reference">A reference section with all forms and functions</a></li>
  24 </ul>
  25
  26 <a name="overview"></a>
  27 <h1>Overview</h1>
  28
  29 <p>The main goal for <strong>lisp-unit</strong> was to make it simple
  30 to use. The advantages of <strong>lisp-unit</strong> are:</p>
  31
  32 <ul>
  33 <li>Written in portable Common Lisp.</li>
  34 <li>Loadable with ASDF.</li>
  35 <li>Dead-simple to define and run tests. See <a href="#example">example</a>.</li>
  36 <li>Supports redefining functions and even macros without reloading tests.</li>
  37 <li>Supports <a href="http://www.extremeprogramming.org/rules/testfirst.html">test-first programming</a>.</li>
  38 <li>Supports testing return values, printed output, macro expansions, and error conditions.</li>
  39 <li>Produces short readable output with a reasonable level of detail.</li>
  40 <li>Groups tests by package for modularity.</li>
  41 </ul>
  42
  43 <h2>How to Use lisp-unit</h2>
  44
  45 <ol>
  46 <li>Load (or compile and load) <code>(asdf:operate 'asdf:load-op :lisp-unit)</code>.</li>
  47
  48 <li>Evaluate <code>(in-package :lisp-unit)</code>.</li>
  49
  50 <li>Load a file of tests.  See below for how to define tests.</li>
  51
  52 <li>Run the tests with <code>run-tests</code>.</li>
  53 </ol>
  54
  55 <p>Any test failures will be printed, along with a summary of how many tests were
  56 run, how many passed, and how many failed.</p>
  57
  58 <p>You define a test with <code>define-test</code>:</p>
  59
  60 <pre class="code-syntax">(define-test <em>name exp<sub>1</sub> exp<sub>2</sub></em> ...)</pre>
  61
  62 <p>This defines a test called <em>name</em>. The expressions can be anything,
  63 but typically most will be
  64 <a href="#assertions">assertion forms</a>.</p>
  65
  66 <p>Tests can be defined before the code they test, even if they're testing
  67 macros. This is to support
  68 <a href="http://www.extremeprogramming.org/rules/testfirst.html">test-first programming</a>.</p>
  69
  70 <p>After defining your tests and the code they test, run
  71 the tests with</p>
  72
  73 <pre>(run-tests)</pre>
  74
  75 <p>This runs every test defined
  76 in the current package. To run just certain specific tests, use</p>
  77
  78 <pre class="code-syntax">(run-tests <em>name<sub>1</sub> name<sub>2</sub></em> ...)</pre>
  79
  80 <p>e.g., <code>(run-tests greater summit)</code>.</p>
  81
  82 <a name="example"></a>
  83 <p>The following example</p>
  84 <ul>
  85 <li>defines some tests to see if <code>my-max</code>
  86 returns the larger of two arguments</li>
  87 <li>defines a deliberately broken version of <code>my-max</code></li>
  88 <li>runs the tests</li>
  89 </ul>
  90
  91 <p>First, we define some tests.</p>
  92
  93 <pre>&gt; <b>(in-package :example)</b>
  94 #&lt;PACKAGE EXAMPLE&gt;
  95 &gt; <b>(define-test test-my-max
  96    (assert-equal 5 (my-max 2 5))
  97    (assert-equal 5 (my-max 5 2))
  98    (assert-equal 10 (my-max 10 10))
  99    (assert-equal 0 (my-max -5 0))
 100    )</b>
 101 TEST-MY-MAX
 102 </pre>
 103
 104 <p>Following good test-first programming practice, we run these
 105 tests <b>before</b> writing any code.</p>
 106
 107 <pre>&gt; <b>(run-tests test-my-max)</b>
 108     TEST-MY-MAX: Undefined function MY-MAX called with arguments (2 5).
 109 </pre>
 110
 111 <p>This shows that we need to do some work. So we define our
 112 broken version of <code>my-max</code>.</p>
 113
 114 <pre>&gt; <b>(defun my-max (x y) x)</b>  ;; <em>deliberately wrong</em>
 115 MY-MAX</pre>
 116
 117 <p>Now we run the tests again:</p>
 118
 119 <pre>&gt; <b>(run-tests my-max)</b>
 120 MY-MAX: (MY-MAX 2 5) failed: Expected 5 but saw 2
 121 MY-MAX: (MY-MAX -5 0) failed: Expected 0 but saw -5
 122 MY-MAX: 2 assertions passed, 2 failed.</pre>
 123
 124 <p>This shows two failures. In both cases, the equality test
 125 returned NIL. In the first case it was because
 126 <code>(my-max 2 5)</code> returned 2 when 5 was expected, and
 127 in the second case, it was because
 128 <code>(my-max -5 0)</code> returned -5 when 0 was expected.</p>
 129
 130 <a name="assertions"></a>
 131 <h2> Assertion Forms</h2>
 132
 133 <p>The most commonly used assertion form is </p>
 134
 135 <pre class="code-syntax">(assert-equal <em>value form</em>)</pre>
 136
 137 <p>This tallies a failure if <em>form</em> returns a value
 138 not <code>equal</code> to <em>value</em>. Both
 139 <em>value</em> and <em>test</em> are
 140 evaluated in the local lexical environment. This means
 141 that you can use local variables in tests. In particular,
 142 you can write loops that run many tests at once:</p>
 143
 144 <pre>&gt; <b>(define-test my-sqrt
 145   (dotimes (i 5)
 146     (assert-equal i (my-sqrt (* i i)))))</b>
 147 MY-SQRT
 148
 149 &gt; <b>(defun my-sqrt (n) (/ n 2))</b>   <em>;; wrong!!</em>
 150
 151 &gt; <b>(run-tests my-sqrt)</b>
 152 MY-SQRT: (MY-SQRT (* I I)) failed: Expected 1 but saw 1/2
 153 MY-SQRT: (MY-SQRT (* I I)) failed: Expected 3 but saw 9/2
 154 MY-SQRT: (MY-SQRT (* I I)) failed: Expected 4 but saw 8
 155 MY-SQRT: 2 assertions passed, 3 failed.</pre>
 156
 157 <p>However, the above output doesn't tell us for which values of <code>i</code>
 158 the code failed. Fortunately, you can fix this by
 159 adding expressions at the end of the <code>assert-equal</code>.
 160 These expression and their values will be printed on failure.</p>
 161
 162 <pre>&gt; <b>(define-test my-sqrt
 163   (dotimes (i 5)
 164     (assert-equal i (my-sqrt (* i i)) i)))</b>  <em>;; added i at the end</em>
 165 MY-SQRT
 166 &gt; <b>(run-tests my-sqrt)</b>
 167 MY-SQRT: (MY-SQRT (* I I)) failed: Expected 1 but saw 1/2
 168    I =&gt; 1
 169 MY-SQRT: (MY-SQRT (* I I)) failed: Expected 3 but saw 9/2
 170    I =&gt; 3
 171 MY-SQRT: (MY-SQRT (* I I)) failed: Expected 4 but saw 8
 172    I =&gt; 4
 173 MY-SQRT: 2 assertions passed, 3 failed.</pre>
 174
 175 <p>The next most useful assertion form is</p>
 176
 177 <pre class="code-syntax">(assert-true <em>test</em>)</pre>
 178
 179 <p>This tallies a failure if <em>test</em> returns false. Again,
 180 if you need to print out extra information, just add expressions
 181 after <em>test</em>.</p>
 182
 183 <p>There are also assertion forms to test
 184 what code prints, what errors code returns, or what
 185 a macro expands into.
 186 A complete list of assertion forms is in
 187 <a href="#ref-assertion">the reference section</a>.</p>
 188
 189 <p class="quote">Do not confuse <code>assert-true</code>
 190 with Common Lisp's <code>assert</code> macro. <code>assert</code>
 191 is used in code to guarantee that some condition is true. If it isn't,
 192 the code halts. <code>assert</code> has options you can use
 193 to let a user fix what's wrong and resume execution. A similar collision
 194 of names exists in JUnit and Java.</p>
 195
 196
 197 <h2>How to Organize Tests with Packages</h2>
 198
 199 <p>Tests are grouped internally by the current package,
 200 so that a set of tests can be defined for one package
 201 of code without interfering with tests for other packages.</p>
 202
 203 <p>If your code is being defined in <code>cl-user</code>,
 204 which is common when learning Common Lisp, but not for
 205 production-level code, then you should define your tests
 206 in <code>cl-user</code> as well.</p>
 207
 208 <p>If your code is being defined in its own package,
 209 you should define your tests either in that same package,
 210 or in another package for test code. The latter
 211 approach has the advantage of making sure that your
 212 tests have access to only the exported symbols of your
 213 code package.</p>
 214
 215 <p>For example, if you were defining a date package,
 216 your <code>date.lisp</code> file would look like this:</p>
 217
 218 <blockquote><pre>(defpackage :date
 219   (:use :common-lisp)
 220   (:export #:date-&gt;string #:string-&gt;date))
 221
 222 (in-package :date)
 223
 224 (defun date-&gt;string (date) ...)
 225 (defun string-&gt;date (string) ...)
 226 </pre></blockquote>
 227
 228 <p>Your <code>date-tests.lisp</code> file would look like this:</p>
 229
 230 <blockquote><pre>(defpackage :date-tests
 231   (:use :common-lisp :lisp-unit :date))
 232
 233 (in-package :date-tests)
 234
 235 (define-test date-&gt;string
 236   (assert-true (string= ... (date-&gt;string ...)))
 237   ...)
 238 ...
 239 </pre></blockquote>
 240
 241
 242 <p>You could then run all your date tests in
 243 the test package:</p>
 244
 245 <blockquote><pre>(in-package :date-tests)
 246
 247 (run-tests)
 248 </pre></blockquote>
 249
 250 <p>Alternately, you could run all your date tests from any package
 251 with:</p>
 252
 253 <blockquote><pre>(lisp-unit:run-all-tests :date-tests)</pre></blockquote>
 254
 255
 256 <a name="reference"></a>
 257 <h1>Reference Section</h1>
 258
 259 <p>Here is a list of the functions and macros
 260 exported by <strong>lisp-unit</strong>.</p>
 261
 262 <a name="ref-tests"></a>
 263 <h3>Functions for managing tests</h3>
 264
 265 <dl>
 266 <dt class="code-syntax">(define-test <em>name exp<sub>1</sub> exp<sub>2</sub></em> ...)</dt>
 267
 268 <dd>This macro defines a test called <em>name</em> with the expressions
 269 specified, in the package specified by
 270 the value of <code>*package*</code>
 271 in effect when <code>define-test</code> is <u>executed</u>.
 272 The expresssions are assembled into runnable code whenever
 273 needed by <code>run-tests</code>. Hence you can define or
 274 redefine macros without reloading tests using those macros.
 275 </dd>
 276
 277 <dt class="code-syntax">(get-tests [<em>package</em>)</dt>
 278 <dd>This function returns the names of all
 279 the tests that have been defined for the <em>package</em>. If no package
 280 is given, the value of <code>*package*</code> is used.</dd>
 281
 282 <dt class="code-syntax">(get-test-code <em>name</em> [<em>package</em>)</dt>
 283 <dd>This function returns the body of the code stored for the test
 284 <em>name</em> under <em>package</em>. If no package
 285 is given, the value of <code>*package*</code> is used.</dd>
 286
 287 <dt class="code-syntax">(remove-tests <em>names</em> [<em>package</em>])</dt>
 288 <dd>This function removes the tests named for the given package.
 289 If no package
 290 is given, the value of <code>*package*</code> is used.</dd>
 291
 292 <dt class="code-syntax">(remove-all-tests [<em>package</em>])</dt>
 293 <dd>This function removes the tests for the given package.
 294 If no package is given, it removes all tests for the current package.
 295 If <code>nil</code> is given, it removes all tests for all packages.</dd>
 296
 297 <dt class="code-syntax">(run-all-tests <em>package</em>)</dt>
 298 <dd>This macro runs all the tests defined in the specified package
 299 and reports the results.</dd>
 300
 301 <dt class="code-syntax">(run-tests <em>name<sub>1</sub> name<sub>2</sub></em> ...)</dt>
 302 <dd>This macro runs the tests named and reports the results.
 303 The package used is the value of <code>*package*</code>
 304 in effect when the macro is <u>expanded</u>. If no names are given,
 305 all tests for that package are run.</dd>
 306
 307 <dt class="code-syntax">(use-debugger [<em>flag</em>])</dt>
 308 <dd>By default, errors that occur while running tests are simply
 309 counted and ignored. You can change this behavior by calling <tt>use-debugger</tt>
 310 with one of three possible flag values: <tt>t</tt> (the default) means
 311 your Lisp's normal error handling routines will be invoked when
 312 errors occur; <tt>:ask</tt> means you will be asked what to do
 313 when an error occurs, and <tt>nil</tt> means errors are counted and
 314 ignored, i.e., the standard behavior.</dd>
 315
 316 </dl>
 317
 318 <a name="ref-assertion"></a>
 319 <h3>Forms for assertions</h3>
 320
 321 <p>All of the assertion forms are macros.
 322 They tally a failure if the associated predication
 323 returns false. Assertions can be made about
 324 return values, printed output, macro expansions, and even
 325 expected errors. Assertion form arguments are evaluated
 326 in the local lexical environment.</p>
 327
 328 <p>All assertion forms
 329 allow you to include additional expressions at the end of the form. These
 330 expressions and their values will
 331 be printed only when the test fails.</p>
 332
 333 <p>Return values are unspecified for all assertion forms.</p>
 334
 335 <dl>
 336
 337 <dt class="code-syntax"> (assert-eq <em>value</em> <em>form</em> [<em>form</em><sub>1</sub> <em>form</em><sub>2</sub> ...])</dt>
 338 <dt class="code-syntax"> (assert-eql <em>value</em> <em>form</em> [<em>form</em><sub>1</sub> <em>form</em><sub>2</sub> ...])</dt>
 339 <dt class="code-syntax"> (assert-equal <em>value</em> <em>form</em> [<em>form</em><sub>1</sub> <em>form</em><sub>2</sub> ...])</dt>
 340 <dt class="code-syntax"> (assert-equalp <em>value</em> <em>form</em> [<em>form</em><sub>1</sub> <em>form</em><sub>2</sub> ...])</dt>
 341 <dt class="code-syntax"> (assert-equality <em>predicate</em> <em>value</em> <em>form</em> [<em>form</em><sub>1</sub> <em>form</em><sub>2</sub> ...])</dt>
 342 <dd>These macros tally a failure if <em>value</em> is not equal to
 343 the result returned by <em>form</em>, using the specified equality predicate.
 344 <br>
 345 In general, <code>assert-equal</code>
 346 is used for most tests.<br>
 347 Example use of <code>assert-equality</code>:
 348 <pre style="margin-left: 2%;">(assert-equality #'set-equal '(a b c) (unique-atoms '((b c) a ((b a) c))))</pre>
 349 </dd>
 350
 351 <dt class="code-syntax">(assert-true <em>test</em> [<em>form</em><sub>1</sub> <em>form</em><sub>2</sub> ...])</dt>
 352 <dt class="code-syntax">(assert-false <em>test</em> [<em>form</em><sub>1</sub> <em>form</em><sub>2</sub> ...])</dt>
 353 <dd><code>assert-true</code>
 354 tallies a failure if <em>test</em> returns false.
 355 <code>assert-false</code>
 356 tallies a failure if <em>test</em> returns true.<br>
 357 </dd>
 358
 359 <dt class="code-syntax"> (assert-prints "<em>output</em>" <em>form</em> [<em>form</em><sub>1</sub> <em>form</em><sub>2</sub> ...])</dt>
 360 <dd>This macro tallies a failure if form does not print to standard output stream
 361 output equal to the
 362 given string, ignoring differences in beginning and ending newlines.</dd>
 363
 364 <dt class="code-syntax"> (assert-expands <em>expansion</em> <em>form</em> [<em>form</em><sub>1</sub> <em>form</em><sub>2</sub> ...])</dt>
 365 <dd>This macro tallies a failure if <code>(macroexpand-1 <em>form</em>)</code>
 366 does not produce a value equal to <em>expansion</em>.
 367 </dd>
 368
 369 <dt class="code-syntax">(assert-error <em>condition-type</em> <em>form</em> [<em>form</em><sub>1</sub> <em>form</em><sub>2</sub> ...])</dt>
 370 <dd>This macro tallies a failure if <em>form</em> does not
 371 signal an error that
 372 is equal to or a subtype of <em>condition-type</em>. Use <code>error</code>
 373 to refer to any kind of error. See
 374 <a href="http://www.lisp.org/HyperSpec/Body/sec_9-1-1.html">condition
 375 types in the Common Lisp Hyperspec</a> for other possible names. For example,
 376
 377 <pre>(assert-error 'arithmetic-error (foo 0))</pre>
 378
 379 would assert that <code>foo</code>
 380 is supposed to signal an arithmetic error when passed zero.</dd>
 381 </dl>
 382
 383
 384 <a name="ref-predicates"></a>
 385 <h3>Utility predicates</h3>
 386
 387 <p>Several predicate functions are exported that are often useful in
 388 writing tests with <code>assert-equality</code>.</p>
 389
 390 <dl>
 391 <dt class="code-syntax">(logically-equal <em>value</em><sub>1</sub> <em>value</em><sub>2</sub>)</dt>
 392 <dd>This predicate returns true of the two values are either
 393 both true, i.e., non-NIL, or both false.</dd>
 394
 395 <dt class="code-syntax">(set-equal <em>list</em><sub>1</sub> <em>list</em><sub>2</sub> [:test])</dt>
 396 <dd>This predicate returns true the first list is a subset of the
 397 second and vice versa. <code>:test</code>
 398 can be used to specify an equality predicate. The default is <code>eql</code>.</dd>
 399
 400
 401 </dl>
 402
 403 <hr>
 404
 405 <p id="closing"><a href="mailto:tmh.public@gmail.com">Comments or suggestions?</a></p>
 406
 407
 408 </body></html>