mingw/html/lib/Test/Harness/TAP.html

   1 <?xml version="1.0" ?>
   2 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
   3 <html xmlns="http://www.w3.org/1999/xhtml">
   4 <head>
   5 <title>Test::Harness::TAP - Documentation for the TAP format</title>
   6 <meta http-equiv="content-type" content="text/html; charset=utf-8" />
   7 <link rev="made" href="mailto:" />
   8 </head>
   9
  10 <body style="background-color: white">
  11 <table border="0" width="100%" cellspacing="0" cellpadding="3">
  12 <tr><td class="block" style="background-color: #cccccc" valign="middle">
  13 <big><strong><span class="block">&nbsp;Test::Harness::TAP - Documentation for the TAP format</span></strong></big>
  14 </td></tr>
  15 </table>
  16
  17 <p><a name="__index__"></a></p>
  18 <!-- INDEX BEGIN -->
  19
  20 <ul>
  21
  22         <li><a href="#name">NAME</a></li>
  23         <li><a href="#synopsis">SYNOPSIS</a></li>
  24         <li><a href="#todo">TODO</a></li>
  25         <li><a href="#the_tap_format">THE TAP FORMAT</a></li>
  26         <li><a href="#harness_behavior">HARNESS BEHAVIOR</a></li>
  27         <li><a href="#tests_lines_and_the_plan">TESTS LINES AND THE PLAN</a></li>
  28         <ul>
  29
  30                 <li><a href="#the_plan">The plan</a></li>
  31                 <li><a href="#the_test_line">The test line</a></li>
  32         </ul>
  33
  34         <li><a href="#directives">DIRECTIVES</a></li>
  35         <ul>
  36
  37                 <li><a href="#todo_tests">TODO tests</a></li>
  38                 <li><a href="#skipping_tests">Skipping tests</a></li>
  39         </ul>
  40
  41         <li><a href="#other_lines">OTHER LINES</a></li>
  42         <ul>
  43
  44                 <li><a href="#bail_out_">Bail out!</a></li>
  45                 <li><a href="#diagnostics">Diagnostics</a></li>
  46                 <li><a href="#anything_else">Anything else</a></li>
  47         </ul>
  48
  49         <li><a href="#examples">EXAMPLES</a></li>
  50         <ul>
  51
  52                 <li><a href="#common_with_explanation">Common with explanation</a></li>
  53                 <li><a href="#unknown_amount_and_failures">Unknown amount and failures</a></li>
  54                 <li><a href="#giving_up">Giving up</a></li>
  55                 <li><a href="#skipping_a_few">Skipping a few</a></li>
  56                 <li><a href="#skipping_everything">Skipping everything</a></li>
  57                 <li><a href="#got_spare_tuits">Got spare tuits?</a></li>
  58                 <li><a href="#creative_liberties">Creative liberties</a></li>
  59         </ul>
  60
  61         <li><a href="#authors">AUTHORS</a></li>
  62         <li><a href="#acknowledgements">ACKNOWLEDGEMENTS</a></li>
  63         <li><a href="#copyright">COPYRIGHT</a></li>
  64 </ul>
  65 <!-- INDEX END -->
  66
  67 <hr />
  68 <p>
  69 </p>
  70 <h1><a name="name">NAME</a></h1>
  71 <p>Test::Harness::TAP - Documentation for the TAP format</p>
  72 <p>
  73 </p>
  74 <hr />
  75 <h1><a name="synopsis">SYNOPSIS</a></h1>
  76 <p>TAP, the Test Anything Protocol, is Perl's simple text-based interface
  77 between testing modules such as Test::More and the test harness
  78 Test::Harness.</p>
  79 <p>
  80 </p>
  81 <hr />
  82 <h1><a name="todo">TODO</a></h1>
  83 <p>Exit code of the process.</p>
  84 <p>
  85 </p>
  86 <hr />
  87 <h1><a name="the_tap_format">THE TAP FORMAT</a></h1>
  88 <p>TAP's general format is:</p>
  89 <pre>
  90     1..N
  91     ok 1 Description # Directive
  92     # Diagnostic
  93     ....
  94     ok 47 Description
  95     ok 48 Description
  96     more tests....</pre>
  97 <p>For example, a test file's output might look like:</p>
  98 <pre>
  99     1..4
 100     ok 1 - Input file opened
 101     not ok 2 - First line of the input valid
 102     ok 3 - Read the rest of the file
 103     not ok 4 - Summarized correctly # TODO Not written yet</pre>
 104 <p>
 105 </p>
 106 <hr />
 107 <h1><a name="harness_behavior">HARNESS BEHAVIOR</a></h1>
 108 <p>In this document, the ``harness'' is any program analyzing TAP output.
 109 Typically this will be Perl's <em>prove</em> program, or the underlying
 110 <code>Test::Harness::runtests</code> subroutine.</p>
 111 <p>A harness must only read TAP output from standard output and not
 112 from standard error.  Lines written to standard output matching
 113 <code>/^(not )?ok\b/</code> must be interpreted as test lines.  All other
 114 lines must not be considered test output.</p>
 115 <p>
 116 </p>
 117 <hr />
 118 <h1><a name="tests_lines_and_the_plan">TESTS LINES AND THE PLAN</a></h1>
 119 <p>
 120 </p>
 121 <h2><a name="the_plan">The plan</a></h2>
 122 <p>The plan tells how many tests will be run, or how many tests have
 123 run.  It's a check that the test file hasn't stopped prematurely.
 124 It must appear once, whether at the beginning or end of the output.</p>
 125 <p>The plan is usually the first line of TAP output and it specifies how
 126 many test points are to follow. For example,</p>
 127 <pre>
 128     1..10</pre>
 129 <p>means you plan on running 10 tests. This is a safeguard in case your test
 130 file dies silently in the middle of its run.  The plan is optional but if
 131 there is a plan before the test points it must be the first non-diagnostic
 132 line output by the test file.</p>
 133 <p>In certain instances a test file may not know how many test points
 134 it will ultimately be running. In this case the plan can be the last
 135 non-diagnostic line in the output.</p>
 136 <p>The plan cannot appear in the middle of the output, nor can it appear more
 137 than once.</p>
 138 <p>
 139 </p>
 140 <h2><a name="the_test_line">The test line</a></h2>
 141 <p>The core of TAP is the test line.  A test file prints one test line test
 142 point executed. There must be at least one test line in TAP output. Each
 143 test line comprises the following elements:</p>
 144 <ul>
 145 <li><strong><a name="item_ok_or_not_ok"><a href="#item_ok"><code>ok</code></a> or <code>not ok</code></a></strong>
 146
 147 <p>This tells whether the test point passed or failed. It must be
 148 at the beginning of the line. <code>/^not ok/</code> indicates a failed test
 149 point. <code>/^ok/</code> is a successful test point. This is the only mandatory
 150 part of the line.</p>
 151 <p>Note that unlike the Directives below, <a href="#item_ok"><code>ok</code></a> and <code>not ok</code> are
 152 case-sensitive.</p>
 153 </li>
 154 <li><strong><a name="item_test_number">Test number</a></strong>
 155
 156 <p>TAP expects the <a href="#item_ok"><code>ok</code></a> or <code>not ok</code> to be followed by a test point
 157 number. If there is no number the harness must maintain
 158 its own counter until the script supplies test numbers again. So
 159 the following test output</p>
 160 <pre>
 161     1..6
 162     not ok
 163     ok
 164     not ok
 165     ok
 166     ok</pre>
 167 <p>has five tests.  The sixth is missing.  Test::Harness will generate</p>
 168 <pre>
 169     FAILED tests 1, 3, 6
 170     Failed 3/6 tests, 50.00% okay</pre>
 171 </li>
 172 <li><strong><a name="item_description">Description</a></strong>
 173
 174 <p>Any text after the test number but before a <code>#</code> is the description of
 175 the test point.</p>
 176 <pre>
 177     ok 42 this is the description of the test</pre>
 178 <p>Descriptions should not begin with a digit so that they are not confused
 179 with the test point number.</p>
 180 <p>The harness may do whatever it wants with the description.</p>
 181 </li>
 182 <li><strong><a name="item_directive">Directive</a></strong>
 183
 184 <p>The test point may include a directive, following a hash on the
 185 test line.  There are currently two directives allowed: <code>TODO</code> and
 186 <code>SKIP</code>.  These are discussed below.</p>
 187 </li>
 188 </ul>
 189 <p>To summarize:</p>
 190 <ul>
 191 <li><strong><a name="item_ok">ok/not ok (required)</a></strong>
 192
 193 <li><strong><a name="item_number">Test number (recommended)</a></strong>
 194
 195 <li><strong>Description (recommended)</strong>
 196
 197 <li><strong>Directive (only when necessary)</strong>
 198
 199 </ul>
 200 <p>
 201 </p>
 202 <hr />
 203 <h1><a name="directives">DIRECTIVES</a></h1>
 204 <p>Directives are special notes that follow a <code>#</code> on the test line.
 205 Only two are currently defined: <code>TODO</code> and <code>SKIP</code>.  Note that
 206 these two keywords are not case-sensitive.</p>
 207 <p>
 208 </p>
 209 <h2><a name="todo_tests">TODO tests</a></h2>
 210 <p>If the directive starts with <code># TODO</code>, the test is counted as a
 211 todo test, and the text after <code>TODO</code> is the explanation.</p>
 212 <pre>
 213     not ok 13 # TODO bend space and time</pre>
 214 <p>Note that if the TODO has an explanation it must be separated from
 215 <code>TODO</code> by a space.</p>
 216 <p>These tests represent a feature to be implemented or a bug to be fixed
 217 and act as something of an executable ``things to do'' list.  They are
 218 <strong>not</strong> expected to succeed.  Should a todo test point begin succeeding,
 219 the harness should report it as a bonus.  This indicates that whatever
 220 you were supposed to do has been done and you should promote this to a
 221 normal test point.</p>
 222 <p>
 223 </p>
 224 <h2><a name="skipping_tests">Skipping tests</a></h2>
 225 <p>If the directive starts with <code># SKIP</code>, the test is counted as having
 226 been skipped.  If the whole test file succeeds, the count of skipped
 227 tests is included in the generated output.  The harness should report
 228 the text after <code> # SKIP\S*\s+</code> as a reason for skipping.</p>
 229 <pre>
 230     ok 23 # skip Insufficient flogiston pressure.</pre>
 231 <p>Similarly, one can include an explanation in a plan line,
 232 emitted if the test file is skipped completely:</p>
 233 <pre>
 234     1..0 # Skipped: WWW::Mechanize not installed</pre>
 235 <p>
 236 </p>
 237 <hr />
 238 <h1><a name="other_lines">OTHER LINES</a></h1>
 239 <p>
 240 </p>
 241 <h2><a name="bail_out_">Bail out!</a></h2>
 242 <p>As an emergency measure a test script can decide that further tests
 243 are useless (e.g. missing dependencies) and testing should stop
 244 immediately. In that case the test script prints the magic words</p>
 245 <pre>
 246     Bail out!</pre>
 247 <p>to standard output. Any message after these words must be displayed
 248 by the interpreter as the reason why testing must be stopped, as
 249 in</p>
 250 <pre>
 251     Bail out! MySQL is not running.</pre>
 252 <p>
 253 </p>
 254 <h2><a name="diagnostics">Diagnostics</a></h2>
 255 <p>Additional information may be put into the testing output on separate
 256 lines.  Diagnostic lines should begin with a <code>#</code>, which the harness must
 257 ignore, at least as far as analyzing the test results.  The harness is
 258 free, however, to display the diagnostics.  Typically diagnostics are
 259 used to provide information about the environment in which test file is
 260 running, or to delineate a group of tests.</p>
 261 <pre>
 262     ...
 263     ok 18 - Closed database connection
 264     # End of database section.
 265     # This starts the network part of the test.
 266     # Daemon started on port 2112
 267     ok 19 - Opened socket
 268     ...
 269     ok 47 - Closed socket
 270     # End of network tests</pre>
 271 <p>
 272 </p>
 273 <h2><a name="anything_else">Anything else</a></h2>
 274 <p>Any output line that is not a plan, a test line or a diagnostic is
 275 incorrect.  How a harness handles the incorrect line is undefined.
 276 Test::Harness silently ignores incorrect lines, but will become more
 277 stringent in the future.</p>
 278 <p>
 279 </p>
 280 <hr />
 281 <h1><a name="examples">EXAMPLES</a></h1>
 282 <p>All names, places, and events depicted in any example are wholly
 283 fictitious and bear no resemblance to, connection with, or relation to any
 284 real entity. Any such similarity is purely coincidental, unintentional,
 285 and unintended.</p>
 286 <p>
 287 </p>
 288 <h2><a name="common_with_explanation">Common with explanation</a></h2>
 289 <p>The following TAP listing declares that six tests follow as well as
 290 provides handy feedback as to what the test is about to do. All six
 291 tests pass.</p>
 292 <pre>
 293     1..6
 294     #
 295     # Create a new Board and Tile, then place
 296     # the Tile onto the board.
 297     #
 298     ok 1 - The object isa Board
 299     ok 2 - Board size is zero
 300     ok 3 - The object isa Tile
 301     ok 4 - Get possible places to put the Tile
 302     ok 5 - Placing the tile produces no error
 303     ok 6 - Board size is 1</pre>
 304 <p>
 305 </p>
 306 <h2><a name="unknown_amount_and_failures">Unknown amount and failures</a></h2>
 307 <p>This hypothetical test program ensures that a handful of servers are
 308 online and network-accessible. Because it retrieves the hypothetical
 309 servers from a database, it doesn't know exactly how many servers it
 310 will need to ping. Thus, the test count is declared at the bottom after
 311 all the test points have run. Also, two of the tests fail.</p>
 312 <pre>
 313     ok 1 - retrieving servers from the database
 314     # need to ping 6 servers
 315     ok 2 - pinged diamond
 316     ok 3 - pinged ruby
 317     not ok 4 - pinged saphire
 318     ok 5 - pinged onyx
 319     not ok 6 - pinged quartz
 320     ok 7 - pinged gold
 321     1..7</pre>
 322 <p>
 323 </p>
 324 <h2><a name="giving_up">Giving up</a></h2>
 325 <p>This listing reports that a pile of tests are going to be run. However,
 326 the first test fails, reportedly because a connection to the database
 327 could not be established. The program decided that continuing was
 328 pointless and exited.</p>
 329 <pre>
 330     1..573
 331     not ok 1 - database handle
 332     Bail out! Couldn't connect to database.</pre>
 333 <p>
 334 </p>
 335 <h2><a name="skipping_a_few">Skipping a few</a></h2>
 336 <p>The following listing plans on running 5 tests. However, our program
 337 decided to not run tests 2 thru 5 at all. To properly report this,
 338 the tests are marked as being skipped.</p>
 339 <pre>
 340     1..5
 341     ok 1 - approved operating system
 342     # $^0 is solaris
 343     ok 2 - # SKIP no /sys directory
 344     ok 3 - # SKIP no /sys directory
 345     ok 4 - # SKIP no /sys directory
 346     ok 5 - # SKIP no /sys directory</pre>
 347 <p>
 348 </p>
 349 <h2><a name="skipping_everything">Skipping everything</a></h2>
 350 <p>This listing shows that the entire listing is a skip. No tests were run.</p>
 351 <pre>
 352     1..0 # skip because English-to-French translator isn't installed</pre>
 353 <p>
 354 </p>
 355 <h2><a name="got_spare_tuits">Got spare tuits?</a></h2>
 356 <p>The following example reports that four tests are run and the last two
 357 tests failed. However, because the failing tests are marked as things
 358 to do later, they are considered successes. Thus, a harness should report
 359 this entire listing as a success.</p>
 360 <pre>
 361     1..4
 362     ok 1 - Creating test program
 363     ok 2 - Test program runs, no error
 364     not ok 3 - infinite loop # TODO halting problem unsolved
 365     not ok 4 - infinite loop 2 # TODO halting problem unsolved</pre>
 366 <p>
 367 </p>
 368 <h2><a name="creative_liberties">Creative liberties</a></h2>
 369 <p>This listing shows an alternate output where the test numbers aren't
 370 provided. The test also reports the state of a ficticious board game in
 371 diagnostic form. Finally, the test count is reported at the end.</p>
 372 <pre>
 373     ok - created Board
 374     ok
 375     ok
 376     ok
 377     ok
 378     ok
 379     ok
 380     ok
 381     # +------+------+------+------+
 382     # |      |16G   |      |05C   |
 383     # |      |G N C |      |C C G |
 384     # |      |  G   |      |  C  +|
 385     # +------+------+------+------+
 386     # |10C   |01G   |      |03C   |
 387     # |R N G |G A G |      |C C C |
 388     # |  R   |  G   |      |  C  +|
 389     # +------+------+------+------+
 390     # |      |01G   |17C   |00C   |
 391     # |      |G A G |G N R |R N R |
 392     # |      |  G   |  R   |  G   |
 393     # +------+------+------+------+
 394     ok - board has 7 tiles + starter tile
 395     1..9</pre>
 396 <p>
 397 </p>
 398 <hr />
 399 <h1><a name="authors">AUTHORS</a></h1>
 400 <p>Andy Lester, based on the original Test::Harness documentation by Michael Schwern.</p>
 401 <p>
 402 </p>
 403 <hr />
 404 <h1><a name="acknowledgements">ACKNOWLEDGEMENTS</a></h1>
 405 <p>Thanks to
 406 Pete Krawczyk,
 407 Paul Johnson,
 408 Ian Langworth
 409 and Nik Clayton
 410 for help and contributions on this document.</p>
 411 <p>The basis for the TAP format was created by Larry Wall in the
 412 original test script for Perl 1.  Tim Bunce and Andreas Koenig
 413 developed it further with their modifications to Test::Harness.</p>
 414 <p>
 415 </p>
 416 <hr />
 417 <h1><a name="copyright">COPYRIGHT</a></h1>
 418 <p>Copyright 2003-2005 by
 419 Michael G Schwern <code>&lt;schwern@pobox.com&gt;</code>,
 420 Andy Lester <code>&lt;andy@petdance.com&gt;</code>.</p>
 421 <p>This program is free software; you can redistribute it and/or
 422 modify it under the same terms as Perl itself.</p>
 423 <p>See <a href="http://www.perl.com/perl/misc/Artistic.html">http://www.perl.com/perl/misc/Artistic.html</a>.</p>
 424 <table border="0" width="100%" cellspacing="0" cellpadding="3">
 425 <tr><td class="block" style="background-color: #cccccc" valign="middle">
 426 <big><strong><span class="block">&nbsp;Test::Harness::TAP - Documentation for the TAP format</span></strong></big>
 427 </td></tr>
 428 </table>
 429
 430 </body>
 431
 432 </html>