mingw/html/lib/Test/Harness.html

   1 <?xml version="1.0" ?>
   2 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
   3 <html xmlns="http://www.w3.org/1999/xhtml">
   4 <head>
   5 <title>Test::Harness - Run Perl standard test scripts with statistics</title>
   6 <meta http-equiv="content-type" content="text/html; charset=utf-8" />
   7 <link rev="made" href="mailto:" />
   8 </head>
   9
  10 <body style="background-color: white">
  11 <table border="0" width="100%" cellspacing="0" cellpadding="3">
  12 <tr><td class="block" style="background-color: #cccccc" valign="middle">
  13 <big><strong><span class="block">&nbsp;Test::Harness - Run Perl standard test scripts with statistics</span></strong></big>
  14 </td></tr>
  15 </table>
  16
  17 <p><a name="__index__"></a></p>
  18 <!-- INDEX BEGIN -->
  19
  20 <ul>
  21
  22         <li><a href="#name">NAME</a></li>
  23         <li><a href="#version">VERSION</a></li>
  24         <li><a href="#synopsis">SYNOPSIS</a></li>
  25         <li><a href="#description">DESCRIPTION</a></li>
  26         <ul>
  27
  28                 <li><a href="#taint_mode">Taint mode</a></li>
  29                 <li><a href="#configuration_variables_">Configuration variables.</a></li>
  30                 <li><a href="#failure">Failure</a></li>
  31                 <li><a href="#functions">Functions</a></li>
  32         </ul>
  33
  34         <li><a href="#export">EXPORT</a></li>
  35         <li><a href="#diagnostics">DIAGNOSTICS</a></li>
  36         <li><a href="#environment_variables_that_test__harness_sets">ENVIRONMENT VARIABLES THAT TEST::HARNESS SETS</a></li>
  37         <li><a href="#environment_variables_that_affect_test__harness">ENVIRONMENT VARIABLES THAT AFFECT TEST::HARNESS</a></li>
  38         <li><a href="#example">EXAMPLE</a></li>
  39         <li><a href="#see_also">SEE ALSO</a></li>
  40         <li><a href="#todo">TODO</a></li>
  41         <li><a href="#bugs">BUGS</a></li>
  42         <li><a href="#authors">AUTHORS</a></li>
  43         <li><a href="#copyright">COPYRIGHT</a></li>
  44 </ul>
  45 <!-- INDEX END -->
  46
  47 <hr />
  48 <p>
  49 </p>
  50 <h1><a name="name">NAME</a></h1>
  51 <p>Test::Harness - Run Perl standard test scripts with statistics</p>
  52 <p>
  53 </p>
  54 <hr />
  55 <h1><a name="version">VERSION</a></h1>
  56 <p>Version 2.56</p>
  57 <p>
  58 </p>
  59 <hr />
  60 <h1><a name="synopsis">SYNOPSIS</a></h1>
  61 <pre>
  62   use Test::Harness;</pre>
  63 <pre>
  64   runtests(@test_files);</pre>
  65 <p>
  66 </p>
  67 <hr />
  68 <h1><a name="description">DESCRIPTION</a></h1>
  69 <p><strong>STOP!</strong> If all you want to do is write a test script, consider
  70 using Test::Simple.  Test::Harness is the module that reads the
  71 output from Test::Simple, Test::More and other modules based on
  72 Test::Builder.  You don't need to know about Test::Harness to use
  73 those modules.</p>
  74 <p>Test::Harness runs tests and expects output from the test in a
  75 certain format.  That format is called TAP, the Test Anything
  76 Protocol.  It is defined in <a href="file://C|\msysgit\mingw\html/lib/Test/Harness/TAP.html">the Test::Harness::TAP manpage</a>.</p>
  77 <p><a href="#item_runtests"><code>Test::Harness::runtests(@tests)</code></a> runs all the testscripts named
  78 as arguments and checks standard output for the expected strings
  79 in TAP format.</p>
  80 <p>The <em>prove</em> utility is a thin wrapper around Test::Harness.</p>
  81 <p>
  82 </p>
  83 <h2><a name="taint_mode">Taint mode</a></h2>
  84 <p>Test::Harness will honor the <code>-T</code> or <code>-t</code> in the #! line on your
  85 test files.  So if you begin a test with:</p>
  86 <pre>
  87     #!perl -T</pre>
  88 <p>the test will be run with taint mode on.</p>
  89 <p>
  90 </p>
  91 <h2><a name="configuration_variables_">Configuration variables.</a></h2>
  92 <p>These variables can be used to configure the behavior of
  93 Test::Harness.  They are exported on request.</p>
  94 <dl>
  95 <dt><strong><a name="item__test__harness__verbose"><code>$Test::Harness::Verbose</code></a></strong>
  96
  97 <dd>
  98 <p>The package variable <a href="#item__test__harness__verbose"><code>$Test::Harness::Verbose</code></a> is exportable and can be
  99 used to let <a href="#item_runtests"><code>runtests()</code></a> display the standard output of the script
 100 without altering the behavior otherwise.  The <em>prove</em> utility's <code>-v</code>
 101 flag will set this.</p>
 102 </dd>
 103 </li>
 104 <dt><strong><a name="item__test__harness__switches"><code>$Test::Harness::switches</code></a></strong>
 105
 106 <dd>
 107 <p>The package variable <a href="#item__test__harness__switches"><code>$Test::Harness::switches</code></a> is exportable and can be
 108 used to set perl command line options used for running the test
 109 script(s). The default value is <code>-w</code>. It overrides <code>HARNESS_SWITCHES</code>.</p>
 110 </dd>
 111 </li>
 112 <dt><strong><a name="item__test__harness__timer"><code>$Test::Harness::Timer</code></a></strong>
 113
 114 <dd>
 115 <p>If set to true, and <code>Time::HiRes</code> is available, print elapsed seconds
 116 after each test file.</p>
 117 </dd>
 118 </li>
 119 </dl>
 120 <p>
 121 </p>
 122 <h2><a name="failure">Failure</a></h2>
 123 <p>When tests fail, analyze the summary report:</p>
 124 <pre>
 125   t/base..............ok
 126   t/nonumbers.........ok
 127   t/ok................ok
 128   t/test-harness......ok
 129   t/waterloo..........dubious
 130           Test returned status 3 (wstat 768, 0x300)
 131   DIED. FAILED tests 1, 3, 5, 7, 9, 11, 13, 15, 17, 19
 132           Failed 10/20 tests, 50.00% okay
 133   Failed Test  Stat Wstat Total Fail  Failed  List of Failed
 134   -----------------------------------------------------------------------
 135   t/waterloo.t    3   768    20   10  50.00%  1 3 5 7 9 11 13 15 17 19
 136   Failed 1/5 test scripts, 80.00% okay. 10/44 subtests failed, 77.27% okay.</pre>
 137 <p>Everything passed but <em>t/waterloo.t</em>.  It failed 10 of 20 tests and
 138 exited with non-zero status indicating something dubious happened.</p>
 139 <p>The columns in the summary report mean:</p>
 140 <dl>
 141 <dt><strong><a name="item_failed_test"><strong>Failed Test</strong></a></strong>
 142
 143 <dd>
 144 <p>The test file which failed.</p>
 145 </dd>
 146 </li>
 147 <dt><strong><a name="item_stat"><strong>Stat</strong></a></strong>
 148
 149 <dd>
 150 <p>If the test exited with non-zero, this is its exit status.</p>
 151 </dd>
 152 </li>
 153 <dt><strong><a name="item_wstat"><strong>Wstat</strong></a></strong>
 154
 155 <dd>
 156 <p>The wait status of the test.</p>
 157 </dd>
 158 </li>
 159 <dt><strong><a name="item_total"><strong>Total</strong></a></strong>
 160
 161 <dd>
 162 <p>Total number of tests expected to run.</p>
 163 </dd>
 164 </li>
 165 <dt><strong><a name="item_fail"><strong>Fail</strong></a></strong>
 166
 167 <dd>
 168 <p>Number which failed, either from ``not ok'' or because they never ran.</p>
 169 </dd>
 170 </li>
 171 <dt><strong><a name="item_failed"><strong>Failed</strong></a></strong>
 172
 173 <dd>
 174 <p>Percentage of the total tests which failed.</p>
 175 </dd>
 176 </li>
 177 <dt><strong><a name="item_list_of_failed"><strong>List of Failed</strong></a></strong>
 178
 179 <dd>
 180 <p>A list of the tests which failed.  Successive failures may be
 181 abbreviated (ie. 15-20 to indicate that tests 15, 16, 17, 18, 19 and
 182 20 failed).</p>
 183 </dd>
 184 </li>
 185 </dl>
 186 <p>
 187 </p>
 188 <h2><a name="functions">Functions</a></h2>
 189 <p>Test::Harness currently only has one function, here it is.</p>
 190 <dl>
 191 <dt><strong><a name="item_runtests"><strong>runtests</strong></a></strong>
 192
 193 <dd>
 194 <pre>
 195   my $allok = runtests(@test_files);</pre>
 196 </dd>
 197 <dd>
 198 <p>This runs all the given <em>@test_files</em> and divines whether they passed
 199 or failed based on their output to STDOUT (details above).  It prints
 200 out each individual test which failed along with a summary report and
 201 a how long it all took.</p>
 202 </dd>
 203 <dd>
 204 <p>It returns true if everything was ok.  Otherwise it will <code>die()</code> with
 205 one of the messages in the DIAGNOSTICS section.</p>
 206 </dd>
 207 </dl>
 208 <p>
 209 </p>
 210 <hr />
 211 <h1><a name="export">EXPORT</a></h1>
 212 <p><code>&amp;runtests</code> is exported by Test::Harness by default.</p>
 213 <p><code>$verbose</code>, <code>$switches</code> and <code>$debug</code> are exported upon request.</p>
 214 <p>
 215 </p>
 216 <hr />
 217 <h1><a name="diagnostics">DIAGNOSTICS</a></h1>
 218 <dl>
 219 <dt><strong><a name="item_all_tests_successful_2e_5cnfiles_3d_25d_2c_tests_3"><code>All tests successful.\nFiles=%d,  Tests=%d, %s</code></a></strong>
 220
 221 <dd>
 222 <p>If all tests are successful some statistics about the performance are
 223 printed.</p>
 224 </dd>
 225 </li>
 226 <dt><strong><a name="item_failed_tests__25s_5cn_5ctfailed__25d_2f_25d_tests_"><code>FAILED tests %s\n\tFailed %d/%d tests, %.2f%% okay.</code></a></strong>
 227
 228 <dd>
 229 <p>For any single script that has failing subtests statistics like the
 230 above are printed.</p>
 231 </dd>
 232 </li>
 233 <dt><strong><a name="item_d"><code>Test returned status %d (wstat %d)</code></a></strong>
 234
 235 <dd>
 236 <p>Scripts that return a non-zero exit status, both <a href="file://C|\msysgit\mingw\html/pod/perlvar.html#item__"><code>$? &gt;&gt; 8</code></a>
 237 and <a href="file://C|\msysgit\mingw\html/pod/perlvar.html#item__"><code>$?</code></a> are printed in a message similar to the above.</p>
 238 </dd>
 239 </li>
 240 <dt><strong><a name="item_failed_1_test_2c__25_2e2f_25_25_okay_2e__25s"><code>Failed 1 test, %.2f%% okay. %s</code></a></strong>
 241
 242 <dt><strong><a name="item_failed__25d_2f_25d_tests_2c__25_2e2f_25_25_okay_2e"><code>Failed %d/%d tests, %.2f%% okay. %s</code></a></strong>
 243
 244 <dd>
 245 <p>If not all tests were successful, the script dies with one of the
 246 above messages.</p>
 247 </dd>
 248 </li>
 249 <dt><strong><a name="item_failed_2d_2dfurther_testing_stopped_3a__25s"><code>FAILED--Further testing stopped: %s</code></a></strong>
 250
 251 <dd>
 252 <p>If a single subtest decides that further testing will not make sense,
 253 the script dies with this message.</p>
 254 </dd>
 255 </li>
 256 </dl>
 257 <p>
 258 </p>
 259 <hr />
 260 <h1><a name="environment_variables_that_test__harness_sets">ENVIRONMENT VARIABLES THAT TEST::HARNESS SETS</a></h1>
 261 <p>Test::Harness sets these before executing the individual tests.</p>
 262 <dl>
 263 <dt><strong><a name="item_harness_active"><code>HARNESS_ACTIVE</code></a></strong>
 264
 265 <dd>
 266 <p>This is set to a true value.  It allows the tests to determine if they
 267 are being executed through the harness or by any other means.</p>
 268 </dd>
 269 </li>
 270 <dt><strong><a name="item_harness_version"><code>HARNESS_VERSION</code></a></strong>
 271
 272 <dd>
 273 <p>This is the version of Test::Harness.</p>
 274 </dd>
 275 </li>
 276 </dl>
 277 <p>
 278 </p>
 279 <hr />
 280 <h1><a name="environment_variables_that_affect_test__harness">ENVIRONMENT VARIABLES THAT AFFECT TEST::HARNESS</a></h1>
 281 <dl>
 282 <dt><strong><a name="item_harness_columns"><code>HARNESS_COLUMNS</code></a></strong>
 283
 284 <dd>
 285 <p>This value will be used for the width of the terminal. If it is not
 286 set then it will default to <code>COLUMNS</code>. If this is not set, it will
 287 default to 80. Note that users of Bourne-sh based shells will need to
 288 <code>export COLUMNS</code> for this module to use that variable.</p>
 289 </dd>
 290 </li>
 291 <dt><strong><a name="item_harness_compile_test"><code>HARNESS_COMPILE_TEST</code></a></strong>
 292
 293 <dd>
 294 <p>When true it will make harness attempt to compile the test using
 295 <code>perlcc</code> before running it.</p>
 296 </dd>
 297 <dd>
 298 <p><strong>NOTE</strong> This currently only works when sitting in the perl source
 299 directory!</p>
 300 </dd>
 301 </li>
 302 <dt><strong><a name="item_harness_debug"><code>HARNESS_DEBUG</code></a></strong>
 303
 304 <dd>
 305 <p>If true, Test::Harness will print debugging information about itself as
 306 it runs the tests.  This is different from <a href="#item_harness_verbose"><code>HARNESS_VERBOSE</code></a>, which prints
 307 the output from the test being run.  Setting <code>$Test::Harness::Debug</code> will
 308 override this, or you can use the <code>-d</code> switch in the <em>prove</em> utility.</p>
 309 </dd>
 310 </li>
 311 <dt><strong><a name="item_harness_fileleak_in_dir"><code>HARNESS_FILELEAK_IN_DIR</code></a></strong>
 312
 313 <dd>
 314 <p>When set to the name of a directory, harness will check after each
 315 test whether new files appeared in that directory, and report them as</p>
 316 </dd>
 317 <dd>
 318 <pre>
 319   LEAKED FILES: scr.tmp 0 my.db</pre>
 320 </dd>
 321 <dd>
 322 <p>If relative, directory name is with respect to the current directory at
 323 the moment <a href="#item_runtests"><code>runtests()</code></a> was called.  Putting absolute path into
 324 <a href="#item_harness_fileleak_in_dir"><code>HARNESS_FILELEAK_IN_DIR</code></a> may give more predictable results.</p>
 325 </dd>
 326 </li>
 327 <dt><strong><a name="item_harness_ignore_exitcode"><code>HARNESS_IGNORE_EXITCODE</code></a></strong>
 328
 329 <dd>
 330 <p>Makes harness ignore the exit status of child processes when defined.</p>
 331 </dd>
 332 </li>
 333 <dt><strong><a name="item_harness_notty"><code>HARNESS_NOTTY</code></a></strong>
 334
 335 <dd>
 336 <p>When set to a true value, forces it to behave as though STDOUT were
 337 not a console.  You may need to set this if you don't want harness to
 338 output more frequent progress messages using carriage returns.  Some
 339 consoles may not handle carriage returns properly (which results in a
 340 somewhat messy output).</p>
 341 </dd>
 342 </li>
 343 <dt><strong><a name="item_harness_perl"><code>HARNESS_PERL</code></a></strong>
 344
 345 <dd>
 346 <p>Usually your tests will be run by <a href="file://C|\msysgit\mingw\html/pod/perlvar.html#item___x"><code>$^X</code></a>, the currently-executing Perl.
 347 However, you may want to have it run by a different executable, such as
 348 a threading perl, or a different version.</p>
 349 </dd>
 350 <dd>
 351 <p>If you're using the <em>prove</em> utility, you can use the <code>--perl</code> switch.</p>
 352 </dd>
 353 </li>
 354 <dt><strong><a name="item_harness_perl_switches"><code>HARNESS_PERL_SWITCHES</code></a></strong>
 355
 356 <dd>
 357 <p>Its value will be prepended to the switches used to invoke perl on
 358 each test.  For example, setting <a href="#item_harness_perl_switches"><code>HARNESS_PERL_SWITCHES</code></a> to <code>-W</code> will
 359 run all tests with all warnings enabled.</p>
 360 </dd>
 361 </li>
 362 <dt><strong><a name="item_harness_verbose"><code>HARNESS_VERBOSE</code></a></strong>
 363
 364 <dd>
 365 <p>If true, Test::Harness will output the verbose results of running
 366 its tests.  Setting <code>$Test::Harness::verbose</code> will override this,
 367 or you can use the <code>-v</code> switch in the <em>prove</em> utility.</p>
 368 </dd>
 369 </li>
 370 </dl>
 371 <p>
 372 </p>
 373 <hr />
 374 <h1><a name="example">EXAMPLE</a></h1>
 375 <p>Here's how Test::Harness tests itself</p>
 376 <pre>
 377   $ cd ~/src/devel/Test-Harness
 378   $ perl -Mblib -e 'use Test::Harness qw(&amp;runtests $verbose);
 379     $verbose=0; runtests @ARGV;' t/*.t
 380   Using /home/schwern/src/devel/Test-Harness/blib
 381   t/base..............ok
 382   t/nonumbers.........ok
 383   t/ok................ok
 384   t/test-harness......ok
 385   All tests successful.
 386   Files=4, Tests=24, 2 wallclock secs ( 0.61 cusr + 0.41 csys = 1.02 CPU)</pre>
 387 <p>
 388 </p>
 389 <hr />
 390 <h1><a name="see_also">SEE ALSO</a></h1>
 391 <p>The included <em>prove</em> utility for running test scripts from the command line,
 392 <a href="file://C|\msysgit\mingw\html/lib/Test.html">the Test manpage</a> and <a href="file://C|\msysgit\mingw\html/lib/Test/Simple.html">the Test::Simple manpage</a> for writing test scripts, <a href="file://C|\msysgit\mingw\html/lib/Benchmark.html">the Benchmark manpage</a> for
 393 the underlying timing routines, and <a href="file://C|\msysgit\mingw\html/Devel/Cover.html">the Devel::Cover manpage</a> for test coverage
 394 analysis.</p>
 395 <p>
 396 </p>
 397 <hr />
 398 <h1><a name="todo">TODO</a></h1>
 399 <p>Provide a way of running tests quietly (ie. no printing) for automated
 400 validation of tests.  This will probably take the form of a version
 401 of <a href="#item_runtests"><code>runtests()</code></a> which rather than printing its output returns raw data
 402 on the state of the tests.  (Partially done in Test::Harness::Straps)</p>
 403 <p>Document the format.</p>
 404 <p>Fix HARNESS_COMPILE_TEST without breaking its core usage.</p>
 405 <p>Figure a way to report test names in the failure summary.</p>
 406 <p>Rework the test summary so long test names are not truncated as badly.
 407 (Partially done with new skip test styles)</p>
 408 <p>Add option for coverage analysis.</p>
 409 <p>Trap STDERR.</p>
 410 <p>Implement Straps <code>total_results()</code></p>
 411 <p>Remember exit code</p>
 412 <p>Completely redo the print summary code.</p>
 413 <p>Implement Straps callbacks.  (experimentally implemented)</p>
 414 <p>Straps-&gt;<code>analyze_file()</code> not taint clean, don't know if it can be</p>
 415 <p>Fix that damned VMS nit.</p>
 416 <p>HARNESS_TODOFAIL to display TODO failures</p>
 417 <p>Add a test for verbose.</p>
 418 <p>Change internal list of test results to a hash.</p>
 419 <p>Fix stats display when there's an overrun.</p>
 420 <p>Fix so perls with spaces in the filename work.</p>
 421 <p>Keeping whittling away at <a href="#item__run_all_tests"><code>_run_all_tests()</code></a></p>
 422 <p>Clean up how the summary is printed.  Get rid of those damned formats.</p>
 423 <p>
 424 </p>
 425 <hr />
 426 <h1><a name="bugs">BUGS</a></h1>
 427 <p>HARNESS_COMPILE_TEST currently assumes it's run from the Perl source
 428 directory.</p>
 429 <p>Please use the CPAN bug ticketing system at <a href="http://rt.cpan.org/">http://rt.cpan.org/</a>.
 430 You can also mail bugs, fixes and enhancements to
 431 <code>&lt;bug-test-harness</code> at <code>rt.cpan.org&gt;</code>.</p>
 432 <p>
 433 </p>
 434 <hr />
 435 <h1><a name="authors">AUTHORS</a></h1>
 436 <p>Either Tim Bunce or Andreas Koenig, we don't know. What we know for
 437 sure is, that it was inspired by Larry Wall's TEST script that came
 438 with perl distributions for ages. Numerous anonymous contributors
 439 exist.  Andreas Koenig held the torch for many years, and then
 440 Michael G Schwern.</p>
 441 <p>Current maintainer is Andy Lester <code>&lt;andy at petdance.com&gt;</code>.</p>
 442 <p>
 443 </p>
 444 <hr />
 445 <h1><a name="copyright">COPYRIGHT</a></h1>
 446 <p>Copyright 2002-2005
 447 by Michael G Schwern <code>&lt;schwern at pobox.com&gt;</code>,
 448 Andy Lester <code>&lt;andy at petdance.com&gt;</code>.</p>
 449 <p>This program is free software; you can redistribute it and/or
 450 modify it under the same terms as Perl itself.</p>
 451 <p>See <a href="http://www.perl.com/perl/misc/Artistic.html">http://www.perl.com/perl/misc/Artistic.html</a>.</p>
 452 <table border="0" width="100%" cellspacing="0" cellpadding="3">
 453 <tr><td class="block" style="background-color: #cccccc" valign="middle">
 454 <big><strong><span class="block">&nbsp;Test::Harness - Run Perl standard test scripts with statistics</span></strong></big>
 455 </td></tr>
 456 </table>
 457
 458 </body>
 459
 460 </html>