doc/overview/outputfiles.html

   1 <HTML
   2 ><HEAD
   3 ><TITLE
   4 >The files DejaGnu produces.</TITLE
   5 ><META
   6 NAME="GENERATOR"
   7 CONTENT="Modular DocBook HTML Stylesheet Version 1.44"><LINK
   8 REL="HOME"
   9 TITLE="DejaGnu"
  10 HREF="book1.html"><LINK
  11 REL="UP"
  12 TITLE="Running Tests"
  13 HREF="runningtests.html"><LINK
  14 REL="PREVIOUS"
  15 TITLE="Runtest"
  16 HREF="runtest.html"><LINK
  17 REL="NEXT"
  18 TITLE="Customizing DejaGnu"
  19 HREF="customizing.html"></HEAD
  20 ><BODY
  21 ><DIV
  22 CLASS="NAVHEADER"
  23 ><TABLE
  24 WIDTH="100%"
  25 BORDER="0"
  26 CELLPADDING="0"
  27 CELLSPACING="0"
  28 ><TR
  29 ><TH
  30 COLSPAN="3"
  31 ALIGN="center"
  32 >DejaGnu: The GNU Testing Framework</TH
  33 ></TR
  34 ><TR
  35 ><TD
  36 WIDTH="10%"
  37 ALIGN="left"
  38 VALIGN="bottom"
  39 ><A
  40 HREF="runtest.html"
  41 >Prev</A
  42 ></TD
  43 ><TD
  44 WIDTH="80%"
  45 ALIGN="center"
  46 VALIGN="bottom"
  47 >Chapter 2. Running Tests</TD
  48 ><TD
  49 WIDTH="10%"
  50 ALIGN="right"
  51 VALIGN="bottom"
  52 ><A
  53 HREF="customizing.html"
  54 >Next</A
  55 ></TD
  56 ></TR
  57 ></TABLE
  58 ><HR
  59 ALIGN="LEFT"
  60 WIDTH="100%"></DIV
  61 ><DIV
  62 CLASS="SECT1"
  63 ><H1
  64 CLASS="SECT1"
  65 ><A
  66 NAME="OUTPUTFILES"
  67 >The files DejaGnu produces.</A
  68 ></H1
  69 ><P
  70 >DejaGnu always writes two kinds of output files: summary
  71     logs and detailed logs.  The contents of both of these are
  72     determined by your tests.</P
  73 ><P
  74 >For troubleshooting, a third kind of output file is useful:
  75     use <TT
  76 CLASS="OPTION"
  77 >--debug</TT
  78 > to request an output file showing
  79     details of what <SPAN
  80 CLASS="PRODUCTNAME"
  81 >Expect</SPAN
  82 > is doing
  83     internally.</P
  84 ><DIV
  85 CLASS="SECT2"
  86 ><H2
  87 CLASS="SECT2"
  88 ><A
  89 NAME="SUM"
  90 >Summary File</A
  91 ></H2
  92 ><P
  93 >DejaGnu always produces a summary output file
  94       <TT
  95 CLASS="FILENAME"
  96 >tool.sum</TT
  97 >. This summary shows the names of
  98       all test files run; for each test file, one line of output from
  99       each <B
 100 CLASS="COMMAND"
 101 >pass</B
 102 > command (showing status
 103       <I
 104 CLASS="EMPHASIS"
 105 >PASS</I
 106 > or <I
 107 CLASS="EMPHASIS"
 108 >XPASS</I
 109 >) or
 110       <B
 111 CLASS="COMMAND"
 112 >fail</B
 113 > command (status
 114       <I
 115 CLASS="EMPHASIS"
 116 >FAIL</I
 117 > or <I
 118 CLASS="EMPHASIS"
 119 >XFAIL</I
 120 >);
 121       trailing summary statistics that count passing and failing tests
 122       (expected and unexpected); and the full pathname and version
 123       number of the tool tested.  (All possible outcomes, and all
 124       errors, are always reflected in the summary output file,
 125       regardless of whether or not you specify
 126       <TT
 127 CLASS="OPTION"
 128 >--all</TT
 129 >.)</P
 130 ><P
 131 >If any of your tests use the procedures
 132       <B
 133 CLASS="COMMAND"
 134 >unresolved</B
 135 >, <B
 136 CLASS="COMMAND"
 137 >unsupported</B
 138 >,
 139       or <B
 140 CLASS="COMMAND"
 141 >runtested</B
 142 >, the summary output also
 143       tabulates the corresponding outcomes.</P
 144 ><P
 145 >For example, after <B
 146 CLASS="COMMAND"
 147 >runtest --tool
 148       binutils</B
 149 >, look for a summary log in
 150       <TT
 151 CLASS="FILENAME"
 152 >binutils.sum</TT
 153 >. Normally, DejaGnu writes this
 154       file in your current working directory; use the
 155       <TT
 156 CLASS="OPTION"
 157 >--outdir</TT
 158 > option to select a different
 159       directory.</P
 160 ><DIV
 161 CLASS="EXAMPLE"
 162 ><P
 163 ><B
 164 >Example 2-1. Here is a short sample summary log</B
 165 ></P
 166 ><TABLE
 167 BORDER="0"
 168 BGCOLOR="#E0E0E0"
 169 WIDTH="100%"
 170 ><TR
 171 ><TD
 172 ><PRE
 173 CLASS="SCREEN"
 174 >       Test Run By rob on Mon May 25 21:40:57 PDT 1992
 175                  === gdb tests ===
 176         Running ./gdb.t00/echo.exp ...
 177         PASS:   Echo test
 178         Running ./gdb.all/help.exp ...
 179         PASS:   help add-symbol-file
 180         PASS:   help aliases
 181         PASS:   help breakpoint "bre" abbreviation
 182         FAIL:   help run "r" abbreviation
 183         Running ./gdb.t10/crossload.exp ...
 184         PASS:   m68k-elf (elf-big) explicit format; loaded
 185         XFAIL:  mips-ecoff (ecoff-bigmips) "ptype v_signed_char" signed C types
 186                 === gdb Summary ===
 187         # of expected passes 5
 188         # of expected failures 1
 189         # of unexpected failures 1
 190         /usr/latest/bin/gdb version 4.6.5 -q
 191       </PRE
 192 ></TD
 193 ></TR
 194 ></TABLE
 195 ></DIV
 196 ></DIV
 197 ><DIV
 198 CLASS="SECT2"
 199 ><H2
 200 CLASS="SECT2"
 201 ><A
 202 NAME="LOG"
 203 >Log File</A
 204 ></H2
 205 ><P
 206 >DejaGnu also saves a detailed log file
 207       <TT
 208 CLASS="FILENAME"
 209 >tool.log</TT
 210 >, showing any output generated by
 211       tests as well as the summary output. For example, after
 212       <B
 213 CLASS="COMMAND"
 214 >runtest --tool binutils</B
 215 >, look for a detailed
 216       log in <TT
 217 CLASS="FILENAME"
 218 >binutils.log</TT
 219 >. Normally, DejaGnu
 220       writes this file in your current working directory; use the
 221       <TT
 222 CLASS="OPTION"
 223 >--outdir</TT
 224 > option to select a different
 225       directory.</P
 226 ><DIV
 227 CLASS="EXAMPLE"
 228 ><P
 229 ><B
 230 >Example 2-2. Here is a brief example showing a detailed log for
 231         <SPAN
 232 CLASS="PRODUCTNAME"
 233 >G++</SPAN
 234 > tests</B
 235 ></P
 236 ><TABLE
 237 BORDER="0"
 238 BGCOLOR="#E0E0E0"
 239 WIDTH="100%"
 240 ><TR
 241 ><TD
 242 ><PRE
 243 CLASS="SCREEN"
 244 >       Test Run By rob on Mon May 25 21:40:43 PDT 1992
 245
 246                 === g++ tests ===
 247
 248         --- Running ./g++.other/t01-1.exp ---
 249         PASS:   operate delete
 250
 251         --- Running ./g++.other/t01-2.exp ---
 252         FAIL:   i960 bug EOF
 253         p0000646.C: In function `int  warn_return_1 ()':
 254         p0000646.C:109: warning: control reaches end of non-void function
 255         p0000646.C: In function `int  warn_return_arg (int)':
 256         p0000646.C:117: warning: control reaches end of non-void function
 257         p0000646.C: In function `int  warn_return_sum (int, int)':
 258         p0000646.C:125: warning: control reaches end of non-void function
 259         p0000646.C: In function `struct foo warn_return_foo ()':
 260         p0000646.C:132: warning: control reaches end of non-void function
 261
 262         --- Running ./g++.other/t01-4.exp ---
 263         FAIL:   abort
 264         900403_04.C:8: zero width for bit-field `foo'
 265         --- Running ./g++.other/t01-3.exp ---
 266         FAIL:   segment violation
 267         900519_12.C:9: parse error before `;'
 268         900519_12.C:12: Segmentation violation
 269         /usr/latest/bin/gcc: Internal compiler error: program cc1plus got fatal signal
 270
 271                 === g++ Summary ===
 272
 273         # of expected passes 1
 274         # of expected failures 3
 275         /usr/latest/bin/g++ version cygnus-2.0.1
 276         </PRE
 277 ></TD
 278 ></TR
 279 ></TABLE
 280 ></DIV
 281 ></DIV
 282 ><DIV
 283 CLASS="SECT2"
 284 ><H2
 285 CLASS="SECT2"
 286 ><A
 287 NAME="DEBUGFILE"
 288 >Debug Log File</A
 289 ></H2
 290 ><P
 291 >With the <TT
 292 CLASS="OPTION"
 293 >--debug</TT
 294 > option, you can request
 295       a log file showing the output from
 296       <SPAN
 297 CLASS="PRODUCTNAME"
 298 >Expect</SPAN
 299 > itself, running in debugging
 300       mode. This file (<TT
 301 CLASS="FILENAME"
 302 >dbg.log</TT
 303 >, in the directory
 304       where you start <B
 305 CLASS="COMMAND"
 306 >runtest</B
 307 >) shows each pattern
 308       <SPAN
 309 CLASS="PRODUCTNAME"
 310 >Expect</SPAN
 311 > considers in analyzing test
 312       output.</P
 313 ><P
 314 >This file reflects each <B
 315 CLASS="COMMAND"
 316 >send</B
 317 > command,
 318       showing the string sent as input to the tool under test; and
 319       each <SPAN
 320 CLASS="PRODUCTNAME"
 321 >Expect</SPAN
 322 > command, showing each
 323       pattern it compares with the tool output.</P
 324 ><DIV
 325 CLASS="EXAMPLE"
 326 ><P
 327 ><B
 328 >Example 2-3. The log messages begin with a message of the form</B
 329 ></P
 330 ><TABLE
 331 BORDER="0"
 332 BGCOLOR="#E0E0E0"
 333 WIDTH="100%"
 334 ><TR
 335 ><TD
 336 ><PRE
 337 CLASS="SCREEN"
 338 >&#13;  expect: does {<SPAN
 339 CLASS="SYMBOL"
 340 >tool output</SPAN
 341 >} (spawn_id <SPAN
 342 CLASS="SYMBOL"
 343 >n</SPAN
 344 >)
 345          match pattern {<I
 346 CLASS="EMPHASIS"
 347 >expected pattern</I
 348 >}?
 349
 350         </PRE
 351 ></TD
 352 ></TR
 353 ></TABLE
 354 ></DIV
 355 ><P
 356 >For every unsuccessful match,
 357       <SPAN
 358 CLASS="PRODUCTNAME"
 359 >Expect</SPAN
 360 > issues a
 361       <I
 362 CLASS="EMPHASIS"
 363 >no</I
 364 > after this message; if other patterns
 365       are specified for the same <SPAN
 366 CLASS="PRODUCTNAME"
 367 >Expect</SPAN
 368 >
 369       command, they are reflected also, but without the first part of
 370       the message (<I
 371 CLASS="EMPHASIS"
 372 >expect... match pattern</I
 373 >).</P
 374 ><P
 375 >When <SPAN
 376 CLASS="PRODUCTNAME"
 377 >Expect</SPAN
 378 > finds a match, the
 379       log for the successful match ends with <I
 380 CLASS="EMPHASIS"
 381 >yes</I
 382 >,
 383       followed by a record of the <SPAN
 384 CLASS="PRODUCTNAME"
 385 >Expect</SPAN
 386 >
 387       variables set to describe a successful match.</P
 388 ><DIV
 389 CLASS="EXAMPLE"
 390 ><P
 391 ><B
 392 >Example 2-4. Here is an excerpt from the debugging log for a
 393         <SPAN
 394 CLASS="PRODUCTNAME"
 395 >GDB</SPAN
 396 > test:</B
 397 ></P
 398 ><TABLE
 399 BORDER="0"
 400 BGCOLOR="#E0E0E0"
 401 WIDTH="100%"
 402 ><TR
 403 ><TD
 404 ><PRE
 405 CLASS="SCREEN"
 406 >       send: sent {break gdbme.c:34\n} to spawn id 6
 407         expect: does {} (spawn_id 6) match pattern {Breakpoint.*at.* file
 408         gdbme.c, line 34.*\(gdb\) $}? no
 409         {.*\(gdb\) $}? no
 410         expect: does {} (spawn_id 0) match pattern {return} ? no
 411         {\(y or n\) }? no
 412         {buffer_full}? no
 413         {virtual}? no
 414         {memory}? no
 415         {exhausted}? no
 416         {Undefined}? no
 417         {command}? no
 418         break gdbme.c:34
 419         Breakpoint 8 at 0x23d8: file gdbme.c, line 34.
 420         (gdb) expect: does {break gdbme.c:34\r\nBreakpoint 8 at 0x23d8:
 421         file gdbme.c, line 34.\r\n(gdb) } (spawn_id 6) match pattern
 422         {Breakpoint.*at.* file gdbme.c, line 34.*\(gdb\) $}? yes
 423         expect: set expect_out(0,start) {18}
 424         expect: set expect_out(0,end) {71}
 425         expect: set expect_out(0,string) {Breakpoint 8 at 0x23d8: file
 426         gdbme.c, line 34.\r\n(gdb) }
 427         epect: set expect_out(spawn_id) {6}
 428         expect: set expect_out(buffer) {break gdbme.c:34\r\nBreakpoint 8
 429         at 0x23d8: file gdbme.c, line 34.\r\n(gdb) }
 430         PASS:   70      0       breakpoint line number in file
 431         </PRE
 432 ></TD
 433 ></TR
 434 ></TABLE
 435 ></DIV
 436 ><P
 437 >This example exhibits three properties of
 438         <SPAN
 439 CLASS="PRODUCTNAME"
 440 >Expect</SPAN
 441 > and
 442         <SPAN
 443 CLASS="PRODUCTNAME"
 444 >DejaGnu</SPAN
 445 > that might be surprising at
 446         first glance:</P
 447 ><P
 448 ></P
 449 ><UL
 450 ><LI
 451 STYLE="list-style-type: disc"
 452 ><P
 453 >Empty output for the first attempted match.  The
 454         first set of attempted matches shown ran against the output
 455         <I
 456 CLASS="EMPHASIS"
 457 >{}</I
 458 > --- that is, no
 459         output. <SPAN
 460 CLASS="PRODUCTNAME"
 461 >Expect</SPAN
 462 > begins
 463         attempting to match the patterns supplied immediately; often,
 464         the first pass is against incomplete output (or completely
 465         before all output, as in this case).</P
 466 ></LI
 467 ><LI
 468 STYLE="list-style-type: disc"
 469 ><P
 470 >Interspersed tool output.  The beginning of
 471         the log entry for the second attempted match may be hard to
 472         spot: this is because the prompt <I
 473 CLASS="EMPHASIS"
 474 >{(gdb) }</I
 475 >
 476         appears on the same line, just before the
 477         <I
 478 CLASS="EMPHASIS"
 479 >expect:</I
 480 > that marks the beginning of the
 481         log entry.</P
 482 ></LI
 483 ><LI
 484 STYLE="list-style-type: disc"
 485 ><P
 486 >Fail-safe patterns.  Many of the patterns
 487         tested are fail-safe patterns provided by
 488         <SPAN
 489 CLASS="PRODUCTNAME"
 490 >GDB</SPAN
 491 > testing utilities, to reduce
 492         possible indeterminacy.  It is useful to anticipate potential
 493         variations caused by extreme system conditions
 494         (<SPAN
 495 CLASS="PRODUCTNAME"
 496 >GDB</SPAN
 497 > might issue the message
 498         <I
 499 CLASS="EMPHASIS"
 500 >virtual memory exhausted</I
 501 > in rare
 502         circumstances), or by changes in the tested program
 503         (<I
 504 CLASS="EMPHASIS"
 505 >Undefined command</I
 506 > is the likeliest
 507         outcome if the name of a tested command changes).</P
 508 ><P
 509 >The pattern <I
 510 CLASS="EMPHASIS"
 511 >{return}</I
 512 > is a
 513         particularly interesting fail-safe to notice; it checks for an
 514         unexpected <B
 515 CLASS="KEYCAP"
 516 >RET</B
 517 > prompt.  This may happen,
 518         for example, if the tested tool can filter output through a
 519         pager.</P
 520 ><P
 521 >These fail-safe patterns (like the debugging log itself)
 522         are primarily useful while developing test scripts.  Use the
 523         <B
 524 CLASS="COMMAND"
 525 >error</B
 526 > procedure to make the actions for
 527         fail-safe patterns produce messages starting with
 528         <I
 529 CLASS="EMPHASIS"
 530 >ERROR</I
 531 > on standard output, and in the
 532         detailed log file.</P
 533 ></LI
 534 ></UL
 535 ></DIV
 536 ></DIV
 537 ><DIV
 538 CLASS="NAVFOOTER"
 539 ><HR
 540 ALIGN="LEFT"
 541 WIDTH="100%"><TABLE
 542 WIDTH="100%"
 543 BORDER="0"
 544 CELLPADDING="0"
 545 CELLSPACING="0"
 546 ><TR
 547 ><TD
 548 WIDTH="33%"
 549 ALIGN="left"
 550 VALIGN="top"
 551 ><A
 552 HREF="runtest.html"
 553 >Prev</A
 554 ></TD
 555 ><TD
 556 WIDTH="34%"
 557 ALIGN="center"
 558 VALIGN="top"
 559 ><A
 560 HREF="book1.html"
 561 >Home</A
 562 ></TD
 563 ><TD
 564 WIDTH="33%"
 565 ALIGN="right"
 566 VALIGN="top"
 567 ><A
 568 HREF="customizing.html"
 569 >Next</A
 570 ></TD
 571 ></TR
 572 ><TR
 573 ><TD
 574 WIDTH="33%"
 575 ALIGN="left"
 576 VALIGN="top"
 577 >Runtest</TD
 578 ><TD
 579 WIDTH="34%"
 580 ALIGN="center"
 581 VALIGN="top"
 582 ><A
 583 HREF="runningtests.html"
 584 >Up</A
 585 ></TD
 586 ><TD
 587 WIDTH="33%"
 588 ALIGN="right"
 589 VALIGN="top"
 590 >Customizing DejaGnu</TD
 591 ></TR
 592 ></TABLE
 593 ></DIV
 594 ></BODY
 595 ></HTML
 596 >