1 o\input texinfo @c -*- Texinfo -*-
3 @setfilename dejagnu.info
5 @c This file documents the GNU Testing Framework ``DejaGnu''
7 @c Copyright (C) 92, 93, 94, 95, 96, 97, 98, 99, 2000 Free Software Foundation, Inc.
9 @c This text may be freely distributed under the terms of the GNU
10 @c General Public License.
13 @c FIXME---MAIN TODO LIST!
15 @c * Revisit organization.
17 @c * discuss Tcl/expect basics---enough to get started (see seminar notes).
18 @c Maybe this would permit abbreviating appendices.
23 * DejaGnu: (dejagnu). The GNU testing framework.
31 @setchapternewpage odd
32 @settitle DejaGnu Testing Framework
34 @title The DejaGnu Testing Framework
35 @subtitle for DejaGnu Version 1.3
42 {\parskip=0pt \hfill Cygnus Support}
45 @vskip 0pt plus 1filll
46 Copyright @copyright{} 92, 93, 94, 95, 96, 97, 98, 99, 2000 Free Software Foundation, Inc.
48 Permission is granted to make and distribute verbatim copies of
49 this manual provided the copyright notice and this permission notice
50 are preserved on all copies.
53 Permission is granted to copy and distribute modified versions of this
54 manual under the conditions for verbatim copying, provided also that
55 the entire resulting derived work is distributed under the terms of a
56 permission notice identical to this one.
58 Permission is granted to copy and distribute translations of this manual
59 into another language, under the above conditions for modified versions.
63 Copyright @copyright{} 92, 93, 94, 95, 96, 97, 98, 99, 2000 Free Software Foundation, Inc.
65 Permission is granted to make and distribute verbatim copies of
66 this manual provided the copyright notice and this permission notice
67 are preserved on all copies.
70 Permission is granted to process this file through TeX and print the
71 results, provided the printed document carries a copying permission
72 notice identical to this one except for the removal of this paragraph
73 (this paragraph not being relevant to the printed manual).
76 Permission is granted to copy and distribute modified versions of this
77 manual under the conditions for verbatim copying, provided also that
78 the entire resulting derived work is distributed under the terms of a
79 permission notice identical to this one.
81 Permission is granted to copy and distribute translations of this manual
82 into another language, under the above conditions for modified versions.
87 DejaGnu is a framework for running test suites on software tools.
89 This file describes version 1.3 of DejaGnu.
92 * Overview:: What is DejaGnu?
93 * What is New:: What is new in this release.
94 * Invoking runtest:: Using `runtest', the main test driver
95 * Customizing:: Setting `runtest' defaults
96 * Internals:: The DejaGnu implementation
97 * Tests:: How to write a test case
98 * Extending:: New tools, new targets, and new hosts
99 * Installation:: Configuring and Installing DejaGnu
109 @chapter What is DejaGnu?
112 DejaGnu is a framework for testing other programs. Its purpose is to
113 provide a single front end for all tests. Beyond this, DejaGnu offers
114 several advantages for testing:
118 The flexibility and consistency of the DejaGnu framework make it easy
119 to write tests for any program.
122 DejaGnu provides a layer of abstraction which allows you to write tests
123 that are portable to any host or target where a program must be tested.
124 For instance, a test for GDB can run (from any Unix based host) on any
125 target architecture that DejaGnu supports. Currently DejaGnu runs tests
126 on several single board computers, whose operating software ranges from
127 just a boot monitor to a full-fledged, Unix-like realtime OS.
130 All tests have the same output format. This makes it easy to integrate
131 testing into other software development processes. DejaGnu's output is
132 designed to be parsed by other filtering script, and it is also human
136 DejaGnu is written in @code{expect}, which in turn uses @dfn{Tcl}---Tool
139 @cindex @code{expect} script names
141 @cindex suffix, @code{expect} scripts
142 Running tests requires two things: the testing framework, and the test
143 suites themselves. Tests are usually written in @code{expect} using
144 Tcl, but you can also use a Tcl script to run a test suite that is not
145 based on @code{expect}. (@code{expect} script filenames conventionally
146 use @samp{.exp} as a suffix; for example, the main implementation of the
147 DejaGnu test driver is in the file @file{runtest.exp}.)
151 * Running Tests:: A first look at running DejaGnu tests
152 * Sample Test:: What does a DejaGnu test case look like?
153 * Design Goals:: Goals behind DejaGnu
154 * Posix:: DejaGnu conforms to POSIX 1003.3
155 * Future Directions:: Where is DejaGnu going?
156 * Tcl and Expect:: Reading more about Tcl and Expect
160 @chapter What is new in this release ?
163 This release has a number of substantial changes over version 1.2. The
164 most visible change is that the version of expect and Tcl included in
165 the release are up-to-date with the current stable net releases. Other
170 @c FIXME: add a link to the config section
171 The config sub-system in DejaGnu has been completely redesigned. It now
172 supports testing on remote hosts as well as remote targets.
175 More builtin support for building target binaries with the correct
176 linker flags. Currently this only works with GCC, preferably with a
177 target support by @code{libgloss}.
180 Lots of little bug fixes from a year of heavy use here at Cygnus
184 DejaGnu now uses @code{autoconf} for configuration.
187 New test cases for DejaGnu have been added for the new features, plus
188 the "--tool" option bug in the 1.2 testsuite has been fixed.
191 The @code{--tool} option is now optional.
194 @code{runtest} when searching for test drivers ignores all directories
195 named SCCS, RCS, and CVS.
198 There is now a generic keyword based test harness that uses comments in
199 source code to control how each test case gets built and run.
202 There is now some support for running a testsuite with multiple passes
203 and multiple targets.
208 @section Running existing tests
209 @cindex existing tests, running
210 @cindex running tests
211 @cindex tests, running
214 To run tests from an existing collection, first use @code{configure} as
215 usual to set up the source directory containing the tests. Then try
222 @cindex @code{check} makefile target
223 If the @code{check} target exists, it usually saves you some
224 trouble---for instance, it can set up any auxiliary programs or other
225 files needed by the tests.
227 @cindex auxiliary files, building
228 Once you have run @samp{make check} to build any auxiliary files, you
229 might want to call the test driver @code{runtest} directly to repeat the
230 tests. You may also have to call @code{runtest} directly for test
231 collections with no @code{check} target in the @file{Makefile}.
233 @c force page break to avoid losing footnote to another page
235 @cindex @code{runtest}, most common options
236 @cindex options for @code{runtest}, common
237 Typically, you must use two command-line options: @samp{--tool}, to
238 specify which set of tests to run@footnote{@samp{--tool} selects a
239 particular suite of tests, @emph{not} the name of the executable program
240 to run. @xref{Config Values,,Configuration dependent values}, for
241 information on the variables that you can use to specify the names of
242 programs to run.}, and @samp{--srcdir}, to specify where to find test
245 For example, if the directory @file{gdb/testsuite} contains a collection
246 of DejaGnu tests for @sc{gdb}, you can run them like this:
250 eg$ runtest --tool gdb
251 @exdent @emph{Test output follows, ending with:}
255 # of expected passes 508
256 # of expected failures 103
257 /usr/latest/bin/gdb version 4.14.4 -nx
260 You can use the option @samp{--srcdir} to point to some other directory
261 containing a collection of tests:
264 eg$ runtest --tool gdb --srcdir /devo/gdb/testsuite
267 @cindex native configuration
268 @cindex cross configuration
269 These examples assume a @dfn{native} configuration, where the same
270 computer runs both @code{runtest} and the tests themselves. When you
271 have a @dfn{cross} configuration, the tests run on a different computer,
272 controlled by the host running @code{runtest}. In this situation, you
273 need the option @samp{--name} to specify the network address for the
277 eg$ runtest --tool gdb --name vx9.munist.com
280 If you always use the same option values, you can record them in a file
281 called @file{site.exp}, rather than typing them each time. @xref{Config
282 Values,,Setting defaults for @code{runtest} options}.
284 By default, @code{runtest} prints only the names of the tests it runs,
285 output from any tests that have unexpected results, and a summary
286 showing how many tests passed and how many failed. To display output
287 from all tests (whether or not they behave as expected), use the
288 @samp{--all} option. For more verbose output about processes being run,
289 communication, and so on, use @samp{--verbose}. To see even more output,
290 use multiple @samp{--verbose} options. @xref{Invoking runtest,,Using
291 @code{runtest}}, for a more detailed explanation of each @code{runtest}
294 Test output goes into two files in your current directory: summary
295 output in @file{@var{tool}.sum}, and detailed output in
296 @file{@var{tool}.log}. (@var{tool} refers to the collection of tests;
297 for example, after a run with @samp{--tool gdb}, look for output files
298 @file{gdb.sum} and @file{gdb.log}.) @xref{Output Files,,The files
302 @section What does a DejaGnu test look like?
305 Each DejaGnu test is an @code{expect} script; the tests vary widely in
306 complexity, depending on the nature of the tool and the feature tested.
308 @kindex gdb.t00/echo.exp
310 Here is a very simple @sc{gdb} test---one of the simplest tests shipped
311 with DejaGnu (extracted from @file{gdb.t00/echo.exp}):@footnote{More
312 recent @sc{gdb} tests use the @samp{gdb_test} procedure.
313 An equivalent test using that procedure is @samp{ gdb_test "echo Hello
314 world!" "Hello world!" }}
315 @c FIXME! include xref in footnote, when gdb_test documented in some manual.
317 @c Extra spaces in @samp above avoid running end ' against " inside.
321 # send a string to the GDB stdin:
322 send "echo Hello world!\n"
324 # inspect the GDB stdout for the correct reply,
325 # and determine whether the test passes or fails:
327 -re "Hello world.*$prompt $" @{ pass "Echo test" @}
328 -re "$prompt $" @{ fail "Echo test" @}
329 timeout @{ fail "(timeout) Echo test" @}
334 Though brief, this example is a complete test. It illustrates some of
335 the main features of DejaGnu test scripts:
339 The test case does not start the tested program (@sc{gdb} in this case);
340 all test scripts for interactive tools can assume the corresponding tool
344 Comments start with @samp{#}.
347 The main commands you use to control a tested program are @code{send}
348 (to give it commands) and @code{expect} (to analyze its responses).
351 The @code{expect} command uses a list of pairs; a pattern (regular
352 expression if @samp{-re} specified), followed by an action to run if the
353 pattern matches output from the program. Only the action for the
354 @emph{first} matching pattern will execute.
357 Test cases use the commands @code{pass} and @code{fail} to record the
362 @section Design goals
365 DejaGnu grew out of the internal needs of Cygnus Support. Cygnus
366 maintains and enhances a variety of free programs in many different
367 environments, and we needed a testing tool that:
371 is useful to developers while fixing bugs;
374 automates running many tests during a software release process;
377 is portable among a variety of host computers;
380 supports cross-development testing;
383 permits testing interactive programs, like @sc{gdb}; and
386 permits testing batch oriented programs, like @sc{gcc}.
389 Some of the requirements proved challenging. For example, interactive
390 programs do not lend themselves very well to automated testing. But all
391 the requirements are important: for instance, it is imperative to make
392 sure that @sc{gdb} works as well when cross-debugging as it does in a
393 native configuration.
395 Probably the greatest challenge was testing in a cross-development
396 environment (which can be a real nightmare). Most cross-development
397 environments are customized by each developer. Even when buying
398 packaged boards from vendors there are many differences. The
399 communication interfaces vary from a serial line to ethernet. DejaGnu
400 was designed with a modular communication setup, so that each kind of
401 communication can be added as required, and supported thereafter. Once
402 a communication procedure is coded, any test can use it. Currently
403 DejaGnu can use @code{rsh}, @code{rlogin}, @code{telnet}, @code{tip},
404 @code{kermit}, and @code{mondfe} for remote communications.
406 @cindex name ``DejaGnu''
407 @cindex DejaGnu, the name
408 @cindex Menapace, Julia
409 Julia Menapace first coined the term ``Deja Gnu'' to describe an earlier
410 testing framework at Cygnus Support. When we replaced it with the
411 Expect-based framework, it was like DejaGnu all over again@dots{}
414 @section A POSIX conforming test framework
416 @cindex POSIX conformance
417 @cindex standard conformance: POSIX 1003.3
418 DejaGnu conforms to the @sc{posix} standard for test frameworks.
421 @sc{posix} standard 1003.3 defines what a testing framework needs to
422 provide, in order to permit the creation of @sc{posix} conformance
423 test suites. This standard is primarily oriented to running @sc{posix}
424 conformance tests, but its requirements also support testing of features
425 not related to @sc{posix} conformance. @sc{posix} 1003.3 does not
426 specify a particular testing framework, but at this time there is only
427 one other @sc{posix} conforming test framework:
428 @sc{tet}.@footnote{@sc{tet} was created by Unisoft for a consortium
429 comprised of X/Open, Unix International, and the Open Software
432 The @sc{posix} documentation refers to @dfn{assertions}. An assertion
433 is a description of behavior. For example, if a standard says ``The sun
434 shall shine'', a corresponding assertion might be ``The sun is
435 shining.'' A test based on this assertion would pass or fail depending
436 on whether it is daytime or nighttime. It is important to note that the
437 standard being tested is never 1003.3; the standard being tested is some
438 other standard, for which the assertions were written.
440 As there is no test suite to test @emph{testing frameworks} for
441 @sc{posix} 1003.3 conformance, verifying conformance to this standard is
442 done by repeatedly reading the standard and experimenting. One of the
443 main things 1003.3 does specify is the set of allowed output messages,
444 and their definitions. Four messages are supported for a required
445 feature of @sc{posix} conforming systems, and a fifth for a conditional
446 feature. DejaGnu supports the use of all five output messages; in this
447 sense a test suite that uses exactly these messages can be considered
448 @sc{posix} conforming. These definitions specify the output of a test
452 @cindex success, POSIX definition
454 A test has succeeded. That is, it demonstrated that the assertion is true.
456 @cindex XFAIL, avoiding for POSIX
458 @sc{posix} 1003.3 does not incorporate the notion of expected failures,
459 so @code{PASS}, instead of @code{XPASS}, must also be returned for test
460 cases which were expected to fail and did not. This means that
461 @code{PASS} is in some sense more ambiguous than if @code{XPASS} is also
462 used. For information on @code{XPASS} and @code{XFAIL}, see
463 @ref{Invoking runtest,,Using @code{runtest}}.
466 @cindex failure, POSIX definition
467 A test @emph{has} produced the bug it was intended to capture. That is,
468 it has demonstrated that the assertion is false. The @code{FAIL}
469 message is based on the test case only. Other messages are used to
470 indicate a failure of the framework.
472 As with @code{PASS}, @sc{posix} tests must return @code{FAIL} rather
473 than @code{XFAIL} even if a failure was expected.
476 @cindex ambiguity, required for POSIX
477 A test produced indeterminate results. Usually, this means the test
478 executed in an unexpected fashion; this outcome requires that a human
479 being go over results, to determine if the test should have passed or
480 failed. This message is also used for any test that requires human
481 intervention because it is beyond the abilities of the testing
482 framework. Any unresolved test should resolved to @code{PASS} or
483 @code{FAIL} before a test run can be considered finished.
485 Note that for @sc{posix}, each assertion must produce a test result
486 code. If the test isn't actually run, it must produce @code{UNRESOLVED}
487 rather than just leaving that test out of the output. This means that
488 you have to be careful when writing tests, to not carelessly use tcl
489 statements like @code{return}---if you alter the flow of control of the
490 tcl code you must insure that every test still produces some result code.
492 Here are some of the ways a test may wind up @code{UNRESOLVED}:
496 A test's execution is interrupted.
499 A test does not produce a clear result. This is usually because there
500 was an @code{ERROR} from DejaGnu while processing the test, or because there
501 were three or more @code{WARNING} messages. Any @code{WARNING} or
502 @code{ERROR} messages can invalidate the output of the test. This
503 usually requires a human being to examine the output to
504 determine what really happened---and to improve the test case.
507 A test depends on a previous test, which fails.
510 The test was set up incorrectly.
514 A test was not run. This is a placeholder, used when there is no
519 The only remaining output message left is intended to test features that
520 are specified by the applicable @sc{posix} standard as conditional:
524 There is no support for the tested case. This may mean that a
525 conditional feature of an operating system, or of a compiler, is not
526 implemented. DejaGnu also uses this message when a testing environment
527 (often a ``bare board'' target) lacks basic support for compiling or
528 running the test case. For example, a test for the system subroutine
529 @code{gethostname} would never work on a target board running only a
533 DejaGnu uses the same output procedures to produce these messages for
534 all test suites, and these procedures are already known to conform to
535 @sc{posix} 1003.3. For a DejaGnu test suite to conform to @sc{posix}
536 1003.3, you must avoid the @code{setup_xfail} procedure as described in
537 the @code{PASS} section above, and you must be careful to return
538 @code{UNRESOLVED} where appropriate, as described in the
539 @code{UNRESOLVED} section above.
541 @node Future Directions
542 @section Future directions
543 @cindex future directions
545 In the near future, there are two parallel directions for DejaGnu
546 development. The first is to add support for more hosts and targets.
548 The second would permit testing programs with a more complex interface,
549 whether text based or GUI based. Two components already exist: a Tcl
550 based X window toolkit, and a terminal package for @code{expect}. Both
551 of these could be merged into DejaGnu in a way that permits testing
552 programs that run in each environment.
554 Meanwhile, we hope DejaGnu enables the creation of test suites for
555 conformance to @sc{ansi} C and C++, to @sc{posix}, and to other
556 standards. We encourage you to make any test suites you create freely
557 available, under the same terms as DejaGnu itself.
560 @section Tcl and Expect
561 @cindex tool command language
563 @cindex Ousterhout, John K.
564 Tcl was introduced in a paper by John K. Ousterhout at the 1990 Winter
565 Usenix conference, @cite{Tcl: An Embeddable Command Language}. That
566 paper is included in PostScript form in the @file{doc} subdirectory of
567 the Tcl distribution. The version of Tcl included in DejaGnu at this time is
570 @cindex @code{expect} scripting language
572 Don Libes introduced @code{expect} in his paper @cite{expect: Curing
573 Those Uncontrollable Fits of Interaction} at the 1990 Summer Usenix
574 conference. The paper is included in PostScript form in the
575 @code{expect} distribution (as are several other papers about
576 @code{expect}). The version of expect included in DejaGnu at this time
579 @node Invoking runtest
580 @chapter Using @code{runtest}
583 @cindex command line options
586 @cindex @code{runtest} description
587 @cindex DejaGnu test driver
588 @code{runtest} is the executable test driver for DejaGnu. You can
589 specify two kinds of things on the @code{runtest} command line: command
590 line options, and Tcl variables for the test scripts. The options are
591 listed alphabetically below.
593 @cindex exit code from @code{runtest}
594 @cindex @code{runtest} exit code
595 @code{runtest} returns an exit code of @code{1} if any test
596 has an unexpected result; otherwise (if all tests pass or fail as
597 expected) it returns @code{0} as the exit code.
599 @code{runtest} flags the outcome of each test as one of these cases.
600 (@xref{Posix,,A POSIX conforming test framework}, for a discussion of
601 how @sc{posix} specifies the meanings of these cases.)
606 @cindex successful test
607 @cindex test, successful
608 The most desirable outcome: the test succeeded, and was expected to
613 @cindex successful test, unexpected
614 @cindex unexpected success
615 A pleasant kind of failure: a test was expected to fail, but succeeded.
616 This may indicate progress; inspect the test case to determine whether
617 you should amend it to stop expecting failure.
621 @cindex failing test, unexpected
622 @cindex test, failing
623 A test failed, although it was expected to succeed. This may indicate
624 regress; inspect the test case and the failing software to locate the bug.
628 @cindex expected failure
629 @cindex failing test, expected
630 A test failed, but it was expected to fail. This result indicates no
631 change in a known bug. If a test fails because the operating system
632 where the test runs lacks some facility required by the test, the
633 outcome is @code{UNSUPPORTED} instead.
637 @cindex test, unresolved outcome
638 Output from a test requires manual inspection; the test suite could not
639 automatically determine the outcome. For example, your tests can report
640 this outcome is when a test does not complete as expected.
644 @cindex untested properties
645 A test case is not yet complete, and in particular cannot yet produce a
646 @code{PASS} or @code{FAIL}. You can also use this outcome in dummy
647 ``tests'' that note explicitly the absence of a real test case
648 for a particular property.
652 @cindex unsupported test
653 @cindex test, unsupported
654 A test depends on a conditionally available feature that does not exist
655 (in the configured testing environment). For example, you can use this
656 outcome to report on a test case that does not work on a particular
657 target because its operating system support does not include a required
661 @code{runtest} may also display the following messages:
666 @cindex problem, detected by test case
667 @cindex test case cannot run
668 Indicates a major problem (detected by the test case itself) in running
669 the test. This is usually an unrecoverable error, such as a missing file
670 or loss of communication to the target. (@sc{posix} test suites should
671 not emit this message; use @code{UNSUPPORTED}, @code{UNTESTED}, or
672 @code{UNRESOLVED} instead, as appropriate.)
676 @cindex test case warnings
677 Indicates a possible problem in running the test. Usually warnings
678 correspond to recoverable errors, or display an important message about
683 @cindex test case messages
684 An informational message about the test case.
687 This is the full set of command line options that @code{runtest}
688 recognizes. Arguments may be abbreviated to the shortest unique string.
690 @cindex @code{runtest} option list
691 @cindex option list, @code{runtest}
693 runtest --tool @var{tool} [ @var{testsuite}.exp @dots{} ]
694 [ @var{testsuite}.exp="testfile1 @dots{}" ]
695 [ @var{tclvar}=@var{value}@dots{} ]
696 [ --all ] [ --baud @var{baud-rate} ] [ --connect @var{type} ]
697 [ --debug ] [ --help ] [ --host @var{string} ]
698 [ --mail "@var{name} @dots{}" ] [ --name @var{string} ]
699 [ --name @var{name} ] [ --outdir @var{path} ]
700 [ --objdir @var{path} ] [ --reboot ]
701 [ --srcdir @var{path} ] [ --strace @var{n} ]
702 [ --target @var{string} --build @var{string} ]
703 [ -v | --verbose ] [ -V | --version ] [ --D@var{n} ]
707 @item --tool @var{tool}
708 @cindex selecting tests for a tool
709 @cindex @code{--tool} (@code{runtest} option)
710 @var{tool} specifies what set of tests to run, and what initialization
711 module to use. @var{tool} is used @emph{only} for these two purposes:
712 it is @emph{not} used to name the executable program to test.
713 Executable tool names (and paths) are recorded in @file{site.exp}
714 (@pxref{Config Values,,Configuration dependent values}), and you can
715 override them by specifying Tcl variables on the command line.
717 For example, including @samp{--tool gcc} on the @code{runtest} command
718 line runs tests from all test subdirectories whose names match
719 @file{gcc.*}, and uses one of the initialization modules named
720 @file{config/*-gcc.exp}. To specify the name of the compiler (perhaps
721 as an alternative path to what @code{runtest} would use by default), use
722 @samp{GCC=@var{binname}} on the @code{runtest} command line.
724 @item @var{testsuite}.exp @dots{}
725 @cindex selecting a range of tests
726 @cindex tests, running specifically
727 @cindex naming tests to run
728 Specify the names of testsuites to run.
729 By default, @code{runtest} runs all tests for the tool, but you can
730 restrict it to particular testsuites by giving the names of the @samp{.exp}
731 @code{expect} scripts that control them.
733 @var{testsuite}.exp may not include path information; use plain filenames.
735 @item @var{testfile}.exp="testfile1 @dots{}"
736 @cindex selecting a range of tests
737 @cindex tests, running specifically
738 @cindex naming tests to run
739 Specify a subset of tests in a suite to run.
740 For compiler or assembler tests, which often use a single @samp{.exp}
741 script covering many different source files, this option allows you to
742 further restrict the tests by listing particular source files to compile.
743 Some tools even support wildcards here. The wildcards supported depend
744 upon the tool, but typically they are @code{?}, @code{*}, and @code{[chars]}.
746 @item @var{tclvar}=@var{value}
747 @kindex @var{tclvar}=@var{value}
748 @cindex Tcl variables, defining for @code{runtest}
749 @cindex command line Tcl variable definition
750 @cindex @code{runtest}, variable defns on cmdline
751 You can define Tcl variables for use by your test scripts in the same
752 style used with @code{make} for environment variables. For example,
753 @samp{runtest GDB=gdb.old} defines a variable called @samp{GDB}; when
754 your scripts refer to @samp{$GDB} in this run, they use the value
757 The default Tcl variables used for most tools are defined in the main
758 DejaGnu @code{Makefile}; their values are captured in the
759 @file{site.exp} file. @xref{Config Values,,Configuration dependent
763 @cindex @code{--all} (@code{runtest} option)
764 @cindex test output, displaying all
765 Display all test output. By default, @code{runtest} shows only the
766 output of tests that produce unexpected results; that is, tests with
767 status @samp{FAIL} (unexpected failure), @samp{XPASS} (unexpected
768 success), or @samp{ERROR} (a severe error in the test case itself).
769 Specify @samp{--all} to see output for tests with status @samp{PASS}
770 (success, as expected) @samp{XFAIL} (failure, as expected), or
771 @samp{WARNING} (minor error in the test case itself).
773 @item --baud @var{baud-rate}
774 @itemx -b @var{baud-rate}
775 @cindex baud rate, specifying
776 @cindex bps, specifying
777 @cindex @code{--baud} (@code{runtest} option)
778 @cindex @code{-b} (@code{runtest} option)
779 Set the default baud rate to something other than 9600. (Some serial
780 interface programs, like @code{tip}, use a separate initialization file
781 instead of this value.)
783 @item --connect @var{type}
784 @cindex connecting to target
785 @cindex @code{--connect} (@code{runtest} option)
786 @cindex remote testbed, connecting to
787 @cindex @code{rlogin}, remote testing via
788 @cindex @code{telnet}, remote testing via
789 @cindex @code{rsh}, remote testing via
790 @cindex @code{tip}, remote testing via
791 @cindex @code{kermit}, remote testing via
792 @cindex @code{mondfe}, remote testing via
793 @cindex remote testing via @code{rlogin}
794 @cindex remote testing via @code{telnet}
795 @cindex remote testing via @code{rsh}
796 @cindex remote testing via @code{tip}
797 @cindex remote testing via @code{kermit}
798 @cindex remote testing via @code{mondfe}
799 Connect to a target testing environment as specified by @var{type}, if
800 the target is not the computer running @code{runtest}. For example, use
801 @samp{--connect} to change the program used to connect to a ``bare
802 board'' boot monitor. The choices for @var{type} in the DejaGnu 1.0
803 distribution are @samp{rlogin}, @samp{telnet}, @samp{rsh}, @samp{tip},
804 @samp{kermit}, and @samp{mondfe}.
807 The default for this option depends on the configuration (@pxref{Cross
808 Targets,,Remote targets supported}). The default is chosen to be the
809 most convenient communication method available, but often other
810 alternatives work as well; you may find it useful to try alternative
811 connect methods if you suspect a communication problem with your testing
815 @cindex @code{--debug} (@code{runtest} option)
816 @cindex debug log for test cases
817 @cindex test cases, debug log
818 @cindex @code{dbg.log} file
819 Turns on the @code{expect} internal debugging output. Debugging output
820 is displayed as part of the @code{runtest} output, and logged to a file
821 called @file{dbg.log}. The extra debugging output does @emph{not}
822 appear on standard output, unless the verbose level is greater than 2
823 (for instance, to see debug output immediately, specify @samp{--debug -v
824 -v}). The debugging output shows all attempts at matching the test
825 output of the tool with the scripted patterns describing expected
826 output. The output generated with @samp{--strace} also goes into
831 @cindex @code{--help} (@code{runtest} option)
832 @cindex help with @code{runtest}
833 @cindex @code{runtest}, listing options
834 Prints out a short summary of the @code{runtest} options, then exits
835 (even if you also specify other options).
837 @item --host @var{string}
838 @cindex @code{--host} (@code{runtest} option)
839 @cindex specifying the host config name
840 @cindex host config name, changing
841 @var{string} is a full configuration ``triple'' name as used by
842 @code{configure}. Use this option to override the default string
843 recorded by your configuration's choice of host. This choice does not
844 change how anything is actually configured unless --build is also
845 specified; it affects @emph{only} DejaGnu procedures that compare the
846 host string with particular values. The procedures @code{ishost},
847 @code{istarget}, @code{isnative}, and @code{setup_xfail} are affected by
848 @samp{--host}. In this usage, @code{host} refers to the machine that the
849 tests are to be run on, which may not be the same as the @code{build}
850 machine. If @code{--build} is also specified, then @code{--host} refers
851 to the machine that the tests wil, be run on, not the machine DejaGnu is
854 @item --build @var{string}
855 @cindex @code{--build} (@code{runtest} option)
856 @cindex specifying the build config name
857 @cindex build config name, changing
858 @var{string} is a full configuration ``triple'' name as used by
859 @code{configure}. This is the type of machine DejaGnu and the tools to
860 be tested are built on. For a normal cross this is the same as the host,
861 but for a canadian cross, they are seperate.
863 @item --name @var{name}
864 @cindex specifying target name
865 @cindex target machine name
866 @cindex @code{--name} (@code{runtest} option)
867 @var{name} is a name for the particular testing target machine (for
868 cross testing). If the testing target has IP network support (for
869 example, @code{RPC} or @code{NFS}), this is the network name for the
870 target itself. (@var{name} is @emph{not the configuration string} you
871 specify as a target with @code{configure}; the @samp{--name} option
872 names a particular target, rather than describing a class of targets.)
873 For targets that connect in other ways, the meaning of the @var{name}
874 string depends on the connection method. @xref{Cross Targets,,Remote
877 @item --name @var{string}
878 @cindex remote test machine name
879 @cindex name for remote test machine
880 Specify a network name of testing target or its host. The particular
881 names that are meaningful with @samp{--name} will depend on your site
882 configuration, and on the connection protocol: for example, @code{tip}
883 connections require names from a serial line configuration file (usually
884 called @file{/etc/remote}), while @code{telnet} connections use IP
887 @item --objdir @var{path}
888 @cindex @code{--objdir} (@code{runtest} option)
889 @cindex object directory
890 @cindex test programs, auxiliary
891 @cindex auxiliary test programs
892 Use @var{path} as the top directory containing any auxiliary compiled
893 test code. This defaults to @file{.}. Use this option to locate
894 pre-compiled test code. You can normally prepare any auxiliary files
895 needed with @code{make}.
897 @item --outdir @var{path}
898 @cindex output directory
899 @cindex @code{--outdir} (@code{runtest} option)
900 @cindex log files, where to write
901 Write output logs in directory @var{path}. The default is @samp{.}, the
902 directory where you start @code{runtest}. This option affects only the
903 summary and the detailed log files @file{@var{tool}.sum} and
904 @file{@var{tool}.log}. The DejaGnu debug log @file{dbg.log} always
905 appears (when requested) in the local directory.
908 @cindex rebooting remote targets
909 @cindex @code{--reboot} (@code{runtest} option)
910 Reboot the target board when @code{runtest} initializes.
911 Usually, when running tests on a separate target board, it is safer to
912 reboot the target to be certain of its state. However, when developing
913 test scripts, rebooting takes a lot of time.
915 @item --srcdir @var{path}
916 @cindex source directory
917 @cindex @code{--srcdir} (@code{runtest} option)
918 Use @var{path} as the top directory for test scripts to run.
919 @code{runtest} looks in this directory for any subdirectory whose name
920 begins with the toolname (specified with @samp{--tool}). For instance,
921 with @samp{--tool gdb}, @code{runtest} uses tests in subdirectories
922 @file{gdb.*} (with the usual shell-like filename expansion). If you do
923 not use @samp{--srcdir}, @code{runtest} looks for test directories under
924 the current working directory.
926 @item --strace @var{n}
927 @cindex @code{--strace} (@code{runtest} option)
928 @cindex tracing Tcl commands
929 @cindex @code{expect} internal tracing
930 Turn on internal tracing for @code{expect}, to @var{n} levels deep. By
931 adjusting the level, you can control the extent to which your output
932 expands multi-level Tcl statements. This allows you to ignore some
933 levels of @code{case} or @code{if} statements. Each procedure call or
934 control structure counts as one ``level''.
936 The output is recorded in the same file, @file{dbg.log}, used for output
939 @item --target @var{string}
940 @cindex @code{--target} (@code{runtest} option)
941 @cindex specifying the target configuration
942 @cindex target configuration, specifying
943 Use this option to override the default setting (running native tests).
944 @var{string} is a full configuration ``triple''
945 name@footnote{Configuration triples have the form
946 @samp{@var{cpu}-@var{vendor}-@var{os}}.} as used by @code{configure}.
947 This option changes the configuration @code{runtest} uses for the
948 default tool names, and other setup information. @xref{Using
949 configure,,Using @code{configure}, configure.info, Cygnus configure},
950 for details about @code{configure} names.
954 @cindex @code{--verbose} (@code{runtest} option)
955 @cindex @code{-v} (@code{runtest} option)
956 @cindex turning on output
957 @cindex output, additional
958 Turns on more output. Repeating this option increases the amount of
959 output displayed. Level one (@samp{-v}) is simply test output. Level
960 two (@samp{-v -v}) shows messages on options, configuration, and process
961 control. Verbose messages appear in the detailed (@file{*.log}) log
962 file, but not in the summary (@file{*.sum}) log file.
966 @cindex @code{-V} (@code{runtest} option)
967 @cindex @code{--version} (@code{runtest} option)
968 @cindex version numbers
969 Prints out the version numbers of DejaGnu, @code{expect} and Tcl, and
970 exits without running any tests.
974 @cindex starting the tcl debugger
976 @c FIXME!!! we should say a *lot* more about this debugger
977 Start the internal Tcl debugger. The Tcl debugger supports breakpoints,
978 single stepping, and other common debugging activities. (See @cite{A
979 Debugger for Tcl Applications} by Don Libes. @footnote{Distributed in
980 PostScript form with @code{expect} as the file@*
981 @file{expect/tcl-debug.ps}.})
983 If you specify @samp{-D1}, the @code{expect} shell stops at a breakpoint
984 as soon as DejaGnu invokes it.
986 If you specify @samp{-D0}, DejaGnu starts as usual, but you can enter
987 the debugger by sending an interrupt (e.g. by typing @key{C-c}).
991 @chapter Setting @code{runtest} defaults
994 @cindex variables of DejaGnu, defaults
995 The site configuration file, @file{site.exp}, captures
996 configuration-dependent values and propagates them to the DejaGnu test
997 environment using Tcl variables. This ties the DejaGnu test scripts
998 into the @code{configure} and @code{make} programs.
1000 @cindex @file{site.exp}, multiple
1001 @cindex overriding @file{site.exp}
1002 DejaGnu supports more than one @file{site.exp} file. The multiple
1003 instances of @file{site.exp} are loaded in a fixed order built into
1004 DejaGnu (the more local last). The first file loaded is the optional
1005 @code{~/.dejagnurc}, then the local files, and finally the global file.
1009 There is am optional ``master'' @file{site.exp}, capturing configuration values
1010 that apply to DejaGnu across the board, in each configuration-specific
1011 subdirectory of the DejaGnu library directory. @code{runtest} loads
1012 these values first. @xref{Installation,,Configuring and Installing
1013 DejaGnu}. The master @file{site.exp} contains the default values for
1014 all targets and hosts supported by DejaGnu. This master file is
1015 identified by setting the environment variable @code{DEJAGNU} to the
1016 name of the file. This is also refered to as the ``global'' config file.
1019 Any directory containing a configured test suite also has a
1020 @file{site.exp}, capturing configuration values specific to the tool
1021 under test. Since @code{runtest} loads these values last, the
1022 individual test configuration can either rely on and use, or override,
1023 any of the global values from the ``master'' @file{site.exp}.
1025 You can usually generate or update the testsuite @file{site.exp} by
1026 typing @samp{make site.exp} in the test suite directory, after the test
1027 suite is configured.
1030 You can also have a file in your home directory called
1031 @code{.dejagnurc}. This gets loaded first before the other config
1032 files. Usually this is used for personal stuff, like setting
1033 @code{all_flag} so all the output gets printed, or verbosity levels.
1036 You can further override the default values in a user-editable section
1037 of any @file{site.exp}, or by setting variables on the @code{runtest}
1041 * Config Values:: Variables used in the configuration file.
1042 * Master Config File:: The master configuration file.
1043 * Local Config File:: The local configuration file.
1044 * Personal Config File:: The personal configuration file.
1047 @node Config Values, Master Config File, , Customizing
1048 @subsection Config Variables
1049 @cindex configuration dependent defaults
1050 @cindex setting defaults for DejaGnu variables
1052 @c NOTE: default values are given via @code{"fubar"} rather than the
1053 @c more conventional @samp{fubar} to permit a consistent and clear
1054 @c notation for the empty string (@code{""}), which will work exactly as
1057 DejaGnu uses a named array in Tcl to hold all the info for each
1058 machine. In the case of a canadian cross, this means host information as
1059 well as target information. The named array is called
1060 @code{target_info}, and it has two indices. The following fields are
1065 The name of the target. (mostly for error messages) This
1066 should also be the string used for this target's array.
1067 It should also be the same as the linker script so we
1068 can find them dynamically. This should be the same as the argument used
1069 for @code{push_target@{@}}.
1072 This is the linker flags required to produce a fully linked
1073 executable. For @code{libgloss} supported targets this is usually just
1074 the name of the linker script.
1077 The target canonical for this target. This is used by some init files to
1078 make sure the target is supported.
1081 The flags required to produce an object file from a source file.
1084 This is the connectmode for this target. This is for both IP and
1085 serial connections. Typically this is either @code{telnet},
1086 @code{rlogin}, or @code{rsh}.
1089 This is the hostname of the target. This is for TCP/IP based connections,
1090 and is also used for version of tip that use /etc/remote.
1093 This is the serial port. This is typically /dev/tty? or com?:.
1096 This is the IP port. This is commonly used for telneting to target
1097 boards that are connected to a terminal server. In that case the IP port
1098 specifies the which serial port to use.
1101 This is the baud rate for a serial port connection.
1104 This is the parameters for an x10 controller. These are simple devices
1105 that let us power cycle or reset a target board remotely.
1108 This is the fileid or spawn id of of the connection.
1111 a glob style pattern to recognize the prompt.
1114 abbreviation for tool init files.
1117 This is the port for I/O on dual port systems. In this configuration,
1118 the main serial port @code{0} is usually used for stdin and stdout,
1119 which the second serial port can be used for debugging.
1122 The first index into the array is the same value as used in the
1123 @code{name} field. This is usually a short version of the name of the
1124 target board. For an example, here's the settings I use for my
1125 @code{Motorola's} @code{IDP} board and my @code{Motorola} 6U VME
1126 @code{MVME135-1} board. (both m68k targets)
1131 set target_info(idp,name) "idp"
1132 set target_info(idp,ldflags) "-Tidp.ld"
1133 set target_info(idp,config) m68k-unknown-aout
1134 set target_info(idp,cflags) ""
1135 set target_info(idp,connect) telnet
1136 set target_info(idp,target) "s7"
1137 set target_info(idp,serial) "tstty7"
1138 set target_info(idp,netport) "wharfrat:1007"
1139 set target_info(idp,baud) "9600"
1141 set target_info(idp,name) "mvme"
1142 set target_info(idp,ldflags) "-Tmvme.ld"
1143 set target_info(idp,config) m68k-unknown-aout
1144 set target_info(idp,cflags) ""
1145 set target_info(idp,connect) telnet
1146 set target_info(idp,target) "s8"
1147 set target_info(idp,serial) "tstty8"
1148 set target_info(idp,netport) "wharfrat:1008"
1149 set target_info(idp,baud) "9600"
1153 DejaGnu can use this information to switch between multiple targets in
1154 one test run. This is done through the use of the @code{push_target}
1155 procedure, which is discussed elsewhere.
1156 @c FIXME: write that section and put an xref here
1158 This array can also hold information for a remote host, which is used
1159 when testing a candain cross. In this case, the only thing different is
1160 the index is just @code{host}. Here's the settings I use to run tests
1161 on my NT machine while running DejaGnu on a Unix machine. (in this case
1166 set target_info(host,name) "nt-host"
1167 set target_info(host,config) "386-unknown-winnt"
1168 set target_info(host,connect) "telnet"
1169 set target_info(host,target) "ripple"
1173 There is more info on how to use these variables in the sections on the
1174 config files. @xref{Master Config File,,Configuration Files}.
1176 @cindex option defaults
1177 @cindex @code{runtest} option defaults
1178 @cindex variables for option defaults
1179 @cindex defaults, option
1180 In the user editable second section of @file{site.exp}, you can not only
1181 override the configuration variables captured in the first section, but
1182 also specify default values for all the @code{runtest} command line
1183 options. Save for @samp{--debug}, @samp{--help}, and @samp{--version},
1184 each command line option has an associated Tcl variable. Use the Tcl
1185 @code{set} command to specify a new default value (as for the
1186 configuration variables). The following table describes the
1187 correspondence between command line options and variables you can set in
1188 @file{site.exp}. @xref{Invoking runtest,,Running the Tests}, for
1189 explanations of the command-line options.
1203 @kindex target_triplet
1204 @kindex host_triplet
1205 @kindex build_triplet
1208 @cindex command line option variables
1209 @cindex Tcl variables for option defaults
1210 @cindex default options, controlling
1211 @cindex options, Tcl variables for defaults
1216 option variable description
1217 __________ ________ ___________________________________________
1219 --all all_flag display all test results if set
1221 --baud baud set the default baud rate to something other
1223 --connect connectmode @samp{rlogin}, @samp{telnet}, @samp{rsh},
1224 @samp{kermit}, @samp{tip}, or @samp{mondfe}
1226 --outdir outdir directory for @file{@var{tool}.sum} and @file{@var{tool}.log}
1228 --objdir objdir directory for pre-compiled binaries
1230 --reboot reboot reboot the target if set to @code{"1"};
1231 do not reboot if set to @code{"0"} (the default)
1233 --srcdir srcdir directory of test subdirectories
1235 --strace tracelevel a number: Tcl trace depth
1237 --tool tool name of tool to test; identifies init, test subdir
1239 --verbose verbose verbosity level. As option, use multiple times;
1240 as variable, set a number, 0 or greater
1241 --target target_triplet The canonical configuration string for the target.
1242 --host host_triplet The canonical configuration string for the host.
1243 --build build_triplet The canonical configuration string for the
1250 \vbox{\halign{\hfil \tt #\quad &\quad\tt #\hfil &\hbox{\vtop{{\raggedright\parindent=0pt\parskip=5pt\hsize=2.75in\rm#\strut\par}}}\hfill\cr
1252 {\it runtest}&{\it Tcl}\cr
1253 {\it option}&{\it variable}&{\it description}\cr
1254 \noalign{\hrule width\hsize}\cr
1255 --all &all\_flag &display all test results if set\cr
1256 --baud &baud &set the default baud rate to something other
1258 --connect &connectmode &@samp{rlogin}, @samp{telnet}, @samp{rsh},
1259 @samp{kermit}, @samp{tip}, or @samp{mondfe}\cr
1260 --mail &mailing\_list&address list for mailing test output\cr
1261 --name &targetname &network name of testing target or its host\cr
1262 --outdir &outdir &directory for @file{@var{tool}.sum} and @file{@var{tool}.log}\cr
1263 --objdir &objdir &directory for compiled binaries\cr
1264 --reboot &reboot &reboot the target if set to @code{"1"};
1265 do not reboot if set to @code{"0"} (the default)\cr
1266 --srcdir &srcdir &directory of test subdirectories\cr
1267 --strace &tracelevel &a number: Tcl trace depth\cr
1268 --tool &tool &name of tool to test; identifies init, test subdir\cr
1269 --verbose &verbose &verbosity level. As option, use multiple times;
1270 as variable, set a number, 0 or greater\cr
1271 --target &target\_triplet
1272 &The canonical configuration string for the target.\cr
1273 --host &host\_triplet &The canonical configuration string for the host.\cr
1274 --build &build\_triplet &The canonical configuration string for the
1279 @node Master Config File, Local Config File, Config Values, Customizing
1280 @subsection Master Config File
1281 @cindex master @file{site.exp}
1282 @cindex @file{site.exp} for all of DejaGnu
1283 The master config file is where all the target specific config variables
1284 get set for a whole site get set. The idea is that for a centralized
1285 testing lab where people have to share a target between multiple
1286 developers. There are settings for both remote targets and remote hosts.
1287 Here's an example of a Master Config File (also called the Global config
1288 file) for a @emph{canadian cross}. A canadian cross is when you build
1289 and test a cross compiler on a machine other than the one it's to be
1292 Here we have the config settings for our California office. Note that
1293 all config values are site dependant. Here we have two sets of values
1294 that we use for testing m68k-aout cross compilers. As both of these
1295 target boards has a different debugging protocol, we test on both of
1303 case "$target_triplet" in @{
1305 set target_abbrev unix
1307 @{ "m68*-unknown-aout" @} @{
1308 set target_abbrev "rom68k"
1309 # IDP target # IDP board with rom68k monitor
1310 set target_info(idp,name) "idp"
1311 set target_info(idp,ldflags) "-Tidp.ld"
1312 set target_info(idp,config) m68k-unknown-aout
1313 set target_info(idp,cflags) ""
1314 set target_info(idp,connect) telnet
1315 set target_info(idp,target) "s7"
1316 set target_info(idp,serial) "tstty12"
1317 set target_info(idp,netport) "truckin:1007"
1318 set target_info(idp,baud) "9600"
1319 # MVME target # Motorola MVME 135 with BUG monitor
1320 set target_info(mvme,name) "mvme"
1321 set target_info(mvme,ldflags) "-Tmvme.ld"
1322 set target_info(mvme,config) m68k-unknown-aout
1323 set target_info(mvme,cflags) ""
1324 set target_info(mvme,connect) telnet
1325 set target_info(mvme,target) "s4"
1326 set target_info(mvme,serial) "tstty8"
1327 set target_info(mvme,netport) "truckin:1004"
1328 set target_info(mvme,baud) "9600"
1334 In this case, we have support for several remote hosts for
1335 our m68k-aout cross compiler. Typically the remote Unix hosts run
1336 DejaGnu locally, but we also use them for debugging the testsuites when
1337 we find problems in running on remote hosts. Expect won't run on NT, so
1338 DejaGnu is run on the local build machine, and it'll connect to the NT
1339 host and run all the tests for this cross compiler on that host.
1343 case "$host_triplet" in @{
1346 "i?86-*-linux*" @{ # Linux host
1347 set target_info(host,name) "linux-host"
1348 set target_info(host,config) $host_triplet
1349 set target_info(host,connect) rlogin
1350 set target_info(host,target) chinadoll
1352 "i?86-*-winnt # NT host
1353 set target_info(host,name) "nt-host"
1354 set target_info(host,config) i386-unknown-winnt
1355 set target_info(host,connect) telnet
1356 set target_info(host,target) ripple
1358 "hppa*-hp-hpux*" @{ # HP-UX host
1359 set target_info(host,name) "hpux-host"
1360 set target_info(host,config) $host_triplet
1361 set target_info(host,connect) rlogin
1362 set target_info(host,target) slipknot
1364 "sparc-sun-sunos*" @{ # SunOS (sun4)
1365 set target_info(host,name) "sunos-host"
1366 set target_info(host,config) $host_triplet
1367 set target_info(host,connect) rlogin
1368 set target_info(host,target) darkstar
1374 @node Local Config File, Personal Config File, Master Config File, Customizing
1375 @subsection Local Config File
1376 @cindex local @file{site.exp}
1377 @cindex @file{site.exp} for each tool
1378 It is usually more convenient to keep these ``manual overrides'' in the
1379 @file{site.exp} local to each test directory, rather than in the
1380 ``master'' @file{site.exp} in the DejaGnu library.
1382 All local @file{site.exp} usually files have two sections, separated by
1383 comment text. The first section is the part that is generated by
1384 @code{make}. It is essentially a collection of Tcl variable definitions
1385 based on @file{Makefile} environment variables. Since they are generated
1386 by @code{make}, they contain the values as specified by
1387 @code{configure}. (You can also customize these values by using the
1388 @samp{--site} option to @code{configure}.) In particular, this section
1389 contains the @file{Makefile} variables for host and target configuration
1390 data. Do not edit this first section; if you do, your changes are replaced
1391 next time you run @code{make}.
1393 The first section starts with:
1397 ## these variables are automatically generated by make ##
1398 # Do not edit here. If you wish to override these values
1399 # add them to the last section
1403 In the second section, you can override any default values (locally to
1404 DejaGnu) for all the variables. The
1405 second section can also contain your preferred defaults for all the
1406 command line options to @code{runtest}. This allows you to easily
1407 customize @code{runtest} for your preferences in each configured
1408 test-suite tree, so that you need not type options repeatedly on the
1409 command line. (The second section may also be empty, if you do not wish
1410 to override any defaults.)
1412 The first section ends with this line:
1416 ## All variables above are generated by configure. Do Not Edit ##
1420 You can make any changes under this line. If you wish to redefine a
1421 variable in the top section, then just put a duplicate value in this
1422 second section. Usually the values defined in this config file are
1423 related to the configuration of the test run. This is the ideal place to
1424 set the variables @code{host_triplet}, @code{build_triplet},
1425 @code{target_triplet}. All other variables are tool dependant. ie for
1426 testing a compiler, the value for @var{CC} might be set to a freshly
1427 built binary, as opposed to one in the user's path.
1429 @node Personal Config File, , Local Config File, Customizing
1430 @subsection Personal Config File
1431 @cindex personal config @file{site.exp}
1432 @cindex @file{site.exp} for each person
1433 The personal config file is used to customize @code{runtest's} behaviour
1434 for each person. It's typically used to set the user prefered setting
1435 for verbosity, and any experimental Tcl procedures. My personal
1436 @file{~/.dejagnurc} file looks like:
1441 set RLOGIN /usr/ucb/rlogin
1442 set RSH /usr/ucb/rsh
1446 Here I set @code{all_flag} so I see all the test cases that PASS along
1447 with the ones that FAIL. I also set @var{RLOGIN} and @code{RSH} to the
1448 BSD version. I have @code{kerberos} installed, and when I rlogin to a
1449 target board, it usually isn't supported. So I use the non secure
1450 versions of these programs rather than the default that's in my path.
1453 @chapter The DejaGnu Implementation
1454 @cindex operating principles
1455 @cindex internal details
1457 DejaGnu is entirely written in @code{expect}, which uses Tcl as a
1458 command language. @code{expect} serves as a very programmable shell;
1459 you can run any program, as with the usual Unix command shells---but
1460 once the program is started, your @code{expect} script has fully
1461 programmable control of its input and output. This does not just apply
1462 to the programs under test; @code{expect} can also run any auxiliary
1463 program, such as @code{diff} or @code{sh}, with full control over its
1466 DejaGnu itself is merely a framework for the set of test suites
1467 distributed separately for each @sc{gnu} tool. Future releases of
1468 @sc{gnu} tools will include even more tests, developed throughout the
1469 free software community.
1472 @code{runtest} is the glue to tie together and manage the test scripts.
1473 The @code{runtest} program is actually a simple Bourne shell script that
1474 locates a copy of the @code{expect} shell and then starts the main Tcl
1475 code, @code{runtest.exp}. @code{runtest.exp} itself has these essential
1480 Parse the command line options, load the library files, and load the
1481 default configuration files.
1484 Locating the individual test scripts. @code{runtest.exp} locates the tests
1485 by exploiting a straightforward naming convention based on the string
1486 you specify with the @samp{--tool} option.
1489 Providing an extended test environment, by defining additional Tcl
1490 procedures beyond those already in @code{expect}.
1493 Locating target-dependent functions, to standardize the test environment
1494 across a wide variety of test platforms.
1498 * Names:: Conventions for using tool names
1499 * Init Module:: Initialization module
1500 * DejaGnu Builtins:: DejaGnu provides these Tcl procedures
1501 * Target Dependent:: Procedures supplied by the init module
1502 * Cross Targets:: Remote targets supported
1503 * Input Files:: The files DejaGnu depends on
1504 * Output Files:: The files DejaGnu produces
1508 @section Conventions for using tool names
1510 @cindex @code{--tool} and naming conventions
1511 @cindex tool names and naming conventions
1512 @cindex naming conventions
1513 DejaGnu uses @samp{$tool}, the name of the tool under test, to tie
1514 together the testing configuration in a straightforward but flexible
1515 way. If there is only one testsuite for a particular application, then
1516 @samp{$tool} is optional.
1518 @samp{$tool} is @emph{not} used to invoke the tool, since sites that run
1519 multiple configurations of a particular tool often call each
1520 configuration by a different name. @code{runtest} uses the
1521 configuration-dependent variables captured in @file{site.exp} to
1522 determine how to call each tool.
1524 @cindex directory names and @code{--tool}
1525 @cindex test directories, naming
1526 @code{runtest} uses tool names to find directories containing tests.
1527 @code{runtest} scans the source directory (specified with
1528 @code{--srcdir}) for all directories whose names start with the tool
1529 name. It is a common practice to put a period after the tool part of the
1530 name. For instance, directories that start with
1531 @samp{g++.} contain @sc{g++} tests. To add a new test, just put it in
1532 any directory (create an entirely new directory, if you wish) whose name
1533 follows this convention.
1535 @cindex @code{exp} filename suffix
1536 @cindex test filename
1537 @cindex filename for test files
1538 A test is any file in an appropriately named subdirectory whose name
1539 ends in @samp{.exp} (the conventional way of naming @code{expect}
1540 scripts). These simple naming conventions make it as simple as possible
1541 to install new tests: all you must do is put the test in the right
1544 @cindex order of tests
1545 @cindex tests, running order
1546 @code{runtest} sorts the tests in each subdirectory by name (using the
1547 Tcl @code{lsort} command) and runs them in the resulting order.
1550 @section Initialization module
1551 @cindex tool initialization
1552 @cindex setting up targets
1554 @c FIXME! should this node be merged with "Target dependent"?
1556 @cindex init file, purpose
1557 @cindex starting interactive tools
1558 @cindex initialization
1559 The initialization module (or ``init file'') has two purposes: to
1560 provide tool and target dependent procedures, and to start up an
1561 interactive tool to the point where it is ready to operate. The latter
1562 includes establishing communications with the target. All the tests for
1563 interactive programs assume that the tool is already running and
1564 communicating. Initialization modules for non-interactive programs may
1565 only need to supply the support functions.
1567 @cindex init file name
1568 @cindex name, initialization module
1569 Each test suite directory must contain (in its @file{config}
1570 subdirectory) a separate initialization module for each target. The
1571 appropriate init file is can be named several ways. The prefered name is
1572 the @emph{os} part of the canonical configuration name with @code{.exp}
1573 as the suffix. An example would be that for an @code{m68k-coff} system,
1574 the @code{target_os} part would be @code{coff}. The next way is for
1575 system where there are short filenames, or a shortcut is desired to
1576 refer to the OS name for that target. This is uses the value of
1577 @code{$target_abbrev} rather than the @code{target_os}.
1579 The final file looked for is simply @file{default.exp}. If there is only
1580 one operating system to support, then this file can be used. It's main
1581 purpose is to offer some support for new operating systems, or for
1582 unsupported cross targets. The last file looked for is
1583 @file{unknown.exp}. This is usually limited to error handling for
1584 unsupported targets. It's whole contents is typically.
1588 perror "Sorry, there is no support for this target"
1593 At the beginning of the init file, you must first determine the proper
1594 executable name of the tool to execute, since the actual name of the
1595 tool to be tested my vary from system to system. Here's an example
1596 for the @sc{GNU} C compiler.
1601 # look for the archiver ar
1602 if ![info exists AR] @{
1603 set AR [findfile $base_dir/../../binutils/ar $base_dir/../../binutils/ar [tr
1605 verbose "AR defaulting to $AR" 2
1610 if ![info exists CFLAGS] then @{
1616 It is always a good idea to first check the variable, and only set it if
1617 it has not yet been defined. Often the proper value of @code{AR} is set
1618 on the command line that invokes @file{runtest}.
1621 The @code{findfile} procedure takes as it's first argument a file name
1622 to look for. The second argument is returned if the file is found, and
1623 the third argument is returned if the file is not found. @code{base_dir}
1624 is set internally by DejaGnu to the top level directory of the object
1628 The @code{transform} procedure takes as its argument the native name of
1629 a tool (such as @samp{gcc} for the compiler), and returns the name as
1630 configured for that tool in the current installation. (For example, a
1631 cross-compiling version of @sc{gnu} CC that generates MIPS code may be
1632 installed with a name like @code{mips-idt-ecoff-gcc}.)
1634 In a test running native, writing the Tcl code for initialization is
1635 usually quite simple. For cross configurations, however, more elaborate
1636 instructions are usually needed to describe how to talk to a remote
1639 Each initialization module defines up to four procedures with standard
1640 names and purposes. The names of these procedures begin with
1641 @samp{$tool}, the string that identifies tests for a particular tool:
1642 @code{$tool_start}, @code{$tool_load}, @code{$tool_exit}, and
1643 @code{$tool_version}. For example, the start procedure for @sc{gdb} is
1644 called @code{gdb_start}. (Since start procedures are used differently
1645 for batch and interactive tools, however, @code{runtest} itself never
1646 calls the start procedure. Init files for interactive tools are
1647 expected to end by running the start procedure.)
1649 @cindex utilities, loading from init file
1650 @cindex defaults, setting in init file
1651 The initialization module is also a good place to call @code{load_lib}
1652 to get any collections of utility procedures meant for a family of test
1653 cases, and to set up default values for any additional Tcl variables
1654 needed for a specific set of tests.
1656 @xref{Target Dependent,,Target dependent procedures}, for full
1657 descriptions of these procedures.
1659 @node DejaGnu Builtins
1660 @section DejaGnu procedures
1661 @cindex built in procedures, DejaGnu
1663 DejaGnu provides these Tcl procedures for use in test scripts.
1664 You can also use any standard @code{expect} or Tcl function. These
1665 procedures are stored in libraries, which DejaGnu loads at
1666 runtime. Here's explanation of the library procedures that get loaded at
1667 runtime. All other librarys are optional, and need to be loaded by the
1671 * framework.exp:: Core Internal Procedures.
1672 * remote.exp:: Procedures for remote communication.
1673 * utils.exp:: Utility procedures.
1674 * target.exp:: Cross target procedures.
1675 * debugger.exp:: Procedures for debugging your Tcl code.
1678 @node framework.exp, remote.exp, ,DejaGnu Builtins
1679 @subsection Core Internal Procedures
1680 @cindex Core Internal Procedures
1682 @xref{Posix,,A POSIX conforming test framework}, for more detailed
1683 explanations of the test outcomes (@samp{FAIL}, @samp{PASS},
1684 @samp{UNTESTED}, @samp{UNRESOLVED}, @samp{UNSUPPORTED}).
1687 @item perror "@var{string} @var{number}"
1688 @cindex test case, ERROR in
1690 Declares a severe error in the testing framework itself.
1691 @code{perror} writes in the log files a message beginning with
1692 @samp{ERROR}, appending the argument @var{string}. If the optional
1693 @var{number} is supplied, then this is used to set the internal count of
1694 errors to that value.
1696 As a side effect, @code{perror} also changes the effect of the next
1697 @code{pass} or @code{fail} command: the test outcome becomes
1698 @samp{UNRESOLVED}, since an automatic @samp{PASS} or @samp{FAIL} cannot
1699 be trusted after a severe error in the test framework. If the optional
1700 numeric value is @samp{0}, then there are no further side effects to
1701 calling this function, and the following test outcome doesn't become
1702 @samp{UNRESOLVED}. This can be used for errors with no known side
1705 @item warning "@var{string} @var{number}"
1706 @cindex test case, WARNING in
1708 Declares detection of a minor error in the test case itself.
1709 @code{warning} writes in the log files a message beginning with
1710 @samp{WARNING}, appending the argument @var{string}. Use @code{warning}
1711 rather than @code{error} for cases (such as communication failure
1712 to be followed by a retry) where the test case can recover from the
1713 error. If the optional @var{number} is supplied, then this is used to
1714 set the internal count of warnings to that value.
1716 As a side effect, @code{warning_threshold} or more calls to
1717 @code{warning} in a single test case also changes the effect of the next
1718 @code{pass} or @code{fail} command: the test outcome becomes
1719 @samp{UNRESOLVED} since an automatic @samp{PASS} or @samp{FAIL} may not
1720 be trustworthy after many warnings. If the optional numeric value is
1721 @samp{0}, then there are no further side effects to calling this
1722 function, and the following test outcome doesn't become
1723 @samp{UNRESOLVED}. This can be used for errors with no known side
1726 @item note "@var{string}"
1727 @cindex test case, informational messages
1729 Appends an informational message to the log file.
1730 @code{note} writes in the log files a message beginning with
1731 @samp{NOTE}, appending the argument @var{string}. Use @code{note}
1732 sparingly. @code{verbose} should be used for most such messages,
1733 but in cases where a message is needed in the log file regardless of
1734 the verbosity level use @code{note}.
1736 @item pass "@var{string}"
1737 @cindex test case, declaring success
1738 Declares a test to have passed. @code{pass} writes in the
1739 log files a message beginning with @samp{PASS} (or @code{XPASS}, if
1740 failure was expected), appending the argument @var{string}.
1742 @item fail "@var{string}"
1743 @cindex test case, declaring failure
1744 Declares a test to have failed. @code{fail} writes in the
1745 log files a message beginning with @samp{FAIL} (or @code{XFAIL}, if
1746 failure was expected), appending the argument @var{string}.
1748 @item unresolved "@var{string}"
1749 @cindex test case, declaring ambiguity
1750 Declares a test to have an unresolved outcome. @code{unresolved} writes
1751 in the log file a message beginning with @samp{UNRESOLVED}, appending
1752 the argument @var{string}. This usually means the test did not execute
1753 as expected, and a human being must go over results to determine if it
1754 passed or failed (and to improve the test case).
1756 @item untested "@var{string}"
1757 @cindex test case, declaring no test
1758 Declares a test was not run. @code{untested} writes in the log file a
1759 message beginning with @samp{UNTESTED}, appending the argument
1760 @var{string}. For example, you might use this in a dummy test whose
1761 only role is to record that a test does not yet exist for some feature.
1763 @item unsupported "@var{string}"
1764 @cindex test case, declaring no support
1765 Declares that a test case depends on some facility that does not exist
1766 in the testing environment. @code{unsupported} writes in the log file a
1767 message beginning with @samp{UNSUPPORTED}, appending the argument
1770 @item get_warning_threshold
1771 @cindex test case, WARNING threshold
1772 Returns the current value of @code{warning_threshold}.
1773 The default value is 3.
1775 @item set_warning_threshold @var{threshold}
1776 @cindex test case, WARNING threshold
1777 Sets the value of @code{warning_threshold}.
1778 A value of @code{0} disables it: calls to @code{warning} will not turn
1779 a @samp{PASS} or @samp{FAIL} into an @samp{UNRESOLVED}.
1781 @item transform "@var{toolname}"
1782 @cindex transform tool name
1783 @cindex installed tool name
1784 @cindex tool name, as installed
1785 @cindex name transformations
1786 Generates a string for the name of a tool as it was configured and
1787 installed, given its native name (as the argument @var{toolname}).
1788 This makes the assumption that all tools are installed using the same
1789 naming conventions: it extrapolates from the invocation name for
1790 @file{runtest}. For example, if you call @code{runtest} as
1791 @file{m68k-vxworks-runtest}, the result of @w{@samp{ transform "gcc" }}
1792 is @samp{m68k-vxworks-gcc}.
1794 @item ishost "@var{host}"
1795 @cindex host configuration test
1796 Tests for a particular @emph{host} environment. If the currently
1797 configured host matches the argument string, the result is @code{1};
1798 otherwise the result is @code{0}. @var{host} must be a full three-part
1799 @code{configure} host name; in particular, you may not use the shorter
1800 nicknames supported by @code{configure} (but you can use wildcard
1801 characters, using shell syntax, to specify sets of names).
1803 @item istarget "@var{target}"
1804 @cindex target configuration test
1805 Tests for a particular @emph{target} environment. If the currently
1806 configured target matches the argument string, the result is @code{1};
1807 otherwise the result is @code{0}. @var{target} must be a full
1808 three-part @code{configure} target name; in particular, you may not use
1809 the shorter nicknames supported by @code{configure} (but you can use
1810 wildcard characters, using shell syntax, to specify sets of names). If it is
1811 passed a @code{NULL} string, then it returns the name of the build
1812 canonical configuration.
1814 @item isbuild "@var{host}"
1815 @cindex build host configuration test
1816 Tests for a particular @emph{build host} environment. If the currently
1817 configured host matches the argument string, the result is @code{1};
1818 otherwise the result is @code{0}. @var{host} must be a full three-part
1819 @code{configure} host name; in particular, you may not use the shorter
1820 nicknames supported by @code{configure} (but you can use wildcard
1821 characters, using shell syntax, to specify sets of names). If it is
1822 passed a @code{NULL} string, then it returns the name of the build
1823 canonical configuration.
1825 item is3way "@var{host}"
1826 @cindex canadian cross configuration test
1827 Tests for a canadian cross. This is when the tests will be run on a
1828 remotly hosted cross compiler. If it is a canadian cross, then the
1829 result is @code{1}; otherwise the result is @code{0}.
1832 @cindex native configuration test
1833 Tests whether the current configuration has the same host and target.
1834 When it runs in a @emph{native} configuration this procedure returns a
1835 @code{1}; otherwise it returns a @code{0}.
1837 @item load_lib "@var{library-file}"
1838 @cindex load library file
1839 Loads the file @var{library-file} by searching a fixed path built into
1840 @code{runtest}. If DejaGnu has been installed, it looks in a path
1841 starting with the installed library directory. If you are running
1842 DejaGnu directly from a source directory, without first running
1843 @samp{make install}, this path defaults to the current directory. In
1844 either case, it then looks in the current directory for a directory
1845 called @code{lib}. If there are duplicate definitions, the last one
1846 loaded takes precedence over the earlier ones.
1848 @item setup_xfail "@var{config} @r{[}@var{bugid}@r{]}"
1849 @c two spaces above to make it absolutely clear there's whitespace---a
1850 @c crude sort of italic correction!
1851 @cindex test case, expecting failure
1852 @cindex failure, expected
1853 @cindex expected failure
1854 Declares that the test is expected to fail on a particular set of
1855 configurations. The @var{config} argument must be a list of full
1856 three-part @code{configure} target name; in particular, you may not use
1857 the shorter nicknames supported by @code{configure} (but you can use the
1858 common shell wildcard characters to specify sets of names). The
1859 @var{bugid} argument is optional, and used only in the logging file
1860 output; use it as a link to a bug-tracking system such as @sc{gnats}
1861 (@pxref{Overview,, Overview, gnats.info, Tracking Bugs With GNATS}).
1863 @cindex @code{XFAIL}, producing
1864 @cindex @code{XPASS}, producing
1865 Once you use @code{setup_xfail}, the @code{fail} and @code{pass}
1866 procedures produce the messages @samp{XFAIL} and @samp{XPASS}
1867 respectively, allowing you to distinguish expected failures (and
1868 unexpected success!) from other test outcomes.
1870 @emph{Warning:} you must clear the expected failure after using
1871 @code{setup_xfail} in a test case. Any call to @code{pass} or
1872 @code{fail} clears the expected failure implicitly; if the test has some
1873 other outcome, e.g. an error, you can call @code{clear_xfail} to clear
1874 the expected failure explicitly. Otherwise, the expected-failure
1875 declaration applies to whatever test runs next, leading to surprising
1878 @item check_conditional_xfail @var{message} @var{targets} @var{includes} @var{excludes}
1879 @cindex test case, expecting a conditional failure
1880 @cindex failure, conditional expected
1881 @cindex conditional expected failure
1883 This procedure adds a condition xfail, based on compiler options used to
1884 create a test case executable. If an include options is found in the
1885 compiler flags, and it's the right architecture, it'll trigger an
1886 XFAIL. Otherwise it'll produce an ordinary FAIL. You can also specify
1887 flags to exclude. This makes a result be a FAIL, even if the included
1888 options are found. To set the conditional, set the variable
1889 @var{compiler_conditional_xfail_data} to the fields "[message string] [targets
1890 list] [includes list] [excludes list]" (descriptions below). This is the
1891 checked at pass/fail decision time, so there is no need to call the
1892 procedure yourself, unless you wish to know if it gets triggered. After
1893 a pass/fail, the variable is reset, so it doesn't effect other tests.
1899 is the message to print with the normal test result
1902 is a string with the targets to activate this conditional on.
1905 is a list of sets of options to search for in the compiler options to
1906 activate this conditional. If any set of the options matches, then this
1907 conditional is true.
1910 is a list of sets of options to search for in the compiler options to
1911 activate this conditional. If any set of the options matches,
1912 (regardless of whether any of the include sets match) then this
1913 conditional is de-activated.
1920 if the conditional is true
1922 if the conditional is false
1925 An example of setting the variable would be:
1929 set compiler_conditional_xfail_data @{@ \
1930 "I sure wish I knew why this was hosed" \
1931 "sparc*-sun*-* *-pc-*-*" \
1932 @{@"-Wall -v" "-O3"@}@ \
1933 @{@"-O1" "-Map" @}@ \
1938 What this does is it matches only for these two targets if "-Wall -v" or
1939 "-O3" is set, but neither "-O1" or "-Map" is set.
1941 For a set to match, the options specified are searched for independantly of
1942 each other, so a "-Wall -v" matches either "-Wall -v" or "-v -Wall". A space
1943 seperates the options in the string. Glob-style regular expressions are also
1946 @item clear_xfail @var{config}
1947 @cindex cancelling expected failure
1948 @cindex expected failure, cancelling
1949 Cancel an expected failure (previously declared with @code{setup_xfail})
1950 for a particular set of configurations. The @var{config} argument is a
1951 list of configuration target names. It is only necessary to call
1952 @code{clear_xfail} if a test case ends without calling either
1953 @code{pass} or @code{fail}, after calling @code{setup_xfail}.
1955 @item verbose @r{[}-log@r{]} @r{[}-n@r{]} @r{[}--@r{]} "@var{string}" @var{number}
1956 @cindex @code{verbose} builtin function
1957 Test cases can use this function to issue helpful messages depending on
1958 the number of @samp{--verbose} options on the @code{runtest} command
1959 line. It prints @var{string} if the value of the variable
1960 @code{verbose} is higher than or equal to the optional @var{number}. The
1961 default value for @var{number} is 1. Use the optional @samp{-log} argument
1962 to cause @var{string} to always be added to the log file, even if it won't
1963 be printed. Use the optional @samp{-n} argument to print @var{string}
1964 without a trailing newline. Use the optional @samp{--} argument if
1965 @var{string} begins with "-".
1970 @node remote.exp, utils.exp, framework.exp, DejaGnu Builtins
1971 @subsection Remote Communication Procedures
1974 @kindex lib/remote.exp
1975 @cindex remote connection procedures
1976 @cindex communications procedures
1977 @file{lib/remote.exp} defines these functions, for establishing and
1978 managing communications:
1980 @emph{Procedures to establish a connection:} Each of these procedures
1981 tries to establish the connection up to three times before returning.
1982 Warnings (if retries will continue) or errors (if the attempt is
1983 abandoned) report on communication failures. The result for any of
1984 these procedures is either @code{-1}, when the connection cannot be
1985 established, or the spawn ID returned by the @code{expect} command
1988 It use the value of the @code{connect} field in the @code{target_info}
1989 array (was @code{connectmode} as the type of connection to make. Current
1990 supported connection types are tip, kermit, telnet, rsh, rlogin, and
1991 netdata. If the @code{--reboot} option was used on the runtest command
1992 line, then the target is rebooted before the connection is made.
1996 @item remote_open @var{type}
1997 @cindex Opening a remote connection
1998 @emph{Remote Connection Procedure.} This is passed @emph{host} or
1999 @emph{target}. Host or target refers to whether it is a connection to a
2000 remote target, or a remote host. This opens the connection to the
2001 desired target or host using the default values in the configuration
2002 system. It returns that @code{spawn_id} of the process that manages the
2003 connection. This value can be used in @code{expect} or @code{exp_send}
2004 statements, or passed to other procedures that need the connection
2005 process's id. This also sets the @code{fileid} field in the
2006 @code{target_info} array.
2009 @item remote_close @var{shellid}
2010 @cindex Closing a remote connection
2011 @emph{shellid} is value returned by a call to @code{remote_open}. This
2012 closes the connection to the target so resources can be used by
2013 others. This parameter can be left off if the @code{fileid} field in the
2014 @code{target_info} array is set.
2016 @item telnet @var{hostname} @var{port}
2017 @itemx rlogin @var{hostname}
2018 @itemx rsh @var{hostname}
2019 @cindex IP network procedures
2020 @cindex network (IP) procedures
2021 @emph{IP network procedures.} @var{hostname} refers to the IP address or
2022 name (for example, an entry in @file{/etc/hosts}) for this target. The
2023 procedure names reflect the Unix utility used to establish a
2024 connection. The optional @var{port} is used to specify the IP port
2025 number. The value of the @code{netport} field in the @code{target_info}
2026 array is used. (was @code{$netport}) This value has two parts, the
2027 hostname and the port number, seperated by a @emph{:}. If @code{host} or
2028 @code{target} is used in the @code{hostname} field, than the config
2029 array is used for all information.
2031 @item tip @var{port}
2032 @cindex serial line connection, @code{tip}
2033 @emph{Serial line procedure.} Connect using the Unix utility @code{tip}.
2034 @var{port} must be a name from the @code{tip} configuration file
2035 @file{/etc/remote}. Often, this is called @samp{hardwire}, or something
2036 like @samp{ttya}. This file holds all the configuration data for
2037 the serial port. The value of the @code{serial} field in the
2038 @code{target_info} array is used. (was @code{$serialport}) If
2039 @code{host} or @code{target} is used in the @code{port} field, than
2040 the config array is used for all information.
2042 @item kermit @var{port} @var{bps}
2043 @cindex serial line connection, @code{kermit}
2044 @emph{Serial line procedure.} Connect using the program @code{kermit}.
2045 @var{port} is the device name, e.g. @file{/dev/ttyb}. @var{bps} is
2046 the line speed to use (in bits per second) for the connection. The value
2047 of the @code{serial} field in the @code{target_info} array is used. (was
2048 @code{$serialport}) If @code{host} or @code{target} is used in the
2049 @code{port} field, than the config array is used for all information.
2054 @emph{Procedures to manage a connection:}
2057 @item tip_download @var{spawnid} @var{file}
2058 @cindex download, @code{tip}
2059 @cindex serial download, @code{tip}
2060 Download @file{@var{file}} to the process @var{spawnid} (the value
2061 returned when the connection was established), using the @code{~put}
2062 command under @code{tip}. Most often used for single board computers
2063 that require downloading programs in @sc{ascii} S-records. Returns
2064 @code{1} if an error occurs, @code{0} otherwise.
2066 @item exit_remote_shell @var{spawnid}
2067 @cindex terminating remote connection
2068 @cindex remote connection, ending
2069 Exits a remote process started by any of the connection procedures.
2070 @var{spawnid} is the result of the connection procedure that started the
2073 @item download @var{file} @r{[} @var{spawnid} @r{]}
2074 @cindex download a file
2075 After you establish a connection to a target, you can download programs
2076 using this command. @code{download} reads in @var{file} (object code in
2077 S-record format) and writes it to the device controlling this
2078 @var{spawnid}. (From the point of view of the target, the S-record file
2079 comes in via standard input.)
2081 If you have more than one target active, you can use the optional argument
2082 @var{spawnid} to specify an alternative target (the default is the most
2083 recently established @var{spawnid}.)
2087 @node utils.exp, target.exp, remote.exp, DejaGnu Builtins
2088 @subsection Utility Procedures
2091 @kindex lib/utils.exp
2092 @file{lib/utils.exp} defines these utility procedures:
2095 @item getdirs @var{dir}
2096 @itemx getdirs @var{dir} @var{pattern}
2097 @cindex directories matching a pattern
2098 @cindex pattern match, directory
2099 Returns a list of all the directories in the single directory @var{dir}
2100 that match @var{pattern}. If you do not specify @var{pattern},
2101 @code{getdirs} assumes @samp{*}. You may use the common shell wildcard
2102 characters in @var{pattern}. If no directories match the pattern, then a
2103 @code{NULL} string is returned.
2105 @item find @var{dir} @var{pattern}
2106 @cindex files matching a pattern
2107 @cindex pattern match, filenames
2108 Search for files whose names match @var{pattern} (using shell wildcard
2109 characters for filename expansion). Search subdirectories recursively,
2110 starting at @var{dir}. The result is the list of files whose names
2111 match; if no files match, the result is empty. Filenames in the result
2112 include all intervening subdirectory names. If no files match the
2113 pattern, then a @code{NULL} string is returned.
2115 @item which @var{binary}
2117 Searches the execution path for an executable file @var{binary}, like
2118 the the BSD @code{which} utility. This procedure uses the shell
2119 environment variable @samp{PATH}. It returns @code{0} if the binary is
2120 not in the path, or if there is no @samp{PATH} environment variable. If
2121 @var{binary} is in the path, it returns the full path to @var{binary}.
2123 @item grep @var{filename} @var{regexp}
2124 @item grep @var{filename} @var{regexp} line
2125 @cindex regular expression, file contents
2126 @cindex searching file contents
2127 Search the file called @var{filename} (a fully specified path) for lines
2128 that contain a match for regular expression @var{regexp}. The result is
2129 a list of all the lines that match. If no lines match, the result is an
2130 empty string. Specify @var{regexp} using the standard regular
2131 expression style used by the Unix utility program @code{grep}.
2133 Use the optional third argument @samp{line} to start lines in the result
2134 with the line number in @var{filename}. (This argument is simply an
2135 option flag; type it just as shown---@samp{line}.)
2137 @item diff @var{filename} @var{filename}
2138 @cindex finding file differences
2139 @cindex comparing files
2140 Compares the two files and returns a 1 if they match, or a 0 if they
2141 don't. If @code{verbose} is set, then it'll print the differences to the
2144 @item slay @var{name}
2145 @cindex slaying processes
2146 This look in the process table for @var{name} and send it a unix
2147 @code{SIGINT}, killing the process.
2149 @item absolute @var{path}
2150 @cindex converting relative paths to absolute
2151 This procedure takes the relative @var{path}, and converts it to an
2154 @item psource @var{filename}
2155 @cindex sourcing Tcl files
2156 This sources the file @var{filename}, and traps all errors. It also
2157 ignores all extraneous output. If there was an error it returns a 1,
2158 otherwise it returns a 0.
2160 @item prune @var{list} @var{pattern}
2161 @cindex list, pruning
2162 Remove elements of the Tcl list @var{list}. Elements are fields
2163 delimited by spaces. The result is a copy of @var{list}, without any
2164 elements that match @var{pattern}. You can use the common shell
2165 wildcard characters to specify @var{pattern}.
2167 @item setenv @var{var} @var{val}
2168 @cindex setting environment variables
2169 Sets the variable @var{var} to the value @var{val}.
2171 @item unsetenv @var{var}
2172 @cindex unsetting environment variables
2173 Unsets the environment variable @var{var}
2175 @item getenv @var{var}
2176 @cindex getting environment variables
2177 returns the value of @var{var} in the environment if it exists,
2178 otherwise it returns @code{NULL}.
2180 @item runtest_file_p @var{runtests} @var{testcase}
2181 @cindex selecting a range of tests
2182 @cindex tests, running specifically
2183 Search @var{runtests} for @var{testcase} and return 1 if found, 0 if not.
2184 @var{runtests} is a list of two elements. The first is the pathname of
2185 the testsuite expect script running. The second is a copy of what was
2186 on the right side of the @code{=} if @samp{foo.exp="@dots{}"} was specified,
2187 or an empty string if no such argument is present.
2188 This is used by tools like compilers where each testcase is a file.
2190 @item prune_system_crud @var{system} @var{text}
2191 @cindex pruning system output, examining program output
2192 For system @var{system}, delete text the host or target operating system might
2193 issue that will interfere with pattern matching of program output in
2194 @var{text}. An example is the message that is printed if a shared library
2200 @node target.exp, debugger.exp, utils.exp, DejaGnu Builtins
2201 @subsection Cross target procedure
2204 @kindex lib/target.exp
2205 @file{lib/target.exp} defines these utility procedures:
2209 @item push_target @emph{name}
2210 @cindex set current target
2211 This makes the target named @emph{name} be the current target
2212 connection. The value of @emph{name} is an index into the
2213 @code{target_info} array and is set in the global config file.
2216 @cindex unset current target
2217 This unsets the current target connection.
2220 @cindex lists supported targets
2221 This lists all the supported targets for this architecture.
2223 @item push_host @emph{name}
2224 @cindex set current host
2225 This makes the host named @emph{name} be the current remote host
2226 connection. The value of @emph{name} is an index into the
2227 @code{target_info} array and is set in the global config file.
2230 @cindex unset current host
2231 This unsets the current host connection.
2233 @c @item compile @emph{file}
2234 @cindex compile a file
2235 This invokes the compiler as set by @code{CC} to compile the file
2236 @emph{file}. The default options for many cross compilation targets are
2237 @emph{guessed} by DejaGnu, and these options can be added to by passing
2238 in more parameters as arguments to @code{compile}. Optionally, this will
2239 also use the value of the @code{cflags} field in the target config
2240 array. If the host is not the same as the build machines, then then
2241 compiler is run on the remote host using @code{execute_anywhere}.
2243 @c @item archive @emph{file}
2244 @cindex archive object files
2245 This produces an archive file. Any parameters passed to @code{archive}
2246 are used in addition to the default flags. Optionally, this will
2247 also use the value of the @code{arflags} field in the target config
2248 array. If the host is not the same as the build machines, then then
2249 archiver is run on the remote host using @code{execute_anywhere}.
2251 @c @item ranlib @emph{file}
2252 @cindex ranlib a file
2253 This generates an index for the archive file for systems that aren't
2254 POSIX yet. Any parameters passed to @code{ranlib} are used in for the
2257 @item execute_anywhere @emph{cmdline}
2258 @cindex executing commands remotely
2259 This executes the @emph{cmdline} on the proper host. This should be used
2260 as a replacement for the Tcl command @code{exec} as this version
2261 utilizes the target config info to execute this command on the build
2262 machine or a remote host. All config information for the remote host
2263 must be setup to have this command work. If this is a canadian cross,
2264 (where we test a cross compiler that runs on a different host then where
2265 DejaGnu is running) then a connection is made to the remote host and
2266 the command is executed there. It returns either @emph{REMOTERROR} (for
2267 an error) or the output produced when the command was executed. This is
2268 used for running the tool to be tested, not a test case.
2272 @node debugger.exp, , target.exp, DejaGnu Builtins
2273 @subsection Debugging Procedures
2275 @kindex debugger.exp
2276 @kindex lib/debugger.exp
2277 @file{lib/debugger.exp} defines these utility procedures:
2281 @item dumpvars @emph{expr}
2282 @cindex Print global variable values
2283 This takes a csh style regular expression (glob rules) and prints the
2284 values of the global variable names that match. It is abbreviated as
2287 @item dumplocals @emph{expr}
2288 @cindex Print local variable value
2289 This takes a csh style regular expression (glob rules) and prints the
2290 values of the local variable names that match. It is abbreviated as
2293 @item dumprocs @emph{expr}
2294 @cindex Print procedure bodies
2295 This takes a csh style regular expression (glob rules) and prints the
2296 body of all procs that match. It is abbreviated as @code{dp}
2298 @item dumpwatch @emph{expr}
2299 @cindex Print watchpoints
2300 This takes a csh style regular expression (glob rules) and prints all
2301 the watchpoints. It is abbreviated as @code{dw}.
2303 @c FIXME: finish these when the code is fixed.
2304 @c @item watcharray @emph{element} @emph{type}
2305 @c @cindex Set a watchpoint on an array
2306 @c This sets an watchpoint of the @emph{element-type} on the
2307 @c @item watchvar v null type
2308 @c @cindex Set a watchpoint on a variable
2310 @item watchunset @emph{var}
2311 @cindex Watch when a variable is unset
2312 This breaks program execution when the variable @emph{var} is unset. It
2313 is abbreviated as @code{wu}.
2315 @item watchwrite @emph{var}
2316 @cindex Watch when a variable is written
2317 This breaks program execution when the variable @emph{var} is
2318 written. It is abbreviated as @code{ww}.
2320 @item watchread @emph{var}
2321 @cindex Watch when a variable is read
2322 This breaks program execution when the variable @emph{var} is read. It
2323 is abbreviated as @code{wr}.
2325 @item watchdel @emph{watch}
2326 @cindex Delete a watchpoint.
2327 This deletes a the watchpoint for @emph{watch}. It is abbreviated as
2330 @item print @emph{var}
2331 @cindex Printing variable values
2332 This prints the value of the variable @emph{var}. It is abbreviated as
2336 @cindex Quiting DejaGnu
2337 This makes runtest exit. It is abbreviated as @code{q}.
2340 @cindex Print a backtrace
2341 This prints a backtrace of the executed Tcl commands.
2345 @node Target Dependent
2346 @section Target dependent procedures
2347 @cindex target dependent procedures
2349 @c FIXME? These may be renamed to just "start", "load", "exit", and
2350 @c "version" eventually.
2352 Each combination of target and tool requires some target-dependent
2353 procedures. The names of these procedures have a common form: the tool
2354 name, followed by an underbar @samp{_}, and finally a suffix describing
2355 the procedure's purpose. For example, a procedure to extract the
2356 version from @sc{gdb} is called @samp{gdb_version}. @xref{Init Module,,
2357 Initialization Module}, for a discussion of how DejaGnu arranges to find
2358 the right procedures for each target.
2360 @code{runtest} itself calls only two of these procedures,
2361 @code{@var{tool}_exit} and @code{@var{tool}_version}; these procedures use
2364 The other two procedures, @code{@var{tool}_start} and
2365 @code{@var{tool}_load}, are only called by the test suites themselves
2366 (or by testsuite-specific initialization code); they may take arguments
2367 or not, depending on the conventions used within each test suite.
2370 @item @var{tool}_start
2371 @cindex start procedure, tested tools
2372 Starts a particular tool. For an interactive tool,
2373 @code{@var{tool}_start} starts and initializes the tool, leaving the
2374 tool up and running for the test cases; an example is @code{gdb_start},
2375 the start function for @sc{gdb}. For a batch oriented tool,
2376 @code{@var{tool}_start} is optional; the recommended convention is to
2377 let @code{@var{tool}_start} run the tool, leaving the output in a
2378 variable called @code{comp_output}. Test scripts can then analyze
2379 @samp{$comp_output} to determine the test results. An example of this
2380 second kind of start function is @code{gcc_start}, the start function
2383 @code{runtest} itself @emph{does not call} @code{@var{tool}_start}. The
2384 initialization module @file{@var{tool}_init.exp} must call
2385 @code{@var{tool}_start} for interactive tools; for batch-oriented tools,
2386 each individual test script calls @code{@var{tool}_start} (or makes
2387 other arrangements to run the tool).
2389 @item @var{tool}_load
2390 @cindex load procedure, tested tools
2391 Loads something into a tool. For an interactive tool, this conditions
2392 the tool for a particular test case; for example, @code{gdb_load} loads
2393 a new executable file into the debugger. For batch oriented tools,
2394 @code{@var{tool}_load} may do nothing---though, for example, the
2395 @sc{gcc} support uses @code{gcc_load} to load and run a binary on the
2396 target environment. Conventionally, @code{@var{tool}_load} leaves the
2397 output of any program it runs in a variable called @samp{exec_output}.
2398 Writing @code{@var{tool}_load} can be the most complex part of extending
2399 DejaGnu to a new tool or a new target, if it requires much communication
2400 coding or file downloading.
2402 Test scripts call @code{@var{tool}_load}.
2404 @item @var{tool}_exit
2405 @cindex exit procedure, tested tools
2406 Cleans up (if necessary) before @code{runtest} exits. For interactive
2407 tools, this usually ends the interactive session. You can also use
2408 @code{@var{tool}_exit} to remove any temporary files left over from the
2411 @code{runtest} calls @code{@var{tool}_exit}.
2413 @item @var{tool}_version
2414 @cindex version procedure, tested tools
2415 Prints the version label and number for @var{tool}. This is called by
2416 the DejaGnu procedure that prints the final summary report. The output
2417 should consist of the full path name used for the tested tool, and its
2420 @code{runtest} calls @code{@var{tool}_version}.
2423 The usual convention for return codes from any of these procedures
2424 (although it is not required by @code{runtest}) is to return @code{0} if
2425 the procedure succeeded, @code{1} if it failed, and @code{-1} if there
2426 was a communication error.
2429 @section Remote targets supported
2432 @cindex remote testing
2433 The DejaGnu distribution includes support for the following remote
2434 targets. You can set the target name and the connect mode in the
2435 @file{site.exp} file (using the Tcl variables @samp{targetname} and
2436 @samp{connectmode}, respectively), or on the @code{runtest} command line
2437 (using @samp{--name} and @samp{--connect}).
2440 @item @sc{amd} 29000, with UDI protocol
2441 Configure DejaGnu for target @samp{a29k-amd-udi}. (Cygnus
2442 @code{configure} also recognizes the abbreviation @samp{udi29k}.) Then,
2443 to run tests, use the @code{runtest} target name to specify whether you
2444 want to use a simulator, or a particular hardware board. The particular
2445 string to use with @samp{--name} will depend on your UDI setup file,
2446 @file{udi_soc} (if @file{udi_soc} is not in your working directory, the
2447 environment variable @samp{UDICONF} should contain a path to this file).
2448 For example, if your UDI setup file includes these lines:
2450 @c table "ends" *only* to allow wider example below
2454 iss AF_UNIX * isstip -r /home/gnu/29k/src/osboot/sim/osboot
2455 mon AF_UNIX * montip -t serial -baud 9600 -com /dev/ttyb
2461 @c fake out table/item into continuing w/same margin as before
2462 You can use @samp{--name iss} to run tests on the simulator, and
2463 @samp{--name mon} to run tests on the 29K hardware. See the
2464 manufacturer's manuals for more information on UDI and @file{udi_soc}.
2465 @c FIXME! Is there a better ref than "the manufacturer's manuals"?
2468 The default connect protocol is @samp{mondfe} with either back end.
2469 @code{mondfe} is the only shell DejaGnu supports for UDI targets.
2470 @code{mondfe} is an @sc{amd} specific monitor program freely available
2473 @emph{Warning:} This target requires @sc{gdb} version 4.7.2 (or
2474 greater). Earlier versions of @sc{gdb} do not fully support the
2475 @code{load} command on this target, so DejaGnu has no way to load
2476 executable files from the debugger.
2478 @item Motorola 680x0 boards, a.out or @sc{coff} object format
2479 Configure DejaGnu for any remote target matching @samp{m68k-*}.
2481 @emph{Warning:} Most @samp{m68k-*} configurations run all tests only for
2482 native testing (when the target is the same as the host). When you
2483 specify most of these targets for a cross configuration, you will only be
2484 able to use tests that run completely within the host (for example,
2485 tests of the binary utilities such as the archiver; or compiler tests
2486 that only generate code rather than running it).
2488 To run a.out or @sc{coff} binaries on a remote M68K, you must configure
2489 DejaGnu for a particular target board. @samp{m68k-abug} is an example.
2490 (In general for an embedded environment, because it does not have absolute
2491 addresses, a.out is not a good choice for output format in any case; most
2492 often S-records or Hex-32 are used instead.)
2494 @item Motorola 68K MVME 135 board running ABug boot monitor
2495 Configure for @samp{m68k-abug-aout} or @samp{m68k-abug-coff} (as a
2496 target). This boot monitor can only download S-records; therefore, the
2497 DejaGnu tests for this environment require a linker command script to
2498 convert either output format to S-records, setting the default addresses
2499 for @code{.text}, @code{.bss}, and @code{.data}.
2501 With this configuration, the default for @samp{--connect} is @samp{tip}.
2502 @samp{tip} is the only communications protocol supported for connecting
2503 to @samp{m68k-abug-*} targets. @samp{tip} uses an @sc{ascii} downloader
2504 (the @code{~put} command) to load S-records into the target board. The
2505 @samp{--name} string must be a machine name that @code{tip}
2506 understands (for example, on some @code{tip} implementations it must be
2507 an entry from the initialization file for @code{tip}; this file is
2508 sometimes called @file{/etc/remote}).
2510 See your system documentation for information on how to create new
2511 entries in @file{/etc/remote}. (Some @sc{unix} systems are distributed
2512 with at least one default entry with a name resembling @samp{hardwire};
2513 if your system has one, you can edit it, or make a modified copy with a
2514 new name.) When you have a working @file{/etc/remote} entry
2515 @var{abugtarget}, you should be able to type @samp{tip
2516 @var{abugtarget}}, and get the prompt @samp{135ABUG>} from the board.
2517 Use the same @var{abugtarget} string with @samp{runtest --name}.
2519 @item Motorola IDP board running the rom68k boot monitor
2520 @c FIXME 1: this doesn't really say anything! OK, so functionality is
2521 @c the same. Is object code the same (srecords)? Do we configure with
2522 @c the same triplets? What is the default for --connect? Is
2523 @c any comms method other than tip supported? What prompt to expect
2524 @c when tip connected?
2525 @c FIXME 2: should @code{BUG} below be @code{ABUG}?
2526 This is the same in functionality as the MVME board running the
2527 @code{BUG} boot monitor. Only the monitor commands and the addresses are
2530 @item VxWorks (Motorola 68K or Intel 960)
2531 Configure DejaGnu for either @samp{m68k-wrs-vxworks} (abbreviated
2532 @samp{vxworks68}) or @samp{i960-wrs-vxworks} (abbreviated
2533 @samp{vxworks960}). Since both targets support IP addressing, specify
2534 the network address (for example, a host name from @file{/etc/hosts})
2537 The default connect protocol is @samp{rlogin}, but you can use any of
2538 @samp{--connect rlogin}, @samp{--connect telnet}, or @samp{--connect
2541 Test scripts need no special code to load programs into these targets;
2542 since VxWorks supports NFS, all you must do is ensure test programs are
2543 on an exported filesystem.
2545 @cindex VxWorks, link with @samp{-r}
2546 When you compile for VxWorks, use the linker @samp{-r} option to make
2547 the linker output relocatable---at least if you want to use library
2548 routines. Many standard C routines are included in VxWorks; often no
2549 additional libraries are needed. See your VxWorks system documentation
2550 for additional details.
2554 @section The files DejaGnu reads
2557 The @code{runtest} program used to invoke DejaGnu is a short shell
2558 script generated by @code{make} during the configuration process. Its
2559 main task is to read the main test framework driver, @file{runtest.exp}.
2561 @file{runtest.exp}, in turn, reads @code{expect} code from certain other
2562 files, in this order:
2566 Each of the @file{site.exp} local definition files available.
2567 @xref{Customizing,,Setting @code{runtest} defaults}, for details.
2570 @file{lib/utils.exp}, a collection of utility procedures. @xref{DejaGnu
2571 Builtins,,DejaGnu Builtins}, for descriptions of these procedures.
2574 @file{lib/framework.exp}, a file of subroutines meant for @code{runtest}
2575 itself rather than for general-purpose use in both @code{runtest} and
2579 @file{debugger.exp}, Don Libes' Tcl Debugger. (See @cite{A Debugger for
2580 Tcl Applications} by Don Libes. This paper is distributed with
2581 @code{expect} in PostScript form as the file
2582 @file{expect/tcl-debug.ps}.)
2585 @file{lib/remote.exp}, a collection of subroutines meant for connecting
2589 @file{lib/target.exp}, a collection of subroutines used for the
2590 configuration systems in DejaGnu. These procedures typically manipulate
2591 or utilize the configuration system.
2594 @c FIXME! A comment in runtest.exp claims a system default is used if
2595 @c no tool-specific init file is not available; I couldn't see where
2596 @c the program flow actually does this, though---pesch 30jul1993.
2597 An initialization file @code{@var{tool}_init.exp}. @xref{Init
2598 Module,,Initialization module}, for more discussion of init files.
2601 @c This hard page break is mainly intended for smallbook formatting;
2602 @c some examples in this section come out better if this starts at a
2606 @section The files DejaGnu writes
2607 @cindex output files
2609 @code{runtest} always writes two kinds of output files: summary logs and
2610 detailed logs. The contents of both of these are determined by your
2613 For troubleshooting, a third kind of output file is useful: use
2614 @samp{--debug} to request an output file showing details of what
2615 @code{expect} is doing internally.
2618 * Summary:: Files that summarize tests
2619 * Detail:: Files that contain complete test results
2620 * Debug:: Logging expect internal actions
2624 @subsection Summary log
2627 @code{runtest} always produces a summary output file
2628 @file{@var{tool}.sum}. This summary shows the names of all test files
2629 run; for each test file, one line of output from each @code{pass}
2630 command (showing status @samp{PASS} or @samp{XPASS}) or @code{fail}
2631 command (status @samp{FAIL} or @samp{XFAIL}); trailing summary
2632 statistics that count passing and failing tests (expected and
2633 unexpected); and the full pathname and version number of the tool
2634 tested. (All possible outcomes, and all errors, are always reflected in
2635 the summary output file, regardless of whether or not you specify
2638 If any of your tests use the procedures @code{unresolved},
2639 @code{unsupported}, or @code{untested}, the summary output also
2640 tabulates the corresponding outcomes.
2642 For example, after @samp{runtest --tool binutils}, look for a summary
2643 log in @file{binutils.sum}. Normally, @code{runtest} writes this file
2644 in your current working directory; use the @samp{--outdir} option to
2645 select a different directory.
2649 Here is a short sample summary log:
2653 Test Run By rob on Mon May 25 21:40:57 PDT 1992
2655 Running ./gdb.t00/echo.exp ...
2657 Running ./gdb.all/help.exp ...
2658 PASS: help add-symbol-file
2660 PASS: help breakpoint "bre" abbreviation
2661 FAIL: help run "r" abbreviation
2662 Running ./gdb.t10/crossload.exp ...
2663 PASS: m68k-elf (elf-big) explicit format; loaded
2664 XFAIL: mips-ecoff (ecoff-bigmips) "ptype v_signed_char" signed
2667 # of expected passes 5
2668 # of expected failures 1
2669 # of unexpected failures 1
2670 /usr/latest/bin/gdb version 4.6.5 -q
2675 @subsection Detailed log
2676 @cindex detailed log
2678 @code{runtest} also saves a detailed log file @file{@var{tool}.log},
2679 showing any output generated by tests as well as the summary output.
2680 For example, after @samp{runtest --tool binutils}, look for a detailed
2681 log in @file{binutils.log}. Normally, @code{runtest} writes this file
2682 in your current working directory; use the @samp{--outdir} option to
2683 select a different directory.
2687 Here is a brief example showing a detailed log for @sc{g++} tests:
2691 Test Run By rob on Mon May 25 21:40:43 PDT 1992
2695 --- Running ./g++.other/t01-1.exp ---
2696 PASS: operate delete
2698 --- Running ./g++.other/t01-2.exp ---
2700 p0000646.C: In function `int warn_return_1 ()':
2701 p0000646.C:109: warning: control reaches end of non-void function
2702 p0000646.C: In function `int warn_return_arg (int)':
2703 p0000646.C:117: warning: control reaches end of non-void function
2704 p0000646.C: In function `int warn_return_sum (int, int)':
2705 p0000646.C:125: warning: control reaches end of non-void function
2706 p0000646.C: In function `struct foo warn_return_foo ()':
2707 p0000646.C:132: warning: control reaches end of non-void function
2709 --- Running ./g++.other/t01-4.exp ---
2711 900403_04.C:8: zero width for bit-field `foo'
2712 --- Running ./g++.other/t01-3.exp ---
2713 FAIL: segment violation
2714 900519_12.C:9: parse error before `;'
2715 900519_12.C:12: Segmentation violation
2716 /usr/latest/bin/gcc: Internal compiler error: program cc1plus got
2721 # of expected passes 1
2722 # of expected failures 3
2723 /usr/ps/bin/g++ version cygnus-2.0.1
2728 @subsection Logging @code{expect} internal actions
2731 With the @samp{--debug} option, you can request a log file showing the
2732 output from @code{expect} itself, running in debugging mode. This file
2733 (@file{dbg.log}, in the directory where you start @code{runtest}) shows
2734 each pattern @code{expect} considers in analyzing test output.
2736 This file reflects each @code{send} command, showing the string sent as
2737 input to the tool under test; and each @code{expect} command, showing
2738 each pattern it compares with the tool output.
2740 The log messages for @code{expect} begin with a message of the form
2743 expect: does @{@var{tool output}@} (spawn_id @var{n}) match pattern
2744 @{@var{expected pattern}@}?
2748 For every unsuccessful match, @code{expect} issues a @samp{no} after
2749 this message; if other patterns are specified for the same
2750 @code{expect} command, they are reflected also, but without the first
2751 part of the message (@samp{expect@dots{}match pattern}).
2753 When @code{expect} finds a match, the log for the successful match ends
2754 with @samp{yes}, followed by a record of the @code{expect} variables set
2755 to describe a successful match. Here is an excerpt from the debugging
2756 log for a @sc{gdb} test:
2758 @c FIXME! Why is the second spawn_id shown 0 rather than 6?
2761 send: sent @{break gdbme.c:34\n@} to spawn id 6
2762 expect: does @{@} (spawn_id 6) match pattern @{Breakpoint.*at.* file
2763 gdbme.c, line 34.*\(gdb\) $@}? no
2765 expect: does @{@} (spawn_id 0) match pattern @{<return>@}? no
2774 Breakpoint 8 at 0x23d8: file gdbme.c, line 34.
2775 (gdb) expect: does @{break gdbme.c:34\r\nBreakpoint 8 at 0x23d8:
2776 file gdbme.c, line 34.\r\n(gdb) @} (spawn_id 6) match pattern
2777 @{Breakpoint.*at.* file gdbme.c, line 34.*\(gdb\) $@}? yes
2778 expect: set expect_out(0,start) @{18@}
2779 expect: set expect_out(0,end) @{71@}
2780 expect: set expect_out(0,string) @{Breakpoint 8 at 0x23d8: file
2781 gdbme.c, line 34.\r\n(gdb) @}
2782 expect: set expect_out(spawn_id) @{6@}
2783 expect: set expect_out(buffer) @{break gdbme.c:34\r\nBreakpoint 8
2784 at 0x23d8: file gdbme.c, line 34.\r\n(gdb) @}
2785 PASS: 70 0 breakpoint line number in file
2790 This example exhibits three properties of @code{expect} and DejaGnu that
2791 might be surprising at first glance:
2795 Empty output for the first attempted match. The first set of attempted
2796 matches shown ran against the output @samp{@{@}}---that is, no output.
2797 @code{expect} begins attempting to match the patterns supplied
2798 immediately; often, the first pass is against incomplete output (or
2799 completely before all output, as in this case).
2802 Interspersed tool output. The beginning of the log entry for the second
2803 attempted match may be hard to spot: this is because the prompt
2804 @samp{(gdb) } appears on the same line, just before the @samp{expect:}
2805 that marks the beginning of the log entry.
2808 Fail-safe patterns. Many of the patterns tested are fail-safe patterns
2809 provided by @sc{gdb} testing utilities, to reduce possible
2810 indeterminacy. It is useful to anticipate potential variations
2811 caused by extreme system conditions (@sc{gdb} might issue the message
2812 @samp{virtual memory exhausted} in rare circumstances), or by changes in
2813 the tested program (@samp{Undefined command} is the likeliest outcome if
2814 the name of a tested command changes).
2816 The pattern @samp{@{<return>@}} is a particularly interesting fail-safe
2817 to notice; it checks for an unexpected @key{RET} prompt. This may
2818 happen, for example, if the tested tool can filter output through a
2821 These fail-safe patterns (like the debugging log itself) are primarily
2822 useful while developing test scripts. Use the @code{error} procedure to
2823 make the actions for fail-safe patterns produce messages starting with
2824 @samp{ERROR} on the @code{runtest} standard output, and in the detailed
2829 @chapter How To Write a Test Cases
2830 @cindex writing a test case
2831 @cindex test case, writing
2834 * Writing:: Writing a test case
2835 * Debugging:: Debugging a test case
2836 * Adding:: Adding a test case to a test suite
2837 * Hints:: Hints on writing a test case
2838 * Variables:: Special variables used by test cases
2842 @section Writing a test case
2844 The easiest way to prepare a new test case is to base it on an existing
2845 one for a similar situation. There are two major categories of tests:
2846 batch or interactive. Batch oriented tests are usually easier to write.
2848 The @sc{gcc} tests are a good example of batch oriented tests. All
2849 @sc{gcc} tests consist primarily of a call to a single common procedure,
2850 since all the tests either have no output, or only have a few warning
2851 messages when successfully compiled. Any non-warning output is a test
2852 failure. All the C code needed is kept in the test directory. The test
2853 driver, written in @code{expect}, need only get a listing of all the C
2854 files in the directory, and compile them all using a generic procedure.
2855 This procedure and a few others supporting for these tests are kept in
2856 the library module @file{lib/c-torture.exp} in the @sc{gcc} test suite.
2857 Most tests of this kind use very few @code{expect} features, and are
2858 coded almost purely in Tcl.
2861 Writing the complete suite of C tests, then, consisted of these steps:
2865 @cindex Granlund, Torbjorn
2866 @cindex C torture test
2867 Copying all the C code into the test directory. These tests were based on
2868 the C-torture test created by Torbjorn Granlund (on behalf of the Free
2869 Software Foundation) for @sc{gcc} development.
2872 Writing (and debugging) the generic @code{expect} procedures for
2876 Writing the simple test driver: its main task is to search the directory
2877 (using the Tcl procedure @code{glob} for filename expansion with
2878 wildcards) and call a Tcl procedure with each filename. It also checks
2879 for a few errors from the testing procedure.
2882 Testing interactive programs is intrinsically more complex. Tests for most
2883 interactive programs require some trial and error before they are complete.
2885 However, some interactive programs can be tested in a simple fashion
2886 reminiscent of batch tests. For example, prior to the creation of
2887 DejaGnu, the @sc{gdb} distribution already included a wide-ranging
2888 testing procedure. This procedure was very robust, and had already
2889 undergone much more debugging and error checking than many recent
2890 DejaGnu test cases. Accordingly, the best approach was simply to
2891 encapsulate the existing @sc{gdb} tests, for reporting purposes.
2892 Thereafter, new @sc{gdb} tests built up a family of @code{expect}
2893 procedures specialized for @sc{gdb} testing.
2895 @file{gdb.t10/crossload.exp} is a good example of an interactive test.
2896 @c FIXME! Check what *kind* of example it is---work-intensive, or generic...
2899 @section Debugging a test case
2900 @cindex debugging a test case
2901 @cindex test case, debugging
2904 These are the kinds of debugging information available from DejaGnu:
2908 Output controlled by test scripts themselves, explicitly allowed for by
2909 the test author. This kind of debugging output appears in the detailed
2910 output recorded in the @file{@var{tool}.log} file. To do the same for
2911 new tests, use the @code{verbose} procedure (which in turn uses the
2912 variable also called @code{verbose}) to control how much output to
2913 generate. This will make it easier for other people running the test to
2914 debug it if necessary. Whenever possible, if @samp{$verbose} is
2915 @code{0}, there should be no output other than the output from
2916 @code{pass}, @code{fail}, @code{error}, and @code{warning}. Then, to
2917 whatever extent is appropriate for the particular test, allow
2918 successively higher values of @samp{$verbose} to generate more
2919 information. Be kind to other programmers who use your tests: provide
2920 for a lot of debugging information.
2923 Output from the internal debugging functions of Tcl and @code{expect}.
2924 There is a command line options for each; both forms of debugging output
2925 are recorded in the file @code{dbg.log} in the current directory.
2927 Use @samp{--debug} for information from the @code{expect} level; it
2928 generates displays of the @code{expect} attempts to match the tool
2929 output with the patterns specified (@pxref{Debug,,Debug Log}). This
2930 output can be very helpful while developing test scripts, since it shows
2931 precisely the characters received. Iterating between the latest attempt
2932 at a new test script and the corresponding @file{dbg.log} can allow you
2933 to create the final patterns by ``cut and paste''. This is sometimes
2934 the best way to write a test case.
2936 Use @samp{--strace} to see more detail at the Tcl level; this shows how Tcl
2937 procedure definitions expand, as they execute. The associated number
2938 controls the depth of definitions expanded; see the discussion of
2939 @samp{--strace} in @ref{Invoking runtest,,Running the Tests}.
2942 Finally, if the value of @samp{verbose} is 3 or greater, @code{runtest}
2943 turns on the @code{expect} command @code{log_user}. This command prints
2944 all @code{expect} actions to the @code{expect} standard output, to the
2945 detailed log file, and (if @samp{--debug} is on) to @file{dbg.log}.
2949 @section Adding a test case to a test suite
2950 @cindex adding a test case
2952 There are two slightly different ways to add a test case. One is to add
2953 the test case to an existing directory. The other is to create a new
2954 directory to hold your test. The existing test directories represent
2955 several styles of testing, all of which are slightly different; examine
2956 the directories for the tool of interest to see which (if any) is most
2959 Adding a @sc{gcc} test can be very simple: just add the C code to any
2960 directory beginning with @samp{gcc.} and it runs on the next
2961 @samp{runtest --tool gcc}.
2963 To add a test to @sc{gdb}, first add any source code you will need to
2964 the test directory. Then you can either create a new @code{expect} file,
2965 or add your test to an existing one (any file with a @samp{.exp}
2966 suffix). Creating a new @samp{.exp} file is probably a better idea if
2967 the test is significantly different from existing tests. Adding it as a
2968 separate file also makes upgrading easier. If the C code has to be
2969 already compiled before the test will run, then you'll have to add it to
2970 the @file{Makefile.in} file for that test directory, then run
2971 @code{configure} and @code{make}.
2973 Adding a test by creating a new directory is very similar:
2977 Create the new directory. All subdirectory names begin with the name of
2978 the tool to test; e.g. @sc{g++} tests might be in a directory called
2979 @file{g++.other}. There can be multiple test directories that start with
2980 the same tool name (such as @samp{g++}).
2983 Add the new directory name to the @samp{configdirs} definition in the
2984 @file{configure.in} file for the test suite directory. This way when
2985 @code{make} and @code{configure} next run, they include the new directory.
2988 Add the new test case to the directory, as above.
2991 To add support in the new directory for configure and make, you must
2992 also create a @code{Makefile.in} and a @code{configure.in}. @xref{What
2993 Configure Does,,What Configure Does, configure.info, Cygnus Configure}.
2996 @c FIXME! Expand this sentence to at least a section, maybe a chapter...
2997 @c The @file{admin} directory contains templates for a few common forms
3001 @section Hints on writing a test case
3002 @cindex hints on test case writing
3004 There may be useful existing procedures already written for your test in
3005 the @file{lib} directory of the DejaGnu distribution. @xref{DejaGnu
3006 Builtins,,DejaGnu Builtins}.
3008 It is safest to write patterns that match @emph{all} the output
3009 generated by the tested program; this is called @dfn{closure}. If a
3010 pattern does not match the entire output, any output that remains will
3011 be examined by the @emph{next} @code{expect} command. In this
3012 situation, the precise boundary that determines which @code{expect}
3013 command sees what is very sensitive to timing between the @code{expect}
3014 task and the task running the tested tool. As a result, the test may
3015 sometimes appear to work, but is likely to have unpredictable results.
3016 (This problem is particularly likely for interactive tools, but can also
3017 affect batch tools---especially for tests that take a long time to finish.)
3018 The best way to ensure closure is to use the @samp{-re} option for the
3019 @code{expect} command to write the pattern as a full regular
3020 expressions; then you can match the end of output using a @samp{$}. It
3021 is also a good idea to write patterns that match all available output by
3022 using @samp{.*\} after the text of interest; this will also match any
3023 intervening blank lines. Sometimes an alternative is to match end of
3024 line using @samp{\r} or @samp{\n}, but this is usually too dependent on
3026 @c FIXME!! explain what "end of output" means for interactive task.
3027 @c (Timeout or EOF, right?)
3029 Always escape punctuation, such as @samp{(} or @samp{"}, in your
3030 patterns; for example, write @samp{\(}. If you forget to escape
3031 punctuation, you will usually see an error message like @samp{extra
3032 characters after close-quote}.
3034 If you have trouble understanding why a pattern does not match the
3035 program output, try using the @samp{--debug} option to @code{runtest},
3036 and examine the debug log carefully. @xref{Debug,,Debug Log}.
3038 Be careful not to neglect output generated by setup rather than by the
3039 interesting parts of a test case. For example, while testing @sc{gdb},
3040 I issue a send @samp{set height 0\n} command. The purpose is simply to
3041 make sure @sc{gdb} never calls a paging program. The @samp{set height}
3042 command in @sc{gdb} does not generate any output; but running @emph{any}
3043 command makes @sc{gdb} issue a new @samp{(gdb) } prompt. If there were
3044 no @code{expect} command to match this prompt, the output @samp{(gdb) }
3045 begins the text seen by the next @code{expect} command---which might
3046 make @emph{that} pattern fail to match.
3048 To preserve basic sanity, I also recommended that no test ever pass if
3049 there was any kind of problem in the test case. To take an extreme
3050 case, tests that pass even when the tool will not spawn are misleading.
3051 Ideally, a test in this sort of situation should not fail either.
3052 Instead, print an error message by calling one of the DejaGnu procedures
3053 @code{error} or @code{warning}.
3056 @section Special variables used by test cases
3057 @cindex special variables
3059 @cindex variables for all tests
3060 Your test cases can use these variables, with conventional meanings (as
3061 well as the variables saved in @file{site.exp}
3062 @pxref{Customizing,,Setting @code{runtest} defaults}):
3065 @emph{These variables are available to all test cases.}
3070 @cindex PRMS bug number
3071 @cindex GNATS bug number
3073 The tracking system (e.g. @sc{gnats}) number identifying a corresponding
3074 bugreport. (@samp{0} if you do not specify it in the test script.)
3077 @cindex bug number, extra
3078 An optional bug id; may reflect a bug identification from another
3079 organization. (@samp{0} if you do not specify it.)
3082 @cindex current test subdirectory
3083 The subdirectory for the current test case.
3087 @emph{These variables should never be changed. They appear in most
3092 @item expect_out(buffer)
3093 @cindex last command output
3094 The output from the last command. This is an internal variable set by
3098 This is the output from a @code{@var{tool}_load} command. This only
3099 applies to tools like @sc{gcc} and @sc{gas} which produce an object
3100 file that must in turn be executed to complete a test.
3103 This is the output from a @code{@var{tool}_start} command. This is
3104 conventionally used for batch oriented programs, like @sc{gcc} and
3105 @sc{gas}, that may produce interesting output (warnings, errors) without
3106 further interaction.
3110 @chapter New Tools, Targets, or Hosts
3112 The most common ways to extend the DejaGnu framework are: adding a suite
3113 of tests for a new tool to be tested; adding support for testing on a
3114 new target; and porting @code{runtest} to a new host.
3117 * Adding Tools:: How to add tests for a new tool
3118 * Adding Targets:: How to add a new target
3119 * Porting:: Porting DejaGnu to a new host
3123 @section Writing tests for a new tool
3125 In general, the best way to learn how to write (code or even prose) is
3126 to read something similar. This principle applies to test cases and to
3127 test suites. Unfortunately, well-established test suites have a way of
3128 developing their own conventions: as test writers become more
3129 experienced with DejaGnu and with Tcl, they accumulate more utilities,
3130 and take advantage of more and more features of @code{expect} and Tcl in
3133 Inspecting such established test suites may make the prospect of
3134 creating an entirely new test suite appear overwhelming. Nevertheless,
3135 it is quite straightforward to get a new test suite going.
3137 @cindex Lupton, Robert
3138 There is one test suite that is guaranteed not to grow more elaborate
3139 over time: both it and the tool it tests were created expressly to
3140 illustrate what it takes to get started with DejaGnu. The
3141 @file{example/} directory of the DejaGnu distribution contains both an
3142 interactive tool called @code{calc}, and a test suite for it. Reading
3143 this test suite, and experimenting with it, is a good way to supplement
3144 the information in this section. (Thanks to Robert Lupton for creating
3145 @code{calc} and its test suite---and also the first version of this
3146 section of the manual!)
3148 To help orient you further in this task, here is an outline of the steps
3149 to begin building a test suite for a program @var{example}.
3153 Create or select a directory to contain your new collection of tests.
3154 Change to that directory (shown here as @code{testsuite}):
3161 Create a @file{configure.in} file in this directory, to control
3162 configuration-dependent choices for your tests. So far as DejaGnu is
3163 concerned, the important thing is to set a value for the variable
3164 @code{target_abbrev}; this value is the link to the init file you will
3165 write soon. (For simplicity, we assume the environment is Unix, and use
3166 @samp{unix} as the value.)
3168 What else is needed in @file{configure.in} depends on the requirements
3169 of your tool, your intended test environments, and which
3170 @code{configure} system you use. This example is a minimal
3171 @code{configure.in} for use with Cygnus Configure. (For an alternative
3172 based on the FSF @code{autoconf} system, see the @code{calc} example
3173 distributed with DejaGnu.) Replace @var{example} with the name of your
3178 # This file is a shell script fragment
3179 # for use with Cygnus configure.
3181 srctrigger="@var{example}.0"
3182 srcname="The DejaGnu @var{example} tests"
3188 # everything defaults to unix for a target
3197 Create @file{Makefile.in}, the source file used by @code{configure} to
3198 build your @file{Makefile}. Its leading section should as usual contain
3199 the values that @code{configure} may override:
3206 exec_prefix = $(prefix)
3207 bindir = $(exec_prefix)/bin
3208 libdir = $(exec_prefix)/lib
3209 tooldir = $(libdir)/$(target_alias)
3211 datadir = $(exec_prefix)/lib/dejagnu
3217 #### host, target, site specific Makefile frags come in here.
3221 This should be followed by the standard targets at your site. To begin
3222 with, they need not do anything---for example, these definitions will
3238 -rm -f *~ core *.info*
3243 It is also a good idea to make sure your @file{Makefile} can rebuild
3244 itself if @file{Makefile.in} changes, with a target like this (which
3245 works for either Cygnus or FSF Configure):
3249 Makefile : $(srcdir)/Makefile.in $(host_makefile_frag) \
3250 $(target_makefile_frag)
3251 $(SHELL) ./config.status
3255 You also need to include two targets important to DejaGnu: @code{check},
3256 to run the tests, and @code{site.exp}, to set up the Tcl copies of
3257 configuration-dependent values. The @code{check} target must run
3258 @samp{runtest --tool @var{example}}:
3263 $(RUNTEST) $(RUNTESTFLAGS) $(FLAGS_TO_PASS) \
3264 --tool @var{example} --srcdir $(srcdir)
3268 The @code{site.exp} target should usually set up (among other things!) a
3269 Tcl variable for the name of your program:
3273 site.exp: ./config.status Makefile
3274 @@echo "Making a new config file..."
3278 -@@mv site.exp site.bak
3279 @@echo "## these variables are automatically\
3280 generated by make ##" > ./tmp0
3281 @@echo "# Do not edit here. If you wish to\
3282 override these values" >> ./tmp0
3283 @@echo "# add them to the last section" >> ./tmp0
3284 @@echo "set host_os $@{host_os@}" >> ./tmp0
3285 @@echo "set host_alias $@{host_alias@}" >> ./tmp0
3286 @@echo "set host_cpu $@{host_cpu@}" >> ./tmp0
3287 @@echo "set host_vendor $@{host_vendor@}" >> ./tmp0
3288 @@echo "set target_os $@{target_os@}" >> ./tmp0
3289 @@echo "set target_alias $@{target_alias@}" >> ./tmp0
3290 @@echo "set target_cpu $@{target_cpu@}" >> ./tmp0
3291 @@echo "set target_vendor $@{target_vendor@}" >> ./tmp0
3292 @@echo "set host_triplet $@{host_canonical@}" >> ./tmp0
3293 @@echo "set target_triplet $@{target_canonical@}">>./tmp0
3294 @@echo "set tool binutils" >> ./tmp0
3295 @@echo "set srcdir $@{srcdir@}" >> ./tmp0
3296 @@echo "set objdir `pwd`" >> ./tmp0
3297 @@echo "set @var{examplename} @var{example}" >> ./tmp0
3298 @@echo "## All variables above are generated by\
3299 configure. Do Not Edit ##" >> ./tmp0
3300 @@cat ./tmp0 > site.exp
3302 -e '1,/^## All variables above are.*##/ d' \
3309 Create a directory (in @file{testsuite/}) called @file{config/}:
3316 Make an init file in this directory; its name must start with the
3317 @code{target_abbrev} value, so call it @file{config/unix.exp}.
3318 This is the file that contains the target-dependent procedures;
3319 fortunately, most of them do not have to do very much in order for
3320 @code{runtest} to run.
3322 If @var{example} is not interactive, you can get away with this minimal
3323 @file{unix.exp} to begin with:
3327 proc foo_exit @{@} @{@}
3328 proc foo_version @{@} @{@}
3332 If @var{example} is interactive, however, you might as well define a
3333 start routine @emph{and invoke it} by using an init file like this:
3337 proc foo_exit @{@} @{@}
3338 proc foo_version @{@} @{@}
3340 proc foo_start @{@} @{
3341 global @var{examplename}
3342 spawn $@var{examplename}
3352 Create a directory whose name begins with your tool's name, to contain
3356 eg$ mkdir @var{example}.0
3360 Create a sample test file in @file{@var{example}.0}. Its name must end
3361 with @samp{.exp}; you can use @samp{first-try.exp} To begin with, just
3362 write there a line of Tcl code to issue a message:
3366 send_user "Testing: one, two...\n"
3371 Back in the @file{testsuite/} (top level) directory, run
3377 (You may have to specify more of a path, if a suitable @code{configure}
3378 is not available in your execution path.)
3381 You are now ready to triumphantly type @samp{make check} or
3382 @samp{runtest --tool @var{example}}. You should see something like this:
3386 Test Run By rhl on Fri Jan 29 16:25:44 EST 1993
3388 === @var{example} tests ===
3390 Running ./@var{example}.0/first-try.exp ...
3391 Testing: one, two...
3393 === @var{example} Summary ===
3398 There is no output in the summary, because so far the example does not
3399 call any of the procedures that establish a test outcome.
3402 Begin writing some real tests. For an interactive tool, you should
3403 probably write a real exit routine in fairly short order; in any case,
3404 you should also write a real version routine soon.
3407 @node Adding Targets
3408 @section Adding a target
3409 @cindex adding a target
3411 DejaGnu has some additional requirements for target support, beyond the
3412 general-purpose provisions of Cygnus @code{configure}. @code{runtest}
3413 must actively communicate with the target, rather than simply generating
3414 or managing code for the target architecture. Therefore, each tool
3415 requires an initialization module for each target. For new targets, you
3416 must supply a few Tcl procedures to adapt DejaGnu to the target. This
3417 permits DejaGnu itself to remain target independent. @xref{Init
3418 Module,,Initialization module}, for a discussion of the naming
3419 conventions that enable DejaGnu to locate and use init files.
3421 Usually the best way to write a new initialization module is to edit an
3422 existing initialization module; some trial and error will be required.
3423 If necessary, you can use the @samp{--debug} option to see what
3426 When you code an initialization module, be generous in printing
3427 information controlled by the @code{verbose} procedure (@pxref{DejaGnu
3428 Builtins, DejaGnu procedures}).
3430 Most of the work is in getting the communications right. Communications
3431 code (for several situations involving IP networks or serial lines) is
3432 available in a DejaGnu library file, @file{lib/remote.exp}.
3433 @xref{DejaGnu Builtins,,DejaGnu Builtins}.
3435 @c FIXME! Say something about Tcl debugger here.
3436 If you suspect a communication problem, try running the connection
3437 interactively from @code{expect}. (There are three ways of running
3438 @code{expect} as an interactive interpreter. You can run @code{expect}
3439 with no arguments, and control it completely interactively; or you can
3440 use @samp{expect -i} together with other command-line options and
3441 arguments; or you can run the command @code{interpreter} from any
3442 @code{expect} procedure. Use @code{return} to get back to the calling
3443 procedure (if any), or @code{return -tcl} to make the calling procedure
3444 itself return to its caller; use @code{exit} or end-of-file to leave
3445 @code{expect} altogether.) Run the program whose name is recorded in
3446 @samp{$connectmode}, with the arguments in @samp{$targetname}, to
3447 establish a connection. You should at least be able to get a prompt
3448 from any target that is physically connected.
3451 @section Porting to a new host
3452 @cindex porting to a new host
3454 The task of porting DejaGnu is basically that of porting Tcl and
3455 @code{expect}. Tcl and @code{expect}, as distributed with DejaGnu, both
3456 use @code{autoconf}; they should port automatically to most Unix
3459 Once Tcl and @code{expect} are ported, DejaGnu should run. Most system
3460 dependencies are taken care of by using @code{expect} as the main
3464 @appendix Installing DejaGnu
3466 @cindex host, explained
3467 @cindex target, explained
3468 @cindex DejaGnu configuration
3469 @cindex configuring DejaGnu
3470 Once you have the DejaGnu source unpacked and available, you must first
3471 configure the software to specify where it is to run (and the associated
3472 defaults); then you can proceed to installing it.
3475 * Configuring DejaGnu::
3476 * Installing DejaGnu::
3479 @node Configuring DejaGnu
3480 @section Configuring the DejaGnu test driver
3482 It is usually best to configure in a directory separate
3483 from the source tree, specifying where to find the source with the
3484 optional @samp{--srcdir} option to @code{configure}. DejaGnu uses the
3485 GNU @code{autoconf} to configure itself. For more info on using
3486 autoconf, read the GNU autoconf manual. To configure, execute the
3487 @file{configure} program, no other options are required. For an example,
3488 to configure in a seperate tree for objects, execute the configure
3489 script from the source tree like this:
3492 ../dejagnu-1.3/configure
3495 DejaGnu doesn't care at config time if it's for testing a native system
3496 or a cross system. That is determined at runtime by using the config
3499 @cindex @code{prefix}, configure options
3500 @cindex @code{exec_prefix}, configure options.
3501 You may also want to use the @code{configure} option @samp{--prefix} to
3502 specify where you want DejaGnu and its supporting code installed. By
3503 default, installation is in subdirectories of @file{/usr/local}, but you
3504 can select any alternate directory @var{altdir} by including
3505 @samp{--prefix=@var{altdir}} on the @code{configure} command line.
3506 (This value is captured in the Makefile variables @code{prefix}
3507 and @code{exec_prefix}.)
3509 @cindex auxiliary programs
3510 @cindex test suite distributions
3511 @cindex @code{make} builds part of tests
3512 Save for a small number of example tests, the DejaGnu distribution
3513 itself does not include any test suites; these are available separately.
3514 Test suites for the @sc{gnu} compiler (testing both GCC and G++) and for
3515 the @sc{gnu} binary utilities are distributed in parallel with the
3516 DejaGnu distribution (but packaged as separate files). The test suite
3517 for the @sc{gnu} debugger is distributed in parallel with each release
3518 of GDB itself, starting with GDB 4.9. After configuring the top-level
3519 DejaGnu directory, unpack and configure the test directories for the
3520 tools you want to test; then, in each test directory, run @code{make} to
3521 build auxiliary programs required by some of the tests.
3523 @node Installing DejaGnu
3524 @section Installing DejaGnu
3526 @cindex installing DejaGnu
3527 To install DejaGnu in your filesystem (either in @file{/usr/local}, or
3528 as specified by your @samp{--prefix} option to @code{configure}), execute
3535 @samp{make install} does these things for DejaGnu:
3539 Look in the path specified for executables (@file{$exec_prefix}) for
3540 directories called @file{lib} and @file{bin}. If these directories do
3541 not exist, @samp{make install} creates them.
3544 Create another directory in the @file{lib} directory, called
3548 Copy the @code{runtest} shell script into @file{$exec_prefix/bin}.
3551 Copy all the library files (used to support the framework) into
3552 @file{$exec_prefix/lib/dejagnu}.
3555 Copy @file{runtest.exp} into @file{$exec_prefix/lib/dejagnu}. This is
3556 the main Tcl code implementing DejaGnu.
3560 Each test suite collection comes with simple installation instructions
3561 in a @file{README} file; in general, the test suites are designed to be
3562 unpacked in the source directory for the corresponding tool, and extract
3563 into a directory called @file{testsuite}.