From 730d3ca912ec9d19d42285799beaf56a1c0d2252 Mon Sep 17 00:00:00 2001 From: Ben Elliston Date: Sat, 16 Aug 2003 06:31:50 +0000 Subject: [PATCH] * doc/overview.sgml (overview): Start overhauling chapter. * doc/README.Writers: New document on style conventions. * doc/ref.sgml: Replace "test suite" with "testsuite" throughout. * doc/user.sgml: Likewise. * doc/overview.sgml: Likewise. * doc/dejagnu.texi (Design Goals): Change "Deja Gnu" to "DejaGnu". --- ChangeLog | 10 ++++ doc/README.Writers | 5 ++ doc/dejagnu.texi | 2 +- doc/overview.sgml | 157 ++++++++++++++++++++++++++--------------------------- doc/ref.sgml | 6 +- doc/user.sgml | 46 ++++++++-------- 6 files changed, 120 insertions(+), 106 deletions(-) create mode 100644 doc/README.Writers diff --git a/ChangeLog b/ChangeLog index 6c4b42f..01a0bf7 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,13 @@ +2003-08-16 Ben Elliston + + * doc/overview.sgml (overview): Start overhauling chapter. + + * doc/README.Writers: New document on style conventions. + * doc/ref.sgml: Replace "test suite" with "testsuite" throughout. + * doc/user.sgml: Likewise. + * doc/overview.sgml: Likewise. + * doc/dejagnu.texi (Design Goals): Change "Deja Gnu" to "DejaGnu". + 2003-08-12 Ben Elliston * Makefile.am (tarball): Update path to dejagnu.spec. diff --git a/doc/README.Writers b/doc/README.Writers new file mode 100644 index 0000000..4f49612 --- /dev/null +++ b/doc/README.Writers @@ -0,0 +1,5 @@ +Conventions: + + * Use DejaGnu, not "Dejagnu" or "Deja Gnu". + * Use "testsuite", not "test suite". + * Refer to the Tcl, Tk and Expect packages using proper capitalisation. diff --git a/doc/dejagnu.texi b/doc/dejagnu.texi index 47443a1..ef433b9 100644 --- a/doc/dejagnu.texi +++ b/doc/dejagnu.texi @@ -407,7 +407,7 @@ DejaGnu can use @code{rsh}, @code{rlogin}, @code{telnet}, @code{tip}, @cindex name ``DejaGnu'' @cindex DejaGnu, the name @cindex Menapace, Julia -Julia Menapace first coined the term ``Deja Gnu'' to describe an earlier +Julia Menapace first coined the term ``DejaGnu'' to describe an earlier testing framework at Cygnus Support. When we replaced it with the Expect-based framework, it was like DejaGnu all over again@dots{} diff --git a/doc/overview.sgml b/doc/overview.sgml index df5bfc1..e1bc35c 100644 --- a/doc/overview.sgml +++ b/doc/overview.sgml @@ -118,22 +118,22 @@ into another language, under the above conditions for modified versions. Abstract - This document attempts to describe the functionality of - DejaGnu, the GNU Testing Framework. DejaGnu is entirely written in + This document describes the functionality of DejaGnu, the + testing framework of the GNU project. DejaGnu is written in Expect, which uses Tcl as a command - language. Expect serves as a very - programmable shell; you can run any program, as with the usual - Unix command shells---but once the program is started, your - test script has fully programmable control of - its input and output. This does not just apply to the programs - under test; expect can also run any auxiliary - program, such as diff or sh, - with full control over its input and output. - - DejaGnu itself is merely a framework for creation of a test - suites. Test suites are distributed separately for each GNU - tool. + language. Expect acts as a very + programmable shell. As with other Unix command shells, you can + run any program, but once the program is started, your test script + has programmable control over its input and output. This does not + just apply to the programs under test; expect + can also run any auxiliary program, such as + diff or sh, with full + control over its input and output. + + DejaGnu itself is merely a framework for the creation of + testsuites. Testsuites are distributed with each + application. @@ -149,16 +149,25 @@ into another language, under the above conditions for modified versions. Tcl procedures crafted to support writing a test harness. A Test Harness is the testing infrastructure that is created to support a specific program - or tool. Each program can have multiple test suites, all + or tool. Each program can have multiple testsuites, all supported by a single test harness. DejaGnu is written in Expect, which in turn uses Tcl -- Tool command language. There is more information on Tcl at the Scriptics web site, and the + URL="http://www.scriptics.com">Scriptics web site and the Expect web site is at NIST. - - DejaGnu offers several advantages for testing: + + Julia Menapace first coined the term ``DejaGnu'' to describe + an earlier testing framework at Cygnus Support she had written + for GDB. When we replaced it with the + Expect-based framework, it was like DejaGnu all over again. + More importantly, it was also named after my daughter, Deja Snow Savoye + (now 13 years old as of September 2003), who was a toddler + during DejaGnu's beginnings. + + DejaGnu offers several advantages for testing: @@ -170,47 +179,37 @@ into another language, under the above conditions for modified versions. DejaGnu provides a layer of abstraction which allows you to write tests that are portable to any host or target where a program must be tested. For instance, a test - for GDB can run (from any Unix - based host) on any target architecture that DejaGnu - supports. Currently DejaGnu runs tests on many single board - computers, whose operating software ranges from just a boot - monitor to a full-fledged, Unix-like realtime OS. - + for GDB can run from any supported host + system on any supported target system. DejaGnu runs tests on + many single board computers, whose operating software ranges + from a simple boot monitor to a real-time OS. + All tests have the same output format. This makes it easy to integrate testing into other software development processes. DejaGnu's output is designed to be - parsed by other filtering script, and it is also human + parsed by other filtering script and it is also human readable. - Using Tcl and expect, it's easy to create wrappers - for existing test suites. By incorporating existing tests under + Using Tcl and Expect, it's easy to create wrappers + for existing testsuites. By incorporating existing tests under DejaGnu, it's easier to have a single set of report analyse programs.. - Running tests requires two things: the testing framework, and - the test suites themselves. Tests are usually written in - Expect using Tcl, but you can also use a - Tcl script to run a test suite that is not based on - Expect. - (expect script filenames conventionally - use .exp as a suffix; for example, the main - implementation of the DejaGnu test driver is in the file + Running tests requires two things: the testing framework and + the testsuites themselves. Tests are usually written in + Expect using Tcl, but you can also use + a Tcl script to run a testsuite that is not based on + Expect. Expect + script filenames conventionally use .exp as a + suffix; for example, the main implementation of the DejaGnu test + driver is in the file runtest.exp.) - Julia Menapace first coined the term ``Deja Gnu'' to describe an - earlier testing framework at Cygnus Support she had written for - GDB. When we replaced it with the Expect-based - framework, it was like DejaGnu all over again. More importantly, it - was also named after my daughter,Deja Snow Savoye (now 12 - years old in Sept of 2002), who was a toddler during DejaGnu's - creation. - @@ -227,7 +226,7 @@ into another language, under the above conditions for modified versions. are: - More builtin support for building target binaries + More built-in support for building target binaries with the correct linker flags. Currently this only works with GCC as the cross compiler, preferably with a target supported by @@ -258,7 +257,7 @@ into another language, under the above conditions for modified versions. To use DejaGnu on Windows, you need to first install the Cygwin release. This works as of the B20.1 release. Cygwin is a POSIX - system for Windows. This covers both utility programs, and a libray + system for Windows. This covers both utility programs and a library that adds POSIX system calls to Windows. Among them is pseudo tty support for Windows that emulates the POSIX pty standard. The latest Cygwin is always available from Design Goals - DejaGnu grew out of the internal needs of Cygnus Solutions. (then - Cygnus Support). Cygnus maintained and enhanced a variety of free - programs in many different environments, and we needed a testing - tool that: + DejaGnu grew out of the internal needs of Cygnus Solutions, + the company formerly known as Cygnus Support. Cygnus maintained + and enhanced a variety of free programs in many different + environments and we needed a testing tool that: was useful to developers while fixing - bugs. + bugs; automated running many tests during a software - release process. + release process; was portable among a variety of host - computers. + computers; supported cross-development - testing. + testing; permitted testing interactive programs, like GDB; and permitted testing batch oriented programs, like @@ -301,18 +300,18 @@ into another language, under the above conditions for modified versions. make sure that GDB works as well when cross-debugging as it does in a native configuration. - Probably the greatest challenge was testing in a cross-development - environment (which can be a real nightmare). Most cross-development - environments are customized by each developer. Even when buying packaged - boards from vendors there are many differences. The communication - interfaces vary from a serial line to ethernet. DejaGnu was designed with - a modular communication setup, so that each kind of communication can be - added as required, and supported thereafter. Once a communication - procedure is coded, any test can use it. Currently DejaGnu can use - rsh, rlogin, - telnet, tip, - kermit, and mondfe for remote - communications. + Probably the greatest challenge was testing in a + cross-development environment. Most cross-development + environments are customized by each developer. Even when buying + packaged boards from vendors there are many differences. The + communication interfaces vary from a serial line to Ethernet. + DejaGnu was designed with a modular communication setup, so that + each kind of communication can be added as required and supported + thereafter. Once a communication procedure is coded, any test can + use it. Currently DejaGnu can use rsh, + rlogin, telnet, + tip, kermit and + mondfe for remote communications. @@ -320,7 +319,7 @@ into another language, under the above conditions for modified versions. A POSIX conforming test framework DejaGnu conforms to the POSIX 1003.3 standard for test - frameworks. I was also a member of that committe. + frameworks. Rob Savoye was a member of that committee. The POSIX standard 1003.3 defines what a testing framework needs to provide, in order to permit the creation of POSIX conformance test @@ -329,23 +328,23 @@ into another language, under the above conditions for modified versions. to POSIX conformance. POSIX 1003.3 does not specify a particular testing framework, but at this time there is only one other POSIX conforming test framework: TET. TET was created by Unisoft for a consortium comprised of - X/Open, Unix International, and the Open Software Foundation. + X/Open, Unix International and the Open Software Foundation. The POSIX documentation refers to assertions. An assertion is a description of behavior. For example, if a standard says ``The sun shall shine'', a corresponding assertion might be ``The sun is shining.'' A test based on this assertion would pass or fail - depending on whether it is daytime or nighttime. It is important to note + depending on whether it is day or night. It is important to note that the standard being tested is never 1003.3; the standard being tested is some other standard, for which the assertions were written. - As there is no test suite to test testing frameworks for POSIX + As there is no testsuite to test testing frameworks for POSIX 1003.3 conformance, verifying conformance to this standard is done by repeatedly reading the standard and experimenting. One of the main - things 1003.3 does specify is the set of allowed output messages, and + things 1003.3 does specify is the set of allowed output messages and their definitions. Four messages are supported for a required feature of - POSIX conforming systems, and a fifth for a conditional feature. DejaGnu - supports the use of all five output messages; in this sense a test suite + POSIX conforming systems and a fifth for a conditional feature. DejaGnu + supports the use of all five output messages. In this sense a testsuite that uses exactly these messages can be considered POSIX conforming. These definitions specify the output of a test case: @@ -393,9 +392,9 @@ into another language, under the above conditions for modified versions. code. If the test isn't actually run, it must produce UNRESOLVED rather than just leaving that test out of the output. This means that you have to be careful when - writing tests, to not carelessly use tcl statements like + writing tests to not carelessly use Tcl commands like return---if you alter the flow of control of the - tcl code you must insure that every test still produces some result + Tcl code you must insure that every test still produces some result code. Here are some of the ways a test may wind up @@ -429,7 +428,7 @@ into another language, under the above conditions for modified versions. UNTESTED - A test was not run. This is a placeholder, used + A test was not run. This is a place-holder, used when there is no real test case yet. @@ -453,10 +452,10 @@ into another language, under the above conditions for modified versions. DejaGnu uses the same output procedures to produce these messages - for all test suites, and these procedures are already known to conform - to POSIX 1003.3. For a DejaGnu test suite to conform to POSIX 1003.3, + for all testsuites and these procedures are already known to conform + to POSIX 1003.3. For a DejaGnu testsuite to conform to POSIX 1003.3, you must avoid the setupxfail} procedure as - described in the PASS section above, and you must + described in the PASS section above and you must be careful to return UNRESOLVED where appropriate, as described in the UNRESOLVED section above. diff --git a/doc/ref.sgml b/doc/ref.sgml index ed642b0..dc9686d 100644 --- a/doc/ref.sgml +++ b/doc/ref.sgml @@ -51,8 +51,8 @@ execprefix}.) Save for a small number of example tests, the DejaGnu distribution - itself does not include any test suites; these are available - separately. Test suites for the GNU development tools are included in + itself does not include any testsuites; these are available + separately. Testsuites for the GNU development tools are included in those releases. After configuring the top-level DejaGnu directory, unpack and configure the test directories for the tools you want to test; then, in each test directory, run make check to build @@ -3161,7 +3161,7 @@ and ${tool}_load}, are only called by the test suites themselves (or by testsuite-specific initialization code); they may take arguments or not, depending on the - conventions used within each test suite. + conventions used within each testsuite. The usual convention for return codes from any of these procedures (although it is not required by diff --git a/doc/user.sgml b/doc/user.sgml index cfafca6..1a4c7f2 100644 --- a/doc/user.sgml +++ b/doc/user.sgml @@ -711,7 +711,7 @@ powerpc-linux-gcc -g -O2 -o calc calc.o Running Tests - There are two ways to execute a test suite. The most + There are two ways to execute a testsuite. The most common way is when there is existing support in the Makefile. This support consists of a check target. The other way is to execute the @@ -812,7 +812,7 @@ powerpc-linux-gcc -g -O2 -o calc calc.o UNRESOLVED Output from a test requires manual inspection; the - test suite could not automatically determine the outcome. For + testsuite could not automatically determine the outcome. For example, your tests can report this outcome is when a test does not complete as expected. @@ -844,7 +844,7 @@ powerpc-linux-gcc -g -O2 -o calc calc.o Indicates a major problem (detected by the test case itself) in running the test. This is usually an unrecoverable error, such as a missing file or loss of communication to the target. (POSIX - test suites should not emit this message; use + testsuites should not emit this message; use UNSUPPORTED, UNTESTED, or UNRESOLVED instead, as appropriate.) @@ -1054,7 +1054,7 @@ powerpc-linux-gcc -g -O2 -o calc calc.o - Specifies which test suite to run, and what + Specifies which testsuite to run, and what initialization module to use. is used only for these two purposes. It is not used to name the executable program to @@ -1163,7 +1163,7 @@ powerpc-linux-gcc -g -O2 -o calc calc.o Typically, you don't need must to use any command-line options. used is only required when there are more than - one test suite in the same directory. The default options are in the + one testsuite in the same directory. The default options are in the local site.exp file, created by "make site.exp". For example, if the directory gdb/testsuite @@ -1465,7 +1465,7 @@ powerpc-linux-gcc -g -O2 -o calc calc.o DejaGnu test environment using Tcl variables. This ties the DejaGnu test scripts into the configure and make programs. If this file is setup correctly, - it is possible to execute a test suite merely by typing + it is possible to execute a testsuite merely by typing runtest. DejaGnu supports two site.exp @@ -1487,7 +1487,7 @@ powerpc-linux-gcc -g -O2 -o calc calc.o DEJAGNU to the name of the file. This is also refered to as the ``global'' config file. - Any directory containing a configured test suite also has a + Any directory containing a configured testsuite also has a local site.exp, capturing configuration values specific to the tool under test. Since runtest loads these values last, the individual test configuration can @@ -1496,7 +1496,7 @@ powerpc-linux-gcc -g -O2 -o calc calc.o You can usually generate or update the testsuite's local site.exp by typing make - site.exp in the test suite directory, after the test + site.exp in the testsuite directory, after the test suite is configured. You can also have a file in your home directory called @@ -1519,7 +1519,7 @@ powerpc-linux-gcc -g -O2 -o calc calc.o local to each test directory, rather than in the global site.exp in the installed DejaGnu library. This file is mostly for supplying tool specific info - that is required by the test suite. + that is required by the testsuite. All local site.exp files have two sections, separated by comment text. The first section is @@ -1601,7 +1601,7 @@ powerpc-linux-gcc -g -O2 -o calc calc.o This file defines the required fields for a local config file, namely the three config triplets, and the srcdir. It also defines several other Tcl variables that are used exclusivly by - the GCC test suite. For most test cases, the CXXFLAGS and LDFLAGS + the GCC testsuite. For most test cases, the CXXFLAGS and LDFLAGS are supplied by DejaGnu itself for cross testing, but to test a compiler, GCC needs to manipulate these itself. @@ -2067,8 +2067,8 @@ powerpc-linux-gcc -g -O2 -o calc calc.o Extending DejaGnu - - Adding A New Test Suite + + Adding A New Testsuite The testsuite for a new tool should always be located in that tools source directory. DejaGnu require the directory be named @@ -2084,29 +2084,29 @@ powerpc-linux-gcc -g -O2 -o calc calc.o In general, the best way to learn how to write (code or even prose) is to read something similar. This principle applies to test cases and - to test suites. Unfortunately, well-established test suites have a way + to testsuites. Unfortunately, well-established testsuites have a way of developing their own conventions: as test writers become more experienced with DejaGnu and with Tcl, they accumulate more utilities, and take advantage of more and more features of Expect and Tcl in general. - Inspecting such established test suites may make the prospect of - creating an entirely new test suite appear overwhelming. Nevertheless, - it is quite straightforward to get a new test suite going. + Inspecting such established testsuites may make the prospect of + creating an entirely new testsuite appear overwhelming. Nevertheless, + it is quite straightforward to get a new testsuite going. - There is one test suite that is guaranteed not to grow more + There is one testsuite that is guaranteed not to grow more elaborate over time: both it and the tool it tests were created expressly to illustrate what it takes to get started with DejaGnu. The example/ directory of the DejaGnu distribution contains both an interactive tool called calc, and a - test suite for it. Reading this test suite, and experimenting with it, + testsuite for it. Reading this testsuite, and experimenting with it, is a good way to supplement the information in this section. (Thanks to - Robert Lupton for creating calc and its test suite---and also the first + Robert Lupton for creating calc and its testsuite---and also the first version of this section of the manual!) To help orient you further in this task, here is an outline of the - steps to begin building a test suite for a program example. + steps to begin building a testsuite for a program example. @@ -2897,8 +2897,8 @@ powerpc-linux-gcc -g -O2 -o calc calc.o - - Adding A Test Case To A Test Suite. + + Adding A Test Case To A Testsuite. There are two slightly different ways to add a test case. One is to add the test case to an existing directory. The @@ -2937,7 +2937,7 @@ powerpc-linux-gcc -g -O2 -o calc calc.o Add the new directory name to the configdirs definition in the - configure.in file for the test suite + configure.in file for the testsuite directory. This way when make and configure next run, they include the new directory. -- 2.11.4.GIT