doc/overview.sgml

   1 <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook V3.1//EN" [
   2
   3 <!-- Begin Document Specific Declarations -->
   4
   5 <?Fm: Validation Off>
   6
   7 <!ENTITY version "1.4">
   8 <!ENTITY dj "DejaGnu">
   9
  10 <!ENTITY dejagnu-copyright "
  11         <YEAR>2001</YEAR>
  12         <HOLDER>Free Software Foundation, Inc.</HOLDER>">
  13
  14 <!ENTITY dejagnu-code-copyright "
  15 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
  16 This file documents the GNU Testing Framework ``DejaGnu''
  17
  18 Copyright (C) 92, 93, 94, 95, 96, 97, 98, 99, 2000, 2001 Free Software
  19 Foundation, Inc.
  20
  21 This text may be freely distributed under the terms of the GNU
  22 General Public License.
  23 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
  24 ">
  25
  26 <!ENTITY dejagnu-copyright "
  27 Copyright 92, 93, 94, 95, 96, 97, 98, 99, 2000, 2001 Free Software
  28 Foundation, Inc.
  29
  30 Permission is granted to make and distribute verbatim copies of
  31 this manual provided the copyright notice and this permission notice
  32 are preserved on all copies.
  33
  34 Permission is granted to copy and distribute modified versions of this
  35 manual under the conditions for verbatim copying, provided also that
  36 the entire resulting derived work is distributed under the terms of a
  37 permission notice identical to this one.
  38
  39 Permission is granted to copy and distribute translations of this manual
  40 into another language, under the above conditions for modified versions.
  41 ">
  42
  43 <!-- The reference material -->
  44 <!entity ref SYSTEM "ref.sgml">
  45
  46 <!-- The user manual -->
  47 <!entity user SYSTEM "user.sgml">
  48
  49 <!-- End Document Specific Declarations -->
  50
  51 ]>
  52
  53 <book>
  54   <bookinfo>
  55     <title>&dj;</title>
  56     <subtitle>The GNU Testing Framework</subtitle>
  57     <date>2001 Feb 16</date>
  58     <edition> &version</edition>
  59     <releaseinfo> for circulation within Cygnus</releaseinfo>
  60     <authorgroup>
  61       <author>
  62         <firstname>Rob Savoye</firstname>
  63         <affiliation>
  64           <orgname>Free Software Foundation</orgname></affiliation>
  65         <!-- <authorblurb>
  66           <title>Rob Savoye</title>
  67           <para>
  68             His home page is at <ulink>
  69             URL="http://www.welcomehome.org/rob.html">this
  70             location</ulink>
  71             </para>
  72         </authorblurb>
  73         -->
  74       </author>
  75     </authorgroup>
  76     <address>
  77       <email>rob@welcomehome.org</email>
  78     </address>
  79     <!-- &cygnus-street-address; -->
  80     <copyright>&dejagnu-copyright;</copyright>
  81     <!-- <legalnotice>
  82       <para> -->
  83         <!-- [FIXME: must put legal notice here] -->
  84       <!-- </para> -->
  85       <!-- &cygnus-legal-notice; -->
  86     <!-- </legalnotice> -->
  87     <revhistory>
  88       <revision>
  89         <revnumber>0.6</revnumber>
  90         <date>2001-2-16</date>
  91         <authorinitials>rob@welcomehome.org</authorinitials>
  92         <revremark>Updated for new release.</revremark>
  93       </revision>
  94       <revision>
  95         <revnumber>0.5</revnumber>
  96         <date>2000-1-24</date>
  97         <authorinitials>rob@welcomehome.org</authorinitials>
  98         <revremark>Initial version after conversion to DocBook.</revremark>
  99       </revision>
 100     </revhistory>
 101
 102   </bookinfo>
 103
 104   <toc></toc>
 105
 106   <preface id=preface>
 107     <title>Abstract</title>
 108
 109     <para>This document attempts to describe the functionality of
 110     DejaGnu, the GNU Testing Framework. DejaGnu is entirely written in
 111     <productname>Expect</productname>, which uses
 112     <productname>Tcl</productname> as a command
 113     language. <productname>Expect</productname> serves as a very
 114     programmable shell; you can run any program, as with the usual
 115     Unix command shells---but once the program is started, your
 116     test script has fully programmable control of
 117     its input and output.  This does not just apply to the programs
 118     under test; <command>expect</command> can also run any auxiliary
 119     program, such as <command>diff</command> or <command>sh</command>,
 120     with full control over its input and output.</para>
 121
 122     <para>DejaGnu itself is merely a framework for creation of a test
 123     suites. Test suites are distributed separately for each GNU
 124     tool.</para>
 125
 126   </preface>
 127
 128   <chapter id=overview xreflabel=Overview>
 129     <title>Overview</title>
 130
 131     <sect1 id=whatis xreflabel="What is &dj; ?">
 132       <title>What is &dj; ?</title>
 133
 134     <para><productname>DejaGnu</productname> is a framework for
 135         testing other programs.  Its purpose is to provide a single
 136         front end for all tests. Think of it as a custom library of
 137         Tcl procedures crafted to support writing a test harness. A
 138         <emphasis>Test Harness</emphasis> is the testing
 139         infrastructure that is created to support a specific program
 140         or tool. Each program can have multiple test suites, all
 141         supported by a single test harness. DejaGnu is written in
 142         <productname>Expect</productname>, which in turn uses
 143         <productname>Tcl</productname> -- Tool command
 144         language. There is more information on Tcl at the <ulink
 145         URL="http://www.scriptics.com">Scriptics</ulink> web site, and the
 146         Expect web site is at <ulink
 147         URL="http://expect.nist.gov">NIST</ulink>.</para>
 148
 149         <para>DejaGnu offers several advantages for testing:</para>
 150
 151     <itemizedlist mark="bullet" spacing="compact">
 152
 153       <listitem><para>The flexibility and consistency of the DejaGnu
 154         framework make it easy to write tests for any program, with
 155         either batch oriented, or interactive programs.</para>
 156       </listitem>
 157
 158       <listitem><para>DejaGnu provides a layer of abstraction which
 159         allows you to write tests that are portable to any host or
 160         target where a program must be tested. For instance, a test
 161         for <command>GDB</command> can run (from any Unix
 162         based host) on any target architecture that DejaGnu
 163         supports. Currently DejaGnu runs tests on many single board
 164         computers, whose operating software ranges from just a boot
 165         monitor to a full-fledged, Unix-like realtime OS.</para>
 166       </listitem>
 167
 168       <listitem><para>All tests have the same output format. This
 169         makes it easy to integrate testing into other software
 170         development processes. DejaGnu's output is designed to be
 171         parsed by other filtering script, and it is also human
 172         readable.</para>
 173       </listitem>
 174
 175       <listitem><para>Using Tcl and expect, it's easy to create wrappers
 176       for existing test suites. By incorporating existing tests under
 177       DejaGnu, it's easier to have a single set of report analyse
 178       programs..</para>
 179
 180       </listitem>
 181     </itemizedlist>
 182
 183     <para>Running tests requires two things: the testing framework, and
 184     the test suites themselves. Tests are usually written in
 185     <productname>Expect</productname> using Tcl, but you can also use a
 186     Tcl script to run a test suite that is not based on
 187     <productname>Expect</productname>.
 188     (<productname>expect</productname> script filenames conventionally
 189     use <emphasis>.exp</emphasis> as a suffix; for example, the main
 190     implementation of the DejaGnu test driver is in the file
 191     <productname>runtest.exp</productname>.)</para>
 192
 193     <para>Julia Menapace first coined the term ``Deja Gnu'' to describe an
 194     earlier testing framework at Cygnus Support she had written for
 195     <command>GDB</command>. When we replaced it with the Expect-based
 196     framework, it was like DejaGnu all over again... But more importantly, it
 197     was also named after my daughter,<ulink
 198     URL="mailto:deja@welcomehome.org">Deja Snow Savoye</ulink> (now 9
 199     years old in Dec of 1998), who was a toddler during DejaGnu's
 200     creation.</para>
 201
 202   </sect1>
 203
 204   <sect1 id=new xreflabel="Release Notes">
 205     <title>What's New In This Release</title>
 206
 207     <para>This release has a number of substantial changes over version
 208     1.3. The most visible change is that the version of Expect and Tcl
 209     included in the release are up-to-date with the current stable net
 210     releases. The biggest change is years of modifications to the
 211     target configuration system, used for cross testing. While this
 212     greatly improved cross testing, is has made that subsystem very
 213     complicated. The goal is to have this entirely rewritten using
 214     <productname>iTcl</productname> by the next release. Other changes
 215     are:</para>
 216
 217     <itemizedlist>
 218       <listitem><para>More builtin support for building target binaries
 219         with the correct linker flags. Currently this only works with
 220         <productname>GCC</productname> as the cross compiler,
 221         preferably with a target supported by
 222         <xref linkend=libgloss>.</para></listitem>
 223
 224       <listitem><para>Lots of little bug fixes from years of heavy
 225         use at Cygnus Solutions.</para></listitem>
 226
 227       <listitem><para>DejaGnu now uses
 228       <productname>Automake</productname> for Makefile
 229       configuration.</para></listitem>
 230
 231       <listitem><para>Updated documentation, now in SGML
 232         (using the <ulink
 233         URL="http://nis-www.lanl.gov/~rosalia/mydocs/docbook-intro.html">free
 234         GNU DocBook tools</ulink>) format.</para></listitem>
 235
 236       <listitem><para>NT support. There is beta level support for NT
 237       that is still a work in progress. This requires the <ulink
 238       URL="http://sources.redhat.com">Cygwin</ulink> POSIX system
 239       for NT.</para></listitem>
 240
 241     </itemizedlist>
 242
 243     <sect2 id=cygwin xreflabel="NT Support">
 244       <title>NT Support</title>
 245
 246       <para>To use DejaGnu on NT, you need to first install the
 247         <ulink URL="http://sources.redhat.com/cygwin">Cygwin</ulink>
 248         release. This works as of the B20.1 release. Cygwin is a POSIX
 249         system for NT. This covers both utility programs, and a libray
 250         that adds POSIX system calls to NT. Among them is pseudo tty
 251         support for NT that emulates the POSIX pty standard. The
 252         latest Cygwin is always available from <ulink
 253         URL="http://sources.redhat.com/cygwin">this location</ulink>. This
 254         works well enough to run <emphasis>"make check"</emphasis> of
 255         the GNU development tree on NT after a native build. But the
 256         nature of pty's on NT is still evolving. Your mileage may
 257         vary...</para>
 258
 259     </sect2>
 260
 261   </sect1>
 262
 263   <sect1 id=designgoals xreflabel="Design Goals">
 264     <title>Design Goals</title>
 265
 266     <para>DejaGnu grew out of the internal needs of Cygnus Solutions. (then
 267     Cygnus Support). Cygnus maintained and enhanced a variety of free
 268     programs in many different environments, and we needed a testing
 269     tool that:</para>
 270
 271     <itemizedlist mark="bullet">
 272       <listitem><para>was useful to developers while fixing
 273       bugs.</para></listitem>
 274       <listitem><para>automated running many tests during a software
 275       release process.</para></listitem>
 276       <listitem><para>was portable among a variety of host
 277       computers.</para></listitem>
 278       <listitem><para>supported cross-development
 279       testing.</para></listitem>
 280       <listitem><para>permited testing interactive programs, like
 281       <command>GDB</command>; and </para></listitem>
 282       <listitem><para>permited testing batch oriented programs, like
 283       <command>GCC</command>.</para></listitem>
 284     </itemizedlist>
 285
 286     <para>Some of the requirements proved challenging.  For example,
 287     interactive programs do not lend themselves very well to automated testing.
 288     But all the requirements are important: for instance, it is imperative to
 289     make sure that <command>GDB</command> works as well when cross-debugging
 290     as it does in a native configuration. </para>
 291
 292     <para>Probably the greatest challenge was testing in a cross-development
 293     environment (which can be a real nightmare).  Most cross-development
 294     environments are customized by each developer.  Even when buying packaged
 295     boards from vendors there are many differences.  The communication
 296     interfaces vary from a serial line to ethernet.  DejaGnu was designed with
 297     a modular communication setup, so that each kind of communication can be
 298     added as required, and supported thereafter.  Once a communication
 299     procedure is coded, any test can use it.  Currently DejaGnu can use
 300     <command>rsh</command>, <command>rlogin</command>,
 301     <command>telnet</command>, <command>tip</command>,
 302     <command>kermit</command>, and <command>mondfe</command> for remote
 303     communications.</para>
 304
 305     </sect1>
 306
 307     <sect1 id=posix xreflabel="A POSIX Conforming Test Framework">
 308       <title>A POSIX conforming test framework</title>
 309
 310       <para>DejaGnu conforms to the POSIX 1003.3 standard for test
 311       frameworks. I was also a member of that committe.</para>
 312
 313       <para>The POSIX standard 1003.3 defines what a testing framework needs to
 314       provide, in order to permit the creation of POSIX conformance test
 315       suites. This standard is primarily oriented to running POSIX conformance
 316       tests, but its requirements also support testing of features not related
 317       to POSIX conformance.  POSIX 1003.3 does not specify a particular testing
 318       framework, but at this time there is only one other POSIX conforming test
 319       framework: TET. TET was created by Unisoft for a consortium comprised of
 320       X/Open, Unix International, and the Open Software Foundation.</para>
 321
 322       <para>The POSIX documentation refers to <firstterm>assertions</firstterm>.
 323       An assertion is a description of behavior.  For example, if a standard
 324       says ``The sun shall shine'', a corresponding assertion might be ``The
 325       sun is shining.''  A test based on this assertion would pass or fail
 326       depending on whether it is daytime or nighttime.  It is important to note
 327       that the standard being tested is never 1003.3; the standard being tested
 328       is some other standard, for which the assertions were written.</para>
 329
 330       <para>As there is no test suite to test testing frameworks for POSIX
 331       1003.3 conformance, verifying conformance to this standard is done by
 332       repeatedly reading the standard and experimenting.  One of the main
 333       things 1003.3 does specify is the set of allowed output messages, and
 334       their definitions.  Four messages are supported for a required feature of
 335       POSIX conforming systems, and a fifth for a conditional feature. DejaGnu
 336       supports the use of all five output messages; in this sense a test suite
 337       that uses exactly these messages can be considered POSIX conforming.
 338       These definitions specify the output of a test
 339       case:</para>
 340
 341       <variablelist>
 342         <varlistentry>
 343           <term>PASS</term>
 344           <listitem><para>A test has succeeded.  That is, it demonstrated that
 345           the assertion is true.</para></listitem>
 346         </varlistentry>
 347
 348         <varlistentry>
 349           <term>XFAIL</term>
 350           <listitem><para>POSIX 1003.3 does not incorporate the notion of
 351           expected failures, so <emphasis>PASS</emphasis>, instead of
 352           <emphasis>XPASS</emphasis>, must also be returned for test cases
 353           which were expected to fail and did not.  This means that
 354           <emphasis>PASS</emphasis> is in some sense more ambiguous than if
 355           <emphasis>XPASS</emphasis> is also used.</para></listitem>
 356         </varlistentry>
 357
 358         <varlistentry>
 359           <term>FAIL</term>
 360           <listitem><para>A test has produced the bug it was intended to
 361           capture.  That is, it has demonstrated that the assertion is false.
 362           The <emphasis>FAIL</emphasis> message is based on the test case only.
 363           Other messages are used to indicate a failure of the framework. As
 364           with <emphasis>PASS</emphasis>, POSIX tests must return
 365           <emphasis>FAIL</emphasis> rather than <emphasis>XFAIL</emphasis> even
 366           if a failure was expected.</para></listitem>
 367         </varlistentry>
 368
 369         <varlistentry>
 370           <term>UNRESOLVED</term>
 371           <listitem><para>A test produced indeterminate results.  Usually, this
 372           means the test executed in an unexpected fashion; this outcome
 373           requires that a human being go over results, to determine if the test
 374           should have passed or failed.  This message is also used for any test
 375           that requires human intervention because it is beyond the abilities
 376           of the testing framework.  Any unresolved test should resolved to
 377           <emphasis>PASS</emphasis> or <emphasis>FAIL</emphasis> before a test
 378           run can be considered finished.</para>
 379
 380           <para>Note that for POSIX, each assertion must produce a test result
 381           code.  If the test isn't actually run, it must produce
 382           <emphasis>UNRESOLVED</emphasis> rather than just leaving that test
 383           out of the output.  This means that you have to be careful when
 384           writing tests, to not carelessly use tcl statements like
 385           <emphasis>return</emphasis>---if you alter the flow of control of the
 386           tcl code you must insure that every test still produces some result
 387           code.</para>
 388
 389           <para>Here are some of the ways a test may wind up
 390           <emphasis>UNRESOLVED</emphasis>:</para></listitem>
 391
 392         </varlistentry>
 393         </variablelist>
 394
 395           <itemizedlist mark=bullet>
 396             <listitem><para>A test's execution is
 397             interrupted.</para></listitem>
 398
 399             <listitem><para>A test does not produce a clear
 400             result. This is usually because there was an
 401             <emphasis>ERROR</emphasis> from DejaGnu while processing
 402             the test, or because there were three or more
 403             <emphasis>WARNING</emphasis> messages. Any
 404             <emphasis>WARNING</emphasis> or <emphasis>ERROR</emphasis>
 405             messages can invalidate the output of the test.  This
 406             usually requires a human being to examine the output to
 407             determine what really happened---and to improve the test
 408             case.</para></listitem>
 409
 410             <listitem><para>A test depends on a previous test, which
 411             fails.</para></listitem>
 412
 413             <listitem><para>The test was set up
 414             incorrectly.</para></listitem>
 415           </itemizedlist>
 416
 417         <variablelist>
 418           <varlistentry>
 419             <term>UNTESTED</term>
 420             <listitem><para>A test was not run.  This is a placeholder, used
 421             when there is no real test case yet.</para></listitem>
 422           </varlistentry>
 423         </variablelist>
 424
 425           <para>The only remaining output message left is intended to test
 426           features that are specified by the applicable POSIX standard as
 427           conditional:</para>
 428
 429         <variablelist>
 430           <varlistentry>
 431             <term>UNSUPPORTED</term>
 432             <listitem><para>There is no support for the tested case.  This may
 433             mean that a conditional feature of an operating system, or of a
 434             compiler, is not implemented.  DejaGnu also uses this message when
 435             a testing environment (often a ``bare board'' target) lacks basic
 436             support for compiling or running the test case.  For example, a
 437             test for the system subroutine <emphasis>gethostname</emphasis>
 438             would never work on a target board running only a boot
 439             monitor.</para></listitem>
 440           </varlistentry>
 441         </variablelist>
 442
 443         <para>DejaGnu uses the same output procedures to produce these messages
 444         for all test suites, and these procedures are already known to conform
 445         to POSIX 1003.3.  For a DejaGnu test suite to conform to POSIX 1003.3,
 446         you must avoid the <emphasis>setup</emphasis>xfail} procedure as
 447         described in the <emphasis>PASS</emphasis> section above, and you must
 448         be careful to return <emphasis>UNRESOLVED</emphasis> where appropriate,
 449         as described in the <emphasis>UNRESOLVED</emphasis> section
 450         above.</para>
 451      </sect1>
 452
 453   </chapter>
 454
 455   <!-- include the user manual -->
 456   &user;
 457
 458   <!-- include the reference manual -->
 459   &ref;
 460
 461 </book>
 462
 463 <!-- Keep this comment at the end of the file
 464 Local variables:
 465 mode: sgml
 466 sgml-omittag:t
 467 sgml-shorttag:t
 468 sgml-namecase-general:t
 469 sgml-general-insert-case:lower
 470 sgml-minimize-attributes:nil
 471 sgml-always-quote-attributes:t
 472 sgml-indent-step:1
 473 sgml-indent-data:nil
 474 sgml-parent-document:nil
 475 sgml-exposed-tags:nil
 476 sgml-local-catalogs:nil
 477 sgml-local-ecat-files:nil
 478 End:
 479 -->