doc/overview.sgml

   1 <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook V3.1//EN" [
   2
   3 <!-- Begin Document Specific Declarations -->
   4
   5 <?Fm: Validation Off>
   6
   7 <!ENTITY version "1.4.3">
   8 <!ENTITY dj "DejaGnu">
   9
  10 <!ENTITY dejagnu-copyright "
  11         <YEAR>2002</YEAR>
  12         <HOLDER>Free Software Foundation, Inc.</HOLDER>">
  13
  14 <!ENTITY dejagnu-code-copyright "
  15 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
  16 This file documents the GNU Testing Framework ``DejaGnu''
  17
  18 Copyright (C) 92 - 2001, 2002 Free Software
  19 Foundation, Inc.
  20
  21 This text may be freely distributed under the terms of the GNU
  22 General Public License.
  23 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
  24 ">
  25
  26 <!ENTITY dejagnu-copyright "
  27 Copyright 92 - 2001, 2002 Free Software
  28 Foundation, Inc.
  29
  30 Permission is granted to make and distribute verbatim copies of
  31 this manual provided the copyright notice and this permission notice
  32 are preserved on all copies.
  33
  34 Permission is granted to copy and distribute modified versions of this
  35 manual under the conditions for verbatim copying, provided also that
  36 the entire resulting derived work is distributed under the terms of a
  37 permission notice identical to this one.
  38
  39 Permission is granted to copy and distribute translations of this manual
  40 into another language, under the above conditions for modified versions.
  41 ">
  42
  43 <!-- The reference material -->
  44 <!entity ref SYSTEM "ref.sgml">
  45
  46 <!-- The user manual -->
  47 <!entity user SYSTEM "user.sgml">
  48
  49 <!-- End Document Specific Declarations -->
  50
  51 ]>
  52
  53 <book>
  54   <bookinfo>
  55     <title>&dj;</title>
  56     <subtitle>The GNU Testing Framework</subtitle>
  57     <date>2002 Sept 04</date>
  58     <edition> &version</edition>
  59     <releaseinfo> New release</releaseinfo>
  60     <authorgroup>
  61       <author>
  62         <firstname>Rob Savoye</firstname>
  63         <affiliation>
  64           <orgname>Free Software Foundation</orgname></affiliation>
  65         <!-- <authorblurb>
  66           <title>Rob Savoye</title>
  67           <para>
  68             His home page is at <ulink>
  69             URL="http://www.welcomehome.org/rob.html">this
  70             location</ulink>
  71             </para>
  72         </authorblurb>
  73         -->
  74       </author>
  75     </authorgroup>
  76     <address>
  77       <email>rob@welcomehome.org</email>
  78     </address>
  79     <!-- &cygnus-street-address; -->
  80     <copyright>&dejagnu-copyright;</copyright>
  81     <!-- <legalnotice>
  82       <para> -->
  83         <!-- [FIXME: must put legal notice here] -->
  84       <!-- </para> -->
  85       <!-- &cygnus-legal-notice; -->
  86     <!-- </legalnotice> -->
  87     <revhistory>
  88       <revision>
  89         <revnumber>0.6.2</revnumber>
  90         <date>2002-7-16</date>
  91         <authorinitials>rob@welcomehome.org</authorinitials>
  92         <revremark>Add new tutorial as a new chapter.</revremark>
  93       </revision>
  94       <revision>
  95         <revnumber>0.6.1</revnumber>
  96         <date>2001-2-16</date>
  97         <authorinitials>rob@welcomehome.org</authorinitials>
  98         <revremark>Add info on the new dejagnu.h file.</revremark>
  99       </revision>
 100       <revision>
 101         <revnumber>0.6</revnumber>
 102         <date>2001-2-16</date>
 103         <authorinitials>rob@welcomehome.org</authorinitials>
 104         <revremark>Updated for new release.</revremark>
 105       </revision>
 106       <revision>
 107         <revnumber>0.5</revnumber>
 108         <date>2000-1-24</date>
 109         <authorinitials>rob@welcomehome.org</authorinitials>
 110         <revremark>Initial version after conversion to DocBook.</revremark>
 111       </revision>
 112     </revhistory>
 113
 114   </bookinfo>
 115
 116   <toc></toc>
 117
 118   <preface id=preface>
 119     <title>Abstract</title>
 120
 121     <para>This document describes the functionality of DejaGnu, the
 122     testing framework of the GNU project. DejaGnu is written in
 123     <productname>Expect</productname>, which uses
 124     <productname>Tcl</productname> as a command
 125     language. <productname>Expect</productname> acts as a very
 126     programmable shell.  As with other Unix command shells, you can
 127     run any program, but once the program is started, your test script
 128     has programmable control over its input and output.  This does not
 129     just apply to the programs under test; <command>expect</command>
 130     can also run any auxiliary program, such as
 131     <command>diff</command> or <command>sh</command>, with full
 132     control over its input and output.</para>
 133
 134     <para>DejaGnu itself is merely a framework for the creation of
 135     testsuites. Testsuites are distributed with each
 136     application.</para>
 137
 138   </preface>
 139
 140   <chapter id=overview xreflabel=Overview>
 141     <title>Overview</title>
 142
 143     <sect1 id=whatis xreflabel="What is &dj; ?">
 144       <title>What is &dj; ?</title>
 145
 146     <para><productname>DejaGnu</productname> is a framework for
 147         testing other programs.  Its purpose is to provide a single
 148         front end for all tests. Think of it as a custom library of
 149         Tcl procedures crafted to support writing a test harness. A
 150         <emphasis>Test Harness</emphasis> is the testing
 151         infrastructure that is created to support a specific program
 152         or tool. Each program can have multiple testsuites, all
 153         supported by a single test harness. DejaGnu is written in
 154         <productname>Expect</productname>, which in turn uses
 155         <productname>Tcl</productname> -- Tool command
 156         language. There is more information on Tcl at the <ulink
 157         URL="http://www.scriptics.com">Scriptics</ulink> web site and the
 158         Expect web site is at <ulink
 159         URL="http://expect.nist.gov">NIST</ulink>.</para>
 160
 161    <para>Julia Menapace first coined the term ``DejaGnu'' to describe
 162         an earlier testing framework at Cygnus Support she had written
 163         for <command>GDB</command>. When we replaced it with the
 164         Expect-based framework, it was like DejaGnu all over again.
 165         More importantly, it was also named after my daughter, <ulink
 166         URL="mailto:deja@welcomehome.org">Deja Snow Savoye</ulink>
 167         (now 13 years old as of September 2003), who was a toddler
 168         during DejaGnu's beginnings.</para>
 169
 170     <para>DejaGnu offers several advantages for testing:</para>
 171
 172     <itemizedlist mark="bullet" spacing="compact">
 173
 174       <listitem><para>The flexibility and consistency of the DejaGnu
 175         framework make it easy to write tests for any program, with
 176         either batch oriented, or interactive programs.</para>
 177       </listitem>
 178
 179       <listitem><para>DejaGnu provides a layer of abstraction which
 180         allows you to write tests that are portable to any host or
 181         target where a program must be tested. For instance, a test
 182         for <command>GDB</command> can run from any supported host
 183         system on any supported target system. DejaGnu runs tests on
 184         many single board computers, whose operating software ranges
 185         from a simple boot monitor to a real-time OS.</para>
 186         </listitem>
 187
 188       <listitem><para>All tests have the same output format. This
 189         makes it easy to integrate testing into other software
 190         development processes. DejaGnu's output is designed to be
 191         parsed by other filtering script and it is also human
 192         readable.</para>
 193       </listitem>
 194
 195       <listitem><para>Using Tcl and Expect, it's easy to create wrappers
 196       for existing testsuites. By incorporating existing tests under
 197       DejaGnu, it's easier to have a single set of report analyse
 198       programs..</para>
 199
 200       </listitem>
 201     </itemizedlist>
 202
 203     <para>Running tests requires two things: the testing framework and
 204     the testsuites themselves. Tests are usually written in
 205     <productname>Expect</productname> using Tcl, but you can also use
 206     a Tcl script to run a testsuite that is not based on
 207     <productname>Expect</productname>. <productname>Expect</productname>
 208     script filenames conventionally use <emphasis>.exp</emphasis> as a
 209     suffix; for example, the main implementation of the DejaGnu test
 210     driver is in the file
 211     <productname>runtest.exp</productname>.)</para>
 212
 213   </sect1>
 214
 215   <sect1 id=new xreflabel="Release Notes">
 216     <title>What's New In This Release</title>
 217
 218     <para>This release has a number of substantial changes over version
 219     1.3. The most visible change is that the version of Expect and Tcl
 220     included in the release are up-to-date with the current stable net
 221     releases. The biggest change is years of modifications to the
 222     target configuration system, used for cross testing. While this
 223     greatly improved cross testing, is has made that subsystem very
 224     complicated. The goal is to have this entirely rewritten using
 225     <productname>iTcl</productname> by the next release. Other changes
 226     are:</para>
 227
 228     <itemizedlist>
 229       <listitem><para>More built-in support for building target binaries
 230         with the correct linker flags. Currently this only works with
 231         <productname>GCC</productname> as the cross compiler,
 232         preferably with a target supported by
 233         <xref linkend=libgloss>.</para></listitem>
 234
 235       <listitem><para>Lots of little bug fixes from years of heavy
 236         use at Cygnus Solutions.</para></listitem>
 237
 238       <listitem><para>DejaGnu now uses
 239       <productname>Automake</productname> for Makefile
 240       configuration.</para></listitem>
 241
 242       <listitem><para>Updated documentation, now in SGML
 243         (using the <ulink
 244         URL="http://nis-www.lanl.gov/~rosalia/mydocs/docbook-intro.html">free
 245         GNU DocBook tools</ulink>) format.</para></listitem>
 246
 247       <listitem><para>Windows support. There is beta level support for
 248       Windows that is still a work in progress. This requires the
 249       <ulink URL="http://www.cygwin.com/">Cygwin</ulink> POSIX
 250       subsystem for Windows.</para></listitem>
 251
 252     </itemizedlist>
 253
 254     <sect2 id=cygwin xreflabel="Windows Support">
 255       <title>Windows Support</title>
 256
 257       <para>To use DejaGnu on Windows, you need to first install the
 258         <ulink URL="http://www.cygwin.com/">Cygwin</ulink>
 259         release. This works as of the B20.1 release. Cygwin is a POSIX
 260         system for Windows. This covers both utility programs and a library
 261         that adds POSIX system calls to Windows. Among them is pseudo tty
 262         support for Windows that emulates the POSIX pty standard. The
 263         latest Cygwin is always available from <ulink
 264         URL="http://www.cygwin.com/">this location</ulink>. This
 265         works well enough to run <emphasis>"make check"</emphasis> of
 266         the GNU development tree on Windows after a native build. But the
 267         nature of ptys on Windows is still evolving. Your mileage may
 268         vary.</para>
 269
 270     </sect2>
 271
 272   </sect1>
 273
 274   <sect1 id=designgoals xreflabel="Design Goals">
 275     <title>Design Goals</title>
 276
 277     <para>DejaGnu grew out of the internal needs of Cygnus Solutions,
 278     the company formerly known as Cygnus Support. Cygnus maintained
 279     and enhanced a variety of free programs in many different
 280     environments and we needed a testing tool that:</para>
 281
 282     <itemizedlist mark="bullet">
 283       <listitem><para>was useful to developers while fixing
 284       bugs;</para></listitem>
 285       <listitem><para>automated running many tests during a software
 286       release process;</para></listitem>
 287       <listitem><para>was portable among a variety of host
 288       computers;</para></listitem>
 289       <listitem><para>supported cross-development
 290       testing;</para></listitem>
 291       <listitem><para>permitted testing interactive programs, like
 292       <command>GDB</command>; and </para></listitem>
 293       <listitem><para>permitted testing batch oriented programs, like
 294       <command>GCC</command>.</para></listitem>
 295     </itemizedlist>
 296
 297     <para>Some of the requirements proved challenging.  For example,
 298     interactive programs do not lend themselves very well to automated testing.
 299     But all the requirements are important: for instance, it is imperative to
 300     make sure that <command>GDB</command> works as well when cross-debugging
 301     as it does in a native configuration. </para>
 302
 303     <para>Probably the greatest challenge was testing in a
 304     cross-development environment.  Most cross-development
 305     environments are customized by each developer.  Even when buying
 306     packaged boards from vendors there are many differences.  The
 307     communication interfaces vary from a serial line to Ethernet.
 308     DejaGnu was designed with a modular communication setup, so that
 309     each kind of communication can be added as required and supported
 310     thereafter.  Once a communication procedure is coded, any test can
 311     use it.  Currently DejaGnu can use <command>rsh</command>,
 312     <command>rlogin</command>, <command>telnet</command>,
 313     <command>tip</command>, <command>kermit</command> and
 314     <command>mondfe</command> for remote communications.</para>
 315
 316     </sect1>
 317
 318     <sect1 id=posix xreflabel="A POSIX Conforming Test Framework">
 319       <title>A POSIX conforming test framework</title>
 320
 321       <para>DejaGnu conforms to the POSIX 1003.3 standard for test
 322       frameworks. Rob Savoye was a member of that committee.</para>
 323
 324       <para>The POSIX standard 1003.3 defines what a testing framework needs to
 325       provide, in order to permit the creation of POSIX conformance test
 326       suites. This standard is primarily oriented to running POSIX conformance
 327       tests, but its requirements also support testing of features not related
 328       to POSIX conformance.  POSIX 1003.3 does not specify a particular testing
 329       framework, but at this time there is only one other POSIX conforming test
 330       framework: TET. TET was created by Unisoft for a consortium comprised of
 331       X/Open, Unix International and the Open Software Foundation.</para>
 332
 333       <para>The POSIX documentation refers to <firstterm>assertions</firstterm>.
 334       An assertion is a description of behavior.  For example, if a standard
 335       says ``The sun shall shine'', a corresponding assertion might be ``The
 336       sun is shining.''  A test based on this assertion would pass or fail
 337       depending on whether it is day or night.  It is important to note
 338       that the standard being tested is never 1003.3; the standard being tested
 339       is some other standard, for which the assertions were written.</para>
 340
 341       <para>As there is no testsuite to test testing frameworks for POSIX
 342       1003.3 conformance, verifying conformance to this standard is done by
 343       repeatedly reading the standard and experimenting.  One of the main
 344       things 1003.3 does specify is the set of allowed output messages and
 345       their definitions.  Four messages are supported for a required feature of
 346       POSIX conforming systems and a fifth for a conditional feature. DejaGnu
 347       supports the use of all five output messages.  In this sense a testsuite
 348       that uses exactly these messages can be considered POSIX conforming.
 349       These definitions specify the output of a test
 350       case:</para>
 351
 352       <variablelist>
 353         <varlistentry>
 354           <term>PASS</term>
 355           <listitem><para>A test has succeeded.  That is, it demonstrated that
 356           the assertion is true.</para></listitem>
 357         </varlistentry>
 358
 359         <varlistentry>
 360           <term>XFAIL</term>
 361           <listitem><para>POSIX 1003.3 does not incorporate the notion of
 362           expected failures, so <emphasis>PASS</emphasis>, instead of
 363           <emphasis>XPASS</emphasis>, must also be returned for test cases
 364           which were expected to fail and did not.  This means that
 365           <emphasis>PASS</emphasis> is in some sense more ambiguous than if
 366           <emphasis>XPASS</emphasis> is also used.</para></listitem>
 367         </varlistentry>
 368
 369         <varlistentry>
 370           <term>FAIL</term>
 371           <listitem><para>A test has produced the bug it was intended to
 372           capture.  That is, it has demonstrated that the assertion is false.
 373           The <emphasis>FAIL</emphasis> message is based on the test case only.
 374           Other messages are used to indicate a failure of the framework. As
 375           with <emphasis>PASS</emphasis>, POSIX tests must return
 376           <emphasis>FAIL</emphasis> rather than <emphasis>XFAIL</emphasis> even
 377           if a failure was expected.</para></listitem>
 378         </varlistentry>
 379
 380         <varlistentry>
 381           <term>UNRESOLVED</term>
 382           <listitem><para>A test produced indeterminate results.  Usually, this
 383           means the test executed in an unexpected fashion; this outcome
 384           requires that a human being go over results, to determine if the test
 385           should have passed or failed.  This message is also used for any test
 386           that requires human intervention because it is beyond the abilities
 387           of the testing framework.  Any unresolved test should resolved to
 388           <emphasis>PASS</emphasis> or <emphasis>FAIL</emphasis> before a test
 389           run can be considered finished.</para>
 390
 391           <para>Note that for POSIX, each assertion must produce a test result
 392           code.  If the test isn't actually run, it must produce
 393           <emphasis>UNRESOLVED</emphasis> rather than just leaving that test
 394           out of the output.  This means that you have to be careful when
 395           writing tests to not carelessly use Tcl commands like
 396           <emphasis>return</emphasis>---if you alter the flow of control of the
 397           Tcl code you must insure that every test still produces some result
 398           code.</para>
 399
 400           <para>Here are some of the ways a test may wind up
 401           <emphasis>UNRESOLVED</emphasis>:</para></listitem>
 402
 403         </varlistentry>
 404         </variablelist>
 405
 406           <itemizedlist mark=bullet>
 407             <listitem><para>A test's execution is
 408             interrupted.</para></listitem>
 409
 410             <listitem><para>A test does not produce a clear
 411             result. This is usually because there was an
 412             <emphasis>ERROR</emphasis> from DejaGnu while processing
 413             the test, or because there were three or more
 414             <emphasis>WARNING</emphasis> messages. Any
 415             <emphasis>WARNING</emphasis> or <emphasis>ERROR</emphasis>
 416             messages can invalidate the output of the test.  This
 417             usually requires a human being to examine the output to
 418             determine what really happened---and to improve the test
 419             case.</para></listitem>
 420
 421             <listitem><para>A test depends on a previous test, which
 422             fails.</para></listitem>
 423
 424             <listitem><para>The test was set up
 425             incorrectly.</para></listitem>
 426           </itemizedlist>
 427
 428         <variablelist>
 429           <varlistentry>
 430             <term>UNTESTED</term>
 431             <listitem><para>A test was not run.  This is a place-holder, used
 432             when there is no real test case yet.</para></listitem>
 433           </varlistentry>
 434         </variablelist>
 435
 436           <para>The only remaining output message left is intended to test
 437           features that are specified by the applicable POSIX standard as
 438           conditional:</para>
 439
 440         <variablelist>
 441           <varlistentry>
 442             <term>UNSUPPORTED</term>
 443             <listitem><para>There is no support for the tested case.  This may
 444             mean that a conditional feature of an operating system, or of a
 445             compiler, is not implemented.  DejaGnu also uses this message when
 446             a testing environment (often a ``bare board'' target) lacks basic
 447             support for compiling or running the test case.  For example, a
 448             test for the system subroutine <emphasis>gethostname</emphasis>
 449             would never work on a target board running only a boot
 450             monitor.</para></listitem>
 451           </varlistentry>
 452         </variablelist>
 453
 454         <para>DejaGnu uses the same output procedures to produce these messages
 455         for all testsuites and these procedures are already known to conform
 456         to POSIX 1003.3.  For a DejaGnu testsuite to conform to POSIX 1003.3,
 457         you must avoid the <emphasis>setup</emphasis>xfail} procedure as
 458         described in the <emphasis>PASS</emphasis> section above and you must
 459         be careful to return <emphasis>UNRESOLVED</emphasis> where appropriate,
 460         as described in the <emphasis>UNRESOLVED</emphasis> section
 461         above.</para>
 462      </sect1>
 463
 464   </chapter>
 465
 466   <!-- include the user manual -->
 467   &user;
 468
 469   <!-- include the reference manual -->
 470   &ref;
 471
 472 </book>
 473
 474 <!-- Keep this comment at the end of the file
 475 Local variables:
 476 mode: sgml
 477 sgml-omittag:t
 478 sgml-shorttag:t
 479 sgml-namecase-general:t
 480 sgml-general-insert-case:lower
 481 sgml-minimize-attributes:nil
 482 sgml-always-quote-attributes:t
 483 sgml-indent-step:1
 484 sgml-indent-data:nil
 485 sgml-parent-document:nil
 486 sgml-exposed-tags:nil
 487 sgml-local-catalogs:nil
 488 sgml-local-ecat-files:nil
 489 End:
 490 -->