main/libstdc++-v3/doc/html/manual/test.html

   1 <?xml version="1.0" encoding="UTF-8" standalone="no"?>
   2 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><title>Test</title><meta name="generator" content="DocBook XSL-NS Stylesheets V1.78.1" /><meta name="keywords" content="ISO C++, test, testsuite, performance, conformance, ABI, exception safety" /><meta name="keywords" content="ISO C++, library" /><meta name="keywords" content="ISO C++, runtime, library" /><link rel="home" href="../index.html" title="The GNU C++ Library" /><link rel="up" href="appendix_porting.html" title="Appendix B.  Porting and Maintenance" /><link rel="prev" href="internals.html" title="Porting to New Hardware or Operating Systems" /><link rel="next" href="abi.html" title="ABI Policy and Guidelines" /></head><body><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Test</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="internals.html">Prev</a> </td><th width="60%" align="center">Appendix B.
   3   Porting and Maintenance
   4
   5 </th><td width="20%" align="right"> <a accesskey="n" href="abi.html">Next</a></td></tr></table><hr /></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a id="manual.intro.setup.test"></a>Test</h2></div></div></div><p>
   6 The libstdc++ testsuite includes testing for standard conformance,
   7 regressions, ABI, and performance.
   8 </p><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="test.organization"></a>Organization</h3></div></div></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="test.organization.layout"></a>Directory Layout</h4></div></div></div><p>
   9   The directory <span class="emphasis"><em>libsrcdir/testsuite</em></span> contains the
  10   individual test cases organized in sub-directories corresponding to
  11   clauses of the C++ standard (detailed below), the dejagnu test
  12   harness support files, and sources to various testsuite utilities
  13   that are packaged in a separate testing library.
  14 </p><p>
  15   All test cases for functionality required by the runtime components
  16   of the C++ standard (ISO 14882) are files within the following
  17   directories.
  18 </p><pre class="programlisting">
  19 17_intro
  20 18_support
  21 19_diagnostics
  22 20_util
  23 21_strings
  24 22_locale
  25 23_containers
  26 25_algorithms
  27 26_numerics
  28 27_io
  29 28_regex
  30 29_atomics
  31 30_threads
  32    </pre><p>
  33       In addition, the following directories include test files:
  34    </p><pre class="programlisting">
  35 tr1               Tests for components as described by the Technical Report on Standard Library Extensions (TR1).
  36 backward          Tests for backwards compatibility and deprecated features.
  37 demangle          Tests for __cxa_demangle, the IA 64 C++ ABI demangler
  38 ext               Tests for extensions.
  39 performance       Tests for performance analysis, and performance regressions.
  40    </pre><p>
  41       Some directories don't have test files, but instead contain
  42       auxiliary information:
  43    </p><pre class="programlisting">
  44 config            Files for the dejagnu test harness.
  45 lib               Files for the dejagnu test harness.
  46 libstdc++*        Files for the dejagnu test harness.
  47 data              Sample text files for testing input and output.
  48 util              Files for libtestc++, utilities and testing routines.
  49    </pre><p>
  50       Within a directory that includes test files, there may be
  51       additional subdirectories, or files.  Originally, test cases
  52       were appended to one file that represented a particular section
  53       of the chapter under test, and was named accordingly. For
  54       instance, to test items related to <code class="code"> 21.3.6.1 -
  55       basic_string::find [lib.string::find]</code> in the standard,
  56       the following was used:
  57    </p><pre class="programlisting">
  58 21_strings/find.cc
  59    </pre><p>
  60       However, that practice soon became a liability as the test cases
  61       became huge and unwieldy, and testing new or extended
  62       functionality (like wide characters or named locales) became
  63       frustrating, leading to aggressive pruning of test cases on some
  64       platforms that covered up implementation errors. Now, the test
  65       suite has a policy of one file, one test case, which solves the
  66       above issues and gives finer grained results and more manageable
  67       error debugging. As an example, the test case quoted above
  68       becomes:
  69    </p><pre class="programlisting">
  70 21_strings/basic_string/find/char/1.cc
  71 21_strings/basic_string/find/char/2.cc
  72 21_strings/basic_string/find/char/3.cc
  73 21_strings/basic_string/find/wchar_t/1.cc
  74 21_strings/basic_string/find/wchar_t/2.cc
  75 21_strings/basic_string/find/wchar_t/3.cc
  76    </pre><p>
  77       All new tests should be written with the policy of one test
  78       case, one file in mind.
  79    </p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="test.organization.naming"></a>Naming Conventions</h4></div></div></div><p>
  80       In addition, there are some special names and suffixes that are
  81       used within the testsuite to designate particular kinds of
  82       tests.
  83    </p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p>
  84    <span class="emphasis"><em>_xin.cc</em></span>
  85   </p><p>
  86       This test case expects some kind of interactive input in order
  87       to finish or pass. At the moment, the interactive tests are not
  88       run by default. Instead, they are run by hand, like:
  89    </p><pre class="programlisting">
  90 g++ 27_io/objects/char/3_xin.cc
  91 cat 27_io/objects/char/3_xin.in | a.out
  92      </pre></li><li class="listitem"><p>
  93      <span class="emphasis"><em>.in</em></span>
  94    </p><p>
  95       This file contains the expected input for the corresponding <span class="emphasis"><em>
  96       _xin.cc</em></span> test case.
  97    </p></li><li class="listitem"><p>
  98    <span class="emphasis"><em>_neg.cc</em></span>
  99   </p><p>
 100       This test case is expected to fail: it's a negative test. At the
 101       moment, these are almost always compile time errors.
 102    </p></li><li class="listitem"><p>
 103    <span class="emphasis"><em>char</em></span>
 104   </p><p>
 105       This can either be a directory name or part of a longer file
 106       name, and indicates that this file, or the files within this
 107       directory are testing the <code class="code">char</code> instantiation of a
 108       template.
 109    </p></li><li class="listitem"><p>
 110    <span class="emphasis"><em>wchar_t</em></span>
 111   </p><p>
 112       This can either be a directory name or part of a longer file
 113       name, and indicates that this file, or the files within this
 114       directory are testing the <code class="code">wchar_t</code> instantiation of
 115       a template. Some hosts do not support <code class="code">wchar_t</code>
 116       functionality, so for these targets, all of these tests will not
 117       be run.
 118    </p></li><li class="listitem"><p>
 119    <span class="emphasis"><em>thread</em></span>
 120   </p><p>
 121       This can either be a directory name or part of a longer file
 122       name, and indicates that this file, or the files within this
 123       directory are testing situations where multiple threads are
 124       being used.
 125    </p></li><li class="listitem"><p>
 126    <span class="emphasis"><em>performance</em></span>
 127   </p><p>
 128       This can either be an enclosing directory name or part of a
 129       specific file name. This indicates a test that is used to
 130       analyze runtime performance, for performance regression testing,
 131       or for other optimization related analysis. At the moment, these
 132       test cases are not run by default.
 133    </p></li></ul></div></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="test.run"></a>Running the Testsuite</h3></div></div></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="test.run.basic"></a>Basic</h4></div></div></div><p>
 134       You can check the status of the build without installing it
 135       using the dejagnu harness, much like the rest of the gcc
 136       tools.</p><pre class="programlisting"> make check</pre><p>in the <span class="emphasis"><em>libbuilddir</em></span> directory.</p><p>or</p><pre class="programlisting"> make check-target-libstdc++-v3</pre><p>in the <span class="emphasis"><em>gccbuilddir</em></span> directory.
 137      </p><p>
 138        These commands are functionally equivalent and will create a
 139        'testsuite' directory underneath
 140        <span class="emphasis"><em>libbuilddir</em></span> containing the results of the
 141        tests. Two results files will be generated: <span class="emphasis"><em>
 142        libstdc++.sum</em></span>, which is a PASS/FAIL summary for each
 143        test, and <span class="emphasis"><em>libstdc++.log</em></span> which is a log of
 144        the exact command line passed to the compiler, the compiler
 145        output, and the executable output (if any).
 146      </p><p>
 147        Archives of test results for various versions and platforms are
 148        available on the GCC website in the <a class="link" href="http://gcc.gnu.org/gcc-4.3/buildstat.html" target="_top">build
 149        status</a> section of each individual release, and are also
 150        archived on a daily basis on the <a class="link" href="http://gcc.gnu.org/ml/gcc-testresults/current" target="_top">gcc-testresults</a>
 151        mailing list. Please check either of these places for a similar
 152        combination of source version, operating system, and host CPU.
 153      </p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="test.run.variations"></a>Variations</h4></div></div></div><p>
 154       There are several options for running tests, including testing
 155       the regression tests, testing a subset of the regression tests,
 156       testing the performance tests, testing just compilation, testing
 157       installed tools, etc. In addition, there is a special rule for
 158       checking the exported symbols of the shared library.
 159     </p><p>
 160       To debug the dejagnu test harness during runs, try invoking with a
 161       specific argument to the variable RUNTESTFLAGS, as below.
 162     </p><pre class="programlisting">
 163 make check-target-libstdc++-v3 RUNTESTFLAGS="-v"
 164 </pre><p>
 165       or
 166     </p><pre class="programlisting">
 167 make check-target-libstdc++-v3 RUNTESTFLAGS="-v -v"
 168 </pre><p>
 169       To run a subset of the library tests, you will need to generate
 170       the <span class="emphasis"><em>testsuite_files</em></span> file by running
 171       <span class="command"><strong>make testsuite_files</strong></span> in the
 172       <span class="emphasis"><em>libbuilddir/testsuite</em></span> directory, described
 173       below.  Edit the file to remove the tests you don't want and
 174       then run the testsuite as normal.
 175     </p><p>
 176       There are two ways to run on a simulator: set up DEJAGNU to point to a
 177       specially crafted site.exp, or pass down --target_board flags.
 178     </p><p>
 179     Example flags to pass down for various embedded builds are as follows:
 180     </p><pre class="programlisting">
 181       --target=powerpc-eabism (libgloss/sim)
 182 make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=powerpc-sim"
 183
 184 --target=calmrisc32 (libgloss/sid)
 185 make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=calmrisc32-sid"
 186
 187 --target=xscale-elf (newlib/sim)
 188 make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=arm-sim"
 189 </pre><p>
 190       Also, here is an example of how to run the libstdc++ testsuite
 191       for a multilibed build directory with different ABI settings:
 192     </p><pre class="programlisting">
 193 make check-target-libstdc++-v3 RUNTESTFLAGS='--target_board \"unix{-mabi=32,,-mabi=64}\"'
 194 </pre><p>
 195       You can run the tests with a compiler and library that have
 196       already been installed.  Make sure that the compiler (e.g.,
 197       <code class="code">g++</code>) is in your <code class="code">PATH</code>.  If you are
 198       using shared libraries, then you must also ensure that the
 199       directory containing the shared version of libstdc++ is in your
 200       <code class="code">LD_LIBRARY_PATH</code>, or equivalent.  If your GCC source
 201       tree is at <code class="code">/path/to/gcc</code>, then you can run the tests
 202       as follows:
 203     </p><pre class="programlisting">
 204 runtest --tool libstdc++ --srcdir=/path/to/gcc/libstdc++-v3/testsuite
 205 </pre><p>
 206       The testsuite will create a number of files in the directory in
 207       which you run this command,.  Some of those files might use the
 208       same name as files created by other testsuites (like the ones
 209       for GCC and G++), so you should not try to run all the
 210       testsuites in parallel from the same directory.
 211     </p><p>
 212       In addition, there are some testing options that are mostly of
 213       interest to library maintainers and system integrators. As such,
 214       these tests may not work on all cpu and host combinations, and
 215       may need to be executed in the
 216       <span class="emphasis"><em>libbuilddir/testsuite</em></span> directory.  These
 217       options include, but are not necessarily limited to, the
 218       following:
 219    </p><pre class="programlisting">
 220    make testsuite_files
 221    </pre><p>
 222     Five files are generated that determine what test files
 223     are run. These files are:
 224   </p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p>
 225          <span class="emphasis"><em>testsuite_files</em></span>
 226        </p><p>
 227          This is a list of all the test cases that will be run. Each
 228          test case is on a separate line, given with an absolute path
 229          from the <span class="emphasis"><em>libsrcdir/testsuite</em></span> directory.
 230        </p></li><li class="listitem"><p>
 231          <span class="emphasis"><em>testsuite_files_interactive</em></span>
 232        </p><p>
 233          This is a list of all the interactive test cases, using the
 234          same format as the file list above. These tests are not run
 235          by default.
 236      </p></li><li class="listitem"><p>
 237          <span class="emphasis"><em>testsuite_files_performance</em></span>
 238        </p><p>
 239          This is a list of all the performance test cases, using the
 240          same format as the file list above. These tests are not run
 241          by default.
 242      </p></li><li class="listitem"><p>
 243          <span class="emphasis"><em>testsuite_thread</em></span>
 244        </p><p>
 245          This file indicates that the host system can run tests which
 246          involved multiple threads.
 247        </p></li><li class="listitem"><p>
 248          <span class="emphasis"><em>testsuite_wchar_t</em></span>
 249        </p><p>
 250          This file indicates that the host system can run the wchar_t
 251          tests, and corresponds to the macro definition <code class="code">
 252          _GLIBCXX_USE_WCHAR_T</code> in the file c++config.h.
 253        </p></li></ul></div><pre class="programlisting">
 254    make check-abi
 255    </pre><p>
 256      The library ABI can be tested. This involves testing the shared
 257      library against an ABI-defining previous version of symbol
 258      exports.
 259    </p><pre class="programlisting">
 260    make check-compile
 261   </pre><p>
 262      This rule compiles, but does not link or execute, the
 263      <span class="emphasis"><em>testsuite_files</em></span> test cases and displays the
 264      output on stdout.
 265    </p><pre class="programlisting">
 266    make check-performance
 267    </pre><p>
 268      This rule runs through the
 269      <span class="emphasis"><em>testsuite_files_performance</em></span> test cases and
 270      collects information for performance analysis and can be used to
 271      spot performance regressions. Various timing information is
 272      collected, as well as number of hard page faults, and memory
 273      used. This is not run by default, and the implementation is in
 274      flux.
 275    </p><p>
 276       We are interested in any strange failures of the testsuite;
 277       please email the main libstdc++ mailing list if you see
 278       something odd or have questions.
 279    </p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="test.run.permutations"></a>Permutations</h4></div></div></div><p>
 280       To run the libstdc++ test suite under the <a class="link" href="debug_mode.html" title="Chapter 17. Debug Mode">debug mode</a>, edit
 281       <code class="filename">libstdc++-v3/scripts/testsuite_flags</code> to add the
 282       compile-time flag <code class="constant">-D_GLIBCXX_DEBUG</code> to the
 283       result printed by the <code class="literal">--build-cxx</code>
 284       option. Additionally, add the
 285       <code class="constant">-D_GLIBCXX_DEBUG_PEDANTIC</code> flag to turn on
 286       pedantic checking. The libstdc++ test suite should produce
 287       precisely the same results under debug mode that it does under
 288       release mode: any deviation indicates an error in either the
 289       library or the test suite.
 290     </p><p>
 291       The <a class="link" href="parallel_mode.html" title="Chapter 18. Parallel Mode">parallel
 292       mode</a> can be tested in much the same manner, substituting
 293       <code class="constant">-D_GLIBCXX_PARALLEL</code> for
 294       <code class="constant">-D_GLIBCXX_DEBUG</code> in the previous paragraph.
 295     </p><p>
 296       Or, just run the testsuites with <code class="constant">CXXFLAGS</code>
 297       set to <code class="constant">-D_GLIBCXX_DEBUG</code> or
 298       <code class="constant">-D_GLIBCXX_PARALLEL</code>.
 299     </p></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="test.new_tests"></a>Writing a new test case</h3></div></div></div><p>
 300     The first step in making a new test case is to choose the correct
 301     directory and file name, given the organization as previously
 302     described.
 303    </p><p>
 304     All files are copyright the FSF, and GPL'd: this is very
 305     important.  The first copyright year should correspond to the date
 306     the file was checked in to SVN.
 307    </p><p>
 308      As per the dejagnu instructions, always return 0 from main to
 309      indicate success.
 310    </p><p>
 311    A bunch of utility functions and classes have already been
 312    abstracted out into the testsuite utility library, <code class="code">
 313    libtestc++</code>. To use this functionality, just include the
 314    appropriate header file: the library or specific object files will
 315    automatically be linked in as part of the testsuite run.
 316    </p><p>
 317    For a test that needs to take advantage of the dejagnu test
 318    harness, what follows below is a list of special keyword that
 319    harness uses. Basically, a test case contains dg-keywords (see
 320    dg.exp) indicating what to do and what kinds of behavior are to be
 321    expected.  New test cases should be written with the new style
 322    DejaGnu framework in mind.
 323    </p><p>
 324     To ease transition, here is the list of dg-keyword documentation
 325     lifted from dg.exp.
 326    </p><pre class="programlisting">
 327 # The currently supported options are:
 328 #
 329 # dg-prms-id N
 330 #       set prms_id to N
 331 #
 332 # dg-options "options ..." [{ target selector }]
 333 #       specify special options to pass to the tool (eg: compiler)
 334 #
 335 # dg-do do-what-keyword [{ target/xfail selector }]
 336 #       `do-what-keyword' is tool specific and is passed unchanged to
 337 #       ${tool}-dg-test.  An example is gcc where `keyword' can be any of:
 338 #       preprocess|compile|assemble|link|run
 339 #       and will do one of: produce a .i, produce a .s, produce a .o,
 340 #       produce an a.out, or produce an a.out and run it (the default is
 341 #       compile).
 342 #
 343 # dg-error regexp comment [{ target/xfail selector } [{.|0|linenum}]]
 344 #       indicate an error message &lt;regexp&gt; is expected on this line
 345 #       (the test fails if it doesn't occur)
 346 #       Linenum=0 for general tool messages (eg: -V arg missing).
 347 #       "." means the current line.
 348 #
 349 # dg-warning regexp comment [{ target/xfail selector } [{.|0|linenum}]]
 350 #       indicate a warning message &lt;regexp&gt; is expected on this line
 351 #       (the test fails if it doesn't occur)
 352 #
 353 # dg-bogus regexp comment [{ target/xfail selector } [{.|0|linenum}]]
 354 #       indicate a bogus error message &lt;regexp&gt; use to occur here
 355 #       (the test fails if it does occur)
 356 #
 357 # dg-build regexp comment [{ target/xfail selector }]
 358 #       indicate the build use to fail for some reason
 359 #       (errors covered here include bad assembler generated, tool crashes,
 360 #       and link failures)
 361 #       (the test fails if it does occur)
 362 #
 363 # dg-excess-errors comment [{ target/xfail selector }]
 364 #       indicate excess errors are expected (any line)
 365 #       (this should only be used sparingly and temporarily)
 366 #
 367 # dg-output regexp [{ target selector }]
 368 #       indicate the expected output of the program is &lt;regexp&gt;
 369 #       (there may be multiple occurrences of this, they are concatenated)
 370 #
 371 # dg-final { tcl code }
 372 #       add some tcl code to be run at the end
 373 #       (there may be multiple occurrences of this, they are concatenated)
 374 #       (unbalanced braces must be \-escaped)
 375 #
 376 # "{ target selector }" is a list of expressions that determine whether the
 377 # test succeeds or fails for a particular target, or in some cases whether the
 378 # option applies for a particular target.  If the case of `dg-do' it specifies
 379 # whether the test case is even attempted on the specified target.
 380 #
 381 # The target selector is always optional.  The format is one of:
 382 #
 383 # { xfail *-*-* ... } - the test is expected to fail for the given targets
 384 # { target *-*-* ... } - the option only applies to the given targets
 385 #
 386 # At least one target must be specified, use *-*-* for "all targets".
 387 # At present it is not possible to specify both `xfail' and `target'.
 388 # "native" may be used in place of "*-*-*".
 389
 390 Example 1: Testing compilation only
 391 // { dg-do compile }
 392
 393 Example 2: Testing for expected warnings on line 36, which all targets fail
 394 // { dg-warning "string literals" "" { xfail *-*-* } 36 }
 395
 396 Example 3: Testing for expected warnings on line 36
 397 // { dg-warning "string literals" "" { target *-*-* } 36 }
 398
 399 Example 4: Testing for compilation errors on line 41
 400 // { dg-do compile }
 401 // { dg-error "no match for" "" { target *-*-* } 41 }
 402
 403 Example 5: Testing with special command line settings, or without the
 404 use of pre-compiled headers, in particular the stdc++.h.gch file. Any
 405 options here will override the DEFAULT_CXXFLAGS and PCH_CXXFLAGS set
 406 up in the normal.exp file.
 407 // { dg-options "-O0" { target *-*-* } }
 408 </pre><p>
 409     More examples can be found in the libstdc++-v3/testsuite/*/*.cc files.
 410    </p></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="test.harness"></a>Test Harness and Utilities</h3></div></div></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="test.harness.dejagnu"></a>Dejagnu Harness Details</h4></div></div></div><p>
 411     Underlying details of testing for conformance and regressions are
 412     abstracted via the GNU Dejagnu package. This is similar to the
 413     rest of GCC.
 414   </p><p>This is information for those looking at making changes to the testsuite
 415 structure, and/or needing to trace dejagnu's actions with --verbose.  This
 416 will not be useful to people who are "merely" adding new tests to the existing
 417 structure.
 418 </p><p>The first key point when working with dejagnu is the idea of a "tool".
 419 Files, directories, and functions are all implicitly used when they are
 420 named after the tool in use.  Here, the tool will always be "libstdc++".
 421 </p><p>The <code class="code">lib</code> subdir contains support routines.  The
 422 <code class="code">lib/libstdc++.exp</code> file ("support library") is loaded
 423 automagically, and must explicitly load the others.  For example, files can
 424 be copied from the core compiler's support directory into <code class="code">lib</code>.
 425 </p><p>Some routines in <code class="code">lib/libstdc++.exp</code> are callbacks, some are
 426 our own.  Callbacks must be prefixed with the name of the tool.  To easily
 427 distinguish the others, by convention our own routines are named "v3-*".
 428 </p><p>The next key point when working with dejagnu is "test files".  Any
 429 directory whose name starts with the tool name will be searched for test files.
 430 (We have only one.)  In those directories, any <code class="code">.exp</code> file is
 431 considered a test file, and will be run in turn.  Our main test file is called
 432 <code class="code">normal.exp</code>; it runs all the tests in testsuite_files using the
 433 callbacks loaded from the support library.
 434 </p><p>The <code class="code">config</code> directory is searched for any particular "target
 435 board" information unique to this library.  This is currently unused and sets
 436 only default variables.
 437 </p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="test.harness.utils"></a>Utilities</h4></div></div></div><p>
 438   </p><p>
 439    The testsuite directory also contains some files that implement
 440    functionality that is intended to make writing test cases easier,
 441    or to avoid duplication, or to provide error checking in a way that
 442    is consistent across platforms and test harnesses. A stand-alone
 443    executable, called <span class="emphasis"><em>abi_check</em></span>, and a static
 444    library called <span class="emphasis"><em>libtestc++</em></span> are
 445    constructed. Both of these items are not installed, and only used
 446    during testing.
 447   </p><p>
 448   These files include the following functionality:
 449   </p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p>
 450        <span class="emphasis"><em>testsuite_abi.h</em></span>,
 451        <span class="emphasis"><em>testsuite_abi.cc</em></span>,
 452        <span class="emphasis"><em>testsuite_abi_check.cc</em></span>
 453        </p><p>
 454         Creates the executable <span class="emphasis"><em>abi_check</em></span>.
 455         Used to check correctness of symbol versioning, visibility of
 456         exported symbols, and compatibility on symbols in the shared
 457         library, for hosts that support this feature. More information
 458         can be found in the ABI documentation <a class="link" href="abi.html" title="ABI Policy and Guidelines">here</a>
 459        </p></li><li class="listitem"><p>
 460        <span class="emphasis"><em>testsuite_allocator.h</em></span>,
 461        <span class="emphasis"><em>testsuite_allocator.cc</em></span>
 462        </p><p>
 463         Contains specialized allocators that keep track of construction
 464         and destruction. Also, support for overriding global new and
 465         delete operators, including verification that new and delete
 466         are called during execution, and that allocation over max_size
 467         fails.
 468        </p></li><li class="listitem"><p>
 469        <span class="emphasis"><em>testsuite_character.h</em></span>
 470        </p><p>
 471         Contains <code class="code">std::char_traits</code> and
 472         <code class="code">std::codecvt</code> specializations for a user-defined
 473         POD.
 474        </p></li><li class="listitem"><p>
 475        <span class="emphasis"><em>testsuite_hooks.h</em></span>,
 476        <span class="emphasis"><em>testsuite_hooks.cc</em></span>
 477        </p><p>
 478        A large number of utilities, including:
 479        </p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: circle; "><li class="listitem"><p>VERIFY</p></li><li class="listitem"><p>set_memory_limits</p></li><li class="listitem"><p>verify_demangle</p></li><li class="listitem"><p>run_tests_wrapped_locale</p></li><li class="listitem"><p>run_tests_wrapped_env</p></li><li class="listitem"><p>try_named_locale</p></li><li class="listitem"><p>try_mkfifo</p></li><li class="listitem"><p>func_callback</p></li><li class="listitem"><p>counter</p></li><li class="listitem"><p>copy_tracker</p></li><li class="listitem"><p>copy_constructor</p></li><li class="listitem"><p>assignment_operator</p></li><li class="listitem"><p>destructor</p></li><li class="listitem"><p>pod_char, pod_int and associated char_traits specializations</p></li></ul></div></li><li class="listitem"><p>
 480          <span class="emphasis"><em>testsuite_io.h</em></span>
 481        </p><p>
 482        Error, exception, and constraint checking for
 483        <code class="code">std::streambuf, std::basic_stringbuf, std::basic_filebuf</code>.
 484        </p></li><li class="listitem"><p>
 485          <span class="emphasis"><em>testsuite_iterators.h</em></span>
 486        </p><p>
 487          Wrappers for various iterators.
 488        </p></li><li class="listitem"><p>
 489          <span class="emphasis"><em>testsuite_performance.h</em></span>
 490        </p><p>
 491        A number of class abstractions for performance counters, and
 492        reporting functions including:
 493        </p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: circle; "><li class="listitem"><p>time_counter</p></li><li class="listitem"><p>resource_counter</p></li><li class="listitem"><p>report_performance</p></li></ul></div></li></ul></div></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="test.special"></a>Special Topics</h3></div></div></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="test.exception.safety"></a>
 494   Qualifying Exception Safety Guarantees
 495   <a id="idm269884641824" class="indexterm"></a>
 496 </h4></div></div></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a id="test.exception.safety.overview"></a>Overview</h5></div></div></div><p>
 497          Testing is composed of running a particular test sequence,
 498          and looking at what happens to the surrounding code when
 499          exceptions are thrown. Each test is composed of measuring
 500          initial state, executing a particular sequence of code under
 501          some instrumented conditions, measuring a final state, and
 502          then examining the differences between the two states.
 503        </p><p>
 504          Test sequences are composed of constructed code sequences
 505          that exercise a particular function or member function, and
 506          either confirm no exceptions were generated, or confirm the
 507          consistency/coherency of the test subject in the event of a
 508          thrown exception.
 509        </p><p>
 510          Random code paths can be constructed using the basic test
 511          sequences and instrumentation as above, only combined in a
 512          random or pseudo-random way.
 513        </p><p> To compute the code paths that throw, test instruments
 514          are used that throw on allocation events
 515          (<code class="classname">__gnu_cxx::throw_allocator_random</code>
 516          and <code class="classname">__gnu_cxx::throw_allocator_limit</code>)
 517          and copy, assignment, comparison, increment, swap, and
 518          various operators
 519          (<code class="classname">__gnu_cxx::throw_type_random</code>
 520          and <code class="classname">__gnu_cxx::throw_type_limit</code>). Looping
 521          through a given test sequence and conditionally throwing in
 522          all instrumented places.  Then, when the test sequence
 523          completes without an exception being thrown, assume all
 524          potential error paths have been exercised in a sequential
 525          manner.
 526        </p></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a id="test.exception.safety.status"></a>
 527     Existing tests
 528 </h5></div></div></div><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p>
 529          Ad Hoc
 530        </p><p>
 531          For example,
 532          <code class="filename">testsuite/23_containers/list/modifiers/3.cc</code>.
 533        </p></li><li class="listitem"><p>
 534          Policy Based Data Structures
 535        </p><p>
 536          For example, take the test
 537          functor <code class="classname">rand_reg_test</code> in
 538          in <code class="filename">testsuite/ext/pb_ds/regression/tree_no_data_map_rand.cc</code>. This uses <code class="classname">container_rand_regression_test</code> in
 539 <code class="filename">testsuite/util/regression/rand/assoc/container_rand_regression_test.h</code>.
 540
 541        </p><p>
 542          Which has several tests for container member functions,
 543 Includes control and test container objects. Configuration includes
 544 random seed, iterations, number of distinct values, and the
 545 probability that an exception will be thrown. Assumes instantiating
 546 container uses an extension
 547 allocator, <code class="classname">__gnu_cxx::throw_allocator_random</code>,
 548 as the allocator type.
 549        </p></li><li class="listitem"><p>
 550          C++11 Container Requirements.
 551        </p><p>
 552          Coverage is currently limited to testing container
 553          requirements for exception safety,
 554          although <code class="classname">__gnu_cxx::throw_type</code> meets
 555          the additional type requirements for testing numeric data
 556          structures and instantiating algorithms.
 557        </p><p>
 558          Of particular interest is extending testing to algorithms and
 559          then to parallel algorithms. Also io and locales.
 560        </p><p>
 561          The test instrumentation should also be extended to add
 562          instrumentation to <code class="classname">iterator</code>
 563          and <code class="classname">const_iterator</code> types that throw
 564          conditionally on iterator operations.
 565        </p></li></ul></div></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a id="test.exception.safety.containers"></a>
 566 C++11 Requirements Test Sequence Descriptions
 567 </h5></div></div></div><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p>
 568          Basic
 569        </p><p>
 570          Basic consistency on exception propagation tests. For
 571          each container, an object of that container is constructed,
 572          a specific member function is exercised in
 573          a <code class="literal">try</code> block, and then any thrown
 574          exceptions lead to error checking in the appropriate
 575          <code class="literal">catch</code> block. The container's use of
 576          resources is compared to the container's use prior to the
 577          test block. Resource monitoring is limited to allocations
 578          made through the container's <span class="type">allocator_type</span>,
 579          which should be sufficient for container data
 580          structures. Included in these tests are member functions
 581          are <span class="type">iterator</span> and <span class="type">const_iterator</span>
 582          operations, <code class="function">pop_front</code>, <code class="function">pop_back</code>, <code class="function">push_front</code>, <code class="function">push_back</code>, <code class="function">insert</code>, <code class="function">erase</code>, <code class="function">swap</code>, <code class="function">clear</code>,
 583          and <code class="function">rehash</code>. The container in question is
 584          instantiated with two instrumented template arguments,
 585          with <code class="classname">__gnu_cxx::throw_allocator_limit</code>
 586          as the allocator type, and
 587          with <code class="classname">__gnu_cxx::throw_type_limit</code> as
 588          the value type. This allows the test to loop through
 589          conditional throw points.
 590        </p><p>
 591          The general form is demonstrated in
 592          <code class="filename">testsuite/23_containers/list/requirements/exception/basic.cc
 593          </code>. The instantiating test object is <code class="classname">__gnu_test::basic_safety</code> and is detailed in <code class="filename">testsuite/util/exception/safety.h</code>.
 594        </p></li><li class="listitem"><p>
 595          Generation Prohibited
 596        </p><p>
 597          Exception generation tests. For each container, an object of
 598          that container is constructed and all member functions
 599          required to not throw exceptions are exercised. Included in
 600          these tests are member functions
 601          are <span class="type">iterator</span> and <span class="type">const_iterator</span> operations, <code class="function">erase</code>, <code class="function">pop_front</code>, <code class="function">pop_back</code>, <code class="function">swap</code>,
 602          and <code class="function">clear</code>. The container in question is
 603          instantiated with two instrumented template arguments,
 604          with <code class="classname">__gnu_cxx::throw_allocator_random</code>
 605          as the allocator type, and
 606          with <code class="classname">__gnu_cxx::throw_type_random</code> as
 607          the value type. This test does not loop, an instead is sudden
 608          death: first error fails.
 609        </p><p>
 610          The general form is demonstrated in
 611          <code class="filename">testsuite/23_containers/list/requirements/exception/generation_prohibited.cc
 612          </code>. The instantiating test object is <code class="classname">__gnu_test::generation_prohibited</code> and is detailed in <code class="filename">testsuite/util/exception/safety.h</code>.
 613        </p></li><li class="listitem"><p>
 614          Propagation Consistent
 615        </p><p>
 616          Container rollback on exception propagation tests. For
 617          each container, an object of that container is constructed,
 618          a specific member function that requires rollback to a previous
 619          known good state is exercised in
 620          a <code class="literal">try</code> block, and then any thrown
 621          exceptions lead to error checking in the appropriate
 622          <code class="literal">catch</code> block. The container is compared to
 623          the container's last known good state using such parameters
 624          as size, contents, and iterator references. Included in these
 625          tests are member functions
 626          are <code class="function">push_front</code>, <code class="function">push_back</code>, <code class="function">insert</code>,
 627          and <code class="function">rehash</code>. The container in question is
 628          instantiated with two instrumented template arguments,
 629          with <code class="classname">__gnu_cxx::throw_allocator_limit</code>
 630          as the allocator type, and
 631          with <code class="classname">__gnu_cxx::throw_type_limit</code> as
 632          the value type. This allows the test to loop through
 633          conditional throw points.
 634        </p><p>
 635          The general form demonstrated in
 636          <code class="filename">testsuite/23_containers/list/requirements/exception/propagation_coherent.cc
 637          </code>. The instantiating test object is <code class="classname">__gnu_test::propagation_coherent</code> and is detailed in <code class="filename">testsuite/util/exception/safety.h</code>.
 638        </p></li></ul></div></div></div></div></div><div class="navfooter"><hr /><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="internals.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="appendix_porting.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="abi.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">Porting to New Hardware or Operating Systems </td><td width="20%" align="center"><a accesskey="h" href="../index.html">Home</a></td><td width="40%" align="right" valign="top"> ABI Policy and Guidelines</td></tr></table></div></body></html>