4 You can use QMTest to test G++. (In the future, it may be possible to
5 test other parts of GCC with QMTest as well, but it is not possible
8 The use of QMTest to run the G++ tests has not been approved as an
9 officially supported testing procedure. Therefore, you must run the
10 tests using DejaGNU (with "make check-g++") before committing changes
13 Differences from DejaGNU
14 ========================
16 Although QMTest can be used to run the G++ test suite, it works
17 somewhat differently from DejaGNU. In particular:
19 - In DejaGNU, a single source file contains many tests. Each
20 line where a diagnostic is expected is considered a separate
21 test. Testing for successful compilation and testing for
22 successful execution of the generated program are considered
23 separate tests. Thus, a test "test.C" could contain, say,
24 seven tests; some of which might pass and some of which might
27 With QMTest, each source file is considered a single test. If any
28 of the seven sub-tests fail, the entire test is considered to fail.
29 However, QMTest does present information about *why* the test
30 failed, so the same information is effectively available.
32 It is true that, therefore, causing an already failing test to "fail
33 more" is not immediately detectable through an additional unexpected
34 failure messages when using QMTest. On the other hand, most people
35 seem to think of each source file as "a test", not "twelve tests",
36 so the model QMTest uses may be more natural.
38 - In DejaGNU, tests themselves keep track of expected and unexpected
39 failures. The QMTest philosophy is that expected failures should be
40 stored separately from the tests themselves; in particular, that
41 tonights results can be tomorrow's expectations. In order to
42 preserve compatibility with DejaGNU, the first time you use QMTest
43 to test G++, QMTest computes the set of expected failures indicated
44 by the tests, and then compares the actual results with these
45 results. Therefore, if you change the expected failure notations in
46 the DejaGNU tests, you must rebuild the set of expected failures.
48 To do this, remove the file "qmtestsuite/gpp-expected.qmr". Then,
49 when you rerun the tests, the expected failures will be
50 automatically recalculated.
55 You must download and install the following software:
57 - Python 2.1 (or greater)
59 See http://www.python.org.
61 You may already have Python on your system; in particular, many
62 GNU/Linux systems ship with Python installed.
64 Installation instructions are available on the web-site.
66 - QMTest 1.1.4 (or greater)
68 See http://www.qmtest.com.
70 QMTest is available at:
72 http://www.codesourcery.com/qm/qmtest_download
76 ftp://ftp.codesourcery.com/pub/qmtest
78 Installation instructions are available on the web-site.
82 This package is available from:
84 ftp://ftp.codesourcery.com/pub/qmtest/qmtc/qmtc-<version>.tar.gz
86 See the file called INSTALL in the distribution.
91 To run the tests, run "make qmtest-g++" in the gcc directory of your
92 build tree. The first time that you do this, QMTest will calculate
93 the set of tests that are expected to fail on your platform, so it
94 will take several minutes before you see any test results. After the
95 first time, QMTest will start running the tests much more quickly.
97 If the test summary printed at the test run indicates no unexpected
98 failures, then G++ is behaving as expected on your target. (Some
99 unexpected passes are normal.)
101 You can obtain detailed information about why tests failed in one
104 1. By invoking QMTest with the "-f full" option. For example:
106 make QMTESTRUNFLAGS="-f full" qmtest-g++
108 2. Examining the log file qmtestsuite/gpp.qmr after the tests have
111 Here are some more advanced usage instructions:
113 1. To run a particular set of tests (rather than all of the tests),
114 use the make variable "QMTEST_GPP_TESTS". For example,
116 make QMTEST_GPP_TESTS="g++.dg" qmtest-g++
118 will run only the tests in the g++.dg subdirectory, and:
120 make QMTEST_GPP_TESTS="g++.dg/special/conpr1.C \
121 g++.old-deja/g++.other/access2.C"
124 will run only the two tests indicated.
126 2. To run qmtest with particular flags, use the make variables
127 "QMTESTFLAGS" and "QMTESTRUNFLAGS". For example:
129 make QMTESTFLAGS="-v" QMTESTRUNFLAGS="-f full" qmtest-g++
131 will run qmtest like this:
133 qmtest -v run -f full ...
135 3. To run the compiler with particular flags, use QMTESTRUNFLAGS to
136 set the QMTest context variable "GPPTest.flags", like this:
138 make QMTESTRUNFLAGS='-c GPPTest.flags="-funroll-loops"' qmtest-g++
140 The compiler will then use the "-funroll-loops" switch when
143 4. If qmtest is not in your path, you can indicate the full path to
144 QMTest by using the make variable "QMTEST_PATH", like this:
146 make QMTEST_PATH=/path/to/qmtest qmtest-g++
148 5. To start the QMTest GUI, use:
152 (Note that this will run the program called "netscape" in your path.
153 If you want to use another browser, you must configure qmtest as
154 described in its manual.)
156 Bear in mind that the QMTest GUI is insecure; malicious users with
157 access to your machine may be able to run commands as if they were
158 you. The QMTest GUI only binds to the loopback IP addresss, which
159 provides a measure of security, but not enough for use in untrusted
162 6. If you have a multiprocessor, you can run the tests in parallel by
163 passing the "-j" option to qmtest:
165 make QMTESTRUNFLAGS="-j 4" qmtest-g++
167 will run tests in four threads. (It is also possible to run tests
168 across multiple machines; for more information see the QMTest
171 7. If you are testing a cross compiler, you must specify an interpreter
172 that is capable of running the generated program. It must be a
173 program "p" such that:
175 p program arg1 arg2 arg3 ...
177 behaves exactly like running:
179 program arg1 arg2 arg3 ...
181 would on the target machine. You specify this program via the
182 "CompilerTest.interpreter" context variable:
184 make QMTESTRUNFLAGS='-c CompilerTest.interpreter=/path/to/interpreter'