1 Unit testing {#page_unittesting}
4 The main goal of unit tests in \Gromacs is to help developers while developing
5 the code. They focus on testing functionality of a certain module or a group
6 of closely related modules. They are designed for quick execution, such that
7 they are easy to run after every change to check that nothing has been broken.
9 Finding, building and running
10 =============================
12 As described in \ref page_codelayout, `src/gromacs/` is divided into modules,
13 each corresponding to a subdirectory. If available, unit tests for that module
14 can be found in a `tests/` subdirectory under the top-level module directory.
15 Typically, tests for code in _file_.h in the module is in a corresponding
16 `tests/`<em>file</em>`.cpp`. Not all files have corresponding tests, as it may not
17 make sense to test that individual file in isolation. Focus of the tests is on
18 functionality exposed outside the module. Some of the tests, in particular for
19 higher-level modules, are more like integration tests, and test the
20 functionality of multiple modules.
21 Shared code used to implement the tests is in `src/external/gmock-1.7.0/` and
22 `src/testutils/` (see below).
24 The tests are built if `BUILD_TESTING=ON` (the default) and
25 `GMX_BUILD_UNITTESTS=ON` (the default if `libxml2` is available) in CMake.
26 Each module produces a separate unit test binary (_module_`-test`) under
27 `bin/`, which can execute all the tests for that module.
29 The tests can be executed in a few different ways:
30 - Build the `test` target (e.g., `make test`):
31 This runs all the tests using CTest. This includes also the regression
32 tests if CMake has been told where to find them (regression tests are not
33 discussed further on this page).
34 If some of the tests fail, this only prints basic summary information (only
35 a pass/fail status for each test binary or regression test class).
36 You can execute the failing test binaries individually to get more
37 information on the failure.
38 Note that `make test` does not rebuild the test binaries if you have changed
39 the source code, so you need to separately run `make` or `make tests`.
40 The latter only builds the test binaries and their dependencies.
41 - Build the `check` target (e.g., `make check`):
42 This behaves the same as the `test` target, with a few extensions:
43 1. Test binaries are rebuilt if they are outdated before the tests are run.
44 2. If a test fails, the output of the test binary is shown.
45 3. If unit tests and/or regression tests are not available, a message is
47 - Directly executing a test binary. This provides the most useful output for
48 diagnosing failures, and allows debugging test failures. The output
49 identifies the individual test(s) that fail, and shows the results of all
50 failing assertions. Some tests also add extra information to failing
51 assertions to make it easier to identify the reason. It is possible to
52 control which tests are run using command line options. Execute the binary
53 with `-h` to get additional information.
55 When executed using CTest, the tests produce XML output in
56 `Testing/Temporary/`, containing the result of each test as well as failure
57 messages. This XML is used by Jenkins for reporting the test status for
58 individual tests. Note that if a test crashes or fails because of an assert or
59 a gmx_fatal() call, no XML is produced for the binary, and Jenkins does not
60 report anything for the test binary. The actual error is only visible in the
63 Unit testing framework
64 ======================
66 The tests are written using [Google Test][], which provides a framework for
67 writing unit tests and compiling them into a test binary. Most of the command
68 line options provided by the test binaries are implemented by Google Test. See
69 the [Google Test Primer][] for an introduction.
70 Some tests also use [Google Mock][], which provides a framework for creating
71 mock implementations of C++ classes. Both components are included in the
72 source tree under `src/external/gmock-1.7.0/`, and are compiled as part of the
75 `src/testutils/` contains \Gromacs-specific shared test code. This includes
77 - CMake macros for declaring test binaries. These take care of providing the
78 %main() method for the test executables and initializing the other parts of
79 the framework, so that the test code in modules can focus on the actual
80 tests. This is the only part of the framework that you need to know to be
81 able to write simple tests: you can use `gmx_add_unit_test()` in CMake to
82 create your test binary and start writing the actual tests right away.
83 See `src/testutils/TestMacros.cmake` and existing CMake code for examples
85 - Generic test fixtures and helper classes. Functionality here includes
86 locating test input files from the source directory and constructing
87 temporary files (gmx::test::TestFileManager), adding custom command line
88 options to the test binary (#GMX_TEST_OPTIONS), some custom test assertions
89 for better exception and floating-point handling (testasserts.h), utilities
90 for constructing command line argument arrays (gmx::test::CommandLine) and
91 test fixtures for tests that need to test long strings for correctness
92 (gmx::test::StringTestBase) and for tests that execute legacy code where
93 `stdin` reading etc. cannot be easily mocked
94 (gmx::test::IntegrationTestFixture).
95 - Some classes and functions to support the above. This code is for internal
96 use of the CMake machinery to build and set up the test binaries, and to
97 customize Google Test to suit our environment.
98 - Simple framework for building tests that check the results against reference
99 data that is generated by the same test code. This can be used if it is not
100 easy to verify the results of the code with C/C++ code alone, but manual
101 inspection of the results is manageable.
103 When the test using the framework is first executed, `-ref-data create` can
104 be passed on command line to create the reference data (also options
105 starting with double dashes are accepted).
106 On later executions, the tests read the reference data and fail if the
107 results are not the same. It is possible to update existing reference data
108 with `-ref-data update`.
110 The reference data is stored in XML files under
111 `src/gromacs/`<em>module</em>`/tests/refdata/`. This part of the framework
112 depends on `libxml2`. For inspecting the reference data in a browser, there
113 are XSLT stylesheets that transform the XML files into HTML. Such custom
114 transformations need to be written for each type of test if the output is
115 not easy to check otherwise. Because of security features in browsers, the
116 transformations may not work for all browsers. For the same reason, the
117 XSLT files must be in the same folder as the XML files. For cases where the
118 XSLT files are shared between multiple modules, `src/testutils/copy_xsl.sh`
119 takes care to synchronize the files after a master copy is edited.
121 The current reference data functionality is quite basic, but it can be extended
122 if/when more control over, e.g., comparison tolerances is needed.
123 See gmx::test::TestReferenceData for documentation and existing tests
124 (e.g., the selection unit tests) for examples of how to use it.
126 In addition to `src/testutils/`, some of the module test directories may
127 provide reusable test code that is used in higher-level tests. For example,
128 the \ref module_analysisdata module provides test fixtures, a mock
129 implementation for gmx::AnalysisDataModuleInterface, and some helper classes
130 that are also used in \ref module_trajectoryanalysis module tests.
131 These cases are handled using CMake object libraries that are linked to all the
132 test binaries that need them.
134 Getting started with new tests
135 ==============================
137 To start working with new tests, you should first read the [Google Test][]
138 documentation to get a basic understanding of the testing framework, and read
139 the above description to understand how the tests are organized in \Gromacs.
140 It is not necessary to understand all the details, but an overall understanding
141 helps to get started.
143 Writing a basic test is straightforward, and you can look at existing tests for
144 examples. The existing tests have a varying level of complexity, so here are
145 some pointers to find tests that use certain functionality:
146 - `src/gromacs/utility/tests/stringutil.cpp` contains very simple tests for
147 functions (e.g., for gmx::formatString() and gmx::replaceAll()). These do
148 not use any fancy functionality, only plain Google Test assertions.
149 The only thing required for these tests is the `TEST()` macro and the block
150 following it, plus headers required to make them compile.
151 - The same file contains also simple tests using the reference framework to
152 check line wrapping (the tests for gmx::TextLineWrapper). The test fixture
153 for these tests is in `src/testutils/stringtest.h`/`.cpp`. The string test
154 fixture also demonstrates how to add a custom command line option to the
155 test binary to influence the test execution.
156 - The \ref module_selection module tests contain more complex use of the
157 reference framework. This is the code the reference framework was
158 originally written for.
159 `src/gromacs/selection/tests/selectioncollection.cpp` is the main file to
161 - For more complex tests that do not use the reference framework, but instead
162 do more complex verification in code, you can look at
163 `src/gromacs/selection/tests/nbsearch.cpp`.
164 - For complex tests with mock-up classes and the reference framework, you can
165 look at \ref module_analysisdata tests.
167 Here are some things to keep in mind when working with the unit tests:
168 - Try to keep the execution time for the tests as short as possible, while
169 covering the most important paths in the code under test. Generally, tests
170 should take seconds instead of minutes to run, so that no one needs to
171 hesitate before running the tests after they have done some changes.
172 Long-running tests should go somewhere else than in the unit test set.
173 Note that Jenkins runs many of the tests under Valgrind, so heavy tests are
174 going to slow down also that part of the verification.
175 - Try to produce useful messages when a test assertion fails. The assertion
176 message should tell what went wrong, with no need to run the _test itself_
177 under a debugger (e.g., if the assertion is within a loop, and the loop
178 index is relevant for understanding why the assertion fails, it should be
179 included in the message). Even better if even a user can understand what
180 goes wrong, but the main audience for the messages is the developer who
181 caused the test to fail.
183 [Google Test]: http://code.google.com/p/googletest/
184 [Google Test Primer]: http://code.google.com/p/googletest/wiki/V1_7_Primer
185 [Google Mock]: http://code.google.com/p/googlemock/