Proof of concept: Physical validation suite
commit6dd0349ff2268a9f8dff96bd48d7563e22d71ce2
authorPascal Merz <pascal.merz@colorado.edu>
Tue, 28 Mar 2017 07:27:40 +0000 (28 01:27 -0600)
committerMark Abraham <mark.j.abraham@gmail.com>
Wed, 22 Nov 2017 15:36:18 +0000 (22 16:36 +0100)
treedff8400bdc9dff317e58eea5a0793966a2d1bd01
parentbd6118d4012a83c2809ec58c586626d78e396263
Proof of concept: Physical validation suite

This is a proof of concept for a physical validation suite for GROMACS
(https://redmine.gromacs.org/issues/2070). It is kept separate from the
remaining code checks, for a number of reasons:
* Like the regression tests, it is separate from the code, i.e. it
  calls the complete binary from an external script.
* Unlike the regression tests, we are not comparing to the results from
  earlier version, but to what is expected physically / mathematically.
* Unlike the existing tests, we are not be able to keep these tests in the
  "seconds, not minutes" time frame, rather aiming for "hours, not days".
  They should therefore be ran periodically, but probably not for every
  build.
  Also, given the long run time, it will in many cases be necessary to
  separate running of the systems (e.g. to run it at a specific time, or
  on a different ressource), such that the make script does give the option
  to
  - prepare run files and an execution script,
  - analyze already present simulations,
  - or prepare, run and analyze in one go.
* Since those tests can not be ran at the same frequency as the current
  tests, they are kept strictly opt-in via `-DGMX_PHYSICAL_VALIDATION=ON`
  (see below for more details).
* Different tests need different simulation run-times - convergence tests
  need significantly shorter simulations than statistical ensemble tests.
  The tests could hence be ran at different frequency, with short tests
  being integrated in the Jekyll environment, and longer tests being
  triggered manually only.

USAGE WITH CMAKE / CTEST
* Run `cmake` in your build directory as usually.
* Run make:
    * All previously existing build targets are unchanged, including
      `make check`
    * `make check-phys` builds the main binaries, then runs the physical
       validation tests, expecting the simulation to have already ran.
    * `make check-all` is the combination of `make check` and
      `make check-phys`. It builds the main binaries and the test
      targets, then runs the Google tests, the regression tests and the
      physical validation tests.
    * `make check-phys-prepare` builds the main binaries, then prepares
      the validation tests by creating a subdirectory
      tests/physicalvalidation in the build directory, which contains the
      GROMACS input files as well as a bash file `run_simulations.sh` to
      run the necessary simulations outside of the build environment.
    * `make check-phys-run` prepares and runs the necessary simulations,
      and runs the tests once the simulations are finished.
A typical sequence of commands to run the physical validation tests would
hence either be
> cmake -DGMX_PHYSICAL_VALIATION=ON ..
> make
> make check-prepare-phys
> # run simulations, possibly on external ressource
> make check-phys
or, to run everything locally (WARNING - can take several hours!)
> cmake -DGMX_PHYSICAL_VALIATION=ON ..
> make
> make check-phys-run

* Keeping the current `make check` target unchanged meant to do some
  changes to the CMake files in the `tests/` directory.
* Note that currently `make check-phys-run` needs just under one hour
  to run on my workstation (10-core Xeon processor)

ADDED FILES
All additional files are located in the `tests/physicalvalidation` folder
or on of its subfolders.
* The physical validation is done with a python script (python3, but
  fully backwards compatible to 2.7), which can also be called
  independently of Ctest. The script is called `gmx_physicalvalidation.py`
  and is relatively extensively documented. Try running it with `-h` and
  `--tests` to see the available options.
* The input files for the systems are stored in the `systems/` subfolder,
  containing a .top file containing the topology, a .gro file containing
  the starting structure, and a .mdp file containing the input options.
  The .mdp files are kept as minimal as possible to be sensitive to changes
  in GROMACS standard values.
* The physical validation script and the CMake files get informations about
  the systems via the `system.json` (single precision) and `systems_d.json`
  (double precision) files. These json-formatted files contain the system
  names, the tests to be performed on the systems, as well as additional
  options where needed.
* The actual computation of the tests is performed by a python package
  stored in the `physical_validation` folder. This is simply a snapshot of
  our code-independent physical validation package.

SYSTEMS:
Currently, the script is only running a few systems, checking convergence
of energy conservation in NVE system with decaying timestep, and the
ensembles generated by a few thermostating and barostating algorithms. We
have systems ready covering a broader combination of settings, but kept the
list shorter for this proof-of-concept to keep runtime short.

OPT-IN VIA `-DGMX_PHYSICAL_VALIDATION=ON`
* Physical validation tests are now strictly opt-in via
  `-DGMX_PHYSICAL_VALIDATION=ON`, as discussed in the developer telco.
  * If `-DGMX_PHYSICAL_VALIDATION=OFF` (default value),
    * `make check`, `make test` and a direct call to `ctest` are
      identical to before the inclusion of physical validation, i.e.
      they run google tests and (if available) the regression tests.
    * `make check-phys` does not run any tests, but issues a note that
      noting will be checked since `-DGMX_PHYSICAL_VALIDATION=OFF`.
    * `make check-all` is identical to `make check`, but issues a note
      that physical validation is not available since
      `-DGMX_PHYSICAL_VALIDATION=OFF`.
  * If `-DGMX_PHYSICAL_VALIDATION=ON`,
    * `make check` is identical to before the inclusion of physical
      validation, i.e. it runs google tests and (if available) the
      regression tests.
    * `make test` and a direct call to `ctest` run all available tests,
      i.e. google tests, possible regression tests, physical validation
      tests.
    * `make check-phys` runs only the physical validation tests.
    * `make check-all` runs all available tests like `make test`, with
      the difference that the tests and all dependencies are built
      before running the tests.
  Since `GMX_PHYSICAL_VALIDATION` is `OFF` by default, existing setups
  should see no change in behavior.

Change-Id: I3155da64b7dfb43a42236030dce1399d36cb9655
102 files changed:
CMakeLists.txt
tests/CMakeLists.txt
tests/CheckTarget.cmake
tests/physicalvalidation/.gitattributes [new file with mode: 0644]
tests/physicalvalidation/gmx_physicalvalidation.py [new file with mode: 0644]
tests/physicalvalidation/physical_validation/README.md [new file with mode: 0644]
tests/physicalvalidation/physical_validation/__init__.py [new file with mode: 0644]
tests/physicalvalidation/physical_validation/data/__init__.py [new file with mode: 0644]
tests/physicalvalidation/physical_validation/data/ensemble_data.py [new file with mode: 0644]
tests/physicalvalidation/physical_validation/data/flatfile_parser.py [new file with mode: 0644]
tests/physicalvalidation/physical_validation/data/gromacs_parser.py [new file with mode: 0644]
tests/physicalvalidation/physical_validation/data/observable_data.py [new file with mode: 0644]
tests/physicalvalidation/physical_validation/data/parser.py [new file with mode: 0644]
tests/physicalvalidation/physical_validation/data/simulation_data.py [new file with mode: 0644]
tests/physicalvalidation/physical_validation/data/system_data.py [new file with mode: 0644]
tests/physicalvalidation/physical_validation/data/trajectory_data.py [new file with mode: 0644]
tests/physicalvalidation/physical_validation/data/unit_data.py [new file with mode: 0644]
tests/physicalvalidation/physical_validation/ensemble.py [new file with mode: 0644]
tests/physicalvalidation/physical_validation/integrator.py [new file with mode: 0644]
tests/physicalvalidation/physical_validation/kinetic_energy.py [new file with mode: 0644]
tests/physicalvalidation/physical_validation/util/__init__.py [new file with mode: 0644]
tests/physicalvalidation/physical_validation/util/checkensemble.py [new file with mode: 0644]
tests/physicalvalidation/physical_validation/util/error.py [new file with mode: 0644]
tests/physicalvalidation/physical_validation/util/gromacs_interface.py [new file with mode: 0644]
tests/physicalvalidation/physical_validation/util/integrator.py [new file with mode: 0644]
tests/physicalvalidation/physical_validation/util/kinetic_energy.py [new file with mode: 0644]
tests/physicalvalidation/physical_validation/util/plot.py [new file with mode: 0644]
tests/physicalvalidation/physical_validation/util/timeseries.py [new file with mode: 0644]
tests/physicalvalidation/systems.json [new file with mode: 0644]
tests/physicalvalidation/systems/ens_argon_md-vv_verlet_pme_nh_mttk/input/system.gro [new file with mode: 0644]
tests/physicalvalidation/systems/ens_argon_md-vv_verlet_pme_nh_mttk/input/system.mdp [new file with mode: 0644]
tests/physicalvalidation/systems/ens_argon_md-vv_verlet_pme_nh_mttk/input/system.top [new file with mode: 0644]
tests/physicalvalidation/systems/ens_argon_md-vv_verlet_pme_vr/input/system.gro [new file with mode: 0644]
tests/physicalvalidation/systems/ens_argon_md-vv_verlet_pme_vr/input/system.mdp [new file with mode: 0644]
tests/physicalvalidation/systems/ens_argon_md-vv_verlet_pme_vr/input/system.top [new file with mode: 0644]
tests/physicalvalidation/systems/ens_argon_md_verlet_pme_vr/input/system.gro [new file with mode: 0644]
tests/physicalvalidation/systems/ens_argon_md_verlet_pme_vr/input/system.mdp [new file with mode: 0644]
tests/physicalvalidation/systems/ens_argon_md_verlet_pme_vr/input/system.top [new file with mode: 0644]
tests/physicalvalidation/systems/ens_argon_md_verlet_pme_vr_pr/input/system.gro [new file with mode: 0644]
tests/physicalvalidation/systems/ens_argon_md_verlet_pme_vr_pr/input/system.mdp [new file with mode: 0644]
tests/physicalvalidation/systems/ens_argon_md_verlet_pme_vr_pr/input/system.top [new file with mode: 0644]
tests/physicalvalidation/systems/ens_water_md-vv_verlet_none_pme_nh_mttk/input/system.gro [new file with mode: 0644]
tests/physicalvalidation/systems/ens_water_md-vv_verlet_none_pme_nh_mttk/input/system.mdp [new file with mode: 0644]
tests/physicalvalidation/systems/ens_water_md-vv_verlet_none_pme_nh_mttk/input/system.top [new file with mode: 0644]
tests/physicalvalidation/systems/ens_water_md-vv_verlet_settle_pme_vr/input/system.gro [new file with mode: 0644]
tests/physicalvalidation/systems/ens_water_md-vv_verlet_settle_pme_vr/input/system.mdp [new file with mode: 0644]
tests/physicalvalidation/systems/ens_water_md-vv_verlet_settle_pme_vr/input/system.top [new file with mode: 0644]
tests/physicalvalidation/systems/ens_water_md_verlet_settle_pme_vr/input/system.gro [new file with mode: 0644]
tests/physicalvalidation/systems/ens_water_md_verlet_settle_pme_vr/input/system.mdp [new file with mode: 0644]
tests/physicalvalidation/systems/ens_water_md_verlet_settle_pme_vr/input/system.top [new file with mode: 0644]
tests/physicalvalidation/systems/ens_water_md_verlet_settle_pme_vr_pr/input/system.gro [new file with mode: 0644]
tests/physicalvalidation/systems/ens_water_md_verlet_settle_pme_vr_pr/input/system.mdp [new file with mode: 0644]
tests/physicalvalidation/systems/ens_water_md_verlet_settle_pme_vr_pr/input/system.top [new file with mode: 0644]
tests/physicalvalidation/systems/int_argon_md-vv_verlet_pme/input/system.gro [new file with mode: 0644]
tests/physicalvalidation/systems/int_argon_md-vv_verlet_pme/input/system.mdp [new file with mode: 0644]
tests/physicalvalidation/systems/int_argon_md-vv_verlet_pme/input/system.top [new file with mode: 0644]
tests/physicalvalidation/systems/int_argon_md-vv_verlet_switch/input/system.gro [new file with mode: 0644]
tests/physicalvalidation/systems/int_argon_md-vv_verlet_switch/input/system.mdp [new file with mode: 0644]
tests/physicalvalidation/systems/int_argon_md-vv_verlet_switch/input/system.top [new file with mode: 0644]
tests/physicalvalidation/systems/int_argon_md_verlet_pme/input/system.gro [new file with mode: 0644]
tests/physicalvalidation/systems/int_argon_md_verlet_pme/input/system.mdp [new file with mode: 0644]
tests/physicalvalidation/systems/int_argon_md_verlet_pme/input/system.top [new file with mode: 0644]
tests/physicalvalidation/systems/int_argon_md_verlet_switch/input/system.gro [new file with mode: 0644]
tests/physicalvalidation/systems/int_argon_md_verlet_switch/input/system.mdp [new file with mode: 0644]
tests/physicalvalidation/systems/int_argon_md_verlet_switch/input/system.top [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md-vv_verlet_lincs_pme_pme/input/system.gro [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md-vv_verlet_lincs_pme_pme/input/system.mdp [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md-vv_verlet_lincs_pme_pme/input/system.top [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md-vv_verlet_lincs_pme_switch/input/system.gro [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md-vv_verlet_lincs_pme_switch/input/system.mdp [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md-vv_verlet_lincs_pme_switch/input/system.top [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md-vv_verlet_none_pme_pme/input/system.gro [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md-vv_verlet_none_pme_pme/input/system.mdp [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md-vv_verlet_none_pme_pme/input/system.top [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md-vv_verlet_none_pme_switch/input/system.gro [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md-vv_verlet_none_pme_switch/input/system.mdp [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md-vv_verlet_none_pme_switch/input/system.top [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md-vv_verlet_settle_pme_pme/input/system.gro [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md-vv_verlet_settle_pme_pme/input/system.mdp [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md-vv_verlet_settle_pme_pme/input/system.top [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md-vv_verlet_settle_pme_switch/input/system.gro [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md-vv_verlet_settle_pme_switch/input/system.mdp [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md-vv_verlet_settle_pme_switch/input/system.top [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md_verlet_lincs_pme_pme/input/system.gro [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md_verlet_lincs_pme_pme/input/system.mdp [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md_verlet_lincs_pme_pme/input/system.top [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md_verlet_lincs_pme_switch/input/system.gro [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md_verlet_lincs_pme_switch/input/system.mdp [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md_verlet_lincs_pme_switch/input/system.top [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md_verlet_none_pme_pme/input/system.gro [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md_verlet_none_pme_pme/input/system.mdp [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md_verlet_none_pme_pme/input/system.top [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md_verlet_none_pme_switch/input/system.gro [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md_verlet_none_pme_switch/input/system.mdp [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md_verlet_none_pme_switch/input/system.top [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md_verlet_settle_pme_pme/input/system.gro [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md_verlet_settle_pme_pme/input/system.mdp [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md_verlet_settle_pme_pme/input/system.top [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md_verlet_settle_pme_switch/input/system.gro [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md_verlet_settle_pme_switch/input/system.mdp [new file with mode: 0644]
tests/physicalvalidation/systems/int_water_md_verlet_settle_pme_switch/input/system.top [new file with mode: 0644]
tests/physicalvalidation/systems_d.json [new file with mode: 0644]