From 1ff17de662e9615822c20be97c3bfe9a4d191f1a Mon Sep 17 00:00:00 2001 From: Martin Beaudoin Date: Sat, 12 Sep 2015 16:55:45 -0400 Subject: [PATCH] testHarness: User guide for the testHarness in Org Mode --- doc/testHarness/Allrun | 6 + doc/testHarness/testHarness.org | 329 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 335 insertions(+) create mode 100644 doc/testHarness/Allrun create mode 100644 doc/testHarness/testHarness.org diff --git a/doc/testHarness/Allrun b/doc/testHarness/Allrun new file mode 100644 index 000000000..cfd473d82 --- /dev/null +++ b/doc/testHarness/Allrun @@ -0,0 +1,6 @@ +# Generate the ASCII version of testHarness.org +pandoc testHarness.org -N -s -S -f org -t asciidoc -o testHarness.txt + +# Generate the Mediawiki version of testHarness.org +pandoc testHarness.org -s -S -f org -t mediawiki -o testHarness.mediawiki + diff --git a/doc/testHarness/testHarness.org b/doc/testHarness/testHarness.org new file mode 100644 index 000000000..71dad5bc8 --- /dev/null +++ b/doc/testHarness/testHarness.org @@ -0,0 +1,329 @@ +# -*- mode: org; -*- +# +#+TITLE: *foam-extend-3.2: A guide to test harness* +#+AUTHOR: Martin Beaudoin +#+DATE: 5 September 2015 +#+LINK: foam-extend https://sourceforge.net/projects/openfoam-extend/ +#+OPTIONS: author:t +#+OPTIONS: toc:2 +#+OPTIONS: ^:{} +#+OPTIONS: _:{} + +############################################################################### + +* A guide to the foam-extend test harness +This file is a guide to the foam-extend test harness. + +The test harness from foam-extend version 3.2 was used as a reference +implementation for this documentation. + +The original version of this file is located at +$WM\_PROJECT\_DIR/doc/testHarness/testHarness.org. + +A plain text version of this file is located at +$WM\_PROJECT\_DIR/doc/testHarness/testHarness.txt. + +A MediaWiki version of this file is located at + +In case any converted versions (text, MediaWiki, etc.) of this file differ from +the original testHarness.org file, the latter should always be considered the +reference version. But we do appreciate your feedback and comments for improving +this text. + +For instance, if you ever decide to improve the Mediawiki version only, please +consider contributing your changes back to the testHarness.org file as +well. Otherwise, your Mediawiki changes might get overwritten by an updated +version of the .org reference file. + +** Introduction +The foam-extend test harness scripts are based mainly on Kitware CMake/CTest +utilities http://www.cmake.org/. + +In its current version, the test harness is designed for running the full suite +of foam-extend tutorials, and report the fail/success results to a CDash Web +service one can consult using a simple browser. + +The CDash Web service for foam-extend 3.2 is available here: +http://foam-extend.sourceforge.net/CDash/index.php + +For a quick introduction of CMake/CTest, see here: +http://www.cmake.org/Wiki/CMake/Testing_With_CTest + +For a more complete introduction to CMake and CTest, one can also refer to the +Kitware book `Mastering CMake: A Cross-Platform Build System` +http://www.kitware.com/products/books.php + +For more information on Kitware CDash, see here: http://www.cdash.org/ + +The rest of this document will provide suficient information so you can run the +foam-extend test harness on your own. + +** Files location and a short description +The foam-extend test harness is currently built from the following files: + +- =$WM_PROJECT_DIR/CMakeLists.txt= :: This is the main CMake file driving the + configuration of the test harness. When processed though the cmake + utility, all the necessary Makefiles and CTest scripts will get + generated. + +- =$WM_PROJECT_DIR/CTestConfig.cmake= :: This file contains the various variables + and definitions necessary to publish your test harness results over the + CDash Web service for foam-extend 3.2. + +- =$FOAM_SITE_DIR/etc/CTestConfig.site.cmake= :: This file allow you to override + the values from the main $WM\_PROJECT\_DIR/CTestConfig.cmake. This file + should be used if you are hosting your own site CDash Web + service. + +- =$WM_PROJECT_USER_DIR/etc/CTestConfig.user.cmake= :: This file allow you to + override the values from the main $WM\_PROJECT\_DIR/CTestConfig.cmake and the + site file $FOAM\_SITE\_DIR/etc/CTestConfig.site.cmake. This file should be + used if you are hosting your own user space personal CDash Web service. + +- =$WM_PROJECT_DIR/testHarness/foam-extend/3.2/CMakeFiles/CMakeLists.txt= :: + This file is a copy of the main $WM\_PROJECT\_DIR/CMakeLists.txt file. + +- =$WM_PROJECT_DIR/testHarness/foam-extend/3.2/CMakeFiles/CTestConfig.cmake.foam-extend= :: This file is a copy of the main $WM\_PROJECT\_DIR/CTestConfig.cmake file. + +- =$WM_PROJECT_DIR/testHarness/foam-extend/3.2/CMakeFiles/Dashboard_Tutorials.cmake.in= :: This file is a template for generating the file `ctest -S` file + $WM\_PROJECT\_DIR/testHarness/foam-extend/3.2/runDir/Dashboard\_Tutorials.cmake + CMake will take care of automatically generating the Dashboard\_Tutorials.cmake file. + +- =$WM_PROJECT_DIR/testHarness/foam-extend/3.2/CMakeFiles/FOAM_Tutorials.cmake= :: This file will configure a copy the foam-extend tutorials for running + under the test harness. The copy will be made under + $WM\_PROJECT\_DIR/testHarness/foam-extend/3.2/runDir. + +- =$WM_PROJECT_DIR/testHarness/foam-extend/3.2/runDir/Allclean= :: Cleanup + script. Will remove all the automatically generated files and directories + for the test harness. Should be called prior to running the Allrun\_\* scripts. + +- =$WM_PROJECT_DIR/testHarness/foam-extend/3.2/runDir/Allrun_CMakeOnly= :: This + script will generate all the necessary files for running the test harness + on the foam-extend tutorials. One can then invoke `make help` to see all + the available make options. + +- =$WM_PROJECT_DIR/testHarness/foam-extend/3.2/runDir/Allrun_Experimental= :: + Same script as `Allrun\_CMakeOnly`, but will also call `make + Experimental`. + +- =$WM_PROJECT_DIR/testHarness/foam-extend/3.2/runDir/Allrun_Nightly= :: Same + script as `Allrun\_CMakeOnly`, but will also call `make Nightly`. + +- =$WM_PROJECT_DIR/testHarness/foam-extend/3.2/scripts/AdditionalRunFunctions= :: Additional `bash` macros for the tutorial Allrun files. + +- =$WM_PROJECT_DIR/testHarness/foam-extend/3.2/scripts/Allrun.default= :: + Default Allrun script for the tutorial when none are provided. + +- =$WM_PROJECT_DIR/testHarness/foam-extend/3.2/scripts/addMissingAllrunFileToTutorial.sh= :: Bash script for adding a default Allrun file to the tutorials that do not + have one. The test harness only run tutorials with an existing Allrun file + +- =$WM_PROJECT_DIR/testHarness/foam-extend/3.2/scripts/prepareCasesForOneTimeStep.sh= :: This script will modify the test cases system/controlDict in order for the + case to run for only 1 time step + +- =$WM_PROJECT_DIR/testHarness/foam-extend/3.2/scripts/prepareCasesForTestHarness.sh= :: This script will modify the test cases Allrun file so it can run properly + under the test harness. + +** Configuring your $WM\_PROJECT\_DIR/etc/prefs.sh file for the test harness +The following environment variables are used for configuring the test harness: + +- =CDASH_SUBMIT_LOCAL_HOST_ID= :: System identifier for the FOAM CDash test harness + on foam-extend. By default, your system FQN/hostname will be used as the + system identifier when publishing your test harness results on the FOAM CDash + server on foam-extend. You can override your identifier using this environment + variable. + +- =CDASH_SCM_INFO= :: Buildname suffix for the FOAM CDash test harness on foam-extend. + By default, the git branch name and git revision number will be appended to the CDash build name. + Otherwise, for users not using git, or wanting to provide additionnal + information, simply initialize the CDASH\_SCM\_INFO with the proper information. + +- =WM_NCOMPPROCS= :: Specify the number of cores to use for the parallel execution + of the test harness. + +- =FOAM_TUTORIALS= :: Directory where the original test cases are located. For + foam-extend, this would be by default $WM\_PROJECT\_DIR/tutorials. + +** The main dashboards : Experimental, Nightly and Continuous +The result of a test run, reformatted for easy review, is called a +`dashboard`. A dashboard can be submitted to a central server, like CDash. Once +properly configured, the test harness will offer 3 main dashboards. + +- =Experimental= :: Will test the current state of the project. An experimental + submission can be performed at any time, usually interactively from the + current working copy of a developer. +- =Nightly= :: Is similar to Experimental, except that the source tree will be set + to the state it was in at a specific nightly time. This ensures that all + `nightly` submissions correspond to the state of the project at the same + point in time. `Nightly` builds are usually done automatically at a preset + time of day. Nightly build will also update your source code to the latest + available revision. So it is best not to run a Nightly dashboard on source + code that is not yet committed. +- =Continuous= :: Means that the source tree is updated to the latest revision, + and a build / test cycle is performed only if any files were actually + updated. Like `Nightly` builds, `Continuous` ones are usually done + automatically and repeatedly in intervals. + +There are also `intermediary dashboards` that allow you to select a specific +test harness intermediary step. The command `make help` will show you that list: + +#+BEGIN_SRC +... ContinuousBuild +... ContinuousConfigure +... ContinuousCoverage +... ContinuousMemCheck +... ContinuousStart +... ContinuousSubmit +... ContinuousTest +... ContinuousUpdate +... ExperimentalBuild +... ExperimentalConfigure +... ExperimentalCoverage +... ExperimentalMemCheck +... ExperimentalStart +... ExperimentalSubmit +... ExperimentalTest +... ExperimentalUpdate +... NightlyBuild +... NightlyConfigure +... NightlyCoverage +... NightlyMemCheck +... NightlyMemoryCheck +... NightlyStart +... NightlySubmit +... NightlyTest +... NightlyUpdate +#+END_SRC + +- Note :: For foam-extend, the MemCheck and Coverage dashboards are not supported. + +** Running the test harness + Running the test harness is pretty simple: + + These commands will configure and run the `Experimental` version of the test harness: + +#+BEGIN_SRC + cd $WM_PROJECT_DIR/testHarness/foam-extend/3.2/runDir + ./Allclean + ./Allrun_Experimental +#+END_SRC + + To run the `Nightly` version of the test harness: + +#+BEGIN_SRC + cd $WM_PROJECT_DIR/testHarness/foam-extend/3.2/runDir + ./Allclean + ./Allrun_Nightly +#+END_SRC + + To see the full range of available options, run the command: + +#+BEGIN_SRC +make help +#+END_SRC + +** Selecting a subset of test cases to run +Instead of using the `make` command to run the test harness, one can also use +command `ctest`. The commmand `ctest` offers additionnal options to select or limit the number +of tests to run. + +The following command will provide all the available options for ctest: + +#+BEGIN_SRC +ctest -N +#+END_SRC + +Here is a list of useful ctest options: + +- =ctest -N= :: this command will list the available tests by their name and + number. By default, all the tests are run in succession following the + numerical order shown. + +- =ctest -R = :: run all the tests whose name are matching the supplied + regular expression. For instance, to run all the tests related to cfMesh, + one can use the following command: `ctest -R cfMesh` + +- =ctest -E = :: run all the tests, but exclude the ones whose name is + matching the supplied regular expression. For instance, to run all the + tests except those for cfMesh, one can use the following command: `ctest -E + cfMesh` + +- =ctest -I [Start,End,Stride,test#,test#]= :: run individual tests by + number. `ctest -I 3,5` will run test 3, 4 and 5. `ctest -I 4,4,,4,7,13` + will run tests 4, 7 and 13. + +- =ctest -D dashboard= :: run a specific dashboard test. For example, the command + `make Experimental` can be replaced by the following suite of ctest + commands: +#+BEGIN_SRC +ctest -D ExperimentalStart +ctest -D ExperimentalConfigure +ctest -D ExperimentalBuild +ctest -D ExperimentalTest +ctest -D ExperimentalSubmit +#+END_SRC + +Here is a more complete example where we will configure, build, test and submit the test +harness results, but only for the incompressible tutorials: +#+BEGIN_SRC +ctest -D Experimental -R incompressible +#+END_SRC + + +** Browsing the CDash service + + The results of the test harness run will be published on the CDash dashboard on foam-extend. + + To see your results: + URL : http://foam-extend.sourceforge.net/CDash/index.php?project=foam-extend-3.2 + + On this interactive Web site, one can then point and click various buttons + and menus to explore the various reports uploaded from your test harness runs. + +** Configuring the test harness for using your own site or personal CDash service +The foam-extend source code comes with a set pre-configured parameters for +uploading your dashboards results on the main project CDash server. One can also +choose to host their own CDash service, either as a site service, or as a +personnal service running in your own user space. Your CDash administrator can +generate a file similar to $WM\_PROJECT\_DIR/CTestConfig.cmake where all the +necessary parameters for connecting to your local service are specified. + +In order to use your site CDash service, simply copy your site CTestConfig.cmake +file to $FOAM\_SITE\_DIR/etc/CTestConfig.site.cmake + +In order to use your personnal CDash service, simply copy your personnal CTestConfig.cmake +file to $WM\_PROJECT\_USER\_DIR/etc/CTestConfig.user.cmake + +As usual, your site configuration will override the default parameters from the +main configuration file $WM\_PROJECT\_DIR/CTestConfig.cmake. + +Likewise, your personnal configuration will override the default parameters from the +main configuration file $WM\_PROJECT\_DIR/CTestConfig.cmake and the default +parameters from the site file $FOAM\_SITE\_DIR/etc/CTestConfig.site.cmake. + + +** New features for the test harness (foam-extend 3.2): + - =Running the test harness in parallel= :: It is now possible to run the test + harness in parallel over a single node or computer. The environment + variable WM\_NCOMPPROCS will specify the number of cores to use for + running the test harness. For `n` cores specified, `n` tutorials will + be running in parallel on your computer. Since all the tests will + still run on the same computer, make sure you have enough ressources to + run `n` tutorials in parallel. Depending on the number of cores + available, one might have to tweak some of the shell `limits` + values. The command `ulimit -a` will show you the actual limits values + imposed on your shell. Some limit values like `open files` (ulimit -n) + or `max user processes` (ulimit -u) might need to be adjusted to some + higher values. In doubt, consult with your local sysadmin. + +** Notes +The MediaWiki version of this file was generated using the following command: + +#+BEGIN_SRC + pandoc testHarness.org -s -S -f org -t mediawiki -o testHarness.mediawiki +#+END_SRC + +The ASCII version of this file was generated using the following command: + +#+BEGIN_SRC + pandoc testHarness.org -N -s -S -f org -t asciidoc -o testHarness.txt +#+END_SRC -- 2.11.4.GIT