generate platform_ops.h from platform_ops.proto, and reduce dependencies

[proto.git] / src / README.prototest

PROTOTEST LONG HELP + TUTORIAL

prototest is a regression tester for MIT Proto.

1. REQUIREMENTS

prototest requires Python v. 2.5


2. BASICS

Dump Directory:
  IF prototest is not in the same directory as proto
  OR you're telling proto to output to a custom location,
  THEN use the -d option to tell prototest the path to the dump files.
  DO NOT use --dump-stem when running proto... this will break prototest
  as it uses its own naming conventions)

Terminology:
  Test File: Is a file containing one or more tests
  Test: is a group of assertions that work on the log generated by the
    same proto command
  Assertion: is a claim about the expected value. Currently only numerical 
    assertions (=, > , <, >=, ~=, !=...) are supported.


3. USING PROTOTEST

The usage is:

  python prototest.py input [other options]

Input can be one of:
  * Test File
  * Test Suite (A file containing a list of paths to test files. Paths can be 
    be absolute paths or relative to prototest)
  * Test Directory 

When you pass a test suite or a test directory all test files (files with 
extension .test) will be run sequentially.

The result of each test file is written to the same directory as the
test file.  The result file has a ".RESULT" appended to the name of
the test file.

Other Options:
    * Verbosity (-v): There are currently 4 verbosity levels, numbered 0 through 3, 
      which control the output verbosity (default is 1). 0 gives you the most
      terse output, while 4 will display the test file parser messages
      (useful if you want to "debug" your test files).
    * Dumpdir (-d): Allows you to manually set proto dump directory. 
    * Recursive (-r): In directory mode, enabling this option will find testfiles
      in subdirectories as well. (WARNING: could be slow if you have a LOT of 
      files in subdirectories. Where LOT means order 50,000 files). 

4. WRITING A TEST FILE
Test files should have a ".test" extension.  Please see the test/
subdirectory for examples of test files.

Adding Comments:
- Use // to start a comment
- Comment must exist in its own line (You cannot attach a comment to
  the end of a non-comment line)

Creating Tests:
- Use 'test: <commands for proto>' to create a test
- WARNING: do not use "dump stem" AT ALL. This will break prototest.
- WARNING: do not change the dump directory, unless you HAVE to. If
  you change the dump directory, be sure to specify its location when
  you run prototest.
- TIP: Use --headless to run the simulator without the GUI (results in
  speedups)
- A test needs to have at least one assertion to be run.

Adding Assertions to a Test:
- Use "<operator> <line#> <col#> <expected value> [<other args>] 
- Note that both LINE and COLUMN numberings start from ZERO. 
- Use -1, -2, ... to access the last, second last, ... columns of line.
- You can use "_" (without the quotes) as your column number if you want to
  compare with the whole line. 
- Numerical operators supported at present: =, > , < , >= , <= , != ,
  ~= (the last two are not equal to, and nearly equal to)
- Use "is_nan" operator to check for NaN
- Nearly equal to takes in "tolerance" as other args.
- You can add as many assertions as necessary (on separate lines)

Other Lines:
- At present, all other lines are ignored, but do not rely on this
  behavior to comment your files; Use // to explicitly mark your
  comments.


5. CREATING TEST SUITES TO RUN TESTS IN BATCH MODE
Instead of simply putting your test files in a single directory and running
all tests in the directory, you can create a test suite file with just the tests
you need.

To create a test suite file, simply create a file with paths (either absolute
or relative to prototest) and name it blah.testsuite

Example of a testsuite:
./tests/math.test
./tests/mathlib.test
./tests/vector.test

6. EXTENDING PROTOTEST BY ADDING YOUR OWN OPERATORS
- Your function should take in either:
    * value in the dumpfile ("actual value) and the "expected value" [IN THAT
      ORDER].
    * -OR- a list of arguments (for functions that require other supporting 
      parameters). This function should return another function that takes in
      an actual value and an expected value.
- The operation will be carried out as: [actual] {OPERATOR} [expected]. E.g.
  if your function is ">", you will be performing [actual] > [expected]. 
- Your function should return "TRUE" if the operator criteria is satisfied, and
  FALSE in all other cases. For example, if your operator is >, your function 
  should return TRUE whenever (actual value) > (expected value).
- Often, your function might require other supporting arguments (besides actual 
  & expected value). In this case, use currying. E.g. the ~= (nearly equal) 
  operator requires "error tolerance" as optional argument. So, we create a 
  function that takes in a list of arguments as values, and returns a function
  of appropriate type: 
  (lambda arg_list: (lambda act, exp: abs(act - exp) <= arg_list[0]))
- Add your function to the appropriate dictionary in ConfigParser in prototest. 
  We currently have two dictionaries: "numeric_fns" and "string_fns". The dict
  entries have the symbol as key, and a tuple of the function and a description
  as the value. If you add new dictionaries, you will need to make necessary
  changes in the "run" method of ConfigParser.

6. CHANGELOG

Changelog:

0.75 [kanak]
* CHANGED: "_" is now used to determine if function takes in whole line, rather
  than negative numbers. This change was necessary because we can use -ve number
  to access the list backwards, which IMO is a powerful feature to have.
  (Note: compiler tests still work because using -1 on a list with only one 
  element gives you that same element).
* REFACTORED: definitions of string functions now
  match the definitions of numeric functions (i.e. no extra "1" or "0" switch
  to determine if function accepts whole line is required).
* ADDED: "has" string function
* CHANGED: Minor indentation and documentation

0.74 [kanak]
* Added a recursively scan directory option.

0.73 [kanak]
* Added an explicit "Test Suite Input mode" for times when running an entire
  test directory is overkill
* Improved Output display: 
    - A primitive progress bar which display a "." for every 5 tests run
    - Instead of just saying passed all tests, it says passed all X tests. where
      X is the number of tests.

0.72 [Jake]
* Whole line testing added (to test p2b)

0.71 [kanak]
* Verbosity Level 0 now makes tests headless if they aren't already.

0.7 Final: UI changes, fixed unwanted seperator lines in log file,
  fixed crash when using string comparisons.
0.7 beta 2: bugs fixed. Other than slight changes to the UI, code is ready.
0.7 beta 1: initial build

0.7 [kanak]
New version number to reflect massive change in functionality;
* Directories are now supported as inputs.
* Class structure was overhauled to Test Suite -> Test File -> Tests
  -> Assertions.
* Major refactoring of the "Main" method, which is now a lot shorter
  since behaviors such as writing to log files have been moved to
  appropriate classes / methods.

0.61 [kanak]
* CHANGED: Curried functions are now generalized (now accept list
  instead of disparate elements)
* ADDED: String comparisons (to support "is_nan" test)
* ADDED: "is_nan" comparison operator which tests if number returned is nan

0.60 [kanak]
* REFACTORED: Code is now much more readable, and adheres to the
  python style guide (http://www.python.org/dev/peps/pep-0008/). Major
  Violations included: Using something other than 4 spaces for tabs,
  too long statements, concatenating statements into a single line,
  too many branches (partially resolved), and overly long functions.
  Non standard tabbing is a big problem because indentation is all too
  important in python.

0.57 [kanak]
* REFACTORED: the "main" method, which was becoming a jungle of "if
  verbosity"s in light of the following changes:
* ADDED: a "failed" attribute in test to determine whether a test was
  successful (instead of len(test.failedAssertions) == 0)
* REMOVED: -n (no proto output) command line switch. The switch was
  redundant in light of new verbosity levels:
* CHANGED: verbosity levels.
  #VERBOSITY LEVELS:
  #0: log failed tests only, don't even display the successful tests
  #1 (Default): log only failed tests, "all assertions passed"
   displayed for successful tests
  #2: log both failed and successful tests
  #3: like #2, but also displays parser output.
  
0.562 [kanak]
* ADDED: -noprotooutput switch to prevent prototest from writing
  proto's stdout/stderr in the results file
* CHANGED: instead of allpass, we now compute the number of tests
  passed. So "X out of Y tests passed" is displayed instead of
  unhelpful FAIL. (If all tests are passed, "ALL TESTS ARE PASSED" is
  still displayed)
* CHANGED: optparse's arguments are now assigned by tuple unpacking
  (more concise).

0.561 [kanak]
* CHANGED: Parser now outputs its guts only when verbosity level is 3
  or greater.

0.56 [kanak]
* FIXED: turns out the verbose mode wasn't really working-> it was
  using an invalid action "check" instead of "store", since "type" was
  not defined it was "storing" strings instead of ints.
* CHANGED: Verbose mode 2 now prints successfull assertions as well
  (useful in verifying mathematical functions)

0.551 [kanak]
* CHANGED: Default value for verbosity is "0" instead of "False" (for
  consistency)
* ADDED: Dump file  location is printed in log by default
* ADDED: "~=" comparison to sampletest. (To test if the comparison
  works... pychecker claimed that the function had an unused
  parameter)

0.55: [jake]
* CHANGED: Output is now prettier
* CHANGED: Before each test, a dot prints on the console, showing how
  execution is running
* CHANGED: verbose -> verbosity, now with multiple levels: verbosity 1
  shows testing, verbosity 2 shows parsing of assertion files

0.54:
* FIXED: Crash when the "actual value" is non-numeric

0.53:
* ADDED: -d command line option to explicitly set dumpfile path
* UPDATED: readme

0.52:
* FIXED: Prototest doesn't find dump directory when executed from a
  different location
* FIXED: Prototest now crashes with "IOError: <testconfig> file
  couldn't be found" instead of a cryptic "Unbound Error" when config
  file doesn't exist
* FIXED: Missing Dumpfile now throws MissingDumpFileException and
  marks all tests as failed instead of "Unforseen Error Occurred"
* FIXED: Unforseen error now appends the name /details of exception
  instead of just saying "Unforseen error"
* FIXED: Unwanted newlines when printing parser output in verbose mode
* FIXED: Linecache doesn't throw IndexOutOfBounds leading to
  "UnboundVariable" exceptions
* VERIFIED: Py3K compliance
* CHANGED: thread is now executed using subprocess instead of
  "deprecated" os.system. Advantages include: proto launched as
  separate thread, can do "nifty" stuff like kill thread etc.
* CHANGED: "Version" is now a variable instead of a hard coded string
  so referencing it is easier.
* CHANGED: verbose mode now writes successful tests as well, and
  displays the "exceptions"
* CHANGED: log file now has a "---" delimiter between tests
* ADDED: MissingDumpFileException to be thrown when dumpfile doesn't exist

0.51:
* FIXED: Missing Paren, and other bugs [JAKE]
* CHANGED: Output is now prettier [JAKE]

0.5:
Major Refactoring: code has been simplified and better documented
* Uses dictionary lookup instead of genAssert
* Uses curried version of "nearly equal to"
* LongHelp and Changelog moved to external file

0.41:
* CHANGED prototest now looks at return code of proto before running
  assertions. A non-zero assertions means tests are not run.

0.4:
* FIXED crash when line/column not found (now adds "exception" to the
  assertion)
* FIXED crash when dump file not found (now adds "exception" to the test)
* FIXED reads long lines due to linecache's weird "indexing starts
  from 1" behavior
* FIXED comparisons fail because numbers were being compared to strings
* FIXED output generator displays "FAILED X TESTS" repeatedly as a
  result of problem with join
* FIXED noisy output even when in non-verbose mode
* CHANGED the output log generation. Log generation is now simplified.
* REVERTED command execution is done by os.system instead of
  check_call because check_call gives weird errors when trying to use
  -dump-stem

0.2:
* ADDED randomized dump file name generation
* ADDED long help option
* CHANGED reading the proto dump log file is done by linecache instead
  of the "naive method".
* CHANGED command line parsing is done by python's optparse module
* CHANGED proto is now executed by using the subprocess module's
  "check_call" method instead of os.system
* and other minor bug fixes and optimizations, primarily in the
  configparser method

0.1: Initial port to python from java