doc/selftest.html

   1 <!--
   2 /* ***** BEGIN LICENSE BLOCK *****
   3  * Version: MPL 1.1/GPL 2.0/LGPL 2.1
   4  *
   5  * The contents of this file are subject to the Mozilla Public License Version
   6  * 1.1 (the "License"); you may not use this file except in compliance with
   7  * the License. You may obtain a copy of the License at
   8  * http://www.mozilla.org/MPL/
   9  *
  10  * Software distributed under the License is distributed on an "AS IS" basis,
  11  * WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License
  12  * for the specific language governing rights and limitations under the
  13  * License.
  14  *
  15  * The Original Code is [Open Source Virtual Machine.].
  16  *
  17  * The Initial Developer of the Original Code is
  18  * Adobe System Incorporated.
  19  * Portions created by the Initial Developer are Copyright (C) 2009
  20  * the Initial Developer. All Rights Reserved.
  21  *
  22  * Contributor(s):
  23  *   Adobe AS3 Team
  24  *
  25  * Alternatively, the contents of this file may be used under the terms of
  26  * either the GNU General Public License Version 2 or later (the "GPL"), or
  27  * the GNU Lesser General Public License Version 2.1 or later (the "LGPL"),
  28  * in which case the provisions of the GPL or the LGPL are applicable instead
  29  * of those above. If you wish to allow use of your version of this file only
  30  * under the terms of either the GPL or the LGPL, and not to allow others to
  31  * use your version of this file under the terms of the MPL, indicate your
  32  * decision by deleting the provisions above and replace them with the notice
  33  * and other provisions required by the GPL or the LGPL. If you do not delete
  34  * the provisions above, a recipient may use your version of this file under
  35  * the terms of any one of the MPL, the GPL or the LGPL.
  36  *
  37  * ***** END LICENSE BLOCK ***** */
  38 -->
  39 <html>
  40 <head>
  41 <title>AVM Selftest Harness</title>
  42 </head>
  43
  44 <body>
  45 <h1> AVM Selftest Harness </h1>
  46
  47 <h2> Purpose </h2>
  48
  49 <P> The selftest infrastructure in the avmplus makes it possible to
  50 write test cases below the level of abstraction of ABC files.  For
  51 example, it can be used to create tests for code generation
  52 optimizations, processor features, and low-level GC features.
  53
  54 <P> The main virtues of the infrastructure is that much of the
  55 drudgery is taken out of writing (and reading!) test code, and that
  56 running all tests across the system is quick and easy and requires no
  57 special knowledge.  Thus everyone can add tests, and they can be run
  58 routinely, whether manually or in a test framework.
  59
  60 <P> The test infrastructure is non-invasive and can be enabled in
  61 release builds without other adverse effects than an increase in code
  62 size and the appearance of the command line switch "-Dselftest".
  63
  64 <h2> Overview </h2>
  65
  66 <h3> Writing and generating tests </h3>
  67
  68 <P> Tests are written in C++, but in order to reduce the amount of
  69 noise and overhead the test code is placed in a specially formatted
  70 ".st" file.  A script then processes all ".st" files at once and
  71 generates C++ code for each, plus common initialization code.  At
  72 this time, each ".st" file lives in the "extensions/" folder and has a
  73 name that starts with "ST_".
  74
  75 <P> For example, "extensions/ST_avmplus_peephole.st" contains tests
  76 for the interpreter's peephole optimizer.
  77
  78 <P> The script that processes the selftest files is
  79 "extensions/selftest.as".  It should be run with "extensions/" as the
  80 current directory, on all the .st files:
  81
  82 <pre>
  83 % pwd
  84 ~/work/redux/extensions
  85 % avmshell selftest.abc -- ST_*.st
  86 %
  87 </pre>
  88
  89 <P> A test file - which can contain many tests - belongs to a
  90 component and a category; each test is in turn named.  By convention,
  91 the ".st" file and the generated ".cpp" file are named using the
  92 component and category, eg, "ST_avmplus_peephole.st".
  93
  94 <h3> Running the tests </h3>
  95
  96 <P> The ST_*.cpp files should be added to the avmshell project, and
  97 AVMPLUS_SELFTEST should be enabled in avmbuild.h.  Once the shell has
  98 been rebuilt, the tests can be invoked either all at once:
  99
 100 <pre>
 101 % avmshell -Dselftest
 102 </pre>
 103
 104 <P> or some can be selected by component, category, and name (using a
 105 trailing '*' as an MS-DOS style suffix wildcard):
 106
 107 <pre>
 108 % avmshell '-Dselftest=avmplus,peephole,get*'
 109 </pre>
 110
 111 <h3> Output format </h3>
 112
 113 <P> The results are formatted as JSON-encoded array data.  At this
 114 time, they are presented on stdout, and there are no postprocessing
 115 facilities (see the "TO DO" section).  Each line is tagged with the
 116 component, category, and (when appropriate) the test name.  For
 117 example, a successful run of the "basic" test case presented below
 118 looks like this:
 119
 120 <pre>
 121 ['start', 'avmplus', 'basics']
 122 ['test', 'avmplus', 'basics', 'unsigned_int']
 123 ['pass', 'avmplus', 'basics', 'unsigned_int']
 124 ['test', 'avmplus', 'basics', 'signed_int']
 125 ['pass', 'avmplus', 'basics', 'signed_int']
 126 ['end', 'avmplus', 'basics']
 127 </pre>
 128
 129 <P> A "start" line is printed as the tests are entered for a component
 130 and category, and an "end" line when those tests are done.  Then for
 131 each test name, a "test" line is printed, followed by a "pass",
 132 "failure", or "exception" line, depending on how the test terminated.
 133 A "crash" line is printed if the prologue or epilogue code (see below)
 134 throws an exception.
 135
 136 <h3> Test format </h3>
 137
 138 <P> (A comment block at the top of "extensions/selftest.as" is the
 139 authoritative definition of the test format.  The information is
 140 repeated and expanded upon here, but the following description is not
 141 normative.)
 142
 143 <h4> Grammar </h4>
 144
 145 <P> Apart from comments and blank lines the input file must match the
 146 nonterminal "file" in the following grammar.
 147
 148 <pre>
 149 file      ::= component category ifdef* (decls | methods | prologue | epilogue | test)*
 150 component ::= SOL "%%component" ident EOL
 151 category  ::= SOL "%%category" ident EOL
 152 ifdef     ::= SOL "%%ifdef" ident EOL
 153 decls     ::= SOL "%%decls" EOL text
 154 methods   ::= SOL "%%methods" EOL text
 155 prologue  ::= SOL "%%prologue" EOL text
 156 epilogue  ::= SOL "%%epilogue" EOL text
 157 test      ::= SOL "%%test" ident EOL (text | verify)*
 158 verify    ::= SOL "%%verify" expr EOL
 159 expr      ::= text but not newline
 160 text      ::= arbitrary text not containing "%%" at SOL
 161 EOL       ::= newline
 162 SOL       ::= beginning of line, possibly with leading spaces
 163 </pre>
 164
 165 <P> Comment lines are C++ style: "//" possibly preceded by blanks, to the end of line.
 166
 167 <P> Blank lines are legal everywhere.
 168
 169 <h4> Semantic constraints on the selftest specification </h4>
 170
 171 <UL>
 172 <LI> There can be an arbitrary number of tests, but at least one.
 173 <LI> There can be at most one prologue or epilogue.
 174 <LI> There can be an arbitrary number of decls and methods, they are just cat'ed together.
 175 <LI> There must be at least one verify in a test.
 176 </UL>
 177
 178 <P> (The requirements for at least one test and at least one verify
 179 are fascistic, but may help catch errors.  Experience will tell.)
 180
 181 <h4> Semantics of the specification </h4>
 182
 183 <P> A selftest file defines one or more test cases for a component
 184 (like "avmplus" or "player") and a category (like "peephole" or
 185 "jit").  Additionally, each test is named.  The test harness lets the
 186 user define the components, categories, and tests to run.
 187
 188 <P> Each generated file is compiled conditionally on the existence of
 189 AVMPLUS_SELFTEST and also each of the names declared in %%ifdef
 190 statements.
 191
 192 <P> The code that is generated comprises a class and its methods, all
 193 within a "namespace avmplus" block. Text in %%decls blocks is inserted
 194 verbatim into the class definition; text in %%methods blocks is
 195 inserted verbatim into the top level of the file. These sections allow
 196 the test case to define and use auxiliary data and methods.
 197
 198 <P> The code in a %%prologue section will be run once, before any test
 199 in the file is run.  The code in an %%epilogue section will also be
 200 run once, after all tests in the file have been run. They are
 201 typically used to initialize instance data (declared by %%decls).
 202
 203 <P> Each %%test is enclosed in an anonymous method of the generated
 204 class.  Each %%verify that appears in the test will test its
 205 condition, and if it does not hold that test function will be aborted
 206 and the error recorded.  Subsequent %%verify statements in that test
 207 will not be executed; however, subsequent tests in the same selftest
 208 spec will be executed.
 209
 210 <P> Comment lines preceding directives are copied verbatim to the output.
 211
 212 <h3> Examples </h3>
 213
 214 <h4> Simple example </h4>
 215
 216 <P> The following trivial example containing two tests tests whether
 217 right shift works as we require it to.
 218
 219 <pre>
 220 %%component avmplus
 221 %%category basics
 222
 223 %%test unsigned_int
 224
 225 // Does right shift of unsigned quantities work?
 226 %%verify (int)(~0U >> 1) > 0
 227
 228 %%test signed_int
 229
 230 // Does right shift of signed quantities work?
 231 %%verify (-1 >> 1) == -1
 232 </pre>
 233
 234 <h4> Complicated example </h4>
 235
 236 <P> The following example tests the peephole optimizer.  It shows how
 237 additional constraints are placed on the test with %%ifdef, how
 238 instance variables are inserted into the generated class, then
 239 initialized, used, and destroyed, and how multiple verifications may
 240 be used in a single test.
 241
 242 <P> (The actual test is that two GETLOCAL instructions are coalesced
 243 as a GET2LOCALS, provided that the original arguments are in the
 244 correct range.  The third GETLOCAL is not folded with the earlier ones
 245 into GET3LOCALS because its argument is too large.)
 246
 247 <pre>
 248 %%component avmplus
 249 %%category  peephole
 250 %%ifdef     AVMPLUS_PEEPHOLE_OPTIMIZER
 251
 252 %%decls
 253
 254 private:
 255 #ifdef AVMPLUS_DIRECT_THREADED
 256     void** opcode_labels;
 257 #endif
 258
 259 %%prologue
 260
 261 #ifdef AVMPLUS_DIRECT_THREADED
 262     int len = WOP_LAST+1
 263     opcode_labels = new void*[len];
 264     for ( int i=0 ; i &lt; len ; i++ )
 265         opcode_labels[i] = (void*)i;
 266 #endif
 267
 268 %%epilogue
 269
 270 #ifdef AVMPLUS_DIRECT_THREADED
 271     delete [] opcode_labels;
 272 #endif
 273
 274 %%test get2locals
 275
 276 #ifdef AVMPLUS_DIRECT_THREADED
 277     Translator* t = new Translator(core, NULL, opcode_labels);
 278 #else
 279     Translator* t = new Translator(core, NULL);
 280 #endif
 281
 282     t->emitOp1(WOP_getlocal, 5);
 283     t->emitOp1(WOP_getlocal, 4);
 284     t->emitOp1(WOP_getlocal, 65536);
 285     uint32_t* code;
 286     uint32_t len = t->epilogue(&code);
 287
 288 %%verify len == 4
 289 %%verify code[0] == NEW_OPCODE(WOP_get2locals)
 290 %%verify code[1] == ((4 &lt;&lt; 16) | 5)
 291 %%verify code[2] == NEW_OPCODE(WOP_getlocal)
 292 %%verify code[3] == 65536
 293
 294     delete t;
 295 </pre>
 296
 297
 298 <h2> TO DO </h2>
 299
 300 <P> <b>Postprocessing facilities:</b> We're really not interested in
 301 knowing about the tests that pass, only those that fail (in some way),
 302 but the output is voluminous.  There needs to be a postprocessing
 303 script.
 304
 305 <P> <b>Negative tests:</b> The only test form is %%verify, but we're
 306 going to need %%verify-not and %%verify-throw before long, I bet.
 307
 308 </body>
 309 </html>