documentation/testing.sgml

   1   <chapter id="testing">
   2     <title>Writing Conformance tests</title>
   3
   4     <para>
   5       Written by &name-francois-gouget; <email>&email-francois-gouget;</email>
   6     </para>
   7     <para>
   8       Note: This part of the documentation is still very much a work in
   9       progress and is in no way complete.
  10     </para>
  11
  12     <sect1 id="testing-intro">
  13       <title>Introduction</title>
  14       <para>
  15         With more The Windows API follows no standard, it is itself a defacto
  16         standard, and deviations from that standard, even small ones, often
  17         cause applications to crash or misbehave in some way. Furthermore
  18         a conformance test suite is the most accurate (if not necessarily
  19         the most complete) form of API documentation and can be used to
  20         supplement the Windows API documentation.
  21       </para>
  22       <para>
  23         Writing a conformance test suite for more than 10000 APIs is no small
  24         undertaking. Fortunately it can prove very useful to the development
  25         of Wine way before it is complete.
  26         <itemizedlist>
  27           <listitem>
  28             <para>
  29               The conformance test suite must run on Windows. This is
  30               necessary to provide a reasonable way to verify its accuracy.
  31               Furthermore the tests must pass successfully on all Windows
  32               platforms (tests not relevant to a given platform should be
  33               skipped).
  34             </para>
  35             <para>
  36               A consequence of this is that the test suite will provide a
  37               great way to detect variations in the API between different
  38               Windows versions. For instance, this can provide insights
  39               into the differences between the, often undocumented, Win9x and
  40               NT Windows families.
  41             </para>
  42             <para>
  43               However, one must remember that the goal of Wine is to run
  44               Windows applications on Linux, not to be a clone of any specific
  45               Windows version. So such variations must only be tested for when
  46               relevant to that goal.
  47             </para>
  48           </listitem>
  49           <listitem>
  50             <para>
  51               Writing conformance tests is also an easy way to discover
  52               bugs in Wine. Of course, before fixing the bugs discovered in
  53               this way, one must first make sure that the new tests do pass
  54               successfully on at least one Windows 9x and one Windows NT
  55               version.
  56             </para>
  57             <para>
  58               Bugs discovered this way should also be easier to fix. Unlike
  59               some mysterious application crashes, when a conformance test
  60               fails, the expected behavior and APIs tested for are known thus
  61               greatly simplifying the diagnosis.
  62             </para>
  63           </listitem>
  64           <listitem>
  65             <para>
  66               To detect regressions. Simply running the test suite regularly
  67               in Wine turns it into a great tool to detect regressions.
  68               When a test fails, one immediately knows what was the expected
  69               behavior and which APIs are involved. Thus regressions caught
  70               this way should be detected earlier, because it is easy to run
  71               all tests on a regular basis, and easier to fix because of the
  72               reduced diagnosis work.
  73             </para>
  74           </listitem>
  75           <listitem>
  76             <para>
  77               Tests written in advance of the Wine development (possibly even
  78               by non Wine developers) can also simplify the work of the
  79               future implementer by making it easier for him to check the
  80               correctness of his code.
  81             </para>
  82           </listitem>
  83           <listitem>
  84             <para>
  85               Conformance tests will also come in handy when testing Wine on
  86               new (or not as widely used) architectures such as FreeBSD,
  87               Solaris x86 or even non-x86 systems. Even when the port does
  88               not involve any significant change in the thread management,
  89               exception handling or other low-level aspects of Wine, new
  90               architectures can expose subtle bugs that can be hard to
  91               diagnose when debugging regular (complex) applications.
  92             </para>
  93           </listitem>
  94         </itemizedlist>
  95       </para>
  96     </sect1>
  97
  98
  99     <sect1 id="testing-what">
 100       <title>What to test for?</title>
 101       <para>
 102         The first thing to test for is the documented behavior of APIs
 103         and such as CreateFile. For instance one can create a file using a
 104         long pathname, check that the behavior is correct when the file
 105         already exists, try to open the file using the corresponding short
 106         pathname, convert the filename to Unicode and try to open it using
 107         CreateFileW, and all other things which are documented and that
 108         applications rely on.
 109       </para>
 110       <para>
 111         While the testing framework is not specifically geared towards this
 112         type of tests, it is also possible to test the behavior of Windows
 113         messages. To do so, create a window, preferably a hidden one so that
 114         it does not steal the focus when running the tests, and send messages
 115         to that window or to controls in that window. Then, in the message
 116         procedure, check that you receive the expected messages and with the
 117         correct parameters.
 118       </para>
 119       <para>
 120         For instance you could create an edit control and use WM_SETTEXT to
 121         set its contents, possibly check length restrictions, and verify the
 122         results using WM_GETTEXT. Similarly one could create a listbox and
 123         check the effect of LB_DELETESTRING on the list's number of items,
 124         selected items list, highlighted item, etc.
 125       </para>
 126       <para>
 127         However, undocumented behavior should not be tested for unless there
 128         is an application that relies on this behavior, and in that case the
 129         test should mention that application, or unless one can strongly
 130         expect applications to rely on this behavior, typically APIs that
 131         return the required buffer size when the buffer pointer is NULL.
 132       </para>
 133     </sect1>
 134
 135
 136     <sect1 id="testing-wine">
 137       <title>Running the tests in Wine</title>
 138       <para>
 139         The simplest way to run the tests in Wine is to type 'make test' in
 140         the Wine sources top level directory. This will run all the Wine
 141         conformance tests.
 142       </para>
 143       <para>
 144         The tests for a specific Wine library are located in a 'tests'
 145         directory in that library's directory. Each test is contained in a
 146         file (e.g. <filename>dlls/kernel/tests/thread.c</>). Each
 147         file itself contains many checks concerning one or more related APIs.
 148       </para>
 149       <para>
 150         So to run all the tests related to a given Wine library, go to the
 151         corresponding 'tests' directory and type 'make test'. This will
 152         compile the tests, run them, and create an '<replaceable>xxx</>.ok'
 153         file for each test that passes successfully. And if you only want to
 154         run the tests contained in the <filename>thread.c</> file of the
 155         kernel library, you would do:
 156 <screen>
 157 <prompt>$ </>cd dlls/kernel/tests
 158 <prompt>$ </>make thread.ok
 159 </screen>
 160       </para>
 161       <para>
 162         Note that if the test has already been run and is up to date (i.e. if
 163         neither the kernel library nor the <filename>thread.c</> file has
 164         changed since the <filename>thread.ok</> file was created), then make
 165         will say so. To force the test to be re-run, delete the
 166         <filename>thread.ok</> file, and run the make command again.
 167       </para>
 168       <para>
 169         You can also run tests manually using a command similar to the
 170         following:
 171 <screen>
 172 <prompt>$ </>../../../tools/runtest -q -M kernel32.dll -p kernel32_test.exe.so thread.c
 173 <prompt>$ </>../../../tools/runtest -p kernel32_test.exe.so thread.c
 174 thread.c: 86 tests executed, 5 marked as todo, 0 failures.
 175 </screen>
 176         The '-P wine' options defines the platform that is currently being
 177         tested. Remove the '-q' option if you want the testing framework
 178         to report statistics about the number of successful and failed tests.
 179         Run <command>runtest -h</> for more details.
 180       </para>
 181     </sect1>
 182
 183
 184     <sect1 id="testing-windows">
 185       <title>Building and running the tests on Windows</title>
 186       <sect2>
 187         <title>Using pre-compiled binaries</title>
 188         <para>
 189           Unfortunately there are no pre-compiled binaries yet. However if
 190           send an email to the Wine development list you can probably get
 191           someone to send them to you, and maybe motivate some kind soul to
 192           put in place a mechanism for publishing such binaries on a regular
 193           basis.
 194         </para>
 195       </sect2>
 196       <sect2>
 197         <title>With Visual C++</title>
 198         <itemizedlist>
 199           <listitem><para>
 200             get the Wine sources
 201           </para></listitem>
 202           <listitem><para>
 203             Run msvcmaker to generate Visual C++ project files for the tests.
 204             'msvcmaker' is a perl script so you may be able to run it on
 205             Windows.
 206 <screen>
 207 <prompt>$ </>./tools/winapi/msvcmaker --no-wine
 208 </screen>
 209           </para></listitem>
 210           <listitem><para>
 211             If the previous steps were done on your Linux development
 212             machine, make the Wine sources accessible to the Windows machine
 213             on which you are going to compile them. Typically you would do
 214             this using Samba but copying them altogether would work too.
 215           </para></listitem>
 216           <listitem><para>
 217             On the Windows machine, open the <filename>winetest.dsw</>
 218             workspace. This will load each test's project. For each test there
 219             are two configurations: one compiles the test with the Wine
 220             headers, and the other uses the Visual C++ headers. Some tests
 221             will compile fine with the former, but most will require the
 222             latter.
 223           </para></listitem>
 224           <listitem><para>
 225             Open the <menuchoice><guimenu>Build</> <guimenu>Batch
 226             build...</></> menu and select the tests and build configurations
 227             you want to build. Then click on <guibutton>Build</>.
 228           </para></listitem>
 229           <listitem><para>
 230             To run a specific test from Visual C++, go to
 231             <menuchoice><guimenu>Project</> <guimenu>Settings...</></>. There
 232             select that test's project and build configuration and go to the
 233             <guilabel>Debug</> tab. There type the name of the specific test
 234             to run (e.g. 'thread') in the <guilabel>Program arguments</>
 235             field. Validate your change by clicking on <guibutton>Ok</> and
 236             start the test by clicking the red exclamation mark (or hitting
 237             'F5' or any other usual method).
 238           </para></listitem>
 239           <listitem><para>
 240             You can also run the tests from the command line. You will find
 241             them in either <filename>Output\Win32_Wine_Headers</> or
 242             <filename>Output\Win32_MSVC_Headers</> depending on the build
 243             method. So to run the kernel 'path' tests you would do:
 244 <screen>
 245 <prompt>C:\&gt;</>cd dlls\kernel\tests\Output\Win32_MSVC_Headers
 246 <prompt>C:\dlls\kernel\tests\Output\Win32_MSVC_Headers&gt;</>kernel32_test path
 247 </screen>
 248           </para></listitem>
 249         </itemizedlist>
 250       </sect2>
 251       <sect2>
 252         <title>With MinGW</title>
 253         <para>
 254           This needs to be documented. The best may be to ask on the Wine
 255           development mailing list and update this documentation with the
 256           result of your inquiry.
 257         </para>
 258       </sect2>
 259       <sect2>
 260         <title>Cross compiling with MinGW on Linux</title>
 261         <para>
 262           Here is how to generate Windows executables for the tests straight
 263           from the comfort of Linux.
 264         </para>
 265         <itemizedlist>
 266           <listitem><para>
 267             First you need to get the MinGW cross-compiler. On Debian all
 268             you need to do is type <command>apt-get install mingw32</>.
 269           </para></listitem>
 270           <listitem><para>
 271             If you had already run <command>configure</>, then delete
 272             <filename>config.cache</> and re-run <command>configure</>.
 273             You can then run <command>make crosstest</>. To sum up:
 274 <screen>
 275 <prompt>$ </><userinput>rm config.cache</>
 276 <prompt>$ </><userinput>./configure</>
 277 <prompt>$ </><userinput>make crosstest</>
 278 </screen>
 279           </para></listitem>
 280           <listitem><para>
 281             If you get an error when compiling <filename>winsock.h</> then
 282             you probably need to apply the following patch:
 283             <ulink url="http://www.winehq.com/hypermail/wine-patches/2002/12/0157.html">http://www.winehq.com/hypermail/wine-patches/2002/12/0157.html</>
 284           </para></listitem>
 285         </itemizedlist>
 286       </sect2>
 287     </sect1>
 288
 289
 290     <sect1 id="testing-test">
 291       <title>Inside a test</title>
 292
 293       <para>
 294         When writing new checks you can either modify an existing test file or
 295         add a new one. If your tests are related to the tests performed by an
 296         existing file, then add them to that file. Otherwise create a new .c
 297         file in the tests directory and add that file to the
 298         <varname>CTESTS</> variable in <filename>Makefile.in</>.
 299       </para>
 300       <para>
 301         A new test file will look something like the following:
 302 <screen>
 303 #include &lt;wine/test.h&gt;
 304 #include &lt;winbase.h&gt;
 305
 306 /* Maybe auxiliary functions and definitions here */
 307
 308 START_TEST(paths)
 309 {
 310    /* Write your checks there or put them in functions you will call from
 311     * there
 312     */
 313 }
 314 </screen>
 315       </para>
 316       <para>
 317         The test's entry point is the START_TEST section. This is where
 318         execution will start. You can put all your tests in that section but
 319         it may be better to split related checks in functions you will call
 320         from the START_TEST section. The parameter to START_TEST must match
 321         the name of the C file. So in the above example the C file would be
 322         called <filename>paths.c</>.
 323       </para>
 324       <para>
 325         Tests should start by including the <filename>wine/test.h</> header.
 326         This header will provide you access to all the testing framework
 327         functions. You can then include the windows header you need, but make
 328         sure to not include any Unix or Wine specific header: tests must
 329         compile on Windows.
 330       </para>
 331       <para>
 332         You can use <function>trace</> to print informational messages. Note
 333         that these messages will only be printed if 'runtest -v' is being used.
 334 <screen>
 335   trace("testing GlobalAddAtomA");
 336   trace("foo=%d",foo);
 337 </screen>
 338       </para>
 339       <para>
 340         Then just call functions and use <function>ok</> to make sure that
 341         they behaved as expected:
 342 <screen>
 343   ATOM atom = GlobalAddAtomA( "foobar" );
 344   ok( GlobalFindAtomA( "foobar" ) == atom, "could not find atom foobar" );
 345   ok( GlobalFindAtomA( "FOOBAR" ) == atom, "could not find atom FOOBAR" );
 346 </screen>
 347         The first parameter of <function>ok</> is an expression which must
 348         evaluate to true if the test was successful. The next parameter is a
 349         printf-compatible format string which is displayed in case the test
 350         failed, and the following optional parameters depend on the format
 351         string.
 352       </para>
 353     </sect1>
 354
 355     <sect1 id="testing-error-messages">
 356       <title>Writing good error messages</title>
 357       <para>
 358         The message that is printed when a test fails is
 359         <emphasis>extremely</> important.
 360       </para>
 361       <para>
 362         Someone will take your test, run it on a Windows platform that
 363         you don't have access to, and discover that it fails. They will then
 364         post an email with the output of the test, and in particular your
 365         error message. Someone, maybe you, will then have to figure out from
 366         this error message why the test failed.
 367       </para>
 368       <para>
 369         If the error message contains all the relevant information that will
 370         be easy. If not, then it will require modifying the test, finding
 371         someone to compile it on Windows, sending the modified version to the
 372         original tester and waiting for his reply. In other words, it will
 373         be long and painful.
 374       </para>
 375       <para>
 376         So how do you write a good error message? Let's start with an example
 377         of a bad error message:
 378 <screen>
 379     ok(GetThreadPriorityBoost(curthread,&amp;disabled)!=0,
 380        "GetThreadPriorityBoost Failed");
 381 </screen>
 382         This will yield:
 383 <screen>
 384 thread.c:123: Test failed: GetThreadPriorityBoost Failed
 385 </screen>
 386       </para>
 387       <para>
 388         Did you notice how the error message provides no information about
 389         why the test failed? We already know from the line number exactly
 390         which test failed. In fact the error message gives strictly no
 391         information that cannot already be obtained by reading the code. In
 392         other words it provides no more information than an empty string!
 393       </para>
 394       <para>
 395         Let's look at how to rewrite it:
 396 <screen>
 397     BOOL rc;
 398 ...
 399     rc=GetThreadPriorityBoost(curthread,&amp;disabled);
 400     ok(rc!=0 && disabled==0,"rc=%d error=%ld disabled=%d",
 401        rc,GetLastError(),disabled);
 402 </screen>
 403         This will yield:
 404 <screen>
 405 thread.c:123: Test failed: rc=0 error=120 disabled=0
 406 </screen>
 407       </para>
 408       <para>
 409         When receiving such a message, one would check the source, see that
 410         it's a call to GetThreadPriorityBoost, that the test failed not
 411         because the API returned the wrong value, but because it returned an
 412         error code. Furthermore we see that GetLastError() returned 120 which
 413         winerror.h defines as ERROR_CALL_NOT_IMPLEMENTED. So the source of
 414         the problem is obvious: this Windows platform (here Windows 98) does
 415         not support this API and thus the test must be modified to detect
 416         such a condition and skip the test.
 417       </para>
 418       <para>
 419         So a good error message should provide all the information which
 420         cannot be obtained by reading the source, typically the function
 421         return value, error codes, and any function output parameter. Even if
 422         more information is needed to fully understand a problem,
 423         systematically providing the above is easy and will help cut down the
 424         number of iterations required to get to a resolution.
 425       </para>
 426       <para>
 427         It may also be a good idea to dump items that may be hard to retrieve
 428         from the source, like the expected value in a test if it is the
 429         result of an earlier computation, or comes from a large array of test
 430         values (e.g. index 112 of _pTestStrA in vartest.c). In that respect,
 431         for some tests you may want to define a macro such as the following:
 432 <screen>
 433 #define eq(received, expected, label, type) \
 434         ok((received) == (expected), "%s: got " type " instead of " type, (label),(received),(expected))
 435
 436 ...
 437
 438     eq( b, curr_val, "SPI_{GET,SET}BEEP", "%d" );
 439 </screen>
 440        </para>
 441     </sect1>
 442
 443
 444     <sect1 id="testing-platforms">
 445       <title>Handling platform issues</title>
 446       <para>
 447         Some checks may be written before they pass successfully in Wine.
 448         Without some mechanism, such checks would potentially generate
 449         hundred of known failures for months each time the tests are being run.
 450         This would make it hard to detect new failures caused by a regression.
 451         or to detect that a patch fixed a long standing issue.
 452       </para>
 453       <para>
 454         Thus the Wine testing framework has the concept of platforms and
 455         groups of checks can be declared as expected to fail on some of them.
 456         In the most common case, one would declare a group of tests as
 457         expected to fail in Wine. To do so, use the following construct:
 458 <screen>
 459 todo_wine {
 460     SetLastError( 0xdeadbeef );
 461     ok( GlobalAddAtomA(0) == 0 && GetLastError() == 0xdeadbeef, "failed to add atom 0" );
 462 }
 463 </screen>
 464         On Windows the above check would be performed normally, but on Wine it
 465         would be expected to fail, and not cause the failure of the whole
 466         test. However. If that check were to succeed in Wine, it would
 467         cause the test to fail, thus making it easy to detect when something
 468         has changed that fixes a bug. Also note that todo checks are accounted
 469         separately from regular checks so that the testing statistics remain
 470         meaningful. Finally, note that todo sections can be nested so that if
 471         a test only fails on the cygwin and reactos platforms, one would
 472         write:
 473 <screen>
 474 todo("cygwin") {
 475     todo("reactos") {
 476         ...
 477     }
 478 }
 479 </screen>
 480         <!-- FIXME: Would we really have platforms such as reactos, cygwin, freebsd & co? -->
 481         But specific platforms should not be nested inside a todo_wine section
 482         since that would be redundant.
 483       </para>
 484       <para>
 485         When writing tests you will also encounter differences between Windows
 486         9x and Windows NT platforms. Such differences should be treated
 487         differently from the platform issues mentioned above. In particular
 488         you should remember that the goal of Wine is not to be a clone of any
 489         specific Windows version but to run Windows applications on Unix.
 490       </para>
 491       <para>
 492         So, if an API returns a different error code on Windows 9x and
 493         Windows NT, your check should just verify that Wine returns one or
 494         the other:
 495 <screen>
 496 ok ( GetLastError() == WIN9X_ERROR || GetLastError() == NT_ERROR, ...);
 497 </screen>
 498       </para>
 499       <para>
 500         If an API is only present on some Windows platforms, then use
 501         LoadLibrary and GetProcAddress to check if it is implemented and
 502         invoke it. Remember, tests must run on all Windows platforms.
 503         Similarly, conformance tests should nor try to correlate the Windows
 504         version returned by GetVersion with whether given APIs are
 505         implemented or not. Again, the goal of Wine is to run Windows
 506         applications (which do not do such checks), and not be a clone of a
 507         specific Windows version.
 508       </para>
 509       <para>
 510         FIXME: What about checks that cause the process to crash due to a bug?
 511       </para>
 512     </sect1>
 513
 514
 515 <!-- FIXME: Strategies for testing threads, testing network stuff,
 516  file handling, eq macro... -->
 517
 518   </chapter>
 519
 520 <!-- Keep this comment at the end of the file
 521 Local variables:
 522 mode: sgml
 523 sgml-parent-document:("wine-devel.sgml" "set" "book" "part" "chapter" "")
 524 End:
 525 -->