documentation/testing.sgml

   1   <chapter id="testing">
   2     <title>Writing Conformance tests</title>
   3
   4     <sect1 id="testing-intro">
   5       <title>Introduction</title>
   6       <para>
   7         The Windows API follows no standard, it is itself a de facto standard,
   8         and deviations from that standard, even small ones, often cause
   9         applications to crash or misbehave in some way.
  10       </para>
  11       <para>
  12         The question becomes, "How do we ensure compliance with that standard?"
  13         The answer is, "By using the API documentation available to us and
  14         backing that up with conformance tests." Furthermore, a conformance
  15         test suite is the most accurate (if not necessarily the most complete)
  16         form of API documentation and can be used to supplement the Windows
  17         API documentation.
  18       </para>
  19       <para>
  20         Writing a conformance test suite for more than 10000 APIs is no small
  21         undertaking. Fortunately it can prove very useful to the development
  22         of Wine way before it is complete.
  23         <itemizedlist>
  24           <listitem>
  25             <para>
  26               The conformance test suite must run on Windows. This is
  27               necessary to provide a reasonable way to verify its accuracy.
  28               Furthermore the tests must pass successfully on all Windows
  29               platforms (tests not relevant to a given platform should be
  30               skipped).
  31             </para>
  32             <para>
  33               A consequence of this is that the test suite will provide a
  34               great way to detect variations in the API between different
  35               Windows versions. For instance, this can provide insights
  36               into the differences between the, often undocumented, Win9x and
  37               NT Windows families.
  38             </para>
  39             <para>
  40               However, one must remember that the goal of Wine is to run
  41               Windows applications on Linux, not to be a clone of any specific
  42               Windows version. So such variations must only be tested for when
  43               relevant to that goal.
  44             </para>
  45           </listitem>
  46           <listitem>
  47             <para>
  48               Writing conformance tests is also an easy way to discover
  49               bugs in Wine. Of course, before fixing the bugs discovered in
  50               this way, one must first make sure that the new tests do pass
  51               successfully on at least one Windows 9x and one Windows NT
  52               version.
  53             </para>
  54             <para>
  55               Bugs discovered this way should also be easier to fix. Unlike
  56               some mysterious application crashes, when a conformance test
  57               fails, the expected behavior and APIs tested for are known thus
  58               greatly simplifying the diagnosis.
  59             </para>
  60           </listitem>
  61           <listitem>
  62             <para>
  63               To detect regressions. Simply running the test suite regularly
  64               in Wine turns it into a great tool to detect regressions.
  65               When a test fails, one immediately knows what was the expected
  66               behavior and which APIs are involved. Thus regressions caught
  67               this way should be detected earlier, because it is easy to run
  68               all tests on a regular basis, and easier to fix because of the
  69               reduced diagnosis work.
  70             </para>
  71           </listitem>
  72           <listitem>
  73             <para>
  74               Tests written in advance of the Wine development (possibly even
  75               by non Wine developers) can also simplify the work of the
  76               future implementer by making it easier for him to check the
  77               correctness of his code.
  78             </para>
  79           </listitem>
  80           <listitem>
  81             <para>
  82               Conformance tests will also come in handy when testing Wine on
  83               new (or not as widely used) architectures such as FreeBSD,
  84               Solaris x86 or even non-x86 systems. Even when the port does
  85               not involve any significant change in the thread management,
  86               exception handling or other low-level aspects of Wine, new
  87               architectures can expose subtle bugs that can be hard to
  88               diagnose when debugging regular (complex) applications.
  89             </para>
  90           </listitem>
  91         </itemizedlist>
  92       </para>
  93     </sect1>
  94
  95
  96     <sect1 id="testing-what">
  97       <title>What to test for?</title>
  98       <para>
  99         The first thing to test for is the documented behavior of APIs
 100         and such as CreateFile. For instance one can create a file using a
 101         long pathname, check that the behavior is correct when the file
 102         already exists, try to open the file using the corresponding short
 103         pathname, convert the filename to Unicode and try to open it using
 104         CreateFileW, and all other things which are documented and that
 105         applications rely on.
 106       </para>
 107       <para>
 108         While the testing framework is not specifically geared towards this
 109         type of tests, it is also possible to test the behavior of Windows
 110         messages. To do so, create a window, preferably a hidden one so that
 111         it does not steal the focus when running the tests, and send messages
 112         to that window or to controls in that window. Then, in the message
 113         procedure, check that you receive the expected messages and with the
 114         correct parameters.
 115       </para>
 116       <para>
 117         For instance you could create an edit control and use WM_SETTEXT to
 118         set its contents, possibly check length restrictions, and verify the
 119         results using WM_GETTEXT. Similarly one could create a listbox and
 120         check the effect of LB_DELETESTRING on the list's number of items,
 121         selected items list, highlighted item, etc.
 122       </para>
 123       <para>
 124         However, undocumented behavior should not be tested for unless there
 125         is an application that relies on this behavior, and in that case the
 126         test should mention that application, or unless one can strongly
 127         expect applications to rely on this behavior, typically APIs that
 128         return the required buffer size when the buffer pointer is NULL.
 129       </para>
 130     </sect1>
 131
 132
 133     <sect1 id="testing-wine">
 134       <title>Running the tests in Wine</title>
 135       <para>
 136         The simplest way to run the tests in Wine is to type 'make test' in
 137         the Wine sources top level directory. This will run all the Wine
 138         conformance tests.
 139       </para>
 140       <para>
 141         The tests for a specific Wine library are located in a 'tests'
 142         directory in that library's directory. Each test is contained in a
 143         file (e.g. <filename>dlls/kernel/tests/thread.c</>). Each
 144         file itself contains many checks concerning one or more related APIs.
 145       </para>
 146       <para>
 147         So to run all the tests related to a given Wine library, go to the
 148         corresponding 'tests' directory and type 'make test'. This will
 149         compile the tests, run them, and create an '<replaceable>xxx</>.ok'
 150         file for each test that passes successfully. And if you only want to
 151         run the tests contained in the <filename>thread.c</> file of the
 152         kernel library, you would do:
 153 <screen>
 154 <prompt>$ </>cd dlls/kernel/tests
 155 <prompt>$ </>make thread.ok
 156 </screen>
 157       </para>
 158       <para>
 159         Note that if the test has already been run and is up to date (i.e. if
 160         neither the kernel library nor the <filename>thread.c</> file has
 161         changed since the <filename>thread.ok</> file was created), then make
 162         will say so. To force the test to be re-run, delete the
 163         <filename>thread.ok</> file, and run the make command again.
 164       </para>
 165       <para>
 166         You can also run tests manually using a command similar to the
 167         following:
 168 <screen>
 169 <prompt>$ </>../../../tools/runtest -q -M kernel32.dll -p kernel32_test.exe.so thread.c
 170 <prompt>$ </>../../../tools/runtest -p kernel32_test.exe.so thread.c
 171 thread.c: 86 tests executed, 5 marked as todo, 0 failures.
 172 </screen>
 173         The '-P wine' options defines the platform that is currently being
 174         tested. Remove the '-q' option if you want the testing framework
 175         to report statistics about the number of successful and failed tests.
 176         Run <command>runtest -h</> for more details.
 177       </para>
 178     </sect1>
 179
 180
 181     <sect1 id="cross-compiling-tests">
 182       <title>Cross-compiling the tests with MinGW</title>
 183       <sect2>
 184         <title>Setup of the MinGW cross-compiling environment</title>
 185         <para>
 186           Here are some instructions to setup MinGW on different Linux
 187           distributions and *BSD.
 188         </para>
 189         <sect3>
 190           <title>Debian GNU/Linux</title>
 191           <para>
 192             On Debian all you need to do is type <command>apt-get install
 193             mingw32</>.
 194           </para>
 195         </sect3>
 196         <sect3>
 197           <title>Red Hat Linux like rpm systems</title>
 198           <para>
 199             This includes Fedora Core, Red Hat Enterprise Linux, Mandrake,
 200             most probably SuSE Linux too, etc. But this list isn't exhaustive;
 201             the following steps should probably work on any rpm based system.
 202           </para>
 203           <para>
 204             Download and install the latest rpm's from
 205             <ulink url="http://mirzam.it.vu.nl/mingw/">MinGW RPM packages</>.
 206             Alternatively you can follow the instructions on that page and
 207             build your own packages from the source rpm's listed there as well.
 208           </para>
 209         </sect3>
 210         <sect3>
 211           <title>*BSD</title>
 212           <para>
 213             The *BSD systems have in their ports collection a port for the
 214             MinGW cross-compiling environment. Please see the documentation
 215             of your system about how to build and install a port.
 216           </para>
 217         </sect3>
 218       </sect2>
 219       <sect2>
 220       <title>Compiling the tests</title>
 221       <para>
 222         Having the cross-compiling environment set up the generation of the
 223         Windows executables is easy by using the Wine build system.
 224       </para>
 225       <para>
 226         If you had already run <command>configure</>, then delete
 227         <filename>config.cache</> and re-run <command>configure</>.
 228         You can then run <command>make crosstest</>. To sum up:
 229 <screen>
 230 <prompt>$ </><userinput>rm config.cache</>
 231 <prompt>$ </><userinput>./configure</>
 232 <prompt>$ </><userinput>make crosstest</>
 233 </screen>
 234       </para>
 235       </sect2>
 236     </sect1>
 237
 238
 239     <sect1 id="testing-windows">
 240       <title>Building and running the tests on Windows</title>
 241       <sect2>
 242         <title>Using pre-compiled binaries</title>
 243         <para>
 244           Unfortunately there are no pre-compiled binaries yet. However if
 245           send an email to the Wine development list you can probably get
 246           someone to send them to you, and maybe motivate some kind soul to
 247           put in place a mechanism for publishing such binaries on a regular
 248           basis.
 249         </para>
 250       </sect2>
 251       <sect2>
 252         <title>With Visual C++</title>
 253         <itemizedlist>
 254           <listitem><para>
 255             If you are using Visual Studio 6, make sure you have the
 256             "processor pack" from
 257             <ulink url="http://msdn.microsoft.com/vstudio/downloads/tools/ppack/default.aspx">http://msdn.microsoft.com/vstudio/downloads/tools/ppack/default.aspx</>.
 258             The processor pack fixes <emphasis>"error C2520: conversion from
 259             unsigned __int64 to double not implemented, use signed __int64"</>.
 260             However note that the "processor pack" is incompatible with
 261             Visual Studio 6.0 Standard Edition, and with the Visual Studio 6
 262             Service Pack 6. If you are using Visual Studio 7 or greater you
 263             do not need the processor pack. In either case it is recommended
 264             to the most recent compatible Visual Studio
 265             <ulink url="http://msdn.microsoft.com/vstudio/downloads/updates/sp/">service pack</>.
 266           </para></listitem>
 267           <listitem><para>
 268             get the Wine sources
 269           </para></listitem>
 270           <listitem><para>
 271             Run msvcmaker to generate Visual C++ project files for the tests.
 272             'msvcmaker' is a perl script so you may be able to run it on
 273             Windows.
 274 <screen>
 275 <prompt>$ </>./tools/winapi/msvcmaker --no-wine
 276 </screen>
 277           </para></listitem>
 278           <listitem><para>
 279             If the previous steps were done on your Linux development
 280             machine, make the Wine sources accessible to the Windows machine
 281             on which you are going to compile them. Typically you would do
 282             this using Samba but copying them altogether would work too.
 283           </para></listitem>
 284           <listitem><para>
 285             On the Windows machine, open the <filename>winetest.dsw</>
 286             workspace. This will load each test's project. For each test there
 287             are two configurations: one compiles the test with the Wine
 288             headers, and the other uses the Microsoft headers.
 289           </para></listitem>
 290           <listitem><para>
 291             If you choose the "Win32 MSVC Headers" configuration, most of the
 292             tests will not compile with the regular Visual Studio headers. So
 293             to use this configuration, download and install a recent
 294             <ulink url="http://www.microsoft.com/msdownload/platformsdk/sdkupdate/">Platform SDK</>
 295             (the download requires Internet Explorer!), as well as the latest
 296             <ulink url="http://msdn.microsoft.com/library/default.asp?url=/downloads/list/directx.asp">DirectX SDK</>.
 297             Then, <ulink url="http://msdn.microsoft.com/library/default.asp?url=/library/EN-US/sdkintro/sdkintro/installing_the_platform_sdk_with_visual_studio.asp">configure Visual Studio</>
 298             to use these SDK's headers and libraries. Alternately you could go
 299             to the <menuchoice><guimenu>Project</> <guimenu>Settings...</></>
 300             menu and modify the settings appropriately, but you would then
 301             have to redo this whenever you rerun msvcmaker.
 302           </para></listitem>
 303           <listitem><para>
 304             Open the <menuchoice><guimenu>Build</> <guimenu>Batch
 305             build...</></> menu and select the tests and build configurations
 306             you want to build. Then click on <guibutton>Build</>.
 307           </para></listitem>
 308           <listitem><para>
 309             To run a specific test from Visual C++, go to
 310             <menuchoice><guimenu>Project</> <guimenu>Settings...</></>. There
 311             select that test's project and build configuration and go to the
 312             <guilabel>Debug</> tab. There type the name of the specific test
 313             to run (e.g. 'thread') in the <guilabel>Program arguments</>
 314             field. Validate your change by clicking on <guibutton>Ok</> and
 315             start the test by clicking the red exclamation mark (or hitting
 316             'F5' or any other usual method).
 317           </para></listitem>
 318           <listitem><para>
 319             You can also run the tests from the command line. You will find
 320             them in either <filename>Output\Win32_Wine_Headers</> or
 321             <filename>Output\Win32_MSVC_Headers</> depending on the build
 322             method. So to run the kernel 'path' tests you would do:
 323 <screen>
 324 <prompt>C:\&gt;</>cd dlls\kernel\tests\Output\Win32_MSVC_Headers
 325 <prompt>C:\dlls\kernel\tests\Output\Win32_MSVC_Headers&gt;</>kernel32_test path
 326 </screen>
 327           </para></listitem>
 328         </itemizedlist>
 329       </sect2>
 330       <sect2>
 331         <title>With MinGW</title>
 332         <para>
 333          Wine's build system already has support for building tests with a MinGW
 334          cross-compiler. See the section above called 'Setup of the MinGW
 335          cross-compiling environment' for instructions on how to set things up.
 336          When you have a MinGW environment installed all you need to do is rerun
 337          configure and it should detect the MinGW compiler and tools. Then run
 338          'make crosstest' to start building the tests.
 339         </para>
 340       </sect2>
 341     </sect1>
 342
 343
 344     <sect1 id="testing-test">
 345       <title>Inside a test</title>
 346
 347       <para>
 348         When writing new checks you can either modify an existing test file or
 349         add a new one. If your tests are related to the tests performed by an
 350         existing file, then add them to that file. Otherwise create a new .c
 351         file in the tests directory and add that file to the
 352         <varname>CTESTS</> variable in <filename>Makefile.in</>.
 353       </para>
 354       <para>
 355         A new test file will look something like the following:
 356 <screen>
 357 #include &lt;wine/test.h&gt;
 358 #include &lt;winbase.h&gt;
 359
 360 /* Maybe auxiliary functions and definitions here */
 361
 362 START_TEST(paths)
 363 {
 364    /* Write your checks there or put them in functions you will call from
 365     * there
 366     */
 367 }
 368 </screen>
 369       </para>
 370       <para>
 371         The test's entry point is the START_TEST section. This is where
 372         execution will start. You can put all your tests in that section but
 373         it may be better to split related checks in functions you will call
 374         from the START_TEST section. The parameter to START_TEST must match
 375         the name of the C file. So in the above example the C file would be
 376         called <filename>paths.c</>.
 377       </para>
 378       <para>
 379         Tests should start by including the <filename>wine/test.h</> header.
 380         This header will provide you access to all the testing framework
 381         functions. You can then include the windows header you need, but make
 382         sure to not include any Unix or Wine specific header: tests must
 383         compile on Windows.
 384       </para>
 385       <para>
 386         You can use <function>trace</> to print informational messages. Note
 387         that these messages will only be printed if 'runtest -v' is being used.
 388 <screen>
 389   trace("testing GlobalAddAtomA");
 390   trace("foo=%d",foo);
 391 </screen>
 392       </para>
 393       <para>
 394         Then just call functions and use <function>ok</> to make sure that
 395         they behaved as expected:
 396 <screen>
 397   ATOM atom = GlobalAddAtomA( "foobar" );
 398   ok( GlobalFindAtomA( "foobar" ) == atom, "could not find atom foobar" );
 399   ok( GlobalFindAtomA( "FOOBAR" ) == atom, "could not find atom FOOBAR" );
 400 </screen>
 401         The first parameter of <function>ok</> is an expression which must
 402         evaluate to true if the test was successful. The next parameter is a
 403         printf-compatible format string which is displayed in case the test
 404         failed, and the following optional parameters depend on the format
 405         string.
 406       </para>
 407     </sect1>
 408
 409     <sect1 id="testing-error-messages">
 410       <title>Writing good error messages</title>
 411       <para>
 412         The message that is printed when a test fails is
 413         <emphasis>extremely</> important.
 414       </para>
 415       <para>
 416         Someone will take your test, run it on a Windows platform that
 417         you don't have access to, and discover that it fails. They will then
 418         post an email with the output of the test, and in particular your
 419         error message. Someone, maybe you, will then have to figure out from
 420         this error message why the test failed.
 421       </para>
 422       <para>
 423         If the error message contains all the relevant information that will
 424         be easy. If not, then it will require modifying the test, finding
 425         someone to compile it on Windows, sending the modified version to the
 426         original tester and waiting for his reply. In other words, it will
 427         be long and painful.
 428       </para>
 429       <para>
 430         So how do you write a good error message? Let's start with an example
 431         of a bad error message:
 432 <screen>
 433     ok(GetThreadPriorityBoost(curthread,&amp;disabled)!=0,
 434        "GetThreadPriorityBoost Failed");
 435 </screen>
 436         This will yield:
 437 <screen>
 438 thread.c:123: Test failed: GetThreadPriorityBoost Failed
 439 </screen>
 440       </para>
 441       <para>
 442         Did you notice how the error message provides no information about
 443         why the test failed? We already know from the line number exactly
 444         which test failed. In fact the error message gives strictly no
 445         information that cannot already be obtained by reading the code. In
 446         other words it provides no more information than an empty string!
 447       </para>
 448       <para>
 449         Let's look at how to rewrite it:
 450 <screen>
 451     BOOL rc;
 452 ...
 453     rc=GetThreadPriorityBoost(curthread,&amp;disabled);
 454     ok(rc!=0 && disabled==0,"rc=%d error=%ld disabled=%d",
 455        rc,GetLastError(),disabled);
 456 </screen>
 457         This will yield:
 458 <screen>
 459 thread.c:123: Test failed: rc=0 error=120 disabled=0
 460 </screen>
 461       </para>
 462       <para>
 463         When receiving such a message, one would check the source, see that
 464         it's a call to GetThreadPriorityBoost, that the test failed not
 465         because the API returned the wrong value, but because it returned an
 466         error code. Furthermore we see that GetLastError() returned 120 which
 467         winerror.h defines as ERROR_CALL_NOT_IMPLEMENTED. So the source of
 468         the problem is obvious: this Windows platform (here Windows 98) does
 469         not support this API and thus the test must be modified to detect
 470         such a condition and skip the test.
 471       </para>
 472       <para>
 473         So a good error message should provide all the information which
 474         cannot be obtained by reading the source, typically the function
 475         return value, error codes, and any function output parameter. Even if
 476         more information is needed to fully understand a problem,
 477         systematically providing the above is easy and will help cut down the
 478         number of iterations required to get to a resolution.
 479       </para>
 480       <para>
 481         It may also be a good idea to dump items that may be hard to retrieve
 482         from the source, like the expected value in a test if it is the
 483         result of an earlier computation, or comes from a large array of test
 484         values (e.g. index 112 of _pTestStrA in vartest.c). In that respect,
 485         for some tests you may want to define a macro such as the following:
 486 <screen>
 487 #define eq(received, expected, label, type) \
 488         ok((received) == (expected), "%s: got " type " instead of " type, (label),(received),(expected))
 489
 490 ...
 491
 492     eq( b, curr_val, "SPI_{GET,SET}BEEP", "%d" );
 493 </screen>
 494        </para>
 495     </sect1>
 496
 497
 498     <sect1 id="testing-platforms">
 499       <title>Handling platform issues</title>
 500       <para>
 501         Some checks may be written before they pass successfully in Wine.
 502         Without some mechanism, such checks would potentially generate
 503         hundred of known failures for months each time the tests are being run.
 504         This would make it hard to detect new failures caused by a regression.
 505         or to detect that a patch fixed a long standing issue.
 506       </para>
 507       <para>
 508         Thus the Wine testing framework has the concept of platforms and
 509         groups of checks can be declared as expected to fail on some of them.
 510         In the most common case, one would declare a group of tests as
 511         expected to fail in Wine. To do so, use the following construct:
 512 <screen>
 513 todo_wine {
 514     SetLastError( 0xdeadbeef );
 515     ok( GlobalAddAtomA(0) == 0 && GetLastError() == 0xdeadbeef, "failed to add atom 0" );
 516 }
 517 </screen>
 518         On Windows the above check would be performed normally, but on Wine it
 519         would be expected to fail, and not cause the failure of the whole
 520         test. However. If that check were to succeed in Wine, it would
 521         cause the test to fail, thus making it easy to detect when something
 522         has changed that fixes a bug. Also note that todo checks are accounted
 523         separately from regular checks so that the testing statistics remain
 524         meaningful. Finally, note that todo sections can be nested so that if
 525         a test only fails on the cygwin and reactos platforms, one would
 526         write:
 527 <screen>
 528 todo("cygwin") {
 529     todo("reactos") {
 530         ...
 531     }
 532 }
 533 </screen>
 534         <!-- FIXME: Would we really have platforms such as reactos, cygwin, freebsd & co? -->
 535         But specific platforms should not be nested inside a todo_wine section
 536         since that would be redundant.
 537       </para>
 538       <para>
 539         When writing tests you will also encounter differences between Windows
 540         9x and Windows NT platforms. Such differences should be treated
 541         differently from the platform issues mentioned above. In particular
 542         you should remember that the goal of Wine is not to be a clone of any
 543         specific Windows version but to run Windows applications on Unix.
 544       </para>
 545       <para>
 546         So, if an API returns a different error code on Windows 9x and
 547         Windows NT, your check should just verify that Wine returns one or
 548         the other:
 549 <screen>
 550 ok ( GetLastError() == WIN9X_ERROR || GetLastError() == NT_ERROR, ...);
 551 </screen>
 552       </para>
 553       <para>
 554         If an API is only present on some Windows platforms, then use
 555         LoadLibrary and GetProcAddress to check if it is implemented and
 556         invoke it. Remember, tests must run on all Windows platforms.
 557         Similarly, conformance tests should nor try to correlate the Windows
 558         version returned by GetVersion with whether given APIs are
 559         implemented or not. Again, the goal of Wine is to run Windows
 560         applications (which do not do such checks), and not be a clone of a
 561         specific Windows version.
 562       </para>
 563       <!--para>
 564         FIXME: What about checks that cause the process to crash due to a bug?
 565       </para-->
 566     </sect1>
 567
 568
 569 <!-- FIXME: Strategies for testing threads, testing network stuff,
 570  file handling, eq macro... -->
 571
 572   </chapter>
 573
 574 <!-- Keep this comment at the end of the file
 575 Local variables:
 576 mode: sgml
 577 sgml-parent-document:("wine-devel.sgml" "set" "book" "part" "chapter" "")
 578 End:
 579 -->