2006-12-06 Chris Toshok <toshok@ximian.com>

[mono-project.git] / web / testing

* Testing

        Daily <a href="http://www.go-mono.com/tests/index.php">test</a> results.

        Testing is an important part of the Mono project: every one of its
        three major components has a test suite tailored for its needs.  This
        is very helpful, because in the course of developing the software it
        is very common to introduce bugs in existing code.  A test suite
        helps us fix the bugs as soon as they are introduced.

        There are various kinds of tests in Mono:
        <ul>
                <li><a href="#unit"><b>Class Library Unit
                Tests:</b></a> These are used to test the class
                libraries.

                <li><a href="#compiler"><b>Compiler tests</b></a>: Both
                tests that should pass and tests that should fail are included. 

                <li><a href="#runtime"><b>Runtime tests</b></a>: Tests for 
                the virtual machine.

                <li><a href="#aspnet"><b>ASP.NET tests</b></a>: ASP.NET tests.

                <li><a href="#ws"><b>Web Services tests</b></a>: Web Services 
                client/server tests.
        </ul>

<a name="unit"></a>
* Class Library Tests

        All classes in Mono libraries should have comprehensive unit test
        suites to go with them.  Unit testing is a software engineering
        methodology that makes it easier to build correct code.  Every
        method in every class should have a set of tests to verify
        that they work correctly.  Mono also needs a testing framework
        to make it easy to write and run lots of tests. 

        In some classes, we might also provide standalone tests because of
        some reasons such as too huge testcases, another downloading and so on.
        (For example, managed XSLT has standalone test which downloads and
        expands some megabytes of OASIS test suite.)

        Here I list them up as long as I know. If you are going to add another
        standalone tests, please add one line here. It is also recommended that
        you add some notes on how to build and run tests.

        <ul>

                * Mono.Data/test/
                * System.Data/Test, and some individual ADO.NET libraries:
                  there are some standalone tests. See the bottom of <a href="ado-net.html">
                  ADO.NET page</a> for detail.
                * System.Web/Test/TestMonoWeb : see README
                * System.Web.Services/Test/standalone : see README
                * System.Windows.Forms/SWFTest/
                * System.XML/Test/System.Xml/standalone_tests : see README
                * System.XML/Test/System.Xml.Schema/standalone_tests : see README
                * System.XML/System.Xml.Serialization/standalone_tests/
                * System.XML/Test/System.Xml.Xsl/standalone_tests : see README
                * Commons.Xml.Relaxng/Test/standalone_tests : see README

        </ul>

** Getting started

        If you are new to writing NUnit tests, there is a template you may use
        to help get started. The file is:

                <b>mcs/class/doc/TemplateTest.cs</b>

        Save a copy of this file in the appropriate test subdirecty
        (see below), and replace all the {text} markers with
        appropriate code. Comments in the template are there to guide
        you. You should also look at existing tests to see how other
        people have written them.
        mcs/class/corlib/Test/System.Collections/CollectionBaseTest.cs
        is a small one that might help.

        The directory that will contain your new file depends on the
        assembly/namespace of the class for which you are creating the
        tests.  Under mcs/class there is a directory for each
        assembly. In each assembly there is a Test directory,
        e.g. mcs/class/corlib/Test. In the Test directory there are
        sub-directories for each namespace in the assembly,
        e.g. mcs/class/corlib/Test/Sytem. Put your new test file in
        the appropriate sub-directory under Test for the class you are
        testing.
        
        Once all of that is done, you can do a 'make test' from the top mcs
        directory.  Your test class needs also to be listed in the
        .sources file at the top of the Test directory.

* Tips on writing Unit tests.

        You should look at the <a href="http://nunit.org">NUnit documentation</a>,
        as it is a fantastic product, and includes fantastic documentation,
        but here are some tips for those of you who are already reading
        this web page.


** Provide an unique error message for Assert()

        Include an unique message for each Assert() so that when the assert
        fails, it is trivial to locate it in the source. Otherwise, it may be
        difficult to determine which part of the test is failing. A good way
        to ensure unique messages is to use something like #A01, #A02 etc.

            Ok:
        <pre>
        
                AssertEquals("array match", compare[0], i1[0]);
                AssertEquals("array match", compare[1], i1[1]);
                AssertEquals("array match", compare[2], i1[2]);
                AssertEquals("array match", compare[3], i1[3]);
        </pre>

            Excellent:
        <pre>
                AssertEquals("#A01", compare[0], i1[0]);
                AssertEquals("#A02", compare[1], i1[1]);
                AssertEquals("#A03", compare[2], i1[2]);
                AssertEquals("#A04", compare[3], i1[3]);
        </pre>
        
        Once you used such a number in an Assert(), don't change it later on -
        people might use it it identify the test in bug reports or in mailing
        lists.

** Use AssertEquals() to compare things, not Assert().

        Do not compare two values with Assert() - if the test fails,
        people have no idea what went wrong while AssertEquals()
        reports the failed value.

        Ok:
        <pre>
                Assert ("A01", myTicks[0] == t1.Ticks);
        </pre>

        Excellent:
        <pre>
                AssertEquals ("A01", myTicks[0], t1.Ticks);
        </pre>

** Test your test with the Microsoft runtime
        
        If possible, try to run your testsuite with the Microsoft runtime on
        .NET on Windows and make sure all tests in it pass. This is especially
        important if you're writing a totally new testcase - without this
        check you can never be sure that your testcase contains no bugs ....
        
        Don't worry if you're writing your test on Linux, other people can
        test it for you on Windows.
        
        Sometimes you may discover that a test doesn't show the expected
        result when run with the Microsoft runtime - either because there is a
        bug in their runtime or something is misleading or wrong in their
        documentation. In this case, please put a detailed description of the
        problem to mcs/class/doc/API-notes and do also report it to the 
        <a href="mailing-lists.html">mailing list</a> - we'll forward this to the
        Microsoft people from time to time to help them fix their documentation
        and runtime.

** Unit tests.

        Why do unit testing? It becomes simple to run automated tests
        for the whole library. Unit tests are a safety net - you can
        change part of the code and verify that you haven't broken
        anything. Ideally, tests are written before the actual library
        code itself. And every time a bug is discovered, a test should
        be written to demonstrate the bug and its fix. Then, if
        you ever reintroduce the bug, you will know immediately. For
        more info, read <a
        href="http://junit.sourceforge.net/doc/testinfected/testing.htm">
        JUnit Test Infected: Programmers Love Writing Tests</a>.


** Getting Started

        We welcome all contributions to the Class Libary Test Suite.

        There is information to help you get started in CVS at
        mcs/class/doc/NUnitGuidelines. Once you have written your test, please
        post it to <a href="mailing-lists.html">mono-list</a>.

        Someone will make sure to add the file or apply the patch as
        appropriate. If you plan to be an on-going contributor and
        would like to get cvs account, email <a href="mailto:miguel@ximian.com">miguel</a>.

        Normally, after you send a couple of well-written new files
        and/or patches to the list, you will be given cvs access.

<a name="compiler"></a>
* Compiler tests

        Mono ships with three compilers: C#, VB.NET and JScript.  The
        tests are ran by running the makefile target `make
        run-test-local' in the appropriate directory.

        The C# compilation tests live in mcs/tests, and the C# error
        tests live in mcs/errors.

        The VB.NET compilation tests live in mcs/btests. 

<a name="runtime"></a>
* Runtime Tests

        These tests verify the virtual machine, to run these tests, do:

<pre>
        cd mono/mono/tests
        make test
</pre>

<a name="aspnet"></a>
* ASP.NET tests

        XSP, the Mono ASP.NET server has tests for ASP.NET pages. It uses
        <a href="http://nunitasp.sourceforge.net">NUnitAsp</a>. Right now
        it only has standalone tests, ie., tests that do not need their own
        global.asax or web.config files.

        If you want to run them, get the xsp CVS module and install it. Then:
<pre>
        cd xsp/nunit-tests
        make
        cd standalone
        xsp
</pre>

        And from another terminal:
<pre>
        cd xsp/nunit-tests/standalone
        nunit-console standalone-tests.dll
</pre>

<a name="ws"></a>
* Web Services tests

        The Test directory for the System.Web.Services assembly contains a
        standalone test suite for testing web services. It tests:

        <ul>
        <li>Proxy generation using the wsdl tool</li>
        <li>Access to web services using the generated client proxies</li>
        <li>Execution of web services in the server</li>
        </ul>

        This suite not only tests web services running on XSP, but it can also test
        services running on other platforms and that are available in internet. This
        will help track down interoperability issues.

        To build the test suite, just run:
        
<pre>
        cd mcs/class/System.Web.Services/Test/standalone
        xsp --root server
</pre>
        
        And from another terminal:
<pre>
        cd mcs/class/System.Web.Services/Test/standalone
        make
        nunit-console testclient.dll
</pre>
        
        This will download the wsdl documents, generate the proxies, build a dll with
        the proxies, and build the nunit tests. Then you can use nunit-console or
        gnunit to run the tests (the nunit dll is testclient.dll).

        Read the README file in mcs/class/System.Web.Services/Test/standalone for
        more info.