WvTest: the dumbest cross-platform test framework that could possibly work
==========================================================================
I have a problem with your unit testing framework. Yes, you.
The person reading this. No, don't look away guiltily. You
know your unit testing framework sucks. You know it has a
million features you don't understand. You know you hate it,
and it hates you. Don't you?
Okay, fine. Let's be honest. Actually, I don't know who you
are or how you feel about your unit testing framework, but I've
tried a lot of them, and I don't like any of them. WvTest is
the first one I don't hate, at least sort of. That might be
because I'm crazy and I only like things I design, or it might
be because I'm crazy and therefore I'm the only one capable of
designing a likable unit testing framework. Who am I to say?
Here are the fundamental design goals of WvTest:
- Be the stupidest thing that can possibly work. People are
way, way too serious about their testing frameworks. Some
people build testing frameworks as their *full time job*.
This is ridiculous. A test framework, at its core, only does
one thing: it runs a program that returns true or false. If
it's false, you lose. If it's true, you win. Everything
after that is gravy. And WvTest has only a minimal amount of
gravy.
- Be a protocol, not an API. If you don't like my API, you can
write your own, and it can still be WvTest and it can still
integrate with other WvTest tools. If you're stuck with
JUnit or NUnit, you can just make your JUnit/NUnit test
produce WvTest-compatible output if you want (although I've
never done this, so you'll have to do it yourself). I'll
describe the protocol below.
- Work with multiple languages on multiple operating systems.
I'm a programmer who programs on Linux, MacOS, and Windows,
to name just three, and I write in lots of programming
languages, including C, C++, C#, Python, Perl, and others.
And worse, some of my projects use *multiple* languages and I
want to have unit tests for *all* of them. I don't know of
any unit testing framework - except maybe some horrendously
overdesigned ones - that work with multiple languages at
once. WvTest does.
- NO UNNECESSARY OBJECT ORIENTATION. The big unit testing
craze seems to have been started by JUnit in Java, which is
object-oriented. Now, that's not a misdesign in JUnit; it's
a misdesign in Java. You see, you can't *not* encapsulate
absolutely everything in Java in a class, so it's perfectly
normal for JUnit to require you to encapsulate everything in
a class. That's not true of almost any other language
(except C#), and yet *every* clone of JUnit in *every*
language seems to have copied its classes and objects. Well,
that's stupid. WvTest is designed around the simple idea of
test *functions*. WvTest runs your function, it checks a
bunch of stuff and it returns or else it dies horribly. If
your function wants to instantiate some objects while it does
that, then that's great; WvTest doesn't care. And yes, you
can assert whether two variables are equal even if your
function *isn't* in a particular class, just as God intended.
- Don't make me name or describe my individual tests. How many
times have you seen this?
assertTrue(thing.works(), "thing didn't work!");
The reasoning there is that if the test fails, we want to be
able to print a user-friendly error message that describes
why. Right? NO!! That is *awful*. That just *doubled* the
amount of work you have to do in order to write a test.
Instead, WvTest auto-generates output including the line
number of the test and the code on that line. So you get a
message like this:
! mytest.t.cc:431 thing.works() FAILED
and all you have to write is this:
WVPASS(thing.works());
(WVPASS is all-caps because it's a macro in C++, but also
because you want your tests to stand out. That's what
you'll be looking for when it fails, after all. And don't
even get me started about the 'True' in assertTrue. Come
on, *obviously* you're going to assert that the condition is
true!)
- No setup() and teardown() functions or fixtures. "Ouch!" you
say. "I'm going to have so much duplicated code!" No, only
if you're an idiot. You know what setup() and teardown() are
code names for? Constructor and destructor. Create some
objects and give them constructors and destructors, and I
think you'll find that, like magic, you've just invented
"test fixtures." Nothing any test framework can possibly do
will make that any easier. In fact, everything test
frameworks *try* to do with test fixtures just makes it
harder to write, read, and understand. Forget it.
- Big long scary test functions. Some test frameworks are
insistent about the rule that "every function should test
only one thing." Nobody ever really explains why. I can't
understand this; it just causes uncontrolled
hormone-imbalance hypergrowth in your test files, and you
have to type more stuff... and run test fixtures over and
over.
My personal theory for why people hate big long test
functions: it's because their assertTrue() implementation
doesn't say which test failed, so they'd like the *name of
the function* to be the name of the failed test. Well,
that's a cute workaround to a problem you shouldn't have had
in the first place. With WvTest, WVPASS() actually tells you
exactly what passed and what failed, so it's perfectly okay -
and totally comprehensible - to have a sequence of five
things in a row where only thing number five failed.
The WvTest Protocol
-------------------
WvTest is a protocol, not really an API. As it happens, the
WvTest project includes several (currently five)
implementations of APIs that produce data in the WvTest format,
but it's super easy to add your own.
The format is really simple too. It looks like this:
Testing "my test function" in mytest.t.cc:
! mytest.t.cc:432 thing.works() ok
This is just some crap that I printed while counting to 3.
! mytest.t.cc.433 3 < 4 FAILED
There are only four kinds of lines in WvTest, and each of the
lines above corresponds to one of them:
- Test function header. A line that starts with the word
Testing (no leading whitespace) and then has a test function
name in double quotes, then "in", then the filename, and then
colon, marks the beginning of a test function.
- A passing assertion. Any line that starts with ! and ends with
" ok" (whitespace, the word "ok", and a newline) indicates
one assertion that passed. The first "word" on that line is
the "name" of that assertion (which can be anything, as long
as it doesn't contain any whitespace). Everything between the
name and the ok is just some additional user-readable detail
about the test that passed.
- Random filler. If it doesn't start with an ! and it doesn't
look like a header, then it's completely ignored by anything
using WvTest. Your program can print all the debug output it
wants, and WvTest won't care, except that you can retrieve it
later in case you're wondering why a test failed. Naturally,
random filler *before* an assertion is considered to be
associated with that assertion; the assertion itself is the
last part of a test.
- A failing assertion. This is just like an 'ok' line, except
augmented with extra detail. A more advanced parser could choose to
parse the extra string to count partial failures:
xfail ok - test was marked as known to fail and failed
(i.e. a known breakage)
xpass ok - test was marked as known to fail and passed
(i.e. previously known breakage is fixed)
skip ok - test was skipped.
and it could be something else instead, if you invent a new and improved way
to fail.
Reading the WvTest Protocol: wvtestrun
--------------------------------------
WvTest provides a simple perl script called wvtestrun, which
runs a test program and parses its output. It works like this:
cd python
../wvtestrun ./wvtest.py t/twvtest.py
(Why can't we just pipe the output to wvtestrun, instead of
having wvtestrun run the test program? Three reasons: first, a
fancier version of wvtestrun could re-run the tests several
times or give a GUI that lets you re-run the test when you push
a button. Second, it handles stdout and stderr separately.
And third, it can kill the test program if it gets stuck
without producing test output for too long.)
If we put the sample output from the previous section through
wvtestrun (and changed the FAILED to ok), it would produce this:
$ ./wvtestrun cat sample-ok
Testing "all" in cat sample-ok:
! mytest.t.cc my ok test function: ..... 0.010s ok
WvTest: 5 tests, 0 failures, total time 0.010s.
WvTest: 0 tests skipped, 0 known breakages, 0 fixed breakages.
WvTest result code: 0
What happened here? Well, wvtestrun took each test header (in
this case, there's just one, which said we're testing "my test
function" in mytest.t.cc) and turns it into a single test line.
Then it prints a dot for each assertion in that test function,
tells you the total time to run that function, and prints 'ok'
if the entire test function failed.
Note that the output of wvtestrun is *also* valid WvTest output.
That means you can use wvtestrun in your 'make test' target in a
subdirectory, and still use wvtestrun as the 'make test' runner
in the parent directory as well. As long as your top-level
'make test' runs in wvtestrun, all the WvTest output will be
conveniently summarized into a *single* test output.
Now, what if the test had failed? Then it would look like this:
$ ./wvtestrun cat sample-error
Testing "all" in cat sample-error:
! mytest.t.cc my error test function: .
! mytest.t.cc:432 thing.works() ok
This is just some crap that I printed while counting to 3.
! mytest.t.cc.433 3 < 4 FAILED
fXs 0.000s ok
WvTest: 5 tests, 1 failure, total time 0.000s.
WvTest: 1 test skipped, 1 known breakage, 1 fixed breakage.
WvTest result code: 0
What happened there? Well, because there were failed tests,
wvtestrun decided you'd probably want to see the detailed output
for that test function, so it expanded it out for you. The line
with the dots is still there, but since it doesn't have an 'ok',
it's considered a failure too, just in case.
Watch what happens if we run a test with both the passing, and
then the failing, test functions:
$ ./wvtestrun cat sample-ok sample-error
Testing "all" in cat sample-ok sample-error:
! mytest.t.cc my ok test function: ..... 0.000s ok
! mytest.t.cc my error test function: .
! mytest.t.cc:432 thing.works() ok
This is just some crap that I printed while counting to 3.
! mytest.t.cc.433 3 < 4 FAILED
fXs 0.000s ok
WvTest: 10 tests, 1 failure, total time 0.000s.
WvTest: 1 test skipped, 1 known breakage, 1 fixed breakage.
WvTest result code: 0
Notice how the messages from sample-ok are condensed; only the
details from sample-error are expanded out, because only that
output is interesting.
How do I actually write WvTest tests?
-------------------------------------
Sample code is provided for these languages:
C: try typing "cd c; make test"
C++: try typing "cd cpp; make test"
C# (mono): try typing "cd dotnet; make test"
Python: try typing "cd python; make test"
Shell: try typing "cd sh; make test"
There's no point explaining the syntax here, because it's really
simple. Just look inside the cpp, dotnet, python, and sh
directories to learn how the tests are written.
How should I embed WvTest into my own program?
----------------------------------------------
The easiest way is to just copy the WvTest source files for your
favourite language into your project. The WvTest protocol is
unlikely to ever change - at least not in a
backwards-incompatible way - so it's no big deal if you end up
using an "old" version of WvTest in your program. It should
still work with updated versions of wvtestrun (or wvtestrun-like
programs).
Another way is to put the WvTest project in a subdirectory of
your project, for example, using 'svn:externals',
'git submodule', or 'git subtree'.
How do I run just certain tests?
--------------------------------
Unfortunately, the command-line syntax for running just *some*
of your tests varies depending which WvTest language you're using.
For C, C++ or C#, you link an executable with wvtestmain.c or
wvtestmain.cc or wvtestmain.cs, respectively, and then you can
provide strings on the command line. Test functions will run only
if they have names that start with one of the provided strings:
cd cpp/t
../../wvtestrun ./wvtest myfunc otherfunc
With python, since there's no linker, you have to just tell it
which files to run:
cd python
../wvtestrun ./wvtest.py ...filenames...
What else can parse WvTest output?
----------------------------------
It's easy to parse WvTest output however you like; for example,
you could write a GUI program that does it. We had a tcl/tk
program that did it once, but we threw it away since the
command-line wvtestrun is better anyway.
One other program that can parse WvTest output is gitbuilder
(http://github.com/apenwarr/gitbuilder/), an autobuilder tool
for git. It reports a build failure automatically if there are
any WvTest-style failed tests in the build output.
Other Assorted Questions
------------------------
What does the "Wv" stand for?
Either "Worldvisions" or "Weaver", both of which were part of the
name of the Nitix operating system before it was called Nitix, and
*long* before it was later purchased by IBM and renamed to Lotus
Foundations.
It does *not* stand for World Vision (sigh) or West Virginia.
Who owns the copyright?
While I (Avery) wrote most of the WvTest framework in C++, C#, and
Python, and I also wrote wvtestrunner, the actual code I wrote is
owned by whichever company I wrote it for at the time. For the most
part, this means:
C++: Net Integration Technologies, Inc. (now part of IBM)
C#: Versabanq Innovations Inc.
Python: EQL Data Inc.
What can I do with it?
WvTest is distributed under the terms of the GNU LGPLv2. See the
file LICENSE for more information.
Basically this means you can use it for whatever you want, but if
you change it, you probably need to give your changes back to the
world. If you *use* it in your program (which is presumably a test
program) you do *not* have to give out your program, only
WvTest itself. But read the LICENSE in detail to be sure.
Where did you get the awesome idea to use a protocol instead of an API?
The perl source code (not to be confused with perlunit)
did a similar trick for the perl interpreter's unit
test, although in a less general way. Naturally, you
shouldn't blame them for how I mangled their ideas, but
I never would have thought of it if it weren't for them.
Why are xfail/xpass/skip needed at all?
Suppose you have a test which you expect to pass, but which is
failing (in the usual sense). You have two choices: 1) fix it right
now, or 2) defer fixing. Sometimes at the start we have lots of
tests failing, and in order to make gradual progress, it makes sense
to mark those presently-failing tests as "I know, it fails".
Another use case is when tests always pass on e.g. Linux, but some
of them fail on Win32 due to differences in environment, and one
does not want to concentrate on fixing win32 yet.
So when you run tests again, you'd like to differentiate between
failing tests marked as xfail (known to fail) and new failing tests.
The latter should produce real FAILURE with details. xfail on the
other hand should produce just a small warning/reminder (I'm failing
here, please don't forget to fix me).
That's the idea.
So now what happens when a test marked as xfail passes anyway
instead of the expected failure? That's not a failure - quite
differently, it _was_ a failure, and now it passes, so what should
we do? Right, we should tell the user that "hey, a test which used
to fail now passes! You probably would want to mark it back as
PASS".
This is how xfail/xpass works.
With introduction of xfail tests are no longer strictly true or
false, but you can ignore the additional information if you want
(the last word is still either "ok" or not).
Who should I complain to about WvTest?
Email me at: Avery Pennarun <apenwarr@gmail.com>
I will be happy to read your complaints, because I actually really
like it when people use my programs, especially if they hate them.
It fills the loneliness somehow and prevents me from writing bad
poetry like this:
Testing makes me gouge out my eyes
But with WvTest, it takes fewer tries.
WvTest is great, wvtest is fun!
Don't forget to call wvtestrun.
