3 I tried to be clever and abstract out the do-testing part from the run-tests
4 and run-test. The trouble is that this abstraction isn't right. I end up
5 needing to wrap error catching at two levels (one at the outer level to handle
6 when multiple tests are happening and one on the inner run-test-internal
7 level). The solution is probably a big refactoring that better models the
8 multi-test and single-test cases individually.
10 it's a failure when :expected-error doesn't error or :expected-problem
13 nicer reporting; espec for ensure-cases
15 turtle -> ntriples -> triple-store -> sparql query -> graphs
18 expect-failure := string | t
19 evaluate-when-defined := t | nil
23 use command line arguments to specify suite, etc..
25 be nice to share message or special formmatting for ensures at
28 dynamic-variable code gets added at every level of the hierarchy
29 only need to add at for direct dynamic variables, not all
30 - need direct-d-v i... actually, no. The dv are set where?
31 I think everyone needs the code, look later
33 Where do options for a test case go? We could just add code to the
34 test case method but this would make it hard to introspect.
35 Could define a method on suite and case... or a regular ht.
37 Do a simple walk on the code to know which "line" of code failed so
38 that you can report that. What about (let ...), we'd want to walk into
39 that. Special case? (let, let*, bind, mvb, db, progX, loop, dotimes, do,
40 hm, quite a few (and I'm sure many more).
42 Introspect the functions / forms a test tests
44 ensure-random-cases (etc) don't really play well with ensure-condition
45 you get to know that the test failed but not how...
47 Lift needs an extra group of parentheses in :tests: e.g.:
48 (deftestsuite twinql.section-data-sort-4 (twinql.section-data-sort)
52 (ensure (null (filter-results))))))
55 compile: lift doesn't catch errors during compile (e.g., symbol x is not
58 report: add consing and times to ouptu (optional)
60 :run-setup :once-per-session
61 (give not implemented error at test def time, not run time
64 need tests that metadata is correctly set too
66 need to test that all variables in a hierarchy are initialized as appropriate
68 We're not checking for class existence in addtest...
69 leads to weird seeming errors
71 LIFT: handle macros better
72 LIFT: code walker -- even a simple one that just tree searched for macros
75 (list lift::*test-is-being-compiled?*
76 lift::*test-is-being-defined?*
77 lift::*test-is-being-executed?*
78 lift::*test-is-being-loaded?*)
80 (setf lift::*test-is-being-compiled?* nil
81 lift::*test-is-being-defined?* nil
82 lift::*test-is-being-executed?* nil
83 lift::*test-is-being-loaded?* nil)
85 may need to change teardownn-test to standard method combination and
87 (defmethod teardown-test :before ((test-suite test-mixin))
88 (setf (current-step test-suite) 'teardown-test))
90 randomly change order of tests
92 (deftestsuite test-addition ()
94 (:cases (:map (a '(1 2 3 4 5))
96 (:test ((ensure-same (+ a b) 10 :test '=))))
98 (deftestsuite test-addition ()
100 (:cases (a '(1 2 3 4 5))
102 (:test ((ensure-same (+ a b) (+ b a) :test '=))))
104 ok 1. Handle simple cases clause -> initialize-prototypes
105 ok 2. Handle run-time evaluation / creation of prototypes
107 ok 3. Look at quickcheck and think a bit
108 4. Determine what a paper on this might look like.
110 make-test-result seems a bit silly too
113 (defun testing-interactively-p ()
116 (and *test-is-being-defined?*
117 (not (or *test-is-being-compiled?*
118 *test-is-being-loaded?*))))
120 test-suite-p, etc could be defined more cleanly if the class
123 Instead of class properties, use the MOP
127 If you define a test suite and a test and then redefine the test-suite,
128 and then try to (run-test), you'll get an error. Should give a nice message
130 ~ need set of lift meta tests and set of lift "UI" tests
133 pending and run-pending method
136 finding suites and tests
138 sharing setup / teardown
141 ~ - non-deterministic
142 e.g., checking for a socket, expect failure occasionally
147 donÕt continue when failure option
149 cache previous results and only rerun failed tests
153 non-determininistic tests
155 [doc] ensure's should have a documentation clause
157 simplify test-suite-p
159 need a better error message for when a superclass that DNE or is not a test-class
162 ?should use same tricks as define-debugging-class to mangle the name
164 shouldn't teardown-test be an :after
166 when using break-on-errors? you should have restarts to allow to cancel out
167 completely, or keep going with the next test or whatever
169 when defining a test with addtest, if no current test case, or if named
170 case doesnt't exist, offer to create it (use a restart or whatever like in debug).
172 remove-test should print a nicer message if the test you're trying to undef DNE
174 remove-test needs to clean up data tables too
176 with-test should take optional test-class
178 clean up all the tables and stuff
180 with-test needs error handling
182 add documentation (at the least to the public interface)
184 Capture errors and have various fun restarts
186 error check addtest if class dne or not test-mixin
188 Improve error messages
191 (deftestsuite test-a ()
195 (defmethod initialize-prototypes :after ((test test-plus))
196 (with-test-slots (a b)
197 (setf (prototypes test)
204 (defmethod initialize-prototypes :after ((test test-plus))
205 (with-test-slots (a b)
206 (setf (prototypes test)
213 (defmethod initialize-prototypes :after ((test test-plus))
214 (with-test-slots (a b)
215 (setf (prototypes test)
218 (list (cons 'a a) (cons 'b b)))))))
220 (deftestsuite test-a ()
223 (deftestsuite test-a ()
225 (:cases (a '((1 2)))))
230 (deftestsuite test-b (test-a)
232 (:cases (b '(4 5 6))))
235 (format t "~% ~A x ~A" a b))
238 (deftestsuite test-e ()
241 (defmethod initialize-prototypes :after ((test test-e))
242 (setf (prototypes test)
245 (list (cons 'a 2)))))
250 (deftestsuite test-f (test-e)
253 (defmethod initialize-prototypes :after ((test test-f))
254 (setf (prototypes test)
258 (list (cons 'b 6)))))
261 (format t "~% ~A x ~A" a b))
273 (defgeneric setup (suite)
274 (:documentation "Setup at the testsuite-level")
275 (:method ((suite test-mixin))
278 (defgeneric testsuite-teardown (suite)
279 (:documentation "Cleanup at the testsuite level.")
280 (:method ((suite test-mixin))
283 (defgeneric testsuite-run (suite result)
284 (:documentation "Run the cases in this suite and it's children."))
286 (defgeneric setup-test (test-case)
287 (:documentation "Setup for a test-case. By default it does nothing."))
289 (defgeneric teardown-test (test-case)
290 (:documentation "Tear-down a test-case. By default it does nothing.")
291 (:method-combination progn :most-specific-first))
293 (defgeneric testsuite-methods (test-case)
294 (:documentation "Returns a list of the test methods defined for test. I.e.,
295 the methods that should be run to do the tests for this test."))
317 ;; here is better but will need to save off more information for results
318 ;; but this fails for child tests
329 ;; so it needs to be here
332 run each method of this suite using run-test-internal
339 push method name on tests-run of the result
352 test teardown plus remove variables
357 ;;; ---------------------------------------------------------------------------
359 Simplest case may be to give test-mixin a slot to hold initial values.
360 For 'regular' tests, this is what it is. For more complex tests, the values
361 are lists (need to differentiate between lists and multiple prototypes).
363 Then all tests work the same way.
365 The questions for the protocol are standard iteration quesionts: what is
366 the next value? and are we done yet?
370 ok - Probably don't need to define accessors or initargs since we're using test-environment
371 ok - I don't think we need *test-result* -- it's not used
372 I think it's a holdover from JUnit
373 ok - use test-code table to prevent some duplication (or require names)
374 ok - when name supplied, don't add "test-" to it
375 ok - get rid of print-test-result {s} and make better use of print-object and describe-object
376 ok - When adding multiple tests, need to store their names immediately (i.e.,
377 before going on to parse the next test).
378 obsolete - Improve error/warning for duplicate code/names
379 ok - run-tests works OK; run-test semi; evaluating an addtest does not
380 ok - need to store more in the test results to see what values were used, etc.
381 ok - Screw run-test being a macro
382 ok - Map case keywords to functions (think of tapir). Each of :map, :cross, etc.
383 is handled separately. This will make it easy to extend and build big monsters
384 Only "hard" part is correctly combining the conses.
385 ok - This prototype stuff at compile time is hard... maybe it would be easier to do it
386 at run time (using stack / CPL to help...)
387 ok - The problem with not using slots of the test class is that I need to duplicate
388 a bunch of functionality like inherited slots to make with-test-slots work
389 ok - I don't think I'm using *test-do-children?*, etc. correctly either
390 ok - Note: LIFT does not use the slots of the instance of the test-class; all looks
391 are via test-environment-value
392 (I no longer remember why this was a good idea)
393 (maybe don't even need the slots... just the defs)
394 ok - Get (deftestsuite test-addition ()
396 (:cases (:map (a '(1 2 3 4 5))
398 (:test ((ensure-same (+ a b) 10 :test '=))))
399 To work... Standardize-Cases-Form is broken
400 ok - parse-test-name-and-class is probably wacked b/c I changed to *current-suite-class-name*
402 ok - Use multiple processes to handle run away tests
403 get reporting to work correctly
404 ok - the whole number of tests thing is stupid and wacky
405 something wrong with loading or compiling addtest, the numbers don't get
406 incremented properly.
407 testsuite-test-count, number-of-tests
408 change to getting count at start of deftestsuite and then add 1 to a special
410 no - use dynamic scope for *test-is-being-compiled?* rather than unwind-protect
411 ok - LIFT: better control of printing (:follow-print)
412 ok - LIFT: variable to control behavior on warning / failure: print full report, etc.
413 ok - add :function to define local functions, use rewriting and macrolet
414 yes - so how do we integrate measuring with the report -- change the test result
415 add test result properties