3 it's a failure wen :expected-error doesn't error or :expected-problem
6 nicer reporting; espec for ensure-cases
8 turtle -> ntriples -> triple-store -> sparql quer -> graps
12 expect-failure := string | t
13 evaluate-when-defined := t | nil
17 use command line arguments to specify suite, etc..
19 be nice to share message or special formmatting for ensures at
22 dynamic-variable code gets added at every level of the hierarchy
23 only need to add at for direct dynamic variables, not all
24 - need direct-d-v i... actually, no. The dv are set where?
25 I think everyone needs the code, look later
27 Where do options for a test case go? We could just add code to the
28 test case method but this would make it hard to introspect.
29 Could define a method on suite and case... or a regular ht.
31 Do a simple walk on the code to know which "line" of code failed so
32 that you can report that. What about (let ...), we'd want to walk into
33 that. Special case? (let, let*, bind, mvb, db, progX, loop, dotimes, do,
34 hm, quite a few (and I'm sure many more).
36 Introspect the functions / forms a test tests
38 ensure-random-cases (etc) don't really play well with ensure-condition
39 you get to know that the test failed but not how...
41 Lift needs an extra group of parentheses in :tests: e.g.:
42 (deftestsuite twinql.section-data-sort-4 (twinql.section-data-sort)
46 (ensure (null (filter-results))))))
49 compile: lift doesn't catch errors during compile (e.g., symbol x is not
52 report: add consing and times to ouptu (optional)
54 :run-setup :once-per-session
55 (give not implemented error at test def time, not run time
58 need tests that metadata is correctly set too
60 need to test that all variables in a hierarchy are initialized as appropriate
62 We're not checking for class existence in addtest...
63 leads to weird seeming errors
65 LIFT: handle macros better
66 LIFT: code walker -- even a simple one that just tree searched for macros
69 (list lift::*test-is-being-compiled?*
70 lift::*test-is-being-defined?*
71 lift::*test-is-being-executed?*
72 lift::*test-is-being-loaded?*)
74 (setf lift::*test-is-being-compiled?* nil
75 lift::*test-is-being-defined?* nil
76 lift::*test-is-being-executed?* nil
77 lift::*test-is-being-loaded?* nil)
79 may need to change teardownn-test to standard method combination and
81 (defmethod teardown-test :before ((test-suite test-mixin))
82 (setf (current-step test-suite) 'teardown-test))
84 randomly change order of tests
86 (deftestsuite test-addition ()
88 (:cases (:map (a '(1 2 3 4 5))
90 (:test ((ensure-same (+ a b) 10 :test '=))))
92 (deftestsuite test-addition ()
94 (:cases (a '(1 2 3 4 5))
96 (:test ((ensure-same (+ a b) (+ b a) :test '=))))
98 ok 1. Handle simple cases clause -> initialize-prototypes
99 ok 2. Handle run-time evaluation / creation of prototypes
101 ok 3. Look at quickcheck and think a bit
102 4. Determine what a paper on this might look like.
104 make-test-result seems a bit silly too
107 (defun testing-interactively-p ()
110 (and *test-is-being-defined?*
111 (not (or *test-is-being-compiled?*
112 *test-is-being-loaded?*))))
114 test-suite-p, etc could be defined more cleanly if the class
117 Instead of class properties, use the MOP
121 If you define a test suite and a test and then redefine the test-suite,
122 and then try to (run-test), you'll get an error. Should give a nice message
124 ~ need set of lift meta tests and set of lift "UI" tests
127 pending and run-pending method
130 finding suites and tests
132 sharing setup / teardown
135 ~ - non-deterministic
136 e.g., checking for a socket, expect failure occasionally
141 donÕt continue when failure option
143 cache previous results and only rerun failed tests
147 non-determininistic tests
149 [doc] ensure's should have a documentation clause
151 simplify test-suite-p
153 need a better error message for when a superclass that DNE or is not a test-class
156 ?should use same tricks as define-debugging-class to mangle the name
158 shouldn't teardown-test be an :after
160 when using break-on-errors? you should have restarts to allow to cancel out
161 completely, or keep going with the next test or whatever
163 when defining a test with addtest, if no current test case, or if named
164 case doesnt't exist, offer to create it (use a restart or whatever like in debug).
166 remove-test should print a nicer message if the test you're trying to undef DNE
168 remove-test needs to clean up data tables too
170 with-test should take optional test-class
172 clean up all the tables and stuff
174 with-test needs error handling
176 add documentation (at the least to the public interface)
178 Capture errors and have various fun restarts
180 error check addtest if class dne or not test-mixin
182 Improve error messages
185 (deftestsuite test-a ()
189 (defmethod initialize-prototypes :after ((test test-plus))
190 (with-test-slots (a b)
191 (setf (prototypes test)
198 (defmethod initialize-prototypes :after ((test test-plus))
199 (with-test-slots (a b)
200 (setf (prototypes test)
207 (defmethod initialize-prototypes :after ((test test-plus))
208 (with-test-slots (a b)
209 (setf (prototypes test)
212 (list (cons 'a a) (cons 'b b)))))))
214 (deftestsuite test-a ()
217 (deftestsuite test-a ()
219 (:cases (a '((1 2)))))
224 (deftestsuite test-b (test-a)
226 (:cases (b '(4 5 6))))
229 (format t "~% ~A x ~A" a b))
232 (deftestsuite test-e ()
235 (defmethod initialize-prototypes :after ((test test-e))
236 (setf (prototypes test)
239 (list (cons 'a 2)))))
244 (deftestsuite test-f (test-e)
247 (defmethod initialize-prototypes :after ((test test-f))
248 (setf (prototypes test)
252 (list (cons 'b 6)))))
255 (format t "~% ~A x ~A" a b))
267 (defgeneric setup (suite)
268 (:documentation "Setup at the testsuite-level")
269 (:method ((suite test-mixin))
272 (defgeneric testsuite-teardown (suite)
273 (:documentation "Cleanup at the testsuite level.")
274 (:method ((suite test-mixin))
277 (defgeneric testsuite-run (suite result)
278 (:documentation "Run the cases in this suite and it's children."))
280 (defgeneric setup-test (test-case)
281 (:documentation "Setup for a test-case. By default it does nothing."))
283 (defgeneric teardown-test (test-case)
284 (:documentation "Tear-down a test-case. By default it does nothing.")
285 (:method-combination progn :most-specific-first))
287 (defgeneric testsuite-methods (test-case)
288 (:documentation "Returns a list of the test methods defined for test. I.e.,
289 the methods that should be run to do the tests for this test."))
311 ;; here is better but will need to save off more information for results
312 ;; but this fails for child tests
323 ;; so it needs to be here
326 run each method of this suite using run-test-internal
333 push method name on tests-run of the result
346 test teardown plus remove variables
351 ;;; ---------------------------------------------------------------------------
353 Simplest case may be to give test-mixin a slot to hold initial values.
354 For 'regular' tests, this is what it is. For more complex tests, the values
355 are lists (need to differentiate between lists and multiple prototypes).
357 Then all tests work the same way.
359 The questions for the protocol are standard iteration quesionts: what is
360 the next value? and are we done yet?
364 ok - Probably don't need to define accessors or initargs since we're using test-environment
365 ok - I don't think we need *test-result* -- it's not used
366 I think it's a holdover from JUnit
367 ok - use test-code table to prevent some duplication (or require names)
368 ok - when name supplied, don't add "test-" to it
369 ok - get rid of print-test-result {s} and make better use of print-object and describe-object
370 ok - When adding multiple tests, need to store their names immediately (i.e.,
371 before going on to parse the next test).
372 obsolete - Improve error/warning for duplicate code/names
373 ok - run-tests works OK; run-test semi; evaluating an addtest does not
374 ok - need to store more in the test results to see what values were used, etc.
375 ok - Screw run-test being a macro
376 ok - Map case keywords to functions (think of tapir). Each of :map, :cross, etc.
377 is handled separately. This will make it easy to extend and build big monsters
378 Only "hard" part is correctly combining the conses.
379 ok - This prototype stuff at compile time is hard... maybe it would be easier to do it
380 at run time (using stack / CPL to help...)
381 ok - The problem with not using slots of the test class is that I need to duplicate
382 a bunch of functionality like inherited slots to make with-test-slots work
383 ok - I don't think I'm using *test-do-children?*, etc. correctly either
384 ok - Note: LIFT does not use the slots of the instance of the test-class; all looks
385 are via test-environment-value
386 (I no longer remember why this was a good idea)
387 (maybe don't even need the slots... just the defs)
388 ok - Get (deftestsuite test-addition ()
390 (:cases (:map (a '(1 2 3 4 5))
392 (:test ((ensure-same (+ a b) 10 :test '=))))
393 To work... Standardize-Cases-Form is broken
394 ok - parse-test-name-and-class is probably wacked b/c I changed to *current-suite-class-name*
396 ok - Use multiple processes to handle run away tests
397 get reporting to work correctly
398 ok - the whole number of tests thing is stupid and wacky
399 something wrong with loading or compiling addtest, the numbers don't get
400 incremented properly.
401 testsuite-test-count, number-of-tests
402 change to getting count at start of deftestsuite and then add 1 to a special
404 no - use dynamic scope for *test-is-being-compiled?* rather than unwind-protect
405 ok - LIFT: better control of printing (:follow-print)
406 ok - LIFT: variable to control behavior on warning / failure: print full report, etc.
407 ok - add :function to define local functions, use rewriting and macrolet
408 yes - so how do we integrate measuring with the report -- change the test result
409 add test result properties