1 :mod:`doctest` --- Test interactive Python examples
2 ===================================================
5 :synopsis: Test pieces of code within docstrings.
6 .. moduleauthor:: Tim Peters <tim@python.org>
7 .. sectionauthor:: Tim Peters <tim@python.org>
8 .. sectionauthor:: Moshe Zadka <moshez@debian.org>
9 .. sectionauthor:: Edward Loper <edloper@users.sourceforge.net>
12 The :mod:`doctest` module searches for pieces of text that look like interactive
13 Python sessions, and then executes those sessions to verify that they work
14 exactly as shown. There are several common ways to use doctest:
16 * To check that a module's docstrings are up-to-date by verifying that all
17 interactive examples still work as documented.
19 * To perform regression testing by verifying that interactive examples from a
20 test file or a test object work as expected.
22 * To write tutorial documentation for a package, liberally illustrated with
23 input-output examples. Depending on whether the examples or the expository text
24 are emphasized, this has the flavor of "literate testing" or "executable
27 Here's a complete but small example module::
30 This is the "example" module.
32 The example module supplies one function, factorial(). For example,
39 """Return the factorial of n, an exact integer >= 0.
41 If the result is small enough to fit in an int, return an int.
44 >>> [factorial(n) for n in range(6)]
46 >>> [factorial(long(n)) for n in range(6)]
49 265252859812191058636308480000000L
51 265252859812191058636308480000000L
53 Traceback (most recent call last):
55 ValueError: n must be >= 0
57 Factorials of floats are OK, but the float must be an exact integer:
59 Traceback (most recent call last):
61 ValueError: n must be exact integer
63 265252859812191058636308480000000L
65 It must also not be ridiculously large:
67 Traceback (most recent call last):
69 OverflowError: n too large
74 raise ValueError("n must be >= 0")
75 if math.floor(n) != n:
76 raise ValueError("n must be exact integer")
77 if n+1 == n: # catch a value like 1e300
78 raise OverflowError("n too large")
87 if __name__ == "__main__":
91 If you run :file:`example.py` directly from the command line, :mod:`doctest`
97 There's no output! That's normal, and it means all the examples worked. Pass
98 :option:`-v` to the script, and :mod:`doctest` prints a detailed log of what
99 it's trying, and prints a summary at the end::
101 $ python example.py -v
108 [factorial(n) for n in range(6)]
110 [1, 1, 2, 6, 24, 120]
113 [factorial(long(n)) for n in range(6)]
115 [1, 1, 2, 6, 24, 120]
118 And so on, eventually ending with::
123 Traceback (most recent call last):
125 OverflowError: n too large
127 2 items passed all tests:
129 8 tests in __main__.factorial
131 9 passed and 0 failed.
135 That's all you need to know to start making productive use of :mod:`doctest`!
136 Jump in. The following sections provide full details. Note that there are many
137 examples of doctests in the standard Python test suite and libraries.
138 Especially useful examples can be found in the standard test file
139 :file:`Lib/test/test_doctest.py`.
142 .. _doctest-simple-testmod:
144 Simple Usage: Checking Examples in Docstrings
145 ---------------------------------------------
147 The simplest way to start using doctest (but not necessarily the way you'll
148 continue to do it) is to end each module :mod:`M` with::
150 if __name__ == "__main__":
154 :mod:`doctest` then examines docstrings in module :mod:`M`.
156 Running the module as a script causes the examples in the docstrings to get
157 executed and verified::
161 This won't display anything unless an example fails, in which case the failing
162 example(s) and the cause(s) of the failure(s) are printed to stdout, and the
163 final line of output is ``***Test Failed*** N failures.``, where *N* is the
164 number of examples that failed.
166 Run it with the :option:`-v` switch instead::
170 and a detailed report of all examples tried is printed to standard output, along
171 with assorted summaries at the end.
173 You can force verbose mode by passing ``verbose=True`` to :func:`testmod`, or
174 prohibit it by passing ``verbose=False``. In either of those cases,
175 ``sys.argv`` is not examined by :func:`testmod` (so passing :option:`-v` or not
178 Since Python 2.6, there is also a command line shortcut for running
179 :func:`testmod`. You can instruct the Python interpreter to run the doctest
180 module directly from the standard library and pass the module name(s) on the
183 python -m doctest -v example.py
185 This will import :file:`example.py` as a standalone module and run
186 :func:`testmod` on it. Note that this may not work correctly if the file is
187 part of a package and imports other submodules from that package.
189 For more information on :func:`testmod`, see section :ref:`doctest-basic-api`.
192 .. _doctest-simple-testfile:
194 Simple Usage: Checking Examples in a Text File
195 ----------------------------------------------
197 Another simple application of doctest is testing interactive examples in a text
198 file. This can be done with the :func:`testfile` function::
201 doctest.testfile("example.txt")
203 That short script executes and verifies any interactive Python examples
204 contained in the file :file:`example.txt`. The file content is treated as if it
205 were a single giant docstring; the file doesn't need to contain a Python
206 program! For example, perhaps :file:`example.txt` contains this::
208 The ``example`` module
209 ======================
214 This is an example text file in reStructuredText format. First import
215 ``factorial`` from the ``example`` module:
217 >>> from example import factorial
224 Running ``doctest.testfile("example.txt")`` then finds the error in this
227 File "./example.txt", line 14, in example.txt
235 As with :func:`testmod`, :func:`testfile` won't display anything unless an
236 example fails. If an example does fail, then the failing example(s) and the
237 cause(s) of the failure(s) are printed to stdout, using the same format as
240 By default, :func:`testfile` looks for files in the calling module's directory.
241 See section :ref:`doctest-basic-api` for a description of the optional arguments
242 that can be used to tell it to look for files in other locations.
244 Like :func:`testmod`, :func:`testfile`'s verbosity can be set with the
245 :option:`-v` command-line switch or with the optional keyword argument
248 Since Python 2.6, there is also a command line shortcut for running
249 :func:`testfile`. You can instruct the Python interpreter to run the doctest
250 module directly from the standard library and pass the file name(s) on the
253 python -m doctest -v example.txt
255 Because the file name does not end with :file:`.py`, :mod:`doctest` infers that
256 it must be run with :func:`testfile`, not :func:`testmod`.
258 For more information on :func:`testfile`, see section :ref:`doctest-basic-api`.
261 .. _doctest-how-it-works:
266 This section examines in detail how doctest works: which docstrings it looks at,
267 how it finds interactive examples, what execution context it uses, how it
268 handles exceptions, and how option flags can be used to control its behavior.
269 This is the information that you need to know to write doctest examples; for
270 information about actually running doctest on these examples, see the following
274 .. _doctest-which-docstrings:
276 Which Docstrings Are Examined?
277 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
279 The module docstring, and all function, class and method docstrings are
280 searched. Objects imported into the module are not searched.
282 In addition, if ``M.__test__`` exists and "is true", it must be a dict, and each
283 entry maps a (string) name to a function object, class object, or string.
284 Function and class object docstrings found from ``M.__test__`` are searched, and
285 strings are treated as if they were docstrings. In output, a key ``K`` in
286 ``M.__test__`` appears with name ::
288 <name of M>.__test__.K
290 Any classes found are recursively searched similarly, to test docstrings in
291 their contained methods and nested classes.
293 .. versionchanged:: 2.4
294 A "private name" concept is deprecated and no longer documented.
297 .. _doctest-finding-examples:
299 How are Docstring Examples Recognized?
300 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
302 In most cases a copy-and-paste of an interactive console session works fine, but
303 doctest isn't trying to do an exact emulation of any specific Python shell. All
304 hard tab characters are expanded to spaces, using 8-column tab stops. If you
305 don't believe tabs should mean that, too bad: don't use hard tabs, or write
306 your own :class:`DocTestParser` class.
308 .. versionchanged:: 2.4
309 Expanding tabs to spaces is new; previous versions tried to preserve hard tabs,
310 with confusing results.
314 >>> # comments are ignored
330 Any expected output must immediately follow the final ``'>>> '`` or ``'... '``
331 line containing the code, and the expected output (if any) extends to the next
332 ``'>>> '`` or all-whitespace line.
336 * Expected output cannot contain an all-whitespace line, since such a line is
337 taken to signal the end of expected output. If expected output does contain a
338 blank line, put ``<BLANKLINE>`` in your doctest example each place a blank line
341 .. versionchanged:: 2.4
342 ``<BLANKLINE>`` was added; there was no way to use expected output containing
343 empty lines in previous versions.
345 * Output to stdout is captured, but not output to stderr (exception tracebacks
346 are captured via a different means).
348 * If you continue a line via backslashing in an interactive session, or for any
349 other reason use a backslash, you should use a raw docstring, which will
350 preserve your backslashes exactly as you type them::
353 ... r'''Backslashes in a raw docstring: m\n'''
355 Backslashes in a raw docstring: m\n
357 Otherwise, the backslash will be interpreted as part of the string. For example,
358 the "\\" above would be interpreted as a newline character. Alternatively, you
359 can double each backslash in the doctest version (and not use a raw string)::
362 ... '''Backslashes in a raw docstring: m\\n'''
364 Backslashes in a raw docstring: m\n
366 * The starting column doesn't matter::
373 and as many leading whitespace characters are stripped from the expected output
374 as appeared in the initial ``'>>> '`` line that started the example.
377 .. _doctest-execution-context:
379 What's the Execution Context?
380 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
382 By default, each time :mod:`doctest` finds a docstring to test, it uses a
383 *shallow copy* of :mod:`M`'s globals, so that running tests doesn't change the
384 module's real globals, and so that one test in :mod:`M` can't leave behind
385 crumbs that accidentally allow another test to work. This means examples can
386 freely use any names defined at top-level in :mod:`M`, and names defined earlier
387 in the docstring being run. Examples cannot see names defined in other
390 You can force use of your own dict as the execution context by passing
391 ``globs=your_dict`` to :func:`testmod` or :func:`testfile` instead.
394 .. _doctest-exceptions:
396 What About Exceptions?
397 ^^^^^^^^^^^^^^^^^^^^^^
399 No problem, provided that the traceback is the only output produced by the
400 example: just paste in the traceback. [#]_ Since tracebacks contain details
401 that are likely to change rapidly (for example, exact file paths and line
402 numbers), this is one case where doctest works hard to be flexible in what it
407 >>> [1, 2, 3].remove(42)
408 Traceback (most recent call last):
409 File "<stdin>", line 1, in ?
410 ValueError: list.remove(x): x not in list
412 That doctest succeeds if :exc:`ValueError` is raised, with the ``list.remove(x):
413 x not in list`` detail as shown.
415 The expected output for an exception must start with a traceback header, which
416 may be either of the following two lines, indented the same as the first line of
419 Traceback (most recent call last):
420 Traceback (innermost last):
422 The traceback header is followed by an optional traceback stack, whose contents
423 are ignored by doctest. The traceback stack is typically omitted, or copied
424 verbatim from an interactive session.
426 The traceback stack is followed by the most interesting part: the line(s)
427 containing the exception type and detail. This is usually the last line of a
428 traceback, but can extend across multiple lines if the exception has a
431 >>> raise ValueError('multi\n line\ndetail')
432 Traceback (most recent call last):
433 File "<stdin>", line 1, in ?
438 The last three lines (starting with :exc:`ValueError`) are compared against the
439 exception's type and detail, and the rest are ignored.
441 Best practice is to omit the traceback stack, unless it adds significant
442 documentation value to the example. So the last example is probably better as::
444 >>> raise ValueError('multi\n line\ndetail')
445 Traceback (most recent call last):
451 Note that tracebacks are treated very specially. In particular, in the
452 rewritten example, the use of ``...`` is independent of doctest's
453 :const:`ELLIPSIS` option. The ellipsis in that example could be left out, or
454 could just as well be three (or three hundred) commas or digits, or an indented
455 transcript of a Monty Python skit.
457 Some details you should read once, but won't need to remember:
459 * Doctest can't guess whether your expected output came from an exception
460 traceback or from ordinary printing. So, e.g., an example that expects
461 ``ValueError: 42 is prime`` will pass whether :exc:`ValueError` is actually
462 raised or if the example merely prints that traceback text. In practice,
463 ordinary output rarely begins with a traceback header line, so this doesn't
464 create real problems.
466 * Each line of the traceback stack (if present) must be indented further than
467 the first line of the example, *or* start with a non-alphanumeric character.
468 The first line following the traceback header indented the same and starting
469 with an alphanumeric is taken to be the start of the exception detail. Of
470 course this does the right thing for genuine tracebacks.
472 * When the :const:`IGNORE_EXCEPTION_DETAIL` doctest option is is specified,
473 everything following the leftmost colon is ignored.
475 * The interactive shell omits the traceback header line for some
476 :exc:`SyntaxError`\ s. But doctest uses the traceback header line to
477 distinguish exceptions from non-exceptions. So in the rare case where you need
478 to test a :exc:`SyntaxError` that omits the traceback header, you will need to
479 manually add the traceback header line to your test example.
481 * For some :exc:`SyntaxError`\ s, Python displays the character position of the
482 syntax error, using a ``^`` marker::
485 File "<stdin>", line 1
488 SyntaxError: invalid syntax
490 Since the lines showing the position of the error come before the exception type
491 and detail, they are not checked by doctest. For example, the following test
492 would pass, even though it puts the ``^`` marker in the wrong location::
495 Traceback (most recent call last):
496 File "<stdin>", line 1
499 SyntaxError: invalid syntax
501 .. versionchanged:: 2.4
502 The ability to handle a multi-line exception detail, and the
503 :const:`IGNORE_EXCEPTION_DETAIL` doctest option, were added.
508 Option Flags and Directives
509 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
511 A number of option flags control various aspects of doctest's behavior.
512 Symbolic names for the flags are supplied as module constants, which can be
513 or'ed together and passed to various functions. The names can also be used in
514 doctest directives (see below).
516 The first group of options define test semantics, controlling aspects of how
517 doctest decides whether actual output matches an example's expected output:
520 .. data:: DONT_ACCEPT_TRUE_FOR_1
522 By default, if an expected output block contains just ``1``, an actual output
523 block containing just ``1`` or just ``True`` is considered to be a match, and
524 similarly for ``0`` versus ``False``. When :const:`DONT_ACCEPT_TRUE_FOR_1` is
525 specified, neither substitution is allowed. The default behavior caters to that
526 Python changed the return type of many functions from integer to boolean;
527 doctests expecting "little integer" output still work in these cases. This
528 option will probably go away, but not for several years.
531 .. data:: DONT_ACCEPT_BLANKLINE
533 By default, if an expected output block contains a line containing only the
534 string ``<BLANKLINE>``, then that line will match a blank line in the actual
535 output. Because a genuinely blank line delimits the expected output, this is
536 the only way to communicate that a blank line is expected. When
537 :const:`DONT_ACCEPT_BLANKLINE` is specified, this substitution is not allowed.
540 .. data:: NORMALIZE_WHITESPACE
542 When specified, all sequences of whitespace (blanks and newlines) are treated as
543 equal. Any sequence of whitespace within the expected output will match any
544 sequence of whitespace within the actual output. By default, whitespace must
545 match exactly. :const:`NORMALIZE_WHITESPACE` is especially useful when a line of
546 expected output is very long, and you want to wrap it across multiple lines in
552 When specified, an ellipsis marker (``...``) in the expected output can match
553 any substring in the actual output. This includes substrings that span line
554 boundaries, and empty substrings, so it's best to keep usage of this simple.
555 Complicated uses can lead to the same kinds of "oops, it matched too much!"
556 surprises that ``.*`` is prone to in regular expressions.
559 .. data:: IGNORE_EXCEPTION_DETAIL
561 When specified, an example that expects an exception passes if an exception of
562 the expected type is raised, even if the exception detail does not match. For
563 example, an example expecting ``ValueError: 42`` will pass if the actual
564 exception raised is ``ValueError: 3*14``, but will fail, e.g., if
565 :exc:`TypeError` is raised.
567 Note that a similar effect can be obtained using :const:`ELLIPSIS`, and
568 :const:`IGNORE_EXCEPTION_DETAIL` may go away when Python releases prior to 2.4
569 become uninteresting. Until then, :const:`IGNORE_EXCEPTION_DETAIL` is the only
570 clear way to write a doctest that doesn't care about the exception detail yet
571 continues to pass under Python releases prior to 2.4 (doctest directives appear
572 to be comments to them). For example, ::
574 >>> (1, 2)[3] = 'moo' #doctest: +IGNORE_EXCEPTION_DETAIL
575 Traceback (most recent call last):
576 File "<stdin>", line 1, in ?
577 TypeError: object doesn't support item assignment
579 passes under Python 2.4 and Python 2.3. The detail changed in 2.4, to say "does
580 not" instead of "doesn't".
585 When specified, do not run the example at all. This can be useful in contexts
586 where doctest examples serve as both documentation and test cases, and an
587 example should be included for documentation purposes, but should not be
588 checked. E.g., the example's output might be random; or the example might
589 depend on resources which would be unavailable to the test driver.
591 The SKIP flag can also be used for temporarily "commenting out" examples.
594 .. data:: COMPARISON_FLAGS
596 A bitmask or'ing together all the comparison flags above.
598 The second group of options controls how test failures are reported:
601 .. data:: REPORT_UDIFF
603 When specified, failures that involve multi-line expected and actual outputs are
604 displayed using a unified diff.
607 .. data:: REPORT_CDIFF
609 When specified, failures that involve multi-line expected and actual outputs
610 will be displayed using a context diff.
613 .. data:: REPORT_NDIFF
615 When specified, differences are computed by ``difflib.Differ``, using the same
616 algorithm as the popular :file:`ndiff.py` utility. This is the only method that
617 marks differences within lines as well as across lines. For example, if a line
618 of expected output contains digit ``1`` where actual output contains letter
619 ``l``, a line is inserted with a caret marking the mismatching column positions.
622 .. data:: REPORT_ONLY_FIRST_FAILURE
624 When specified, display the first failing example in each doctest, but suppress
625 output for all remaining examples. This will prevent doctest from reporting
626 correct examples that break because of earlier failures; but it might also hide
627 incorrect examples that fail independently of the first failure. When
628 :const:`REPORT_ONLY_FIRST_FAILURE` is specified, the remaining examples are
629 still run, and still count towards the total number of failures reported; only
630 the output is suppressed.
633 .. data:: REPORTING_FLAGS
635 A bitmask or'ing together all the reporting flags above.
637 "Doctest directives" may be used to modify the option flags for individual
638 examples. Doctest directives are expressed as a special Python comment
639 following an example's source code:
641 .. productionlist:: doctest
642 directive: "#" "doctest:" `directive_options`
643 directive_options: `directive_option` ("," `directive_option`)\*
644 directive_option: `on_or_off` `directive_option_name`
645 on_or_off: "+" \| "-"
646 directive_option_name: "DONT_ACCEPT_BLANKLINE" \| "NORMALIZE_WHITESPACE" \| ...
648 Whitespace is not allowed between the ``+`` or ``-`` and the directive option
649 name. The directive option name can be any of the option flag names explained
652 An example's doctest directives modify doctest's behavior for that single
653 example. Use ``+`` to enable the named behavior, or ``-`` to disable it.
655 For example, this test passes::
657 >>> print range(20) #doctest: +NORMALIZE_WHITESPACE
658 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
659 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
661 Without the directive it would fail, both because the actual output doesn't have
662 two blanks before the single-digit list elements, and because the actual output
663 is on a single line. This test also passes, and also requires a directive to do
666 >>> print range(20) # doctest:+ELLIPSIS
669 Multiple directives can be used on a single physical line, separated by commas::
671 >>> print range(20) # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE
674 If multiple directive comments are used for a single example, then they are
677 >>> print range(20) # doctest: +ELLIPSIS
678 ... # doctest: +NORMALIZE_WHITESPACE
681 As the previous example shows, you can add ``...`` lines to your example
682 containing only directives. This can be useful when an example is too long for
683 a directive to comfortably fit on the same line::
685 >>> print range(5) + range(10,20) + range(30,40) + range(50,60)
686 ... # doctest: +ELLIPSIS
687 [0, ..., 4, 10, ..., 19, 30, ..., 39, 50, ..., 59]
689 Note that since all options are disabled by default, and directives apply only
690 to the example they appear in, enabling options (via ``+`` in a directive) is
691 usually the only meaningful choice. However, option flags can also be passed to
692 functions that run doctests, establishing different defaults. In such cases,
693 disabling an option via ``-`` in a directive can be useful.
695 .. versionchanged:: 2.4
696 Constants :const:`DONT_ACCEPT_BLANKLINE`, :const:`NORMALIZE_WHITESPACE`,
697 :const:`ELLIPSIS`, :const:`IGNORE_EXCEPTION_DETAIL`, :const:`REPORT_UDIFF`,
698 :const:`REPORT_CDIFF`, :const:`REPORT_NDIFF`,
699 :const:`REPORT_ONLY_FIRST_FAILURE`, :const:`COMPARISON_FLAGS` and
700 :const:`REPORTING_FLAGS` were added; by default ``<BLANKLINE>`` in expected
701 output matches an empty line in actual output; and doctest directives were
704 .. versionchanged:: 2.5
705 Constant :const:`SKIP` was added.
707 There's also a way to register new option flag names, although this isn't useful
708 unless you intend to extend :mod:`doctest` internals via subclassing:
711 .. function:: register_optionflag(name)
713 Create a new option flag with a given name, and return the new flag's integer
714 value. :func:`register_optionflag` can be used when subclassing
715 :class:`OutputChecker` or :class:`DocTestRunner` to create new options that are
716 supported by your subclasses. :func:`register_optionflag` should always be
717 called using the following idiom::
719 MY_FLAG = register_optionflag('MY_FLAG')
721 .. versionadded:: 2.4
724 .. _doctest-warnings:
729 :mod:`doctest` is serious about requiring exact matches in expected output. If
730 even a single character doesn't match, the test fails. This will probably
731 surprise you a few times, as you learn exactly what Python does and doesn't
732 guarantee about output. For example, when printing a dict, Python doesn't
733 guarantee that the key-value pairs will be printed in any particular order, so a
737 {"Hermione": "hippogryph", "Harry": "broomstick"}
739 is vulnerable! One workaround is to do ::
741 >>> foo() == {"Hermione": "hippogryph", "Harry": "broomstick"}
744 instead. Another is to do ::
746 >>> d = foo().items()
749 [('Harry', 'broomstick'), ('Hermione', 'hippogryph')]
751 There are others, but you get the idea.
753 Another bad idea is to print things that embed an object address, like ::
755 >>> id(1.0) # certain to fail some of the time
758 >>> C() # the default repr() for instances embeds an address
759 <__main__.C instance at 0x00AC18F0>
761 The :const:`ELLIPSIS` directive gives a nice approach for the last example::
763 >>> C() #doctest: +ELLIPSIS
764 <__main__.C instance at 0x...>
766 Floating-point numbers are also subject to small output variations across
767 platforms, because Python defers to the platform C library for float formatting,
768 and C libraries vary widely in quality here. ::
772 >>> print 1./7 # safer
774 >>> print round(1./7, 6) # much safer
777 Numbers of the form ``I/2.**J`` are safe across all platforms, and I often
778 contrive doctest examples to produce numbers of that form::
780 >>> 3./4 # utterly safe
783 Simple fractions are also easier for people to understand, and that makes for
784 better documentation.
787 .. _doctest-basic-api:
792 The functions :func:`testmod` and :func:`testfile` provide a simple interface to
793 doctest that should be sufficient for most basic uses. For a less formal
794 introduction to these two functions, see sections :ref:`doctest-simple-testmod`
795 and :ref:`doctest-simple-testfile`.
798 .. function:: testfile(filename[, module_relative][, name][, package][, globs][, verbose][, report][, optionflags][, extraglobs][, raise_on_error][, parser][, encoding])
800 All arguments except *filename* are optional, and should be specified in keyword
803 Test examples in the file named *filename*. Return ``(failure_count,
806 Optional argument *module_relative* specifies how the filename should be
809 * If *module_relative* is ``True`` (the default), then *filename* specifies an
810 OS-independent module-relative path. By default, this path is relative to the
811 calling module's directory; but if the *package* argument is specified, then it
812 is relative to that package. To ensure OS-independence, *filename* should use
813 ``/`` characters to separate path segments, and may not be an absolute path
814 (i.e., it may not begin with ``/``).
816 * If *module_relative* is ``False``, then *filename* specifies an OS-specific
817 path. The path may be absolute or relative; relative paths are resolved with
818 respect to the current working directory.
820 Optional argument *name* gives the name of the test; by default, or if ``None``,
821 ``os.path.basename(filename)`` is used.
823 Optional argument *package* is a Python package or the name of a Python package
824 whose directory should be used as the base directory for a module-relative
825 filename. If no package is specified, then the calling module's directory is
826 used as the base directory for module-relative filenames. It is an error to
827 specify *package* if *module_relative* is ``False``.
829 Optional argument *globs* gives a dict to be used as the globals when executing
830 examples. A new shallow copy of this dict is created for the doctest, so its
831 examples start with a clean slate. By default, or if ``None``, a new empty dict
834 Optional argument *extraglobs* gives a dict merged into the globals used to
835 execute examples. This works like :meth:`dict.update`: if *globs* and
836 *extraglobs* have a common key, the associated value in *extraglobs* appears in
837 the combined dict. By default, or if ``None``, no extra globals are used. This
838 is an advanced feature that allows parameterization of doctests. For example, a
839 doctest can be written for a base class, using a generic name for the class,
840 then reused to test any number of subclasses by passing an *extraglobs* dict
841 mapping the generic name to the subclass to be tested.
843 Optional argument *verbose* prints lots of stuff if true, and prints only
844 failures if false; by default, or if ``None``, it's true if and only if ``'-v'``
847 Optional argument *report* prints a summary at the end when true, else prints
848 nothing at the end. In verbose mode, the summary is detailed, else the summary
849 is very brief (in fact, empty if all tests passed).
851 Optional argument *optionflags* or's together option flags. See section
852 :ref:`doctest-options`.
854 Optional argument *raise_on_error* defaults to false. If true, an exception is
855 raised upon the first failure or unexpected exception in an example. This
856 allows failures to be post-mortem debugged. Default behavior is to continue
859 Optional argument *parser* specifies a :class:`DocTestParser` (or subclass) that
860 should be used to extract tests from the files. It defaults to a normal parser
861 (i.e., ``DocTestParser()``).
863 Optional argument *encoding* specifies an encoding that should be used to
864 convert the file to unicode.
866 .. versionadded:: 2.4
868 .. versionchanged:: 2.5
869 The parameter *encoding* was added.
872 .. function:: testmod([m][, name][, globs][, verbose][, report][, optionflags][, extraglobs][, raise_on_error][, exclude_empty])
874 All arguments are optional, and all except for *m* should be specified in
877 Test examples in docstrings in functions and classes reachable from module *m*
878 (or module :mod:`__main__` if *m* is not supplied or is ``None``), starting with
881 Also test examples reachable from dict ``m.__test__``, if it exists and is not
882 ``None``. ``m.__test__`` maps names (strings) to functions, classes and
883 strings; function and class docstrings are searched for examples; strings are
884 searched directly, as if they were docstrings.
886 Only docstrings attached to objects belonging to module *m* are searched.
888 Return ``(failure_count, test_count)``.
890 Optional argument *name* gives the name of the module; by default, or if
891 ``None``, ``m.__name__`` is used.
893 Optional argument *exclude_empty* defaults to false. If true, objects for which
894 no doctests are found are excluded from consideration. The default is a backward
895 compatibility hack, so that code still using :meth:`doctest.master.summarize` in
896 conjunction with :func:`testmod` continues to get output for objects with no
897 tests. The *exclude_empty* argument to the newer :class:`DocTestFinder`
898 constructor defaults to true.
900 Optional arguments *extraglobs*, *verbose*, *report*, *optionflags*,
901 *raise_on_error*, and *globs* are the same as for function :func:`testfile`
902 above, except that *globs* defaults to ``m.__dict__``.
904 .. versionchanged:: 2.3
905 The parameter *optionflags* was added.
907 .. versionchanged:: 2.4
908 The parameters *extraglobs*, *raise_on_error* and *exclude_empty* were added.
910 .. versionchanged:: 2.5
911 The optional argument *isprivate*, deprecated in 2.4, was removed.
913 There's also a function to run the doctests associated with a single object.
914 This function is provided for backward compatibility. There are no plans to
915 deprecate it, but it's rarely useful:
918 .. function:: run_docstring_examples(f, globs[, verbose][, name][, compileflags][, optionflags])
920 Test examples associated with object *f*; for example, *f* may be a module,
921 function, or class object.
923 A shallow copy of dictionary argument *globs* is used for the execution context.
925 Optional argument *name* is used in failure messages, and defaults to
928 If optional argument *verbose* is true, output is generated even if there are no
929 failures. By default, output is generated only in case of an example failure.
931 Optional argument *compileflags* gives the set of flags that should be used by
932 the Python compiler when running the examples. By default, or if ``None``,
933 flags are deduced corresponding to the set of future features found in *globs*.
935 Optional argument *optionflags* works as for function :func:`testfile` above.
938 .. _doctest-unittest-api:
943 As your collection of doctest'ed modules grows, you'll want a way to run all
944 their doctests systematically. Prior to Python 2.4, :mod:`doctest` had a barely
945 documented :class:`Tester` class that supplied a rudimentary way to combine
946 doctests from multiple modules. :class:`Tester` was feeble, and in practice most
947 serious Python testing frameworks build on the :mod:`unittest` module, which
948 supplies many flexible ways to combine tests from multiple sources. So, in
949 Python 2.4, :mod:`doctest`'s :class:`Tester` class is deprecated, and
950 :mod:`doctest` provides two functions that can be used to create :mod:`unittest`
951 test suites from modules and text files containing doctests. These test suites
952 can then be run using :mod:`unittest` test runners::
956 import my_module_with_doctests, and_another
958 suite = unittest.TestSuite()
959 for mod in my_module_with_doctests, and_another:
960 suite.addTest(doctest.DocTestSuite(mod))
961 runner = unittest.TextTestRunner()
964 There are two main functions for creating :class:`unittest.TestSuite` instances
965 from text files and modules with doctests:
968 .. function:: DocFileSuite([module_relative][, package][, setUp][, tearDown][, globs][, optionflags][, parser][, encoding])
970 Convert doctest tests from one or more text files to a
971 :class:`unittest.TestSuite`.
973 The returned :class:`unittest.TestSuite` is to be run by the unittest framework
974 and runs the interactive examples in each file. If an example in any file
975 fails, then the synthesized unit test fails, and a :exc:`failureException`
976 exception is raised showing the name of the file containing the test and a
977 (sometimes approximate) line number.
979 Pass one or more paths (as strings) to text files to be examined.
981 Options may be provided as keyword arguments:
983 Optional argument *module_relative* specifies how the filenames in *paths*
984 should be interpreted:
986 * If *module_relative* is ``True`` (the default), then each filename specifies
987 an OS-independent module-relative path. By default, this path is relative to
988 the calling module's directory; but if the *package* argument is specified, then
989 it is relative to that package. To ensure OS-independence, each filename should
990 use ``/`` characters to separate path segments, and may not be an absolute path
991 (i.e., it may not begin with ``/``).
993 * If *module_relative* is ``False``, then each filename specifies an OS-specific
994 path. The path may be absolute or relative; relative paths are resolved with
995 respect to the current working directory.
997 Optional argument *package* is a Python package or the name of a Python package
998 whose directory should be used as the base directory for module-relative
999 filenames. If no package is specified, then the calling module's directory is
1000 used as the base directory for module-relative filenames. It is an error to
1001 specify *package* if *module_relative* is ``False``.
1003 Optional argument *setUp* specifies a set-up function for the test suite. This
1004 is called before running the tests in each file. The *setUp* function will be
1005 passed a :class:`DocTest` object. The setUp function can access the test
1006 globals as the *globs* attribute of the test passed.
1008 Optional argument *tearDown* specifies a tear-down function for the test suite.
1009 This is called after running the tests in each file. The *tearDown* function
1010 will be passed a :class:`DocTest` object. The setUp function can access the
1011 test globals as the *globs* attribute of the test passed.
1013 Optional argument *globs* is a dictionary containing the initial global
1014 variables for the tests. A new copy of this dictionary is created for each
1015 test. By default, *globs* is a new empty dictionary.
1017 Optional argument *optionflags* specifies the default doctest options for the
1018 tests, created by or-ing together individual option flags. See section
1019 :ref:`doctest-options`. See function :func:`set_unittest_reportflags` below for
1020 a better way to set reporting options.
1022 Optional argument *parser* specifies a :class:`DocTestParser` (or subclass) that
1023 should be used to extract tests from the files. It defaults to a normal parser
1024 (i.e., ``DocTestParser()``).
1026 Optional argument *encoding* specifies an encoding that should be used to
1027 convert the file to unicode.
1029 .. versionadded:: 2.4
1031 .. versionchanged:: 2.5
1032 The global ``__file__`` was added to the globals provided to doctests loaded
1033 from a text file using :func:`DocFileSuite`.
1035 .. versionchanged:: 2.5
1036 The parameter *encoding* was added.
1039 .. function:: DocTestSuite([module][, globs][, extraglobs][, test_finder][, setUp][, tearDown][, checker])
1041 Convert doctest tests for a module to a :class:`unittest.TestSuite`.
1043 The returned :class:`unittest.TestSuite` is to be run by the unittest framework
1044 and runs each doctest in the module. If any of the doctests fail, then the
1045 synthesized unit test fails, and a :exc:`failureException` exception is raised
1046 showing the name of the file containing the test and a (sometimes approximate)
1049 Optional argument *module* provides the module to be tested. It can be a module
1050 object or a (possibly dotted) module name. If not specified, the module calling
1051 this function is used.
1053 Optional argument *globs* is a dictionary containing the initial global
1054 variables for the tests. A new copy of this dictionary is created for each
1055 test. By default, *globs* is a new empty dictionary.
1057 Optional argument *extraglobs* specifies an extra set of global variables, which
1058 is merged into *globs*. By default, no extra globals are used.
1060 Optional argument *test_finder* is the :class:`DocTestFinder` object (or a
1061 drop-in replacement) that is used to extract doctests from the module.
1063 Optional arguments *setUp*, *tearDown*, and *optionflags* are the same as for
1064 function :func:`DocFileSuite` above.
1066 .. versionadded:: 2.3
1068 .. versionchanged:: 2.4
1069 The parameters *globs*, *extraglobs*, *test_finder*, *setUp*, *tearDown*, and
1070 *optionflags* were added; this function now uses the same search technique as
1073 Under the covers, :func:`DocTestSuite` creates a :class:`unittest.TestSuite` out
1074 of :class:`doctest.DocTestCase` instances, and :class:`DocTestCase` is a
1075 subclass of :class:`unittest.TestCase`. :class:`DocTestCase` isn't documented
1076 here (it's an internal detail), but studying its code can answer questions about
1077 the exact details of :mod:`unittest` integration.
1079 Similarly, :func:`DocFileSuite` creates a :class:`unittest.TestSuite` out of
1080 :class:`doctest.DocFileCase` instances, and :class:`DocFileCase` is a subclass
1081 of :class:`DocTestCase`.
1083 So both ways of creating a :class:`unittest.TestSuite` run instances of
1084 :class:`DocTestCase`. This is important for a subtle reason: when you run
1085 :mod:`doctest` functions yourself, you can control the :mod:`doctest` options in
1086 use directly, by passing option flags to :mod:`doctest` functions. However, if
1087 you're writing a :mod:`unittest` framework, :mod:`unittest` ultimately controls
1088 when and how tests get run. The framework author typically wants to control
1089 :mod:`doctest` reporting options (perhaps, e.g., specified by command line
1090 options), but there's no way to pass options through :mod:`unittest` to
1091 :mod:`doctest` test runners.
1093 For this reason, :mod:`doctest` also supports a notion of :mod:`doctest`
1094 reporting flags specific to :mod:`unittest` support, via this function:
1097 .. function:: set_unittest_reportflags(flags)
1099 Set the :mod:`doctest` reporting flags to use.
1101 Argument *flags* or's together option flags. See section
1102 :ref:`doctest-options`. Only "reporting flags" can be used.
1104 This is a module-global setting, and affects all future doctests run by module
1105 :mod:`unittest`: the :meth:`runTest` method of :class:`DocTestCase` looks at
1106 the option flags specified for the test case when the :class:`DocTestCase`
1107 instance was constructed. If no reporting flags were specified (which is the
1108 typical and expected case), :mod:`doctest`'s :mod:`unittest` reporting flags are
1109 or'ed into the option flags, and the option flags so augmented are passed to the
1110 :class:`DocTestRunner` instance created to run the doctest. If any reporting
1111 flags were specified when the :class:`DocTestCase` instance was constructed,
1112 :mod:`doctest`'s :mod:`unittest` reporting flags are ignored.
1114 The value of the :mod:`unittest` reporting flags in effect before the function
1115 was called is returned by the function.
1117 .. versionadded:: 2.4
1120 .. _doctest-advanced-api:
1125 The basic API is a simple wrapper that's intended to make doctest easy to use.
1126 It is fairly flexible, and should meet most users' needs; however, if you
1127 require more fine-grained control over testing, or wish to extend doctest's
1128 capabilities, then you should use the advanced API.
1130 The advanced API revolves around two container classes, which are used to store
1131 the interactive examples extracted from doctest cases:
1133 * :class:`Example`: A single python :term:`statement`, paired with its expected
1136 * :class:`DocTest`: A collection of :class:`Example`\ s, typically extracted
1137 from a single docstring or text file.
1139 Additional processing classes are defined to find, parse, and run, and check
1142 * :class:`DocTestFinder`: Finds all docstrings in a given module, and uses a
1143 :class:`DocTestParser` to create a :class:`DocTest` from every docstring that
1144 contains interactive examples.
1146 * :class:`DocTestParser`: Creates a :class:`DocTest` object from a string (such
1147 as an object's docstring).
1149 * :class:`DocTestRunner`: Executes the examples in a :class:`DocTest`, and uses
1150 an :class:`OutputChecker` to verify their output.
1152 * :class:`OutputChecker`: Compares the actual output from a doctest example with
1153 the expected output, and decides whether they match.
1155 The relationships among these processing classes are summarized in the following
1159 +------+ +---------+
1160 |module| --DocTestFinder-> | DocTest | --DocTestRunner-> results
1161 +------+ | ^ +---------+ | ^ (printed)
1164 DocTestParser | Example | OutputChecker
1168 .. _doctest-doctest:
1174 .. class:: DocTest(examples, globs, name, filename, lineno, docstring)
1176 A collection of doctest examples that should be run in a single namespace. The
1177 constructor arguments are used to initialize the member variables of the same
1180 .. versionadded:: 2.4
1182 :class:`DocTest` defines the following member variables. They are initialized by
1183 the constructor, and should not be modified directly.
1186 .. attribute:: examples
1188 A list of :class:`Example` objects encoding the individual interactive Python
1189 examples that should be run by this test.
1192 .. attribute:: globs
1194 The namespace (aka globals) that the examples should be run in. This is a
1195 dictionary mapping names to values. Any changes to the namespace made by the
1196 examples (such as binding new variables) will be reflected in :attr:`globs`
1197 after the test is run.
1202 A string name identifying the :class:`DocTest`. Typically, this is the name
1203 of the object or file that the test was extracted from.
1206 .. attribute:: filename
1208 The name of the file that this :class:`DocTest` was extracted from; or
1209 ``None`` if the filename is unknown, or if the :class:`DocTest` was not
1210 extracted from a file.
1213 .. attribute:: lineno
1215 The line number within :attr:`filename` where this :class:`DocTest` begins, or
1216 ``None`` if the line number is unavailable. This line number is zero-based
1217 with respect to the beginning of the file.
1220 .. attribute:: docstring
1222 The string that the test was extracted from, or 'None' if the string is
1223 unavailable, or if the test was not extracted from a string.
1226 .. _doctest-example:
1232 .. class:: Example(source, want[, exc_msg][, lineno][, indent][, options])
1234 A single interactive example, consisting of a Python statement and its expected
1235 output. The constructor arguments are used to initialize the member variables
1238 .. versionadded:: 2.4
1240 :class:`Example` defines the following member variables. They are initialized by
1241 the constructor, and should not be modified directly.
1244 .. attribute:: source
1246 A string containing the example's source code. This source code consists of a
1247 single Python statement, and always ends with a newline; the constructor adds
1248 a newline when necessary.
1253 The expected output from running the example's source code (either from
1254 stdout, or a traceback in case of exception). :attr:`want` ends with a
1255 newline unless no output is expected, in which case it's an empty string. The
1256 constructor adds a newline when necessary.
1259 .. attribute:: exc_msg
1261 The exception message generated by the example, if the example is expected to
1262 generate an exception; or ``None`` if it is not expected to generate an
1263 exception. This exception message is compared against the return value of
1264 :func:`traceback.format_exception_only`. :attr:`exc_msg` ends with a newline
1265 unless it's ``None``. The constructor adds a newline if needed.
1268 .. attribute:: lineno
1270 The line number within the string containing this example where the example
1271 begins. This line number is zero-based with respect to the beginning of the
1275 .. attribute:: indent
1277 The example's indentation in the containing string, i.e., the number of space
1278 characters that precede the example's first prompt.
1281 .. attribute:: options
1283 A dictionary mapping from option flags to ``True`` or ``False``, which is used
1284 to override default options for this example. Any option flags not contained
1285 in this dictionary are left at their default value (as specified by the
1286 :class:`DocTestRunner`'s :attr:`optionflags`). By default, no options are set.
1289 .. _doctest-doctestfinder:
1291 DocTestFinder objects
1292 ^^^^^^^^^^^^^^^^^^^^^
1295 .. class:: DocTestFinder([verbose][, parser][, recurse][, exclude_empty])
1297 A processing class used to extract the :class:`DocTest`\ s that are relevant to
1298 a given object, from its docstring and the docstrings of its contained objects.
1299 :class:`DocTest`\ s can currently be extracted from the following object types:
1300 modules, functions, classes, methods, staticmethods, classmethods, and
1303 The optional argument *verbose* can be used to display the objects searched by
1304 the finder. It defaults to ``False`` (no output).
1306 The optional argument *parser* specifies the :class:`DocTestParser` object (or a
1307 drop-in replacement) that is used to extract doctests from docstrings.
1309 If the optional argument *recurse* is false, then :meth:`DocTestFinder.find`
1310 will only examine the given object, and not any contained objects.
1312 If the optional argument *exclude_empty* is false, then
1313 :meth:`DocTestFinder.find` will include tests for objects with empty docstrings.
1315 .. versionadded:: 2.4
1317 :class:`DocTestFinder` defines the following method:
1320 .. method:: find(obj[, name][, module][, globs][, extraglobs])
1322 Return a list of the :class:`DocTest`\ s that are defined by *obj*'s
1323 docstring, or by any of its contained objects' docstrings.
1325 The optional argument *name* specifies the object's name; this name will be
1326 used to construct names for the returned :class:`DocTest`\ s. If *name* is
1327 not specified, then ``obj.__name__`` is used.
1329 The optional parameter *module* is the module that contains the given object.
1330 If the module is not specified or is None, then the test finder will attempt
1331 to automatically determine the correct module. The object's module is used:
1333 * As a default namespace, if *globs* is not specified.
1335 * To prevent the DocTestFinder from extracting DocTests from objects that are
1336 imported from other modules. (Contained objects with modules other than
1337 *module* are ignored.)
1339 * To find the name of the file containing the object.
1341 * To help find the line number of the object within its file.
1343 If *module* is ``False``, no attempt to find the module will be made. This is
1344 obscure, of use mostly in testing doctest itself: if *module* is ``False``, or
1345 is ``None`` but cannot be found automatically, then all objects are considered
1346 to belong to the (non-existent) module, so all contained objects will
1347 (recursively) be searched for doctests.
1349 The globals for each :class:`DocTest` is formed by combining *globs* and
1350 *extraglobs* (bindings in *extraglobs* override bindings in *globs*). A new
1351 shallow copy of the globals dictionary is created for each :class:`DocTest`.
1352 If *globs* is not specified, then it defaults to the module's *__dict__*, if
1353 specified, or ``{}`` otherwise. If *extraglobs* is not specified, then it
1357 .. _doctest-doctestparser:
1359 DocTestParser objects
1360 ^^^^^^^^^^^^^^^^^^^^^
1363 .. class:: DocTestParser()
1365 A processing class used to extract interactive examples from a string, and use
1366 them to create a :class:`DocTest` object.
1368 .. versionadded:: 2.4
1370 :class:`DocTestParser` defines the following methods:
1373 .. method:: get_doctest(string, globs, name, filename, lineno)
1375 Extract all doctest examples from the given string, and collect them into a
1376 :class:`DocTest` object.
1378 *globs*, *name*, *filename*, and *lineno* are attributes for the new
1379 :class:`DocTest` object. See the documentation for :class:`DocTest` for more
1383 .. method:: get_examples(string[, name])
1385 Extract all doctest examples from the given string, and return them as a list
1386 of :class:`Example` objects. Line numbers are 0-based. The optional argument
1387 *name* is a name identifying this string, and is only used for error messages.
1390 .. method:: parse(string[, name])
1392 Divide the given string into examples and intervening text, and return them as
1393 a list of alternating :class:`Example`\ s and strings. Line numbers for the
1394 :class:`Example`\ s are 0-based. The optional argument *name* is a name
1395 identifying this string, and is only used for error messages.
1398 .. _doctest-doctestrunner:
1400 DocTestRunner objects
1401 ^^^^^^^^^^^^^^^^^^^^^
1404 .. class:: DocTestRunner([checker][, verbose][, optionflags])
1406 A processing class used to execute and verify the interactive examples in a
1409 The comparison between expected outputs and actual outputs is done by an
1410 :class:`OutputChecker`. This comparison may be customized with a number of
1411 option flags; see section :ref:`doctest-options` for more information. If the
1412 option flags are insufficient, then the comparison may also be customized by
1413 passing a subclass of :class:`OutputChecker` to the constructor.
1415 The test runner's display output can be controlled in two ways. First, an output
1416 function can be passed to :meth:`TestRunner.run`; this function will be called
1417 with strings that should be displayed. It defaults to ``sys.stdout.write``. If
1418 capturing the output is not sufficient, then the display output can be also
1419 customized by subclassing DocTestRunner, and overriding the methods
1420 :meth:`report_start`, :meth:`report_success`,
1421 :meth:`report_unexpected_exception`, and :meth:`report_failure`.
1423 The optional keyword argument *checker* specifies the :class:`OutputChecker`
1424 object (or drop-in replacement) that should be used to compare the expected
1425 outputs to the actual outputs of doctest examples.
1427 The optional keyword argument *verbose* controls the :class:`DocTestRunner`'s
1428 verbosity. If *verbose* is ``True``, then information is printed about each
1429 example, as it is run. If *verbose* is ``False``, then only failures are
1430 printed. If *verbose* is unspecified, or ``None``, then verbose output is used
1431 iff the command-line switch :option:`-v` is used.
1433 The optional keyword argument *optionflags* can be used to control how the test
1434 runner compares expected output to actual output, and how it displays failures.
1435 For more information, see section :ref:`doctest-options`.
1437 .. versionadded:: 2.4
1439 :class:`DocTestParser` defines the following methods:
1442 .. method:: report_start(out, test, example)
1444 Report that the test runner is about to process the given example. This method
1445 is provided to allow subclasses of :class:`DocTestRunner` to customize their
1446 output; it should not be called directly.
1448 *example* is the example about to be processed. *test* is the test
1449 *containing example*. *out* is the output function that was passed to
1450 :meth:`DocTestRunner.run`.
1453 .. method:: report_success(out, test, example, got)
1455 Report that the given example ran successfully. This method is provided to
1456 allow subclasses of :class:`DocTestRunner` to customize their output; it
1457 should not be called directly.
1459 *example* is the example about to be processed. *got* is the actual output
1460 from the example. *test* is the test containing *example*. *out* is the
1461 output function that was passed to :meth:`DocTestRunner.run`.
1464 .. method:: report_failure(out, test, example, got)
1466 Report that the given example failed. This method is provided to allow
1467 subclasses of :class:`DocTestRunner` to customize their output; it should not
1470 *example* is the example about to be processed. *got* is the actual output
1471 from the example. *test* is the test containing *example*. *out* is the
1472 output function that was passed to :meth:`DocTestRunner.run`.
1475 .. method:: report_unexpected_exception(out, test, example, exc_info)
1477 Report that the given example raised an unexpected exception. This method is
1478 provided to allow subclasses of :class:`DocTestRunner` to customize their
1479 output; it should not be called directly.
1481 *example* is the example about to be processed. *exc_info* is a tuple
1482 containing information about the unexpected exception (as returned by
1483 :func:`sys.exc_info`). *test* is the test containing *example*. *out* is the
1484 output function that was passed to :meth:`DocTestRunner.run`.
1487 .. method:: run(test[, compileflags][, out][, clear_globs])
1489 Run the examples in *test* (a :class:`DocTest` object), and display the
1490 results using the writer function *out*.
1492 The examples are run in the namespace ``test.globs``. If *clear_globs* is
1493 true (the default), then this namespace will be cleared after the test runs,
1494 to help with garbage collection. If you would like to examine the namespace
1495 after the test completes, then use *clear_globs=False*.
1497 *compileflags* gives the set of flags that should be used by the Python
1498 compiler when running the examples. If not specified, then it will default to
1499 the set of future-import flags that apply to *globs*.
1501 The output of each example is checked using the :class:`DocTestRunner`'s
1502 output checker, and the results are formatted by the
1503 :meth:`DocTestRunner.report_\*` methods.
1506 .. method:: summarize([verbose])
1508 Print a summary of all the test cases that have been run by this DocTestRunner,
1509 and return a :term:`named tuple` ``TestResults(failed, attempted)``.
1511 The optional *verbose* argument controls how detailed the summary is. If the
1512 verbosity is not specified, then the :class:`DocTestRunner`'s verbosity is
1515 .. versionchanged:: 2.6
1519 .. _doctest-outputchecker:
1521 OutputChecker objects
1522 ^^^^^^^^^^^^^^^^^^^^^
1525 .. class:: OutputChecker()
1527 A class used to check the whether the actual output from a doctest example
1528 matches the expected output. :class:`OutputChecker` defines two methods:
1529 :meth:`check_output`, which compares a given pair of outputs, and returns true
1530 if they match; and :meth:`output_difference`, which returns a string describing
1531 the differences between two outputs.
1533 .. versionadded:: 2.4
1535 :class:`OutputChecker` defines the following methods:
1538 .. method:: check_output(want, got, optionflags)
1540 Return ``True`` iff the actual output from an example (*got*) matches the
1541 expected output (*want*). These strings are always considered to match if
1542 they are identical; but depending on what option flags the test runner is
1543 using, several non-exact match types are also possible. See section
1544 :ref:`doctest-options` for more information about option flags.
1547 .. method:: output_difference(example, got, optionflags)
1549 Return a string describing the differences between the expected output for a
1550 given example (*example*) and the actual output (*got*). *optionflags* is the
1551 set of option flags used to compare *want* and *got*.
1554 .. _doctest-debugging:
1559 Doctest provides several mechanisms for debugging doctest examples:
1561 * Several functions convert doctests to executable Python programs, which can be
1562 run under the Python debugger, :mod:`pdb`.
1564 * The :class:`DebugRunner` class is a subclass of :class:`DocTestRunner` that
1565 raises an exception for the first failing example, containing information about
1566 that example. This information can be used to perform post-mortem debugging on
1569 * The :mod:`unittest` cases generated by :func:`DocTestSuite` support the
1570 :meth:`debug` method defined by :class:`unittest.TestCase`.
1572 * You can add a call to :func:`pdb.set_trace` in a doctest example, and you'll
1573 drop into the Python debugger when that line is executed. Then you can inspect
1574 current values of variables, and so on. For example, suppose :file:`a.py`
1575 contains just this module docstring::
1582 ... import pdb; pdb.set_trace()
1587 Then an interactive Python session may look like this::
1589 >>> import a, doctest
1590 >>> doctest.testmod(a)
1592 > <doctest a[1]>(3)g()->None
1593 -> import pdb; pdb.set_trace()
1597 3 -> import pdb; pdb.set_trace()
1603 > <doctest a[0]>(2)f()->None
1613 > <doctest a[2]>(1)?()->None
1619 .. versionchanged:: 2.4
1620 The ability to use :func:`pdb.set_trace` usefully inside doctests was added.
1622 Functions that convert doctests to Python code, and possibly run the synthesized
1623 code under the debugger:
1626 .. function:: script_from_examples(s)
1628 Convert text with examples to a script.
1630 Argument *s* is a string containing doctest examples. The string is converted
1631 to a Python script, where doctest examples in *s* are converted to regular code,
1632 and everything else is converted to Python comments. The generated script is
1633 returned as a string. For example, ::
1636 print doctest.script_from_examples(r"""
1637 Set x and y to 1 and 2.
1647 # Set x and y to 1 and 2.
1655 This function is used internally by other functions (see below), but can also be
1656 useful when you want to transform an interactive Python session into a Python
1659 .. versionadded:: 2.4
1662 .. function:: testsource(module, name)
1664 Convert the doctest for an object to a script.
1666 Argument *module* is a module object, or dotted name of a module, containing the
1667 object whose doctests are of interest. Argument *name* is the name (within the
1668 module) of the object with the doctests of interest. The result is a string,
1669 containing the object's docstring converted to a Python script, as described for
1670 :func:`script_from_examples` above. For example, if module :file:`a.py`
1671 contains a top-level function :func:`f`, then ::
1674 print doctest.testsource(a, "a.f")
1676 prints a script version of function :func:`f`'s docstring, with doctests
1677 converted to code, and the rest placed in comments.
1679 .. versionadded:: 2.3
1682 .. function:: debug(module, name[, pm])
1684 Debug the doctests for an object.
1686 The *module* and *name* arguments are the same as for function
1687 :func:`testsource` above. The synthesized Python script for the named object's
1688 docstring is written to a temporary file, and then that file is run under the
1689 control of the Python debugger, :mod:`pdb`.
1691 A shallow copy of ``module.__dict__`` is used for both local and global
1694 Optional argument *pm* controls whether post-mortem debugging is used. If *pm*
1695 has a true value, the script file is run directly, and the debugger gets
1696 involved only if the script terminates via raising an unhandled exception. If
1697 it does, then post-mortem debugging is invoked, via :func:`pdb.post_mortem`,
1698 passing the traceback object from the unhandled exception. If *pm* is not
1699 specified, or is false, the script is run under the debugger from the start, via
1700 passing an appropriate :func:`execfile` call to :func:`pdb.run`.
1702 .. versionadded:: 2.3
1704 .. versionchanged:: 2.4
1705 The *pm* argument was added.
1708 .. function:: debug_src(src[, pm][, globs])
1710 Debug the doctests in a string.
1712 This is like function :func:`debug` above, except that a string containing
1713 doctest examples is specified directly, via the *src* argument.
1715 Optional argument *pm* has the same meaning as in function :func:`debug` above.
1717 Optional argument *globs* gives a dictionary to use as both local and global
1718 execution context. If not specified, or ``None``, an empty dictionary is used.
1719 If specified, a shallow copy of the dictionary is used.
1721 .. versionadded:: 2.4
1723 The :class:`DebugRunner` class, and the special exceptions it may raise, are of
1724 most interest to testing framework authors, and will only be sketched here. See
1725 the source code, and especially :class:`DebugRunner`'s docstring (which is a
1726 doctest!) for more details:
1729 .. class:: DebugRunner([checker][, verbose][, optionflags])
1731 A subclass of :class:`DocTestRunner` that raises an exception as soon as a
1732 failure is encountered. If an unexpected exception occurs, an
1733 :exc:`UnexpectedException` exception is raised, containing the test, the
1734 example, and the original exception. If the output doesn't match, then a
1735 :exc:`DocTestFailure` exception is raised, containing the test, the example, and
1738 For information about the constructor parameters and methods, see the
1739 documentation for :class:`DocTestRunner` in section :ref:`doctest-advanced-api`.
1741 There are two exceptions that may be raised by :class:`DebugRunner` instances:
1744 .. exception:: DocTestFailure(test, example, got)
1746 An exception thrown by :class:`DocTestRunner` to signal that a doctest example's
1747 actual output did not match its expected output. The constructor arguments are
1748 used to initialize the member variables of the same names.
1750 :exc:`DocTestFailure` defines the following member variables:
1753 .. attribute:: DocTestFailure.test
1755 The :class:`DocTest` object that was being run when the example failed.
1758 .. attribute:: DocTestFailure.example
1760 The :class:`Example` that failed.
1763 .. attribute:: DocTestFailure.got
1765 The example's actual output.
1768 .. exception:: UnexpectedException(test, example, exc_info)
1770 An exception thrown by :class:`DocTestRunner` to signal that a doctest example
1771 raised an unexpected exception. The constructor arguments are used to
1772 initialize the member variables of the same names.
1774 :exc:`UnexpectedException` defines the following member variables:
1777 .. attribute:: UnexpectedException.test
1779 The :class:`DocTest` object that was being run when the example failed.
1782 .. attribute:: UnexpectedException.example
1784 The :class:`Example` that failed.
1787 .. attribute:: UnexpectedException.exc_info
1789 A tuple containing information about the unexpected exception, as returned by
1790 :func:`sys.exc_info`.
1793 .. _doctest-soapbox:
1798 As mentioned in the introduction, :mod:`doctest` has grown to have three primary
1801 #. Checking examples in docstrings.
1803 #. Regression testing.
1805 #. Executable documentation / literate testing.
1807 These uses have different requirements, and it is important to distinguish them.
1808 In particular, filling your docstrings with obscure test cases makes for bad
1811 When writing a docstring, choose docstring examples with care. There's an art to
1812 this that needs to be learned---it may not be natural at first. Examples should
1813 add genuine value to the documentation. A good example can often be worth many
1814 words. If done with care, the examples will be invaluable for your users, and
1815 will pay back the time it takes to collect them many times over as the years go
1816 by and things change. I'm still amazed at how often one of my :mod:`doctest`
1817 examples stops working after a "harmless" change.
1819 Doctest also makes an excellent tool for regression testing, especially if you
1820 don't skimp on explanatory text. By interleaving prose and examples, it becomes
1821 much easier to keep track of what's actually being tested, and why. When a test
1822 fails, good prose can make it much easier to figure out what the problem is, and
1823 how it should be fixed. It's true that you could write extensive comments in
1824 code-based testing, but few programmers do. Many have found that using doctest
1825 approaches instead leads to much clearer tests. Perhaps this is simply because
1826 doctest makes writing prose a little easier than writing code, while writing
1827 comments in code is a little harder. I think it goes deeper than just that:
1828 the natural attitude when writing a doctest-based test is that you want to
1829 explain the fine points of your software, and illustrate them with examples.
1830 This in turn naturally leads to test files that start with the simplest
1831 features, and logically progress to complications and edge cases. A coherent
1832 narrative is the result, instead of a collection of isolated functions that test
1833 isolated bits of functionality seemingly at random. It's a different attitude,
1834 and produces different results, blurring the distinction between testing and
1837 Regression testing is best confined to dedicated objects or files. There are
1838 several options for organizing tests:
1840 * Write text files containing test cases as interactive examples, and test the
1841 files using :func:`testfile` or :func:`DocFileSuite`. This is recommended,
1842 although is easiest to do for new projects, designed from the start to use
1845 * Define functions named ``_regrtest_topic`` that consist of single docstrings,
1846 containing test cases for the named topics. These functions can be included in
1847 the same file as the module, or separated out into a separate test file.
1849 * Define a ``__test__`` dictionary mapping from regression test topics to
1850 docstrings containing test cases.
1852 .. rubric:: Footnotes
1854 .. [#] Examples containing both expected output and an exception are not supported.
1855 Trying to guess where one ends and the other begins is too error-prone, and that
1856 also makes for a confusing test.