Lib/test/regrtest.py

   1 #! /usr/bin/env python
   2
   3 """Regression test.
   4
   5 This will find all modules whose name is "test_*" in the test
   6 directory, and run them.  Various command line options provide
   7 additional facilities.
   8
   9 Command line options:
  10
  11 -v: verbose    -- run tests in verbose mode with output to stdout
  12 -w: verbose2   -- re-run failed tests in verbose mode
  13 -q: quiet      -- don't print anything except if a test fails
  14 -x: exclude    -- arguments are tests to *exclude*
  15 -s: single     -- run only a single test (see below)
  16 -S: slow       -- print the slowest 10 tests
  17 -r: random     -- randomize test execution order
  18 -f: fromfile   -- read names of tests to run from a file (see below)
  19 -l: findleaks  -- if GC is available detect tests that leak memory
  20 -u: use        -- specify which special resource intensive tests to run
  21 -h: help       -- print this text and exit
  22 -t: threshold  -- call gc.set_threshold(N)
  23 -T: coverage   -- turn on code coverage using the trace module
  24 -D: coverdir   -- Directory where coverage files are put
  25 -N: nocoverdir -- Put coverage files alongside modules
  26 -L: runleaks   -- run the leaks(1) command just before exit
  27 -R: huntrleaks -- search for reference leaks (needs debug build, v. slow)
  28 -M: memlimit   -- run very large memory-consuming tests
  29
  30 If non-option arguments are present, they are names for tests to run,
  31 unless -x is given, in which case they are names for tests not to run.
  32 If no test names are given, all tests are run.
  33
  34 -T turns on code coverage tracing with the trace module.
  35
  36 -D specifies the directory where coverage files are put.
  37
  38 -N Put coverage files alongside modules.
  39
  40 -s means to run only a single test and exit.  This is useful when
  41 doing memory analysis on the Python interpreter (which tend to consume
  42 too many resources to run the full regression test non-stop).  The
  43 file /tmp/pynexttest is read to find the next test to run.  If this
  44 file is missing, the first test_*.py file in testdir or on the command
  45 line is used.  (actually tempfile.gettempdir() is used instead of
  46 /tmp).
  47
  48 -f reads the names of tests from the file given as f's argument, one
  49 or more test names per line.  Whitespace is ignored.  Blank lines and
  50 lines beginning with '#' are ignored.  This is especially useful for
  51 whittling down failures involving interactions among tests.
  52
  53 -L causes the leaks(1) command to be run just before exit if it exists.
  54 leaks(1) is available on Mac OS X and presumably on some other
  55 FreeBSD-derived systems.
  56
  57 -R runs each test several times and examines sys.gettotalrefcount() to
  58 see if the test appears to be leaking references.  The argument should
  59 be of the form stab:run:fname where 'stab' is the number of times the
  60 test is run to let gettotalrefcount settle down, 'run' is the number
  61 of times further it is run and 'fname' is the name of the file the
  62 reports are written to.  These parameters all have defaults (5, 4 and
  63 "reflog.txt" respectively), so the minimal invocation is '-R ::'.
  64
  65 -M runs tests that require an exorbitant amount of memory. These tests
  66 typically try to ascertain containers keep working when containing more than
  67 2 billion objects, which only works on 64-bit systems. There are also some
  68 tests that try to exhaust the address space of the process, which only makes
  69 sense on 32-bit systems with at least 2Gb of memory. The passed-in memlimit,
  70 which is a string in the form of '2.5Gb', determines howmuch memory the
  71 tests will limit themselves to (but they may go slightly over.) The number
  72 shouldn't be more memory than the machine has (including swap memory). You
  73 should also keep in mind that swap memory is generally much, much slower
  74 than RAM, and setting memlimit to all available RAM or higher will heavily
  75 tax the machine. On the other hand, it is no use running these tests with a
  76 limit of less than 2.5Gb, and many require more than 20Gb. Tests that expect
  77 to use more than memlimit memory will be skipped. The big-memory tests
  78 generally run very, very long.
  79
  80 -u is used to specify which special resource intensive tests to run,
  81 such as those requiring large file support or network connectivity.
  82 The argument is a comma-separated list of words indicating the
  83 resources to test.  Currently only the following are defined:
  84
  85     all -       Enable all special resources.
  86
  87     audio -     Tests that use the audio device.  (There are known
  88                 cases of broken audio drivers that can crash Python or
  89                 even the Linux kernel.)
  90
  91     curses -    Tests that use curses and will modify the terminal's
  92                 state and output modes.
  93
  94     lib2to3 -   Run the tests for 2to3 (They take a while.)
  95
  96     largefile - It is okay to run some test that may create huge
  97                 files.  These tests can take a long time and may
  98                 consume >2GB of disk space temporarily.
  99
 100     network -   It is okay to run tests that use external network
 101                 resource, e.g. testing SSL support for sockets.
 102
 103     bsddb -     It is okay to run the bsddb testsuite, which takes
 104                 a long time to complete.
 105
 106     decimal -   Test the decimal module against a large suite that
 107                 verifies compliance with standards.
 108
 109     compiler -  Test the compiler package by compiling all the source
 110                 in the standard library and test suite.  This takes
 111                 a long time.  Enabling this resource also allows
 112                 test_tokenize to verify round-trip lexing on every
 113                 file in the test library.
 114
 115     subprocess  Run all tests for the subprocess module.
 116
 117     urlfetch -  It is okay to download files required on testing.
 118
 119 To enable all resources except one, use '-uall,-<resource>'.  For
 120 example, to run all the tests except for the bsddb tests, give the
 121 option '-uall,-bsddb'.
 122 """
 123
 124 import cStringIO
 125 import getopt
 126 import os
 127 import random
 128 import re
 129 import sys
 130 import time
 131 import traceback
 132 import warnings
 133
 134 # I see no other way to suppress these warnings;
 135 # putting them in test_grammar.py has no effect:
 136 warnings.filterwarnings("ignore", "hex/oct constants", FutureWarning,
 137                         ".*test.test_grammar$")
 138 if sys.maxint > 0x7fffffff:
 139     # Also suppress them in <string>, because for 64-bit platforms,
 140     # that's where test_grammar.py hides them.
 141     warnings.filterwarnings("ignore", "hex/oct constants", FutureWarning,
 142                             "<string>")
 143
 144 # Ignore ImportWarnings that only occur in the source tree,
 145 # (because of modules with the same name as source-directories in Modules/)
 146 for mod in ("ctypes", "gzip", "zipfile", "tarfile", "encodings.zlib_codec",
 147             "test.test_zipimport", "test.test_zlib", "test.test_zipfile",
 148             "test.test_codecs", "test.string_tests"):
 149     warnings.filterwarnings(module=".*%s$" % (mod,),
 150                             action="ignore", category=ImportWarning)
 151
 152 # MacOSX (a.k.a. Darwin) has a default stack size that is too small
 153 # for deeply recursive regular expressions.  We see this as crashes in
 154 # the Python test suite when running test_re.py and test_sre.py.  The
 155 # fix is to set the stack limit to 2048.
 156 # This approach may also be useful for other Unixy platforms that
 157 # suffer from small default stack limits.
 158 if sys.platform == 'darwin':
 159     try:
 160         import resource
 161     except ImportError:
 162         pass
 163     else:
 164         soft, hard = resource.getrlimit(resource.RLIMIT_STACK)
 165         newsoft = min(hard, max(soft, 1024*2048))
 166         resource.setrlimit(resource.RLIMIT_STACK, (newsoft, hard))
 167
 168 from test import test_support
 169
 170 RESOURCE_NAMES = ('audio', 'curses', 'lib2to3', 'largefile', 'network',
 171                   'bsddb', 'decimal', 'compiler', 'subprocess', 'urlfetch')
 172
 173
 174 def usage(code, msg=''):
 175     print __doc__
 176     if msg: print msg
 177     sys.exit(code)
 178
 179
 180 def main(tests=None, testdir=None, verbose=0, quiet=False,
 181          exclude=False, single=False, randomize=False, fromfile=None,
 182          findleaks=False, use_resources=None, trace=False, coverdir='coverage',
 183          runleaks=False, huntrleaks=False, verbose2=False, print_slow=False):
 184     """Execute a test suite.
 185
 186     This also parses command-line options and modifies its behavior
 187     accordingly.
 188
 189     tests -- a list of strings containing test names (optional)
 190     testdir -- the directory in which to look for tests (optional)
 191
 192     Users other than the Python test suite will certainly want to
 193     specify testdir; if it's omitted, the directory containing the
 194     Python test suite is searched for.
 195
 196     If the tests argument is omitted, the tests listed on the
 197     command-line will be used.  If that's empty, too, then all *.py
 198     files beginning with test_ will be used.
 199
 200     The other default arguments (verbose, quiet, exclude,
 201     single, randomize, findleaks, use_resources, trace, coverdir, and
 202     print_slow) allow programmers calling main() directly to set the
 203     values that would normally be set by flags on the command line.
 204     """
 205
 206     test_support.record_original_stdout(sys.stdout)
 207     try:
 208         opts, args = getopt.getopt(sys.argv[1:], 'hvgqxsSrf:lu:t:TD:NLR:wM:',
 209                                    ['help', 'verbose', 'quiet', 'exclude',
 210                                     'single', 'slow', 'random', 'fromfile',
 211                                     'findleaks', 'use=', 'threshold=', 'trace',
 212                                     'coverdir=', 'nocoverdir', 'runleaks',
 213                                     'huntrleaks=', 'verbose2', 'memlimit=',
 214                                     ])
 215     except getopt.error, msg:
 216         usage(2, msg)
 217
 218     # Defaults
 219     if use_resources is None:
 220         use_resources = []
 221     for o, a in opts:
 222         if o in ('-h', '--help'):
 223             usage(0)
 224         elif o in ('-v', '--verbose'):
 225             verbose += 1
 226         elif o in ('-w', '--verbose2'):
 227             verbose2 = True
 228         elif o in ('-q', '--quiet'):
 229             quiet = True;
 230             verbose = 0
 231         elif o in ('-x', '--exclude'):
 232             exclude = True
 233         elif o in ('-s', '--single'):
 234             single = True
 235         elif o in ('-S', '--slow'):
 236             print_slow = True
 237         elif o in ('-r', '--randomize'):
 238             randomize = True
 239         elif o in ('-f', '--fromfile'):
 240             fromfile = a
 241         elif o in ('-l', '--findleaks'):
 242             findleaks = True
 243         elif o in ('-L', '--runleaks'):
 244             runleaks = True
 245         elif o in ('-t', '--threshold'):
 246             import gc
 247             gc.set_threshold(int(a))
 248         elif o in ('-T', '--coverage'):
 249             trace = True
 250         elif o in ('-D', '--coverdir'):
 251             coverdir = os.path.join(os.getcwd(), a)
 252         elif o in ('-N', '--nocoverdir'):
 253             coverdir = None
 254         elif o in ('-R', '--huntrleaks'):
 255             huntrleaks = a.split(':')
 256             if len(huntrleaks) != 3:
 257                 print a, huntrleaks
 258                 usage(2, '-R takes three colon-separated arguments')
 259             if len(huntrleaks[0]) == 0:
 260                 huntrleaks[0] = 5
 261             else:
 262                 huntrleaks[0] = int(huntrleaks[0])
 263             if len(huntrleaks[1]) == 0:
 264                 huntrleaks[1] = 4
 265             else:
 266                 huntrleaks[1] = int(huntrleaks[1])
 267             if len(huntrleaks[2]) == 0:
 268                 huntrleaks[2] = "reflog.txt"
 269         elif o in ('-M', '--memlimit'):
 270             test_support.set_memlimit(a)
 271         elif o in ('-u', '--use'):
 272             u = [x.lower() for x in a.split(',')]
 273             for r in u:
 274                 if r == 'all':
 275                     use_resources[:] = RESOURCE_NAMES
 276                     continue
 277                 remove = False
 278                 if r[0] == '-':
 279                     remove = True
 280                     r = r[1:]
 281                 if r not in RESOURCE_NAMES:
 282                     usage(1, 'Invalid -u/--use option: ' + a)
 283                 if remove:
 284                     if r in use_resources:
 285                         use_resources.remove(r)
 286                 elif r not in use_resources:
 287                     use_resources.append(r)
 288     if single and fromfile:
 289         usage(2, "-s and -f don't go together!")
 290
 291     good = []
 292     bad = []
 293     skipped = []
 294     resource_denieds = []
 295
 296     if findleaks:
 297         try:
 298             import gc
 299         except ImportError:
 300             print 'No GC available, disabling findleaks.'
 301             findleaks = False
 302         else:
 303             # Uncomment the line below to report garbage that is not
 304             # freeable by reference counting alone.  By default only
 305             # garbage that is not collectable by the GC is reported.
 306             #gc.set_debug(gc.DEBUG_SAVEALL)
 307             found_garbage = []
 308
 309     if single:
 310         from tempfile import gettempdir
 311         filename = os.path.join(gettempdir(), 'pynexttest')
 312         try:
 313             fp = open(filename, 'r')
 314             next = fp.read().strip()
 315             tests = [next]
 316             fp.close()
 317         except IOError:
 318             pass
 319
 320     if fromfile:
 321         tests = []
 322         fp = open(fromfile)
 323         for line in fp:
 324             guts = line.split() # assuming no test has whitespace in its name
 325             if guts and not guts[0].startswith('#'):
 326                 tests.extend(guts)
 327         fp.close()
 328
 329     # Strip .py extensions.
 330     if args:
 331         args = map(removepy, args)
 332     if tests:
 333         tests = map(removepy, tests)
 334
 335     stdtests = STDTESTS[:]
 336     nottests = NOTTESTS[:]
 337     if exclude:
 338         for arg in args:
 339             if arg in stdtests:
 340                 stdtests.remove(arg)
 341         nottests[:0] = args
 342         args = []
 343     tests = tests or args or findtests(testdir, stdtests, nottests)
 344     if single:
 345         tests = tests[:1]
 346     if randomize:
 347         random.shuffle(tests)
 348     if trace:
 349         import trace
 350         tracer = trace.Trace(ignoredirs=[sys.prefix, sys.exec_prefix],
 351                              trace=False, count=True)
 352     test_times = []
 353     test_support.verbose = verbose      # Tell tests to be moderately quiet
 354     test_support.use_resources = use_resources
 355     save_modules = sys.modules.keys()
 356     for test in tests:
 357         if not quiet:
 358             print test
 359             sys.stdout.flush()
 360         if trace:
 361             # If we're tracing code coverage, then we don't exit with status
 362             # if on a false return value from main.
 363             tracer.runctx('runtest(test, verbose, quiet,'
 364                           '        test_times, testdir)',
 365                           globals=globals(), locals=vars())
 366         else:
 367             try:
 368                 ok = runtest(test, verbose, quiet, test_times,
 369                              testdir, huntrleaks)
 370             except KeyboardInterrupt:
 371                 # print a newline separate from the ^C
 372                 print
 373                 break
 374             except:
 375                 raise
 376             if ok > 0:
 377                 good.append(test)
 378             elif ok == 0:
 379                 bad.append(test)
 380             else:
 381                 skipped.append(test)
 382                 if ok == -2:
 383                     resource_denieds.append(test)
 384         if findleaks:
 385             gc.collect()
 386             if gc.garbage:
 387                 print "Warning: test created", len(gc.garbage),
 388                 print "uncollectable object(s)."
 389                 # move the uncollectable objects somewhere so we don't see
 390                 # them again
 391                 found_garbage.extend(gc.garbage)
 392                 del gc.garbage[:]
 393         # Unload the newly imported modules (best effort finalization)
 394         for module in sys.modules.keys():
 395             if module not in save_modules and module.startswith("test."):
 396                 test_support.unload(module)
 397
 398     # The lists won't be sorted if running with -r
 399     good.sort()
 400     bad.sort()
 401     skipped.sort()
 402
 403     if good and not quiet:
 404         if not bad and not skipped and len(good) > 1:
 405             print "All",
 406         print count(len(good), "test"), "OK."
 407     if print_slow:
 408         test_times.sort(reverse=True)
 409         print "10 slowest tests:"
 410         for time, test in test_times[:10]:
 411             print "%s: %.1fs" % (test, time)
 412     if bad:
 413         print count(len(bad), "test"), "failed:"
 414         printlist(bad)
 415     if skipped and not quiet:
 416         print count(len(skipped), "test"), "skipped:"
 417         printlist(skipped)
 418
 419         e = _ExpectedSkips()
 420         plat = sys.platform
 421         if e.isvalid():
 422             surprise = set(skipped) - e.getexpected() - set(resource_denieds)
 423             if surprise:
 424                 print count(len(surprise), "skip"), \
 425                       "unexpected on", plat + ":"
 426                 printlist(surprise)
 427             else:
 428                 print "Those skips are all expected on", plat + "."
 429         else:
 430             print "Ask someone to teach regrtest.py about which tests are"
 431             print "expected to get skipped on", plat + "."
 432
 433     if verbose2 and bad:
 434         print "Re-running failed tests in verbose mode"
 435         for test in bad:
 436             print "Re-running test %r in verbose mode" % test
 437             sys.stdout.flush()
 438             try:
 439                 test_support.verbose = True
 440                 ok = runtest(test, True, quiet, test_times, testdir,
 441                              huntrleaks)
 442             except KeyboardInterrupt:
 443                 # print a newline separate from the ^C
 444                 print
 445                 break
 446             except:
 447                 raise
 448
 449     if single:
 450         alltests = findtests(testdir, stdtests, nottests)
 451         for i in range(len(alltests)):
 452             if tests[0] == alltests[i]:
 453                 if i == len(alltests) - 1:
 454                     os.unlink(filename)
 455                 else:
 456                     fp = open(filename, 'w')
 457                     fp.write(alltests[i+1] + '\n')
 458                     fp.close()
 459                 break
 460         else:
 461             os.unlink(filename)
 462
 463     if trace:
 464         r = tracer.results()
 465         r.write_results(show_missing=True, summary=True, coverdir=coverdir)
 466
 467     if runleaks:
 468         os.system("leaks %d" % os.getpid())
 469
 470     sys.exit(len(bad) > 0)
 471
 472
 473 STDTESTS = [
 474     'test_grammar',
 475     'test_opcodes',
 476     'test_dict',
 477     'test_builtin',
 478     'test_exceptions',
 479     'test_types',
 480     'test_unittest',
 481     'test_doctest',
 482     'test_doctest2',
 483    ]
 484
 485 NOTTESTS = [
 486     'test_support',
 487     'test_future1',
 488     'test_future2',
 489     ]
 490
 491 def findtests(testdir=None, stdtests=STDTESTS, nottests=NOTTESTS):
 492     """Return a list of all applicable test modules."""
 493     if not testdir: testdir = findtestdir()
 494     names = os.listdir(testdir)
 495     tests = []
 496     for name in names:
 497         if name[:5] == "test_" and name[-3:] == os.extsep+"py":
 498             modname = name[:-3]
 499             if modname not in stdtests and modname not in nottests:
 500                 tests.append(modname)
 501     tests.sort()
 502     return stdtests + tests
 503
 504 def runtest(test, verbose, quiet, test_times,
 505             testdir=None, huntrleaks=False):
 506     """Run a single test.
 507
 508     test -- the name of the test
 509     verbose -- if true, print more messages
 510     quiet -- if true, don't print 'skipped' messages (probably redundant)
 511     test_times -- a list of (time, test_name) pairs
 512     testdir -- test directory
 513     huntrleaks -- run multiple times to test for leaks; requires a debug
 514                   build; a triple corresponding to -R's three arguments
 515     Return:
 516         -2  test skipped because resource denied
 517         -1  test skipped for some other reason
 518          0  test failed
 519          1  test passed
 520     """
 521
 522     try:
 523         return runtest_inner(test, verbose, quiet, test_times,
 524                              testdir, huntrleaks)
 525     finally:
 526         cleanup_test_droppings(test, verbose)
 527
 528 def runtest_inner(test, verbose, quiet, test_times,
 529                   testdir=None, huntrleaks=False):
 530     test_support.unload(test)
 531     if not testdir:
 532         testdir = findtestdir()
 533     if verbose:
 534         capture_stdout = None
 535     else:
 536         capture_stdout = cStringIO.StringIO()
 537
 538     try:
 539         save_stdout = sys.stdout
 540         try:
 541             if capture_stdout:
 542                 sys.stdout = capture_stdout
 543             if test.startswith('test.'):
 544                 abstest = test
 545             else:
 546                 # Always import it from the test package
 547                 abstest = 'test.' + test
 548             start_time = time.time()
 549             the_package = __import__(abstest, globals(), locals(), [])
 550             the_module = getattr(the_package, test)
 551             # Old tests run to completion simply as a side-effect of
 552             # being imported.  For tests based on unittest or doctest,
 553             # explicitly invoke their test_main() function (if it exists).
 554             indirect_test = getattr(the_module, "test_main", None)
 555             if indirect_test is not None:
 556                 indirect_test()
 557             if huntrleaks:
 558                 dash_R(the_module, test, indirect_test, huntrleaks)
 559             test_time = time.time() - start_time
 560             test_times.append((test_time, test))
 561         finally:
 562             sys.stdout = save_stdout
 563     except test_support.ResourceDenied, msg:
 564         if not quiet:
 565             print test, "skipped --", msg
 566             sys.stdout.flush()
 567         return -2
 568     except (ImportError, test_support.TestSkipped), msg:
 569         if not quiet:
 570             print test, "skipped --", msg
 571             sys.stdout.flush()
 572         return -1
 573     except KeyboardInterrupt:
 574         raise
 575     except test_support.TestFailed, msg:
 576         print "test", test, "failed --", msg
 577         sys.stdout.flush()
 578         return 0
 579     except:
 580         type, value = sys.exc_info()[:2]
 581         print "test", test, "crashed --", str(type) + ":", value
 582         sys.stdout.flush()
 583         if verbose:
 584             traceback.print_exc(file=sys.stdout)
 585             sys.stdout.flush()
 586         return 0
 587     else:
 588         # Except in verbose mode, tests should not print anything
 589         if verbose or huntrleaks:
 590             return 1
 591         output = capture_stdout.getvalue()
 592         if not output:
 593             return 1
 594         print "test", test, "produced unexpected output:"
 595         print "*" * 70
 596         print output
 597         print "*" * 70
 598         sys.stdout.flush()
 599         return 0
 600
 601 def cleanup_test_droppings(testname, verbose):
 602     import shutil
 603
 604     # Try to clean up junk commonly left behind.  While tests shouldn't leave
 605     # any files or directories behind, when a test fails that can be tedious
 606     # for it to arrange.  The consequences can be especially nasty on Windows,
 607     # since if a test leaves a file open, it cannot be deleted by name (while
 608     # there's nothing we can do about that here either, we can display the
 609     # name of the offending test, which is a real help).
 610     for name in (test_support.TESTFN,
 611                  "db_home",
 612                 ):
 613         if not os.path.exists(name):
 614             continue
 615
 616         if os.path.isdir(name):
 617             kind, nuker = "directory", shutil.rmtree
 618         elif os.path.isfile(name):
 619             kind, nuker = "file", os.unlink
 620         else:
 621             raise SystemError("os.path says %r exists but is neither "
 622                               "directory nor file" % name)
 623
 624         if verbose:
 625             print "%r left behind %s %r" % (testname, kind, name)
 626         try:
 627             nuker(name)
 628         except Exception, msg:
 629             print >> sys.stderr, ("%r left behind %s %r and it couldn't be "
 630                 "removed: %s" % (testname, kind, name, msg))
 631
 632 def dash_R(the_module, test, indirect_test, huntrleaks):
 633     # This code is hackish and inelegant, but it seems to do the job.
 634     import copy_reg, _abcoll, io
 635
 636     if not hasattr(sys, 'gettotalrefcount'):
 637         raise Exception("Tracking reference leaks requires a debug build "
 638                         "of Python")
 639
 640     # Save current values for dash_R_cleanup() to restore.
 641     fs = warnings.filters[:]
 642     ps = copy_reg.dispatch_table.copy()
 643     pic = sys.path_importer_cache.copy()
 644     abcs = {}
 645     modules = _abcoll, io
 646     for abc in [getattr(mod, a) for mod in modules for a in mod.__all__]:
 647         # XXX isinstance(abc, ABCMeta) leads to infinite recursion
 648         if not hasattr(abc, '_abc_registry'):
 649             continue
 650         for obj in abc.__subclasses__() + [abc]:
 651             abcs[obj] = obj._abc_registry.copy()
 652
 653     if indirect_test:
 654         def run_the_test():
 655             indirect_test()
 656     else:
 657         def run_the_test():
 658             reload(the_module)
 659
 660     deltas = []
 661     nwarmup, ntracked, fname = huntrleaks
 662     repcount = nwarmup + ntracked
 663     print >> sys.stderr, "beginning", repcount, "repetitions"
 664     print >> sys.stderr, ("1234567890"*(repcount//10 + 1))[:repcount]
 665     dash_R_cleanup(fs, ps, pic, abcs)
 666     for i in range(repcount):
 667         rc = sys.gettotalrefcount()
 668         run_the_test()
 669         sys.stderr.write('.')
 670         dash_R_cleanup(fs, ps, pic, abcs)
 671         if i >= nwarmup:
 672             deltas.append(sys.gettotalrefcount() - rc - 2)
 673     print >> sys.stderr
 674     if any(deltas):
 675         msg = '%s leaked %s references, sum=%s' % (test, deltas, sum(deltas))
 676         print >> sys.stderr, msg
 677         refrep = open(fname, "a")
 678         print >> refrep, msg
 679         refrep.close()
 680
 681 def dash_R_cleanup(fs, ps, pic, abcs):
 682     import gc, copy_reg
 683     import _strptime, linecache
 684     dircache = test_support.import_module('dircache', deprecated=True)
 685     import urlparse, urllib, urllib2, mimetypes, doctest
 686     import struct, filecmp
 687     from distutils.dir_util import _path_created
 688
 689     # Clear the warnings registry, so they can be displayed again
 690     for mod in sys.modules.values():
 691         if hasattr(mod, '__warningregistry__'):
 692             del mod.__warningregistry__
 693
 694     # Restore some original values.
 695     warnings.filters[:] = fs
 696     copy_reg.dispatch_table.clear()
 697     copy_reg.dispatch_table.update(ps)
 698     sys.path_importer_cache.clear()
 699     sys.path_importer_cache.update(pic)
 700
 701     # clear type cache
 702     sys._clear_type_cache()
 703
 704     # Clear ABC registries, restoring previously saved ABC registries.
 705     for abc, registry in abcs.items():
 706         abc._abc_registry = registry.copy()
 707         abc._abc_cache.clear()
 708         abc._abc_negative_cache.clear()
 709
 710     # Clear assorted module caches.
 711     _path_created.clear()
 712     re.purge()
 713     _strptime._regex_cache.clear()
 714     urlparse.clear_cache()
 715     urllib.urlcleanup()
 716     urllib2.install_opener(None)
 717     dircache.reset()
 718     linecache.clearcache()
 719     mimetypes._default_mime_types()
 720     filecmp._cache.clear()
 721     struct._clearcache()
 722     doctest.master = None
 723
 724     # Collect cyclic trash.
 725     gc.collect()
 726
 727 def findtestdir():
 728     if __name__ == '__main__':
 729         file = sys.argv[0]
 730     else:
 731         file = __file__
 732     testdir = os.path.dirname(file) or os.curdir
 733     return testdir
 734
 735 def removepy(name):
 736     if name.endswith(os.extsep + "py"):
 737         name = name[:-3]
 738     return name
 739
 740 def count(n, word):
 741     if n == 1:
 742         return "%d %s" % (n, word)
 743     else:
 744         return "%d %ss" % (n, word)
 745
 746 def printlist(x, width=70, indent=4):
 747     """Print the elements of iterable x to stdout.
 748
 749     Optional arg width (default 70) is the maximum line length.
 750     Optional arg indent (default 4) is the number of blanks with which to
 751     begin each line.
 752     """
 753
 754     from textwrap import fill
 755     blanks = ' ' * indent
 756     print fill(' '.join(map(str, x)), width,
 757                initial_indent=blanks, subsequent_indent=blanks)
 758
 759 # Map sys.platform to a string containing the basenames of tests
 760 # expected to be skipped on that platform.
 761 #
 762 # Special cases:
 763 #     test_pep277
 764 #         The _ExpectedSkips constructor adds this to the set of expected
 765 #         skips if not os.path.supports_unicode_filenames.
 766 #     test_socket_ssl
 767 #         Controlled by test_socket_ssl.skip_expected.  Requires the network
 768 #         resource, and a socket module with ssl support.
 769 #     test_timeout
 770 #         Controlled by test_timeout.skip_expected.  Requires the network
 771 #         resource and a socket module.
 772 #
 773 # Tests that are expected to be skipped everywhere except on one platform
 774 # are also handled separately.
 775
 776 _expectations = {
 777     'win32':
 778         """
 779         test__locale
 780         test_bsddb185
 781         test_bsddb3
 782         test_commands
 783         test_crypt
 784         test_curses
 785         test_dbm
 786         test_dl
 787         test_fcntl
 788         test_fork1
 789         test_epoll
 790         test_gdbm
 791         test_grp
 792         test_ioctl
 793         test_largefile
 794         test_kqueue
 795         test_mhlib
 796         test_openpty
 797         test_ossaudiodev
 798         test_pipes
 799         test_poll
 800         test_posix
 801         test_pty
 802         test_pwd
 803         test_resource
 804         test_signal
 805         test_threadsignals
 806         test_timing
 807         test_wait3
 808         test_wait4
 809         """,
 810     'linux2':
 811         """
 812         test_bsddb185
 813         test_curses
 814         test_dl
 815         test_largefile
 816         test_kqueue
 817         test_ossaudiodev
 818         """,
 819    'mac':
 820         """
 821         test_atexit
 822         test_bsddb
 823         test_bsddb185
 824         test_bsddb3
 825         test_bz2
 826         test_commands
 827         test_crypt
 828         test_curses
 829         test_dbm
 830         test_dl
 831         test_fcntl
 832         test_fork1
 833         test_epoll
 834         test_grp
 835         test_ioctl
 836         test_largefile
 837         test_locale
 838         test_kqueue
 839         test_mmap
 840         test_openpty
 841         test_ossaudiodev
 842         test_poll
 843         test_popen
 844         test_popen2
 845         test_posix
 846         test_pty
 847         test_pwd
 848         test_resource
 849         test_signal
 850         test_sundry
 851         test_tarfile
 852         test_timing
 853         """,
 854     'unixware7':
 855         """
 856         test_bsddb
 857         test_bsddb185
 858         test_dl
 859         test_epoll
 860         test_largefile
 861         test_kqueue
 862         test_minidom
 863         test_openpty
 864         test_pyexpat
 865         test_sax
 866         test_sundry
 867         """,
 868     'openunix8':
 869         """
 870         test_bsddb
 871         test_bsddb185
 872         test_dl
 873         test_epoll
 874         test_largefile
 875         test_kqueue
 876         test_minidom
 877         test_openpty
 878         test_pyexpat
 879         test_sax
 880         test_sundry
 881         """,
 882     'sco_sv3':
 883         """
 884         test_asynchat
 885         test_bsddb
 886         test_bsddb185
 887         test_dl
 888         test_fork1
 889         test_epoll
 890         test_gettext
 891         test_largefile
 892         test_locale
 893         test_kqueue
 894         test_minidom
 895         test_openpty
 896         test_pyexpat
 897         test_queue
 898         test_sax
 899         test_sundry
 900         test_thread
 901         test_threaded_import
 902         test_threadedtempfile
 903         test_threading
 904         """,
 905     'riscos':
 906         """
 907         test_asynchat
 908         test_atexit
 909         test_bsddb
 910         test_bsddb185
 911         test_bsddb3
 912         test_commands
 913         test_crypt
 914         test_dbm
 915         test_dl
 916         test_fcntl
 917         test_fork1
 918         test_epoll
 919         test_gdbm
 920         test_grp
 921         test_largefile
 922         test_locale
 923         test_kqueue
 924         test_mmap
 925         test_openpty
 926         test_poll
 927         test_popen2
 928         test_pty
 929         test_pwd
 930         test_strop
 931         test_sundry
 932         test_thread
 933         test_threaded_import
 934         test_threadedtempfile
 935         test_threading
 936         test_timing
 937         """,
 938     'darwin':
 939         """
 940         test__locale
 941         test_bsddb
 942         test_bsddb3
 943         test_curses
 944         test_epoll
 945         test_gdbm
 946         test_largefile
 947         test_locale
 948         test_kqueue
 949         test_minidom
 950         test_ossaudiodev
 951         test_poll
 952         """,
 953     'sunos5':
 954         """
 955         test_bsddb
 956         test_bsddb185
 957         test_curses
 958         test_dbm
 959         test_epoll
 960         test_kqueue
 961         test_gdbm
 962         test_gzip
 963         test_openpty
 964         test_zipfile
 965         test_zlib
 966         """,
 967     'hp-ux11':
 968         """
 969         test_bsddb
 970         test_bsddb185
 971         test_curses
 972         test_dl
 973         test_epoll
 974         test_gdbm
 975         test_gzip
 976         test_largefile
 977         test_locale
 978         test_kqueue
 979         test_minidom
 980         test_openpty
 981         test_pyexpat
 982         test_sax
 983         test_zipfile
 984         test_zlib
 985         """,
 986     'atheos':
 987         """
 988         test_bsddb185
 989         test_curses
 990         test_dl
 991         test_gdbm
 992         test_epoll
 993         test_largefile
 994         test_locale
 995         test_kqueue
 996         test_mhlib
 997         test_mmap
 998         test_poll
 999         test_popen2
1000         test_resource
1001         """,
1002     'cygwin':
1003         """
1004         test_bsddb185
1005         test_bsddb3
1006         test_curses
1007         test_dbm
1008         test_epoll
1009         test_ioctl
1010         test_kqueue
1011         test_largefile
1012         test_locale
1013         test_ossaudiodev
1014         test_socketserver
1015         """,
1016     'os2emx':
1017         """
1018         test_audioop
1019         test_bsddb185
1020         test_bsddb3
1021         test_commands
1022         test_curses
1023         test_dl
1024         test_epoll
1025         test_kqueue
1026         test_largefile
1027         test_mhlib
1028         test_mmap
1029         test_openpty
1030         test_ossaudiodev
1031         test_pty
1032         test_resource
1033         test_signal
1034         """,
1035     'freebsd4':
1036         """
1037         test_bsddb
1038         test_bsddb3
1039         test_epoll
1040         test_gdbm
1041         test_locale
1042         test_ossaudiodev
1043         test_pep277
1044         test_pty
1045         test_socket_ssl
1046         test_socketserver
1047         test_tcl
1048         test_timeout
1049         test_urllibnet
1050         """,
1051     'aix5':
1052         """
1053         test_bsddb
1054         test_bsddb185
1055         test_bsddb3
1056         test_bz2
1057         test_dl
1058         test_epoll
1059         test_gdbm
1060         test_gzip
1061         test_kqueue
1062         test_ossaudiodev
1063         test_tcl
1064         test_zipimport
1065         test_zlib
1066         """,
1067     'openbsd3':
1068         """
1069         test_bsddb
1070         test_bsddb3
1071         test_ctypes
1072         test_dl
1073         test_epoll
1074         test_gdbm
1075         test_locale
1076         test_normalization
1077         test_ossaudiodev
1078         test_pep277
1079         test_tcl
1080         """,
1081     'netbsd3':
1082         """
1083         test_bsddb
1084         test_bsddb185
1085         test_bsddb3
1086         test_ctypes
1087         test_curses
1088         test_dl
1089         test_epoll
1090         test_gdbm
1091         test_locale
1092         test_ossaudiodev
1093         test_pep277
1094         test_tcl
1095         """,
1096 }
1097 _expectations['freebsd5'] = _expectations['freebsd4']
1098 _expectations['freebsd6'] = _expectations['freebsd4']
1099 _expectations['freebsd7'] = _expectations['freebsd4']
1100 _expectations['freebsd8'] = _expectations['freebsd4']
1101
1102 class _ExpectedSkips:
1103     def __init__(self):
1104         import os.path
1105         from test import test_timeout
1106
1107         self.valid = False
1108         if sys.platform in _expectations:
1109             s = _expectations[sys.platform]
1110             self.expected = set(s.split())
1111
1112             # expected to be skipped on every platform, even Linux
1113             self.expected.add('test_linuxaudiodev')
1114
1115             if not os.path.supports_unicode_filenames:
1116                 self.expected.add('test_pep277')
1117
1118             try:
1119                 from test import test_socket_ssl
1120             except ImportError:
1121                 pass
1122             else:
1123                 if test_socket_ssl.skip_expected:
1124                     self.expected.add('test_socket_ssl')
1125
1126             if test_timeout.skip_expected:
1127                 self.expected.add('test_timeout')
1128
1129             if sys.maxint == 9223372036854775807L:
1130                 self.expected.add('test_imageop')
1131
1132             if not sys.platform in ("mac", "darwin"):
1133                 MAC_ONLY = ["test_macos", "test_macostools", "test_aepack",
1134                             "test_plistlib", "test_scriptpackages",
1135                             "test_applesingle"]
1136                 for skip in MAC_ONLY:
1137                     self.expected.add(skip)
1138             elif len(u'\0'.encode('unicode-internal')) == 4:
1139                 self.expected.add("test_macostools")
1140
1141
1142             if sys.platform != "win32":
1143                 # test_sqlite is only reliable on Windows where the library
1144                 # is distributed with Python
1145                 WIN_ONLY = ["test_unicode_file", "test_winreg",
1146                             "test_winsound", "test_startfile",
1147                             "test_sqlite"]
1148                 for skip in WIN_ONLY:
1149                     self.expected.add(skip)
1150
1151             if sys.platform != 'irix':
1152                 IRIX_ONLY = ["test_imageop", "test_al", "test_cd", "test_cl",
1153                              "test_gl", "test_imgfile"]
1154                 for skip in IRIX_ONLY:
1155                     self.expected.add(skip)
1156
1157             if sys.platform != 'sunos5':
1158                 self.expected.add('test_sunaudiodev')
1159                 self.expected.add('test_nis')
1160
1161             if not sys.py3kwarning:
1162                 self.expected.add('test_py3kwarn')
1163
1164             self.valid = True
1165
1166     def isvalid(self):
1167         "Return true iff _ExpectedSkips knows about the current platform."
1168         return self.valid
1169
1170     def getexpected(self):
1171         """Return set of test names we expect to skip on current platform.
1172
1173         self.isvalid() must be true.
1174         """
1175
1176         assert self.isvalid()
1177         return self.expected
1178
1179 if __name__ == '__main__':
1180     # Remove regrtest.py's own directory from the module search path.  This
1181     # prevents relative imports from working, and relative imports will screw
1182     # up the testing framework.  E.g. if both test.test_support and
1183     # test_support are imported, they will not contain the same globals, and
1184     # much of the testing framework relies on the globals in the
1185     # test.test_support module.
1186     mydir = os.path.abspath(os.path.normpath(os.path.dirname(sys.argv[0])))
1187     i = pathlen = len(sys.path)
1188     while i >= 0:
1189         i -= 1
1190         if os.path.abspath(os.path.normpath(sys.path[i])) == mydir:
1191             del sys.path[i]
1192     if len(sys.path) == pathlen:
1193         print 'Could not find %r in sys.path to remove it' % mydir
1194     main()