Lib/test/regrtest.py

   1 #! /usr/bin/env python
   2
   3 """
   4 Usage:
   5
   6 python -m test.regrtest [options] [test_name1 [test_name2 ...]]
   7 python path/to/Lib/test/regrtest.py [options] [test_name1 [test_name2 ...]]
   8
   9
  10 If no arguments or options are provided, finds all files matching
  11 the pattern "test_*" in the Lib/test subdirectory and runs
  12 them in alphabetical order (but see -M and -u, below, for exceptions).
  13
  14 For more rigorous testing, it is useful to use the following
  15 command line:
  16
  17 python -E -tt -Wd -3 -m test.regrtest [options] [test_name1 ...]
  18
  19
  20 Options:
  21
  22 -h/--help       -- print this text and exit
  23
  24 Verbosity
  25
  26 -v/--verbose    -- run tests in verbose mode with output to stdout
  27 -w/--verbose2   -- re-run failed tests in verbose mode
  28 -W/--verbose3   -- re-run failed tests in verbose mode immediately
  29 -q/--quiet      -- no output unless one or more tests fail
  30 -S/--slow       -- print the slowest 10 tests
  31
  32 Selecting tests
  33
  34 -r/--random     -- randomize test execution order (see below)
  35 -f/--fromfile   -- read names of tests to run from a file (see below)
  36 -x/--exclude    -- arguments are tests to *exclude*
  37 -s/--single     -- single step through a set of tests (see below)
  38 -u/--use RES1,RES2,...
  39                 -- specify which special resource intensive tests to run
  40 -M/--memlimit LIMIT
  41                 -- run very large memory-consuming tests
  42
  43 Special runs
  44
  45 -l/--findleaks  -- if GC is available detect tests that leak memory
  46 -L/--runleaks   -- run the leaks(1) command just before exit
  47 -R/--huntrleaks RUNCOUNTS
  48                 -- search for reference leaks (needs debug build, v. slow)
  49 -j/--multiprocess PROCESSES
  50                 -- run PROCESSES processes at once
  51 -T/--coverage   -- turn on code coverage tracing using the trace module
  52 -D/--coverdir DIRECTORY
  53                 -- Directory where coverage files are put
  54 -N/--nocoverdir -- Put coverage files alongside modules
  55 -t/--threshold THRESHOLD
  56                 -- call gc.set_threshold(THRESHOLD)
  57 -F/--forever    -- run the specified tests in a loop, until an error happens
  58
  59
  60 Additional Option Details:
  61
  62 -r randomizes test execution order. You can use --randseed=int to provide a
  63 int seed value for the randomizer; this is useful for reproducing troublesome
  64 test orders.
  65
  66 -s On the first invocation of regrtest using -s, the first test file found
  67 or the first test file given on the command line is run, and the name of
  68 the next test is recorded in a file named pynexttest.  If run from the
  69 Python build directory, pynexttest is located in the 'build' subdirectory,
  70 otherwise it is located in tempfile.gettempdir().  On subsequent runs,
  71 the test in pynexttest is run, and the next test is written to pynexttest.
  72 When the last test has been run, pynexttest is deleted.  In this way it
  73 is possible to single step through the test files.  This is useful when
  74 doing memory analysis on the Python interpreter, which process tends to
  75 consume too many resources to run the full regression test non-stop.
  76
  77 -f reads the names of tests from the file given as f's argument, one
  78 or more test names per line.  Whitespace is ignored.  Blank lines and
  79 lines beginning with '#' are ignored.  This is especially useful for
  80 whittling down failures involving interactions among tests.
  81
  82 -L causes the leaks(1) command to be run just before exit if it exists.
  83 leaks(1) is available on Mac OS X and presumably on some other
  84 FreeBSD-derived systems.
  85
  86 -R runs each test several times and examines sys.gettotalrefcount() to
  87 see if the test appears to be leaking references.  The argument should
  88 be of the form stab:run:fname where 'stab' is the number of times the
  89 test is run to let gettotalrefcount settle down, 'run' is the number
  90 of times further it is run and 'fname' is the name of the file the
  91 reports are written to.  These parameters all have defaults (5, 4 and
  92 "reflog.txt" respectively), and the minimal invocation is '-R :'.
  93
  94 -M runs tests that require an exorbitant amount of memory. These tests
  95 typically try to ascertain containers keep working when containing more than
  96 2 billion objects, which only works on 64-bit systems. There are also some
  97 tests that try to exhaust the address space of the process, which only makes
  98 sense on 32-bit systems with at least 2Gb of memory. The passed-in memlimit,
  99 which is a string in the form of '2.5Gb', determines howmuch memory the
 100 tests will limit themselves to (but they may go slightly over.) The number
 101 shouldn't be more memory than the machine has (including swap memory). You
 102 should also keep in mind that swap memory is generally much, much slower
 103 than RAM, and setting memlimit to all available RAM or higher will heavily
 104 tax the machine. On the other hand, it is no use running these tests with a
 105 limit of less than 2.5Gb, and many require more than 20Gb. Tests that expect
 106 to use more than memlimit memory will be skipped. The big-memory tests
 107 generally run very, very long.
 108
 109 -u is used to specify which special resource intensive tests to run,
 110 such as those requiring large file support or network connectivity.
 111 The argument is a comma-separated list of words indicating the
 112 resources to test.  Currently only the following are defined:
 113
 114     all -       Enable all special resources.
 115
 116     audio -     Tests that use the audio device.  (There are known
 117                 cases of broken audio drivers that can crash Python or
 118                 even the Linux kernel.)
 119
 120     curses -    Tests that use curses and will modify the terminal's
 121                 state and output modes.
 122
 123     largefile - It is okay to run some test that may create huge
 124                 files.  These tests can take a long time and may
 125                 consume >2GB of disk space temporarily.
 126
 127     network -   It is okay to run tests that use external network
 128                 resource, e.g. testing SSL support for sockets.
 129
 130     bsddb -     It is okay to run the bsddb testsuite, which takes
 131                 a long time to complete.
 132
 133     decimal -   Test the decimal module against a large suite that
 134                 verifies compliance with standards.
 135
 136     compiler -  Test the compiler package by compiling all the source
 137                 in the standard library and test suite.  This takes
 138                 a long time.  Enabling this resource also allows
 139                 test_tokenize to verify round-trip lexing on every
 140                 file in the test library.
 141
 142     subprocess  Run all tests for the subprocess module.
 143
 144     urlfetch -  It is okay to download files required on testing.
 145
 146     gui -       Run tests that require a running GUI.
 147
 148     xpickle -   Test pickle and cPickle against Python 2.4, 2.5 and 2.6 to
 149                 test backwards compatibility. These tests take a long time
 150                 to run.
 151
 152 To enable all resources except one, use '-uall,-<resource>'.  For
 153 example, to run all the tests except for the bsddb tests, give the
 154 option '-uall,-bsddb'.
 155 """
 156
 157 import StringIO
 158 import getopt
 159 import json
 160 import os
 161 import random
 162 import re
 163 import sys
 164 import time
 165 import traceback
 166 import warnings
 167 import unittest
 168 import tempfile
 169 import imp
 170 import platform
 171 import sysconfig
 172
 173
 174 # Some times __path__ and __file__ are not absolute (e.g. while running from
 175 # Lib/) and, if we change the CWD to run the tests in a temporary dir, some
 176 # imports might fail.  This affects only the modules imported before os.chdir().
 177 # These modules are searched first in sys.path[0] (so '' -- the CWD) and if
 178 # they are found in the CWD their __file__ and __path__ will be relative (this
 179 # happens before the chdir).  All the modules imported after the chdir, are
 180 # not found in the CWD, and since the other paths in sys.path[1:] are absolute
 181 # (site.py absolutize them), the __file__ and __path__ will be absolute too.
 182 # Therefore it is necessary to absolutize manually the __file__ and __path__ of
 183 # the packages to prevent later imports to fail when the CWD is different.
 184 for module in sys.modules.itervalues():
 185     if hasattr(module, '__path__'):
 186         module.__path__ = [os.path.abspath(path) for path in module.__path__]
 187     if hasattr(module, '__file__'):
 188         module.__file__ = os.path.abspath(module.__file__)
 189
 190
 191 # MacOSX (a.k.a. Darwin) has a default stack size that is too small
 192 # for deeply recursive regular expressions.  We see this as crashes in
 193 # the Python test suite when running test_re.py and test_sre.py.  The
 194 # fix is to set the stack limit to 2048.
 195 # This approach may also be useful for other Unixy platforms that
 196 # suffer from small default stack limits.
 197 if sys.platform == 'darwin':
 198     try:
 199         import resource
 200     except ImportError:
 201         pass
 202     else:
 203         soft, hard = resource.getrlimit(resource.RLIMIT_STACK)
 204         newsoft = min(hard, max(soft, 1024*2048))
 205         resource.setrlimit(resource.RLIMIT_STACK, (newsoft, hard))
 206
 207 # Test result constants.
 208 PASSED = 1
 209 FAILED = 0
 210 ENV_CHANGED = -1
 211 SKIPPED = -2
 212 RESOURCE_DENIED = -3
 213 INTERRUPTED = -4
 214
 215 from test import test_support
 216
 217 RESOURCE_NAMES = ('audio', 'curses', 'largefile', 'network', 'bsddb',
 218                   'decimal', 'compiler', 'subprocess', 'urlfetch', 'gui',
 219                   'xpickle')
 220
 221 TEMPDIR = os.path.abspath(tempfile.gettempdir())
 222
 223
 224 def usage(code, msg=''):
 225     print __doc__
 226     if msg: print msg
 227     sys.exit(code)
 228
 229
 230 def main(tests=None, testdir=None, verbose=0, quiet=False,
 231          exclude=False, single=False, randomize=False, fromfile=None,
 232          findleaks=False, use_resources=None, trace=False, coverdir='coverage',
 233          runleaks=False, huntrleaks=False, verbose2=False, print_slow=False,
 234          random_seed=None, use_mp=None, verbose3=False, forever=False):
 235     """Execute a test suite.
 236
 237     This also parses command-line options and modifies its behavior
 238     accordingly.
 239
 240     tests -- a list of strings containing test names (optional)
 241     testdir -- the directory in which to look for tests (optional)
 242
 243     Users other than the Python test suite will certainly want to
 244     specify testdir; if it's omitted, the directory containing the
 245     Python test suite is searched for.
 246
 247     If the tests argument is omitted, the tests listed on the
 248     command-line will be used.  If that's empty, too, then all *.py
 249     files beginning with test_ will be used.
 250
 251     The other default arguments (verbose, quiet, exclude,
 252     single, randomize, findleaks, use_resources, trace, coverdir,
 253     print_slow, and random_seed) allow programmers calling main()
 254     directly to set the values that would normally be set by flags
 255     on the command line.
 256     """
 257
 258     test_support.record_original_stdout(sys.stdout)
 259     try:
 260         opts, args = getopt.getopt(sys.argv[1:], 'hvqxsSrf:lu:t:TD:NLR:FwWM:j:',
 261             ['help', 'verbose', 'verbose2', 'verbose3', 'quiet',
 262              'exclude', 'single', 'slow', 'random', 'fromfile', 'findleaks',
 263              'use=', 'threshold=', 'trace', 'coverdir=', 'nocoverdir',
 264              'runleaks', 'huntrleaks=', 'memlimit=', 'randseed=',
 265              'multiprocess=', 'slaveargs=', 'forever'])
 266     except getopt.error, msg:
 267         usage(2, msg)
 268
 269     # Defaults
 270     if random_seed is None:
 271         random_seed = random.randrange(10000000)
 272     if use_resources is None:
 273         use_resources = []
 274     for o, a in opts:
 275         if o in ('-h', '--help'):
 276             usage(0)
 277         elif o in ('-v', '--verbose'):
 278             verbose += 1
 279         elif o in ('-w', '--verbose2'):
 280             verbose2 = True
 281         elif o in ('-W', '--verbose3'):
 282             verbose3 = True
 283         elif o in ('-q', '--quiet'):
 284             quiet = True;
 285             verbose = 0
 286         elif o in ('-x', '--exclude'):
 287             exclude = True
 288         elif o in ('-s', '--single'):
 289             single = True
 290         elif o in ('-S', '--slow'):
 291             print_slow = True
 292         elif o in ('-r', '--randomize'):
 293             randomize = True
 294         elif o == '--randseed':
 295             random_seed = int(a)
 296         elif o in ('-f', '--fromfile'):
 297             fromfile = a
 298         elif o in ('-l', '--findleaks'):
 299             findleaks = True
 300         elif o in ('-L', '--runleaks'):
 301             runleaks = True
 302         elif o in ('-t', '--threshold'):
 303             import gc
 304             gc.set_threshold(int(a))
 305         elif o in ('-T', '--coverage'):
 306             trace = True
 307         elif o in ('-D', '--coverdir'):
 308             coverdir = os.path.join(os.getcwd(), a)
 309         elif o in ('-N', '--nocoverdir'):
 310             coverdir = None
 311         elif o in ('-R', '--huntrleaks'):
 312             huntrleaks = a.split(':')
 313             if len(huntrleaks) not in (2, 3):
 314                 print a, huntrleaks
 315                 usage(2, '-R takes 2 or 3 colon-separated arguments')
 316             if not huntrleaks[0]:
 317                 huntrleaks[0] = 5
 318             else:
 319                 huntrleaks[0] = int(huntrleaks[0])
 320             if not huntrleaks[1]:
 321                 huntrleaks[1] = 4
 322             else:
 323                 huntrleaks[1] = int(huntrleaks[1])
 324             if len(huntrleaks) == 2 or not huntrleaks[2]:
 325                 huntrleaks[2:] = ["reflog.txt"]
 326         elif o in ('-M', '--memlimit'):
 327             test_support.set_memlimit(a)
 328         elif o in ('-u', '--use'):
 329             u = [x.lower() for x in a.split(',')]
 330             for r in u:
 331                 if r == 'all':
 332                     use_resources[:] = RESOURCE_NAMES
 333                     continue
 334                 remove = False
 335                 if r[0] == '-':
 336                     remove = True
 337                     r = r[1:]
 338                 if r not in RESOURCE_NAMES:
 339                     usage(1, 'Invalid -u/--use option: ' + a)
 340                 if remove:
 341                     if r in use_resources:
 342                         use_resources.remove(r)
 343                 elif r not in use_resources:
 344                     use_resources.append(r)
 345         elif o in ('-F', '--forever'):
 346             forever = True
 347         elif o in ('-j', '--multiprocess'):
 348             use_mp = int(a)
 349         elif o == '--slaveargs':
 350             args, kwargs = json.loads(a)
 351             try:
 352                 result = runtest(*args, **kwargs)
 353             except BaseException, e:
 354                 result = INTERRUPTED, e.__class__.__name__
 355             print   # Force a newline (just in case)
 356             print json.dumps(result)
 357             sys.exit(0)
 358         else:
 359             print >>sys.stderr, ("No handler for option {}.  Please "
 360                 "report this as a bug at http://bugs.python.org.").format(o)
 361             sys.exit(1)
 362     if single and fromfile:
 363         usage(2, "-s and -f don't go together!")
 364     if use_mp and trace:
 365         usage(2, "-T and -j don't go together!")
 366     if use_mp and findleaks:
 367         usage(2, "-l and -j don't go together!")
 368     if use_mp and max(sys.flags):
 369         # TODO: inherit the environment and the flags
 370         print "Warning: flags and environment variables are ignored with -j option"
 371
 372     good = []
 373     bad = []
 374     skipped = []
 375     resource_denieds = []
 376     environment_changed = []
 377     interrupted = False
 378
 379     if findleaks:
 380         try:
 381             import gc
 382         except ImportError:
 383             print 'No GC available, disabling findleaks.'
 384             findleaks = False
 385         else:
 386             # Uncomment the line below to report garbage that is not
 387             # freeable by reference counting alone.  By default only
 388             # garbage that is not collectable by the GC is reported.
 389             #gc.set_debug(gc.DEBUG_SAVEALL)
 390             found_garbage = []
 391
 392     if single:
 393         filename = os.path.join(TEMPDIR, 'pynexttest')
 394         try:
 395             fp = open(filename, 'r')
 396             next_test = fp.read().strip()
 397             tests = [next_test]
 398             fp.close()
 399         except IOError:
 400             pass
 401
 402     if fromfile:
 403         tests = []
 404         fp = open(os.path.join(test_support.SAVEDCWD, fromfile))
 405         for line in fp:
 406             guts = line.split() # assuming no test has whitespace in its name
 407             if guts and not guts[0].startswith('#'):
 408                 tests.extend(guts)
 409         fp.close()
 410
 411     # Strip .py extensions.
 412     removepy(args)
 413     removepy(tests)
 414
 415     stdtests = STDTESTS[:]
 416     nottests = NOTTESTS.copy()
 417     if exclude:
 418         for arg in args:
 419             if arg in stdtests:
 420                 stdtests.remove(arg)
 421             nottests.add(arg)
 422         args = []
 423
 424     # For a partial run, we do not need to clutter the output.
 425     if verbose or not (quiet or single or tests or args):
 426         # Print basic platform information
 427         print "==", platform.python_implementation(), \
 428                     " ".join(sys.version.split())
 429         print "==  ", platform.platform(aliased=True), \
 430                       "%s-endian" % sys.byteorder
 431         print "==  ", os.getcwd()
 432
 433     alltests = findtests(testdir, stdtests, nottests)
 434     selected = tests or args or alltests
 435     if single:
 436         selected = selected[:1]
 437         try:
 438             next_single_test = alltests[alltests.index(selected[0])+1]
 439         except IndexError:
 440             next_single_test = None
 441     if randomize:
 442         random.seed(random_seed)
 443         print "Using random seed", random_seed
 444         random.shuffle(selected)
 445     if trace:
 446         import trace
 447         tracer = trace.Trace(ignoredirs=[sys.prefix, sys.exec_prefix],
 448                              trace=False, count=True)
 449
 450     test_times = []
 451     test_support.use_resources = use_resources
 452     save_modules = sys.modules.keys()
 453
 454     def accumulate_result(test, result):
 455         ok, test_time = result
 456         test_times.append((test_time, test))
 457         if ok == PASSED:
 458             good.append(test)
 459         elif ok == FAILED:
 460             bad.append(test)
 461         elif ok == ENV_CHANGED:
 462             bad.append(test)
 463             environment_changed.append(test)
 464         elif ok == SKIPPED:
 465             skipped.append(test)
 466         elif ok == RESOURCE_DENIED:
 467             skipped.append(test)
 468             resource_denieds.append(test)
 469
 470     if forever:
 471         def test_forever(tests=list(selected)):
 472             while True:
 473                 for test in tests:
 474                     yield test
 475                     if bad:
 476                         return
 477         tests = test_forever()
 478     else:
 479         tests = iter(selected)
 480
 481     if use_mp:
 482         try:
 483             from threading import Thread
 484         except ImportError:
 485             print "Multiprocess option requires thread support"
 486             sys.exit(2)
 487         from Queue import Queue
 488         from subprocess import Popen, PIPE
 489         debug_output_pat = re.compile(r"\[\d+ refs\]$")
 490         output = Queue()
 491         def tests_and_args():
 492             for test in tests:
 493                 args_tuple = (
 494                     (test, verbose, quiet, testdir),
 495                     dict(huntrleaks=huntrleaks, use_resources=use_resources)
 496                 )
 497                 yield (test, args_tuple)
 498         pending = tests_and_args()
 499         def work():
 500             # A worker thread.
 501             try:
 502                 while True:
 503                     try:
 504                         test, args_tuple = next(pending)
 505                     except StopIteration:
 506                         output.put((None, None, None, None))
 507                         return
 508                     # -E is needed by some tests, e.g. test_import
 509                     popen = Popen([sys.executable, '-E', '-m', 'test.regrtest',
 510                                    '--slaveargs', json.dumps(args_tuple)],
 511                                    stdout=PIPE, stderr=PIPE,
 512                                    universal_newlines=True, close_fds=True)
 513                     stdout, stderr = popen.communicate()
 514                     # Strip last refcount output line if it exists, since it
 515                     # comes from the shutdown of the interpreter in the subcommand.
 516                     stderr = debug_output_pat.sub("", stderr)
 517                     stdout, _, result = stdout.strip().rpartition("\n")
 518                     if not result:
 519                         output.put((None, None, None, None))
 520                         return
 521                     result = json.loads(result)
 522                     if not quiet:
 523                         stdout = test+'\n'+stdout
 524                     output.put((test, stdout.rstrip(), stderr.rstrip(), result))
 525             except BaseException:
 526                 output.put((None, None, None, None))
 527                 raise
 528         workers = [Thread(target=work) for i in range(use_mp)]
 529         for worker in workers:
 530             worker.start()
 531         finished = 0
 532         try:
 533             while finished < use_mp:
 534                 test, stdout, stderr, result = output.get()
 535                 if test is None:
 536                     finished += 1
 537                     continue
 538                 if stdout:
 539                     print stdout
 540                 if stderr:
 541                     print >>sys.stderr, stderr
 542                 if result[0] == INTERRUPTED:
 543                     assert result[1] == 'KeyboardInterrupt'
 544                     raise KeyboardInterrupt   # What else?
 545                 accumulate_result(test, result)
 546         except KeyboardInterrupt:
 547             interrupted = True
 548             pending.close()
 549         for worker in workers:
 550             worker.join()
 551     else:
 552         for test in tests:
 553             if not quiet:
 554                 print test
 555                 sys.stdout.flush()
 556             if trace:
 557                 # If we're tracing code coverage, then we don't exit with status
 558                 # if on a false return value from main.
 559                 tracer.runctx('runtest(test, verbose, quiet, testdir)',
 560                               globals=globals(), locals=vars())
 561             else:
 562                 try:
 563                     result = runtest(test, verbose, quiet,
 564                                      testdir, huntrleaks)
 565                     accumulate_result(test, result)
 566                     if verbose3 and result[0] == FAILED:
 567                         print "Re-running test %r in verbose mode" % test
 568                         runtest(test, True, quiet, testdir, huntrleaks)
 569                 except KeyboardInterrupt:
 570                     interrupted = True
 571                     break
 572                 except:
 573                     raise
 574             if findleaks:
 575                 gc.collect()
 576                 if gc.garbage:
 577                     print "Warning: test created", len(gc.garbage),
 578                     print "uncollectable object(s)."
 579                     # move the uncollectable objects somewhere so we don't see
 580                     # them again
 581                     found_garbage.extend(gc.garbage)
 582                     del gc.garbage[:]
 583             # Unload the newly imported modules (best effort finalization)
 584             for module in sys.modules.keys():
 585                 if module not in save_modules and module.startswith("test."):
 586                     test_support.unload(module)
 587
 588     if interrupted:
 589         # print a newline after ^C
 590         print
 591         print "Test suite interrupted by signal SIGINT."
 592         omitted = set(selected) - set(good) - set(bad) - set(skipped)
 593         print count(len(omitted), "test"), "omitted:"
 594         printlist(omitted)
 595     if good and not quiet:
 596         if not bad and not skipped and not interrupted and len(good) > 1:
 597             print "All",
 598         print count(len(good), "test"), "OK."
 599     if print_slow:
 600         test_times.sort(reverse=True)
 601         print "10 slowest tests:"
 602         for time, test in test_times[:10]:
 603             print "%s: %.1fs" % (test, time)
 604     if bad:
 605         bad = set(bad) - set(environment_changed)
 606         if bad:
 607             print count(len(bad), "test"), "failed:"
 608             printlist(bad)
 609         if environment_changed:
 610             print "{} altered the execution environment:".format(
 611                 count(len(environment_changed), "test"))
 612             printlist(environment_changed)
 613     if skipped and not quiet:
 614         print count(len(skipped), "test"), "skipped:"
 615         printlist(skipped)
 616
 617         e = _ExpectedSkips()
 618         plat = sys.platform
 619         if e.isvalid():
 620             surprise = set(skipped) - e.getexpected() - set(resource_denieds)
 621             if surprise:
 622                 print count(len(surprise), "skip"), \
 623                       "unexpected on", plat + ":"
 624                 printlist(surprise)
 625             else:
 626                 print "Those skips are all expected on", plat + "."
 627         else:
 628             print "Ask someone to teach regrtest.py about which tests are"
 629             print "expected to get skipped on", plat + "."
 630
 631     if verbose2 and bad:
 632         print "Re-running failed tests in verbose mode"
 633         for test in bad:
 634             print "Re-running test %r in verbose mode" % test
 635             sys.stdout.flush()
 636             try:
 637                 test_support.verbose = True
 638                 ok = runtest(test, True, quiet, testdir,
 639                              huntrleaks)
 640             except KeyboardInterrupt:
 641                 # print a newline separate from the ^C
 642                 print
 643                 break
 644             except:
 645                 raise
 646
 647     if single:
 648         if next_single_test:
 649             with open(filename, 'w') as fp:
 650                 fp.write(next_single_test + '\n')
 651         else:
 652             os.unlink(filename)
 653
 654     if trace:
 655         r = tracer.results()
 656         r.write_results(show_missing=True, summary=True, coverdir=coverdir)
 657
 658     if runleaks:
 659         os.system("leaks %d" % os.getpid())
 660
 661     sys.exit(len(bad) > 0 or interrupted)
 662
 663
 664 STDTESTS = [
 665     'test_grammar',
 666     'test_opcodes',
 667     'test_dict',
 668     'test_builtin',
 669     'test_exceptions',
 670     'test_types',
 671     'test_unittest',
 672     'test_doctest',
 673     'test_doctest2',
 674 ]
 675
 676 NOTTESTS = {
 677     'test_support',
 678     'test_future1',
 679     'test_future2',
 680 }
 681
 682 def findtests(testdir=None, stdtests=STDTESTS, nottests=NOTTESTS):
 683     """Return a list of all applicable test modules."""
 684     testdir = findtestdir(testdir)
 685     names = os.listdir(testdir)
 686     tests = []
 687     others = set(stdtests) | nottests
 688     for name in names:
 689         modname, ext = os.path.splitext(name)
 690         if modname[:5] == "test_" and ext == ".py" and modname not in others:
 691             tests.append(modname)
 692     return stdtests + sorted(tests)
 693
 694 def runtest(test, verbose, quiet,
 695             testdir=None, huntrleaks=False, use_resources=None):
 696     """Run a single test.
 697
 698     test -- the name of the test
 699     verbose -- if true, print more messages
 700     quiet -- if true, don't print 'skipped' messages (probably redundant)
 701     test_times -- a list of (time, test_name) pairs
 702     testdir -- test directory
 703     huntrleaks -- run multiple times to test for leaks; requires a debug
 704                   build; a triple corresponding to -R's three arguments
 705     Returns one of the test result constants:
 706         INTERRUPTED      KeyboardInterrupt when run under -j
 707         RESOURCE_DENIED  test skipped because resource denied
 708         SKIPPED          test skipped for some other reason
 709         ENV_CHANGED      test failed because it changed the execution environment
 710         FAILED           test failed
 711         PASSED           test passed
 712     """
 713
 714     test_support.verbose = verbose  # Tell tests to be moderately quiet
 715     if use_resources is not None:
 716         test_support.use_resources = use_resources
 717     try:
 718         return runtest_inner(test, verbose, quiet,
 719                              testdir, huntrleaks)
 720     finally:
 721         cleanup_test_droppings(test, verbose)
 722
 723
 724 # Unit tests are supposed to leave the execution environment unchanged
 725 # once they complete.  But sometimes tests have bugs, especially when
 726 # tests fail, and the changes to environment go on to mess up other
 727 # tests.  This can cause issues with buildbot stability, since tests
 728 # are run in random order and so problems may appear to come and go.
 729 # There are a few things we can save and restore to mitigate this, and
 730 # the following context manager handles this task.
 731
 732 class saved_test_environment:
 733     """Save bits of the test environment and restore them at block exit.
 734
 735         with saved_test_environment(testname, verbose, quiet):
 736             #stuff
 737
 738     Unless quiet is True, a warning is printed to stderr if any of
 739     the saved items was changed by the test.  The attribute 'changed'
 740     is initially False, but is set to True if a change is detected.
 741
 742     If verbose is more than 1, the before and after state of changed
 743     items is also printed.
 744     """
 745
 746     changed = False
 747
 748     def __init__(self, testname, verbose=0, quiet=False):
 749         self.testname = testname
 750         self.verbose = verbose
 751         self.quiet = quiet
 752
 753     # To add things to save and restore, add a name XXX to the resources list
 754     # and add corresponding get_XXX/restore_XXX functions.  get_XXX should
 755     # return the value to be saved and compared against a second call to the
 756     # get function when test execution completes.  restore_XXX should accept
 757     # the saved value and restore the resource using it.  It will be called if
 758     # and only if a change in the value is detected.
 759     #
 760     # Note: XXX will have any '.' replaced with '_' characters when determining
 761     # the corresponding method names.
 762
 763     resources = ('sys.argv', 'cwd', 'sys.stdin', 'sys.stdout', 'sys.stderr',
 764                  'os.environ', 'sys.path')
 765
 766     def get_sys_argv(self):
 767         return id(sys.argv), sys.argv, sys.argv[:]
 768     def restore_sys_argv(self, saved_argv):
 769         sys.argv = saved_argv[1]
 770         sys.argv[:] = saved_argv[2]
 771
 772     def get_cwd(self):
 773         return os.getcwd()
 774     def restore_cwd(self, saved_cwd):
 775         os.chdir(saved_cwd)
 776
 777     def get_sys_stdout(self):
 778         return sys.stdout
 779     def restore_sys_stdout(self, saved_stdout):
 780         sys.stdout = saved_stdout
 781
 782     def get_sys_stderr(self):
 783         return sys.stderr
 784     def restore_sys_stderr(self, saved_stderr):
 785         sys.stderr = saved_stderr
 786
 787     def get_sys_stdin(self):
 788         return sys.stdin
 789     def restore_sys_stdin(self, saved_stdin):
 790         sys.stdin = saved_stdin
 791
 792     def get_os_environ(self):
 793         return id(os.environ), os.environ, dict(os.environ)
 794     def restore_os_environ(self, saved_environ):
 795         os.environ = saved_environ[1]
 796         os.environ.clear()
 797         os.environ.update(saved_environ[2])
 798
 799     def get_sys_path(self):
 800         return id(sys.path), sys.path, sys.path[:]
 801     def restore_sys_path(self, saved_path):
 802         sys.path = saved_path[1]
 803         sys.path[:] = saved_path[2]
 804
 805     def resource_info(self):
 806         for name in self.resources:
 807             method_suffix = name.replace('.', '_')
 808             get_name = 'get_' + method_suffix
 809             restore_name = 'restore_' + method_suffix
 810             yield name, getattr(self, get_name), getattr(self, restore_name)
 811
 812     def __enter__(self):
 813         self.saved_values = dict((name, get()) for name, get, restore
 814                                                    in self.resource_info())
 815         return self
 816
 817     def __exit__(self, exc_type, exc_val, exc_tb):
 818         for name, get, restore in self.resource_info():
 819             current = get()
 820             original = self.saved_values[name]
 821             # Check for changes to the resource's value
 822             if current != original:
 823                 self.changed = True
 824                 restore(original)
 825                 if not self.quiet:
 826                     print >>sys.stderr, (
 827                           "Warning -- {} was modified by {}".format(
 828                                                  name, self.testname))
 829                     if self.verbose > 1:
 830                         print >>sys.stderr, (
 831                               "  Before: {}\n  After:  {} ".format(
 832                                                   original, current))
 833             # XXX (ncoghlan): for most resources (e.g. sys.path) identity
 834             # matters at least as much as value. For others (e.g. cwd),
 835             # identity is irrelevant. Should we add a mechanism to check
 836             # for substitution in the cases where it matters?
 837         return False
 838
 839
 840 def runtest_inner(test, verbose, quiet,
 841                   testdir=None, huntrleaks=False):
 842     test_support.unload(test)
 843     testdir = findtestdir(testdir)
 844     if verbose:
 845         capture_stdout = None
 846     else:
 847         capture_stdout = StringIO.StringIO()
 848
 849     test_time = 0.0
 850     refleak = False  # True if the test leaked references.
 851     try:
 852         save_stdout = sys.stdout
 853         try:
 854             if capture_stdout:
 855                 sys.stdout = capture_stdout
 856             if test.startswith('test.'):
 857                 abstest = test
 858             else:
 859                 # Always import it from the test package
 860                 abstest = 'test.' + test
 861             with saved_test_environment(test, verbose, quiet) as environment:
 862                 start_time = time.time()
 863                 the_package = __import__(abstest, globals(), locals(), [])
 864                 the_module = getattr(the_package, test)
 865                 # Old tests run to completion simply as a side-effect of
 866                 # being imported.  For tests based on unittest or doctest,
 867                 # explicitly invoke their test_main() function (if it exists).
 868                 indirect_test = getattr(the_module, "test_main", None)
 869                 if indirect_test is not None:
 870                     indirect_test()
 871                 if huntrleaks:
 872                     refleak = dash_R(the_module, test, indirect_test,
 873                         huntrleaks)
 874                 test_time = time.time() - start_time
 875         finally:
 876             sys.stdout = save_stdout
 877     except test_support.ResourceDenied, msg:
 878         if not quiet:
 879             print test, "skipped --", msg
 880             sys.stdout.flush()
 881         return RESOURCE_DENIED, test_time
 882     except unittest.SkipTest, msg:
 883         if not quiet:
 884             print test, "skipped --", msg
 885             sys.stdout.flush()
 886         return SKIPPED, test_time
 887     except KeyboardInterrupt:
 888         raise
 889     except test_support.TestFailed, msg:
 890         print "test", test, "failed --", msg
 891         sys.stdout.flush()
 892         return FAILED, test_time
 893     except:
 894         type, value = sys.exc_info()[:2]
 895         print "test", test, "crashed --", str(type) + ":", value
 896         sys.stdout.flush()
 897         if verbose:
 898             traceback.print_exc(file=sys.stdout)
 899             sys.stdout.flush()
 900         return FAILED, test_time
 901     else:
 902         if refleak:
 903             return FAILED, test_time
 904         if environment.changed:
 905             return ENV_CHANGED, test_time
 906         # Except in verbose mode, tests should not print anything
 907         if verbose or huntrleaks:
 908             return PASSED, test_time
 909         output = capture_stdout.getvalue()
 910         if not output:
 911             return PASSED, test_time
 912         print "test", test, "produced unexpected output:"
 913         print "*" * 70
 914         print output
 915         print "*" * 70
 916         sys.stdout.flush()
 917         return FAILED, test_time
 918
 919 def cleanup_test_droppings(testname, verbose):
 920     import shutil
 921     import stat
 922
 923     # Try to clean up junk commonly left behind.  While tests shouldn't leave
 924     # any files or directories behind, when a test fails that can be tedious
 925     # for it to arrange.  The consequences can be especially nasty on Windows,
 926     # since if a test leaves a file open, it cannot be deleted by name (while
 927     # there's nothing we can do about that here either, we can display the
 928     # name of the offending test, which is a real help).
 929     for name in (test_support.TESTFN,
 930                  "db_home",
 931                 ):
 932         if not os.path.exists(name):
 933             continue
 934
 935         if os.path.isdir(name):
 936             kind, nuker = "directory", shutil.rmtree
 937         elif os.path.isfile(name):
 938             kind, nuker = "file", os.unlink
 939         else:
 940             raise SystemError("os.path says %r exists but is neither "
 941                               "directory nor file" % name)
 942
 943         if verbose:
 944             print "%r left behind %s %r" % (testname, kind, name)
 945         try:
 946             # if we have chmod, fix possible permissions problems
 947             # that might prevent cleanup
 948             if (hasattr(os, 'chmod')):
 949                 os.chmod(name, stat.S_IRWXU | stat.S_IRWXG | stat.S_IRWXO)
 950             nuker(name)
 951         except Exception, msg:
 952             print >> sys.stderr, ("%r left behind %s %r and it couldn't be "
 953                 "removed: %s" % (testname, kind, name, msg))
 954
 955 def dash_R(the_module, test, indirect_test, huntrleaks):
 956     """Run a test multiple times, looking for reference leaks.
 957
 958     Returns:
 959         False if the test didn't leak references; True if we detected refleaks.
 960     """
 961     # This code is hackish and inelegant, but it seems to do the job.
 962     import copy_reg, _abcoll, _pyio
 963
 964     if not hasattr(sys, 'gettotalrefcount'):
 965         raise Exception("Tracking reference leaks requires a debug build "
 966                         "of Python")
 967
 968     # Save current values for dash_R_cleanup() to restore.
 969     fs = warnings.filters[:]
 970     ps = copy_reg.dispatch_table.copy()
 971     pic = sys.path_importer_cache.copy()
 972     try:
 973         import zipimport
 974     except ImportError:
 975         zdc = None # Run unmodified on platforms without zipimport support
 976     else:
 977         zdc = zipimport._zip_directory_cache.copy()
 978     abcs = {}
 979     modules = _abcoll, _pyio
 980     for abc in [getattr(mod, a) for mod in modules for a in mod.__all__]:
 981         # XXX isinstance(abc, ABCMeta) leads to infinite recursion
 982         if not hasattr(abc, '_abc_registry'):
 983             continue
 984         for obj in abc.__subclasses__() + [abc]:
 985             abcs[obj] = obj._abc_registry.copy()
 986
 987     if indirect_test:
 988         def run_the_test():
 989             indirect_test()
 990     else:
 991         def run_the_test():
 992             imp.reload(the_module)
 993
 994     deltas = []
 995     nwarmup, ntracked, fname = huntrleaks
 996     fname = os.path.join(test_support.SAVEDCWD, fname)
 997     repcount = nwarmup + ntracked
 998     print >> sys.stderr, "beginning", repcount, "repetitions"
 999     print >> sys.stderr, ("1234567890"*(repcount//10 + 1))[:repcount]
1000     dash_R_cleanup(fs, ps, pic, zdc, abcs)
1001     for i in range(repcount):
1002         rc_before = sys.gettotalrefcount()
1003         run_the_test()
1004         sys.stderr.write('.')
1005         dash_R_cleanup(fs, ps, pic, zdc, abcs)
1006         rc_after = sys.gettotalrefcount()
1007         if i >= nwarmup:
1008             deltas.append(rc_after - rc_before)
1009     print >> sys.stderr
1010     if any(deltas):
1011         msg = '%s leaked %s references, sum=%s' % (test, deltas, sum(deltas))
1012         print >> sys.stderr, msg
1013         with open(fname, "a") as refrep:
1014             print >> refrep, msg
1015             refrep.flush()
1016         return True
1017     return False
1018
1019 def dash_R_cleanup(fs, ps, pic, zdc, abcs):
1020     import gc, copy_reg
1021     import _strptime, linecache
1022     dircache = test_support.import_module('dircache', deprecated=True)
1023     import urlparse, urllib, urllib2, mimetypes, doctest
1024     import struct, filecmp
1025     from distutils.dir_util import _path_created
1026
1027     # Clear the warnings registry, so they can be displayed again
1028     for mod in sys.modules.values():
1029         if hasattr(mod, '__warningregistry__'):
1030             del mod.__warningregistry__
1031
1032     # Restore some original values.
1033     warnings.filters[:] = fs
1034     copy_reg.dispatch_table.clear()
1035     copy_reg.dispatch_table.update(ps)
1036     sys.path_importer_cache.clear()
1037     sys.path_importer_cache.update(pic)
1038     try:
1039         import zipimport
1040     except ImportError:
1041         pass # Run unmodified on platforms without zipimport support
1042     else:
1043         zipimport._zip_directory_cache.clear()
1044         zipimport._zip_directory_cache.update(zdc)
1045
1046     # clear type cache
1047     sys._clear_type_cache()
1048
1049     # Clear ABC registries, restoring previously saved ABC registries.
1050     for abc, registry in abcs.items():
1051         abc._abc_registry = registry.copy()
1052         abc._abc_cache.clear()
1053         abc._abc_negative_cache.clear()
1054
1055     # Clear assorted module caches.
1056     _path_created.clear()
1057     re.purge()
1058     _strptime._regex_cache.clear()
1059     urlparse.clear_cache()
1060     urllib.urlcleanup()
1061     urllib2.install_opener(None)
1062     dircache.reset()
1063     linecache.clearcache()
1064     mimetypes._default_mime_types()
1065     filecmp._cache.clear()
1066     struct._clearcache()
1067     doctest.master = None
1068
1069     # Collect cyclic trash.
1070     gc.collect()
1071
1072 def findtestdir(path=None):
1073     return path or os.path.dirname(__file__) or os.curdir
1074
1075 def removepy(names):
1076     if not names:
1077         return
1078     for idx, name in enumerate(names):
1079         basename, ext = os.path.splitext(name)
1080         if ext == '.py':
1081             names[idx] = basename
1082
1083 def count(n, word):
1084     if n == 1:
1085         return "%d %s" % (n, word)
1086     else:
1087         return "%d %ss" % (n, word)
1088
1089 def printlist(x, width=70, indent=4):
1090     """Print the elements of iterable x to stdout.
1091
1092     Optional arg width (default 70) is the maximum line length.
1093     Optional arg indent (default 4) is the number of blanks with which to
1094     begin each line.
1095     """
1096
1097     from textwrap import fill
1098     blanks = ' ' * indent
1099     # Print the sorted list: 'x' may be a '--random' list or a set()
1100     print fill(' '.join(str(elt) for elt in sorted(x)), width,
1101                initial_indent=blanks, subsequent_indent=blanks)
1102
1103 # Map sys.platform to a string containing the basenames of tests
1104 # expected to be skipped on that platform.
1105 #
1106 # Special cases:
1107 #     test_pep277
1108 #         The _ExpectedSkips constructor adds this to the set of expected
1109 #         skips if not os.path.supports_unicode_filenames.
1110 #     test_timeout
1111 #         Controlled by test_timeout.skip_expected.  Requires the network
1112 #         resource and a socket module.
1113 #
1114 # Tests that are expected to be skipped everywhere except on one platform
1115 # are also handled separately.
1116
1117 _expectations = {
1118     'win32':
1119         """
1120         test__locale
1121         test_bsddb185
1122         test_bsddb3
1123         test_commands
1124         test_crypt
1125         test_curses
1126         test_dbm
1127         test_dl
1128         test_fcntl
1129         test_fork1
1130         test_epoll
1131         test_gdbm
1132         test_grp
1133         test_ioctl
1134         test_largefile
1135         test_kqueue
1136         test_mhlib
1137         test_openpty
1138         test_ossaudiodev
1139         test_pipes
1140         test_poll
1141         test_posix
1142         test_pty
1143         test_pwd
1144         test_resource
1145         test_signal
1146         test_threadsignals
1147         test_timing
1148         test_wait3
1149         test_wait4
1150         """,
1151     'linux2':
1152         """
1153         test_bsddb185
1154         test_curses
1155         test_dl
1156         test_largefile
1157         test_kqueue
1158         test_ossaudiodev
1159         """,
1160     'unixware7':
1161         """
1162         test_bsddb
1163         test_bsddb185
1164         test_dl
1165         test_epoll
1166         test_largefile
1167         test_kqueue
1168         test_minidom
1169         test_openpty
1170         test_pyexpat
1171         test_sax
1172         test_sundry
1173         """,
1174     'openunix8':
1175         """
1176         test_bsddb
1177         test_bsddb185
1178         test_dl
1179         test_epoll
1180         test_largefile
1181         test_kqueue
1182         test_minidom
1183         test_openpty
1184         test_pyexpat
1185         test_sax
1186         test_sundry
1187         """,
1188     'sco_sv3':
1189         """
1190         test_asynchat
1191         test_bsddb
1192         test_bsddb185
1193         test_dl
1194         test_fork1
1195         test_epoll
1196         test_gettext
1197         test_largefile
1198         test_locale
1199         test_kqueue
1200         test_minidom
1201         test_openpty
1202         test_pyexpat
1203         test_queue
1204         test_sax
1205         test_sundry
1206         test_thread
1207         test_threaded_import
1208         test_threadedtempfile
1209         test_threading
1210         """,
1211     'riscos':
1212         """
1213         test_asynchat
1214         test_atexit
1215         test_bsddb
1216         test_bsddb185
1217         test_bsddb3
1218         test_commands
1219         test_crypt
1220         test_dbm
1221         test_dl
1222         test_fcntl
1223         test_fork1
1224         test_epoll
1225         test_gdbm
1226         test_grp
1227         test_largefile
1228         test_locale
1229         test_kqueue
1230         test_mmap
1231         test_openpty
1232         test_poll
1233         test_popen2
1234         test_pty
1235         test_pwd
1236         test_strop
1237         test_sundry
1238         test_thread
1239         test_threaded_import
1240         test_threadedtempfile
1241         test_threading
1242         test_timing
1243         """,
1244     'darwin':
1245         """
1246         test__locale
1247         test_bsddb
1248         test_bsddb3
1249         test_curses
1250         test_epoll
1251         test_gdbm
1252         test_largefile
1253         test_locale
1254         test_kqueue
1255         test_minidom
1256         test_ossaudiodev
1257         test_poll
1258         """,
1259     'sunos5':
1260         """
1261         test_bsddb
1262         test_bsddb185
1263         test_curses
1264         test_dbm
1265         test_epoll
1266         test_kqueue
1267         test_gdbm
1268         test_gzip
1269         test_openpty
1270         test_zipfile
1271         test_zlib
1272         """,
1273     'hp-ux11':
1274         """
1275         test_bsddb
1276         test_bsddb185
1277         test_curses
1278         test_dl
1279         test_epoll
1280         test_gdbm
1281         test_gzip
1282         test_largefile
1283         test_locale
1284         test_kqueue
1285         test_minidom
1286         test_openpty
1287         test_pyexpat
1288         test_sax
1289         test_zipfile
1290         test_zlib
1291         """,
1292     'atheos':
1293         """
1294         test_bsddb185
1295         test_curses
1296         test_dl
1297         test_gdbm
1298         test_epoll
1299         test_largefile
1300         test_locale
1301         test_kqueue
1302         test_mhlib
1303         test_mmap
1304         test_poll
1305         test_popen2
1306         test_resource
1307         """,
1308     'cygwin':
1309         """
1310         test_bsddb185
1311         test_bsddb3
1312         test_curses
1313         test_dbm
1314         test_epoll
1315         test_ioctl
1316         test_kqueue
1317         test_largefile
1318         test_locale
1319         test_ossaudiodev
1320         test_socketserver
1321         """,
1322     'os2emx':
1323         """
1324         test_audioop
1325         test_bsddb185
1326         test_bsddb3
1327         test_commands
1328         test_curses
1329         test_dl
1330         test_epoll
1331         test_kqueue
1332         test_largefile
1333         test_mhlib
1334         test_mmap
1335         test_openpty
1336         test_ossaudiodev
1337         test_pty
1338         test_resource
1339         test_signal
1340         """,
1341     'freebsd4':
1342         """
1343         test_bsddb
1344         test_bsddb3
1345         test_epoll
1346         test_gdbm
1347         test_locale
1348         test_ossaudiodev
1349         test_pep277
1350         test_pty
1351         test_socketserver
1352         test_tcl
1353         test_tk
1354         test_ttk_guionly
1355         test_ttk_textonly
1356         test_timeout
1357         test_urllibnet
1358         test_multiprocessing
1359         """,
1360     'aix5':
1361         """
1362         test_bsddb
1363         test_bsddb185
1364         test_bsddb3
1365         test_bz2
1366         test_dl
1367         test_epoll
1368         test_gdbm
1369         test_gzip
1370         test_kqueue
1371         test_ossaudiodev
1372         test_tcl
1373         test_tk
1374         test_ttk_guionly
1375         test_ttk_textonly
1376         test_zipimport
1377         test_zlib
1378         """,
1379     'openbsd3':
1380         """
1381         test_ascii_formatd
1382         test_bsddb
1383         test_bsddb3
1384         test_ctypes
1385         test_dl
1386         test_epoll
1387         test_gdbm
1388         test_locale
1389         test_normalization
1390         test_ossaudiodev
1391         test_pep277
1392         test_tcl
1393         test_tk
1394         test_ttk_guionly
1395         test_ttk_textonly
1396         test_multiprocessing
1397         """,
1398     'netbsd3':
1399         """
1400         test_ascii_formatd
1401         test_bsddb
1402         test_bsddb185
1403         test_bsddb3
1404         test_ctypes
1405         test_curses
1406         test_dl
1407         test_epoll
1408         test_gdbm
1409         test_locale
1410         test_ossaudiodev
1411         test_pep277
1412         test_tcl
1413         test_tk
1414         test_ttk_guionly
1415         test_ttk_textonly
1416         test_multiprocessing
1417         """,
1418 }
1419 _expectations['freebsd5'] = _expectations['freebsd4']
1420 _expectations['freebsd6'] = _expectations['freebsd4']
1421 _expectations['freebsd7'] = _expectations['freebsd4']
1422 _expectations['freebsd8'] = _expectations['freebsd4']
1423
1424 class _ExpectedSkips:
1425     def __init__(self):
1426         import os.path
1427         from test import test_timeout
1428
1429         self.valid = False
1430         if sys.platform in _expectations:
1431             s = _expectations[sys.platform]
1432             self.expected = set(s.split())
1433
1434             # expected to be skipped on every platform, even Linux
1435             self.expected.add('test_linuxaudiodev')
1436
1437             if not os.path.supports_unicode_filenames:
1438                 self.expected.add('test_pep277')
1439
1440             if test_timeout.skip_expected:
1441                 self.expected.add('test_timeout')
1442
1443             if sys.maxint == 9223372036854775807L:
1444                 self.expected.add('test_imageop')
1445
1446             if sys.platform != "darwin":
1447                 MAC_ONLY = ["test_macos", "test_macostools", "test_aepack",
1448                             "test_plistlib", "test_scriptpackages",
1449                             "test_applesingle"]
1450                 for skip in MAC_ONLY:
1451                     self.expected.add(skip)
1452             elif len(u'\0'.encode('unicode-internal')) == 4:
1453                 self.expected.add("test_macostools")
1454
1455
1456             if sys.platform != "win32":
1457                 # test_sqlite is only reliable on Windows where the library
1458                 # is distributed with Python
1459                 WIN_ONLY = ["test_unicode_file", "test_winreg",
1460                             "test_winsound", "test_startfile",
1461                             "test_sqlite"]
1462                 for skip in WIN_ONLY:
1463                     self.expected.add(skip)
1464
1465             if sys.platform != 'irix':
1466                 IRIX_ONLY = ["test_imageop", "test_al", "test_cd", "test_cl",
1467                              "test_gl", "test_imgfile"]
1468                 for skip in IRIX_ONLY:
1469                     self.expected.add(skip)
1470
1471             if sys.platform != 'sunos5':
1472                 self.expected.add('test_sunaudiodev')
1473                 self.expected.add('test_nis')
1474
1475             if not sys.py3kwarning:
1476                 self.expected.add('test_py3kwarn')
1477
1478             self.valid = True
1479
1480     def isvalid(self):
1481         "Return true iff _ExpectedSkips knows about the current platform."
1482         return self.valid
1483
1484     def getexpected(self):
1485         """Return set of test names we expect to skip on current platform.
1486
1487         self.isvalid() must be true.
1488         """
1489
1490         assert self.isvalid()
1491         return self.expected
1492
1493 if __name__ == '__main__':
1494     # Simplification for findtestdir().
1495     assert __file__ == os.path.abspath(sys.argv[0])
1496
1497     # When tests are run from the Python build directory, it is best practice
1498     # to keep the test files in a subfolder.  It eases the cleanup of leftover
1499     # files using command "make distclean".
1500     if sysconfig.is_python_build():
1501         TEMPDIR = os.path.join(sysconfig.get_config_var('srcdir'), 'build')
1502         TEMPDIR = os.path.abspath(TEMPDIR)
1503         if not os.path.exists(TEMPDIR):
1504             os.mkdir(TEMPDIR)
1505
1506     # Define a writable temp dir that will be used as cwd while running
1507     # the tests. The name of the dir includes the pid to allow parallel
1508     # testing (see the -j option).
1509     TESTCWD = 'test_python_{}'.format(os.getpid())
1510
1511     TESTCWD = os.path.join(TEMPDIR, TESTCWD)
1512
1513     # Run the tests in a context manager that temporary changes the CWD to a
1514     # temporary and writable directory. If it's not possible to create or
1515     # change the CWD, the original CWD will be used. The original CWD is
1516     # available from test_support.SAVEDCWD.
1517     with test_support.temp_cwd(TESTCWD, quiet=True):
1518         main()