When open_urlresource() fails, HTTPException is another possible error
[python.git] / Lib / subprocess.py
blobf1434609900fbb680a413f4c2fdca4373328bdd2
1 # subprocess - Subprocesses with accessible I/O streams
3 # For more information about this module, see PEP 324.
5 # This module should remain compatible with Python 2.2, see PEP 291.
7 # Copyright (c) 2003-2005 by Peter Astrand <astrand@lysator.liu.se>
9 # Licensed to PSF under a Contributor Agreement.
10 # See http://www.python.org/2.4/license for licensing details.
12 r"""subprocess - Subprocesses with accessible I/O streams
14 This module allows you to spawn processes, connect to their
15 input/output/error pipes, and obtain their return codes. This module
16 intends to replace several other, older modules and functions, like:
18 os.system
19 os.spawn*
20 os.popen*
21 popen2.*
22 commands.*
24 Information about how the subprocess module can be used to replace these
25 modules and functions can be found below.
29 Using the subprocess module
30 ===========================
31 This module defines one class called Popen:
33 class Popen(args, bufsize=0, executable=None,
34 stdin=None, stdout=None, stderr=None,
35 preexec_fn=None, close_fds=False, shell=False,
36 cwd=None, env=None, universal_newlines=False,
37 startupinfo=None, creationflags=0):
40 Arguments are:
42 args should be a string, or a sequence of program arguments. The
43 program to execute is normally the first item in the args sequence or
44 string, but can be explicitly set by using the executable argument.
46 On UNIX, with shell=False (default): In this case, the Popen class
47 uses os.execvp() to execute the child program. args should normally
48 be a sequence. A string will be treated as a sequence with the string
49 as the only item (the program to execute).
51 On UNIX, with shell=True: If args is a string, it specifies the
52 command string to execute through the shell. If args is a sequence,
53 the first item specifies the command string, and any additional items
54 will be treated as additional shell arguments.
56 On Windows: the Popen class uses CreateProcess() to execute the child
57 program, which operates on strings. If args is a sequence, it will be
58 converted to a string using the list2cmdline method. Please note that
59 not all MS Windows applications interpret the command line the same
60 way: The list2cmdline is designed for applications using the same
61 rules as the MS C runtime.
63 bufsize, if given, has the same meaning as the corresponding argument
64 to the built-in open() function: 0 means unbuffered, 1 means line
65 buffered, any other positive value means use a buffer of
66 (approximately) that size. A negative bufsize means to use the system
67 default, which usually means fully buffered. The default value for
68 bufsize is 0 (unbuffered).
70 stdin, stdout and stderr specify the executed programs' standard
71 input, standard output and standard error file handles, respectively.
72 Valid values are PIPE, an existing file descriptor (a positive
73 integer), an existing file object, and None. PIPE indicates that a
74 new pipe to the child should be created. With None, no redirection
75 will occur; the child's file handles will be inherited from the
76 parent. Additionally, stderr can be STDOUT, which indicates that the
77 stderr data from the applications should be captured into the same
78 file handle as for stdout.
80 If preexec_fn is set to a callable object, this object will be called
81 in the child process just before the child is executed.
83 If close_fds is true, all file descriptors except 0, 1 and 2 will be
84 closed before the child process is executed.
86 if shell is true, the specified command will be executed through the
87 shell.
89 If cwd is not None, the current directory will be changed to cwd
90 before the child is executed.
92 If env is not None, it defines the environment variables for the new
93 process.
95 If universal_newlines is true, the file objects stdout and stderr are
96 opened as a text files, but lines may be terminated by any of '\n',
97 the Unix end-of-line convention, '\r', the Macintosh convention or
98 '\r\n', the Windows convention. All of these external representations
99 are seen as '\n' by the Python program. Note: This feature is only
100 available if Python is built with universal newline support (the
101 default). Also, the newlines attribute of the file objects stdout,
102 stdin and stderr are not updated by the communicate() method.
104 The startupinfo and creationflags, if given, will be passed to the
105 underlying CreateProcess() function. They can specify things such as
106 appearance of the main window and priority for the new process.
107 (Windows only)
110 This module also defines some shortcut functions:
112 call(*popenargs, **kwargs):
113 Run command with arguments. Wait for command to complete, then
114 return the returncode attribute.
116 The arguments are the same as for the Popen constructor. Example:
118 retcode = call(["ls", "-l"])
120 check_call(*popenargs, **kwargs):
121 Run command with arguments. Wait for command to complete. If the
122 exit code was zero then return, otherwise raise
123 CalledProcessError. The CalledProcessError object will have the
124 return code in the returncode attribute.
126 The arguments are the same as for the Popen constructor. Example:
128 check_call(["ls", "-l"])
130 check_output(*popenargs, **kwargs):
131 Run command with arguments and return its output as a byte string.
133 If the exit code was non-zero it raises a CalledProcessError. The
134 CalledProcessError object will have the return code in the returncode
135 attribute and output in the output attribute.
137 The arguments are the same as for the Popen constructor. Example:
139 output = subprocess.check_output(["ls", "-l", "/dev/null"])
141 Exceptions
142 ----------
143 Exceptions raised in the child process, before the new program has
144 started to execute, will be re-raised in the parent. Additionally,
145 the exception object will have one extra attribute called
146 'child_traceback', which is a string containing traceback information
147 from the childs point of view.
149 The most common exception raised is OSError. This occurs, for
150 example, when trying to execute a non-existent file. Applications
151 should prepare for OSErrors.
153 A ValueError will be raised if Popen is called with invalid arguments.
155 check_call() and check_output() will raise CalledProcessError, if the
156 called process returns a non-zero return code.
159 Security
160 --------
161 Unlike some other popen functions, this implementation will never call
162 /bin/sh implicitly. This means that all characters, including shell
163 metacharacters, can safely be passed to child processes.
166 Popen objects
167 =============
168 Instances of the Popen class have the following methods:
170 poll()
171 Check if child process has terminated. Returns returncode
172 attribute.
174 wait()
175 Wait for child process to terminate. Returns returncode attribute.
177 communicate(input=None)
178 Interact with process: Send data to stdin. Read data from stdout
179 and stderr, until end-of-file is reached. Wait for process to
180 terminate. The optional input argument should be a string to be
181 sent to the child process, or None, if no data should be sent to
182 the child.
184 communicate() returns a tuple (stdout, stderr).
186 Note: The data read is buffered in memory, so do not use this
187 method if the data size is large or unlimited.
189 The following attributes are also available:
191 stdin
192 If the stdin argument is PIPE, this attribute is a file object
193 that provides input to the child process. Otherwise, it is None.
195 stdout
196 If the stdout argument is PIPE, this attribute is a file object
197 that provides output from the child process. Otherwise, it is
198 None.
200 stderr
201 If the stderr argument is PIPE, this attribute is file object that
202 provides error output from the child process. Otherwise, it is
203 None.
206 The process ID of the child process.
208 returncode
209 The child return code. A None value indicates that the process
210 hasn't terminated yet. A negative value -N indicates that the
211 child was terminated by signal N (UNIX only).
214 Replacing older functions with the subprocess module
215 ====================================================
216 In this section, "a ==> b" means that b can be used as a replacement
217 for a.
219 Note: All functions in this section fail (more or less) silently if
220 the executed program cannot be found; this module raises an OSError
221 exception.
223 In the following examples, we assume that the subprocess module is
224 imported with "from subprocess import *".
227 Replacing /bin/sh shell backquote
228 ---------------------------------
229 output=`mycmd myarg`
231 output = Popen(["mycmd", "myarg"], stdout=PIPE).communicate()[0]
234 Replacing shell pipe line
235 -------------------------
236 output=`dmesg | grep hda`
238 p1 = Popen(["dmesg"], stdout=PIPE)
239 p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
240 output = p2.communicate()[0]
243 Replacing os.system()
244 ---------------------
245 sts = os.system("mycmd" + " myarg")
247 p = Popen("mycmd" + " myarg", shell=True)
248 pid, sts = os.waitpid(p.pid, 0)
250 Note:
252 * Calling the program through the shell is usually not required.
254 * It's easier to look at the returncode attribute than the
255 exitstatus.
257 A more real-world example would look like this:
259 try:
260 retcode = call("mycmd" + " myarg", shell=True)
261 if retcode < 0:
262 print >>sys.stderr, "Child was terminated by signal", -retcode
263 else:
264 print >>sys.stderr, "Child returned", retcode
265 except OSError, e:
266 print >>sys.stderr, "Execution failed:", e
269 Replacing os.spawn*
270 -------------------
271 P_NOWAIT example:
273 pid = os.spawnlp(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg")
275 pid = Popen(["/bin/mycmd", "myarg"]).pid
278 P_WAIT example:
280 retcode = os.spawnlp(os.P_WAIT, "/bin/mycmd", "mycmd", "myarg")
282 retcode = call(["/bin/mycmd", "myarg"])
285 Vector example:
287 os.spawnvp(os.P_NOWAIT, path, args)
289 Popen([path] + args[1:])
292 Environment example:
294 os.spawnlpe(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg", env)
296 Popen(["/bin/mycmd", "myarg"], env={"PATH": "/usr/bin"})
299 Replacing os.popen*
300 -------------------
301 pipe = os.popen("cmd", mode='r', bufsize)
303 pipe = Popen("cmd", shell=True, bufsize=bufsize, stdout=PIPE).stdout
305 pipe = os.popen("cmd", mode='w', bufsize)
307 pipe = Popen("cmd", shell=True, bufsize=bufsize, stdin=PIPE).stdin
310 (child_stdin, child_stdout) = os.popen2("cmd", mode, bufsize)
312 p = Popen("cmd", shell=True, bufsize=bufsize,
313 stdin=PIPE, stdout=PIPE, close_fds=True)
314 (child_stdin, child_stdout) = (p.stdin, p.stdout)
317 (child_stdin,
318 child_stdout,
319 child_stderr) = os.popen3("cmd", mode, bufsize)
321 p = Popen("cmd", shell=True, bufsize=bufsize,
322 stdin=PIPE, stdout=PIPE, stderr=PIPE, close_fds=True)
323 (child_stdin,
324 child_stdout,
325 child_stderr) = (p.stdin, p.stdout, p.stderr)
328 (child_stdin, child_stdout_and_stderr) = os.popen4("cmd", mode,
329 bufsize)
331 p = Popen("cmd", shell=True, bufsize=bufsize,
332 stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
333 (child_stdin, child_stdout_and_stderr) = (p.stdin, p.stdout)
335 On Unix, os.popen2, os.popen3 and os.popen4 also accept a sequence as
336 the command to execute, in which case arguments will be passed
337 directly to the program without shell intervention. This usage can be
338 replaced as follows:
340 (child_stdin, child_stdout) = os.popen2(["/bin/ls", "-l"], mode,
341 bufsize)
343 p = Popen(["/bin/ls", "-l"], bufsize=bufsize, stdin=PIPE, stdout=PIPE)
344 (child_stdin, child_stdout) = (p.stdin, p.stdout)
346 Return code handling translates as follows:
348 pipe = os.popen("cmd", 'w')
350 rc = pipe.close()
351 if rc != None and rc % 256:
352 print "There were some errors"
354 process = Popen("cmd", 'w', shell=True, stdin=PIPE)
356 process.stdin.close()
357 if process.wait() != 0:
358 print "There were some errors"
361 Replacing popen2.*
362 ------------------
363 (child_stdout, child_stdin) = popen2.popen2("somestring", bufsize, mode)
365 p = Popen(["somestring"], shell=True, bufsize=bufsize
366 stdin=PIPE, stdout=PIPE, close_fds=True)
367 (child_stdout, child_stdin) = (p.stdout, p.stdin)
369 On Unix, popen2 also accepts a sequence as the command to execute, in
370 which case arguments will be passed directly to the program without
371 shell intervention. This usage can be replaced as follows:
373 (child_stdout, child_stdin) = popen2.popen2(["mycmd", "myarg"], bufsize,
374 mode)
376 p = Popen(["mycmd", "myarg"], bufsize=bufsize,
377 stdin=PIPE, stdout=PIPE, close_fds=True)
378 (child_stdout, child_stdin) = (p.stdout, p.stdin)
380 The popen2.Popen3 and popen2.Popen4 basically works as subprocess.Popen,
381 except that:
383 * subprocess.Popen raises an exception if the execution fails
384 * the capturestderr argument is replaced with the stderr argument.
385 * stdin=PIPE and stdout=PIPE must be specified.
386 * popen2 closes all filedescriptors by default, but you have to specify
387 close_fds=True with subprocess.Popen.
390 import sys
391 mswindows = (sys.platform == "win32")
393 import os
394 import types
395 import traceback
396 import gc
397 import signal
399 # Exception classes used by this module.
400 class CalledProcessError(Exception):
401 """This exception is raised when a process run by check_call() or
402 check_output() returns a non-zero exit status.
403 The exit status will be stored in the returncode attribute;
404 check_output() will also store the output in the output attribute.
406 def __init__(self, returncode, cmd, output=None):
407 self.returncode = returncode
408 self.cmd = cmd
409 self.output = output
410 def __str__(self):
411 return "Command '%s' returned non-zero exit status %d" % (self.cmd, self.returncode)
414 if mswindows:
415 import threading
416 import msvcrt
417 if 0: # <-- change this to use pywin32 instead of the _subprocess driver
418 import pywintypes
419 from win32api import GetStdHandle, STD_INPUT_HANDLE, \
420 STD_OUTPUT_HANDLE, STD_ERROR_HANDLE
421 from win32api import GetCurrentProcess, DuplicateHandle, \
422 GetModuleFileName, GetVersion
423 from win32con import DUPLICATE_SAME_ACCESS, SW_HIDE
424 from win32pipe import CreatePipe
425 from win32process import CreateProcess, STARTUPINFO, \
426 GetExitCodeProcess, STARTF_USESTDHANDLES, \
427 STARTF_USESHOWWINDOW, CREATE_NEW_CONSOLE
428 from win32process import TerminateProcess
429 from win32event import WaitForSingleObject, INFINITE, WAIT_OBJECT_0
430 else:
431 from _subprocess import *
432 class STARTUPINFO:
433 dwFlags = 0
434 hStdInput = None
435 hStdOutput = None
436 hStdError = None
437 wShowWindow = 0
438 class pywintypes:
439 error = IOError
440 else:
441 import select
442 _has_poll = hasattr(select, 'poll')
443 import errno
444 import fcntl
445 import pickle
447 # When select or poll has indicated that the file is writable,
448 # we can write up to _PIPE_BUF bytes without risk of blocking.
449 # POSIX defines PIPE_BUF as >= 512.
450 _PIPE_BUF = getattr(select, 'PIPE_BUF', 512)
453 __all__ = ["Popen", "PIPE", "STDOUT", "call", "check_call",
454 "check_output", "CalledProcessError"]
456 try:
457 MAXFD = os.sysconf("SC_OPEN_MAX")
458 except:
459 MAXFD = 256
461 _active = []
463 def _cleanup():
464 for inst in _active[:]:
465 if inst._internal_poll(_deadstate=sys.maxint) >= 0:
466 try:
467 _active.remove(inst)
468 except ValueError:
469 # This can happen if two threads create a new Popen instance.
470 # It's harmless that it was already removed, so ignore.
471 pass
473 PIPE = -1
474 STDOUT = -2
477 def call(*popenargs, **kwargs):
478 """Run command with arguments. Wait for command to complete, then
479 return the returncode attribute.
481 The arguments are the same as for the Popen constructor. Example:
483 retcode = call(["ls", "-l"])
485 return Popen(*popenargs, **kwargs).wait()
488 def check_call(*popenargs, **kwargs):
489 """Run command with arguments. Wait for command to complete. If
490 the exit code was zero then return, otherwise raise
491 CalledProcessError. The CalledProcessError object will have the
492 return code in the returncode attribute.
494 The arguments are the same as for the Popen constructor. Example:
496 check_call(["ls", "-l"])
498 retcode = call(*popenargs, **kwargs)
499 if retcode:
500 cmd = kwargs.get("args")
501 if cmd is None:
502 cmd = popenargs[0]
503 raise CalledProcessError(retcode, cmd)
504 return 0
507 def check_output(*popenargs, **kwargs):
508 """Run command with arguments and return its output as a byte string.
510 If the exit code was non-zero it raises a CalledProcessError. The
511 CalledProcessError object will have the return code in the returncode
512 attribute and output in the output attribute.
514 The arguments are the same as for the Popen constructor. Example:
516 >>> check_output(["ls", "-l", "/dev/null"])
517 'crw-rw-rw- 1 root root 1, 3 Oct 18 2007 /dev/null\n'
519 The stdout argument is not allowed as it is used internally.
520 To capture standard error in the result, use stderr=subprocess.STDOUT.
522 >>> check_output(["/bin/sh", "-c",
523 "ls -l non_existent_file ; exit 0"],
524 stderr=subprocess.STDOUT)
525 'ls: non_existent_file: No such file or directory\n'
527 if 'stdout' in kwargs:
528 raise ValueError('stdout argument not allowed, it will be overridden.')
529 process = Popen(stdout=PIPE, *popenargs, **kwargs)
530 output, unused_err = process.communicate()
531 retcode = process.poll()
532 if retcode:
533 cmd = kwargs.get("args")
534 if cmd is None:
535 cmd = popenargs[0]
536 raise CalledProcessError(retcode, cmd, output=output)
537 return output
540 def list2cmdline(seq):
542 Translate a sequence of arguments into a command line
543 string, using the same rules as the MS C runtime:
545 1) Arguments are delimited by white space, which is either a
546 space or a tab.
548 2) A string surrounded by double quotation marks is
549 interpreted as a single argument, regardless of white space
550 or pipe characters contained within. A quoted string can be
551 embedded in an argument.
553 3) A double quotation mark preceded by a backslash is
554 interpreted as a literal double quotation mark.
556 4) Backslashes are interpreted literally, unless they
557 immediately precede a double quotation mark.
559 5) If backslashes immediately precede a double quotation mark,
560 every pair of backslashes is interpreted as a literal
561 backslash. If the number of backslashes is odd, the last
562 backslash escapes the next double quotation mark as
563 described in rule 3.
566 # See
567 # http://msdn.microsoft.com/en-us/library/17w5ykft.aspx
568 # or search http://msdn.microsoft.com for
569 # "Parsing C++ Command-Line Arguments"
570 result = []
571 needquote = False
572 for arg in seq:
573 bs_buf = []
575 # Add a space to separate this argument from the others
576 if result:
577 result.append(' ')
579 needquote = (" " in arg) or ("\t" in arg) or ("|" in arg) or not arg
580 if needquote:
581 result.append('"')
583 for c in arg:
584 if c == '\\':
585 # Don't know if we need to double yet.
586 bs_buf.append(c)
587 elif c == '"':
588 # Double backslashes.
589 result.append('\\' * len(bs_buf)*2)
590 bs_buf = []
591 result.append('\\"')
592 else:
593 # Normal char
594 if bs_buf:
595 result.extend(bs_buf)
596 bs_buf = []
597 result.append(c)
599 # Add remaining backslashes, if any.
600 if bs_buf:
601 result.extend(bs_buf)
603 if needquote:
604 result.extend(bs_buf)
605 result.append('"')
607 return ''.join(result)
610 class Popen(object):
611 def __init__(self, args, bufsize=0, executable=None,
612 stdin=None, stdout=None, stderr=None,
613 preexec_fn=None, close_fds=False, shell=False,
614 cwd=None, env=None, universal_newlines=False,
615 startupinfo=None, creationflags=0):
616 """Create new Popen instance."""
617 _cleanup()
619 self._child_created = False
620 if not isinstance(bufsize, (int, long)):
621 raise TypeError("bufsize must be an integer")
623 if mswindows:
624 if preexec_fn is not None:
625 raise ValueError("preexec_fn is not supported on Windows "
626 "platforms")
627 if close_fds and (stdin is not None or stdout is not None or
628 stderr is not None):
629 raise ValueError("close_fds is not supported on Windows "
630 "platforms if you redirect stdin/stdout/stderr")
631 else:
632 # POSIX
633 if startupinfo is not None:
634 raise ValueError("startupinfo is only supported on Windows "
635 "platforms")
636 if creationflags != 0:
637 raise ValueError("creationflags is only supported on Windows "
638 "platforms")
640 self.stdin = None
641 self.stdout = None
642 self.stderr = None
643 self.pid = None
644 self.returncode = None
645 self.universal_newlines = universal_newlines
647 # Input and output objects. The general principle is like
648 # this:
650 # Parent Child
651 # ------ -----
652 # p2cwrite ---stdin---> p2cread
653 # c2pread <--stdout--- c2pwrite
654 # errread <--stderr--- errwrite
656 # On POSIX, the child objects are file descriptors. On
657 # Windows, these are Windows file handles. The parent objects
658 # are file descriptors on both platforms. The parent objects
659 # are None when not using PIPEs. The child objects are None
660 # when not redirecting.
662 (p2cread, p2cwrite,
663 c2pread, c2pwrite,
664 errread, errwrite) = self._get_handles(stdin, stdout, stderr)
666 self._execute_child(args, executable, preexec_fn, close_fds,
667 cwd, env, universal_newlines,
668 startupinfo, creationflags, shell,
669 p2cread, p2cwrite,
670 c2pread, c2pwrite,
671 errread, errwrite)
673 if mswindows:
674 if p2cwrite is not None:
675 p2cwrite = msvcrt.open_osfhandle(p2cwrite.Detach(), 0)
676 if c2pread is not None:
677 c2pread = msvcrt.open_osfhandle(c2pread.Detach(), 0)
678 if errread is not None:
679 errread = msvcrt.open_osfhandle(errread.Detach(), 0)
681 if p2cwrite is not None:
682 self.stdin = os.fdopen(p2cwrite, 'wb', bufsize)
683 if c2pread is not None:
684 if universal_newlines:
685 self.stdout = os.fdopen(c2pread, 'rU', bufsize)
686 else:
687 self.stdout = os.fdopen(c2pread, 'rb', bufsize)
688 if errread is not None:
689 if universal_newlines:
690 self.stderr = os.fdopen(errread, 'rU', bufsize)
691 else:
692 self.stderr = os.fdopen(errread, 'rb', bufsize)
695 def _translate_newlines(self, data):
696 data = data.replace("\r\n", "\n")
697 data = data.replace("\r", "\n")
698 return data
701 def __del__(self, sys=sys):
702 if not self._child_created:
703 # We didn't get to successfully create a child process.
704 return
705 # In case the child hasn't been waited on, check if it's done.
706 self._internal_poll(_deadstate=sys.maxint)
707 if self.returncode is None and _active is not None:
708 # Child is still running, keep us alive until we can wait on it.
709 _active.append(self)
712 def communicate(self, input=None):
713 """Interact with process: Send data to stdin. Read data from
714 stdout and stderr, until end-of-file is reached. Wait for
715 process to terminate. The optional input argument should be a
716 string to be sent to the child process, or None, if no data
717 should be sent to the child.
719 communicate() returns a tuple (stdout, stderr)."""
721 # Optimization: If we are only using one pipe, or no pipe at
722 # all, using select() or threads is unnecessary.
723 if [self.stdin, self.stdout, self.stderr].count(None) >= 2:
724 stdout = None
725 stderr = None
726 if self.stdin:
727 if input:
728 self.stdin.write(input)
729 self.stdin.close()
730 elif self.stdout:
731 stdout = self.stdout.read()
732 self.stdout.close()
733 elif self.stderr:
734 stderr = self.stderr.read()
735 self.stderr.close()
736 self.wait()
737 return (stdout, stderr)
739 return self._communicate(input)
742 def poll(self):
743 return self._internal_poll()
746 if mswindows:
748 # Windows methods
750 def _get_handles(self, stdin, stdout, stderr):
751 """Construct and return tuple with IO objects:
752 p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite
754 if stdin is None and stdout is None and stderr is None:
755 return (None, None, None, None, None, None)
757 p2cread, p2cwrite = None, None
758 c2pread, c2pwrite = None, None
759 errread, errwrite = None, None
761 if stdin is None:
762 p2cread = GetStdHandle(STD_INPUT_HANDLE)
763 if p2cread is None:
764 p2cread, _ = CreatePipe(None, 0)
765 elif stdin == PIPE:
766 p2cread, p2cwrite = CreatePipe(None, 0)
767 elif isinstance(stdin, int):
768 p2cread = msvcrt.get_osfhandle(stdin)
769 else:
770 # Assuming file-like object
771 p2cread = msvcrt.get_osfhandle(stdin.fileno())
772 p2cread = self._make_inheritable(p2cread)
774 if stdout is None:
775 c2pwrite = GetStdHandle(STD_OUTPUT_HANDLE)
776 if c2pwrite is None:
777 _, c2pwrite = CreatePipe(None, 0)
778 elif stdout == PIPE:
779 c2pread, c2pwrite = CreatePipe(None, 0)
780 elif isinstance(stdout, int):
781 c2pwrite = msvcrt.get_osfhandle(stdout)
782 else:
783 # Assuming file-like object
784 c2pwrite = msvcrt.get_osfhandle(stdout.fileno())
785 c2pwrite = self._make_inheritable(c2pwrite)
787 if stderr is None:
788 errwrite = GetStdHandle(STD_ERROR_HANDLE)
789 if errwrite is None:
790 _, errwrite = CreatePipe(None, 0)
791 elif stderr == PIPE:
792 errread, errwrite = CreatePipe(None, 0)
793 elif stderr == STDOUT:
794 errwrite = c2pwrite
795 elif isinstance(stderr, int):
796 errwrite = msvcrt.get_osfhandle(stderr)
797 else:
798 # Assuming file-like object
799 errwrite = msvcrt.get_osfhandle(stderr.fileno())
800 errwrite = self._make_inheritable(errwrite)
802 return (p2cread, p2cwrite,
803 c2pread, c2pwrite,
804 errread, errwrite)
807 def _make_inheritable(self, handle):
808 """Return a duplicate of handle, which is inheritable"""
809 return DuplicateHandle(GetCurrentProcess(), handle,
810 GetCurrentProcess(), 0, 1,
811 DUPLICATE_SAME_ACCESS)
814 def _find_w9xpopen(self):
815 """Find and return absolut path to w9xpopen.exe"""
816 w9xpopen = os.path.join(os.path.dirname(GetModuleFileName(0)),
817 "w9xpopen.exe")
818 if not os.path.exists(w9xpopen):
819 # Eeek - file-not-found - possibly an embedding
820 # situation - see if we can locate it in sys.exec_prefix
821 w9xpopen = os.path.join(os.path.dirname(sys.exec_prefix),
822 "w9xpopen.exe")
823 if not os.path.exists(w9xpopen):
824 raise RuntimeError("Cannot locate w9xpopen.exe, which is "
825 "needed for Popen to work with your "
826 "shell or platform.")
827 return w9xpopen
830 def _execute_child(self, args, executable, preexec_fn, close_fds,
831 cwd, env, universal_newlines,
832 startupinfo, creationflags, shell,
833 p2cread, p2cwrite,
834 c2pread, c2pwrite,
835 errread, errwrite):
836 """Execute program (MS Windows version)"""
838 if not isinstance(args, types.StringTypes):
839 args = list2cmdline(args)
841 # Process startup details
842 if startupinfo is None:
843 startupinfo = STARTUPINFO()
844 if None not in (p2cread, c2pwrite, errwrite):
845 startupinfo.dwFlags |= STARTF_USESTDHANDLES
846 startupinfo.hStdInput = p2cread
847 startupinfo.hStdOutput = c2pwrite
848 startupinfo.hStdError = errwrite
850 if shell:
851 startupinfo.dwFlags |= STARTF_USESHOWWINDOW
852 startupinfo.wShowWindow = SW_HIDE
853 comspec = os.environ.get("COMSPEC", "cmd.exe")
854 args = comspec + " /c " + args
855 if (GetVersion() >= 0x80000000L or
856 os.path.basename(comspec).lower() == "command.com"):
857 # Win9x, or using command.com on NT. We need to
858 # use the w9xpopen intermediate program. For more
859 # information, see KB Q150956
860 # (http://web.archive.org/web/20011105084002/http://support.microsoft.com/support/kb/articles/Q150/9/56.asp)
861 w9xpopen = self._find_w9xpopen()
862 args = '"%s" %s' % (w9xpopen, args)
863 # Not passing CREATE_NEW_CONSOLE has been known to
864 # cause random failures on win9x. Specifically a
865 # dialog: "Your program accessed mem currently in
866 # use at xxx" and a hopeful warning about the
867 # stability of your system. Cost is Ctrl+C wont
868 # kill children.
869 creationflags |= CREATE_NEW_CONSOLE
871 # Start the process
872 try:
873 hp, ht, pid, tid = CreateProcess(executable, args,
874 # no special security
875 None, None,
876 int(not close_fds),
877 creationflags,
878 env,
879 cwd,
880 startupinfo)
881 except pywintypes.error, e:
882 # Translate pywintypes.error to WindowsError, which is
883 # a subclass of OSError. FIXME: We should really
884 # translate errno using _sys_errlist (or simliar), but
885 # how can this be done from Python?
886 raise WindowsError(*e.args)
888 # Retain the process handle, but close the thread handle
889 self._child_created = True
890 self._handle = hp
891 self.pid = pid
892 ht.Close()
894 # Child is launched. Close the parent's copy of those pipe
895 # handles that only the child should have open. You need
896 # to make sure that no handles to the write end of the
897 # output pipe are maintained in this process or else the
898 # pipe will not close when the child process exits and the
899 # ReadFile will hang.
900 if p2cread is not None:
901 p2cread.Close()
902 if c2pwrite is not None:
903 c2pwrite.Close()
904 if errwrite is not None:
905 errwrite.Close()
908 def _internal_poll(self, _deadstate=None):
909 """Check if child process has terminated. Returns returncode
910 attribute."""
911 if self.returncode is None:
912 if WaitForSingleObject(self._handle, 0) == WAIT_OBJECT_0:
913 self.returncode = GetExitCodeProcess(self._handle)
914 return self.returncode
917 def wait(self):
918 """Wait for child process to terminate. Returns returncode
919 attribute."""
920 if self.returncode is None:
921 obj = WaitForSingleObject(self._handle, INFINITE)
922 self.returncode = GetExitCodeProcess(self._handle)
923 return self.returncode
926 def _readerthread(self, fh, buffer):
927 buffer.append(fh.read())
930 def _communicate(self, input):
931 stdout = None # Return
932 stderr = None # Return
934 if self.stdout:
935 stdout = []
936 stdout_thread = threading.Thread(target=self._readerthread,
937 args=(self.stdout, stdout))
938 stdout_thread.setDaemon(True)
939 stdout_thread.start()
940 if self.stderr:
941 stderr = []
942 stderr_thread = threading.Thread(target=self._readerthread,
943 args=(self.stderr, stderr))
944 stderr_thread.setDaemon(True)
945 stderr_thread.start()
947 if self.stdin:
948 if input is not None:
949 self.stdin.write(input)
950 self.stdin.close()
952 if self.stdout:
953 stdout_thread.join()
954 if self.stderr:
955 stderr_thread.join()
957 # All data exchanged. Translate lists into strings.
958 if stdout is not None:
959 stdout = stdout[0]
960 if stderr is not None:
961 stderr = stderr[0]
963 # Translate newlines, if requested. We cannot let the file
964 # object do the translation: It is based on stdio, which is
965 # impossible to combine with select (unless forcing no
966 # buffering).
967 if self.universal_newlines and hasattr(file, 'newlines'):
968 if stdout:
969 stdout = self._translate_newlines(stdout)
970 if stderr:
971 stderr = self._translate_newlines(stderr)
973 self.wait()
974 return (stdout, stderr)
976 def send_signal(self, sig):
977 """Send a signal to the process
979 if sig == signal.SIGTERM:
980 self.terminate()
981 else:
982 raise ValueError("Only SIGTERM is supported on Windows")
984 def terminate(self):
985 """Terminates the process
987 TerminateProcess(self._handle, 1)
989 kill = terminate
991 else:
993 # POSIX methods
995 def _get_handles(self, stdin, stdout, stderr):
996 """Construct and return tuple with IO objects:
997 p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite
999 p2cread, p2cwrite = None, None
1000 c2pread, c2pwrite = None, None
1001 errread, errwrite = None, None
1003 if stdin is None:
1004 pass
1005 elif stdin == PIPE:
1006 p2cread, p2cwrite = os.pipe()
1007 elif isinstance(stdin, int):
1008 p2cread = stdin
1009 else:
1010 # Assuming file-like object
1011 p2cread = stdin.fileno()
1013 if stdout is None:
1014 pass
1015 elif stdout == PIPE:
1016 c2pread, c2pwrite = os.pipe()
1017 elif isinstance(stdout, int):
1018 c2pwrite = stdout
1019 else:
1020 # Assuming file-like object
1021 c2pwrite = stdout.fileno()
1023 if stderr is None:
1024 pass
1025 elif stderr == PIPE:
1026 errread, errwrite = os.pipe()
1027 elif stderr == STDOUT:
1028 errwrite = c2pwrite
1029 elif isinstance(stderr, int):
1030 errwrite = stderr
1031 else:
1032 # Assuming file-like object
1033 errwrite = stderr.fileno()
1035 return (p2cread, p2cwrite,
1036 c2pread, c2pwrite,
1037 errread, errwrite)
1040 def _set_cloexec_flag(self, fd):
1041 try:
1042 cloexec_flag = fcntl.FD_CLOEXEC
1043 except AttributeError:
1044 cloexec_flag = 1
1046 old = fcntl.fcntl(fd, fcntl.F_GETFD)
1047 fcntl.fcntl(fd, fcntl.F_SETFD, old | cloexec_flag)
1050 def _close_fds(self, but):
1051 if hasattr(os, 'closerange'):
1052 os.closerange(3, but)
1053 os.closerange(but + 1, MAXFD)
1054 else:
1055 for i in xrange(3, MAXFD):
1056 if i == but:
1057 continue
1058 try:
1059 os.close(i)
1060 except:
1061 pass
1064 def _execute_child(self, args, executable, preexec_fn, close_fds,
1065 cwd, env, universal_newlines,
1066 startupinfo, creationflags, shell,
1067 p2cread, p2cwrite,
1068 c2pread, c2pwrite,
1069 errread, errwrite):
1070 """Execute program (POSIX version)"""
1072 if isinstance(args, types.StringTypes):
1073 args = [args]
1074 else:
1075 args = list(args)
1077 if shell:
1078 args = ["/bin/sh", "-c"] + args
1080 if executable is None:
1081 executable = args[0]
1083 # For transferring possible exec failure from child to parent
1084 # The first char specifies the exception type: 0 means
1085 # OSError, 1 means some other error.
1086 errpipe_read, errpipe_write = os.pipe()
1087 try:
1088 try:
1089 self._set_cloexec_flag(errpipe_write)
1091 gc_was_enabled = gc.isenabled()
1092 # Disable gc to avoid bug where gc -> file_dealloc ->
1093 # write to stderr -> hang. http://bugs.python.org/issue1336
1094 gc.disable()
1095 try:
1096 self.pid = os.fork()
1097 except:
1098 if gc_was_enabled:
1099 gc.enable()
1100 raise
1101 self._child_created = True
1102 if self.pid == 0:
1103 # Child
1104 try:
1105 # Close parent's pipe ends
1106 if p2cwrite is not None:
1107 os.close(p2cwrite)
1108 if c2pread is not None:
1109 os.close(c2pread)
1110 if errread is not None:
1111 os.close(errread)
1112 os.close(errpipe_read)
1114 # Dup fds for child
1115 if p2cread is not None:
1116 os.dup2(p2cread, 0)
1117 if c2pwrite is not None:
1118 os.dup2(c2pwrite, 1)
1119 if errwrite is not None:
1120 os.dup2(errwrite, 2)
1122 # Close pipe fds. Make sure we don't close the same
1123 # fd more than once, or standard fds.
1124 if p2cread is not None and p2cread not in (0,):
1125 os.close(p2cread)
1126 if c2pwrite is not None and c2pwrite not in (p2cread, 1):
1127 os.close(c2pwrite)
1128 if errwrite is not None and errwrite not in (p2cread, c2pwrite, 2):
1129 os.close(errwrite)
1131 # Close all other fds, if asked for
1132 if close_fds:
1133 self._close_fds(but=errpipe_write)
1135 if cwd is not None:
1136 os.chdir(cwd)
1138 if preexec_fn:
1139 preexec_fn()
1141 if env is None:
1142 os.execvp(executable, args)
1143 else:
1144 os.execvpe(executable, args, env)
1146 except:
1147 exc_type, exc_value, tb = sys.exc_info()
1148 # Save the traceback and attach it to the exception object
1149 exc_lines = traceback.format_exception(exc_type,
1150 exc_value,
1152 exc_value.child_traceback = ''.join(exc_lines)
1153 os.write(errpipe_write, pickle.dumps(exc_value))
1155 # This exitcode won't be reported to applications, so it
1156 # really doesn't matter what we return.
1157 os._exit(255)
1159 # Parent
1160 if gc_was_enabled:
1161 gc.enable()
1162 finally:
1163 # be sure the FD is closed no matter what
1164 os.close(errpipe_write)
1166 if p2cread is not None and p2cwrite is not None:
1167 os.close(p2cread)
1168 if c2pwrite is not None and c2pread is not None:
1169 os.close(c2pwrite)
1170 if errwrite is not None and errread is not None:
1171 os.close(errwrite)
1173 # Wait for exec to fail or succeed; possibly raising exception
1174 data = os.read(errpipe_read, 1048576) # Exception limited to 1M
1175 finally:
1176 # be sure the FD is closed no matter what
1177 os.close(errpipe_read)
1179 if data != "":
1180 os.waitpid(self.pid, 0)
1181 child_exception = pickle.loads(data)
1182 for fd in (p2cwrite, c2pread, errread):
1183 if fd is not None:
1184 os.close(fd)
1185 raise child_exception
1188 def _handle_exitstatus(self, sts):
1189 if os.WIFSIGNALED(sts):
1190 self.returncode = -os.WTERMSIG(sts)
1191 elif os.WIFEXITED(sts):
1192 self.returncode = os.WEXITSTATUS(sts)
1193 else:
1194 # Should never happen
1195 raise RuntimeError("Unknown child exit status!")
1198 def _internal_poll(self, _deadstate=None):
1199 """Check if child process has terminated. Returns returncode
1200 attribute."""
1201 if self.returncode is None:
1202 try:
1203 pid, sts = os.waitpid(self.pid, os.WNOHANG)
1204 if pid == self.pid:
1205 self._handle_exitstatus(sts)
1206 except os.error:
1207 if _deadstate is not None:
1208 self.returncode = _deadstate
1209 return self.returncode
1212 def wait(self):
1213 """Wait for child process to terminate. Returns returncode
1214 attribute."""
1215 if self.returncode is None:
1216 pid, sts = os.waitpid(self.pid, 0)
1217 self._handle_exitstatus(sts)
1218 return self.returncode
1221 def _communicate(self, input):
1222 if self.stdin:
1223 # Flush stdio buffer. This might block, if the user has
1224 # been writing to .stdin in an uncontrolled fashion.
1225 self.stdin.flush()
1226 if not input:
1227 self.stdin.close()
1229 if _has_poll:
1230 stdout, stderr = self._communicate_with_poll(input)
1231 else:
1232 stdout, stderr = self._communicate_with_select(input)
1234 # All data exchanged. Translate lists into strings.
1235 if stdout is not None:
1236 stdout = ''.join(stdout)
1237 if stderr is not None:
1238 stderr = ''.join(stderr)
1240 # Translate newlines, if requested. We cannot let the file
1241 # object do the translation: It is based on stdio, which is
1242 # impossible to combine with select (unless forcing no
1243 # buffering).
1244 if self.universal_newlines and hasattr(file, 'newlines'):
1245 if stdout:
1246 stdout = self._translate_newlines(stdout)
1247 if stderr:
1248 stderr = self._translate_newlines(stderr)
1250 self.wait()
1251 return (stdout, stderr)
1254 def _communicate_with_poll(self, input):
1255 stdout = None # Return
1256 stderr = None # Return
1257 fd2file = {}
1258 fd2output = {}
1260 poller = select.poll()
1261 def register_and_append(file_obj, eventmask):
1262 poller.register(file_obj.fileno(), eventmask)
1263 fd2file[file_obj.fileno()] = file_obj
1265 def close_unregister_and_remove(fd):
1266 poller.unregister(fd)
1267 fd2file[fd].close()
1268 fd2file.pop(fd)
1270 if self.stdin and input:
1271 register_and_append(self.stdin, select.POLLOUT)
1273 select_POLLIN_POLLPRI = select.POLLIN | select.POLLPRI
1274 if self.stdout:
1275 register_and_append(self.stdout, select_POLLIN_POLLPRI)
1276 fd2output[self.stdout.fileno()] = stdout = []
1277 if self.stderr:
1278 register_and_append(self.stderr, select_POLLIN_POLLPRI)
1279 fd2output[self.stderr.fileno()] = stderr = []
1281 input_offset = 0
1282 while fd2file:
1283 try:
1284 ready = poller.poll()
1285 except select.error, e:
1286 if e.args[0] == errno.EINTR:
1287 continue
1288 raise
1290 for fd, mode in ready:
1291 if mode & select.POLLOUT:
1292 chunk = input[input_offset : input_offset + _PIPE_BUF]
1293 input_offset += os.write(fd, chunk)
1294 if input_offset >= len(input):
1295 close_unregister_and_remove(fd)
1296 elif mode & select_POLLIN_POLLPRI:
1297 data = os.read(fd, 4096)
1298 if not data:
1299 close_unregister_and_remove(fd)
1300 fd2output[fd].append(data)
1301 else:
1302 # Ignore hang up or errors.
1303 close_unregister_and_remove(fd)
1305 return (stdout, stderr)
1308 def _communicate_with_select(self, input):
1309 read_set = []
1310 write_set = []
1311 stdout = None # Return
1312 stderr = None # Return
1314 if self.stdin and input:
1315 write_set.append(self.stdin)
1316 if self.stdout:
1317 read_set.append(self.stdout)
1318 stdout = []
1319 if self.stderr:
1320 read_set.append(self.stderr)
1321 stderr = []
1323 input_offset = 0
1324 while read_set or write_set:
1325 try:
1326 rlist, wlist, xlist = select.select(read_set, write_set, [])
1327 except select.error, e:
1328 if e.args[0] == errno.EINTR:
1329 continue
1330 raise
1332 if self.stdin in wlist:
1333 chunk = input[input_offset : input_offset + _PIPE_BUF]
1334 bytes_written = os.write(self.stdin.fileno(), chunk)
1335 input_offset += bytes_written
1336 if input_offset >= len(input):
1337 self.stdin.close()
1338 write_set.remove(self.stdin)
1340 if self.stdout in rlist:
1341 data = os.read(self.stdout.fileno(), 1024)
1342 if data == "":
1343 self.stdout.close()
1344 read_set.remove(self.stdout)
1345 stdout.append(data)
1347 if self.stderr in rlist:
1348 data = os.read(self.stderr.fileno(), 1024)
1349 if data == "":
1350 self.stderr.close()
1351 read_set.remove(self.stderr)
1352 stderr.append(data)
1354 return (stdout, stderr)
1357 def send_signal(self, sig):
1358 """Send a signal to the process
1360 os.kill(self.pid, sig)
1362 def terminate(self):
1363 """Terminate the process with SIGTERM
1365 self.send_signal(signal.SIGTERM)
1367 def kill(self):
1368 """Kill the process with SIGKILL
1370 self.send_signal(signal.SIGKILL)
1373 def _demo_posix():
1375 # Example 1: Simple redirection: Get process list
1377 plist = Popen(["ps"], stdout=PIPE).communicate()[0]
1378 print "Process list:"
1379 print plist
1382 # Example 2: Change uid before executing child
1384 if os.getuid() == 0:
1385 p = Popen(["id"], preexec_fn=lambda: os.setuid(100))
1386 p.wait()
1389 # Example 3: Connecting several subprocesses
1391 print "Looking for 'hda'..."
1392 p1 = Popen(["dmesg"], stdout=PIPE)
1393 p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
1394 print repr(p2.communicate()[0])
1397 # Example 4: Catch execution error
1399 print
1400 print "Trying a weird file..."
1401 try:
1402 print Popen(["/this/path/does/not/exist"]).communicate()
1403 except OSError, e:
1404 if e.errno == errno.ENOENT:
1405 print "The file didn't exist. I thought so..."
1406 print "Child traceback:"
1407 print e.child_traceback
1408 else:
1409 print "Error", e.errno
1410 else:
1411 print >>sys.stderr, "Gosh. No error."
1414 def _demo_windows():
1416 # Example 1: Connecting several subprocesses
1418 print "Looking for 'PROMPT' in set output..."
1419 p1 = Popen("set", stdout=PIPE, shell=True)
1420 p2 = Popen('find "PROMPT"', stdin=p1.stdout, stdout=PIPE)
1421 print repr(p2.communicate()[0])
1424 # Example 2: Simple execution of program
1426 print "Executing calc..."
1427 p = Popen("calc")
1428 p.wait()
1431 if __name__ == "__main__":
1432 if mswindows:
1433 _demo_windows()
1434 else:
1435 _demo_posix()