The bug is discussed here:
Transcript to reproduce in Python 2.6 .5:
>>>
import subprocess, sys >>>
subprocess.call(('echo', 'foo'), stderr = sys.stdout)
echo: write: Bad file descriptor
1
>>>
Expected behavior:
>>>
import subprocess, sys >>>
subprocess.call(('echo', 'foo'), stderr = sys.stdout)
foo
0
>>>This happens because we've asked the child's stderr to be redirected, but not its stdout. So in _execute_child, errwrite is 1 while c2pwrite is None. So fd 1 (errwrite) correctly gets duped to 2. But then, since errwrite is not None and it's not in (p2cread, c2pwrite, 2), the child closes fd 1.
The equivalent thing happens if you supply stdout=sys.stderr and the child attempts to write to its stderr.
I've attached a patch to fix this. It simply adds 2 and 2 to the list of fds not to close for c2pwrite and errwrite, respectively.
This patch is against the 2.6.5 release.
There is also a workaround, in case anyone else is affected by this bug before a fix has been released:
>>>
import os, subprocess, sys
>>>
subprocess.call(('echo', 'foo'), stderr = os.dup(sys.stdout.fileno()))
foo
0
>>>If check is true, and the process exits with a non-zero exit code, a CalledProcessError exception will be raised. Attributes of that exception hold the arguments, the exit code, and stdout and stderr if they were captured.,In addition, the replacements using check_output() will fail with a CalledProcessError if the requested operation produces a non-zero return code. The output is still available as the output attribute of the raised exception.,If the process does not terminate after timeout seconds, a TimeoutExpired exception will be raised. Catching this exception and retrying communication will not lose any output.,If the process does not terminate after timeout seconds, raise a TimeoutExpired exception. It is safe to catch this exception and retry the wait.
os.system os.spawn *
>>> subprocess.run(["ls", "-l"]) # doesn 't capture output
CompletedProcess(args = ['ls', '-l'], returncode = 0)
>>>
subprocess.run("exit 1", shell = True, check = True)
Traceback(most recent call last):
...
subprocess.CalledProcessError: Command 'exit 1'
returned non - zero exit status 1
>>>
subprocess.run(["ls", "-l", "/dev/null"], capture_output = True)
CompletedProcess(args = ['ls', '-l', '/dev/null'], returncode = 0,
stdout = b 'crw-rw-rw- 1 root root 1, 3 Jan 23 16:23 /dev/null\n', stderr = b '')Popen(["/usr/bin/git", "commit", "-m", "Fixes a bug."])>>> import shlex, subprocess >>> command_line = input() / bin / vikings - input eggs.txt - output "spam spam.txt" - cmd "echo '$MONEY'" >>> args = shlex.split(command_line) >>> print(args)['/bin/vikings', '-input', 'eggs.txt', '-output', 'spam spam.txt', '-cmd', "echo '$MONEY'"] >>> p = subprocess.Popen(args) # Success!
Popen(['/bin/sh', '-c', args[0], args[1], ...])with Popen(["ifconfig"], stdout = PIPE) as proc:
log.write(proc.stdout.read())This issue tracker has been migrated to GitHub, and is currently read-only. For more information, see the GitHub FAQs in the Python's Developer Guide.
Hello,
Here is a code that exhibits an invalid behavior(Python 2.6 .6):
-- - 8 < -- -
import subprocess, os
os.close(0) # Works correctly
if any of these two are commented out.
os.close(2)
print subprocess.Popen('echo foo>&2', shell = True,
stdout = subprocess.PIPE,
stderr = subprocess.PIPE).communicate()
-- - > 8-- -
When run, the output is:
('', '')
While it should be:
('', 'foo\n')
When analyzing the code with strace the problem gets clearer:
$ strace - f - e pipe, fork, dup2, close. / Popen - bug.py[...]
5085 pipe([0, 2]) = 0 # Creates the pipes.
5085 pipe([3, 4]) = 0
5085 pipe([5, 6]) = 0[...] # In this skipped part Popen() closes useless pipe endpoints.
5086 dup2(2, 1) = 1 # stdout setup.
5086 dup2(4, 2) = 2 # stderr setup.
5086 close(2) = 0[...]
The last "close(2)"
is the error: apparently Popen() tries to close the remaining pipe endpoints(as should theoretically be done) but fails to see that the endpoint created by pipe([0, 2]) has already been closed during the previous dup2(4, 2) and that the file descriptor 2 is now the standard error.Therefore, Popen incorrectly closes the standard error.
To fix that, Popen should check, before closing the remaining pipe endpoints, that these endpoints are not the one that just get closed by the two previous dup2s.
Best regards,issue6610 looks very similar.issue9074 may also be related.
OK here is a patch + tests.Basically, it makes sure that the fd that it is closing is not 0, 1 or 2. I 've set it for 2.7, 3.1 and 3.2.
Fixed in issue10806.
This is basically just like the Popen class and takes all of the same arguments, but it simply wait until the command completes and gives us the return code.,Run the command described by args. Wait for command to complete, then return the returncode attribute.,Here is another example for a piped command. The code gets window ID for the currently active window. The command looks like this:,The underlying process creation and management in this module is handled by the Popen class. It offers a lot of flexibility so that developers are able to handle the less common cases not covered by the convenience functions.
Popen(['/bin/sh', '-c', args[0], args[1], ...])Special value that can be used as the stderr argument to Popen and indicates that standard error should go into the same handle as standard output.,Special value that can be used as the stdin, stdout or stderr argument to Popen and indicates that a pipe to the standard stream should be opened.,Do not use stdout=PIPE or stderr=PIPE with this function as that can deadlock based on the child process output volume. Use Popen with the communicate() method when you need pipes.,Do not use stderr=PIPE with this function as that can deadlock based on the child process error volume. Use Popen with the communicate() method when you need a stderr pipe.
os.system os.spawn * os.popen * popen2.* commands.*
>>> subprocess.call(["ls", "-l"])
0
>>>
subprocess.call("exit 1", shell = True)
1>>> subprocess.check_call(["ls", "-l"])
0
>>>
subprocess.check_call("exit 1", shell = True)
Traceback(most recent call last):
...
subprocess.CalledProcessError: Command 'exit 1'
returned non - zero exit status 1
Traceback(most recent call last):
...
subprocess.CalledProcessError: Command 'exit 1'
returned non - zero exit status 1>>> subprocess.check_output(["echo", "Hello World!"])
'Hello World!\n'
>>>
subprocess.check_output("exit 1", shell = True)
Traceback(most recent call last):
...
subprocess.CalledProcessError: Command 'exit 1'
returned non - zero exit status 1
Traceback(most recent call last):
...
subprocess.CalledProcessError: Command 'exit 1'
returned non - zero exit status 1>>> subprocess.check_output(
..."ls non_existent_file; exit 0",
...stderr = subprocess.STDOUT,
...shell = True)
'ls: non_existent_file: No such file or directory\n'>>> from subprocess
import call
>>>
filename = input("What file would you like to display?\n")
What file would you like to display ?
non_existent;
rm - rf / # >>>
call("cat " + filename, shell = True) # Uh - oh.This will end badly...Like Popen.wait(), this will deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data.,This will deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that.,Note that if you want to send data to the process’s stdin, you need to create the Popen object with stdin=PIPE. Similarly, to get anything other than None in the result tuple, you need to give stdout=PIPE and/or stderr=PIPE too.,If the stderr argument was PIPE, this attribute is a file object that provides error output from the child process. Otherwise, it is None.
os.system os.spawn * os.popen * popen2.* commands.*
>>> import shlex, subprocess >>> command_line = raw_input() / bin / vikings - input eggs.txt - output "spam spam.txt" - cmd "echo '$MONEY'" >>> args = shlex.split(command_line) >>> print args['/bin/vikings', '-input', 'eggs.txt', '-output', 'spam spam.txt', '-cmd', "echo '$MONEY'"] >>> p = subprocess.Popen(args) # Success!
Popen(['/bin/sh', '-c', args[0], args[1], ...])>>> from subprocess
import call
>>>
filename = input("What file would you like to display?\n")
What file would you like to display ?
non_existent;
rm - rf / # >>>
call("cat " + filename, shell = True) # Uh - oh.This will end badly...>>> retcode = subprocess.call(["ls", "-l"])>>> subprocess.check_call(["ls", "-l"])
0To also capture standard error in the result, use stderr=subprocess.STDOUT:,communicate() returns a tuple (stdout_data, stderr_data).,Command that was used to spawn the child process.,The subprocess module exposes the following constants.
os.system os.spawn *
>>> subprocess.call(["ls", "-l"])
0
>>>
subprocess.call("exit 1", shell = True)
1>>> subprocess.check_call(["ls", "-l"])
0
>>>
subprocess.check_call("exit 1", shell = True)
Traceback(most recent call last):
...
subprocess.CalledProcessError: Command 'exit 1'
returned non - zero exit status 1>>> subprocess.check_output(["echo", "Hello World!"])
b 'Hello World!\n'
>>>
subprocess.check_output(["echo", "Hello World!"], universal_newlines = True)
'Hello World!\n'
>>>
subprocess.check_output(["sed", "-e", "s/foo/bar/"],
...input = b "when in the course of fooman events\n")
b 'when in the course of barman events\n'
>>>
subprocess.check_output("exit 1", shell = True)
Traceback(most recent call last):
...
subprocess.CalledProcessError: Command 'exit 1'
returned non - zero exit status 1>>> subprocess.check_output(
..."ls non_existent_file; exit 0",
...stderr = subprocess.STDOUT,
...shell = True)
'ls: non_existent_file: No such file or directory\n'>>> import shlex, subprocess >>> command_line = input() / bin / vikings - input eggs.txt - output "spam spam.txt" - cmd "echo '$MONEY'" >>> args = shlex.split(command_line) >>> print(args)['/bin/vikings', '-input', 'eggs.txt', '-output', 'spam spam.txt', '-cmd', "echo '$MONEY'"] >>> p = subprocess.Popen(args) # Success!