Question or problem about Python programming:
To launch programs from my Python-scripts, I’m using the following method:
def execute(command): process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) output = process.communicate()[0] exitCode = process.returncode if (exitCode == 0): return output else: raise ProcessException(command, exitCode, output)
So when i launch a process like Process.execute(“mvn clean install”), my program waits until the process is finished, and only then i get the complete output of my program. This is annoying if i’m running a process that takes a while to finish.
Can i let my program write the process output line by line, by polling the process output before it finishes in a loop or something?
**[EDIT]
Sorry i didn’t search very well before posting this question. Threading is actually the key. Found an example here which shows how to do it: **
Python Subprocess.Popen from a thread
How to solve the problem:
Solution 1:
You can use iter to process lines as soon as the command outputs them: lines = iter(fd.readline, "")
. Here’s a full example showing a typical use case (thanks to @jfs for helping out):
from __future__ import print_function # Only Python 2.x import subprocess def execute(cmd): popen = subprocess.Popen(cmd, stdout=subprocess.PIPE, universal_newlines=True) for stdout_line in iter(popen.stdout.readline, ""): yield stdout_line popen.stdout.close() return_code = popen.wait() if return_code: raise subprocess.CalledProcessError(return_code, cmd) # Example for path in execute(["locate", "a"]): print(path, end="")
Solution 2:
Ok i managed to solve it without threads (any suggestions why using threads would be better are appreciated) by using a snippet from this question Intercepting stdout of a subprocess while it is running
def execute(command): process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) # Poll process for new output until finished while True: nextline = process.stdout.readline() if nextline == '' and process.poll() is not None: break sys.stdout.write(nextline) sys.stdout.flush() output = process.communicate()[0] exitCode = process.returncode if (exitCode == 0): return output else: raise ProcessException(command, exitCode, output)
Solution 3:
To print subprocess’ output line-by-line as soon as its stdout buffer is flushed in Python 3:
from subprocess import Popen, PIPE, CalledProcessError with Popen(cmd, stdout=PIPE, bufsize=1, universal_newlines=True) as p: for line in p.stdout: print(line, end='') # process line here if p.returncode != 0: raise CalledProcessError(p.returncode, p.args)
Notice: you do not need p.poll()
— the loop ends when eof is reached. And you do not need iter(p.stdout.readline, '')
— the read-ahead bug is fixed in Python 3.
See also, Python: read streaming input from subprocess.communicate().
Solution 4:
There is actually a really simple way to do this when you just want to print the output:
import subprocess import sys def execute(command): subprocess.check_call(command, stdout=sys.stdout, stderr=subprocess.STDOUT)
Here we’re simply pointing the subprocess to our own stdout, and using existing succeed or exception api.
Solution 5:
@tokland
tried your code and corrected it for 3.4 and windows
dir.cmd is a simple dir command, saved as cmd-file
import subprocess c = "dir.cmd" def execute(command): popen = subprocess.Popen(command, stdout=subprocess.PIPE,bufsize=1) lines_iterator = iter(popen.stdout.readline, b"") while popen.poll() is None: for line in lines_iterator: nline = line.rstrip() print(nline.decode("latin"), end = "\r\n",flush =True) # yield line execute(c)