My python script uses subprocess to call a linux utility that is very noisy. I want to store all of the output to a log file and show some of it to the user. I thought the following would work, but the output doesn't show up in my application until the utility has produced a significant amount of output.
#fake_utility.py, just generates lots of output over time import time i = 0 while True: print hex(i)*512 i += 1 time.sleep(0.5) #filters output import subprocess proc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE) for line in proc.stdout: #the real code does filtering here print "test:", line.rstrip()
The behavior I really want is for the filter script to print each line as it is received from the subprocess. Sorta like what
tee does but with python code.
What am I missing? Is this even possible?
sys.stdout.flush() is added to fake_utility.py, the code has the desired behavior in python 3.1. I'm using python 2.6. You would think that using
proc.stdout.xreadlines() would work the same as py3k, but it doesn't.
Here is the minimal working code.
#fake_utility.py, just generates lots of output over time import sys, time for i in range(10): print i sys.stdout.flush() time.sleep(0.5) #display out put line by line import subprocess proc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE) #works in python 3.0+ #for line in proc.stdout: for line in iter(proc.stdout.readline,''): print line.rstrip()
It's been a long time since I last worked with Python, but I think the problem is with the statement
for line in proc.stdout, which reads the entire input before iterating over it. The solution is to use
#filters output import subprocess proc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE) while True: line = proc.stdout.readline() if not line: break #the real code does filtering here print "test:", line.rstrip()
Of course you still have to deal with the subprocess' buffering.
Note: according to the documentation the solution with an iterator should be equivalent to using
readline(), except for the read-ahead buffer, but (or exactly because of this) the proposed change did produce different results for me (Python 2.5 on Windows XP).
Bit late to the party, but was surprised not to see what I think is the simplest solution here:
import io import subprocess proc = subprocess.Popen(["prog", "arg"], stdout=subprocess.PIPE) for line in io.TextIOWrapper(proc.stdout, encoding="utf-8"): # or another encoding # do something with line