read subprocess stdout line by line


My python script uses subprocess to call a linux utility that is very noisy. I want to store all of the output to a log file and show some of it to the user. I thought the following would work, but the output doesn't show up in my application until the utility has produced a significant amount of output., just generates lots of output over time
import time
i = 0
while True:
   print hex(i)*512
   i += 1

#filters output
import subprocess
proc = subprocess.Popen(['python',''],stdout=subprocess.PIPE)
for line in proc.stdout:
   #the real code does filtering here
   print "test:", line.rstrip()

The behavior I really want is for the filter script to print each line as it is received from the subprocess. Sorta like what tee does but with python code.

What am I missing? Is this even possible?


If a sys.stdout.flush() is added to, the code has the desired behavior in python 3.1. I'm using python 2.6. You would think that using proc.stdout.xreadlines() would work the same as py3k, but it doesn't.

Update 2:

Here is the minimal working code., just generates lots of output over time
import sys, time
for i in range(10):
   print i

#display out put line by line
import subprocess
proc = subprocess.Popen(['python',''],stdout=subprocess.PIPE)
#works in python 3.0+
#for line in proc.stdout:
for line in iter(proc.stdout.readline,''):
   print line.rstrip()
12/29/2014 4:19:24 PM

Accepted Answer

It's been a long time since I last worked with Python, but I think the problem is with the statement for line in proc.stdout, which reads the entire input before iterating over it. The solution is to use readline() instead:

#filters output
import subprocess
proc = subprocess.Popen(['python',''],stdout=subprocess.PIPE)
while True:
  line = proc.stdout.readline()
  if not line:
  #the real code does filtering here
  print "test:", line.rstrip()

Of course you still have to deal with the subprocess' buffering.

Note: according to the documentation the solution with an iterator should be equivalent to using readline(), except for the read-ahead buffer, but (or exactly because of this) the proposed change did produce different results for me (Python 2.5 on Windows XP).

6/2/2019 4:02:44 PM

Bit late to the party, but was surprised not to see what I think is the simplest solution here:

import io
import subprocess

proc = subprocess.Popen(["prog", "arg"], stdout=subprocess.PIPE)
for line in io.TextIOWrapper(proc.stdout, encoding="utf-8"):  # or another encoding
    # do something with line

Licensed under: CC-BY-SA with attribution
Not affiliated with: Stack Overflow