How do I redirect stdout to an arbitrary file in Python?
When a long-running Python script (e.g, web application) is started from within the ssh session and backgounded, and the ssh session is closed, the application will raise IOError and fail the moment it tries to write to stdout. I needed to find a way to make the application and modules output to a file rather than stdout to prevent failure due to IOError. Currently, I employ nohup to redirect output to a file, and that gets the job done, but I was wondering if there was a way to do it without using nohup, out of curiosity.
I have already tried
sys.stdout = open('somefile', 'w'), but this does not seem to prevent some external modules from still outputting to terminal (or maybe the
sys.stdout = ... line did not fire at all). I know it should work from simpler scripts I've tested on, but I also didn't have time yet to test on a web application yet.
If you want to do the redirection within the Python script, set
sys.stdout to an file object does the trick:
import sys sys.stdout = open('file', 'w') print('test')
A far more common method is to use shell redirection when executing (same on Windows and Linux):
$ python foo.py > file
contextlib.redirect_stdout() function in Python 3.4:
from contextlib import redirect_stdout with open('help.txt', 'w') as f: with redirect_stdout(f): print('it now prints to `help.text`')
It is similar to:
import sys from contextlib import contextmanager @contextmanager def redirect_stdout(new_target): old_target, sys.stdout = sys.stdout, new_target # replace sys.stdout try: yield new_target # run some code with the replaced stdout finally: sys.stdout = old_target # restore to the previous value
that can be used on earlier Python versions. The latter version is not reusable. It can be made one if desired.
It doesn't redirect the stdout at the file descriptors level e.g.:
import os from contextlib import redirect_stdout stdout_fd = sys.stdout.fileno() with open('output.txt', 'w') as f, redirect_stdout(f): print('redirected to a file') os.write(stdout_fd, b'not redirected') os.system('echo this also is not redirected')
b'not redirected' and
'echo this also is not redirected' are not redirected to the
To redirect at the file descriptor level,
os.dup2() could be used:
import os import sys from contextlib import contextmanager def fileno(file_or_fd): fd = getattr(file_or_fd, 'fileno', lambda: file_or_fd)() if not isinstance(fd, int): raise ValueError("Expected a file (`.fileno()`) or a file descriptor") return fd @contextmanager def stdout_redirected(to=os.devnull, stdout=None): if stdout is None: stdout = sys.stdout stdout_fd = fileno(stdout) # copy stdout_fd before it is overwritten #NOTE: `copied` is inheritable on Windows when duplicating a standard stream with os.fdopen(os.dup(stdout_fd), 'wb') as copied: stdout.flush() # flush library buffers that dup2 knows nothing about try: os.dup2(fileno(to), stdout_fd) # $ exec >&to except ValueError: # filename with open(to, 'wb') as to_file: os.dup2(to_file.fileno(), stdout_fd) # $ exec > to try: yield stdout # allow code to be run with the redirected stdout finally: # restore stdout to its previous value #NOTE: dup2 makes stdout_fd inheritable unconditionally stdout.flush() os.dup2(copied.fileno(), stdout_fd) # $ exec >&copied
The same example works now if
stdout_redirected() is used instead of
import os import sys stdout_fd = sys.stdout.fileno() with open('output.txt', 'w') as f, stdout_redirected(f): print('redirected to a file') os.write(stdout_fd, b'it is redirected now\n') os.system('echo this is also redirected') print('this is goes back to stdout')
The output that previously was printed on stdout now goes to
output.txt as long as
stdout_redirected() context manager is active.
stdout.flush() does not flush
C stdio buffers on Python 3 where I/O is implemented directly on
write() system calls. To flush all open C stdio output streams, you could call
libc.fflush(None) explicitly if some C extension uses stdio-based I/O:
try: import ctypes from ctypes.util import find_library except ImportError: libc = None else: try: libc = ctypes.cdll.msvcrt # Windows except OSError: libc = ctypes.cdll.LoadLibrary(find_library('c')) def flush(stream): try: libc.fflush(None) stream.flush() except (AttributeError, ValueError, IOError): pass # unsupported
You could use
stdout parameter to redirect other streams, not only
sys.stdout e.g., to merge
def merged_stderr_stdout(): # $ exec 2>&1 return stdout_redirected(to=sys.stdout, stdout=sys.stderr)
from __future__ import print_function import sys with merged_stderr_stdout(): print('this is printed on stdout') print('this is also printed on stdout', file=sys.stderr)
To answer, your edit: you could use
python-daemon to daemonize your script and use
logging module (as @erikb85 suggested) instead of