The Remote Job Printed Nothing for 48 Hours. stdout Had No TTY.

Run it two ways before you touch pytest, Docker files, or an agent prompt.

python reproduce_buffer.py python reproduce_buffer.py | cat python -u reproduce_buffer.py | cat PYTHONUNBUFFERED=1 python reproduce_buffer.py | cat 

On a laptop TTY, the first command prints starting batch immediately, then sleeps, then raises RuntimeError. Piped through cat, that same print often stays hidden until the crash unwinds and the buffer finally flushes. stderr still speaks during the sleep, which is why my proof that the process was alive kept coming from the wrong stream.

Hour 12 to 24: I asked a model, then I verified without a TTY

I did not want another laptop-only green run, because a laptop TTY will lie about buffering every single time. Disclosure: This article was prepared as part of MonkeyCode’s product outreach. I used MonkeyCode’s free model access to list the real buffering knobs I should measure. Then I ran the same script on the free server option, where stdout was not my terminal. That pairing mattered more than any generated explanation: the model named candidates, and the non-TTY server made the failure visible.

The checklist I kept was short, and I refused to accept any item I had not measured on the stream itself.

• python -u
• PYTHONUNBUFFERED=1
• print(..., flush=True)
• sys.stdout.reconfigure(line_buffering=True)
• move heartbeats to stderr or to a FileHandler

The model was a checklist generator, not an oracle for my process table, and that distinction saved another wrong patch. I still printed isatty() on stderr, because a confident story you never measured is just fan fiction with syntax highlighting. If your remote runner allocates a pseudo-TTY, you will not see this bug, and you will distrust people who claim it exists. Ask the stream, not the dashboard, before you start renaming locks.

Hour 24 to 36: what actually broke

Three things broke during those two days, and only one of them was CPython’s default policy for pipes.

1. I treated print as a log line. It is a write to a buffered stream, and the policy depends on isatty().
2. I captured stdout through a pipe. CI, nohup, docker logs without a TTY, and subprocess.PIPE all take that path.
3. I trusted a local terminal session. My laptop was line-buffered, while the remote job was block-buffered.

CPython’s usual rule is simple enough to memorize, and I still forgot it the moment the log looked empty.

• Interactive stdout attached to a TTY is line-buffered, so a newline often appears right now.
• stdout attached to a pipe or a file is block-buffered, so you wait for a flush, a full buffer, or exit.
• stderr is typically unbuffered, which is why tracebacks arrive while ordinary prints do not.
• python -u or PYTHONUNBUFFERED=1 keeps stdin, stdout, and stderr unbuffered for that process.

Current CPython also lets you be explicit without changing the entire process personality.

import sys sys.stdout.reconfigure(line_buffering=True) print("starting batch") 

Or keep the process defaults and only flush the checkpoints you actually need to see.

print("starting batch", flush=True) 

I would not sprinkle flush=True on every debug line inside a hot inner loop that writes thousands of records. I would move those checkpoints to logging with a StreamHandler on stderr, or a FileHandler whose flush behavior I control. print() is a convenience for a TTY session, and I had been using it as infrastructure.

A decision table I wish I had at hour one

Copy that table into the runbook before you open a deadlock ticket against code that is merely quiet. The table is not a personality test for your service. It is a launch-environment checklist.

A tiny test plan you can keep in the repo

I do not unit-test CPython buffering itself, because that contract belongs to the interpreter and to the launch environment. I do test that a checkpoint I claim is durable cannot vanish into a pipe. The snippet below is a regression tripwire, not a proof that production logging is complete.

# test_checkpoint_stream.py import subprocess import sys SCRIPT = r""" import sys print("checkpoint") sys.stdout.flush() print("after-flush") """ def test_checkpoint_survives_a_pipe(): proc = subprocess.run( [sys.executable, "-c", SCRIPT], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, check=True, ) assert proc.stdout.splitlines()[0] == "checkpoint" 

Pair that test with a manual pipe run whenever somebody changes how the job is launched.

python -c "import sys; print(sys.stdout.isatty())" python -c "import sys; print(sys.stdout.isatty())" | cat 

If those two commands disagree, your mental model of the log is already wrong, and you should stop arguing from a laptop screenshot. Ask the stream. Do not ask the dashboard. A green local run with a TTY is not evidence that CI will ever see the same bytes.

What I would repeat, and what I would not

I would repeat the isatty() probe on stderr before I touch locks, retries, or an agent loop that looks stuck. I would repeat running the job under a pipe on purpose, because that is what CI will do later. I would repeat putting heartbeats on stderr or into a file, not into buffered stdout. I would not repeat waiting two days for a buffer to fill.

Should you export PYTHONUNBUFFERED=1 in every Dockerfile and every systemd unit as a new house style? Not if the process writes large binary payloads to stdout and you care about syscall cost. Not if a downstream consumer assumes block-sized writes and you just changed the framing. Not if you already have structured logging and you are only avoiding the migration. Unbuffered mode is a debugging lever, not a personality trait for every Python process.

Limitations you should actually respect when you copy this notebook:

• This write-up is about missing text checkpoints, not about a real deadlock if mutexes are involved.
• sys.stdout.reconfigure is a CPython convenience, so do not assume every interpreter you support implements it.
• A remote server without a TTY is a good reproduction box, not a substitute for your production log pipeline.
• Language models will invent a hung-thread story if you paste an empty log and ask them why.

Who should not use this approach? Anyone whose stdout is a protocol rather than a log, including tools that emit length-prefixed binary frames. Also skip the unbuffered hammer if your runtime already wraps the process with a PTY and you have measured isatty() as True.

Closing the notebook

The remote job was not frozen for forty-eight hours, because the interpreter was only being polite to a pipe. I was reading that politeness as death, which is an expensive way to learn about block buffering. Next time the log is empty and ps still shows Python, I will print isatty() on stderr before I name a villain. If you need a second machine that is not your laptop TTY, MonkeyCode’s free server option can reproduce this silence.