I Spent 40 Minutes at 11pm Debugging a Deploy That Wasn’t Broken

The whole engine is until "$@"; do ... done. until runs the command and executes the loop body only when it fails, exiting the instant it succeeds. Passing the command as "$@" (after shift-ing past the attempt count) means the function retries anything — a curl, an ssh, a port check, your own script — without caring what it is.

The backoff is the three lines at the bottom of the loop: sleep for the current delay plus a little jitter, then double the delay, capped at max_delay. That gives you 1s, 2s, 4s, 8s, 16s, 30s, 30s…

The jitter is not decoration

RANDOM % 3 looks trivial, and it’s the line people delete to “clean up.” Don’t. Without jitter, a fleet of machines that all failed at the same instant — because the same service went down — will all retry at the same instant, and the same instant after that, producing a synchronized thundering herd that knocks the recovering service straight back over on every round. A few hundred milliseconds of randomness per client spreads the retries out so the service actually gets room to recover. At one machine it does nothing; at fifty it’s the difference between recovery and a retry storm.

The mistake that makes retries dangerous

# Good: a transient failure that retrying can fix retry 6 nc -z -w 2 db.internal 5432 # Bad: retrying a deterministic failure just delays the error 30 seconds retry 6 curl -fsS https://api.example.com/v1/thing-that-returns-404 

A retry loop is only as smart as what you point it at. A port check belongs in a loop because the answer changes — “no” until the database boots, then “yes.” A request that returns 404 returns 404 on attempt one and attempt six; the loop just postpones the failure and buries the real status under retry noise. Retry transient failures — timeouts, connection-refused, 429, 5xx, DNS hiccups. Don’t retry deterministic ones — a 404, a 401, a syntax error, a missing file. When you can, branch on the exit code or HTTP status and loop only on the codes worth looping on.

For plain curl, its built-in --retry 5 --retry-delay 2 does most of this and is simpler; reach for the function when the thing you’re retrying isn’t curl, or when you want one backoff policy across a database probe, an ssh call, and a download at once.

Back to 11pm

That deploy never paged me again once the migration waited for the port instead of assuming it. The database still took its six seconds to boot, the network still blipped occasionally — retrying didn’t make the dependencies faster. It stopped a normal, transient slowness from being treated as a fatal error, which is most of what “production-ready” means for a script.

Full function with the wait-for-port pattern and the guidance on which failures to retry: https://bashsnippets.xyz/snippets/bash-retry-with-backoff

Retries are the third leg of an unattended job: flock stops overlap, timeout stops hangs, retry rides out the blip. The Hardened Cron Wrapper Generator wires all three into one wrapper, Bash Scripts That Survive Cron is the end-to-end version, and the rest of the library is at https://bashsnippets.xyz