Same job. Same model, same resolution, same prompt length. At 10pm each unit took four minutes. By 3am the identical work took eleven.
Nothing had changed except the machine's memory state — and that turned out to be the entire story of the night. If part one was the skeleton of an overnight loop, this is what you bolt on so the loop doesn't quietly eat itself between midnight and dawn.
Measured on a 36GB M4 Max. Cutting resolution by 45% bought 5%. Memory state swung it 3×.
Read those four bars again, because they invert every instinct you have. The knob everyone reaches for — resolution — is noise. The knob nobody thinks about — whether the OS is compressing your working set — is a 3× multiplier.
Failure one: paying the cold start on every unit
The naive loop spawns a fresh process per unit. Clean, isolated, obviously correct. It's also catastrophic when the unit needs a 27GB model in memory, because you allocate and free 27GB on every single iteration.
On this box the free never fully happened. macOS accumulated compressed pages it wouldn't reclaim, and renders drifted from 389 seconds to 801 over one session — a self-inflicted slowdown that looked exactly like thermal throttling and wasn't.
The fix is a persistent worker. Load once, keep it resident, iterate inside the process:
model = ZImage(model_config=ModelConfig.z_image_turbo(), quantize=4, lora_paths=[LORA], lora_scales=[0.9])saver = MemorySaver(model=model, keep_transformer=True,1 cache_limit_bytes=1000 ** 3, num_seeds=len(todo))model.callbacks.register(saver)while True: if os.path.exists(STOP):2 os.remove(STOP); log("⏹ clean exit, nothing lost"); break todo = pending() # re-derive from disk, every iteration if not todo: break seed, dim, name, prompt = todo[0] t = time.time() try: model.generate_image(seed=seed, prompt=prompt, ...).save(path=out) except Exception as exc:3 log("❌ %s FAILED: %s" % (name, str(exc)[-250:])); continue log("✅ %s (%.0fs)" % (name, time.time() - t))keep_transformer=Trueis the whole point. Set it False and you reintroduce the per-unit 27GB reload this script exists to eliminate.- The stop flag is checked between units, never mid-unit — so a stop request never destroys work in flight.
- Not defensive padding. Unattended, one unhandled exception at 1am costs you five hours of queue. Catch, log, continue, count it in the morning summary.
Failure two: the thing you can only fix by leaving
Here's the counter-intuitive one. Once I had a resident model, I tried purging the memory compressor between units to keep it fast. It recovered 0.3GB. Useless.
Because the compressed pages belong to the live process. You cannot reclaim the memory of a program that is still running. Purge with nothing loaded and it frees tens of gigabytes; purge with the model resident and you're rearranging deck chairs.
So the worker deliberately kills itself on a schedule, and the babysitter brings it back:
recycle_after = int(cfg("recycle_after", 8))
if recycle_after and (ok + fail) >= recycle_after:
log("♻️ recycling after %d units so a purge can actually reclaim memory")
break # exit clean -> supervisor purges (nothing loaded) -> reloads warm in ~2s
The pattern
Persistent for throughput, periodically recycled for hygiene. Long-lived enough to skip the cold start eight times, short-lived enough that the OS gets a clean slate before the memory pressure compounds. Every long-running worker you have ever admired does some version of this — it's why PHP-FPM and Gunicorn ship max_requests.
Tune recycle_after to where your throughput curve starts bending. Mine bends at about eight.
Failure three: you will want to edit the queue at 1am
You'll spot a typo in unit 60's prompt. The old move is stop, edit, restart — and the restart costs a cold load plus whatever unit was in flight.
Instead, re-derive the work list from disk every iteration and hot-reload the module that declares it:
def pending(reload=False):
"""Units with no artefact yet, in priority order."""
if reload:
try:
importlib.reload(P) # picks up edits + brand-new units, live
except Exception as exc: # a syntax error mid-save must NOT kill the run
log("⚠️ reload failed (%s) — carrying on with the old list" % str(exc)[-120:])
return [j for j in ordered(P.UNITS) if not os.path.exists(artefact_for(j))]
Two details make this safe. The declarations file is pure data at module level, so re-importing has no side effects. And the reload is wrapped — because you will hit the moment the loop reads the file halfway through your save, and a syntax error must degrade to "keep using the old list", never to a dead night.
Same trick for tuning. Read the config inside the loop rather than at startup, and you can change quality settings live:
steps = max(1, int(cfg("steps", 9))) # re-read per unit -> quality tunable, no restart
At 11pm I dropped steps from 9 to 6 for volume. No restart, no lost unit, takes effect on the next iteration.
Failure four: the babysitter fights the stop button
Any serious night run has a supervisor that restarts the worker after an OOM kill or crash. Mine checks every 60 seconds and exits only when every artefact exists.
while true; do
LEFT=$(remaining) # count missing artefacts
[ "$LEFT" = "0" ] && { echo "🎉 all done"; exit 0; }
if ! pgrep -f "_par/serve.py" >/dev/null; then
echo "⚠️ worker down, $LEFT left — restarting" >> "$LOG"
nohup "$PYBIN" _par/serve.py >>_par/serve.log 2>&1 & disown
fi
sleep 60
done
Which creates a beautiful bug: you ask the worker to stop, it stops, and 60 seconds later the babysitter cheerfully starts it again. So stopping is a two-step, and the order matters:
Kill the supervisor first
pkill -f keepalive.sh — otherwise every graceful exit gets undone a minute later.
Then raise the stop flag
touch STOP. The worker sees it between units and exits after finishing the one in flight.
Then wait, don't assume
while pgrep -f serve.py; do sleep 5; done, then print "stopped cleanly — nothing lost". A stop command that returns before the thing has stopped is a lie.
Failure five: restarting for every fix you spot
The last one is behavioural, and it's the one I break most. You're watching the log, you notice unit 80's wording is off, and you fix it right now. Then again for unit 84. Then 91.
Every restart has a cost even when it's done safely. So fixes go on a pending list and get applied in one batch — either when I'm already making a change, or at the end of the run.
Measured cost of impatience
Ten careless restarts in a single session binned about 45 minutes of in-flight work. That is more than every failed optimisation experiment that week put together. The night doesn't die from big mistakes; it dies from a hundred small interruptions.
Noticing that kind of thing at all is a job for a cheap always-on watcher rather than a human — nobody is reading a 30,000-line log at 1am.
What the morning looks like
Queue armed, 221 units. Supervisor up. Purge-then-load, model resident in 41s.
Throughput down 46% — a browser GPU helper at 390% CPU. Logged, not fixed: I'm asleep and killing it needs a human.
Worker recycles for the 14th time. Compressor purged with nothing loaded, back to ~4-minute units.
145 of 221 landed — and because the queue was value-sorted, they're the 145 I'd have chosen.
That 1:07 line is the point of all of this. An unattended loop can't fix a resource rival, but it absolutely can notice one and leave the evidence. Most of the value of a hardened night run isn't the throughput — it's waking up to a log that explains itself.
The verdict
Overnight loops fail at the boring layer. Not the prompt, not the model — the cold start, the memory state, the restart discipline, and the supervisor arguing with the stop button. Fix those five and a night run becomes a machine you can trust rather than a lottery you check nervously at 6am.
The queue this rig was built for is a batch of photoreal stills of a real person for ad creative, which needed a character LoRA trained on their face — and taught me why faceswap loses to a LoRA once you look at it above thumbnail size.
Running something long and unattended? I'd genuinely like to hear what. Or have a poke around what these loops have shipped.