The Overnight AI Loop That Ships While I Sleep

There are roughly eight hours every night where my machine sits at 2% CPU, doing nothing but rendering a screensaver. That's a third of my life, and until last year I was throwing it away.

Now the box works nights. Not "I left a chat window open and hoped" — an actual loop: a queue, a worker, a stop flag, a log, and a summary waiting for me at 7am. Some nights it's an agent writing and committing blog units. Some nights it's a diffusion model rendering 221 ad stills. Same skeleton either way.

Here's the shape of it, the code, and the three failure modes that will eat your night run if you don't design for them.

Rule one: the unit is the atom

The single biggest design decision is what one iteration produces. Get this wrong and you wake up to either a giant unreviewable blob or a pile of half-finished nothing.

A unit must be independently useful, independently verifiable, and independently revertible. One blog post. One rendered still. One migrated file. If unit 40 is garbage, I git revert unit 40 and the other 39 stand.

That's also why every unit gets its own commit. A night run that produces one commit at the end is a night run you have to accept or reject wholesale. Fifteen commits is fifteen independent decisions over coffee.

Rule two: resumable by construction

Your loop will be interrupted. The machine sleeps, the model API 500s, an OOM kills the worker, or you get up for a glass of water and mash Ctrl-C out of habit.

The fix isn't checkpointing state. It's making the output itself the state. Before doing a unit, ask the filesystem whether it's already done:

#!/bin/bashset -u; cd "$(dirname "$0")"LOG=overnight_$(date +%Y%m%d_%H%M).logwhile read -r unit; do  [ -f STOP ] && { echo "⏹  stop flag — exiting clean" | tee -a "$LOG"; break; }1  out="out/${unit}.json"  [ -f "$out" ] && { echo "⏭  $unit (exists)" | tee -a "$LOG"; continue; }2  start=$(date +%s)  if ! ./do_unit.sh "$unit" > "$out.tmp" 2>>"$LOG"; then    echo "❌ $unit failed" | tee -a "$LOG"; rm -f "$out.tmp"; continue  fi  python3 -c "import json; json.load(open('$out.tmp'))" || { rm -f "$out.tmp"; continue; }3  mv "$out.tmp" "$out"4  git add "$out" && git commit -q -m "content($unit): overnight unit"5  echo "✅ $unit ($(($(date +%s)-start))s)" | tee -a "$LOG"done < queue.txt

Rule three: a stop flag, never a kill

This one cost me 45 minutes of GPU in a single session before I learned it. My image renderer writes its PNG at the very end of an 8-minute generation. Kill it at 7:59 and you get nothing — not a partial image, nothing.

So the worker checks for a STOP file between units and exits clean. Stopping is a request, not a signal:

#!/bin/bash
# stop.sh — ask nicely. NEVER pkill.
pkill -f keepalive.sh 2>/dev/null   # else the babysitter just relaunches it
touch STOP
echo "⏹  requested — will finish the current unit (up to ~10 min) then exit."
while pgrep -f overnight.sh >/dev/null; do sleep 5; done
echo "✅ stopped cleanly — nothing lost."

Ten careless pkills in one night binned more work than every failed experiment that week combined. The lesson generalises: in any long-running loop, the only safe stop is a cooperative one.

The log format is the interface

You are not going to read 30,000 lines of stdout in the morning. You're going to read the last screen of it and want to know: what landed, what broke, how long each thing took.

22:48  ▶  queue: 15 units · stop flag clear · budget 900k tok22:59  ✅ unit 01 halls-and-rooms          (11m 04s) → commit a91f3c223:14  ✅ unit 02 hooks-beat-prompts       (14m 51s) → commit 4d0b11823:22  ❌ unit 03 token-economy            (draft failed schema check, requeued)23:41  ✅ unit 03 token-economy            (18m 12s) → commit 77ae90101:07  ⚠️  throughput down 46% — GPU rival at 390% (chrome_helper)06:52  ▶  queue drained: 15/15 · 0 open failures06:52  📝 wrote overnight_SUMMARY.md

Three columns: timestamp, status glyph, unit plus duration. Grep-able, skim-able, and the durations tell you where the night actually went. Mine caught a browser process stealing 390% CPU and halving render throughput — invisible in any per-unit view, obvious in a column of times.

The last line matters most. Every run writes a SUMMARY.md: what landed with paths, what failed and why, what's queued for tomorrow. That file is the entire morning handover.

Order the queue by what you'd rather have

Here's the bit nobody mentions. Your loop will probably not finish. On one 221-unit render run I could see by 1am that contention meant only ~145 would land by morning.

That's fine — if the queue is sorted so the 145 that land are the ones I actually wanted. So the runner orders by stated value, not by category size or alphabetically:

Sorting a queue by expected value costs ten lines of Python and is the highest-leverage change in this entire post.

The catch: unsupervised means undifferentiated

Now the honest part. An agent left alone for eight hours produces work that is individually fine and collectively repetitive. My 15-post night run shipped with the same closing CTA byte-for-byte in every post, one catchphrase in 15 out of 15, and — the real miss — zero internal links between siblings.

None of that is visible from inside a single unit. Every post passed its own review. The defect only exists at the level of the set.

Pair that with a verify-before-done checklist inside each unit and deterministic hooks for the rules you refuse to negotiate, and the output stops needing a full re-read.

The verdict

An overnight loop is not an AI trick. It's the oldest idea in computing — a work queue with a worker — wearing a model as its execution engine. The parts that make it survive the night are boring: idempotency, atomic writes, a cooperative stop, a legible log, and a queue sorted by what you'd actually miss.

Build those five and the machine earns its keep in the dark. I've since pointed the same skeleton at image generation, where the units are 9-minute renders instead of blog posts — that's part two, on the hardening rules, and the reason I ended up training a LoRA on a real person's face to feed it.

Got eight hours a night going spare? Tell me what you'd point them at — or see what the loops have already built over on the front page.