I Started Grading My AI Every Day. The Results Were Ugly

For months my assessment of AI coding tools was a feeling. Some days brilliant, some days maddening, and when someone asked how it was going I'd say "mixed" and change the subject. That's not an assessment, it's a mood.

So I started grading it. One row per day, two counts, a letter, and a note. Six days in, the vague frustration had resolved into three specific, fixable patterns — and my grades were much worse than my vibes had been telling me.

The format

Deliberately tiny, because a tracker that takes effort doesn't survive a week:

date,smoke_breaks,disappointments,grade,notes

Two metrics, and the definitions are the whole trick:

Named after what it feels like: an employee wandering off for a smoke mid-job. Two counts, because collapsing them into one "bad day" score loses the diagnosis — which is the only thing the tracker is for.

Two counts, because they have completely different causes and completely different fixes. Stalling is a pacing and orchestration problem. Disappointment is a verification problem. Collapsing them into one "bad day" score loses the diagnosis, which is the only thing the tracker is for.

Then a letter grade — a gut summary, not a formula. And notes, which is where the actual value lives.

Six real days

Bar length is disappointments; the value column is the count and the letter grade for the day.

Note day 4. One stall, seven disappointments, and a grade below both D days. That's not an inconsistency — it's the tracker doing its job. The morning shipped genuinely good work. The afternoon lost roughly an hour to a flaky file download, with repeated "it's working / no it isn't" flip-flops, before switching to the obvious resumable approach that should have been the first move on a known-throttled connection.

An hour lost to a file download, on a day that also shipped real deliverables. The letter grade captures the felt experience; the counts explain it; the note names the fix.

What six days of notes actually revealed

Three patterns, none of which I'd have named from memory.

1. Re-explaining things already in the window

   The single most aggravating category, and it showed up on four of six days. Asking for clarification on something stated explicitly twenty turns earlier. Not a knowledge failure — an attention failure, and one that reads as insulting when you're the one who already said it.

2. Forcing a constraint that had been explicitly ruled out

   One day was largely consumed by an agent snapping geographic regions to administrative boundaries after being told repeatedly that the regions were contiguous operational zones that deliberately cross those boundaries. Same wrong assumption, all day, despite correction — the correction never made it into the working model.

3. Batching the only visible output to the end

   The best thing the tracker found. A long job wrote its results at the end of each unit with the largest unit queued first, so I watched nothing happen with no way to tell working from hung — then restarted it several times, each restart binning in-flight work. Nothing was technically broken; the design was just invisible, which for a human watching is functionally identical to broken.

The fixes the tracker produced

This is the payoff. Vague frustration produces no fixes. Named patterns produce these:

• Make progress visible from the first unit of work. Never batch the only observable output to the end of a long job. Write results incrementally, and put the smallest unit first so something appears fast. This is now a standing rule and it's fixed more perceived-quality complaints than any model change.
• One reusable watcher, killed before another spawns. Born from a day of stacked background pollers competing with the real work.
• Answer the question actually asked, before moving on. From a day where I was asked a direct question, opened with "let me give you a straight answer", and then never answered it. Had to be asked twice.
• Reach for the boring resumable tool immediately on known-flaky infrastructure. The hour-long download detour, in one line.

Every one of those is now a rule in my setup. None of them would exist without the notes column.

That third pattern is the one that changed the most downstream. It's why every overnight loop I run now writes one artefact per unit and puts the smallest unit first.

The catch: don't let the AI grade itself

Two honest caveats, because this method has a sharp edge.

The numbers must be yours. I had the agent draft some of these entries and the self-assessed counts were consistently generous — of course they were. Models over-trust their own output; that's well documented, and it applies to self-scoring as much as to self-verification. The agent can draft the notes. You set the numbers. If it grades itself you're measuring its optimism, not its performance.

Grade the system, not the vendor. The temptation is to read a run of D grades as "this model is bad". Mostly it wasn't the model — it was my harness: no verification gate, no progress visibility, rules living in prompts that should have been hooks. The grades improved when I fixed the harness, not when I switched models. If you use the tracker as ammunition for a vendor argument, you'll skip the part that actually raises the score.

The verdict

You cannot improve a workflow you're only assessing by feel, and "AI is frustrating sometimes" is not a diagnosis. Two counts, one letter and one honest note per day converts that into a list of named failure modes — and a named failure mode is one you can build a guard against.

Separate stalling from wrongness, because they have different fixes. Write the notes while it's fresh. Set the numbers yourself. And expect the first week to be uglier than you thought.

My first two days were both a D.

Want an honest audit of where your AI workflow is actually losing you time? That's most of what I get hired for — start a conversation, or see recent projects.