For months my assessment of AI coding tools was a feeling. Some days brilliant, some days maddening, and when someone asked how it was going I'd say "mixed" and change the subject. That's not an assessment, it's a mood.
So I started grading it. One row per day, two counts, a letter, and a note. Six days in, the vague frustration had resolved into three specific, fixable patterns — and my grades were much worse than my vibes had been telling me.
The format
Deliberately tiny, because a tracker that takes effort doesn't survive a week:
date,smoke_breaks,disappointments,grade,notesTwo metrics, and the definitions are the whole trick:
smoke_breaks — times the agent stalled, deferred, or idled when it had everything it needed to proceed. Waiting on a timer instead of acting. "I'll do that as a focused pass later." Explaining why it wasn't doing the task instead of doing it. Named after what it feels like: an employee wandering off for a smoke mid-job.
disappointments — times the output was wrong, unverified, or off-target. Claimed done without checking. Built the wrong thing. Broke something adjacent. Made me find the break.
Two counts, because they have completely different causes and completely different fixes. Stalling is a pacing and orchestration problem. Disappointment is a verification problem. Collapsing them into one "bad day" score loses the diagnosis, which is the only thing the tracker is for.
Then a letter grade — a gut summary, not a formula. And notes, which is where the actual value lives.
Six real days
day 1 10 breaks 20+ disappointments D
day 2 6 breaks 15+ disappointments D
day 3 1 break 3 disappointments B-
day 4 1 break 7 disappointments E-
day 5 0 breaks 1 disappointment B
day 6 0 breaks 3 disappointments D+Note day 4. One stall, seven disappointments, and a grade below both D days. That's not an inconsistency — it's the tracker doing its job. The morning shipped genuinely good work. The afternoon lost roughly an hour to a flaky file download, with repeated "it's working / no it isn't" flip-flops, before switching to the obvious resumable approach that should have been the first move on a known-throttled connection.
An hour lost to a file download, on a day that also shipped real deliverables. The letter grade captures the felt experience; the counts explain it; the note names the fix.
What six days of notes actually revealed
Three patterns, none of which I'd have named from memory.
1. Re-explaining things already in the window. The single most aggravating category, and it showed up on four of six days. Asking for clarification on something stated explicitly twenty turns earlier. Not a knowledge failure — an attention failure, and one that reads as insulting when you're the one who already said it.
2. Forcing a constraint that had been explicitly ruled out. One day was largely consumed by an agent snapping geographic regions to administrative boundaries after being told repeatedly that the regions were contiguous operational zones that deliberately cross those boundaries. Same wrong assumption, all day, despite correction. The correction never made it into the working model.
3. Batching the only visible output to the end. This one's subtle and it's the best thing the tracker found. A long job was designed to write its results at the end of each unit, and the largest unit was queued first. So I sat watching nothing happen for a long stretch with no way to tell whether it was working or hung. Then it got restarted several times for fixes, each restart binning in-flight work.
Nothing was technically broken. The design was just invisible, which for a human watching it is functionally identical to broken.
The fixes the tracker produced
This is the payoff. Vague frustration produces no fixes. Named patterns produce these:
- Make progress visible from the first unit of work. Never batch the only observable output to the end of a long job. Write results incrementally, and put the smallest unit first so something appears fast. This is now a standing rule and it's fixed more perceived-quality complaints than any model change.
- One reusable watcher, killed before another spawns. Born from a day of stacked background pollers competing with the real work.
- Answer the question actually asked, before moving on. From a day where I was asked a direct question, opened with "let me give you a straight answer", and then never answered it. Had to be asked twice.
- Reach for the boring resumable tool immediately on known-flaky infrastructure. The hour-long download detour, in one line.
Every one of those is now a rule in my setup. None of them would exist without the notes column.
The catch: don't let the AI grade itself
Two honest caveats, because this method has a sharp edge.
The numbers must be yours. I had the agent draft some of these entries and the self-assessed counts were consistently generous — of course they were. Models over-trust their own output; that's well documented, and it applies to self-scoring as much as to self-verification. The agent can draft the notes. You set the numbers. If it grades itself you're measuring its optimism, not its performance.
Grade the system, not the vendor. The temptation is to read a run of D grades as "this model is bad". Mostly it wasn't the model — it was my harness: no verification gate, no progress visibility, rules living in prompts that should have been hooks. The grades improved when I fixed the harness, not when I switched models. If you use the tracker as ammunition for a vendor argument, you'll skip the part that actually raises the score.
The verdict
You cannot improve a workflow you're only assessing by feel, and "AI is frustrating sometimes" is not a diagnosis. Two counts, one letter and one honest note per day converts that into a list of named failure modes — and a named failure mode is one you can build a guard against.
Separate stalling from wrongness, because they have different fixes. Write the notes while it's fresh. Set the numbers yourself. And expect the first week to be uglier than you thought.
My first two days were both a D.
Want an honest audit of where your AI workflow is actually losing you time? That's most of what I get hired for — start a conversation, or see recent projects.