562 specs later: what changed when the agent's work had to pass a gate to count as done
Since August, every feature an agent builds here goes through a Specsfy spec. I measured 562 specs, my requests, the token cost and delivery on GitHub. I decide at the start, the agent chains phases on its own, and about half the cost goes to defining and changing the spec.
On October 29, an audit found tests without a valid RED in five Fuad Next specs. Most had been written after the code, and the agent had marked those tasks as done. The method’s rule is the opposite: the test comes first, fails for the right reason (the RED), and only then does the code go in.
I answered four numbered questions and granted an exception only for the tasks the audit pointed to. The tests stayed as regression tests, without faking a RED that never happened. The audit was only possible because every task, every acceptance criterion and every test are tied together in a single file. That is what Specsfy has done in my workflow since August, and I measured what changed with it.
Where we came from
Until August, the workflow was Superpowers: process skills for coding agents, with brainstorming, a written plan, execution by subagents and verification before finishing. Around it, we built our own practices as problems showed up: a handoff at the end of each session so the next one could pick up where it stopped, decisions in numbered files, acceptance testing in the real app, driven by the agent itself, and messages exchanged between the Mac and the Windows laptop.
It worked, but the plan, the decisions, the tests and the proof that something was done lived in different places. In the end, the one saying a task was done was the agent.
One path, with gates
Specsfy is a specification-driven development framework created by Luiz Eduardo Fonseca, from Promovaweb, who cites the GitHub Spec Kit as one of its inspirations. It isn’t ours, but we have contributed to it. Since August 20, every feature follows this path:
- The inbox stores an idea right away, with no questions, in its original wording.
- The backlog turns the capture into a decision through numbered questions, and authorizes no implementation.
- The spec is a single file with the problem, requirements, acceptance criteria, tasks, tests and evidence. A plan or task list in a separate file is forbidden.
- The spec goes through three gates: define (Definition Gate), prove it with tests failing before the code (Plan Gate) and deliver with evidence (Delivery Gate). Changing the behavior later reopens all three.
In a new product, the flow starts one step earlier, with an interview that defines the MVP and splits it into milestones: states of the product that can be demonstrated, each with an exit condition. The agent can implement a milestone’s specs one after another, but the milestone only closes when I confirm the condition was demonstrated.
Capturing in the middle of the work
The use I like most is the inbox in the middle of something else. I’m in a session implementing a spec, a problem shows up that doesn’t belong to it, and I ask the agent to analyze the case and read a source I point to. At the end, it records an inbox item. The capture comes out with the context of that moment, I don’t stop what I was doing, and in the next session the item already starts with the case explained. About one in four inbox calls came in the middle of another step of the flow, in the same session.
How much it is used
| On November 7, across the nine repositories | Total |
|---|---|
| Specs | 562, of which 463 completed |
| Acceptance criteria | 11,417 |
| Tasks | 18,387 |
| Markers linking a test to the criterion it covers | 20,096 |
The specs aren’t small. In Fuad Next, the median spec has 16 acceptance criteria and Effort 6. Effort is the difficulty score, from 1 to 10, recorded in each spec, and has nothing to do with the model’s reasoning effort, which shows up further down.
Where I come in
My part is concentrated at the start. Questions arrive numbered, one at a time, with the options ready, and I often answer with just a number. The question below is an example: Specsfy requires Pest in PHP projects, this project’s worker is Node, and the agent brought three ways to fit one into the other.
To see whether this really happens, I counted my requests by phase of the flow and how many of them were just an option number. The first column is that share; the second, how many requests fell in each phase.
| Arm | Just the option number (%) | Requests |
|---|---|---|
| Capture and refinement (inbox, backlog, interviews) | 49.6 | 1516 |
| Setup | 28.0 | 322 |
| From spec creation to implementation | 22.2 | 1246 |
| Other skills (documentation, specialists) | 19.5 | 512 |
| Late change | 17.6 | 255 |
| Spec work without calling a skill | 17.2 | 937 |
| Before the session's first skill | 6.3 | 824 |
| Sessions unrelated to any spec | 3.4 | 591 |
Source: Local Claude Code transcripts: 6,203 human requests in 680 sessionscommit 17d0066MethodologyDownload JSONDownload CSV
In capture and refinement, half of my requests were just a number. In sessions unrelated to any spec, it was 3%. The difference comes from the format: when the agent arrives with the options already thought through, deciding becomes choosing. From spec creation on, the share drops to about 20%, because that is where I confirm, answer and correct course in sentences. And part of the work doesn’t even go through me: of the 158 sessions that worked on a spec without calling a skill, 123 were opened by another session, the one that coordinates the work streams, and 79 have no request from me at all.
After a request, the agent keeps going on its own. In 117 requests, it went through three or more skills of the flow without a word from me in between, and the median was 45 minutes of work. Diário, the pipeline that produces this site’s drafts, closed 15 specs in a little over a day, from the first inbox item to the last acceptance.
What changed in delivery
Chaining phases is only worth it if delivery gets better, and I wanted to know whether it did. In Fuad Next, I compared, using git and GitHub, the 83 days before the switch with the 80 days after it, with four measures:
- PRs merged per day, to see throughput;
- PRs with a finding from the Codex review, to see what started being caught before the merge;
- Typecheck failures in CI, the only workflow that exists in both periods;
- rework, the PRs that received, within 7 days, a fix to a file they themselves created, the closest signal to a defect that slipped through.
Along with Specsfy came a rule of ours: each spec becomes a PR in its own worktree, with a Codex review before the merge. The table measures that package, not the method alone.
| Arm | Merged PRs per day | PRs with a Codex review finding (%) | Typecheck CI failure (%) | PRs with a fix to a file they created within 7 days (%) |
|---|---|---|---|---|
| After (Specsfy) | 8.2 | 55.0 | 21.8 | 40.2 |
| Before (Superpowers) | 2.1 | 0.0 | 60.0 | 36.5 |
Source: Fuad Next git and GitHub: 175 PRs merged before and 655 aftercommit 17d0066MethodologyDownload JSONDownload CSV
Before, almost everything went straight to main. After, almost everything went through a PR, and merged PRs went from 2.1 to 8.2 per day, with the same median size, about 840 lines. In 55% of them, the Codex review found something before the merge. Typecheck failures fell from 60% to 22% of runs, but the before period has only 45 runs.
Rework didn’t change: 37% of PRs before and 40% after. The review catches things before the merge, but I didn’t see fewer fixes after it. My reading is that more PRs in quick succession touch the same core files, and the measure can’t tell a defect fix from an adjustment made by the next spec. Per request, the difference is also small: 19.3 requests from me per Superpowers plan before, and 16.8 per completed spec after, with delivery documents about 50% larger. The gain I can claim is in throughput and review, not in fewer defects.
How far the agent goes on its own
A request where the agent works more than half an hour without me is a request where I can do something else. So I measured, month by month, the share of my requests that went past 30 minutes of agent work before the next request. In Codex, I only counted the sessions I opened and typed into: reviews one agent asked of the other and automations were left out. Hover over or focus a point to see the month, the value and how many requests went into it.
Requests over 30 min (%)
Month, 2026
* Aug: Codex mixes days before and after Aug 20; Claude Code from Aug 21
* Nov 1 to 7
See the numbers in a table
| Month | Codex | Claude Code |
|---|---|---|
| May 2026 | 9.3 | — |
| Jun 2026 | 2.1 | — |
| Jul 2026 | 13.1 | — |
| Aug 2026* | 7.3 | 11.1 |
| Sep 2026 | 9.9 | 7.9 |
| Oct 2026 | 12.3 | 11.3 |
| Nov 2026* | — | 18.2 |
Source: Local transcripts: Claude Code (all sessions) and Codex (interactive typed sessions)commit 17d0066MethodologyDownload JSONDownload CSV
The curve shows no jump when Specsfy arrived. Codex was already at 13% in July, before it, and from September to October both curves stayed between 8% and 12%. The jump came in November, in Claude Code: 18% of requests went past half an hour, and one in ten went past 66 minutes. That was when I started coordinating several work streams at once, each in its own session. I read this as the evolution of the whole workflow: model, agent, method and the way I work changed together, every month, and I have no way to separate the share of each.
Claude Code versus Codex
Between August and October, I used both agents with Specsfy. To compare without mixing periods, I compared them within the same month, on four signals of how the agent worked:
- loop, the same command repeated three times within ten calls, the most direct signal of a stuck agent;
- call with an error, a command that ended in an error, or a rejected edit;
- failed test, which includes the RED, the test TDD says to write failing before the code;
- minutes per request, how long it worked on its own, at the median.
| Arm | Loops per 100 calls | Calls with an error, per 100 | Test runs that failed (%) | Minutes per request (median) |
|---|---|---|---|---|
| Aug · Claude Code (301 requests) | 0.0 | 1.8 | 34.2 | 4.1 |
| Sep · Claude Code (442 requests) | 0.0 | 2.1 | 35.5 | 0.8 |
| Oct · Claude Code (3,074 requests) | 0.1 | 2.0 | 30.9 | 1.4 |
| Aug · Codex (382 requests) | 0.5 | 8.7 | 40.9 | 2.0 |
| Oct · Codex (121 requests) | 0.6 | 7.1 | 22.3 | 3.1 |
| Sep · Codex (1,024 requests) | 0.9 | 10.2 | 30.9 | 2.6 |
Source: Local transcripts: Claude Code sessions that called a Specsfy skill and interactive Codex sessions that read a Specsfy skillcommit 17d0066MethodologyDownload JSONDownload CSV
On the two signals of an agent spinning its wheels, Claude Code came out lower in every month: at most 0.1 loops per 100 calls, versus 0.5 to 0.9 in Codex, and 1.8 to 2.1 calls with an error, versus 7.1 to 10.2. Part of the error gap is about the unit: in Codex, one call can run several commands and counts as an error if any of them fails. On failed tests, there is no winner: the order flipped from one month to the next, and the number carries the RED the method itself asks for.
Per spec, Claude Code needed fewer requests from me. In Effort 7-8 specs, the median was 9 with it and 18 with Codex, which had only 16 specs in that band. In exchange, it used more tokens, 1.3 to 1.9 times as many depending on the cut, and the two agents count tokens differently. The models were different on both sides, and I picked the agent according to the task, so this compares each one’s workflow in my use, not the agents in a controlled test.
Models and machines in Codex
Within Codex, I switched models and machines over the months, from the Windows laptop to the Mac. I wanted to know whether any of those switches explains the difference. The chart crosses two measures: on the vertical axis, the share of test runs that failed; on the horizontal, the share of requests I interrupted midway, the most direct signal that I had to step in. The lower and further left, the less friction. Each point is a model on a machine in a period: hover over or focus a point to see the numbers.
Interrupted requests (%)
See the numbers in a table
| Arm | Test runs that failed (%) | Interrupted requests (%) | Frontier |
|---|---|---|---|
| gpt-5.6-sol, Windows, before Specsfy (Jul–Aug) | 41.9 | 7.9 | off the frontier |
| gpt-5.6-sol, Mac, after Specsfy (Aug–Sep) | 40.4 | 2.0 | off the frontier |
| gpt-6-astra, Mac, after Specsfy (Sep) | 27.0 | 1.1 | on the frontier |
| gpt-5.5, Windows, before Specsfy (Jun–Jul) | 38.3 | 6.0 | off the frontier |
| gpt-6.1-sol, Mac, after Specsfy (Sep–Nov) | 23.1 | 4.2 | on the frontier |
Source: Local Codex transcripts: interactive sessions with a human request, cells with 100 requests or morecommit 17d0066MethodologyDownload JSONDownload CSV
The clearest case is gpt-5.6-sol, the only model that shows up on both sides of the switch. On Windows, before Specsfy, I interrupted 7.9% of its requests; on the Mac, after, 2.0%. The share of failed tests stayed near 41% in both. Same model, same level of test failures, a quarter of the interruptions. My reading is that, with Specsfy, a good part of the requests became answers to numbered questions or requests for a whole phase, which there is no reason to cut midway. But the machine and the period changed together, and Windows ran almost everything in xhigh, so the data can’t separate one thing from the other.
The two lower points, joined by the line, are the models that came later, from September on: gpt-6.1-sol and gpt-6-astra, with 23% and 27% of tests failing. The line joins the points that no other point beats on both measures at once, but gpt-6.1-sol has only 144 requests, and each point covers its own period. The chart shows what happened in my use. Choosing a model would take the same tasks run on each one, and that is for another post. In Claude Code there isn’t even anything to compare: Opus 5.5 was the predominant model in 394 of the 415 sessions with a spec.
Reasoning effort
Both agents let you choose how much the model thinks before answering, from low to xhigh, and record that level on every turn. This is not the Effort from 1 to 10, which measures the spec’s difficulty. The question here is whether thinking more is worth the cost. The only clean comparison I have is one Codex cell where only the level changed: gpt-6-astra, on the Mac, in September.
In the chart, the horizontal axis is the cost, in tokens per request, and the vertical axis is the share of test runs that failed. If thinking more paid off, the points on the right would sit lower.
Tokens per request, thousands (median)
See the numbers in a table
| Arm | Test runs that failed (%) | Tokens per request, thousands (median) | Frontier |
|---|---|---|---|
| medium (210 requests) | 16.9 | 438 | on the frontier |
| high (214 requests) | 36.5 | 935 | off the frontier |
| xhigh (251 requests) | 24.8 | 1232 | off the frontier |
Source: Local Codex transcripts: human requests to gpt-6-astra on the Mac in September, by the effort set for the turncommit 17d0066MethodologyDownload JSONDownload CSV
In that cell, high used twice as many tokens per request as medium, and xhigh almost three times as many. Time per request went from 1 to 4 minutes, and xhigh repeated more commands, 2.4 loops per 100 calls. Failed tests didn’t drop with more reasoning: 17% in medium, 37% in high and 25% in xhigh, counting the RED, and the kind of request also changes from one level to another. Thinking more cost more and didn’t show up as fewer errors. In Claude Code, almost everything ran in high, 205 of the 218 completed specs, so there the effect of reasoning can’t be separated.
What it costs
I added up the tokens of every Claude Code message between August 21 and November 7 and multiplied them by the API price list. Usage is on a subscription, so the amount isn’t what was paid: it serves to compare the parts of the work. Sessions that called a Specsfy skill added up to about US$ 25 thousand equivalent, and the ones that worked on a spec without calling a skill, another US$ 6 thousand. Together, they took 93% of the period’s cost. The table splits the cost of sessions with a skill by the phase in which each message happened.
| Arm | Equivalent cost (US$ thousand) | Share of cost (%) |
|---|---|---|
| Before the session's first skill | 2.6 | 10.4 |
| Documentation, specialists and other skills | 3.2 | 12.8 |
| Late change (reopening a defined spec) | 3.9 | 15.5 |
| Tests and implementation | 6.0 | 24.0 |
| Defining the spec: setup, inbox, backlog, creation, validation and tasks | 9.3 | 37.3 |
Source: Local Claude Code transcripts, 290 session files with a Specsfy skill, public API pricescommit 17d0066MethodologyDownload JSONDownload CSV
Defining the spec and changing it later took about 53% of the cost of sessions with a skill, and tests and implementation took 24%. Defining is expensive because, to write the spec, the agent reads a lot of code, and that reading counts as defining. In total, late changes cost US$ 3.9 thousand, close to the US$ 4.7 thousand for implementing, and that figure includes implementing the change itself. Per spec completed in Claude Code, the median was US$ 51 equivalent. The method spends more on the conversation than on the code, and that is why reopening a spec is costly.
When it pays off
A spec’s cost grows with its difficulty, but the price of each acceptance criterion changes little.
| Arm | Equivalent cost per spec (US$) | Specs with a late change (%) | Minutes of agent work | Cost per acceptance criterion (US$) |
|---|---|---|---|---|
| Effort 3-4 (58 specs) | 24 | 9 | 22 | 2.66 |
| Effort 5-6 (76 specs) | 51 | 20 | 114 | 2.73 |
| Effort 7-8 (67 specs) | 77 | 46 | 201 | 3.17 |
Source: Local Claude Code transcripts and current spec.md files: 201 completed specs with Effort 3-8commit 17d0066MethodologyDownload JSONDownload CSV
From Effort 3-4 to 7-8, the median spec goes from US$ 24 to US$ 77 equivalent, and the agent’s work from 22 to 201 minutes. Within those bands, each acceptance criterion costs between US$ 2.66 and US$ 3.17. What grows fastest is late change: 9% of Effort 3-4 specs were reopened, compared with 46% of Effort 7-8 ones.
One question I had was whether a spec with a screen, an interface for people, costs more just because it is bigger. To separate the two, I compared specs with and without a screen within the same Effort band. In the chart, each point is a band with or without a screen: the horizontal axis is the median cost per spec, and the vertical one, the share of specs reopened. The lower left corner is the cheapest and the one that comes back least.
Equivalent cost per spec, median (US$)
See the numbers in a table
| Arm | Specs with a late change (%) | Equivalent cost per spec, median (US$) | Frontier |
|---|---|---|---|
| Effort 3-4, with a screen (11 specs) | 0 | 22 | on the frontier |
| Effort 3-4, no screen (44 specs) | 11 | 23 | off the frontier |
| Effort 5-6, with a screen (37 specs) | 30 | 58 | off the frontier |
| Effort 5-6, no screen (38 specs) | 16 | 36 | off the frontier |
| Effort 7-8, with a screen (51 specs) | 55 | 68 | off the frontier |
| Effort 7-8, no screen (24 specs) | 50 | 66 | off the frontier |
Source: Local Claude Code transcripts and current spec.md files: completed specs with Effort 3-8, median equivalent cost per speccommit 17d0066MethodologyDownload JSONDownload CSV
Screens matter in the middle of the scale. At Effort 5-6, specs with a screen cost US$ 58 at the median, versus US$ 36 without one, and were reopened almost twice as often, 30% versus 16%. In small and large specs, the points with and without a screen nearly touch: the cost is practically the same, and the band is what matters. My reading is that, in a medium spec, the screen is still being discovered while the agent implements, and each discovery becomes a late change; in large specs, late change is already high either way.
What I take from this:
- Split before starting. At Effort 7-8, almost half the specs came back after being defined.
- Talk more about the screen in medium specs. That is where the interface adds the most cost and the most reopening.
- Keep small fixes out. Across the nine repositories, 187 sessions worked on no spec at all, with a median of US$ 2.94 each. Effort 1-2 specs are too few, only 8, to say what the floor costs.
- Keep the gate. In the October audit, the agent had skipped the RED and said it was done. Because task, criterion and test are in the same file, the review found it.
Done, but never filed
When I checked the specs that went only through Codex, many looked open, and I suspected they were done and simply hadn’t been closed. A spec only leaves the open list when it moves to specs/completed/, and that step depends on someone asking for it. So I looked at every spec outside that folder in the nine repositories and sorted them by their real situation.
| Arm | Specs | Idle for more than 30 days |
|---|---|---|
| Marked complete, but not moved | 37 | 8 |
| One or two tasks left | 36 | 7 |
| Real work still open | 20 | 5 |
| All tasks done, delivery gate open | 10 | 3 |
Source: Remote main branch of the nine repositories: 103 specs outside specs/completed/commit 17d0066MethodologyDownload JSONDownload CSV
At the end of November 7, there were 103. Of these, 37 were already marked complete and 10 had every task done with the delivery gate still open: 47 specs done, or nearly, sitting in the wrong place. In 36, one or two tasks were left, and only 20 had real work ahead. The 18 idle for more than a month, not counting the ones with open work, are all in Fuad Next, from late August to early October. The gate says what is done, but moving the spec is a separate step, and it gets forgotten once attention has moved on to the next one. Closing has to become routine, the same way the PR did.
What remained our job
- Handing over between sessions. Specsfy doesn’t handle continuity from one session to the next, or from one machine to another, so we kept our handoff.
- Delivery discipline. A PR per spec in its own worktree, the agent’s own acceptance in the real app and the Codex review before the merge are our rules.
- Patches to the tool. Traceability got a wrapper, because the same test marker often applied to more than one spec. The framework’s contract got a single patch, protected by a test: allowing inert types and signatures before the RED, without loosening TDD.
Methodology
The data was measured on November 7, 2026, without any new model runs. The requests come from a single person.
- Specs, criteria, tasks and markers come from the
spec.mdfiles and the versioned tests in the nine repositories, counted by script. Criteria count unique IDs per spec. Inbox, backlog and milestones are the files inspecs/inbox/,specs/backlog/andspecs/milestones/. The opening audit comes from the decisions recorded in Fuad Next that day. - In Claude Code, the local transcripts from August 21 to November 7 are included: 680 session files, in three groups. There are 290 that called a Specsfy skill, 158 that edited spec files without calling a skill and 232 unrelated to any spec, 45 of them outside the nine repositories. Human requests number 6,203, excluding duplicates, notifications, automatic messages, compaction summaries and the briefing a coordinating session sends to another. The phase of a request or message is the last skill called before it, and a session counts toward the spec it edited most.
- The 117 chained requests, the requests per delivery and the cost split by phase come from a first count, made before removing the duplicates of resumed sessions; that is why the cost by phase adds up to US$ 25.0 thousand, versus US$ 25.2 thousand in the new count. Requests from before August come from the typed-prompt history, which survived the transcript cleanup.
- In Codex, the 366 interactive sessions with a human request are included, from April to November 7, on the Mac, the Windows laptop and WSL. Subagent, voice and automation sessions are left out, as are those called by another agent. A loop is the same command repeated three times within ten calls, not counting waits. Failed tests include the RED expected by TDD, and until August 19, 72% of Codex test runs didn’t record an exit code, so the failure comes from the text.
- The cost uses the API price list as of October 6, 2026, with cache writes at 1.25 times the input price (5 minutes) and 2 times (1 hour). Reading code to write the spec counts as defining. The cost per spec ignores work done in Codex on those specs, and Effort and interface are those in the current
spec.md. - Claude Code versus Codex compares, from August 21 to November 7, the Claude Code sessions that called a Specsfy skill with the interactive Codex sessions that read a Specsfy skill, main thread only. Loops use the same rule in both. Reasoning effort comes from the
effortfield on each turn in both agents, and each request takes the most frequent level in its window. - The specs outside
completed/were read from the remote main branch of each repository at the end of November 7, when there were already 569 specs. “Marked complete” means Status Complete or an approved Delivery Gate, and “idle” means a spec whose folder hasn’t received a commit in more than 30 days. - In the Codex model chart, each point is a combination of the session’s predominant model, machine and period, before or after August 20, with 100 requests or more; 13 smaller combinations, with 122 requests, were left out. An interrupted request is one I cut off before the agent finished. The two agents count tokens differently, so tokens only compare orders of magnitude.
- Autonomy is the time between one of my requests and the agent’s last action before the next request. Claude Code transcripts from before August no longer exist, which is why there is no before-and-after cost comparison.
- The Fuad Next before and after uses git and GitHub from May 29 to August 19 and from August 20 to November 7. The GitHub repository has existed since June 1, and CI barely existed before. A Codex review finding is a Codex review that left a comment before the merge. Rework is a fix commit that reaches
mainwithin 7 days and touches a production file the PR created. - Requests per delivery add up Claude Code and Codex in Fuad Next, without slash commands. Specs still open are left out of the denominator, which favors the before period.
Limits that matter: it is one person, in projects from the same company, over a period when model, agent, machine and method changed together. The numbers describe the workflow. They don’t prove that Specsfy, on its own, makes anything better or worse.
Agent draft from the listed sources; reviewed and approved by Lucas Fuad