What a real product refactor revealed about AI coding agents

ReviewWithAI engineering case study · 15–16 September 2026** By Aashish Bhandari, with AI-assisted analysis by Goku (Codex, Astra) and independent review by Naruto (Claude, Fable 5.1). Executive summary I worked with two AI coding agents to refactor ReviewWithAI, an alpha application for reviewing Markdown documents and handing changes to external agents. Goku acted as principal architect and implementing developer. Naruto independently reviewed the design and code. I set priorities, resolved material decisions and authorized progress through the review checkpoints. The work implemented eleven low-level designs across three milestones. It covered server and browser structure, authorization, persistence, testing, operational diagnostics and release preparation. The candidate passed independent checks, including tests, browser workflows and reproduction of the packaged application. Those results establish a reviewed engineering outcome; they do not establish production readiness or prove that the process was efficient. After implementation, we examined the runtime evidence to understand how the work had been orchestrated. The analysis used collected telemetry first, followed by a bounded deterministic inspection of fields missing from the collector. Its scope was Goku's implementation conversation, delegated workers and automatic approval reviewers. Naruto's separate review sessions and my time were outside that measurement. The clearest finding concerned waiting. When the parent agent had no independent work, it repeatedly resumed after short waits for its workers. Each resumption could carry conversation history back into a model invocation. Activations that issued waits accounted for more than a quarter of the parent's recorded tokens. Some waits returned completed work, so that share cannot simply be called waste or promised as recoverable savings. A second finding concerned worker reuse. Several delegated conversations accumulated assignments across multiple stages. Their recorded consumption became concentrated, but the evidence does not tell us whether starting fresh workers would have produced the same quality with less effort. Continuity may help correctness while increasing context costs. That tradeoff needs a controlled comparison. Compaction reduced the context entering the next invocation, but later context growth and recurring waits continued. Choosing cheaper worker models also lowered a calculation based on published rates, without demonstrating lower total work. Most recorded input was cached. Processed tokens therefore need to be distinguished from unique content, uncached computation and actual charges. The investigation exposed limitations in our own measurement process. The collector omitted compaction activity, and its interruption counter did not describe every terminal failure. Automatic approval reviewers also consumed resources separately from implementation workers. Even evaluating the session had a material cost: the analysis conversation included other work, preventing a clean estimate of analysis overhead. We report that uncertainty rather than presenting precision. The practical lesson is to evaluate the mechanisms around coding agents alongside their output. Instructions to avoid polling did not reliably prevent the observed behavior. Before testing optimizations, we need better deterministic counters, evaluation budgets and comparisons that include accepted quality, recovery and human effort. This report is a baseline from one project, with delivery and unresolved efficiency questions. The next step is to test changes against that baseline and publish the results, including any changes that fail to help. The detailed evidence below lets other developers inspect our reasoning and judge where these observations might apply to their workflows. Boring stuff ahead. Proceed with caution. The rest contains the methods, tables and caveats for readers who want to check the evidence. The product and the development arrangement ReviewWithAI is a Markdown review application: a user selects text, attaches comments, hands work to an external coding agent, checks changed anchors, records repairs and accepts a specific source revision. The starting point was an alpha-grade, mid-sized product, as described by its developer. The work included server behavior, browser interactions, persistence, authorization, testing, documentation and release tooling. It was an existing-product refactor and remediation effort, rather than initial application generation. The eleven low-level designs (LLDs) addressed twelve review findings labelled Q1–Q12; Q3 and Q4 shared one design. Design preparation and agreement preceded the measured Phase B implementation task. The roles were: Participant Responsibility Included in the measured task? One human developer, Aashish (Max) Product direction, scope, material decisions, milestone authorization and final candidate acceptance Human time is not measured Goku, Codex parent on GPT-6 Astra Principal architect, implementing developer, delegation, integration and verification Yes Sixteen Goku worker threads on Sol, Luna and Terra Bounded research, implementation, integration, tests and release work; some threads were later reused Yes Naruto, Claude coding agent; final review identifies Fable 5.1 Independent principal architect for design and code review, including reproduction of delivery evidence No; separate review sessions Automatic Codex approval reviewers Review of eligible approval requests Yes; separate from implementation workers “Two agents” describes the two principal collaborators. It does not mean two model processes: Goku's implementation task alone produced sixteen worker threads and 38 approval-review components. flowchart TD H["Human developer: scope, decisions and acceptance"] --> G["Goku: architecture and implementation"] H --> N["Naruto: independent design and code review"] subgraph measured["Measured Codex task"] G --> W["16 delegated worker threads"] G -. "approval requests across the task" .-> A["38 approval-review components"] end G --> E["Candidate code, package and verification evidence"] W --> E E --> N N --> R["Review findings and milestone verdicts"] R --> H R --> G Eleven designs, three review milestones The milestones were review checkpoints rather than individual commits. The LLD index and checkpoint records establish the sequence: Milestone Implemented scope Review boundary 1. Housekeeping and engineering controls Engineering-standard amendments, LLD navigation and citations, documentation checker and checker tests Human and Naruto approval before merge and runtime implementation 2. First implementation group Q5 handoff-route tests; Q10 transaction guard; combined Q3/Q4 typed operations, browser structure, retries and credential rotation Independent midpoint review; approved work integrated at c9215c9 3. Remaining implementation and release candidate Q2 diagnostics; Q6 portable tooling; Q8 strict agent inputs; Q9 health version; Q11 bounded discovery; Q12 handoff provenance; Q7 contributor docs; Q1 release curation Final independent closure review after integration, packaging and artifact checks Formatting and documentation-baseline amendments arrived after the initial Q1 freeze. They were applied and release curation was repeated at 64aa194 ; the final review explicitly accepted that sequence deviation. This additional work is included in the task's recorded consumption. Design Finding(s) Engineering change Handoff endpoint verification Q5 Seven route tests and expiry sensitivity Non-nested transaction contract Q10 Explicit transaction ownership, rollback and cleanup behavior Structural refactor Q3, Q4 Typed handlers, browser decomposition, shared retry and credential-rotation ownership Operational diagnostics Q2 Redacted, correlated and rate-bounded diagnostics Portable tooling Q6 Browser checks and separate source/script typechecks Strict agent inputs Q8 Shared action policy and strict integer validation Health version Q9 Health response derived from package version Bounded document discovery Q11 Short-lived cache with capacity and recovery boundaries Handoff provenance Q12 Persisted provenance, authorization checks and schema migration Public contributor documentation Q7 Accurate contributor guides and explicit documentation gates Release curation Q1 Curated source history, portable evidence and reproducible package Scope and method The target was the closed task [Complete] [Phase B] Design fixes for Q1 to Q12 , parent 01a0a59d-ca16-76b3-8a9a-5f95389d22e3 . The analysis included its sixteen worker threads and associated approval reviewers. Prior design work, Naruto's independent review sessions, human effort and subsequent evaluation/editorial work fall outside that population. Routine totals come from the canonical derived record , captured by codex-hook-v1 and closed by SessionEnd at 2026-09-16 04:13:19 UTC. The derived record does not retain worker task names, wait outcomes, compaction boundaries or terminal error reasons. A targeted accuracy audit therefore used deterministic jq , rg , find , Git and arithmetic against only this parent's 15,355,576-byte (approximately 15.4 MB) runtime file and selected child metadata. No model worker parsed logs. Prompt bodies, source bodies, secrets and raw output were not copied into this report. Product outcome and authorship were checked against the contribution record , Goku's final response and Naruto's final review . Product files remained read-only. The measured wall window is 47,022 seconds, from 15 September 15:09:37 UTC to 16 September 04:13:19 UTC. It includes a roughly 7h33m gap between completed parent turns, external test waits and user pauses; it is not active model time. Reading the counters Term Meaning in this report Activation A recorded model invocation, rather than a user prompt or a tool call Processed tokens Cached input + uncached input + output; cumulative across invocations Cached input Input tokens served from the model's prompt cache; still included in processed-token accounting Output Recorded output tokens, including reasoning where the counter exposes it Worker thread A delegated conversation that may contain multiple follow-up assignments Guardian An automatic approval-review component, distinct from a worker-health watchdog Wait-generating activation A parent invocation that issued a wait; its entire usage is associated with that invocation, not isolated instruction-by-instruction Frozen price equivalent Recorded token categories multiplied by a dated API rate schedule; an analytical comparison rather than a measured subscription charge Repeatedly presenting a large context contributes to the cumulative token count even when much of that input is cached. A 158M-token total therefore does not imply 158M tokens of unique code, prose or uncached processing. The analysis proceeded from derived aggregates to component/model reconciliation, then to a targeted deterministic audit of missing orchestration fields, and finally to comparison with independent product-review evidence. This editorial revision uses those existing findings; it does not reopen the raw transcripts or rerun the product tests. Delivered outcome The compared Git range 69686a3..bb7af7d contains 27 commits and changes 118 files: 16,369 insertions and 3,229 deletions. The engineering-file subset covers 46 files, 11,903 insertions and 2,767 deletions. These figures include tests and tooling and the full range also includes reviews and records; neither is a pure backend-line count. Naruto's independent closure review found no P0 or P1 and reproduced: source and script typecheck plus build; 100/100 TAP tests; 73/73 browser checks; 48 documentation files and 476 links with zero errors; clean Prettier and git diff --check ; a one-commit, 61-file curated repository with clean history scans; and a byte-identical 133,263-byte package with SHA-256 485f87aceaa5f4bebf7c2c49e7d55d44b68d1cab00fec1de5fc29442db911ae3 . These are candidate-level checks. They do not establish production reliability or absence of defects. Naruto recorded five non-blocking observations, including handoff-citation semantics, package reproduction instructions and retained module-size debt. Human candidate acceptance and publication remained separate decisions at the final-review snapshot. Canonical accounting The collector records 1,299 tool calls across the measured task. Token and activation totals are separated below by component; the total excludes Naruto's separate review sessions and the later performance analysis. Component class Components Activations Processed Cached input Uncached input Output Share Parent Astra 1 342 36,090,696 34,588,032 1,337,219 165,445 22.82% Worker threads 16 1,011 110,255,046 106,565,888 3,211,069 478,089 69.72% Approval-review guardians 38, of which 20 non-zero 152 11,791,577 9,690,112 2,081,891 19,574 7.46% Total 55 1,505 158,137,319 150,844,032 6,630,179 663,108 100% Reasoning output is a 238,390-token subset of output, not an additional category. Cache writes are zero. Processed tokens are not bytes transmitted, inference FLOPs, energy, quota usage or a subscription bill. Model mix and frozen price equivalent Rates are the dossier's 15 September 2026 frozen standard API rates. They may change. codex-auto-review has no recorded public rate and is excluded from dollars. Model Components Activations Processed Share Frozen standard equivalent GPT-6 Astra 1 342 36,090,696 22.82% $56.23 GPT-5.6 Sol 9 592 69,756,461 44.11$42.21 GPT-5.6 Luna 6 414 40,236,736 25.44% $1.15 GPT-5.6 Terra 1 5 261,849 0.17$0.21 codex-auto-review 20 non-zero 152 11,791,577 7.46% Unknown Named-model total: 146,345,742 tokens and $99.80 . Pricing the same named tokens entirely as Astra gives$218.81 , a 54.39% price substitution reduction . Worker-only usage is $43.57 at the actual model mix versus$162.58 at Astra rates, a 73.20% price substitution reduction . The frozen calculation assigns a lower price to the recorded model mix. It holds token counts constant and supplies no evidence about the amount of work an actual all-Astra run would perform; Sol and Luna together processed 109.99M tokens in the observed run. All sixteen worker threads Static task names became incomplete descriptions because the parent issued 28 followup_task calls and 24 send_message calls . The “actual work” column follows the completed worker-turn records and product contribution evidence, not the original label alone. Worker label Model Actual observed work and terminal state Processed Activations Cache read design_evidence Terra Read-only Q1-Q12 evidence map; no files changed 261,849 5 73.81% commit1_standard Sol B1-B4 engineering-standard document 246,654 6 73.59% commit1_checker Sol Documentation checker, tests and bounded corrections 1,500,331 24 94.83% commit1_navigation Luna Eleven-LLD index, plan link and citation corrections 559,111 12 89.74% commit1_gate Luna Commit-1 integrated gate; 56/56 tests 525,185 15 95.20% midpoint_merge Sol Reused for Q10 transaction guard and regressions 1,871,363 34 96.66% q5_handoff Sol Seven Q5 handoff-route tests and expiry sensitivity 2,775,134 34 96.53% q34_browser Sol Q3/Q4 browser split, mutation lifecycle and regressions 9,609,041 81 98.07% q34_server Sol Q3/Q4 server types, handlers, retry and credential rotation 9,473,162 68 96.72% midpoint_gates Luna Reused for integration, verification and midpoint records 3,968,535 68 93.92% final_merge Luna Rebase/integration, Q2-Q12 gates, packaging, two long artifact runs and final freeze 33,364,102 287 97.85% q6_portability Sol Reused for Q6, Q12 and B-F/B-D baseline work; final turn ended at model capacity after earlier deliverables 20,249,883 165 96.05% q2_diagnostics Sol Reused for Q2, Q8, Q9 and integrated Q11 work 10,394,239 83 96.43% q11_discovery Sol Reused for Q11, Q1 tooling and curation; final closing-record turn ended at model capacity after earlier deliverables 13,636,654 97 96.51% closing_records Luna Six-document closing packet 918,820 15 91.19% closure_merge Luna Final fast-forward while preserving five local edits 900,983 17 94.19% Four threads— final_merge , q6_portability , q11_discovery and q2_diagnostics —were used across multiple stages and together consumed 77,644,878 tokens: 70.42% of worker usage and 49.10% of the task. A follow-up can carry accumulated conversation context into later invocations. The aggregate concentration is observed; the fraction caused by reuse, and the benefit of starting fresh workers, remain unmeasured. Two Sol threads ended their last follow-up with Selected model is at capacity after completing earlier stages. The derived record still reports zero interruptions. This is not a contradiction in the current schema—capacity failure is not counted as interruptions —but it is a telemetry gap: component terminal outcomes should be explicit rather than inferred from raw child events. The approved telemetry-extension work, ED-6, is the relevant workstream. Its current enumerated fields do not explicitly include terminal outcomes or follow-up counts; those additions need an explicit specification rather than being described as already installed or already enumerated in the decision. Parent orchestration The parent recorded 330 tool calls: Parent action Calls Shell/tool execution 170 Spawn worker 16 Follow-up assignment 28 Worker message/steering 24 wait_agent 89 sleep 3 Orchestration and waiting were 160 of 330 tool calls, 48.48% . Tool-call count is not token cost, but it demonstrates how much of the parent's control flow became coordination rather than direct engineering. The canonical lifecycle record contains 12 parent Stop events. Dividing 342 parent activations by those events gives 28.5 activations per recorded stop . This is a descriptive average, not a per-prompt compliance result: it hides the distribution and uses lifecycle events as its denominator. The one-minute polling loop wait_agent : 89 calls; 88 × 60 seconds, 1 × 10 seconds. Outcomes: 66 timeout-like, 23 completion-like. Wait-generating activations: 10,281,999 processed; 10,153,856 cached input; 96,038 uncached input; 32,105 output, of which 28,105 reasoning. Share: 28.49% of canonical parent tokens and 6.50% of the whole task. Mean per wait-generating activation: 115,528 processed tokens. Frozen Astra price equivalent: $12.72 , or 22.62% of parent Astra price equivalent and 12.74% of all known-price usage. The largest parent work turn issued 62 waits, 50 of which timed out. Those wait-generating activations consumed 7,443,509 tokens. Another turn issued 23 waits, 14 timeouts, consuming 2,377,824 tokens. The runtime also recorded 42 SubagentStop lifecycle events while only 23 waits returned completion-like output. These counts are not one-to-one because one worker thread can run multiple follow-ups and events can batch, but they establish that completion delivery existed independently of a fresh one-minute status check. The polling loop was not the only way completions reached the parent. Finding: the global declarative no-poll guidance did not enforce event-driven behavior when the parent had no independent work. This task reproduces the previously observed active-versus-idle split at much larger scale. The causal contribution of platform instructions versus parent choice is not isolated here. flowchart LR A["Parent activation"] --> B["Issue wait: usually 60 seconds"] B --> C{"Wait result"} C -->|"Timeout: 66 returns"| D["Parent resumes; may issue another wait"] D --> A C -->|"Completion-like: 23 returns"| E["Process worker result"] The diagram shows the observed return categories; it does not count uninterrupted timeout-to-wait sequences. 66 timeouts is not automatically 66 repeated waits with no intervening work. The minimum requested timeout was 10 seconds , while the dominant cadence was