The Evidence This Series Has Already Gathered Can Now Answer the Engineering Questions
The previous 13 articles each verified a specific project — each one answering "what is the LLM actually doing in this scenario, and what can't traditional tooling do here." This article doesn't introduce a new open-source project. Instead, it lays the already-verified evidence side by side to answer the three more engineering-flavored questions the planning doc saved for last: what order of magnitude is token cost in testing scenarios compared to code-generation scenarios; where in the pipeline should the human-confirmation gate sit; and which scenarios should explicitly not use an LLM for testing at all. None of these three questions need a new OSS verification — what they need is putting concrete numbers and concrete designs, already read in prior research, next to each other.
Cost Order of Magnitude: Thousands of Dollars vs. a Dime — the Difference Isn't the Model, It's How the Task Boundary Is Drawn
Two sets of already-verified real numbers, side by side.
Code-generation scenario: an enterprise-grade end-to-end Bug-fix workflow I documented earlier (ai-bug-e2e-workflow-enterprise) — 12 nodes, fully AI-driven, from Jira ticket intake to Gerrit submission — burned thousands of dollars in tokens during its debugging phase. One concrete failure case makes the point well: in one real run, after a single Turn racked up 117 tool calls, right as the Sonar scan finished and the workflow tried to move to the next step, the API request was aborted server-side outright — not because the agent didn't know what to do next, but because the Turn's accumulated context had grown large enough that the server rejected the request. The root cause was "long-chain context explosion": reading the Jira ticket, downloading and unpacking logs, analyzing log text, reading multiple source files, Code Review results, Sonar scan reports — all of it piling up linearly inside a single Turn.
Testing scenario: two numbers already verified in this series. From article 11, AutoRestTest's README, verbatim — "testing an average API with ~15 operations using GPT-4o-mini, the cost was approximately $0.1" — testing a 15-operation API for about a dime. From article 13, pytest-triage's default configuration — --ai-budget=10, meaning no matter how many test cases fail in a single run, it spends at most 10 model calls, and each call's prompt size has a hard ceiling ("4 KB of traceback, 2 KB of each output tail, 1 KB of the exception message," with typical prompt size "well under 2 KB").
These two sets of numbers differ by several orders of magnitude — not because testing scenarios use a cheaper model, but because the task boundary is drawn completely differently. In the Bug-fix workflow, the LLM's job is end-to-end: read logs, locate root cause, modify code, understand Code Review feedback, fix code again, then understand CI failure output — every step's output feeds the next step as input, context accumulates linearly, and there's no hard budget ceiling, because "fixing a bug" has no natural cap on how many calls it takes. In testing scenarios, the pattern this series keeps confirming is confining the LLM to one narrow, clearly-bounded slice: AutoRestTest only lets the LLM generate "realistic-looking" field values, leaving sequencing decisions and dependency inference to reinforcement learning and vector similarity; pytest-triage only lets the LLM append a diagnostic note after a test has already failed, and architecturally wraps that one slice in a hard budget ceiling and defensive layering. A narrow task boundary is itself what makes cost budgetable — you can calculate in advance the worst case for how much this system will spend. An end-to-end task boundary can't do this, because how long the chain runs or how many retry rounds it takes isn't known up front.
This also answers the core of the planning doc's "testing-scenario token cost magnitude" question: it's not that "testing is inherently cheaper than code generation" — it's that nearly every testing project verified in this series that actually made it to production adopted a "confine the LLM to one narrow slice + hard budget ceiling" design, and that design itself is what makes the cost order of magnitude calculable and low. Conversely, if someone tried to build a genuine "end-to-end AI testing workflow" (read requirements, write tests, run tests, analyze failures, fix tests, rerun), it would very likely run into the same context-explosion problem as the Bug-fix workflow — that's not a privilege unique to testing scenarios, it's a consequence of task-boundary design.
Where the Human-Confirmation Gate Belongs: This Series' Answer Is "Only Review Failure Cases," Not "Review After Generation" or "Review Before Execution"
The planning doc frames this as a three-way choice: review after generation, review before execution, or review only failure cases. Placing this series' evidence next to the human-confirmation design verified in the Bug-fix workflow, the answer clearly leans toward the third option.
The three human-confirmation gates in the Bug-fix workflow are none of them "review immediately after generation" or "approve before execution" — instead, they all trigger only after the automated path has already run its full course and the system itself determines an escalation is needed: Gate B triggers only after Code Review's automatic retry loop has run three full rounds without passing, at which point the system presents a comparison report and offers "accept current code" or "escalate to human fix" as the two options; Gate C triggers only after Gerrit receives a failing vote from the CI pipeline. Not one of these gates is a "let me take a quick look" inserted right after AI generates code, and not one is a "you need my approval before you can act" inserted before AI starts executing — every one of them is "the automated path has run out of road, the system has hit a case it can't judge on its own, and only then does a human get pulled in."
The testing projects verified in this series follow the same path. pytest-triage's design is even more extreme — it doesn't really have a "confirmation gate" in the conventional sense at all: AI judgment only appends a diagnostic note after a test has already failed, and whether a human reads that note or acts on it is entirely optional — the AI's judgment is architecturally forbidden from feeding back into the test run's actual result ("AI never affects the test verdict"). vlmkit's two-stage pipeline follows the same pattern — Stage 2 produces FIX: suggestion lines, a proposal for a human to consider, never a patch applied automatically; and this AI judgment only triggers at all once a visual difference has been detected — that is, only after something has already failed or changed.
Why does "only review failure cases" turn out to be the answer both scenarios converge on, rather than "review after generation" or "review before execution"? Because the cost structure of the other two modes doesn't scale — if every generation needs a human glance, labor cost grows linearly with automation's call frequency, defeating the point of automating in the first place; if every execution needs prior approval, the wait for approval eats up whatever time automation was supposed to save. "Only review failure cases" ties human effort to the signal "the system genuinely can't judge this on its own / the automated path has run out of road" — human attention is needed only when both signals appear together, and not otherwise. That's also exactly why all three gates in the Bug-fix workflow are set on triggers like "retries exhausted" or "an external system has issued an explicit failure signal (a CI vote)," rather than "after every single AI output."
Which Scenarios Should Explicitly Not Use an LLM for Testing
The planning doc names three categories. Combined with the evidence verified across this series' 13 prior articles, concrete reasons can be given rather than just intuition.
High-frequency regression testing — this is a cost-structure problem, not a capability problem. Article 11 verified Schemathesis: a Hypothesis-based property testing engine that enumerates schema-driven boundary values, with zero external API calls, fully deterministic, and rerunnable at zero marginal cost, no matter how many times regression needs to run. If the same boundary-value mining were done with an LLM instead, every regression run would cost one API call's worth of money and latency — and once call frequency is tied to CI trigger frequency (every commit, every PR), cost grows linearly or worse with team size and commit frequency — which directly inverts the "testing-scenario cost is low" conclusion established above, because that conclusion's precondition was the LLM staying in a narrow slice at low trigger frequency. Once the LLM is pushed into a high-frequency regression path triggered on every commit, the narrow-slice design can no longer absorb the call volume.
Pure-logic unit tests — this is a task-structure problem. Article 11 already identified the core reason: a schema/type system can exhaustively enumerate "valid" boundaries, deriving boundary values, extreme values, and type mismatches — the classic defect categories — without needing any language understanding at all. Pure logic code (a sort function, a state-machine transition) has a correctness standard that's explicit and formalizable; there's no fuzzy zone requiring common-sense judgment like "does this value look like real business data" — which is exactly where this series has repeatedly found LLMs adding the least incremental value.
Scenarios demanding extreme determinism — this is a risk-tolerance problem, and pytest-triage's architecture already gives the most concrete answer. If a judgment result needs to be 100% reproducible with zero chance of model-hallucination-driven variance, the LLM's output can't be allowed to enter the judgment chain directly — it can only serve as an out-of-band suggestion sitting outside that chain. pytest-triage turns this principle into an "invariant" — AI is never allowed to affect a test's pass/fail result, and even if the model errors, times out, or returns garbage, the entire run's outcome remains byte-identical to a run without the plugin installed. This isn't "the LLM isn't accurate enough yet, so it can't be used" — it's that even if the LLM eventually becomes more accurate, as long as a judgment requires absolute determinism, the risk exposure of putting an LLM inside that judgment chain never drops to zero on its own. Excluding it from the judgment chain at the architecture level is the only way to lock that risk exposure at zero.
Summary
- The end-to-end Bug-fix workflow (a real enterprise practice) burned thousands of dollars in tokens during its debugging phase, and the root cause was an end-to-end task boundary (read logs, modify code, understand CR, modify code again) that accumulates context linearly with no natural cap on call count — in one real run, a single Turn was aborted server-side after 117 tool calls.
- The testing scenarios verified across this series come in orders of magnitude cheaper (AutoRestTest testing an API for about a dime, pytest-triage's default budget capping any single run at 10 model calls) — not because testing is inherently cheaper, but because these projects overwhelmingly confine the LLM to one narrow slice paired with a hard budget ceiling, and that narrow boundary is what makes cost calculable in advance.
- The human-confirmation gate belongs at "only review failure cases," not review-after-generation or review-before-execution — all three gates in the Bug-fix workflow trigger only after automated retries are exhausted or an external system issues an explicit failure signal, and the AI judgments in pytest-triage and vlmkit likewise only produce suggestions after a failure or change has already been detected, never applying automatically or feeding back into the actual result.
- High-frequency regression testing shouldn't use an LLM, because the narrow-slice-plus-low-frequency cost structure can't absorb high-frequency triggering, and deterministic options like Schemathesis already rerun at zero marginal cost; pure-logic unit tests shouldn't use an LLM, because their correctness standard is formalizable with no fuzzy zone requiring common-sense judgment; judgment chains demanding extreme determinism shouldn't let an LLM participate directly, because only excluding it from the chain entirely locks hallucination-driven risk exposure at zero — waiting for the model to get accurate enough isn't a substitute.
- The same conclusion running through all 14 articles closes out here: every project in test automation that's actually made it to production, kept costs under control, and stayed reliable, without exception, confines the LLM to one narrow slice that genuinely needs semantic, visual, or language understanding — and hands everything else — sequencing decisions, dependency inference, correctness judgment, the final pass/fail result — to cheaper, more deterministic, more auditable traditional techniques. That's not a capability gap in LLMs. It's that most of what "testing" actually involves never needed a language model in the first place.
Check out PrimeSkills — a curated marketplace for AI agents and skills, all validated in real enterprise workflows. No fluff, just what actually works.
Find more useful knowledge and interesting products on my Homepage