LLM-Driven Automated Testing Series (13): Flaky Tests Aren't a Rerun-Count Problem, They're a Classification Problem

pytest-rerunfailures (477 stars) and flaky (397 stars) handle test instability with the same one-liner: if it fails, rerun it a few times. That doesn't answer what the planning doc actually asks — was this failure a real bug, environment jitter, or a badly designed test? The only project actually attempting that root-cause classification with an LLM is a 2-star, less-than-two-months-old repo called pytest-triage — and its most instructive feature turns out to be the four safety invariants it wraps around AI judgment, guaranteeing AI is never allowed to affect whether a test actually passes or fails.

·11 min read·AI Engineering
LLMAutomated TestingFlaky TestAI Engineering

Rerun-Count Statistics Can't Answer "Why Did It Fail"

This series has been verifying the same pattern over and over: the more structured a domain (API testing), the more conservative LLM adoption stays; the more a domain genuinely needs semantic understanding (visual regression, UI manipulation), the more real LLM value shows up. Flaky tests — the same code, the same test, run repeatedly, sometimes passing and sometimes not — are a good domain to test this pattern against, because the planning doc's question is very specific: when a test failure happens, which of three root causes is it — a genuine bug in the code, an environment problem (network jitter, timing races, concurrency contention), or a test case that's simply not robust enough by design? How do traditional tools currently make this call, and can an LLM replace or assist with that classification?

The finding up front: the highest-starred traditional tools (pytest-rerunfailures, 477 stars; flaky, 397 stars) handle flaky tests with essentially one sentence — if it fails, rerun it a few times, and if it passes once, pretend it never failed. That answers nothing about "why it failed" — it just statistically smooths instability away. The only open-source implementation actually attempting LLM-based root-cause classification is a project released less than two months ago with 2 stars, pytest-triage — and it turns out to be more disciplined than expected, disciplined enough that it directly bakes the "degradation strategy when classification accuracy is insufficient" question this article wants to discuss into its own architecture.

pytest-rerunfailures and flaky: 477 and 397 Stars, Built on "Just Rerun It"

pytest-rerunfailures (477 stars, maintained by the official pytest-dev organization, last commit 2026-09-24) and Box's flaky (397 stars, Apache 2.0, last commit 2025-09-08) are the de facto industry-standard plugins in this space — an extremely simple mechanism: tag a test (e.g. @pytest.mark.flaky(reruns=3)), auto-rerun it up to N times on failure, and count it as passed if any single attempt passes.

This mechanism performs zero root-cause judgment — it doesn't distinguish "this failure was a network timeout" from "this failure was a concurrency deadlock" from "this failure genuinely means the code broke." Its assumption is blunt: a real bug reproduces reliably, so if it still fails after several reruns it's probably real; a flaky issue clears up after one rerun. That assumption holds often enough in practice, which is exactly why these two tools have earned several hundred stars each — cheap, zero configuration overhead, no need to understand the failure to use it. But the planning doc's question is precisely this mechanism's ceiling: if a real bug only triggers under certain timing conditions (a race condition that fires rarely), the rerun mechanism will wave it through as flaky; if a test itself is poorly designed (depends on execution order, or uses a fixed sleep() to wait for something async), the rerun mechanism will just keep "happening" to pass without ever telling you the test needs a rewrite. Rerun-count statistics solve the short-term problem of "should CI go green right now" but do nothing for the long-term problem of "what's actually causing this instability" — and that accumulated long-term problem is exactly the tech debt so many teams end up with: nobody knows which tests are genuinely flaky, why, or when they should be fixed.

DeFlaker: No LLM, But a Genuine Root-Cause Distinction — Cross-Referencing Coverage Against the Diff

Worth a separate mention on the traditional side: DeFlaker (gmu-swe/deflaker, 39 stars, a Maven extension out of George Mason University's software engineering lab). What makes it smarter than "just rerun it" is that it genuinely tries to answer "is this failure related to this particular code change" — using a purely deterministic technique: cross-referencing lines changed in a git diff against per-test code coverage collected during the run.

The logic is straightforward: if a test fails and its coverage data shows it actually executed lines changed in the diff, DeFlaker flags it as a likely real regression tied to that change; if a test fails but never touched any changed lines, DeFlaker marks it as apparently flaky and unrelated to the diff. This requires no understanding of code semantics at all — purely a computable fact about whether the test's execution path crossed the changed code. Compared to plain rerun mechanisms, this is a genuine step toward "root cause," but the question it can answer is still limited to "is this related to the current change" — it can't tell you, if unrelated, whether the cause was an environment issue or a test-design issue, the finer three-way split this article is after.

pytest-triage: 2 Stars, But the Most Careful Thinking I've Read on "How Much Power Should AI Judgment Get"

IKrysanov/pytest-triage (2 stars, Apache 2.0, created 2026-07-23, last commit 2026-09-21 — one of the newest, smallest projects researched in this series) states its purpose plainly in the README's first line: "collects a machine-readable report of every failed test and — optionally — enriches each failure with an LLM verdict (regression/flaky/environment/test bug)." Its classification enum maps almost word-for-word onto the planning doc's three root causes — the category field's enum values are regression, flaky, env (environment), test_bug (test design problem), and unknown (can't determine), paired with confidence, hypothesis (a concrete claim, e.g. "ConnectionError in tests/test_shop.py::test_db_connection"), and suggested_fix. It calls the Anthropic or OpenAI API and forces the model to return this structured schema via tool use, not free text — the Anthropic provider's schema sets "strict": True and "additionalProperties": False, and forces tool_choice={"type": "tool", "name": "record_verdict"} so the model can only invoke that one tool, never reply in free text.

What's genuinely worth recording is the four "safety invariants" wrapped around this LLM judgment — the project's own word is "invariants," and it explicitly states these are enforced by tests, not just promised in documentation:

  1. "AI never affects the test verdict" — with the example: "A provider raising, timing out, or returning garbage leaves the run byte-identical to a run without the plugin." This directly answers the planning doc's question about "the degradation strategy when root-cause classification accuracy is insufficient" — pytest-triage's answer isn't "escalate to human review when the model is wrong," it's cutting the causal chain between AI judgment and the test's actual outcome at the architecture level: AI only ever appends a diagnostic note after a failure has already happened; it is never allowed to decide whether the run counts as pass or fail.
  2. Disabled by default — installing the plugin changes nothing about existing test suite behavior unless explicitly turned on.
  3. A hard budget ceiling — default --ai-budget=10, capping the run at 10 model calls regardless of how many tests fail; test #11 onward returns straight to unknown with no call made. Setting --ai-budget=0 keeps the reporting feature active while spending nothing.
  4. Four layers of defensive wrapping — CachingClient (deduplicates identical failure signatures so repeats don't cost budget) wraps CircuitBreakerClient (trips after a timeout or two consecutive errors, returning unknown for all subsequent calls) wraps BudgetedClient (rejects once the budget is spent) wraps TimedOutClient (a hard wall-clock cap per call, run in its own thread, so a timeout or crash never stalls the whole test run). The README states the rationale plainly: "Cache is outermost, so a cache hit costs neither budget nor time; the breaker sits above the budget, so a tripped breaker spends nothing at all."

Together, these four invariants answer one question: if the LLM's root-cause judgment can itself be wrong (hallucination, API timeout, malformed output), what should the system's default posture be? pytest-triage's answer isn't "make the AI more accurate" — it's confining the cost of a wrong AI judgment strictly to "this one diagnostic note is inaccurate," never letting it propagate into "the CI pipeline's pass/fail result" or "the total runtime/cost of the test run." This is the third reappearance of the same engineering discipline seen in AutoRestTest confining its LLM to SmartValueGenerator (article 11) and vlmkit confining its VLM judgment to "the narrowest slice deterministic data doesn't cover" (article 12) — except this time the constraint isn't about what the LLM should be allowed to decide, it's about what the system should do when the LLM's decision turns out to be wrong.

Three Root Causes, Statistical Degradation, Scope of Judgment: More Concrete Than Expected

The planning doc's three questions now all have concrete grounding:

What LLM-based three-way root-cause classification from failure logs does that traditional rerun statistics can't: the rerun mechanism's only signal is "did it still fail after rerunning" — it never reads the content of the failure at all. pytest-triage's LLM verdict actually reads the real traceback, exception type, and stdout/stderr tail logs, producing a concrete hypothesis ("this is a ConnectionError in tests/test_shop.py") and a suggested fix — something rerun-count statistics cannot do in principle, because they never look at what actually failed.

What manual triage cost is saved compared to traditional rerun-count-based flaky detection: DeFlaker already proves "partial root-cause distinction via deterministic methods" is viable (diff-vs-coverage cross-referencing), but it can't produce human-readable diagnostic text — it can only tell you whether the test ran the changed code or not. pytest-triage's value is turning that step into "a human-readable hypothesis plus an actionable fix suggestion," saving the engineer from manually digging through tracebacks and guessing at causes — but that value depends entirely on classification accuracy; once the classification is wrong, the saved triage cost can turn into an added cost of being misled.

The degradation strategy when root-cause classification accuracy is insufficient: pytest-triage gives the most concrete answer found in this research — not a binary choice between "escalate to human review" and "mark for quarantine directly," but an architecture that tightly bounds the blast radius of "the classification was wrong": budget exhausted, breaker tripped, or timed out, all fall straight through to unknown — no guessing, no retrying, no effect on the test's actual result. This is more practical than a debate over "should we trust the AI's judgment" — it reframes the question as a pure engineering one: even if the AI's judgment accuracy were zero, what's the worst case for this system? pytest-triage's answer is: the worst case is a pile of unknowns, and the test run itself is completely unaffected.

Summary

  1. pytest-rerunfailures (477 stars) and flaky (397 stars) are the de facto standard tools in this space — rerun N times on failure, count a pass on any single success, with zero root-cause judgment. They solve the short-term problem of "should CI go green" but not the long-term problem of "what's actually causing this instability."
  2. DeFlaker (39 stars) takes a step forward on root-cause judgment using a purely deterministic technique (cross-referencing git diff against code coverage) — it can distinguish "is this failure related to the current change," but not the finer split of "if unrelated, was it an environment issue or a test-design issue."
  3. pytest-triage (2 stars, created July 2026 — one of the newest and smallest projects researched in this series) is the only verified open-source implementation that genuinely calls an LLM API for root-cause classification — outputting an enum of regression/flaky/env/test_bug/unknown alongside confidence, hypothesis, and suggested fix, with tool-use forcing structured output.
  4. What's most worth recording about pytest-triage isn't its classification ability, but the four safety invariants wrapped around its AI judgment: AI never affects the actual test pass/fail result, disabled by default, a hard budget ceiling (default 10 calls per run max), and four layers of defensive wrapping (cache/circuit-breaker/budget/timeout) — a concrete engineering answer to "what happens when root-cause classification accuracy falls short": not improving accuracy or escalating to humans, but confining the blast radius of a wrong judgment to "this diagnostic note is inaccurate," never letting it propagate into the test run's actual outcome.
  5. Placed side by side, the three projects echo the pattern this series keeps confirming: high-star traditional tools solve statistical stability problems without touching semantic understanding; the one project doing semantic-level root-cause classification has the fewest stars and the shortest track record, yet its architectural restraint about "AI judgment can be wrong" exceeds that of most mature tools.

Check out PrimeSkills — a curated marketplace for AI agents and skills, all validated in real enterprise workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

← Back to Blog