LLM-Driven Automated Testing Series (03): Unit Test Generation — TestGen-LLM and Qodo Cover

Tell an LLM to 'just write some tests' and you usually get code that doesn't compile, asserts something trivially true, or contributes nothing to coverage. Meta's TestGen-LLM paper doesn't try to make the model smarter — it bolts on a mechanical filter pipeline that discards anything that fails to prove its own worth. This post breaks down that filter design and how the open-source implementation, Qodo Cover, turned it into a runnable CLI tool.

·9 min read·AI Engineering
LLMAutomated TestingUnit TestingTestGen-LLMAI Engineering

Where "just write some tests" goes wrong

Hand a function to an LLM and ask it to "generate unit tests to improve coverage," and the most common outcome isn't tests that are merely mediocre — it's tests that look fine but test nothing at all:

def test_calculate_discount():
    result = calculate_discount(100, 0.1)
    assert result is not None  # this assertion will never fail
 
def test_process_order():
    # doesn't even compile: the mock's return type doesn't match the real interface
    mock_client = Mock(return_value="success")
    process_order(mock_client)

These tests have coverage numbers attached to them — sometimes they even make the coverage report look "better" — but they provide zero protection against real regressions. assert result is not None will never fail, which means it asserts nothing. Worse still is a second failure mode: LLM-generated tests sometimes don't even compile or run, and dropping them straight into CI just manufactures new red X's.

Meta's 2024 paper "Automated Unit Test Improvement using Large Language Models at Meta" tackles exactly this problem, with a tool called TestGen-LLM. Its core idea isn't "train a smarter model to write tests" — it's the opposite: assume the LLM's output is probably garbage, build a mechanical filter pipeline that discards anything substandard, and keep only the small fraction that can prove its own value.

TestGen-LLM's four-layer filter

The paper calls this approach Assured Offline LLM-Based Software Engineering (Assured LLMSE): the LLM is just one generation step in a pipeline, and its output is never trusted directly — it has to pass a series of verifiable checks, and only what clears every check gets recommended to a human. That word "assured" is the key distinction from "just use the model's output as the result."

Concretely, this is a four-stage cascade of elimination:

LLM generates a candidate test
      ↓
① Build Filter
   The candidate code must fully compile within the app's existing build infrastructure
   Fails to build → discarded immediately
      ↓
② Pass Filter (first execution)
   A test that compiles must pass on its first run
   — this step doesn't try to judge "did the failure reveal a real bug,"
     because there's currently no automated way to distinguish
     a real bug from a bad assertion, so the conservative default is:
     fails on first run → discard
      ↓
③ Flakiness Filter (stability)
   Run the test 5 times in a row — it must pass every single time
   Even one failure out of five → deemed flaky, discarded
      ↓
④ Coverage Filter
   Must prove it actually increases code coverage
   Fails to improve coverage → discarded, even if it passed the first three filters
      ↓
Only tests that survive all four layers get recommended to an engineer for review

The design philosophy is narrow the funnel at every stage — it's better to lose a valuable test than let a garbage test through. Layer three is particularly worth noting — running a test five times to filter flakiness is a technique Qodo Cover's roadmap later borrowed directly (more on that below).

One point worth clarifying, since it's easy to get wrong: mutation testing is not part of TestGen-LLM's filter pipeline. The paper does mention that mutation coverage would theoretically be the best-performing criterion, but it explicitly lists this as future work, noting that "it is challenging to deploy such computationally demanding techniques at the scale we would require." In other words, the mutation-testing idea is directionally right, but too expensive to run in practice — so this version relies on the much lighter "did coverage improve" signal instead of the more precise but far costlier "were the mutants of each line actually killed."

Real numbers: how this performed inside Meta

The paper reports two sets of numbers — one from a lab-style evaluation, one from real engineer usage — and the gap between them is itself informative.

Lab evaluation: Instagram's Reels/Stories components

Tested on 86 Kotlin components (31 Stories, 55 Reels):

75%  of test classes: got at least one new test case that compiles correctly
57%  of test classes: compiles AND passes reliably (cleared the first three filters)
25%  of test classes: compiles, passes reliably, AND increases line coverage (cleared all four filters)

This tells a very practical story: there's a big drop-off between "can an LLM generate a test" and "can an LLM generate a test that's actually worth something." Three-quarters of components got a test that "looks like it compiles," but only a quarter actually improved coverage — the middle three filters are precisely what's cutting away the "looks useful but isn't" output.

Real-world usage: three test-a-thons

More convincing than the lab numbers is data from three internal test-a-thons — sprint events where engineers actually reviewed the generated diffs:

                        Diffs generated   Accepted by engineers   Acceptance rate
──────────────────────────────────────────────────────────────────────────────
Instagram (1st round)        17                   16                  94%
Instagram (2nd round)        42                   36                  86%
Facebook                    280                   144                 51% (~70% among reviewed diffs)
──────────────────────────────────────────────────────────────────────────────
Total: 196 out of 1,979 test classes were successfully improved (~10%)
       73% of TestGen-LLM's improvement suggestions were accepted for production

Comparing the two datasets side by side: in the lab, roughly 25% of test classes "clear all four filters." In the real-world test-a-thons, 73% of improvements were accepted by engineers. These numbers don't contradict each other — the filters eliminate output that's mechanically substandard (doesn't build, unstable, doesn't improve coverage), while "engineer acceptance" measures whether a test that's already cleared the mechanical bar is actually worth merging. In other words, the four filters solve "don't let garbage through" — they don't guarantee "everything generated is valuable." That judgment still needs a human.

The open-source implementation: how Qodo Cover turns this into a CLI tool

TestGen-LLM itself is an internal Meta tool and was never open-sourced. Qodo Cover (formerly Codium's Cover-Agent) is the first public open-source implementation of this same approach, runnable directly against your own project.

Its architecture maps the paper's "filter" concept onto four concrete components:

Prompt Builder
   Gathers necessary context from the codebase (source file, existing tests,
   coverage report) and assembles it into a prompt for the LLM
        ↓
AI Caller
   Calls the LLM with the assembled prompt to generate a candidate test
   (via LiteLLM, so the underlying model provider is swappable)
        ↓
Test Runner
   Actually runs the project's test command against the newly generated test
        ↓
Coverage Parser
   Parses the coverage report (Cobertura/JaCoCo, etc.) and checks
   whether coverage genuinely increased
        ↓
   Passes → keep the test, move to next iteration
   Fails  → discard, generate a new candidate

This is effectively a concrete implementation of TestGen-LLM's Pass Filter + Coverage Filter (the Build Filter is implicit in a failed Test Runner execution; a Flakiness Filter is on the roadmap, explicitly described as planned to "check test flakiness, e.g. by running 5 times as suggested by TestGen-LLM").

Usage is a CLI-driven loop: specify the source file, test file, coverage-report path, a target coverage percentage (--desired-coverage), and a max iteration count (--max-iterations); the tool repeatedly generates candidate tests, runs them, and checks coverage until it hits the target or runs out of iterations. This loop can be wired straight into a CI pipeline, or run locally as a one-shot "bump this module's coverage" command.

Worth flagging: Qodo Cover, the open-source project, is no longer actively maintained as of mid-2025 (the README explicitly marks it "no longer maintained" and suggests forking if you want to keep developing it). That doesn't diminish its value as a reference implementation of the TestGen-LLM approach, but it does raise a maintainability risk if you'd depend on it in production — a recurring theme in this series: open-source proof-of-concept projects and production-ready tools are often not the same thing.

Where this approach actually holds up

Putting Meta's lab data and test-a-thon data together, you can map out where this method genuinely works and where it doesn't.

Works well: modules with relatively self-contained logic and clear dependencies — the LLM can reasonably infer input/output relationships from function signatures and existing test style, and the four-layer filter effectively strips out most garbage output.

Doesn't work well:

  • Code that depends heavily on external state or complex mocks — LLM-generated mocks frequently don't match the real interface's actual behavior, which tends to get caught by the Build Filter or Pass Filter, resulting in a low survival rate
  • Code that requires deep business context to write a meaningful assertion — a test might "run successfully" but assert something meaningless (like assert result is not None in the opening example). The coverage filter only checks whether the code path was executed, not whether the assertion itself means anything — this is a blind spot the four-layer filter never addresses
  • Scenarios demanding extreme determinism and reproducibility — the flakiness filter only runs five times, so statistically it can still miss a test that's fundamentally unstable

One point worth emphasizing: the four-layer filter guarantees that a test is mechanically valid — it does not guarantee that the test's assertions are semantically correct for the business logic. Coverage improvement is an automatable proxy metric, but "did this test actually check the thing I care about" still needs human review. That's exactly why Meta's own data treats "cleared all filters" and "accepted by engineers" as two distinct numbers that each need to be examined separately.

Summary

  1. LLM-generated tests are probably garbage by default — they may not compile, assert something trivial, or contribute nothing to coverage. TestGen-LLM's fix isn't a smarter model, it's a four-layer mechanical filter (build, first-run, stability, coverage) that eliminates substandard output stage by stage
  2. Mutation testing is not currently part of TestGen-LLM's methodology — the paper explicitly lists it as future work constrained by computational cost, and uses the much lighter "did coverage improve" signal instead
  3. The lab number (25% clear all filters) and the real-world number (73% accepted by engineers) measure two different things — the former is mechanical validity, the latter is human judgment of value, and the four-layer filter only solves the former
  4. The open-source implementation Qodo Cover turns this idea into a runnable CLI/CI iteration loop, but the project itself is no longer maintained — maintainability risk needs to be weighed before relying on it in production

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

← Back to Blog