Teardown of Anthropic's Official Evaluation Methodology: From Defining Success Criteria to claude plugin eval

Two Docs, One Complete Chain

Anthropic has two official documents on evaluation, and they are more interesting read together:

• "Define success criteria and build evaluations" (Claude developer platform): the methodology. Define what "good" means first, then design evaluations to measure it.
• "Test plugins with evals" (Claude Code): the engineering. A ready-made command, claude plugin eval, for plugin and Skill authors.

The first answers "what to evaluate and how to score it"; the second answers "how to make evaluation a repeatable, gate-able engineering step". This article follows that line: the methodology first, then the tool, then the design ideas worth borrowing.

Part 1: Methodology — Define Success Before You Evaluate

The four SMAR traits of success criteria

Good success criteria should be:

The contrast the doc gives is vivid. Even something fuzzy like "safety" can be quantified:

The sentiment-analysis example uses all four: an F1 of at least 0.85 (specific, measurable) on a held-out test set of 10,000 diverse Twitter posts (relevant), a 5% improvement over the current baseline (achievable).

One application, several dimensions

The doc stresses that most use cases need multi-dimensional criteria, and lists eight common ones: task fidelity, consistency, relevance and coherence, tone and style, privacy preservation, context utilization, latency, and price. Its "good" criterion for sentiment analysis combines them:

• F1 ≥ 0.85
• 99.5% of outputs are non-toxic
• 90% of errors are merely inconvenient rather than severe
• 95% of response times are under 200 ms

Note the third item: where "inconvenient" ends and "severe" begins has to be defined in your own project. Errors must be classified by severity — this theme comes back later.

Three principles for building evaluations

1. Be task-specific. The test set should mirror the real task distribution and include edge cases: irrelevant or nonexistent input, overly long input, poor-quality or harmful input, and ambiguous cases where even humans would struggle to agree.
2. Automate when possible. Structure questions so they can be graded automatically.
3. Volume over quality. More questions with slightly lower-signal automated grading beats fewer questions with high-quality human grading.

The third is easy to overlook. Many teams polish 20 questions, review them all by hand, and then dare not change models. The official stance: coverage matters more; each individual signal can be coarser.

Choosing among six grading methods

The doc maps each grading method to an evaluation dimension. The table works as a quick selection guide:

The pattern is clear: use code wherever judgment can be deterministic (exact match, cosine similarity, ROUGE-L) and hand only subjective dimensions to an LLM (tone, privacy judgment, context use).

Exact match is almost embarrassingly simple, which is precisely the point:

def evaluate_exact_match(model_output, correct_answer):
    return model_output.strip().lower() == correct_answer.lower()

Paired with an "output only the label" constraint in the prompt, a classification evaluation is done.

Edge-case design in the test sets

The edge cases in the doc's examples are worth copying:

• Sentiment: sarcasm ("I just love it when my flight gets delayed for 5 hours #bestdayever"), mixed sentiment (terrible plot, phenomenal acting)
• FAQ consistency: typos ("Wut's yur retrn polcy?"), long rambling questions, questions padded with irrelevant detail
• Summarization: multi-topic articles, misleading titles
• Privacy: explicit PHI, hypothetical PHI, implicit PHI ("my son was prescribed the same medication as his father last year")
• Support: angry customers, complex issues, a compliment phrased as a complaint

What they share: they sit exactly where models are most likely to trip semantically, which random sampling rarely reaches.

Two practical notes on LLM grading

A code comment carries two small but important points. When grading with an LLM, it's usually best to use a different model from the one that produced the output. And grading prompts ask for only a number or yes/no, which makes parsing reliable.

Those correspond to two real problems: self-preference bias and parsing stability.

Part 2: claude plugin eval — Turning Methodology into a Command

claude plugin eval runs your plugin against a suite of test cases and scores the results. Its core design breaks into four layers.

1. Case: one directory per scenario

An eval suite lives in the plugin's evals/ directory, with each case a subdirectory:

my-plugin/
├── .claude-plugin/plugin.json
├── skills/...
└── evals/
    ├── first-case/
    │   ├── prompt.md          # frontmatter: case fields; body: the prompt
    │   ├── graders/
    │   │   ├── criteria.md    # frontmatter: type + options; body: rubric
    │   │   └── skill-fired.md
    │   └── case.yaml          # optional: only for context.* fields
    └── results/               # written by each run; add to .gitignore

Design points worth noting:

• Files are the config. The prompt is the body of prompt.md; run limits (max_turns, timeout_seconds, allowed_tools, etc.) sit in frontmatter. No database or platform dependency, so cases go into Git and through code review naturally.
• Every case needs at least one grader, or it fails to load. A case without a scoring standard isn't allowed to exist.
• Write prompts the way users really talk, without naming the Skill. The doc emphasizes this; otherwise you can't measure how reliably the Skill triggers.

There's also a scaffolding command, claude plugin eval init. It reads your plugin, asks what outcome you want, proposes prompts that should and shouldn't trigger the plugin, designs graders for each, does a trial run to check the graders behave, and then writes the files. Note that "should not trigger" is a first-class citizen: both false triggers and missed triggers are defects.

2. Run: isolated, repeated, bounded

For each case run, Claude Code starts a fresh, isolated non-interactive session that loads only your plugin, sends the prompt, and lets Claude work until it finishes or hits the case's turn or time limit.

Each case runs three times by default, because "a single run of a non-deterministic agent tells you very little". A run's score is the fraction of graders that passed (weighted if you set weights); a case's score is the mean across its runs; the case passes when it reaches --threshold (default 1.0).

The isolation is thorough:

The value is reproducibility: your local config can't contaminate results, and a different person on a different machine should reach the same conclusion.

3. Grader: six types, two cost tiers

A grader is a pass/fail check on what Claude produced. Six types:

The first four are computed from the transcript and files and incur no model cost; the last two call a judge model and add cost. The llm grader uses majority voting (two of three) to dampen the judge's own randomness.

What a grader can look at is categorized too (target / focus): the last message, the full session trace, the list of created file paths, the content of a specific file, and the recorded calls to mocked MCP tools.

A practical combination from the doc is the two-grader pattern:

---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:[\w-]+:)?your-skill-name"'
---

One grader checks the outcome (final message or artifact); the other checks the process (whether your Skill was actually invoked). Together they answer "is the answer correct" and "did your plugin produce it".

Stability advice from the doc

This overlaps heavily with pitfalls we've hit in our own evaluations:

• For long outputs (e.g. generated files), grade file content with regex, which checks the whole file the same way every time; keep llm graders for short outputs and write rubrics with concrete PASS and FAIL conditions.
• If a case's tool_used: Skill grader passes but Δ is negative, suspect the judge model before the plugin — a small judge can mark a correct answer wrong because the format differs from the rubric's description. Re-run with a stronger judge and tighten the rubric so format doesn't decide the verdict.
• To check that a build or test passed, have the prompt ask Claude to run it and write the result to a file, grade that file, and assert with tool_used plus input_match that the command actually ran.
• There are no custom code graders, the doc says explicitly. The capability surface is deliberately narrowed.

4. Baseline: with vs without, and Δ

This is the most elegant part of the design.

A high score alone doesn't tell you whether the plugin helped, because Claude might do well without it.

So each case runs in two arms by default — with the plugin loaded (with-arm) and without it (without-arm) — yielding WITH and W/OUT scores. Their difference, Δ, is the plugin's net contribution:

CASE        WITH  W/OUT Δ      RUNS COST    NOTES
first-case  1.00  0.33  +0.67  6    $0.41

If a case scores 1.0 both with and without the plugin, the plugin isn't why it passes.

One detail is handled carefully: a check like "the Skill was invoked" can never pass without the plugin, so counting it would push the without-arm toward zero and inflate Δ. Such graders are therefore excluded from the score in both arms and shown only as pass/fail indicators in the with-arm. When needed, arm: both forces both sides to score them (for example a "must not invoke this Skill" check with min: 0, max: 0).

The typical first finding in the doc: Δ near zero while tool_used: Skill fails — meaning that under natural phrasing, Claude didn't pick your Skill. The fix isn't code; it's the Skill's description. Edit it and re-run for comparison.

Fixtures and mocks: testing Skills that depend on external services

Skills often call MCP tools, and real services aren't controllable. plugin eval handles this with Markdown files at evals/mocks/<server>/<tool>.md, whose body is what the tool returns to Claude:

---
expect:
  title: string
  priority: [low, medium, high]
---
 
Created issue #4821: {{input.title}}

• {{input.<field>}} echoes call input into the result
• expect: validates the input Claude sent to the tool; a violation aborts the run with a score of 0 and records the reason, letting you assert that what your plugin asks the server to do is correct
• error: true returns a tool error, to test error-handling paths
• type: agent has the judge model answer as the server, and answers can be recorded and replayed (mock-recordings/ → mocks/.replay/), making CI runs reproducible with no further model calls

Tools without a mock file are unavailable to Claude, and real MCP servers don't start unless you pass --allow-real-servers or --mocks off.

When a case needs files or git state, use case.yaml: context.scaffold_script (runs outside the agent sandbox, so it only executes with --scaffold), context.history_file (resume a prior conversation), and context.add_dirs (read-only fixture directories).

Part 3: Into CI — Exit Codes Are the Contract

The recommended gating command:

claude plugin eval . \
  --trust-plugin \
  --json results.json \
  --threshold 0.8 \
  --model claude-sonnet-5 \
  --judge-model claude-haiku-4-5 \
  --no-publish \
  --max-cost-usd 20

The engineering reasoning behind these flags:

• --model and --judge-model are pinned separately: so a model rollout isn't mistaken for a plugin regression, and scores stay comparable over time.
• --max-cost-usd caps cost: it's checked before each run starts; once exhausted no new runs start, though runs in flight finish, so actual spend can slightly exceed the cap.
• --trust-plugin keeps the job from stalling on the first-run trust prompt.
• -j concurrency maxes out at 8; it shortens wall-clock time but shares your account's rate limit, so it doesn't raise throughput beyond it.

The exit codes are clean:

Two principles worth remembering:

1. Δ is reported but never changes the exit code. The gate checks whether the with-arm score meets the bar; baseline comparison is left to humans watching trends.
2. API errors aren't quietly folded into zeros. A run that ends on a usage or rate-limit error is scored 0 without grading, and the command exits non-zero even if every case's score is high enough. That prevents rate-limiting from producing false passes or false failures.

The JSON result is versioned (schemaVersion: 1); scripts should ignore fields they don't recognize, and trend charts should exclude documents with partial: true and runs with skippedPaidGraders, since their scores aren't comparable.

The advice for keeping CI cost predictable is concrete: for the fast suite that runs on every change, use only graders that don't call the judge model, and use --ablation none wherever you don't need Δ.

Design Ideas Worth Borrowing

Reading the two docs side by side, I distilled these:

1. Success criteria come before scoring. Without numbers like "0.85 F1, 99.5% non-toxic, 95% under 200 ms", even a beautiful grader has nothing to judge against.
2. Deterministic grading first, LLM grading as fallback. Only three of the six methods in the official examples use an LLM, and four of the six plugin-eval graders are free. LLMs are reserved for tone, privacy judgment, and other things you can't write rules for — with majority voting to reduce randomness.
3. Grade outcome and process in the same case. A correct answer doesn't mean your Skill produced it, and vice versa.
4. Use a with-vs-without baseline to separate the plugin's contribution from the model's own ability. A score alone says little; Δ says a lot.
5. Absorb non-determinism with repeated sampling. Three by default — not out of pedantry, but because one run carries too little information.
6. Keep evaluation config as files under version control. Cases, rubrics, and mocks are Markdown/YAML: reviewable, diffable, traceable.
7. Mark "failed, but not the subject's fault" separately. Rate limits, cost ceilings, credential failures, and partial results each get their own exit code or field instead of leaking into scores.
8. Constrain the evaluation itself. Cost caps, sandboxes, trust prompts, least-privilege tools: evaluation really calls models and really runs tools, so treat it like a production system.

If you're building your own evaluation platform, these eight make a decent checklist. When we re-tested a Jira Skill ourselves, we hit "an AI judge lets paraphrasing errors through" and "single-case conclusions aren't reliable". Our fixes — character-level script comparison, 100 cases × 3 samples — point the same direction as the official approach: never hand a model what can be judged deterministically, and use enough samples to expose rare but fatal errors.

Summary

1. Methodology layer: SMAR success criteria → a test set covering edge cases → grading methods chosen per dimension (deterministic first, LLM for subjective ones).
2. Tool layer: claude plugin eval makes cases, graders, and baselines a file-based, repeatable, isolated pipeline, with repeated sampling and with/without comparison built in.
3. Engineering layer: pinning the model under test and the judge, capping cost, and using explicit exit codes to separate "really failed" from "didn't finish running" are what make evaluation a trustworthy CI gate.

Together the two docs say one thing: evaluation isn't running a test and reading off a score; it's an engineering system that makes scores trustworthy, comparable, and traceable.

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage