How It Started: A Report That Looked Complete
Not long ago the team produced a "high-frequency skill model adaptation evaluation" that picked a best model for each of the Jira Skill's 8 task scenarios (D1–D8). It had tables, accuracy numbers, token and latency comparisons. It was neatly formatted and the conclusions were clear.
I was ready to use it as the basis for model selection. But the more I read it, the less comfortable I got.
The core data looked like this (40 judgments per model):
| Model | Accuracy | Avg Tokens | Avg Latency |
|---|---|---|---|
| deepseek-v4-pro | 40/40 | 1,539 | 32.7 s |
| claude-sonnet-4-6 | 39/40 | 1,791 | 24.7 s |
| deepseek-v4-flash | 39/40 | 2,160 | 33.2 s |
| glm-5.3 | 40/40 | 2,775 | 43.3 s |
| doubao-2.0-lite | 33/40 | 1,076 | 29.4 s |
| kimi-k2 | 32/40 | 1,092 | 30.1 s |
The top four models all sit at 97.5% or above, essentially tied. When every strong model scores near-perfect, it usually isn't because they're all great. It's because the task is too easy or the grading too lenient to tell them apart.
With that suspicion, I took the methodology apart.
What Was Wrong With It
Problem 1: One sample
Each scenario had a single case, run 5 times. That tests "is this one case stable?", not "does it still work on a different case?". Five runs are not five samples; they're five draws from the same sample.
Doing well on one bug ticket says nothing about a ticket with 50 attachments and special characters in the description.
Problem 2: AI judges its own answers
"Accuracy" was defined loosely: an AI decided whether the output was right. That's fine for semantic tasks, but risky for tasks like "reproduce the bug's content verbatim". A real example comes below.
Problem 3: Tested tasks didn't match real usage
In the bugfix workflow only 3 of the Skill's tasks are used often: fetch bug info and download attachments, add a comment, change the assignee. The best model across 8 tasks isn't necessarily the best for those 3, and a cheaper model might be good enough.
Problem 4: Missing candidates
The strongest model, sonnet-5-5, wasn't tested, and neither was ds-v4.1-flash, which is both strong and cheap. A "best" picked from an incomplete candidate set isn't very convincing.
None of these is outrageous alone, but stacked together the conclusion doesn't hold. So I decided to re-test.
Redesigning the Evaluation
Scope: answer one question at a time
I started with scenario 1, "fetch bug info and download attachments", the most frequently called and most failure-prone step. One question at a time keeps the conclusion from being diluted by other tasks.
Metric: scripts judge, not AI
The metric is information retrieval accuracy, with deliberately rigid scoring:
- A script diffs the Agent's output against ground truth character by character
- Exact match → 5 points; any difference → 0
- Ground truth has two parts:
- Text such as bug title and description must match exactly
- Attachment count, names and file sizes must match exactly
Scale: 100 cases × 3 samples
100 different bug tickets, each sampled 3 times per model. 100 cases cover diversity; 3 samples show stability on the same case.
Why not AI scoring
Consider a real failure. The cheapest model, doubao-seed-2.0-mini, makes subtle word-level mistakes when restating bug content in this task.
That raises an awkward question: if an AI judges whether the output matches ground truth, it will likely let such errors slide. If it makes mistakes when restating, why trust it not to make them when comparing?
AI scoring is good at semantic judgments (is it relevant, is it helpful) and weak at exact character-level checking. "The bug description is identical, character for character" is exactly the latter.
The rule is simple: if a script can judge it, don't ask an AI. A script not only decides right from wrong, it also pinpoints where and why it's wrong. The failure reasons in the screenshots below all come from the script, for example first diff at char 44, which says exactly where the text first diverges.
Results: The Top Score Isn't Necessarily the Best Choice
sonnet-5-5: highest score, but its errors are fatal
Across 100 cases × 3 samples, sonnet-5-5 made only 3 errors, the best of all models.
But those 3 errors fall into two patterns, both fatal for this task:
- Fetched the wrong Jira ticket. It got another ticket's information, and everything downstream is built on wrong input.
- The Agent couldn't finish the task. It stalled mid-flow with no result.
Here is the second pattern. The reply says it can't finish the requested output format. The ticket and all 50 attachments were actually fetched, but the final result block is incomplete, the script can't parse it, and the case scores 0:
The first pattern is the one to worry about: with a single case you'd almost never hit it. Rare fatal errors only show up in a large enough sample. That is exactly where the original report's method breaks down.
ds-v4.1-flash: best overall value
Per the pricing from IT, ds-v4.1-flash is cheaper than both v4-pro and v4-flash, making it the best value on paper. On this task its score is second only to sonnet-5-5.
Side by side in the run history (red box is ds-v4.1-flash; v2-5 is sonnet-5-5):
| Model | Accuracy score | Avg duration | Avg tokens |
|---|---|---|---|
| sonnet-5-5 | 4.93 | 44,414 ms | 174,485 |
| ds-v4.1-flash | 4.78 | 34,649 ms | 45,995 |
The score gap is 0.15, but ds-v4.1-flash is faster and uses about a quarter of the tokens of sonnet-5-5.
Its errors fall into two patterns:
| Error pattern | Count | Severity |
|---|---|---|
| Minor word-level restating errors | 10 | Acceptable |
| Did not follow the output format | 2 | Minor |
A typical restating error: the description is 188 characters in both, but differs from character 44 onward, meaning a few words were swapped for near-synonyms.
A format error is when the model didn't output the JSON verbatim and wrote a line like "step 2's stdout is the JSON below" instead, so the script can't parse it:
What matters is the nature of the errors. Most of ds-v4.1-flash's errors are small word-level deviations. Here the Agent fetches bug info for later analysis and fixing; a few words off doesn't change what the bug points to, so it isn't fatal.
By contrast, sonnet-5-5 errs less often, but both of its error types invalidate the whole flow.
Selection result
For "fetch bug info and download attachments", ds-v4.1-flash wins on success rate, cost and speed combined.
This is counter-intuitive: the top score isn't the best pick. Model selection isn't about the score, it's about "cost of an error × how often it happens + cost of the call".
Lessons From This Re-test
- Sample size decides how far you can trust a conclusion. Many runs of one case show stability, not generalization. Fatal but rare errors (like fetching the wrong ticket) need large samples to surface.
- Don't use AI to judge what a script can. An AI judge lets through the subtle mistakes it makes itself. Exact diffing is both accurate and explains the cause.
- Classify errors by pattern and severity. "3 errors" and "12 errors" aren't directly comparable; ask whether each kind invalidates the flow.
- Align selection with tasks actually used. The best model across 8 tasks isn't necessarily best for the 3 you use. The eval task mix should match production usage.
- Be willing to overturn conclusions. With more data and a stricter method, old conclusions should give way.
But the biggest takeaway from this re-test was something else: if you hand-roll this process every time, it won't last.
From a One-off Re-test to a Platform
This re-test meant configuring 100 cases, running 3 sampling rounds, comparing several models, tallying error patterns, and writing it all up. With ad-hoc scripts, every new model or every Skill change means starting over, and large-sample evaluation can't be sustained.
So I turned the whole process into the Skill Evaluation module of my AI workbench, and the Jira re-test above ran inside it. Below I walk through the full flow using a commit-message Skill as the example.
1. Eval plan: define what "good" means first
Each Skill gets an eval plan whose core is metrics and weights. Every metric uses one of two scoring modes:
- AI score: an AI scores 1–5 against a rubric, suited to semantic judgments like title quality or motivation
- Rule check: regex or other assertions, suited to precisely checkable things like format compliance
This matches the lesson above: use rules wherever a rule can decide, hand only the semantic judgments to AI, and write the rubric into the plan rather than going by feel.
2. Test cases: write by hand, batch import, or AI-generate
The cases page supports three sources: add manually, batch import, and AI auto-generate. Each case has an input, an expected result, and the metrics that apply to it.
The right side of each case summarizes its best and worst score across configurations, so it's easy to spot which cases discriminate well.
3. Runs: both version and model are variables
When triggering an eval you pick the Skill version and the model. The platform runs it in batch and computes each metric, the spread (σ), average duration and tokens. In run history each column is a run, colored by metric score, with the best value per metric marked.
The Head-to-head section below supports two kinds of comparison:
- Vertical (same model, different Skill versions): the platform labels it "Same condition: only the skill version differs", so there's one variable. It answers "did this Skill change actually help?"
- Horizontal (same version, different models): it answers "can this Skill run on a cheaper model?", which is the question this Jira re-test set out to answer
4. AI-written report: conclusion, recommendation, evidence
After runs complete, you pick some runs and let AI generate a comparison report with three layers:
- Conclusion: with effect size and confidence interval
- Recommendation: default choice, conditional choices, and next steps to firm up the conclusion
- Evidence: overview, stability, per-group performance, failure analysis, efficiency and tokens, limitations; each section expands to the analysis and underlying tables
Reports can be downloaded as Markdown or a data pack for archiving or further analysis.
One detail: this sample report states its own limits in the "Limitations" section, such as response contents being omitted and too few cases, so conclusions rest on scores alone. A good eval report states its boundaries instead of just handing you a pretty number.
💬 Want to try the Skill evaluation platform, or talk it over? Leave a comment below. I'd love to hear what problems you've run into evaluating Skills, or how you'd want to use it.
Wrap-up
Looking back, the whole thing is three steps:
- Question it: near-perfect scores for everyone, a single case, and AI grading its own answers are all red flags in the method
- Redo it: narrow the scope, score with scripts, enlarge the sample, match real tasks
- Make it durable: turn the method into a platform so the next evaluation is just "add cases, pick models, hit run"
Evaluation shouldn't be a one-off verdict. It should be a repeatable engineering step that keeps pace as Skills and models evolve.
Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.
Find more useful knowledge and interesting products on my Homepage