Making AI Skills Better Over Time: Evaluation, Tuning, and Self-Evolution System Design

Preface

Once an Agent platform accumulates a significant number of Skills and Workflows, the next critical question is: How do we make this system continuously improve with use, rather than being frozen at the capability level it had when initially deployed?

Many teams' approach is to have another LLM "review" the quality of new Skills, or rely on developers' "gut feeling" to decide whether to publish.

Two research papers published in May 2026 used experimental data to thoroughly debunk the reliability of this approach. This article is an interpretation of their core findings and a guide to implementing their methodology in enterprise AI platforms.

LLM Self-Evaluation is Unreliable: A Disturbing Discovery

SkillLens, a paper jointly published by Microsoft Research and Fudan University studying the full lifecycle of agent skills, contains one of its most important findings:

LLM comparative judgments without explicit evaluation criteria have an accuracy of only 46.4% — equivalent to random guessing.

This is concerning enough. What's more alarming: in cases where the actual performance gap between two Skills is larger, LLMs are more likely to choose incorrectly — in the group with the largest performance delta, the probability of the LLM choosing the worse Skill reached 84.2%.

Why does this happen? LLMs tend to select Skills that "look good": clear wording, complete structure, logical flow. But "looking good" has no correlation with "executing well."

Direct impact: Many teams currently rely on developer gut feeling to publish Skills, or use another LLM to "review" Skill quality. These two approaches are equivalent in terms of evaluation results — both approximately equal to random guessing.

The Three Dimensions That Actually Determine Skill Quality

SkillLens ran large-scale experiments (5 domains, 6 target models, 5 extraction models) and found only three dimensions that actually predict Skill downstream performance:

Other seemingly important dimensions — clarity, completeness, conciseness, format (ordered lists/unordered lists/prose/checklists) — had no statistically significant impact on performance in experiments (p > 0.34).

This means: a Skill with neat formatting and clear wording that only provides general advice performs far worse than a Skill with rough formatting but containing specific failure path analysis.

The Three Phases of Skill Lifecycle

SkillLens formalizes the complete Skill lifecycle into three phases, each with independent key findings:

Phase 1: Experience Generation
  Target model executes training tasks → produces (task, execution trajectory, result) triples
  Key: Collect both success and failure trajectories
  Finding: All-failure trajectories are the worst starting point; optimal success/failure ratio varies by domain
 
Phase 2: Skill Extraction
  Extraction model analyzes trajectory pool → distills transferable patterns → generates structured Skill docs
  Finding: Extraction capability is orthogonal to task execution capability — the strongest execution model
           isn't necessarily the best extraction model
 
Phase 3: Skill Consumption
  Target model uses Skill at inference time → measure performance improvement
  Finding: The same Skill shows dramatically different effects across different target models

SkillOpt: Treating Skills as "Trainable External State"

SkillOpt (Microsoft, May 2026) proposes a framework that uses the logic of deep learning training loops to design Skill tuning:

Seven Core Components

Forward Pass (Rollout Evidence): Execute a batch of training tasks with the current Skill, recording complete execution trajectories — tool calls, intermediate outputs, final results, validation feedback.

Backward Pass (Minibatch Reflection): The optimizer model analyzes execution trajectories, grouping failure and success trajectories to extract specific edit suggestions. The reason for using mini-batches rather than single trajectories: single trajectories only produce case-specific fixes, while mini-batches expose reusable procedural error patterns.

Bounded Text Updates (Bounded Edit Budget): The number of edits allowed per step is controlled by a "learning rate," supporting constant, linear, cosine, and autonomous scheduling. Default cosine scheduling: larger edits initially (exploration), smaller edits later (convergent refinement). Unbounded rewrites erase effective rules or introduce conflicting instructions.

Validation Gate + Rejection Buffer: After each edit, score on the held-out set — must strictly improve over the current version (tie is rejected). Rejected edits enter a "rejection buffer," which future reflection calls can see, preventing the optimizer from repeatedly trying already-proven-ineffective directions.

Epoch-wise Slow Update: Fast updates learn within-batch patterns; slow updates learn across epochs. At the end of each epoch, compare the current Skill and the previous epoch's Skill on the same tasks, writing longitudinal patterns (which types of edits work, which consistently fail) into the Skill's protected fields.

Experimental Results

Across 52 evaluation units spanning 6 domains, 7 target models, and 3 execution environments (direct conversation, Codex, Claude Code), SkillOpt achieved best or tied-best results on all units, with zero failures.

Representative results (GPT-5.5 direct conversation):

• SearchQA: 77.7 → 87.3 (+9.6)
• SpreadsheetBench: 41.8 → 80.7 (+38.9)
• OfficeQA: 33.1 → 72.1 (+39.0)
• ALFWorld: 83.6 → 95.5 (+11.9)

Cross-environment transfer: Skills trained on SpreadsheetBench in the Codex environment transferred to Claude Code with an absolute improvement of +59.7 (22.1 → 81.8), exceeding the results from training directly in the Claude Code environment (80.4). This shows that Skills encode task procedural patterns rather than environment-specific preferences.

Final Skill documents are compact (379–1995 tokens, median ~920 tokens), with training costs paid once and zero additional inference-time overhead.

Mapping to Enterprise Agent Platforms

The above methodology is validated on academic benchmarks. Adapting it to enterprise platforms requires the following adjustments.

Building an Evaluation System

Step 1: Build a test set for each Skill, including:

• Typical success cases: representative standard inputs and expected outputs
• Edge failure cases: real failure scenarios extracted from historical execution logs
• Regression cases: issues that occurred before and were fixed, preventing reintroduction

Without a reliable test set, all subsequent optimization is building on sand.

Step 2: Establish evaluation metrics, prioritizing deterministic signals:

Step 3: Audit existing Skill library using the three-dimensional standard — failure paths, executable specificity, dangerous operation blacklist. Skills that don't satisfy these three criteria, regardless of how they look on other dimensions, will most likely perform poorly in actual execution.

Tuning Process

Phase 1: Reactive Repair (based on explicit failure signals)

1. Collect failure trajectories (failing inputs + incorrect outputs + expected outputs)
2. Organize mini-batch reflection: multiple failure cases form a batch, analyzing common failure patterns
3. Generate improvement suggestions, limiting edit budget (only change 1-2 rules per round)
4. Validate on test set, strictly requiring score improvement (tie not accepted)
5. Publish new version after human approval

Critical principle: The agent that edits the Skill and the agent that scores it must be completely isolated — they cannot be the same session.

Phase 2: Proactive Pattern Discovery (based on pattern aggregation, suitable after accumulating sufficient execution data)

• Aggregate failure records for the same Skill across different tasks, looking for systemic patterns
• Identify Skills that "consistently fail on a certain type of input," and add test cases for that input type
• Cross-Skill analysis: does Skill A's output quality degradation cause increased failure rates in downstream Skill B?

Four Levels Toward Autonomous Evolution

Safety Rails: Inherent Risks of Automatic Optimization Systems

Automatic optimization systems have several failure modes that require designed guardrails:

Metric Gaming: Optimizing test set metrics while quietly degrading capabilities not covered by the test set. Counter-measure: test sets must continuously be supplemented from real failure cases — they can't be static.

Distribution Drift: Skill performs well on the test set but degrades on edge cases from real-world inputs. Counter-measure: at Level 3 and above, must simultaneously monitor real online quality metrics.

Cascading Degradation: Skill A's minor degradation causes Skill B's failure rate to rise, but each Skill individually meets benchmarks. Counter-measure: must simultaneously monitor Workflow end-to-end quality in addition to Skill-level monitoring.

Edit Velocity Limits: The same Skill shouldn't allow consecutive edits in a short period. Each edit must go through an observation window (at least one normal usage cycle) before the next edit.

Mandatory Version Records and Rollback: All Skill changes must have version records, supporting one-click rollback to any historical version. This is the safety net for the entire autonomous evolution system.

Implementation Roadmap

Prerequisites (must be built first)

• Skill test dataset construction (at least 20 representative test cases per Skill)
• Skill version management and rollback capability
• Structured collection of execution logs (inputs, outputs, execution time, intermediate states)
• Three-dimensional assessment audit (comprehensive health check of existing Skill library)

Phase 1: Evaluation System

• Build test datasets for P0/P1 Skills
• Build automated evaluation pipeline based on deterministic signals
• Build Skill quality dashboard with real-time quality view
• Establish baseline: run all existing Skills through evaluation to know the current quality level

Phase 2: Tuning Process

• Build failure case collection and archiving system
• Implement Level 1 automatic monitoring + human-triggered tuning
• Introduce SkillOpt methodology, implement Level 2 automatic optimization + human approval

Phase 3: Autonomous Evolution

• Level 3: autonomous optimization + human spot-check, pilot on low-risk Skills first
• Execution log analysis, semi-automated Skill extraction opportunity identification
• Cross-Agent Skill knowledge sharing mechanism

Reference Resources

Visit PrimeSkills — a curated AI Agent and skills marketplace where all content is validated through real enterprise workflows. No hype, just what actually works.

For more practical knowledge and interesting products, visit my personal homepage