Open Source Project #197: skill-up — Alibaba's Agent Skills Evaluation and Evolution Tool, Closed-Loop Testing with Auto-Repair

Introduction

"Evaluation makes Skill quality measurable. Evolution turns those results into the next improvement."

This is article #197 in the "One Open Source Project a Day" series. Today's project is skill-up — Alibaba's open-source evaluation and evolution tool for Agent Skills, 655 Stars, written in Go, Apache-2.0.

The core problem skill-up solves: you've written an Agent Skill (SKILL.md) — how do you know it actually works? If it doesn't, where exactly does it fail? And after you fix it, how do you prevent it from regressing?

skill-up closes the entire loop: skill-up CLI runs declarative test cases; skill-upper (a built-in Agent Skill) reads failure reports, automatically repairs the Skill or expands test cases, reruns the evaluation, and keeps iterating — all driven through conversation.

What You'll Learn

• Evaluation side: the declarative eval.yaml + cases/*.yaml config structure
• Three judging strategies: rule_based, script, agent_judge
• Evolution side: how skill-upper drives an automatic repair loop through conversation
• Supported Agent engines: Claude Code, Codex, Qoder CLI, Qwen Code
• GitHub Action integration: cross-engine Skill evaluation in CI
• Compatibility with Anthropic's evals.json format

Prerequisites

• Familiarity with Agent Skills basics (SKILL.md format)
• Basic command-line experience
• Some CI/CD background helps for the GitHub Actions section

Project Background

Overview

skill-up is the official evaluation toolchain for the Agent Skills standard. Agent Skills let coding agents (Claude Code, Codex, etc.) load capabilities on demand — described in Markdown, the agent reads the SKILL.md and knows what to do. skill-up solves the quality control and evaluation problem in this ecosystem.

The official evaluation guide describes the right workflow: write realistic cases, run with and without the Skill, grade outputs, aggregate results, and iterate. skill-up turns that workflow into a reusable CLI.

Author / Team

• Organization: Alibaba
• Primary language: Go 1.25+
• License: Apache-2.0
• Documentation: alibaba.github.io/skill-up

Project Stats

• ⭐ GitHub Stars: 655+
• 🍴 Forks: 44+
• 📄 License: Apache-2.0
• 📅 Created: 2026-05-09

Two Core Capabilities

skill-up is designed around two complementary capabilities:

Evaluation: Makes Skill quality measurable and repeatable. Declarative YAML configuration, runs test cases across multiple Agent engines, uses rule/script/agent judging strategies, generates structured reports locally or in CI.

Evolution: Turns evaluation results into the next improvement. skill-upper reads failure reports, automatically repairs or expands the eval suite, reruns skill-up, and keeps iterating — all through conversation.

Write SKILL.md
    ↓
skill-upper generates eval.yaml + cases/*.yaml
    ↓
skill-up run → result.json
    ↓
skill-upper analyzes failures → fixes Skill or fixes eval cases
    ↓
skill-up run (again) → all important behaviors pass

Installation

Install the skill-up CLI

curl -fsSL https://raw.githubusercontent.com/alibaba/skill-up/main/install.sh | bash

Install skill-upper (recommended entry point)

# Codex, global install
npx skills add https://github.com/alibaba/skill-up/tree/main/skills/skill-upper -g -a codex -y
 
# Claude Code, global install
npx skills add https://github.com/alibaba/skill-up/tree/main/skills/skill-upper -g -a claude-code -y

skill-upper checks for the skill-up CLI at runtime; if it's not installed, it guides the agent through installation automatically.

Declarative Evaluation Configuration

Directory Layout

my-skill/
  SKILL.md              ← the Skill being evaluated
  evals/
    eval.yaml           ← environment, engine, global settings
    cases/
      case-001.yaml     ← individual test case
      case-002.yaml

eval.yaml Structure

schema_version: v1alpha1
kind: EvalConfig
 
environment:
  type: local            # local sandbox
 
engine:
  type: claude_code      # use Claude Code as the agent engine
  model: claude-sonnet-5
 
skill:
  path: ../SKILL.md      # the Skill being evaluated
 
cases:
  - cases/               # all yaml files under cases/

A Test Case (case-001.yaml)

schema_version: v1alpha1
kind: EvalCase
 
id: case-001
description: "Basic behavior verification"
 
input:
  role: user
  content: "Help me complete this task..."
 
judge:
  type: rule_based
  rules:
    - type: contains
      value: "expected output keyword"
    - type: not_contains
      value: "content that should not appear"

Three Judging Strategies

rule_based

Simplest and fastest — ideal for cases with clear, deterministic output expectations:

judge:
  type: rule_based
  rules:
    - type: contains
      value: "function"
    - type: regex
      pattern: "def\\s+\\w+\\("
    - type: not_contains
      value: "error"

script

Custom logic for complex judgments — runs an arbitrary script against the output:

judge:
  type: script
  script: |
    #!/bin/bash
    # $OUTPUT contains the agent's output
    echo "$OUTPUT" | grep -q "expected_pattern"
    exit $?

agent_judge

Uses an LLM as judge — for subjective or nuanced evaluation criteria that are hard to express as rules:

judge:
  type: agent_judge
  criteria: |
    Evaluation criteria:
    1. Did the output correctly complete the task?
    2. Are there any security concerns?
    3. Is the code quality acceptable?
  pass_threshold: 0.8   # score threshold from 0 to 1

Supported Agent Engines

Custom engines connect via local transport, suitable for internal agents or tools not in the built-in list. See docs/design/custom-engine.md.

Evolution: Driving the Repair Loop with skill-upper

This is skill-up's most distinctive feature. skill-upper is itself an Agent Skill — it runs inside Claude Code or Codex, and turns the evaluate-and-fix cycle into a conversational workflow.

Step 1: Create and Run the First Evals

In a project containing your SKILL.md, ask your agent:

Use skill-upper to evaluate this Skill.
Read SKILL.md, identify its most important behaviors, create realistic eval
cases with appropriate judges, validate the configuration, and run skill-up.
Summarize the results and the highest-impact failures.

skill-upper will:

1. Read SKILL.md and analyze its key behaviors
2. Generate evals/eval.yaml and cases/*.yaml
3. Run skill-up validate to verify the configuration
4. Run skill-up run to execute the evaluation
5. Report results and highest-priority failures

The generated workspace looks like:

my-skill-workspace/
  iteration-1/
    result.json       ← detailed results
    grading.json      ← Anthropic-compatible format
    benchmark.md      ← human-readable summary

Step 2: Fix, Regress, and Iterate

Continue in the same conversation:

Review the latest skill-up results. For each failure, determine whether the
Skill or the eval is wrong. Fix SKILL.md and supporting files, or repair the
eval case and judge as appropriate. Add regression cases for the bugs you
found, rerun skill-up, and continue until the important behaviors pass.

skill-upper judges each failure: is the Skill itself broken, or is the test case wrong? It fixes SKILL.md for the former, repairs cases/*.yaml for the latter, adds regression cases, and reruns.

This is a genuine eval-to-evolution closed loop: reports → fixes → regression cases → rerun — every iteration makes both the Skill and its eval suite stronger.

Report Output Formats

skill-up run generates multiple report formats for different use cases:

Regenerate reports from an existing result without re-running:

skill-up report result.json

GitHub Action CI Integration

skill-up ships a GitHub Action at its repository root, triggering automatic Skill evaluation on every PR — with cross-engine comparison in a single step:

# .github/workflows/skill-eval.yml
name: Skill Eval
on:
  pull_request:
    paths: ['skills/**', 'evals/**', '**/SKILL.md']
jobs:
  eval:
    runs-on: ubuntu-latest   # Docker container action — Linux only
    steps:
      - uses: actions/checkout@v4
      - uses: alibaba/skill-up@main
        with:
          engine: claude_code          # or codex / qodercli / qwen_code
          api-key: ${{ secrets.ANTHROPIC_API_KEY }}
          base-url: https://api.anthropic.com
          skill-target: evals/eval.yaml

The runner image pre-installs skill-up CLI and the three engine CLIs, so a run is "pull image, eval" — no additional setup required.

CLI Reference

Configuration Precedence

skill-up supports four layers of configuration, from lowest to highest precedence:

embed (empty built-in defaults)
    < user (~/.config/skill-up/config.yaml)
    < project ($PWD/.skill-up.yaml)
    < explicit (--config command-line flag)

skill-up init           # write to ~/.config/skill-up/config.yaml
skill-up init --local   # write to $PWD/.skill-up.yaml
skill-up init --print   # print the config template to stdout

Anthropic evals.json Compatibility

If you already have Anthropic-format eval files, import them to skill-up's YAML layout in one step:

skill-up import ./evals/evals.json --output ./evals

skill-up also outputs grading.json (Anthropic-compatible), making it easy to integrate with the Anthropic eval toolchain without a full migration.

Resources

Official Links

• 🌟 GitHub: alibaba/skill-up
• 📖 User Manual: alibaba.github.io/skill-up
• 📖 用户手册（中文）: alibaba.github.io/skill-up/zh
• 🌐 Agent Skills standard: agentskills.io

Summary

skill-up addresses a real engineering problem in the Agent Skills ecosystem: writing a Skill is easy; verifying that its behaviors match expectations is hard; and ensuring it doesn't regress across iterations is harder still.

Three design decisions worth noting:

Evaluation and evolution are separate but connected: The skill-up CLI only does evaluation; skill-upper Agent Skill only does diagnosis and repair. Clear separation of concerns, but connected through file formats (result.json), forming a complete closed loop. This is cleaner than cramming everything into a single tool.

skill-upper is itself a Skill: It installs via npx skills add, runs inside Claude Code/Codex, reads result.json, and drives the repair loop through conversation. The "use a Skill to evaluate and fix a Skill" design makes the entire toolchain self-consistent.

Graduated judging strategies: rule-based (zero cost) → script (low cost) → agent judge (high accuracy). Choose the judging method based on the case's complexity rather than blanket-using LLM evaluation for everything.

Anthropic compatibility is an explicit design goal: Importing evals.json and outputting grading.json shows the project was designed for interoperability with existing toolchains, not forcing users to migrate everything at once.

If you're building Agent Skills, skill-up provides a complete path from one-off manual testing to continuous CI evaluation, with skill-upper's automatic repair loop making the process nearly fully automated.

Explore PrimeSkills — a curated marketplace of AI agents and skills, each validated against real enterprise workflows. No hype, just what actually works.

Visit my personal site for more insights and interesting products.