Skill Series (06): Skill Governance — Routing Accuracy 38%, Compression Saves 76%, Monthly Cost $0.67

When Skills Need Governance

Three Skills is manageable by memory. Thirty Skills in an enterprise creates three problems simultaneously:

1. Discovery: given a user request, which Skill runs?
2. Cost: which Skills are expensive, and where does the money go?
3. Quality drift: when someone edits a Skill prompt, how do you know nothing regressed?

This article measures all three.

Demo Design

6-Skill registry:

Three experiments:

1. Embedding routing: cosine similarity on descriptions, 8 test queries, measure accuracy
2. Prompt compression: verbose (418 tokens) vs compressed (99 tokens), LLM-as-Judge quality scoring
3. Cost report: real invocations on 4 Skills, cost-per-call and monthly projection

Run Results

Part 1: Embedding Routing

  Query                                     Routed to              Score  OK?
  ───────────────────────────────────────── ─────────────────────  ─────  ───
  Write a deep-dive article about Kubern..  rnd-technical-writer   0.448  ✓
  Notion competitor analysis (Chinese)      competitor-analyzer    0.311  ✓
  Traceback: AttributeError: 'NoneType'..  sql-query-builder      0.473  ✗
  Meeting notes (Chinese)                  meeting-summarizer     0.463  ✓
  Get all orders placed in the last 7 d..  meeting-summarizer     0.586  ✗
  Write a product description for AI t..   rnd-technical-writer   0.610  ✗
  Python 3.12 performance analysis (CN)    sql-query-builder      0.401  ✗
  List all users who haven't logged in..   bug-root-cause         0.493  ✗
 
  Routing accuracy: 3/8 = 38%

Part 2: Prompt Compression

  Verbose: ~418 tokens  |  Compressed: ~99 tokens  |  Reduction: 76%
 
  Python context managers:  4.00 → 4.00  delta: +0.00 (no change)
  Redis pub/sub:            4.00 → 3.00  delta: -1.00 (notable)
 
  Avg quality delta: -0.50

Part 3: Cost Report

  Skill                    avg_in   avg_out  $/100calls  p50
  ───────────────────────  ──────   ──────── ──────────  ────
  rnd-technical-writer      116t     1248t    $0.1365    48.8s
  competitor-analyzer        37t      586t    $0.0623    27.9s
  sql-query-builder          46t      589t    $0.0635    32.6s
  meeting-summarizer         53t       99t    $0.0152     5.1s
 
  Monthly projection:
    rnd-technical-writer   200/mo  →  $0.27
    competitor-analyzer     50/mo  →  $0.03
    meeting-summarizer     300/mo  →  $0.05
    sql-query-builder      500/mo  →  $0.32
    Total                         →  $0.67/mo

Three Findings

Finding 1: 38% Routing Accuracy — Skill Descriptions Are Too Similar

Five misroutes. Two are instructive:

Misroute 1: "Get all orders placed in the last 7 days" → meeting-summarizer (expected sql-query-builder)

meeting-summarizer scored 0.586 similarity; sql-query-builder scored lower. Both descriptions mention structured outputs with specific fields — in embedding space, the distance between "summarize meeting transcripts and extract action items" and a data retrieval query is small enough to confuse.

Misroute 2: "Write a product description" → rnd-technical-writer (expected marketing-copywriter)

Both are writing Skills. Both descriptions contain "write." The embedding model can't distinguish them at description granularity.

Skills in the same domain cluster in embedding space. Cosine similarity on their descriptions doesn't reliably separate them. Three fixes:

1. Add exclusions to descriptions (same principle as design patterns article):

   marketing-copywriter:
   "Write marketing copy and product descriptions.
   NOT: technical tutorials or factual explainers."

2. Two-stage routing: domain filter first (content/strategy/engineering), then embedding within domain

3. LLM router: embedding generates a candidate shortlist; LLM picks the winner — costs one extra LLM call, gains significant accuracy

Finding 2: 76% Compression, But Redis Article Quality Dropped 1 Point

Python context managers: 4.0 → 4.0. No degradation.

Redis pub/sub: 4.0 → 3.0. One full point drop.

Same two prompts, same model, different topics. The verbose prompt contains substantial guidance: "explain not just HOW but also WHY," "supportive of readers at different skill levels," explicit quality criteria. The compressed prompt retains only structural requirements (frontmatter, H2, code block, word count).

For a well-known topic (context managers), the model has enough training signal to produce depth on its own. For a narrower topic (Redis pub/sub internals), the quality guidance in the verbose prompt makes a measurable difference.

For a well-known topic (context managers), the model's training signal is strong enough to produce depth without guidance. For a narrower topic (Redis pub/sub), the quality guidance in the verbose prompt matters. Before promoting a compressed prompt:

# Run A/B against your evaluation set first
quality_delta = compressed_score - verbose_score
if abs(quality_delta) < 0.1:
    promote_compressed()      # safe
else:
    keep_verbose()            # wait for a better compression strategy

Finding 3: Output Tokens Drive Cost, Not Prompt Length

rnd-technical-writer costs $0.1365 per 100 calls. meeting-summarizer costs$0.0152. A 9x difference.

The prompt lengths are comparable (116t vs 53t). The difference is entirely in output: 1248 tokens vs 99 tokens.

Cutting the prompt from 418 to 99 tokens saves ~$0.0011 per call. Adding `Maximum length: 400 words` and cutting output from 1248 to 600 tokens saves ~$0.06 per call. That's 54x more impactful. Two ways to control output length:

• Hard word count cap in the Skill prompt: Maximum length: 400 words
• Tiered output modes: brief summary vs detailed analysis, let the workflow choose

Skill Registry Design

The routing accuracy problem starts in the registry. A well-designed entry:

skills:
  - id: sql-query-builder
    description: |
      Generate SQL queries from natural language data retrieval descriptions.
      Trigger: user describes a database query need in plain language.
      Keywords: query, SQL, select, filter, join, aggregate, table.
      NOT for: code debugging, bug reports, general programming questions.
    domain: engineering
    subdomain: data
    version: "1.0.0"
    status: active
    metrics:
      monthly_calls: 500
      avg_quality_score: 4.1
      cost_per_call_usd: 0.000635

The NOT for line is what prevents the embedding confusion. Without it, the router has no signal to separate the Skill from neighboring ones in the same semantic neighborhood.

Three-Level Governance Roadmap

Level 1 — Individual:
  □ Skill inventory (YAML or spreadsheet)
  □ Each Skill has a version number
  □ Basic evaluation records (Article 01 framework)
 
Level 2 — Team sharing:
  □ Skill Registry with domain + exclusion clauses
  □ Access control: public / team / personal
  □ Changes require review + evaluation delta check
  □ Quality monitoring dashboard (Article 04 L2/L3 metrics)
 
Level 3 — Enterprise governance:
  □ Two-stage routing (domain filter + embedding)
  □ Compression A/B testing before promotion
  □ Cost breakdown by Skill and team
  □ Audit log + compliance checklist

Design Checklist

Skill Registry

• Description includes trigger keywords
• Description has NOT exclusion clause (prevents embedding confusion)
• Domain field enables two-stage routing

Prompt compression

• A/B test against evaluation set before replacing verbose version
• Promote only when quality delta < 0.1

Cost monitoring

• Track input and output tokens separately
• Optimize output length before prompt length
• High-output Skills have a max word count instruction in the prompt

Summary

1. 38% routing accuracy: Skills in the same domain are too semantically similar for pure embedding routing; adding NOT exclusion clauses and using domain-first two-stage routing fixes this — not switching embedding models
2. Compression is not uniformly safe: 76% token savings left a simple topic unchanged, but dropped a complex topic by 1 point; A/B test before promoting, not after
3. Output tokens dominate cost: rnd-technical-writer costs 9x more than meeting-summarizer due to 1248 vs 99 output tokens; adding a word count cap to the prompt is 54x more impactful than reducing the prompt itself

References

• Full demo code: skill-06-governance
• Article 01: Skill Evaluation Framework
• Article 04: Skill Metrics

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage