Open Source Project #176: Better Harness — A Five-Dimension Workflow Evaluator for AI Coding Agents That Reviews the Loop, Not the Diff

Introduction

"Your AI coding agent generates code fast. Your workflow is the bottleneck."

This is article #176 in the "One Open Source Project a Day" series. Today's project is Better Harness — QoderAI's open-source evaluation tool that analyzes the workflow surrounding an AI coding agent, not just the code it produces.

Most evaluations of AI coding agents focus on output quality: test pass rates, defect density, functional correctness. Better Harness takes a different position: agents fail not because the model is weak, but because the surrounding workflow has gaps. Fuzzy goals, no reusable execution paths, unvalidated changes, bypassed quality checks, lessons that evaporate after each session — these problems don't show up in diffs. They only appear when you audit the workflow itself.

Better Harness collects project and session evidence, evaluates workflow health across five dimensions, and outputs prioritized findings — each with a scoped, actionable repair plan.

1,500 Stars. MIT license. Supports Claude Code, Codex, GitHub Copilot, Cursor, and Qwen Code.

What You'll Learn

• Better Harness's core model: feedforward guides and feedback sensors
• What each of the five dimensions evaluates and what evidence counts
• The architecture: three independent evidence agents running in parallel
• Report structure: findings, repair plans, historical trends
• Why it's "deliberately honest": never inferring usage from configuration presence
• Installation and usage in Claude Code and Codex

Prerequisites

• Experience using Claude Code, Codex, Cursor, or a similar AI coding tool
• Familiarity with AGENTS.md, Hooks, and Skills helps but isn't required
• Basic awareness of software quality practices (CI/CD, testing, code review)

Project Background

The Problem: Agents Move Fast, Workflows Stay Weak

AI coding agents introduce a new failure mode: speed. An agent can finish in minutes what previously took hours — but that speed also skips the slow steps that were actually valuable. Careful goal understanding, working within proven paths, validating changes, passing human review.

Better Harness identifies five common workflow failure patterns:

These five problems are typically invisible in code diffs. The code passes review. The workflow problems persist and repeat on the next task.

QoderAI and Qoder

Better Harness is built by QoderAI, who also makes a desktop AI coding agent called Qoder. Better Harness is natively integrated in Qoder; the open-source version runs as a plugin in other major agents.

Project Stats

• ⭐ GitHub Stars: 1,500+
• 🍴 Forks: 123+
• 📄 License: MIT
• Runtime: Node.js 22.20.0–25.0.0

Core Concept: Feedforward + Feedback Dual Signals

Better Harness's evaluation model rests on one framework: effective agent workflows need two signal types working together.

Before work starts                After / during work
──────────────────                ──────────────────
Feedforward guides                Feedback sensors
 
AGENTS.md                         Linters
Spec documents                    Test suites
Skills (reusable steps)           Hooks (event-triggered)
Acceptance criteria               Evaluation agents
                                  Diagnostics

Feedforward guides steer the agent before it acts — AGENTS.md establishes rules and goals, specs define task scope, Skills provide proven execution paths, acceptance criteria define what "done" looks like.

Feedback sensors observe results after the agent acts — linters check conventions, test suites validate functionality, Hooks capture signals on trigger events, evaluation agents score output quality.

The core indicator of a healthy workflow: both sides are operating, and their results leave evidence.

The Five Dimensions

Dimension 1: Task Understanding

Core question: Does the agent know what the goal is and what "done" looks like?

What gets evaluated:

• Whether AGENTS.md exists and contains effective rules and goal definitions
• Whether spec documents define task scope
• Whether explicit acceptance criteria tell the agent when to stop
• Whether the agent can identify the project starting point and appropriate change granularity

Common failure: Vague goal descriptions ("improve login flow" vs. "add specific error messages for each error state") leave the agent without a clear stopping point, producing over-engineering or repeated course corrections.

Dimension 2: Controlled Execution

Core question: Is the agent working on supported, repeatable paths?

What gets evaluated:

• Skills configuration: whether reusable SDLC execution steps exist
• MCP tool availability and boundary settings
• Sandbox boundaries: whether agent permissions are appropriately scoped
• Whether the agent uses proven paths rather than improvising each time

Common failure: Reinventing the execution process for each task; agent with excessive permissions making changes beyond task scope; no reusable steps, so similar tasks produce inconsistent quality.

Dimension 3: Change Validation

Core question: Is there evidence the change actually works?

What gets evaluated:

• Whether tests actually ran after the change (not just "tests exist")
• Whether lint checks actually ran after the change
• Whether Hooks captured validation signals
• Whether re-validation happened after validation failures
• Whether diagnostic tools were actually used

Key distinction: Better Harness separates "tests are configured" from "tests were executed." A project can have a complete test suite, but without evidence the agent ran tests after changes, this dimension doesn't score for that.

Dimension 4: Reliable Delivery

Core question: Did AI speed bypass quality gates?

What gets evaluated:

• Whether task acceptance has verifiable evidence (not just "code is done")
• Whether high-risk operations have human approval paths
• Whether rollback mechanisms exist
• Whether CI/CD pipelines are part of the agent's workflow
• Whether human review actually happened

Core concern: An agent can make extensive modifications without anyone noticing. Reliable Delivery evaluates what validation gates those modifications passed before delivery.

Dimension 5: Learning Capture

Core question: Do lessons from this task improve the next one?

What gets evaluated:

• Whether recurring issues get distilled into reusable Rules or Skills
• Whether Loop Discovery is working (pattern recognition generating suggestions)
• Whether the Memory system is in use
• Whether similar tasks reuse existing experience or start from scratch each time

One signal: Better Harness flags "long sessions" (over 45 minutes) for human review — these often indicate the agent spent significant effort on exploration that should be captured to avoid repetition.

Analysis Architecture: Three Independent Evidence Agents

Better Harness doesn't use a single agent for all analysis. Three independent read-only sub-agents collect different evidence categories in parallel, and a lead agent synthesizes the results only after independent collection completes.

Three independent sub-agents (parallel)
├── Agent 1: Customization asset analysis
│       → Completeness of Rules, Skills, Hooks, and configs
│
├── Agent 2: Real task session analysis
│       → What the agent actually did and how it performed
│
└── Agent 3: Project engineering foundation analysis
        → Whether project structure supports the agent workflow
 
        ↓ (after independent collection)
 
Lead Agent: Unified analysis + report generation

Why keep them independent: Running the three sub-agents separately prevents one category's conclusions from skewing another's interpretation. If Agent 1 finds complete Skills configuration, that shouldn't influence how Agent 2 reads the actual session records — Agent 2 looks at execution evidence only.

Missing evidence handling: Unobserved behavior is never inferred. No test execution records → Change Validation is unknown, not assumed based on the presence of test files in the project.

Report Structure

Running the analysis produces three files:

• report.html: Self-contained visual report (open directly in a browser)
• report.md: Markdown format for version control and team sharing
• findings.json: Structured data for programmatic processing

What the Report Contains

Five-dimension overview: Scored bar chart per dimension + count of related findings

Scope snapshot: Current configured asset inventory — Rules count, Skills count, custom Agents, MCP tools, Memories, Hooks

Prioritized findings: Each finding includes:

• Priority (High / Medium / Low)
• Owning dimension
• Cause: the specific gap in current configuration
• Expected Output: what a successful fix achieves
• Fix instructions: an editable pre-filled prompt starting with /harness

Session observations: Representative patterns extracted from analyzed sessions; sessions over 45 minutes flagged separately

Historical trend: Results compared across multiple runs, showing dimension-level change over time

Deliberately Conservative Scoring

Better Harness has an explicit scoring constraint:

"Configured assets can establish that a mechanism exists, but only linked task evidence can establish that it was used."

A project with complete Skills configuration doesn't get full marks on Controlled Execution without evidence of actual usage. Passing a current check proves the intervention was exercised; only a comparable later result can prove the loop improved. The history view shows recorded trends, not causal proof of improvement.

Installation and Usage

Claude Code

/plugin marketplace add QoderAI/better-harness

Other Platforms

Run Analysis

After installation, in any supported agent:

/better-harness analyze this project's AI coding workflow and generate an evidence-backed report

Outputs a self-contained report.html + report.md + findings.json.

The Repair Workflow

Better Harness never modifies anything directly — it identifies gaps and provides the fix starting point:

High-priority finding → click "Plan a fix"
    ↓
Fix detail opens:
  - Cause: the specific config gap
  - Expected Output: what a fix achieves
  - Fix instructions: editable pre-filled prompt
    ↓
Click "Start Fix" → launches as a Quest task
    ↓
Agent executes fix in an inspectable, reversible Quest task
    ↓
Re-run /better-harness → confirm the loop actually improved

Fix results can be distilled further into Rules, Skills, and Memories, letting subsequent tasks benefit directly.

Resources

• 🌟 GitHub: QoderAI/better-harness
• 📖 Docs: docs.qoder.com/user-guide/knowledge-engine/better-harness

Summary

Better Harness solves a meta-level problem: an AI coding agent's output quality depends on the surrounding workflow, not just the model's capability. The same Claude Sonnet in a workflow with complete AGENTS.md, clear acceptance criteria, post-change test runs, and experience distilled into Skills produces significantly better outcomes than in a workflow without any of these.

The five-dimension framework makes "workflow health" measurable — not "the workflow feels off" but "Change Validation scored low because no test execution records were found." Priority ordering tells you what to fix first. Repair plans tell you how. The historical trend confirms fixes actually worked.

The deliberately conservative scoring is what makes this tool trustworthy. It doesn't infer "tests ran" from "test files exist." It doesn't claim "the repair caused the improvement" from historical score increases. That honesty means the output can be acted on rather than second-guessed.

If you're using Claude Code, Codex, or Cursor and your sessions occasionally feel inefficient — goals that took too long to clarify, changes that broke unexpectedly, the same problems recurring — /better-harness surfaces which part of the loop is actually broken.

Explore PrimeSkills — A marketplace for handpicked AI Agents and skills. Each is validated in real enterprise workflows, stripping away hype and keeping only what truly works.

Welcome to my Homepage for more useful insights and interesting products.