Open Source Project #222: security-audit — Cloudflare's Skill That Turns Your Coding Agent into a Six-Phase Security Auditor, 13k Stars

Introduction

"A coding-agent skill for multi-phase security audits with independently verified, machine-readable findings."

This is article #222 in the "One Open Source Project a Day" series. Today's project is security-audit — Cloudflare's coding-agent skill, 13,825 Stars, MIT license.

security-audit addresses a critical question: how do you make AI-generated security audit results trustworthy enough to hand to a security team? The answer isn't "a smarter model" — it's a structured process: six audit phases, adversarial validation, and machine-readable finding records. It turns "the model says there's a problem here" into "here's a confirmed vulnerability with source evidence, a reproducible path, and a priority ranking."

What You'll Learn

The six-phase audit workflow, from reconnaissance to target-neutral reporting
The adversarial validation principle (the checker is never the finder)
The difference between three verdicts: confirmed / needs_validation / rejected
How the coverage ledger makes multi-run audits additive
The sandbox requirement: why "can't execute target code" means needs_validation

Prerequisites

Experience with Claude Code or similar coding agents and the skill mechanism
Basic understanding of security audit concepts (attack surface, trust boundaries, vulnerability confirmation)
Node.js fundamentals

Project Background

Overview

This skill is the single-repo starting point of Cloudflare's vulnerability discovery harness. Cloudflare's official blog Build your own vulnerability harness describes how that system evolved into a multi-stage, fleet-wide vulnerability discovery platform — and security-audit is the single-repo version it evolved from.

It's not a "generate a security report" prompt template — it's an orchestration system: isolated sub-agents run reconnaissance, hunting, validation, and verification, and every step produces structured records with independent validators.

Author / Team

Organization: Cloudflare
Primary language: JavaScript (zero-dependency validators, written for Node.js)
License: MIT
Created: 2026-06-18

Project Stats

⭐ GitHub Stars: 13,825+
🍴 Forks: 740+
📄 License: MIT
📅 Created: 2026-06-18

Quick Start

Installation

# Install with the Skills CLI
npx skills add https://github.com/cloudflare/security-audit-skill \
  --skill security-audit
 
# User-level installation
npx skills add https://github.com/cloudflare/security-audit-skill \
  --skill security-audit \
  --global

Usage

Start your coding agent pointed at the codebase you want to audit, then say:

security audit this codebase

Or:

find security vulnerabilities in ./src

do a security review, output to ~/audits/my-project

The skill activates automatically when the request matches its trigger (security audit / find vulnerabilities / pen-test, etc.).

Two Modes

Mode	Trigger	Behavior
guidance	Security questions, focused reviews, methodology	Use only relevant parts; don't run the full workflow or write files
full audit	Explicit audit/pen-test request, end-to-end review	Run all six phases and write report files

Core: The Six-Phase Audit Workflow

Phase 1: Reconnaissance

Launch multiple research agents in parallel, each returning structured source facts with file:line references:

Agent 1a: product type, tech stack, build commands, subsystem boundaries
Agent 1b: principals, authority, trust boundaries, control locations
Agent 1c: entry surfaces, copies, and sinks

Outputs architecture.md and coverage-ledger.json. Reconnaissance is read-only — no external services are contacted.

Phase 2: Coverage-Led Hunting

Assigns ledger "coverage units" to isolated general agents. Each hunter reads only its assigned source blocks, writes only to its own scratch/, and returns one structured result.

Key: coverage critics find gaps — which units were missed, which assignments overlap.

Phase 3: Independent Candidate Validation

Every unique candidate (after deduplication) goes to a fresh verifier that did not hunt it, tasked with trying to refute it.

The verifier's prompt says explicitly: "You did not write this candidate. Try to refute it from repository source and bounded local evidence."

Phase 4: Structured Output

Writes three verdict records to findings.json and validates them with validate-findings.cjs.

Phase 5: Independent Record Verification

Fresh agents verify final source claims. Material replacements receive another independent verifier.

Phase 6: Target-Neutral Reporting

Derives REPORT.md, FINDINGS-DETAIL.md, and NEEDS-VALIDATION.md from the verified records and coverage ledger.

Three Verdicts: Clear Semantics

This is the most instructive design point — verdicts aren't "high/medium/low risk"; they're "evidence completeness":

Verdict	Meaning
`confirmed`	Complete source trace with a bounded, reproducible observation
`needs_validation`	An exact unresolved fact, but no severity assigned
`rejected`	A disproved candidate (records why it was rejected)

The key principle: "a source-grounded suspicion" ≠ "a confirmed vulnerability." When a lead can't be validated because of sandbox constraints, it stays needs_validation — never hastily marked confirmed, never silently dropped.

Design Principles

1. Adversarial Validation

The agent that checks a finding is never the agent that found it. This prevents model self-confirmation — when a model validates its own finding, it tends to confirm it.

2. Severity Requires Impact

Severity = likelihood × impact, not deviation from a checklist. A problem that matches a checklist but has no actual impact is not a vulnerability.

3. Defense-in-Depth Gaps Are Not Vulnerabilities

If Layer A already prevents the attack, the absence of Layer B is a hardening note, not a vulnerability.

4. Multiple Runs Improve Coverage

In Cloudflare's test runs, a single run found roughly half the vulnerabilities that repeated runs found in total. So the skill is designed for additive multi-run coverage — each run uses prior ledgers and findings to target gaps.

The Coverage Ledger: Making Audits Additive

coverage-ledger.json is the skill's core data asset.

A normal security audit is one-shot — run once, produce a report, and running again means starting from zero. security-audit's ledger makes audits incremental:

First audit run:
  → generates coverage-ledger.json (records which units are covered, confirmed, or pending)
  → finds N vulnerabilities
 
Second audit run (same repo):
  → reads the prior ledger and findings
  → only re-hunts "uncovered gaps" and "changed source"
  → carries forward prior findings still backed by current source
  → does NOT treat stale or unresolved prior work as covered

Each unit has a state: planned → in_progress → completed / deferred. If a unit can't be assigned because of agent-count limits, it's explicitly marked deferred with a reason — never silently dropped.

Machine-Readable Finding Records

findings.json follows the schema in report-schema.json, paired with zero-dependency validators:

File	Purpose
`report-schema.json`	JSON schema for all three verdicts
`validate-findings.cjs`	Zero-dependency validator (used in Phases 4/5)
`validate-coverage-ledger.cjs`	Coverage ledger validator (used in Phases 1–5)

The parent runs validate-coverage-ledger.cjs after creating the ledger and after each update, and validate-findings.cjs in Phase 4 and after each Phase 5 replacement.

This "machine-readable + independently validated" design lets audit output feed downstream tooling, rather than being a PDF only humans can read.

Sandbox Requirements: An Honest Boundary

security-audit explicitly requires: executing target-controlled code must happen inside an OS-enforced sandbox.

The sandbox must:

Disable external networking
Use a sanitized allowlisted environment
Enforce resource limits
Allow writes only to assigned scratch paths

If these controls are unavailable, the workflow does not execute target code and keeps the lead as needs_validation.

This is an honest engineering decision: better to leave something unconfirmed than to run potentially malicious code without a safe boundary. In AI security tooling, this explicit stance on the "validation boundary" is more trustworthy than "I can auto-run the PoC."

Resources

🌟 GitHub: cloudflare/security-audit-skill
📝 Background blog: Build your own vulnerability harness
🔧 Skills CLI: skills.sh
✉️ Contact: security-ai-research@cloudflare.com

Summary

security-audit represents a specific judgment: the bottleneck in AI security auditing isn't "can the model find vulnerabilities" — it's "can the findings be trusted."

Three things worth noting:

Adversarial validation is the core of credibility. The problem with many AI audit tools is that discovery and validation are done by the same model — the model "finds" a vulnerability, then validates its own finding, naturally biased toward confirmation. security-audit cuts this bias at the process level with the "checker ≠ finder" structural constraint.

Verdict semantics = evidence completeness, not risk level. confirmed / needs_validation / rejected describe "where the evidence chain breaks," not "how severe this is." Severity (likelihood × impact) is a field inside confirmed. This separation makes audit results machine-processable and trusted by security teams.

Additive coverage solves the "audits are one-shot" problem. Traditional audits go stale the moment they're done. security-audit's coverage ledger makes audits incremental: each run builds on prior coverage, only filling gaps and revalidating changes. This is closer to "continuous security" than "periodic audit."

If you're doing security work with a coding agent, or want to understand how to make AI-generated security conclusions trustworthy, security-audit is the most complete open-source reference available.

Explore PrimeSkills — a curated marketplace of AI agents and skills, each validated against real enterprise workflows. No hype, just what actually works.

Visit my personal site for more insights and interesting products.