Open Source Project #181: Open Code Review — Alibaba's Battle-Tested AI Code Review Tool, 1/9 the Tokens of a General Agent

Introduction

"The problem with general-purpose agents doing code review isn't that they miss bugs. It's that they don't know which files to read, they drift on line numbers, and they burn 9x the tokens doing it."

This is article #181 in the "One Open Source Project a Day" series. Today's project is Open Code Review — Alibaba's open-sourcing of its internal AI code review tool, released in 2026 after years of serving tens of thousands of engineers internally and identifying "millions of code defects."

Tools that use LLMs for code review are common, but most just dump the diff into a model and wait for output. Open Code Review starts from a different place: a hybrid architecture where deterministic engineering pipelines handle the things that must not be wrong (file selection, line positioning, rule matching), and the LLM agent handles only what genuinely requires dynamic judgment. The result: same underlying model, higher precision and F1, roughly one-ninth the token consumption of a general-purpose agent.

18,100 Stars. 1,200 Forks. Apache 2.0.

What You'll Learn

• The hybrid architecture design rationale: why separate "deterministic" from "agent"
• ocr review (incremental diff review) vs ocr scan (full-file audit)
• Delegation mode: your existing agent runs the review; no OCR API key needed
• Benchmark numbers comparing against Claude Code as a general agent
• GitHub Actions integration and three SLA review levels
• Defect patterns specific to AI-generated code

Prerequisites

• Familiarity with Git workflows (branch, diff, PR)
• Basic understanding of CI/CD concepts
• Experience with Claude Code or similar AI coding tools is helpful

Background

From Internal Tooling to Open Source

Open Code Review wasn't purpose-built for open-source release — it's Alibaba's production-running internal system published externally. That distinction matters: its design decisions come from real large-scale operational experience, not from first-principles speculation.

Before open-sourcing, the internal system:

• Served tens of thousands of engineers
• Identified millions of code defects
• Handled the genuine complexity of production-scale codebases

The Problem with General Agents for Code Review

Sending a diff directly to Claude Code or GPT for code review carries systematic weaknesses:

The root cause: general agents hand all decisions to the LLM for flexibility, including decisions that deterministic code could handle precisely.

Core Architecture: Hybrid Design

Open Code Review's design philosophy: let deterministic things be handled deterministically; let the agent handle only what genuinely requires dynamic judgment.

Git changes
    ↓
[Deterministic Layer]
  ├── Precise file selection (no misses, no extras)
  ├── Smart bundle grouping (related files → isolated sub-agent contexts)
  ├── Template engine rule matching (NPE, thread safety, XSS, SQL injection)
  └── External positioning + reflection modules (accurate line-level placement)
    ↓
[Agent Layer]
  ├── Scenario-tuned prompts and toolsets
  ├── Optimized from analysis of production tool-call traces
  └── Dynamic context retrieval and tool calls
    ↓
Code review output (precise line-level comments)

Deterministic layer handles:

• Selecting exactly the right files from git changes — no gaps, no noise
• Grouping related files into Bundles, each processed in an isolated sub-agent context (divide and conquer)
• Matching common defect rules via template engine rather than asking LLM to reason from scratch each time
• Ensuring comment line numbers are accurate with no position drift

Agent layer handles:

• Prompts and toolsets tuned from analyzing production tool-call traces at scale
• Complex cross-file reasoning that genuinely needs inference
• Dynamic tool calls

The result: higher precision (the deterministic layer removes LLM randomness from structure), lower token usage (the agent processes only the genuinely hard parts).

Benchmark Data

The Open Code Review team built a benchmark from:

• 50 open-source repositories
• 200 PRs
• 10 programming languages
• 1,505 annotated issues (hand-labeled by 80+ engineers)

Compared against Claude Code (general agent mode) using the same underlying model:

On the recall tradeoff: Open Code Review deliberately designs for lower recall than a general agent. This isn't a limitation — it's a choice. Fewer false alarms and higher precision are more valuable than comprehensive coverage in a CI/CD gate. Noise that blocks PRs creates review fatigue faster than any genuine bug.

What the token difference means in practice: For 100 PRs, Open Code Review consumes roughly 11% of what a general agent would. In CI/CD where every PR triggers a review, this determines whether the tool is financially viable. Nine-to-one cost difference changes the calculation entirely.

Two Review Modes

ocr review: Incremental Diff Review (Most Common)

Reviews git diffs — staged changes, unstaged changes, branch ranges, or single commits:

# Review current working tree changes (staged + unstaged)
ocr review
 
# Review a branch range
ocr review --from main --to feature/new-auth
 
# Review a single commit
ocr review --commit abc1234
 
# Output formats
ocr review --output json
ocr review --output sarif   # importable into GitHub Security

ocr scan: Full-File Audit

No git history required — reviews file content directly:

# Audit a directory
ocr scan --path internal/
 
# Audit a single file
ocr scan --path src/auth/handler.go
 
# Generate HTML report
ocr scan --path src/ --output html

Good fit for: security audits on inherited codebases, quality assessment of legacy code with no git history, reviewing code snippets without a repository context.

Delegation Mode

One of Open Code Review's most interesting design choices.

Standard mode: OCR's agent layer uses an LLM API you configure (Anthropic/OpenAI key required) to run the review.

Delegation mode: OCR handles only the deterministic layer (file selection, bundle grouping, rule resolution), then delegates the review task to your existing agent (Claude Code, Codex, Cursor, etc.), using that agent's own LLM.

# Preview the delegation plan (see how OCR intends to split the task)
ocr delegate preview
 
# Generate rule descriptions for specific files for the agent to review
ocr delegate rule src/main.go src/handler.go

Why it's useful:

1. You already have a Claude Code subscription or API key — no separate OCR key needed
2. OCR's deterministic layer handles file selection and rule resolution; your agent focuses on understanding and judgment
3. Integrates into your existing agent workflow without switching tools

GitHub Actions Integration

Thirty-second setup. Every PR triggers a review automatically:

name: AI Code Review
on: [pull_request]
 
jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - uses: raye-deng/open-code-review@v1
        with:
          sla: L2          # Review depth: L1 / L2 / L3
          threshold: 60    # Fail if quality score drops below 60
          github-token: ${{ secrets.GITHUB_TOKEN }}

Three SLA Levels

AI-Generated Code Defect Detection

Open Code Review positions itself on GitHub Marketplace as "the first open-source CI/CD quality gate built specifically for AI-generated code" — detecting defect patterns common in LLM output that traditional linters miss entirely:

These problems appear at significantly higher rates in AI-generated code than in human-written code — standard linters aren't looking for them.

Supported languages: TypeScript/JavaScript, Python, Java, Go, Kotlin (6 languages).

Installation and Configuration

Install

# npm (recommended)
npm install -g @alibaba-group/open-code-review
 
# Requires Git >= 2.41
git --version

Configure the LLM

ocr config provider
# Interactive setup: choose Anthropic / OpenAI / custom compatible endpoint

Fine-grained configuration via .ocrrc.yml:

sla: L3
ai:
  embedding:
    provider: ollama
    model: nomic-embed-text
  llm:
    provider: ollama       # local Ollama supported
    model: qwen3-coder     # any OpenAI-compatible model

First Run

cd my-project
 
# Review current changes
ocr review
 
# List sessions (resume support built in)
ocr session list

Resources

• 🌟 GitHub: alibaba/open-code-review
• 🌐 Docs: open-codereview.ai
• 🏪 GitHub Marketplace: Open Code Review Action
• 📦 npm: @alibaba-group/open-code-review

Summary

Open Code Review makes one thing clear: in the specific domain of code review, using engineering constraints around the LLM produces better results than letting the LLM operate freely.

The problem with general agents for code review isn't insufficient LLM capability — it's that handing all decisions to the LLM introduces unnecessary randomness and token waste where deterministic code could be precise. Handling file selection, line positioning, and rule matching with deterministic pipelines focuses the LLM's attention on the parts that actually need inference.

This design principle is worth generalizing: not "how do we make the AI do better" but "which parts should the AI never have been doing." It's a conclusion that most AI tooling teams arrive at after hitting the same walls at scale. Alibaba packaged the lessons from that experience and released them along with the code.

Explore PrimeSkills — A marketplace for handpicked AI Agents and skills. Each is validated in real enterprise workflows, stripping away hype and keeping only what truly works.

Welcome to my Homepage for more useful insights and interesting products.