Knowledge Share
Technical articles, tutorials, and insights
LLM-Driven Automated Testing Series (14): Production Engineering — Why Testing Turns Out to Be the Cheapest Line Item in This Whole Series
A code-generation workflow I documented earlier burned thousands of dollars in tokens during debugging. The testing scenarios verified across this series — AutoRestTest testing a 15-operation API for about a dime, pytest-triage capping itself at 10 model calls per run by default — are off by orders of magnitude. This article puts the evidence from 13 prior articles side by side to answer the three questions the planning doc left for last: why testing-scenario cost is what it is, where the human-in-the-loop gate actually belongs, and which scenarios should explicitly avoid LLMs altogether.
LLM-Driven Automated Testing Series (13): Flaky Tests Aren't a Rerun-Count Problem, They're a Classification Problem
pytest-rerunfailures (477 stars) and flaky (397 stars) handle test instability with the same one-liner: if it fails, rerun it a few times. That doesn't answer what the planning doc actually asks — was this failure a real bug, environment jitter, or a badly designed test? The only project actually attempting that root-cause classification with an LLM is a 2-star, less-than-two-months-old repo called pytest-triage — and its most instructive feature turns out to be the four safety invariants it wraps around AI judgment, guaranteeing AI is never allowed to affect whether a test actually passes or fails.
LLM-Driven Automated Testing Series (12): Beyond Pixel Diffs — Who's Actually Making LLMs 'Understand' UI Changes
BackstopJS, at 7.2k stars, handles rendering noise with exactly four numeric config knobs — misMatchThreshold, requireSameDimensions, ignoreAntialiasing, usePreciseMatching — which is still just fighting font-rendering jitter with pixel-level tolerance. The project that actually wires a VLM/LLM into the visual-regression judgment loop is a 23-star repo called vlmkit, whose own internal benchmark notes state outright that one model hallucinates red into red. Putting these two side by side answers, concretely, what semantic visual regression actually solves — and what new problems it introduces.
LLM-Driven Automated Testing Series (11): Why API Testing Is Actually the Most Conservative Battleground for LLMs
Keploy has 18.5k stars and its README says 'Expand API Coverage using AI' — but grep the entire open-source Go codebase for LLM/AI keywords and you'll find zero implementation code. Schemathesis, at 3.6k stars, uses no LLM at all — it finds real server 500 errors purely through Hypothesis-based property-based fuzzing. The projects that actually call an LLM API in their core logic turn out to be two sub-100-star projects: AutoRestTest and api-automation-agent. This article pins down why: structured API input/output is naturally suited to traditional fuzzing, so what can LLMs actually do here that traditional tools can't — and why does this domain stay cautious instead of going all-in the way UI automation has?
LLM-Driven Automated Testing Series (10): Self-Healing Locators — Where Heuristic Scoring Hits the Ceiling of Semantic Understanding
Healenium is the most established open-source project in self-healing locators, and its README claims it 'leverages machine learning' — but reading the code directly reveals the core mechanism is a weighted scoring formula over DOM tree similarity, with hardcoded constants for weights and thresholds, nothing to do with machine learning. More notably, its 2026 addition of an AI-based XPath generation endpoint is gated behind a hard-coded exception: 'you must have a paid hlm-ai service.' This article first cracks open exactly how Healenium's scoring formula works, then examines how MarketSquare/robotframework-selfhealing-agents — a genuinely LLM-based self-healing project — designs its multi-agent architecture, and finally answers the question the planning doc raised: where exactly does the tradeoff between healing success rate and false-positive rate actually bite.
Teardown of Anthropic's Official Evaluation Methodology: From Defining Success Criteria to claude plugin eval
Anthropic published two evaluation docs: 'Define success criteria and build evaluations' on the Claude developer platform (methodology) and 'Test plugins with evals' in Claude Code (engineering). This article reads them together: SMAR success criteria, how to choose among six grading methods, the case/grader/baseline structure of claude plugin eval, the with-vs-without Δ design, sandbox isolation and CI exit-code conventions, and what they mean for anyone building their own evaluation platform.
A Skill Evaluation Post-Mortem: I Overturned an Existing Model Selection, Then Built My Own Eval Platform
A teammate's skill-model adaptation report looked polished, but on closer inspection it had real problems: one case per scenario, and AI judging its own correctness. I redesigned the evaluation for the Jira Skill's most frequent task — fetching a bug and downloading attachments — with 100 cases × 3 samples, scored by scripts that diff against ground truth. The result was counter-intuitive: the top-scoring sonnet-5-5 was not the best choice; ds-v4.1-flash won on success rate, cost and speed combined. The post ends with the Skill evaluation platform behind the re-test: eval plans, cases, runs, metrics, AI-written reports, plus version-over-version and model-vs-model comparison.
Open Source Project #245: Raven — The 'Harness of Harnesses' That Orchestrates Claude Code, Codex and More, and Improves Itself Recursively
Raven, from EverMind-AI, is an open-source Host Agent: four built-in agents (Research / Code / Design / Oncall), 13 preset third-party agents (Claude Code, Codex, Hermes Agent, etc.) connected over ACP, CLI or OpenAI-compatible APIs, an Evolver that improves the harness itself through propose → evaluate → adopt, and cross-session memory from EverOS. 5.7k Stars, Apache-2.0.
Open Source Project #246: rec — Recursive's Purpose-Built Local Inference Engine for Running Flint on a Mac, Claiming 1.4–2.6x Over llama.cpp
rec is a local inference engine from recursive-oss built to run Recursive's Flint language model on Apple Silicon Macs, aimed at single-session coding-agent use. One command installs it, rec launch pi / opencode starts it, and it exposes local OpenAI- and Anthropic-compatible APIs. The project claims 1.4–2.6x the throughput of llama.cpp on an M4 Pro. Apache-2.0, 151 Stars.
Open Source Project #247: wxa-runner — Run WeChat Mini Programs in a Plain Browser on the Official Base Library, with Headless Playwright E2E
wxa-runner, from open-wxa, is a browser runner for WeChat Mini Programs. It implements only the WeChat client's native layer (message bridge, page stack, native APIs) and reuses the official base library, compiler, and bridge protocol, so mini programs run in an ordinary browser for local debugging and Playwright-based headless E2E testing — no developer tools needed on Linux CI. Supports native mini programs and uni-app / Taro build output. MIT, 556 Stars.
Open Source Project #244: HowToLiveBetter — 767 Pieces of Advice, Each With Cost, Payoff, Evidence Grade, and a Primary Source
A Chinese-language life guide on GitHub covering 34 topics — longevity, first aid, saving money, legal red lines, layoffs and work injury, health and social insurance, relationships and parenting, business compliance — in 767 pieces of advice. Each entry states cost, payoff, an evidence grade (A/B/C), and an original source, citing only journal papers and official documents. Ships with a search page, PDF/EPUB/Anki deck, and a skill that lets AI assistants answer from the book. CC BY 4.0, 67k+ Stars.
Open Source Project #243: ArtCraft — An AI Creation IDE for Artists: Build the Scene First, Then Generate, Turning Prompt-Gambling into Craft
A desktop AI image and video creation IDE written in Rust + Tauri: composite in 2D, stage scenes and character poses in 3D, then pick a model to render. Aggregates 62 models (Nano Banana, GPT Image, Seedance, Kling, Veo, Suno, Hunyuan 3D and more) plus Grok, Midjourney, Sora, and World Labs entry points. Dual-licensed MIT/Apache-2.0, 34k+ Stars.