Knowledge Share
Technical articles, tutorials, and insights
LLM-Driven Automated Testing Series (14): Production Engineering — Why Testing Turns Out to Be the Cheapest Line Item in This Whole Series
A code-generation workflow I documented earlier burned thousands of dollars in tokens during debugging. The testing scenarios verified across this series — AutoRestTest testing a 15-operation API for about a dime, pytest-triage capping itself at 10 model calls per run by default — are off by orders of magnitude. This article puts the evidence from 13 prior articles side by side to answer the three questions the planning doc left for last: why testing-scenario cost is what it is, where the human-in-the-loop gate actually belongs, and which scenarios should explicitly avoid LLMs altogether.
LLM-Driven Automated Testing Series (13): Flaky Tests Aren't a Rerun-Count Problem, They're a Classification Problem
pytest-rerunfailures (477 stars) and flaky (397 stars) handle test instability with the same one-liner: if it fails, rerun it a few times. That doesn't answer what the planning doc actually asks — was this failure a real bug, environment jitter, or a badly designed test? The only project actually attempting that root-cause classification with an LLM is a 2-star, less-than-two-months-old repo called pytest-triage — and its most instructive feature turns out to be the four safety invariants it wraps around AI judgment, guaranteeing AI is never allowed to affect whether a test actually passes or fails.
LLM-Driven Automated Testing Series (12): Beyond Pixel Diffs — Who's Actually Making LLMs 'Understand' UI Changes
BackstopJS, at 7.2k stars, handles rendering noise with exactly four numeric config knobs — misMatchThreshold, requireSameDimensions, ignoreAntialiasing, usePreciseMatching — which is still just fighting font-rendering jitter with pixel-level tolerance. The project that actually wires a VLM/LLM into the visual-regression judgment loop is a 23-star repo called vlmkit, whose own internal benchmark notes state outright that one model hallucinates red into red. Putting these two side by side answers, concretely, what semantic visual regression actually solves — and what new problems it introduces.
LLM-Driven Automated Testing Series (11): Why API Testing Is Actually the Most Conservative Battleground for LLMs
Keploy has 18.5k stars and its README says 'Expand API Coverage using AI' — but grep the entire open-source Go codebase for LLM/AI keywords and you'll find zero implementation code. Schemathesis, at 3.6k stars, uses no LLM at all — it finds real server 500 errors purely through Hypothesis-based property-based fuzzing. The projects that actually call an LLM API in their core logic turn out to be two sub-100-star projects: AutoRestTest and api-automation-agent. This article pins down why: structured API input/output is naturally suited to traditional fuzzing, so what can LLMs actually do here that traditional tools can't — and why does this domain stay cautious instead of going all-in the way UI automation has?
LLM-Driven Automated Testing Series (10): Self-Healing Locators — Where Heuristic Scoring Hits the Ceiling of Semantic Understanding
Healenium is the most established open-source project in self-healing locators, and its README claims it 'leverages machine learning' — but reading the code directly reveals the core mechanism is a weighted scoring formula over DOM tree similarity, with hardcoded constants for weights and thresholds, nothing to do with machine learning. More notably, its 2026 addition of an AI-based XPath generation endpoint is gated behind a hard-coded exception: 'you must have a paid hlm-ai service.' This article first cracks open exactly how Healenium's scoring formula works, then examines how MarketSquare/robotframework-selfhealing-agents — a genuinely LLM-based self-healing project — designs its multi-agent architecture, and finally answers the question the planning doc raised: where exactly does the tradeoff between healing success rate and false-positive rate actually bite.
LLM-Driven Automated Testing Series (09): Mobile Automation (V) — When a Generic Agent Framework Sinks Down into a Testing Scenario
MetaGPT is a 'multi-agent software company' framework built for requirements analysis and code generation, but it ships a built-in Android testing demo. Reading the source directly overturns one premise from the planning doc: this demo does not reuse MetaGPT's usual Product Manager/Engineer/QA role division — it has exactly one custom Role, reusing instead the lower-level Team/Environment/Role/Action scheduling scaffolding. More importantly, the README states plainly that it 'referenced ideas and code from AppAgent' — this piece breaks down exactly what this generic framework inherited from AppAgent, what reusing generic scaffolding actually bought it, and what it cost.
LLM-Driven Automated Testing Series (08): Mobile Automation (IV) — AppAgent: Combining Parser Trees with Visual Features
AppAgent is Tencent's open-source CHI 2025 project. Unlike the pure-visual (ARTEMIS/Mobilerun) or pure-coordinate (Mobile-Agent-v3) approaches covered earlier, it uses the accessibility tree to draw numeric labels on screenshot elements, and the model only needs to pick a number — the 'parser tree' and 'vision' aren't two independent perception channels at all; structural information handles label placement while the vision model handles the decision. This piece breaks down how the autonomous-exploration phase builds documentation via a four-way reflection (BACK/INEFFECTIVE/CONTINUE/SUCCESS), how the grid-fallback mechanism degrades when the parser tree fails, and honestly points out a real coordinate-conversion bug living inside that grid-fallback path.
One Open Source Project a Day (No. 231): MiniMind — Train a 64M-Parameter LLM From Scratch in 2 Hours for About $0.43
MiniMind is a fully from-scratch, native-PyTorch LLM training project whose core claim is "train a 64-million-parameter language model from scratch in 2 hours for about ¥3." It covers the complete modern training pipeline — pretraining, SFT, LoRA, DPO, RLAIF (PPO/GRPO/CISPO), Agentic RL, and knowledge distillation — and spawns sibling projects for vision, multimodal, diffusion, and linear attention. 63.4k Stars, Apache-2.0 License.
One Open Source Project a Day (No. 230): HandRaw-Style — Numbering 327 Hand-Drawn Styles, 165 Layouts, and 36 Color Palettes So AI Art Stops Drifting Every Time
HandRaw-Style is a "numbered gallery of hand-drawn art styles paired with a bilingual AI-prompt-generation skill." It catalogs 327 hand-drawn illustration styles, 165 layout templates, and 36 classic monochrome color themes, all numbered — users just say a code plus a topic and get tested bilingual prompts, solving the "vague style description, every generation drifts" problem. It supports smart recommendation, precise selection, poster design, article illustration, photography planning, and more modes. 4.7k Stars.
Open Source Project #229: e2e — Write E2E Tests in Plain English, Then Replay Verified Agent Actions With Zero Model Calls, 6k+ Stars
TesterArmy's end-to-end testing framework for web and mobile. The core innovation: agent.act drives the app toward a natural-language goal, agent.assert judges a statement against the screen, and once a goal is verified, its actions are recorded into a replay cache — subsequent runs replay with no model call at all, until the app changes enough that the agent has to take over again. TypeScript, Apache-2.0, 6k+ Stars.
LLM-Driven Automated Testing Series (07): Mobile Automation Part 3 — Mobile-Agent-v3 and the Self-Trained GUI-Owl Approach
Mobile-Agent-v3 is Alibaba Tongyi Lab's multi-agent framework built on the self-trained GUI-Owl model — a completely different bet from the previous two articles' reliance on general-purpose VLMs (Gemini/GPT/Claude): train a dedicated GUI-grounding model that outputs 'absolute pixel coordinates' instead of guessing. This piece dissects the Manager/Executor/ActionReflector/Notetaker role split, the 'two consecutive failures before escalating to Manager' fault-tolerance threshold, and a real, verified HarmonyOS input bug.
LLM-Driven Automated Testing Series (06): Mobile Automation Part 2 — DroidRun/Mobilerun's Role-Level Model Split
DroidRun (now renamed Mobilerun) uses a unified device-driver abstraction to fit Android and iOS behind the same tool interface, but it takes a completely different path from ARTEMIS in the previous post: it doesn't pick 'Flash vs Pro' at the task level — it splits models 'by role,' with Manager, Executor, and FastAgent each getting their own model and temperature. This post digs into its multi-agent workflow, the Portal accessibility service, the App Card mechanism, and also states plainly that its '91.4%' benchmark badge has no locally verifiable documentation anywhere in the repo.