Two ways to write the same test step
A traditional Playwright script needs a stable selector before it can click a button:
await page.locator('[data-testid="submit-btn"]').click();This line assumes a stable data-testid actually exists on the page. If that button is a bare icon, drawn on a <canvas>, or buried inside a cross-origin iframe, this approach breaks down entirely — with no semantic annotation, a selector has nothing to grab onto.
Midscene, open-sourced by ByteDance's web-infra team, answers this differently: skip the selector, hand the full screenshot to a vision model, and describe what to do in natural language:
await agent.aiTap('submit button');This maps directly onto Path 1: visual grounding from the vocabulary built in the previous post (02) — no dependency on any structured interface the app exposes, purely "looking." This post digs into what that path actually looks like in a real, production-shaped testing framework.
Four core APIs, four levels of granularity
Midscene isn't a single "control the page with natural language" black box — it splits different levels of need into four APIs with clearly separated responsibilities:
aiTap / aiInput —— Atomic actions: precise, single-step "click" or "type"
↓
aiAct —— Autonomous flow: hand a multi-step natural-language task
to the agent and let it plan its own execution
(e.g. "search and sort by price ascending")
↓
aiQuery —— Structured extraction: "read" structured data off the page
(e.g. extract all item names and prices in the current cart)
↓
aiAssert —— Visual assertion: verify whether "what a user would actually see"
matches expectations (e.g. is this button highlighted, is the layout broken)The design intent behind this split: not everything in a test script should be handed off to autonomous agent decision-making. Atomic actions (aiTap/aiInput) preserve the determinism and readability of traditional scripts, where every single step is explicitly specified by a human. The autonomous flow (aiAct) is reserved for exploratory tasks where the number of steps isn't fixed in advance. This is fundamentally different from a pure Computer Use approach, where a single model decides every action end to end — Midscene lets the test engineer choose exactly where uncertainty gets introduced, rather than betting the entire test flow on the model's planning ability.
aiAssert deserves a callout of its own: it directly addresses the oracle problem from post 01. A lot of visual correctness can't be judged by pixel diffing or checking a DOM attribute — e.g., "does this button look disabled?" aiAssert hands that judgment to the vision model as a semantic decision, instead of hardcoding a CSS class check. This idea gets expanded further in post 12 (visual and semantic regression testing).
Core implementation one: three model roles, not one model doing everything
The previous section said aiAct "hands a multi-step task to the agent to plan and execute on its own," but glossed over how that "planning" actually maps onto model calls. Midscene's answer is to split a single automation task across three roles (its Model Strategy), each of which can be bound to a different underlying model:
Default model —— Handles element locating, plus whatever workload
Planning/Insight don't take over — the base model
that carries most automation needs by default
Planning model —— Enhances planning for complex goals, multi-step
tasks, and branching scenarios
Insight model —— Enhances data extraction, assertion judgment,
and page understandingThe key design choice is that these three roles can each be bound to a different underlying model — the official docs put it as "users can layer Planning and Insight models on top of the Default model," letting each model play to whichever role it's actually good at: pair "planning" with a model that reasons better, pair "locating" with a model that's more precise visually, instead of forcing one general-purpose model to carry both entirely different capability demands at once.
That flexibility isn't free, though. The official docs explicitly warn: "this collaboration extends Midscene's ability to handle complex tasks, but also increases task latency and token consumption." The recommended usage is therefore incremental: get things working with the Default model first, and only bring in a Planning or Insight model once you hit a clear capability bottleneck — not turn all three on by default.
This role split is itself an engineering-level response to post 02's conclusion that "locating precision isn't the only variable": explicitly separating "planning" and "locating" as independent capability needs, each with its own model choice, makes it easier to optimize precisely against a bottleneck than a single model trying to cover both — at the cost of extra latency and call overhead for that flexibility.
Core implementation two: aiAct's execution loop — when to replan, when to split out locating
aiAct doesn't just call the model once, get back a full step list, and execute it end to end. The official docs describe it as continuously using AI to plan the next action from the latest UI state, until the goal is complete — meaning it re-observes the current screen after every single step before deciding the next one, rather than sticking rigidly to whatever plan got generated when the task started. The benefit: if a permission dialog pops up mid-flow or a loading delay shifts the layout, planning naturally adjusts from the latest state, without needing extra branch logic hand-written for "what if an unexpected dialog appears."
This loop isn't infinite replanning, either — a replanningCycleLimit (max replanning cycles) parameter backstops it, with a default that varies by model type (20 for standard models, 40 for UI-TARS, 100 for AutoGLM). Once that limit is exceeded, or an assertion embedded in the prompt fails, aiAct throws immediately rather than continuing to retry indefinitely. This is the same design philosophy as post 03's "fail conservatively" idea: better to surface an error quickly for a human to judge than let a task that can't produce a result spin silently.
By default, aiAct handles "plan the next step" and "locate the target element" in a single model call. But complex tasks sometimes need those two split into separate calls — that's what the deepThink parameter does: turning it on splits planning and locating into two independent calls, trading more calls and latency for better stability on complex tasks.
There's a historical wrinkle here worth clarifying, since it's an easy source of confusion: the parameter name deepThink has meant two different things across different APIs. In aiAct(), it has always been the "planning mode" switch; but in single-step methods like aiTap, the older deepThink meant "enhanced element locating." To resolve that ambiguity, Midscene renamed the "enhanced locating" concept to its own deepLocate parameter, leaving deepThink consistently meaning "planning mode" — and inside aiAct(), deepThink (whether to split out planning) and deepLocate (whether to boost locating precision) coexist as two independent parameters, not mutually exclusive ones.
Relationship to Playwright: augmentation, not replacement
Midscene didn't reinvent a browser-control protocol — it wraps the existing Playwright page object via PlaywrightAgent. Everything Playwright already does (navigation, waits, traditional selector-based assertions) stays fully intact; Midscene's four AI APIs are simply layered on top. The same integration approach also supports Puppeteer.
The practical implication: migration cost is incremental, not a rewrite. An existing Playwright test suite can start by swapping aiAssert into the handful of assertion points most likely to break on a UI redesign, leaving everything else unchanged — you don't need to rip out the whole framework just to introduce visual locating.
Going further, Midscene turns this Agent API into a unified cross-platform interface: "the same API covers Web, Android, iOS, HarmonyOS, and desktop applications." That means a test engineer who's already internalized aiTap/aiQuery/aiAssert doesn't need to learn a new API when switching to mobile or desktop automation — only the underlying screenshot-capture and event-injection implementation changes, not the call pattern. (Specific mobile projects, each with their own mobile-specific optimizations, get their own detailed treatment in posts 05-09.)
The caching mechanism: how the biggest pain point of visual approaches gets mitigated
The previous post already covered the cost of Path 1 (visual grounding): every locating step requires a model inference pass — slower and more expensive than reading structured data. If a test suite has hundreds of test cases, each with dozens of steps, and every single step makes a real vision-model call, that cost is untenable to run in CI.
Midscene's answer is to cache two distinct kinds of output separately, rather than lumping everything under a generic "cache the locate result":
Planning cache (for ai / aiAct)
key = the raw natural-language instruction the user wrote
value = the execution plan the model returned (a sequence of concrete steps)
Locate cache (for aiLocate / aiTap and other single-step locating calls)
key = the locate prompt
value = the matched element's XPathQuery-type operations (aiQuery/aiBoolean/aiAssert) are explicitly excluded from caching altogether — because the whole point of those calls is to "read the page's actual current state," and caching a past judgment would be semantically wrong.
The locate cache isn't "cache the XPath and trust it blindly." Every run first re-validates whether that XPath is still valid on the current page, and invalidates it if either "the text content of the element at that same XPath has changed since it was cached" or "the page's DOM structure has changed relative to when it was cached." This is a deliberately strict matching policy — better to invalidate a bit too often and trigger a fresh locate than to click, on a page that's already changed, an element that merely looks visually similar but is no longer semantically the right one. The planning-cache side has a similarly conservative design: if a cached aiAct plan fails during execution, Midscene falls back to the normal AI planning flow and clears the stale cache entry; but the freshly-regenerated plan from that fallback doesn't get written back into the cache under the original prompt, since it may not represent a complete, correct plan starting "from the task's original state."
There's one more boundary condition worth calling out on its own: the locate cache is structurally unusable in rendering scenarios with unstable structure — shapes drawn inside a Canvas have no corresponding DOM nodes, so there's naturally no XPath to cache; cross-origin iframes are inaccessible internally due to browser security policy; Closed Shadow DOM is entirely inaccessible from the outside; WebGL/dynamic SVG have inherently unstable DOM structure, so even a cached result would likely invalidate on the very next run. This lines up exactly with the point made at the top of this post: canvas/iframe/custom rendering are precisely the scenarios where a pure-visual approach has its core advantage — and also precisely the scenarios where the caching mechanism is structurally unavailable and every run genuinely needs a model call. The cost savings caching offers and the pure-visual approach's core advantage scenarios sit in structural tension with each other — that isn't a design flaw in Midscene, it's an inevitable consequence of those interfaces lacking structured information in the first place.
This design effectively downgrades "visual locating" from "must be used every single time" to "a fallback mechanism." Under normal conditions, tests run a cached, deterministic sequence of operations — with speed and cost essentially indistinguishable from a traditional selector-based script. Only when the UI genuinely changes and invalidates the cache does it fall back to the vision model for fresh locating — though in canvas/iframe and other structurally-missing scenarios, that "fallback mechanism" is in fact the only mechanism available. In a sense, this already carries a flavor of the "self-healing locators" topic covered in post 10 — the difference is that post 10's self-healing logic focuses on "how to recover after a locate failure," whereas this caching mechanism solves a different problem: "most of the time, locating shouldn't need to be triggered at all."
Real benchmark numbers
Midscene's official site publishes three benchmark results, worth comparing against the ScreenSpot/OSWorld numbers from post 02:
AndroidWorld Benchmark Pass@1 93.1% Pass@3 97.4%
MobileWorld Benchmark Pass@1 78.6% (92/117 passed)
AppControlBench Benchmark Pass@1 96.7% (58 items passed)These are all mobile benchmarks (AndroidWorld gets a full treatment in post 05), but there's a comparison worth flagging here: post 02 mentioned OSWorld's early baseline models achieving only a 12.24% task success rate, while Midscene achieves a 93.1% Pass@1 on AndroidWorld. This isn't saying Midscene's underlying model is dramatically stronger than what OSWorld tested — the task types and evaluation paradigms differ. Benchmarks like AndroidWorld/AppControlBench, purpose-built for mobile GUI agents, have more clearly-scoped tasks, and combined with dedicated planning + retry mechanisms (Pass@3 being 4 points higher than Pass@1 shows that allowing retries genuinely recovers some single-attempt locating failures), can hit success rates far above OSWorld's open-ended desktop tasks. This reinforces post 02's conclusion: end-to-end task success depends on more than grounding precision alone — planning, retries, and task-scope design matter just as much.
Comparing paths against Stagehand
Midscene isn't the only open-source project in this space. Stagehand (open-sourced by Browserbase) is a worthwhile comparison — it also layers a natural-language API (act/extract/observe) on top of Playwright, but its underlying technical path leans more toward Path 2 from post 02: DOM/accessibility-tree semantic understanding — it prioritizes structured information exposed by the browser for locating and action, with visual capability as a supplement rather than the core.
Primary locating method Best fit Cost
──────────────────────────────────────────────────────────────────────────────────────────
Midscene Pure vision (screenshot+model) canvas/iframe/icon-only scenarios Requires a model
lacking semantic annotation inference call per
locate (mitigated by cache)
Stagehand DOM/accessibility tree first, Conventional web apps with Weaker fallback than a
vision as backup reliable structured info pure-visual approach when
structure is missingThis comparison echoes post 02's core conclusion: the two paths aren't a matter of one being technically superior — they're trade-offs suited to different scenarios. If the test target is a conventional web app with complete structural info and proper semantic annotation, a DOM-first approach (like Stagehand) is usually faster and cheaper. If the test target itself involves heavy canvas rendering, custom-rendered widgets, or needs one unified test logic maintained across Web/mobile/desktop, a pure-visual approach (like Midscene) has a clearer edge in generality.
Summary
- Midscene takes the pure visual grounding path — no dependency on selectors or semantic annotations, feeding screenshots directly to a vision model — which lets it cover canvas rendering, icon-only elements, and cross-origin iframes that traditional approaches can't locate at all
- The four core APIs (
aiTap/aiInputfor atomic actions,aiActfor autonomous flows,aiQueryfor structured extraction,aiAssertfor visual assertions) let the test engineer choose exactly which layer to introduce uncertainty into, rather than betting the entire flow on the model's planning ability - The Default/Planning/Insight model-role split lets "planning" and "locating" be bound to different models and tuned independently, at the cost of extra latency and token spend — the official recommendation is to adopt Planning/Insight incrementally, not enable all three by default
aiAct's execution loop replans from the latest UI state after every step, rather than sticking to whatever plan was generated at the start; thedeepThinkparameter controls whether planning and locating happen in one model call or two separate ones, trading extra calls for stability on complex tasks- The caching mechanism caches planning output (keyed on the raw instruction) and locate results (keyed on the locate prompt, valued as XPath) separately, with the locate cache enforcing strict invalidation on text or DOM-structure changes; it's structurally unusable in canvas/iframe and similar structure-missing scenarios — precisely where the pure-visual approach's core advantage and the cache mechanism's blind spot overlap
- Compared against Stagehand (DOM/accessibility-tree first), the two paths aren't a matter of one being better — they map onto two different scenarios: unreliable structural information versus complete, reliable structural information
Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.
Find more useful knowledge and interesting products on my Homepage