LLM-Driven Automated Testing Series (02): The Unified Technical Foundation — Visual Grounding / DOM Semantics / Computer Use

Same problem, three answers

"Find the submit button on this page and click it."

Hand that instruction to three different open-source GUI agent projects and you'll get three entirely different implementations:

• One feeds the full screenshot to a specially-trained visual grounding model, which directly outputs a coordinate like (842, 613)
• Another parses the page's DOM or accessibility tree, finds the node with role=button, name="Submit", and calls its click handler directly
• A third skips any dedicated locating step entirely — a general-purpose multimodal model just looks at the screenshot and decides "click here" on its own

These three paths aren't interchangeable implementation details — they're distinct architectural decisions, each with clear trade-offs and boundaries. This post builds the vocabulary the next 12 case-study posts will reuse. Without it, every post would have to re-explain "which locating approach this project uses" from scratch.

Path 1: Pixel-Based Visual Grounding Models

How it works

This path treats the GUI as an image. Given a natural-language reference ("the submit button") and a screenshot, the model directly outputs the element's coordinates or bounding box on screen.

This is a dedicated sub-task in the literature called GUI grounding: it doesn't decide "what to do next" — it only maps a natural-language reference precisely onto screen coordinates. Representative open-source models include UGround, Aria-UI, and Alibaba's self-trained GUI-Owl (used in Mobile-Agent-v3, covered in a later post).

The typical architecture is two-stage:

User task: "Submit this form"
        ↓
Planning model (can be a general LLM): "Next step is to click the submit button"
        ↓
Visual grounding model (dedicated, e.g. UGround): "submit button" + screenshot → (842, 613)
        ↓
Execution layer: click at (842, 613)

Planning and grounding are two separate models, each specialized for one job.

Trade-offs

Advantages: no dependency on any application-internal structured data — pure "looking," which makes it naturally cross-platform (web, mobile, desktop, even game UIs — anywhere you can take a screenshot). A key result from the UGround paper: this path beats accessibility-tree-dependent approaches (Path 2 below) by up to 20 absolute percentage points on multiple GUI grounding tasks, because real-world accessibility trees are often incomplete or unreliable.

Costs:

1. Every locating step requires a visual model inference pass — slower and more expensive than reading structured data directly
2. Accuracy drops sharply for small elements and dense layouts (the ScreenSpot-Pro numbers below quantify how much)
3. The grounding model itself needs dedicated training and maintenance — you can't get high accuracy "for free" from a general-purpose model

Path 2: DOM / Accessibility-Tree Structured Understanding

How it works

This path doesn't look at pixels — it looks at structure. Web pages have a DOM tree; desktop and mobile apps have an accessibility tree (the semantic view the OS exposes for screen readers and other assistive tools). Serialize that tree into text and hand it to an LLM directly: "Given this structure, which node is the submit button?"

Once the target node is found, subsequent actions don't need coordinates at all — you call the node's exposed click(), setValue(), etc. directly, the same mechanism traditional Selenium/Playwright scripts already use. The only thing that changes is that "finding the node" moves from a hardcoded selector to LLM-based semantic matching.

Trade-offs

Advantages:

1. Structured data is text, so token cost is far lower than images, and it's faster
2. Locating accuracy doesn't degrade with screen resolution or element size — if the node is in the tree, semantic matching is precise regardless
3. Can reuse an existing automation framework's execution layer (Playwright, Appium) — the change is isolated to the "locating" step

Costs:

1. Strongly dependent on the app exposing complete, reliable structured information. Canvas-rendered content, embedded WebViews, and heavily custom-rendered mobile widgets often show up as an opaque black-box node in the accessibility tree with no semantic information at all — at that point this path simply fails and must fall back to a visual approach
2. Real-world DOM/accessibility trees can be enormous (modern SPAs routinely have thousands of nodes); stuffing the whole thing into an LLM's context is expensive, requiring pruning or downsampling — this is exactly the problem work like "DOM Downsampling for LLM-Based Web Agents" sets out to solve
3. Tree structures differ significantly across platforms (Web DOM, Android View tree, iOS UIKit tree), so one semantic-understanding pipeline doesn't transfer cleanly across platforms

Path 3: Computer Use — Raw Coordinate Clicking

How it works

This is the most "brute-force," yet also the most commercially deployed path of the past two years: Anthropic's Computer Use and OpenAI's Operator/CUA both fall into this category.

The key difference from Path 1: there's no separate "grounding model" step. A general-purpose multimodal model takes the screenshot directly as input and directly outputs the next action — "click at (842, 613)," "type text," "press Tab" — perception and decision-making happen in a single pass by the same model. There's no "planning model + dedicated grounding model" division of labor.

Screenshot → general multimodal model (perception + decision combined) → directly outputs: click(842, 613)

Trade-offs

Advantages:

1. Simplest architecture — no separate grounding model to maintain, no platform-specific structured data to parse
2. Because it has zero dependency on any interface the app exposes, it has, in principle, the highest ceiling on generality — "if you can screenshot it, you can operate it" — and it's the technical basis for turning an agent from "a web automation tool" into "a general assistant that operates a computer"

Costs:

1. Locating accuracy is the weakest of the three paths on public benchmarks — the numbers in the next section quantify this
2. Highest latency and cost: every single action requires a full multimodal inference pass, and tasks often need many consecutive steps to complete — this is the most expensive path of the three to run
3. No structured-data fallback: Path 2 can approach near-100% precision when the structure is reliable; Path 3 has no such ceiling guarantee — it's purely bounded by the model's visual grounding ability

Quantifying the gap with benchmarks

Saying "one path is better" without numbers is meaningless. A few public benchmarks let us compare the three paths on the same scale.

The ScreenSpot family: testing "locating" in isolation

ScreenSpot was the earliest GUI grounding benchmark: given a natural-language reference and a screenshot, the model must point to the target element's coordinates. Leading models already score above 90% accuracy on this benchmark.

ScreenSpot-Pro is the hardened version: all screenshots come from real professional desktop software (CAD tools, IDEs, data-analysis apps), at higher resolution with smaller, denser interface elements. The same leading models drop to 37.1% (one of the highest scores reported by a 2025-generation model — most models score far lower).

Benchmark          What it measures              Leading model accuracy
─────────────────────────────────────────────────────────────────────
ScreenSpot          Locating in simple UIs         90%+
ScreenSpot-Pro       Locating in professional UIs   ~37% (best case)

That cliff-edge drop reveals a key fact: "locating accuracy" isn't a fixed property of a model — it collapses sharply as interface density and element size push in the wrong direction. This is exactly why mobile automation (covered in five dedicated posts, 05-09) leans so heavily on dedicated grounding models — icons on a phone screen tend to be smaller and denser than buttons on desktop software.

OSWorld: testing end-to-end task success

ScreenSpot only tests "can you find the element." OSWorld tests whether a full task actually gets completed — executing a multi-step task in a real OS environment (e.g., "clean up this data in LibreOffice Calc and export it as a PDF"), which requires planning, locating, execution, and error recovery across the whole chain.

The OSWorld paper reports a human baseline of 72.36% task completion, while early baseline models (e.g. GPT-4V) achieved only 12.24% — a gap of nearly 60 percentage points. The paper explicitly attributes the bottleneck to GUI grounding and operational knowledge, not task comprehension itself. That number has since closed rapidly as dedicated agent architectures (e.g. the Agent S series) improved — by 2025-2026, leading approaches reach roughly 65%-70%+, closing in on the human baseline, but that gain came from agent-architecture-level engineering — planning, reflection, retry loops — not purely from a better underlying grounding model.

                     Human baseline    Early baseline model    2026 leading approaches
──────────────────────────────────────────────────────────────────────────────────
OSWorld task success   72.36%           12.24%                  ~65%-70%

The takeaway: grounding accuracy and task success rate are two different things. A perfectly accurate grounding model still won't get you a high end-to-end success rate if the agent can't re-plan after a failure or verify whether its action achieved the intended effect. That's why "planning + reflection mechanisms" will be just as important a topic as "which grounding model was chosen" when we get to mobile automation projects like ARTEMIS and Mobile-Agent-v3.

The three paths aren't mutually exclusive

In practice, mature open-source projects rarely bet purely on one path. The common pattern is layered fallback: try Path 2 first (DOM/accessibility tree — fast and precise), and fall back to Path 1 or Path 3 (visual grounding) whenever reliable structured data isn't available. AppAgent, covered in a later post (08), is a textbook example of this "parser tree + visual features, dual input" design — the two paths aren't a mutually exclusive technology choice, but two tiers of fallback ranked by reliability within a single system.

Concept vocabulary (reused across the series)

Summary

1. There are three technical paths for GUI element locating: dedicated visual grounding models (broadly cross-platform, but slow and expensive), DOM/accessibility-tree structured understanding (fast and precise, but dependent on the platform exposing reliable structured data), and Computer Use raw coordinate clicking (simplest architecture, highest generality ceiling, but the lowest locating-precision ceiling)
2. The ScreenSpot → ScreenSpot-Pro cliff (90%+ dropping to ~37%) shows that locating accuracy collapses sharply with interface density — it isn't a fixed model capability
3. OSWorld's human-vs-model gap (72.36% vs. an early baseline of 12.24%) shows that end-to-end task success depends on more than grounding precision alone — planning and error-recovery mechanisms matter just as much
4. Mature systems typically use layered fallback rather than betting on a single path — this vocabulary will recur throughout every case study that follows

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage