What ARTEMIS actually is, and what problem it solves
Google's open-source ARTEMIS describes itself in one line: "Let AI assistants and test suites use real phones like a human." Its core use case: give it one natural-language instruction, and it carries out cross-app operations and verification on a real or emulated Android device — not a hand-written UI automation script, but an agent that plans, executes, and verifies on its own.
A few of its headline features (from the README's Key Highlights) are worth laying out up front, since they're the entry point for everything this post digs into:
- Cross-app automation: executes cross-app test flows and everyday tasks directly from natural-language instructions
- Multimodal targeting: prefers element indices (structured info from the accessibility tree), falling back to coordinate and visual locating when structured info isn't available — this maps directly onto the "Path 1 / Path 2" vocabulary built in post 02
- Native MCP integration: via the Model Context Protocol, AI IDEs like Antigravity, Claude Code, and Windsurf can drive test devices directly and pull Logcat output and screenshots
- Two execution profiles: Flash (a reactive loop, 3–5s per step) and Pro (a multi-agent graph with planning and verification, 15–40s per step)
- AndroidWorld 99%+: claims a 99%+ completion rate on Google Research's AndroidWorld benchmark (20+ apps, 100+ multi-step tasks)
This post doesn't repeat a feature tour — instead it answers three harder questions: How do Flash and Pro actually work internally? Which specific technical choices causally drive that claimed 99%+ completion rate? And what less-common, distinctive technology does it actually use (say, a dedicated locating model)?
Worth noting upfront: ARTEMIS's repository explicitly states it includes source code from Minitap, Inc.'s open-source project mobile-use — this isn't an architecture invented from scratch, it's engineering and extension on top of an existing open-source foundation. Keeping that in mind helps separate "common practice in the mobile-agent space" from ARTEMIS's own incremental contributions.
A bimodal distribution, not "which mode is better"
Mobile UI automation task complexity follows a bimodal distribution: a large chunk of tasks are deterministic short flows like "open Settings, flip a toggle," with a predictable number of steps; a small chunk are exploratory, 20+ step flows that might get interrupted by a dialog halfway through. Using one architecture for both extremes wastes something either way — a reactive loop is fast but can't absorb the errors that accumulate over a long-horizon task; a multi-agent planning loop is robust but paying 15-40 seconds of latency to flip a toggle is pure overhead.
ARTEMIS answers this by splitting the two scenarios into two fully independent execution paths: Flash and Pro.
Flash: why the loop can afford to have no turn cap
Flash mode is a single-model reactive loop: observe the screen → decide the next step → execute → repeat. In the code, FlashRunner has no planning node and no dedicated validation node, and agent.flash.max_turns defaults to 0 (unlimited).
At first glance, "no turn cap" looks like a missing guardrail. But it only works because of a specific precondition: history is managed by compression, not truncation. Flash and Pro share the same TranscriptLedger / history-compression machinery (artemis/memory/chunking.py):
Old screenshots → replaced with visual summaries (pixels discarded, semantics kept)
Completed steps → compressed into searchable "chunks"
Aging chunks → folded into "eras" — a one-line synopsis plus a minimal
per-step index (step number, session-relative time offset,
one-line action phrase)The module docstring describes the folding rule precisely as monotonic: later compression events only append new chunk blocks after the already-folded region, and never restore folded content back to its detailed form. That guarantees context usage has a gradually-tightening ceiling rather than growing linearly with step count; when the agent needs to recall earlier detail, it actively retrieves it via search_history/replay_steps instead of that detail permanently occupying context space.
This clarifies a design choice that's easy to overlook: Flash can set its turn cap to zero not because it's "simpler and doesn't need a limit," but because the memory layer has already solved the "will context blow up" problem — so the turn cap is no longer needed as a blast-radius control, and termination can be left entirely to the task's own completion condition.
Pro: the graph is not a straight line from planner to operator to checker
If you only look at the README's diagram, it's easy to picture Pro mode as a straight pipeline: Planner plans → Operator executes → Checker verifies → done. Reading the actual node/edge wiring in artemis/graph/graph.py, the real topology is a cyclic graph:
planner → convergence → perception → operator → execution_check
↑ ↓
└────────── (validation failed) ← validator ┘
↓
summarizer
↓
convergence
↓
perception (next round) | exit_settlement | ENDTwo nodes are worth pulling apart individually.
convergence_node's entire body is return {} — a node that does nothing at all. Its purpose isn't to run any logic; it exists as a merge point: whether the previous step was the beginning of planning or the end of executing an action about to head into the next perception round, both paths converge here first, and this single node decides whether to head back into perception or toward exit. It's a graph pattern for collapsing branching logic into one decision point — the benefit is that adding a new branch only requires changing the routing logic in this one node, instead of maintaining a separate "where does this go next" decision on every branch-producing path.
execution_check_node treats a failed plan-validation as advisory, not blocking. This node:
- Unconditionally records every action from the Operator's turn to the DataEngine (
_record_turn) — regardless of whether a safety check fired or passed this turn - Awaits and harvests the async planner-validation result (
ctx.planner_task) - If plan validation flags this step as a deviation from the plan — that signal only surfaces as
operator_feedbackin the next turn's Operator prompt; it never rolls back the action already taken, and never forces the current flow to halt - Along the way, harvests/spawns checkpoint-check tasks (
harvest_finished_checkpoints/spawn_pending_checkpoints)
The trade-off behind this design: a lot of "deviation from plan" on mobile UI is actually normal — dialogs, permission prompts, loading delays that cause a transient shift. Forcing a rollback on every deviation would burn a huge amount of retry cost on what are, most of the time, benign shifts. Feeding deviation info in as soft feedback for the next decision hands the judgment of "does this actually need correcting" back to the Operator itself, instead of letting a rule engine decide on its behalf.
exit_settlement_node is a two-phase exit, not a single verdict. Phase one is an unconditional "settlement barrier" that always runs regardless of whether the task met its goal; phase two is the conditional "final review" — pulling the user's original goal and every declared check item, and comparing them against the current device state. If a verify-type check item fails, execution loops back for another attempt (bounded by final_check_max_attempts); but if an assert-type check item fails, no retry loop is triggered — the failure is recorded verbatim as a legitimate test result.
That last point ties directly into the plan grammar itself — the next section unpacks it.
A key distinction baked into syntax, not left to prompt persuasion: verify vs. assert
Pro mode's Planner produces a Markdown task plan that's a "dual-channel document" (in the words of the artemis/utils/plan_grammar.py module docstring):
Machine channel: checkbox micro-grammar (status characters, indentation,
[Loop] tags) — only harness code ever reads this layer to
decide termination, loop protection, and when to trigger
validation/checker runs
Semantic channel: all remaining free text — only LLM agents interpret this
layer; harness code never branches on the wording itselfThe point of this split: any deterministic decision — should the loop end, should validation fire — never depends on an LLM's interpretation of natural language to drive it; code parses structured markers directly instead. That's more reliable than "let the model decide for itself whether to exit the loop," and much easier to test.
Within this grammar, check items come in two forms that look almost identical but mean something completely different:
- verify: <expected state> — an acceptance criterion. Judged at the moment
its parent milestone completes (unless suffixed
with @end, which defers judgment to task exit
using final device state). Failure → reopens
the milestone and triggers repair
- assert: <expected observation> — a test assertion. Failure is recorded
verbatim as a legitimate test result; the
assertion itself is "unrepairable" — its
failure never triggers an extra retry loopThis distinction maps directly onto a real problem in test engineering that's often left fuzzy: "this step didn't go right, fix it and keep going" and "this step's result is literally what the test exists to verify — if it fails, the test failed, and it shouldn't be quietly patched away" are two entirely different semantics. If a framework conflates the two — say, treating every check item as "retry until it passes" — the failure rate gets artificially suppressed, because assertions that should legitimately go red get silently re-run until they don't. ARTEMIS encodes this distinction directly into the grammar the Planner must follow, rather than relying on a prompt line like "please remember to distinguish acceptance criteria from test assertions" and hoping the model exercises the right judgment — a concrete implementation, at the plan-grammar layer, of the "deterministic decisions go to code, semantic decisions go to the model" principle.
The pre-execution safety net: two layers, each with its own failure taxonomy
In Pro mode, before the Operator executes a potentially destructive action (tap, long press, text input), it passes through a "Safety Net" check, split into two independent modules in the code.
precondition_xml.py — XML-hierarchy validation, covering only four action types: tap, long_press_on, focus_and_input_text, focus_and_clear_text. Its core is a four-way weighted score: W_ID = 0.5 (resource-ID match), W_TEXT = 0.4 (text match), W_BOUNDS = 0.3 (bounding-box match), and W_COORD = 0.3 (whether the original coordinate still falls within the element's bounds) — each signal independently judges whether the target the Operator is about to act on is still where it was in the live UI tree, and a missing signal (say the target has no resource-ID at all) simply drops out of the weighted sum rather than forcing a default score. The docstring explicitly mentions "coordinate self-healing for small drifts," and maps failures into a structured taxonomy: shifted (the element is still there, but has moved), occupied (something else now sits at that position), disappeared (the element is gone entirely) — that three-way split is itself useful signal, giving the Operator more to act on than a generic "validation failed."
precondition_pixel.py — pixel-level VLM fallback, covering a broader action set (click_coordinate, click, long_press, input_text, and more). It takes over when the XML check is skipped due to timeout or fetch failure (the XML_BYPASSED category — the code explicitly does not retry here, it falls straight through to this layer), comparing a before/after screenshot of the target region via a vision model.
The division of labor: use structured information (the XML tree) whenever it's available and reliable, since it's faster and more deterministic than a vision-model call; only fall back to the pricier but broader-coverage pixel comparison when the structured information itself can't be trusted (timeout, empty tree, malformed data). This is the same "Path 1 vs. Path 2" trade-off from post 02's vocabulary showing up again — not at the macro level of "how does the whole test suite locate elements," but at the micro level of "the last check before each individual action fires."
Recovery: no dedicated "repair agent"
When a safety-net check fails, it generates an ExecutionIncident (a dataclass in artemis/agents/validator/incidents.py) carrying a kind (safety_net or exec_error), a category (the shifted/occupied/disappeared taxonomy above), a reason, failure counts, the turn it occurred on, and more. This record is stored in state.open_incident — and then no second agent gets dispatched to handle it.
The implementation is direct: as long as state.open_incident isn't cleared, every turn re-renders this unresolved incident into the Operator's prompt (ExecutionIncidentPromptComponent); once an action the Operator itself takes succeeds, the incident is marked closed, and the very next turn gets a one-time "CLOSED" notice nudging the Operator to reconnect with whatever intent got interrupted by the incident. The README states this design plainly: "recovery is handled by the Operator itself with no separate repair agent."
The trade-off: dispatching a dedicated "repair agent" means synchronizing context between the Operator and the repair agent — what was it trying to do, where did it get stuck, what's already been tried — and that synchronization layer is itself a new failure surface. Letting the same Operator keep deciding with the unresolved incident still visible means the context is naturally continuous — it already knows what it was trying to do, and now it just has an added note: "that action failed last turn, here's why." The cost is a more complex prompt for the Operator to work with, but it avoids maintaining a handoff protocol between two agents.
Core technology: a dedicated Google embodied-reasoning model for locating
Everything so far has been about workflow and validation design. But there's a lower-level technical choice that's easy to miss entirely: ARTEMIS doesn't let the general-purpose model doing planning/execution reasoning also handle element locating on the side — it routes "pointing at coordinates" to a separate, Google-built model specialized in spatial localization.
artemis/agents/object_detector/object_detector.py defines an independent object_detector node, invoked via get_llm(ctx, name="object_detector", is_utils=True) — a model instance completely separate from the Operator/Planner. It receives a screenshot plus a prompt like "Point to the following objects: {labels_str}", and asks the model to return [y, x] coordinates normalized to 0–1000. The config file (artemis/resources/config/artemis.jsonc) is blunt about which model this node needs:
// CRITICAL REQUIREMENT: Standard LLMs (including standard Gemini Flash,
// GPT-4o, Claude 3.7) lack sub-pixel spatial coordinate fine-tuning.
// To achieve accurate [x, y] point and bounding box localization, this
// node MUST use specialized Gemini ER (Embodied Reasoning / Robotics)
// models: e.g., "gemini-robotics-er-2-preview". Non-ER models will fail
// spatial coordinate detection.
"object_detector": {
"model": "gemini-robotics-er-2-preview"
}Gemini Robotics-ER isn't a model ARTEMIS trained itself — it's an "embodied reasoning" model from Google DeepMind's robotics product line. Per Google's own documentation, this model family specializes in "pointing at objects, drawing bounding boxes, tracking trajectories," returning structured spatial coordinates meant to feed directly into a robot control system — originally built so a robotic arm could "see" the physical world well enough to know where to grasp. What ARTEMIS reuses is exactly that "point at the object, give me coordinates" core capability, repurposing it from physical-world object localization to on-screen UI-element localization — fundamentally the same spatial-pointing capability being transplanted, not a UI-specific locating model built from scratch. That's a genuinely different path from a general-purpose VLM (standard Gemini Flash, GPT-4o) "guessing" coordinates purely off its language-model-driven image/text understanding: the former has been specifically fine-tuned for coordinate precision, the latter hasn't — the config comment's line "non-ER models will fail spatial coordinate detection" is not a throwaway warning.
The detect_objects tool (artemis/agents/explorer/perception_tools.py, which calls _run_object_detection) is invoked directly by the Explorer's flash tier, and the pro/ultra tiers reach it indirectly through ask_perception_tool — meaning whichever tier is active, the underlying coordinate fallback can bottom out at this same specialized locating model.
One detail worth stating honestly here. The repository's currently-active top-level config/artemis.jsonc (a different file from the template config bundled under the resources directory) has the object_detector node reconfigured to inherit from default (a local llamacpp-served Qwen3.8) instead. Its comment reads, verbatim in intent: previously forced to Google's native ER model for sub-pixel spatial coordinate fine-tuning; now inherits default per an explicit user choice, accepting reduced coordinate accuracy. What this means: Gemini Robotics-ER is the officially recommended locating model, and very likely the one actually in play when the 99%+ score was measured — but it is not the out-of-the-box default under this repo's current local-development config. That distinction is worth keeping in mind, so as not to conflate "the configuration used to produce the benchmark number" with "what you get by default the moment you clone the repo."
Explorer's three tiers: a knob set by the user, not chosen by the agent
GUI grounding precision directly determines end-to-end success rate, and ARTEMIS addresses this with three Explorer tiers (artemis/agents/explorer/tiers.py). An easy misconception here: these three tiers are not a runtime decision the agent makes for itself about "which tier fits this scenario" — the README states plainly that the tier is "never chosen by the agent" — it's set upfront by the user, either in a config file (config/artemis.jsonc's pro.explorer.mode) or a CLI flag (--explorer-pro-mode):
flash engine=oneshot max 1 turn tools=empty set no caching
— one-shot visual detection on the current screenshot (calls the
object_detector above directly); fastest, no reasoning loop
pro engine=loop max 3 turns tools={ask_perception_tool} no caching
— a short reasoning loop combining UI-tree search, coordinate
audit, and the object detection from the previous section;
the balanced default
ultra engine=loop max 8 turns tools=full perception toolset caching on
— a deep reasoning loop with zooming, OCR, and pixel-level image
processing, for layout-critical searches; the slowest optionThe pro tier's ask_perception_tool deserves its own callout — internally it runs three searches concurrently (asyncio.gather): a text-based UI-tree search, a coordinate-based element hit test, and the object detection covered above, with each result returned independently and then stitched into one unified text description. This differs from the Safety Net's "graduated fallback" logic above — this is parallel redundancy, not sequential degradation — because the grounding step's goal is "gather as much candidate evidence as possible," while the validation step's goal is "rule out the common case as cheaply as possible." Those two goals lead to different cost/benefit judgments about whether to query multiple sources in parallel.
Making the grounding-precision tier an explicit, user-controlled setting rather than something the agent judges dynamically at runtime seems to rest on this reasoning: the precision/latency/cost trade-off is fundamentally a business decision, not a purely technical one — a regression test running in CI might happily trade ultra-tier precision for flash-tier speed, while an offline exploratory debugging session might want the opposite. Leaving that trade-off to an explicit user setting is more predictable than letting the model guess at runtime "how carefully should I look this time," and makes it far easier to pin down which config layer is responsible when costs run over.
Checker: a read-only, zero-side-effect verdict agent
There's one more independent role in Pro mode that's easy to overlook: the Checker. It never participates in execution. The checker.py module docstring defines it as an "independent, zero-side-effect verdict agent," with two entry points sharing one tool loop:
run_checkpoint_check— audits a just-completed milestone'son_completecheck items, using evidence recorded at the moment the milestone was declared complete, deliberately without access to the live screen (a restriction enforced at the tool-table level, not by a prompt instruction telling it not to look)run_final_check— audits the user's original goal plus every declared check item at task exit, where final device state is available
Both entry points only ever read step history, notes, and read-only device probes — no device actions, no note writes, no spawning sub-agents. This "zero side effect" constraint draws a hard line against the Operator: the Operator changes device state, the Checker renders judgment on facts that have already happened without changing anything. Splitting "execution that produces side effects" and "verdict that produces none" into two roles that can't cross into each other's territory is a structural safeguard against a subtle class of bug — a verdict-rendering agent quietly tampering with the evidence it's supposed to be judging — rather than something enforced by convention or a prompt reminder.
Plugging into AI IDEs via MCP: test capability becomes native IDE capability
ARTEMIS exposes 5 tools through mcp_server/ (built on FastMCP): mobile_run_task, mobile_manage_task, mobile_get_device_state, mobile_inspect_trace, mobile_diagnose. The real signature of mobile_run_task (mcp_server/tools/task_runner.py) takes task_desc (the natural-language task description), model (values "Flash"/"Pro", mapping to the two modes above), locked_app_package, app_path, expected_output_desc, device_serial, verification_level, and explorer_mode.
The architectural significance here isn't "there's now an MCP interface" in itself — it's that operating a real device and verifying the result stops being an external test process that needs its own startup and its own maintained context, and becomes a tool call that Claude Code, Antigravity, Codex, and similar AI IDEs can invoke directly. For an AI coding assistant that's actively writing Android feature code, verifying "does this feature I just wrote actually work on a real device" no longer requires switching manually into a separate test framework — running the code and verifying the code can happen in the same conversation, through the same tool protocol.
Behind the 99%+ on AndroidWorld: the causal chain from technique to score
Stringing the mechanisms from every section above back together into one causal chain makes it much clearer where that number actually comes from — rather than settling for a vague "these mechanisms all sound good" impression:
- It starts with locating precision. A large share of AndroidWorld failures trace back to "the tap landed in the wrong place." ARTEMIS routes that specific step to Gemini Robotics-ER, a model specialized for spatial coordinates, instead of letting a general-purpose reasoning model guess — which directly suppresses the error rate at the very top of the task chain, so every downstream mechanism only has to deal with the harder errors left over after that
- The Safety Net turns "locating still went wrong" from a blind failure into diagnosable information. Even when the locating model's initial coordinates were fine, the UI can still change before the action actually executes; the four-way weighted score plus the shifted/occupied/disappeared taxonomy turns a generic "that didn't work" into something specific enough for the Operator to act on
- The recovery mechanism absorbs those diagnosable failures without extra coordination overhead. The unresolved
ExecutionIncidentjust stays in the Operator's own prompt — no repair agent to spawn, no cross-agent context to sync — so the path from failure to recovery is as short as it can be - The verify/assert grammar distinction keeps the retry budget spent where it actually belongs. A fixable acceptance-criterion failure gets a bounded number of retries; an unfixable assertion failure doesn't — avoiding two opposite failure modes: giving up too early on a deviation that was genuinely fixable, or quietly re-running an assertion that should have legitimately failed
- Explorer's tiers turn "how much should this step cost" into an explicit, adjustable knob — long-horizon, high-risk exploratory tasks can dial up to the ultra tier for a better locating success rate, while routine tasks don't have to pay that extra time
These five layers aren't independent bonus points stacked on top of each other — they're one complete chain running from "was the locate correct" through "what happens when it wasn't" to "does this failure deserve a retry." Drop any single link, and the effort spent on the earlier links can get quietly cancelled out downstream. And precisely because the most upstream link in that chain — which locating model gets used — has already been swapped for a lower-precision local model under this repo's current local-development config, the point raised in the previous section is worth repeating: when you read a 99%+ number, it's worth asking "under which configuration was this measured," rather than assuming it's the out-of-the-box behavior.
Summary
- ARTEMIS's stated goal is to "let AI assistants and test suites use real phones like a human." The Flash/Pro split maps onto the bimodal distribution of task complexity — Flash can afford an unlimited turn count only because the history-compression mechanism (chunk → era, one-directional folding) has already solved context bloat
- Pro mode's real graph is cyclic:
convergence_nodeis a pure routing merge point, andexecution_check_nodetreats plan deviation as advisory feedback rather than a blocking rollback - The plan grammar's
verify(a fixable acceptance criterion) vs.assert(an unfixable test assertion, recorded verbatim) distinction is enforced by code parsing structured syntax — not by hoping the model exercises the right judgment from a prompt instruction - The pre-execution safety net splits into XML-structure validation (a four-way weighted score plus a shifted/occupied/disappeared failure taxonomy) and a pixel-level VLM fallback, following the "structured info first, fall back only when unreliable" trade-off; recovery has no dedicated repair agent — the unresolved incident just keeps reappearing in the Operator's own prompt until it succeeds on its own
- The technical point most write-ups skip: ARTEMIS routes element locating to Google's own embodied-reasoning model, Gemini Robotics-ER, rather than letting the general reasoning model guess coordinates on the side — but that model has been swapped out for a lower-precision local model under this repo's current local-development default config, so the 99%+ score more likely corresponds to the recommended configuration with ER enabled
- The 99%+ on AndroidWorld is the output of a complete chain: the locating model suppresses upstream error rate → the Safety Net turns residual errors into diagnosable information → the recovery mechanism absorbs them cheaply → verify/assert keeps the retry budget spent correctly → Explorer's tiers let cost scale to the scenario — and dropping any single link cancels out the effort spent on the rest
Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.
Find more useful knowledge and interesting products on my Homepage