One Open Source Project a Day (No. 224): ARTEMIS — Google's Open-Source Mobile AI Automation Framework, Letting AI Assistants Operate Phones Like a Human

Introduction

"A test script breaks the moment a button moves — an AI that actually understands the interface doesn't."

This is the 224th article in the "One Open Source Project a Day" series. Today's project is ARTEMIS.

Mobile UI automation testing has long lived with a fundamental tension: traditional scripts based on element IDs or coordinates are fast, but extremely brittle — a small layout tweak breaks test cases en masse, and maintenance cost grows linearly with how fast the app iterates. Vision-model-driven approaches are more robust, but slow and expensive, making them a poor fit for large-scale regression testing.

ARTEMIS is Google's attempt to find a balance between these two extremes. It turns natural-language instructions — like "open Settings, find the Battery option, tell me the current level" — into reliable Android automation, while offering two execution modes: one optimized for speed on routine deterministic tasks, another optimized for accuracy and verifiability on complex, long-running ones. More notably, it plugs directly into Claude Code, Antigravity, Codex, and other AI IDEs via MCP, giving AI coding assistants the ability to operate real phones.

8.1k Stars, Apache-2.0, built in Python, achieving 99%+ task completion on the AndroidWorld benchmark.

What You Will Learn

• How ARTEMIS turns natural-language instructions into Android automation
• The architectural differences between the Flash and Pro execution modes, and when to use each
• Multi-modal target localization: combining the accessibility hierarchy, OCR, and vision models
• How it integrates with Claude Code and other AI IDEs via MCP
• How the Python SDK plugs into existing automated testing frameworks

Prerequisites

• Basic understanding of Android automation testing concepts (ADB, Accessibility services)
• Familiarity with Python async programming (asyncio)
• Optional: basic understanding of MCP (Model Context Protocol)

Project Background

What It Is

ARTEMIS's official positioning: "ARTEMIS turns natural-language instructions into reliable Android automation" — letting AI assistants and test suites operate real phones like a human would.

One clarification worth making up front: this is not a security testing or fuzzing tool. It's an intelligent agent system focused on mobile UI automation testing and everyday task execution, squarely in the AI-driven mobile testing/automation framework space.

Team and Background

• Organization: Google
• License: Apache License 2.0
• Primary language: Python (requires Python 3.12+)
• Code provenance: the project notes it includes source code developed by Minitap, Inc., building on the open-source mobile-use project

Project Stats

• ⭐ GitHub Stars: 8,100+
• 🍴 Forks: 770+
• 👀 Watchers: 71
• 📄 License: Apache-2.0
• 🏆 Benchmark: 99%+ task completion on AndroidWorld (covering 20+ apps, 100+ multi-step tasks)

What It Does

The Problem It Solves

Traditional element-ID/coordinate-based automation scripts:
  UI layout shifts slightly → element lookup fails → tests break en masse
  ↑ Fast, but extremely brittle, high maintenance cost
 
Pure vision-model-driven approaches:
  Every step relies on a vision model to read the screen → accurate but slow
  ↑ Good robustness, but slow and expensive, hard to scale
 
ARTEMIS's approach:
  Natural-language instruction → multi-modal localization (element index first,
  vision/coordinates as fallback)
  ↓ Flash mode: routine deterministic tasks, 3-5 sec/step, fast reactive loop
  ↓ Pro mode: complex long-running tasks, 15-40 sec/step, plan-execute-verify workflow
  ↑ Choose the right mode for task complexity, balancing speed and reliability

Use Cases

1. Mobile app end-to-end test automation

   • Describe test steps in natural language instead of maintaining brittle element-locator scripts

2. Bug reproduction and diagnosis

   • Combined with Logcat log and screenshot capture, quickly reproduce and pinpoint issues

3. Exploratory stability testing

   • Pro mode supports long-running, continuously monitored exploratory testing

4. CI/CD-integrated automated testing

   • Integrate via the Python SDK into pytest and similar frameworks, folded into continuous integration pipelines

5. AI IDE-driven real device operation

   • Let Claude Code, Antigravity, and other AI coding assistants directly operate a phone to verify functionality

Quick Start

Clone and one-command start:

git clone https://github.com/google/artemis.git && cd artemis
 
# macOS/Linux one-command start (auto-detects and installs ADB, scrcpy, FFmpeg, etc.)
./start.sh

Run a task directly via CLI:

uv run artemis run "Open Settings, find Battery and tell me current level" --profile flash

Install MCP integration for an IDE:

# Install for Antigravity
uv run artemis mcp --install antigravity
 
# Install for all supported IDEs (including Codex)
uv run artemis mcp --install all

On first task execution, ARTEMIS installs an "Artemis Accessibility Helper" service on the device — it only reads the screen layout locally and uploads no data, and can be pre-installed, checked, or uninstalled via command.

Core Features

1. Four Ways to Use It

2. Dual Execution Modes: Flash and Pro

3. Python SDK Integration Example

uv add "artemis-client @ git+https://github.com/google/artemis.git#subdirectory=packages/artemis-client"

import asyncio
from artemis_client import ArtemisClient
 
 
async def main():
    client = ArtemisClient(
        "http://artemis-host:8000",
        device_serial="emulator-5554",  # optional: target specific device serial
        default_profile="flash",  # "flash" (fast reactive) or "pro" (deep reasoning)
    )
 
    result = await client.run(
        "Open System Settings, go to 'Battery', verify battery percentage is displayed, and check for any crash dialogs.",
    )
 
    assert result.succeeded, f"Test failed: {result.error or result.status}"
    print(f"✅ Test Passed! Device: {result.device_serial} | Trace ID: {result.trace_id}")
 
 
if __name__ == "__main__":
    asyncio.run(main())

The SDK claims "zero runtime dependencies" — ADB, the agent, models, and image processing all stay on the device host. It supports strongly typed Pydantic structured outputs and assertions, and plugs directly into frameworks like pytest.

4. MCP Integration Configuration

Using Claude Desktop as an example (claude_desktop_config.json):

{
  "mcpServers": {
    "artemis": {
      "command": "/path/to/artemis/.venv/bin/python",
      "args": ["-m", "mcp_server"],
      "cwd": "/path/to/artemis"
    }
  }
}

The docs also recommend mounting a behavior rules file (mcp_server/rules.md) for AI agents, covering proactive exploration before coding, Flash/Pro routing strategy, and latency compensation logic.

A Deeper Look

Flash vs. Pro: A Pragmatic Speed/Reliability Trade-off

ARTEMIS doesn't use a single mode for every task — it explicitly splits into two architecturally distinct execution paths:

Flash mode (reactive):
  Observe screen → think about next step → execute action → loop
  ↑ Single-model loop, no planning, no safety net, no final report
  ↑ Loop count is unbounded by default (history is compressed, not truncated)
  ↑ Good for short, deterministic tasks like "open Settings and flip a switch"
 
Pro mode (plan-and-verify):
  Planner drafts a Markdown task plan (with milestones and verification items)
      ↓
  Operator runs a "Safety Net" check before each action
      ↓ (checks against the live UI tree first, then pixel fallback)
  Hits a blocker → opens an "execution incident," Operator handles recovery autonomously
      ↓
  Checker (read-only) verifies plan checkpoints, supports a final review before exit

The core insight behind this design: mobile automation task complexity is bimodal. A large share of tasks are deterministic short tasks — "open a page and confirm a state" — where a simple reactive loop is fast and sufficient. A minority are complex workflows that need multi-step planning, fault tolerance, and long-running monitoring, where only a multi-agent plan-execute-verify structure can guarantee reliability. Using one architecture for both extremes wastes something either way.

Fault-Tolerant Design in Multi-Modal Target Localization

ARTEMIS's element localization mechanism reflects a "prefer precision, degrade gracefully" fault-tolerance philosophy:

Localization priority:
  1. Accessibility Hierarchy — most precise, the go-to for standard Android UI
  2. OCR text recognition — handles cases where the hierarchy tree lacks information
  3. Vision model recognition — fallback, handles custom Canvas/Compose/Flutter UIs

Standard Android widgets are usually exposed accurately by accessibility services, making that the fastest and most reliable localization method. But more and more apps render their UI with Compose, Flutter, or custom Canvas drawing, which is unfriendly to accessibility services — that's when OCR and vision models step in. This layered fallback strategy avoids the waste of "use the most expensive method for everything" while also avoiding the fragility of "just fail outright when standard widgets aren't recognized."

Shared History Compression: Memory Management for Long-Running Tasks

Flash and Pro share the same history compression mechanism, and this detail is worth noting:

Problem: long-running tasks (especially Pro's 100+ step workflows) accumulate
a large volume of screenshots and action logs
  ↑ Stuffing them directly into context blows past the window limit
  ↑ Simple truncation loses important early information
 
ARTEMIS's approach:
  Old screenshots → replaced with visual summaries (keep semantics, drop pixel detail)
  Completed steps → compressed into retrievable history chunks
  ↑ The agent can retrieve past context when it needs to look back,
    instead of always carrying the full history

This design lets Flash mode run stably long-term even with no loop cap, and is also the memory-management foundation that makes Pro's 100+ step workflows possible.

How It Compares to Similar Mobile Automation Approaches

ARTEMIS's differentiation is combining "natural-language driven + multi-modal fault-tolerant localization + native AI IDE integration" into one tool, using its dual-mode design to avoid the dilemma of "either too slow with pure vision models, or too brittle with fixed scripts."

Project Links and Resources

Official Resources

• 🌟 GitHub: https://github.com/google/artemis
• 📄 License: Apache License 2.0
• 🐛 Issues: GitHub Issues

Related Resources

• AndroidWorld — Google Research's mobile agent benchmark project, and ARTEMIS's evaluation benchmark
• Model Context Protocol — The standard protocol underlying ARTEMIS's integration with Claude Code and other AI IDEs
• mobile-use — The open-source project (from Minitap, Inc.) that ARTEMIS is built on

Summary

Key Takeaways

1. Natural-language-driven mobile automation: replaces brittle element-locator scripts with instructions, lowering test maintenance cost
2. Dual Flash/Pro mode design: choose a fast reactive loop or a plan-execute-verify workflow based on task complexity, balancing speed and reliability
3. Multi-modal fault-tolerant localization: accessibility hierarchy first, OCR and vision models as graceful fallbacks, adapting from standard widgets to custom Canvas UIs
4. Native MCP integration: gives Claude Code, Antigravity, Codex, and other AI IDEs the ability to directly operate real Android devices
5. 99%+ on AndroidWorld: reliability validated on a standard mobile agent benchmark

Who This Is For

• Mobile app testing teams: want to escape brittle element-locator scripts and reduce maintenance cost from UI changes
• AI application developers: want AI coding assistants to be able to operate real Android devices for feature verification or bug reproduction
• CI/CD pipeline maintainers: need to seamlessly fold mobile automation testing into existing test frameworks and continuous integration
• Exploratory/stability testing teams: need long-running, continuously monitored mobile testing capability

One-Line Verdict

ARTEMIS doesn't pick a side between "fast but brittle" and "robust but slow" — it splits the tension apart directly with its Flash/Pro dual-mode design, which may be a pragmatic middle answer as mobile automation moves toward AI-driven approaches.

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage