What Mobile-Agent-v3 Is and What Problem It Solves
Mobile-Agent-v3 is an open-source cross-platform GUI agent framework from Alibaba's Tongyi Lab (X-PLUG team), built around the self-trained GUI-Owl model — a multimodal model series trained specifically for GUI perception, grounding, and end-to-end operation, claiming SOTA results across benchmarks like ScreenSpot-v2, ScreenSpot-Pro, OSWorld-G, Android World, and OSWorld. The Mobile-Agent-v3 framework itself instantiates GUI-Owl into several specialized roles (Manager/Executor/ActionReflector/Notetaker), forming a multi-agent pipeline capable of planning, executing, reflecting, and note-taking.
Compared to the previous two articles' ARTEMIS and Mobilerun, the fundamental difference isn't in how multi-agent responsibilities are split — it's in the philosophy of the perception layer: ARTEMIS and Mobilerun both hand a screenshot to a general-purpose conversational model (Gemini/GPT/Claude) and hope it also nails coordinate grounding as a side effect. Mobile-Agent-v3 does the reverse — it first trains a model dedicated to GUI grounding, then fits that model into every role of the multi-agent pipeline. This article won't repeat the "what" — it goes straight into three questions: How does the plan-execute-reflect loop actually work? What did the self-trained grounding model buy, and at what cost? What engineering compromises, visible directly in the code, were made going from paper to open-source tool?
The Four-Role Pipeline: Manager Plans, Executor Acts, ActionReflector Judges Outcomes, Notetaker Records
The main control loop lives in run_instruction inside mobile_v3/run_mobileagentv3.py, a fixed-cap for step in range(max_step) loop (max_step=25 by default). Each iteration calls the four roles in sequence:
Screenshot → error detection → Manager (plan/replan) → Executor (pick action) → execute action
→ ActionReflector (judge A/B/C) → if success and Notetaker enabled, take notes → next stepAll four roles inherit from the same abstract base class BaseAgent (mobile_v3/utils/mobile_agent_e.py), with just two methods:
class BaseAgent(ABC):
@abstractmethod
def get_prompt(self, info_pool: InfoPool) -> str:
pass
@abstractmethod
def parse_response(self, response: str) -> dict:
passEvery role shares a single InfoPool dataclass as the only state container — no message queue, no dedicated inter-agent communication protocol, just "read from and write to the same shared state":
@dataclass
class InfoPool:
instruction: str = ""
summary_history: list = field(default_factory=list)
action_history: list = field(default_factory=list)
action_outcomes: list = field(default_factory=list)
error_descriptions: list = field(default_factory=list)
important_notes: str = ""
error_flag_plan: bool = False
plan: str = ""
completed_plan: str = ""
progress_status: str = ""
err_to_manager_thresh: int = 2This design is plain, but that plainness is exactly what makes the whole pipeline fully legible: there's no state transition hidden inside a message bus or event system — for every field in InfoPool, you can grep the code directly to see who writes it and who reads it.
Manager (the Manager class) is responsible for planning and deciding whether the task is complete. Its system prompt states the role directly: "You are an agent who can operate an Android phone on behalf of a user. Your goal is to track progress and devise high-level plans to achieve the user's requests." If Manager decides the task is done, it marks the Plan output with "Finished," and the main loop breaks out of the loop when it detects that word.
Executor (the Executor class) works from the first subgoal of the current Plan and picks one action from a fixed atomic-action table — it does no planning, only single-step action selection.
ActionReflector (the ActionReflector class) judges whether the previous action met its expectation, taking before/after screenshots as input:
prompt += "The two attached images are phone screenshots taken before and after your last action. \n"
...
prompt += "A: Successful or Partially Successful. The result of the last action meets the expectation.\n"
prompt += "B: Failed. The last action results in a wrong page. I need to return to the previous state.\n"
prompt += "C: Failed. The last action produces no changes.\n\n"This A/B/C three-way classification is the pivot of the entire fault-tolerance mechanism — not a simple success/failure binary. B (landed on the wrong page) and C (nothing changed at all, e.g. swiping when already at the bottom) are two distinct failure modes that call for different recovery strategies.
Notetaker (the Notetaker class) is only invoked when the action is judged A (success) and the user passed --notetaker True. It accumulates information from the current screenshot relevant to the user's goal into info_pool.important_notes, which gets read back into Manager's next planning prompt — this is "working memory" accumulated during task execution, not knowledge from a training phase.
The Fault-Tolerance Threshold: Two Consecutive Failures Before Escalating to Manager
This is the single most worth-unpacking piece of Mobile-Agent-v3's fault-tolerance design. At the start of every step, the main loop checks the most recent err_to_manager_thresh (default 2) entries of action_outcomes:
info_pool.error_flag_plan = False
err_to_manager_thresh = info_pool.err_to_manager_thresh
if len(info_pool.action_outcomes) >= err_to_manager_thresh:
latest_outcomes = info_pool.action_outcomes[-err_to_manager_thresh:]
count = 0
for outcome in latest_outcomes:
if outcome in ["B", "C"]:
count += 1
if count == err_to_manager_thresh:
info_pool.error_flag_plan = TrueOnly when the most recent two outcomes are both B or C (not one B mixed with one A) does error_flag_plan get set to True. Once that flag is set, Manager's next planning prompt gets an extra dedicated "potentially stuck" block:
if info_pool.error_flag_plan:
prompt += "### Potentially Stuck! ###\n"
prompt += "You have encountered several failed attempts. Here are some logs:\n"
k = info_pool.err_to_manager_thresh
recent_actions = info_pool.action_history[-k:]
recent_summaries = info_pool.summary_history[-k:]
recent_err_des = info_pool.error_descriptions[-k:]
for i, (act, summ, err_des) in enumerate(zip(recent_actions, recent_summaries, recent_err_des)):
prompt += f"- Attempt: Action: {act} | Description: {summ} | Outcome: Failed | Feedback: {err_des}\n"Notably, even once this flag is set, the instruction Manager receives is only "carefully assess the current status... think step by step about whether the overall plan needs to be revised" — the framework itself doesn't force a replan; whether to scrap the original plan or just retry a different action is left entirely to the model's own judgment. This is a form of lightweight tiered escalation: at the Executor level, small retry noise is tolerated (a single failure doesn't bother Manager); only sustained failure pulls decision-making back up to the planning layer — matching a human operator's intuition of "try it myself twice first, only call the supervisor if it still fails." This is a different order of design than ARTEMIS's dedicated Checker node doing post-hoc validation, or Mobilerun's ActionReflector-style reflection — Mobile-Agent-v3's threshold mechanism is plainer, but because it's fully exposed in the body of the loop in run_mobileagentv3.py, it's the easiest fault-tolerance logic to read end-to-end across all three articles.
There's also a small short-circuit optimization in the main loop: if the previous action was marked "invalid" due to a parsing failure, the current step skips Manager and retries Executor directly — avoiding letting a pure formatting error (not an actual operation failure) bother the planning layer.
The Action Space: Six Atomic Actions, Coordinates Only — No Index-Based Grounding
The action table Mobile-Agent-v3 actually uses at runtime is much narrower than the full constant list (16+ entries) defined in new_json_action.py. The six that are actually presented in Executor's prompt and executed by the main loop live in ATOMIC_ACTION_SIGNITURES_noxml:
ATOMIC_ACTION_SIGNITURES_noxml = {
ANSWER: {"arguments": ["text"], ...}, # answer the user's question
CLICK: {"arguments": ["coordinate"], ...}, # tap (x, y)
LONG_PRESS: {"arguments": ["coordinate"], ...}, # long-press (x, y)
TYPE: {"arguments": ["text"], ...}, # type text
SYSTEM_BUTTON: {"arguments": ["button"], ...}, # system button (Back/Home)
SWIPE: {"arguments": ["coordinate", "coordinate2"], ...}, # swipe
}Compared to Mobilerun (previous article), also on the general-VLM path — Mobilerun's action set includes both click(index) (structural index-based click) and click_at(x, y) (coordinate click), and by default blocks coordinate-based tools unless specific conditions are met. Mobile-Agent-v3 takes the opposite strategy — there's no index-click option at all; coordinate clicking is the only way to locate elements. The logic behind this tradeoff is direct: GUI-Owl is a model trained specifically for pixel-level visual grounding. Index-based clicking depends on structured information like the accessibility tree, which is frequently missing or unreliable in real-world scenarios (custom-rendered widgets, games, WebViews). Once you've already invested in training a model with strong coordinate-grounding ability, there's no need to maintain a structural-index fallback path. This is the direct fingerprint the "self-trained grounding model" choice leaves on action-space design.
Coordinate execution lands in the main loop:
if action_object['action'] == "click":
controller.tap(action_object['coordinate'][0], action_object['coordinate'][1])
elif action_object['action'] == "swipe":
controller.slide(action_object['coordinate'][0], action_object['coordinate'][1],
action_object['coordinate2'][0], action_object['coordinate2'][1])GUI-Owl Outputs Absolute Pixel Coordinates — The Key Difference From Qwen-VL and Seed-VL
There's an easy-to-miss but information-dense line in the README:
"If the model you are using outputs relative coordinates from 0 to 1000, such as Seed-VL or Qwen-VL-2 or Qwen-VL-3, please set
--coor_type "qwen-vl"... If the model you are using outputs absolute coordinates. such as Qwen-VL-2.5 or GUI-Owl, please do not set coordinate mapping."
GUI-Owl outputs absolute pixel coordinates aligned to the device's actual resolution by default, rather than relative coordinates normalized to 0-1000 that need conversion, the way Qwen-VL-2/Seed-VL do. The corresponding coordinate-mapping code only runs when coor_type != "abs":
if coor_type != "abs":
if "coordinate" in action_object:
action_object['coordinate'] = [int(action_object['coordinate'][0] / 1000 * width), int(action_object['coordinate'][1] / 1000 * width)]
if "coordinate2" in action_object:
action_object['coordinate2'] = [int(action_object['coordinate2'][0] / 1000 * width), int(action_object['coordinate2'][1] / 1000 * height)]Worth pointing out here is a real code defect (not speculation — confirmed by directly reading the code): the Y-component of the first line's coordinate conversion also uses width instead of height — the coordinate field's Y-axis conversion is wrong, while the coordinate2 line right below it is correct (it does use height). This bug only triggers when the user explicitly passes a non-"abs" --coor_type (i.e., switches to a Qwen-VL-style relative-coordinate model) — it's never hit under the default GUI-Owl absolute-coordinate mode. This lines up with the structural read that "absolute coordinates are this framework's first-class citizen; the relative-coordinate compatibility path was added later and has thinner test coverage."
Direct comparison across all three articles: ARTEMIS pairs a general VLM with a dedicated Gemini Robotics-ER model to reinforce coordinate grounding. Mobilerun relies entirely on the general model's own visual understanding, with no extra investment. Mobile-Agent-v3 takes a third path — instead of adding a grounding-reinforcement layer on top of a general model, it bakes "output pixel coordinates directly usable for clicking" into the model's own capability from the training stage. Three routes, three different cost structures: ARTEMIS trades an extra model call for precision; Mobilerun has zero extra cost but precision fully depends on the main model; Mobile-Agent-v3 front-loads grounding precision into model training, which actually makes its runtime the simplest of the three — one model call, coordinates ready to use.
Platform Abstraction: One Abstract Base Class, Android via ADB, HarmonyOS via HDC — But HarmonyOS's Input Implementation Has a Real Bug
mobile_v3/utils/controller.py defines a minimal abstract base class:
class Controller(ABC):
@abstractmethod
def get_screenshot(self, save_path): pass
@abstractmethod
def tap(self, x, y): pass
@abstractmethod
def type(self, text): pass
@abstractmethod
def slide(self, x1, y1, x2, y2): pass
@abstractmethod
def back(self): pass
@abstractmethod
def home(self): passAndroidController implements everything via adb shell input ... commands; HarmonyOSController implements everything via hdc shell uitest uiInput ... commands — both implementations have identical method signatures, and the caller (run_mobileagentv3.py) doesn't care which platform is underneath, only depending on the Controller interface. This is a standard driver-abstraction pattern, the same engineering approach as Mobilerun's DeviceDriver abstraction from the previous article.
But reading harmonyos_controller.py's type method turns up a real implementation defect:
def type(self, text):
text = text.replace("\\n", "_").replace("\n", "_")
for char in text:
if char == ' ':
command = self.adb_path + f" shell uitest uiInput keyEvent 2050"
subprocess.run(command, capture_output=True, text=True, shell=True)
elif char == '_':
command = self.hdc_path + f" shell uitest uiInput keyEvent 2054"
...The branch handling the space character builds command using self.adb_path — but HarmonyOSController doesn't have an adb_path attribute at all; only self.hdc_path gets assigned in __init__. Which means — the moment text to be typed on HarmonyOS contains a space, this line throws an AttributeError — every other character-type branch correctly uses self.hdc_path. This isn't speculation; it's confirmed by directly reading the code: the other five methods (tap/slide/back/home/get_screenshot) all correctly use self.hdc_path — only this one branch in type, handling spaces, has the wrong variable name. A bug like this usually signals that the HarmonyOS path's test coverage isn't as thorough as the Android path's — most likely a leftover from copy-pasting the Android implementation and forgetting one substitution, with no test case covering "typing text that contains a space."
How GUI-Owl Is Called: OpenAI-Compatible Protocol, Temperature Locked to 0
mobile_v3/utils/call_mobile_agent_e.py's GUIOwlWrapper class wraps GUI-Owl as a standard OpenAI-compatible client call:
class GUIOwlWrapper(LlmWrapper, MultimodalLlmWrapper):
RETRY_WAITING_SECONDS = 20
def __init__(self, api_key, base_url, model_name, max_retry=10, temperature=0.0):
self.max_retry = min(max_retry, 10)
self.temperature = temperature
self.model = model_name
self.bot = OpenAI(api_key=api_key, base_url=base_url, timeout=30)temperature defaults hard-coded to 0.0 — all four roles (Manager/Executor/ActionReflector/Notetaker) share the same GUIOwlWrapper instance, and thus the same "determinism-first" sampling policy. This contrasts with the previous article's Mobilerun, which configures per-role temperatures (Manager 0.2, Executor 0.1) — Mobile-Agent-v3's current open-source version hasn't gone down the "role-level differentiated sampling" path; the four roles differ in responsibility but call the same model with the same sampling parameters.
Image preprocessing calls qwen_vl_utils.smart_resize(), adaptively scaling screenshots within a min/max pixel budget (MIN_PIXELS=3136, MAX_PIXELS=10035200) before base64 encoding — the standard image-preprocessing pipeline for the Qwen model family, consistent with GUI-Owl's Qwen-VL training lineage.
From Paper to Tool: Engineering Fingerprints Left in the Code
Reading through the files one by one, mobile_v3/ doesn't have many "unfinished" traces, but there is at least one — in MobileUse.parameters's action enum list, the answer entry is followed directly by a # todo comment:
"enum": [
"key", "click", "long_press", "swipe", "type",
"answer", # todo
"system_button", "open", "wait", "terminate",
],The comment doesn't state what specifically is pending. Combined with the more concrete finding below, it suggests this file's completeness genuinely lags behind the files that actually drive execution in mobile_v3/:
The tool class in function_call_mobile_answer.py is a pure schema definition with no real execution logic behind it — this file defines a MobileUse(BaseTool) class, registered into Qwen-Agent's tool system via @register_tool("mobile_use"), whose description and parameters fields fully describe ten action types (including key/open/wait/terminate, four more than the six actually supported by the main loop). But every concrete action method that call() dispatches to — _key/_click/_long_press/_swipe/_type/_answer/_system_button/_open/_wait/_terminate — is a single-line raise NotImplementedError(), with none actually implemented:
def _click(self, coordinate: Tuple[int, int]):
raise NotImplementedError()
def _type(self, text: str):
raise NotImplementedError()That means calling this class is guaranteed to raise. Its value is limited to giving the Qwen-Agent framework a description/parameters schema (the if __name__ == "__main__" demo at the bottom of the file only prints the function_call structure parsed by NousFnCallPrompt — it never actually calls mobile_use.call() and hits these empty methods). The real execution logic is implemented separately in run_mobileagentv3.py's main loop; the two have similar action naming and parameter structure but aren't the same code path — this schema looks more like a documentation artifact prepared for an alternate integration route ("wire GUI-Owl into the Qwen-Agent framework"), not code Mobile-Agent-v3's main pipeline actually depends on.
Notetaker's prompt has task-specific rules left over from benchmark evaluation runs mixed in, for example:
if "transactions" in info_pool.instruction and "Simple Gallery" in info_pool.instruction:
prompt += "### Guideline ###\nYou can only record the transaction information in DCIM..."This kind of hard-coded branch targeting a specific benchmark task (a specific AndroidWorld question) mixed into general framework code is a common trace of academic-evaluation code — special-case logic added to score well on a benchmark, left uncleaned before open-sourcing. This differs from Mobile-Agent-E (Mobile-Agent-v3's predecessor project), which kept a persistent "tips/shortcuts" cross-task experience mechanism: Mobile-Agent-E has dedicated ExperienceRetrieverShortCut/ExperienceRetrieverTips roles that persist cross-task experience to files and retrieve it before the next task starts. Mobile-Agent-v3's mobile_agent_e.py (despite the file name carrying over from the old project) has zero tips/shortcuts-related fields at all — this cross-task experience-persistence mechanism was removed entirely in v3; Notetaker now only accumulates notes within a single task, no longer reusing them across tasks.
Evaluation code lives in its own separate directories: os_world_v3/ and android_world_v3/ each contain their own evaluation scripts (run_guiowl.sh/run_ma3.sh), physically separated from mobile_v3/ (real-device runtime) — a sensible piece of project organization, since benchmark scoring and real-device deployment are naturally two separate entry points.
A Brief Note: v3.5 Folds Multi-Role Capability Into the Model Itself
The repository also contains a newer Mobile-Agent-v3.5 directory, worth a brief mention. v3.5's corresponding model is GUI-Owl-1.5, and the README's key description is:
"Multi-agent ready: Serves both as a standalone end-to-end agent and as specialized roles (planner, executor, verifier, notetaker) within the Mobile-Agent-v3.5 framework."
But actually reading mobile_use/run_gui_owl_1_5_for_mobile.py (v3.5's only current mobile run script, 287 lines) reveals: this script has no separate Manager/Executor/ActionReflector classes at all — just a single loop inside main(), repeating: take a screenshot, call the model once, parse a <tool_call>-formatted output, execute the action, log history. Planning, execution, and verification are no longer separate explicit role calls in the pipeline — they're described as already "converged into the model itself," meaning the model performs all of them within a single inference pass, and the framework no longer needs a separate model call per role.
This is a direct contrast with v3: v3 is "one model, called multiple times, playing different roles" (four calls per step); v3.5's mobile script is "one model, called once, roles internalized" (one call per step). Whether this represents the Mobile-Agent series' future direction — abandoning explicit multi-agent orchestration and moving that complexity into model training — can currently only be inferred from the way this one run script is written. There's no separate v3.5 Manager/Executor orchestration code in the local repository to compare against, and the evaluation code (android_world_v3.5/) would need to be separately checked to confirm whether it validates this direction — this should be treated as a direction worth watching, not a settled conclusion.
Summary
- Mobile-Agent-v3 defines four roles through a shared
BaseAgentabstract base class — Manager (planning), Executor (action selection), ActionReflector (A/B/C outcome judgment), Notetaker (note-taking) — all reading from and writing to the sameInfoPoolstate, with no dedicated inter-agent communication protocol. - The fault-tolerance mechanism is a concrete, plain, fully legible threshold design: two consecutive B/C judgments before escalating to Manager for replanning; a single failure lets Executor retry on its own without bothering the planning layer.
- The action space has only six atomic actions (answer/click/long_press/type/system_button/swipe), and coordinate-based localization is the only method — there's no index-click option at all, a direct design consequence of investing in the self-trained GUI-Owl grounding model.
- GUI-Owl outputs absolute pixel coordinates aligned to device resolution by default, unlike the 0-1000 relative coordinates from Qwen-VL-2/Seed-VL; there's a coordinate-conversion bug in the code (Y-axis incorrectly uses
width) that only triggers under the relative-coordinate compatibility mode. - Android and HarmonyOS share the same
Controllerabstract base class, but HarmonyOS'stypemethod incorrectly references a nonexistentself.adb_pathwhen handling the space character — a real code defect that directly causes anAttributeError, pointing to insufficient test coverage on the HarmonyOS path. - Three articles, three different grounding-precision cost structures: ARTEMIS is "general model + dedicated grounding model reinforcement," Mobilerun is "entirely dependent on the general model's own capability," Mobile-Agent-v3 is "grounding precision baked into the model at training time." Signs from the newer v3.5 in the repository suggest planning/execution/verification role capabilities are also being converged into a single model inference pass — but this direction can currently only be observed from one run script and doesn't yet constitute complete evidence.
Check out PrimeSkills — a curated marketplace for AI agents and skills, all validated in real enterprise workflows. No fluff, just what actually works.
Find more useful knowledge and interesting products on my Homepage