LLM-Driven Automated Testing Series (09): Mobile Automation (V) — When a Generic Agent Framework Sinks Down into a Testing Scenario

What Is MetaGPT's Android Assistant, and a Premise to Correct First

MetaGPT is an open-source multi-agent framework best known as a "Multi-Agent Software Company" — a preset team of roles (Product Manager, Architect, Engineer, QA) that collaborate to turn a one-line requirement into a runnable code project. Buried in its repository is a less-publicized submodule: metagpt/ext/android_assistant/, a testing-agent demo that can autonomously explore or learn from human demonstration to operate Android apps and complete phone tasks.

This piece needs to clear up a premise before going further. Per the series plan, article 09 was supposed to ask "how does reusing MetaGPT's generic Product Manager/Engineer/QA role division for test automation compare — in gains and losses — to the specialized tools covered in articles 05-08." Reading android_assistant/roles/android_assistant.py directly shows this premise doesn't hold: the entire demo defines exactly one custom Role class, AndroidAssistant — no Product Manager, Engineer, or QA roles appear anywhere. What actually gets reused is MetaGPT's lower-level scheduling scaffolding: the Role/Action base classes, Team/Environment orchestration, and the ActionNode structured-output system.

So this piece's three questions need reframing: How exactly does the testing demo reuse MetaGPT's generic scheduling scaffolding? What's the actual relationship to article 08's AppAgent — how far does the README's "referenced" go in practice? And having sunk down into a testing scenario, what did reusing the generic framework's design actually buy, and what baggage came with it that has nothing to do with this specific scenario?

What's Reused Isn't Role Division — It's the Role/Action/Team Scheduling Scaffolding

First, separate "role" from "scaffolding" clearly.

AndroidAssistant(Role)'s __init__ reads stage (learn/act) and mode (manual/auto) from config.extra, combines them into three valid configurations, and calls self.set_actions([...]) once to wire up a different Action list for each:

if stage == "learn" and mode == "manual":
    # choose ManualRecord and then run ParseRecord
    # Remember, only run each action only one time, no need to run n_round.
    self.set_actions([ManualRecord, ParseRecord])
elif stage == "learn" and mode == "auto":
    # choose SelfLearnAndReflect to run
    self.set_actions([SelfLearnAndReflect])
elif stage == "act":
    # choose ScreenshotParse to run
    self.set_actions([ScreenshotParse])
else:
    raise ValueError(f"invalid stage: {stage}, mode: {mode}")

That's the entirety of the "role design" — a single Role class, dispatching to a different Action combination via an if/elif branch. There's no scenario where multiple roles collaborate or message each other; _watch([UserRequirement, AndroidActionOutput]) just lets this one Role respond to user input and its own prior-round output.

What's genuinely, thoroughly reused is the role-agnostic scheduling infrastructure underneath:

• Team.run(n_round)'s driving loop: while n_round > 0: ... await self.env.run(), with each round calling Environment.run(), which gathers all roles concurrently via asyncio.gather(role.run() for role in self.roles.values()). This loop was designed for multiple roles collaborating; here it's driving a single Role and hasn't been simplified for that at all.
• Role.react()'s two reaction modes: RoleReactMode.BY_ORDER lets _think() advance state sequentially (self._set_state(self.rc.state + 1)), matching the [ManualRecord, ParseRecord] case where each Action runs exactly once. When set_actions() is given a single Action ([SelfLearnAndReflect] or [ScreenshotParse]), _think() unconditionally returns state 0, and the same Action instance re-executes every round — iteration count is entirely controlled from the outside by Team.run(n_round=...). This is exactly what the source comment means by "only run each action only one time, no need to run n_round": in the two modes, "round" belongs to different owners — one is managed by the Role itself, the other by the outer Team.
• _observe() overridden for cross-round memory control:

async def _observe(self, ignore_memory=True) -> int:
    """ignore old memory to make it run multi rounds inside a role"""
    newest_msgs = self.rc.memory.get(k=1)
    newest_msg = newest_msgs[0] if newest_msgs else None
    if newest_msg and (RunState.SUCCESS.value.upper() not in newest_msg.content):
        ignore_memory = False
        ...
    return await super()._observe(ignore_memory)

Default is ignore_memory=True, but if the previous round's result wasn't SUCCESS, it flips to False, carrying memory into the next round — this is the concrete, MetaGPT-native implementation of "the same Action runs repeatedly across rounds, but context must be preserved on failure/incompletion," not a mechanism designed fresh for testing.

• ActionNode structured output: SCREENSHOT_PARSE_NODE/SELF_LEARN_REFLECT_NODE/RECORD_PARSE_NODE correspond respectively to "parse a screenshot during the act stage," "reflect during autonomous exploration," and "parse a human-demonstration record" — each is a composite node built from several ActionNode fields (e.g. OBSERVATION/THOUGHT/ACTION/SUMMARY). Calling .fill(context=..., llm=..., images=[...]) returns a pydantic-validated structured dict directly (.instruct_content.model_dump()). This is a fundamentally different parsing approach from AppAgent's re.findall(r"Field: (.*?)$", rsp, re.MULTILINE) single-line regex from article 08 — MetaGPT's generic framework turned "getting the model to output structured fields" into a framework-level capability, and the testing demo gets that benefit for free without writing its own fragile regex.

This is the core judgment to establish before going further: MetaGPT sinking into a testing scenario reuses "scheduling scaffolding + structured output" — domain-agnostic framework-level capabilities — not the "Product Manager/Engineer/QA" domain role design, which simply doesn't appear in this demo.

How Far the AppAgent Borrowing Actually Goes — From Design Philosophy Down to Implementation Details

metagpt/ext/android_assistant/README.md states this plainly:

"The MetaGPT Android Assistant has referenced some ideas and code from the AppAgent project. We thank the developers of the Appagent project."

This isn't a token acknowledgment — direct code comparison shows agreement running from design philosophy down to implementation detail:

Element de-duplication logic is near-line-for-line identical. MetaGPT's traverse_xml_tree() walks clickable/focusable nodes and de-duplicates via a Euclidean-distance threshold (min_dist, default 30) — the same design and even the same default value as AppAgent's traverse_tree() from article 08.

The four-way reflection decision space maps one-to-one. The Decision enum defines BACK/INEFFECTIVE/CONTINUE/SUCCESS, matching AppAgent's four-way classification semantics exactly: INEFFECTIVE generates no documentation; BACK/CONTINUE/SUCCESS all do; BACK/CONTINUE both blacklist the element into useless_list; BACK additionally triggers a real system-back action. This is a full port of the "active pruning" design covered in article 08.

The human-demonstration compound-parameter delimiter is preserved verbatim. manual_record.py records a text-input action as text(3:sep:'hello'):::<uid> (element index, input text, and element uid joined by different delimiters) — the custom ":sep:" delimiter matches AppAgent byte-for-byte.

The documentation-refinement asymmetry is preserved verbatim. parse_record.py (the human-demonstration path) checks extra_config.get("doc_refine", False) and merges/rewrites existing documentation when enabled; self_learn_and_reflect.py (the autonomous-exploration path), on encountering existing documentation, just logs "already exists" and returns FAIL — no refinement logic at all. This is exactly the "human demonstration can refine, autonomous exploration writes once and never touches it again" asymmetry from article 08, down to the naming convention of the config flag.

The grid-fallback mechanism is ported wholesale, including the nine-way compass subdivision. draw_grid() uses the same [120, 180]-pixel divisor-search to size the grid; area_to_xy()'s nine named positions (center plus eight compass directions) match AppAgent's grid fallback exactly — GridOpParam/TapGridOpParam/LongPressGridOpParam/SwipeGridOpParam carry AppAgent's same fallback action space straight into MetaGPT's BaseOpParam hierarchy. And the grid-mode prompt template (screenshot_parse_with_grid_template) still has no text() action — article 08's finding that "grid fallback shrinks the action space itself, not just precision" holds here too.

There are also a few spots that aren't copy-paste but genuine improvements made during the port — worth noting fairly, alongside the bugs and dead weight covered below:

The uid generation scheme adds a parent-node prefix, making it more collision-resistant. AppAgent's get_id_from_element() uses only resource-id or class_width_height as the primary key. MetaGPT's same-named function has identical logic, but traverse_xml_tree() adds an extra step:

parent_prefix = ""
if len(path) > 1:
    parent_prefix = get_id_from_element(path[-2])
...
if parent_prefix:
    elem_id = parent_prefix + "_" + elem_id
if add_index:
    elem_id += f"_{elem.attrib['index']}"

Prepending the immediate parent node's own id, and optionally appending the raw XML index attribute — this means the same resource-id recurring under different parent containers (e.g. every row of a list sharing one resource-id) won't be misidentified as the same element. It's a strictly more rigorous element-identity scheme than AppAgent's.

Structured output replaces fragile regex, as covered above — ActionNode.fill() swapping out AppAgent's single-line regex parsing is a genuine improvement made possible by leaning on MetaGPT's own framework capability, not a straight copy-paste.

So the accurate weight of "referenced" is: the core design philosophy and key mechanisms (four-way reflection, pruning, grid fallback, refinement asymmetry) are fully inherited, while two specific spots — element-identity resolution and response parsing — got targeted hardening by leaning on what MetaGPT's own framework already provides.

The Cost of Sinking Into a Generic Framework: One Unfixed Crash Bug, and a Whole Set of Models Loaded for Nothing

Reusing a generic framework isn't free. Reading the code turns up two specific, verifiable costs — not subjective judgment, confirmed directly from source.

First: a crash bug the maintainers flagged themselves, but never fixed. Inside self_learn_and_reflect.py's run_reflect():

logger.info(
    f"reflect_parse_extarct decision: {op_param.decision}, "
    f"elem_list size: {len(self.elem_list)}, ui_area: {self.ui_area}"
)
# TODO here will cause `IndexError: list index out of range`.
#  Maybe you should clink back to the desktop in the simulator
resource_id = self.elem_list[int(self.ui_area) - 1].uid

This TODO comment is original to the repository — not something added locally during debugging. It's an explicit admission from the maintainers: if the model's returned ui_area exceeds elem_list's actual length, this line throws IndexError and crashes the entire task outright, with the suggested workaround being "manually switch the simulator back to the home screen." This differs in kind from article 08's swipe_precise() coordinate bug in AppAgent, which nobody flagged and had to be discovered by reading source — that one was "silently wrong, nobody noticed"; this one is "maintainers noticed, wrote a TODO, and never fixed it." Both point to the same underlying conclusion: bounds-checking on model output is a widely under-engineered corner across open-source autonomous-exploration agents, even when maintainers are aware of the gap.

Second: every run unconditionally loads three models the testing path never actually uses. AndroidExtEnv.__init__ contains this line:

def __init__(self, **data: Any):
    super().__init__(**data)
    device_id = data.get("device_id")
    self.ocr_detection, self.ocr_recognition, self.groundingdino_model = load_cv_model()
    ...

load_cv_model() unconditionally loads an OCR-detection model, an OCR-recognition model, and a GroundingDINO icon-localization model (paired with CLIP). These three models are used only by user_click_icon(), user_click_text(), and user_open_app(). Checking every EnvActionType value constructed by the four Action classes (manual_record.py/parse_record.py/screenshot_parse.py/self_learn_and_reflect.py) turns up only SYSTEM_TAP/USER_INPUT/USER_LONGPRESS/USER_SWIPE/SYSTEM_BACK/USER_SWIPE_TO — none of which route to those three OCR/GroundingDINO-dependent methods. In other words, every single run of this testing demo pays the startup cost of loading three models that this code path will never call.

This is the most direct sample of "generic framework sinking into a specialized scenario" costing something real: AndroidExtEnv was clearly built for a broader action space (tapping by icon, tapping by text), and some other, testing-unrelated agent inside MetaGPT presumably uses that capability — but this testing demo inherits the entire environment class, and with it the startup cost, with no way to load only the part it actually needs. On the flip side, AndroidExtEnv.user_swipe_to() deserves a positive note: its implementation is f"{self.adb_prefix_si} swipe {start[0]} {start[1]} {end[0]} {end[1]} {duration}" — coordinate order is entirely correct, and it does not reproduce AppAgent's swipe_precise() X/Y-coordinate-swap bug from article 08. The port isn't "copy-paste preserving every bug" — at least this one spot was implemented correctly.

A Technical-Selection Matrix Across Articles 05-09

The mobile-automation mini-series closes here. Rather than repeating details each article already covered, here's the five projects from 05-09 placed side by side across the same dimensions.

What's worth reading here isn't the listing itself — it's that the 09 and 08 rows are nearly identical, which is exactly the confirmation of this piece's core judgment: across the dimensions that actually determine a testing agent's capability boundary — how it perceives, how it grounds, how it handles failure, how it accumulates knowledge — MetaGPT's Android Assistant offers no design independent of AppAgent. Its distinct value sits entirely on a different axis: taking an already-validated design and dropping it into a more generic multi-agent scheduling framework, which buys framework-level infrastructure (structured output, cross-round memory management) but also inherits costs unrelated to the testing scenario itself (like the unconditional model loading above). This is the most honest difference between "a specialized tool" and "one application of a generic framework": every design choice in the former serves the current scenario directly, while much of the latter's design is inherited — both the benefits and the costs weren't custom-tailored for this particular use case.

Summary

1. The planning doc's premise of "reusing Product Manager/Engineer/QA role division" doesn't hold — the android_assistant demo has exactly one custom AndroidAssistant Role; what's actually reused is MetaGPT's lower-level Team/Environment/Role/Action/ActionNode scheduling scaffolding, plus the BY_ORDER reaction mode and cross-round memory control — all domain-agnostic framework capabilities.
2. The README states plainly that this demo "referenced ideas and code from AppAgent," and direct source comparison confirms this isn't a token claim: element de-duplication logic, four-way reflection (BACK/INEFFECTIVE/CONTINUE/SUCCESS), useless_list active pruning, the ":sep:" compound-parameter delimiter, the grid-fallback mechanism, and the human-demonstration/autonomous-exploration refinement asymmetry are all ported wholesale; two spots — uid generation (adding a parent-node prefix for stronger collision resistance) and response parsing (ActionNode structured output replacing fragile regex) — received genuine improvements during the port.
3. Sinking into the generic framework carried two verifiable costs: an IndexError crash risk in self_learn_and_reflect.py that maintainers flagged themselves but never fixed, and AndroidExtEnv unconditionally loading three OCR/GroundingDINO models on every run that this testing path never actually invokes — the latter a textbook sample of inheriting a generic environment class while using only a fraction of its capability.
4. user_swipe_to()'s coordinate implementation is correct and does not reproduce AppAgent's swipe_precise() X/Y-swap bug — the port wasn't mindless copy-paste; at least this one spot was implemented correctly.
5. The 05-09 technical-selection matrix shows MetaGPT's Android Assistant is nearly identical to AppAgent across perception input, grounding method, fault-tolerance design, and knowledge-base persistence — the dimensions that actually determine a testing agent's capability boundary. Its distinct contribution is confined to "dropping into a more generic multi-agent scheduling framework" — both the benefit (structured output, cross-round memory) and the cost (unrelated model-loading overhead) come from that layer, not from the testing scenario itself.
6. This closes out the mobile-automation mini-series (05-09): the five projects represent five distinct paths — dual-model collaboration (ARTEMIS), system-layer abstraction (DroidRun), self-trained coordinate models (Mobile-Agent-v3), parser-tree-plus-vision dual input (AppAgent), and generic-framework reuse (MetaGPT) — none is a strict best choice; it depends on whether a team can afford to train its own grounding model, how complete the target app's accessibility tree is, and whether integration with a larger multi-agent system is actually needed.

Check out PrimeSkills — a curated marketplace for AI agents and skills, all validated in real enterprise workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage