LLM-Driven Automated Testing Series (10): Self-Healing Locators — Where Heuristic Scoring Hits the Ceiling of Semantic Understanding

Healenium is the most established open-source project in self-healing locators, and its README claims it 'leverages machine learning' — but reading the code directly reveals the core mechanism is a weighted scoring formula over DOM tree similarity, with hardcoded constants for weights and thresholds, nothing to do with machine learning. More notably, its 2026 addition of an AI-based XPath generation endpoint is gated behind a hard-coded exception: 'you must have a paid hlm-ai service.' This article first cracks open exactly how Healenium's scoring formula works, then examines how MarketSquare/robotframework-selfhealing-agents — a genuinely LLM-based self-healing project — designs its multi-agent architecture, and finally answers the question the planning doc raised: where exactly does the tradeoff between healing success rate and false-positive rate actually bite.

·14 min read·AI Engineering
LLMAutomated TestingSelf-HealingLocatorsAI Engineering

What Self-Healing Locators Solve, and a Claim Worth Debunking First

The most fragile part of Web UI test automation isn't assertion logic — it's element locating. IDs, classes, and DOM hierarchy break the moment a page gets redesigned, and the test doesn't fail because the business logic is wrong; it fails because "this button's selector can't be found anymore." Self-Healing Locators exist to solve exactly this: when a locator fails, the system tries to automatically find a new way to locate what looks like the same element, instead of letting the test go red and waiting for a human to fix the selector.

The oldest and highest-starred open-source project in this space is Healenium (healenium/healenium-web, 201 stars, last commit 2026-09-07, still actively maintained). Its README contains the line: "leverages machine learning to allow Selenium tests to self-heal." The first thing this article does is verify that claim directly against the source — and the conclusion is: the claim isn't false in a marketing sense, but it's misleading in a technical sense. Healenium's core self-healing mechanism isn't machine learning at all — it's a weighted-sum heuristic formula whose weights are hardcoded floating-point constants. The basic elements of machine learning — training data, model inference — are entirely absent from this formula.

Let's set out the three questions this article answers up front: How exactly does Healenium's "so-called machine learning" heuristic scoring compute a score, and where's the ceiling? What form does its 2026 AI addition actually take, and why does the code hard-lock it behind a paywall? By contrast, how does a genuinely LLM-based open-source project — MarketSquare/robotframework-selfhealing-agents — use "what the element does" rather than "where the element is" to relocate it?

Healenium's Scoring Formula: Tree Similarity + Levenshtein Distance, No Machine Learning Anywhere

Healenium's self-healing pipeline has two layers: SelfHealingEngine (in the main healenium-web repository) intercepts a locator failure during test execution, parses the current page's DOM into a Node tree (via JsoupHTMLParser), and hands it to a separately published artifact — tree-comparing (Maven coordinate com.epam.healenium:tree-comparing, pinned at version 0.4.14 in healenium-web's pom.xml) — to do the tree matching and scoring.

The matching logic lives in PathFinder.findScoresToNodes(): it first uses LCSPathDistance (longest common subsequence) to compare "the failed element's historical path" against "the paths of every leaf node on the current page," finding the structurally most similar candidate paths, then calls HeuristicNodeDistance.distance() on every node along those candidate paths to compute a score. I verified this scoring function directly against decompiled bytecode (from tree-comparing-0.4.14.jar on Maven Central — the floating-point constants in the constant pool match the source exactly), and the formula is this:

private static final double POINTS_FOR_TAG = 100.0D;
private static final double POINTS_FOR_ID = 50.0D;
private static final double POINTS_FOR_CLASS = 40.0D;
private static final double POINTS_FOR_VALUE = 30.0D;
private static final double POINTS_FOR_OTHER_ATTRIBUTE = 30.0D;
// maximumScore = 350.0
 
double score = LCSDistance / curPathHeight * 100.0;   // structural path similarity
if (tagsMatch) score += 100.0;
if (idPresent) score += 50.0 * levenshteinScore(id1, id2, 0.3);
score += 30.0 * levenshteinScore(innerText1, innerText2, 0.3);
score += classIntersectionRatio * 40.0;
score += avgLevenshteinOfOtherAttributes * 30.0;
return score / maximumScore;   // normalized to [0, 1]

calculateLevenshteinScore() internally uses Apache Commons Text's LevenshteinDistance, converting the edit distance between two strings into a [0,1] similarity value; the threshold parameter (0.3 or 0.75) determines how many edits before two strings are judged dissimilar. There is nothing "learned" anywhere in this formula — the weights (100, 50, 40, 30) are hardcoded constants, not parameters fitted from data; the similarity algorithm is string edit distance, not word embeddings or semantic vectors. If an element's id changes from login-btn to signin-btn, the Levenshtein distance is small, the score is high, and it's judged as "the same element." But if it changes to something semantically identical but character-wise unrelated, like submit-auth, the edit distance maxes out and the score collapses — even though to a human this is obviously still the same login button. This is exactly the ceiling of heuristic scoring: it measures surface-level string and structural similarity, not functional semantic similarity of the element — which is precisely why the gap the planning doc identified ("LLM semantic-match self-healing: relocate by 'what this element does' rather than 'where it is'") is a real technical gap, not a marketing invention.

The final score is filtered by HealingService against a threshold from config (score-cap, defaulting to .6 in application.conf, .55 in the test config); candidate nodes above the threshold are sorted descending by score and tried one at a time, converted into either a CSS selector or an XPath, and validated via driver.findElements() to check that it uniquely locates exactly one element — the first candidate that validates successfully becomes the healing result. This is a fully deterministic rules engine: fast, explainable, no external API calls required — but its boundary is equally clear: as soon as whatever changed falls outside the comparable dimensions of "tag, id, class, text, attributes" — for example, an entire component gets re-implemented while its meaning stays the same — this formula has no way to reason about it.

The 2026 AI Endpoint: A Path Directly Blocked by a Paywall

There's a method in RestClient.java that's entirely separate from the scoring engine above:

public String getXpathSelector(Node node, String sessionId) {
    HttpRequest request = new HttpRequest(HttpMethod.POST, "/selectors/xpath");
    ...
    HttpResponse response = aiServiceExecute(request);
    if (HTTP_NOT_FOUND == response.getStatus()) {
        throw new RuntimeException("[Get Xpath Selector] Compatibility error. You must have a paid hlm-ai service.");
    }
    ...
}

This method POSTs DOM node information to a separate aiServiceUrl (config key hlm.ai.url, defaulting to http://localhost:6565 in application.conf), completely decoupled from the scoring engine — HealingService.createXPathFromElement() only calls it when useXPath(engine) evaluates true (i.e., selector-type is configured as xpath), as an alternative path to CSS selector construction. This piece of code confirms two things by itself: first, that even the Healenium team recognizes pure rule-based scoring isn't enough, and has opened a dedicated channel for "letting an AI service generate XPath"; second, that this channel is explicitly gated behind a paywall — a 404 response triggers an exception whose message is stated with no softening at all: "You must have a paid hlm-ai service." This hlm-ai service itself is a closed-source commercial component, outside the open-source repository's scope — what model it uses, how it does semantic understanding, is entirely unverifiable from the outside.

This detail is more valuable than any judgment about whether a high-star tool's AI marketing is accurate: it shows that even the oldest, longest-maintained open-source project in this space recognizes that pure rule-based scoring has hit a wall and needs semantic understanding — but the team chose to make that capability closed-source and commercial, rather than open and community-built. This isn't the same pattern we'll see later in article 11 (where a high-star tool's "intelligence" marketing often wraps a non-LLM mechanism): Healenium isn't misrepresenting anything — it's honestly acknowledging its limitation, then turning the fix into a paid feature.

Genuine LLM-Based Healing: The Orchestrator-Locator Dual-Agent Architecture of MarketSquare/robotframework-selfhealing-agents

MarketSquare/robotframework-selfhealing-agents (27 stars, Apache-2.0, created 2025-04, last commit 2026-02, MarketSquare being the official organization for the Robot Framework ecosystem) is one of the rare open-source implementations in this space where the source code confirms real LLM API calls end to end. It's a Robot Framework Listener plugin that handles locator failures via a two-tier agent architecture:

Tier 1: OrchestratorAgent, built on PydanticAI, is responsible for deciding "is this failure actually a locator problem." Its decision logic itself delegates to a downstream locator_agent.is_failed_locator_error() call — for example, in the Selenium case, checking whether the error text matches patterns like "with locator" ... "not found" — and if it's not a locator failure (say, an assertion failure or an unrelated timeout), it returns a NoHealingNeededResponse directly, avoiding wasted tokens on a repair flow that wouldn't help. Only once a failure is classified as a locator problem does the request get routed to a pydantic-ai Agent, with output_type=[self._get_healed_locators] registering "heal the locator" as a callable tool — going through a genuine LLM tool-call loop.

Tier 2: BaseLocatorAgent (and its subclasses SeleniumLocatorAgent/BrowserLocatorAgent) is where the actual semantic locating happens, and it supports two modes, switched by the config flag use_llm_for_locator_generation (default True):

  • Pure LLM generation mode: the DOM tree at failure time, the error message, the failed locator string, and the list of already-tried-but-still-failing locators are all packed into a prompt, and the model is asked to output 3 new locator candidates directly. The system prompt is explicit: "Using the elements in the DOM at failure time, suggest 3 new locators... Make sure you do not suggest a locator that is on that list." — this is pure language-model semantic reasoning, with no rule-based scoring involved at all.
  • DOM-tool-generation-plus-LLM-selection mode: deterministic code (_dom_utility.get_locator_proposals()) first enumerates candidate locators based on DOM structure, and then a separate selection_agent picks the "best" one from that candidate list. This path is "deterministic generation + LLM judgment" combined — compared to pure LLM generation, this lowers the risk of hallucinating an invalid selector, but also narrows the space of semantic understanding, since the LLM can only pick from an already-enumerated candidate set — it can't produce an answer outside that set.

The tradeoff between these two modes maps directly onto a concrete answer to the planning doc's question about "the tradeoff between self-healing success rate and false-positive rate": pure LLM generation mode has broader coverage (the model can invent locating strategies that a candidate enumeration might have missed), but is more prone to hallucinating syntactically valid but non-matching selectors; the DOM-tool-generation mode is more conservative and more reliable, but its ceiling is capped by whatever a deterministic enumeration is capable of finding. The project chose to expose this switch to the user rather than deciding for them — that design decision itself is an acknowledgment that neither path can achieve both "high coverage" and "low false-positive rate" at the same time.

There's another engineering detail worth recording in the response-validation layer: generation_agent.output_validator runs an additional deterministic filtering pass over the model's output — _sort_locators() uses is_locator_unique() to rank candidates that "uniquely locate exactly one element" ahead of the rest, and _filter_clickable_locators() specifically filters for genuinely clickable elements when the failure came from a click/tap/select-type keyword. If no candidate passes validation at the end, the code raises ModelRetry, triggering pydantic-ai's built-in retry mechanism rather than returning a potentially wrong locator outright. This is different from simply "trusting whatever the model says" — the LLM's semantic judgment still has to pass through a layer of deterministic code validation before it's accepted. This splits "semantic understanding" and "structural correctness" into two separate responsibilities, rather than expecting a single model call to get both right at once.

The config layer (SelfhealingAgents/utils/cfg.py) confirms this is a genuine multi-provider architecture: orchestrator_agent_provider/locator_agent_provider both support openai/azure, the default model is gpt-4o-mini, and there are separate hard cost constraints — request_limit (default 5) and total_tokens_limit (default 6000). These hard constraints are themselves a direct answer to the concern "will LLM-based healing run out of control and burn tokens" — every healing request has an explicit token ceiling, and it terminates outright when exceeded rather than retrying indefinitely.

Where the Tradeoff Between Healing Success Rate and False-Positive Rate Actually Bites

Looking at both projects' mechanisms side by side, the planning doc's question — "when does self-healing end up masking a real UI regression bug" — has a concrete answer: neither heuristic scoring nor LLM-based semantic judgment can tell the difference between "a harmless implementation detail changed" and "a genuine functional UI regression." Both are ultimately answering "is there something on the new page that looks like the old element" rather than "is this change consistent with expected product behavior." Healenium's scoring formula might assign a similarly high score to a pure id rename and to a case where a button was accidentally removed and replaced with a visually similar but functionally different new button. LLM-based semantic judgment, when facing "a new button appeared on the page with similar wording but completely different logic," can be misled by the same kind of surface-level semantic similarity.

This means the reasonable boundary for a self-healing mechanism isn't "push judgment accuracy toward 100%" — it's treating every healing result as a signal that needs to be logged and reviewed by a human, not an automatic fix that can be fully trusted and let a test go green silently. robotframework-selfhealing-agents's reporting system (reports/report_generator.py and its family of report_types) specifically produces healed_files_report/diff_files_report — this design itself is an acknowledgment that "healing succeeded" doesn't mean "there's no problem," and instead records every healing event as a change worth reviewing, so a human can judge whether a real UI regression is hiding behind that fix. This mirrors Healenium's backlight-healing config option (defaulting to true, which highlights in the test report which elements were recovered through self-healing) — two projects with completely different technical approaches converged on the same engineering decision: self-healing results must be traceable and reviewable. That convergence says more about the consensus boundary of this field than any single scoring formula or prompt design could.

Summary

  1. Healenium's README claim of "leverages machine learning" is a marketing overstatement — the core healing mechanism (HeuristicNodeDistance.distance()) is a weighted-sum formula with hardcoded weights (TAG=100, ID=50, CLASS=40, VALUE=30), using Levenshtein edit distance for similarity — nothing resembling the train/infer paradigm of machine learning. This was verified by decompiling the bytecode constant pool of tree-comparing-0.4.14.jar on Maven Central against the mirrored source.
  2. Healenium's 2026 addition, RestClient.getXpathSelector(), is a separate AI-based XPath generation channel — the code throws "You must have a paid hlm-ai service" directly on a 404 response. Even the oldest open-source project in this space acknowledges that pure rule-based scoring can't meet the demand for semantic understanding — but chose to make that capability a closed-source paid service rather than open-source it.
  3. MarketSquare/robotframework-selfhealing-agents is a genuinely LLM-based open-source healing implementation, using a two-tier architecture — an orchestrator agent (deciding whether healing is needed) and a locator agent (generating or selecting new locators) — that supports both "pure LLM generation" and "DOM enumeration plus LLM selection" modes. The two modes trade off coverage against false-positive rate differently, and the project leaves that choice to the user rather than baking in a single default answer.
  4. The project doesn't trust LLM output directly — its output_validator layer runs deterministic code to verify candidate locators' uniqueness and clickability, raising ModelRetry to trigger a retry rather than returning a potentially wrong result outright when validation fails. This splits semantic judgment and structural correctness verification into two separate responsibilities.
  5. Neither heuristic scoring nor LLM-based semantic judgment can distinguish "a harmless implementation change" from "a genuine functional regression" — two projects with entirely different technical approaches both converged on the same response: backlight-healing highlighting on one side, and structured healed/diff reports on the other, treating every healing result as a signal requiring traceability and human review rather than an automatic fix that can be fully trusted.

Check out PrimeSkills — a curated marketplace for AI agents and skills, all validated in real enterprise workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

← Back to Blog