# Locating UI elements so the automation survives

Every GUI action an agent takes starts with a reference to an element — "click *this*". Which reference you pick, and how you wait for it, decides whether the script still works tomorrow. Nothing else about desktop automation matters as much.

Order the ways to name an element by how rarely they change:

1. **Stable machine identifier** — UI Automation `AutomationId` (WPF `x:Name`, WinForms `AccessibleName`, Win32 control ID). Immune to localization, layout, theming.
2. **Visible text / accessible name** — readable, but breaks on copy edits and language switches.
3. **Class name + index** — breaks when anyone adds a control earlier in the tree.
4. **Coordinates / XPath position** — breaks on DPI change, window move, resolution, resize. Treat as last resort only.

Search narrowly (the target window's subtree, direct children), not from the desktop root, and cache the resolved reference — re-resolving a lookup on every action on a large tree is slow enough to cause timeouts that look like locator failures. If a cached reference throws, re-find it once before declaring the element gone; a re-render stale-references children without changing the tree's shape.

Then wait for a condition, never for a duration. `sleep(3)` before a click is wrong twice over: on a slow machine the app isn't ready and the step throws; on a fast machine you burned three seconds and nobody notices until CI is thirty minutes long. Poll the predicate — visible, enabled, text equals — with a deadline and an interval, and fail with the condition that wasn't met. Same rule when reading results: assert on state (label text, control enabled), not on pixel position.

## When the locator ladder itself fails

The ladder assumes a UIA tree exists. It doesn't for self-drawn controls — `paintEvent`-only widgets, game engines, embedded web content. Those report as one opaque rectangle. The correct move is to scope the fallback to the unreachable control only (screenshot + template match, or OCR, on that region), not to throw away UIA for the whole app the moment one control resists. If you template-match: capture templates at the same display scale as the target (a 125%-scaled screenshot will not match a 100%-captured template, and resizing the template does not rescue it — resampling artifacts destroy the match score), and record the scale factor alongside each artifact, or a postmortem is guesswork.

A second shape trap: dialogs and dropdown popups are usually *separate top-level windows*, not children of the app window. Searching for "the OK button" inside the app's subtree finds nothing while the button sits plainly on screen. Search the desktop root by title for anything that popped up.

## How this typically goes wrong

The failure mode of coordinate locators is not a clean error — it's a *click somewhere else*, or a test that passes for months then dies on a display-scaling change or an OS update that shifts a control twelve pixels. The failure mode of sleep-based waits is "flaky": right code, random death, and teams respond by retrying until green and adding longer sleeps, which hides the real fix (the missing condition wait). The failure mode of text locators is silent wrongness after a copy change: the script clicks *some* button that used to be named "Save". A locator that cannot fail — coordinates that always hit *something*, a sleep that always completes — is not a check; name what you expect to see and assert you saw it.

---

Generalizations collected while vetting community skills for an agent-run Windows sandbox, 2026-09-09: locator ladder and DPI rules from the `windows-desktop-e2e` skill (affaan-m/everything-claude-code, via skills.sh); caching/scope-limiting from `windows-ui-automation` (martinholovsky/claude-skills-generator). Neither is verified against a live app by this page's author; check against current UIA docs before trusting a specific pattern. See skills/windows-desktop-driver for the driver side (threading, kill switch, audit log).