AI Browser Automation: DOM vs Accessibility Tree vs Vision

AI browser automation means a language model operating a web browser: it reads the page, decides what to do, and clicks or types. Agents read the page in one of three ways: the raw DOM (the page's HTML structure), the accessibility tree (a pruned list of roles, names and states), or screenshots with pixel coordinates. Use the accessibility tree by default, the DOM for extraction, and vision as the fallback.

If you want ideas like these working inside your own business, see our AI development and automation service.

If the word DOM is new to you, start with our primer on what the DOM is and why AI agents need it. This piece is about the next choice: which view of the page to feed the model, what it costs, and where it breaks.

How does an AI agent actually operate a browser?

An AI agent operates a browser through a loop: capture the page state, send it to the model, get back an action, execute it, repeat. Underneath sits Playwright, Puppeteer or the Chrome DevTools Protocol. The model never touches the browser. It sees only what your code shows it, and that choice decides cost, speed and failure modes.

1. Raw DOM or HTML. The markup, or a filtered slice of it, as text. The model picks elements by CSS selector, XPath or a numeric index.

2. Accessibility tree. The simplified tree the browser builds for screen readers: each element's role (button, link, textbox), its name and its state, with a short reference ID the agent acts on.

3. Screenshot and coordinates. The model sees an image of the viewport and replies "click at x 640, y 380". Vendors call this computer use.

What does each approach cost in tokens?

A raw DOM is by far the most expensive view, a pruned accessibility snapshot is roughly an order of magnitude smaller, and a single screenshot is cheapest per step but shows only one viewport. We measured this on three public pages with Playwright 1.64 in a 1280 by 800 window, counting text tokens with OpenAI's o200k tokenizer (other vendors' tokenizers will differ somewhat, so treat these as ratios, not invoices).

• Wikipedia's "Document Object Model" article (3,402 elements): rendered HTML 146,227 tokens, accessibility snapshot 23,095 tokens.

• The Playwright MCP repository page on GitHub (4,790 elements): rendered HTML 273,313 tokens, accessibility snapshot 22,283 tokens.

• books.toscrape.com, a scraping practice store (542 elements): rendered HTML 9,832 tokens, accessibility snapshot 8,611 tokens.

For images, Anthropic's vision documentation gives the formula: an image costs width divided by 28, times height divided by 28, in visual tokens, each rounded up. A 1280 by 800 screenshot is 46 times 29, which is 1,334 tokens. Anthropic's computer use docs put typical screenshots at roughly 1,000 to 1,800 input tokens each and warn that long agent loops pile them up quickly.

On heavy pages the raw HTML is 6 to 12 times larger than the snapshot, because most of it is styling, tracking and layout wrappers. On a lean page the gap nearly vanishes. A screenshot looks cheap until you remember it covers only what is on screen, so a long page means scrolling and paying again.

Which is faster: DOM, accessibility tree or screenshots?

Structured text approaches are faster per task because one step can target any element on the page, while vision agents must look, act, and look again after every few actions. Anthropic's computer use tool documentation says plainly that current computer use latency "might be too slow" compared with a human, and recommends it for work where speed is not critical, such as background research and automated testing.

The DOM route has its own latency trap: a huge prompt takes longer to process. That is why DOM based tools prune before sending.

Where does each approach break?

Each view has different blind spots, which is the whole argument for blending them.

Raw DOM failure modes

• Size. Large single page apps produce markup that blows past useful context.

• Shadow DOM. Web components hide their insides in a shadow root. Playwright's locator documentation says its locators pierce shadow DOM by default, but XPath does not, and closed shadow roots are not supported.

• Iframes. Payment fields and consent dialogs often live in separate frames that need explicit handling.

Accessibility tree failure modes

• Bad markup, bad tree. A div with a click handler and no role shows up as a nameless "generic" node.

• Canvas apps. Design editors, maps, charts and games draw to a canvas, so the tree is nearly empty.

• Virtualized lists and cross origin iframes. Anthropic's browser tool docs name these, plus embedded video, as places where the tree often has no useful node.

Vision failure modes

• Coordinates drift. A banner loads, the layout shifts 40 pixels, and the click lands on the wrong button.

• Resolution limits. Oversized screenshots get downscaled or rejected, and small text becomes illegible.

• Only what is visible. Anything below the fold or in a closed menu does not exist until the agent scrolls or opens it.

Failure modes all three share

Dynamic JavaScript, anti bot walls and prompt injection hit every approach. Content rendered after a network call needs a wait condition whichever view you use, and a bot challenge blocks a vision agent as firmly as a DOM agent. Many stores also block AI agents by accident, which we cover in why you probably blocked AI shopping agents. And Anthropic's docs warn that instructions on webpages or inside images can override yours, which is as true for a screenshot as for HTML.

Which real tools use which approach (as of November 2026)?

As of November 2026, most developer tools lead with the accessibility tree or a pruned DOM and add screenshots as a fallback, while the big lab "computer use" products lead with vision. Here is what each vendor's own documentation says.

• Playwright MCP (Microsoft). The Playwright MCP server lets models act "through structured accessibility snapshots, bypassing the need for screenshots or visually tuned models". Actions target element references; coordinate clicking is opt in via a vision capability flag. The README adds that coding agents increasingly prefer the Playwright CLI because full trees are verbose.

• Browser Use. An MIT licensed open source Python library. Its click tool targets elements "by their index", and the use_vision setting decides whether screenshots are always sent, sent on request, or never.

• Stagehand (Browserbase). Natural language act, extract and observe primitives built on trimmed accessibility trees. Its agent documentation describes three modes: dom, hybrid (structure plus coordinates) and cua for vendor computer use models.

• Anthropic. The computer use toolset works from screenshots, mouse and keyboard across a whole desktop. The newer browser use toolset stays in the browser and reads the accessibility tree, page text and screenshots. Its advice is the clearest rule any vendor publishes: prefer element references where the tree is usable, fall back to coordinates where it is not.

• OpenAI. The Computer Using Agent model powered Operator, which became ChatGPT agent in July 2025. OpenAI's API guide now offers a screenshot based computer tool and, for its newest model, recommends letting the model write Playwright code against the live page.

• Google. Project Mariner, its screenshot driven browser agent, was shut down in May 2026, with its features moving into the Gemini API and Gemini Agent. Developers use the Gemini API computer use tool, which lists Gemini 3.8 Flash as its recommended model.

The pattern is convergence from both directions.

A worked example: one task done three ways

The task: on the first page of books.toscrape.com, find the cheapest book rated four stars or more. The page lists 20 books. We ran this against the live site.

The DOM way

Each book is an article element with a paragraph whose class reads star-rating Five, the full title in a title attribute and the price in price_color. The model, or code it writes, filters on the class and sorts by price. Answer: Set Me Free, five stars, £17.46. About 9,800 tokens, one model call, exact.

The accessibility tree way

The snapshot is about 8,600 tokens, with every link and "Add to basket" button tagged with a clickable ref. But the star rating is a paragraph of styled empty icons, and in the snapshot it becomes five empty "generic" nodes. The rating has vanished, and titles are cut to the visible "A Light in the ...". The agent can read prices and click perfectly, but cannot answer without opening each product page or asking for the DOM. Data that is visual only is the classic way tree agents fail.

The vision way

The first 1280 by 800 screenshot shows the header, the sidebar and only 4 of the 20 books. The stars are plainly visible, exactly what the tree missed. Seeing all 20 takes about four screenshots at 1,334 tokens each, roughly 5,300 tokens, plus a model round trip per scroll, and the agent must count small star icons correctly in each image. It works, but slower, and the counting is the weak link.

What the example teaches

The DOM was the cheapest correct route, the tree would be the cheapest for clicking, and vision was the only one that saw the stars unaided. A blended agent, tree for acting plus DOM or a screenshot for the field the tree loses, beats any single view.

How do you choose? A decision rule

Start with the accessibility tree, drop to the DOM when you need data the tree hides, and use vision only when neither structure describes the thing you must touch.

1. Is the target outside a web page (a desktop app, remote desktop, native dialog)? Use a computer use model. Vision is the only option.

2. Mostly clicking and filling forms? Use accessibility snapshots with element references. Cheap, stable across layout shifts, deterministic.

3. Mostly extracting data? Use the DOM pruned to the relevant region, or have the model write selector code once and rerun it without a model.

4. Is the key element a canvas, map, chart or unlabelled widget? Add screenshots for that step only.

5. Will it run thousands of times on one site? Let the agent explore once, save the path as a plain Playwright script, and call the model only when the script breaks.

In the agent builds we see at AISmith, step 5 is where most of the money goes or stays. An agent that rediscovers the same checkout flow every run pays for that discovery every time.

If you want an agent like this on your own stack, see AI agents and automation. For a fully autonomous open source agent that drives a browser and the rest of a computer, read about OpenClaw, the open source autonomous agent.

What this means if you run a website

Agents read your site through these same three views, so accessible markup is agent friendly markup: real buttons, labelled inputs, and visible text for prices, ratings and stock. If you sell online, also check whether agents can reach and buy from you at all with our free agentic commerce checker, and read which agent protocol your store needs if you are deciding how far to go beyond the browser.

Key takeaways

• Agents see a page as raw DOM, an accessibility tree, or screenshots with coordinates.

• On heavy pages we measured raw HTML at 6 to 12 times the accessibility snapshot; a 1280 by 800 screenshot is about 1,334 tokens on Claude.

• Tree for acting, DOM for extracting, vision for canvas, unlabelled widgets and anything outside the browser.

• Data shown only through styling, such as star icons, disappears from the tree.

• As of November 2026 the leading tools blend structure with screenshots.

Frequently asked questions

What is a DOM agent?

A DOM agent is an AI agent that reads a web page's HTML structure as text and acts by selecting elements, usually by CSS selector or an assigned index. It is precise and good at extraction. Its weakness is size, since modern pages can run to hundreds of thousands of tokens unless pruned.

Is the accessibility tree the same as the DOM?

No. The accessibility tree is derived from the DOM by the browser and keeps only what assistive technology needs: roles, names, states and structure. That makes it far smaller, but anything conveyed only through styling or images without labels can be missing from it.

Do AI web automation APIs handle JavaScript heavy sites?

Yes, as long as they drive a real browser rather than fetching raw HTML. Tools built on Playwright or the Chrome DevTools Protocol run the page's JavaScript and read the rendered result. You still need waits for content that loads after network calls, and you still face bot challenges on protected sites.

Is computer use better than DOM based automation?

Not in general. Computer use works anywhere a screen exists, including desktop and canvas apps, but it is slower and its clicks depend on coordinates that shift when layouts move. For ordinary websites, structure first tools are usually faster and more reliable.

What does Playwright MCP send to the model?

By default it sends an accessibility snapshot: an indented list of elements with roles, names and a reference ID for each. The model replies with an action such as click on a given reference. Coordinate based actions are available only if you enable the vision capability.