An AI agent can navigate a marketplace like Amazon by exploring the page source, capturing network requests, and writing its own plugin definitions. The agent receives a discovery prompt, launches a browser via MCP with an authenticated session, and begins inspecting the DOM for elements that match the goal. It cross-references selectors across multiple page loads to identify stable ones, distinguishing auto-generated class hashes from intentional semantic markers like data-testid attributes and aria labels. The agent also captures the network layer, mapping API endpoints that fire when buttons are clicked. This dual-layer approach gives resilience that neither layer alone provides. The critical insight is that narrow, specific prompts produce dramatically better results than broad exploration. A prompt asking the agent to find the Prime filter gives it a testable hypothesis and validation criteria. A prompt asking it to explore the marketplace leads to indiscriminate cataloging of irrelevant elements, filling the context window with noise. The agent can only self-validate when it knows what success looks like, which is why scoped discovery prompts that name specific skills and imply validation criteria are the key to building plugins that survive site redesigns.
#4752: How AI Agents Map Websites to Build Stable Plugins
An AI agent explores DOM and network layers to build self-healing browser plugins.
Episode Details
- Episode ID
- MWP-4931
- Published
- Duration
- 24:36
- Audio
- Direct link
- Pipeline
- V5
- TTS Engine
-
chatterbox-regular - Script Writing Agent
- deepseek-v4-pro
AI-Generated Content: This podcast is created using AI personas. Please verify any important information independently.
Downloads
Transcript (TXT)
Plain text transcript file
Transcript (PDF)
Formatted PDF with styling
You Might Also Like
Never miss an episode
New episodes drop daily — subscribe on your favorite platform
New to the show? Start here#4752: How AI Agents Map Websites to Build Stable Plugins
Daniel's been deep in the weeds building Claude Code plugins this week, and he's hit on a pattern that I think is worth pulling apart. He's building tools that let an AI agent navigate marketplaces — Amazon, Ivory, that sort of thing — not by writing the navigation logic himself, but by pointing the agent at the website and telling it to figure out the mapping on its own. Give it an authenticated browser session, tell it what skills you want — find the Prime filter, find the Next Day delivery button — and the agent explores the page source, captures network requests, and writes its own plugin definition using the divs and classes it discovers. And it works.
It does work. That's the part that sounds like it shouldn't.
Right. The question he's asking is why it works, and more specifically, how the way you prompt the agent during that discovery phase determines whether you get a reusable tool or a pile of brittle selectors that break the next time someone pushes a CSS update. He wants to get into the mechanics of agent-led UI discovery, why specificity in the discovery prompt is the difference-maker, and how to build plugins that survive a site redesign.
So we're talking about the difference between telling an agent "go look at this website" and telling it "find the elements that let a user filter search results by delivery speed, map them to stable selectors, and build me a skill that clicks the right one." The difference in output quality between those two prompts is enormous, and the reasons why are genuinely interesting.
Let's start with what this pattern actually looks like in practice.
The core loop is this: you've got an AI agent — Claude, in Daniel's case — connected to a browser via the Model Context Protocol. That browser integration is the critical piece. It's not a headless scraper fetching raw HTML. It's a real browser rendering JavaScript, handling state changes, maintaining an authenticated session. The agent can see the page the way a user sees it, but it can also read the source, inspect the DOM tree, and watch the network tab. All three layers at once.
So it's got X-ray vision on a page it's also interacting with normally.
The agent receives a discovery prompt — here's the marketplace, here's where deliveries go, here are the skills I want. It navigates to the relevant pages. It starts inspecting the DOM, looking for elements that match the goal. A checkbox with a label that says Prime. A button that says Next Day Delivery. It reads the class names, the aria labels, any data attributes. And crucially, it does this across multiple page loads and multiple navigation states. It clicks the filter, watches what changes in the DOM, watches what network request fires off. Then it synthesizes all of that into a plugin definition — a set of named skills with selectors and action sequences.
And the network request capture is doing something different from the DOM inspection. It's mapping the API layer underneath the UI.
Right — the agent sees that clicking the Next Day Delivery button triggers a GET request to some endpoint with a query parameter like speed equals next underscore day. That's a second, independent way to understand what the button does. Even if the button's visual design changes, the API endpoint might stay the same. Or vice versa. Having both layers gives you resilience that neither layer alone provides.
So this isn't web scraping in the traditional sense.
Not even close. Traditional scraping is a human writing a fixed extraction script — go to this URL, find this CSS selector, pull the text. If the selector changes, the script breaks and a human has to fix it. Selenium-style automation is the same thing with more clicking. The agent here isn't executing a pre-written script. It's doing exploratory development. It's deciding which elements matter based on a high-level goal, inferring which selectors are stable by comparing across page loads, and then writing the automation code itself. The human never inspects a single div.
That's the part that feels new. The agent is doing the development work, not just running someone else's script. But that also means the quality of the development work depends on how well you frame the task. To understand why specificity matters so much, we need to get into the mechanics of how the agent discovers and maps UI elements.
Let's walk through the pipeline step by step, because each stage has a failure mode that the prompting strategy needs to address. Stage one: the agent receives the discovery prompt. If the prompt is good, it includes context about the marketplace, the delivery location, and a narrow list of desired skills — filter by Prime, filter by Next Day delivery. Stage two: the agent launches a browser via the MCP server using the user's authenticated session. This is the enabler. Without authentication, the agent sees the logged-out version of Amazon, which doesn't even show Prime filters. With it, it sees exactly what the user sees.
And it sees the fully rendered page, not just the initial HTML payload.
That's the key technical advantage. A lot of modern websites are single-page apps — the initial HTML is basically an empty shell with a JavaScript bundle. React or Vue or whatever framework renders the actual UI in the browser. If you're just fetching HTML, you get none of that. The browser MCP integration gives the agent the post-render DOM, including every element that JavaScript generated. It can also interact with the page — click, scroll, type in a search box — and observe the resulting DOM mutations and network calls. That interactivity is what lets it test hypotheses about what elements do.
So stage three is where it actually starts inspecting. It's looking at the DOM and trying to figure out which selectors are stable.
This is the selector discovery problem, and it's the hardest part of the whole pipeline. The agent is looking at a page full of elements, most of which have class names that look like gibberish — css dash one a b c two d, or some hash that changes on every deploy. Modern front-end frameworks generate these automatically. React uses something called CSS-in-JS that produces unique class hashes. If the agent latches onto one of those, the plugin breaks the next time the site rebuilds its asset bundle. Could be tomorrow, could be in an hour.
So how does it distinguish the stable ones from the ephemeral ones?
By cross-referencing. The agent loads the same page multiple times — different sessions, different navigations — and compares the DOM. Selectors that stay the same across loads are candidates for stability. Selectors that change are flagged as ephemeral. It also looks for semantic markers that developers intentionally put in for testing and accessibility. Data dash test id attributes are the gold standard — they're specifically designed to be stable across UI changes. Aria labels and roles are nearly as good, because they're tied to accessibility requirements and tend not to change casually. Semantic HTML structure — like a button element inside a form with a specific action — is more stable than a div with a class name.
Daniel mentioned the Amazon Prime filter as a concrete example. Walk me through what the agent actually finds there.
So the agent navigates to a search results page on Amazon while authenticated with Prime. It's looking for the filter that narrows results to Prime-eligible items. It inspects the checkbox element. The checkbox might have a class like a dash checkbox dash label — that's a pattern that appears across Amazon's UI and has been stable for years. But right next to it, there might be a wrapper div with a class like css dash one a b c two d that's clearly auto-generated. The agent observes that a dash checkbox dash label appears on every page load, across multiple search queries, while the hash-based class changes. It learns to prefer the stable one. It also captures the network request that fires when you click that checkbox — something hitting an endpoint that includes a parameter like prime equals true. That confirms it's found the right element and gives it a second way to verify the interaction.
And it's doing this for every skill you asked for.
For every skill in the scoped prompt. That's the key constraint. If you told the agent "explore Amazon and tell me what you find," it would try to catalog every interactive element on every page it visits. Search bar, cart button, language selector, department dropdown, wishlist heart, every single filter in the sidebar — there are dozens. Most of them are irrelevant to a plugin that just needs to filter by delivery speed. The agent wastes its context window on noise, and the resulting plugin definition is bloated with low-confidence mappings it never actually tested against the goal.
So the narrow prompt isn't just about efficiency. It's about accuracy.
It's about whether the agent can validate its own work. When the agent knows the skill is supposed to filter search results by Prime eligibility, it can test candidate selectors against that goal. It clicks the element and checks whether the results actually changed. It can ask "did the number of displayed items drop?" or "did the Prime badge appear on the remaining products?" That validation step is what separates a high-confidence mapping from a guess. A broad prompt doesn't give the agent a clear success criterion, so it can't self-validate. It just catalogs everything and hopes.
There's a deeper point here about the relationship between goal specificity and selector quality. The agent isn't just picking elements that look clickable — it's picking elements that achieve a specific outcome.
And the outcome is defined by the prompt. "Find the Prime filter" gives the agent a testable hypothesis. "Explore the marketplace" gives it an unbounded observation task. The first produces maybe fifteen candidate elements, twelve of which are useful. The second produces two hundred candidates, three of which are useful, and the agent can't tell you which three because it never tested them.
That ratio is brutal. Three out of two hundred.
And those three might be the wrong three. The agent might map a visually similar but functionally different element — like a "Sponsored" badge that happens to use the same CSS class as the Prime indicator on a different page. Without a specific goal to validate against, it has no way to catch that error.
So the agent can do the discovery. The mechanics work. But the quality of what it finds depends entirely on how you point it. That's where prompting strategy comes in.
Let's talk about what makes a discovery prompt good. Daniel's prompt works because it does three things. First, it specifies the marketplace and the delivery context — this is Amazon, deliveries go to this address. Second, it names the desired skills — filter by Prime, filter by Next Day delivery. Third, it implies the validation criteria — the skill should actually narrow the results to items that match the delivery constraint. That's a scoped discovery prompt. It constrains the agent's search space to the relevant DOM subtrees and API calls.
And what happens when you get lazier than that?
The failure pattern is what I just described — the agent explores indiscriminately. It lands on the homepage and starts cataloging the navigation bar. It follows a link to Today's Deals and catalogs that page. It finds the footer and catalogs the help links. None of this helps build a delivery filter plugin. The context window fills up with irrelevant DOM dumps, and by the time the agent reaches the actual search results page where the Prime filter lives, it's either out of capacity or out of attention. The plugin it produces is either incomplete or wrong.
There's a sub-agent pattern here that Daniel alluded to. Instead of one agent doing all the discovery, you spawn sub-agents for different parts of the plugin.
This is where the pattern gets really powerful. You can have one sub-agent map the search flow — how do you enter a query and get to a results page. Another sub-agent maps the filter flow — what are the available filters and how do you activate them. A third maps the checkout flow. Each sub-agent gets its own scoped prompt, its own browser session, and its own narrow goal. They work in parallel, and each one's context window stays focused on a single task. The parent agent then assembles the plugin from the sub-agent outputs.
That parallelization also means you can re-run individual sub-agents when something breaks without redoing the whole discovery.
If Amazon changes their checkout flow but leaves the search filters alone, you only need to re-discover the checkout mapping. The filter sub-agent's work is still valid. That's a maintenance strategy, not just a development strategy.
Which brings us to the brittleness problem. Any third-party UI change can break the plugin. Daniel flagged this up front. What can the agent actually do about it?
A few things. The first is selector preference — always prefer semantic selectors over positional ones. A data dash test id attribute is more likely to survive a redesign than "the third div inside the second list item." The agent should also include fallback selectors. If the primary selector fails, try the aria label. If that fails, try a specific class pattern that's been stable historically. The plugin definition can include a ranked list of selectors for each element, not just one.
You can build in validation.
This is the self-healing pattern. On each run, before the plugin executes any skill, it validates its selectors. It checks whether the expected element exists on the page. If the selector fails, the plugin doesn't just crash — it can either fall back to an alternative selector, or trigger a lightweight re-discovery pass. The re-discovery is scoped to just the broken element, not the whole plugin. The agent re-navigates to the relevant page, re-inspects the DOM around where the element should be, and finds the new selector.
The maintenance cost shifts from "developer manually inspects the DOM and updates the code" to "agent re-discovers the selector automatically."
That's the trade that makes this pattern viable long-term. The traditional approach — a developer writing Selenium tests or Puppeteer scripts — has the same brittleness problem. Every UI change breaks something. The difference is who fixes it and how fast. With the agent-driven approach, the fix can happen automatically on the next plugin run. The user might not even notice the site changed.
That's a pretty fundamental shift. The brittleness hasn't gone away — the UI can still change at any time — but the cost of fixing it just dropped to near zero.
Provided the discovery prompt was specific enough to produce quality mappings in the first place. A self-healing plugin built on bad selectors is just going to re-discover bad selectors. The foundation has to be solid.
The specificity of the initial prompt has downstream effects on maintainability.
Everything flows from that initial prompt. It determines selector quality, which determines how often the plugin breaks, which determines whether the self-healing mechanism can actually find the right replacement element. A vague prompt produces brittle selectors that break frequently and heal poorly. A specific prompt produces stable selectors that break rarely and heal accurately.
I want to go back to something you said about the agent validating its own work by checking whether clicking an element actually produced the expected result. That's a form of testing built into the discovery process.
It's test-driven development, but the agent is both writing the tests and the implementation. The discovery prompt defines the acceptance criteria — "this skill should filter results to Prime-eligible items." The agent explores, forms a hypothesis about which element does that, tests the hypothesis by clicking and observing, and only includes the mapping in the plugin if the test passes. That's a much more rigorous process than "here are all the clickable things I found."
The network request capture adds a second validation layer. Even if the DOM changes, the API endpoint might stay the same.
The dual-layer understanding is a resilience strategy that I don't think enough people are exploiting yet. The agent maps the UI element to the API call. If the UI element changes, the agent can still identify the correct interaction by looking for the API endpoint. If the API changes but the UI stays the same, the DOM mapping still works. You only lose the mapping if both layers change simultaneously, which is rare.
That's a hedge against different kinds of site updates. A visual redesign might change the DOM but leave the API alone. A backend migration might change the API but leave the front-end selectors intact.
The agent can detect which kind of change happened by checking which layer broke. If the selector fails but the API endpoint still exists, it's a front-end change. If the API endpoint returns a four-oh-four but the selector still works, it's a back-end change. The re-discovery can target the broken layer specifically.
This is starting to sound less like a plugin and more like a living thing that adapts to its environment.
That's not a bad way to think about it. The plugin isn't a static script anymore. It's a lightweight agent that validates its own assumptions on every run and repairs itself when those assumptions break. The initial discovery prompt is like its genome — it defines what the plugin is supposed to do and how it should verify that it's doing it. Everything else can be re-derived from that.
Hilbert: Nineteen ninety-eight. I was running QA automation for the orange e-commerce platform — not the blue one, the orange one. We had twelve thousand Selenium tests. Twelve thousand. Every Tuesday at two in the morning, the engineering team pushed a new deploy, and by two-fifteen my phone was ringing because four hundred tests had gone red. Every single Tuesday.
Hilbert: Nineteen ninety-eight. I was running QA automation for the orange e-commerce platform — not the blue one, the orange one. We had twelve thousand Selenium tests. Twelve thousand. Every Tuesday at two in the morning, the engineering team pushed a new deploy, and by two-fifteen my phone was ringing because four hundred tests had gone red. Every single Tuesday.
Four hundred failures per deploy.
Hilbert: On a good week. Bad week, maybe eight hundred. And every one of them was a selector that stopped matching. A button moved. A div got renamed. Some front-end developer in Seattle changed a class name from "add-to-cart" to "add-to-cart-primary" and thirty-seven tests broke because they were all targeting the old class. I had a team of six people whose entire job was updating selectors.
Hilbert: On a good week. Bad week, maybe eight hundred. And every one of them was a selector that stopped matching. A button moved. A div got renamed. Some front-end developer in Seattle changed a class name from "add-to-cart" to "add-to-cart-primary" and thirty-seven tests broke because they were all targeting the old class. I had a team of six people whose entire job was updating selectors.
You're hearing this conversation and feeling some deja vu.
Hilbert: I'm hearing you talk about selector brittleness like it's a new problem. We had this exact conversation in twenty-twelve about Selenium locators. The difference is that back then, a human had to go find the new selector and update the test. Open the page, inspect element, copy the new class name, paste it into the test script, commit, deploy. Twenty minutes per test, times four hundred tests. What Daniel's describing is an agent that can do that re-discovery automatically. That's not a minor improvement. That's the whole game. The brittleness hasn't gone away, but the cost of fixing it just dropped from a team of six to zero.
Hilbert: I'm hearing you talk about selector brittleness like it's a new problem. We had this exact conversation in twenty-twelve about Selenium locators. The difference is that back then, a human had to go find the new selector and update the test. Open the page, inspect element, copy the new class name, paste it into the test script, commit, deploy. Twenty minutes per test, times four hundred tests. What Daniel's describing is an agent that can do that re-discovery automatically. That's not a minor improvement. That's the whole game. The brittleness hasn't gone away, but the cost of fixing it just dropped from a team of six to zero.
The cost goes from a human's time to the agent's compute.
Hilbert: Which is cheaper and faster by a factor of... I don't know, a lot. The real trick nobody's mentioned is that you should have the agent log why it chose each selector. The reasoning trace.
Hilbert: Which is cheaper and faster by a factor of... I don't know, a lot. The real trick nobody's mentioned is that you should have the agent log why it chose each selector. The reasoning trace.
Say more about that.
Hilbert: When the UI changes and the selector breaks, you don't just want the agent to go find a new element that looks similar. Similar-looking elements might do completely different things. You want the agent to know what the selector was supposed to do. "I chose this element because it was the checkbox labeled Prime inside the delivery filter sidebar, and clicking it triggered a request to the delivery options endpoint with prime equals true." That reasoning trace tells the agent what function it's trying to restore, not just what pixels it was pointing at. That's how you get a fix that actually works instead of the agent picking the first element with the word Prime in it.
Hilbert: When the UI changes and the selector breaks, you don't just want the agent to go find a new element that looks similar. Similar-looking elements might do completely different things. You want the agent to know what the selector was supposed to do. "I chose this element because it was the checkbox labeled Prime inside the delivery filter sidebar, and clicking it triggered a request to the delivery options endpoint with prime equals true." That reasoning trace tells the agent what function it's trying to restore, not just what pixels it was pointing at. That's how you get a fix that actually works instead of the agent picking the first element with the word Prime in it.
The reasoning trace is the specification. It's what survives when the selector doesn't.
Hilbert: Right. The selector is just the current address of the thing you want. The reasoning trace is what the thing is. Addresses change. The thing itself doesn't.
Hilbert: Right. The selector is just the current address of the thing you want. The reasoning trace is what the thing is. Addresses change. The thing itself doesn't.
That's a really concrete mechanism we didn't touch. The agent writes a comment in the plugin definition — "this selector targets the Prime filter checkbox; validated by checking that results count decreased and Prime badge appeared." When the selector breaks, the re-discovery agent reads that comment and knows exactly what functional element it's looking for.
Hilbert: We tried something like this in two thousand three with a rules engine that described what each test was supposed to verify, separate from how it verified it. It sort of worked, but the rules engine was hand-maintained and nobody updated it. With an agent, the reasoning trace gets written automatically during discovery. It's part of the output. You get it for free.
Hilbert: We tried something like this in two thousand three with a rules engine that described what each test was supposed to verify, separate from how it verified it. It sort of worked, but the rules engine was hand-maintained and nobody updated it. With an agent, the reasoning trace gets written automatically during discovery. It's part of the output. You get it for free.
That reasoning trace idea is worth sitting with. But it also raises a bigger question about where this pattern breaks. As websites get more dynamic — single-page apps, shadow DOM, canvas-rendered UIs — will agent-led DOM inspection keep working?
Shadow DOM is already a challenge. It encapsulates DOM trees so the agent can't traverse into them with standard selectors. Canvas-rendered UIs are worse — there's no DOM to inspect at all. The agent would need to use visual recognition, screenshot analysis, to identify elements. That's a different capability entirely from reading class names and aria labels.
The pattern works brilliantly for traditional and even React-style web apps, but hits a wall with certain modern architectures.
For canvas UIs, you'd need the agent to use the browser's screenshot capability and run computer vision on the result. Find the button visually, get its bounding box, click at those coordinates. That's fragile in a different way — it breaks if the layout shifts even slightly. But for the vast majority of web applications, the DOM is still there and still inspectable. Shadow DOM can be pierced with the right browser APIs. The pattern has a lot of runway.
The broader point, and I think this is where Daniel was driving, is that the specificity principle extends beyond plugin development. Any time you ask an agent to explore an unknown space and produce a structured output, the quality of that output is bounded by the precision of the prompt. A vague exploration prompt produces vague, low-confidence results. A specific, goal-oriented prompt produces testable, high-confidence results. Daniel's plugin pattern is a case study in that broader truth.
It's a truth that applies whether you're building a delivery filter plugin or asking an agent to analyze a codebase or research a topic. Tell it what success looks like. Give it a way to validate its own work. Constrain the search space. The agent is capable of doing the discovery, but it needs to know what it's looking for.
If you've built a plugin this way, or if you've tried and hit the brittleness wall, we want to hear about it. Send your war stories to show at my weird prompts dot com.
This has been My Weird Prompts. Thanks to our producer Hilbert Flumingtop.
We'll be back soon.
This episode was generated with AI assistance. Hosts Herman and Corn are AI personalities.