Daniel's been deep in the browser automation trenches again, and I think he's hit that specific frustration that every developer working on AI agents eventually runs into. He builds tools for marketplaces he actually uses — real businesses he has relationships with — using the browser as a workaround when there's no documented API. And he keeps hitting anti-bot walls designed for spammers and data miners. His question is practical first and philosophical second: what are the specific technical blockers, what workarounds actually work, what tools beyond Puppeteer should be in the arsenal, and then the bigger one — why should anti-bot measures loosen up when the bot is just a user's AI agent doing what the user would do anyway?
And that last part is where this gets genuinely interesting. This isn't about scraping product catalogs to undercut someone's business. It's about an AI agent navigating a website on your behalf, with your credentials, your session, your intent — and the web's entire security model can't tell the difference between that and a scalper bot buying fifty pairs of sneakers.
The web was built for humans with keyboards. AI agents driving browsers break that assumption at the foundation.
Let's start with the core tension, because it explains why this is so hard and why it's not going to get easier anytime soon. The fundamental problem is that the web's authentication and access model assumes a human at the keyboard with a browser. You log in, you get a session cookie, you click around. That's the contract. But AI agents need programmatic access — they need to navigate pages, fill forms, click buttons, extract data — and the services they're trying to use often have no public API, or the API is undocumented, or rate-limited to the point of uselessness, or just doesn't expose the specific action the agent needs to perform.
So the developer does the rational thing — they use Puppeteer or something similar to automate a real browser instance, piggyback on the user's own session and authentication, and mimic human interaction patterns. It works until it doesn't.
Right. And the reason it stops working is that this exact pattern — automated browser, session hijacking, mimicking human behavior — has been the playbook for spammers, data miners, ticket scalpers, and credential stuffers for decades. Anti-bot companies like Cloudflare, Akamai, and PerimeterX have built multi-million-dollar detection systems specifically to identify and block exactly this behavior. So the thing Daniel's doing — the legitimate, user-consented, non-malicious thing — looks identical to the thing those systems were built to stop.
That's the false equivalence at the heart of this. The detection systems can't distinguish intent from method. A bot actively controlled by a user is not meaningfully different from a human user navigating the site — same HTTP requests, same browser APIs, same session cookies. The difference is purely in the pattern of interaction. And the systems are optimized to flag anything that doesn't look human, regardless of why it doesn't look human.
To understand why this is so hard to work around, we need to look at what anti-bot systems are actually checking. It's not just User-Agent strings anymore — that era ended a decade ago. Modern bot detection is a multi-layered stack, and if you're building tools like Daniel is, you're going to hit every layer.
Walk me through the stack.
First layer, and the one most developers hit immediately: browser fingerprinting. Anti-bot systems collect dozens of signals from the browser — WebGL renderer, canvas fingerprint, screen resolution, installed fonts, timezone, navigator dot plugins, audio context. Headless browsers leak detectable differences all over the place. The most obvious one is navigator dot webdriver — in headless Chrome, that flag is set to true, and it's trivial for any site to check. The User-Agent often contains the string HeadlessChrome. And the Chrome DevTools Protocol detection is straightforward — there are JavaScript checks that can determine whether the browser is being controlled via the DevTools protocol, which is exactly how Puppeteer and Playwright operate.
So before you've even loaded the page, you've announced yourself.
Before you've made a single HTTP request in some cases. And it gets worse. Second layer: behavioral analysis. This is where the really sophisticated detection happens. Services like DataDome and Shape Security model user behavior in real-time — mouse movement patterns, scroll speed, click timing, navigation flow. A human moves their mouse in micro-jittery arcs with variable acceleration. An automated script tends to move in straight lines with consistent timing. A human scrolls unevenly, pauses to read, sometimes scrolls back up a bit. A script scrolls at a constant rate to a predetermined position. A human clicks with slight variations in timing and position. A script clicks at exactly the same coordinates with exactly the same delay every time.
The consistency is the tell.
The inhuman consistency. Even if you randomize your delays, the distribution of those delays often doesn't match human patterns — humans are inconsistent in specific, statistically modelable ways. Third layer: CAPTCHA escalation. reCAPTCHA version three assigns a score based on user behavior without ever showing a challenge. You don't even know you're being scored. Automated browsers consistently score low — the behavioral signals just don't look human enough — and once your score drops below a threshold, you get challenged, or blocked outright. Cloudflare Turnstile and hCaptcha add additional layers on top of this. Turnstile in particular has gotten very aggressive about detecting automated browsers.
And at that point you're in a doom loop — the challenge itself is designed to be hard for bots, and failing it confirms you're a bot, which makes future challenges more aggressive.
Fourth layer, and this one catches people by surprise: IP reputation and proxy detection. If you're running your automation from a datacenter IP — AWS, Google Cloud, DigitalOcean — you're already in a heavily penalized range. Known VPN and proxy ranges are similarly flagged. Cloud provider autonomous system numbers are basically blacklists at this point. The workaround is residential proxy networks — Bright Data, Oxylabs, Smartproxy — which provide IPs from real ISPs, but they're expensive, and some sites have started blocking known residential proxy ranges too. Mobile proxies, using four G and five G connections, are even harder to detect but even more costly.
And then there's the ethical dimension of residential proxies — using someone else's IP without their informed consent is legally gray in a lot of jurisdictions.
Yeah, that's a whole separate conversation. Fifth layer, and this is the one that surprises even experienced developers: TLS fingerprinting. The JA3 fingerprint of the TLS handshake can identify specific browser versions and automation libraries before any HTTP traffic even begins. Different versions of Chrome, Firefox, and Safari produce different JA3 fingerprints. Automation tools like curl with custom TLS settings produce fingerprints that don't match any real browser. You can be blocked purely on the characteristics of your TLS handshake — the site never even sees your HTTP request.
So the block happens at the transport layer before you've sent a single header.
Before you've sent anything. It's a handshake-level rejection. And the fingerprints for tools like Python's requests library or Node's axios are well-known and heavily penalized.
Alright. So you're a developer like Daniel, you're hitting all five of these layers, and you need to get through. What actually works?
The first line of defense is undetected-chromedriver. This is a patched version of ChromeDriver that modifies the Chrome DevTools Protocol to hide automation flags. It spoofs navigator dot webdriver to false, patches common detection vectors in the DevTools protocol, and handles a lot of the basic fingerprinting checks. It's not perfect, but for many sites it's enough to get past the first layer of detection.
And when it's not enough?
Then you layer on playwright-stealth or puppeteer-extra-plugin-stealth. These inject patches at the browser level — they normalize WebGL fingerprint signals, override canvas and audio context outputs to look like a real browser, and randomize behavioral parameters. The key thing here is that these are actively maintained. The developers track what anti-bot systems are checking for and update the patches accordingly. It's an arms race, but these projects have stayed ahead of most basic detection for years now.
I want to pause on that arms race for a second, because it's the dynamic that makes all of this temporary.
It's absolutely temporary. Anti-bot companies actively monitor GitHub repositories for new stealth techniques. When a new patch drops, they analyze it, update their detection models, and push the update to their customers. The window where a particular evasion technique works is measured in weeks or months, not years. If you're building a tool that needs to work reliably over time, you have to plan for constant maintenance.
So what does the maintenance look like in practice?
Browser-level randomization, mostly. You randomize viewport size within a realistic range — not exactly nineteen twenty by ten eighty every time, but something like nineteen eighteen by ten seventy-six, then nineteen twenty-three by ten eighty-one. You rotate user agents from a list of real browser versions. You randomize timezone and geolocation to match the IP you're coming from. And then the behavioral stuff — you introduce random mouse jitter with variable acceleration curves, you vary typing speed with occasional pauses and corrections, you add realistic scroll patterns with micro-adjustments.
The corrections are a nice touch. Humans backspace.
Humans backspace. A script that types perfectly at a constant speed is a dead giveaway. A script that types most of a word, backspaces two characters, retypes them, and continues — that's much harder to distinguish from a human. But here's the thing: even with all of that, some sites are still going to block you. The marketplaces Daniel's working with — eBay, Craigslist, any site with high-value inventory — they have aggressive, multi-layered detection that's been tuned against exactly these evasion techniques.
So at that point you're into the proxy and CAPTCHA-solving layer.
Which is where things get expensive and ethically complicated. Residential proxy networks charge by the gigabyte — you're looking at anywhere from five to fifteen dollars per gigabyte depending on the provider and the targeting options. CAPTCHA solving services like Two Captcha, CapMonster, and Anti-Captcha add another cost layer. They use either human workers or machine learning models to solve CAPTCHAs in real-time, and the latency is five to thirty seconds per challenge. For an AI agent that's supposed to be doing something useful for a user, adding a thirty-second delay every time it hits a CAPTCHA is... not great.
It breaks the experience. The whole point of an AI agent is that it's faster and more convenient than doing it yourself.
If your agent takes longer to book a flight than you would take doing it manually, what's the point? And that's before we even get into the cost. If you're paying for residential proxies plus CAPTCHA solving plus the compute to run the agent, you could easily be spending a few dollars per task. For high-value transactions like booking flights or buying limited-release products, that might be worth it. For everyday use, it's not sustainable.
So those are the blockers and the basic workarounds. But Daniel specifically asked about tools beyond Puppeteer — the rest of the iceberg.
Right. Puppeteer is the most well-known, but it's not always the best choice, and in some cases it's actively the wrong choice. Let me run through the alternatives and when you'd reach for each one. Playwright is the big one. It's cross-browser — Chromium, Firefox, and WebKit — which matters because some anti-bot systems are tuned specifically against Chromium-based automation. Using Firefox via Playwright can get you past detection that's looking for Chrome-specific tells. It also has better multi-tab handling, native network interception, and a more modern API. Microsoft maintains it, and the development velocity is high.
And the stealth story?
playwright-stealth exists and is actively maintained. It's not quite as battle-tested as the Puppeteer stealth ecosystem, but it's catching up fast. For new projects, I'd probably start with Playwright just for the cross-browser support alone. Selenium is still out there, still relevant for legacy systems, but it's less stealthy out of the box and the WebDriver protocol it uses is easier to detect than the DevTools protocol that Puppeteer and Playwright use. You'd only reach for Selenium if you're maintaining something old or if you need to support a very specific browser version.
And then the specialized tools.
Browserless is interesting — it's headless browser as a service with built-in stealth features. You don't manage the browser instance yourself; you send it commands and it handles the anti-detection on its end. The value proposition is that they stay on top of the arms race so you don't have to. FlareSolverr is another specialized tool — it's a proxy-aware CAPTCHA-solving wrapper around Puppeteer, specifically designed to get past Cloudflare challenges. You send it a URL, it handles the Cloudflare challenge page, solves the CAPTCHA if needed, and returns the page content. It's not a general-purpose automation tool, but for the specific problem of Cloudflare-protected sites, it's very effective.
So the arsenal expands based on the specific defenses you're up against.
And based on your tolerance for maintenance burden. Every tool in the stack is another thing that can break when a detection model updates. The more layers you add, the more fragile the whole system becomes.
This brings us back to Daniel's philosophical argument. If a bot is actively controlled by a user, acting on their behalf with their credentials and their intent, blocking it is functionally blocking the user. The detection systems are looking at the how, not the why.
And the industry's counter-argument is that they can't tell the difference. A scalper bot buying concert tickets is also acting on behalf of a user — the scalper. The actions are identical. The HTTP requests are identical. The session cookies are identical. The only difference is intent and scale, and neither of those is visible to the detection system. From the server's perspective, a thousand requests from a thousand different AI agents look exactly like a thousand requests from a thousand scalper bots.
But that's a scale problem, not an identity problem. Rate-limiting solves scale. Authentication solves identity. If you know who the user is and you can limit how many requests they make, you don't need to detect whether they're a bot.
That's the argument for a different model entirely. Instead of "detect and block automation," you move to "authenticate and rate-limit automation." The way OAuth tokens work for APIs — you prove who you are, you get a token with specific permissions and rate limits, and you use that token to access the service. Some sites already do this. GitHub has API access tiers. Twitter, or X, has API access tiers. The question is whether the rest of the web will follow.
The web standards angle is interesting here. WebDriver BiDi is a W3C standard for browser automation protocols — it standardizes how automation tools communicate with browsers, which sounds like it would make detection easier, not harder. But the idea is that if automation is standardized, sites can build opt-in access models around the standard rather than trying to detect and block it.
Right. The vision is something like a standardized agent identity header or API endpoint. A site could say "I accept automated access from authenticated users with these rate limits and these permitted actions." The AI agent identifies itself honestly — no spoofing, no stealth patches, no cat-and-mouse — and the site either allows it or doesn't. The user knows what's happening, the site knows what's happening, and the detection arms race becomes unnecessary.
That's a clean vision. What's stopping it?
Buy-in, mostly. Browser vendors would need to implement the standard. Website operators would need to adopt it. And the anti-bot industry — which is a multi-billion-dollar industry — would need to... adapt, let's say. There's a lot of money in the current model. Cloudflare's bot management is a significant revenue stream. Moving to an opt-in model where bots identify themselves honestly doesn't generate the same kind of recurring revenue.
So the incentives are misaligned. The companies that could build the solution profit from the problem.
That's a bit cynical, but not entirely wrong. The more charitable reading is that the web has spent twenty years treating all automation as hostile, and changing that assumption requires changing infrastructure, standards, and business models simultaneously. That's slow work.
Let me try a concrete example. Daniel builds a tool for a marketplace he uses regularly. He's logged in, he has purchase history, he has a reputation score. His AI agent wants to check if a specific item is available and place an order if it is. That's exactly what Daniel would do manually. Under the current model, the marketplace's anti-bot systems flag the agent, block it, and maybe flag Daniel's account. Under the opt-in model, Daniel's agent presents an identity token, the marketplace says "authenticated user, permitted actions: browse, search, purchase, rate limit: sixty requests per minute," and the agent operates within those constraints.
And the marketplace is happy because they know who's accessing their site and they can enforce limits. Daniel's happy because his agent works reliably. The anti-bot vendor is... less happy, but that's a business model problem, not a technical one.
The technical problem is that the opt-in model requires the marketplace to build and maintain that infrastructure. Right now, they outsource the whole problem to Cloudflare or Akamai and don't think about it. Moving to an identity-based model means building new endpoints, managing tokens, setting rate limits per user and per action — it's real engineering work.
Which is why it'll happen first at the big platforms that already have API programs. Google, Meta, Amazon — they already have the infrastructure for authenticated API access. Extending that to browser-based agentic access is a smaller step for them than it is for a mid-size marketplace.
The future Daniel wants — an internet where user-agent spoofing isn't necessary — probably arrives unevenly. The big platforms build it because they can. The long tail of the web keeps blocking bots because the detection model is cheaper than the identity model.
In the meantime, developers like Daniel keep maintaining their stealth patches and proxy rotations and CAPTCHA solvers. The arms race continues.
This idea that the web needs to adapt to AI agents — it's not just a technical argument. Hilbert, you've been through some version of this.
Hilbert: iMacros. Two thousand seven. I was running a small arbitrage operation — buy clearance items from big retailers, resell on eBay. Built a bot to check inventory and auto-purchase. Got banned from three retailers in a month.
What gave you away?
Hilbert: Target caught me because my script clicked the Add to Cart button exactly two point four seconds after page load. Every single time. I didn't even notice until they sent the ban email. I went back and looked at the logs — two point four seconds, two point four seconds, two point four seconds. A human would never do that. But here's the thing — I was sitting there watching it. I was the user. I just wanted to know if the item was in stock without refreshing the page fifty times.
The bot was doing what you would have done.
Hilbert: Just faster. And Target couldn't tell the difference between me checking inventory and a scalper buying fifty units. They still can't. That's the problem. The detection systems are looking at the wrong thing. They're looking at the how, not the why.
Did you try to fix it?
Hilbert: I added a random delay. Between one and four seconds. Still got banned. The distribution was too flat — real humans cluster around certain delays, they don't spread evenly across a range. I didn't know that at the time. I just thought random meant random.
The statistical modeling has gotten a lot more sophisticated since then.
Hilbert: I know. That's what worries me. I still have those scripts on an old hard drive. Been thinking about dusting them off for an AI agent project — nothing commercial, just automating some things I do manually. But I don't think iMacros even works anymore, and the anti-bot systems are... I mean, you just spent twenty minutes listing all the ways they'd catch me. TLS fingerprinting wasn't even a thing back then.
What would you build if you could?
Hilbert: Something for estate sales. My sister-in-law runs an estate sale business — she's the one who got me into the arbitrage thing in the first place, actually. She posts inventory lists on her site before the sales, and the good stuff goes fast. I want an agent that watches her site, alerts me when certain items show up, and if it's something I really want, places a hold. It's all stuff I'd do manually anyway. I just can't sit there refreshing a page all day.
Your sister-in-law would be fine with this?
Hilbert: She'd be fine with me doing it. She'd be less fine if everyone did it. That's the tension, isn't it? One person automating is fine. A thousand people automating is a problem. But the detection systems can't tell the difference between one and a thousand until it's too late.
Even in your own hypothetical, the scale problem is real.
Hilbert: It's real. I'm not saying it's easy. I'm saying the current approach — block everything that doesn't look human — is lazy. My sister-in-law could give me an API key. She won't, because she doesn't have the infrastructure, but she could. The technology exists. What doesn't exist is the will to build it for every small business on the internet.
That's exactly the gap. The big platforms can build it. The small ones can't, or won't, until it's packaged as a service they can buy.
Hilbert: Cloudflare could sell authenticated agent access the same way they sell bot blocking. Same infrastructure, different rule set. They just haven't decided it's worth the revenue shift yet.
They might not have a choice for much longer. If AI agents become as common as browsers, blocking them all becomes blocking a significant chunk of legitimate traffic.
Hilbert: My sister-in-law would notice if half her customers couldn't reach the site. She'd call someone. That's when it changes — when the blocking hurts the business more than the bots do.
That's the inflection point Daniel's betting on. The question is how far away it is.
Hilbert: Further than I'd like. I'm still not dusting off those iMacros scripts.
The question remains — will the web adapt to AI agents, or will the arms race continue indefinitely? The standardized agent identity header sounds clean on paper, but it requires coordination across browser vendors, website operators, and the anti-bot industry, and the incentives aren't aligned yet.
The counter-force is that as AI agents become more capable and more common, the pressure on anti-bot systems increases. Either they develop the ability to distinguish legitimate automation from malicious automation — which is an intent-detection problem, not a behavior-detection problem — or they risk blocking a growing portion of legitimate web traffic. And the second one hurts revenue.
The misconception I keep seeing is that user-agent spoofing is a reliable workaround. It's not. Modern fingerprinting goes far beyond the User-Agent string — TLS handshake analysis, behavioral modeling, IP reputation, and dozens of browser fingerprint signals all feed into the detection decision. Changing the User-Agent accomplishes almost nothing on its own.
The real workaround is layering multiple evasion techniques — stealth browser patches, behavioral randomization, residential proxies, CAPTCHA solving — and accepting that the whole stack needs constant maintenance. It's not a one-time fix. It's an ongoing arms race.
Daniel's vision of an internet where none of this is necessary may be idealistic, but the technical and philosophical arguments are sound. The web needs a new model for agentic access. Who builds it first — the browser vendors, the big platforms, or the anti-bot companies themselves — is still an open question.
My bet is on the platforms. Google and have the most to gain from a web that works well with AI agents, and they already have the identity infrastructure. But it'll be years, not months.
Thanks to our producer Hilbert Flumingtop for keeping this show running. This has been My Weird Prompts. Find every episode at my weird prompts dot com, or email the show at show at my weird prompts dot com. We'll be back soon.
