Build a moat with browser automation

Everybody can use a Next.js boilerplate. Everybody can use an API. Everybody can add an AI chatbot to their product.

Even if you land on a business that works with these methods, you still don't have a moat. You still aren't differentiated from the competition, let alone all the copycats that come after you.

If you really want to make something that sticks, you need to look for features that are too hard to replicate. Luckily, these also tend to be the features that are the most valuable to your users.

The question is, where do you find these features? And how do you build them?

Browser automation is unique

When I think about a moat, my mind always goes to the browser. Why? Because everything happens in the browser, even the things that you can't do with APIs.

For example, you can't automate the process of signing up for a new account on a website. You can't automate the process of filling out a form. You can't automate the process of clicking on a button.

You also can't scrape a website's valued data, interact on social media, track competitors' prices, ensure a website's performance, or generate beautiful images, videos, and interactive PDFs.

You also can't give an LLM direct access to a persistent browser session, to take actions on a user's behalf.

Yet these are all things you can do with a headless browser and a little bit of code.

Browser automation is defensible

Why then aren't lots of people building these features?

Well, browser automation is a bit of a pain.

First, the web is constantly changing, so you need to write your automation code to be resilient, and you need to ensure you're notified when some of your resilience fails.

Second, your target websites probably don't want you to interact with them programmatically. USA law is clear: your code is free to interact with the web just as a human is. But the law doesn't prohibit sites from finding ways to block or ban you. So you also need a bit of skill in avoiding detection and working around anti-bot measures.

Third, it's a pain to host headless browsers. They're very large, very heavy, and they don't scale in the same way as your typical API. For example, your API may support 1,000 requests per second, but with the same CPU/memory profile, your headless browser may only support 10 requests per second. They also don't work so well in serverless environments. So it's better to host the browsers separately.

Browser automation is valuable

All that said, when you get browser automation right, you can charge an arm and a leg for it. Just look up the pricing for proprietary data sets, LinkedIn automation, or enterprise RPA (Robotic Process Automation). Cha-ching!

The best part is that there's low hanging fruit for browser automation in every niche. Talk to anyone, in any industry, about the pointing and clicking they have to do for their job, and you'll find a wealth of opportunities.

Remember, most knowledge workers spend the vast majority of their time occupied by repetitive tasks in the browser... and they hate it!

It's just a bit annoying to build, so most people don't.

Browser automation can be easy

I've worked with browser automation for years, from major corporate gigs to indie startups to personal side projects. I've faced pretty much every challenge there is, and I've learned a lot about how to make it easy.

Let's look back at our list of challenges:

1. Writing resilient code
2. Avoiding detection
3. Hosting headless browsers

Writing resilient code

Even if you're only a "decent" dev, these practices aren't out of reach. After all, the web doesn't change that much, or that often.

It boils down to a few simple practices:

1. Handle all possible errors. You never want your job to fail when it could have succeeded. Always think through ways your code could fail, and handle them preemptively.
2. Write idempotent code. You may have to re-run your jobs, and you want to ensure this doesn't create more problems down the line.
3. Support partial successes. In complex jobs, a lot can go wrong. But you've got your users to think about. Ensure that even if a job fails, you track and report the successful portions.
4. Select elements with fallbacks. For example, you target a button by its text, its ID, and its location on the page. Then you take the first match.
5. Run interactions with fallbacks. For example, if your first strategy for submitting a form fails, try an alternative method.
6. Notify yourself of failures. At the end of a run, aggregate data about what failed and why. Your scripts should always have fallbacks, so it's not okay if due to changes, only a single path still works. Am error tracking service like Sentry is great for this.

Like all things in programming, these practices seem more daunting than they are. Once you've done them a few times, they become second nature.

I've begun writing about them on the BrowserCat blog. Check out my guide to strengthening your selectors for a taste.

Avoiding detection

This task may seem even more daunting, but it's actually much easier than you think. I listed the following recommendations in descending order of effectiveness. Only implement them as you find you actually need them.

1. Use a headless browser. It's easy to detect simple fetch() requests are bots. But a real headless browser is completely indistinguishable from a normal browser instance. This is why most scraping APIs charge you 5x the credits for the privilege.
2. Throttle your requests. Normal humans don't make 100 requests per second. Neither should your jobs. They're too easy to detect, so most adversaries (including Amazon, LinkedIn, and X) rely on this metric as their first or only line of defense.
3. Use a residential proxy rotation. If you're making a lot of requests from a single commercial IP address, you're going to get detected. But by rotating through numerous residential IPs in your target country, you're going to seem that much more like a real user.
4. Simulate human behavior. Some sophisticated anti-bot measures will look at mouse movement, keyboard patterns, and "speed-running" through on-site actions. Thankfully, with a headless browser, some helper methods, and a random number generator, you can beat most detection. Open-source libraries will help with the few remaining edge-cases.
5. Solve captchas. If you're still getting detected, you can rely on third-party APIs to solve captchas for you. Some will use real humans, while others will use AI. Either way, you'll get a solution in just a few seconds. Nevertheless, it's better never to get to this step. Most won't need it.

Hosting headless browsers

This is the most annoying part of browser automation.

As mentioned above, browsers are heavy dependencies, and they must be hosted separately from your code. In serverless, they have terrible warm up times, they're costly to run, and they're limited in many cumbersome ways. But when deployed in docker, you've got to manage scaling to zero, or you're going to spend a lot of money for nothing.

Meanwhile, if you want to generate videos, download large files, reuse persistent user sessions, or in any other way take full advantage of the browser, there are many other bits of infrastructure for you to develop.

The biggest problem isn't that it's hard to do, but rather that you have to do it instead of building your product. It's a huge, costly, painful distraction. (Even if you're starting from a baseline like Playwright's docker image.)

This is the main reason why browser automation isn't more popular.

PITCH: That's why I'm building BrowserCat. BrowserCat hosts and scales headless browsers so you don't have to. Point your code at our servers, and pay only for what you use. I aim to make it cheap, fast, and easy to work with headless browsers, so reach out with any questions or feedback you have.

Write your first browser automation

If you're interested in building a moat with browser automation, check out my article on writing your first script. It shows how to use the open-source Playwright automation library to control a headless browser on your local machine.

And when you're ready to deploy, jump to production with BrowserCat.

Ask me anything!

If you made it this far, you're clearly interested in browser automation. I'll be hanging out in the comments below, and I want to help. Ask me anything!