“It will be the greatest theft in the entire history of humanity.” Indie hackers weigh in on big AI companies scraping the web

Big AI is scraping content made by creators like you and me and ingesting it for their own gain.

Google's a good example. It recently updated its privacy policy to say that it’ll use… well… everything on the internet.

Obviously, this practice brings up not only privacy concerns but copyright concerns too; in fact, a class action lawsuit against Google for copyright infringement is already underway.

I caught up with fellow indie hackers to get their thoughts.

Is scraping for AI good or bad?

Courtland Allen of Indie Hackers:

I’m a big fan of AI. It’s fun, it’s interesting, and it gives me superpowers as a developer and business owner.

I don’t care if AI ingests my content. It’s also a moot point. It’s 2023. We’ve seen this battle play out enough times to know that technology will ultimately win. Let’s just accept the inevitable and focus on making the best of it.

Class-action lawsuit statement:

Google does not own the internet, it does not own our creative works, it does not own our expressions of our personhood, pictures of our families and children, or anything else simply because we share it online.

Alexander Isora of Unicorn Platform ($13K/mo):

It will be the greatest theft in the entire history of humanity. But is that good or bad?

Creativity is the process of creating something new based on the existing thing(s). If I get inspired by Rembrandt and Andy Warhol and use the neural network in my brain to create a new style, am I a thief?

What if I teach a neural network in my computer to do this?

What if I make a service so anyone can use my network to create millions of works of art in the new style?

I believe it is neither bad nor good. This is progress. It's pointless to judge it. And there is no way to prevent it. We need to get used to the new reality and find a way to adapt to it.

That means we need to at least be aware of the progress and the possibilities of GPT. This is why I advocate following AI news and joining AI discussions like this one.

Is what Big AI is doing a crime?

Mateusz Buda of Scraping Fish ($5K/mo) and Narf AI:

I hold the view that if the content is public and all legal requirements are met, everyone should be allowed to obtain it, store it, process it, and even sell it in one form or another.

The stuff that Google does is totally legal, as it does only the things that website owners allow them to do and it’s pretty easy to prevent your data from getting scraped at the cost of not being exposed for free and allowed to be found by other people.

Oleg Kulyk of ScrapingAnt ($14k/mo):

Google robots (web scraping spiders) should follow sitemaps and robot.txt files in order to not index web pages that website owners restricted it from. So you should be able to protect even publicly available data from scraping, but I’m not quite sure that Google would follow the same mechanisms for indexing and data scraping for AI learning. Still, I assume that they should.

What does all this mean for copyrights and innovation?

Courtland Allen:

I’m not a big fan of copyright. It puts unnecessary restrictions on ideas, stories, and art, which are things that are naturally better off if they spread.

I don’t buy the argument that copyright is necessary to spur creativity. Lots of people create art without any expectation of profit from licensing, because they love art. And lots of people get paid for their writing without relying on licensing, because they’re good at business.

Expecting a paycheck because ChatGPT or Bard or Claude scraped your writing seems like the worst of both worlds — wanting to do art for the sake of business, but not being good enough at business to figure out how to profit.

I don’t see why it benefits society to have laws that support that kind of behavior.

What does this mean for other scrapers?

Mateusz Buda:

The landscape of web scraping isn't equitable for all players. Large corporations like Google and OpenAI enjoy an advantage compared to individual indie hackers. Legal challenges associated with web scraping disproportionately affect these smaller entities, thereby limiting their access to content. Small companies with no marketing budget also cannot afford lawyers when they face cease-and-desist letters."

Oleg Kulyk:

Other web scraping market players are constantly struggling against anti-bot systems, as they don’t have Google’s access.

Google has a great benefit over all other companies that scrape publicly available data, as their indexing robots are not banned by anti-bot systems like Cloudflare, Akamai, etc., so they can get the needed data pretty easily.

From the other side, all other companies that extract publicly available data cannot follow robots.txt and just run the spiders to follow all the website links and extract the publicly available data. So I’d not say that it should be considered a crime, but it’s not the best way to stay on the ethical side.

What does the future hold?

Mateusz Buda:

All things considered, as long as Google doesn't use its position to access data not available to the public, I don't think it is theft. However, the future will necessitate regulations around the use of publicly available data, similar to the GDPR or CCPA that protect personal data. Ideally, authors would retain the rights to their written content, and explicit consent would be required for its usage. Hopefully, this won't result in even more cluttered websites with additional consent banners that end users have to navigate.

Importantly, tech giants like Google or OpenAI should not receive preferential treatment. All entities, regardless of size, should operate under the same rules concerning data access and usage.

Keep moving forward

So there you have it. AI, and the scraping that powers it, is problematic. But ultimately, these indie hackers are on board. What other option do they have?

We’re in a new frontier and we’re figuring it out as we go. There’s no going back, so all we can do is stay informed, stay ethical, and adapt to this new environment.

At least that’s what these founders said.

Subscribe for more founder interviews, roundtables, case studies, and tips from people who are in the thick of it. 🪤

The Boot's Trap 🪤

Learn from indie hackers who are in the thick of it

Subscribe