They better don’t attack too much, because all of the internet is built on FOSS infrastructure, and they might stop working, lol.
I have a small site that mirrors hacker news but with dark mode and stuff, and it is getting blasted by bot traffic. All the data is freely available from the official api but they’re scraping my piddling site which runs on an anemic VPS because it looks like user generated content. Bot protection does little to help from my provider. Gonna have to rethink my whole architecture now. Very annoying
In a blogpost called, “AI crawlers need to be more respectful”, they claim that blocking all AI crawlers immediately decreased their traffic by 75%, going from 800GB/day to 200GB/day. This made the project save up around $1500 a month.
“AI” companies are a plague on humanity. From now on, I’m mentally designating them as terrorists.
that’s the pure definition of a parasite
If you’re wondering if it’s really that bad, have this quote:
GNOME sysadmin, Bart Piotrowski, kindly shared some numbers to let people fully understand the scope of the problem. According to him, in around two hours and a half they received 81k total requests, and out of those only 3% passed Anubi’s proof of work, hinting at 97% of the traffic being bots
And this is just one quote. The article is full of quotes of people all over reporting they can’t focus on their work because either the infra they rely on is constantly down, or because they’re the ones fighting to keep it functional.
This shit is unsustainable. Fuck all of these AI companies.
Its absolutely sustainable. Just cache it. Done.
Cache size is limited and can usually only hold a limited number of most recently viewed pages. But these bots go through every single page on the website, even old ones that are never viewed by users. As they only send one request per page, caching doesnt really help.
I’m sure that if it was that simple people would be doing it already…
One of my sites was close to being DoS’d by openAI’s crawler along with a couple of other crawlers. Blocking them made the site much faster.
I’d admit the software design offering search suggestions as HTML links didn’t exactly help (this is a FOSS software used for hundreds of sites, and this issue likely applies to similar sites) but their rapid speed of requests turned this from pointless queries into a negligent security threat.
Great write-up by Niccolò.
I actually agree with the commenter on that post, the lack of quoting and using images is pretty bad, especially for screen-readers (which I use), and not directly linking sources (though they are made clear regardless) is a bit of a pain.
Definitely agree. Love TheLibre as it covers subjects I don’t see hit on as often but the lack of actually linking to sources and proper quotes blows.
You’d think these centralised LLM search providers would be caching a lot of this stuff, eg perplexity or claude.
There’s two prongs to this
-
Caching is an optimization strategy used by legitimate software engineers. AI dorks are anything but.
-
Crippling information sources outside of service means information is more easily “found” inside the service.
So if it was ever a bug, it’s now a feature.
Third prong, looking constantly for new information. Yeah, most of these sites may be basically static, but it’s probably cheaper and easier to just constantly recrawl things.
-
They’re absolutely not crawling it every time they nee to access the data. That’s an incredible waste of processing power on their end as well.
In the case of code though that does change somewhat often. They’d still need to check if the code has been updated at the bare minimum.
Hashes for cached content. Anyone know what sort of DB makes sense here?
Sad there’s no mention of running an Onion Service. That has built-in PoW for DoS protection. So you dont have to be an asshole and block all if Brazil or China or Edge users.
Just use Tor, silly sysadmins
Proof of work is what those modern captchas tend to do I believe. Not useful to stop creating accounts and such, but very effective to stop crawlers.
Have the same problem at work, and Cloudflare does jack shit about it. Half that traffic uses user agents that have no chance to even support TLS1.3, I see some IE5, IE6, Opera with their old Presto engine, I’ve even seen Netscape. Complete and utter bullshit. At this point if you’re not on an allow list of known common user agents or logged in, you get a PoW captcha.
If I was a bot author intent on causing misery I’d just use the user agent from the latest version of Firefox/Chrome/Edge that legitimate users would use.
It’s just a string controlled by the client at the end of the day and I’m surprised the GPT and OpenAI bots announce themselves in it. Associating meaning on the server side is always going to be problematic if the client can control the value
Yeah but Tor’s doesn’t require JavaScript, so you dont have to block at-risk users and opress them further
