I’m getting really tired of “verify you’re not a bot” checks everywhere.
I understand these checks help prevent bots, but they seem to appear on so many websites now. Does anyone else find them frustrating?
I’m getting really tired of “verify you’re not a bot” checks everywhere.
I understand these checks help prevent bots, but they seem to appear on so many websites now. Does anyone else find them frustrating?
I created a bot black hole. Specific recursive link that if the click it three times it blocks the IP. And the link is “click if ai” URL at the bottom. Works like a charm.
No; the traffic i get seems like it had stored my sitemap from ~10 years ago because the requested urls there are long gone, and now tries to guess other url’s with a similar pattern. It does not seem to follow links from the frontpage.
That’s fascinating. It almost sounds like they’re training a scraper. Which, really, is far and away the worst possible bot to have.
Training a scraper? Sounds more like they’re just scraping training data.
You’re entirely right, but I just wanted to post something innocuous to confirm that I picked up a stalker. Sorry for involving you!
Maybe they’re dating train scrapings
They’re straining craping dats
I have fail2ban take care if those. 404s and all that.
I already looked into something similar, IP banning does not work. One day i got ~120k unique IPs, but the most active one only made ~100 requests. I would gain nothing if i run any script to lookup the IP with each requests (and using up more resources in the process), if it fails 99% of the time because the next hit is a different IP.
Ok. Sounds like your in a unique situation. Good luck. You seem fucked.
All the above worked for me.
I also have the same problem as who you’re replying to on a few of my websites. The only way to block them is Anubis or Cloudflare WAF.
I think you might be in the unique situation, to be honest. It may only be a matter of time until your solution also stops working.
Agreed. The current situation seems to be a gazillion bots each making a single request. I’ve had some luck blocking whole ASNs (Alibaba, I’m glaring at you), and Anubis. I’ve had pretty good uptime the last few months.