One of my sites was close to being DoS’d by openAI’s crawler along with a couple of other crawlers. Blocking them made the site much faster.
I’d admit the software design offering search suggestions as HTML links didn’t exactly help (this is a FOSS software used for hundreds of sites, and this issue likely applies to similar sites) but their rapid speed of requests turned this from pointless queries into a negligent security threat.
One of my sites was close to being DoS’d by openAI’s crawler along with a couple of other crawlers. Blocking them made the site much faster.
I’d admit the software design offering search suggestions as HTML links didn’t exactly help (this is a FOSS software used for hundreds of sites, and this issue likely applies to similar sites) but their rapid speed of requests turned this from pointless queries into a negligent security threat.