The “It’s not clear who’s doing it” section is what’s baffling to me, like I’m very far from into networking and I’ve definitely done some crude scraping the few times I’ve needed it, but even I am pretty sure I could do better than this. Makes me think someone with more money than sense hooked up an “Agentic” AI and told it to gather data to train the next AI then just let it decide how – except not because I’m pretty sure the AIs would have more sense. It’s really weird.
Anubis is the way to go
Detect whether the user is a human, but instead of blocking the request (which is going to be obvious to the scraper operator and will just cause the bot developers to go work on better human emulation until they get the data), poison the response. Just as blocking scrapers is hard for website operators, so is separating useful data from not-useful data for people building AI training corpuses.
https://www.cloudflare.com/learning/ai/data-poisoning/
Data poisoning involves injecting malicious information into training datasets to manipulate an AI model’s behavior, compromising its accuracy, reliability, and the overall integrity of machine learning results.
This is the way. Computer use agents are common and can easily ‘browse’ to a page and grab the content.
The really exasperating thing is that Wikipedia makes all of their data available in compressed database form, and I suspect that other MediaWiki wikis could do so too. I checked one of the wikis being complained about, the Minecraft wiki, and it’s a MediaWiki wiki.
This isn’t even a case where the server operators have any problem with the data being available to the bots. In fact, people are going out of their way to package up all the data in a form that is optimal for computer processing (better for the bots) and which doesn’t create additional load on the wiki servers (better for the wiki operators).
But the bots aren’t using them! And on top of that, they aren’t even written to follow traditional conventions for being polite in scraping. We’ve had lots of software spider the Web — that’s not new — but normally that spidering software has followed basic politeness to try to avoid excessive disruption to the servers being spidered.
It’s as if someone has set up a table out front of their house with a big sign reading “Free cookies!” with boxes of pre-packaged cookies. Instead of taking the damn cookies, the bots are obtaining the largest trucks possible and then ramming holes in the house with a truck and scavenging through the ruins, desperately trying to find cookies.
That compressed database would have to be updated frequently, I don’t know how well it’d work.
Wikipedia has started striking deals with AI companies as a means to cover the cost of their lookups which, as much as I may not like it, is totally fair.
A wiki for some obscure indie game won’t have the same leverage.
That compressed database would have to be updated frequently
They are.
It’s a weird phenomenon I encountered as a data engineer inside corporations. Teams wanted to run endless queries off our API instead of using the snapshots we provided. Most of the apps didn’t have a justifiable reason for wanting up to the minute information. 🤷♂️
Ai scrapers are making it kinda suck to run anything on the internet to be honest.
Making it kinda suck to use the Internet at all. I try to look things up and its page after page of AI slop. We are going to end up back in the days of Altavista.
For some reason, for the past month or so, when I google stuff the first result is ‘Can’t generate an AI overview right now, try again later’. Maybe one time in 50 it’ll do the AI response.
I don’t know if I changed a setting and forgot, or if they just don’t like me, or if their local processing can’t hack it. Either way, it makes me actually go to the websites, like the good old days of a couple of years ago! And I find better info doing that.
Honestly, the AI overview is crap. One time, searching for legal stuff about a potentially precedent setting case I’m involved in, it responded referring specifically to my case - as though it had been resolved and the precedent was set.
So yeah, I wouldn’t trust that shit to provide me a recipe for biscuits.
i am actively looking for a google alternative, they are horrible with their summaries and because it sounds confident, it throws you off and derails actually finding an answer. their search has been seo optimized clickbait for years but now with reddit becoming ai slop and another billionaire persona cleansing space its looking bleak
Ai
scrapersaremakingitkinda sucktorunanythingontheinternetto be honest.
I wonder if it would be possible to filter traffic by limiting the rate at which links can be followed after the initial connection.
I suppose one problem with that is it would also limit the crawlers you want, like search engine indexing. Maybe if enough sites were doing this it would generate some pressure on the AI companies to behave better.
If they have 10k URLs and in parallel find out all links via different IPs, they get new links.
When they do that again with the new links, there is no connection between visits, because the IPs differ. It looks like someone else is requesting what is behind the link.
If they have 10K IPs, then you need a DDOS filter, which would require a huge infrastructure no matter what.
The issue with wiki’s and source forges is that there is a maze of links to all past versions of everything, each generated on demand from a cpu-expensive database query. You basically have to limit the pages anonymous users can spider into. Forgejo has a setting to block expensive pages from non-logged in users for example.
Probably can configure anubis to require challenge that is proportional to the CPU time needed to render each page?
That seems to be a lot of peoples approach, but if they cared about time or bandwidth they wouldn’t be spidering Dow into your commit history multiple times a day. They have more patience and resources than your human readers.
I could see maybe caching that and providing it to a not-clearly-human user if it is in cache. That lets someone do something like link to a particular version of a file in a discussion here on the Threadiverse. The first user loading it will cause it to be cached.
Sure. Its just the thousands of obscure page edit history pages that ai crawlers hit every hour that cause the problem.
Wait a second…I already read this… 2 months ago
