Bots are currently scraping the internet for LLM training data at unprecedented rates[1][2][3], driving up costs and destabilizing public-facing websites. I want to talk about how this has been particularly difficult for wikis, and has gotten much worse in the last few months.
Probably can configure anubis to require challenge that is proportional to the CPU time needed to render each page?
That seems to be a lot of peoples approach, but if they cared about time or bandwidth they wouldn’t be spidering Dow into your commit history multiple times a day. They have more patience and resources than your human readers.