It seems a bit naive for some reason and doesn't do performance back-off the way I would expect from Google Bot. It just kept repeatedly requesting more and more until my server crashed, then it would back off for a minute and then request more again.
My solution was to add a Cloudflare rule to block requests from their User-Agent. I also added more nofollow rules to links and a robots.txt but those are just suggestions and some bots seem to ignore them.
This is already a thing for basically all of the second[0] and third worlds. A non-trivial amount of Cloudflare's security value is plausible algorithmic discrimination and collective punishment as a service.
[0] Previously Soviet-aligned countries; i.e. Russia and eastern Europe.
If 90% of your problem users come from 1-2 countries, seems pretty sensible to block that country. I know I have 0 paying users in those countries, so why deal with it? Let them go fight it out doing bot wars in local sites
Anecdatally, by default, we now block all Chinese and Russian IPs across our servers.
After doing so, all of our logs, like ssh auth etc, are almost completely free and empty of malicious traffic. It’s actually shocking how well a blanket ban worked for us.
Being slightly annoyed by noise in SSH logs I’ve blocked APNIC IPs and now see a comparable number of brute force attempts from ARIN IPs (mostly US ones). Geo blocks are totally ineffective against TAs which use a global network of proxies.
~20 years ago I worked for a small IT/hosting firm, and the vast majority of our hostile traffic came from APNIC addresses. I seriously considered blocking all of it, but I don’t think I ever pulled the trigger.
> Anecdatally, by default, we now block all Chinese and Russian IPs across our servers.
This. Just get several countries' entire IP address space and block these. I've posted I was doing just that only to be told that this wasn't in the "spirit" of the Internet or whatever similar nonsense.
In addition to that only allow SSH in from the few countries / ISPs legit trafic shall legitimately be coming from. This quiets the logs, saves bandwidth, saves resources, saves the planet.
I agree with your approach. It’s easy to empathize with innocent people in say, Russia, blocked from a site which has useful information to them. However the thing these “spirit/openness” people miss is that many sites have a narrow purpose which makes no sense to open it up to people across the world. For instance, local government. Nobody in India or Russia needs to see the minutes from some US city council meeting, or get building permit information. Likewise with e-commerce. If I sell chocolate bars and ship to US and Canada, why wouldn’t I turn off all access from overseas? You might say “oh, but what if some friend in $COUNTRY wants to order a treat for someone here?” And the response to that is always “the hypothetical loss from that is minuscule compared to the cost of serving tons of bot traffic as well as possible exploits those bots might do.
(Yes, yes, VPNs and proxies exist and can be used by both good and bad actors to evade this strategy, and those are another set of IPs widely banned for the same reason. It’s a cat and mouse game but you can’t argue with the results)
Having a door with a lock on it prevents other people from committing crime in my house. This metaphor has the added benefit of making some amount of sense in context.
It's unclear that there are actors below the regional-conglomerate-of-nation-states level that could credibly resolve the underlying issues, and given legislation and enforcement regimes sterling track record of resolving technological problems realistically it seems questionable that solutions could exist in practice. Anyway this kind of stuff is well outside the bounds of what a single org hosting an online forum could credibly address. Pragmatism uber alles.
The underlying issue is that countries like russia support abuse like this. So by blocking them perhaps the people there will demand that their govt stops supporting crimes and absuse so that they can be allowed back into the internet.
(In the case of russians though i guess they will never change)
> people there will demand that their govt stops supporting crimes and absuse so that they can be allowed back into the internet
Sure. It doesn't work that way, not in Russia or China. First they have to revert back to 1999 when Putin took over. Then they have to extradite criminals and crack down on cybercrime. Then maybe they could be allowed back onto the open Internet.
In my country one would be exradited to the US in no time. In fact the USSS came over for a guy who had been laundering money through BTC from a nearby office. Not a month passed and he got extradited to the US, never to be heard from again.
It's of course trivially bypassable with a VPN, but getting a 403 for an innocent get request of a public resource makes me angry every time nonetheless.
No, Russia is by definition the 2nd world. It's about spheres of influence, not any kind of economic status. The First World is the Western Bloc centered around the US, the Second World is the Eastern Bloc centered around then-USSR and now-Russia (although these days more centered on China), the Third World is everyone else.
By which definition? Here’s the first result in google: “The term "second world" was initially used to refer to the Soviet Union and countries of the communist bloc. It has subsequently been revised to refer to nations that fall between first and third world countries in terms of their development status and economic indicators.” https://www.investopedia.com/terms/s/second-world.asp#:~:tex....
What do you mean crushing risk? Just solve these 12 puzzles by moving tiny icons on tiny canvas while on the phone and you are in the clear for a couple more hours!
If you live in a region which it is economically acceptable to ignore the existence of (I do), you sometimes get blocked by website r̶a̶c̶k̶e̶t̶ protection for no reason at all, simply because some "AI" model saw a request coming from an unusual place.
I have come across some websites that block me using Cloudflare with no way of solving it. I’m not sure why, I’m in a large first-world country, I tried a stock iPhone and a stock Windows PC, no VPN or anything.
I saw GDPR related blockage like literally twice in a few years and I connect from EU IP almost all the time
Overload of captcha is not about GDPR...
but the issue is strange. @benhurmarcel I would check if there is somebody or some company nearby abusing stuff and you got under the hammer. Maybe unscrupulous VPN company. Using a good VPN can in fact make things better (but will cost money) or if you have a place to put your own all the better. otherwise check if you can change your IP with provider or change providers or move I guess...
not to excuse CF racket but as this thread shows the data hungry artificial stupidity leaves no choice to some sites
This may be too paranoid, but if your mobile IP is persistent and phone was compromised and is serving as a proxy for bots then it could explain why your IP fell out of favor
If it clears you at all. I accidentally set a user agent switcher on for every site instead of the one I needed it for, and Cloudflare would give me an infinite loop of challenges. At least turning it off let me use the Internet again.
These features are opt-in and often paid features. I struggle to see how this is a "crushing risk," although I don't doubt that sufficiently unskilled shops would be completely crushed by an IP/userAgent block. Since Cloudflare has a much more informed and broader view of internet traffic than maybe any other company in the world, I'll probably use that feature without any qualms at some point in the future. Right now their normal WAF rules do a pretty good job of not blocking legitimate traffic, at least on enterprise.
The risk is not to the company using Cloudflare; the risk is to any legitimate individual who Cloudflare decides is a bot. Hopefully their detection is accurate because a false positive would cause great difficulties for the individual.
For months, my Firefox was locked out of gitlab.com and some other sites I wanted to use, because CloudFlare didn't like my browser.
Lesson learned: even when you contact the sales dept. of multiple companies, they just don't/can't care about random individuals.
Even if they did care, a company successfully doing an extended three-way back-and-forth troubleshooting with CloudFlare, over one random individual, seems unlikely.
I see a lot of traffic I can tell are bots based on the URL patterns they access. They do not include the "bot" user agent, and often use residential IP pools.
I haven't found an easy way to block them. They nearly took out my site a few days ago too.
You could run all of your content through an LLM to create a twisted and purposely factually incorrect rendition of your data. Forward all AI bots to the junk copy.
Everyone should start doing this. Once the AI companies engorge themselves on enough garbage and start to see a negative impact to their own products, they'll stop running up your traffic bills.
Maybe you don't even need a full LLM. Just a simple transformer that inverts negative and positive statements, changes nouns such as locations, and subtly nudges the content into an erroneous state.
Self plug, but I made this to deal with bots on my site: https://marcusb.org/hacks/quixotic.html. It is a simple markov generator to obfuscate content (static-site friendly, no server-side dynamic generation required) and an optional link-maze to send incorrigible bots to 100% markov-generated non-sense (requires a server-side component.)
This is cool! It'd have been funny for this to become mainstream somehow and mess with LLM progression. I guess that's already happening with all the online AI slop that is being re-fed into its training.
I tested it on your site and I'm curious, is there a reason why the link-maze links are all gibberish (as in "oNvUcPo8dqUyHbr")? I would have had links be randomly inserted in the generated text going to "[random-text].html" so they look a bit more "real".
Its unfinished. At the moment, the links are randomly generated because that was an easy way to get a bunch of unique links. Sooner or later, I’ll just get a few tokens from the markov generator and use those for the link names.
I’d also like to add image obfuscation on the static generator side - as it stands now, anything other than text or html gets passed through unchanged.
> You could run all of your content through an LLM to create a twisted and purposely factually incorrect rendition of your data. Forward all AI bots to the junk copy.
> Everyone should start doing this. Once the AI companies engorge themselves on enough garbage and start to see a negative impact to their own products, they'll stop running up your traffic bills.
I agree, and not just to discourage them running up traffic bills. The end-state of what they hope to build is very likely to be extremely for most regular people [1], so we shouldn't cooperate in building it.
[1] And I mean end state. I don't care how much value you say you get from some AI coding assistant today, the end state is your employer happily gets to fire you and replace you with an evolved version of the assistant at a fraction of your salary. The goal is to eliminate the cost that is our livelihoods. And if we're lucky, in exchange we'll get a much reduced basic income sufficient to count the rest of our days from a dense housing project filled with cheap minimum-quality goods and a machine to talk to if we're sad.
Or maybe solve a small sha2(sha2()) leading zeroes challenge, taking ~1 second of computer time. Normal users won't notice, and bots will earn you Bitcoins :)
> Everyone should start doing this. Once the AI companies engorge themselves on enough garbage and start to see a negative impact to their own products, they'll stop running up your traffic bills
Or just wait for after the AI flood has peaked & most easily scrapable content has been AI generated (or at least modified).
We should seriously start discussing the future of the public web & how to not leave it to big tech before it's too late. It's a small part of something i am working on, but not central. So i haven't spend enough time to have great answers. If anyone reading this seriously cares, i am waiting desperately to exchange thoughts & approaches on this.
Very tangential but you should check out the old game “Hacker BS Replay”.
It’s basically about how in 2012, with the original internet overrun by spam, porn and malware, all the large corporations and governments got together and created a new, tightly-controlled clean internet. Basically how modern Apple & Disneyland would envision the internet. On this internet you cannot choose your software, host your own homepage or have your own e-mail server. Everyone is linked to a government ID.
We’re not that far off:
- SaaS
- Gmail blocking self-hosted mailservers
- hosting your own site becoming increasingly cumbersome, and before that MySpace and then Meta gobbled up the idea of a home page a la GeoCities.
- Secure Boot (if Microsoft locked it down and Apple locked theirs, we would have been screwed before ARM).
- Government ID-controlled access is already commonplace in Korea and China, where for example gaming is limited per day.
In the Hacker game, as a response to the new corporate internet, hackers started using the infrastructure of the old internet (“old copper lines”) and set something up called the SwitchNet, with bridges to the new internet.
Agree. The bots are already significantly better at passing almost every supposed "Are You Human?" test than the actual humans. "Can you find the cars in this image?" Bots are already better. "Can you find the incredibly convoluted text in this color spew?" Bots are already better. Almost every test these days is the same "These don't make me feel especially 'human'. Not even sure what that's an image of. Are there even letters in that image?"
Part of the issue, the humans all behaved the same way previously. Just slower.
All the scraping, and web downloading. Humans have been doing that for a long time. Just slower.
It's the same issue with a lot of society. Mean, hurtful humans, made mean hurtful bots.
Always the same excuses too. Company / researchers make horrible excrement, knowing full well its going harm everybody on the world wide web. Then claim they had no idea. "Thoughts and prayers."
The torture that used to exist on the world wide web of copy-pasta pages and constant content theft, is now just faster copy-pasta pages and content theft.
My cheap and dirty way of dealing with bots like that is to block any IP address that accesses any URLs in robots.txt. It's not a perfect strategy but it gives me pretty good results given the simplicity to implement.
I don't understand this. You don't have routes your users might need in robots.txt? This article is about bots accessing resources that other might use.
Too many ways to list here, and implementation details will depend on your hosting environment and other requirements. But my quick-and-dirty trick involves a single URL which, when visited, runs a script which appends "deny from foo" (where foo is the naughty IP address) to my .htaccess file. The URL in question is not publicly listed, so nobody will accidentally stumble upon it and accidentally ban themselves. It's also specifically disallowed in robots.txt, so in theory it will only be visited by bad bots.
Another related idea: use fail2ban to monitor the server access logs. There is one filter that will ban hosts that request non-existent URLs like WordPress login and other PHP files. If your server is not hosting PHP at all it's an obvious sign that the requests are from bots that are probing maliciously.
TLS fingerprinting still beats most of them. For really high compute endpoints I suppose some sort of JavaScript challenge would be necessary. Quite annoying to set up yourself. I hate cloudflare as a visitor but they do make life so much easier for administrators
You rate limit them and then block the abusers. Nginx allows rate limiting. You can then block them using fail2ban for an hour if they're rate limited 3 times. If they get blocked 5 times you can block them forever using the recidive jail.
I've had massive AI bot traffic from M$, blocked several IPs by adding manual entries into the recidive jail. If they come back and disregard robots.txt with disallow * I will run 'em through fail2ban.
Whatever M$ was doing still baffles me. I still have several azure ranges in my blocklist because whatever this was appeared to change strategie once I implemented a ban method.
They were hammering our closed ticketing system for some reason. I blocked an entire C block and an individual IP. If needed I will not hesitate banning all their ranges, which means we won't get any mail from Azure, M$ office 365, since this is also our mail server. But scew'em, I'll do it anyway until someone notices, since it's clearly abuse.
Maybe, but impact can also make a pretty viable case.
For instance, if you own a home you may have an easement on part of your property that grants other cars from your neighborhood access to pass through it rather than going the long way around.
If Amazon were to build a warehouse on one side of the neighborhood, however, it's not obvious that they would be equally legally justified to send their whole fleet back and forth across it every day, even though their intent is certainly not to cause you any discomfort at all.
It's like these AI companies have to invent scraping spiders again from scratch. I don't know how often I have been ddosed to complete site failure and still ongoing by random scrapers just the last few months.
If I make a physical robot and it runs someone over, I'm still liable, even though it was a delivery robot, not a running over people robot.
If a bot sends so many requests that a site completely collapses, the owner is liable, even though it was a scraping bot and not a denial of service bot.
Doubt it, a vanilla cease-and-desist letter would probably be the approach there. I doubt any large AI company would pay attention though, since, even if they're in the wrong, they can outspend almost anyone in court.
You can also block by IP. Facebook traffic comes from a single ASN and you can kill it all in one go, even before user agent is known. The only thing this potentially affects that I know of is getting the social card for your site.
> If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really).
It's really absurd that they seem to think this is acceptable.
> Oh, and of course, they don’t just crawl a page once and then move on. Oh, no, they come back every 6 hours because lol why not. They also don’t give a single flying fuck about robots.txt, because why should they. And the best thing of all: they crawl the stupidest pages possible. Recently, both ChatGPT and Amazon were - at the same time - crawling the entire edit history of the wiki.
Most administrators have no idea or no desire to correctly configure Cloudflare, so they just slap it on the whole site by default and block all the legitimate access to e.g. rss feeds.
Some of them, and initially only by accident. And without the ingredients to create your own.
Meta is trying to kill OpenAI and any new FAANG contenders. They'll commoditize their complement until the earth is thoroughly salted, and emerge as one of the leading players in the space due to their data, talent, and platform incumbency.
They're one of the distribution networks for AI, so they're going to win even by just treading water.
I'm glad Meta is releasing models, but don't ascribe their position as one entirely motivated by good will. They want to win.
And I doubt Facebook implemented something that actually saturates the network, usually a scraper implements a limit on concurrent connections and often also a delay between connections (e.g. max 10 concurrent, 100ms delay).
Chances are the website operator implemented a webserver with terrible RAM efficiency that runs out of RAM and crashes after 10 concurrent requests, or that saturates the CPU from simple requests, or something like that.
I've seen concurrency in excess of 500 from Metas crawlers to a single site. That site had just moved all their images so all the requests hit the "pretty url" rewrite into a slow dynamic request handler. It did not go very well.
Can't every webserver crash due to being overloaded? There's an upper limit to performance of everything. My website is a hobby and has a budget of $4/mo budget VPS.
Perhaps I'm saying crash and you're interpreting that as a bug but really it's just an OOM issue cause of too many in-flight requests. IDK, I don't care enough to handle serving my website at Facebook's scale.
I wouldn't expect it to crash in any case, but I'd generally expect that even an n100 minipc should bottleneck on the network long before you manage to saturate CPU/RAM (maybe if you had 10Gbit you could do it). The linked post indicates they're getting ~2 requests/second from bots, which might as well be zero. Even low powered modern hardware can do thousands to tens of thousands.
I've worked on multiple sites like this over my career.
Our pages were expensive to generate, so what scraping did is blew out all our caches by yanking cold pages/images into memory. Page caches, fragment caches, image caches, but also the db working set in ram, making every single thing on the site slow.
Usually ones that are written in a slow language, do lots of IO to other webservices or databases in a serial, blocking fashion, maybe don't have proper structure or indices in their DBs, and so on. I have seen some really terribly performing spaghetti web sites, and have experience with them collapsing under scraping load. With a mountain of technical debt in the way it can even be challenging to fix such a thing.
Even if you're doing serial IO on a single thread, I'd expect you should be able to handle hundreds of qps. I'd think a slow language wouldn't be 1000x slower than something like functional scala. It could be slow if you're missing an index, but then I'd expect the thing to barely run for normal users; scraping at 2/s isn't really the issue there.
Run a mediawiki, as described in the post. It's very heavy.
Specifically for history I'm guessing it has to re-parse the entire page and do all link and template lookups because previous versions of the page won't be in any cache
The original post says it's not actually a burden though; they just don't like it.
If something is so heavy that 2 requests/second matters, it would've been completely infeasible in say 2005 (e.g. a low power n100 is ~20x faster than the athlon xp 3200+ I used back then. An i5-12600 is almost 100x faster. Storage is >1000x faster now). Or has mediawiki been getting less efficient over the years to keep up with more powerful hardware?
> And I mean that - they indexed every single diff on every page for every change ever made. Frequently with spikes of more than 10req/s. Of course, this made MediaWiki and my database server very unhappy, causing load spikes, and effective downtime/slowness for the human users.
Does MW not store diffs as diffs (I'd think it would for storage efficiency)? That shouldn't really require much computation. Did diffs take 30s+ to render 15-20 years ago?
For what it's worth my kiwix copy of Wikipedia has a ~5ms response time for an uncached article according to Firefox. If I hit a single URL with wrk (so some caching at least with disks. Don't know what else kiwix might do) at concurrency 8, it does 13k rps on my n305 with a 500 us average response time. That's over 20Gbit/s, so basically impossible to actually saturate. If I load test from another computer it uses ~0.2 cores to max out 1Gbit/s. Different code bases and presumably kiwix is a bit more static, but at least provides a little context to compare with for orders of magnitude. A 3 OOM difference seems pretty extreme.
Incidentally, local copies of things are pretty great. It really makes you notice how slow the web is when links open in like 1 frame.
According to MediaWiki it gzips diffs [1]. So to render a previous version of the page I guess it'd have to unzip and apply all diffs in sequence to render the final version of the page.
And then it depends on how efficient the queries are at fetching etc.
Yeah, this is the sort of thing that a caching and rate limiting load balancer (e.g. nginx) could very trivially mitigate. Just add a request limit bucket based on the meta User Agent allowing at most 1 qps or whatever (tune to 20% of your backend capacity), returning 429 when exceeded.
Of course Cloudflare can do all of this for you, and they functionally have unlimited capacity.
I did read the article. I'm skeptical of the claim though. The author was careful to publish specific UAs for the bots, but then provided no extra information of the non-bot UAs.
>If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet.
I'm also skeptical of the need for _anyone_ to access the edit history at 10 qps. You could put an nginx rule on those routes that just limits the edit history page to 0.5 qps per IP and 2 qps across all IPs, which would protect your site from both bad AI bots and dumb MediaWiki script kiddies at little impact.
>Oh, and of course, they don’t just crawl a page once and then move on. Oh, no, they come back every 6 hours because lol why not.
And caching would fix this too, especially for pages that are guaranteed not to change (e.g. an edit history diff page).
Don't get me wrong, I'm not unsympathetic to the author's plight, but I do think that the internet is an unsafe place full of bad actors, and a single bad actor can easily cause a lot of harm. I don't think throwing up your arms and complaining is that helpful. Instead, just apply the mitigations that have existed for this for at least 15 years, and move on with your life. Your visitors will be happier and the bots will get boned.
It seems a bit naive for some reason and doesn't do performance back-off the way I would expect from Google Bot. It just kept repeatedly requesting more and more until my server crashed, then it would back off for a minute and then request more again.
My solution was to add a Cloudflare rule to block requests from their User-Agent. I also added more nofollow rules to links and a robots.txt but those are just suggestions and some bots seem to ignore them.
Cloudflare also has a feature to block known AI bots and even suspected AI bots: https://blog.cloudflare.com/declaring-your-aindependence-blo... As much as I dislike Cloudflare centralization, this was a super convenient feature.