> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.
It's massively less annoying than a captcha, which is both a longer delay (typically, at present) and a massive cognitive distraction/roadblock.
The anubis author has stated they recognize it's an arms race, but PoW scales. Captchas and other signals are already at the end of the road; any additional difficulty increases false bot-positives, which are already unacceptably high.
For websites running dynamic languages, a binary (anubis is in go) sentry that operates before[1] the website is forced to expend any resources, is usually a large improvement over a site-hosted captcha. I would rather, and I think most humans would agree, have to wait a few seconds, maybe even closer to a minute in the future, to get a website access token good for a day or a week, than be forced to solve a captcha.
The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
[1] this is true regardless of whether anubis is in reverse proxy mode or auth mode.
Proof of work does not scale. It trades something fungible and incredibly cheap (CPU) for something incredibly expensive (user-visible latency). There is no set of parameters where the cost is going to be a meaningful deterrent to any kind of abuse (even something as low-yield as scraping) without adding crippling amounts of latency to real users.
> The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
There is no dilemma. They get a token, they maybe do some automated multi-armed bandit per-site to figure out how to maximize the extraction rate they get from a single token, and then they use an IP for that many requests / that amount of time before ditching it.
The success of the ongoing Anubis rollout proves the opposite. People are used to slowly-loading websites - the rise of garbage SPAs has seen to that. Staring at a spinner for a second every once in a while is not an issue for genuine users.
On the other hand, the additional CPU usage rises the compute cost of scraping by several orders of magnitude. If you don't have to scrape this specific website, you'd be stupid not to move on and hit someone else.
Ideally the general cost of scraping would be high enough that it isn't affordable any more - especially anonymized - but considering the amount of money brainlessly being pumped into AI I doubt that'll happen any time soon.
You could of course also make the argument that the user's time is worth something as well and should be included as part of the cost, but that ship sailed a loooong time ago. If you care about that, you should be calling for the death of client-side Javascript and any form of advertising.
Scrapers generally aren't looking for random websites to scrape. They have a specific URL in mind. Only if the goal was DDoS would they not care which URL was accessed.
It doesn't make the economics any different. In a browser environment, you're maybe looking at the acceptable lease being 100MB for 1 second. Much more than that, and you start hitting limits of what browsers will let you do on low-end phones. Longer than that, and we're back to the user-observable latencies being too long.
100MB for 1 second just is not much of a deterrent.
Sure, but end users as a group still have a significant amount of RAM.
Even on a low-specced machine you can afford to have the currently-active website tab consume a few hundreds of megabytes of RAM. It was mostly sitting idle anyways, so most people aren't even going to notice. Multiply that by a few thousand concurrent visitors and you're burning hundreds of gigabytes without anyone caring.
Some scraper being forced to install hundreds of gigabytes of extra RAM in their crawling node? They will notice having to go from a $0.50 / h instance to a $15.00 / h one, solely for the extra RAM requirement.
Anubis is by far the least annoying throttler I encounter. Entirely agreed, just crank it up when you get a flood, I much prefer waiting a couple seconds to interacting with custom UI for tens of seconds.
I'm so glad to see that (essentially) HashCash is coming back. Now we just need it for email, like it was originally designed for...
I looked this up and realised it’s the page I’d seen briefly on a range of websites lately. It’s not annoyed me at all. Not nearly as much as having to complete captchas with slow refreshing tiles.
The captcha itself (matching pictures to text) is mostly for ML training data. I think pass/fail is mostly based on heuristics like how you moved your mouse which could get analyzed before you complete the captcha. https://www.techradar.com/news/captcha-if-you-can-how-youve-...
reason why is 1. Google and others really needed the training data, and 2. it probably helped justify the cost of providing the captcha service for free worldwide (old free tier was 1M/mo)
That was Louis von Ahn’s stated goal when he announced it - i think it was his NIPS (now NeurIPS) talk where he basically outlined the training data paying for the infrastructure as a fair exchange.
A person's testimony is a source. I can add mine: you can tell when you're truly blocked because if you click for the accessibility audio-based captcha it will actually tell you you're blocked (but, if you did the visual captcha, would simply loop forever while telling you you did it wrong).
I don't think you'll find an article by Google saying "yes, we sometimes completely block users while making it look like they're not blocked and wasting their time".
I implemented something similar for my bot defences. If headless chrome is detected you still get the same anubis-style PoW but even if you submit the right answer you get rejected.
Imo the worst is recaptcha. At least with cloudflare the work you have to provide is minimal. With recaptcha it can take me much longer than 5 seconds, and lately I have trouble even completing their challenges correctly. Nowadays if I see a (recaptcha) captcha I drop the site unless I must visit it for some reason, it is not worth the time, the effort or the annoyance.
Most CF / Recaptcha problems are users going "off the golden path", and not realizing that their config changes are at fault.
If you're on a consumer router, using a mainstream stock browser with stock settings (maybe plus uBlock Origin), with your Google account logged in, it's very, very likely to just work. If you're part of the .01% of users with opinions about that sort of thing... you're not worth optimizing for.
At least for me, CF is fine; recaptcha is the only one I really have problems with.
I dont care what recaptcha wants to optimize for. I dont think that using a vpn is that a rare thing anyway. If others have figured out how to do it without requiring spending 30 seconds to solve a captcha, I dont see why websites still use recaptcha/captchas for that.
And that it is "my fault" not being logged into google I was least expecting to see here.
A lot of users also run cheap cracked fire sticks and other low reputation hardware that's proxying their residential traffic for nefarious reasons which makes all the big providers put up their guard.
A few months ago there was a story posted here about someone who completely eliminated crawlers on their website with Anubis.
I think it was getting upvoted before users were clicking the article because if you did, you had to leave the Anubis PoW page open for several minutes before you could get into the site. The Anubis difficulty scale is unintuitive and the difference between a small delay and becoming unusable is easy to cross.
From my understanding this is also how cloudflare bot protection has worked for a long time, and then they look for entropy in user input to confirm the user is human. Also how recaptcha without images works.
Google and Cloudflare both are not just looking at entropy of mouse movements, that was cracked years ago, they are fingerprinting you and correlating your session with all your activity cross domains to score your botlike behavior.
I doubt they are doing it. You just have to get on a VPN to and see yourself being flooded with captchas despite browsing the web like a normal human and solving dozens of captchas along the way.
Profiling visitor traffic to protect against abuse is a legitimate use case, even for GDPR. It's only a violation if that usage is not disclosed in the ToS.
Supposedly, but not really. I regularly encounter sites where cloudflare serves me with an ambiguous ban notice rather than a proof of work. What's worse is that these apparent IP bans take effect even if I already had a valid active session (ie previously passed the check).
Yes, a VPN involved. That doesn't make it okay and notice that anubis by default works without issue (though possibly with a more difficult challenge) in the exact same scenario.
Regardless of the precise logic it's no excuse for the policy. Simply hand out a sufficiently difficult PoW to prevent widespread abuse.
I'm quite certain it isn't a generic "datacenter" list though because a given VPN exit that was working will suddenly stop. Meanwhile I have a valid cookie yet that is disregarded.
That is an entirely different matter. They only block (AFAIK) editing which is a scenario where PoW would not be expected to solve the problem. What I was talking about here is IP banning as a (boneheaded) mitigation against high volume scrapers.
Except when it throws you into a reload loop. It's pretty buggy, and trivial to bypass.
And contrary to grandparent, PoW only worked because it was a novel thing to work around, a simple "type human" prompt would've worked as well.
When anubis gets widespread enough users will still run the PoW in javascript or whatever while the scrapers will run much more optimized native code, so no, it doesn't scale.
Putting aside the question of whether it will continue to work, even somewhat, against botnets, I find your first paragraph confusing.
Reload loops, or being able to "bypass" anubis (unless you merely mean bypassing it for the token validity period by solving a challenge), sound like misconfigurations. There's no reason for anubis itself to cause reload loops; it's tricky to configure a webserver to use it in some scenarios.
Any ability to bypass anubis probably means the site is using it in auth/challenge mode only, and then misconfigured their webserver's auth checking. Or it's a bug. If you mean the double-spend tavis mentioned in his blog post which previously made the HN frontpage, that was patched right after it was reported to the maintainer almost a year ago.
PoW barely affects the "residential proxies" aka. malware botfarms. The IPs are free for them and siphoning additional system resources for PoW doesn't matter at all for them. PoW only affects the large centralised scraping by the AI providers, which are not operating behind "residential proxies".
Most users of residential proxies just get a SOCKS5 address and routing, they don't actually get computational resources of the infected systems beyond that. The user of the proxies, the operator of what the article describes as a control node, would be the device responsible for the PoW.
Do you have any evidence that AI providers aren't using residential proxies?
If you pop my machine and use it to route 100 MBit/s, I might not notice for months.
If I hear the fan spinning at night, you're probably getting caught immediately.
If you pop my mom's TV box and use it to route data within the connection's capabilities, you're getting away with it. If you consume a little bit of resources, still. If you consume enough to be useful for these kind of challenges, chances are her TV playback will start to stutter, which will be resolved by taking the compromised TV box, and removing the malware using advanced mechanical means called "a trash compactor".
>chances are her TV playback will start to stutter
video decoding is hardware accelerated, and there's probably enough excess compute to be able to do some sort of PoW challenge. Besides, unlike humans, bots aren't in a hurry, so they can spread out the work across a long time to minimize disruption.
> The anubis author has stated they recognize it's an arms race, but PoW scales.
The scraper wars are largely between script kiddies and people with both deep intimate networking and DOM knowledge. Yes greyhairs, I’m looking at you.
The problem is, you can’t PoW every page load and resource request because the user experience will suck and people will run away. And that window - the gap between what people will tolerate vs draconian enforcement - is exactly what the scrapers exploit.
And looking at the PoW options out there - I’ve seen at least one PoW WAF (honestly can’t remember if azure or amazon) have their PoW boil down to repeated trigonometric functions, ie very optimisable.
It’s a neat concept, but the answer and future to my eyes look bleak.
Your typical end user doesn't switch IPs that often, so it's fine to Anubis them again when they do. A scraper, on the other hand, has a tradeoff to make between rotating ips often (requiring a challenge on every request) or keeping only a few IPs (making cross-request identification much more valuable and reliable).
> That’s why Anubis does an up front challenge and then you’re good for a while. It’s a really low cost for the scrapers.
Except that doing hundreds of requests from the same IP makes it pretty trivial to detect scraping, opening you up to being banned, or fun stuff like a slowloris or being fed poisoned data.
There are major websites that take more than 2-3 seconds to load every new page. Google, for example. Reddit. Facebook. Instagram. Doesn't seem to have hurt them.
Anubis's default 1-week token lifetime may not be nearly enough to dissuade enough scraper networks to make a difference, particularly with the default weight->difficulty level hierarchy, but that's for individual site admins to determine.
We can all argue based on how we envision "ideal" scraper networks being run and whether the web-PoW concept would stand up to that. However, what matters at present is that anubis helps many sites cope with misbehaving bot scrapers written by the script kiddies you mention, who don't care if the internet burns as long as they finish their scrape 1 hour faster. If anubis motivates them to devote a few brain cells to make their scrapers smarter, they may also fix the scrapers to not take down the sites they're scraping.
I don't think PoW scales, because if the bot authors get serious they'll start using native implementations that are much more efficient than the web ones real users are running. In theory maybe Anubis could start using WebGPU to help close that gap, but then anyone without WebGPU support is out of luck.
Then again, a large portion of the problem seems to be bots making way too many requests and in general not being optimized in the first place, and this does help filter those out.
If that happens the browser engines (all what, 3 of them?) can add a PoW API to call into native code. Or a pathologically scalar algorithm can be adopted so that wasm is good enough. RandomX or something close to it probably qualifies.
It's not about traffic identification at all, but rather a hashing algorithm that is deliberately resistant to parallelization and GPU/ASIC acceleration, which shrinks the gap in solving speed between the fastest systems (i.e. datacenter-class compute resources) and typical systems (e.g. the CPU in your smartphone or laptop).
Uh, is it resistant to parallelization across multiple sites? Because that's the situation for the scrapers. They're not trying to solve a single PoW challenge across many cores.
Typically, the machine doing the content processing, including solving PoW, is the centralized "control" node described in the article, not the machines who's IP addresses are being used. In typical residential proxy networks, the residential proxies are exposed to the customer (the person paying for and using the proxies) as just SOCKS5 addresses, and no computational power from those compromised devices is made available for the scraper besides that used to power the SOCKS5 server itself, the customer is just paying for the transport and address (and indeed, is often billed on either a per-GB or per-IP basis).
In effect, if the customer (the entity paying for and using the proxies) wants to solve PoW challenges through those connections, it is indeed the customer who must pay that compute cost, not the compromised devices.
Note that this is the case for a majority of, but not all, residential proxy networks, which often are built through quasi-voluntary distribution channels, including SDKs included in otherwise legitimate mobile applications distributed through Apple's App Store and Google Play.
These distribution channels tend to be categorically unavailable (or at least unreliable) for true RAT-style malware that enables remote operators to dynamically assign arbitrary computational workloads to client devices.
This isn't to say that true botnets built with actual malware delivered through either software exploits, phishing attacks, or watering hole attacks don't also perform as residential proxy networks, but such categories are a relatively small subset of all residential proxy networks, and there are much higher ROI malicious activities to be performed on these devices rather than serving as relatively mundane traffic networks for scraping.
Ah, I see what you're getting at. Yes. You can think of any given computer aa having a fixed amount of compute budget for these types of acceleration-resistant hashing algorithms. Let's say the scraper can perform 10,000 hashing operations per second total on their machine, and needs an average of 1,000 hashing operations to solve the PoW. It's a minor detail, but note that these PoW challenges non-deterministically vary in the number of hashing operations needed to produce a valid hash, not dissimilar to bitcoin mining, where a hash with a certain number of 0s prefixed is sought, and the scraper essentially has to brute force through all possible inputs until an input that produces a valid hash is found.
In a well-designed PoW systems, there is a per-site prefix or suffix that is required to be prepended or appended to these random inputs, and it may change not only between websites, but even between PoW sessions on the same website, and should not be predictable - only being disclosed to the client at the time the PoW challenge is issued. In such a case, the scraper cannot simply precompute a bunch of valid hashes that work across multiple sites, nor a bunch of valid hashes that will always be good for even one site, the scraper operator will need to compute these hashes (with a limited budget to do so) upon initiating each PoW session.
So, in your example, each request needs 0.1 core-seconds of PoW. Is that right? Or, since you're saying 10000 hashes across the entire machine and 1000 needed, then on a typical 96-core server you need 9.6 core-seconds of PoW? The former means you pay about $0.0000005 per request at standard cloud rates; the latter means you pay about $0.00005 per request _and_ your site is totally unusable by legitimate clients. Both are _easily_ worth it for someone backed by VC billions and hungry for data. The network and storage fees are likely to be more significant than that already, not to mention actually training a model; they don't balk at downloading terabytes of crap already.
Note that none of this assumes any sort of acceleration from parallelization (be it through GPUs or reusing work across servers) or precomputation relative to what a normal client does. Compute is just really cheap in dollars compared to the cost of having a user wait, and these companies _also_ have a lot of appetite for spending dollars compared to that of a normal user. As others have pointed out, the only reason why Anubis works (sort-of; not for everyone) right now is that it is uncommon enough, essentially “proof that you bothered to have your crawler run JavaScript at all”. It's a confusion measure.
You raise some good points here, but PoW is meant to be one tool in the toolbox, not the only line of defense. You can still maintain blocklists of known scrapers (or better yet, have your PoW system be aware of them and silently adjust the difficulty to an impossible level, such that the scraper gets stuck trying to solve your PoW challenge until it hits a timeout configured by the scraper, if they were wise enough to configure one). It's also courteous to not only build and maintain your own blocklists, but to share them with e.g. vtotal and spamhaus, to help protect others.
Similarly, you have tarpits, which generate infinite mazes of garbage data, or even deliberately poisoning training data (should the scrapers be training LLMs) though this don't entirely eliminate the deleterious effects of scraping on the host's web server (more info: https://arstechnica.com/tech-policy/2025/01/ai-haters-build-...).
If the premise was evaluating whether or not PoW would be a magic silver bullet that stops scrapers all by itself, then you are correct, it does not stop all scrapers. Scraping and anti-scraping is fundamentally a constantly evolving cat and mouse game that demands adaptability and punishes complacency from all participants trying not to lose.
I think you just described most of the reason I like it. Plain text sites tend to scale text, allow pinch to zoom, and are just generally faster.
Post deletion we've seen used for manipulation on other sites, it's a sticky topic. HN seems to want you to stand by your word, so it makes sense here.
That’s a terrible justification, and I would question if it’s even true. Is there anything to support that conclusion or is it just a guess you’ve made?
Because I don’t buy it. There is no time limit for discussions. One can take as much time as they like to enter text on their phone. Also if they’re motivated by the discussion they may take the time and make the effort to write thoughtful replies.
Finally, there’s no reason that users visiting on desktop computers won’t make equally shallow remarks. Shallow online discussions have existed long before smartphones existed.
Well, we don't use a captcha either. If it were a choice between a captcha and a proof of work system, we'd have to reevaluate things. Luckily, for now, we're able to get away with a much lighter touch.
Not if the honest party is doing it in a browser: The same computer can so any POW so much faster in C than any amount jf JS and WASM that it will never ever ever be a contest.
> becoming much more obvious and easy to block, or they have to use massive amounts of compute.
If you believe this, please contact me: I think compute is free[1] and can probably help you out.
Can you not design a PoW that is most efficient in a browser? Don't brute force hashes like Hashcash/Bitcoin, do something similar to RandomX instead but in JS. Browsers ought to run the fastest JS interpreters already so if interpreting JS becomes the bulk of the work, that attack might not work. Maybe even involve the DOM or whatever else makes sense.
> The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
You can't do that any more. Too many ISPs, especially mobile carriers, don't hand out anything resembling a fixed IP address any more. It's CGNAT and constantly changing IP addresses alllll the time now.
On a small enough site (even LWN might qualify) the chance of two random sets of client IPs intersecting can be quite low.
Private trackers do this. If they ban a user that geolocates to a certain city and ISP, they'll ban new signups from that city and ISP because there's probably only a few users from the same city and ISP. And then report to their friends at other trackers, that a user with that city and ISP is trying to evade a ban.
Anubis appears to be a temporarily-useful stopgap that has been cargo culted into prominence and an expectation of permanent usefulness, for reasons I don't fully understand.
The cost of solving the default Anubis PoW is negligible on cloud servers, and it's even lower if you use native code rather than JavaScript to solve it, which Tavis Ormandy helpfully demonstrated last year (https://lock.cmpxchg8b.com/anubis.html). If Anubis were to be even more widely adopted, botnet operators would surely adopt and optimize native code solvers en masse.
So Anubis doesn't do much to stop bots, but it makes otherwise lightweight websites (little JavaScript or interactivity) almost unusable on low-resource systems like my old phone or an old Atom-based nettop.
> when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape
This "IP-bound proof-of-work" thing is gonna kill multipath TCP and bring down IPv6 with it. Uffff.
> If Anubis were to be even more widely adopted, botnet operators would surely adopt and optimize native code solvers en masse.
Then anubis adopts it itself, increases the amount of work that needs to be done and the bar stays the same again for everyone? Seems like mostly a non-issue unless there is an arms race towards ever more optimized solvers which I don't believe is possible.
PoW can theoretically scale effectively infinite because it can mine cryptocurrency. Millions of compromised IoT devices hitting your server? Now you have enough money for a faster server.
It doesn’t matter that the challenge must be verified: present multiple challenges, some are verified while others mine crypto.
But why? Obviously an unjustified cryptominer is bad, like unnecessarily slow JavaScript, but this one has a good purpose and to the user is no different than PoW.
It's massively less annoying than a captcha, which is both a longer delay (typically, at present) and a massive cognitive distraction/roadblock.
The anubis author has stated they recognize it's an arms race, but PoW scales. Captchas and other signals are already at the end of the road; any additional difficulty increases false bot-positives, which are already unacceptably high.
For websites running dynamic languages, a binary (anubis is in go) sentry that operates before[1] the website is forced to expend any resources, is usually a large improvement over a site-hosted captcha. I would rather, and I think most humans would agree, have to wait a few seconds, maybe even closer to a minute in the future, to get a website access token good for a day or a week, than be forced to solve a captcha.
The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
[1] this is true regardless of whether anubis is in reverse proxy mode or auth mode.