Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
jsheard
on Dec 30, 2024
|
parent
|
context
|
favorite
| on:
AI companies cause most of traffic on forums
It won't help with the more egregious scrapers, but this list is handy for telling the ones that do respect robots.txt to kindly fuck off:
https://github.com/ai-robots-txt/ai.robots.txt
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
https://github.com/ai-robots-txt/ai.robots.txt