Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

It won't help with the more egregious scrapers, but this list is handy for telling the ones that do respect robots.txt to kindly fuck off:

https://github.com/ai-robots-txt/ai.robots.txt



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: