Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Scrapers generally aren't looking for random websites to scrape. They have a specific URL in mind. Only if the goal was DDoS would they not care which URL was accessed.


They also don't care which URL was accessed when the goal is scraping as much text as possible for AI training.




Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: