https://blog.cloudflare.com/cloudflares-always-online-and-the-internet-archive-t... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		jgrahamc on Sept 17, 2020 \| parent \| context \| favorite \| on: Cloudflare and the Wayback Machine, joining forces... https://blog.cloudflare.com/cloudflares-always-online-and-th... We tell archive.org about the URI, they crawl it. They handle robots.txt.

AnonHP on Sept 18, 2020 [–]

archive.org doesn't handle robots.txt in any meaningful way (see my comment above at https://news.ycombinator.com/item?id=24516875 ). If that's changed recently, I'd like to know more.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact