Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
ziftface
on March 26, 2021
|
parent
|
context
|
favorite
| on:
Common Crawl
Well that's to be expected, but if a lot of search engines start using it, it's likely that websites will start allowing it to crawl and index their pages. So there might be potential there.
ianbutler
on March 26, 2021
[–]
It's not an issue of being allowed to crawl.
ziftface
on March 26, 2021
|
parent
[–]
Oh I guess I misunderstood then. Why don't they crawl at Google's scale?
ianbutler
on March 26, 2021
|
root
|
parent
[–]
Mostly it comes down to their crawler itself and the cost of storing all of that data, it's sufficiently prohibitive.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
