Hacker Newsnew | past | comments | ask | show | jobs | submit | OlikEten's commentslogin

there are many web scraping companies doing this, among them Apify too - https://apify.com/compass/crawler-google-places



("Scraping Fish" owns this site but it isn't disclosed anywhere, and this guy "rustdeveloper" seems associated with them, most of his comments push that service)


This is correct, my friends from Scraping Fish are hosting https://compareproxy.com to help people find proxy for web scraping. I'm happy to "push" for Scraping Fish as I'm also a satisfied user who received a lot of help from the founders for my web scraping projects.


I am surprised nobody mentioned https://apify.com/ and they even offer discount for YC startups as ex-graduate from the YC Combinator program


Nice! Finally a web scraping library for the programming language most websites use. It was about time lol


Thank you! And exactly as you say, using the same language as the websites gives us some advantages - we have HTTP-only (no client-side JS) crawlers based on the Cheerio library, which mimics jQuery API, and if you later find out that you need to use a full headless browser with Puppeteer, you can just call the utility function injectJquery, and there's very little you have to modify to keep your script working


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: