Coscrape
An AE Studio lead-generation scraper for AngelList and Crunchbase: Node workers with PhantomJS/Horseman, Tor and rotating proxies, MongoDB, a company ranking score, email validation and CSV export, with a React frontend and docker-compose scaling.
- BPM
- 169
- GRADE
- B
- LEVEL
- 6
- RELEASES
- 0
- YEARS
- 2017–2017
Scraping For Our Own Leads
At Agency Enterprise we sold product development to startups, so we wanted to know which startups existed and which ones had just raised money. In September 2017 we built a tool for it.
Coscrape pulled companies out of AngelList and Crunchbase. Node workers drove PhantomJS through Horseman. Everything landed in MongoDB. On top of that was a scoring pass that ranked companies by how good a fit they looked, an email validation step to weed out dead addresses, and CSV export at the end. There was a React frontend and docker-compose to scale the workers out.
It ran from September 21st to the end of November, about ten weeks. It was the same fall as BitFax, the crypto tooling and the Woodpecker API client. Woodpecker was a cold email platform, so the scraper fed the top of the funnel and our other internal tools handled the rest.