Company Profiler.
Post it a URL and it comes back with the emails, phone numbers, social accounts, locations, company name and sector. Two services, one job each.
What it is
Two Flask services that talk to each other over HTTP. scraper-app takes a URL, goes looking for the pages that usually hold something useful, contact, about, and pulls out the text and the links while throwing away the icons and the decoration.
extractor-app takes that text and finds the emails, the phone numbers, the social accounts, the addresses, the company name and the sector. One POST endpoint each.
Why I made it
I was doing this by hand. Open the site, hunt for the contact page, copy the email, copy the phone number, guess the sector, paste it into a sheet, then do it again forty more times.
How I made it
Python and Flask. Splitting it across two services meant the reading and the understanding could change without disturbing each other, which is the whole reason it survived past the first version.
Everything writes to a log file. That turned out to matter more than anything clever in the parsing, because almost every failure is one strange page rather than a bug.
What I learned
Websites are not built to be read this way, and they never will be. Half the work is deciding which page is the contact page when the site calls it something else entirely. The other half is deciding that a run of digits is a phone number and not one of the four other things a run of digits can be.
It reads public pages, and it should only be pointed at sites whose terms allow it.
