PROBLEM
STATEMENT
The client needed the data for all the real estate agents in the United States of America. The difficult part was to retrieve such a huge dataset without getting blocked. Also, it was essential to cover all the states and avoid duplication because certain agents are working in more than one state.
SOLUTION
We implemented a smart algorithm with a multi-level crawler to make sure that all the real estate agents are being found. We scraped multiple websites to gather an extensive amount of data and used proxies to prevent blocking and other issues. The extracted information included bio, area, reviews, experience, jobs completed, and company name. The data was then cleaned and checked for duplication before sending the final reports to the client.
Input
List of things that were to be found for the real estate agents of the United States
Output
Excel sheet containing all the required information about the real estate agents of the United States
Tools &
Technologies
Python (Scrapy)
Selenium
