Well this was a mammoth task at the first look but we organized the job into smaller scrapers and guess what. We ended up with 21345 micro scrapers which scraped the entire listed 5 sites in less than 36 hours.
In total there are approx 18 Million search results and after removing duplicates we got almost 11.8Million results.
The scraper is built in php and can be set up as a cron job and uses 3 MySQL databases to complete the job.
If you are also looking for scraping these sites then we can offer an extremely competitive rates and also configure it on your server[s].
Source: http://www.escalateseo.com/scraping-top-real-estate-sites-from-us/
No comments:
Post a Comment