Week 01 (W46 Nov16) London RE - Rostlab/DM_CS_WS_2016-17 GitHub Wiki
The initial proposal for mining the data included crawling the data from rightmove.co.uk. We found out that the platform zoopla.co.uk, which also lists properties for sale and rent provides an easy to use api, which would allow us to fetch the data we need with less effort. The documentation of the api can be found here here. The api provides different formats for the results (xml, json, ...) what will be very convenient. The only downside is that the api has a rate limit of 1000 calls per day (or hour - it was not clearly mentioned), but if this is an issue we will spilt the requests to be done on several days.
The following steps are:
- registering for an api key (already done)
- using the
GET Search Property Listings /api/v1/property_listings.jsonto get the data - convert the
jsontocsv - set up an sqlite database
Our request will contain the following parameters:
http://api.zoopla.co.uk/api/v1/property_listings.json?api_key=our_api_key&listing_status=rent&page_size=100&area=London&page_number=1
| parameter | meaning |
|---|---|
| api_key=our_api_key | our personal api key |
| listing_status=rent | we only want properties that are for rent |
| page_size=100 | how many objects can be fetched with one call (has to be set btwn 1 an 100) |
| area=London | retrieve objects from London |
| page_number=1 | the current page number (we'll fetch around 700 pages) |