All Things API - nsaid-team/gotopaws GitHub Wiki

##Petfinder

Full petfinder api documentation is available at (https://www.petfinder.com/developers/api-docs). Petfinder allows a variety of GET calls against their api, requiring only simple key authentication (POST, PUT, and DELETE are not implemented at this time). A developer key can be requested from (https://www.petfinder.com/developers/api-key). Obtaining a key will entitle you to:

• Total requests per day: 10,000
• Records per request: 1,000
• Maximum records per search: 2,000

Responses are available in XML (by default) and JSON, which we used (specified using the fomrat=json argument and value). Several calls are available, but these are the ones we used:

• shelter.find
• shelter.getPets
• pet.get

Example call:

http://api.petfinder.com/shelter.get?key=2933122e170793b4d4b60358e67ecb65&id=TX1148&format=json

Example output:

`{
"@encoding": "iso-8859-1",
"@version": "1.0",
"petfinder": {
    "@xmlns:xsi": "http://www.w3.org/2001/XMLSchema-instance",
    "shelter": {
        "country": {
            "$t": "US"
        },
        "longitude": {
            "$t": "-97.7428"
        },
        "name": {
            "$t": "Small Chance Rescue"
        },
        "phone": {
            "$t": "512-699-7244    "
        },
        "state": {
            "$t": "TX"
        },
        "address2": {},
        "email": {
            "$t": "[email protected]"
        },
        "city": {
            "$t": "Austin"
        },
        "zip": {
            "$t": "78766"
        },
        "fax": {},
        "latitude": {
            "$t": "30.2669"
        },
        "id": {
            "$t": "TX1148"
        },
        "address1": {
            "$t": "P.O. Box 10033"
        }
    },
    "header": {
        "timestamp": {
            "$t": "2015-07-21T21:49:55Z"
        },
        "status": {
            "message": {},
            "code": {
                "$t": "100"
            }
        },
        "version": {
            "$t": "0.1"
        }
    },
    "@xsi:noNamespaceSchemaLocation": "http://api.petfinder.com/schemas/0.9/petfinder.xsd"
}
}`

##Yelp

The yelp api documentation is available at (https://www.yelp.com/developers/documentation/v2/overview)

Consumer Key KUvBeJ9O5nV23Bg5lBGLUA Consumer Secret K-xQZ9Y8moKw1C71Qsn61MaM0t0 Token VchJDQRV1LyH_ZHoawrotybgyuuVN-IW Token Secret eWqGCYQrFV9QcE2Ok6LuTvx6p7g

API v1.0 (deprecated) YWSID Key Y8oEk4_tEQD_vJIEgXj3Gg

We weren't able to grab these with just a rest call in a browser on account of oauth, so we created a short python script, dataset_creator.py, to rely on python modules to do it for us.

Google Search API

We used calls to (http://ajax.googleapis.com/ajax/services/search/web) to grab the external urls for shelters, since they were unavailable via the yelp api.

Scraping strategy

Our strategy for scraping APIs was to iterate over a static list of 5 cities, and use the Petfinder API to find several shelters per city. We would then grab several attributes from our petfinder request, and use the attributes of that request to further query the google api for a link to the shelter website, and query the yelp api for a short blurb describing the shelter. Then we wrote the final shelter objects to nsaid/fixtures/shelters_fixture.json.

Once we had a fixture file for our shelters, we parsed it, and iterated over the shelters, running a series of shelter.getPets calls to grab up to 10 pets per shelter. We created json objects out of the pets and dumped them to nsaid/fixtures/shelters_fixture.json with the appropriate fixture heading (https://docs.djangoproject.com/en/1.8/howto/initial-data/).

Finally, we created a city fixture file by iterating over the city list, and running a series of yelp queries to grab the top rated veterinarians, pet groomers, and dog parks in each city.

Each call also required that we use python replaces to sanitize the json of unicode characters so the charsets would play nicely with mysql, and to trim extra newlines so when the frontend of our site eventually displayed the objects they were formatted nicely.

Once we had fixture files, we were able to completely drop all tables, recreate them, and reload all of our json-backed records with the following commands:

 `python manage.py sqlclear nsaid | python manage.py dbshell ; python manage.py sql nsaid | python manage.py dbshell`
 `python ~/gotopaws/manage.py loaddata ~/gotopaws/nsaid/fixtures/{cities,shelters,pets}_fixture.json`

It was handy!

GotoPaws

API Documentation: http://docs.gotopaws1.apiary.io/#

API Blueprint Editor: https://app.apiary.io/gotopaws1/editor