• Goals:
  • Understanding better the API REST
  • Practicing with public APIs
• Duration: 2h
• Date: Week 10: Tuesday, March-26th-2019
• This session consist of the teacher's guidelines for driving the Lecture, following the learn by doing approach

Contents

• Introduction
• Accessing to remote data: API REST
  • Finding public APIs
  • Accessing the services
• Practicing
  • RandomCat
  • Exercise 1: Chuck Norris
  • Playing with the weather
    • Exercise 2: What's the weather like i the capitals of the world?
  • Getting information from github
    • Exercise 3: Informtion about a github user
• Authors
• Credits
• License

Introduction

It is time to develop simple python clients for getting data from servers around the world. We will learn by practicing

Accessing to remote data: API REST

For accessing to remote data, first we need to know what servers are available, and what is the web interface they provide

Finding public APIs

Here you can find a list of public APIs for accessing the data: weather, animals, human gnome, drugs, github...

List of public web APIs

Let's have a look!

On the left we find the link to the server that provides the service. On the right we find some information on the type of access to the server:

• HTTPS: Some services are provided through the HTTP protocol, that we already know, but others use the HTTPS. It is similar to HTTP, but the information is encrypted, so its more secure. We should know if the service is provided by HTTP or HTTPS in order to program our clients

• AUTH: For using some APIs, authentication is required. Other APIs, like github, have some data available without authentication, and is required for another. The authentication can be:

  • OAuth
  • API Key

Accessing the services

For accessing to the services/data provided by the server, we should read its API documentation carefully. Every server defines its own names and parameters. It is important to understand the terminology:

• Hostname: We need to know the hostname of the server we want to access. This name should NOT include http:// or a slash in the end. This is an example of a correct hostname: aws.random.cat

• HTTP method: We should know the method used for the request: GET or POST

• Headers: Some services require the client to send information on the request heathers

• The endpoints: This is the name given to the resources provided by the server. For example: "/meow", or "/users"

• The parameters: Some endpoints require parameters. Some parameters are optional, others mandatory. Parameters can be given in two places:

  • As part of the endpoint name: For example, for accessing to the user repos in github, the endpoint is like this: /users/:user/repo. In this example, :user is the parameter and should be changed for the real username
  • As an external parameter: This parameters are located after the ? symbol, in the end of the endpoint. The syntax is: "parameter=value". Different parameters are separated by the & or + symbols. For example: /sequence/id/ENSG00000157764?content-type=application/json

• The data FORMAT: We should know in which format the DATA is returned: JSON, XML, other...

Practicing

Practicing is everything in the programer's world. So, Let's practice!

RandomCat

We all know that the fast development of Internet was due to the cats. It is a fact! :-) If we have a look at the public APIs we will find some of them related to cats. Let's chose this one: RandomCat. This is a services that returns the URL of a random cat image. So, everytime we invoke the service, we will receive a different URL (that correspond to a different image)

The service provided by the RandomCat server is so simple, that there is not have any documentation. So, let's try to figure out the information by ourselves. Before programming anything, it is important to test the API with the Browser, to check if we understand it correctly. Then we can proceed to program the client

Let's open this URL in the browser: RandomCat

We analyze it and write down all the required information for accessing to it:

• Protocol: HTTPS. We can see it in the URL
• Host name: aws.random.cat. Also from the URL
• HTTP method: We do not know. Let's check the headers:

Now we can see more information:

• HTTP method: GET. This is what the header Access-Control-Allow-Methods say
• DATA FORMAT: JSON. We have checked in on the Content-Type header: application/json
• Endpoint: /meow
• Parameters: none

Analyzing the JSON file, we see that it represents an object, which has the file property. The value of this property is the URL where the image is located

Now that we have collected all the informatio, we can proceed to program the client. It will connect to the server, get the URL and print it on the console

# Example of accessing to the RandomCAT service for getting an URL
# of a random image of a CAT. This clients just print it on
# the console

import http.client
import json

# -- API information
HOSTNAME = "aws.random.cat"
ENDPOINT = "/meow"
METHOD = "GET"

# -- Here we can define special headers if needed
headers = {'User-Agent': 'http-client'}

# -- Connect to the server
# -- NOTICE it is an HTTPS connection!
# -- If we do not specify the port, the standar one
# -- will be used
conn = http.client.HTTPSConnection(HOSTNAME)

# -- Send the request. No body (None)
# -- Use the defined headers
conn.request(METHOD, ENDPOINT, None, headers)

# -- Wait for the server's response
r1 = conn.getresponse()

# -- Print the status
print()
print("Response received: ", end='')
print(r1.status, r1.reason)

# -- Read the response's body and close
# -- the connection
text_json = r1.read().decode("utf-8")
conn.close()

# -- Optionally you can print the
# -- received json file for testing
# print(text_json)

# -- Generate the object from the json file
cat = json.loads(text_json)

# -- Print the received URL
print("Cat image: ", cat['file'])

Run the program. This is what you should get (with a different URL)

If you click on the URL, the image will be opened in the browser

In this animation you can see how the program works

Exercise 1: Chuck Norris

Study this API: http://www.icndb.com/api/ and write a python program that should do the following, each time you execute it:

• Print on the console the following information:
  • The number of total jokes about Chuck Norris avaiable at the database
  • The number and names of the different categories
  • A random joke :-)

Playing with the weather

There are many servers that let us getting information about the weather. One of them is Metaweather

The Endpoint /api/location/:woeid/ is used for retrieving the data. The input parameter is the woeid identifier

For example, the Madrid WOEID is 766273. Let's test it first in the browser, accessing to the following URL: https://www.metaweather.com/api/location/766273/

This client will get the information about Madrid and print the time, the weather description and the temperature on the console:

# Example of getting information about the weather of
# a location

import http.client
import json

# -- API information
HOSTNAME = "www.metaweather.com"
ENDPOINT = "/api/location/"

# -- For the location we have to use the
# -- Were on earth identifier
# -- London woeid = 44418
# -- Madrid woeid = 766273
LOCATION_WOEID = "766273"
METHOD = "GET"

# -- Here we can define special headers if needed
headers = {'User-Agent': 'http-client'}

# -- Connect to the server
# -- NOTICE it is an HTTPS connection!
# -- If we do not specify the port, the standar one
# -- will be used
conn = http.client.HTTPSConnection(HOSTNAME)

# -- Send the request. No body (None)
# -- Use the defined headers
conn.request(METHOD, ENDPOINT + LOCATION_WOEID + '/', None, headers)

# -- Wait for the server's response
r1 = conn.getresponse()

# -- Print the status
print()
print("Response received: ", end='')
print(r1.status, r1.reason)

# -- Read the response's body and close
# -- the connection
text_json = r1.read().decode("utf-8")
conn.close()

# -- Optionally you can print the
# -- received json file for testing
# print(text_json)

# -- Generate the object from the json file
weather = json.loads(text_json)

# -- Get the data
time = weather['time']

temp0 = weather['consolidated_weather'][0]
description = temp0['weather_state_name']
temp = temp0['the_temp']
place = weather['title']

print()
print("Place: {}".format(place))
print("Time: {}".format(time))
print("Weather description: {}".format(description))
print("Current temp: {} degrees".format(temp))

Run the program. You should get something like this:

Using the endpoint /api/location/search/ with the argument query it is possible to find the woeid of some cities (Not all the cities are available. Try with the capitals)

Let's find the woeid of Rome. This is the URL used: https://www.metaweather.com/api/location/search/?query=rome. We get this information in the browser:

With the woeid we can get the information about the weather in Rome

Exercise 2: What's the weather like in the capitals of the world?

Using the Metaweather server, write a python program that ask the user for a capital and print on the console the information about the current time, the temperature and the sunset in that city

The program should first find the woeid of the city, and then accessing to the weather info. If the woeid is not know, the program you show an error message

Getting information from Github

You can also access to the information stored in Github. You will need authorization in general, but there are some endpoints that can be accessed by anymone

The github API is available here: https://developer.github.com/v3/

You can find information about a github user with the endpoint /users/:username. This is an example of use: https://api.github.com/users/Obijuan

This is an example of a client that get some information about a github user:

# Example of getting information stored in github

import http.client
import json

# -- API information
HOSTNAME = "api.github.com"
ENDPOINT = "/users/"
GITHUB_ID = "Obijuan"
METHOD = "GET"

# -- Here we can define special headers if needed
headers = {'User-Agent': 'http-client'}

# -- Connect to the server
# -- NOTICE it is an HTTPS connection!
# -- If we do not specify the port, the standar one
# -- will be used
conn = http.client.HTTPSConnection(HOSTNAME)

# -- Send the request. No body (None)
# -- Use the defined headers
conn.request(METHOD, ENDPOINT + GITHUB_ID, None, headers)

# -- Wait for the server's response
r1 = conn.getresponse()

# -- Print the status
print()
print("Response received: ", end='')
print(r1.status, r1.reason)

# -- Read the response's body and close
# -- the connection
text_json = r1.read().decode("utf-8")
conn.close()

# -- Optionally you can print the
# -- received json file for testing
# print(text_json)

# -- Generate the object from the json file
user = json.loads(text_json)

# -- Get some data
login = user['login']
name = user['name']
bio = user['bio']
nrepos = user['public_repos']

print()
print("User: {}".format(login))
print("Name: {}".format(name))
print("Repos: {}".format(nrepos))
print("Bio: \n{}".format(bio))

Run the program. This is what you will seen on the console:

Exercise 3: Information about a github user

Create a python program that ask the user for a github username and print on the console the following information about that username:

• Real name
• The list with the names of all the repos the user has
• The total number of commits to the 2018-19-PNE-repo

Authors

• Boni García
• Juan González-Gómez (Obijuan)

Credits

• Alvaro del Castillo. He designed and created the original content of this subject. Thanks a lot :-)

License

Links

• Universidad Rey Juan Carlos de Madrid
• Escuela Técnica Superior de Ingeniería de Telecomunicaciones (URJC)