Instructions

Step 1: microservices introductory demo

Often, we want to contribute to or further develop an open-source project in isolation from the original. We achieve this by forking a project for our personal use (always check license), which has some benefits over branching. Click Fork at the top right corner of this project's home page, and then you have a personal copy of the repository which you can change to your heart's content. Clone the repository to your Azure VM (or you can also use your own laptop if you have python3 and Docker installed ** It will not work on Windows unless you have Windows server** It will likely work with MacOS and Linux.).

You will see that you are on the master branch, with two files movies.py and showtimes.py. Look through their source code to see how they function. movies.py fetches movie data from a json file in /database. showtimes.py fetches showtimes for the movies from another json file in /database. Each of them is a microservice which exposes RESTful API endpoints. You can run them as follows:

python3 movies.py

Now, the movies microservice is running. You can query it by sending HTTP GET requests. Note that we are using the port 5001 on localhost (127.0.0.1) as that is how the server is configured (see towards the bottom of the file in movies.py)

python3 movies.py

Now, the movies microservice is running. You can query it by sending HTTP GET requests. Note that we are using the port 5001 on localhost (127.0.0.1) as that is how the server is configured (see towards the bottom of the file in movies.py)

curl http://127.0.0.1:5001/ #returns list of endpoints exposed
curl http://127.0.0.1:5001/movies #returns list of all movies
curl http://127.0.0.1:5001/movies/720d006c-3a57-4b6a-b18f-9b713b073f3c #returns details of one movie

Similarly, you can run the showtimes microservice

python3 showtimes.py

You can query its endpoints too:

python3 showtimes.py

You can query its endpoints too:

$ curl 127.0.0.1:5002/      
{"uri": "/", "subresource_uris": {"showtimes": "/showtimes", "showtime": "/showtimes/<date>"}}%                                                                 
$ curl 127.0.0.1:5002/showtimes
{"20151130": ["720d006c-3a57-4b6a-b18f-9b713b073f3c", "a8034f44-aee4-44cf-b32c-74cf452aaaae", "39ab85e5-5e8e-4dc5-afea-65dc368bd7ab"], "20151201": ["267eedb8-0f5d-42d5-8f43-72426b9fb3e6", "7daf7208-be4d-4944-a3ae-c1c2f516f3e6", "39ab85e5-5e8e-4dc5-afea-65dc368bd7ab", "a8034f44-aee4-44cf-b32c-74cf452aaaae"], "20151202": ["a8034f44-aee4-44cf-b32c-74cf452aaaae", "96798c08-d19b-4986-a05d-7da856efb697", "39ab85e5-5e8e-4dc5-afea-65d

Step 2: Making microservices talk to each other

As with the movies, the showtimes microservice allows you to query one single record. For a date that it knows about (e.g., 30 Nov 2015 or 20151130), you can ask what movies were shown on that date.

$ curl 127.0.0.1:5002/showtimes/20151130 
[
    "720d006c-3a57-4b6a-b18f-9b713b073f3c",
    "a8034f44-aee4-44cf-b32c-74cf452aaaae",
    "39ab85e5-5e8e-4dc5-afea-65dc368bd7ab"
]%  

You get back a list of movie IDs. This is nice, but not wholly satisfactory. Your next task is to modify the showtimes_record function to contact the movies microservice to get the movie title given its ID, and make the showtimes service return back a more user-friendly response that looks like this:

$ curl 127.0.0.1:5002/showtimes/20151130
[
    "The Good Dinosaur",
    "The Martian",
    "Spectre"
]%               

The main change needed is simple. We just have to issue an HTTP GET request to the movies API endpoint. We do this by changing showtime_record as follows:

movies_service = "http://127.0.0.1:5001/movies/{}" #we know this is where the service is running. We replace the {} with the actual ID of the movie we want.

def showtimes_record(date):
    if date not in showtimes:
        raise NotFound
    print(showtimes[date])
    result = []
    for movie_id in showtimes[date]:
        resp = requests.get(movies_service.format(movie_id))
        result.append(resp.json()["title"])
    return nice_json(result)

Step 3: Dockerization of microservices

We now have two microservices running on localhost and talking to each other. If we need to move these services, it is not easy. Your next task is to wrap these microservices as a docker image and make them portable. Essentially, this involves specifying all the dependencies and running parameters explicitly in a Dockerfile. You can have a go at this by copying from this tutorial.

In this branch, we have created two directories movieservice and stservice that create docker images for movies and showtimes microservices respectively. The Dockerfile for the movie microservice is shown here. The other one is very similar.

FROM python:3.8-alpine
WORKDIR /
COPY movieservice/requirements.txt .
RUN pip install -r requirements.txt
COPY ./movies.py /
COPY ./database /database
EXPOSE 5001
CMD python movies.py

Docker images are created in layers. The FROM command says this image is derived from a base python 3.8 image. WORKDIR sets the working directory for the main process in the running container. The COPY commands are interesting -- the first argument is in the host machine, and the second argument is in the container. Thus, for example, COPY ./movies.py / is an instruction to copy the movies.py to the / directory in the container, which happens to be the working directory. So, when CMD python movies.py runs the main command, it finds movies.py in the / directory which is its working directory.

Also interesting is the command to EXPOSE 5001 which is a declaration that we want the port 5001, on which the microservice is listening, to be exposed to the outside world. However, if you do this, build and run the docker container, and then try to query the dockerized service (e.g., with curl http://127.0.0.1:5001/movies), we get a connection refused error. There is a fundamental difference between exposing and publishing a port, which is done with the -p flag which we saw in last week's live session. This distinction is discussed here

The Makefile provided in this branch has a target make movieservice which provides the right set of parameters. But even after publishing the port 5001, we still are not able to access the microservice. (Try issuing another curl). The reason is because of Docker Network Namespaces. The container is given a different network namespace to isolate it, and because in movies.py we have asked the Flask server to listen in on the localhost IP address, it does not respond to queries for the external IP address. The solution for this is to have the HTTP Server listen on 0.0.0.0 (See the differences in movies.py between this branch and the simpleservices branch).

The short solution/summary: Checkout this branch, and run

$ make movieservice stservice # This brings up Docker containers for both services
$ echo "After this we can query each service separately as with the curls below"
$ curl http://127.0.0.1:5001/movies  
$ curl 127.0.0.1:5002/showtimes

A simple technical explanation of what is going on at the network level, together with illuminating visuals can be found in this blog

Step 4: How to make two dockerized microservices talk to each other

The above has just demonstrated that we can talk to each microservice from the host machine. Therefore, you may be under the impression that the two microservices should also be able to talk to each other, as we were able to, before dockerizing them. Test this by invoking the showtimes API endpoint that invokes the movies service (from Step 2 above): curl 127.0.0.1:5002/showtimes/20151130. Notice that this causes an exception. Interestingly, although we can see it from the host machine, the dockerized showtimes service cannot access the movie service docker container!

What is going on? Essentially, this is due to the excellent isolation provided by Docker through namespaces, which is making development slightly harder. An overview of the intricacies of Docker Networking can be found here. The simple solution for us is to tell the Docker daemon to create a network that encompasses both the containers we have created. The Makefile has a target for this: make network.

Once the make target sets up the networking connections between the containers, the curl 127.0.0.1:5002/showtimes/20151130 will return the results as it did in Step 2.

Further steps

The Makefile in the above steps stitches together a number of different things - the ports to expose, the data directory, and creating a network. All this becomes very complicated when you have more services running that talk to each other. To make this simpler, we can use docker compose. To see how to use this, move to the compose branch of this repository.

credits

The original code is taken from https://github.com/umermansoor/microservices It has been lightly modified for Python3 compatibility, and further simplified to showcase microservice communications. We have also added a demo of dockerization