Deploy Magma Orchestrator using Docker - caprivm/virtualization GitHub Wiki

caprivm ([email protected])

Description

This page explains all the steps required to deploy the orchestrator using Docker and register it containers in a Docker Hub account. In addition, it explains how to manage the certificates necessary for the operation of the orchestrator. The hardware it was tested on is:

Feature Value
OS Used Ubuntu 18.04 LTS
vCPU 4
RAM (GB) 8
Disk (GB) 60
Home user ubuntu
Magma Tag v1.6

The contents of the page are:

Prerequisites

Before starting this guide, you should have installed the following tools. You can check the adjacent links if you haven't already:

We'll go over how to publish images to Docker Hub. For this you'll need:

Environment Variables

In this case, consider the next environment variables before continue the procedure:

export MAGMA_ROOT=~/magma_v1.6
export MAGMA_TAG=v1.6

Changes in Source Code

Any changes to the source code must be made before compiling the images to create the containers in the registry. This means that changes should be made at this point, and do not continue until all changes are made.

Add the needed LTE bands

If you need more bands for configuring networks in the orchestrator, you can add them in lte_bands.go file. Consider the next steps:

cd $MAGMA_ROOT/lte/cloud/go/lte/lte_bands.go
sudo vi lte_bands.go

Add the next lines in the file:

var bands = [...]LTEBand{
        // FDDMode
        {ID: 1, Mode: FDDMode, StartEarfcnDl: 0, StartEarfcnUl: 18000, CountEarfcn: 600},
        {ID: 2, Mode: FDDMode, StartEarfcnDl: 600, StartEarfcnUl: 18600, CountEarfcn: 600},
        {ID: 3, Mode: FDDMode, StartEarfcnDl: 1200, StartEarfcnUl: 19200, CountEarfcn: 750},
        {ID: 4, Mode: FDDMode, StartEarfcnDl: 1950, StartEarfcnUl: 19950, CountEarfcn: 450},
+       {ID: 5, Mode: FDDMode, StartEarfcnDl: 2400, StartEarfcnUl: 20400, CountEarfcn: 250},
+       {ID: 6, Mode: FDDMode, StartEarfcnDl: 2650, StartEarfcnUl: 20560, CountEarfcn: 100},
+       {ID: 7, Mode: FDDMode, StartEarfcnDl: 2750, StartEarfcnUl: 20750, CountEarfcn: 700},
+       {ID: 8, Mode: FDDMode, StartEarfcnDl: 3450, StartEarfcnUl: 21450, CountEarfcn: 350},
+       {ID: 9, Mode: FDDMode, StartEarfcnDl: 3800, StartEarfcnUl: 21800, CountEarfcn: 350},
+       {ID: 10, Mode: FDDMode, StartEarfcnDl: 4150, StartEarfcnUl: 22150, CountEarfcn: 600},
+       {ID: 11, Mode: FDDMode, StartEarfcnDl: 4750, StartEarfcnUl: 22750, CountEarfcn: 260},
+       {ID: 12, Mode: FDDMode, StartEarfcnDl: 5010, StartEarfcnUl: 23010, CountEarfcn: 170},
+       {ID: 13, Mode: FDDMode, StartEarfcnDl: 5180, StartEarfcnUl: 23180, CountEarfcn: 100},
+       {ID: 14, Mode: FDDMode, StartEarfcnDl: 5280, StartEarfcnUl: 23280, CountEarfcn: 450},
+       {ID: 17, Mode: FDDMode, StartEarfcnDl: 5730, StartEarfcnUl: 23730, CountEarfcn: 120},
+       {ID: 18, Mode: FDDMode, StartEarfcnDl: 5850, StartEarfcnUl: 23850, CountEarfcn: 150},
+       {ID: 19, Mode: FDDMode, StartEarfcnDl: 6000, StartEarfcnUl: 24000, CountEarfcn: 150},
+       {ID: 20, Mode: FDDMode, StartEarfcnDl: 6150, StartEarfcnUl: 24150, CountEarfcn: 300},
+       {ID: 21, Mode: FDDMode, StartEarfcnDl: 6450, StartEarfcnUl: 24450, CountEarfcn: 150},
+       {ID: 22, Mode: FDDMode, StartEarfcnDl: 6600, StartEarfcnUl: 24600, CountEarfcn: 900},
+       {ID: 23, Mode: FDDMode, StartEarfcnDl: 7500, StartEarfcnUl: 25500, CountEarfcn: 200},
+       {ID: 24, Mode: FDDMode, StartEarfcnDl: 7700, StartEarfcnUl: 25700, CountEarfcn: 340},
+       {ID: 25, Mode: FDDMode, StartEarfcnDl: 8040, StartEarfcnUl: 26040, CountEarfcn: 650},
+       {ID: 26, Mode: FDDMode, StartEarfcnDl: 8690, StartEarfcnUl: 26690, CountEarfcn: 350},
+       {ID: 27, Mode: FDDMode, StartEarfcnDl: 9040, StartEarfcnUl: 27040, CountEarfcn: 170},
        {ID: 28, Mode: FDDMode, StartEarfcnDl: 9210, StartEarfcnUl: 27210, CountEarfcn: 450},
+       {ID: 71, Mode: FDDMode, StartEarfcnDl: 68586, StartEarfcnUl: 133122, CountEarfcn: 350},
        // TDDMode
        {ID: 33, Mode: TDDMode, StartEarfcnDl: 36000, CountEarfcn: 200},
        {ID: 34, Mode: TDDMode, StartEarfcnDl: 36200, CountEarfcn: 150},
        {ID: 35, Mode: TDDMode, StartEarfcnDl: 36350, CountEarfcn: 600},
        {ID: 36, Mode: TDDMode, StartEarfcnDl: 36950, CountEarfcn: 600},
        {ID: 37, Mode: TDDMode, StartEarfcnDl: 37550, CountEarfcn: 200},
        {ID: 38, Mode: TDDMode, StartEarfcnDl: 37750, CountEarfcn: 500},
        {ID: 39, Mode: TDDMode, StartEarfcnDl: 38250, CountEarfcn: 400},
        {ID: 40, Mode: TDDMode, StartEarfcnDl: 38650, CountEarfcn: 1000},
        {ID: 41, Mode: TDDMode, StartEarfcnDl: 39650, CountEarfcn: 1940},
        {ID: 42, Mode: TDDMode, StartEarfcnDl: 41590, CountEarfcn: 2000},
        {ID: 43, Mode: TDDMode, StartEarfcnDl: 43590, CountEarfcn: 2000},
        {ID: 44, Mode: TDDMode, StartEarfcnDl: 45590, CountEarfcn: 1000},
        {ID: 45, Mode: TDDMode, StartEarfcnDl: 46590, CountEarfcn: 200},
        {ID: 46, Mode: TDDMode, StartEarfcnDl: 46790, CountEarfcn: 7750},
        {ID: 47, Mode: TDDMode, StartEarfcnDl: 54540, CountEarfcn: 700},
        {ID: 48, Mode: TDDMode, StartEarfcnDl: 55240, CountEarfcn: 1500},
        {ID: 49, Mode: TDDMode, StartEarfcnDl: 56740, CountEarfcn: 1500},
        {ID: 50, Mode: TDDMode, StartEarfcnDl: 58240, CountEarfcn: 850},
        {ID: 51, Mode: TDDMode, StartEarfcnDl: 59090, CountEarfcn: 50},
        {ID: 52, Mode: TDDMode, StartEarfcnDl: 59140, CountEarfcn: 1000},
        // Adding Band #53 require changes in the python code cause it's
        // start_freq_dl is float value.
        //{ID: 53, Mode: TDDMode, StartEarfcnDl: 60140, CountEarfcn: 115},
}

Close and save the file. Go with the next steps to deploy the orchestrator.

Clone and configure the Magma repository

First, start up the Docker daemon to log in to the Docker Hub account. Before execute this command you should have installed Docker as explained in section Docker Installation. In password put your Docker Hub access token.

docker login 
# Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
# Username:
# Password:
# Login Succeeded

Next, clone the magma repository and checkout the $MAGMA_TAG branch:

cd ~ && sudo apt update && sudo apt upgrade -y
sudo apt install git -y
git clone https://github.com/magma/magma.git $MAGMA_ROOT
cd $MAGMA_ROOT
git checkout $MAGMA_TAG
# You can check if you chose the correct branch using the following command:
git branch
#   master
# * v1.6

Build the container images

Build the containers with all configurations. Remember that this command needs Docker and Python:

cd $MAGMA_ROOT/orc8r/cloud/docker
./build.py --all

Wait for the command to finish executing before continuing to the next section.

Limit log writing

The Prometheus and MariaDB services can write excessively logs filling the disk of the host hosting the Orchestrator. To limit writing, consider the following changes:

cd $MAGMA_ROOT/orc8r/cloud/docker
sudo vi docker-compose.metrics.yml

In prometheus service add the next lines:

[...]
    prometheus:
    image: prom/prometheus:v2.20.1
    ports:
      - 9090:9090/tcp
    volumes:
      - $PWD/../../../orc8r/cloud/docker/metrics-configs:/etc/prometheus:ro
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.enable-lifecycle'
+   restart: always
+   logging:
+     driver: "json-file"
+     options:
+       max-file: "5"
+       max-size: "100m"
[...]

Next:

cd $MAGMA_ROOT/orc8r/cloud/docker
sudo vi docker-compose.yml

In postgres service add the next lines:

[...]
    postgres:
    image: postgres
+   logging:
+     options:
+       max-size: 10m
[...]

Deploy the ochestrator backend containers

After build, deploy the backend orchestrator applications using the next command:

cd $MAGMA_ROOT/orc8r/cloud/docker
./run.py --metrics
# Wait until the command finish. After finished, you should see the certificates
ls -1 $MAGMA_ROOT/.cache/test_certs/.
# admin_operator.key.pem
# admin_operator.pem
# admin_operator.pfx
# bootstrapper.key
# certifier.key
# certifier.pem
# controller.crt
# controller.csr
# controller.key
# rootCA.key
# rootCA.pem
# rootCA.srl
# vpn_ca.crt
# vpn_ca.key

NOTE: Please consider download the admin_operator.pfx to import and add the client cert to keychain. The password for the cert is magma. If you use Firefox, you'll have to import this admin_operator.pfx file into your browser's installed client certificates. See here for instructions.

Deploy the orchestrator frontend containers

To deploy the rest of the orchestrator containers, build and deploy the frontend containers:

cd $MAGMA_ROOT/nms/app/packages/magmalte
docker-compose build magmalte
# To set up the NMS
docker-compose up -d
# Wait a couple of minutes to up all features. Validate the health
docker ps -a | grep magmalte_magmalte | awk  '{ print $10 }'  # <-- The output must be (healthy)
# Create the user: [email protected] password: password1234
./scripts/dev_setup.sh

When the script finishes, you can execute the next command for validate the state of the containers.

docker ps -a
# CONTAINER ID   IMAGE                                                 COMMAND                  CREATED      STATUS                  PORTS                                                                                                                                                    NAMES
# bc49228c1bb1   nginx                                                 "/docker-entrypoint.…"   3 days ago   Up 3 days               80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp                                                                                                            magmalte_nginx-proxy_1
# 7fe71628d1c9   magmalte_magmalte                                     "docker-entrypoint.s…"   3 days ago   Up 3 days (healthy)     0.0.0.0:8081->8081/tcp, :::8081->8081/tcp                                                                                                                magmalte_magmalte_1
# e74cf9a0a611   mariadb:10.4.12                                       "docker-entrypoint.s…"   3 days ago   Up 3 days (healthy)     3306/tcp                                                                                                                                                 magmalte_mariadb_1
# 07f0b89d4395   60d16b088c54                                          "/bin/bash -lc 'echo…"   3 days ago   Exited (0) 3 days ago                                                                                                                                                            orc8r_test_1
# f2bd0099041b   orc8r_controller                                      "/bin/sh -c '/usr/lo…"   3 days ago   Up 3 days                                                                                                                                                                        orc8r_controller_1
# 93bc68f06830   orc8r_fluentd                                         "tini -- /bin/entryp…"   3 days ago   Up 3 days               5140/tcp, 0.0.0.0:24224-24225->24224-24225/tcp, 0.0.0.0:24224-24225->24224-24225/udp, :::24224-24225->24224-24225/tcp, :::24224-24225->24224-24225/udp   fluentd
# 021eb61abe05   docker.elastic.co/kibana/kibana:7.3.1                 "/usr/local/bin/dumb…"   3 days ago   Up 3 days               0.0.0.0:5601->5601/tcp, :::5601->5601/tcp                                                                                                                orc8r_kibana_1
# 3571640d032c   prom/prometheus:v2.20.1                               "/bin/prometheus --c…"   3 days ago   Up 3 days               0.0.0.0:9090->9090/tcp, :::9090->9090/tcp                                                                                                                orc8r_prometheus_1
# 88e1be6184b7   postgres                                              "docker-entrypoint.s…"   3 days ago   Up 3 days               0.0.0.0:5432->5432/tcp, :::5432->5432/tcp                                                                                                                orc8r_postgres_1
# 5c59542f7135   grafana/grafana:6.6.2                                 "/run.sh"                3 days ago   Up 3 days               0.0.0.0:3000->3000/tcp, :::3000->3000/tcp                                                                                                                orc8r_user-grafana_1
# 0b5591b8e058   prom/alertmanager                                     "/bin/alertmanager -…"   3 days ago   Up 3 days               0.0.0.0:9093->9093/tcp, :::9093->9093/tcp                                                                                                                orc8r_alertmanager_1
# 82749a24dffb   facebookincubator/alertmanager-configurer:1.0.0       "alertmanager_config…"   3 days ago   Up 3 days                                                                                                                                                                        orc8r_alertmanager-configurer_1
# 3e4039c54b10   facebookincubator/prometheus-edge-hub:1.1.0           "prometheus-edge-hub…"   3 days ago   Up 3 days               0.0.0.0:9091-9092->9091-9092/tcp, :::9091-9092->9091-9092/tcp                                                                                            orc8r_prometheus-cache_1
# 59ac4c893d8f   orc8r_nginx                                           "/bin/sh -c '/usr/lo…"   3 days ago   Up 3 days               80/tcp, 0.0.0.0:9443->9443/tcp, :::9443->9443/tcp, 0.0.0.0:7443->8443/tcp, :::7443->8443/tcp, 0.0.0.0:7444->8444/tcp, :::7444->8444/tcp                  orc8r_nginx_1
# 2d4698b762e5   docker.elastic.co/elasticsearch/elasticsearch:7.3.1   "/usr/local/bin/dock…"   3 days ago   Up 3 days               0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 0.0.0.0:9300->9300/tcp, :::9300->9300/tcp                                                                     elasticsearch
# b0fa840f0546   facebookincubator/prometheus-configurer:1.0.0         "prometheus_configur…"   3 days ago   Up 3 days                                                                                                                                                                        orc8r_prometheus-configurer_1
# 893e39575fca   postgres                                              "docker-entrypoint.s…"   3 days ago   Up 3 days               0.0.0.0:5433->5432/tcp, :::5433->5432/tcp                                                                                                                orc8r_postgres_test_1
# 74278cd1ceb1   mariadb:10.2                                          "docker-entrypoint.s…"   3 days ago   Up 3 days               3306/tcp                                                                                                                                                 orc8r_maria_1

If get any issue in containers status, execute docker restart $(docker ps -a -q) to restart all containers and verify.

Deploy a crontab to remove logs

The orchestrator, deployed in Docker, logs excessively in the /var/lib/docker/ folder. For this reason, one task is generated in crontab to eliminate the logs that are being written. Make sure you have cron installed:

sudo apt update
sudo apt install cron
sudo systemctl enable cron

Create the next script with the name delete_logs_docker_container.sh:

#!/bin/bash
# Script to delete container logs

for i in $(sudo find /var/lib/docker/ -name *-json.log); do sudo rm $i; done

Executes the crontab job in minutes 20 of every hour every day in the year.

crontab -e
# Add the next line to end:
20 * * * * /root/delete_logs_docker_container.sh

This periodic task ensures that the orchestrator remains stable.

Access to NMS UI and Controller API

In the host that you use to access to UI and Controller API, put in yout /etc/hosts file the next lines:

<NMS_IP> magma-test
<NMS-IP> master

Now, go to your web browser (Firefox was tested) and access to https://magma-test/. Accept the self signed certificate and continue. If you want to go to the Swagger orchestrator documentation, need the admin_operator.pfx certificate imported in your web browser. With this, you can access to https://magma-test:9443/apidocs/v1/ (select the admin_operator.pfx certificate) and get the Swagger documentation.

Magma NMS Interface

When you access to https://magma-test/ you should see the next interface:

magma-test interface

Now, when you access to NMS (default credentials are [email protected]/password1234) you should see the next interface:

magma-test interface once login

You can navigate through the graphical interface by following the recommendations provided in the official Magma documentation.

Magma Controller API

Please note that you must import the certificate admin_operator.pfx in your browser. The certificate password is magma. When you access to https://magma-test:9443/apidocs/v1/ you should see the following interface:

magma API interface

For an idea of how the API can be used, check out this magma documentation. Although the documentation in this link is for the Federation Gateway, it gives an idea of how the API can be used.

Publish the container images

NOTE: This step is necessary only if the images have not been published before.

The Magma repository provide some scripts to build and publish images to Docker Hub account. Before use, it's necessary define some environment variables. Add the the next variables:

export PUBLISH=$MAGMA_ROOT/orc8r/tools/docker/publish.sh
export REGISTRY=docker.io/caprivm

Once the deployment finish, execute the next command to registry only the controller and nginx containers images:

cd $MAGMA_ROOT/orc8r/cloud/docker
for image in controller nginx ; do ${PUBLISH} -r ${REGISTRY} -i ${image} -v ${MAGMA_TAG} ; done

The next step consist on registry the magmalte container. For this, execute the next commands:

cd $MAGMA_ROOT/nms/app/packages/magmalte/
COMPOSE_PROJECT_NAME=magmalte ${PUBLISH} -r ${REGISTRY} -i magmalte -v ${MAGMA_TAG}

In your docker hub account you can validate that the images are already register as can be shown in the next figure:

Docker Images validation

Cleaning process

To clear the deployed containers for the Magma orchestrator and Magma NMS, run the following commands:

docker stop $(docker ps -a -q)
docker rm $(docker ps -a -q)
docker system prune --all
# WARNING! This will remove:
#   - all stopped containers
#   - all networks not used by at least one container
#   - all images without at least one container associated to them
#   - all build cache

# Are you sure you want to continue? [y/N] y

Troubleshooting

These are some errors that have been detected with this type of installation.

Fluentd bad starting

It is normal that the fluentd container does not start after the execution of the command ./run.py --metrics. So, this problem is related to certificate permissions. Execute the next commands to start the container without any issue:

cd $MAGMA_ROOT/.cache/test_certs/
sudo chmod 644 * && cd
docker restart fluentd

After this you can see the fluentd container in Up status using docker ps -a command.

⚠️ **GitHub.com Fallback** ⚠️