Simple Deployment on AWS - LibraryOfCongress/scribeAPI GitHub Wiki

Build & Publish Docker Image

(Replace tongwang with your own Docker Hub account.)

# build and tag as the latest and a specific version
docker build -t tongwang/scribeapi_loc:latest -t tongwang/scribeapi_loc:1.0 .

# push the image to Docker Hub
docker login
docker push tongwang/scribeapi_loc

Install Docker and Docker Compose on EC2

Install Docker:

sudo yum update -y
sudo yum install -y docker
sudo service docker start

# add user to docker group
sudo usermod -aG docker $USER

Install Docker Compose:

# https://github.com/docker/compose/releases
sudo su
curl -L https://github.com/docker/compose/releases/download/1.8.1/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

Prepare AWS EC2 EBS volume as MongoDB storage

When launching a new EC2 instance, add a new volume:

• Volume type: EBS
• Device: /dev/sdb
• Size: 100GB
• Volume Type: Provisioned IOPS SSD

To see the volume:

sudo fdisk -l

The first time you use a newly created EBS volume, you must create a file system:

sudo mkfs.ext4 /dev/sdb

After the file system has been created, you can mount the device:

sudo mkdir /mnt/data/
sudo mount /dev/sdb /mnt/data/

To see mounted volumes:

df -h

Change owner:

sudo chown ec2-user:ec2-user /mnt/data/

To automatically mount an EBS volume upon starting an Amazon EC2 Linux instance, make an entry to /etc/fstab:

sudo
echo "/dev/xvdf   /mnt/data   ext4    defaults,nofail        0       2" >> /etc/fstab

To unmount:

sudo umount /mnt/data/

Run ScribeAPI on an AWS EC2 Instance

Create .env file with following environment variables:

RAILS_ENV=    # production, development, test
MONGOLAB_URI=     # for production only
DEVISE_SECRET_TOKEN=   # generated by 'rake secret'
SECRET_KEY_BASE_TOKEN=   # generated by 'rake secret'
GOOGLE_ID=
GOOGLE_SECRET=

MONGOLAB_URI should be something like this: mongodb://scribe:password@mongodb:27017/scribe_loc

For the first time, create db folder:

mkdir -p /mnt/data/db

Create file docker-compose.yml with following content (see file docker-compose-ec2.yml):

mongodb:
  image: mongo:2.6
  ulimits:
    nproc: 64000
    nofile: 64000
  volumes:
    - /mnt/data/db:/data/db
  command: mongod --auth --setParameter textSearchEnabled=true

scribe:
  image: tongwang/scribeapi_loc:latest
  links:
    - mongodb:mongodb
  ports:
    - 80:80
  environment:
    - RAILS_ENV
    - MONGOLAB_URI
    - DEVISE_SECRET_TOKEN
    - SECRET_KEY_BASE_TOKEN
    - GOOGLE_ID
    - GOOGLE_SECRET

Start Scribe the first time:

docker-compose up -d

Scribe container will fail because of user account is not set up in Mongo.

Create MongoDB admin user and scribeAPI user

SSH into MongoDB container first:

docker exec -it ec2user_mongodb_1 bash

mongo

use admin

db.addUser( { user: "root",
              pwd: "password",
              roles: [ "userAdminAnyDatabase", "dbAdminAnyDatabase", "clusterAdmin" ] } )

# login as admin user
mongo -u root -p password admin

# switched to db scribe_loc
use scribe_loc

db.addUser( { user: "scribe",
              pwd: "password",
              roles: [ "readWrite", "dbAdmin" ]
            } )

Restart:

docker-compose restart

Set up Scribe project:

docker exec -it ec2user_scribe_1 bash
rake project:load[loc]
rake project:create_lc_indexes

You can also load project content, workflow or subjects only:

rake project:load[loc,content]
rake project:load[loc,workflows]
rake project:load[loc,subjects]

Before load a large set of subjects, create the following indexes in mongodb:

db.subjects.createIndex({subject_set_id: 1, "location.standard": 1, type: 1})
db.subject_sets.createIndex({group_id: 1, key: 1})

Restart:

docker-compose restart

Upgrade

# pull new images
docker-compose down
docker-compose pull
docker-compose up -d --build

Logs

By default, Docker will write the output wrapped into JSON into a specific file named after the container ID, in a directory under /var/lib/docker/containers named after the container ID. You can view the log by:

sudo cat /var/lib/docker/containers/$CID/$CID-json.log

It can also log to syslog (/var/log/syslog). Add the following to docker-compose.yml for each container:

logging:
  driver: syslog