[DEPRECATED] 8. Docker and AWS ElasticBeanStalk - Rothamsted/knetminer GitHub Wiki

Deprecated!

We have significantly reviewed the architecture of our Knetminer for Docker and this is documented in this page.

This page will be removed in future.

It is now possible to spin up a new dedicated Knetminer instance with a custom OXL file using Docker, or to the cloud using AWS ElasticBeanStalk.

Follow the simple instructions below to see how:

Before starting:

  1. You will need to have your OXL file, your SemanticMotifs.txt, and any other files required by the configuration (see below) accessible via an HTTP URL, i.e. saved on a web server or in an S3 bucket. They must be publicly accessible for the duration of the deployment but can be hidden or deleted once deployment is complete
  2. If using Docker, you will need to install Docker locally
  3. If using AWS ElasticBeanStalk, you will need an AWS account. The instructions below assume you will deploy KnetMiner using the ElasticBeanStalk command-line interface (eb-cli), but you can also achieve the same results through the AWS console web interface

To install using Docker:

  1. Into an empty folder download the Dockerfile
  2. Edit the downloaded Dockerfile and in the first ENV section at the top update the settings:
    1. Note that this includes the branch of the KnetMiner GitHub repository to use, and in most circumstances this should be the same branch that you got the Dockerfile from in the first place (knetminer-3.0 in this example) otherwise behaviour could be unpredictable
    2. The defaults shown in the file will also work if you just want to try it and see - the example URLs it references are in a public S3 bucket in the KnetMiner AWS account
  3. On the command line inside the folder where you downloaded the Dockerfile, type:
docker image build --squash -t knetminer .
docker run -p8080:8080 -it --rm knetminer
  1. After waiting about ten minutes for the previous step to complete you’ll be able to access KnetMiner at http://localhost:8080/client/

To install using AWS ElasticBeanStalk:

  1. Into an empty folder download two files:
    1. Dockerfile
    2. Dockerrun.aws.json
  2. Edit the downloaded Dockerfile and in the first ENV section at the top update the settings
    1. Note that this includes the branch of the knetminer Git repository to use, and in most circumstances this should be the same branch that you got the Dockerfile from in the first place (knetminer-3.0 in this example) otherwise behaviour could be unpredictable
    2. The defaults shown in the file will also work if you just want to try it and see - the example URLs it references are in a public S3 bucket in the Knetminer AWS account
  3. If you are using the AWS ElasticBeanStalk web interface, zip the folder containing the two files into a single zip archive, then go to the AWS console to upload and deploy it (the type is Generic->Docker, and your source bundle is the zip file you just created)
  4. If you are using the ElasticBeanStalk command-line method instead, then from within the folder containing the two files type the following commands and follow any instructions they give you:
eb init
eb create
  1. After waiting about ten minutes for it to start up, scan the output of the commands for the details of the load balancer it has created, or otherwise go to the AWS console in your web browser and look for the load balancer details there. Make a note of the load balancer's hostname (public DNS)
  2. You can now access KnetMiner at: http://\<your-load-balancer-hostname\>/client/

(RRes) - To install using Docker on a local VM (production):

To install using Docker on a VM (e.g., CentOS) with locally stored OXL/files, for production instances:

  1. Create a folder for your KnetMiner species, and copy into it the species OXL file, SemanticMotifs.txt, basemap.xml and other required files (e.g., build-docker.sh).
  2. Also copy Dockerfile-local to this folder
  3. You can edit the GitHub repo and branch to use in Dockerfile-local
  4. Run build-docker.sh in this folder. This takes all the local data in this folder and invokes Dockerfile-local (with some build-args) which pulls code from GitHub and builds a Docker image, e.g, knetminer-arabidopsis-v3.0 using a specific OXL and on a specific port.
  5. Once built, use docker run -p8080:8080 -it --rm knetminer-arabidopsis-v3.0 to run it on localhost:8080 (change port number & image_name as defined in build-docker.sh). Note: when container is starting up Tomcat and deploying OXL, press Ctrl+P and Ctrl+Q to exit to terminal while keeping interactive container running.
  6. Access the deployed KnetMiner GUI on localhost:8080/client and back-end API (api_url) via localhost:8080/ws/. Note: These can later be re-directed in organisation/group's external DNS to generic url's to mask the host and port.
  7. Debugging: Check logs inside the running container via docker exec -it containerID /bin/sh. You will be at /root which has: data.oxl and Tomcat logs at /usr/local/tomcat/logs/ (seen via, e.g., tail -f /usr/local/tomcat/logs/ws.log). Docker logs can also be followed via: docker logs -f containerID.

To install using Docker on a local VM (test):

To install using Docker on a VM, with locally stored data (OXL/files) + application code (useful for debug mode), for test/dev instances:

  1. Clone KnetMiner code on the VM in a new folder (e.g., in /home/usern/), e.g, clone latest branch manually from GitHub via: git clone --single-branch -b knetminer-3.0 https://github.com/Rothamsted/knetminer.git
  2. For a particular species, e.g., Arabidopsis, ensure the latest files (semantic motifs, image, example queries) are in the correct species/ folder locally, e.g., at species-arabidopsis-url
  3. Upload your latest species OXL into this folder as well.
  4. Run build-docker-dev.sh in this folder (knetminer/species/arabidopsis/). This takes all the local data in this folder and invokes Dockerfile-dev (with some build-args) which builds a Docker image, e.g, knetminer-arabidopsis-v3.0 using a specific OXL and on a specific port, using KnetMiner code you pulled on this VM.
  5. Once built, use docker run -p8080:8080 -it --rm knetminer-arabidopsis-v3.0 to run it on localhost:8080 (change port number & image_name as defined in build-docker-dev.sh). Note: when container is starting up Tomcat and deploying OXL, press Ctrl+P and Ctrl+Q to exit to terminal while keeping interactive container running.
  6. Access the deployed KnetMiner GUI on localhost:8080/client and back-end API (api_url) via localhost:8080/ws/. Note: These can later be re-directed in organisation/group's external DNS to generic url's to mask the host and port.
  7. Debugging: Check logs inside the running container via docker exec -it containerID /bin/sh. You will be at /root which has: knetminer/common/ code and data.oxl while Tomcat logs will be at /usr/local/tomcat/logs/ (seen via, e.g., tail -f /usr/local/tomcat/logs/ws.log). Docker logs can also be followed via: docker logs -f containerID.

Tips: for Docker tidy-up on a local VM:

  1. To view all images on a VM, use docker images
  2. To view all containers on a VM, use docker ps
  3. To follow logs: docker logs -f containerID
  4. To stop/remove a running container, use docker stop containerID (containerID can be listed first via docker ps).
  5. To delete an old image, use docker image rm imageID (imageID can be listed first via docker images).
  6. To remove "dangling images" (after image deletion), use docker image prune
  7. Optional: To clean-up a VM, use docker system prune and then enter 'y' (removes all stopped containers, dangling images and unused volumes to free up max. space).
  8. Optional: To delete an old, stopped container, use docker container rm containerID
  9. further info. at: linux-Docker-cleanup and cleanup-dangling-containers-images