Variant Mining Setup - tuiaswath/karyosoft GitHub Wiki

Variant Mining platform Installation

Machine Configuration

AWS Tier : t2.2XLarge
Platform : Ubuntu 20.04 (Linux)
Root Partition : 20GB
SSD : 200GB
RAM : 32GB

1. Check the space on the disk and root partitions

sudo lsblk
  • Drive:/mnt/disks/
  • It should have 200GB or more

2. Update the local repository sources to get the latest versions of the packages needed to install later

sudo apt update
sudo apt upgrade

3. Install Docker using APT

sudo apt install docker.io

4. Start the docker service and enable auto-start on reboot

sudo systemctl start docker 
sudo systemctl enable docker

4.1. Check Docker version

docker --version

6. Install Docker-Compose using APT.

6.1. Install curl if it is not yet installed

sudo apt-get update 
sudo apt install curl

6.2. Download docker-compose into the system Path location

sudo curl -L "https://github.com/docker/compose/releases/download/v2.5.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

6.3. Add execution permission to the docker-compose executable and change owner to the local user

sudo chmod +x /usr/local/bin/docker-compose

6.4 Change the owner of the docker socket and executable

sudo usermod -aG docker ubuntu 
sudo chown ubuntu:ubuntu /var/run/docker.sock

6.5. Check the docker-compose version

docker-compose --version

Changing the location of Docker's Data directory

7. Stop Docker Service

sudo systemctl stop docker

8. Verify if above command succeeded. The following command will yield no output only if Docker service is stopped:

sudo ps aux | grep -i docker | grep -v grep

9. Create this file /etc/docker/daemon.json

sudo nano /etc/docker/daemon.json

{ 
  "graph": "/mnt/disks/docker/" 
}

10. Copy Existing Docker Data

sudo rsync -aP /var/lib/docker/ /mnt/disks/docker/

11. Verification

11.1. Rename the original data directory

sudo mv /var/lib/docker /var/lib/docker.orig

11.2. Start our docker service

sudo systemctl start docker

11.3. Verify with docker info

docker info | grep -i root

  • You should see something like this

    Docker Root Dir: /mnt/disks/docker/

  • Once you have verified the data directory has moved and it reflects the same in the docker info, you may proceed to remove the original directory.

11.4. Remove original docker data directory

sudo rm -fr /var/lib/docker.orig

12. Clone the repository

TO-DO

13. Check the environment configuration file for current IP address for the API URL

cd /mnt/disks/variantmining
sudo nano ./client/src/environments/environment.prod.ts

13.1 Check and change the API URL to current IP address of the VM

APIURL: "http://XX.XX.XX.XX/"

14. Download and extract the latest "MongoDB Database Tools" from the MongoDB website.

As of 05/12/2022 The latest version is 100.5.2

14.1 Download the latest archive and extact it to folder outside the repo location.

cd /mnt/disks/
curl https://fastdl.mongodb.org/tools/db/mongodb-database-tools-ubuntu2004-x86_64-100.5.2.tgz -o mongodb-database-tools.tgz && tar zxvf mongodb-database-tools.tgz
mv mongodb-database-tools-* mongodb-database-tools

14.2 Copy the mongoimport file to the the following folder

sudo cp mongodb-database-tools/bin/mongoimport ./variantmining/api/scripts/mongoimport

14.3 Check and add execution permission for the copied mongoimport file

sudo chmod +x ./variantmining/api/scripts/mongoimport

14.4 Remove the unnecessary files

rm mongodb-database-tools.tgz
rm -r mongodb-database-tools

15. Verify the configured path of the mongoimport and other scripts in the follow files

  • variantmining/api/scripts/VCFtoJSON.py

    snpsLoaderScript = "./scripts/loadSNPSToMongo.sh"
indelsLoaderScript = "./scripts/loadIndelsToMongo.sh"
filterLoaderScript = "./scripts/loadFiltersToMongo.sh"
JSONGeneratorScript = "./scripts/VCFtoJSONGenerator.py"
FilterJSONGeneratorScript = "./scripts/FilterJSONGenerator.py"

  • variantmining/api/scripts/loadSNPSToMongo.sh

    /api/scripts/mongoimport --uri`

  • variantmining/api/scripts/loadIndelsToMongo.sh

    /api/scripts/mongoimport --uri

  • variantmining/api/scripts/loadFiltersToMongo.sh

    /api/scripts/mongoimport --uri

16. Build the application

cd /mnt/disks/variantmining
docker-compose build --no-cache

17. Run the appplication

 docker-compose up -d

17.1 Check the status of the application to make sure it is running and check the logs for any errors

docker ps
docker logs variantminingapi

Note : If you have log entries like below. you need to restart the application with sudo permissions

ERROR: Could not write to directory: [Error: ENOENT: no such file or directory, stat '/uploads/.writeTest'] {
  [stack]: "Error: ENOENT: no such file or directory, stat '/uploads/.writeTest'",
  [message]: "ENOENT: no such file or directory, stat '/uploads/.writeTest'",
  errno: -2,
  code: 'ENOENT',
  syscall: 'stat',
  path: '/uploads/.writeTest'
}

17.2 To Restart the application

docker-compose down
sudo docker-compose up -d