Setup MongoDB

The platform and the annotation system will have MongoDB as the backend storage system. This section describes installation and customization of MongoDB in the backend server. We mostly follow the official instructions.

Number of open files and processes

MongoDB requires a minimum number of allowed opened files and processes to guarantee good functionality. These are either controlled temporarily by the ulimit command or permanently by modifying the /etc/security/limits.conf file.

To work with the installation, the following is enough:

ulimit -n 65536

To make the required changes permanent, enter (as root) the following lines in /etc/security/limits.conf (before # End of file):

* soft nofile 65536
* hard nofile 65536
* soft memlock unlimited
* hard memlock unlimited

MongoDB installation

We then follow steps 1-4 here. After completion and prior to starting the MonfoDB service, we need to configure where the MongoDB files will be stored. As the expected data will be of great volumes, we must ensure that they will live in a filesystem with sufficient space. In the following, it is assumed as $DBHOME. Other dynamic variables which are commited here only as examples are the database ports.

MongoDB storage

We assume that:

• The storage volume with adequate storage is in /media/big_storage
• User (or main app database) port is 27017
• Bacground knowledge database port is 27018

DBHOME=/media/big_storage/mongodb
USERDB_PORT=27017
BACKDB_PORT=27018

sudo mkdir -p $DBHOME
sudo chown -R mongodb:mongodb $DBHOME

Create separate spaces for user database and background knowledge database as well as the configurations:

sudo mkdir -p $DBHOME/userdb && sudo chown -R mongodb:mongodb $DBHOME/userdb
sudo mkdir -p $DBHOME/backdb && sudo chown -R mongodb:mongodb $DBHOME/backdb
sudo mkdir -p $DBHOME/config && sudo chown -R mongodb:mongodb $DBHOME/config
sudo mkdir -p $DBHOME/logs && sudo chown -R mongodb:mongodb $DBHOME/logs

We then proceed to create root users and mongod configuration files.

Root users

Prior to enabling security in MongoDB we create root users for each database:

sudo /usr/bin/mongod \
  --port $USERDB_PORT \
  --dbpath $DBHOME/userdb \
  --pidfilepath /tmp/user.pid \
  --logpath /tmp/user.log \
  --noauth --quiet &

Then create the main root user in the MongoDB shell:

sudo /usr/bin/mongosh --port $USERDB_PORT --authenticationDatabase admin

followed by:

use admin
db.createUser({
  user: "root",
  pwd: passwordPrompt(),
  roles: ["root"]
})
exit

And stop the instance:

sudo /usr/bin/mongod \
  --port $USERDB_PORT \
  --dbpath $DBHOME/userdb \
  --pidfilepath /tmp/user.pid \
  --shutdown

Repeat for the background database by replacing user with back in --dbpath, --pidfilepath and --logpath as well as the port.

mongod config files

The data storage model we follow consists of two instances:

• An instance to hold main application data (user data, analyses, VCF file contents, annotations etc.)
• An instance to store the background annotation knowledge gathered from public resources

For the first instance (user data) the file will be stored in $DBHOME/config/mongod_userdb.conf. To create it, use the following:

echo -e \
"# mongod_userdb.conf

storage:
  dbPath: $DBHOME/userdb

systemLog:
  destination: file
  logAppend: true
  path: $DBHOME/logs/mongod_userdb.log
  logRotate: rename
  timeStampFormat: iso8601-local

net:
  port: $USERDB_PORT
  bindIp: 127.0.0.1 # More addresses will be added

processManagement:
  pidFilePath: $DBHOME/config/mongod_userdb.pid

security:
  authorization: enabled

#operationProfiling:

#replication:

#sharding:
" | sudo tee $DBHOME/config/mongod_userdb.conf && \
sudo chown mongodb:mongodb $DBHOME/config/mongod_userdb.conf

For the second instance (background data) the file will be stored in $DBHOME/config/mongod_backdb.conf:

echo -e \
"# mongod_backdb.conf

storage:
  dbPath: $DBHOME/backdb

systemLog:
  destination: file
  logAppend: true
  path: $DBHOME/logs/mongod_backdb.log
  logRotate: rename
  timeStampFormat: iso8601-local

net:
  port: $BACKDB_PORT
  bindIp: 127.0.0.1 # More addresses will be added

processManagement:
  pidFilePath: $DBHOME/config/mongod_backdb.pid

security:
  authorization: enabled

#operationProfiling:

#replication:

#sharding:
" | sudo tee $DBHOME/config/mongod_backdb.conf && \
sudo chown mongodb:mongodb $DBHOME/config/mongod_backdb.conf

Start the daemons

Firstly start the daemons:

sudo /usr/bin/mongod --config $DBHOME/config/mongod_userdb.conf &
sudo /usr/bin/mongod --config $DBHOME/config/mongod_backdb.conf &

And then change the ownerships of the log and pid files so that mongodb user can take over:

sudo chown mongodb:mongodb $DBHOME/logs/mongod_userdb.log
sudo chown mongodb:mongodb $DBHOME/logs/mongod_backdb.log
sudo chown mongodb:mongodb $DBHOME/config/mongod_userdb.pid
sudo chown mongodb:mongodb $DBHOME/config/mongod_backdb.pid

Stop the daemons

To stop the daemons:

sudo /usr/bin/mongod --shutdown --config $DBHOME/config/mongod_userdb.conf
sudo /usr/bin/mongod --shutdown --config $DBHOME/config/mongod_backdb.conf

Initialization of default users

We need three users:

• A global user to read/write in both static and user databases
• A user to read/write in the user production database
• A user to read/write in the user test database

Instructions to create these users are placed in a non-commited file. The main MongoDB commands are something like the following:

Firstly open the MongoDB shell:

sudo /usr/bin/mongosh --host HOST --port USERDB_PORT \
  --username ADMIN_ROOT_USER --password ADMIN_ROOT_USER \
  --authenticationDatabase admin

And then create the users:

use dbname
db.createUser({
    user: "UserName",
    pwd: "StrongPassword",
    roles: [{ 
        role: "readWrite", 
        db: "dbname" 
    },{ 
        role: "readWrite",
        db: "anotherdbname" 
    },{ 
        role: "read", 
        db: "readonlydbname" 
    }]
})