Continuous Running Migrations - tsgrp/OpenMigrate GitHub Wiki

There are multiple ways you can have migrations that run forever in OM. The use cases and complexities differ but here are a few options you have.

Active MQ Migration In this migration your queue is an ActiveMQ instance. This instance of activeMQ is triggered by the start of your OM job and it maintains persistence using Kaha DB which is a file based persistence database. This is one of the best options since the main job of Active MQ is to be a queue and handle traffic. The queue is populated by adding JMS messages, which is then distributed to the OMs running.

Chron Jobs One can also schedule a recurring chron job or scheduled task to run on the server at a specific time interval. This would re-trigger the OM process every X minutes and thereby pick up any new information to be migrated. This is technically not a continuous running migrations, so you do have some overheads like re-initializing everything and also waiting for the job to kick off to pick up new information. There are various examples of these scripts you can schedule to use this method of continuous migration.

_This article helps set it up _ https://tecadmin.net/crontab-in-linux-with-20-examples-of-cron-schedule/

In windows we can use scheduled jobs.

MigrationPipeline Change This can be done by commenting out the disengage in the MigrationPipeline.java and simply calling the engage function again to restart the migration. This would create a continuous running OM scenario but this has not been fully implemented for large systems and there could thread locks and timeouts that we aren't aware of. In case of this though, one has to manually shut down the migration and restart.

Infinitely Running OM Script A shell or batch script can be created to run in an infinite loop to trigger the OM job. This can be controlled with the number of instances active at any particular point. This is similar to a chron job except this triggers the next OM job right after the first one ends. A possible detriment of this is that since this is not a controlled scheduled job, killing it might require manual steps of finding the PID and killing it.

_an example of this in a shell file. _ #!/usr/bin/env bash

loop_counter=16 while true; do om_count=$(ps -ef | grep -e 'projectname' | grep -v 'grep' | wc -l) if (( om_count < 1 )) && (( loop_counter-- > 0 )); then sh projectlocation/om.sh & sleep 5 else break fi done