System Structure

The application instance is a scala project based on spray and quartz. The codes are the same for application 1, application 2, application 3 .... It does 2 things

1. Create the trigger/job in database.

2. Execute the job on the trigger time based on the data in database.

Once the triggers and jobs are in database, we can manage them and search them in the DB. No mater all the applications are restarted or stopped. Once there is a available instance running again, our jobs will be picked and executed again.

And there is only one bottle neck. It is the DB. We may also have another system structure like this.(Not demo yet, but it is only some configuration changes in quartz.propeties)

Data Flow

Trigger and JobDetail

There are 2 kinds of jobs.

1. Produce Message Job.

Job to take the first 4 steps in the data flow chart. One campaign will have one "Produce Message Job".

It will usually triggered right after we create campaign or 30 minutes before the campaign event date time. If this job is not created, we will know that immediately and fix that. If this job is created, it will stored in the DB. No matter how the applications healthy or out of memory. Once we reboot the application, the system will execute this task.

And the purpose for this job is to produce the "Send Message Job". For example, for a certain campaign event at 10:00 am, this job may be triggered at 9:30 am and load all the data from database, filter by profile, we get 1 million devices to send our message at last. We put some logic here to create "Send Message Job". In each "Send Message Job", we will have these things in detail:

1. Trigger at 10:00 am

2. 1000 device Ids in this job detail

3. A meaningful name for this job, eg. SendMessage_CampaignName_1000, SendMessage_CampaignName_2000, ...

If we have 1 million devices, we may create 1000 "Send Message Job" in our databases. Hope this will be done in 30 minutes.

If our server fails down during creating "Send Message Job", we can redo this work, because we know how many "Send Message Job" are in our database with the meaningful names. If we already have SendMessage_CampaignName_90000, the system will begin from SendMessage_CampaignName_100000.

2. Send Message Job.

There will be 1000 "Send Message Job" in our database if we have 1 million devices to send. It is not a big problem for multiple applications with multiple threads based on quartz. All the application 1, 2, 3 ... n will use multiple threads, one thread will load one job and execute and keep on working on the pending jobs in database. If one application fail, other applications will take the task and go on. Even we restart all the instances(we will not always do that), the system will re-pick the jobs from database once there is at least one application is running.

TODOs and What We Have

1. (Done) Integrate Quartz, Spray together in Scala Project.

2. (Done) SQL to create the tables, all other configuration for Quartz cluster/DB storage.

3. (TODO) Load the device Ids or other information from DB based on GUTS. We may not have all the methods in groovy LocalpointModel project to load the device information matching the campaign.

4. (Ready) Filter the devices by profile based on GUTS. We can directly use GUTS.

5. (TODO) Some logic to create meaningful names of "Send Message Job", then the creating task can be picked up where the system fails.

6. (Ready) Send the message based on PushNotification Project. Both are scala projects, we can directly use PushNotification.

7. (TODO) Better configuration and extends Logging to record all the jobs are finished by our system. Maybe for performance reason, once the job is executed and finished, it is removed from the Quartz DB. Quartz only provide a listener after finish the job to move the job detail data to our history DB. (If we need the history data after we successfully send our the message)

8. (TODO) More tests and Performance tests.

Performance

On my slowly laptop, 4 instances with 10 threads in each instance, on top of one MYSQL database instance will execute 4000 jobs in 15 minutes.

TODO