Clustering Processes - Sotera/watchman GitHub Wiki

Clustering: Scheduler and Worker processes

1. every 15 mins. a scheduler process submits a job via redis with attributes: ES host, start time, end time
2. on completion, the worker writes job result to redis, either an error message or clustering output.
3. the scheduler regularly polls redis for job results and writes the results metadata to a job record (mongo?) with attributes: job id, job status, job params: start, end
4. the scheduler, on boot up, can fill in missing time periods in the jobs table by submitting multiples jobs to the worker.