Scheduler - michaelmarconi/oncue GitHub Wiki

The Scheduler's key responsibility is to assign jobs on the unscheduled queue to the registered agents. Let's take a look at the key components:

images/scheduler-components.png

Data structures

• UnscheduledJobs: Maintains a PriorityQueue of all jobs that remain unscheduled, ordered by placing the oldest jobs at the head of the queue.
• ScheduledJobs: Maps the registered agents to the jobs their have been scheduled to work on.

Backing store

This is an optional component that allows the scheduler data structures to maintain their state persistently. There is currently only one implementation: the RedisBackingStore, which uses Redis lists and hashes to ensure that no jobs are lost, even in the case where the Scheduler component explodes.

Current implementations

• SimpleQueuePopScheduler: A super-simple scheduler that will pop the first job off the unscheduled jobs queue and hand it to the first agent that responds to a WORK_AVAILABLE broadcast. Useful for testing.

• ThrottledScheduler: This scheduler employs a throttling strategy to ensure that agents are never overwhelmed with work. It is designed to work with a ThrottledAgent, which will request a limited number of jobs using a ThrottledWorkRequest depending on how many jobs it is currently tackling. The scheduler will pop just enough jobs off the unscheduled queue to satisfy the agent's work request.

• JVMCapacityScheduler: This scheduler works in tandem with the JVMCapacityAgent to schedule jobs according to the amount of memory available in each Agent's JVM. It is currently a proof-of-concept and should not be used for production!

New implementations

To create a new kind of Scheduler, extend AbstractScheduler and override the following key method:

protected abstract void scheduleJobs(AbstractWorkRequest workRequest);

This method must create a new Schedule object, which maps jobs to agents. Once the Schedule has been constructed, the jobs must be dispatched by calling the dispatchJobs(schedule) method!