Overview

The Vehicle-Node software is a Scala program that uses Kafka producers and consumers in an asynchronous event-driven system. The major components are the loaders, executors, and the managers.

Loaders are responsible for accessing data such as the current set of map/reduce queries. Executors are responsible for transforming streams of messages consumed from a set of input topics and publishing the results to a set of output topics. Managers are part of the main event loop. Each manager is responsible for performing one piece of the event loop's operations such as updating the executors, updating the clusterhead relays, or publishing sensor data.

Loaders

Currently the only loader of importance is the QueryLoader which can be found in the queries package.

Executors

Executors consume data from a set of Kafka topics (which can be on arbitrary Kafka clusters), perform subclass-specific transformations on the data, and produce the results to another set of Kafka topics. The set of input and output topics are specified as sets of GlobalTopic case classes, which contain the topic name and cluster connection information. By using GlobalTopics it is possible to create executors that can chain together multiple Kafka clusters arbitrarily.

The current set of Executors are Mappers, KVReducers, and Relays. The executor source code can be found in the kafka.executors package.

Managers

Managers are responsible for handling individual asynchronous components of the main event loop. The current set of managers is the StartupManager, QueryManager and the ClusterheadRelayManager. The StartupManager is responsible for waiting for local and cloud Kafka clusters to come online before starting the main loop. The QueryManager is responsible for getting the current set of active map/reduce queries and starting/stopping Mapper and KVReducer executors as necessary. THe ClusterheadRelayManager is responsible for updating the clusterhead Relay executor when the clusterhead changes or when the set of mappers changes.