Dev Environment Architecture

The Dev environment consists of the following parts:

1. The middle-ware device with Apache Kafka + Zookeeper running on a local machine, basically analogous to the central server. This Kafka + Zookeeper setup listens for the messages from the cluster-heads on the Advertised Listener port on some Kafka topic.
2. The Cluster Head - One such running docker container is designated as the cluster head, manually. That means that rest of the vehicle nodes (i.e. instance of running docker containers) send their messages to the cluster head. The cluster head listens to all of the incoming message requests in the Advertised Listener port and relays these messages combined with its own message to the middle-ware.
3. The Vehicle Nodes are being simulated on the same local machine where the middle ware is hosted on. We are using Docker to achieve this. The main Spindle vehicle jar file, which is supposed to run on each Raspberry Pi in the real world setup, is being encapsulated within a Docker image. An docker image is a snapshot of a docker container. A docker container is a running instance of a docker image, and is a sort of like a VM except that it does not bundle a full operating system but only the necessary libraries. This makes it lightweight and possible to deploy scores of docker containers on low system resources. We can run multiple copies of such containers on a single computer, each isolated from the others and emulating a vehicle node. For more information on docker please visit What is docker?. The vehicles send their messages to the cluster-head node through an Apache Kafka+ Zookeeper setup running inside each container natively, and isolated from other containers.
4. Finally, the Spark program - This can be run on basically any device as long as it is pointed to the middle-ware Kafka and ZOokeeper ports for data ingestion. However, we are also running it on the same local machine as above, by pointing it to the configuration of the Zookeeper and Kafka running directly n the local machine (not inside the docker containers).
5. At the current moment, all the vehicles generate stub data i.e. some dummy data. The emphasis is on how they are connected, how they can send and aggregate in the network and how much data they are sending. On stop of the simulation, all the Log files are pulled from the docker containers in to the host machine.
6. Following is a diagram of the dev environment architecture: images/dev-architecture.jpg