Standard Runtime Environment Architecture with the Raspberry Pis

The Standard Runtime Environment consists of the following parts:

1. The middle-ware device with Apache Kafka + Zookeeper running on some machine, basically analogous to the central server. This Kafka + Zookeeper setup listens for the messages from the cluster-heads on the Advertised Listener port on some Kafka topic. It is absolutely imperative that this machine is reachable by all the cluster-head raspberry pis and that the Zookeeper + Kafka setup on this machine is running before any of the Raspberry Pis are being fired up.
2. The Cluster Head - One particular raspberry pi is manually designated as the cluster head. That means that rest of the vehicle nodes (i.e. instance of running docker containers) send their messages to this cluster head pi. The cluster head listens to all of the incoming message requests in the Advertised Listener port and relays these messages combined with its own message to the middle-ware. Also all the other raspberry pis, as well as the cluster head itself needs to know where the cluster head and middle ware kafkas are running i.e. the corresponding IP addresses and the ports. This has to be done in every raspberry pi by setting up 3 environment variables as described in the guidelines.
3. The Raspberry Pis a.k.a. the Vehicle Nodes are being simulated on the raspberry pis. Each Raspberry pi simulates a single vehicle. The spindle jar file which is prepared in the dev machine, is deployed to each of the raspberry pis maintaining the folder structure as described in the guidelines .
   The jar basically contains the files needed for Spindle as well as Apache Kafka and Zookeeper encapsulated inside it so that it does not have to depend on external resources. The vehicles (or pis) thus send their messages to the cluster-head node through the Apache Kafka already natively present in the jar, and then starts logging the transmission details.
4. Finally, the Spark program - This can be run on basically any device as long as it is pointed to the middle-ware Kafka and ZOokeeper ports for data ingestion. However, right now, we are running it on the middle-ware machine, by pointing it to the configuration of the Zookeeper and Kafka running on the same machine.
5. At the current moment, all the vehicles generate stub data i.e. some dummy data. The emphasis is on how they are connected, how they can send and aggregate in the network and how much data they are sending. During the simulation all the log files are generated inside the individual raspberry pis.
6. The following is a schematic of the standard runtime environment with the raspberry pi + middleware + Apache Spark setup: images/standard-archtecture.jpg