Element state and element operations are fully separated.
The command interface and the “state store” are completely independent, being normally run on different hosts.
The system has architectural provisions for extending the operation set for different kinds of objects (e.g., “VM migrate”, “VM suspend”, “AI fail”).
A new instance of the CLI can be “attached” to a running experiment at any moment, for the execution of operations on an element (e.g., start an experiment at the office, attach a CLI instance later from home to check the progress).
Element state information stored as key-value pairs.
Use of in-memory key-value store (Redis) allows high-throughput (10K’s of inserts/sec) on low-power servers.
Key-value servers have (typically) a scalable (scale-out) architecture.
The execution of an operation is delegated to the element that is closer to the actual locus of the action.
The load management (i.e., how high is the load, and for how is it going to be executed?) is performed by the “driver VM” on each AI.
Performance collection for all VMs on an AI is also performed by the “driver VM”.