XD Distributed Runtime - garyrussell/spring-xd GitHub Wiki

This document describes what’s happening "under the hood" of the Spring XD Distributed Runtime (DIRT) and in particular, how the runtime architecture achieves high availability and failover in a clustered production environment. See Running in Distributed Mode for more information on installing and running Spring XD in distributed mode.

This discussion focuses on Spring XD’s core runtime components and the role of ZooKeeper in managing the state of the Spring XD cluster and enabling automatic recovery from failures.

A production Spring XD environment is typically distributed among multiple hosts in a clustered environment. Spring XD scales horizontally when you add container instances. In the simplest case, all containers are replicas, that is each instance is running on an identically configured host and modules are deployed to any available container in a round-robin fashion. However, this simplifying assumption does not address real production scenarios in which more control is requred in order to optimize resource utilization. To this end, Spring XD supports a flexible algorithm which allows you to match module deployments to specific container configurations. The container matching algorithm will be covered in more detail later, but for now, let’s assume the simple case. Running multiple containers not only enables horizontal scalability, but enables failure recovery. If a container becomes unavailable due to an unrecoverable connection loss, any modules currently deployed to that container will be deployed automatically to the other available instances.

Spring XD requires that a single active Admin server handle interactions with the containers, such as stream deployment requests, as these types of operations must be processed serially in the order received. Without a backup, the Admin server becomes single point of failure. Therefore, two (or more for the risk averse) Admin servers are recommended for a production environment. Note that every Admin server can handle all requests via REST endpoints but only one instance, the "Leader", will actually perform requests that update the runtime state. If the Leader goes down, another available Admin server will assume the leader role. Leader Election is an example of a common feature for distributed systems provided by the Curator Framework which sits on top of ZooKeeper.

An HA Spring XD installation also requires that external servers - ZooKeeper, messaging middleware, and data stores needed for running Spring XD in distributed mode must be configured for HA as well. Please consult the product documentation for specific recommendations regarding each of these external components. Also see Message Bus Configuration for tips on configuring the MessageBus for HA, error handling, etc.

In the previous section, we claimed that if a container goes down, Spring XD will redeploy any modules deployed on that instance to another available container. We also claimed that if the Admin Leader goes down, another Admin server will assume that role. ZooKeeper is what makes this all possible. ZooKeeper is a widely used Apache project designed primarily for distributed system management and coordination. This section will cover some basic concepts necessary to understand its role in Spring XD. See The ZooKeeper Wiki for a more complete overview.

ZooKeeper is based on a simple hierarchical data structure, formally a tree, and conceptually and semantically similar to a file directory structure. As such, data is stored in nodes. A node is referenced via a path, for example, /xd/streams/mystream. Each node can store additional data, serialized as a byte array. In Spring XD, all data is a java.util.Map serialized as JSON. The following figure shows the Spring XD schema:

A ZooKeeper node is either ephemeral or persistent. An ephemeral node exists only as long as the session that created it remains active. A persistent node is, well, persistent. Ephemeral nodes are appropriate for registering Container instances. When an Spring XD container starts up it creates an ephemeral node, /xd/containers/<container-id>, using an internally generated container id. When the container’s session is closed due to a connection loss, for example, the container process terminates, its node is removed. The ephemeral container node also holds metadata such as its hostname and IP address, runtime metrics, and user defined container attributes. Persistent nodes maintain state needed for normal operation and recovery. This includes data such as stream definitions, job definitions, deployment manifests, module deployments, and deployment state for streams and jobs.

Obviously ZooKeeper is a critical piece of the Spring XD runtime and must itself be HA. ZooKeeper itself supports a distributed architecture, called an ensemble. The details are beyond the scope of this document but for the sake of this discussion it is worth mentioning that there should be at least three ZooKeeper server instances running (an odd number is always recommended) on dedicated hosts. The Container and Admin nodes are clients to the ZooKeeper ensemble and must connect to ZooKeeper at startup. Spring XD components are configured with a zk.client.connect property which may designate a single <host>:<port> or a comma separated list. The ZooKeeper client will attempt to connect to each server in order until it succeeds. If it is unable to connect, it will keep trying. If a connection is lost, the ZooKeeper client will attempt to reconnect to one of the servers. The ZooKeeper cluster guarantees consistent replication of data across the ensemble. Specifically, ZooKeeper guarantees:

• Sequential Consistency - Updates from a client will be applied in the order that they were sent.

• Atomicity - Updates either succeed or fail. No partial results.

• Single System Image - A client will see the same view of the service regardless of the server that it connects to.

• Reliability - Once an update has been applied, it will persist from that time forward until a client overwrites the update.

• Timeliness - The clients view of the system is guaranteed to be up-to-date within a certain time bound.

ZooKeeper maintains data primarily in memory backed by a disk cache. Updates are logged to disk for recoverability, and writes are serialized to disk before they are applied to the in-memory database.

In addition to performing basic CRUD operations on nodes, A ZooKeeper client can register a callback on a node to respond to any events or state changes to that node or any of its children. Such node operations and callbacks are the mechanism that control the Spring XD runtime.

Assuming more than one Admin instance is running, Each instance requests leadership at start up. If there is already a designated leader, the instance will watch the xd/admin node to be notified if the Leader goes away. The instance designated as the "Leader", using the Leader Selector recipe provided by Curator, a ZooKeeper client library that implements some common patterns. Curator also provides some Listener callback interfaces that the client can register on a node. The AdminServer creates the top level nodes, depicted in the figure above:

• /xd/admins - children are ephemeral nodes for each available Admin instance and used for Leader Selector

• /xd/containers - children are ephemeral nodes containing runtime attributes including hostname,process id, ip address, and user defined attributes for each container instance.

• /xd/streams - children are persistent nodes containing the definition (DSL) for each stream.

• /xd/jobs - children are persistent nodes containing the definition (DSL) for each job.

• /xd/taps - children are persistent nodes describing each deployed tap.

• /xd/deployments/streams - children are nodes containing stream deployment status (leaf nodes are ephemeral).

• /xd/deployments/jobs - children are nodes containing job deployment status (leaf nodes are ephemeral).

• /xd/deployments/modules/requested - stores module deployment requests including deployment criteria.

• /xd/deployments/modules/allocated - stores information describing currently deployed modules.

The admin leader creates a DeploymentSupervisor which registers listeners on /xd/deployments/modules/requested to handle module deployment requests related to stream and job deployments, and xd/containers/ to be notified when containers are added and removed from the cluster. Note that any Admin instance can handle user requests. For example, if you enter the following commands via XD shell,

This command will invoke a REST service on its connected Admin instance to create a new node /xd/streams/ticktock

Assuming the deployment is successful, This will result in the creation of several nodes used to manage deployed resources, for example, /xd/deployments/streams/ticktock. The details are discussed in the example below.

If the Admin instance connected to the shell is not the Leader, it will perform no further action. The Leader’s DeploymentSupervisor will attempt to deploy each module in the stream definition, in accordance with the deployment manifest, to an available container, and update the runtime state.

This section describes how the Spring XD runtime manages deployment internally. For more details on how to deploy streams and jobs see Deployment.

To process a stream deployment request, the StreamDeploymentListener invokes its ContainerMatcher to select a container instance for each module and records the module’s deployment properties under /xd/deployments/modules/requested/. If a match is found, the StreamDeploymentListener creates a node for the module under /xd/deployments/modules/allocated/<container_id>. The Container includes a DeploymentListener that monitors the container node for new modules to deploy. If the deployment is successful, the Container writes the ephemeral nodes status and metadata under the new module node.

When a container departs, the ephemeral nodes are deleted so its modules are now undeployed. The ContainerListener responds to the deleted nodes and attempts to redeploy any affected modules to another instance.