Aquarium Fish: A Distributed Resource Management System

Aquarium Fish is a core component of the Aquarium platform, serving as a dynamic and heterogeneous resource management system. It functions as a userspace daemon, managing resources from both local hardware pools and remote resources such as cloud environments.

CO: Fish lives in the Aquarium.

The Aquarium Fish components are bundled into a single binary per supported operating system, eliminating the need for additional configurations. This streamlined packaging simplifies deployment, enabling developers to quickly run the required environment locally, debug CI system issues, or experiment with pipelines in real-world CI/CD build and release environments.

Key Features

• CI/Release-Oriented: Implements best DevOps practices for managing build/test environments.
• Distributed: Utilizes cluster synchronization consensus to distribute information between nodes.
• Single Binary: A Go-based codebase simplifies integration and distribution.
• Embedded Key-Value Database: No need for a separate database deployment and management.
• REST & RPC+streaming API: Provides a simple communication interface (Basic Auth + TLS transport included) auto-generated from Protobuf specifications.
• Driver Support: Plugin interface for external schedulers and compute resources, including:
  • Gates: Facilitates communication with schedulers (e.g., GitHub) or enables additional user interfaces (e.g., proxyssh).
  • Providers: Works with various compute resources (VMWare, Docker, AWS, native).
• Simple Start: Requires only a TCP port (API) and additional configuration for drivers if needed.
• Modular: Easily replaceable components for custom implementations.
• Secure: Utilizes SSL certificates and API keys for secure access to functionality.

General Usage

When the cluster receives a resource request (Application), each node computes if it can provide the required resource and shares this information with the cluster. The cluster then selects the optimal node to fulfill the request.

Each node also manages resources to quickly respond to requests. For example, if the demand for a resource like xcode12.2 is high, some nodes can predict this and begin provisioning resources ahead of time to meet demand. While there's still work to be done, implementing such functionality is more straightforward than it may appear.

A typical usage path involves the following steps (see the example scripts for more details):

1. Run Fish Cluster: Start the desired number of nodes on an available pool of machines (or just one machine) and join them. You can designate one node as the seed and connect others to it. The cluster will form automatically.
2. Create a Label: Labels describe the environment, ensuring that the specified label is exactly what you need. While labels are immutable, you can create new versions of labels.
3. Create an Application: To request a resource, form an application. The cluster will review the request, and each node will vote during the election process. A node will be selected to provide the resource.
4. Monitor Resource Allocation: The election process may take some time depending on the cluster load. Be sure to check the application status and ensure it hasn't encountered an error.
5. Use the Resource: Once the resource is allocated, you can typically begin using it (e.g., in a Jenkins CI Agent). However, resource availability depends on the cluster setup and driver configurations.
6. Deallocate the Resource: Once you've completed your workload, remember to deallocate the resource to free it up for others.

For a deeper understanding of internal resources, check out the Protobuf spec documentation. It contains valuable information about the objects used by the Fish cluster.

Permissions

API is enforced with Role-based permission system and contains 3 built-in roles: Administrator, User and Power:

• Administrator - gives full control to the user
• User - allows user basic operations with Applications allocation and control of it's own resources
• Power - is addition to User role that gives user access to ApplicationTasks and request for Access Credentials

Also you can create your own roles to give users more or less privileges, like only managing Labels or view-only access to the system.

Cluster Networking

The diagram below illustrates how the cluster interacts with each node, the exposed interfaces, and the internal structure of the nodes: