Deploying Scylla Manager and connecting it to a Scylla Cluster

This guide will follow the steps required to deploy Scylla Manager using the ansible-scylla-manager role

Requirements

• An inventory file in a supported format (ini/yaml)
• Ansible 2.8 or higher
• SSH access to all the Scylla Manager hosts (usually just one host)

Variables

The defaults/main.yml file of the role is highly commented. It can be found here: https://github.com/scylladb/scylla-ansible-roles/blob/master/ansible-scylla-manager/defaults/main.yml

Since different operating systems require different dependencies, keys and configuration, the role is designed to be flexible, but might require some careful data gathering before it is used.

To keep things easy, it is recommended to keep the defaults and set only the necessary variables. Open https://scylladb.com and move to the downloads page. There, select your preferred version and platform:

Scylla Manager Download page and the variables that can be populated from it

For RHEL/CentOS, copy the repo URL:

scylla_manager_repo_url: "http://downloads.scylladb.com/rpm/centos/scylladb-manager-2.1.repo"

For Debian/Ubuntu, some additional data needs to be set:

scylla_manager_repo_url: "http://downloads.scylladb.com/deb/debian/scylladb-manager-2.1-stretch.list"
scylla_repo_keyserver: "keyserver.ubuntu.com"
scylla_manager_repo_keys:
  - 6B2BFD3660EF3F5B
  - 17723034C56D4B19

Internal Scylla database variables

The role installs Scylla for the manager's use, and that requires some basic settings as well:

scylla_manager_db_vars:
  scylla_repos:
    - http://downloads.scylladb.com.s3.amazonaws.com/rpm/centos/scylla-4.1.repo
  scylla_dependencies:
    - curl
    - wget
  scylla_edition: oss

NOTE: if there are any dependencies listed for the Manager in the Downloads page, the dependencies can be added to the list of scylla_dependencies

Scylla Clusters to be attached to the new Manager

Scylla Manager is there to manager Scylla clusters. This role can connect existing clusters to Scylla Manager after deploying the Manager itself.

The variable requires the following:

• cluster_name: name for the cluster (not necessarily the same name set under cluster_name in Scylla itself)
• host: an address of a node in the cluster (the Manager will connect to a single node in order to extract the required information)
• auth_token_file: A text file containing the authentication token configured on all the nodes in the cluster. If you have used the ansible-scylla-node role, this file should be available after the role finished running
• without_repair: when cluster is added, Manager schedules repair to repeat every 7 days. To create a cluster without a scheduled repair, set this flag to true

scylla_clusters:
  - cluster_name: "MyCluster"
    host: 10.128.0.2
    auth_token_file: scyllamgr_auth_token.txt
    without_repair: false

Parameters file:

# parameters.yml
scylla_clusters:
  - cluster_name: "MyCluster"
    host: 10.128.0.2
    auth_token_file: scyllamgr_auth_token.txt
    without_repair: false

scylla_manager_db_vars:
  scylla_repos:
    - http://downloads.scylladb.com.s3.amazonaws.com/rpm/centos/scylla-4.1.repo
  scylla_dependencies:
    - curl
    - wget
  scylla_edition: oss

scylla_manager_repo_url: "http://downloads.scylladb.com/rpm/centos/scylladb-manager-2.1.repo"

Inventory

The inventory file should contain a scylla-manager section. Example expanding upon the inventory presented in the ansible-scylla-node documentation:

#inventory.ini
[scylla-manager]
10.142.0.2
[scylla]
10.142.0.3 dc=us-east1 rack=rack-b
10.142.0.4 dc=us-east1 rack=rack-b
10.142.0.5 dc=us-east1 rack=rack-b
10.128.0.2 dc=us-central1 rack=rack-a
10.128.0.3 dc=us-central1 rack=rack-a
10.128.0.4 dc=us-central1 rack=rack-a

Defining Alert Manager rules

• The role is always going to install rules that come with a Monitoring release: prometheus.rules.yml.
• One can provide a file with a custom set of rules via prometheus_rules_file parameter, e.g.

prometheus_rules_file: my_awesome_alerts.rules.yml

IMPORTANT: When provided these rules are going to be installed in addition to rules in prometheus.rules.yml!

• In order to filter out some rules from the prometheus.rules.yml one can use a scylla_monitoring_default_alert_reject_filter parameter and set it to a regular expression that will be used to filter out rules by according to an alert field value.

For example to filter out every rule that ends with "Full" or starts with "cql":

scylla_monitoring_default_alert_reject_filter: ".*Full|cql.*"

Example of a custom alerts rules file:

groups:
- name: scylla.rules
  rules:
  - alert: DiskFull
    expr: node_filesystem_avail_bytes{mountpoint="/var/lib/scylla"} / node_filesystem_size_bytes{mountpoint="/var/lib/scylla"}
      * 100 < 25
    for: 30s
    labels:
      severity: "warn"
    annotations:
      description: '{{ $labels.instance }} has less than 25% free disk space.'
      summary: Instance {{ $labels.instance }} low disk space
  - alert: DiskFull
    expr: node_filesystem_avail_bytes{mountpoint="/var/lib/scylla"} / node_filesystem_size_bytes{mountpoint="/var/lib/scylla"}
      * 100 < 20
    for: 30s
    labels:
      severity: "error"
    annotations:
      description: '{{ $labels.instance }} has less than 20% free disk space.'
      summary: Instance {{ $labels.instance }} low disk space

Example playbook

# manager.yml
- hosts: scylla-manager
  roles:
    - role: ansible-scylla-manager

NOTE: You should have the ansible-scylla-node role available in your local roles directory

Executing the playbook

ansible-playbook -i inventory.ini -e '@parameters.yml' manager.yml