Bring in the slave check service - ksgh/dkrb-slave-check GitHub Wiki

Let's plant the slave-check.yml compose file:

version: '3'

networks:
  mysql_net-generic:
    external: true

## All env's passed in should be prefixed with "SC_"... with the exception of http(s)?_proxy (if required)
## Also see: https://docs.docker.com/compose/compose-file/#restart_policy

services:
  check:
    image: kshenk/slave-check:latest
    networks:
      - mysql_net-generic
    environment:
      PS1: '[\[\e[1;32m\]\u\[\e[m\]@\[\e[1;32m\]slave-check\[\e[m\] \[\e[33m\]\w\[\e[m\]]# '

      ## Match mode means we're either going to be looking for actual AWS instances,
      ## or mysql instances running within the same docker network
      ## SC_MATCH_MODE: <aws|docker>
      SC_MATCH_MODE: docker

      ## If we're matching in "docker mode", we're going to list the containers/nodes/hosts to watch
      ## Currently these need to be resolvable within docker's network mesh.
      SC_DOCKER_NODES: mysql-01,mysql-02

      ## I think these are self-explanatory
      SC_SLAVE_USERNAME: slave-check
      SC_SLAVE_PASSWORD: fermyg00ber
      SC_SLAVE_PORT: 3306

      ## Redis Host (if more than one are provided we'll assume redis is running sentinels
      SC_REDIS_HOST: redis

      ## If we want the "slave_status" key to expire, set this (in seconds). otherwise comment it out to turn that off.
      SC_REDIS_KEY_EXPIRE: 25

      ## how many times to allow the app to loop
      SC_ITERATIONS: 10

      ## how long to sleep in between iterations
      SC_CYCLE_SLEEP: 2

      ## current has no bearing on anything
      #SC_SLAVE_LAG_THRESHOLD: int

      ## Time in seconds for lag to be "in the green"
      SC_SLAVE_MAX_GREEN: 3

      ## range(start, len) for lag to be "in the yellow"
      ## This is optional and the range will be calculated if left blank
      #SC_SLAVE_RANGE_YELLOW: start,length

      ## If slave lag is greater than this (in seconds) we're in the red
      SC_SLAVE_MIN_RED: 7

    restart: always
    # The below was used when I was running this as a swarm service. It will be ignored by `docker-compose`.
    deploy:
      replicas: 1
      update_config:
        parallelism: 1
        delay: 3s
      restart_policy:
        condition: any
        delay: 2s
        max_attempts: 5
        window: 60s

There's quite a few knobs to tweak here, and in my opinion it's much easier to do this with environment variables in regards to running this as a docker service than it would be to accept all this with argparse.

So the important stuff

SC_DOCKER_NODES: mysql-01,mysql-02: We're going to be concerned with watching replication on our two slaves (note I'm using the node/host address as opposed to the service addresses)

SC_REDIS_HOST: redis: Here I'm using the network alias. The only reason being I'm going to run one redis (standalone) node at any one point in time and the nodename could change. Any application addressing my redis service will likely be using this network alias as well as opposed to the nodename so this makes sense.

SC_REDIS_KEY_EXPIRE: 25: This is another safeguard. The information in redis will expire after 25 seconds if we haven't been updating it. This way if something stops reporting slave status in it won't be possible for us to accidentally read stale slave status information. Any external application can handle this however it wants.

So we'll fire this one up:

docker stack deploy -c slave-check.yml slave

You'll likely see something of the sort

Creating service slave_check

So now that we have that up and running..... let's get the name of our redis service again

[centos@node-01 ~]$ docker ps --filter name=redis --format '{{ .Names }}'
redis_redis-00.1.a831m0adxxolcxqnkm8h4dlre

And we can check to see that we're reporting slave status in

docker exec redis_redis-00.1.a831m0adxxolcxqnkm8h4dlre get slave_status | base64 --decode | jq
{
  "status": "green",
  "hosts": [
    {
      "seconds": 0,
      "host": "mysql-01",
      "datetime": "2019-10-15 01:10:16"
    },
    {
      "seconds": 0,
      "host": "mysql-02",
      "datetime": "2019-10-15 01:10:16"
    }
  ],
  "avg_seconds": 0,
  "updated_at": "2019-10-15 01:10:16"
}

Run that a couple times and you should notice the datetime/update_at is changing. We've got this rolling!

Now, with your programming language of choice, you can access redis ask for the slave_status key, base64 decode it and make whatever decisions you need with this information.

⚠️ **GitHub.com Fallback** ⚠️