activity cache replication - Orodan/Hilary GitHub Wiki
If there is a communications fault for the master redis instance, normally activity-cache0, haproxy will automatically start using the slave, activity-cache1. However when the redis master re-joins the cluster haproxy does not switch back to the master.
This procedure should reset the slave-master setup to normal, activity-cache0=master (now called 0) and activity-cache1=slave (now called 1).
Obviously check the for the private IPs of activity-cache{0,1} in case they've changed for some reason.
- first swap master/slave relationship;
redis-cli slaveof no one on 1...
redis-cli slaveof 10.224.14.104 6379 redis-cli sync on 0...
- wait until replication is normalized
redis-cli info | grep master_ on 0
if master_sync_in_progress=0 then no sync is in progress
if master_last_io_seconds_ago>0 but only a few seconds then a sync has been done recently
A better definitive check would be nice
- swap master/slave back again
redis-cli slaveof no one on 0...
redis-cli slaveof 10.224.14.103 6379 redis-cli sync on 1...
- reload haproxy
oldpid=$(ps -A | awk '/redis/ { print $1 }') service haproxy reload kill $oldpid on proxy0...
Checking the activity cache is easy, log in and upload a doc and it should appear in your activity stream.