Orchestrator first steps - grierj/orchestrator GitHub Wiki
You have Orchestrator installed and deployed. What can you do with it?
A walk through common commands, mostly on the CLI side
Must
Discover
You need to discover your MySQL hosts. Either browse to your http://orchestrator:3000/web/discover
page and submti an instance for discovery, or:
$ orchestrator -c discover -i some.mysql.instance.com:3306
The :3306
is not required, since the DefaultInstancePort
configuration is 3306
. You may also
$ orchestrator -c discover -i some.mysql.instance.com
This discovers a single instance. But: do you also have an orchestrator service running? It will pick up from there and will interrogate this instance for its master and slaves, recursively moving on until the entire topology is revealed.
By the way, you can also run a service-like orchestrator from the command line:
orchestrator -c continuous
The above does all the polling and other service activities, without providing HTTP access.
Information
We now assume you have topologies known to orchestrator (you have discovered it). Let's say some.mysql.instance.com
belongs to one topology. a.slave.3.instance.com
belongs to another. You may ask the following questions:
$ orchestrator -c clusters
topology1.master.instance.com:3306
topology2.master.instance.com:3306
$ orchestrator -c which-master -i some.mysql.instance.com
some.master.instance.com:3306
$ orchestrator -c which-slaves -i some.mysql.instance.com
a.slave.instance.com:3306
another.slave.instance.com:3306
$ orchestrator -c which-cluster -i a.slave.3.instance.com
topology2.master.instance.com:3306
$ orchestrator -c which-cluster-instances -i a.slave.3.instance.com
topology2.master.instance.com:3306
a.slave.1.instance.com:3306
a.slave.2.instance.com:3306
a.slave.3.instance.com:3306
a.slave.4.instance.com:3306
a.slave.5.instance.com:3306
a.slave.6.instance.com:3306
a.slave.7.instance.com:3306
a.slave.8.instance.com:3306
$ orchestrator -c topology -i a.slave.3.instance.com
topology2.master.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
+ a.slave.1.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
+ a.slave.2.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
+ a.slave.3.instance.com:3306 [OK,5.6.17-log,STATEMENT]
+ a.slave.4.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
+ a.slave.5.instance.com:3306 [OK,5.6.17-log,STATEMENT]
+ a.slave.6.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
+ a.slave.7.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
+ a.slave.8.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
Move stuff around
You may move servers around using various commands. The generic "figure things out automatically" commands are
relocate
and relocate-slaves
# Move a.slave.3.instance.com to replicate from a.slave.4.instance.com
$ orchestrator -c relocate -i a.slave.3.instance.com:3306 -d a.slave.4.instance.com
a.slave.3.instance.com:3306<a.slave.4.instance.com:3306
$ orchestrator -c topology -i a.slave.3.instance.com
topology2.master.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
+ a.slave.1.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
+ a.slave.2.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
+ a.slave.4.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
+ a.slave.3.instance.com:3306 [OK,5.6.17-log,STATEMENT]
+ a.slave.5.instance.com:3306 [OK,5.6.17-log,STATEMENT]
+ a.slave.6.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
+ a.slave.7.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
+ a.slave.8.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
# Move the slaves of a.slave.2.instance.com to replicate from a.slave.6.instance.com
$ orchestrator -c relocate-slaves -i a.slave.2.instance.com:3306 -d a.slave.6.instance.com
a.slave.4.instance.com:3306
a.slave.5.instance.com:3306
$ orchestrator -c topology -i a.slave.3.instance.com
topology2.master.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
+ a.slave.1.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
+ a.slave.2.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
+ a.slave.6.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
+ a.slave.4.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
+ a.slave.3.instance.com:3306 [OK,5.6.17-log,STATEMENT]
+ a.slave.5.instance.com:3306 [OK,5.6.17-log,STATEMENT]
+ a.slave.7.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
+ a.slave.8.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
relocate
and relocate-slaves
automatically figure out hwo to repoint a slave. Perhaps via GTID; perhaps normal binlog file:pos.
Or maybe there's Pseudo GTID, or is there a binlog server involved? Other variations also supported.
If you want to have greater control:
- Normal file:pos operations are done via
move-up
,move-below
- Pseudo-GTID specific slave relocation, use
match
,match-slaves
,regroup-slaves
. - Binlog server operations are typically done with
repoint
,repoint-slaves
Replication control
You are easily able to see what the following do:
$ orchestrator -c stop-slave -i a.slave.8.instance.com
$ orchestrator -c start-slave -i a.slave.8.instance.com
$ orchestrator -c restart-slave -i a.slave.8.instance.com
$ orchestrator -c set-read-only -i a.slave.8.instance.com
$ orchestrator -c set-writeable -i a.slave.8.instance.com
Break replication by messing with a slave's binlog coordinates:
$ orchestrator -c detach-slave -i a.slave.8.instance.com
Don't worry, this is reversible:
$ orchestrator -c reattach-slave -i a.slave.8.instance.com
This works for normal file:pos as well as GTID setups:
$ orchestrator -c skip-query -i a.slave.8.instance.com
Toggle GTID mode (Oracle & MariaDB):
$ orchestrator -c disable-gtid -i a.slave.8.instance.com
$ orchestrator -c enable-gtid -i a.slave.8.instance.com
Crash analysis & recovery
Are your clusters healty?
$ orchestrator -c replication-analysis
some.master.instance.com:3306 (cluster some.master.instance.com:3306): DeadMaster
a.slave.6.instance.com:3306 (cluster topology2.master.instance.com:3306): DeadIntermediateMaster
$ orchestrator -c topology -i a.slave.6.instance.com
topology2.master.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
+ a.slave.1.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
+ a.slave.2.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
+ a.slave.6.instance.com:3306 [last check invalid,5.6.17-log,STATEMENT,>>]
+ a.slave.4.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
+ a.slave.3.instance.com:3306 [OK,5.6.17-log,STATEMENT]
+ a.slave.5.instance.com:3306 [OK,5.6.17-log,STATEMENT]
+ a.slave.7.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
+ a.slave.8.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
Ask orchestrator to recover the above dead intermediate master:
$ orchestrator -c recover -i a.slave.6.instance.com:3306
a.slave.8.instance.com:3306
$ orchestrator -c topology -i a.slave.8.instance.com
topology2.master.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
+ a.slave.1.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
+ a.slave.2.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
+ a.slave.6.instance.com:3306 [last check invalid,5.6.17-log,STATEMENT,>>]
+ a.slave.8.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
+ a.slave.4.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
+ a.slave.3.instance.com:3306 [OK,5.6.17-log,STATEMENT]
+ a.slave.5.instance.com:3306 [OK,5.6.17-log,STATEMENT]
+ a.slave.7.instance.com:3306 [OK,5.6.17-log,STATEMENT,>>]
More
The above should get you up and running. For more please consult the Manual. For full CLI commands listing jusr run
orchestrator