Horcrux Installation and Usage Guide - Cumulo-pro/Horcrux-Architecture GitHub Wiki

This guide is intended for Validator operators and outlines the usage of Horcrux as a remote signing cluster with three nodes to enhance both high availability and security. This approach eliminates single points of failure and strengthens the security of the validator's signing keys.

+info https://github.com/strangelove-ventures/horcrux

3 Sentry Nodes & 3 Signer Nodes

• Sentry Nodes: Baremetal with the chain's requirements
• Signer Nodes: VPS 4 vCore/ 4 GB/ 80 GB SSD NVMe/ 1 Gb/s

  - SIGNERS open port 2222, close all ports except SSH and prometheus.
  - SENTRYS should expose :1234 (private validation port) to SIGNERS, but not to the open Internet. 26656 port (p2p) to the open Internet

Run the following on your local machine. If you are using the binary on the co-signers instead of the container image, run it on each virtual machine on the co-signer node as well.

✔️ Create a working directory:

mkdir horcruxChain  
cd horcruxChain

✔️Install the las release:

wget https://github.com/strangelove-ventures/horcrux/releases/download/v3.2.3/horcrux_linux-amd64  
mv horcrux_linux-amd64 horcrux  
sudo cp horcrux /usr/bin/  
sudo cp horcrux /usr/local/sbin/horcrux  

✔️Fix permissions error:

sudo chmod +x /usr/bin/horcrux  
sudo chmod +x /usr/local/sbin/horcrux  

✔️Initialise the shared configuration for the co-signers on your local machine:

horcrux config init --node "tcp://192.168.1.100:1234" --node "tcp://192.168.1.101:1234" --node "tcp://192.168.1.102:1234" --cosigner "tcp://192.168.1.103:2222" --cosigner "tcp://192.168.1.104:2222" --cosigner "tcp://192.168.1.105:2222" --threshold 2 --grpc-timeout 1000ms --raft-timeout 1000ms

✔️Generate encryption keys for p2p co-signer-to-signer communications:

horcrux create-ecies-shards --shards 3  

files generated

ls -R  

  ./cosigner_1:
     ecies_keys.json
  ./cosigner_2:
    ecies_keys.json
  ./cosigner_3:
     ecies_keys.json

✔️Fragments the priv_validator_key.json: Copy the priv_validator_key.json file from the validator node into your horcruxChain working directory.

❗️CAUTION: this is a delicate operation and must be done safely.

horcrux create-ed25519-shards --chain-id <chain-id> --key-file ./priv_validator_key.json --threshold 2 --shards 3   

files generated

ls –R  

  ./cosigner_1:
    <CHAIN_ID>shard.json ecies_keys.json
  ./cosigner_2:_
    <CHAIN_ID>shard.json ecies_keys.json
  ./cosigner_3:_
    _<CHAIN_ID>shard.json ecies_keys.json

The contents of the configuration files up to this step are:

✔️Now move your priv_validator_key.json to a safe location as we no longer need it.

Each Signer must have the horcrux.service installed.

✔️Create the horcrux.service:

sudo tee /etc/systemd/system/horcrux.service > /dev/null << EOF  
[Unit]  
Description=MPC Signer node  
After=network-online.target  
[Service]  
User=$USER  
WorkingDirectory=/home/$USER  
ExecStart=$(which horcrux) start  
Restart=on-failure  
RestartSec=3  
LimitNOFILE=4096  
[Install]  
WantedBy=multi-user.target  
EOF

✔️activate the service:

sudo systemctl daemon-reload  
sudo systemctl enable horcrux.service  
sudo systemctl status horcrux.service  

The files must be moved to their corresponding signatory nodes in the ~/.horcrux/ directory.
❗️It is important that each cosigner's files are placed correctly on the corresponding cosigner's node in the initial init order (see node/ips distribution table).

Each of the signatory nodes will have to have the following configuration files: ~/.horcrux/
    config.yaml
    ecies_keys.json
    <CHAIN_ID>_shard.json
    /state
       <CHAIN_ID>_priv_validator_state.json

You need to share the content of the $NODE_HOME/data/priv_validator_state.json file. This file represents the last time your validator key was used to sign by consensus and acts as a "maximum limit" flag to prevent your validator from double signing.

✔️ Stop the validator node and ❗️ONLY after it stops open the priv_validator_state.json file and check the "height" number.

At the validator node:

sudo systemctl stop <noded>  
cat .<NODE_HOME>/data/priv_validator_state.json 

Example:
{
   "height": "361402",
   "round": 0,
  "step": 3,
  "signature": "IEOS7EJ8C6ZZxwwXiGeMhoO8mwtgTiq6VPR/F1cpLZuz0ZvUZdsgQjTt0GniAIgosfEjC5izKw4Nvvs3ZIceAw==",
   "signbytes": "6B080211BA8305000000000022480A205D4E1F722F53A3FD9E0D28639D7CE7B588338570EBA5C340687C30609C47BCA41224080112208283B6E16BEA46797F8AD4EE0ACE424AC7A4827202446B2D56E7F4438541B7BD2A0C08E4ACE28B0610CCD0AC830232066A756E6F2D31"
}

✔️You will need to replace the contents of _~/.horcrux/state/<CHAIN_ID>priv_validator_state.json each signing node with a truncated and slightly modified version of the file. Note especially the "round" value:

{
   "height": "361402",
   "round": "0",
  "step": 3
}

✔️Once you have all the co-signatory nodes fully configured, it is time to start them. Start them all at approximately the same time:

sudo systemctl start horcrux   
sudo journalctl -u horcrux -f  

The signer will continue to try to communicate with the sentinels until we activate the listener of the sentinel's priv_validator in the next step.

✔️On each node, enable the private validator detector and verify the configuration changes with the following commands:

sed -i 's#priv_validator_laddr = ""#priv_validator_laddr = "tcp://0.0.0.0:1234"#g' .<NODE_HOME>/config/config.toml  
cat .<NODE_HOME>/config/config.toml | grep priv_validator_laddr  

priv_validator_laddr = "tcp://0.0.0.0:1234"

❗️ FOR ALL RUNNING NODES IN THE CLUSTER, MAKE SURE YOU ARE USING priv_validator_laddr = "0.0.0.0.0:1234" AND REMOVE THE ORIGINAL priv_validator_key.json FROM ALL NODES.

✔️start Sentinel nodes

sudo systemctl restart {node_service} && journalctl -u {node_service} -f  

✔️check Horcrux logs:

sudo journalctl -u horcrux -f

If all goes well, you will see something like this: