CoreOS Experimentation and Notes - tooltwist/documentation GitHub Wiki
This tool is great for monitoring the performance (CPU, memory, network) of Docker containers.
This is enabled by setting extra parameters to the CoreOS config (fleet.socket).
#cloud-config
coreos:
etcd:
# generate a new token for each unique cluster from https://discovery.etcd.io/new
discovery: https://discovery.etcd.io/21c642a255f72bd566439367785cc45c
# multi-region deployments, multi-cloud deployments, and droplets without
# private networking need to use $public_ipv4
addr: $private_ipv4:4001
peer-addr: $private_ipv4:7001
units:
- name: etcd.service
command: start
- name: fleet.socket
command: start
content: |
[Socket]
ListenStream=7070
Service=fleet.service
[Install]
WantedBy=sockets.target
- name: fleet.service
command: start
This allows fleet to be accessed via http://:7070/v1-alpha/machines
{"machines":[
{"id":"0277bc7109c84589be8bdf3ce58ba01d","primaryIP":"169.254.30.228"},
{"id":"dc7d6423c97843d29b704911f187d18c","primaryIP":"169.254.156.94"}, <---- (A)
{"id":"f3f94fdeb45040cbb0c513d1fe694301","primaryIP":"169.254.13.134"}
]}
From my laptop I cannot ping any of these addresses.
Strangely, the command correctly returns details about all the servers in the coreOS cluster, but with IP addresses unrelated to the private or public IP addresses of the DigitalOcean servers on which they were running. From the Digital Ocean console:
Public Network
IP Address: 128.199.190.70 <----- (B)
Gateway: 128.199.128.1
Netmask: 255.255.192.0
Private Network
Private IP: 10.130.207.238 <----- (C)
Netmask: 255.255.0.0
Logging onto the CoreOS box which is running fleet, the ifconfig command returned the following:
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.17.42.1 netmask 255.255.0.0 broadcast 0.0.0.0 <----- (D)
...
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 169.254.156.94 netmask 255.255.0.0 broadcast 169.254.255.255 <----- (A)
...
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.130.207.238 netmask 255.255.0.0 broadcast 10.130.255.255 <----- (C)
...
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
...
Here eth1 matches the private IP address shown by Digital Ocean, and eth0 matches the address returned by the Fleet API. From one CoreOS instance I could not ping the others due to a No Route to Host error (probably due to the netmask).
Within a Docker Container:
# ifconfig
eth0 Link encap:Ethernet HWaddr 36:12:7b:27:e6:e8
inet addr:172.17.0.2 Bcast:0.0.0.0 Mask:255.255.0.0 <----- (E)
...
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
...
Ping tests from with CoreOS:
Who can ping these addresses?
A - 169.254.156.94 - same CoreOS - Docker container - Destination Host Unreachable -
B - 128.199.190.70 - same CoreOS - Docker container - other CoreOS
C - 10.130.207.238 - same CoreOS - Docker container - other CoreOS
D - 172.17.42.1 - same CoreOS - Docker container - other CoreOS
E - 172.17.0.2 - not CoreOS - Destination Host Unreachable - not other CoreOS
So, what do all these IP addresses mean?
A - 169.254.156.94 - [eth0] some other IP, perhaps mapped via DO's load balancer to the public IP
B - 128.199.190.70 - official DO public IP
C - 10.130.207.238 - [eth1] official DO private IP
D - 172.17.42.1 - internal Docker
E - 172.17.0.2 - internal Docker
Result: cannot access the eth0 port from within the DO cloud, nor from outside. The IP addresses returned by Fleet are not usable, so we'll have to try /etcd and have each cadvisor register itself.
Update: The discovery URL can provide the private IP addresses of the machines.