BUM Tree not remove compute vhost0 down - lujiang2016/contrail GitHub Wiki
BUM Tree not remove vhost0 down compute VR discussion
The BUM tree is being built on 1 of the 3 control nodes there. It is then being downloaded into the VR of a compute.
I did a quick test to down vhost0 on comp46 in my lab testbed. All the XMPP connections will be down and vr-agent be initializing. As the XMPP connection all down from comp46, then the control node will remove the broadcast entry from comp46 (172.18.41.46) out of the ermvpn.0, hence will remove comp46 out of the multicast BUM tree.
root@cont41:~# ist ctr route show -t default-domain:jlu:vn-left:vn-left.ermvpn.0
default-domain:jlu:vn-left:vn-left.ermvpn.0: 6 destinations, 6 routes (5 primary, 1 secondary, 0 infeasible)
…
0-172.18.41.46:3-0.0.0.0,255.255.255.255,0.0.0.0, age: 18:21:16.363157, last_modified: 2018-Jul-24 00:09:06.947311
[XMPP|comp46] age: 18:21:16.366089, localpref: 100, nh: 172.18.41.46, encap: ['gre', 'udp'], label: 0, AS path: None
Thought the problem in the field RDM3 testbed is some kind design issue. As I remember the vr-agent is still active at the time before restart the vr-agent on the compute 172.29.5.151 there? Thought there is chance the XMPP connections bw compute 172.29.5.151 and control nodes could be not above vhost0 and bond0 there? But be above br-mesh0 or some other NICs?
root@comp46:~# contrail-status -d
== Contrail vRouter ==
supervisor-vrouter: active
contrail-vrouter-agent active pid 2179, uptime 2 days, 22:45:17
contrail-vrouter-nodemgr active pid 2178, uptime 2 days, 22:45:17
root@comp46:~# ist vr status -d
module_id: contrail-vrouter-agent
state: Functional
description
Connetion Info:
+-----------+---------------------------+---------------------+--------+-------------------+
| type | name | server_addrs | status | description |
+-----------+---------------------------+---------------------+--------+-------------------+
| XMPP | control-node:172.18.41.41 | 172.18.41.41:5269 | Up | OpenSent |
| XMPP | control-node:172.18.41.42 | 172.18.41.42:5269 | Up | OpenSent |
| XMPP | dns-server:172.18.41.42 | 172.18.41.42:53 | Up | OpenSent |
| XMPP | dns-server:172.18.41.43 | 172.18.41.43:53 | Up | OpenSent |
| Collector | n/a | 172.18.41.43:8086 | Up | Established |
| Discovery | Collector | 172.18.41.40:5998 | Up | SubscribeResponse |
| Discovery | dns-server | 172.18.41.40:5998 | Up | SubscribeResponse |
| Discovery | xmpp-server | 172.18.41.40:5998 | Up | SubscribeResponse |
+-----------+---------------------------+---------------------+--------+-------------------+
root@comp46:~# ifconfig vhost0
vhost0 Link encap:Ethernet HWaddr 00:50:56:9e:35:f2
inet addr:172.18.41.46 Bcast:172.18.41.255 Mask:255.255.255.0
inet6 addr: fe80::250:56ff:fe9e:35f2/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2640856 errors:0 dropped:0 overruns:0 frame:0
TX packets:1446630 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:397051489 (397.0 MB) TX bytes:580825860 (580.8 MB)
root@comp46:~# ifconfig vhost0 down
root@comp46:~# ifconfig vhost0
vhost0 Link encap:Ethernet HWaddr 00:50:56:9e:35:f2
inet addr:172.18.41.46 Bcast:172.18.41.255 Mask:255.255.255.0
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:2640936 errors:0 dropped:26 overruns:0 frame:0
TX packets:1446654 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:397056493 (397.0 MB) TX bytes:580849592 (580.8 MB)
7: vhost0: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether 00:50:56:9e:35:f2 brd ff:ff:ff:ff:ff:ff
inet 172.18.41.46/24 brd 172.18.41.255 scope global vhost0
valid_lft forever preferred_lft forever
root@comp46:~# ping 172.18.41.41
PING 172.18.41.41 (172.18.41.41) 56(84) bytes of data.
^C
--- 172.18.41.41 ping statistics ---
6 packets transmitted, 0 received, 100% packet loss, time 5040ms
root@comp46:~# ist vr status -d
module_id: contrail-vrouter-agent
state: Functional
description: Collector connection down
Connetion Info:
+-----------+---------------------------+---------------------+--------------+-------------------------------------+
| type | name | server_addrs | status | description |
+-----------+---------------------------+---------------------+--------------+-------------------------------------+
| XMPP | control-node:172.18.41.41 | 172.18.41.41:5269 | Up | OpenSent |
| XMPP | control-node:172.18.41.42 | 172.18.41.42:5269 | Up | OpenSent |
| XMPP | dns-server:172.18.41.42 | 172.18.41.42:53 | Up | OpenSent |
| XMPP | dns-server:172.18.41.43 | 172.18.41.43:53 | Up | OpenSent |
| Collector | n/a | 172.18.41.43:8086 | Initializing | Established : EvTcpClose -> Connect |
| Discovery | Collector | 172.18.41.40:5998 | Up | SubscribeResponse |
| Discovery | dns-server | 172.18.41.40:5998 | Up | SubscribeResponse |
| Discovery | xmpp-server | 172.18.41.40:5998 | Up | SubscribeResponse |
+-----------+---------------------------+---------------------+--------------+-------------------------------------+
root@comp46:~# contrail-status -d
== Contrail vRouter ==
supervisor-vrouter: active
contrail-vrouter-agent active (Collector connection down)pid 2179, uptime 2 days, 22:46:44
contrail-vrouter-nodemgr active pid 2178, uptime 2 days, 22:46:44
root@comp46:~# contrail-status -d
== Contrail vRouter ==
supervisor-vrouter: active
contrail-vrouter-agent initializing (XMPP:control-node:172.18.41.41, XMPP:control-node:172.18.41.42, XMPP:dns-server:172.18.41.42, XMPP:dns-server:172.18.41.43, Collector, Discovery:dns-server, Discovery:xmpp-server connection down)pid 2179, uptime 2 days, 22:48:34
contrail-vrouter-nodemgr active pid 2178, uptime 2 days, 22:48:34
root@comp46:~# ist vr status -d
module_id: contrail-vrouter-agent
state: Non-Functional
description: XMPP:control-node:172.18.41.41, XMPP:control-node:172.18.41.42, XMPP:dns-server:172.18.41.42, XMPP:dns-server:172.18.41.43, Collector, Discovery:Collector, Discovery:dns-server, Discovery:xmpp-server connection down
Connetion Info:
+-----------+---------------------------+---------------------+--------+---------------------------------+
| type | name | server_addrs | status | description |
+-----------+---------------------------+---------------------+--------+---------------------------------+
| XMPP | control-node:172.18.41.41 | 172.18.41.41:5269 | Down | Connect |
| XMPP | control-node:172.18.41.42 | 172.18.41.42:5269 | Down | Connect |
| XMPP | dns-server:172.18.41.42 | 172.18.41.42:53 | Down | Connect |
| XMPP | dns-server:172.18.41.43 | 172.18.41.43:53 | Down | Connect |
| Collector | n/a | 0.0.0.0:0 | Down | Connect : EvConnectTimerExpired |
| Discovery | Collector | 172.18.41.40:5998 | Down | Timeout was reached |
| Discovery | dns-server | 172.18.41.40:5998 | Down | Timeout was reached |
| Discovery | xmpp-server | 172.18.41.40:5998 | Down | Timeout was reached |
+-----------+---------------------------+---------------------+--------+---------------------------------+
root@comp46:~# contrail-status -d
== Contrail vRouter ==
supervisor-vrouter: active
contrail-vrouter-agent initializing (XMPP:control-node:172.18.41.41, XMPP:control-node:172.18.41.42, XMPP:dns-server:172.18.41.42, XMPP:dns-server:172.18.41.43, Collector, Discovery:Collector, Discovery:dns-server, Discovery:xmpp-server connection down)pid 2179, uptime 2 days, 22:57:30
contrail-vrouter-nodemgr active pid 2178, uptime 2 days, 22:57:30
note comp46 being removed in vn-left.ermvpn.0, hence comp46 will be removed from the BUM tree
root@cont41:/var/log/contrail# ist ctr route show -t default-domain:jlu:vn-left:vn-left.ermvpn.0
default-domain:jlu:vn-left:vn-left.ermvpn.0: 5 destinations, 5 routes (4 primary, 1 secondary, 0 infeasible)
0-172.18.41.45:3-0.0.0.0,255.255.255.255,0.0.0.0, age: 3 days 3:09:19.948788, last_modified: 2018-Jul-23 19:17:23.719035
[XMPP|comp45] age: 3 days 3:09:19.951029, localpref: 100, nh: 172.18.41.45, encap: ['gre', 'udp'], label: 0, AS path: None
1-0:0-172.18.41.41,255.255.255.255,0.0.0.0, age: 0:08:41.154509, last_modified: 2018-Jul-26 22:18:02.513314
[Local|None] age: 0:08:41.157191, localpref: 100, nh: 172.18.41.41, encap: [], label: 0, AS path: None
1-0:0-172.18.41.42,255.255.255.255,0.0.0.0, age: 3 days 3:09:19.928689, last_modified: 2018-Jul-23 19:17:23.739134
[BGP|172.18.41.42] age: 3 days 3:09:19.931714, localpref: 100, nh: 0.0.0.0, encap: [], label: 0, AS path: None
2-0:0-172.18.41.41,255.255.255.255,0.0.0.0, age: 0:08:41.154232, last_modified: 2018-Jul-26 22:18:02.513591
[Local|None] age: 0:08:41.157671, localpref: 100, nh: 172.18.41.41, encap: [], label: 0, AS path: None
2-0:0-172.18.41.42,255.255.255.255,0.0.0.0, age: 0:08:41.154224, last_modified: 2018-Jul-26 22:18:02.513599
[Local|None] age: 0:08:41.158015, localpref: 100, nh: 172.18.41.41, encap: [], label: 0, AS path: None
root@cont41:/var/log/contrail# ist ctr mcast -t vn-left table
+---------------------------------------------+-------------+---------+------------+
| name | total_trees | deleted | deleted_at |
+---------------------------------------------+-------------+---------+------------+
| default-domain:jlu:vn-left:vn-left.ermvpn.0 | 1 | false | n/a |
+---------------------------------------------+-------------+---------+------------+
note comp46 removed out from the BUM tree. Now only comp44 and comp45 in the BUM tree
root@cont41:/var/log/contrail# ist ctr mcast -t default-domain:jlu:vn-left:vn-left.ermvpn.0 tree
+-----------------+---------+----------------------------+---------------------------------+
| group | source | level0_forwarders | level1_forwarders |
+-----------------+---------+----------------------------+---------------------------------+
| 255.255.255.255 | 0.0.0.0 | ShowMulticastForwarder | ShowMulticastForwarder |
| | | address: 172.18.41.45 | address: 172.18.41.45 |
| | | label_block: 4609-5119 | label_block: 4639-4639 |
| | | label: 4639 | label: 4639 |
| | | router_id: 0.0.0.0 | router_id: 172.18.41.41 |
| | | links | links |
| | | | ShowMulticastTreeLink |
| | | | address: 172.18.41.44 |
| | | | label: 4609 |
| | | | ShowMulticastForwarder |
| | | | address: 172.18.41.44 |
| | | | label_block: 4609-4609 |
| | | | label: 4609 |
| | | | router_id: 172.18.41.42 |
| | | | links |
| | | | ShowMulticastTreeLink |
| | | | address: 172.18.41.45 |
| | | | label: 4639 |
+-----------------+---------+----------------------------+---------------------------------+
Following capture is before the comp46 vhost0 down
root@cont41:~# ist ctr route show -t default-domain:jlu:vn-left:vn-left.ermvpn.0
default-domain:jlu:vn-left:vn-left.ermvpn.0: 6 destinations, 6 routes (5 primary, 1 secondary, 0 infeasible)
0-172.18.41.45:3-0.0.0.0,255.255.255.255,0.0.0.0, age: 23:12:59.591433, last_modified: 2018-Jul-23 19:17:23.719035
[XMPP|comp45] age: 23:12:59.593994, localpref: 100, nh: 172.18.41.45, encap: ['gre', 'udp'], label: 0, AS path: None
0-172.18.41.46:3-0.0.0.0,255.255.255.255,0.0.0.0, age: 18:21:16.363157, last_modified: 2018-Jul-24 00:09:06.947311
[XMPP|comp46] age: 18:21:16.366089, localpref: 100, nh: 172.18.41.46, encap: ['gre', 'udp'], label: 0, AS path: None
1-0:0-172.18.41.41,255.255.255.255,0.0.0.0, age: 18:21:16.363102, last_modified: 2018-Jul-24 00:09:06.947366
[Local|None] age: 18:21:16.366312, localpref: 100, nh: 172.18.41.41, encap: [], label: 0, AS path: None
1-0:0-172.18.41.42,255.255.255.255,0.0.0.0, age: 23:12:59.571334, last_modified: 2018-Jul-23 19:17:23.739134
[BGP|172.18.41.42] age: 23:12:59.574805, localpref: 100, nh: 0.0.0.0, encap: [], label: 0, AS path: None
2-0:0-172.18.41.41,255.255.255.255,0.0.0.0, age: 18:21:16.362984, last_modified: 2018-Jul-24 00:09:06.947484
[Local|None] age: 18:21:16.366713, localpref: 100, nh: 172.18.41.41, encap: [], label: 0, AS path: None
2-0:0-172.18.41.42,255.255.255.255,0.0.0.0, age: 18:21:16.362970, last_modified: 2018-Jul-24 00:09:06.947498
[Local|None] age: 18:21:16.366950, localpref: 100, nh: 172.18.41.41, encap: [], label: 0, AS path: None
root@cont41:~# ist ctr mcast -t default-domain:jlu:vn-left:vn-left.ermvpn.0 table
+---------------------------------------------+-------------+---------+------------+
| name | total_trees | deleted | deleted_at |
+---------------------------------------------+-------------+---------+------------+
| default-domain:jlu:vn-left:vn-left.ermvpn.0 | 1 | false | n/a |
+---------------------------------------------+-------------+---------+------------+
root@cont41:~# ist ctr mcast -t default-domain:jlu:vn-left:vn-left.ermvpn.0 tree
+-----------------+---------+---------------------------------+---------------------------------+
| group | source | level0_forwarders | level1_forwarders |
+-----------------+---------+---------------------------------+---------------------------------+
| 255.255.255.255 | 0.0.0.0 | ShowMulticastForwarder | ShowMulticastForwarder |
| | | address: 172.18.41.45 | address: 172.18.41.46 |
| | | label_block: 4609-5119 | label_block: 4101-4101 |
| | | label: 4633 | label: 4101 |
| | | router_id: 0.0.0.0 | router_id: 172.18.41.41 |
| | | links | links |
| | | ShowMulticastTreeLink | ShowMulticastTreeLink |
| | | address: 172.18.41.46 | address: 172.18.41.44 |
| | | label: 4101 | label: 4609 |
| | | ShowMulticastForwarder | ShowMulticastForwarder |
| | | address: 172.18.41.46 | address: 172.18.41.44 |
| | | label_block: 4098-4608 | label_block: 4609-4609 |
| | | label: 4101 | label: 4609 |
| | | router_id: 0.0.0.0 | router_id: 172.18.41.42 |
| | | links | links |
| | | ShowMulticastTreeLink | ShowMulticastTreeLink |
| | | address: 172.18.41.45 | address: 172.18.41.46 |
| | | label: 4633 | label: 4101 |
+-----------------+---------+---------------------------------+---------------------------------+