This article explains the scenario and behavior for Fabric-Controller when one of the Director-device is rebooted. Fabric-Controllers are the Route-Reflectors in Qfabric devices, which has internal IBGP peering with all the other VMs and Node-Groups in a Qfabric System.
Consider this scenario:
DG1 is primary and DG0 is backup for director-group cluster. FC-1 is running on DG-1 and FC-0 is running on DG-0 respectively.
[root@dg1 ~]# lsvm NODE ACTIVE TAG UUID
dg0 1 _DCF_default___RR-INE-0_RE0_ 0bbf7b2a-5859-11e6-bad6-f3a9e63c1a6a dg1 1 _DCF_default___RR-INE-1_RE0_ 0ce190ec-5859-11e6-9773-a72e24c42352
All tests are done on version 14.1D17.1
1-Scenario-1 is in expected condition.
DG1 (primary) is rebooted, FC-1 is shifted/respawned on DG0 (Expected behaviour).
Before reboot Both FC-0 and FC-1 has BGP session established to all node-groups and between RRs too:
root@FC-0> show bgp summary fabric | grep "est|idl|act|con" Unconfigured peers: 6 Table Tot Paths Act Paths Suppressed History Damp State Pending Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped... 128.0.128.4 100 32433 32033 0 6 1w3d 4:13:54 Establ 128.0.128.8 100 119040 119045 0 0 5w2d 13:10:00 Establ 128.0.130.0 100 1768 1731 0 2 13:09:10 Establ 128.0.130.6 100 119657 119339 0 0 5w2d 13:09:58 Establ 128.0.130.8 100 119660 119340 0 0 5w2d 13:09:58 Establ 128.0.130.10 100 90385 89199 0 0 4w0d 8:43:35 Establ 128.0.130.18 100 90625 89472 0 0 4w0d 10:28:19 Establ
root@FC-1> show bgp summary fabric | no-more | grep "Est|Act|conn|idl" Table Tot Paths Act Paths Suppressed History Damp State Pending Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped... 128.0.128.4 100 32428 32009 0 6 1w3d 4:05:09 Establ 128.0.128.6 100 119045 119033 0 1 5w2d 13:01:20 Establ 128.0.130.0 100 1763 1729 0 2 13:06:17 Establ 128.0.130.6 100 119667 119181 0 0 5w2d 13:07:57 Establ 128.0.130.8 100 119670 119181 0 0 5w2d 13:07:57 Establ 128.0.130.10 100 90379 89182 0 0 4w0d 8:34:50 Establ 128.0.130.18 100 90619 89461 0 0 4w0d 10:19:34 Establ
During DG1 reboot:
FC-1 VM shuts down too and respawned on DG0. Now both FC VMs are on DG0:
[root@dg0 ~]# lsvm NODE ACTIVE TAG UUID dg0 1 _DCF_default___NW-INE-0_RE1_ 0a7b58ec-5859-11e6-aa3b-03617c167afc dg0 1 _DCF_default___RR-INE-0_RE0_ 0bbf7b2a-5859-11e6-bad6-f3a9e63c1a6a dg0 1 _TAG_DCF_ROOT_RE0_ c5816470-5858-11e6-a26a-ef094445ab07 dg1 1 _DCF_default___RR-INE-1_RE0_ 0ce190ec-5859-11e6-9773-a72e24c42352 ? dg1 1 _TAG_DRE_ c9fe7f38-5858-11e6-8ed1-470321454c61 dg1 1 _DCF_default___NW-INE-0_RE0_ 09474094-5859-11e6-b704-573a3d7bca43 dg1 1 _TAG_DCF_ROOT_RE1_ c58669e8-5858-11e6-a653-3792a1592d6a [root@dg0 ~]# lsvm NODE ACTIVE TAG UUID dg0 1 _DCF_default___NW-INE-0_RE1_ 0a7b58ec-5859-11e6-aa3b-03617c167afc dg0 1 _DCF_default___RR-INE-0_RE0_ 0bbf7b2a-5859-11e6-bad6-f3a9e63c1a6a dg0 1 _TAG_DCF_ROOT_RE0_ c5816470-5858-11e6-a26a-ef094445ab07 --- 0 _DCF_default___RR-INE-1_RE0_ 0ce190ec-5859-11e6-9773-a72e24c42352 ? --- 0 _TAG_DRE_ c9fe7f38-5858-11e6-8ed1-470321454c61 --- 0 _DCF_default___NW-INE-0_RE0_ 09474094-5859-11e6-b704-573a3d7bca43 --- 0 _TAG_DCF_ROOT_RE1_ c58669e8-5858-11e6-a653-3792a1592d6a [root@dg0 ~]# date Fri Jan 27 18:20:13 UTC 2017 [root@dg0 ~]# lsvm NODE ACTIVE TAG UUID dg0 1 _DCF_default___NW-INE-0_RE1_ 0a7b58ec-5859-11e6-aa3b-03617c167afc dg0 1 _DCF_default___RR-INE-0_RE0_ 0bbf7b2a-5859-11e6-bad6-f3a9e63c1a6a dg0 1 _TAG_DCF_ROOT_RE0_ c5816470-5858-11e6-a26a-ef094445ab07 dg0 1 _DCF_default___RR-INE-1_RE0_ 0ce190ec-5859-11e6-9773-a72e24c42352 ? dg0 1 _TAG_DRE_ c9fe7f38-5858-11e6-8ed1-470321454c61 dg0 1 _DCF_default___NW-INE-0_RE0_ 09474094-5859-11e6-b704-573a3d7bca43 --- 0 _TAG_DCF_ROOT_RE1_ c58669e8-5858-11e6-a653-3792a1592d6a
As FC-1 VM shuts down, FC-0 lost BGP to peer RR FC-1:
128.0.128.4 100 13 18 0 7 3:52 Establ 128.0.128.8 100 119210 119215 0 1 4:19 Connect <<<<< 128.0.130.0 100 1931 1920 0 2 14:22:23 Establ 128.0.130.6 100 119819 119527 0 0 5w2d 14:23:11 Establ 128.0.130.8 100 119822 119527 0 0 5w2d 14:23:11 Establ
As FC-1 VM respawned on DG0 now, FC-1 established BGP session to FC-0:
---(refreshed at 2017-01-27 18:26:52 UTC)--- Unconfigured peers: 4 Table Tot Paths Act Paths Suppressed History Damp State Pending Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped... 128.0.128.4 100 14 20 0 7 4:08 Establ 128.0.128.8 100 4 17 0 1 1 Establ <<<<< 128.0.130.0 100 1932 1921 0 2 14:22:39 Establ 128.0.130.6 100 119820 119528 0 0 5w2d 14:23:27 Establ 128.0.130.8 100 119822 119529 0 0 5w2d 14:23:27 Establ
Now both FC-0 and FC-1 are residing on DG0 and have all BGP sessions established:
NODE ACTIVE TAG UUID dg0 1 _DCF_default___NW-INE-0_RE1_ 0a7b58ec-5859-11e6-aa3b-03617c167afc dg0 1 _DCF_default___RR-INE-0_RE0_ 0bbf7b2a-5859-11e6-bad6-f3a9e63c1a6a <<<< dg0 1 _TAG_DCF_ROOT_RE0_ c5816470-5858-11e6-a26a-ef094445ab07 dg0 1 _DCF_default___RR-INE-1_RE0_ 0ce190ec-5859-11e6-9773-a72e24c42352 <<<< dg0 1 _TAG_DRE_ c9fe7f38-5858-11e6-8ed1-470321454c61 dg0 1 _DCF_default___NW-INE-0_RE0_ 09474094-5859-11e6-b704-573a3d7bca43 --- 0 _TAG_DCF_ROOT_RE1_ c58669e8-5858-11e6-a653-3792a1592d6a
Post DG1 reboot:
To load balance VMs, FC-0 and NNG RE0 VMs are released by DG-0(current primary) and spawned on DG-1(current Backup)
If you notice VMs that are released by DG-0, NNG RE VM is the newly spawned one whereas FC VM is the native one (FC with longest uptime) but not the newly spawned one:
Fri Jan 27 18:27:05 UTC 2017 [root@dg0 ~]# lsvm NODE ACTIVE TAG UUID dg0 1 _DCF_default___NW-INE-0_RE1_ 0a7b58ec-5859-11e6-aa3b-03617c167afc dg0 1 _DCF_default___RR-INE-0_RE0_ 0bbf7b2a-5859-11e6-bad6-f3a9e63c1a6a dg0 1 _TAG_DCF_ROOT_RE0_ c5816470-5858-11e6-a26a-ef094445ab07 dg0 1 _DCF_default___RR-INE-1_RE0_ 0ce190ec-5859-11e6-9773-a72e24c42352 dg0 1 _TAG_DRE_ c9fe7f38-5858-11e6-8ed1-470321454c61 dg0 1 _DCF_default___NW-INE-0_RE0_ 09474094-5859-11e6-b704-573a3d7bca43 --- 0 _TAG_DCF_ROOT_RE1_ c58669e8-5858-11e6-a653-3792a1592d6a [root@dg0 ~]# lsvm NODE ACTIVE TAG UUID dg0 1 _DCF_default___NW-INE-0_RE1_ 0a7b58ec-5859-11e6-aa3b-03617c167afc --- 0 _DCF_default___RR-INE-0_RE0_ 0bbf7b2a-5859-11e6-bad6-f3a9e63c1a6a dg0 1 _TAG_DCF_ROOT_RE0_ c5816470-5858-11e6-a26a-ef094445ab07 dg0 1 _DCF_default___RR-INE-1_RE0_ 0ce190ec-5859-11e6-9773-a72e24c42352 dg0 1 _TAG_DRE_ c9fe7f38-5858-11e6-8ed1-470321454c61 --- 0 _DCF_default___NW-INE-0_RE0_ 09474094-5859-11e6-b704-573a3d7bca43 dg1 1 _TAG_DCF_ROOT_RE1_ c58669e8-5858-11e6-a653-3792a1592d6a
As a result FC-0 shuts down and BGP peering between FC-0 and FC-1 went down:
FC-1: ---(refreshed at 2017-01-27 18:30:06 UTC)--- Unconfigured peers: 4 Table Tot Paths Act Paths Suppressed History Damp State Pending Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped... 128.0.128.4 100 26 17 0 0 5:45 Establ 128.0.128.6 100 30 15 0 1 2:44 Connect 128.0.130.0 100 24 19 0 0 5:47 Establ 128.0.130.6 100 17 17 0 0 5:46 Establ 128.0.130.8 100 17 17 0 0 5:46 Establ Unconfigured peers: 4 Table Tot Paths Act Paths Suppressed History Damp State Pending Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped... 128.0.128.4 100 26 18 0 0 5:47 Establ 128.0.128.6 100 5 6 0 1 2 Establ 128.0.130.0 100 24 20 0 0 5:49 Establ 128.0.130.6 100 18 18 0 0 5:48 Establ 128.0.130.8 100 17 18 0 0 5:48 Establ 4. Final State:
Only FC VMs UUID getting swapped between DGs prior and post reboot of the actual primary DG (DG-1) whereas NNG VMs UUID remained on actual DG:
[root@dg0 ~]# lsvm NODE ACTIVE TAG UUID dg0 1 _DCF_default___NW-INE-0_RE1_ 0a7b58ec-5859-11e6-aa3b-03617c167afc dg1 1 _DCF_default___RR-INE-0_RE0_ 0bbf7b2a-5859-11e6-bad6-f3a9e63c1a6a << dg0 1 _TAG_DCF_ROOT_RE0_ c5816470-5858-11e6-a26a-ef094445ab07 dg0 1 _DCF_default___RR-INE-1_RE0_ 0ce190ec-5859-11e6-9773-a72e24c42352 << dg0 1 _TAG_DRE_ c9fe7f38-5858-11e6-8ed1-470321454c61 dg1 1 _DCF_default___NW-INE-0_RE0_ 09474094-5859-11e6-b704-573a3d7bca43 dg1 1 _TAG_DCF_ROOT_RE1_ c58669e8-5858-11e6-a653-3792a1592d6a root@FC-1> show bgp summary fabric | match "est|act|idl|con" | refresh 2 Jan 27 19:11:04 ---(refreshed at 2017-01-27 19:11:04 UTC)--- Unconfigured peers: 4 Table Tot Paths Act Paths Suppressed History Damp State Pending Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped... 128.0.128.4 100 117 113 0 0 46:42 Establ 128.0.128.6 100 113 107 0 1 40:57 Establ 128.0.130.0 100 114 117 0 0 46:44 Establ 128.0.130.6 100 107 120 0 0 46:43 Establ 128.0.130.8 100 107 120 0 0 46:43 Establ root@FC-0> show bgp summary fabric | match "est|act|ild|con" Unconfigured peers: 4 Table Tot Paths Act Paths Suppressed History Damp State Pending Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped... 128.0.128.4 100 104 100 0 0 41:22 Establ 128.0.128.8 100 111 110 0 0 41:24 Establ 128.0.130.0 100 103 104 0 0 41:23 Establ 128.0.130.6 100 95 107 0 0 41:22 Establ 128.0.130.8 100 96 106 0 0 41:22 Establ
Both FCs show change in uptime during recent VM spawned time:
root@FC-0> show system uptime Current time: 2017-01-27 19:21:50 UTC System booted: 2017-01-27 18:28:45 UTC (00:53:05 ago) Protocols started: 2017-01-27 18:29:44 UTC (00:52:06 ago) Last configured: 2017-01-27 18:29:49 UTC (00:52:01 ago) by root 7:21PM up 53 mins, 1 user, load averages: 0.00, 0.00, 0.00 root@FC-1> show system uptime Jan 27 19:22:06 Current time: 2017-01-27 19:22:06 UTC System booted: 2017-01-27 18:22:14 UTC (00:59:52 ago) Protocols started: 2017-01-27 18:23:13 UTC (00:58:53 ago) Last configured: 2017-01-27 18:23:16 UTC (00:58:50 ago) by root 7:22PM up 1 hr, 1 user, load averages: 0.00, 0.01, 0.00
This can cause disruption in traffic at times.
The reason behind this is that the existing logic of selection and construction of a list of only lesser priority VMs to powerdown and activate on peer DG.
This will be fixed in version 14.1X53-D43 and higher.