ISSUE: Server reachability issues between old and new datacenter
ISSUE TRIGGER: Bringing up new datacenter
ISSUE IMPACT1: Communication between servers in old and new datacenter are affected
APSTRA/MIST managed?: No
DEVICE:
Old DC Spines
Model: QFX10K2-36q
Code: 20.2R3-S3.6 (EOL & EOS)
New DC Spines
Model: QFX5120-32c
Code: 22.2R3-S5.4
TOPOLOGY:
Nutanix HCI Servers < --- > | NEW DC| Collapsed Spines QFX5120-32c 1/2 |NEW DC| < --- DCI --- > |OLD DC1| Collapsed Spines QFX5120-32c 1/2 |OLD DC1| (Nutanix HCI servers connected to OLD DC1) < --- > |OLD DC2| Collapsed Spines QFX5700 1/2 |OLD DC2| < --- > Nutanix HCI Servers
TROUBLESHOOTING SUMMARY:
1. Customer is running EVPN/VXLAN with a collapsed core design in their new datacenter (pair of QFX5120's) along with old DC1 and DC2 (pair of QFX10K2's in each DC). New DC is connected to old DC1 and DC2 with DCI connection and in turn OLD DC1 is connected to OLD DC2 via DCI connection using both type 2 and type 5. L3 gateway irb's are stretched among these DC's.
2. Customer reported server reachability issues between new datacenter and servers behind old DC1 and DC2.
3. We reviewed the irb and VXLAN configurations and routes and confirmed the reachability between irb's and reachability from irb to the servers is working as expected.
4. We tried blocking the type 5 host route advertisement for one of the affected host route to see if the reachability works and it did not help.
5. We switched troubleshooting from windows server to linux server to use the flood ping option and narrow down packet drops in the switch. We did not see any packet drops in OLD DC1 QFX5120's and noticed packet drop increments in OLD DC1 QFX10K2 spine1 matching the initiated traffic.
Jsample on the switch confirmed the packets were matching the initiated traffic.
FPC0(oldDC1-jnpr-qfx1 vty)# show pechip 2 trapstats
DEV TRAPCODE count rate
=================================================================================
2 ( 252) sw.nh.discard_sampling_trap 1085159169 568
6. We checked the next hop ID programming in the switch along with the route imports and exports.
7. Customer attached the configuration, logs and session outputs to the case for further JTAC review.
9. We further looked into the next hop programming in OLD DC1 QFX1 switch and confirmed that the next hop route for the non-working flow was pointing to trapcode 252 discard.
8. Based on the issue symptoms, we matched it to this exisiting PR1649841 and proposed the workaround to execute the following caommand: "set shim jnh sort-and-defer disable". and flap (disable then re-enable) the ae0 link between OLD DC1 QFX1 and QFX2. For permanent fix, upgrade the switch software code to latest recommended EVPN releases.
9. Customer performed the above workaround and the application reachability between the servers were restored.
Based on the issue symptoms, we matched it to this exisiting PRPR1649841 and proposed the workaround to execute the following caommand: "set shim jnh sort-and-defer disable". and flap (disable then re-enable) the ae0 link between CAR QFX1 and QFX2. For permanent fix, upgrade the switch software code to latest recommended EVPN releases.
Customer performed the above workaround and the application reachability between the servers were restored.