The customer was performing upgrade on MX204 router and they noticed the BGP peering inside the Virtual Switch of the MX204 taking approximately 30 minutes to come up.
The MAC address for the BGP neighbors had been learnt when the customer collected the outputs:
root@D1HMDC-AG02> show arp no-resolve
MAC Address Address Interface Flags
24:fc:4e:bd:44:72 10.42.7.4 xe-0/1/7.10 none
24:fc:4e:bc:72:63 10.42.7.6 et-0/0/0.10 none
00:1a:f0:a1:f0:4e 10.42.7.19 xe-0/1/6.10 none
24:fc:4e:bd:44:72 10.42.8.130 xe-0/1/7.20 none
b4:0c:25:e7:80:10 10.48.226.1 ae0.2280 none
00:10:db:ff:40:02 10.48.240.17 ae0.2298 none
00:10:db:ff:60:03 10.48.240.33 irb.2297 [.local..5] permanent remote
24:fc:4e:bc:7a:74 10.48.240.34 irb.2297 [.local..5] permanent remote
00:10:db:ff:10:00 10.52.5.254 fxp0.0 none
00:10:db:ff:10:02 10.52.8.1 irb.2100 [.local..7] permanent remote
24:fc:4e:bc:7a:74 10.52.8.2 irb.2100 [.local..7] permanent remote
fe:54:00:2b:8a:6f 128.0.0.16 em3.0 none
fe:54:00:2b:8a:6f 128.0.0.101 em3.0 none
fe:54:00:7e:88:66 192.168.1.2 em2.32768 none
Total entries: 14
<<<<<<<< the traffic should be sent out of interface xe-0/1/1.2297 via L3 irb.2297.
routing-instances {
VS-2297 {
instance-type virtual-switch;
protocols {
evpn {
default-gateway no-gateway-community;
extended-vlan-list 2297;
}
bridge-domains {
BD-2297 {
domain-type bridge;
vlan-id 2297;
routing-interface irb.2297;
interface xe-0/1/1.2297;
route-distinguisher 65240:222297;
vrf-target target:65240:222297;
<<<<<< Based on the xe-0/1/1 interface, there is no dropped packets, nor errors.
The issue is NO received packets!
Physical interface: xe-0/1/1, Enabled, Physical link is Up
Interface index: 155, SNMP ifIndex: 560, Generation: 158
Input errors:
Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 0
Output errors:
Carrier transitions: 1, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0
Egress queues: 8 supported, 4 in use
Queue counters: Queued packets Transmitted packets Dropped packets
0 126 126 0
1 0 0 0
2 0 0 0
3 79 79 0
MAC statistics: Receive Transmit
Total octets 0 37434
Total packets 0 205
Unicast packets 0 5
Broadcast packets 0 126
Multicast packets 0 74
CRC/Align errors 0 0
FIFO errors 0 0
MAC control frames 0 0
MAC pause frames 0 0
Oversized frames 0
Jabber frames 0
Fragment frames 0
VLAN tagged frames 0
Code violations 0
Total errors 0 0
Filter statistics:
Input packet count 0
Input packet rejects 0
Input DA rejects 0
Input SA rejects 0
Output packet count 205
Output packet pad count 0
Output packet error count 0
CAM destination filters: 0, CAM source filters: 0
Before upgrade ---- Looked fine
========================
Total octets 1472097702500 590724080604
Total packets 1039420137 521444818
Unicast packets 1039418492 520578860
Broadcast packets 1645 16533
Multicast packets 0 849425
VLAN tagged frames 1039420137
The issue was related to underlayer L2 problem, for example, some L2 issues caused the packets dropping in the middle. JTAC can only see the output/transmit packets/Octets, not see any receive/incoming packets/octets in interface xe-0/1/1 in the RSI file after upgrade. This is abnormal and some thing wrong at L2/L1 layer.