Description

EVPN/VxLAN Composite Next-hop entries sometimes are not installed in PFE after reboot.

 

Symptoms

{master:0}
labroot@jtac-qfx5110-48s-4c-r2104> show evpn database 
Jun 02 10:11:01
Instance: default-switch
VLAN  DomainId  MAC address        Active source                  Timestamp        IP address
     1101       00:46:00:00:00:01  192.168.1.5                    Jun 02 09:20:03
     1101       00:47:00:00:00:02  xe-0/0/0.0                     Jun 02 08:41:04

{master:0}
labroot@jtac-qfx5110-48s-4c-r2104> show route forwarding-table destination 00:46:00:00:00:01        
Jun 02 10:11:09
Routing table: default-switch.evpn-vxlan
Bridging domain: T0001101R.evpn-vxlan
VPLS:
Enabled protocols: Bridging, ACKed by all peers, EVPN VXLAN, 
Destination        Type RtRef Next hop           Type Index    NhRef Netif
00:46:00:00:00:01/48 user     0                  comp     1772   159    <<<<<<< Destination MAC

{master:0}
labroot@jtac-qfx5110-48s-4c-r2104> request pfe execute target fpc0 command "show nh id 1772 rec"    
Jun 02 10:11:16
SENT: Ukern command: show nh id 1772 rec

No nexthop for ID 1772 could be found    <<<<<<<< No nexthop information


{master:0}
labroot@jtac-qfx5110-48s-4c-r2104> show log messages| grep 1772                                     
Jun 02 10:11:32
Jun  1 06:42:18  jtac-qfx5110-48s-4c-r2104 kernel: rnh_comp_request,4396: op 2, rnh 1772 transition from Public->Private unallowed. comp function=14 subtype=11
Jun  2 06:59:52  jtac-qfx5110-48s-4c-r2104 kernel: rnh_comp_request,4396: op 2, rnh 1772 transition from Public->Private unallowed. comp function=14 subtype=11
Jun  2 08:31:11  jtac-qfx5110-48s-4c-r2104 kernel: rnh_comp_request,4396: op 2, rnh 1772 transition from Public->Private unallowed. comp function=14 subtype=11
Jun  2 08:41:21  jtac-qfx5110-48s-4c-r2104 kernel: rnh_comp_request,4396: op 2, rnh 1772 transition from Private->Public unallowed. comp function=14 subtype=11

{master:0}
labroot@jtac-qfx5110-48s-4c-r2104> 
{master:0}
labroot@jtac-qfx5110-48s-4c-r2104> start shell 
Jun 02 10:13:17
% nhinfo -di 1772
Nexthop is: private
Nexthop subtype is: vxlan
full length of message in bytes: 276, version: 99
message type: NH-comp
message id: unknown
message op: get
message sequence number: 2
next hop type: [0x18] "comp"
next hop index: 1772
logical system id: 0
reference count: 159
next hop flags: [0x4] free
next hop kernel-flags: [0x11]
next hop user flags: [0x20]
Rtbl id: 7    next hop interface AF: vpls
next hop AF: vpls
hardware token: 0000000000
NH-comp list, num-NHs 1
NHs in list: 324,
Composition function id 14 (Vxlan Encap NH with forwarding NH being Unicast) 
Opaque data len 48 bytes:
Hexdump                                                 ASCII           
-------                                                 -----           
00 01 00 04 00 00 00 00 00 02 00 04 c0 a8 01 07         ................
00 03 00 04 c0 a8 01 05 00 04 00 04 00 00 00 00         ................
00 05 00 04 00 00 03 39 00 06 00 04 00 00 00 9a         .......9........
ipv4 svtep:0xc0a80107    rvtep:0xc0a80105
l3 rtt index:0 dest UDP port:0
svtep ifl index:815 rvtep ifl index:825
%


% less messages | egrep "Jun  2 08:4" | egrep -B10 -A10 1772
Jun  2 08:41:13  jtac-qfx5110-48s-4c-r2104 kernel: rnh_comp_request,4396: op 2, rnh 1766 transition from Public->Private unallowed. comp function=1 subtype=1
Jun  2 08:41:13  jtac-qfx5110-48s-4c-r2104 kernel: rnh_comp_request,4396: op 2, rnh 1767 transition from Public->Private unallowed. comp function=1 subtype=1
Jun  2 08:41:13  jtac-qfx5110-48s-4c-r2104 kernel: rnh_comp_request,4396: op 2, rnh 1768 transition from Public->Private unallowed. comp function=1 subtype=1
Jun  2 08:41:13  jtac-qfx5110-48s-4c-r2104 kernel: rnh_comp_request,4396: op 2, rnh 1769 transition from Public->Private unallowed. comp function=1 subtype=1
Jun  2 08:41:14  jtac-qfx5110-48s-4c-r2104 l2ald[9549]: L2ALD_ST_CTL_IN_EFFECT: xe-0/0/0.0: storm control in effect on the port
Jun  2 08:41:15  jtac-qfx5110-48s-4c-r2104 bfdd[9530]: BFDD_TRAP_MHOP_STATE_UP: local discriminator: 17, new state: up, peer addr: 192.168.1.2
Jun  2 08:41:16  jtac-qfx5110-48s-4c-r2104 l2ald[9549]: L2ALD_ST_CTL_IN_EFFECT: xe-0/0/0.0: storm control in effect on the port
Jun  2 08:41:16  jtac-qfx5110-48s-4c-r2104 bfdd[9530]: BFDD_TRAP_MHOP_STATE_UP: local discriminator: 16, new state: up, peer addr: 192.168.1.1
Jun  2 08:41:18  jtac-qfx5110-48s-4c-r2104 l2ald[9549]: L2ALD_ST_CTL_IN_EFFECT: xe-0/0/0.0: storm control in effect on the port
Jun  2 08:41:20  jtac-qfx5110-48s-4c-r2104 l2ald[9549]: L2ALD_ST_CTL_IN_EFFECT: xe-0/0/0.0: storm control in effect on the port
Jun  2 08:41:21  jtac-qfx5110-48s-4c-r2104 kernel: rnh_comp_request,4396: op 2, rnh 1772 transition from Private->Public unallowed. comp function=14 subtype=11
Jun  2 08:41:22  jtac-qfx5110-48s-4c-r2104 l2ald[9549]: L2ALD_ST_CTL_IN_EFFECT: xe-0/0/0.0: storm control in effect on the port
Jun  2 08:41:30  jtac-qfx5110-48s-4c-r2104 last message repeated 4 times
Jun  2 08:41:31  jtac-qfx5110-48s-4c-r2104 rpd[9519]: RPD_RT_HWM_NOTICE: New RIB highwatermark for unique destinations: 72 [2023-06-02 08:41:21]
Jun  2 08:41:31  jtac-qfx5110-48s-4c-r2104 rpd[9519]: RPD_RT_HWM_NOTICE: New RIB highwatermark for routes: 96 [2023-06-02 08:41:25]
Jun  2 08:41:32  jtac-qfx5110-48s-4c-r2104 l2ald[9549]: L2ALD_ST_CTL_IN_EFFECT: xe-0/0/0.0: storm control in effect on the port
Jun  2 08:41:38  jtac-qfx5110-48s-4c-r2104 last message repeated 3 times
Jun  2 08:41:39  jtac-qfx5110-48s-4c-r2104 rpd[9519]: RPD_RT_HWM_NOTICE: New FIB highwatermark for routes: 12 [2023-06-02 08:41:21]
Jun  2 08:41:40  jtac-qfx5110-48s-4c-r2104 l2ald[9549]: L2ALD_ST_CTL_IN_EFFECT: xe-0/0/0.0: storm control in effect on the port
Jun  2 08:41:44  jtac-qfx5110-48s-4c-r2104 last message repeated 2 times
Jun  2 08:41:44  jtac-qfx5110-48s-4c-r2104 vhclient.11309.daemon[11331]: Error: /dev/sdb: unrecognised disk label :error[0]
%  

Solution

Upgrade to one of the fixed versions listed below of PR1739686.

evo:21.3R3-S5-EVO
evo:21.4R3-S6-EVO
evo:22.1R3-S4-EVO
evo:22.2R3-S2-EVO
evo:22.3R3-S1-EVO
evo:22.4R3-EVO
evo:23.1R2-EVO
evo:23.2R1-S2-EVO
evo:23.2R2-EVO
evo:23.3R1-EVO
evo:23.4R1-EVO
junos:20.4R3-S9
junos:21.3R3-S5
junos:21.4R3-S4-J17
junos:21.4R3-S6
junos:22.1R3-S4
junos:22.2R3-S2
junos:22.3R3-S1
junos:22.4R3
junos:23.1R2
junos:23.2R1-S2
junos:23.2R2
junos:23.3R1
junos:23.4R1

External link to the PR itself:

https://prsearch.juniper.net/problemreport/PR1739686

Modification History

Publish

Related Information

Root cause (as per AT#80 of PR1739686):
1. RPD Sending stale Route Information about INH to vtep-ip, followed immediately by correct INH
2. L2ald creates VENH->Private_NH
3. L2ald tried to upate it to Public_NH Change of Next Hops Disallowed by PR-1451968:
VENH->Private_NH -> VENH->Public_NH
4. Traffic to VTEP-IP Dropped