In EVPN VXLAN CRB topology with QFX10008 running 20.4R3-S3.4, some of the bfd sessions between the IRB interfaces of spine routers shows init/down state.
Issue is related to multi FPC and When BFD runs in distributed mode on QFX10008, one FPC is selected as anchor FPC. All the BFD packets received on other FPCs are given to anchor FPC for processing but on PFE hostpath , for BFD pkts over vtep , incoming port is derived from (cpu_number + stream_number) which is not valid in case of BFD anchoring.
user@qfx-10008-1> show bfd session extensive
Aug 27 10:02:49
Detect Transmit
Address State Interface Time Interval Multiplier
10.9.68.220 Init irb.992 12.000 2.000 6
Client BGP, TX interval 1.000, RX interval 1.000
Local diagnostic AdminDown, remote diagnostic None
Remote state Down, version 1
Replicated
Session type: Single hop BFD
Min async interval 1.000, min slow interval 2.000
Adaptive async TX interval 2.000, RX interval 2.000
Local min TX interval 2.000, minimum RX interval 1.000, multiplier 6
Remote min TX interval 1.000, min RX interval 1.000, multiplier 6
Local discriminator 68, remote discriminator 1089667074
Echo TX interval 2.000, echo detection interval 12.000
Echo mode disabled/inactiveno-absorb, no-refresh
Remote is control-plane independent <<<BFD running in Distributed Mode
Session ID: 0x1b8
53 sessions, 53 clients
Cumulative transmit rate 48.5 pps, cumulative receive rate 44.5 pps
BFD Config:
set routing-instances MIS-L3VPN protocols bgp group VENDOR-FIREWALLS bfd-liveness-detection minimum-interval 1000
set routing-instances MIS-L3VPN protocols bgp group ENDOR-FIREWALLS bfd-liveness-detection multiplier 6
IRB Config:
set interfaces irb unit 992 proxy-macip-advertisement
set interfaces irb unit 992 virtual-gateway-accept-data
set interfaces irb unit 992 family inet address 10.9.68.218/29 preferred
set interfaces irb unit 992 family inet address 10.9.68.218/29 virtual-gateway-address 10.9.68.217
set routing-instances MY-L3VPN interface irb.992
set vlans v992 description PAFW_10.9.68.216/29
set vlans v992 vlan-id 992
set vlans v992 l3-interface irb.992
set vlans v992 vxlan vni 2010992
Traceoption logs show QFX is sending BFD packets to 10.9.68.220:
user@qfx-10008-1> clear bfd session address 10.9.68.220
Aug 27 09:42:16 <<<<SESSION GETS CLEARED AND GOES FROM Init -> Down
Aug 27 09:42:16 Session 10.9.68.220 (IFL 700) state Init -> Down
Aug 27 09:42:16 Session 10.9.68.220 (IFL 700) flap count 1
Aug 27 09:42:16 (bfdd_update_tx_intervals:529) Session 10.9.68.220 (IFL 700): cur tx ivl 1000000, new_invl 0(1,0,1000000,1000000)
Aug 27 09:42:16 Session 10.9.68.220 (IFL 700) starting version negotiation
Aug 27 09:42:16 [SLOW_START]bfdd_slow_start_start: Session 10.9.68.220 <<<<BFD SESSION START
Aug 27 09:42:16 [SLOW_START]bfdd_slow_start_set: Session 10.9.68.220 pre intervals are mix rx = 1000000 adpt rx = 1000000 min tx = 1000000 adpt tx = 1000000
Aug 27 09:42:16 Sent Downstream SetAdj (11) len 144:
Aug 27 09:42:16 IfIndex (3) len 4: 700
Aug 27 09:42:16 SrcAddr (5) len 8: 10.9.68.220
Aug 27 09:42:16 HoldTime (14) len 8: 6 sec 0 nsec
Aug 27 09:42:16 NoAbsorb (15) len 1: True
Aug 27 09:42:16 NoRefresh (16) len 1: True
Aug 27 09:42:16 ForceRefresh (17) len 1: True
Aug 27 09:42:16 DoNotAge (18) len 1: True
Aug 27 09:42:16 Distribute (27) len 1: True
Aug 27 09:42:16 LooseAuth (122) len 1: (hex) 00
Aug 27 09:42:16 Discriminator (63) len 4: 0x44
Aug 27 09:42:16 DestAddr (8) len 8: 10.9.68.218
Aug 27 09:42:16 RtblIdx (24) len 4: 10
Aug 27 09:42:16 MinRecvTTL (68) len 1: 255
Aug 27 09:42:16 Echo (260) len 1: False
Aug 27 09:42:16 RecvOnMhopPort (101) len 1: 0
Aug 27 09:42:16 Unknown (153) len 1: (hex) 00
Aug 27 09:42:16 Unknown (154) len 4: (hex) 00 00 01 b8
Aug 27 09:42:16 Unknown (165) len 4: (hex) 00 00 00 06
Aug 27 09:42:16 Unknown (211) len 1: (hex) 00
Aug 27 09:42:16 Unknown (167) len 1: (hex) 01
Aug 27 09:42:16 rx flow: echo mode inactive for session 10.9.68.220
Aug 27 09:42:16 program_thread, final 0, poll 0
Aug 27 09:42:16 (bfdd_build_packet:2354) : Session 10.9.68.220 (IFL 700): cur tx ivl 1000000
Aug 27 09:42:16 BFD packet to 10.9.68.220 (IFL 700), len 24
++Firewall filter will show QFX is not sending BFD packets on irb.
user@qfx-10008-1# show | compare
[edit interfaces irb unit 992 family inet]
+ filter {
+ input BFD;
+ output BFD;
+ }
[edit firewall family inet]
+ filter BFD {
+ interface-specific;
+ term In {
+ from {
+ source-address {
+ 10.9.68.220/32;
+ destination-address {
+ 10.9.68.218/32;
+ protocol udp;
+ port [ 4784 3784 3785 ];
+ then {
+ count BFD-In;
+ log;
+ accept;
+ term Out {
+ term ANY {
+ then accept;
user@qfx-10008-1> show firewall filter BFD-irb.992-i | refresh 3
---(refreshed at 2025-10-01 08:00:36 UTC)---
Filter: BFD-irb.992-i
Counters:
Name Bytes Packets
BFD-In-irb.992-i 13832 266
---(refreshed at 2025-10-01 08:00:39 UTC)---
BFD-In-irb.992-i 14092 271
---(refreshed at 2025-10-01 08:00:42 UTC)---
BFD-In-irb.992-i 14248 274
---(refreshed at 2025-10-01 08:00:45 UTC)---
BFD-In-irb.992-i 14456 278
---(refreshed at 2025-10-01 08:00:48 UTC)---
user@qfx-10008-1> show firewall filter BFD-irb.992-o | refresh 3
---(refreshed at 2025-10-01 08:01:24 UTC)---
Filter: BFD-irb.992-o
BFD-In-irb.992-o 0 0
---(refreshed at 2025-10-01 08:01:27 UTC)---
---(refreshed at 2025-10-01 08:01:30 UTC)---
---(*more 100%)---[abort]
{master}
user@qfx-10008-1> show firewall filter BFD-irb.992-o
Hostpath is different than BFD anchor FPC. In this case. hostpath is through FPC1, below shows BFD anchor FPC is FPC0:
FPC0(qfx-10008-1 vty)# show filter
Program Filters:
---------------
Index Dir Cnt Text Bss Name
-------- ------ ------ ------ ------ --------
Term Filters:
------------
Index Semantic Name
-------- ----------------
...
54525952 Classic ppmd-bfd-inet4
Resolve Filters:
Index
--------
FPC0(qfx-10008-1 vty)# show filter index 54525952 program
....
action next-hop, type (nh-id)
2672 <<<<<<<check the nh-id
count ppmd-v4-47-4784-count
FPC1(qfx-10008-1 vty)# show nhdb id 2672 extensive
ID Type Interface Next Hop Addr Protocol Encap MTU Flags PFE internal Flags
----- -------- ------------- --------------- ---------- ------------ ---- ------------------ ------------------
2672 Unicast pfh-0/0/0.16384 128.0.0.16 IPv4 Unspecified 0 0x0000000000000000 0x0000000000000000 <<<<Anchor FPC is FPC0 which is different than the hostpath
There is a PR for this: https://prsearch.juniper.net/problemreport/PR1680757
https://prsearch.juniper.net/problemreport/PR1680757
set routing-options ppm no-delegate-processing