This article outlines the behavior of Bidirectional Forwarding Detection (BFD) when used in aggregated Ethernet (ae) bundles that have child links in different Flexible PIC Concentrator (FPC) cards.
In case of aggregated Ethernet or IRB interfaces, where the member links are hosted by different FPCs, one FPC will be chosen as the FPC that is responsible for sending and receiving BFD packets. A firewall filter rule will be programmed on the other FPCs to redirect packets to the responsible FPC on the "pfh" interface. The responsible FPC is not changed when the links from the respective FPC come out of the bundle. Consider a topology where R1 and R2 are connected back to back.
R1 has an ae bundle towards R2, and the ae bundle in R1 has child links from three FPCs:
[edit] lab@R1# show | match ae10 | display set Dec 02 08:30:44 set interfaces xe-0/2/0 gigether-options 802.3ad ae10 set interfaces xe-1/2/0 gigether-options 802.3ad ae10 set interfaces xe-3/2/0 gigether-options 802.3ad ae10 set interfaces ae10 aggregated-ether-options lacp active set interfaces ae10 unit 0 family inet address 172.19.1.1/24 set protocols ospf area 0.0.0.0 interface ae10.0 bfd-liveness-detection minimum-interval 1000 set protocols ospf area 0.0.0.0 interface ae10.0 bfd-liveness-detection multiplier 5 [edit] lab@R1# run show interfaces ae10.0 extensive Dec 02 08:30:50 Logical interface ae10.0 (Index 323) (SNMP ifIndex 6858) (Generation 241) Flags: SNMP-Traps 0x4004000 Encapsulation: ENET2 Statistics Packets pps Bytes bps Bundle: Input : 16556 4 1675388 3720 Output: 5750 1 323076 416 Link: xe-0/2/0.0 Input : 2239 1 268092 1312 Output: 263 0 23442 0 xe-1/2/0.0 Input : 4821 1 583472 992 Output: 2896 0 152110 0 xe-3/2/0.0 Input : 4353 2 373092 1416 Output: 2282 1 118706 416 LACP info: Role System System Port Port Port priority identifier priority number key xe-0/2/0.0 Actor 127 b0:a8:6e:36:9f:c0 127 3 11 xe-0/2/0.0 Partner 127 3c:94:d5:0c:07:c0 127 11 11 xe-1/2/0.0 Actor 127 b0:a8:6e:36:9f:c0 127 2 11 xe-1/2/0.0 Partner 127 3c:94:d5:0c:07:c0 127 10 11 xe-3/2/0.0 Actor 127 b0:a8:6e:36:9f:c0 127 4 11 xe-3/2/0.0 Partner 127 3c:94:d5:0c:07:c0 127 12 11 LACP Statistics: LACP Rx LACP Tx Unknown Rx Illegal Rx xe-0/2/0.0 2042 2029 0 0 xe-1/2/0.0 4640 4676 0 0 xe-3/2/0.0 2038 2032 0 0 Marker Statistics: Marker Rx Resp Tx Unknown Rx Illegal Rx xe-0/2/0.0 0 0 0 0 xe-1/2/0.0 0 0 0 0 xe-3/2/0.0 0 0 0 0 Protocol inet, MTU: 1500, Generation: 342, Route table: 0 Flags: Sendbcast-pkt-to-re Addresses, Flags: Is-Preferred Is-Primary Destination: 172.19.1/24, Local: 172.19.1.1, Broadcast: 172.19.1.255, Generation: 185 Protocol multiservice, MTU: Unlimited, Generation: 343, Route table: 0 Flags: Is-Primary Policer: Input: __default_arp_policer__
OSPF is running between R1 and R2 with BFD. The uptime for OSPF and BPD is seen below:
[edit] lab@R1# run show ospf neighbor detail Dec 02 08:31:05 Address Interface State ID Pri Dead 172.19.1.2 ae10.0 Full 66.110.0.74 128 32 Area 0.0.0.0, opt 0x52, DR 172.19.1.1, BDR 172.19.1.2 Up 00:36:35, adjacent 00:35:59 [edit] lab@R1# run show bfd session detail Dec 02 08:31:15 Detect Transmit Address State Interface Time Interval Multiplier 172.19.1.2 Up ae10.0 5.000 1.000 5 Client OSPF realm ospf-v2 Area 0.0.0.0, TX interval 1.000, RX interval 1.000 Session up time 00:36:39 Local diagnostic None, remote diagnostic None Remote state Up, version 1 1 sessions, 1 clients Cumulative transmit rate 1.0 pps, cumulative receive rate 1.0 pps
The filter that is programmed in the FPC can be seen from the FPC shell:
[edit] lab@R1# run start shell pfe network fpc0 NPC platform (1067Mhz MPC 8548 processor, 2048MB memory, 512KB flash) NPC0# show filter Program Filters: --------------- Index Dir Cnt Text Bss Name -------- ------ ------ ------ ------ -------- Term Filters: ------------ Index Semantic Name -------- ---------- ------ 1 Classic __default_bpdu_filter__ 17000 Classic __default_arp_policer__ 57008 Classic __cfm_filter_shared_lc__ 65280 Classic __auto_policer_template__ 65281 Classic __auto_policer_template_1__ 65282 Classic __auto_policer_template_2__ 65283 Classic __auto_policer_template_3__ 65284 Classic __auto_policer_template_4__ 65285 Classic __auto_policer_template_5__ 65286 Classic __auto_policer_template_6__ 65287 Classic __auto_policer_template_7__ 65288 Classic __auto_policer_template_8__ 16777216 Classic fnp-filter-level-all 46137345 Classic HOSTBOUND_IPv4_FILTER 46137346 Classic HOSTBOUND_IPv6_FILTER 54525952 Classic ppmd-bfd-agg-inet4 Resolve Filters: --------------- Index --------
Looking at the detailed configuration of the filter named "ppmd-bfd-agg-inet4," the following is observed:
NPC0# show filter index 54525952 program Filter index = 54525952 Optimization flag: 0xf7 Filter notify host id = 0 Filter properties: None Filter state = CONSISTENT term ppmd-172.19.1.2 term priority 0 destination-port 3784 false branch to match action in rule ppmd-default source-address 172.19.1.2/32 false branch to match action in rule ppmd-default then action next-hop, type (nh-id) 968 term ppmd-default term priority 0 then accept
The next-hop index points to the responsible FPC on all three FPCs where the ae has child links:
NPC0# show nhdb id 968 ID Type Interface Next Hop Addr Protocol Encap MTU Flags PFE internal Flags ----- -------- ------------- --------------- ---------- ------------ ---- ---------- -------------------- 968 Unicast pfh-1/0/0.16383 128.0.0.17 IPv4 Unspecified 65535 0x00000000 0x00000000 lab@R1# run start shell pfe network fpc3 NPC platform (1067Mhz MPC 8548 processor, 2048MB memory, 512KB flash) NPC3# show filter index 54525952 program Filter index = 54525952 Optimization flag: 0xf7 Filter notify host id = 0 Filter properties: None Filter state = CONSISTENT term ppmd-172.19.1.2 term priority 0 destination-port 3784 false branch to match action in rule ppmd-default source-address 172.19.1.2/32 false branch to match action in rule ppmd-default then action next-hop, type (nh-id) 968 term ppmd-default term priority 0 then accept NPC3# show nhdb id 968 ID Type Interface Next Hop Addr Protocol Encap MTU Flags PFE internal Flags ----- -------- ------------- --------------- ---------- ------------ ---- ---------- -------------------- 968 Unicast pfh-1/0/0.16383 128.0.0.17 IPv4 Unspecified 65535 0x00000000 0x00000000 From the pfe shell of FPC1 it is observed the bfd packets are transmitted and received >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> NPC1( vty)# show ppm adjacencies protocol bfd PPM Adjacency information for BFD IFL-index Holdtime (msec) PPM handle Discr Absorbed Packets RX Packet Length Dist Do-dist 342 5000 10 20 459 462 0x532f8858 24 no-dist no-do-dist NPC1( vty)# show ppm adjacencies protocol bfd PPM Adjacency information for BFD IFL-index Holdtime (msec) PPM handle Discr Absorbed Packets RX Packet Length Dist Do-dist 342 5000 10 20 475 478 0x532f8858 24 no-dist no-do-dist
Then, link xe-1/2/0 is removed from the bundle:
[edit] lab@R1# show | compare Dec 02 08:33:44 [edit interfaces] - xe-1/2/0 { - gigether-options { - 802.3ad ae10; - } - } [edit] lab@R1# commit Dec 02 08:33:47 commit complete
After the link comes out of the bundle, the ae link with OSPF and BFD is up and has not reset:
[edit] lab@R1# run show interfaces terse ae10 Dec 02 08:34:06 Interface Admin Link Proto Local Remote ae10 up up ae10.0 up up inet 172.19.1.1/24 multiservice [edit] lab@R1# run show ospf neighbor detail Dec 02 08:34:09 Address Interface State ID Pri Dead 172.19.1.2 ae10.0 Full 66.110.0.74 128 31 Area 0.0.0.0, opt 0x52, DR 172.19.1.1, BDR 172.19.1.2 Up 00:39:40, adjacent 00:39:04 [edit] lab@R1# run show bfd session detail Dec 02 08:34:14 Detect Transmit Address State Interface Time Interval Multiplier 172.19.1.2 Up ae10.0 5.000 1.000 5 Client OSPF realm ospf-v2 Area 0.0.0.0, TX interval 1.000, RX interval 1.000 Session up time 00:39:39 Local diagnostic None, remote diagnostic None Remote state Up, version 1 1 sessions, 1 clients Cumulative transmit rate 1.0 pps, cumulative receive rate 1.0 pps
Even when the link from FPC 1 has come out of the bundle, the responsible FPC has not changed, which can be confirmed from the filter programming on the PFE:
NPC0# show filter index 54525952 program Filter index = 54525952 Optimization flag: 0xf7 Filter notify host id = 0 Filter properties: None Filter state = CONSISTENT term ppmd-172.19.1.2 term priority 0 destination-port 3784 false branch to match action in rule ppmd-default source-address 172.19.1.2/32 false branch to match action in rule ppmd-default then action next-hop, type (nh-id) 968 term ppmd-default term priority 0 then accept NPC0# show nhdb id 968 ID Type Interface Next Hop Addr Protocol Encap MTU Flags PFE internal Flags ----- -------- ------------- --------------- ---------- ------------ ---- ---------- -------------------- 968 Unicast pfh-1/0/0.16383 128.0.0.17 IPv4 Unspecified 65535 0x00000000 0x00000000 It is observed that the ppm adjacency can be still found on FPC1 and bfd packets are transmitted and received NPC1( vty)# show ppm adjacencies protocol bfd PPM Adjacency information for BFD IFL-index Holdtime (msec) PPM handle Discr Absorbed Packets RX Packet Length Dist Do-dist 342 5000 10 20 5876 5879 0x532f8858 24 no-dist no-do-dist NPC1( vty)# show ppm adjacencies protocol bfd PPM Adjacency information for BFD IFL-index Holdtime (msec) PPM handle Discr Absorbed Packets RX Packet Length Dist Do-dist 342 5000 10 20 5957 5960 0x532f8858 24 no-dist no-do-dist
Now, restarting fpc in slot 1 will cause a BFD flap, which will in turn cause a protocol flap:
lab@R1# run request chassis fpc slot 1 restart Dec 02 08:35:21 Restart initiated, use "show chassis fpc" to verify [edit] lab@R1# run show chassis fpc Dec 02 08:35:24 Temp CPU Utilization (%) Memory Utilization (%) Slot State (C) Total Interrupt DRAM (MB) Heap Buffer 0 Online 39 7 0 2048 21 13 1 Offline ---Restarted by cli command--- 2 Empty 3 Online 33 22 0 2048 12 13 4 Online 42 6 0 1024 7 41 5 Online 35 5 0 1024 19 29 [edit] lab@R1# run show bfd session extensive Dec 02 08:35:28 Detect Transmit Address State Interface Time Interval Multiplier 172.19.1.2 Up ae10.0 5.000 1.000 5 Client OSPF realm ospf-v2 Area 0.0.0.0, TX interval 1.000, RX interval 1.000 Session up time 00:00:04, previous down time 00:00:01 Local diagnostic None, remote diagnostic None Remote state Up, version 1 Min async interval 1.000, min slow interval 1.000 Adaptive async TX interval 1.000, RX interval 1.000 Local min TX interval 1.000, minimum RX interval 1.000, multiplier 5 Remote min TX interval 1.000, min RX interval 1.000, multiplier 5 Local discriminator 4, remote discriminator 21 Echo mode disabled/inactive Remote is control-plane independent 1 sessions, 1 clients Cumulative transmit rate 1.0 pps, cumulative receive rate 1.0 pps
The log reports:
Dec 2 08:35:23 R1 bfdd[1525]: BFDD_TRAP_SHOP_STATE_DOWN: local discriminator: 4, new state: down, interface: ae10.0, peer addr: 172.19.1.2 Dec 2 08:35:23 R1 rpd[1560]: RPD_OSPF_NBRDOWN: OSPF neighbor 172.19.1.2 (realm ospf-v2 ae10.0 area 0.0.0.0) state changed from Full to Down due to InActiveTimer (event reason: BFD session timed out and neighbor was declared dead) Dec 2 08:35:23 R1 rpd[1560]: RPD_OSPF_NBRUP: OSPF neighbor 172.19.1.2 (realm ospf-v2 ae10.0 area 0.0.0.0) state changed from Init to ExStart due to 2WayRcvd (event reason: neighbor detected this router) Dec 2 08:35:23 R1 rpd[1560]: RPD_OSPF_NBRUP: OSPF neighbor 172.19.1.2 (realm ospf-v2 ae10.0 area 0.0.0.0) state changed from Exchange to Full due to ExchangeDone (event reason: DBD exchange of master completed) Dec 2 08:35:24 R1 bfdd[1525]: BFDD_TRAP_SHOP_STATE_UP: local discriminator: 4, new state: up, interface: ae10.0, peer addr: 172.19.1.2
This is a day one behavior and cannot be changed. Customers are requested to check the FPC using the above process to ensure that it is not the responsible FPC before any event for a specific FPC is planned.