BFD session is flapping for ISIS adjacency with the other node where the other BFD sessions were stable.

Initial suspect was the receiving logical interface index and another internal interface (lc - interface) was having the same IFD index but that is not the case here.

we initially applied the BFD filter and we don't see issues in sending the packet out but remote router is receiving lower incoming packets than the out packets, hence the problem of delayed packet scheduling was the suspect from Originating router.

root@router-MX> show firewall | match BFD    

Filter: BFD-IN-ae121.0-i                                      

BFD-PKT-IN-ae121.0-i                                                          129012                2481

Filter: BFD-OUT-ae121.0-o                                     

BFD-PKT-OUT-ae121.0-o                                                         129012                2481

BFD session down logs was seen as the reason for ISIS adjacency flaps and its due to the intermittent long ICMP reply times between the 2 chassis and also indicating a High CPU on FPC which could have led to the BFD packets scheduling delays as well resulting in session flaps.

root@router-MX > show chassis fpc no-forwarding

                    Temp CPU Utilization (%)  CPU Utilization (%) Memory   Utilization (%)

Slot State           (C) Total Interrupt     1min  5min  15min DRAM (MB) Heap    Buffer

 0 Online           54    92        34       90    89    89   3168      25        46

 1 Online           53    30         4      30    32    33   3168      25        28

SMPC0(router-MX vty)# show sched   

Total system uptime 5+14:49:53, (485359768 ms), 652059766 thread dispatches

CPU load is 33% (1 second), 38% (5 second)

           36% (10 second), 35% (1 minute)

Total interrupt time 164917856 (ms)

CPU      Name     Time(ms)

11%      Idle     55626704

54%   Threads     264848611

33%       ISR     164917856

33%   Level 1     164360074

 0%   Level 6     557781

Last Long Running Thread Event time 100 ms

CPU      Name     Time(ms)   Count

Top Thread:

  pid     = 64

  name    = ARP Proxy

  time    = 122959646 ms

  cpu     = 25%

High CPU usage on FPC 0 was the key reason for BFD flaps due to below symptoms.

1. High ICMP response time beyond 4000 ms.
2. ARP storm is observed on one of FPC 0 interface.
3. ARP storm issue is fixed by applying the ARP policer at interface level on the interested interface and post that CPU got stabilized and the BFD/ ISIS session as well.

root@router-MX> show ddos-protection protocols statistics terse                              

Packet types: 253, Received traffic: 60, Currently violated: 0

Protocol   Packet     Received       Dropped       Rate    Violation State

group      type       (packets)      (packets)     (pps)   counts

arp        aggregate  63258342206     57699955156    408     1        ok  

arp        ucast      34260436       0             16      0        ok  

arp        bcast      63224081770     57699955156    391     1        ok  

root@router-MX> show chassis fpc no-forwarding

                     Temp CPU Utilization (%)  CPU Utilization (%) Memory   Utilization (%)

Slot State           (C) Total Interrupt     1min  5min  15min DRAM (MB) Heap    Buffer

 0 Online           54    38         3      36    34    33   3168      25        46

 1 Online           53    31         4      32    33    32   3168      25        28