Description

On EX9200, LDP and BGP flap would be seen when receiving high number of ARP packets. This KB describes the details and provide the solution.

Symptoms

On EX9200, LDP and BGP flap would be seen randomly without user operations. At that time, EX9200 is receiving a high number of ARP packets, but DDOS protection for ARP packets doesn't detect the violation. As a result, you might overlook the issue as a problem specific to ARP issues.

Here is the log when LDP down is seen. There are two different logs when the issue is seen, but the cause is the same, not receiving LDP keepalive in a timely manner.

Jun 17 16:42:08 Switch rpd[1802]: %DAEMON-5-RPD_LDP_SESSIONDOWN: LDP session X.X.X.X is down, reason: hold time expired
Jun 17 17:40:35 Switch rpd[1802]: %DAEMON-5-RPD_LDP_SESSIONDOWN: LDP session X.X.X.X is down, reason: all adjacencies down

The similar log can be seen with BGP protocols with the error, hold time expired.

Jun 18 10:13:33  Switch rpd[1802]: %DAEMON-4-RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer X.X.X (Internal AS XXXX) changed state from Established to Idle (event HoldTime) (instance TEST)
Check the output below, and see there are many ARP packets filtered by default policer.
user@Switch> show policer __default_arp_policer__
Policers:
Name                                                Bytes              Packets
__default_arp_policer__                       20537083520            387136313

Solution

This is working as a design as DDOS protection is coming after default ARP policer.
For example, when EX9200 is receiving ARP packets at 1000 pps, DDOS output shows only around 440 pps.

user@Switch> show ddos-protection protocols arp
Packet types: 1, Modified: 0, Received traffic: 1, Currently violated: 0
Currently tracked flows: 0, Total detected flows: 0
* = User configured value

Protocol Group: ARP
~snip~
    FPC slot 1 information:
      Bandwidth: 100% (20000 pps), Burst: 100% (20000 packets), enabled
      Hostbound queue 2
      Aggregate policer is never violated
      Received:  0                   Arrival rate:     399 pps
      Dropped:   0                   Max arrival rate: 438 pps
    Dropped by individual policers: 0
    Dropped by flow suppression:    0

And the counter on default ARP policer is increasing.

user@Switch> show policer __default_arp_policer__
Policers:
Name                                                Bytes              Packets
__default_arp_policer__                       20537083520            387136313

On EX9200, default ARP policer has below parameters on each PFE. And from the test result, if exceeding 400 pps, there is a chance that ARP packets would be dropped by default ARP policer.

  • bandwidth-limit 150000 bits/sec
  • burst-size-limit 15000 bytes

As a result, LDP and BGP flap could be observed since fails to send keepalive messages.
Keep receiving ARP packets with high rate is not ideal in the network. To identify which source is sending that packets, below config would be useful.

set system ddos-protection global flow-detection
set system ddos-protection global flow-detection-mode on

<Sample log>

Jun 19 14:42:08 Switch jddosd[29827]: %DAEMON-4-DDOS_SCFD_FLOW_FOUND: A new flow of protocol ARP:aggregate on ge-2/1/0.0 with source addr 00:00:00:00:00:00 is found at 2023-06-19 14:42:00 IST

Modification History

2023-07-20 : Created a KB