Description

This article explain fabric drops on the MX10K devices.We will be seeing latency issues around 400G on the LAG interfaces. And this would lead to fabric drops.

Symptoms

Farbic drops would be noticed when traffic exceed 400G on the interface.

 

labroot@testing> show interfaces vt-0/0/0.5984 extensive  

 

 Logical interface vt-0/0/0.5984 (Index 380) (SNMP ifIndex 602) (Generation 201)

 Flags: Up Point-To-Point 0x4000 Encapsulation: Virtual-loopback-tunnel

 Traffic statistics:

Input bytes :    93327691610835

   Output bytes :          0

   Input packets:     154898926820

   Output packets:          0

 

  Transit statistics:

   Input bytes :    93327691610835     415748425352 bps. >>>>> more than 400G

 

   Output bytes :          0          0 bps

Input packets:     154898926820       51864823 pps

Output packets:          0          0 pps

 

  Protocol inet, MTU: Unlimited

 

  Max nh cache: 0, New hold nh limit: 0, Curr nh cnt: 0, Curr new hold cnt: 0, NH drop cnt: 0

  Generation: 268, Route table: 13

   Flags: Sendbcast-pkt-to-re

  Protocol mpls, MTU: Unlimited, Maximum labels: 3, Generation: 271, Route table: 13

   Flags: Is-Primary

 

 

labroot@testing> show pfe statistics traffic | match fabric

 

  Fabric Input :     331400532316      215679852 pps

  Fabric Output :     331370310308      215698085 pps

  Fabric drops        :      3373634890 >>>>>>fabric drop noticed.

 

 

labroot@testing> show pfe statistics traffic | match fabric  

 

  Fabric Input :     332694844651      215730135 pps

  Fabric Output :     332664652658      215745895 pps

  Fabric drops        :      3436072900

 

Solution

 Packet Forwarding Engine on MX10003 is based on the third generation of the Trio chipset architecture---namely, the EA (Eagle) ASIC. The Packet Forwarding Engine offers 400 Gbps of WAN and fabric bandwidth each.

Since the single PFE instance has the capacity of only 400G the PFE instance hosting the interface is getting oversubscribed leading to the pressure and the fabric drops reported by multiple source PFEs. This is expected behaviour as the fabric traffic being sent is beyond EA's 400G capacity.

 

Modification History

2023-12-29 : Article Created