Description

This article describes the issue of fabric drops increasing on the trio platform.

Symptoms

After cleaning the PFE statistics record, an increase in fabric drops is noticed, exactly during the traffic spike on the PFE engine.  To confirm this, use the following commands in the PFE statistics traffic parameters: 

user@re0-router> show pfe statistics traffic fpc 0 | match "cell|drops"
Info cell drops            :               104888
Fabric drops               :              1434592

You will also be able to notice the tail drops or RED drops via the egress queue and shell commands, as shown below:

user@re0-router> show interfaces queue ge-0/1/0 egress               

Physical interface: ge-0/1/0, Enabled, Physical link is Up
  Interface index: 156, SNMP ifIndex: 523
Forwarding classes: 16 supported, 6 in use
Egress queues: 8 supported, 6 in use
Queue: 0, Forwarding classes: be1
  Queued:
    Packets              :           30452605752                 16239 pps
    Bytes                :        23786610358133              96295936 bps
  Transmitted:
    Packets              :           30452520268                 16239 pps
    Bytes                :        23786536137026              96295936 bps
    Tail-dropped packets :                     0                     0 pps
    RED-dropped packets  :                 85484                     0 pps
     Low                 :                 85484                     0 pps
     Medium-low          :                     0                     0 pps
     Medium-high         :                     0                     0 pps
     High                :                     0                     0 pps
    RED-dropped bytes    :              74221107                     0 bps
     Low                 :              74221107                     0 bps
     Medium-low          :                     0                     0 bps
     Medium-high         :                     0                     0 bps
     High                :                     0                     0 bps
NPC0(re0-router vty)# show mqchip 0 fi stats    

FI Counters:

    Stream  Mask Match
           0x000 0x000          Packets     Pkt Rate            Cells    Cell Rate
  -------------------- ---------------- ------------ ---------------- ------------
              Received    1927762707901       500081   33098984718091      8560244
               Dropped          6002156            0         91466419            0
            Pkts to PT    1927756720187       500081
    Errored Pkts to PT            16581            0

Solution

  • Those counters are very different, depending on different platforms.

  • On the MX Trio platform, fabric drops can be displayed by the MQ-chip and the interface queue info.

  • The 256-byte packet header as notification cells was forwarded from the fabric to the LU-chip for lookup via MQ-chipset; so MQ-chip will also be requested to drop the extra packet header, if the traffic spike to the LU-chip performance limit occurs.

  • In the tests, the more output direction from firewall filters with long terms, the easier it is to hit hardware limitationof dropping the cells from the fabric.

  • Also, the more complex the traffic, such as carrying TCP or UDP, is matched by firewall filters, the easier it is to hit the performance limitation.


 The following outputs can be examined to identify the cause of this message:

  • request support information | no-more
  • show firewall filter <fliter-name>
  • show interface queue <interface index> egress
  • start shell pfe network fpc <x>
  • show mqchip <x> fi stats
  • show mqchip <x> counters
  • show mqchip <x> counters error
  • exit

To resolve this issue, make the firewall filter term shorter; which means the reduction of the terms or filters on specific interface's output direction, if possible. If the above scenario is observed, create a JTAC case for further analysis of the issue.