This article explains how to identify microburst traffic occurring in a network, which could lead to packet drops in the Class of Service (CoS) queue.

Dropped packets are seen on the class-of-service (CoS) interface.

root@router> show interfaces queue ge-0/1/0    
Physical interface: ge-0/1/0, Enabled, Physical link is Up
Interface index: 649, SNMP ifIndex: 518
Forwarding classes: 16 supported, 5 in use
Egress queues: 12 supported, 5 in use
Queue: 0, Forwarding classes: best-effort
  Queued:
    Packets              :                     0                     0 pps
    Bytes                :                     0                     0 bps
  Transmitted:
    Packets              :           92299418556                122542 pps
    Bytes                :        92299407936208             980338976 bps
    Tail-dropped packets : Not Available  
    RL-dropped packets   :                     0                     0 pps
    RL-dropped bytes     :                     0                     0 bps
    Total-dropped packets:          830775607880               1217312 pps
    Total-dropped bytes  :       830775515656432            9738503624 bps

The total bandwidth of this interface is 10 GBPS and 5% is allocated to queue 0. When traffic exceeds 10Gbps, the CoS buffers allocated for queue 0 will appear. Then the server drops the traffic that crosses the buffer limit.

You can monitor interface utilization with Simple Network Management Protocol (SNMP) traps to plot a graph, which will send a query varying from 30 seconds to 5 minutes. The data utilization graph, however, will not show the bandwidth spike that occurred in the interval of a millisecond to a second, making users think that the interface utilization never crossed the defined limit.

If a queue drop is seen, you can use the following command to check the buffer and data utilization on the node in FPC mode after running "start shell network pfe fpc0" on the EX device. For a QFX device, use "start shell pfe network fpc0," which is a hidden command.

set dcbcm bcmshell "for i=0,200 'sleep 0 10000; dump chg MMU_THDU_XPIPE_COUNTER_QUEUE'"

Note: Here we are executing the above BCM counter dump command 100 times at intervals of 10ms. This will provide the buffer utilization during microburst conditions.

Example

When the output bandwidth of an interface is set to 10Gbps and traffic is pushed to approximately 11Gbps, the command provides the following output:

TFXPC0(vty)# ...000; dump chg MMU_THDU_XPIPE_COUNTER_QUEUE'"                  

HW (unit 0)
Sleeping for 0.010000 seconds
MMU_THDU_XPIPE_COUNTER_QUEUE.mmu0[492]: <SHARED_COUNT=0x19db,PARITY=1,MIN_COUNT=9,ECCP=0x51,ECC=0x11,DATA=0x0033b60009>
Sleeping for 0.010000 seconds
MMU_THDU_XPIPE_COUNTER_QUEUE.mmu0[492]: <SHARED_COUNT=0x19ec,MIN_COUNT=9,DATA=0x0033d80009>
Sleeping for 0.010000 seconds
MMU_THDU_XPIPE_COUNTER_QUEUE.mmu0[492]: <SHARED_COUNT=0x19b3,MIN_COUNT=9,ECCP=0xa,ECC=0xa,DATA=0x0033660009>
Sleeping for 0.010000 seconds
MMU_THDU_XPIPE_COUNTER_QUEUE.mmu0[492]: <SHARED_COUNT=0x1a06,MIN_COUNT=9,ECCP=0x23,ECC=0x23,DATA=0x00340c0009>
MMU_THDU_XPIPE_COUNTER_QUEUE.mmu0[504]: <PARITY=1,MIN_COUNT=1,ECCP=0x43,ECC=3,DATA=0x0000000001>

When traffic is brought down to a lower value, around 10Gbps, then the output indicates there was no microburst:

Sleeping for 0.010000 seconds
Sleeping for 0.010000 seconds
Sleeping for 0.010000 seconds
Sleeping for 0.010000 seconds
Sleeping for 0.010000 seconds
Sleeping for 0.010000 seconds
Sleeping for 0.010000 seconds
Sleeping for 0.010000 seconds​

2023-01-17: Added command to access PFE from CLI for QFX and also noted that they are hidden commands