ISSUE: RED packet drops in some of the interfaces in QFX5100  in Fusion setup

ISSUE TRIGGER: No changes made to the environment.

IMPACT: Packet drops are impacting their customer traffic

ENVIRONMENT: Production

DEVICE:

AD - MX204 Code: 20.2R2.11 (EOL)

SD - QFX5100 Code: 20.2R2.11 (EOL)

TROUBLESHOOTING SUMMARY:

1. Customer is running Fusion setup where they are noticing RED packet drops for some of their interfaces in QFX5100 SD devices.
2. Customer reported that their end users are experiencing issues while accessing the internet due to these incrementing drops.
3. For troubleshooting purpose customer shared one of the affected interface.
4. DDOS violation logs have been observed, but they don't appear to be relevant, as the traffic volume isn't significant enough to trigger the ARP policer.
5. MX case was opened to troubleshoot with MX team as AD is MX204. Per MX engineer, no issues were seen on MX side.
6. We discussed about this PR https://gnats.juniper.net/web/default/1788493#description_tab to understand the impact of adding this command: set class-of-service shared-buffer egress buffer-partition lossy dynamic-threshold 9. Adding this command did not help with packet drops.
7. Using dmirror we dumped the ukern traces and checked for packet drops which showed the packets as SFLOW and nothing related to production/impacted traffic.
8. Issue got recovered by itself without any intervention.

The issue happens during the production hours when the customer used the link to have around 10% of link utilization.

>> The issue happens during the production hours when the customer used the link to have around 10% of link utilization.

>> we see RED drops which are bound to happen when the buffers are around 98% full.

>> To investigate the same, we will wait for the issue to come back in.

>> Once the issue is back we can log into the satellite device and enter the vty fpc0.

>> run the command : bcmshell "dump chg MMU_THDU_XPIPE_COUNTER_QUEUE"

>> The above command shows how many packets are in the shared buffers

"https://supportportal.juniper.net/s/article/EX-QFX-Traffic-drop-due-to-microburst-in-COS-Classification?language=en_US"

>> During the regular operations when drops are not seen, the shared buffers had 0-7 packets.

>> In case the buffer has more packets which means interface dedicated buffers were entirely used and then shared was also being used.

>> Once the utilization crosses 98% the red will start dropping random packets.

>> Customer to also check on possibility of iperf tests in order to see the behavior under controlled environment.