On QFX5120 switches, SNMP polling may stop returning interface traffic statistics, and access to the FPC shell using vty fpc0 may become unresponsive

One or more QFX5120 switches may exhibit the following symptoms:

• SNMP polling stations stop receiving interface traffic statistics.
• SNMP MIB walk requests for interface counters fail with a general error.
• Access to the FPC shell using vty fpc0 becomes unresponsive.
• Interface statistics retrieved through several standard interface MIB objects are unavailable.

root@QFX5120> show snmp mib walk ifInOctets

ifInOctets.4 = 0
ifInOctets.5 = 0
ifInOctets.6 = 3056639632
ifInOctets.7 = 0
ifInOctets.8 = 0
ifInOctets.9 = 0
ifInOctets.10 = 0
ifInOctets.11 = 0
ifInOctets.12 = 0

Request failed: General error

The issue may affect interface statistics queried through MIB objects such as:

ifOutOctets
ifInUcastPkts
ifOutUcastPkts
ifInBroadcastPkts
ifOutBroadcastPkts
ifInMulticastPkts
ifOutMulticastPkts

Attempts to access the FPC shell may fail:

root@QFX5120> start shell user root
root@QFX5120:RE:0% vty fpc0
exit on signal 2

The following logs are reported:

MMM DD HH:MM:SS YYYY QFX5120 fpc0 TVP_PROC_STAT_FILE_OPEN_ERR: Error opening /proc/stat file 

MMM DD HH:MM:SS YYYY QFX5120 fpc0 TVP_PROC_STAT_FILE_OPEN_ERR: Error opening /proc/stat file 

MMM DD HH:MM:SS YYYYQFX5120 fpc0 TVP_PROC_STAT_FILE_OPEN_ERR: Error opening /proc/stat file 

MMM DD HH:MM:SS YYYY QFX5120 snmpd[13546]: LIBJSNMP_NS_LOG_INFO: INFO: AgentX session, /var/run/pfed-14, noticed 1 request timeout.

Upgrade the switch to a Junos OS release containing the fix for PR1925996. Additionally, investigate and address the source of frequent DDoS violations, as they are the trigger for this issue.

WORKAROUND: 

The following workaround can temporarily restore functionality until the switch is rebooted:

>start shell
%vty fpc0
#set ukern_trace <# of BRCM_PKT> file-logging disable
#set ukern_trace <# of BRCM_PKT> logging disable

Note: Remember you have to disable these settings after every reboot of the switch to avoid hitting this problem.