Description

On EX3200/EX4200 Series switches, the Flexible PIC Concentrator (FPC) can be detached from the Virtual Chassis (VC) or cause other traffic related issues. A high number of pause frames received on interfaces may cause a soft reset of the switch.

Symptoms

The following (PFE) messages will be seen in /var/log when the switch undergoes a soft reset:

Feb 12 10:30:01 W-Core /kernel: simulated intr
Feb 12 10:30:01 W-Core chassism[1467]: cm_java_pfe_critical_error_check: Soft-resetting device 0

Solution

If the pause frames are continuous and frequent, this might result in continuous soft-reset of the PFE device. The impact on traffic of a soft reset of the PFE device is minor; However, continuous soft resets due to pause frames might cause the FPC to detach from the VC, and cause other traffic related issues.

This error message also indicates that the PFE device undergoes a graceful restart to recover from the buffer stuck or critical error state. The system may undergo buffer stuck state in certain race conditions. A PFE reset is implemented to recover the system automatically. This particular condition can be caused by high rate of paused frames received on an interface. Pause frames are the method of flow control for signaling the upstream devise to slow down. A periodic, high rate of these frames can cause TxUnderun (which can cause CRC errors on any of the internal ports) or Tx Queue Watchdog Expiry [Tx - Transmit]. These errors can also be seen on EX4200s running 12.3R6 onwards, under no production impact environments. In a virtual chassis, it can even cause FPC reboots. These errors can also point to hardware errors, if CRC errors are seen on both VCPs and internal ports.  

There are a few options:

  • Isolate and stop the root cause of pause frames (traffic bottlenecks). The pause frames are from the device on the other end. If the other end interface is congested, it will send the MAC PAUSE notification to the connected peers to halt the transmissions to avoid packet drops.
    Check the FPC to confirm that it is receiving a high rate of pause frames; look for PAUSE errors.

    root> show interfaces <int-name> extensive | match pause

    OR
    root> show interfaces "[gfx]e-*" media detail | grep "(physical|pause)"     >>> filter the MAC pause frame section for all interfaces
    Physical interface: ge-0/0/0, Enabled, Physical link is Down
    MAC pause frames 15 0
    Physical interface: ge-0/0/1, Enabled, Physical link is Down
    MAC pause frames 30 0
    Physical interface: ge-0/0/2, Enabled, Physical link is Down
    MAC pause frames 0 0
    Physical interface: ge-0/0/3, Enabled, Physical link is Down
    MAC pause frames 0 0
    Physical interface: ge-0/0/4, Enabled, Physical link is Down
    MAC pause frames 0 0
    Physical interface: ge-0/0/5, Enabled, Physical link is Down
    MAC pause frames 0 0
    Physical interface: ge-0/0/6, Enabled, Physical link is Down
    MAC pause frames 0 0
    Physical interface: ge-0/0/7, Enabled, Physical link is Down
    MAC pause frames 0 0
  • Consider disabling flow control on the interfaces of which the counter is increasing. To disable flow control, include the no-flow-control statement under the interface hierarchy.
      root# set interfaces <interface-name> ether-options no-flow-control
  • Make sure there are no high rates of paused frames before replacing hardware due to CRC errors.
 

Related Information