Description

This article explains the meaning of the syslog message "fpc[X] MIC[X] MEZS PCI Fatal Error detected" that you may observe on MX devices.

When you encounter the MEZS PCI Fatal Error, you should restart the Flexible PIC Concentrator (FPC). If the error continues after FPC restart and physical re-seat, open a case with JTAC Support for troubleshooting the possibility of performing an RMA of the faulty FPC.

Symptoms

When a fatal error is encountered on a MEZS card with MIC installed, the following alarm is generated:

root@RE0> show chassis alarms
Alarm time Class Description
2023-01-20 07:03:15 UTC Major FPC 12 Major Error

When you see the "fpc[X] MIC[X] MEZS PCI Fatal Error detected" syslog message, the following log messages are also reported along with a chassis major alarm:

Jan 19 09:43:15  re0 fpc12 MIC0 MEZS PCI Fatal Error detected.
Jan 19 09:43:15  re0 fpc12 mpcs_pio_read_u32: MPCS(1) failed: pio_handle 0x41594ef8 addr 0x0000040c
Jan 19 09:43:15  re0 fpc12 mpcs_mdio_single_io: MPCS(1) ctlr 1 is busy
Jan 19 09:43:15  re0 fpc12 cmic_bcm8454x_mezs_mdio_read: cmic-bcm84544-12/0/3-PHY3: phy idx 3, chan 0, paddr 0xc: error reading reg 1.8000  (MEZS idx 1, ctlr 1, grp 0)
Jan 19 09:43:15  re0 fpc12 cmic_sfpp_present: CMIC(12/0) failed to get mod_abs for xe-12/0/12
Jan 19 09:43:15   re0 fpc12 cmic_sfpp_periodic: Failed to set up SFP+ link 0
Jan 19 09:43:15  re0 fpc12 bcm8454x_reg_read: cmic-bcm84544-12/0/3-PHY3(0) xe-12/0/12: error reading MDIO reg 1.8000
Jan 19 09:43:15  re0 fpc12 bcm8454x_wait_i2c_optics_data: cmic-bcm84544-12/0/3-PHY3, chan 0: error reading reg 1.8000
Jan 19 09:43:15  re0 fpc12 bcm8454x_get_sfpp_i2c_data: cmic-bcm84544-12/0/3-PHY3(0) xe-12/0/12: error getting module I2C data
Jan 19 09:43:15  re0 fpc12 cmic_bcm8454x_get_sfpp_i2c_data: CMIC(12/0)(12) - error getting SFP+ module I2C data
Jan 19 09:43:16  re0 fpc12 PCI ERROR: 0:0:0:0 PIO Active, Timestamp 6785018483 msec.
Jan 19 09:43:16  re0 fpc12 PCI ERROR: 0:0:0:0 (0x0006)                         Status : 0x00004010
Jan 19 09:43:16  re0 fpc12 PCI ERROR: 0:0:0:0 (0x001e)           Secondary bus status : 0x00004000
Jan 19 09:43:16  re0 fpc12 PCI ERROR: 0:0:0:0 (0x005e)                    Link status : 0x00000041
Jan 19 09:43:16  re0 fpc12 PCI ERROR: 0:0:0:0 (0x0130)              Root error status : 0x00000055
Jan 19 09:43:16  re0 fpc12 PCI ERROR: 0:0:0:0 (0x0134)                Error source ID : 0x05000218
Jan 19 09:43:16  re0 fpc12 PCI ERROR: 0:5:0:0 PIO Active, Timestamp 6785018483 msec.
Jan 19 09:43:16  re0 fpc12 PCI ERROR: 0:5:0:0 (0x0006)                         Status : 0x00004010
Jan 19 09:43:17  re0 fpc12 PCI ERROR: 0:5:0:0 (0x009a)                  Device status : 0x00000004
Jan 19 09:43:17  re0 fpc12 PCI ERROR: 0:5:0:0 (0x00a2)                    Link status : 0x00001011
Jan 19 09:43:17  re0 fpc12 PCI ERROR: 0:2:3:0 PIO Active, Timestamp 6785018483 msec.
Jan 19 09:43:17  re0 fpc12 PCI ERROR: 0:2:3:0 (0x0006)                         Status : 0x00004010
Jan 19 09:43:17  re0 fpc12 PCI ERROR: 0:2:3:0 (0x001e)           Secondary bus status : 0x00004000
Jan 19 09:43:17  re0 fpc12 PCI ERROR: 0:2:3:0 (0x0072)                  Device status : 0x00000001
Jan 19 09:43:17  re0 fpc12 PCI ERROR: 0:2:3:0 (0x007a)                    Link status : 0x00002011
 
Jan 20 07:03:15  re0 alarmd[31535]: Alarm set: FPC id=150998120, color=RED, class=CHASSIS, reason=FPC 12 Major Errors
Jan 20 07:03:15  re0 craftd[27772]:  Major alarm set, FPC 12 Major Errors. ---------------> Major alarm set

Indications

  • A major alarm is raised.

  • Message logs may be overwhelmed with the above messages and traffic might be impacted on the ports.

Solution

Cause

  • The WAN/physical interfaces portion of the MX MPC resides on a MEZZ (Mezzanine) card.

  • The MEZZ card is installed at the factory and is not removable during operation.

  • The MEZZ card is connected to the local processor system on the MPC with two PCIe interfaces.

  • The underlying PCI driver on the local processor system is responsible for communication with the MEZZ card.

  • This driver can set a fatal error status while carrying the communication to and from the cards. When the fatal errors status is set, access to the failing device is disabled and hence a major alarm is generated on the device or local processor system.

Solution

You can clear this error state of the PCI drivers by rebooting the affected Flexible PIC Concentrator (FPC), which clears the PCI bus system as these are usually transient in nature.

To resolve the issue, restart the affected FPC during a Maintenance Window.

To restart the FPC:

root@RE0> request chassis fpc (offline | online | restart) slot <slot-number>

If the issue does not resolve after the restart, or the errors continue after FPC restart and physical re-seat, collect the following show command outputs to investigate further.

  • show log messages
  • show log chassisd
  • start shell pfe network <fpc#> [# - indicates FPC number]
  • show nvram
  • show syslog messages

If the issue is seen after restarting the FPC, open a case with JTAC Support for further investigation and assistance. This might need detailed investigation to rule out the possibility of permanent hardware error due to issues at the driver level.

Modification History

2023-03-30: Initial version

2025-05-07 : Modified the command from "start shell network pfe fpc#" to "start shell pfe network fpc#"