Description

This article explains the meaning of the syslog message "464 TPCI PIO Read errors seen in last second" that is seen on MX routers and what users must do when they encounter this error.

Symptoms

The Routing Engine continuously logs the following messages: 

​Jul 17 17:15:03.657 2021  Test : %PFE-3: fpc2 cc_mic_periodic_raw: CC_MIC(2/2) 464 TPCI PIO Read errors seen in last second
Jul 17 17:15:03.861 2021  Test : %PFE-3: fpc2 PCI ERROR: 2:2:11:0 PIO Active, Timestamp 127585408743 msec.
Jul 17 17:15:03.958 2021  Test : %PFE-3: fpc2 PCI ERROR: 2:2:11:0 (0x0006)                         Status : 0x00004010
Jul 17 17:15:04.075 2021  Test : %PFE-3: fpc2 PCI ERROR: 2:2:11:0 (0x004a)                  Device status : 0x0000000c
Jul 17 17:15:04.176 2021  Test : %PFE-3: fpc2 PCI ERROR: 2:2:11:0 (0x0052)                    Link status : 0x00001811
Jul 17 17:15:04.286 2021  Test : %PFE-3: fpc2 PCI ERROR: 2:2:11:0 (0x0104)     Uncorrectable error status : 0x00100000
Jul 17 17:15:04.395 2021  Test : %PFE-3: fpc2 PCI ERROR: 2:2:11:0 (0x0118)       Advanced error cap & ctl : 0x000001f4
Jul 17 17:15:04.504 2021  Test : %PFE-3: fpc2 PCI ERROR: 2:2:11:0 (0x011c)                   Header log 0 : 0x40008001

Solution

The error indicates a permanent or temporary PCI block error. A PIO error message is reported in the system message file whenever a memory integrity check is performed by the LU driver and encounters an error.

If this log appears only once, it is most likely that this is a transient issue. In this case, reboot the affected MIC and monitor. Ideally, in the case of a transient issue, a reboot will re-initiate all functions and processes.

If the log continues to appear after a reboot, it is suggested to re-seat/replace the MIC. If the log persists even after replacement, contact Support for additional assistance.

 

Note: Procedure to determined the affected MIC module.

The error log format is '%PFE-3: fpc2 cc_mic_periodic_raw: CC_MIC(<FPC_Slot>/<PIC_Slot>)'. 

 

The observed error can be used to identify affected PIC module.

The PIC can then be used to determine the parent MIC module on which the mitigation action is to be performed.

 

For Example:

Error Log: fpc1 cc_mic_periodic_raw: CC_MIC(1/2) 260 TPCI PIO <Read/Write> errors seen in last second


<show chassis hardware>
FPC 1            REV 14   750-063183   <S/N>             MPC2E NG HQoS
  CPU            REV 13   711-045719   <S/N>             RMPC PMB
  MIC 0          REV 27   750-049846   <S/N>             3D 20x 1GE(LAN)-E,SFP
    PIC 0                 BUILTIN      BUILTIN           10x 1GE(LAN) -E  SFP
    <>
    PIC 1                 BUILTIN      BUILTIN           10x 1GE(LAN) -E  SFP
    <>
  MIC 1          REV 05   750-060558   <S/N>             MIC-3D-4OC3OC12-1OC48
    PIC 2                 BUILTIN      BUILTIN           MIC-3D-4OC3OC12-1OC48
    <>

The above error 'CC_MIC(1/2)' indicates that FPC1 PIC2 is affected.

Thus mitigation actions will need to be performed on parent MIC1.

 

Modification History

Mar 27: Procedure to determine the affected MIC card has been added to the Solution Section.