Peripheral Component Interconnect (PCI) is an interface that provides connectivity between the FRU's. The PCI Fatal Error Encountered by message reports that the specified hardware is not able to handle some transit traffic internally.

We might see the below PCI fatal errors detected on the MX104 device:

 %PFE-6: afeb0 ifp ge-1/1/8 ifd_mdown

 %PFE-3: afeb0 stats_lu_counter_group_delete(1484): pfe 0: group = 0

 %PFE-6: afeb0 ifp xe-2/0/1 ifd_mdown

 %PFE-6: afeb0 ifp ge-1/0/0 ifd_mdown

 %PFE-6: afeb0 ifp ge-1/1/9 ifd_mdown

 %PFE-6: afeb0 mic_mac_periodic:ge-1/0/0: ifd_mup

 %PFE-3: afeb0 MIC2 IX PCI Fatal Error detected.

 %PFE-3: afeb0 Ixchip(4): pio_handle(0x4f690ba8); pio_read_u32() failed: 20(input/output error)! ge-addr=011048c8

 %PFE-3: afeb0 MIC2 IX PCI Fatal Error detected.

 %PFE-3: afeb0 Ixchip(4): pio_handle(0x4f690ba8); pio_read_u32() failed: 20(input/output error)! ge-addr=011048cc

 %PFE-3: afeb0 MIC2 IX PCI Fatal Error detected.

 %PFE-3: afeb0 Ixchip(4): pio_handle(0x4f690ba8); pio_read_u32() failed: 20(input/output error)! ge-addr=011048d0

 %PFE-3: afeb0 Ixchip(4): pio_handle(0x4f690ba8); pio_read_u32() failed: 20(input/output error)! ge-addr=011048dc

 %PFE-0: afeb0 ixchip_env_check: Too many IXCHIP 4 IO errors, stop subsequent IXCHIP READ/WRITE operations

 %PFE-3: afeb0 stats_lu_counter_group_delete(1484): pfe 0: group = 0

 %PFE-3: afeb0 stats_lu_counter_group_delete(1484): pfe 0: group = 0

The above fatal errors could be caused due to any of the below component:

• MIC/PIC
• FPC
• 3rd party or NON-JNPR SFP's
• AFEB.

In order to find the exact faulty part, we may need to collect the below command output:

>show chassis alarms

>show system core-dumps

>show chassis fpc pic-status

>show chassis hardware detail

>show log messages

>show log chassisd

> start shell pfe network afeb0 

#show syslog messages

#show syslog info

#show nvram

#show ixchip all

#show ixchip ifd

#show hsl2 statistics

#show hsl2 stat detail

#show luchip #

(Replace the '#' character in the above list with the corresponding FPC or PFE chip number, as provided in the syslog messages).

#show pq3 pci

# show sfp list 

# show sfp <index from show sfp list> info 

    (Check i2c failure count on each SFP. If the SFP shows a non-zero i2c failure count, replace that specific SFP). 

# show sfp 3 info 

#exit

• Analyze the show command output. Look for any related events that occurred at or just before the Fatal Error Encountered by message in the syslog or chassisd outputs.
• If a core dump was generated, open a case with your technical support representative, and upload the core file with all the above listed outputs.
• For troubleshooting the 3rd Party/NON-JNPR SFP issue, please refer the KB31238 [juniper.net]: https://supportportal.juniper.net/s/article/PCI-Fatal-errors-or-Too-many-IXCHIP-x-IO-errors-are-seen-on-MX-104-MICs-when-using-3rd-party-NON-JNPR-optics
• If you see an issue on the particular MIC card as below, restart the particular MIC or the FPC, to see if the alarms are cleared during the MW time:

> show chassis fpc pic-status 

Slot 0 Online   

Slot 1 Online   

 PIC 0 Present   10x 1GE(LAN) SFP- Hardware Error

 PIC 1 Present   10x 1GE(LAN) SFP- Hardware Error

>request chassis fpc slot 1 restart

• If you still the issue is not resolved and you see multiple MIC cards on the MX device, affected with fatal errors, then the issue could be on the AFEB0. A reboot of the FEB is required to clear the errored condition. Please perform this during a maintence window as this will impact traffic. 

>request chassis afeb restart

• If the issue persists still, please open a TAC case to confirm if the MX104 midplane is faulty.

KB26111 [juniper.net] : https://supportportal.juniper.net/s/article/Syslog-message-Fatal-Error-Encountered-by

KB31238 [juniper.net]: https://supportportal.juniper.net/s/article/PCI-Fatal-errors-or-Too-many-IXCHIP-x-IO-errors-are-seen-on-MX-104-MICs-when-using-3rd-party-NON-JNPR-optics