Description

The article explains the meaning of PCI errors observed on customer box and steps to resolve the issue . 

Symptoms

Customer reported FPC is rebooting and crashing with below logs seen on the box . 

Mar 20 09:39:46 2024   pe5 PCI Error detected. 
Mar 20 09:39:46 2024   pechip_hmc_mem_read:1300: PECHIP[5]: fail to rd hmcif access control 
Mar 20 09:39:46 2024   pe5 PCI Error detected. 
Mar 20 09:39:46 2024   pe5 PCI Error detected. 
Mar 20 09:39:46 2024   pechip_hmc_mem_read:1340: PECHIP[5]: index:27866000 fail to read hmcif access data 
Mar 20 09:39:46 2024   pechip_q_cm_per_voq_drop_stat_get:3827: PE[5]: CM VOQ Stat HMC read(hmc_addr = 27866000) failed 7 (invalid argument) 
Mar 20 09:39:46 2024   pechip_q_stats_voq_update:298: PE_STAT_CM: per_voq_stat get failed for PE[5] VOQ=0x16600 
Mar 20 09:39:46 2024   pe5 PCI Error detected. 
Mar 20 09:39:46 2024   pe5 PCI Error detected. 
Mar 20 09:39:46 2024   pechip_hmc_mem_read:1300: PECHIP[5]: fail to rd hmcif access control 
Mar 20 09:39:46 2024   pe5 PCI Error detected. 
Mar 20 09:39:46 2024   pe5 PCI Error detected. 
Mar 20 09:39:46 2024   pechip_hmc_mem_read:1340: PECHIP[5]: index:27866010 fail to read hmcif access data 
Mar 20 09:39:46 2024   pechip_q_cm_per_voq_drop_stat_get:3827: PE[5]: CM VOQ Stat HMC read(hmc_addr = 27866010) failed 7 (invalid argument) 
Mar 20 09:39:46 2024   pechip_q_stats_voq_update:298: PE_STAT_CM: per_voq_stat get failed for PE[5] VOQ=0x16601 
Mar 20 09:39:46 2024   pechip_hmc_mem_read:1274: PECHIP[5]: index:27866020 fail to set hmcif access control 
Mar 20 09:39:46 2024   pechip_q_cm_per_voq_drop_stat_get:3827: PE[5]: CM VOQ Stat HMC read(hmc_addr = 27866020) failed 1 (generic failure) 
Mar 20 09:39:46 2024   pechip_q_stats_voq_update:298: PE_STAT_CM: per_voq_stat get failed for PE[5] VOQ=0x16602 
Mar 20 09:39:46 2024   pechip_hmc_mem_read:1274: PECHIP[5]: index:27866030 fail to set hmcif access control 
Mar 20 09:39:46 2024   pechip_q_cm_per_voq_drop_stat_get:3827: PE[5]: CM VOQ Stat HMC read(hmc_addr = 27866030) failed 1 (generic failure) 
Mar 20 09:39:46 2024   pechip_q_stats_voq_update:298: PE_STAT_CM: per_voq_stat get failed for PE[5] VOQ=0x16603 
Mar 20 09:39:46 2024   pechip_hmc_mem_read:1274: PECHIP[5]: index:27866040 fail to set hmcif access control iled 1 (generic failure) 
   

show system core-dumps no-forwarding
 
-rw-r--r--  1 root  wheel  291790173 Mar 8  09:32 /var/crash/core-PE1-gldfpc.elf.0.tgz
-rw-r--r--  1 root  wheel  252364092 Mar 13 15:58 /var/crash/core-PE1.sjc10-fpc1-gldfpc.elf.1.tgz
-rw-r--r--  1 root  wheel  253843953 Mar 20 09:42 /var/crash/core-PE1.sjc10-fpc1-gldfpc.elf.2.tgz

Mar 20 09:39:46 2024   fpc1 pe5 PCI Error detected.
Mar 13 15:56:10 2024   fpc1 pe5 PCI Error detected.
Mar  8 09:30:00 2024   fpc1 pe5 PCI Error detected.

#2  0x0832c030 in posix_interface_abort () at ../../../../../../../src/pfe/platform/linux/posix_interface.c:2641
#3  0x0832309b in do_abort () at ../../../../../src/pfe/platform/linux/linux_panic.c:34
#4  0x083230b8 in panic (format_string=format_string@entry=0xb380280 <pci_user_pio_panic_buf> "pe5 PCI Fatal Error limit exceeded.\n") at ../../../../../src/pfe/platform/linux/linux_panic.c:76

Solution

PCI is a high-speed bus which manages the communication with ukern of the box. The above log indicates PCI errors has led to data corruption on HMC as it has lost communication with ukern . If there are multiple occurrence of PCI fault causing FPC reboot and core-dumps it indicates hardware issue and the FPC needs to be replaced . 

Modification History

2024-03-21 : Article Created