Description

The user saw error log for FPC 10

"toe_interrupt_errors:405 LU TOE chip 48 Thread 0 ifetch err"

 

>> A TOE_ERR_MEM_READ error is reported when there is a transient memory error in TOE for LU.

>> TOE or Trinity Offload Engine is a mechanism which moves data between ASICs and the FPC microkernel. This helps in the interaction of the FPC with the internal chips and the hardware on the underlying chip.

>>In this case the error seems to be generated because of the ifetch error and memory error.

Symptoms

FPC 10 generated below error logs.

Jul 30 18:58:40 Router fpc10 toe_interrupt_errors:405 LU TOE chip 48 Thread 0 ifetch err 
Jul 30 18:58:40 Router fpc10 toe_interrupt_errors:453 LU TOE chip 48 Memory Error Thread 0, INSTR Addr 0x1564 
Jul 30 18:58:40 Router fpc10 toe_coredump_read_regs(481): failed(1) read gpr-32 th-0 
Jul 30 18:58:40 Router tnp.tftpd[96942]: TFTPD_CONNECT_INFO: TFTP write from address 26 port 3 file /var/tmp/core-FPC10-PFE3-LU0-TOE0
Jul 30 18:58:40 Router fpc10 toe_dump_core(1259): failed(1) read regs data 

Jul 30 18:58:40 Router chassisd[13400]: CHASSISD_FPC_ASIC_ERROR: <FPC 10> ASIC Error detected errorno 0x00010004 (null)
Jul 30 18:58:40 Router alarmd[13546]: Alarm set: FPC id=167774824, color=YELLOW, class=CHASSIS, reason=FPC 10 Minor Errors
Jul 30 18:58:40 Router craftd[13404]: Minor alarm set, FPC 10 Minor Errors
Jul 30 18:58:40 Router chassisd[13400]: CHASSISD_FPC_ASIC_ERROR: <FPC 10> ASIC Error detected errorno 0x00010002 (null)

Jul 30 18:58:40 Router fpc10 PFE-3 LU-0 TOE-0 Coredump Scheduled for thread mask 0x1 in 9361 usec (9277+84) 
Jul 30 18:58:40 Router fpc10 ** TOE ERROR 0x14 DETECTED IN PFE 3 TOE LU Stats 
Jul 30 18:58:41 Router fpc10 CMError: /fpc/10/pfe/0/cm/0/TOE-LU-3:0:0/3/TOE_ERR_INSTRUCTION_READ[0] (0x10004), scope: pfe, category: processing, severity: minor, module: TOE-LU-3:0:0, type: LU_TOE Instruction Read Minor error 
Jul 30 18:58:41 Router tnp.tftpd[96942]: TFTPD_RECVCOMPLETE_INFO: Received 30 blocks of 1024 size for file '/var/tmp/core-FPC10-PFE3-LU0-TOE0.0'
Jul 30 18:58:42 Router fpc10 Performing action log for error /fpc/10/pfe/0/cm/0/TOE-LU-3:0:0/3/TOE_ERR_INSTRUCTION_READ[0] (0x10004) in module: TOE-LU-3:0:0 with scope: pfe category: processing level: minor 
Jul 30 18:58:42 Router fpc10 Performing action cmalarm for error /fpc/10/pfe/0/cm/0/TOE-LU-3:0:0/3/TOE_ERR_INSTRUCTION_READ[0] (0x10004) in module: TOE-LU-3:0:0 with scope: pfe category: processing level: minor 
Jul 30 18:58:43 Router fpc10 Cmerror Op Set: TOE-LU-3:0:0: TOE LU.3.0.0 : SetErr - thread 0 ifetch err (URI: /fpc/10/pfe/0/cm/0/TOE-LU-3:0:0/3/TOE_ERR_INSTRUCTION_READ[0]) 
Jul 30 18:58:43 Router mosquitto[13462]: Allocated node for mosq : 0x79f500, Client : client-2-re0-NA_periodic_subscriber-re0, topic : /1001/1/0, max bytes in queue : 10485760, hash_size is 500, hashIndex is 0x1214000
Jul 30 18:58:43 Router fpc10 CMError: /fpc/10/pfe/0/cm/0/TOE-LU-3:0:0/3/TOE_ERR_MEM_READ_ERR[0] (0x10002), scope: pfe, category: processing, severity: minor, module: TOE-LU-3:0:0, type: LU_TOE Memory Read Minor error 
Jul 30 18:58:44 Router fpc10 Performing action log for error /fpc/10/pfe/0/cm/0/TOE-LU-3:0:0/3/TOE_ERR_MEM_READ_ERR[0] (0x10002) in module: TOE-LU-3:0:0 with scope: pfe category: processing level: minor 
Jul 30 18:58:44 Router fpc10 Cmerror Op Set: TOE-LU-3:0:0: TOE LU.3.0.0 : SetErr - LU TOE chip 48 Memory Error Thread 0, INSTR Addr 0x1564 (URI: /fpc/10/pfe/0/cm/0/TOE-LU-3:0:0/3/TOE_ERR_MEM_READ_ERR[0]) 
Jul 30 18:58:45 Router fpc10 Starting Coredump /var/tmp/core-FPC10-PFE3-LU0-TOE0 
Jul 30 18:58:45 Router fpc10 Finished Coredump /var/tmp/core-FPC10-PFE3-LU0-TOE0 
Jul 30 18:58:45 Router fpc10 ucode running: TRINITY_LU-TOE pfe 3 toe 0 toe_idx 48 
Jul 30 18:58:45 Router fpc10 toe_lu_stats_recover:1226 RECOVERY in PFE 3 TOE LU Stats successful 
Jul 30 18:58:45 Router fpc10 trinity_pio: 1 PIO errors occurred 
Jul 30 18:58:45 Router fpc10 trinity_pio: Last error: 13  LU-3 Trinity PCI   0x01081180 Read PCIe  4  

 

 

 

Solution

Next Actions:

To clear this error, use the following command:

clear chassis fpc errors fpc-slot 10 all

 

If the error reappears even after executing the clear command, reset the FPC by using the following command under a MW.

request chassis fpc slot 10 offline
request chassis fpc slot 10 online

 

 

This should clear the alarm. If you still see the alarm, then raise a case with JTAC for RMA.

 

Modification History

2024-08-01 : Article Created