Description

Our qfx5110-48s-4c has failed and spitting out the below errors repeatedly, and dcpfe core was also generated.

 

ada1:ata0:0:1:0): Retrying command, 3 more tries remain

(ada1:ata0:0:1:0): WRITE_DMA. ACB: ca 00 62 29 1b 40 00 00 00 00 28 00

(ada1:ata0:0:1:0): CAM status: ATA Status Error

(ada1:ata0:0:1:0): ATA status: 41 (DRDY ERR), error: 04 (ABRT )

(ada1:ata0:0:1:0): RES: 41 04 62 29 1b 00 00 00 00 28 00

(ada1:ata0:0:1:0): Retrying command, 2 more tries remain

(ada1:ata0:0:1:0): WRITE_DMA. ACB: ca 00 62 29 1b 40 00 00 00 00 28 00

(ada1:ata0:0:1:0): CAM status: ATA Status Error

(ada1:ata0:0:1:0): ATA status: 41 (DRDY ERR), error: 04 (ABRT )

(ada1:ata0:0:1:0): RES: 41 04 62 29 1b 00 00 00 00 28 00

(ada1:ata0:0:1:0): Retrying command, 1 more tries remain

(ada1:ata0:0:1:0): WRITE_DMA. ACB: ca 00 62 29 1b 40 00 00 00 00 28 00

(ada1:ata0:0:1:0): CAM status: ATA Status Error

 


 

 

Solution

1. DMA write errors were seen on console.

2. There DCPFE crash happened due to DMA errors and the fact that DCPFED could not access the disk.

3. The DCPFE restarted repeatedly unsuccessfully, citing ASIC initialization error, also generating a core dumps. So, the core-dumps mentioned ASIC initialization errors.

    ...(dcpfe getting repawned with the pid 14724)
        2024-05-01T23:57:58+00:00 dcpfe0 logger: fpc Stopping dcpfe: --------------------------->restarting dcpfe
        2024-05-01T23:57:58+00:00 dcpfe0 logger: has_lxc_platform is TRUE
        2024-05-01T23:57:58+00:00 dcpfe0 logger: fpc Starting dcpfe with option(s)[  -D ]: 
        2024-05-01T23:57:58+00:00 dcpfe0 dcpfe[14724]: Successfully Connected to syslog 
        2024-05-01T23:57:58+00:00 dcpfe0 dcpfe[14724]: tvp_get_cpu_speed_mhz: CPU speed 1800 MHz
        2024-05-01T23:58:00+00:00 dcpfe0 dcpfe[14724]: tvp_is_accton_type: Platform Name: QFX5_48X10G_4X40G_2P

    ...
        2024-05-01T23:58:07+00:00 dcpfe0 dcpfe[14724]: bcm_sdk_init: device ID: dev: 0xb862, rev: 0x2
        2024-05-01T23:58:07+00:00 dcpfe0 dcpfe[14724]: bcm_sdk_init:2978  unit:0,bus_type:2
        2024-05-01T23:58:07+00:00 dcpfe0 dcpfe[14724]: bcm_sdk_init: device unit no: 0

    ...
        2024-05-01T23:57:18+00:00 dcpfe0 dcpfe[14724]: bcore_init: after soc_reset_init   ------------------->when dcpfe restart happened it tried 3 times for asic intialization. As it was unsuccessful resulted in core-dump.
        2024-05-01T23:57:19+00:00 dcpfe0 dcpfe[14724]: bcore_init: before soc_reset_init
        2024-05-01T23:57:20+00:00 dcpfe0 dcpfe[14724]: bcore_init: after soc_reset_init
        2024-05-01T23:57:21+00:00 dcpfe0 dcpfe[14724]: bcore_init: before soc_reset_init
        2024-05-01T23:57:22+00:00 dcpfe0 dcpfe[14724]: bcore_init: after soc_reset_init


    ==> The crash got generated at the same time due to asic not getting detected.
                
        23:57 /var/crash/core-dcpfe-dcpfe0-0-1714607843.core.tgz

 

4. The issue was resolved when a power cycle restored the DMA error condition.

 

 

 

 

 

Modification History

2024-06-17 : Article Created

Related Information

View 1809372 - QFX5110 | dcpfe core | #4 0x0808d135 in panic (format_string=0xb9d6424 "Assertion failed:../../../../../src/pfe/common/drivers/dc-bcm-pic/dcbcm_sdk_core.c:3115: %s\n") at ../../../../../src/pfe/platform/tvp/tvp_ panic.c:71 - Bug database (juniper.net)