How to troubleshoot PE Error code: 0x210202
root@QFX10008_RE0> show log messages | match 210202 Apr 10 10:50:56 QFX10008_RE0 craftd[5394]: Major alarm set, FPC 1 Major Errors - PE Error code: 0x210202 Apr 10 10:52:36 QFX10008_RE0 craftd[5394]: Major alarm cleared, FPC 1 Major Errors - PE Error code: 0x210202 Apr 10 11:29:33 QFX10008_RE0 craftd[5394]: Major alarm set, FPC 1 Major Errors - PE Error code: 0x210202 Apr 10 11:31:14 QFX10008_RE0 craftd[5394]: Major alarm cleared, FPC 1 Major Errors - PE Error code: 0x210202 Apr 10 12:52:17 QFX10008_RE0 craftd[5394]: Major alarm set, FPC 1 Major Errors - PE Error code: 0x210202 Apr 10 12:53:57 QFX10008_RE0 craftd[5394]: Major alarm cleared, FPC 1 Major Errors - PE Error code: 0x210202 Apr 10 12:54:34 QFX10008_RE0 craftd[5394]: Major alarm set, FPC 1 Major Errors - PE Error code: 0x210202 Apr 10 12:56:59 QFX10008_RE0 craftd[5394]: Major alarm cleared, FPC 1 Major Errors - PE Error code: 0x210202 Apr 10 13:21:31 QFX10008_RE0 craftd[5394]: Major alarm set, FPC 1 Major Errors - PE Error code: 0x210202
On QFX10008 platforms, an FPC major alarm might be seen indicating a transient cache read error when accessing DLU entry (Destination lookup). The following error messages might be seen, and the alarm might be cleared without intervention.
root@QFX10008_RE0> show log messages | find "Jan 10 10:50" Apr 10 10:50:56 QFX10008_RE0 fpc1 fpc1 dcpfe: Cmerror Op Set: PE Chip: PE1[1]: DLU: elp memory cache error#012 Apr 10 10:50:56 QFX10008_RE0 fpc1 Cmerror Op Set: PE Chip: PE1[1]: DLU: elp memory cache error Apr 10 10:50:56 QFX10008_RE0 alarmd[5393]: Alarm set: FPC color=RED, class=CHASSIS, reason=FPC 1 Major Errors - PE Error code: 0x210202 Apr 10 10:50:56 QFX10008_RE0 craftd[5394]: Major alarm set, FPC 1 Major Errors - PE Error code: 0x210202 Apr 10 10:50:59 QFX10008_RE0 fpc1 fpc1 dcpfe: Cmerror Op Set: PE Chip: PE1[1]: DLU: elp memory cache error#012 Apr 10 10:50:59 QFX10008_RE0 fpc1 Cmerror Op Set: PE Chip: PE1[1]: DLU: elp memory cache error Apr 10 10:51:05 QFX10008_RE0 fpc1 fpc1 dcpfe: Cmerror Op Set: PE Chip: PE1[1]: DLU: elp memory cache error#012 Apr 10 10:51:05 QFX10008_RE0 fpc1 Cmerror Op Set: PE Chip: PE1[1]: DLU: elp memory cache error Apr 10 10:51:18 QFX10008_RE0 fpc1 Cmerror Op Set: PE Chip: PE1[1]: DLU: elp memory cache error Apr 10 10:51:17 QFX10008_RE0 fpc1 fpc1 dcpfe: Cmerror Op Set: PE Chip: PE1[1]: DLU: elp memory cache error#012 Apr 10 10:52:36 QFX10008_RE0 fpc1 Cmerror Op Clear: PE Chip: PE1[1]: DLU: elp clear memory cache error Apr 10 10:52:36 QFX10008_RE0 fpc1 Cmerror Op Clear: PE Chip: PE1[1]: DLU: elp clear memory cache error Apr 10 10:52:36 QFX10008_RE0 fpc1 fpc1 dcpfe: Cmerror Op Clear: PE Chip: PE1[1]: DLU: elp clear memory cache error#012 Apr 10 10:52:36 QFX10008_RE0 last message repeated 3 times Apr 10 10:52:36 QFX10008_RE0 fpc1 Cmerror Op Clear: PE Chip: PE1[1]: DLU: elp clear memory cache error Apr 10 10:52:36 QFX10008_RE0 fpc1 Cmerror Op Clear: PE Chip: PE1[1]: DLU: elp clear memory cache error Apr 10 10:52:36 QFX10008_RE0 alarmd[5393]: Alarm cleared: FPC color=RED, class=CHASSIS, reason=FPC 1 Major Errors - PE Error code: 0x210202 Apr 10 10:52:36 QFX10008_RE0 craftd[5394]: Receive FX craftd clear alarm message: color: Major class: 100 slot: 1 Apr 10 10:52:36 QFX10008_RE0 craftd[5394]: Major alarm cleared, FPC 1 Major Errors - PE Error code: 0x210202
You can check the frequency of this error by issuing the following cli command
root@QFX10008_RE0> show chassis fpc errors FPC Level Occurred Cleared Threshold Action-Taken Action 0 Minor 0 0 10 0 LOG| Major 0 0 1 0 CM ALARM| Fatal 0 0 1 0 CM ALARM| Pfe-State: pfe-0 -ENABLED | pfe-1 -ENABLED | pfe-2 -ENABLED | 1 Minor 0 0 10 0 LOG| Major 1038 1038 1 2289 CM ALARM| Fatal 0 0 1 0 CM ALARM| Pfe-State: pfe-0 -ENABLED | pfe-1 -ENABLED | pfe-2 -ENABLED | Note that this major error may also be accompanied by traffic loss.
The PEChip errors reported indicate a transient cache read error when accessing DLU entry (Destination lookup). The system tried to correct these on subsequent reads. Hence the alarms have cleared. If the error logs reappear or issue is reported, try the following to clear this alarm in a Maintenance Window:
Soft reset/reload the FPC slot X (request chassis fpc slot 1 restart)
Physically reseat (remove and insert) the FPC
Before reseating the card in the MW, collect the following: show pechip trapstats (run it 2-3 times) show cmerror module brief (to get the Module# for "PE Chip") show cmerror module X error 0x210202 pfe 0 (X is Module# for "PE Chip") show cmerror module X error 0x210202 pfe 1 show cmerror module X error 0x210202 pfe 2 show pechip X stats error dlu (X is the PE CHIP number where the error has occurred) Apr 10 10:51:17 QFX10008_RE0 fpc1 fpc1 dcpfe: Cmerror Op Set: PE Chip: PE1[1]: DLU: elp memory cache error#012 {master} root@QFX10008_RE0> start shell pfe network fpc1 Switching platform (1600 Mhz QorIQ P5020 processor, 3071MB memory, 8192KB flash) FPC1(QFX10008_RE0 vty)# show pechip trapstats DEV TRAPCODE count rate ========================================================================================= 0 ( 12) dlu.ucode.pkt_to_cpu 21401413 15 0 ( 71) igp.tcp_hdr_len 31 0 0 ( 165) irp.pdct.pol_drop_trapcode 7976354 0 0 ( 213) edf.pkt_chk.same_port_l2d 10998493 1 0 ( 252) sw.nh.discard_sampling_trap 13190 0 1 ( 4) dlu.ucode.discard 3 0 1 ( 12) dlu.ucode.pkt_to_cpu 1292431 0 1 ( 40) dlu.elp.hw_err 2668 0 1 ( 210) edf.pkt_chk_stp_blkd 10 0 1 ( 213) edf.pkt_chk.same_port_l2d 1582341 0 2 ( 4) dlu.ucode.discard 4 0 2 ( 12) dlu.ucode.pkt_to_cpu 18266477 13 2 ( 151) slu.stp_blocked 2 0 2 ( 186) irp.ttl_ig_ip4_expired 3 0 2 ( 210) edf.pkt_chk_stp_blkd 32 0 2 ( 213) edf.pkt_chk.same_port_l2d 16894409265 329989 2 ( 250) sw.egnh.cfg_discard 4 0 FPC1(QFX10008_RE0 vty)# show cmerror module brief --------------------------------------- Module Name Active Errors --------------------------------------- 1 Generic HMC 0 2 TOE-PE-0:0:0 0 3 TOE-PE-1:0:0 0 4 TOE-PE-0:0:1 0 5 TOE-PE-2:0:0 0 6 TOE-PE-1:0:1 0 7 TOE-PE-0:0:2 0 8 TOE-PE-2:0:1 0 9 TOE-PE-1:0:2 0 10 TOE-PE-0:0:3 0 11 TOE-PE-2:0:2 0 12 TOE-PE-1:0:3 0 13 TOE-PE-2:0:3 0 14 PE Chip 0 FPC1(QFX10008_RE0 vty)# show cmerror module 14 error 0x210202 pfe 1 Error-id : 0x210202 Description : PFE : 1 Level : Major Count : 0 Threshold : 1 Error Limit : 0 Occur Count : 1430 Clear Count : 1430 Last-occurred(ms ago) : 4936606 Logs: ---------------------------------------------------------- Index Time Sub-Err State Description ---------------------------------------------------------- 0 04/10/23 04:23:31 0 Clear PE1[1]: DLU: elp clear memory cache error 1 04/10/23 04:23:31 0 Clear PE1[1]: DLU: elp clear memory cache error 2 04/10/23 04:23:31 0 Clear PE1[1]: DLU: elp clear memory cache error 3 04/10/23 04:23:31 0 Clear PE1[1]: DLU: elp clear memory cache error 4 04/10/23 04:22:12 0 Set PE1[1]: DLU: elp memory cache error 5 04/10/23 04:22:00 0 Set PE1[1]: DLU: elp memory cache error 6 04/10/23 04:21:55 0 Set PE1[1]: DLU: elp memory cache error 7 04/10/23 04:21:52 0 Set PE1[1]: DLU: elp memory cache error 8 04/10/23 04:16:07 0 Clear PE1[1]: DLU: elp clear memory cache error 9 04/10/23 04:16:07 0 Clear PE1[1]: DLU: elp clear memory cache error 10 04/10/23 04:16:07 0 Clear PE1[1]: DLU: elp clear memory cache error 11 04/10/23 04:16:07 0 Clear PE1[1]: DLU: elp clear memory cache error 12 04/10/23 04:14:49 0 Set PE1[1]: DLU: elp memory cache error 13 04/10/23 04:14:36 0 Set PE1[1]: DLU: elp memory cache error 14 04/10/23 04:14:31 0 Set PE1[1]: DLU: elp memory cache error 15 04/10/23 04:14:28 0 Set PE1[1]: DLU: elp memory cache error 16 04/10/23 02:20:02 0 Clear PE1[1]: DLU: elp clear memory cache error 17 04/10/23 02:20:02 0 Clear PE1[1]: DLU: elp clear memory cache error 18 04/10/23 02:20:02 0 Clear PE1[1]: DLU: elp clear memory cache error 19 04/10/23 02:20:02 0 Clear PE1[1]: DLU: elp clear memory cache error 20 04/10/23 02:18:44 0 Set PE1[1]: DLU: elp memory cache error 21 04/10/23 02:18:32 0 Set PE1[1]: DLU: elp memory cache error 22 04/10/23 02:18:25 0 Set PE1[1]: DLU: elp memory cache error 23 04/10/23 02:18:22 0 Set PE1[1]: DLU: elp memory cache error 24 04/10/23 01:32:41 0 Clear PE1[1]: DLU: elp clear memory cache error 25 04/10/23 01:32:41 0 Clear PE1[1]: DLU: elp clear memory cache error 26 04/10/23 01:32:41 0 Clear PE1[1]: DLU: elp clear memory cache error 27 04/10/23 01:32:41 0 Clear PE1[1]: DLU: elp clear memory cache error 28 04/10/23 01:32:41 0 Clear PE1[1]: DLU: elp clear memory cache error 29 04/10/23 01:31:04 0 Set PE1[1]: DLU: elp memory cache error 30 04/10/23 01:31:01 0 Set PE1[1]: DLU: elp memory cache error 31 04/10/23 01:30:49 0 Set PE1[1]: DLU: elp memory cache error 32 04/10/23 01:30:43 0 Set PE1[1]: DLU: elp memory cache error 33 04/10/23 01:30:40 0 Set PE1[1]: DLU: elp memory cache error 34 04/10/23 01:18:12 0 Clear PE1[1]: DLU: elp clear memory cache error 35 04/10/23 01:18:12 0 Clear PE1[1]: DLU: elp clear memory cache error 36 04/10/23 01:18:12 0 Clear PE1[1]: DLU: elp clear memory cache error 37 04/10/23 01:18:12 0 Clear PE1[1]: DLU: elp clear memory cache error 38 04/10/23 01:18:12 0 Clear PE1[1]: DLU: elp clear memory cache error 39 04/10/23 01:16:20 0 Set PE1[1]: DLU: elp memory cache error FPC1(QFX10008_RE0 vty)# show pechip 1 stats error dlu periodic ***************DLU stats*************** Register Name index description value rate pe.dlu.ipipe.err.cfg_err 0 Configuration Error Register 0 0 pe.dlu.ilp.err.cfg_err_log 0 Configuration Error Log Register 0 0 pe.dlu.ilp.err.tcp_fail_log 0 TCP Fail Error Log Register 0 0 pe.dlu.ilp.err.eof_oflow_log 0 End Of Flow Fifo Overflow Log Register 0 0 pe.dlu.ilp.err.mc_err_log 0 MC Interface Error Log Register 0 0 pe.dlu.elp.err.cfg_err_log 0 Configuration Error Log Register 0 0 pe.dlu.elp.err.tcp_fail_log 0 TCP Fail Error Log Register 0 0 pe.dlu.elp.err.eof_oflow_log 0 End Of Flow Fifo Overflow Log Register 0 0 pe.dlu.elp.err.mc_err_log 0 MC Interface Error Log Register 2723 20