This article explains the meaning of TOE_ERR_TX_BLOCKED error on AFEB0 in MX104 and the steps to address it
TOE or Trinity Offload Engine is a mechanism which moves data between ASICs and the FPC microkernel. This helps in the interaction of the FPC with the internal chips and the hardware on the underlying chip. This TOE error usually occurs due to memory error in any other chip example: XM chip, MQ chip, etc.., An issue with the TOE functionality will cause the thread that moves data between ASIC and FPC to be blocked/hampered and that affects the traffic flow from FPC towards the ASIC. The nature of this error is purely stochastic and can recover post restart/reseat in ideal cases.
In the 'show log messages', review the events that occurred at or just before the appearance of the TOE error message. Frequently these events help identify the cause.
In the below scenario, MX104 afeb 0 encountered hsl2 link errors towards fpc 0 pfe 0, resulting in hostloopback wedge. This eventually led to TOE_ERR_TX_BLOCKED. Here is a snippet of the logs for reference:
Apr 24 20:51:19 mx104 chassisd[2077]: %DAEMON-3-CHASSISD_FPC_ASIC_ERROR: <FPC 0> ASIC Error detected errorno 0x00200001 (null)
Apr 24 20:51:19 mx104 : %PFE-5: afeb0 Error: /fpc/0/pfe/0/cm/0/HSL2/0/CMTFPC_REGISTER_HSL2_FAULT[0] (0x200001), scope: board, category: functional, severity: major, module: HSL2, type: HSL2 Failure in PFE 0
Apr 24 20:51:19 mx104 : %PFE-5: afeb0 Performing action log for error /fpc/0/pfe/0/cm/0/HSL2/0/CMTFPC_REGISTER_HSL2_FAULT[0] (0x200001) in module: HSL2 with scope: board category: functional level: major
Apr 24 20:51:19 mx104 tnp.tftpd[3841]: %DAEMON-6-TFTPD_CONNECT_INFO: TFTP write from address 50 port 3 file /var/tmp/pfe_debug_info_MX104-ABB-0
Apr 24 20:51:20 mx104 : %PFE-5: afeb0 Performing action get-state for error /fpc/0/pfe/0/cm/0/HSL2/0/CMTFPC_REGISTER_HSL2_FAULT[0] (0x200001) in module: HSL2 with scope: board category: functional level: major
Apr 24 20:51:20 mx104 : %PFE-3: afeb0 LUCHIP(0): Secondary PPE 5 zone 23 timeout.
Apr 24 20:51:46 mx104 : %PFE-3: afeb0 Cmerror Op Set: HSL2: HSL2 Fault IXCHIP(0)-chan-rx-0 <= MQCHIP(0)-chan-tx-41 - set (URI: /fpc/0/pfe/0/cm/0/HSL2/0/CMTFPC_REGISTER_HSL2_FAULT[0])
Apr 24 20:51:52 mx104 : %PFE-5: afeb0 Performing action log for error /fpc/0/pfe/0/cm/0/Host_Loopback/0/HOST_LOOPBACK_MAKE_CMERROR_ID[0] (0x20001) in module: Host Loopback with scope: pfe category: functional level: major
Apr 24 20:51:52 mx104 : %PFE-5: afeb0 Performing action get-state for error /fpc/0/pfe/0/cm/0/Host_Loopback/0/HOST_LOOPBACK_MAKE_CMERROR_ID[0] (0x20001) in module: Host Loopback with scope: pfe category: functional level: major
Apr 24 20:51:52 mx104 : %PFE-5: afeb0 Performing action cmalarm for error /fpc/0/pfe/0/cm/0/Host_Loopback/0/HOST_LOOPBACK_MAKE_CMERROR_ID[0] (0x20001) in module: Host Loopback with scope: pfe category: functional level: major
Apr 24 20:51:52 mx104 : %PFE-3: afeb0 Cmerror Op Set: Host Loopback: G٫0@^C(URI: /fpc/0/pfe/0/cm/0/Host_Loopback/0/HOST_LOOPBACK_MAKE_CMERROR_ID[0])
Apr 24 20:51:52 mx104 : %PFE-3: afeb0 LUCHIP(0): Secondary PPE 3 zone 20 timeout.
Apr 24 20:51:52 mx104 : %PFE-3: afeb0 PPE Thread Timeout Trap: Count 7, PC 3f2, 0x03f2: select_continuation_processing
Apr 24 20:51:52 mx104 : %PFE-3: afeb0 LUCHIP(0) PPE_3 Errors thread timeout error
Apr 24 20:51:52 mx104 : %PFE-3: afeb0 LUCHIP(0) PPE_6 Errors thread timeout error
Apr 24 20:51:51 mx104 rpd[2084]: %DAEMON-5-RPD_RT_HWM_NOTICE: New RIB highwatermark for routes: 307893 [2024-04-24 20:51:21]
Apr 24 20:51:52 mx104 : %PFE-3: afeb0 LUCHIP(0) PPE_12 Errors thread timeout error
Apr 24 20:51:53 mx104 : %PFE-5: afeb0 Error: /fpc/0/pfe/0/cm/0/TOE-MQ-0:0:0/0/TOE_ERR_TX_BLOCKED[1] (0x10409), scope: pfe, category: processing, severity: major, module: TOE-MQ-0:0:0, type: MQ_TOE TX Blocked Major error
Apr 24 20:51:53 mx104 : %PFE-5: afeb0 Performing action log for error /fpc/0/pfe/0/cm/0/TOE-MQ-0:0:0/0/TOE_ERR_TX_BLOCKED[1] (0x10409) in module: TOE-MQ-0:0:0 with scope: pfe category: processing level: major
Apr 24 20:51:53 mx104 : %PFE-5: afeb0 Performing action get-state for error /fpc/0/pfe/0/cm/0/TOE-MQ-0:0:0/0/TOE_ERR_TX_BLOCKED[1] (0x10409) in module: TOE-MQ-0:0:0 with scope: pfe category: processing level: major
Apr 24 20:51:54 mx104 : %PFE-5: afeb0 Performing action cmalarm for error /fpc/0/pfe/0/cm/0/TOE-MQ-0:0:0/0/TOE_ERR_TX_BLOCKED[1] (0x10409) in module: TOE-MQ-0:0:0 with scope: pfe category: processing level: major
Apr 24 20:51:54 mx104 : %PFE-3: afeb0 Cmerror Op Set: TOE-MQ-0:0:0: TOE MQ.0.0.0 : SetErr - ** WEDGE DETECTED IN PFE 0 stream 0 TOE host packet transfer: TOE to Asic path blocked (code 0x9) (URI: /fpc/0/pfe/0/cm/0/TOE-MQ-0:0:0/0/TOE_ERR_TX_BLOCKED[1])
Apr 24 20:51:55 mx104 : %PFE-5: afeb0 Error: /fpc/0/pfe/0/cm/0/Host_Loopback/0/HOST_LOOPBACK_MAKE_CMERROR_ID[0] (0x20001), scope: pfe, category: functional, severity: major, module: Host Loopback, type: Host Loopback Path Id 0
Apr 24 20:51:55 mx104 : %PFE-5: afeb0 Performing action log for error /fpc/0/pfe/0/cm/0/Host_Loopback/0/HOST_LOOPBACK_MAKE_CMERROR_ID[0] (0x20001) in module: Host Loopback with scope: pfe category: functional level: major
labroot@mx104> start shell pfe network afeb0
MX platform (1000Mhz QorIQ P2020 processor, 2048MB memory, 512KB flash)
MX104-ABB-0(mx104 vty)# sh nvram
System NVRAM :
32751 available bytes, 32751 used, 0 free
Contents:
MPC: Reset reason: RE Initiated Reset
[LOG] SCHED: Thread 89 (PFE Manager) aborted, hogged 2505 ms
[Apr 25 04:43:43.940 LOG: Debug] <167>Cmerror: processing the task op_type 1 for scope 0 and category 0 level 1 level_count 5 occur_count 5 clear_count 0 level_threshold 1 level_action 0x7
item errid 2097153 item_threshold 1 item_count 5 item_sub_err_state 0 sub_item errid 0 sub_[Apr 25 04:43:43.940 LOG: Debug] <167>Cmerror: Level 1 count increment 6 occur_count 6 clear_count 0
[Apr 25 04:43:43.940 LOG: Notice] <165>Error: /fpc/0/pfe/0/cm/0/HSL2/0/CMTFPC_REGISTER_HSL2_FAULT[0] (0x200001), scope: board, category: functional, severity: major, module: HSL2, type: HSL2 Failure in PFE 0
[Apr 25 04:43:43.940 LOG: Debug] <167>Cmerror: Level 1 count 6 (occur_count 6 clear_count 0)crossed threshold 1 action 0x7
[Apr 25 04:43:43.940 LOG: Debug] <167>alarm count for level 1 is incremented and set to 6
[Apr 25 04:43:43.940 LOG: Notice] <165>Performing action log for error /fpc/0/pfe/0/cm/0/HSL2/0/CMTFPC_REGISTER_HSL2_FAULT[0] (0x200001) in module: HSL2 with scope: board category: functional level: major
[Apr 25 04:43:43.940 LOG: Debug] <167>cmerror_take_action_helper: performing action 1 for scope 0 category 0 level 1 err_id /fpc/0/pfe/0/cm/0/HSL2/0/CMTFPC_REGISTER_HSL2_FAULT[0](0x200001) module id 10
[Apr 25 04:43:43.940 LOG: Notice] <165>Performing action get-state for error /fpc/0/pfe/0/cm/0/HSL2/0/CMTFPC_REGISTER_HSL2_FAULT[0] (0x200001) in module: HSL2 with scope: board category: functional level: major
[Apr 25 04:43:43.940 LOG: Debug] <167>cmerror_take_action_helper: performing action 2 for scope 0 category 0 level 1 err_id /fpc/0/pfe/0/cm/0/HSL2/0/CMTFPC_REGISTER_HSL2_FAULT[0](0x200001) module id 10
[Apr 25 04:43:43.940 LOG Notice] <165>Performing action cmalarm for error /fpc/0/pfe/0/cm/0/HSL2/0/CMTFPC_REGISTER_HSL2_FAULT[0] (0x200001) in module: HSL2 with scope: board category: functional level: major
[Apr 25 04:43:43.940 LOG: Debug] <167>cmerror_take_action_helper: performing action 4 for scope 0 category 0 level 1 err_id /fpc/0/pfe/0/cm/0/HSL2/0/CMTFPC_REGISTER_HSL2_FAULT[0](0x200001) module id 10
[Apr 25 04:43:43.940 LOG: Err] <163>Cmerror Op Set: HSL2: HSL2 Fault MQCHIP(0)-chan-rx-3 <= LUCHIP(0)-chan-tx-3 - set (URI: /fpc/0/pfe/0/cm/0/HSL2/0/CMTFPC_REGISTER_HSL2_FAULT[0])
[Apr 25 04:43:43.940 LOG: Debug] <167>cmerror_process_queue: module = HSL2
[Apr 25 04:43:43.940 LOG: Debug] <167>Cmerror: processing the task op_type 1 for scope 0 and category 0 level 1 level_count 6 occur_count 6 clear_count 0 level_threshold 1 level_action 0x7
item errid 2097153 item_threshold 1 item_count 6 item_sub_err_state 0 sub_item errid 0 sub_
[Apr 25 04:43:43.940 LOG: Debug] <167>Cmerror: Level 1 count increment 7 occur_count 7 clear_count 0
[Apr 25 04:43:43.940 LOG: Debug] <167>Cmerror: Level 1 count 7 (occur_count 7 clear_count 0)crossed threshold 1 action 0x7
[Apr 25 04:43:43.940 LOG: Debug] <167>alarm count for level 1 is incremented and set to 7
[Apr 25 04:43:43.940 LOG: Notice] <165>Performing action cmalarm for error /fpc/0/pfe/0/cm/0/HSL2/0/CMTFPC_REGISTER_HSL2_FAULT[0] (0x200001) in module: HSL2 with scope: board category: functional level: major
[Apr 25 04:43:43.941 LOG: Debug] <167>cmerror_take_action_helper: performing action 4 for scope 0 category 0 level 1 err_id /fpc/0/pfe/0/cm/0/HSL2/0/CMTFPC_REGISTER_HSL2_FAULT[0](0x200001) module id 10
[Apr 25 04:43:43.941 LOG: Err] <163>Cmerror Op Set: HSL2: HSL2 Fault MQCHIP(0)-chan-rx-13 <= IXCHIP(0)-chan-tx-3 - set (URI: /fpc/0/pfe/0/cm/0/HSL2/0/CMTFPC_REGISTER_HSL2_FAULT[0])
[Apr 25 04:43:43.941 LOG: Debug] <167>cmerror_process_queue: module = HSL2
[Apr 25 04:43:43.941 LOG: Debug] <167>Cmerror: processing the task op_type 1 for scope 0 and category 0 level 1 level_count 7 occur_count 7 clear_count 0 level_threshold 1 level_action 0x7
item errid 2097153 item_threshold 1 item_count 7 item_sub_err_state 0 sub_item errid 0 sub_
[Apr 25 04:43:43.941 LOG: Debug] <167>Cmerror: Level 1 count increment 8 occur_count 8 clear_count 0
[Apr 25 04:43:43.941 LOG: Notice] <165>Error: /fpc/0/pfe/0/cm/0/HSL2/0/CMTFPC_REGISTER_HSL2_FAULT[0] (0x200001), scope: board, category: functional, severity: major, module: HSL2, type: HSL2 Failure in PFE 0
[Apr 25 04:43:43.941 LOG: Debug] <167>Cmerror: Level 1 count 8 (occur_count 8 clear_count 0)crossed threshold 1 action 0x7
[Apr 25 04:43:43.941 LOG: Debug] <167>alarm count for level 1 is incremented and set to 8
[Apr 25 04:43:43.941 LOG: Notice] <165>Performing action log for error /fpc/0/pfe/0/cm/0/HSL2/0/CMTFPC_REGISTER_HSL2_FAULT[0] (0x200001) in module: HSL2 with scope: board category: functional level: major
[Apr 25 04:43:43.941 LOG: Debug] <167>cmerror_take_action_helper: performing action 1 for scope 0 category 0 level 1 err_id /fpc/0/pfe/0/cm/0/HSL2/0/CMTFPC_REGISTER_HSL2_FAULT[0](0x200001) module id 10
[Apr 25 04:43:43.941 LOG: Notice] <165>Performing action get-state for error /fpc/0/pfe/0/cm/0/HSL2/0/CMTFPC_REGISTER_HSL2_FAULT[0] (0x200001) in module: HSL2 with scope: board category: functional level: major
[Apr 25 04:43:43.941 LOG: Debug] <167>cmerror_take_action_helper: performing action 2 for scope 0 category 0 level 1 err_id /fpc/0/pfe/0/cm/0/HSL2/0/CMTFPC_REGISTER_HSL2_FAULT[0](0x200001) module id 10
[Apr 25 04:43:43.941 LOG: Notice] <165>Performing action cmalarm for error /fpc/0/pfe/0/cm/0/HSL2/0/CMTFPC_REGISTER_HSL2_FAULT[0] (0x200001) in module: HSL2 with scope: board category: functional level: major
[Apr 25 04:43:43.941 LOG: Err] <163>Cmerror Op Set: HSL2: HSL2 Fault IXCHIP(0)-chan-rx-0 <= MQCHIP(0)-chan-tx-41 - set (URI: /fpc/0/pfe/0/cm/0/HSL2/0/CMTFPC_REGISTER_HSL2_FAULT[0])
[Apr 25 04:43:43.941 LOG: Debug] <167>Cmerror: Draining ASIC error message queue
[Apr 25 04:43:45.265 LOG: Err] <163>LUCHIP(0): Secondary PPE 8 zone 10 timeout.
[Apr 25 04:43:45.265 LOG: Err] <163>LUCHIP(0) PPE_8 Errors thread timeout error
[Apr 25 04:43:45.265 LOG: Err] <163>LUCHIP(0) PPE_9 Errors thread timeout error
[Apr 25 04:43:45.265 LOG: Err] <163>LUCHIP(0) PPE_15 Errors thread timeout error
[Apr 25 04:43:46.285 LOG: Err] <163>LUCHIP(0): Secondary PPE 8 zone 10 timeout.
[Apr 25 04:43:46.285 LOG: Err] <163>LUCHIP(0) PPE_8 Errors thread timeout error
[Apr 25 04:43:46.285 LOG: Err] <163>LUCHIP(0) PPE_9 Errors thread timeout error
[Apr 25 04:43:46.285 LOG: Err] <163>LUCHIP(0) PPE_15 Errors thread timeout error
[Apr 25 04:43:46.460 LOG: Debug] <167>Cmerror: Draining ASIC error message queue
[Apr 25 04:43:46.460 LOG: Debug] <167>cmerror_process_queue: module = TOE-MQ-0:0:0
[Apr 25 04:43:46.460 LOG: Debug] <167>Cmerror: processing the task op_type 1 for scope 1 and category 5 level 1 level_count 0 occur_count 0 clear_count 0 level_threshold 1 level_action 0x7
item errid 66569 item_threshold 1 item_count 0 item_sub_err_state 0 sub_item errid 0 sub_it
[Apr 25 04:43:46.460 LOG: Debug] <167>Cmerror: Level 1 count increment 1 occur_count 1 clear_count 0
[Apr 25 04:43:46.460 LOG: Notice] <165>Error: /fpc/0/pfe/0/cm/0/TOE-MQ-0:0:0/0/TOE_ERR_TX_BLOCKED[1] (0x10409), scope: pfe, category: processing, severity: major, module: TOE-MQ-0:0:0, type: MQ_TOE TX Blocked Major error
[Apr 25 04:43:46.460 LOG: Debug] <167>Cmerror: Level 1 count 1 (occur_count 1 clear_count 0)crossed threshold 1 action 0x7
[Apr 25 04:43:46.460 LOG: Debug] <167>alarm count for level 1 is incremented and set to 9
[Apr 25 04:43:46.460 LOG: Notice] <165>Performing action log for error /fpc/0/pfe/0/cm/0/TOE-MQ-0:0:0/0/TOE_ERR_TX_BLOCKED[1] (0x10409) in module: TOE-MQ-0:0:0 with scope: pfe category: processing level: major
[Apr 25 04:43:46.460 LOG: Debug] <167>cmerror_take_action_helper: performing action 1 for scope 1 category 5 level 1 err_id /fpc/0/pfe/0/cm/0/TOE-MQ-0:0:0/0/TOE_ERR_TX_BLOCKED[1](0x10409) module id 9
[Apr 25 04:43:46.460 LOG: Notice] <165>Performing action get-state for error /fpc/0/pfe/0/cm/0/TOE-MQ-0:0:0/0/TOE_ERR_TX_BLOCKED[1] (0x10409) in module: TOE-MQ-0:0:0 with scope: pfe category: processing level: major
[Apr 25 04:43:46.460 LOG: Debug] <167>cmerror_take_action_helper: performing action 2 for scope 1 category 5 level 1 err_id /fpc/0/pfe/0/cm/0/TOE-MQ-0:0:0/0/TOE_ERR_TX_BLOCKED[1](0x10409) module id 9
[Apr 25 04:43:46.460 LOG: Notice] <165>Performing action cmalarm for error /fpc/0/pfe/0/cm/0/TOE-MQ-0:0:0/0/TOE_ERR_TX_BLOCKED[1] (0x10409) in module: TOE-MQ-0:0:0 with scope: pfe category: processing level: major
[Apr 25 04:43:46.460 LOG: Debug] <167>cmerror_take_action_helper: performing action 4 for scope 1 category 5 level 1 err_id /fpc/0/pfe/0/cm/0/TOE-MQ-0:0:0/0/TOE_ERR_TX_BLOCKED[1](0x10409) module id 9
[Apr 25 04:43:46.460 LOG: Err] <163>Cmerror Op Set: TOE-MQ-0:0:0: TOE MQ.0.0.0 : SetErr - ** WEDGE DETECTED IN PFE 0 stream 0 TOE host packet transfer: TOE toAsic path blocked (code 0x9)
(URI: /fpc/0/pfe/0/cm/0/TOE-MQ-0:0:0/0/TOE_ERR_TX_BLOCKED[1])
[Apr 25 04:43:47.285 LOG: Err] <163>LUCHIP(0): Secondary PPE 8 zone 10 timeout.
[Apr 25 04:43:47.285 LOG: Err] <163>LUCHIP(0) PPE_8 Errors thread timeout error
[Apr 25 04:43:47.285 LOG: Err] <163>LUCHIP(0) PPE_9 Errors thread timeout error
[Apr 25 04:43:47.285 LOG: Err] <163>LUCHIP(0) PPE_15 Errors thread timeout error
[Apr 25 04:43:48.285 LOG: Err] <163>LUCHIP(0): Secondary PPE 8 zone 10 timeout.
[Apr 25 04:43:48.285 LOG: Err] <163>LUCHIP(0) PPE_8 Errors thread timeout error
[Apr 25 04:43:48.285 LOG: Err] <163>LUCHIP(0) PPE_9 Errors thread timeout error
[Apr 25 04:43:48.285 LOG: Err] <163>LUCHIP(0) PPE_15 Errors thread timeout error
[Apr 25 04:43:49.285 LOG: Err] <163>LUCHIP(0): Secondary PPE 8 zone 10 timeout.
[Apr 25 04:43:49.285 LOG: Err] <163>LUCHIP(0) PPE_8 Errors thread timeout error
[Apr 25 04:43:49.285 LOG: Err] <163>LUCHIP(0) PPE_9 Errors thread timeout error
[Apr 25 04:43:49.285 LOG: Err] <163>LUCHIP(0) PPE_15 Errors thread timeout error
[Apr 25 04:43:50.290 LOG: Err] <163>LUCHIP(0): Secondary PPE 8 zone 10 timeout.
[Apr 25 04:43:50.290 LOG: Err] <163>LUCHIP(0) PPE_8 Errors thread timeout error
[Apr 25 04:43:50.290 LOG: Err] <163>LUCHIP(0) PPE_9 Errors thread timeout error
[Apr 25 04:43:50.290 LOG: Err] <163>LUCHIP(0) PPE_15 Errors thread timeout error
[Apr 25 04:43:50.466 LOG: Info] <166>ifp xe-2/0/0 ifd_mdown: 355160 ms
[Apr 25 04:43:50.659 LOG: Info] <166>mic_mac_periodic:xe-2/0/0: ifd_mup
[Apr 25 04:43:51.290 LOG: Err] <163>LUCHIP(0): Secondary PPE 8 zone 10 timeout.
[Apr 25 04:43:51.290 LOG: Err] <163>LUCHIP(0) PPE_8 Errors thread timeout error
[Apr 25 04:43:51.290 LOG: Err] <163>LUCHIP(0) PPE_9 Errors thread timeout error
[Apr 25 04:43:51.290 LOG: Err] <163>LUCHIP(0) PPE_15 Errors thread timeout error
[Apr 25 04:43:52.290 LOG: Err] <163>LUCHIP(0): Secondary PPE 8 zone 10 timeout.
[Apr 25 04:43:53.290 LOG: Err] <163>LUCHIP(0): Secondary PPE 8 zone 10 timeout.
[Apr 25 04:43:54.136 LOG: Info] <166>ifp xe-2/0/0 ifd_mdown: 358830 ms
[Apr 25 04:43:54.155 LOG: Info] <166>ifp xe-2/0/1 ifd_mdown: 358850 ms
[Apr 25 04:43:54.234 LOG: Err] <163>MQ PCI Fatal Error detected.
[Apr 25 04:43:54.234 LOG: Err] <163>MQ(0): pio_handle(0x48b42db8); pio_read_u32() failed: 20(input/output error)! addr=00000000070ff000
[Apr 25 04:43:54.234 LOG: Err] <163>pfe_ifd_link_updown: Handling IFD link DOWN failed - status 20, ifd xe-2/0/0
System Exception: Vector/Code 0x00900, Signal -1
Event occurred at: Apr 25 04:43:56.827492
Juniper Embedded Microkernel Version 20.4R3-S2.6
Built by builder on 2022-02-06 17:53:14 UTC
Copyright (C) 1998-2022, Juniper Networks, Inc.
All rights reserved.
Reason string: "Panic"
Context: Thread (PFE Manager)
Registers:
R00: 0x00000001 R01: 0x48afc4f0 R02: 0x48af48b8 R03: 0x45b46ca0
R04: 0x00000000 R05: 0x48afc668 R06: 0x00000000 R07: 0x00000000
R08: 0x010892a8 R09: 0x45b46c60 R10: 0x010892a9 R11: 0xe2000000
R12: 0x00fffc00 R13: 0x804719d7 R14: 0x00000000 R15: 0x44c35b18
R16: 0x48afc560 R17: 0x00000001 R18: 0x48afc668 R19: 0x43a86398
R20: 0x00000000 R21: 0x00000020 R22: 0x00000000 R23: 0x00000001
R24: 0x48b3fab8 R25: 0x45b46d00 R26: 0x45b46ce0 R27: 0x48afc668
R28: 0x00000000 R29: 0x00000004 R30: 0x00000000 R31: 0x48b3fab8
MSR: 0x00029200 CTR: 0x4003a704 Link: 0x40aa25f4 SP: 0x48afc4f0
CR: 0x28884022 XER: 0x00000000 DEAR: 0x00000000 PC: 0x40aa287c
ESR: 0x00000000 K_MSR: 0x00021200
Stack Traceback:
Frame 01: sp = 0x48afc4f0, pc = 0x40aa25f4
Frame 02: sp = 0x48afc518, pc = 0x40aa1fe0
Frame 03: sp = 0x48afc548, pc = 0x40e39a80
Frame 04: sp = 0x48afc640, pc = 0x40e06a78
Frame 05: sp = 0x48afc660, pc = 0x40e30628
Frame 06: sp = 0x48afc688, pc = 0x40dd9a80
Frame 07: sp = 0x48afc6e0, pc = 0x42920740
Frame 08: sp = 0x48afc6f0, pc = 0x40135608
Frame 09: sp = 0x48afc718, pc = 0x400fb78c
Frame 10: sp = 0x48afc740, pc = 0x400c87a0
Frame 11: sp = 0x48afc758, pc = 0x400d6134
Frame 12: sp = 0x48afc7c0, pc = 0x400d61d4
Frame 13: sp = 0x48afc7d0, pc = 0x400d6250
Frame 14: sp = 0x48afc7e0, pc = 0x41a43700
Frame 15: sp = 0x48afc8b8, pc = 0x4003355c
[LOG] syslog called with interrupts off (caller pc:0x40037148)
[LOG] Dumping core-MX104-ABB-0 to 1
[LOG] Coredump finished!
[LOG] altius_pcieswitch_preconfig: Ignoring For Altius for now
[LOG] Set the IP IRI for table #1 to 0x80000032
[LOG] IPV4 Init: Set the IP IRI to 0x80000032
[LOG] igmp_pfe_thread:541 starting IGMP thread with pid 7 ...
[LOG] GN2405: JSPEC V 1.0 Module Init.
[LOG] WP module init
[LOG] otn_priodic ON
[LOG] ddos_issu_helper_register: issu state is 0 at ddos startup
[LOG] ddos sock support init ...
[LOG] ddos sock connection proto in use 4 ...
[LOG] RSMON rsmon_msg_thread_init
[LOG] if_module_init: Zeroing out jam vectors
[LOG] pic_jam_plugin_module_init: PIC toolkit module init
[LOG] pic_jam_plugin_module_init: PIC toolkit MQ jam vector init
ROM NVRAM:
0 available bytes, 0 used, 0 free
MX104-ABB-0(mx104 vty)# exit
This error is due to a transient hardware issue observed in afeb 0.
Follow below steps to address the error:
1. Collect below outputs for analysis:
2. During MW, restart afeb 0 to clear the TOE wedge condition:
request chassis afeb 0 offline
*** 30 secs wait***
request chassis afeb 0 online
3. If errors continue to occur even after afeb 0 restart, please open a case with JTAC with logs collected in step 1
If similar errors are seen for any other MX_series FPC models, then refer to below KB's on how to address it:
https://supportportal.juniper.net/s/article/MX-FPC-x-Major-Error-TOE-Error-code-0x10404
https://supportportal.juniper.net/s/article/MXTOE-ERR-TX-BLOCKED1-error-0x010409