This Article outlines the recovery steps we should follow in the event that logs and alarms mentioned in the symptoms section are reported.
Hostname: labrouter.RD.SD-RE0
Model: mx10008
Junos: 21.4R3-S6.1
[email protected]> show system alarms
1 alarms currently active
Alarm time Class Description
2025-04-20 07:49:16 UTC Minor Potential slow peers are: XDPC1
FPC 1 REV 42 750-114437 XXXXX578 JNP10K-LC9600
Apr 20 07:43:36.159 2025 labrouter-NEW-fpc1 kernel: NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [Direct-root:12940]
Apr 20 07:43:36.162 2025 labrouter-NEW-fpc1 kernel: Modules linked in: uio_pci_hostif macvlan 8021q ebtable_filter amd64_edac_mod edac_core crct10dif_pclmul crct10dif_common jnx_pmb_nvram jnx_f1_bcpld jnx_cbios_debugfs k10temp snd_hda_intel snd_hda_codec snd_hda_core i2c_piix4 tmp401 max6697 ucd9000 ltc2945 tpm_rng ofpart vt7505 pmbus max20751 pmbus_core m25p80 spi_nor leds_supercon_led2 gpio_supercon_hotswap leds_supercon_ledx uio_pdrv_genirq jnx_mastership_supercon uio tpm_crb acpi_cpufreq sam_core mtdoops mtd i2c_dev gpio_sam gpio_ptxpmb_cpld
Apr 20 07:43:36.162 2025 labrouter-NEW-fpc1 kernel: CPU: 14 PID: 12940 Comm: Direct-root Not tainted 4.8.28-WR2.2.1_standard-gad6c183 #1
Apr 20 07:43:36.162 2025 labrouter-NEW-fpc1 kernel: Hardware name: Juniper Networks Inc. 0D88/0D7A, BIOS CASN_P_PMB1_00.29.01 09/12/2022
Apr 20 07:49:12.159 2025 labrouter-NEW-fpc1 kernel: NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [Direct-root:12940]
Apr 20 07:49:12.159 2025 labrouter-NEW-fpc1 kernel: Modules linked in: uio_pci_hostif macvlan 8021q ebtable_filter amd64_edac_mod edac_core crct10dif_pclmul crct10dif_common jnx_pmb_nvram jnx_f1_bcpld jnx_cbios_debugfs k10temp snd_hda_intel snd_hda_codec snd_hda_core i2c_piix4 tmp401 max6697 ucd9000 ltc2945 tpm_rng ofpart vt7505 pmbus max20751 pmbus_core m25p80 spi_nor leds_supercon_led2 gpio_supercon_hotswap leds_supercon_ledx uio_pdrv_genirq jnx_mastership_supercon uio tpm_crb acpi_cpufreq sam_core mtdoops mtd i2c_dev gpio_sam gpio_ptxpmb_cpld
Apr 20 07:49:12.159 2025 labrouter-NEW-fpc1 kernel: CPU: 14 PID: 12940 Comm: Direct-root Tainted: G L 4.8.28-WR2.2.1_standard-gad6c183 #1
Apr 20 07:49:12.159 2025 labrouter-NEW-fpc1 kernel: Hardware name: Juniper Networks Inc. 0D88/0D7A, BIOS CASN_P_PMB1_00.29.01 09/12/2022
Apr 20 07:49:12.159 2025 labrouter-NEW-fpc1 kernel: task: 00000000c286bcee task.stack: 0000000035312692
Apr 20 07:49:12.159 2025 labrouter-NEW-fpc1 kernel: RIP: 0010:[<ffffffff806b2057>] [<000000009e81c065>] copy_page+0x7/0x10
Apr 20 07:49:12.159 2025 labrouter-NEW-fpc1 kernel: RSP: 0000:ffffc90004bfbd68 EFLAGS: 00000286
Apr 20 07:49:12.159 2025 labrouter-NEW-fpc1 kernel: RAX: 0000160000000000 RBX: ffffea00084d6500 RCX: 00000000000001f8
Apr 20 07:49:12.159 2025 labrouter-NEW-fpc1 kernel: RDX: ffff8806ed79d4c0 RSI: ffff880213594040 RDI: ffff8804bc772040
Apr 20 07:49:12.159 2025 labrouter-NEW-fpc1 kernel: RBP: ffffc90004bfbdc8 R08: ffff88087eb9b9f8 R09: 0000000000000001
Apr 20 07:49:12.159 2025 labrouter-NEW-fpc1 kernel: R10: 00000000024200ca R11: 0000000000000081 R12: ffffea0012f1dc80
Apr 20 07:49:12.159 2025 labrouter-NEW-fpc1 kernel: R13: ffffc90004bfbe70 R14: ffff88044bd88cc0 R15: 8000000213594065
Apr 20 07:49:12.159 2025 labrouter-NEW-fpc1 kernel: FS: 00007f87835fb700(0000) GS:ffff88087eb80000(0000) knlGS:0000000000000000
Apr 20 07:49:12.159 2025 labrouter-NEW-fpc1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 20 07:49:12.159 2025 labrouter-NEW-fpc1 kernel: CR2: 00007f865584a000 CR3: 00000006ee5b4000 CR4: 00000000003406f0
Apr 20 07:49:12.160 2025 labrouter-NEW-fpc1 kernel: Stack:
Apr 20 07:49:12.160 2025 labrouter-NEW-fpc1 kernel: ffffffff803b9c80 00007f865584a000 00007f8655849000 ffff8806ed79d4c0
Apr 20 07:49:12.160 2025 labrouter-NEW-fpc1 kernel: 00007f865584a000 ffff880792d74ec0 ffff8807b66ba800 ffffea00084d6500
Apr 20 07:49:12.160 2025 labrouter-NEW-fpc1 kernel: ffffc90004bfbe70 8000000213594065 ffff88044bd88cc0 ffffc90004bfbe70
Apr 20 07:49:12.160 2025 labrouter-NEW-fpc1 kernel: Call Trace:
Apr 20 07:49:12.160 2025 labrouter-NEW-fpc1 kernel: [<00000000919b0b85>] ? wp_page_copy+0x110/0x710
Apr 20 07:49:12.160 2025 labrouter-NEW-fpc1 kernel: [<000000007df678ef>] do_wp_page+0x135/0x6a0
Apr 20 07:49:13.162 2025 labrouter.RD.SD-RE0 snmpd[11673]: LIBJSNMP_NS_LOG_INFO: INFO: AgentX session, /var/run/mib2d-38, noticed 1 request timeout.
Apr 20 07:49:12.160 2025 labrouter-NEW-fpc1 kernel: [<00000000b3df1a45>] handle_mm_fault+0x742/0xfe0
Apr 20 07:49:12.160 2025 labrouter-NEW-fpc1 kernel: [<00000000967fe668>] __do_page_fault+0x260/0x550
Apr 20 07:49:12.160 2025 labrouter-NEW-fpc1 kernel: [<000000007af2cf93>] do_page_fault+0xc/0x10
Apr 20 07:49:12.160 2025 labrouter-NEW-fpc1 kernel: [<00000000464a1bca>] page_fault+0x22/0x30
Apr 20 07:49:12.160 2025 labrouter-NEW-fpc1 kernel: Code: 90 90 90 90 9c fa 65 48 3b 06 75 14 65 48 3b 56 08 75 0d 65 48 89 1e 65 48 89 4e 08 9d b0 01 c3 9d 30 c0 c3 66 90 b9 00 02 00 00 <f3> 48 a5 c3 0f 1f 44 00 00 48 83 ec 10 48 89 1c 24 4c 89 64 24
Apr 20 07:49:14.740 2025 labrouter.RD.SD-RE0 snmpd[11673]: LIBJSNMP_NS_LOG_INFO: INFO: AgentX session, /var/run/mib2d-38, noticed 1 request timeout.
Apr 20 07:49:16.062 2025 labrouter.RD.SD-RE0 alarmd[11672]: Alarm set: RE id=570425449, color=YELLOW, class=CHASSIS, reason=Potential slow peers are: XDPC1
Apr 20 07:49:16.063 2025 labrouter.RD.SD-RE0 craftd[11163]: Receive FX craftd set alarm message: color: 2 class: 100 object: 105 slot: 0 silent: 0 short_reason= long_reason=Potential slow peers are: XDPC1 id=570425449 reason=570425344
Apr 20 07:49:16.063 2025 labrouter.RD.SD-RE0 craftd[11163]: Minor alarm set, Potential slow peers are: XDPC1
Apr 20 08:00:24.159 2025 labrouter-NEW-fpc1 kernel: NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [Direct-root:12940]
Apr 20 08:00:24.159 2025 labrouter-NEW-fpc1 kernel: Modules linked in: uio_pci_hostif macvlan 8021q ebtable_filter amd64_edac_mod edac_core crct10dif_pclmul crct10dif_common jnx_pmb_nvram jnx_f1_bcpld jnx_cbios_debugfs k10temp snd_hda_intel snd_hda_codec snd_hda_core i2c_piix4 tmp401 max6697 ucd9000 ltc2945 tpm_rng ofpart vt7505 pmbus max20751 pmbus_core m25p80 spi_nor leds_supercon_led2 gpio_supercon_hotswap leds_supercon_ledx uio_pdrv_genirq jnx_mastership_supercon uio tpm_crb acpi_cpufreq sam_core mtdoops mtd i2c_dev gpio_sam gpio_ptxpmb_cpld
Apr 20 08:00:24.159 2025 labrouter-NEW-fpc1 kernel: CPU: 14 PID: 12940 Comm: Direct-root Tainted: G L 4.8.28-WR2.2.1_standard-gad6c183 #1
Apr 20 08:00:24.159 2025 labrouter-NEW-fpc1 kernel: Hardware name: Juniper Networks Inc. 0D88/0D7A, BIOS CASN_P_PMB1_00.29.01 09/12/2022
Apr 20 08:00:24.159 2025 labrouter-NEW-fpc1 kernel: task: 00000000c286bcee task.stack: 0000000035312692
Apr 20 08:00:24.159 2025 labrouter-NEW-fpc1 kernel: RIP: 0010:[<ffffffff806b2057>] [<000000009e81c065>] copy_page+0x7/0x10
Apr 20 08:00:24.159 2025 labrouter-NEW-fpc1 kernel: RSP: 0000:ffffc90004bfbd68 EFLAGS: 00000286
Apr 20 08:00:24.159 2025 labrouter-NEW-fpc1 kernel: RAX: 0000160000000000 RBX: ffffea00084d6500 RCX: 00000000000001f8
Apr 20 08:00:24.159 2025 labrouter-NEW-fpc1 kernel: RDX: ffff8806ed79d4c0 RSI: ffff880213594040 RDI: ffff8804bc772040
Apr 20 08:00:24.159 2025 labrouter-NEW-fpc1 kernel: RBP: ffffc90004bfbdc8 R08: ffff88087eb9b9f8 R09: 0000000000000001
Apr 20 08:00:24.159 2025 labrouter-NEW-fpc1 kernel: R10: 00000000024200ca R11: 0000000000000081 R12: ffffea0012f1dc80
Apr 20 08:00:24.159 2025 labrouter-NEW-fpc1 kernel: R13: ffffc90004bfbe70 R14: ffff88044bd88cc0 R15: 8000000213594065
Apr 20 08:00:24.160 2025 labrouter-NEW-fpc1 kernel: FS: 00007f87835fb700(0000) GS:ffff88087eb80000(0000) knlGS:0000000000000000
Apr 20 08:00:24.160 2025 labrouter-NEW-fpc1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 20 08:00:24.160 2025 labrouter-NEW-fpc1 kernel: CR2: 00007f865584a000 CR3: 00000006ee5b4000 CR4: 00000000003406f0
Apr 20 08:00:24.160 2025 labrouter-NEW-fpc1 kernel: Stack:
Apr 20 08:00:24.160 2025 labrouter-NEW-fpc1 kernel: ffffffff803b9c80 00007f865584a000 00007f8655849000 ffff8806ed79d4c0
Apr 20 08:00:24.160 2025 labrouter-NEW-fpc1 kernel: 00007f865584a000 ffff880792d74ec0 ffff8807b66ba800 ffffea00084d6500
Apr 20 08:00:24.160 2025 labrouter-NEW-fpc1 kernel: ffffc90004bfbe70 8000000213594065 ffff88044bd88cc0 ffffc90004bfbe70
Apr 20 08:00:24.160 2025 labrouter-NEW-fpc1 kernel: Call Trace:
Apr 20 08:00:24.160 2025 labrouter-NEW-fpc1 kernel: [<00000000919b0b85>] ? wp_page_copy+0x110/0x710
Apr 20 08:00:24.160 2025 labrouter-NEW-fpc1 kernel: [<000000007df678ef>] do_wp_page+0x135/0x6a0
Apr 20 08:00:24.160 2025 labrouter-NEW-fpc1 kernel: [<00000000b3df1a45>] handle_mm_fault+0x742/0xfe0
Apr 20 08:00:24.160 2025 labrouter-NEW-fpc1 kernel: [<00000000967fe668>] __do_page_fault+0x260/0x550
Apr 20 08:00:24.160 2025 labrouter-NEW-fpc1 kernel: [<000000007af2cf93>] do_page_fault+0xc/0x10
Apr 20 08:00:24.160 2025 labrouter-NEW-fpc1 kernel: [<00000000464a1bca>] page_fault+0x22/0x30
Apr 20 08:00:24.160 2025 labrouter-NEW-fpc1 kernel: Code: 90 90 90 90 9c fa 65 48 3b 06 75 14 65 48 3b 56 08 75 0d 65 48 89 1e 65 48 89 4e 08 9d b0 01 c3 9d 30 c0 c3 66 90 b9 00 02 00 00 <f3> 48 a5 c3 0f 1f 44 00 00 48 83 ec 10 48 89 1c 24 4c 89 64 24
Apr 20 08:00:43.930 2025 labrouter.RD.SD-RE0 lfmd[11188]: LFMD_3AH_STATS_FETCH_FAILING: Stats fetch failed 3 consecutive times for ifd et-1/3/3
Apr 20 08:00:49.130 2025 labrouter.RD.SD-RE0 lfmd[11188]: LFMD_3AH_STATS_FETCH_FAILING: Stats fetch failed 3 consecutive times for ifd et-1/0/1
Apr 20 08:00:49.130 2025 labrouter.RD.SD-RE0 lfmd[11188]: LFMD_3AH_STATS_FETCH_FAILING: Stats fetch failed 3 consecutive times for ifd et-1/0/2
Apr 20 08:00:49.131 2025 labrouter.RD.SD-RE0 lfmd[11188]: LFMD_3AH_STATS_FETCH_FAILING: Stats fetch failed 3 consecutive times for ifd et-1/3/0
Apr 20 08:00:49.131 2025 labrouter.RD.SD-RE0 lfmd[11188]: LFMD_3AH_STATS_FETCH_FAILING: Stats fetch failed 3 consecutive times for ifd et-1/3/1
Apr 20 08:00:49.131 2025 labrouter.RD.SD-RE0 lfmd[11188]: LFMD_3AH_STATS_FETCH_FAILING: Stats fetch failed 3 consecutive times for ifd et-1/3/2
It appears that the issue may be related to a hardware problem with CPU#14. The recommended course of action is to reboot the affected FPC to attempt recovery. If the problem persists after the reboot, it would be necessary to replace the FPC