This article describes a scenario on MX routers where "jlock (R) hog" and "Scheduler Oinker" messages are intermittently reported in the syslog. Although the messages are seen on the master RE's syslog, it is needed to identify if those ones are reported by the RE0 or RE1.
The root cause is traced back to hardware memory errors on the RE. These messages indicate that a thread (often ipintr()) holds a kernel lock for an extended period, leading to scheduler warnings, even though direct CPU or process issues are not immediately apparent. The underlying cause is typically corrected memory hardware errors on the affected RE.
The following log messages are observed in the syslog of the MX10008 router:
May 21 12:31:42 mx10008-re0 re1 a03.frnkge13.de.bb-re1 kernel: net_jlock_remove_reader: jlock (R) hog reported for 2019 msecs: ticks (acq=2274035393, rel=2274037412). Thread (100061) held it for 2019 msecs. No. of times read jlock shared in this duration 0.
May 21 12:31:42 mx10008-re0 re1 a03.frnkge13.de.bb-re1 kernel: jlock (R) hog reported for 2019 msecs: time_uptime (acq=127150, rel=127152), ticks (acq=2274035393, rel=2274037412), net_jlock_uptime (acq=125900, rel=125900) td 0xfffff80105cf8000 (intr): td_prio 28, pri_promo 0, CPU 7, td_lastrt 2274035393, jlock (acq=2274035393, cur=2274037412): offending fns (acq=ipintr(), rel=ipintr()). td_runtime 2010059 usecs, nvcsw 0, nivcsw 0 (../../../../../../../../../src/junos/bsd/sys/netinet/ip_input.c, line 1849): { 0xffffffff8130203f 0xffffffff813086ca 0xffffffff81307f4b 0xffffffff81308993 0xffffffff8155a000 0xffffffff81402eda 0xffffffff81407102 0xffffffff80475035 0xffffffff80471f01 0xffffffff8062b4d
May 21 12:31:42 mx10008-re0 spmb1 SCHED: Thread 2 (Idle) ran for 1842 ms without yielding
May 21 12:31:42 mx10008-re0 spmb1 Scheduler Oinker
May 21 12:31:42 mx10008-re0 spmb1 Frame 0: sp = 0xdeb1ffc8, pc = 0x8072cb2
May 21 12:31:42 mx10008-re0 spmb1 Frame 1: sp = 0xdeb1ffe8, pc = 0x8134a00
May 21 12:31:42 mx10008-re0 spmb1 Frame 2: sp = 0xdeb1fff8, pc = 0x808ad86
May 21 12:31:42 mx10008-re0 spmb1 Frame 3: sp = 0xdeb12fd8, pc = 0x81316c8
May 21 12:31:42 mx10008-re0 spmb1 Frame 4: sp = 0xdeb12ff8, pc = 0x808ad86
May 21 12:31:42 mx10008-re0 spmb1 Frame 5: sp = 0xfff628b8, pc = 0x813326c
May 21 12:31:42 mx10008-re0 spmb1 Frame 6: sp = 0xfff628d8, pc = 0x8076785
May 21 12:31:42 mx10008-re0 spmb1 Frame 7: sp = 0xfff628e8, pc = 0x804e2fe
May 21 12:31:42 mx10008-re0 spmb1 Frame 8: sp = 0xfff62908, pc = 0xf7afbea1
Additionally, upon examining the Routing Engine's System Information (RSI) on the affected RE (RE1 in the example), hardware error messages related to memory are observed:
[1357578.968160] {88354}[Hardware Error]: node: 0
[1357588.791332] {88355}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 0
[1357588.791334] {88355}[Hardware Error]: It has been corrected by h/w and requires no further action
[1357588.791335] {88355}[Hardware Error]: event severity: corrected
[1357588.791336] {88355}[Hardware Error]: Error 0, type: corrected
[1357588.791337] {88355}[Hardware Error]: section_type: memory error
[1357588.791338] {88355}[Hardware Error]: error_status: 0x0000000000000400
These symptoms occur randomly without an exact trigger, and direct CPU or process-related issues are not immediately found.
To resolve this issue:
1. Identify the affected RE: Determine which Routing Engine is reporting the memory hardware errors by checking its RSI output. In the provided context, RE1 was the affected RE, even though RE0 (master) was reporting the syslog messages.
2. Replace the affected component: Since RE in MX10008 built into the Control Board (CB), the entire CB needs to be replaced.