Description

This article explains the meaning of the "FPC Major Error: PLL Error: XQ-0 PLL_LOCK:PWUP_CTRL_PLL_STATUS" alarm that is seen on MX Series routers and indicates if any actions need to be taken.

Symptoms

MX router reports the following FPC Major Errors alarm, in the example below the alarm was reported by MPC5E card in slot 2:
  • show system alarms
user@device> show chassis alarms
1 alarms currently active
Alarm time Class Description
<date> <time> Major FPC 2 Major Errors
  • show chassis errors active detail fpc-slot 2
user@device> show chassis errors active detail fpc-slot 2
Slot 2
--------------------------------------------------
Location : FPC 2
Identifier : /fpc/2/pfe/0/cm/0/PLL_Error/0/PLL_CMERROR_XQ_PWUP_CTRL_PLL_STATUS
Error : PLL_CMERROR_XQ_PWUP_CTRL_PLL_STATUS
Scope : board
Category : functional
Severity : Major
Details : XQ PWUP_CTRL_PLL
Count : 1
Support : No help info provided
  • show log messages
fpc2 PLL ERROR MONITOR: Unable to get the XQ-0 status
fpc2 CMError: /fpc/2/pfe/0/cm/0/PLL_Error/0/PLL_CMERROR_XQ_PWUP_CTRL_PLL_STATUS (0x170003), scope: board, category: functional, severity: major, module: PLL Error, type: XQ PWUP_CTRL_PLL
fpc2 Performing action get-state for error /fpc/2/pfe/0/cm/0/PLL_Error/0/PLL_CMERROR_XQ_PWUP_CTRL_PLL_STATUS (0x170003) in module: PLL Error with scope: board category: functional level: major
fpc2 Performing action cmalarm for error /fpc/2/pfe/0/cm/0/PLL_Error/0/PLL_CMERROR_XQ_PWUP_CTRL_PLL_STATUS (0x170003) in module: PLL Error with scope: board category: functional level: major
fpc2 Cmerror Op Set: PLL Error: XQ-0 PLL_LOCK:PWUP_CTRL_PLL_STATUS (0x0)  (URI: /fpc/2/pfe/0/cm/0/PLL_Error/0/PLL_CMERROR_XQ_PWUP_CTRL_PLL_STATUS)

Solution

In the output of "show log messages" review the events that occurred at or just before the appearance of the "FPC Major Error: PLL Error: XQ-0 PLL_LOCK:PWUP_CTRL_PLL_STATUS" alarm. Frequently these events help identify the cause.
 

This alarm is reported when XQ chip is not able to read PLL (phased-locked loop) status of the board.
This is typically a transient hardware issue, no RMA is required.

Recommended actions:

  1. No RMA required.
  2. By default, disable-pfe action will be taken if major error alarm is raised.
  3. An FPC reset during a maintenance window is needed to clear this error and bring the PFE back into service.

If the issue is seen repeatedly even after restarting the FPC, contact Support for further investigation and assistance.

In the example below, XM, XL and XQ chips were not able to read PLL status due to continuous I2C read failures caused by bad parity at the same address. FPC reload resolved the issue. 
Jul 24 05:22:56  device : %PFE-3: fpc2 trinity_i2c_read_reg: read from group 0x55 device 0x61 reg 0x86, primary addr 0x00808108 failed too many times, giving up
Jul 24 05:22:56  device : %PFE-3: fpc2 trinity_i2c_read32:  Failed byte 0 read for primary address 0x00808108
Jul 24 05:22:56  device : %PFE-3: fpc2 JGCI_IOERR : XR2CHIP(0)-chan-rx0-sc0 read  lkr.stats.tl 0x808108 -> 0x0 failed
Jul 24 05:22:56  device : %PFE-3: fpc2 jgci_chan_get_rx_tl_frames() Reading 'lkr.stats.tl' failed: XR2CHIP(0)-chan-rx0-sc0 err 1
Jul 24 05:22:57  device : %PFE-3: fpc2 trinity_i2c_read_reg: i2c_read(0x55, 0x62, 1) for reg 0x06, primary addr 0x00808108 had bad parity: 0x6f, retrying
Jul 24 05:22:58  device last message repeated 9 times
Jul 24 05:22:58  device : %PFE-3: fpc2 trinity_i2c_read_reg: read from group 0x55 device 0x62 reg 0x86, primary addr 0x00808108 failed too many times, giving up
Jul 24 05:22:58  device : %PFE-3: fpc2 trinity_i2c_read32:  Failed byte 0 read for primary address 0x00808108
Jul 24 05:22:58  device : %PFE-3: fpc2 JGCI_IOERR : XR2CHIP(1)-chan-rx0-sc0 read  lkr.stats.tl 0x808108 -> 0x0 failed
Jul 24 05:22:58  device : %PFE-3: fpc2 jgci_chan_get_rx_tl_frames() Reading 'lkr.stats.tl' failed: XR2CHIP(1)-chan-rx0-sc0 err 1
Jul 24 05:22:59  device : %PFE-3: fpc2 trinity_i2c_read_reg: i2c_read(0x56, 0x61, 1) for reg 0x06, primary addr 0x00808108 had bad parity: 0x6f, retrying
Jul 24 05:23:00  device last message repeated 9 times
Jul 24 05:23:00  device : %PFE-3: fpc2 trinity_i2c_read_reg: read from group 0x56 device 0x61 reg 0x86, primary addr 0x00808108 failed too many times, giving up
Jul 24 05:23:00  device : %PFE-3: fpc2 trinity_i2c_read32:  Failed byte 0 read for primary address 0x00808108
Jul 24 05:23:00  device : %PFE-3: fpc2 JGCI_IOERR : XR2CHIP(2)-chan-rx0-sc0 read  lkr.stats.tl 0x808108 -> 0x0 failed
Jul 24 05:23:01  device : %PFE-3: fpc2 jgci_chan_get_rx_tl_frames() Reading 'lkr.stats.tl' failed: XR2CHIP(2)-chan-rx0-sc0 err 1
Jul 24 05:23:01  device : %PFE-3: fpc2 trinity_i2c_read_reg: i2c_read(0x57, 0x62, 1) for reg 0x06, primary addr 0x00808108 had bad parity: 0x6f, retrying
Jul 24 05:23:01  device last message repeated 4 times
Jul 24 05:23:01  device : %PFE-5: fpc4 user.notice logrotate: ALERT exited abnormally with [1]
Jul 24 05:23:01  device : %PFE-3: fpc2 trinity_i2c_read_reg: i2c_read(0x57, 0x62, 1) for reg 0x06, primary addr 0x00808108 had bad parity: 0x6f, retrying
Jul 24 05:23:02  device last message repeated 4 times
Jul 24 05:23:02  device : %PFE-3: fpc2 trinity_i2c_read_reg: read from group 0x57 device 0x62 reg 0x86, primary addr 0x00808108 failed too many times, giving up
Jul 24 05:23:02  device : %PFE-3: fpc2 trinity_i2c_read32:  Failed byte 0 read for primary address 0x00808108
Jul 24 05:23:02  device : %PFE-3: fpc2 JGCI_IOERR : XR2CHIP(3)-chan-rx0-sc0 read  lkr.stats.tl 0x808108 -> 0x0 failed
Jul 24 05:23:03  device : %PFE-3: fpc2 jgci_chan_get_rx_tl_frames() Reading 'lkr.stats.tl' failed: XR2CHIP(3)-chan-rx0-sc0 err 1
Jul 24 05:23:03  device : %PFE-3: fpc2 trinity_i2c_read_reg: i2c_read(0x55, 0x61, 1) for reg 0x06, primary addr 0x00808108 had bad parity: 0x6f, retrying
Jul 24 05:23:04  device last message repeated 9 times
Jul 24 05:23:04  device : %PFE-3: fpc2 trinity_i2c_read_reg: read from group 0x55 device 0x61 reg 0x86, primary addr 0x00808108 failed too many times, giving up
Jul 24 05:23:04  device : %PFE-3: fpc2 trinity_i2c_read32:  Failed byte 0 read for primary address 0x00808108
Jul 24 05:23:04  device : %PFE-3: fpc2 JGCI_IOERR : XR2CHIP(0)-chan-rx0-sc0 read  lkr.stats.tl 0x808108 -> 0x0 failed
Jul 24 05:23:05  device : %PFE-3: fpc2 jgci_chan_get_rx_tl_frames() Reading 'lkr.stats.tl' failed: XR2CHIP(0)-chan-rx0-sc0 err 1
Jul 24 05:23:05  device : %PFE-3: fpc2 trinity_i2c_read_reg: i2c_read(0x55, 0x78, 1) for reg 0x06, secondary addr 0x00000010 had bad parity: 0x6f, retrying
Jul 24 05:23:06  device last message repeated 9 times
Jul 24 05:23:06  device : %PFE-3: fpc2 trinity_i2c_read_reg: read from group 0x55 device 0x78 reg 0x86, secondary addr 0x00000010 failed too many times, giving up
Jul 24 05:23:06  device : %PFE-3: fpc2 trinity_i2c_read32:  Failed byte 0 read for secondary address 0x00000010
Jul 24 05:23:07  device : %PFE-3: fpc2 xlchip_get_pll_status:  XL[0:0] failed to read PLL status from pwup_ctrl.pll_status.
Jul 24 05:23:07  device : %PFE-3: fpc2 PLL ERROR MONITOR: Unable to get the XL-0 status
Jul 24 05:23:07  device : %PFE-3: fpc2 trinity_i2c_read_reg: i2c_read(0x4e, 0x74, 1) for reg 0x06, secondary addr 0x00000010 had bad parity: 0x6f, retrying
Jul 24 05:23:08  device last message repeated 6 times
Jul 24 05:23:08  device alarmd[27390]: %DAEMON-4: Alarm set: FPC id=150995560, color=RED, class=CHASSIS, reason=FPC 2 Major Errors
Jul 24 05:23:08  device craftd[24237]: %DAEMON-4:  Major alarm set, FPC 2 Major Errors
Jul 24 05:23:08  device : %PFE-3: fpc2 trinity_i2c_read_reg: i2c_read(0x4e, 0x74, 1) for reg 0x06, secondary addr 0x00000010 had bad parity: 0x6f, retrying
Jul 24 05:23:08  device last message repeated 2 times
Jul 24 05:23:08  device : %PFE-3: fpc2 trinity_i2c_read_reg: read from group 0x4e device 0x74 reg 0x86, secondary addr 0x00000010 failed too many times, giving up
Jul 24 05:23:08  device : %PFE-3: fpc2 trinity_i2c_read32:  Failed byte 0 read for secondary address 0x00000010
Jul 24 05:23:09  device : %PFE-3: fpc2 XMCHIP(0): xmchip_pio_i2c_sec_read_u32: Reading 32-bit register failed - status 1, pio_handle_i2c_sec 0x1c662b10, addr 0x10
Jul 24 05:23:09  device : %PFE-3: fpc2 XMCHIP(0): xmchip_get_pll_lock_status: Getting PLL lock status failed - status 1
Jul 24 05:23:09  device : %PFE-3: fpc2 PLL ERROR MONITOR: Unable to get the XM-0 status
Jul 24 05:23:09  device : %PFE-3: fpc2 trinity_i2c_read_reg: i2c_read(0x56, 0x7c, 1) for reg 0x06, secondary addr 0x00000010 had bad parity: 0x6f, retrying
Jul 24 05:23:10  device last message repeated 9 times
Jul 24 05:23:10  device : %PFE-3: fpc2 trinity_i2c_read_reg: read from group 0x56 device 0x7c reg 0x86, secondary addr 0x00000010 failed too many times, giving up
Jul 24 05:23:11  device : %PFE-3: fpc2 trinity_i2c_read32:  Failed byte 0 read for secondary address 0x00000010
Jul 24 05:23:11  device : %PFE-3: fpc2 xqchip_pio_i2c_sec_read_u32:299 XQ-chip(0): pio_handle_i2c_sec(0x1c662cc8); pio_read_u32() failed: 1(generic failure)! addr=0000000000000010
Jul 24 05:23:11  device : %PFE-3: fpc2 xqchip_get_pll_status:1694 XQ-0 pwup_ctrl.pll_status read failed
Jul 24 05:23:11  device : %PFE-3: fpc2 PLL ERROR MONITOR: Unable to get the XQ-0 status
Jul 24 05:23:12  device : %PFE-5: fpc2 CMError: /fpc/2/pfe/0/cm/0/PLL_Error/0/PLL_CMERROR_XQ_PWUP_CTRL_PLL_STATUS (0x170003), scope: board, category: functional, severity: major, module: PLL Error, type: XQ PWUP_CTRL_PLL

Modification History

2024-08-06 : Article Created