Description

All interfaces went down but device was in operational state. Required full restart to restore.

Symptoms

root@Router> show system errors active detail

 

System Active Errors Detail Information

CB 0

----------------------------------------------------------------

Error Name : PLL input failure

Identifier : /cb/0/clockd/0/C_PLL_LOCK/0/clocking/0/PLL input failure

Description : PLL input failure

State : enabled

Scope : board

Category : functional

Level : major

Threshold : 1

Error limit : 1

Occur count : 1

Clear count : 0

Last occurred(ms ago) : 2025-09-16 04:45:56 BST (16424295 ms ago)

System Active Errors Detail Information

CB 0

----------------------------------------------------------------

Error Name : PLL input failure

Identifier : /cb/0/clockd/0/NIF_PLL_LOCK/0/clocking/0/PLL input failure

Description : PLL input failure

State : enabled

Scope : board

Category : functional

Level : major

Threshold : 1

Error limit : 1

Occur count : 1

Clear count : 0

Last occurred(ms ago) : 2025-09-16 04:45:56 BST (16424295 ms ago)

System Active Errors Detail Information

CB 0

----------------------------------------------------------------

Error Name : PLL input failure

Identifier : /cb/0/clockd/0/Q2U_BS_PLL_LOCK/0/clocking/0/PLL input failure

Description : PLL input failure

State : enabled

Scope : board

Category : functional

Level : major

Threshold : 1

Error limit : 1

Occur count : 1

Clear count : 0

Last occurred(ms ago) : 2025-09-16 04:45:56 BST (16424294 ms ago)

System Active Errors Detail Information

CB 0

----------------------------------------------------------------

Error Name : PLL input failure

Identifier : /cb/0/clockd/0/U_PLL_LOCK/0/clocking/0/PLL input failure

Description : PLL input failure

State : enabled

Scope : board

Category : functional

Level : major

Threshold : 1

Error limit : 1

Occur count : 1

Clear count : 0

Last occurred(ms ago) : 2025-09-16 04:45:56 BST (16424294 ms ago)

System Active Errors Detail Information

RE 0

----------------------------------------------------------------

Error Name : PCIe AER Uncorrected error threshold exceeded

Identifier : /re/0/emfd-fpa/0/0000:00:09.0/0/pci_resiliency/0/PCIe AER Uncorrected error threshold exceeded

Description : PCIe AER Uncorrected error threshold exceeded

State : enabled

Scope : board

Category : io

Level : major

Threshold : 1

Error limit : 1

Occur count : 1

Clear count : 0

Last occurred(ms ago) : 2025-09-16 04:45:39 BST (16441743 ms ago)

System Active Errors Detail Information

RE 0

----------------------------------------------------------------

Error Name : PCIe AER Uncorrected error threshold exceeded

Identifier : /re/0/emfd-fpa/0/0000:00:0a.0/0/pci_resiliency/0/PCIe AER Uncorrected error threshold exceeded

Description : PCIe AER Uncorrected error threshold exceeded

State : enabled

Scope : board

Category : io

Level : major

Threshold : 1

Error limit : 1

Occur count : 1

Clear count : 0

Last occurred(ms ago) : 2025-09-16 04:45:36 BST (16444033 ms ago)

System Active Errors Detail Information

RE 0

----------------------------------------------------------------

Error Name : PCIe AER Uncorrected error threshold exceeded

Identifier : /re/0/emfd-fpa/0/0000:00:0b.0/0/pci_resiliency/0/PCIe AER Uncorrected error threshold exceeded

Description : PCIe AER Uncorrected error threshold exceeded

State : enabled

Scope : board

Category : io

Level : major

Threshold : 1

Error limit : 1

Occur count : 1

Clear count : 0

Last occurred(ms ago) : 2025-09-16 04:45:39 BST (16441743 ms ago)

System Active Errors Detail Information

RE 0

----------------------------------------------------------------

Error Name : PCIe AER Uncorrected error threshold exceeded

Identifier : /re/0/emfd-fpa/0/0000:00:0c.0/0/pci_resiliency/0/PCIe AER Uncorrected error threshold exceeded

Description : PCIe AER Uncorrected error threshold exceeded

State : enabled

Scope : board

Category : io

Level : major

Threshold : 1

Error limit : 1

Occur count : 1

Clear count : 0

Last occurred(ms ago) : 2025-09-16 04:45:39 BST (16441744 ms ago)

 

Solution

Below alarms are HW specific & not related to any software issue.

 

Alarms:

2025-09-16 04:45:39 BST Major RE 0 PCI Uncorrected error on dev 0000:00:09.0

2025-09-16 04:45:36 BST Major RE 0 PCI Uncorrected error on dev 0000:00:0a.0

2025-09-16 04:45:39 BST Major RE 0 PCI Uncorrected error on dev 0000:00:0b.0

2025-09-16 04:45:39 BST Major RE 0 PCI Uncorrected error on dev 0000:00:0c.0

2025-09-16 04:45:56 BST Major CB 0 C_PLL_LOCK PLL Input Failure

2025-09-16 04:45:56 BST Major CB 0 NIF_PLL_LOCK PLL Input Failure

2025-09-16 04:45:56 BST Major CB 0 Q2U_BS_PLL_LOCK PLL Input Failure

2025-09-16 04:45:56 BST Major CB 0 U_PLL_LOCK PLL Input Failure

 

This is BCM chip reporting PCIe issues which in turn results in hang of pfe/interfaces.

 

An AER (Advanced Error Reporting) uncorrected error indicates a PCI Express communication problem that cannot be automatically fixed by the hardware and may lead to system instability or failure. 

These errors, often fatal to the PCIe link, require system intervention like resetting a component, the link, or even rebooting the system to resolve.

 

To detect the current kind of issue, below CLIs can help, when issue is hit.

show system alarms

show chassis alarms

show system errors active detail

 

Workaround for recovery is to do full restart or power cycle.

 

We may need to do proactive RMA for chassis if customer requests same.

Modification History

2025-10-24 : Article Created