This article details a concerning issue that one of our customers has encountered after upgrading from version 19.4X12.1 to 21.2X8.1. They are experiencing problems related to the FPC core and have reported instances of log spamming.
Model: mx480
Junos: 21.2X8.1
Router Console (inband/Outband) spammed with below logs
[email protected]>
Message from [email protected] at Jan 8 10:26:38.815 2025 ...
re1.labrouter fpc1 Freeing an already freed small node!
re1.labrouter fpc1 Frame 0: sp = 0xe15f6db8, pc = 0xa137574
re1.labrouter fpc1 Frame 1: sp = 0xe15f6de8, pc = 0x96b78e0
re1.labrouter fpc1 Frame 2: sp = 0xe15f6e18, pc = 0x96be64c
re1.labrouter fpc1 Frame 3: sp = 0xe15f6e58, pc = 0x96be84c
re1.labrouter fpc1 Frame 4: sp = 0xe15f6f08, pc = 0x96c428c
re1.labrouter fpc1 Frame 5: sp = 0xe15f6f28, pc = 0x96c4892
Message Logs
...
Jan 8 10:24:41.297 2025 re1.labrouter fpc1 Invalid pfe(239)
Jan 8 10:24:41.297 2025 re1.labrouter fpc1 Frame -1: sp = 0xe15f6cc8, pc = 0x8120bee
Jan 8 10:24:41.297 2025 re1.labrouter fpc1 Frame 0: sp = 0xe15f6d68, pc = 0x97e99f2
Jan 8 10:24:41.297 2025 re1.labrouter fpc1 Frame 1: sp = 0xe15f6d98, pc = 0xa154393
Jan 8 10:24:41.297 2025 re1.labrouter fpc1 Frame 2: sp = 0xe15f6db8, pc = 0xa137574
Jan 8 10:24:41.297 2025 re1.labrouter fpc1 Frame 3: sp = 0xe15f6de8, pc = 0x96b78e0
Jan 8 10:24:41.297 2025 re1.labrouter fpc1 Frame 4: sp = 0xe15f6e18, pc = 0x96be64c
Jan 8 10:24:41.297 2025 re1.labrouter fpc1 Frame 5: sp = 0xe15f6e58, pc = 0x96be84c
Jan 8 10:24:41.297 2025 re1.labrouter fpc1 Frame 6: sp = 0xe15f6f08, pc = 0x96c428c
Jan 8 10:24:41.297 2025 re1.labrouter fpc1 Frame 7: sp = 0xe15f6f28, pc = 0x96c4892
Jan 8 10:24:41.297 2025 re1.labrouter fpc1 Frame 8: sp = 0xe15f6f48, pc = 0x9b0ceb5
…
Core dumps
[email protected]> show system core-dumps
-rw-r--r-- 1 root wheel 74920693 Jan 8 14:41 /var/crash/core-re1.labrouter-fpc1-smpc.elf.22.tgz
-rw-r--r-- 1 root wheel 74901598 Jan 8 15:12 /var/crash/core-re1.labrouter-fpc1-smpc.elf.23.tgz
-rw-r--r-- 1 root wheel 74890942 Jan 8 15:20 /var/crash/core-re1.labrouter-fpc1-smpc.elf.24.tgz
-rw-r--r-- 1 root wheel 74932888 Jan 8 15:39 /var/crash/core-re1.labrouter-fpc1-smpc.elf.25.tgz
When an Inline CFM UP MEP session is configured on an AE interface with VLAN normalisation and bridge or VPLS encapsulation, a reboot of the FPC or learning a new CFM peer over VPLS may trigger heap memory-related errors. These errors can occur consistently and, over time, may result in the FPC process crash.
This issue might be seen if the following conditions are met:
* On Junos MX platforms with MPC1- MPC9 line cards
* Inline CFM UP MEP is configured on AE with VLAN normalisation
* Bridge/VPLS encapsulation
* FPC reboot or learning a new CFM peer over VPLS
Issue can be mitigated by deactivating CFM or by upgrading to version that has the fix for the issue