Description

Devices may intermittently report SPC3 flowd heartbeat‑loss events, particularly after configuration commits involving large security policies. These delays can cause PFEs to miss heartbeats, potentially triggering redundancy group failover

Symptoms

PFE / Terminal Logs :

Administrators may observe scheduler latency and thread‑stall messages on the PFE during or shortly after commit operations:


 FPC Slot 1, PIC Slot 0) Frame 1: sp = 0x400905f0, pc = 0x769a20

<160>1 2026-02-25T23:07:37.720+11:00 FIREWALL - - - - (FPC Slot 1, PIC Slot 0) SCHED: Thread 4 (Module Init) ran for 3181 ms without yielding

<160>1 2026-02-25T23:07:37.720+11:00 FIREWALL - - - - (FPC Slot 1, PIC Slot 0) Scheduler Oinker

<160>1 2026-02-25T23:07:37.720+11:00 FIREWALL - - - - (FPC Slot 1, PIC Slot 0) Frame 0: sp = 0x400905a0, pc = 0x7635dc

<160>1 2026-02-25T23:07:37.720+11:00 FIREWALL - - - - (FPC Slot 1, PIC Slot 0) Frame 1: sp = 0x400905f0, pc = 0x769a20


Messages:

spu_spc_x86_64_check_heartbeat: fpc 1 pic1 flowd lost heartbeat for 5 consecutive seconds
spu_spc_x86_64_check_heartbeat: fpc 9 pic1 flowd lost heartbeat for 10 consecutive seconds. Service may go down.

Impact :

Because heartbeat messages from RE → PFE are delayed, the SPC3 flowd process appears unresponsive.

This can lead to:


> Redundancy Group (RG) Failover

Triggered when the system interprets the delayed heartbeats as:

  • Interface failures, or
  • FPC restarts / unresponsiveness

This results in the RG transitioning to the backup node, causing:

  • Brief service interruption
  • Failover events in high‑availability clusters

If a heartbeat timeout aligns with an interface flap or a real failover activity, the impact can be more visible.

Solution

This issue occurs when the Routing Engine (RE) processes large security policy files (typically 10MB+) during a commit.

Policy compilation and file‑serialization introduce significant CPU and I/O load on the RE, slowing down RE-to-PFE communication channels.

As a result:

  • Heartbeat messages are delayed.
  • SPC3 flowd considers the PFE unhealthy.
  • Failover decisions may be triggered prematurely.

Workaround

Disable security policy file serialization to reduce RE processing load:

 

#set security policies no-file-serialization

#commit

 

This prevents large policy files from being continuously serialized, reducing commit-time delays and avoiding heartbeat misses.

This issue is fixed on junos:24.4R2-S4, junos:25.2R2-S1 junos:25.4R2 junos:26.2R1 and further releases. 

Modification History

2026-02-28 : Article Created