Description

In this article, we explain a scenario where inserting an optic can cause instability on platforms with limited memory, such as the EX4300. This instability may result in a PFE disconnection due to high memory utilization at the time of insertion, which can trigger further system issues.

Symptoms

  • Optic insertion can cause instability on the device.
  • Memory threshold logs recorded before the incident indicate high memory utilization. 
  • PFE disconnection caused by memory instability.
  • Scenario has been observed across both standalone devices and Virtual Chassis.

The following logs were observed immediately after a QSFP was inserted:

 

May  5 17:59:04 2026  Switch fpc0 Rear QSFP+ PIC plugged in 
May  5 17:59:12 2026  Switch fpc0 Rear QSFP+ PIC Chan# 0: Rx power low alarm set 
May  5 17:59:12 2026  Switch fpc0 Rear QSFP+ PIC Chan# 0: Rx power low warning set 
May  5 17:59:12 2026  Switch fpc0 Rear QSFP+ PIC Chan# 1: Rx power low alarm set 
May  5 17:59:12 2026  Switch fpc0 Rear QSFP+ PIC Chan# 1: Rx power low warning set 
May  5 17:59:12 2026  Switch fpc0 Rear QSFP+ PIC Chan# 2: Rx power low alarm set 
May  5 17:59:12 2026  Switch fpc0 Rear QSFP+ PIC Chan# 2: Rx power low warning set 
May  5 17:59:12 2026  Switch fpc0 Rear QSFP+ PIC Chan# 3: Rx power low alarm set 
May  5 17:59:12 2026  Switch fpc0 Rear QSFP+ PIC Chan# 3: Rx power low warning set 
May  5 17:59:13 2026  Switch vccpd[1584]: VCCPD_IFL_SUCCESS: Creation of ifl vcp-255/1/1.32768 Success 
May  5 17:59:14 2026  Switch pfex: inteface name: vcp-255/1/1
May  5 17:59:14 2026  Switch fpc0 inteface name: vcp-255/1/1 
May  5 17:59:21 2026  Switch /kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 1000/1000/5
May  5 17:59:21 2026  Switch /kernel: tcp_timer_keep:Local(0x8000003f:7001) Foreign(0x80000010:55944)
May  5 17:59:21 2026  Switch /kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 300/300/5
May  5 17:59:21 2026  Switch /kernel: tcp_timer_keep:Local(0x80000010:49272) Foreign(0x80000001:33015)
May  5 17:59:21 2026  Switch /kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 1000/1000/5
May  5 17:59:21 2026  Switch /kernel: tcp_timer_keep:Local(0x80000010:62254) Foreign(0x8000003f:6234)
May  5 17:59:21 2026  Switch /kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 1000/1000/5
May  5 17:59:21 2026  Switch /kernel: tcp_timer_keep:Local(0x8000003f:6234) Foreign(0x80000010:62254)
May  5 17:59:21 2026  Switch /kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 1000/1000/5
May  5 17:59:21 2026  Switch /kernel: tcp_timer_keep:Local(0x80000001:7000) Foreign(0x80000010:56958)
May  5 17:59:21 2026  Switch /kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 1000/1000/5
May  5 17:59:21 2026  Switch /kernel: tcp_timer_keep:Local(0x80000010:55944) Foreign(0x8000003f:7001)
May  5 17:59:21 2026  Switch chassism[1586]: JTASK_SCHED_SLIP_KEVENT: 7 sec 51759 usec kevent block
May  5 17:59:21 2026  Switch chassism[1586]: CMLC: No Master IPC pipe/thread for msg
May  5 17:59:21 2026  Switch rpd[1752]: JTASK_SCHED_SLIP_KEVENT: 7 sec 603684 usec kevent block
May  5 17:59:21 2026  Switch l2cpd[1769]: JTASK_SCHED_SLIP_KEVENT: 7 sec 667489 usec kevent block
May  5 17:59:21 2026  Switch mplsoamd[1774]: JTASK_SCHED_SLIP_KEVENT: 8 sec 193016 usec kevent block
May  5 17:59:21 2026  Switch mcsnoopd[1781]: JTASK_SCHED_SLIP_KEVENT: 7 sec 763706 usec kevent block
May  5 17:59:21 2026  Switch vccpd[1584]: JTASK_SCHED_SLIP_KEVENT: 7 sec 763631 usec kevent block
May  5 17:59:21 2026  Switch dot1xd[1779]: JTASK_SCHED_SLIP: 7 sec scheduler slip, user: 0 sec 0 usec, system: 0 sec, 612 usec
May  5 17:59:21 2026  Switch pfex: SCHED: Thread 2 (Idle) aborted, hogged 7183 ms
May  5 17:59:21 2026  Switch pfex: IPC Socket is closed 
May  5 17:59:21 2026  Switch fpc0 SCHED: Thread 2 (Idle) aborted, hogged 7183 ms 
May  5 17:59:22 2026  Switch chassisd[1579]: CHASSISD_IPC_FLUSH_ERROR: ch_flush_fru_pipe: flush operation failed for LCD 0
May  5 17:59:22 2026  Switch chassisd[1579]: CHASSISD_IPC_FLUSH_ERROR: ch_flush_fru_pipe: flush operation failed for LCD 0
May  5 17:59:22 2026  Switch chassisd[1579]: CHASSISD_IPC_CONNECTION_DROPPED: Dropped IPC connection for FPC 0

 

FPC chassis output confirms the PFE disconnection based on the uptime:

 

root@Switch> show chassis fpc detail

Slot 0 information:
  State                               Online
  Total CPU DRAM                 2048 MB
  Start time                          2026-05-05 17:59:34 UTC
  Uptime                              2 days, 20 hours, 39 minutes, 18 seconds

 

Before this, the following log was raised, which indicates the device was experiencing high memory utilization just prior to the event:

 

Mar 5 07:18:50 2026 Switch /kernel: Percentage memory available(19)less than threshold(20 %)- 3

Solution

There is an increase in memory footprint across different daemons after an image upgrade resulting increase in the system memory. This is due to software architecture change in 20.4 which led to the increase in memory usage for some of the daemons:

 

Please check: https://prsearch.juniper.net/problemreport/PR1694522

 

The root cause of the instability following an optic insertion is high memory utilization at the time of insertion, which can trigger system instability. Since the memory footprint on this platform has already increased, is recommended to avoid multiple concurrent CLI sessions, as these can significantly raise memory usage and make the device unstable. This is also documented in the PR I referenced, which explains the rise in memory consumption.

 

Here is an example of a device consuming excessive resources due to multiple CLI sessions being opened:

 

root@Switch> show system memory 
fpc0:
--------------------------------------------------------------------------
System memory usage distribution:
       Total memory: 1943552 Kbytes (100%)
    Reserved memory:   38572 Kbytes (  1%)
       Wired memory:  161660 Kbytes (  8%)
      Active memory: 1436120 Kbytes ( 73%)
    Inactive memory:  118524 Kbytes (  6%)
       Cache memory:   76416 Kbytes (  3%)
        Free memory:  111556 Kbytes (  5%)
Memory disk resident memory:  472960 Kbytes
VM-Kbytes(  %  ) Resident(  %  ) Map-name
  1048576(99.99)   884064(84.31) kernel map
   524288(50.00)    77368(14.76) kmem map
     1216(00.12)     1216(99.99) exec map
    26212(02.50)     1228(04.68) pipe map
   115488(11.01)   114784(99.39) buffer map
    32768(03.13)    32768(99.99) pager map
----
     88844(11.92)    54192(02.76) -cli
   299164(16.52)    17420(00.89) mgd: (mgd) (user)/dev/ttyp2
    88792(11.92)    53800(02.74) -cli
    63076(04.01)     6908(00.35) mgd: (mgd) (user/dev/ttyp3
    88792(11.92)    54256(02.76) cli
    63080(04.01)     9880(00.50) mgd: (mgd) (user)/dev/ttyp3
    92940(12.40)    54120(02.75) cli
    63160(04.01)    11316(00.58) mgd: (mgd) (user)/dev/ttyp3
    92940(12.40)    55924(02.84) cli
    72316(04.59)    11920(00.61) mgd: (mgd) (user)/dev/ttyp3
    93920(04.53)    54372(02.77) cli
    63408(03.06)     5992(00.30) mgd: (mgd) (user)/dev/ttyp3

 

With 1 or 2 CLI session, below is the memory consumption and system is stable:

 

root@Switch> show system memory 
fpc0:
--------------------------------------------------------------------------
System memory usage distribution:
        Total memory: 1943552 Kbytes (100%)
     Reserved memory:   38572 Kbytes (  1%)
        Wired memory:  172052 Kbytes (  8%)
       Active memory: 1278924 Kbytes ( 65%)
     Inactive memory:  101176 Kbytes (  5%)
        Cache memory:  102120 Kbytes (  5%)
         Free memory:  250120 Kbytes ( 12%)
Memory disk resident memory:  462488 Kbytes
VM-Kbytes(  %  ) Resident(  %  ) Map-name
   1048576(99.99)   883344(84.24) kernel map
    524288(50.00)    82516(15.74) kmem map
      1216(00.12)     1216(99.99) exec map
     26212(02.50)     1160(04.43) pipe map
    115488(11.01)   114784(99.39) buffer map
     32768(03.13)    32768(99.99) pager map

 

On EX switches with limited memory, each new release adds more features, which reduces the amount of available resident memory. When multiple CLI sessions are launched, the system may become unstable, and low‑memory logs can appear. The CLI feature allows you to use the command "set system max-cli-sessions x" to restrict the number of concurrent sessions and help reduce memory consumption.

Modification History

2026-05-13 : Article Created