In this article, we explain a scenario where different behaviors are observed on a Virtual Chassis related to disruptions in the traffic with no clear indicators of a fundamental issue within the VC itself.
We highlight the types of logs that consistently appear and explain how the connection between a member acting as the master and a line card can introduce additional load on VC operations when errors are present. This behavior underscores the importance of monitoring VC‑port stability, as persistent errors not only generate recurring log events but also place stress on the control plane, leading to intermittent traffic disruptions.
On this scenario we have a VC of 7 members, where FPC 1 is acting as the Master and is connected to FPC 7 acting as Linecard:
root@user> show virtual-chassis status
Preprovisioned Virtual ChassisVirtual Chassis ID: xxxx.xxxx.xxxxVirtual Chassis Mode: Enabled Mstr Mixed Route Neighbor ListMember ID Status Serial No Alias-Name Model prio Role Mode Mode ID Interface0 (FPC 0) Prsnt XXXXXXXXXXXX 0 ex4100-24p 129 Backup* N VC 1 vcp-255/1/0 2 vcp-255/1/11 (FPC 1) Prsnt XXXXXXXXXXXX 1 ex4100-48p 129 Master N VC 7 vcp-255/1/0 0 vcp-255/1/12 (FPC 2) Prsnt XXXXXXXXXXXX 2 ex4100-24p 0 Linecard N VC 3 vcp-255/1/2 0 vcp-255/1/33 (FPC 3) Prsnt XXXXXXXXXXXX 3 ex4100-48p 0 Linecard N VC 5 vcp-255/1/2 2 vcp-255/1/35 (FPC 5) Prsnt XXXXXXXXXXXX 5 ex4100-48p 0 Linecard N VC 3 vcp-255/1/2 6 vcp-255/1/36 (FPC 6) Prsnt XXXXXXXXXXXX 6 ex4100-48p 0 Linecard N VC 5 vcp-255/1/0 7 vcp-255/1/17 (FPC 7) Prsnt XXXXXXXXXXXX 7 ex4100-48p 0 Linecard N VC 1 vcp-255/1/0 6 vcp-255/1/1
root@user> show virtual-chassis vc-port
fpc1:--------------------------------------------------------------------------Interface Type Trunk Status Speed Neighboror ID (mbps) ID InterfacePIC / Port1/0 Configured -1 Up 25000 7 vcp-255/1/01/1 Configured -1 Up 25000 0 vcp-255/1/0
--------------------------------------------------------------------------Interface Type Trunk Status Speed Neighboror ID (mbps) ID InterfacePIC / Port1/0 Configured -1 Up 25000 1 vcp-255/1/01/1 Configured -1 Up 25000 6 vcp-255/1/1
Following messages are seen on FPC 7 and FPC 1:
Jun 23 07:57:12 user kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 1000/1000/5Jun 23 07:57:12 user kernel: tcp_timer_keep:Local(0x80000016:61958) Foreign(0x80000001:6157)Jun 23 08:04:50 user dc-pfe[17457]: PFEMAN: Master socket closedJun 23 08:04:50 user dc-pfe[17457]: PFEMAN disconnected; PFEMAN socket closed abruptlyJun 23 08:04:51 user dc-pfe[17457]: Routing engine PFEMAN reconnection succeeded after 1 triesJun 23 08:04:51 user dc-pfe[17457]: PFEMAN master RE reconnection made with 0x1000080Jun 23 08:04:51 user dc-pfe[17457]: Probe exists for str bulkget_dfwJun 23 08:13:44 user dc-pfe[17457]: PFEMAN: Master socket closed
Each Packet Forwarding Engine (PFE) is connected to the primary Routing Engine (RE) and continuously sends TCP keepalives. When the kernel becomes busy, the PFE cannot process these keepalives, which leads to TCP timer expirations. As a result, the PFEs report TCP connection timeouts and disconnect their sockets, but they reconnect quickly once the kernel load subsides. In this case, the timeout occurred and corrected itself immediately.
If the Kernel remains busy for a longer period, it may continue dropping TCP keepalives, leaving TCP connections to all PFEs broken for an extended time. This condition can sometimes cause the Flexible PIC Concentrators (FPCs) on the device to reset and come back online.
No additional logs beyond those already observed were generated, and no daemon leaks were detected to explain the source of these messages and the inconsistent traffic behavior. However, when reviewing the VC‑ports, drops are incrementing on vcp‑255/1/0, which connects FPC7 directly to FPC1 (the master):
root@user> show virtual-chassis vc-port statistics extensive member 7 vcp-255/1/0
fpc7:
-------------------------------------------------------------------------- RX TX
Port: vcp-255/1/0MAC control frames: 0 0CRC alignment errors: 863740Oversize packets: 0Undersize packets: 0Jabber packets: 529612Fragments received: 18776
fpc7:-------------------------------------------------------------------------- RX TX
Port: vcp-255/1/0 CRC alignment errors: 1088581 > IncreasingOversize packets: 0 Undersize packets: 0 Jabber packets: 616495 > IncreasingFragments received: 24694 > Increasing
-------------------------------------------------------------------------- RX TXPort: vcp-255/1/0 CRC alignment errors: 1091332 > Increasing Oversize packets: 0 Undersize packets: 0 Jabber packets: 617888 > Increasing Fragments received: 24775 > Increasing
fpc7:-------------------------------------------------------------------------- RX TXPort: vcp-255/1/0 CRC alignment errors: 1093677 > IncreasingOversize packets: 0 Undersize packets: 0 Jabber packets: 619087 > IncreasingFragments received: 24837 > Increasing
These VC‑port errors are typically associated with physical issues involving the cable, optic, or the port itself. They can significantly contribute to dropped control and data plane traffic between FPC 7 and FPC 1 (the master), resulting in instability across the entire VC. Since the errors are consistently increasing, the following troubleshooting steps are recommended:
After replacing the optic on vcp‑255/1/0 on FPC 1, the issue was resolved: log events stopped, and ping, SSH, and SNMP connectivity were recovered.