This article explains that if we are not using any protect-RE filter and distributed denial‐of‐service (DDoS) is disabled for the Routing Engine (RE) and the Flexible PIC Concentrator (FPC), keepalive messages between the RE and the FPC may be dropped whenever there is excessive host-bound traffic on any device.
The following logs indicate a high number of traffic type profiles (TTPs) or host-bound packets.
fpc1 localttp_pkt_stat_update: 30001 TTP packet to RE in 350ms fpc1 localttp_pkt_stat_update: 30001 TTP packet to RE in 380ms fpc1 localttp_pkt_stat_update: 30001 TTP packet to RE in 429ms fpc1 localttp_pkt_stat_update: 30001 TTP packet to RE in 406ms
As shown below, you can see IPC connection drops post expiration of these TCP keepalive timer messages:
kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 3000/1000/3 kernel: tcp_timer_keep:Local(0x80000001:6985) Foreign(0x80000017:14444) kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 1000/1000/5 kernel: tcp_timer_keep:Local(0x80000001:6157) Foreign(0x80000017:32422) kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 1000/1000/5 kernel: tcp_timer_keep:Local(0x80000001:6166) Foreign(0x80000017:32421) kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 1000/1000/5 kernel: tcp_timer_keep:Local(0x80000001:6234) Foreign(0x80000015:46173) kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 1000/1000/5 kernel: tcp_timer_keep:Local(0x80000001:6234) Foreign(0x80000017:32423) kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 1000/1000/5 kernel: tcp_timer_keep:Local(0x80000001:6157) Foreign(0x80000015:46175) kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 1000/1000/5 kernel: tcp_timer_keep:Local(0x80000001:6166) Foreign(0x80000015:46174) chassisd[6878]: CHASSISD_IPC_CONNECTION_DROPPED: Dropped IPC connection for FPC 8 chassisd[6878]: CHASSISD_IPC_CONNECTION_DROPPED: Dropped IPC connection for FPC 9 chassisd[6878]: CHASSISD_IPC_CONNECTION_DROPPED: Dropped IPC connection for FPC 4 chassisd[6878]: CHASSISD_IPC_CONNECTION_DROPPED: Dropped IPC connection for FPC 10 chassisd[6878]: CHASSISD_IPC_CONNECTION_DROPPED: Dropped IPC connection for FPC 1 chassisd[6878]: CHASSISD_IPC_CONNECTION_DROPPED: Dropped IPC connection for FPC 7 chassisd[6878]: CHASSISD_IPC_CONNECTION_DROPPED: Dropped IPC connection for FPC 5 fpc9 PFEMAN disconnected; PFEMAN socket closed abruptly fpc1 PFEMAN disconnected; PFEMAN socket closed abruptly fpc7 PFEMAN disconnected; PFEMAN socket closed abruptly fpc5 PFEMAN disconnected; PFEMAN socket closed abruptly fpc10 PFEMAN disconnected; PFEMAN socket closed abruptly fpc8 PFEMAN disconnected; PFEMAN socket closed abruptly fpc4 PFEMAN disconnected; PFEMAN socket closed abruptly
You see the keepalive messages between the RE and the FPC expire mainly due to excessive host-bound traffic, which congests the RE-PFE link and therefore results in the keepalives of the IPC connection being dropped. In some cases, the excessive host-bound traffic will keep the RE too busy to be able to serve the keepalives.
To resolve this problem:
Configure the protect-RE filter if it is missing and monitor the FPC. For more information about configuring a protect-RE filter, refer to Example: Configuring a Stateless Firewall Filter to Protect Against TCP and ICMP Floods.
Check whether the DDoS configuration is enabled or not (enabled by default). For more information about DDoS configuration, refer to Configuring Control Plane DDoS Protection.
If the issue is seen repeatedly even after configuring protect-RE and DDoS, contact Support for further investigation and assistance.
2023-03-15: Initial version