Description

On ACX5448 platform, when FPC (Flexible PIC Concentrator) lost communication with Routing Engine, FPC remained online and it's physical ports stayed up as well. But the traffic through this interface will be black-holed. because ACX5448 is a fixed platform router, all the traffic on acx5448 gets drooped during that time.

Symptoms

On the ACX5448 platform, FPC remains online and its physical ports stayed up as well even though FPC lost communication with RE. This issue caused the all the traffic drop. 
Below log messages indicate that the connection between FPC and RE is lost.

Sep 10 10:47:03 3024  acx5448-01 kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 1000/1000/5
Sep 10 10:47:03 3024  acx5448-01 kernel: tcp_timer_keep:Local(0x80000001:6157) Foreign(0x80000010:60924)
Sep 10 10:47:04 3024  acx5448-01 kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 1000/1000/5
Sep 10 10:47:04 3024  acx5448-01 kernel: tcp_timer_keep:Local(0x80000001:6234) Foreign(0x80000010:57438)
Sep 10 10:47:04 3024  acx5448-01 kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 1000/1000/5
Sep 10 10:47:04 3024  acx5448-01 kernel: tcp_timer_keep:Local(0x80000001:6166) Foreign(0x80000010:52156)
Sep 10 10:47:08 3024  acx5448-01 kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 2000/2000/5
Sep 10 10:47:08 3024  acx5448-01 kernel: tcp_timer_keep:Local(0x80000001:6011) Foreign(0x80000010:57824)
Sep 10 12:56:57 3024  acx5448-01 kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 7200000/75000/8
Sep 10 12:56:57 3024  acx5448-01 kernel: tcp_timer_keep:Local(0x80000001:7000) Foreign(0x80000010:58160)
Sep 10 12:56:58 3024  acx5448-01 chassisd[20495]: CHASSISD_SHUTDOWN_NOTICE: Shutdown reason: DFEB connection lost
Sep 10 12:56:58 3024  acx5448-01 chassisd[20495]: CHASSISD_IFDEV_DETACH_FPC: ifdev_detach_fpc(0)

Solution

The root cause is PR1659949 which clarifies the following:

To end communication between RE and RIOFEB, we turned down the BME0 interface.
CMLC thread watchdog triggers hardware reset after 20 seconds when RE and RIOFEB connections are terminated. Hardware reset terminates RIOFEB and restarts it. In spite of this, the peer device port remains active even after exiting RIOFEB.

This issue might be seen if the following conditions are met:
* On ACX5448 platform
* Trigger communication disruption between RE and FPC

This issue resolved in the following version:
junos:20.3X2-J1 
junos:21.3R3-S1 
junos:21.4R3 
junos:22.1R3 
junos:22.2R2 
junos:22.3R1 
junos:22.4R1

Modification History

KB created on Sep 11