Customer suffered an impact on MX104 devices due to multiple AFEB disconnections:
Aug 29 10:45:27.245 2024 ROUTER : %PFE-3: afeb0 CMLC: Master closed connection (errno=29)
Aug 29 10:45:28.660 2024 ROUTER : %PFE-6: afeb0 CMLC: Master RE reconnection made
Aug 29 10:51:01.576 2024 ROUTER : %PFE-3: afeb0 CMLC: Master closed connection (errno=29)
Aug 29 10:51:03.015 2024 ROUTER : %PFE-6: afeb0 CMLC: Master RE reconnection made
Aug 29 10:56:26.369 2024 ROUTER : %PFE-3: afeb0 CMLC: Master closed connection (errno=29)
Aug 29 10:56:27.430 2024 ROUTER : %PFE-6: afeb0 CMLC: Master RE reconnection made
Aug 29 11:02:14.542 2024 ROUTER : %PFE-3: afeb0 CMLC: Master closed connection (errno=29)
Aug 29 11:02:14.552 2024 ROUTER : %PFE-6: afeb0 CMLC: Master RE reconnection made
The AFEB nvram output does not show any particular reason for the AFEB disconnecting.
It only shows that the reset was initiated by the RE.
MPC: Reset reason: RE Initiated Reset
[Aug 29 07:45:27.172 LOG: Emergency] <160>CMLC: Going disconnected; Routing engine chassis socket closed abruptly
[Aug 29 07:51:01.498 LOG: Emergency] <160>CMLC: Going disconnected; Routing engine chassis socket closed abruptly
[Aug 29 07:56:26.216 LOG: Emergency] <160>CMLC: Going disconnected; Routing engine chassis socket closed abruptly
[Aug 29 08:01:54.835 LOG: Emergency] <160>CMLC: Going disconnected; Routing engine chassis socket closed abruptly
Also at the same time other symptoms appear:
1- Scheduler slips
Aug 29 10:52:51.338 2024 ROUTER rpd[2571]: %DAEMON-3-JTASK_SCHED_SLIP: 4 sec scheduler slip, user: 0 sec 95338 usec, system: 0 sec, 0 usec
Aug 29 10:52:56.947 2024 ROUTER rpd[2571]: %DAEMON-3-JTASK_SCHED_SLIP: 4 sec scheduler slip, user: 0 sec 95655 usec, system: 0 sec, 2055 usec
Aug 29 10:53:09.344 2024 ROUTER rpd[2571]: %DAEMON-3-JTASK_SCHED_SLIP: 4 sec scheduler slip, user: 0 sec 94036 usec, system: 0 sec, 37 usec
Aug 29 10:53:13.838 2024 ROUTER rpd[2571]: %DAEMON-3-JTASK_SCHED_SLIP: 4 sec scheduler slip, user: 0 sec 108361 usec, system: 0 sec, 8 usec
Aug 29 10:53:19.493 2024 ROUTER rpd[2571]: %DAEMON-3-JTASK_SCHED_SLIP: 5 sec scheduler slip, user: 0 sec 124586 usec, system: 0 sec, 2269 usec
Aug 29 10:53:23.940 2024 ROUTER rpd[2571]: %DAEMON-3-JTASK_SCHED_SLIP: 4 sec scheduler slip, user: 0 sec 98964 usec, system: 0 sec, 0 usec
2- Alarm raised for loss of communication with the backup
Aug 29 10:40:39.699 2024 ROUTER craftd[2112]: %DAEMON-4: Minor alarm set, Loss of communication with Backup RE
Aug 29 11:01:19.305 2024 ROUTER craftd[2112]: %DAEMON-4: Minor alarm set, Loss of communication with Backup RE
Analysis of the log shows that, at the same time that the AFEB disconnections and the other symptoms were occurring, there were aggressive activity of SSH connections to the node.
(several connections per minute, for the purpose of information collection)
This comes from one particular host/server in the network that was later found to be misbehaving.
The symptoms stopped when that activity stopped, and the services recovered.
To mitigate the effect of such connections on the MX104 node, (example: limit the number of concurrent ssh connections), the configuration settings under "system->services->ssh" can be modified (reduced)
In particular:
services {
ssh {
max-sessions-per-connection <*>;
connection-limit <*>;
rate-limit <*>;
}