MX2K fabric link errors on various SFBs and FPCs

Description

In a rare case fabric link errors may be seen on a number of separate planes and FPCs.
Those link errors will trigger several chassis alarms:

user@router> show chassis alarms
5 alarms currently active
Alarm time Class Description
2024-04-12 17:27:27 CEST Minor Check plane 3 Fabric Chip
2024-04-09 21:54:34 CEST Minor Check plane 0 Fabric Chip
2024-04-09 09:17:12 CEST Minor Check plane 1 Fabric Chip
2024-04-07 19:33:48 CEST Minor Check plane 6 Fabric Chip
2024-03-28 11:14:21 CET Minor Check plane 7 Fabric Chip

Symptoms

Usually, link errors are seen on one fabric link or on a single fabric plane or on a single FPC. Which is mostly caused by a faulty FPC or SFB.
However, in case of clocking source failure (or deterioration) on master control board (CB) we may see link errors on many FPCs and SFBs.

user@router> show log messages | match HSL
Apr 7 19:33:48.221 router spmb1 CMTY_FM_HSL2: FM_HSL2_LINK_ERROR on pb_slot[11], plane[20], pfe[2]
Apr 8 05:26:09.057 router spmb1 CMTY_FM_HSL2: FM_HSL2_LINK_ERROR on pb_slot[13], plane[22], pfe[2]
Apr 10 04:56:47.340 router spmb1 CMTY_FM_HSL2: CRC error on pb_slot[15], plane[4], pfe[2]
Apr 10 04:56:47.340 router spmb1 CMTY_FM_HSL2: FM_HSL2_LINK_ERROR on pb_slot[15], plane[4], pfe[2]
Apr 10 06:23:30.455 router spmb1 CMTY_FM_HSL2: FM_HSL2_LINK_ERROR on pb_slot[11], plane[23], pfe[3]
Apr 12 17:27:26.101 router spmb1 CMTY_FM_HSL2: FM_HSL2_LINK_ERROR on pb_slot[15], plane[11], pfe[1]
Apr 13 10:05:04.233 router spmb1 CMTY_FM_HSL2: FM_HSL2_LINK_ERROR on pb_slot[11], plane[23], pfe[2]

user@router> show chassis fabric summary extended
Plane State Link Link Destination errors Uptime
Error TF Local / Remote
0 Online NO NO NO / NO 27 days, 23 hours, 50 minutes, 18 seconds
1 Online NO NO NO / NO 27 days, 23 hours, 50 minutes, 18 seconds
2 Online NO NO NO / YES 27 days, 23 hours, 50 minutes, 17 seconds
3 Online NO NO NO / NO 18 days, 4 hours, 24 minutes, 20 seconds
4 Online YES NO NO / NO 18 days, 4 hours, 24 minutes, 20 seconds
5 Online NO NO NO / YES 18 days, 4 hours, 24 minutes, 19 seconds
6 Online NO NO NO / NO 65 days, 20 hours, 53 minutes, 56 seconds
7 Online NO NO NO / NO 65 days, 20 hours, 53 minutes, 56 seconds
8 Online NO NO NO / NO 65 days, 20 hours, 53 minutes, 55 seconds
9 Online NO NO NO / NO 335 days, 23 hours, 20 minutes, 32 seconds
10 Online NO NO NO / NO 335 days, 23 hours, 20 minutes, 32 seconds
11 Online YES NO NO / NO 335 days, 23 hours, 20 minutes, 32 seconds
12 Online NO NO NO / NO 335 days, 23 hours, 20 minutes, 32 seconds
13 Online NO NO NO / NO 335 days, 23 hours, 20 minutes, 32 seconds
14 Online NO NO NO / NO 335 days, 23 hours, 20 minutes, 32 seconds
15 Online NO NO NO / NO 18 days, 4 hours, 20 minutes, 29 seconds
16 Online NO NO NO / NO 18 days, 4 hours, 20 minutes, 29 seconds
17 Online NO NO NO / NO 18 days, 4 hours, 20 minutes, 29 seconds
18 Online NO NO NO / NO 18 days, 4 hours, 17 minutes, 1 second
19 Online NO NO NO / NO 18 days, 4 hours, 17 minutes
20 Online YES NO NO / NO 18 days, 4 hours, 16 minutes, 59 seconds
21 Online NO NO NO / NO 18 days, 4 hours, 14 minutes, 17 seconds
22 Online YES NO YES / NO 18 days, 4 hours, 14 minutes, 17 seconds
23 Online YES NO YES / YES 18 days, 4 hours, 14 minutes, 16 seconds

Solution

In a maintenance window:

Clear existing errors by restarting fabric planes with errors one by one
- request chassis fabric plane offline <plane_number>
- wait
- request chassis fabric plane online <plane_number>
In a clean state after clearing all fabric errors, perform RE switchover so that another CB becomes master.
- request chassis routing-engine master switch
On another CB acting as master, monitor to see whether the errors return.
If the errors stop, replace the faulty CB-RE (combined Routing Engine and Control Board unit)

Modification History

2024-05-29 : Article Created