Description

A lot of intermittent CB alarms are seen, which are set and cleared.

Symptoms

From the log messages, a lot of CB alarms logs are seen. The alarms set and cleared almost at the same time. The same occurs on RE1 as well.

May 25 13:24:48.386 2021 hostname craftd[7281]: %DAEMON-4: Major alarm set, CB 0 Failure 
May 25 13:24:49.386 2021 hostname alarmd[10448]: %DAEMON-4: Alarm cleared: CB color=RED, class=CHASSIS, reason=CB 0 Failure 
May 25 13:24:49.386 2021 hostname craftd[7281]: %DAEMON-4: Major alarm cleared, CB 0 Failure 
May 25 13:24:50.977 2021 hostname alarmd[10448]: %DAEMON-4: Alarm set: CB color=RED, class=CHASSIS, reason=CB 0 Failure
May 25 13:24:50.977 2021 hostname craftd[7281]: %DAEMON-4: Major alarm set, CB 0 Failure 
May 25 13:24:51.978 2021 hostname alarmd[10448]: %DAEMON-4: Alarm cleared: CB color=RED, class=CHASSIS, reason=CB 0 Failure 
May 25 13:24:51.978 2021 hostname craftd[7281]: %DAEMON-4: Major alarm cleared, CB 0 Failure 
May 25 13:25:10.284 2021 hostname alarmd[10448]: %DAEMON-4: Alarm set: CB color=RED, class=CHASSIS, reason=CB 0 Failure
May 25 13:25:10.284 2021 hostname craftd[7281]: %DAEMON-4: Major alarm set, CB 0 Failure 
May 25 13:25:12.285 2021 hostname alarmd[10448]: %DAEMON-4: Alarm cleared: CB color=RED, class=CHASSIS, reason=CB 0 Failure 
May 25 13:25:12.285 2021 hostname craftd[7281]: %DAEMON-4: Major alarm cleared, CB 0 Failure 
May 25 13:32:45.031 2021 hostname alarmd[10448]: %DAEMON-4: Alarm set: CB color=RED, class=CHASSIS, reason=CB 0 Failure
May 25 13:32:45.031 2021 hostname craftd[7281]: %DAEMON-4: Major alarm set, CB 0 Failure 
May 25 13:32:46.032 2021 hostname alarmd[10448]: %DAEMON-4: Alarm cleared: CB color=RED, class=CHASSIS, reason=CB 0 Failure 
May 25 13:32:46.032 2021 hostname craftd[7281]: %DAEMON-4: Major alarm cleared, CB 0 Failure 
May 25 13:32:49.890 2021 hostname alarmd[10448]: %DAEMON-4: Alarm set: CB color=RED, class=CHASSIS, reason=CB 0 Failure
May 25 13:32:49.890 2021 hostname craftd[7281]: %DAEMON-4: Major alarm set, CB 0 Failure 
May 25 13:32:50.890 2021 hostname alarmd[10448]: %DAEMON-4: Alarm cleared: CB color=RED, class=CHASSIS, reason=CB 0 Failure 
May 25 13:32:50.890 2021 hostname craftd[7281]: %DAEMON-4: Major alarm cleared, CB 0 Failure 
May 25 13:32:56.497 2021 hostname alarmd[10448]: %DAEMON-4: Alarm set: CB color=RED, class=CHASSIS, reason=CB 0 Failure
May 25 13:32:56.497 2021 hostname craftd[7281]: %DAEMON-4: Major alarm set, CB 0 Failure 
May 25 13:32:57.510 2021 hostname alarmd[10448]: %DAEMON-4: Alarm cleared: CB color=RED, class=CHASSIS, reason=CB 0 Failure 

Solution

The chassisd log shows the CB alarm is accompanied by FPM logs. Too many interrupts were raised due interaction with FPM. The logs do not indicate whether the FPM itself is faulty or if a CB component caused the failed interrupts to be raised. 

Remove FPM first, then observe if the logs stop. Install the FPM back to the chassis. If the logs still appear, then a RMA is needed for this FPM.
 

May 25 02:03:43  stormchaser_get_cps: last call time=521289:924708   now time=521290:146367  dt=0:221659
May 25 02:03:44 CHASSISD_FPM_STORM_STATUS: fpm_tiny_tcb_storm_state_change: tcb storm active
May 25 02:03:44  send: red alarm set, device CB 0, reason CB 0 Failure
May 25 02:03:44  stormchaser_get_cps: last call time=521289:924708   now time=521291:148371  dt=1:223663
May 25 02:03:44  fpm_tiny_tcb_intr tcb_ints_pending 0x30210000
 
May 25 02:03:44  fpm_tiny_CH_PRS_CHG handling CH_PRS interrupt
 
May 25 02:03:44  fpm_tiny_tcb_intr re-enabling interrupts (0x00038000)
 
May 25 02:03:44  exit fpm_tiny_tcb_intr ...
 
May 25 02:03:45 CHASSISD_FPM_STORM_STATUS: fpm_tiny_tcb_storm_state_change: tcb storm active
May 25 02:03:45  send: red alarm clear, device CB 0, reason CB 0 Failure
May 25 02:03:46  stormchaser_get_cps: last call time=521291:148371   now time=521293:546373  dt=2:398002
May 25 02:03:46  fpm_tiny_tcb_intr tcb_ints_pending 0x30210000
 
May 25 02:03:46  fpm_tiny_CH_PRS_CHG handling CH_PRS interrupt
 
May 25 02:03:46  fpm_tiny_tcb_intr re-enabling interrupts (0x00038000)
 
May 25 02:03:46  exit fpm_tiny_tcb_intr ...