Description

During normal operation of MX960 with single NGRE, 2 nos SCBE3 and 4 MPC, we can see less than10% chassisd. This increases with redundant RE and additional SCBE3.

Symptoms

Greater than 17% chassisd is observed in a setup with Dual NGRE, 3*SCBE3, 7*MPC (4 MPC4E, 2 MPC10E, 1 MPC3E). 

​root@router> show system processes extensive | match chass
8304 root 30 0 876M 99392K select 2 11:54 17.29% chassisd{CHASSISD.EN_US}
8304 root -26 r26 876M 99392K select 2 0:01 0.00% chassisd{CHASSISD.EN_US}

Solution

SCBE3
The SCBE3 is a next generation board with 24 retimers on it, which means there are aproximately 59 sensors to be periodically monitored.

SCBE2
The SCBE2 has a previous generation fabric chip and no retimers. On SCBE2, there are only 6 sensors to monitor. 

==== SCBE3 59 sensors ===
root@router> show chassis hardware | match CB                       
CB 0             REV 03   750-070866   CAJF0312          Enhanced MX SCB 3
CB 1             REV 07   750-070866   CAJP0446          Enhanced MX SCB 3

root@router> show chassis environment | match "CB 1" | count    
Count: 59 lines

root@router> show chassis environment | match "CB 1"            
      CB 1 Intake                    OK         40 degrees C / 104 degrees F
      CB 1 Exhaust A                 OK         41 degrees C / 105 degrees F
      CB 1 Exhaust B                 OK         50 degrees C / 122 degrees F
      CB 1 PMBus POLs (Highest)      OK         59 degrees C / 138 degrees F
      CB 1 MAX20743_RT POL (Highest) OK         52 degrees C / 125 degrees F
      CB 1 RT BCM59850 (Highest)     OK         52 degrees C / 125 degrees F
      CB 1 ZF 0 External MAX6657     OK         57 degrees C / 134 degrees F
      CB 1 ZF 0 Internal (Highest)   OK         55 degrees C / 131 degrees F
      CB 1 ZF 0 Internal Main        OK         53 degrees C / 127 degrees F
      CB 1 ZF 0 Internal Remote 0    OK         54 degrees C / 129 degrees F
      CB 1 ZF 0 Internal Remote 1    OK         55 degrees C / 131 degrees F
      CB 1 ZF 0 Internal Remote 2    OK         54 degrees C / 129 degrees F
      CB 1 ZF 0 Internal Remote 3    OK         54 degrees C / 129 degrees F
      CB 1 ZF 0 Internal Remote 4    OK         54 degrees C / 129 degrees F
      CB 1 ZF 0 Internal Remote 5    OK         54 degrees C / 129 degrees F
      CB 1 ZF 0 Internal Remote 6    OK         54 degrees C / 129 degrees F
      CB 1 RT BCM59850_00 Core 0     OK         50 degrees C / 122 degrees F
      CB 1 RT BCM59850_00 Core 1     OK         52 degrees C / 125 degrees F
      CB 1 RT BCM59850_01 Core 0     OK         49 degrees C / 120 degrees F
      CB 1 RT BCM59850_01 Core 1     OK         49 degrees C / 120 degrees F
      CB 1 RT BCM59850_02 Core 0     OK         45 degrees C / 113 degrees F
      CB 1 RT BCM59850_02 Core 1     OK         46 degrees C / 114 degrees F
      CB 1 RT BCM59850_15 Core 0     OK         50 degrees C / 122 degrees F
      CB 1 RT BCM59850_15 Core 1     OK         47 degrees C / 116 degrees F
      CB 1 RT BCM59850_16 Core 0     OK         46 degrees C / 114 degrees F
      CB 1 RT BCM59850_16 Core 1     OK         48 degrees C / 118 degrees F
      CB 1 RT BCM59850_17 Core 0     OK         42 degrees C / 107 degrees F
      CB 1 RT BCM59850_17 Core 1     OK         47 degrees C / 116 degrees F
      CB 1 MAX20743_PLL_VDD_1V25 T1  OK         46 degrees C / 114 degrees F
      CB 1 MAX20730_VDD_5V0 T1       OK         36 degrees C / 96 degrees F
      CB 1 MAX20730_VDD_3V3 T1       OK         42 degrees C / 107 degrees F
      CB 1 MAX20730_VDD_2V5 T1       OK         45 degrees C / 113 degrees F
      CB 1 MAX20730_VDD_1V8 T1       OK         43 degrees C / 109 degrees F
      CB 1 MAX20743_VDD_1V2 T1       OK         50 degrees C / 122 degrees F
      CB 1 MAX20730_VDD_1V0 T1       OK         37 degrees C / 98 degrees F
      CB 1 MAX20754_ZF0_VDD_0V75 T1  OK         44 degrees C / 111 degrees F
      CB 1 MAX20754_ZF0_VDD_0V75 T2  OK         51 degrees C / 123 degrees F
      CB 1 MAX20754_ZF0_VDDM_0V8 T1  OK         44 degrees C / 111 degrees F
      CB 1 MAX20754_ZF0_VDDM_0V8 T2  OK         48 degrees C / 118 degrees F
      CB 1 MAX20754_ZF0_AVDD_0V9 T1  OK         43 degrees C / 109 degrees F
      CB 1 MAX20754_ZF0_AVDD_0V9 T2  OK         49 degrees C / 120 degrees F
      CB 1 MAX20743_ZF0_AVDDH_1V1 T1 OK         50 degrees C / 122 degrees F
      CB 1 MAX20754_RT00_11_AVDD_0V8 T1 OK      43 degrees C / 109 degrees F
      CB 1 MAX20754_RT00_11_AVDD_0V8 T2 OK      59 degrees C / 138 degrees F
      CB 1 MAX20754_RT12_23_AVDD_0V8 T1 OK      41 degrees C / 105 degrees F
      CB 1 MAX20754_RT12_23_AVDD_0V8 T2 OK      50 degrees C / 122 degrees F
      CB 1 MAX20743_RT00_02_DVDD_0V8 T1 OK      49 degrees C / 120 degrees F
      CB 1 MAX20743_RT03_05_DVDD_0V8 T1 OK      52 degrees C / 125 degrees F
      CB 1 MAX20743_RT06_08_DVDD_0V8 T1 OK      50 degrees C / 122 degrees F
      CB 1 MAX20743_RT09_11_DVDD_0V8 T1 OK      46 degrees C / 114 degrees F
      CB 1 MAX20743_RT12_14_DVDD_0V8 T1 OK      47 degrees C / 116 degrees F
      CB 1 MAX20743_RT15_17_DVDD_0V8 T1 OK      47 degrees C / 116 degrees F
      CB 1 MAX20743_RT18_20_DVDD_0V8 T1 OK      46 degrees C / 114 degrees F
      CB 1 MAX20743_RT21_23_DVDD_0V8 T1 OK      45 degrees C / 113 degrees F
      CB 1 BMR453_A T1               OK         47 degrees C / 116 degrees F
      CB 1 BMR453_A T2               OK         45 degrees C / 113 degrees F
      CB 1 BMR453_B T1               OK         53 degrees C / 127 degrees F
      CB 1 BMR453_B T2               OK         52 degrees C / 125 degrees F
      CB 1 PIM4820 T1                OK         53 degrees C / 127 degrees F

==== SCBE2 6 sensors ===
root@router> show chassis hardware | match "CB"                 
CB 0             REV 07   750-062572   CAHV9495          Enhanced MX SCB 2
CB 1             REV 01   750-055976   CACZ9092          Enhanced MX SCB 2

root@router> show chassis environment | match "CB 0" | count    
Count: 6 lines

root@router> show chassis environment | match "CB 0"            
      CB 0 Intake                    OK         39 degrees C / 102 degrees F
      CB 0 Exhaust A                 OK         38 degrees C / 100 degrees F
      CB 0 Exhaust B                 OK         43 degrees C / 109 degrees F
      CB 0 ACBC                      OK         42 degrees C / 107 degrees F
      CB 0 XF A                      OK         47 degrees C / 116 degrees F
      CB 0 XF B                      OK         47 degrees C / 116 degrees F

This is expected behavior with SCBE3. SCBE3 is the next generation control board. This comes with more sensors and monitors many sensors compared to older SCBE2. Hence, the chassisd is expected to spend more time on SCBE3.

  • SCBE2 has 6 sensors/CB being monitored.
  • SCBE3 has 59 sensors being monitored by CB.

Comparing SCBE3 vs SCBE2 and explanation of why the chassisd utilization is more in SCBE3 vs SCBE2:

  1. On SCBE3, the chipset itself has more sensors compared to that on SCBE2.
  2. On SCBE3, there are 24 retimers on the board vs 0 retimer on SCBE2.
  3. The number of sensors to be monitored on SCBE3 has increased by ~10 times vs the SCBE2 for each CB.
  4. The higher number of SCBE3 in the system will result in more work for the EM policy in chassisd.

This high % of chassisd will not have any major impact on the memory and CPU usage of the other daemons in the device.

Modification History

2022-02-25: Updated title with more clarity.