On MX2008 Series routers that has two Next Generation Routing Engines installed running different Junos OS versions, the backup SPMB x may not come online after Routing Engine switchover. This issue is seen if the old primary RE's version is Junos OS Release 17.4R3, but NOT seen if the old primary RE's version is Junos OS Release 18.1R1 +.
This article clarifies that this is not a hardware issue, but a software one. If the Junos OS version is the same on both Routing Engines, the issue is not seen after RE switchover.
The new backup SPMB 0 does not come online after RE switchover, if the old primary RE's version is Junos OS Release 17.4.
########## Junos version Master re0:17.4R3.16 / Backup re1:19.1R3.9 ########## User@mx2008-re0> show system information Model: mx2008 Family: junos Junos: 17.4R3.16 Hostname: mx2008-re0 User@mx2008-re1> show system information Model: mx2008 Family: junos Junos: 19.1R3.9 Hostname: mx2008-re1 User@mx2008-re0> show chassis alarms No alarms currently active User@mx2008-re0> show chassis routing-engine Routing Engine status: Slot 0: Current state Master Election priority Master (default) CPU temperature 66 degrees C / 150 degrees F DRAM 49108 MB (49152 MB installed) Memory utilization 5 percent 5 sec CPU utilization: User 0 percent Background 0 percent Kernel 1 percent Interrupt 0 percent Idle 99 percent 1 min CPU utilization: User 0 percent Background 0 percent Kernel 1 percent Interrupt 0 percent Idle 98 percent 5 min CPU utilization: User 0 percent Background 0 percent Kernel 1 percent Interrupt 0 percent Idle 59 percent 15 min CPU utilization: User 0 percent Background 0 percent Kernel 0 percent Interrupt 0 percent Idle 26 percent Model RE-MX2008-X8-64G Serial ID BUILTIN Start time 2020-10-22 20:47:28 IST Uptime 7 minutes, 22 seconds Last reboot reason 0x2000:hypervisor reboot Load averages: 1 minute 5 minute 15 minute 0.23 0.31 0.20 Routing Engine status: Slot 1: Current state Backup Election priority Backup (default) CPU temperature 61 degrees C / 141 degrees F DRAM 49104 MB (49152 MB installed) Memory utilization 5 percent 5 sec CPU utilization: User 0 percent Background 0 percent Kernel 0 percent Interrupt 0 percent Idle 100 percent Model RE-MX2008-X8-64G Serial ID BUILTIN Start time 2020-10-22 18:26:11 IST Uptime 2 hours, 28 minutes, 40 seconds Last reboot reason 0x2000:hypervisor reboot Load averages: 1 minute 5 minute 15 minute 0.20 0.21 0.23 User@mx2008-re0> show chassis spmb Slot 0 information: State Online Start time: 2020-10-22 20:51:09 IST Uptime: 3 minutes, 42 seconds Slot 1 information: State Online - Standby Start time: 2020-10-22 20:51:09 IST Uptime: 3 minutes, 42 seconds User@mx2008-re0> show chassis environment cb CB 0 status: State Online Master Inlet1 Temperature 39 degrees C / 102 degrees F Inlet2 Temperature 44 degrees C / 111 degrees F Inlet3 Temperature 39 degrees C / 102 degrees F Inlet4 Temperature 40 degrees C / 104 degrees F Exhaust1 Temperature 40 degrees C / 104 degrees F Exhaust2 Temperature 46 degrees C / 114 degrees F Exhaust3 Temperature 43 degrees C / 109 degrees F Exhaust4 Temperature 43 degrees C / 109 degrees F Power 1 1.0 V PHY 995 mV 1.15 V 1153 mV 1.2 V bias 1195 mV 1.5 V 1501 mV 1.8 V 1801 mV 2.5 V 2468 mV 3.3 V bias 3296 mV VCCIO 1040 mV Power 2 1.1 V 1099 mV 3.3 V 3300 mV Power 3 0.95 V XL710 919 mV 1.05 V 1050 mV Power 4 1.2 V 1199 mV 5.0 V 5000 mV Power 5 1.0 V 1000 mV 1.2 V PHY 1199 mV Bus Revision 116 FPGA Revision 2 CB 1 status: State Online Standby Inlet1 Temperature 37 degrees C / 98 degrees F Inlet2 Temperature 41 degrees C / 105 degrees F Inlet3 Temperature 38 degrees C / 100 degrees F Inlet4 Temperature 39 degrees C / 102 degrees F Exhaust1 Temperature 37 degrees C / 98 degrees F Exhaust2 Temperature 43 degrees C / 109 degrees F Exhaust3 Temperature 42 degrees C / 107 degrees F Exhaust4 Temperature 42 degrees C / 107 degrees F Power 1 1.0 V PHY 986 mV 1.15 V 1150 mV 1.2 V bias 1189 mV 1.5 V 1488 mV 1.8 V 1788 mV 2.5 V 2471 mV 3.3 V bias 3296 mV VCCIO 1028 mV Power 2 1.1 V 1100 mV 3.3 V 3300 mV Power 3 0.95 V XL710 949 mV 1.05 V 1050 mV Power 4 1.2 V 1199 mV 5.0 V 5000 mV Power 5 1.0 V 1000 mV 1.2 V PHY 1200 mV Bus Revision 116 FPGA Revision 2 User@mx2008-re0> request chassis routing-engine master switch warning: Traffic will be interrupted while the PFE is re-initialized Toggle mastership between routing engines ? [yes,no] (no) yes Resolving mastership... User@mx2008-re1> show chassis routing-engine Routing Engine status: Slot 0: Current state Backup Election priority Master (default) CPU temperature 65 degrees C / 149 degrees F DRAM 49108 MB (49152 MB installed) Memory utilization 5 percent 5 sec CPU utilization: User 1 percent Background 0 percent Kernel 4 percent Interrupt 0 percent Idle 94 percent Model RE-MX2008-X8-64G Serial ID BUILTIN Start time 2020-10-22 20:47:28 IST Uptime 10 minutes, 28 seconds Last reboot reason 0x2000:hypervisor reboot Load averages: 1 minute 5 minute 15 minute 0.29 0.29 0.20 Routing Engine status: Slot 1: Current state Master Election priority Backup (default) CPU temperature 62 degrees C / 143 degrees F DRAM 49104 MB (49152 MB installed) Memory utilization 5 percent 5 sec CPU utilization: User 0 percent Background 0 percent Kernel 2 percent Interrupt 0 percent Idle 98 percent 1 min CPU utilization: User 1 percent Background 0 percent Kernel 2 percent Interrupt 0 percent Idle 89 percent 5 min CPU utilization: User 0 percent Background 0 percent Kernel 0 percent Interrupt 0 percent Idle 38 percent 15 min CPU utilization: User 0 percent Background 0 percent Kernel 0 percent Interrupt 0 percent Idle 15 percent Model RE-MX2008-X8-64G Serial ID BUILTIN Start time 2020-10-22 18:26:10 IST Uptime 2 hours, 31 minutes, 53 seconds Last reboot reason 0x2000:hypervisor reboot Load averages: 1 minute 5 minute 15 minute 0.25 0.23 0.23 User@mx2008-re1> show chassis alarms 2 alarms currently active Alarm time Class Description 2020-10-22 20:57:26 IST Minor Backup RE Active 2020-10-22 20:57:26 IST Major SPMB 0 not online User@mx2008-re1> show chassis spmb Slot 1 information: State Online Start time: 2020-10-22 20:57:28 IST Uptime: 39 seconds User@mx2008-re1> show chassis environment cb CB 0 status: State Online Standby Inlet1 Temperature 39 degrees C / 102 degrees F Inlet2 Temperature 44 degrees C / 111 degrees F Inlet3 Temperature 40 degrees C / 104 degrees F Inlet4 Temperature 40 degrees C / 104 degrees F Exhaust1 Temperature 40 degrees C / 104 degrees F Exhaust2 Temperature 46 degrees C / 114 degrees F Exhaust3 Temperature 43 degrees C / 109 degrees F Exhaust4 Temperature 44 degrees C / 111 degrees F Power 1 1.0 V PHY 989 mV 1.15 V 1153 mV 1.2 V bias 1195 mV 1.5 V 1495 mV 1.8 V 1798 mV 2.5 V 2468 mV 3.3 V bias 3296 mV VCCIO 1037 mV Power 2 1.1 V 1099 mV 3.3 V 3300 mV Power 3 0.95 V XL710 919 mV 1.05 V 1050 mV Power 4 1.2 V 1200 mV 5.0 V 5000 mV Power 5 1.0 V 1000 mV 1.2 V PHY 1199 mV Bus Revision 116 FPGA Revision 2 CB 1 status: State Online Master Inlet1 Temperature 37 degrees C / 98 degrees F Inlet2 Temperature 41 degrees C / 105 degrees F Inlet3 Temperature 38 degrees C / 100 degrees F Inlet4 Temperature 39 degrees C / 102 degrees F Exhaust1 Temperature 37 degrees C / 98 degrees F Exhaust2 Temperature 43 degrees C / 109 degrees F Exhaust3 Temperature 42 degrees C / 107 degrees F Exhaust4 Temperature 43 degrees C / 109 degrees F Power 1 1.0 V PHY 989 mV 1.15 V 1150 mV 1.2 V bias 1189 mV 1.5 V 1492 mV 1.8 V 1795 mV 2.5 V 2471 mV 3.3 V bias 3296 mV VCCIO 1028 mV Power 2 1.1 V 1099 mV 3.3 V 3300 mV Power 3 0.95 V XL710 949 mV 1.05 V 1050 mV Power 4 1.2 V 1200 mV 5.0 V 5000 mV Power 5 1.0 V 1000 mV 1.2 V PHY 1200 mV Bus Revision 116 FPGA Revision 2
The spmb_restart_timer executes spmb_hw_restart to bring the SPMB online, but SPMB does not come online even after the spmb_hw_restart function is executed continuously.
spmb_restart_timer
spmb_hw_restart
*** messages *** Oct 22 20:59:24.804 mx2008-re1 chassisd[25421]: CHASSISD_SPMB_RESTART_TIMEOUT: Attempt 2 to restart SPMB 0 timed out; trying again Oct 22 21:00:24.806 mx2008-re1 chassisd[25421]: CHASSISD_SPMB_RESTART_TIMEOUT: Attempt 3 to restart SPMB 0 timed out; trying again Oct 22 21:01:24.823 mx2008-re1 chassisd[25421]: CHASSISD_SPMB_RESTART_TIMEOUT: Attempt 4 to restart SPMB 0 timed out; trying again Oct 22 21:02:24.832 mx2008-re1 chassisd[25421]: CHASSISD_SPMB_RESTART_TIMEOUT: Attempt 5 to restart SPMB 0 timed out; trying again Oct 22 21:03:24.846 mx2008-re1 chassisd[25421]: CHASSISD_SPMB_RESTART_TIMEOUT: Attempt 6 to restart SPMB 0 timed out; trying again Oct 22 21:04:24.857 mx2008-re1 chassisd[25421]: CHASSISD_SPMB_RESTART_TIMEOUT: Attempt 7 to restart SPMB 0 timed out; trying again ########## Junos version Master re0:18.1R3.3 / Backup re1:19.1R3.9 ##########
The new backup SPMB 0 comes online after RE switchover, if the old primary RE's Junos OS version is 18.1 +.
User@mx2008-re0> show system information Model: mx2008 Family: junos Junos: 18.1R3.3 Hostname: mx2008-re0 User@mx2008-re1> show system information Model: mx2008 Family: junos Junos: 19.1R3.9 Hostname: mx2008-re1 User@mx2008-re0> show chassis alarms No alarms currently active User@mx2008-re0> show chassis routing-engine |no-more Routing Engine status: Slot 0: Current state Master Election priority Master (default) CPU temperature 64 degrees C / 147 degrees F DRAM 49108 MB (49152 MB installed) Memory utilization 5 percent 5 sec CPU utilization: User 0 percent Background 0 percent Kernel 1 percent Interrupt 0 percent Idle 99 percent 1 min CPU utilization: User 0 percent Background 0 percent Kernel 1 percent Interrupt 0 percent Idle 99 percent 5 min CPU utilization: User 0 percent Background 0 percent Kernel 1 percent Interrupt 0 percent Idle 69 percent 15 min CPU utilization: User 0 percent Background 0 percent Kernel 0 percent Interrupt 0 percent Idle 32 percent Model RE-MX2008-X8-64G Serial ID BUILTIN Start time 2020-10-22 20:17:33 IST Uptime 14 minutes Last reboot reason 0x2000:hypervisor reboot Load averages: 1 minute 5 minute 15 minute 0.24 0.27 0.20 Routing Engine status: Slot 1: Current state Backup Election priority Backup (default) CPU temperature 60 degrees C / 140 degrees F DRAM 49104 MB (49152 MB installed) Memory utilization 5 percent 5 sec CPU utilization: User 0 percent Background 0 percent Kernel 0 percent Interrupt 0 percent Idle 100 percent Model RE-MX2008-X8-64G Serial ID BUILTIN Start time 2020-10-22 18:26:11 IST Uptime 2 hours, 5 minutes, 18 seconds Last reboot reason 0x2000:hypervisor reboot Load averages: 1 minute 5 minute 15 minute 0.12 0.21 0.23 User@mx2008-re0> show chassis spmb Slot 0 information: State Online Start time: 2020-10-22 20:28:05 IST Uptime: 3 minutes, 32 seconds Slot 1 information: State Online - Standby Start time: 2020-10-22 20:28:06 IST Uptime: 3 minutes, 31 seconds User@mx2008-re0> show chassis environment cb CB 0 status: State Online Master Inlet1 Temperature 39 degrees C / 102 degrees F Inlet2 Temperature 44 degrees C / 111 degrees F Inlet3 Temperature 39 degrees C / 102 degrees F Inlet4 Temperature 39 degrees C / 102 degrees F Exhaust1 Temperature 39 degrees C / 102 degrees F Exhaust2 Temperature 46 degrees C / 114 degrees F Exhaust3 Temperature 43 degrees C / 109 degrees F Exhaust4 Temperature 43 degrees C / 109 degrees F Power 1 1.0 V PHY 989 mV 1.15 V 1153 mV 1.2 V bias 1195 mV 1.5 V 1498 mV 1.8 V 1804 mV 2.5 V 2468 mV 3.3 V bias 3296 mV VCCIO 1037 mV Power 2 1.1 V 1099 mV 3.3 V 3299 mV Power 3 0.95 V XL710 919 mV 1.05 V 1050 mV Power 4 1.2 V 1199 mV 5.0 V 5000 mV Power 5 1.0 V 1000 mV 1.2 V PHY 1199 mV Bus Revision 116 FPGA Revision 2 CB 1 status: State Online Standby Inlet1 Temperature 37 degrees C / 98 degrees F Inlet2 Temperature 41 degrees C / 105 degrees F Inlet3 Temperature 38 degrees C / 100 degrees F Inlet4 Temperature 39 degrees C / 102 degrees F Exhaust1 Temperature 37 degrees C / 98 degrees F Exhaust2 Temperature 43 degrees C / 109 degrees F Exhaust3 Temperature 42 degrees C / 107 degrees F Exhaust4 Temperature 42 degrees C / 107 degrees F Power 1 1.0 V PHY 982 mV 1.15 V 1150 mV 1.2 V bias 1189 mV 1.5 V 1492 mV 1.8 V 1791 mV 2.5 V 2471 mV 3.3 V bias 3296 mV VCCIO 1031 mV Power 2 1.1 V 1099 mV 3.3 V 3300 mV Power 3 0.95 V XL710 949 mV 1.05 V 1050 mV Power 4 1.2 V 1200 mV 5.0 V 5000 mV Power 5 1.0 V 1000 mV 1.2 V PHY 1200 mV Bus Revision 116 FPGA Revision 2 User@mx2008-re0> request chassis routing-engine master switch warning: Traffic will be interrupted while the PFE is re-initialized Toggle mastership between routing engines ? [yes,no] (no) yes Resolving mastership... User@mx2008-re1> show chassis routing-engine Routing Engine status: Slot 0: Current state Backup Election priority Master (default) DRAM 49108 MB (49152 MB installed) Memory utilization 0 percent 5 sec CPU utilization: User 0 percent Background 0 percent Kernel 2 percent Interrupt 0 percent Idle 98 percent Model RE-MX2008-X8-64G Serial ID BUILTIN Start time 2020-10-22 20:17:33 IST Uptime 16 minutes, 1 second Last reboot reason 0x2000:hypervisor reboot Load averages: 1 minute 5 minute 15 minute 0.27 0.26 0.20 Routing Engine status: Slot 1: Current state Master Election priority Backup (default) CPU temperature 60 degrees C / 140 degrees F DRAM 49104 MB (49152 MB installed) Memory utilization 5 percent 5 sec CPU utilization: User 0 percent Background 0 percent Kernel 1 percent Interrupt 0 percent Idle 98 percent 1 min CPU utilization: User 2 percent Background 0 percent Kernel 2 percent Interrupt 0 percent Idle 83 percent 5 min CPU utilization: User 1 percent Background 0 percent Kernel 1 percent Interrupt 0 percent Idle 31 percent 15 min CPU utilization: User 0 percent Background 0 percent Kernel 0 percent Interrupt 0 percent Idle 12 percent Model RE-MX2008-X8-64G Serial ID BUILTIN Start time 2020-10-22 18:26:11 IST Uptime 2 hours, 7 minutes, 33 seconds Last reboot reason 0x2000:hypervisor reboot Load averages: 1 minute 5 minute 15 minute 0.38 0.24 0.23 User@mx2008-re1> show chassis alarms 1 alarms currently active Alarm time Class Description 2020-10-22 20:33:20 IST Minor Backup RE Active User@mx2008-re1> show chassis spmb Slot 0 information: State Online - Standby Start time: 2020-10-22 20:33:23 IST Uptime: 25 seconds Slot 1 information: State Online Start time: 2020-10-22 20:33:23 IST Uptime: 25 seconds User@mx2008-re1> show chassis environment cb CB 0 status: State Online Standby Inlet1 Temperature 39 degrees C / 102 degrees F Inlet2 Temperature 43 degrees C / 109 degrees F Inlet3 Temperature 39 degrees C / 102 degrees F Inlet4 Temperature 40 degrees C / 104 degrees F Exhaust1 Temperature 39 degrees C / 102 degrees F Exhaust2 Temperature 45 degrees C / 113 degrees F Exhaust3 Temperature 42 degrees C / 107 degrees F Exhaust4 Temperature 43 degrees C / 109 degrees F Power 1 1.0 V PHY 989 mV 1.15 V 1153 mV 1.2 V bias 1195 mV 1.5 V 1495 mV 1.8 V 1798 mV 2.5 V 2468 mV 3.3 V bias 3296 mV VCCIO 1040 mV Power 2 1.1 V 1099 mV 3.3 V 3300 mV Power 3 0.95 V XL710 919 mV 1.05 V 1050 mV Power 4 1.2 V 1200 mV 5.0 V 5000 mV Power 5 1.0 V 1000 mV 1.2 V PHY 1200 mV Bus Revision 116 FPGA Revision 2 CB 1 status: State Online Master Inlet1 Temperature 37 degrees C / 98 degrees F Inlet2 Temperature 40 degrees C / 104 degrees F Inlet3 Temperature 38 degrees C / 100 degrees F Inlet4 Temperature 39 degrees C / 102 degrees F Exhaust1 Temperature 37 degrees C / 98 degrees F Exhaust2 Temperature 42 degrees C / 107 degrees F Exhaust3 Temperature 42 degrees C / 107 degrees F Exhaust4 Temperature 42 degrees C / 107 degrees F Power 1 1.0 V PHY 986 mV 1.15 V 1150 mV 1.2 V bias 1189 mV 1.5 V 1488 mV 1.8 V 1782 mV 2.5 V 2471 mV 3.3 V bias 3296 mV VCCIO 1028 mV Power 2 1.1 V 1099 mV 3.3 V 3300 mV Power 3 0.95 V XL710 950 mV 1.05 V 1050 mV Power 4 1.2 V 1199 mV 5.0 V 5000 mV Power 5 1.0 V 1000 mV 1.2 V PHY 1199 mV Bus Revision 116 FPGA Revision 2 User@mx2008-re0> show chassis hardware Hardware inventory: Item Version Part number Serial number Description Chassis JN1263D3CAFL MX2008 Midplane REV 21 750-067705 ABAD3885 Midplane PMP REV 01 711-051406 ABAD4065 Power Midplane FPM Board REV 07 760-068193 ABDL4524 Front Panel Display PSM 0 REV 08 740-050037 1EDB6490CBV DC 52V Power Supply Module PSM 1 REV 08 740-050037 1EDB6490C26 DC 52V Power Supply Module PSM 2 REV 08 740-050037 1EDB6490BX7 DC 52V Power Supply Module PSM 3 REV 08 740-050037 1EDB6490C5R DC 52V Power Supply Module PSM 4 REV 08 740-050037 1EDB707005K DC 52V Power Supply Module PSM 5 REV 08 740-050037 1EDB6490C8R DC 52V Power Supply Module PSM 6 REV 08 740-050037 1EDB6490C8L DC 52V Power Supply Module PSM 7 REV 08 740-050037 1EDB6490BTX DC 52V Power Supply Module PDM 0 REV 03 740-050036 1EFD6471150 DC Power Dist Module PDM 1 REV 03 740-050036 1EFD3390110 DC Power Dist Module Routing Engine 0 BUILTIN BUILTIN RE-MX2008-X8-64G Routing Engine 1 BUILTIN BUILTIN RE-MX2008-X8-64G CB 0 REV 17 750-067373 ABDN3022 Control Board CB 1 REV 13 750-067373 ABDK8165 Control Board SFB 0 REV 13 750-067371 ABDK5367 MX2008 Switch Fabric Board 2 SFB 1 REV 13 750-067371 ABDK5425 MX2008 Switch Fabric Board 2 SFB 2 REV 13 750-067371 ABDN3287 MX2008 Switch Fabric Board 2 SFB 3 REV 13 750-067371 ABDK5362 MX2008 Switch Fabric Board 2 SFB 4 REV 13 750-067371 ABDK5424 MX2008 Switch Fabric Board 2 SFB 5 REV 13 750-067371 ABDK5358 MX2008 Switch Fabric Board 2 SFB 6 REV 13 750-067371 ABDK5419 MX2008 Switch Fabric Board 2 SFB 7 REV 13 750-067371 ABDK5349 MX2008 Switch Fabric Board 2 ADC 4 REV 19 750-043596 ABCV1289 Adapter Card Fan Tray 0 REV 03 760-052467 ACAZ1393 172mm FanTray - 6 Fans Fan Tray 1 REV 03 760-052467 ACAZ1446 172mm FanTray - 6 Fans
The problem is seen if the old primary RE's Junos OS version is 17.4R3, but NOT seen if the old primary RE's Junos OS version is 18.1R1 +, which means that the issue is fixed in Junos OS Release 18.1R1.
This is not a hardware issue, but a software issue. If the Junos OS version is the same on both Routing Engines, the SPMB x problem is not seen after RE switchover.
Note: Replaced words that failed to represent the inclusion and diversity Juniper values; in this case master changed to primary in context