Customer found the license is missing after the MX304 vmhost reboot. /config/license doesnt have the license file. This is seen on JUNOS 22.2R3-S1.9.

Before the vmhost rebooting, license is installed.

{master}
root@mx304> show system license
License usage:
                                 Licenses     Licenses    Licenses
                                  Feature      Feature     Feature
  Feature name                       used    installed      needed  Expiry
  scale-subscriber                      0           10           0    permanent
  scale-l2tp                            0         1000           0    permanent
  bgp                                   1            1           0    2024-12-27 09:00:00 JST
  isis                                  0            1           0    2024-12-27 09:00:00 JST
  bfd-liveness-detection                0            1           0    2024-12-27 09:00:00 JST
  connectivity-fault-management         0            1           0    2024-12-27 09:00:00 JST
  vrrp                                  0            1           0    2024-12-27 09:00:00 JST
  dot1q-tunneling                       0            1           0    2024-12-27 09:00:00 JST
  services-rpm                          0            1           0    2024-12-27 09:00:00 JST
  vpls                                  1            1           0    2024-12-27 09:00:00 JST
  l3vpn                                 1            1           0    2024-12-27 09:00:00 JST
  vxlan                                 0            1           0    2024-12-27 09:00:00 JST
  PCEP                                  0            1           0    2024-12-27 09:00:00 JST
  mc-lag                                0            1           0    2024-12-27 09:00:00 JST
  esi-lag                               0            1           0    2024-12-27 09:00:00 JST
  link-fault-management                 0            1           0    2024-12-27 09:00:00 JST
  ldp                                   0            1           0    2024-12-27 09:00:00 JST
  rsvp                                  1            1           0    2024-12-27 09:00:00 JST
  l2circuit                             0            1           0    2024-12-27 09:00:00 JST
  l3static                              1            1           0    2024-12-27 09:00:00 JST
  l3vpn-instance-capacity               1      1048576           0    2024-12-27 09:00:00 JST
  evpn-vxlan                            0            1           0    2024-12-27 09:00:00 JST
  igmp-multicast                        0            1           0    2024-12-27 09:00:00 JST
  evpn-mpls                             0            1           0    2024-12-27 09:00:00 JST
  sr                                    0            1           0    2024-12-27 09:00:00 JST
  pim                                   0            1           0    2024-12-27 09:00:00 JST
  gre_tunnel                            0            1           0    2024-12-27 09:00:00 JST
  static-mpls                           0            1           0    2024-12-27 09:00:00 JST
  rip                                   0            1           0    2024-12-27 09:00:00 JST
  ospf                                  1            1           0    2024-12-27 09:00:00 JST
  srv6                                  0            1           0    2024-12-27 09:00:00 JST
  qos-license                           1            1           0    2024-12-27 09:00:00 JST
  video-monitoring-MDI                  0            1           0    2024-12-27 09:00:00 JST
  mplsoudp                              0            1           0    2024-12-27 09:00:00 JST
  ipinip                                0            1           0    2024-12-27 09:00:00 JST
  bgp-pic                               0            1           0    2024-12-27 09:00:00 JST
  inline-jflow                          0            1           0    2024-12-27 09:00:00 JST
  PWHT L2                               0            1           0    2024-12-27 09:00:00 JST
  PWHT L3                               0            1           0    2024-12-27 09:00:00 JST
  ADV Bandwidth (in gbps)            1600            0        1500    invalid
  PREM Bandwidth (in gbps)              0          100           0    2024-12-27 09:00:00 JST

After "request vmhost reboot",

root@mx304> show system license |no-more
License usage:
                                 Licenses     Licenses    Licenses
                                  Feature      Feature     Feature
  Feature name                       used    installed      needed  Expiry
  scale-subscriber                      0           10           0    permanent
  scale-l2tp                            0         1000           0    permanent
  bgp                                   1            0           1    invalid
  vpls                                  1            0           1    invalid
  l3vpn                                 1            0           1    invalid
  rsvp                                  1            0           1    invalid
  l3static                              1            0           1    invalid
  ospf                                  1            0           1    invalid
  qos-license                           1            0           1    invalid

Licenses installed: none

Problem can also be seen in a very corner case with very specific sequence as below.
 

1. Both NG-REs are halted;
2. RE1 is booted up online first and remains as backup RE for a while. But as its role is backup, license check will fail and license file will be removed;
3. RE0 is booted up bit later, and for some reason it takes a longer time to be booted online;
4. RE1 takes the master role as no response from RE0;
5. RE0 is booted online and switches to the backup;
6. As license had been removed from the RE1 already, system wont have license installed;
7. Both REs are rebooted again;
8. RE0 is booted as master, and again, since license files had been removed, no license will be installed.

This issue can be seen on all VMHOST RE platform. There have multiple RCA/PRs why the vmhost system deleted the license file after reboot.

Scenario 1:

TSB78046 [juniper.net]/PR1775463, under centain timing related sequecne post a system reboot, the device is not able to validate the license since the chassis serial ID is not available to license-check process, hence the license get removed from the system. If the license is configured via "set system license keys key <>", the system will go into amnesiac mode.

Scenario 2:

Even with PR1775463 fixed release, if the license is installed via "request system license add <>" only, the license might get removed. This is due to a bug, when both REs are rebooted, the master RE comes up as Standby momentarily and in that mode, its missing to add chassis serial ID hence cause the license -check process to remove the licese.

In this scenario, if the license is installed via "set system license keys key <>", then there wont have such probelm.

This problem is fixed via PR1792672.

In both scenarios, customer can manually add the license again to restore the service.