This document helps to resolve application dot1xd and rpd-sa-agent failure alarm seen after downgrade from version 23.4R2.14-EVO to 21.4R3.13-EVO.

• "request system software rollback" command is used to downgrade version from 23.4R2.14-EVO to 21.4R3.13-EVO.
• Validation failure error occurred while software rollback.
• Used "no-validate" option to continue downgrade to 21 version.
• After downgrade to version 21.4R3.13-EVO, below alarms are present on device:

2024-10-10 06:55:11 UTC Major Application dot1xd fail on node Re0

2024-10-10 06:55:11 UTC Major Application rpd-sa-agent fail on node Re0

2024-10-11 08:48:05 UTC Major Application bfdd fail on node Re1

2024-10-11 08:58:48 UTC Major Application mustd fail on node Re1

• Checked error message in "/var/log/validation_config.log" file. The validation check during software rollback was failing due to license key error.

 <progress-indicator>

<timestamp junos:seconds="1728820969">2024-10-13 12:02:49.151593 UTC</timestamp>

<message>Started running mustd validator hooks</message>

</progress-indicator>

[edit system license keys]

  'key "DemolabJUNOS603464990 aeaqia qfjbdd anrrcc akcaub bthctt 4xaab6 qaacqe g44kop s4ac3r waakd5 dtrjz6 lqaaaa aebdbt rjz6lq aaaaae bc7trj z6lqaa aaaebi cfwofh hzoaaa aaaqfa ixzyu4 7fyaaa vdaaub dlhctt 4xaaaa aaicqe n44kop s4aaad 5aakar ztrjz6 lqaab6 qabicd 6ofhhz oaaaap uafaiq zyu47f yaaab6 qaud5h hctt4x aaaaaa iclxhc tt4xaa aaaaic kdhctt 4xaaaa aaimcb fhk3tj obsxei cjnz2g k4tomf waokao tj742k psupf4 fctpv4 ibsng7 fom2ga qnkovh 5cuoxm bkr5xj 22fmdc rszinr 3tz6my"'

    ConnectError(host: None, msg: Capability exchange timed out)

<progress-indicator>

<timestamp junos:seconds="1728821034">2024-10-13 12:03:54.813366 UTC</timestamp>

<message>Finished running mustd validator hooks</message>

</progress-indicator>

Exception in thread session (most likely raised during interpreter shutdown):

mgd: error: commit failed: (validation hook evaluation failed)

mgd: Commit failed, cleanup checked out files



 

Root-Cause:

While installing the license key via config mode, validation of the license key is done by mustd using the licenseValidation.py file.

The customer is installing a node-lock license key which requires a device serial ID to validate the license key.

Device serial ID can be fetched in two ways:

1. "show chassis hardware" rpc

2. Reading "/tmp/root/chas_ser_num.id"

But, during software upgradation with no "no-validate" option, mustd script was unable to fetch the device serial ID due to mgd -N not being up and "/tmp/root/chas_ser_num.id" is not available . This leads to issue to install the license key during the software upgrades.

To fix the issue, license-infra will store the device serial ID info in the "/var/db/script/validator" path which is the persistent path, and fetch the device serial ID when installation the license key via config mode.

Note: Both the current running device and upgrade device should have a fix to avoid the issue.

If the current running device doesn't have a fix, the user needs to copy the serial ID manually in the /var/db/scripts/validator path.

cp /tmp/root/chas_ser_num.id /var/db/scripts/validator/device_serial_id

Restoration-Steps:

Installing the license key using op mode "request system license add"