Customer hit an issue on a PTX10001-36MR (EV631) running 21.2R2-S2 in which once a commit is made to the box, can be anything as small as an interface description change, we see the OSPF LSDB and LDP labels flushed/reallocated. This causes downstream devices to drop L2VPN services and causes actual service impact. The PTX is a P router, and connected downstream are some Cisco NCS and Cisco ASR9k platforms that do the PE function.
Before configuration change, ospf routes are up stable. labroot@mr-r2009# run show route protocol ospf | match metric 192.168.1.5/32 *[OSPF/10/10] 00:02:20, metric 1000 make any conf changes and commit labroot@mr-r2009# set interfaces et-0/0/1 description 111111 OSPF routes are refreshed labroot@mr-r2009# run show route protocol ospf | match metric 192.168.1.5/32 *[OSPF/10/10] 00:00:08, metric 1000
Customer's problem has a very specific configuration with macsec and a very special configuration sequence. Problem can be seen with below sequence:
Due to a bug, it causes every configuration field being marked as "changed", hence leads to the routes flapping after each commit.
For example,
macsec {
connectivity-association test {
cipher-suite gcm-aes-256;
security-mode static-cak;
mka {
sak-rekey-interval 1800;
key-server-priority 4;
}
include-sci;
pre-shared-key {
ckn 01;
cak "$9$I8TRylKM8"; ## SECRET-DATA
After commit, warning can be seen:
[edit security macsec]
'connectivity-association test'
warning: To maximize security, recommend configuring all 64 hexadecimal digits of pre-shared-key ckn
commit complete
Now make any configure change
labroot@mr-r2009# set interfaces et-0/0/1 description 111111
If we check the MGD view of the change, everything is marked with "changed" after.
labroot@mr-r2009# show | display xml | display mark-changed | display changed | match changed
<configuration junos:changed-seconds="1714525323" junos:changed-localtime="2024-05-01 11:02:03 AEST" junos:changed="changed">
<version junos:changed="changed">23.3R1.8-EVO</version>
<groups protect="protect" junos:changed="changed">
<name junos:changed="changed" junos:mark-changed="mark-changed">default</name>
<system junos:changed="changed" junos:mark-changed="mark-changed">
<host-name junos:changed="changed">mr-r2009</host-name>
<root-authentication junos:changed="changed">
<encrypted-password junos:changed="changed">$$$$$$$$$$$/</encrypted-password>
<login junos:changed="changed">
<user junos:changed="changed">
<name junos:changed="changed">labroot</name>
<uid junos:changed="changed">2012</uid>
<class junos:changed="changed">super-user</class>
<authentication junos:changed="changed">
<encrypted-password junos:changed="changed">$$$$$$$$$$/</encrypted-password>
<services junos:changed="changed">
<ssh junos:changed="changed">
<ftp junos:changed="changed">
<telnet junos:changed="changed">
<netconf junos:changed="changed">
<web-management junos:changed="changed">
<undocumented><http junos:changed="changed">
<backup-router junos:changed="changed">
<address junos:changed="changed">10.219.38.1</address>
<time-zone junos:changed="changed">Asia/Calcutta</time-zone>
<name-server junos:changed="changed">
<name junos:changed="changed">10.219.2.101</name>
<name junos:changed="changed">10.219.2.102</name>
<name junos:changed="changed">10.215.194.50</name>
<ntp junos:changed="changed">
<server junos:changed="changed">
<name junos:changed="changed">10.219.0.35</name>
<interfaces junos:changed="changed" junos:mark-changed="mark-changed">
<interface junos:changed="changed">
<name junos:changed="changed">re0:mgmt-0</name>
<unit junos:changed="changed">
<name junos:changed="changed">0</name>
<family junos:changed="changed">
<inet junos:changed="changed">
<address junos:changed="changed">
<name junos:changed="changed">10.219.38.17/26</name>
<routing-options junos:changed="changed" junos:mark-changed="mark-changed">
<static junos:changed="changed">
<route junos:changed="changed">
<name junos:changed="changed">0.0.0.0/0</name>
<next-hop junos:changed="changed">10.219.38.1</next-hop>
...
<truncated, including protocols configure>
This problem is fixed as part of PR1744917, which had been committed in 23.4R1 and onwards.
At this stage, back port the fix into earlier release is in discussion but need careful consideration to avoid causing other problem.
With the JUNOS release prior to 23.4R1, the workaround can be set the macsec pre-shared-key ckn/cak with all the 64 hexadecimal digits to get out of this issue.