Description

Customer hit an issue on a PTX10001-36MR (EV631) running 21.2R2-S2 in which once a commit is made to the box, can be anything as small as an interface description change, we see the OSPF LSDB and LDP labels flushed/reallocated. This causes downstream devices to drop L2VPN services and causes actual service impact. The PTX is a P router, and connected downstream are some Cisco NCS and Cisco ASR9k platforms that do the PE function.

Symptoms

Before configuration change, ospf routes are up stable.
    labroot@mr-r2009# run show route protocol ospf | match metric   
    192.168.1.5/32     *[OSPF/10/10] 00:02:20, metric 1000

make any conf changes and commit
    labroot@mr-r2009# set interfaces et-0/0/1 description 111111

OSPF routes are refreshed
    labroot@mr-r2009# run show route protocol ospf | match metric   
    192.168.1.5/32     *[OSPF/10/10] 00:00:08, metric 1000


Solution

Customer's problem has a very specific configuration with macsec and a very special configuration sequence. Problem can be seen with below sequence:

  1. [macsec connectivity-association xxx pre-shared-key ckn|cak] is configured, while the ckn or cak does not use all the 64 hexadecimal digits, which will generate waring message as can be seen in the following example;
  2. routing protocols or other fileds are configured after.

Due to a bug, it causes every configuration field being marked as "changed", hence leads to the routes flapping after each commit.


For example,

macsec {

  connectivity-association test {

    cipher-suite gcm-aes-256;

    security-mode static-cak;

    mka {

      sak-rekey-interval 1800;

      key-server-priority 4;

    }

    include-sci;

    pre-shared-key {

      ckn 01;

      cak "$9$I8TRylKM8"; ## SECRET-DATA

    }

  }

}


After commit, warning can be seen:

[edit security macsec]

 'connectivity-association test'

  warning: To maximize security, recommend configuring all 64 hexadecimal digits of pre-shared-key ckn

commit complete


Now make any configure change

labroot@mr-r2009# set interfaces et-0/0/1 description 111111


If we check the MGD view of the change, everything is marked with "changed" after.

labroot@mr-r2009# show | display xml | display mark-changed | display changed | match changed   

  <configuration junos:changed-seconds="1714525323" junos:changed-localtime="2024-05-01 11:02:03 AEST" junos:changed="changed">

      <version junos:changed="changed">23.3R1.8-EVO</version>

      <groups protect="protect" junos:changed="changed">

        <name junos:changed="changed" junos:mark-changed="mark-changed">default</name>

        <system junos:changed="changed" junos:mark-changed="mark-changed">

          <host-name junos:changed="changed">mr-r2009</host-name>

          <root-authentication junos:changed="changed">

            <encrypted-password junos:changed="changed">$$$$$$$$$$$/</encrypted-password>

          <login junos:changed="changed">

            <user junos:changed="changed">

              <name junos:changed="changed">labroot</name>

              <uid junos:changed="changed">2012</uid>

              <class junos:changed="changed">super-user</class>

              <authentication junos:changed="changed">

                <encrypted-password junos:changed="changed">$$$$$$$$$$/</encrypted-password>

          <services junos:changed="changed">

            <ssh junos:changed="changed">

            <ftp junos:changed="changed">

            <telnet junos:changed="changed">

            <netconf junos:changed="changed">

              <ssh junos:changed="changed">

            <web-management junos:changed="changed">

              <undocumented><http junos:changed="changed">

          <backup-router junos:changed="changed">

            <address junos:changed="changed">10.219.38.1</address>

          <time-zone junos:changed="changed">Asia/Calcutta</time-zone>

          <name-server junos:changed="changed">

            <name junos:changed="changed">10.219.2.101</name>

          <name-server junos:changed="changed">

            <name junos:changed="changed">10.219.2.102</name>

          <name-server junos:changed="changed">

            <name junos:changed="changed">10.215.194.50</name>

          <ntp junos:changed="changed">

            <server junos:changed="changed">

              <name junos:changed="changed">10.219.0.35</name>

        <interfaces junos:changed="changed" junos:mark-changed="mark-changed">

          <interface junos:changed="changed">

            <name junos:changed="changed">re0:mgmt-0</name>

            <unit junos:changed="changed">

              <name junos:changed="changed">0</name>

              <family junos:changed="changed">

                <inet junos:changed="changed">

                  <address junos:changed="changed">

                    <name junos:changed="changed">10.219.38.17/26</name>

        <routing-options junos:changed="changed" junos:mark-changed="mark-changed">

          <static junos:changed="changed">

            <route junos:changed="changed">

              <name junos:changed="changed">0.0.0.0/0</name>

              <next-hop junos:changed="changed">10.219.38.1</next-hop>

      ...

<truncated, including protocols configure>



This problem is fixed as part of PR1744917, which had been committed in 23.4R1 and onwards.

At this stage, back port the fix into earlier release is in discussion but need careful consideration to avoid causing other problem.

With the JUNOS release prior to 23.4R1, the workaround can be set the macsec pre-shared-key ckn/cak with all the 64 hexadecimal digits to get out of this issue.

Modification History

2024-05-07 : Article Created