Article Detail

Description

After the customer configured two iso addresses on lo0, it changed to the second ISO address right after upgrading MW.

Because the second ISO address is duplicated to another alive router in the network, it made the isis route/node flapping and LSP refresh.

The RR bgp sessions on this router got flapping due to ISIS route being withdrawn periodically due to sysid collision.

Symptoms

From log messages, re0 was upgraded around time:
---------------
Mar 20 01:38:48 2025 re0.test rpd[47628]: JTASK_TASK_BEGIN: Commencing routing updates version 21.2X8.1, built 2024-04-07 05:47:31 UTC by builder
---------------

From log mastership, RE0 acts as master as below:
---------------
Mar 20 02:48:33 new state = master
Mar 20 02:48:33 event = E_ORE_B, state = master, param = 0x0x6c06c08
Mar 20 02:48:33 currentAction = A_NOOP
---------------

From isis-debug in var-log, it shows up “collision” as below:
---------------
Mar 20 02:49:21.398112 ERROR: Possible sysid collision
<snipped>
Mar 20 05:29:12.293712 ERROR: Possible sysid collision
Mar 20 05:32:28.466535 ERROR: Possible sysid collision
Mar 20 05:33:51.625772 ERROR: Possible sysid collision
--------------

From messages in var-log, RR bgp flapping displays as below:

--------------
Mar 20 01:55:14 2025 re0.test rpd[47628]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer 172.16.54.133 (Internal AS 30001) changed state from Established to Idle (event JsrUnreplicate) (instance master)

Mar 20 02:02:47 2025 re0.test rpd[47628]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer 172.16.54.133 (Internal AS 30001) changed state from Idle to Established (event RecvEstab) (instance master)
--------------

by checking the route table, the RR route keeps changing all the time.

-------------

[email protected]> show route 172.16.54.133
Mar 20 06:54:29

inet.0: 496 destinations, 499 routes (494 active, 0 holddown, 2 hidden)
+ = Active Route, - = Last Active, * = Both

172.16.54.133/32 *[IS-IS/18] 00:00:07, metric 16908264
> to 76.58.25.25 via ae10.0

[email protected]> show route 172.16.54.133
Mar 20 06:21:24
inet.0: 334216 destinations, 364538 routes (334214 active, 7492 holddown, 2 hidden)
+ = Active Route, - = Last Active, * = Both

172.16.54.133/32 *[IS-IS/18] 00:00:38, metric 130100
> to 76.58.25.120 via ae26.0

-------------

Lo0 configuration as below:

------------

lo0 {
unit 0 {
family inet {
address 127.0.0.1/32;
address 10.43.180.50/32;
}
family iso {
address 49.1010.0100.4318.0050.00; <<<<< correct address.
address 49.7390.0100.4318.0052.00;<<<<<<duplicated and misconfigured ISO address.
}
family inet6 {
address 1005:e000:0:0:0:0:f:32/128;
}
}
}

------------

Solution

Solution as below:

The logical explanation for this issue is that the correct iso address should be configured before this duplicated ISO address was input, but there was no rebooting event since then, so it keeps the original ISO address with sysid for a long time.

But in the customer MW, the router was upgraded and rebooted, so the duplicated iso address was rephrased and elected as the new SYSID, then the accident happened with bgp flapping.

By the lab test, these actions (router rebooting, restarting-routing, and deactivating/activating protocols isis) could be proved to make the ISO sysid change accordingly.

LAB Test Results:

=============

broot@waio-re0> show version
Hostname: waio-re0
Model: mx960
Junos: 21.2X8.1

===========
The first step: Configure ISIS on this router with only correct ISO address.

------------------------------
labroot@waio-re0> show configuration interfaces lo0
description "Loopback Interface";
unit 0 {
family inet {
address 127.0.0.1/32;
address 10.43.180.50/32;
}
family iso {
address 49.1010.0100.4318.0050.00; <<<<<<<<< only only ISO address
}
family inet6 {
address 1005:e000:0:0:0:0:f:32/128;
}
}
------------------------------

isis {
interface et-4/3/0.0 {
ldp-synchronization;
level 2 {
te-metric 100;
metric 1000;
ipv6-unicast-metric 1000;
hello-authentication-type md5;
hello-interval 10;
hold-time 30;
}
level 1 disable;
hello-padding disable;
point-to-point;
}
interface lo0.0 {
passive;
}
level 2 wide-metrics-only;
level 1 {
disable;
wide-metrics-only;
}
topologies ipv6-unicast;
overload {
timeout 300;
advertise-high-metrics;
}
}
------------------------------
labroot@waio-re0> show isis adjacency
Interface System L State Hold (secs) SNPA
et-4/3/0.0 raven 2 Up 29
--------------

labroot@waio-re0> show isis overview
Instance: master
Router ID: 10.43.180.50
IPv6 Router ID: 1005:e000::f:32
Hostname: waio-re0
Sysid: 0100.4318.0050 <<<<< the correct sysid as design.
Areaid: 49.1010
Adjacency holddown: enabled
Maximum Areas: 3
LSP life time: 1200
Attached bit evaluation: enabled
SPF delay: 200 msec, SPF holddown: 5000 msec, SPF rapid runs: 3
Overload bit at startup is set
Overload high metrics: enabled
Allow route leaking: disabled
Allow internal prefix overloading: disabled
Allow external prefix overloading: disabled
Overload timeout: 300 sec
IPv4 is enabled, IPv6 is enabled
Traffic engineering: enabled
Traffic engineering v6: disabled
Restart: Disabled
Helper mode: Enabled
Layer2-map: Disabled
Source Packet Routing (SPRING): Disabled
Post Convergence Backup: Disabled
Level 1
Internal route preference: 15
External route preference: 160
Prefix export count: 0
Wide metrics are enabled
Level 2
Internal route preference: 18
External route preference: 165
Prefix export count: 0
Wide metrics are enabled
------------------------------

The second step: Add one more ISO address into lo0 and this iso address as production configuration.
This lab router keeps stable without any further change, so the original sysid keeps there.

--------------
labroot@waio-re0> show configuration interfaces lo0
Apr 03 14:57:43
description "Loopback Interface";
unit 0 {
family inet {
address 127.0.0.1/32;
address 10.43.180.50/32;
}
family iso {
address 49.1010.0100.4318.0050.00;
address 49.7390.0100.4318.0052.00; <<<<<<<<<<<<<< one more iso address
}
family inet6 {
address 1005:e000:0:0:0:0:f:32/128;
}
}
--------------

labroot@waio-re0> show isis overview
Apr 03 14:58:03
Instance: master
Router ID: 10.43.180.50
IPv6 Router ID: 1005:e000::f:32
Hostname: waio-re0
Sysid: 0100.4318.0050 <<<<<<< still the same as original as the first step.
<snipped>
--------------

The third step: Verify the ISO sysid changing conditions based on STEP TWO works.

A. Rebooting this router can change the sysid, which is as similar as production work.

---------------
labroot@waio-re0> request vmhost reboot routing-engine both
Reboot the vmhost ? [yes,no] (no) yes
----------------

labroot@waio-re0> show isis overview
Instance: master
Router ID: 10.43.180.50
IPv6 Router ID: 1005:e000::f:32
Hostname: waio-re0
Sysid: 0100.4318.0052 <<<<<<< changed to the NEW ISO sysid.
Areaid: 49.1010
Adjacency holddown: enabled
Maximum Areas: 3
LSP life time: 1200
Attached bit evaluation: enabled
SPF delay: 200 msec, SPF holddown: 5000 msec, SPF rapid runs: 3
Overload bit at startup is set
Overload high metrics: enabled
Allow route leaking: disabled
Allow internal prefix overloading: disabled
Allow external prefix overloading: disabled
Overload timeout: 300 sec
IPv4 is enabled, IPv6 is enabled
Traffic engineering: enabled
Traffic engineering v6: disabled
Restart: Disabled
Helper mode: Enabled
Layer2-map: Disabled
Source Packet Routing (SPRING): Disabled
Post Convergence Backup: Disabled
Level 1
Internal route preference: 15
External route preference: 160
Prefix export count: 0
Wide metrics are enabled
Level 2
Internal route preference: 18
External route preference: 165
Prefix export count: 0
Wide metrics are enabled
--------------

B. Restart-routing can also change the ISO address as below.

--------------
labroot@waio-re0> restart routing
Routing protocols process started, pid 15179

--------------

labroot@waio-re0> show isis overview
Instance: master
Router ID: 10.43.180.50
IPv6 Router ID: 1005:e000::f:32
Hostname: waio-re0
Sysid: 0100.4318.0052 <<<< changed
<snipped>
--------------

C. deactivating/activating protocol isis can change it as well.

--------------
labroot@waio-re0> show system rollback compare 1 0
[edit protocols]
! active: isis { ... }

labroot@waio-re0> show system rollback compare 2 1
[edit protocols]
! inactive: isis { ... }
--------------

labroot@waio-re0> show isis overview
Instance: master
Router ID: 10.43.180.50
IPv6 Router ID: 1005:e000::f:32
Hostname: waio-re0
Sysid: 0100.4318.0052 <<<<<<<<<<<changed
<snipped>
--------------
=============================================================

Modification History

2025-04-04 : Article Created