This article describes a scenario in which a customer reported being unable to reach their hosts in the data center, resulting in a major service impact. During the troubleshooting process, routing loops were identified between Juniper core routers. The root cause was traced to incorrect BGP next-hop values being advertised by the route reflectors.
#Topology
#Traceroute reporting routing loops between core routers
[labuser@labhost ~]$ traceroute XXX.XX.5.103
traceroute to XXX.XX.5.103 (XXX.XX.5.103), 30 hops max, 60 byte packets
...
8 lag-5.Core_Router1 (XX.XX.15.18) 54.691 ms 54.681 ms 53.715 ms
9 lag-10.Core_Router2 (XX.XX.13.231) 65.536 ms lag-11.Core_Router2 (XX.XX.13.233) 58.787 ms lag-10.Core_Router2 (XX.XX.13.231) 65.496 ms
10 lag-10.Core_Router1 (XX.XX.13.230) 56.261 ms 54.878 ms lag-11.Core_Router1 (XX.XX.13.232) 57.763 ms
11 lag-11.Core_Router2 (XX.XX.13.233) 96.585 ms 91.941 ms 116.510 ms
12 lag-11.Core_Router1 (XX.XX.13.232) 73.221 ms 73.167 ms lag-10.Core_Router1 (XX.XX.13.230) 70.036 ms
13 lag-11.Core_Router2 (XX.XX.13.233) 103.627 ms 103.554 ms 103.512 ms
14 lag-10.Core_Router1 (XX.XX.13.230) 88.602 ms lag-11.Core_Router1 (XX.XX.13.232) 87.331 ms lag-10.Core_Router1 (XX.XX.13.230) 82.981 ms
15 lag-10.Core_Router2 (XX.XX.13.231) 104.955 ms 102.678 ms lag-11.Core_Router2 (XX.XX.13.233) 146.707 ms
16 lag-10.Core_Router1 (XX.XX.13.230) 104.234 ms 133.986 ms lag-11.Core_Router1 (XX.XX.13.232) 100.006 ms
17 lag-11.Core_Router2 (XX.XX.13.233) 136.375 ms 136.308 ms lag-10.Core_Router2 (XX.XX.13.231) 117.563 ms
18 lag-11.Core_Router1 (XX.XX.13.232) 120.115 ms lag-10.Core_Router1 (XX.XX.13.230) 145.506 ms 144.359 ms
#Core_Router1 pointing nexthop towards Core_Router2
labuser@Core_Router1> show route XXX.XX.5.101
Jun 10 06:16:47XXX.XX.5.101
inet.0: 1093770 destinations, 2597417 routes (1093753 active, 7 holddown, 19 hidden)
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both
XXX.XX.4.0/22 *[BGP/170] 05:29:07, MED 0, localpref 50, from XX.XX.214.31
AS path: I, validation-state: unverified
> to XX.XX.13.231 via ae10.0
to XX.XX.13.233 via ae11.0
[BGP/170] 05:29:07, MED 0, localpref 50, from XX.XX.214.32
Interface Admin Link Description
ae10 up up [TYPE=CORE][BW=2300G][GRNT=PLHD][ZLOC=Core_Router2][ZPRT=AE10][IP=XX.XX.213.0]
ae11 up up [TYPE=CORE][BW=2300G][GRNT=PLHD][ZLOC=Core_Router2][ZPRT=AE11][IP=XX.XX.213.0]
labuser@Core_Router1> show route XXX.XX.5.101 detail
Jun 10 06:20:38
inet.0: 1093755 destinations, 2597318 routes (1093740 active, 5 holddown, 19 hidden)
XXX.XX.4.0/22 (2 entries, 1 announced)
*BGP Preference: 170/-51
Next hop type: Indirect, Next hop index: 0
Address: 0x3102ff5c
Next-hop reference count: 12
Source: XX.XX.214.31
Next hop type: Router, Next hop index: 0
Next hop: XX.XX.13.231 via ae10.0 weight 0x1, selected
Session Id: 0
Next hop: XX.XX.13.233 via ae11.0 weight 0x1
Protocol next hop: XX.XX.213.240
Indirect next hop: 0x1811c908 2097512 INH Session ID: 59071
State: <Active Int Ext>
Local AS: XXXX Peer AS: XXXX
Age: 5:32:59 Metric: 0 Metric2: 66536
Validation State: unverified
ORR Generation-ID: 0
Task: BGP_20115.XX.XX.214.31
Announcement bits (4): 0-KRT 8-Resolve tree 3 9-RT 10-BGP_RT_Background
AS path: I (Originator)
Cluster list: XX.XX.214.31 XX.XX.213.3
Originator ID: XX.XX.213.240
Communities: .... .... ....
Accepted
Localpref: 50
Router ID: XX.XX.214.31
Thread: junos-main
Route Record: Recorded nexthop 0x3102ff5c
as-index: 706549 (Recorded) com-index 187 (Not-Recorded)
BGP Preference: 170/-51
labuser@Core_Router1> show route receive-protocol bgp XX.XX.214.31 XXX.XX.5.101
Jun 10 06:25:26
inet.0: 1093488 destinations, 2596993 routes (1093477 active, 1 holddown, 19 hidden)
Prefix Nexthop MED Lclpref AS path
* XXX.XX.4.0/22 XX.XX.213.240 0 50 I
labuser@Core_Router1> show route XX.XX.213.240
Jun 10 06:21:56
inet.0: 1093749 destinations, 2597437 routes (1093736 active, 3 holddown, 19 hidden)
XX.XX.213.240/32 *[IS-IS/15] 05:35:16, metric 66536
#Core_Router2 pointing nexthop towards Core_Router1
labuser@Core_Router2> show route XXX.XX.5.101
Jun 10 06:18:28
inet.0: 1093815 destinations, 5208322 routes (1093774 active, 0 holddown, 236 hidden)
XXX.XX.5.96/28 *[BGP/170] 2d 18:10:59, localpref 100, from XX.XX.213.1
AS path: XXXXX XXXXX ?, validation-state: unverified
> to XX.XX.13.230 via ae10.0
to XX.XX.13.232 via ae11.0
[BGP/170] 09:32:27, localpref 100, from XX.XX.213.3
ae10 up up [TYPE=CORE][BW=2300G][GRNT=PLHD][ZLOC=Core_Router1][ZPRT=AE10][IP=XX.XX.214.142]
ae11 up up [TYPE=CORE][BW=2300G][GRNT=PLHD][ZLOC=Core_Router1][ZPRT=AE11][IP=XX.XX.214.142]
labuser@Core_Router2> show route XXX.XX.5.101 detail | no-more
Jun 10 06:22:38
inet.0: 1093554 destinations, 5207448 routes (1093511 active, 2 holddown, 236 hidden)
XXX.XX.5.96/28 (2 entries, 1 announced)
*BGP Preference: 170/-101
Address: 0x56d2d7cc
Next-hop reference count: 441
Source: XX.XX.213.1
Next hop: XX.XX.13.230 via ae10.0, selected
Session Id: 0x0
Next hop: XX.XX.13.232 via ae11.0
Protocol next hop: XX.XX.213.160
Indirect next hop: 0x87e10fc08 2104070 INH Session ID: 0x21445d
Age: 2d 18:15:09 Metric2: 66001
Task: BGP_20115.XX.XX.213.1
Announcement bits (4): 0-KRT 6-Resolve tree 3 8-BGP_RT_Background 9-RT
AS path: XXXX XXXX ? (Originator)
Cluster list: XX.XX.213.1
Originator ID: XX.XX.213.160
Localpref: 100
Router ID: XX.XX.213.1
BGP Preference: 170/-101
labuser@Core_Router2> show route receive-protocol bgp XX.XX.213.1 XXX.XX.5.101
Jun 10 06:26:15
inet.0: 1093719 destinations, 5208071 routes (1093675 active, 3 holddown, 236 hidden)
* XXX.XX.5.96/28 XX.XX.213.160 100 XXXXX XXXXX ?
labuser@Core_Router2> show route XX.XX.213.160
Jun 10 06:22:56
inet.0: 1093554 destinations, 5207460 routes (1093512 active, 1 holddown, 236 hidden)
XX.XX.213.160/32 *[IS-IS/15] 2d 08:54:50, metric 66001
Both core routers received the same subnet from their respective regional route reflectors (RRs). However, Core_Router1 received a less specific route (/22), while Core_Router2 received a more specific route (/28). As a result, Core_Router1's protocol next-hop pointed to Core_Router2, and Core_Router2's protocol next-hop pointed back to Core_Router1, creating a routing loop.
The root cause was traced to an incorrect configuration recently applied on the route reflector, which is a non-Juniper device in the production environment. The issue was resolved after correcting the configuration on the affected route reflector.