Description

This article explains the difference on route selection between external BGP and external BGP in confederation setup.

Symptoms

AS65000 has BGP confederation and it is logically divided into sub-AS 65001 and 65003.

R1 and R2 belongs to sub-AS 65001 and R3 belongs to sub-AS 65003.

                                              +-+-+
          +------------------------+            |
          |            R3          |            |
          |         10.0.0.3       |            | sub-AS 65003
          +------------------------+            |
          .3|                    |.3          +-+-+
            |ebgp            ebgp|
192.168.1/24|                    |192.168.2/24
            |    confederation   |
          .1|                    |.2          +-+-+
    +---------+                +---------+      |
    |    R1   |.1    ibgp    .2|    R2   |      |
    | 10.0.0.1|----------------| 10.0.0.2|      | sub-AS 65001
    +---------+  192.168.0/24  +---------+      |
                                              +-+-+

When you check route to 10.0.0.3 (R3's loopback) on R1, you will see only one route in the routing table and it has next-hop 192.168.1.3. But when you check the same on R2, it has two routes to 10.0.0.3 and selects the route via R1, instead of the route via direct link to R3. Unlike R1, R2 does not select the most effective route.
 

lab@R1# run show route 10.0.0.3    
inet.0: 14 destinations, 19 routes (14 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.0.0.3/32         *[BGP/170] 10:59:01, localpref 100
                      AS path: (65003) I, validation-state: unverified
                    > to 192.168.1.3 via ge-0/0/5.0

lab@R2# run show route 10.0.0.3
inet.0: 14 destinations, 20 routes (14 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.0.0.3/32         *[BGP/170] 10:59:08, localpref 100
                      AS path: (65003) I, validation-state: unverified
                    > to 192.168.0.1 via ge-0/0/4.0
                    [BGP/170] 10:59:08, localpref 100
                      AS path: (65003) I, validation-state: unverified
                    > to 192.168.2.3 via ge-0/0/6.0

lab@R1# run show route advertising-protocol bgp 192.168.0.2
inet.0: 14 destinations, 19 routes (14 active, 0 holddown, 0 hidden)
  Prefix      Nexthop        MED     Lclpref    AS path
* 10.0.0.1/32              Self                         100        I
* 10.0.0.3/32              Self                         100        (65003) I
* 192.168.0.0/24          Self                         100        I
* 192.168.1.0/24          Self                         100        I

lab@R2# run show route advertising-protocol bgp 192.168.0.1
inet.0: 14 destinations, 20 routes (14 active, 0 holddown, 0 hidden)
  Prefix      Nexthop        MED     Lclpref    AS path
* 10.0.0.2/32              Self                         100        I
* 192.168.0.0/24          Self                         100        I
* 192.168.2.0/24          Self                         100        I

When you setup the same topology with real external BGP, contrary to the confederation scenario, R2 selects route via direct link to R3.

lab@R1# run show route 10.0.0.3             
inet.0: 14 destinations, 20 routes (14 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.0.0.3/32         *[BGP/170] 00:00:52, localpref 100
                      AS path: 65003 I, validation-state: unverified
                    > to 192.168.1.3 via ge-0/0/5.0
                    [BGP/170] 00:00:58, localpref 100
                      AS path: 65003 I, validation-state: unverified
                    > to 192.168.0.2 via ge-0/0/4.0

lab@R2# run show route 10.0.0.3
inet.0: 14 destinations, 20 routes (14 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.0.0.3/32         *[BGP/170] 00:01:05, localpref 100
                      AS path: 65003 I, validation-state: unverified
                    > to 192.168.2.3 via ge-0/0/6.0
                    [BGP/170] 00:00:58, localpref 100
                      AS path: 65003 I, validation-state: unverified
                    > to 192.168.0.1 via ge-0/0/4.0

Solution

The confederation BGP peer is treated as an internal, even if the peer belongs to a different sub-AS and has external BGP configured.

The reason why R2 selects route via R1 to reach R3's loopback is that R2 uses the smallest router-ID as a tie breaker in a route selection.

According to the documentation on Understanding BGP Path Selection , BGP path selection is done by "Prefer the path from the peer with the lowest router ID".

Based on the description in the  aforementioned documentation, let's review route information with extensive option.

Check 'Inactive reason' in the output.

Inactive reason: Not Best in its group - Router ID

lab@R2# run show route 10.0.0.3 detail
inet.0: 14 destinations, 20 routes (14 active, 0 holddown, 0 hidden)
10.0.0.3/32 (2 entries, 1 announced)
        *BGP    Preference: 170/-101
                Next hop type: Indirect, Next hop index: 0
                Address: 0xbebd9f0
                Next-hop reference count: 8
                Source: 192.168.0.1
                Next hop type: Router, Next hop index: 576
                Next hop: 192.168.0.1 via ge-0/0/4.0, selected
                Session Id: 0x14e
                Protocol next hop: 192.168.0.1
                Indirect next hop: 0xae42f00 1048575 INH Session ID: 0x14f
                State: <Active Int Ext>
                Local AS: 65001 Peer AS: 65001
                Age: 43   Metric2: 0
                Validation State: unverified
                Task: BGP_1.192.168.0.1
                Announcement bits (3): 0-KRT 2-BGP_RT_Background 3-Resolve tree 1
                AS path: (65003) I
                Accepted
                Localpref: 100
                Router ID: 10.0.0.1
         BGP    Preference: 170/-101
                Next hop type: Indirect, Next hop index: 0
                Address: 0xbebd030
                Next-hop reference count: 4
                Source: 192.168.2.3
                Next hop type: Router, Next hop index: 577
                Next hop: 192.168.2.3 via ge-0/0/6.0, selected
                Session Id: 0x14c
                Protocol next hop: 192.168.2.3
                Indirect next hop: 0xae43200 1048574 INH Session ID: 0x14d
                State: <NotBest Int Ext>
                Inactive reason: Not Best in its group - Router ID
                Local AS: 65001 Peer AS: 65003
                Age: 1:01   Metric2: 0
                Validation State: unverified
                Task: BGP_3.192.168.2.3
                AS path: (65003) I
                Accepted
                Localpref: 100
                Router ID: 10.0.0.3

Here's route information from normal external BGP setup.

Inactive reason: Not Best in its group - Interior > Exterior > Exterior via Interior

lab@R2# run show route 10.0.0.3 detail
inet.0: 14 destinations, 20 routes (14 active, 0 holddown, 0 hidden)
10.0.0.3/32 (2 entries, 1 announced)
        *BGP    Preference: 170/-101
                Next hop type: Router, Next hop index: 586
                Address: 0xbebd750
                Next-hop reference count: 5
                Source: 192.168.2.3
                Next hop: 192.168.2.3 via ge-0/0/6.0, selected
                Session Id: 0x153
                State: <Active Ext>
                Local AS: 65001 Peer AS: 65003
                Age: 2:55
                Validation State: unverified
                Task: BGP_3.192.168.2.3
                Announcement bits (3): 0-KRT 2-BGP_RT_Background 3-Resolve tree 1
                AS path: 65003 I
                Accepted
                Localpref: 100
                Router ID: 10.0.0.3
         BGP    Preference: 170/-101
                Next hop type: Indirect, Next hop index: 0
                Address: 0xbebd5d0
                Next-hop reference count: 7
                Source: 192.168.0.1
                Next hop type: Router, Next hop index: 585
                Next hop: 192.168.0.1 via ge-0/0/4.0, selected
                Session Id: 0x150
                Protocol next hop: 192.168.0.1
                Indirect next hop: 0xae43680 1048574 INH Session ID: 0x154
                State: <NotBest Int Ext Changed>
                Inactive reason: Not Best in its group - Interior > Exterior > Exterior via Interior
                Local AS: 65001 Peer AS: 65001
                Age: 2:48   Metric2: 0
                Validation State: unverified
                Task: BGP_1.192.168.0.1
                AS path: 65003 I
                Accepted
                Localpref: 100
                Router ID: 10.0.0.1

To make R2's route selection more effective, you have two options.

[a] Changing router-ID of R3 to set the lowest.
[b] Setting local-preference (or some other BGP control measures as you like).

[a] Apply the following into R3:

lab@R3# set routing-options router-id 1.0.0.3
lab@R3# commit
lab@R1# run show route 10.0.0.3    
inet.0: 14 destinations, 21 routes (14 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.0.0.3/32         *[BGP/170] 00:00:09, localpref 100
                      AS path: (65003) I, validation-state: unverified
                    > to 192.168.1.3 via ge-0/0/5.0
                    [BGP/170] 00:00:09, localpref 100
                      AS path: (65003) I, validation-state: unverified
                    > to 192.168.0.2 via ge-0/0/4.0

lab@R2# run show route 10.0.0.3    
inet.0: 14 destinations, 21 routes (14 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.0.0.3/32         *[BGP/170] 00:00:14, localpref 100
                      AS path: (65003) I, validation-state: unverified
                    > to 192.168.2.3 via ge-0/0/6.0
                    [BGP/170] 00:00:14, localpref 100
                      AS path: (65003) I, validation-state: unverified
                    > to 192.168.0.1 via ge-0/0/4.0

lab@R2# run show route 10.0.0.3 detail
inet.0: 14 destinations, 21 routes (14 active, 0 holddown, 0 hidden)
10.0.0.3/32 (2 entries, 1 announced)
        *BGP    Preference: 170/-101
                Next hop type: Indirect, Next hop index: 0
                Address: 0xbebd150
                Next-hop reference count: 6
                Source: 192.168.2.3
                Next hop type: Router, Next hop index: 586
                Next hop: 192.168.2.3 via ge-0/0/6.0, selected
                Session Id: 0x158
                Protocol next hop: 192.168.2.3
                Indirect next hop: 0xae43500 1048575 INH Session ID: 0x159
                State: <Active Int Ext>
                Local AS: 65001 Peer AS: 65003
                Age: 45   Metric2: 0
                Validation State: unverified
                Task: BGP_3.192.168.2.3
                Announcement bits (3): 0-KRT 2-BGP_RT_Background 3-Resolve tree 1
                AS path: (65003) I
                Accepted
                Localpref: 100
                Router ID: 1.0.0.3
         BGP    Preference: 170/-101
                Next hop type: Indirect, Next hop index: 0
                Address: 0xbebd810
                Next-hop reference count: 7
                Source: 192.168.0.1
                Next hop type: Router, Next hop index: 585
                Next hop: 192.168.0.1 via ge-0/0/4.0, selected
                Session Id: 0x156
                Protocol next hop: 192.168.0.1
                Indirect next hop: 0xae42f00 1048574 INH Session ID: 0x157
                State: <NotBest Int Ext Changed>
                Inactive reason: Not Best in its group - Router ID
                Local AS: 65001 Peer AS: 65001
                Age: 45   Metric2: 0
                Validation State: unverified
                Task: BGP_1.192.168.0.1
                AS path: (65003) I
                Accepted
                Localpref: 100
                Router ID: 10.0.0.1
 

[b] Apply the following into R1 and R2:

set policy-options policy-statement lowpref term 10 from protocol bgp
set policy-options policy-statement lowpref term 10 then local-preference 100
set policy-options policy-statement lowpref term 10 then accept
set policy-options policy-statement highpref term 10 from protocol bgp
set policy-options policy-statement highpref term 10 then local-preference 200
set policy-options policy-statement highpref term 10 then accept
set protocols bgp group ibgp import lowpref
set protocols bgp group ebgp import highpref
lab@R1# run show route 10.0.0.3
inet.0: 14 destinations, 21 routes (14 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.0.0.3/32         *[BGP/170] 00:03:42, localpref 200
                      AS path: (65003) I, validation-state: unverified
                    > to 192.168.1.3 via ge-0/0/5.0
                    [BGP/170] 00:01:15, localpref 100
                      AS path: (65003) I, validation-state: unverified
                    > to 192.168.0.2 via ge-0/0/4.0

lab@R2# run show route 10.0.0.3
inet.0: 14 destinations, 21 routes (14 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.0.0.3/32         *[BGP/170] 00:03:51, localpref 200
                      AS path: (65003) I, validation-state: unverified
                    > to 192.168.2.3 via ge-0/0/6.0
                    [BGP/170] 00:01:26, localpref 100
                      AS path: (65003) I, validation-state: unverified
                    > to 192.168.0.1 via ge-0/0/4.0

lab@R2# run show route 10.0.0.3 detail
inet.0: 14 destinations, 21 routes (14 active, 0 holddown, 0 hidden)
10.0.0.3/32 (2 entries, 1 announced)
        *BGP    Preference: 170/-201
                Next hop type: Indirect, Next hop index: 0
                Address: 0xbebd870
                Next-hop reference count: 6
                Source: 192.168.2.3
                Next hop type: Router, Next hop index: 586
                Next hop: 192.168.2.3 via ge-0/0/6.0, selected
                Session Id: 0x158
                Protocol next hop: 192.168.2.3
                Indirect next hop: 0xae43500 1048575 INH Session ID: 0x15a
                State: <Active Int Ext>
                Local AS: 65001 Peer AS: 65003
                Age: 3:54   Metric2: 0
                Validation State: unverified
                Task: BGP_3.192.168.2.3
                Announcement bits (3): 0-KRT 2-BGP_RT_Background 3-Resolve tree 1
                AS path: (65003) I
                Accepted
                Localpref: 200
                Router ID: 10.0.0.3
         BGP    Preference: 170/-101
                Next hop type: Indirect, Next hop index: 0
                Address: 0xbebd810
                Next-hop reference count: 7
                Source: 192.168.0.1
                Next hop type: Router, Next hop index: 585
                Next hop: 192.168.0.1 via ge-0/0/4.0, selected
                Session Id: 0x156
                Protocol next hop: 192.168.0.1
                Indirect next hop: 0xae42f00 1048574 INH Session ID: 0x157
                State: <Int Ext Changed>
                Inactive reason: Local Preference
                Local AS: 65001 Peer AS: 65001
                Age: 1:29   Metric2: 0
                Validation State: unverified
                Task: BGP_1.192.168.0.1
                AS path: (65003) I
                Accepted
                Localpref: 100
                Router ID: 10.0.0.1

Modification History

2022-12-05: Updated non-compliant IP addresses to RFC standards.