Two CRPD instances are seeing an issue where an IBGP peering between then using loopback addresses will not come up
Addresses tied to each instance:
crpd01lo0 = 172.17.188.1610.64.94.64/31 dev ens4
crpd02lo0 = 172.17.191.1610.64.94.48/31 dev ens4
crpd01 configuration:
interfaces {
lo0 {
unit 0 {
family inet {
address 172.17.191.16/32 {
primary;
preferred;
}
protocols {
bgp {
path-selection l2vpn-use-bgp-rules;
group RR {
type internal;
multihop {
ttl 255;
local-address 172.17.191.16;
unicast;
family inet-vpn {
family inet6-vpn {
neighbor 172.17.188.16 {
tcp-mss 1400;
crpd02 configuration:
address 172.17.188.16/32 {
local-address 172.17.188.16;
neighbor 172.17.191.16 {
BGP Logs from crpd02 show a Send for the OPEN using the Loopback:
Aug 30 21:17:52.102713 BGP SEND 172.17.191.16+41171 -> 172.17.188.16+179
Aug 30 21:17:52.102717 BGP SEND message type 1 (Open) length 111
Aug 30 21:17:52.102720 BGP SEND version 4 as 11158 holdtime 90 id 172.17.191.16 parmlen 82
Aug 30 21:17:52.102723 BGP SEND MP capability AFI=1, SAFI=1
Aug 30 21:17:52.102727 BGP SEND MP capability AFI=1, SAFI=128
Aug 30 21:17:52.102730 BGP SEND MP capability AFI=25, SAFI=65
Aug 30 21:17:52.102733 BGP SEND MP capability AFI=2, SAFI=128
Aug 30 21:17:52.102736 BGP SEND MP capability AFI=1, SAFI=5
Aug 30 21:17:52.102738 BGP SEND MP capability AFI=2, SAFI=5
Aug 30 21:17:52.102741 BGP SEND MP capability AFI=1, SAFI=66
Aug 30 21:17:52.102744 BGP SEND Refresh capability, code=128
Aug 30 21:17:52.102747 BGP SEND Refresh capability, code=2
Aug 30 21:17:52.102751 BGP SEND Restart capability, code=64, time=120, flags=Notification
Aug 30 21:17:52.102754 BGP SEND 4 Byte AS-Path capability (65), as_num 11158
Aug 30 21:17:52.102757 BGP SEND Long-Lived Graceful Restart capability, code=71
Aug 30 21:17:52.102786 BGP_11158.172.17.188.16: send proc: writev 111/111 bytes, rc 1
Aug 30 21:17:52.102791 task_timer_ucreate: created timer BGP_11158.172.17.188.16_Hold flags <>
Aug 30 21:17:52.102796 task_timer_uset: timer BGP_11158.172.17.188.16_Hold <Touched> set to interval 0 offset 1:30 with jitter 0 at 21:19:22.102634
Aug 30 21:17:52.172529 task_process_events_internal: recv ready for BGP_11158.172.17.188.16
Aug 30 21:17:52.172536 bgp_recv_open: called for peer 172.17.188.16 (Internal AS 11158)
Aug 30 21:17:52.172547 bgp_process_open: peer 172.17.188.16 (Internal AS 11158): rcv bgp pkt type 0, expected OPEN.
Aug 30 21:17:52.172557 task_process_events_internal: recv ready for BGP_11158.172.17.188.16
Aug 30 21:17:52.172560 bgp_recv_open: called for peer 172.17.188.16 (Internal AS 11158)
Aug 30 21:17:52.172569
Aug 30 21:17:52.172569 BGP RECV 172.17.188.16+179 -> 172.17.191.16+41171
Aug 30 21:17:52.172573 BGP RECV message type 1 (Open) length 29
Aug 30 21:17:52.172576 BGP RECV version 4 as 11158 holdtime 90 id 172.17.188.16 parmlen 0
Aug 30 21:17:52.172582 bgp_process_4byte_aspath_cap():332 AS4-Peer 172.17.188.16 (Internal AS 11158)(RECV): 4 byte AS Cap value stored 0
Aug 30 21:17:52.172591 bgp_event: peer 172.17.188.16 (Internal AS 11158) old state OpenSent event RecvOpen new state OpenConfirm
Aug 30 21:17:52.172612 task_set_socket: task BGP_11158.172.17.188.16 socket 141
Aug 30 21:17:52.172616 task_timer_delete: BGP_11158.172.17.188.16_Hold <Touched>
Aug 30 21:17:52.172632 io-141-1651-BGP_11158.172.17.188.16: Set queue attrs qidx=0
Aug 30 21:17:52.172653 io-141-1651-BGP_11158.172.17.188.16: Success: Initialized queue 0
Aug 30 21:17:52.172656 io-141-1651-BGP_11158.172.17.188.16: Set queue attrs qidx=1
Aug 30 21:17:52.172670 io-141-1651-BGP_11158.172.17.188.16: Success: Initialized queue 1
Aug 30 21:17:52.172672 io-141-1651-BGP_11158.172.17.188.16: Set queue attrs qidx=2
Aug 30 21:17:52.172686 io-141-1651-BGP_11158.172.17.188.16: Success: Initialized queue 2
Aug 30 21:17:52.172692 io-141-1651-BGP_11158.172.17.188.16: Success: Encoded msg op=3
Aug 30 21:17:52.172706 io-141-1651-BGP_11158.172.17.188.16: Sent create request: msgid=4306
Aug 30 21:17:52.172710 io-141-1651-BGP_11158.172.17.188.16: Success: Created session
Aug 30 21:17:52.172712 bgp_thrio_peer_init: peer 172.17.188.16 (Internal AS 11158): Initialized I/O session
Aug 30 21:17:52.172715 Inserts 172.17.188.16 (Internal AS 11158), ios_id 8d00000673, into ios_id tree
Aug 30 21:17:52.172723 BGP_11158.172.17.188.16: send proc: send via threaded I/O
Aug 30 21:17:52.172725 sending 19 bytes
Aug 30 21:17:52.172727
.
But, a Ceas Notification is received shortly thereafter indicating unconfigured neighbor:
Aug 30 21:17:52.173018
Aug 30 21:17:52.173018 BGP RECV 172.17.188.16+179 -> 172.17.191.16+41171
Aug 30 21:17:52.173020 BGP RECV message type 3 (Notification) length 21
Aug 30 21:17:52.173026 BGP RECV Notification code 6 (Cease) subcode 5 (Connection Rejected)
Aug 30 21:17:52.173035 Notify received from 172.17.188.16 (Internal AS 11158), code 6, subcode 5
crpd01 report reception of the packet from 10.64.94.48 and not the peer loopback, which explains the error:
Aug 30 21:17:04 VSGSNG5-RRR-2 rpd: bgp_peer_addr_ok: Connection attempt from unconfigured neighbor: 10.64.94.48+40609
Aug 30 21:17:04 VSGSNG5-RRR-2 rpd: bgp_listen_accept:7522: NOTIFICATION sent to 10.64.94.48+40609 (proto): code 6 (Cease) subcode 5 (Connection Rejected), Reason: Connection attempt from unconfigured neighbor: 10.64.94.48+40609
Aug 30 21:17:36 VSGSNG5-RRR-2 rpd: bgp_listen_accept:7522: NOTIFICATION sent to 10.64.94.48+52635 (proto): code 6 (Cease) subcode 5 (Connection Rejected), Reason: Connection attempt from unconfigured neighbor: 10.64.94.48+52635
With the same thing happening in reverse:
Aug 30 21:17:07 VSGSNG5-RRR-2 rpd: bgp_handle_notify:5223: NOTIFICATION received from 172.17.191.16 (Internal AS 11158): code 6 (Cease) subcode 5 (Connection Rejected)
Aug 30 21:17:36 VSGSNG5-RRR-2 rpd: bgp_peer_addr_ok: Connection attempt from unconfigured neighbor: 10.64.94.48+52635
Packet captures were collected at each and they confirm the above:
o. Time Source Destination Protocol Length Info
8825 0.000663 10.64.94.48 172.17.188.16 TCP 80 43867 ? 179 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM TSval=2175172576 TSecr=0 WS=2
8826 0.000076 172.17.188.16 10.64.94.48 TCP 80 179 ? 43867 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM TSval=2600360270 TSecr=2175172576 WS=2
8832 0.055253 10.64.94.48 172.17.188.16 BGP 183 OPEN Message
8833 0.000068 172.17.188.16 10.64.94.48 TCP 72 179 ? 43867 [ACK] Seq=1 Ack=112 Win=65050 Len=0 TSval=2600360339 TSecr=2175172646
8834 0.000244 172.17.188.16 10.64.94.48 BGP 122 OPEN Message, NOTIFICATION Message
8835 0.000022 172.17.188.16 10.64.94.48 TCP 72 179 ? 43867 [RST, ACK] Seq=51 Ack=112 Win=65050 Len=0 TSval=2600360339 TSecr=2175172646
35856 0.006427 10.64.94.64 172.17.191.16 TCP 80 42113 ? 179 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM TSval=3198916250 TSecr=0 WS=2
35861 0.051302 172.17.191.16 10.64.94.64 TCP 80 179 ? 42113 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM TSval=3170295057 TSecr=3198916250 WS=2
35862 0.000379 10.64.94.64 172.17.191.16 BGP 183 OPEN Message
35863 0.069446 172.17.191.16 10.64.94.64 TCP 72 179 ? 42113 [ACK] Seq=1 Ack=112 Win=65050 Len=0 TSval=3170295127 TSecr=3198916320
35864 0.000146 172.17.191.16 10.64.94.64 BGP 122 OPEN Message, NOTIFICATION Message
35865 0.000020 172.17.191.16 10.64.94.64 TCP 72 179 ? 42113 [RST, ACK] Seq=51 Ack=112 Win=65050 Len=0 TSval=3170295127 TSecr=3198916320
45725 0.001353 10.64.94.48 172.17.188.16 TCP 80 60121 ? 179 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM TSval=2175320717 TSecr=0 WS=2
45726 0.000070 172.17.188.16 10.64.94.48 TCP 80 179 ? 60121 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM TSval=2600508411 TSecr=2175320717 WS=2
45775 0.008362 10.64.94.48 172.17.188.16 BGP 183 OPEN Message
45776 0.000062 172.17.188.16 10.64.94.48 TCP 72 179 ? 60121 [ACK] Seq=1 Ack=112 Win=65050 Len=0 TSval=2600508481 TSecr=2175320787
45777 0.000286 172.17.188.16 10.64.94.48 BGP 122 OPEN Message, NOTIFICATION Message
45778 0.000022 172.17.188.16 10.64.94.48 TCP 72 179 ? 60121 [RST, ACK] Seq=51 Ack=112 Win=65050 Len=0 TSval=2600508481 TSecr=2175320787
71222 0.043426 10.64.94.64 172.17.191.16 TCP 80 58409 ? 179 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM TSval=3199064393 TSecr=0 WS=2
71273 0.016341 172.17.191.16 10.64.94.64 TCP 80 179 ? 58409 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM TSval=3170443199 TSecr=3199064393 WS=2
71274 0.000334 10.64.94.64 172.17.191.16 BGP 183 OPEN Message
71278 0.034996 172.17.191.16 10.64.94.64 TCP 72 179 ? 58409 [ACK] Seq=1 Ack=112 Win=65050 Len=0 TSval=3170443269 TSecr=3199064463
71279 0.000185 172.17.191.16 10.64.94.64 BGP 122 OPEN Message, NOTIFICATION Message
71280 0.000025 172.17.191.16 10.64.94.64 TCP 72 179 ? 58409 [RST, ACK] Seq=51 Ack=112 Win=65050 Len=0 TSval=3170443269 TSecr=3199064463
81687 0.077656 10.64.94.48 172.17.188.16 TCP 80 46141 ? 179 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM TSval=2175468858 TSecr=0 WS=2
81688 0.000082 172.17.188.16 10.64.94.48 TCP 80 179 ? 46141 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM TSval=2600656553 TSecr=2175468858 WS=2
81733 0.028130 10.64.94.48 172.17.188.16 BGP 183 OPEN Message
81734 0.000096 172.17.188.16 10.64.94.48 TCP 72 179 ? 46141 [ACK] Seq=1 Ack=112 Win=65050 Len=0 TSval=2600656623 TSecr=2175468928
81735 0.000299 172.17.188.16 10.64.94.48 BGP 122 OPEN Message, NOTIFICATION Message
81736 0.000022 172.17.188.16 10.64.94.48 TCP 72 179 ? 46141 [RST, ACK] Seq=51 Ack=112 Win=65050 Len=0 TSval=2600656623 T
The issue seen above comes down to an overlap of the loopback addresses with the address and NAT rules installed by Docker.
The CRPD instances are running in Host mode as they are the only instances on each server:
docker run --rm --detach --name crpd01 -h crpd01 --net=host --privileged -v crpd01-config:/config -v crpd01-varlog:/var/log -it crpd:23.4R1.9
docker run --rm --detach --name crpd02 -h crpd02 --net=host --privileged -v crpd02-config:/config -v crpd02-varlog:/var/log -it crpd:23.4R1.9
In Host mode, all of the interfaces in the host OS are visible, including docker0 which is marked as DOWN as it is only used in bridge mode:
root@crpd01> show interfaces terse
Interface@link Oper State Addresses
br-58ebec70e7a3 DOWN 172.19.0.1/16
br-ee5df69f95b0 DOWN 172.18.0.1/16
docker0 DOWN 172.17.0.1/16
ens160 UP 10.85.211.75/24 fe80::250:56ff:fe9e:cfa1/64
ens192 UP 100.64.94.64/24 fe80::250:56ff:fe9e:563b/64
erspan0@NONE DOWN
gre0@NONE UNKNOWN fe80::ac12:1/64 fe80::ac11:1/64 fe80::ac13:1/64 fe80::6440:5e40/64 fe80::a55:d34b/64 fe80::7f00:1/64
gretap0@NONE DOWN
ip6tnl0@NONE UNKNOWN fe80::8c8f:15ff:fe26:d757/64
irb UNKNOWN fe80::9c36:10ff:fed4:b031/64
lo UNKNOWN 127.0.0.1/8 ::1/128
lo0.0 UNKNOWN 172.17.188.16/32 fe80::54b1:57ff:fe14:640d/64
lsi UNKNOWN fe80::dc0c:9cff:fea5:828c/64
sit0@NONE UNKNOWN ::172.18.0.1/96 ::172.17.0.1/96 ::172.19.0.1/96 ::100.64.94.64/96 ::10.85.211.75/96 ::127.0.0.1/96
tunl0@NONE UNKNOWN
But, as seen above, the 172.17.0.0/16 subnet is tied to docker0, which is the default behavior.
Additionally, Docker installs a NAT rule in the host OS iptables, as seen below:
root@ubuntu:~# iptables -t nat -L -n -v --line-numbers
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination
1 2957 177K DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
1 0 0 DOCKER all -- * * 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
1 2405 144K MASQUERADE all -- * !docker0 172.17.0.0/16 0.0.0.0/0
2 0 0 MASQUERADE all -- * !br-ee5df69f95b0 172.18.0.0/16 0.0.0.0/0
3 0 0 MASQUERADE all -- * !br-58ebec70e7a3 172.19.0.0/16 0.0.0.0/0
Chain DOCKER (2 references)
1 0 0 RETURN all -- docker0 * 0.0.0.0/0 0.0.0.0/0
2 0 0 RETURN all -- br-ee5df69f95b0 * 0.0.0.0/0 0.0.0.0/0
3 0 0 RETURN all -- br-58ebec70e7a3 * 0.0.0.0/0 0.0.0.0/0
root@ubuntu:~#
That NAT rule, which helps avoid issues with different containers overlapping with the same subnet is the reason for the address change. The lo0 sourced BGP packets match the rule and are in turn translated.
Junos running in CRPD is not aware of any of these factors, so there is no fix available there. However, there are a number of ways to get around this scenario:
1. Renumber the loopback addresses to avoid the conflict with the 172.17.0.0 subnet.
2. Remove the NAT rule in iptables with the command 'iptables -t nat -D POSTROUTING <chain num>
For example:
root@ubuntu:~# iptables -t nat -D POSTROUTING 1
1 1614 96824 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
1 0 0 MASQUERADE all -- * !br-eb43c96e8a4d 172.19.0.0/16 0.0.0.0/0
2 0 0 RETURN all -- br-eb43c96e8a4d * 0.0.0.0/0 0.0.0.0/0
The issue with this approach is that docker would reinstall the rule upon a system reboot or application restart.
3. Change the address used by Docker on docker0. There is a file that can be created or modified that Docker uses for configuration, /etc/docker/daemon.json
The following syntax added to the file will change the subnet to 192.16.1.0/24:
{
"bip": "192.168.1.1/24"
4. Disable the NAT rule installation by Docker with the following in /etc/docker/daemon.json:
"ip-masq": false
5. Disable the creation of the docker0 interface completely with the following in /etc/docker/daemon.json:
"bridge": "none"