Description

Contrail supports the BGP-as-a-service (BaaS) feature. The Juniper manual, BGP as a Service shows how to configure BaaS from the control node perspective.

KB33304 - Configuring BGP as a Service [juniper.net] provides an example of how to configure BGPaaS using vSRX as VNF along with some basic verification. In a production environment, the feature may be deployed with other features to provide high availability and scalablity:

  • BGP GR (graceful restart)
  • AAP (Allowed Address Pair)
  • VRRP (Virtual Router Redundent Protocol)

This article explains the AAP use case. The focus of this article is the configuration and verification of BGPaaS. More details about the AAP feature is beyond the scope of this KB.

Solution

With only 1 BGP session configured on 1 VNF, the BGP session is susceptible to any temporary network/VNF outages. Once the only BGP session gets flapped, any network entities relying on this session will go through route churns and will experience corresponding service impacts. In production, a best practice to bring more robustness to the BGPaaS feature is to use AAP (Allowed Address Pair). For example, 20 VMIs used by 20 VNFs can share a same "AAP IP", one or more BGP session (s) can be established from the same AAP IP by different VNFs. The number of BGP sessions needed depends on the desire of the network design, but at least two is needed to avoid single point of failure situations in case one session is down.

Example with the following configuration parameters:

  • Configure 1 VN: pings-net
  • Configure 1 AAP IP on the VN: 4.4.4.100
  • Configure 2 VMIs under the AAP IP each VMI is attached to 1 vSRX VM
  • Each vSRX VM is configured a lo0 interface with the AAP IP.
  • Each vSRX VM is configured 1 BGP session from the AAP IP towards the .1
  • Vrouter GW IP.

Setup diagram

        contrail controllers               computes             VNF
     ...............................    .................   ..............
     .   +-------+                 .    .  +-------+    .   .  +-------+ .
     .   |       |       XMPP      .    .  |       |.1  .   .  |       | .4.4.4.100
     .   |cont101+-------------------------+bcomp79+----.BGP.--+ vsrx1 +-.---
     .   |       |       BGP       .    .  |       | AAP.   .  |       | .
     .   +-------+                 . ___.__+-------+    .   .  +-------+ .
     .                          __./   .               .   .            .
     .                 +-------+/XMPP   .               .   .            .
     .                 |       |   .    .               .   .            .
     .                 |cont103|   .    .               .   .            .
     .                 |       |   .    .               .   .            .
     .                 +-------+_ XMPP  .               .   .            .
     .                           \_._   .               .   .            .
     .    +-------+            BGP . \__.__+-------+    .   .  +-------+ .
     .    |       |                .    .  |       |.1  .   .  |       | .4.4.4.100
     .    |cont102+------------------------+bcomp80+----.BGP.--+ vsrx2 +-.----
     .    |       |      XMPP      .    .  |       | AAP.   .  |       | .
     .    +-------+                .    .  +-------+    .   .  +-------+ .
     .                             .    .               .   .            .
     ...............................    .................   ..............

Contrail BGPaaS with AAP configuration:



VNF(VSRX) configuration. The 2 vSRX are with the same BGP configurations:

    protocols {
        bgp {
            group baas {
                local-address 4.4.4.100;
                family inet {
                    unicast;
                }
                family inet6 {
                    unicast;
                }
                neighbor 4.4.4.1 {
                    peer-as 60100;
                }
            }
        }
    }

AAP configuration. To enable AAP on one port(VMI):




Following the same steps to enable AAP on any ports. In this example, we enabled AAP on 1 port on each compute.

Verification method:

BGPaaS verification can be done via multiple methods:
  • Contrail GUI: control node
  • Contrail introspect: control node, compute node
  • Contrail flow table: compute node
  • Unix CLI: control node unix CLI "netstat"
  • VNF: in this case vSRX Junos CLI "show bgp summary"

The following Introspect can be used to monitor and verify BGP/XMPP status:
  • control node: http://127.0.0.1:8083/Snh_BgpNeighborReq?
  • compute node: http://127.0.0.1:8085/Snh_AgentXmppConnectionStatusReq
  • compute node: http://127.0.0.1:8085/Snh_BgpAsAServiceSandeshReq
  • control node: http://127.0.0.1:8085/Snh_SandeshTraceRequest?x=ControllerRxRouteXmppMessage1
  • control node: http://127.0.0.1:8085/Snh_SandeshTraceRequest?x=ControllerRxRouteXmppMessage2

Contrail GUI

Verify the BGP connection status on control node GUI:

cont101:


cont103:


cont102


Compute node: xmpp session

Compute node selects one control node from one of its XMPP sessions as the BGP neighbor. Therefore, before BGP session can be established, we need to verify XMPP sessions and make sure the status are up. XMPP session list can be displayed with the following introspect:

    curl http://127.0.0.1:8085/Snh_AgentXmppConnectionStatusReq
    <AgentXmppConnectionStatus type="sandesh">
      <peer type="list" identifier="1">
        <list type="struct" size="2">
        ......
       
     
      <more type="bool" identifier="0"> false
   

After converting XML format to text table, the following information is seen:

bcomp79:

    +----------------+-------------+-------------------------------------+---------------------+----------------+------------+-----------------------------+
    | controller_ip  | state       | peer_name                           | peer_address        | cfg_controller | flap_count | flap_time                   |
    +----------------+-------------+-------------------------------------+---------------------+----------------+------------+-----------------------------+
    | 172.18.101.103 | Established | [email protected] | 172.18.101.103:5269 | No             | 0          | 1974-Jun-07 16:17:45.985120 |
    | 172.18.101.101 | Established | [email protected] | 172.18.101.101:5269 | Yes            | 0          | n/a                         |
    +----------------+-------------+-------------------------------------+---------------------+----------------+------------+-----------------------------+

bcomp80:

    +----------------+-------------+-------------------------------------+---------------------+----------------+------------+-----------+
    | controller_ip  | state       | peer_name                           | peer_address        | cfg_controller | flap_count | flap_time |
    +----------------+-------------+-------------------------------------+---------------------+----------------+------------+-----------+
    | 172.18.101.101 | Established | [email protected] | 172.18.101.101:5269 | Yes            | 0          | n/a       |
    | 172.18.101.102 | Established | [email protected] | 172.18.101.102:5269 | No             | 0          | n/a       |
    +----------------+-------------+-------------------------------------+---------------------+----------------+------------+-----------+

From the above capture, both XMPP sessions on two compute nodes are up. Later,we'll see that bcomp79 and bcomp80 select cont103 and cont101 as BGP neighbors respectively.

Compute node: tcp source port

The source tcp port after NAT operation can be verified from vrouter introspect:

    curl http://127.0.0.1:8085/Snh_BgpAsAServiceSandeshReq
    <BgpAsAServiceSandeshResp type="sandesh">
      <bgp_as_a_service_list type="list" identifier="1">
        <list type="struct" size="1">
          <BgpAsAServiceSandeshList>
            <vm_bgp_peer_ip type="string" identifier="1"> 4.4.4.100
            <vm_nat_source_port type="i32" identifier="2"> 50003
            <vmi_uuid type="string" identifier="3" link="ItfReq"> 76437055-f294-4640-ba8c-d9744c12d649
            <is_shared type="bool" identifier="4"> false
         
       
     
      <more type="bool" identifier="0"> false
   

Collect the introspect info on both compute nodes and put them into a table:

    +-------+----------------+--------------------+--------------------------------------+-----------+
    |compute| vm_bgp_peer_ip | vm_nat_source_port | vmi_uuid                             | is_shared |
    +-------+----------------+--------------------+--------------------------------------+-----------+
    |bcomp79| 4.4.4.100      | 50003              | 76437055-f294-4640-ba8c-d9744c12d649 | false     |
    +-------+----------------+--------------------+--------------------------------------+-----------+
    |bcomp80| 4.4.4.100      | 50002              | d173cfb2-5752-4a99-bf8e-861e3c98a3d1 | false     |
    +-------+----------------+--------------------+--------------------------------------+-----------+

So the session from VNF on bcomp79 will end up with a tcp session without source port as 50003 toward contrail controller; VNF on bcomp80 will use 50002. Same can be verified by checking the flow table in the two compute nodes.

Compute node: TCP flow table

Compute node will select one contrail controller and forward the session after performing NAT operation. The compute node does not terminate the session or act as a session endpoint. Therefore, from compute node there is no direct way to verify the BGP tcp session directly. However, the flow table is maintained in vrouter and that can be used to check the BGP status indirectly, which is demonstrated in KB33304 - Configuring BGP as a Service [juniper.net]

bcomp79:

    root@bcomp79:~# flow --match proto tcp
    Flow table(size 80609280, entries 629760)

    Entries: Created 9326 Added 9326 Deleted 18504 Changed 18504 Processed 9326 Used Overflow entries 0
    (Created Flows/CPU: 1166 746 565 652 675 601 661 580 533 550 522 530 96 47 101 70 60 53 78 80 83 51 54 64 4 59 57 19 37 18 16 4 7 1 18 395 15 11 7 0 3 0 2 1 2 3 0 29)(oflows 0)

    Action:F=Forward, D=Drop N=NAT(S=SNAT, D=DNAT, Ps=SPAT, Pd=DPAT, L=Link Local Port)
     Other:K(nh)=Key_Nexthop, S(nh)=RPF_Nexthop
     Flags:E=Evicted, Ec=Evict Candidate, N=New Flow, M=Modified Dm=Delete Marked
    TCP(r=reverse):S=SYN, F=FIN, R=RST, C=HalfClose, E=Established, D=Dead

    Listing flows matching (Protocol TCP)

        Index                Source:Port/Destination:Port                      Proto(V)
    -----------------------------------------------------------------------------------
       406996<=>522980       4.4.4.100:53488                                     6 (6->0)
                             4.4.4.1:179
    (Gen: 1, K(nh):88, Action:N(SPsD), Flags:, TCP:SSrEEr, QOS:-1, S(nh):0,
     Stats:16347/1242816,  SPort 53775, TTL 255, Sinfo 18.0.0.0)

       522980<=>406996       172.18.101.103:179                                  6 (0->6)
                             172.18.79.79:50003
    (Gen: 1, K(nh):5, Action:N(SDPd), Flags:, TCP:SSrEEr, E:1, QOS:-1, S(nh):0,
     Stats:16369/999622,  SPort 54373, TTL 2, Sinfo 0.0.0.0)

bcomp80:

    root@bcomp80:~# flow --match proto tcp
    Flow table(size 80609280, entries 629760)

    Entries: Created 1894 Added 1894 Deleted 3632 Changed 3640 Processed 1894 Used Overflow entries 0
    (Created Flows/CPU: 0 0 0 0 0 0 0 0 0 0 148 246 214 183 293 291 266 253)(oflows 0)

    Action:F=Forward, D=Drop N=NAT(S=SNAT, D=DNAT, Ps=SPAT, Pd=DPAT, L=Link Local Port)
     Other:K(nh)=Key_Nexthop, S(nh)=RPF_Nexthop
     Flags:E=Evicted, Ec=Evict Candidate, N=New Flow, M=Modified Dm=Delete Marked
    TCP(r=reverse):S=SYN, F=FIN, R=RST, C=HalfClose, E=Established, D=Dead

    Listing flows matching (Protocol TCP)

        Index                Source:Port/Destination:Port                      Proto(V)
    -----------------------------------------------------------------------------------
       105680<=>373352       4.4.4.100:56411                                     6 (1->0)
                             4.4.4.1:179
    (Gen: 2, K(nh):45, Action:N(SPsD), Flags:, TCP:SSrEEr, QOS:-1, S(nh):0,
     Stats:16316/1044594,  SPort 59260, TTL 255, Sinfo 32.0.0.0)

       373352<=>105680       172.18.101.101:179                                  6 (0->1)
                             172.18.102.80:50002
    (Gen: 5, K(nh):5, Action:N(SDPd), Flags:, TCP:SSrEEr, E:0, QOS:-1, S(nh):0,
     Stats:16346/853793,  SPort 62386, TTL 2, Sinfo 0.0.0.0)

Control node: BGP session and netstat

The BGP session endpoints will be control node and VNF. To check the BGP status on control node, use the following control node introspect. Unix "netstat" command can also be used to check the socket connectivity status.

    curl http://127.0.0.1:8083/Snh_BgpNeighborReq?search_string=4.4.4
    <BgpNeighborListResp type="sandesh">
      <neighbors type="list" identifier="1">
      ......
     
      <more type="bool" identifier="0"> false
   

NOTE: In the capture below, we have converted the information carried in the Introspect XML output into a text format table for better viewing effect.

cont101:

    root@cont101:~# netstat -lnap | grep :179
    tcp  0  0 0.0.0.0:179            0.0.0.0:*               LISTEN      4837/contrail-contr
    tcp  0  0 172.18.101.101:38600   192.168.0.250:179       ESTABLISHED 4837/contrail-contr
    tcp  0  0 172.18.101.101:55575   192.168.0.204:179       ESTABLISHED 4837/contrail-contr
    tcp  0  0 172.18.101.101:179     172.18.101.102:36711    ESTABLISHED 4837/contrail-contr
    tcp  0  0 172.18.101.101:179     172.18.102.80:50002     ESTABLISHED 4837/contrail-contr #<--bcomp79
    tcp  0  0 172.18.101.101:179     172.18.101.103:38647    ESTABLISHED 4837/contrail-contr

+-------------------------------------+--------------+----------+----------+-----------+-------+-------------+------------+-------------+
| peer                                | peer_address | peer_asn | encoding | peer_type | state | send_state  | flap_count | flap_time   |
+-------------------------------------+-------------+----------+-----+-----------+-------------+----------------+---------+------------------------+
|76437055-f294-4640-ba8c-d9744c12d649 | 4.4.4.100   | 60101    | BGP | external  | Active      | not advertising | 0 | n/a                         |
|d173cfb2-5752-4a99-bf8e-861e3c98a3d1 | 4.4.4.100   | 60101    | BGP | external  | Established | in sync         | 2 | 2018-Dec-08 03:33:59.521988 |
  --------------------------------------+--------------+--------+----+-----------+-------------+-----------------+---+-----------------------------+

cont103:

    root@cont103:~# netstat -lnap | grep :179
    tcp  0  0 0.0.0.0:179             0.0.0.0:*              LISTEN      19695/contrail-cont
    tcp  0  0 172.18.101.103:45725    172.18.101.102:179     ESTABLISHED 19695/contrail-cont
    tcp  0  0 172.18.101.103:179      172.18.79.79:50003     ESTABLISHED 19695/contrail-cont #<--bcomp79
    tcp  0  0 172.18.101.103:38647    172.18.101.101:179     ESTABLISHED 19695/contrail-cont
    tcp  0  0 172.18.101.103:42664    192.168.0.204:179      ESTABLISHED 19695/contrail-cont
    tcp  0  0 172.18.101.103:35039    192.168.0.250:179      ESTABLISHED 19695/contrail-cont

    +--------------------------------------+--------------+----------+----------+-----------+-------------+-----------------+------------+-----------+
    | peer                                 | peer_address | peer_asn | encoding | peer_type | state       | send_state      | flap_count | flap_time |
    +--------------------------------------+--------------+----------+----------+-----------+-------------+-----------------+------------+-----------+
    | 76437055-f294-4640-ba8c-d9744c12d649 | 4.4.4.100    | 60101    | BGP      | external  | Established | in sync         | 0          | n/a       |
    | d173cfb2-5752-4a99-bf8e-861e3c98a3d1 | 4.4.4.100    | 60101    | BGP      | external  | Active      | not advertising | 0          | n/a       |
    +--------------------------------------+--------------+----------+----------+-----------+-------------+-----------------+------------+-----------+

cont102:

    root@cont102:~# netstat -lnap | grep :179
    tcp        0      0 0.0.0.0:179             0.0.0.0:*               LISTEN      27676/contrail-cont
    tcp        0      0 172.18.101.102:36711    172.18.101.101:179      ESTABLISHED 27676/contrail-cont
    tcp        0      0 172.18.101.102:59266    192.168.0.250:179       ESTABLISHED 27676/contrail-cont
    tcp        0      0 172.18.101.102:179      172.18.101.103:45725    ESTABLISHED 27676/contrail-cont
    tcp        0      0 172.18.101.102:52798    192.168.0.204:179       ESTABLISHED 27676/contrail-cont

+--------------------------------------+--------------+----------+----------+-----------+--------+-----------------+------------+---------------------+
| peer                                 | peer_address | peer_asn | encoding | peer_type | state  | send_state      | flap_count          | flap_time  |
+--------------------------------------+--------------+----------+----------+-----------+--------+-----------------+----+-----------------------------+
| 76437055-f294-4640-ba8c-d9744c12d649 | 4.4.4.100    | 60101    | BGP      | external  | Active | not advertising | 1  | 2018-Dec-08 03:27:49.293429 |
| d173cfb2-5752-4a99-bf8e-861e3c98a3d1 | 4.4.4.100    | 60101    | BGP      | external  | Active | not advertising | 0  | n/a                         |
+--------------------------------------+--------------+----------+----------+-----------+--------+-------------+---+----+-----------------------------+

From the capture above, the following facts can be observed:
  • 2 vSRX BGP sessions arrive at the 2 controllers, cont101 and cont103, respectively. 
  • The other controller, cont102 is not selected by any compute nodes and so it does not hold any BGP sessions.
  • On all control nodes, the port/VMI UUID is displayed to indicate from which VNF the other end of the session comes from.

Ideally, under this design any one of the below conditions won't tear down all BGP sessions:
  • One controller gets restarted
  • One compute node gets restarted
  • One VNF gets restarted

Therefore, the BGPaaS feature becomes more robust.