Description

This article provides a workaround for not being able to establish eBGP peering over a logical tunnel interface (lt-0/0/0) in flow mode on SRX platforms.

Symptoms

BGP peering over lt-0/0/0 is not coming up. The logical interfaces belongs to two routing instances (virtual routers). 

[edit]
root@jtac# show interfaces 
lt-0/0/0 {
    unit 1 {
        encapsulation ethernet;
        peer-unit 2;
        family inet {
            address 10.20.30.1/30;
        }
    }
    unit 2 {
        encapsulation ethernet;
        peer-unit 1;
        family inet {
            address 10.20.30.2/30;
        }
    }
}
ge-0/0/1 {
    unit 0 {
        family inet {
            address 192.168.1.1/24;
        }
    }
}
ge-0/0/2 {                              
    unit 0 {
        family inet {
            address 192.168.2.1/24;
        }
    }
}

[edit]
root@jtac# show routing-instances 
R1 {
    instance-type virtual-router;
    interface lt-0/0/0.1;
    interface ge-0/0/1.0;
    routing-options {
        autonomous-system 100;
    }
    protocols {
        bgp {
            group mygroup {
                neighbor 10.20.30.2 {
                    peer-as 200;
                }
            }
        }
    }
}
R2 {
    instance-type virtual-router;
    interface lt-0/0/0.2;
    interface ge-0/0/2.0;
    routing-options {
        autonomous-system 200;
    }                                   
    protocols {
        bgp {
            traceoptions {
                file R2;
                flag all;
            }
            group mygroup {
                neighbor 10.20.30.1 {
                    peer-as 100;
                }
            }
        }
    }
}

[edit]
root@jtac# run show bgp neighbor instance R1   
Peer: 10.20.30.2+179 AS 200    Local: 10.20.30.1 AS 100  
  Type: External    State: Connect        Flags: <>
  Last State: Active        Last Event: ConnectRetry
  Last Error: Cease
  Options: <Preference PeerAS Refresh>
  Holdtime: 90 Preference: 170
  Number of flaps: 1
  Last flap event: Stop
  Error: 'Cease' Sent: 1 Recv: 0


BGP State toggles between Connect and Active. It will not reach an Established State.

Security Flow Traceoptions reports:  packet dropped: for self but not interested 
Jun 17 09:39:20 09:39:19.1388266:CID-0:RT:<10.20.30.1/53542->10.20.30.2/179;6> :
Jun 17 09:39:20 09:39:19.1388277:CID-0:RT:packet [64] ipid = 53177, @44e77b92
Jun 17 09:39:20 09:39:19.1388277:CID-0:RT:---- flow_process_pkt: (thd 10): flow_ctxt type 0, common flag 0x0, mbuf 0x44e79480, 
rtbl_idx = 0
Jun 17 09:39:20 09:39:19.1388331:CID-0:RT: flow process pak fast ifl 75 in_ifp lt-0/0/0.2
Jun 17 09:39:20 09:39:19.1388331:CID-0:RT:  lt-0/0/0.2:10.20.30.1/53542->10.20.30.2/179, tcp, flag 2 syn
Jun 17 09:39:20 09:39:19.1388362:CID-0:RT: find flow: table 0x5cec9ba8, hash 7310(0xffff), sa 10.20.30.1, da 10.20.30.2, sp 53542, 
dp 179, proto 6, tok 520
Jun 17 09:39:20 09:39:19.1388377:CID-0:RT:  no session found, start first path. in_tunnel - 0, from_cp_flag - 0
Jun 17 09:39:20 09:39:19.1388411:CID-0:RT:self ip check: ip=0a141e02, laddr=0a141e02
Jun 17 09:39:20 09:39:19.1388429:CID-0:RT:check self-traffic on lt-0/0/0.2, in_tunnel 0x0
Jun 17 09:39:20 09:39:19.1388429:CID-0:RT:retcode: 0xa04
Jun 17 09:39:20 09:39:19.1388429:CID-0:RT:pak_for_self : proto 6, dst port 179, action 0x4
Jun 17 09:39:20 09:39:19.1388429:CID-0:RT:  flow_first_create_session
Jun 17 09:39:20 09:39:19.1388429:CID-0:RT:(flow_first_create_session) usp_tagged set session as mng session
Jun 17 09:39:20 09:39:19.1388429:CID-0:RT:  flow_first_in_dst_nat: in <lt-0/0/0.2>, out <N/A> dst_adr 10.20.30.2, sp 53542, dp 179
Jun 17 09:39:20 09:39:19.1388429:CID-0:RT:  chose interface lt-0/0/0.2 as incoming nat if.
Jun 17 09:39:20 09:39:19.1388429:CID-0:RT:flow_first_rule_dst_xlate: DST no-xlate: 0.0.0.0(0) to 10.20.30.2(179)
Jun 17 09:39:20 09:39:19.1388429:CID-0:RT:flow_first_routing: call flow_route_lookup(): src_ip 10.20.30.1, x_dst_ip 10.20.30.2, 
in ifp lt-0/0/0.2, out ifp N/A sp 53542, dp 179, ip_proto 6, tos c0
Jun 17 09:39:20 09:39:19.1388429:CID-0:RT:Doing DESTINATION addr route-lookup
Jun 17 09:39:20 09:39:19.1388429:CID-0:RT:  routed (x_dst_ip 10.20.30.2) from Z2 (lt-0/0/0.2 in 0) to .local..4, Next-hop: 10.20.30.2
Jun 17 09:39:20 09:39:19.1388429:CID-0:RT:  policy search from zone Z2-> zone junos-self (0x0,0xd12600b3,0xb3)
Jun 17 09:39:20 09:39:19.1388429:CID-0:RT:  app 0, timeout 1800s, curr ageout 20s
Jun 17 09:39:20 09:39:19.1388429:CID-0:RT:flow_first_src_xlate: 10.20.30.1/53542 -> 10.20.30.2/179 | 10.20.30.2/179 -> 
0.0.0.0/53542: nat_src_xlated: False, nat_src_xlate_failed: False
Jun 17 09:39:20 09:39:19.1388429:CID-0:RT:flow_first_src_xlate: src nat 0.0.0.0(53542) to 10.20.30.2(179) returns status: 0, 
rule/pool id: 0/0, pst_nat: False.
Jun 17 09:39:20 09:39:19.1388429:CID-0:RT:  dip id = 0/0, 10.20.30.1/53542->10.20.30.1/53542
Jun 17 09:39:20 09:39:19.1388429:CID-0:RT:  choose interface .local..4 as outgoing phy if
Jun 17 09:39:20 09:39:19.1388429:CID-0:RT:is_loop_pak: No loop: ifp doesnt match .local..4 vs looked-up: lt-0/0/0.2, addr: 
10.20.30.2, rtt_idx: 4, addr_type:0x3
Jun 17 09:39:20 09:39:19.1388744:CID-0:RT:jsf sess interest check. regd plugins 10
Jun 17 09:39:20 09:39:19.1388744:CID-0:RT: Allocating plugin info block for 12 plugin(s) from OL
Jun 17 09:39:20 09:39:19.1388744:CID-0:RT:-jsf int check: plugin id  1, svc_req 0x0. rc 4
Jun 17 09:39:20 09:39:19.1388744:CID-0:RT:-jsf int check: plugin id  2, svc_req 0x2. rc 4                                      
Jun 17 09:39:20 09:39:19.1388744:CID-0:RT:-jsf int check: plugin id  3, svc_req 0x0. rc 4
Jun 17 09:39:20 09:39:19.1388744:CID-0:RT:-jsf int check: plugin id  5, svc_req 0x0. rc 4
Jun 17 09:39:20 09:39:19.1388744:CID-0:RT:-jsf int check: plugin id  6, svc_req 0x0. rc 4
Jun 17 09:39:20 09:39:19.1388744:CID-0:RT:-jsf int check: plugin id  7, svc_req 0x0. rc 4
Jun 17 09:39:20 09:39:19.1388744:CID-0:RT:-jsf int check: plugin id  9, svc_req 0x0. rc 4
Jun 17 09:39:20 09:39:19.1388744:CID-0:RT:-jsf int check: plugin id 10, svc_req 0x0. rc 2
Jun 17 09:39:20 09:39:19.1388744:CID-0:RT: No JSF plugins enabled for session
Jun 17 09:39:20 09:39:19.1388744:CID-0:RT: Releasing plugin info block for 12 plugin(s) to OL                                   
Jun 17 09:39:20 09:39:19.1388744:CID-0:RT:flow_first_service_lookup(): natp(0x5e8d8d68): app_id, 0(0).
Jun 17 09:39:20 09:39:19.1388744:CID-0:RT:  service lookup identified service 0.
Jun 17 09:39:20 09:39:19.1388744:CID-0:RT:  flow_first_final_check: in <lt-0/0/0.2>, out <.local..4>
Jun 17 09:39:20 09:39:19.1388744:CID-0:RT:  existing vector list 2-503399e8.
Jun 17 09:39:20 09:39:19.1388744:CID-0:RT:  Session (id:2556) created for first pak 2
Jun 17 09:39:20 09:39:19.1388744:CID-0:RT:  flow_first_install_session======> 0x5e8d8d68
Jun 17 09:39:20 09:39:19.1388744:CID-0:RT: nsp 0x5e8d8d68, nsp2 0x5e8d8dcc
Jun 17 09:39:20 09:39:19.1388744:CID-0:RT:  make_nsp_ready_no_resolve()
Jun 17 09:39:20 09:39:19.1388744:CID-0:RT:  route lookup: dest-ip 10.20.30.1 orig ifp lt-0/0/0.2 output_ifp 
lt-0/0/0.2 orig-zone 8 out-zone 8 vsd 0                                       
Jun 17 09:39:20 09:39:19.1388744:CID-0:RT:  route to 10.20.30.1
Jun 17 09:39:20 09:39:19.1388744:CID-0:RT:Doing jsf sess create notify
Jun 17 09:39:20 09:39:19.1388744:CID-0:RT:Installing c2s NP session wing
Jun 17 09:39:20 09:39:19.1388744:CID-0:RT:Installing s2c NP session wing
Jun 17 09:39:20 09:39:19.1388744:CID-0:RT:  flow got session.
Jun 17 09:39:20 09:39:19.1388744:CID-0:RT:  flow session id 2556
Jun 17 09:39:20 09:39:19.1388744:CID-0:RT:mbuf 0x44e79480, exit nh 0xfffb0006
Jun 17 09:39:20 09:39:19.1388744:CID-0:RT: ----- flow_process_pkt rc 0x0 (fp rc 0)
Jun 17 09:39:23 09:39:23.426825:CID-0:RT:<10.20.30.2/179->10.20.30.1/53542;6> :
Jun 17 09:39:23 09:39:23.426859:CID-0:RT:packet [64] ipid = 53180, @44e77b92
Jun 17 09:39:23 09:39:23.426867:CID-0:RT:---- flow_process_pkt: (thd 7): flow_ctxt type 0, common flag 0x0, 
mbuf 0x44e79480, rtbl_idx = 0
Jun 17 09:39:23 09:39:23.426886:CID-0:RT: flow process pak fast ifl 74 in_ifp lt-0/0/0.1
Jun 17 09:39:23 09:39:23.426886:CID-0:RT:  lt-0/0/0.1:10.20.30.2/179->10.20.30.1/53542, tcp, flag 12 syn ack
Jun 17 09:39:23 09:39:23.426924:CID-0:RT: find flow: table 0x5cec9ba8, hash 31919(0xffff), sa 10.20.30.2, da 10.20.30.1, 
sp 179, dp 53542, proto 6, tok 394
Jun 17 09:39:23 09:39:23.426961:CID-0:RT:  no session found, start first path. in_tunnel - 0, from_cp_flag - 0
Jun 17 09:39:23 09:39:23.426968:CID-0:RT:self ip check: ip=0a141e01, laddr=0a141e01
Jun 17 09:39:23 09:39:23.426968:CID-0:RT:check self-traffic on lt-0/0/0.1, in_tunnel 0x0
Jun 17 09:39:23 09:39:23.426968:CID-0:RT:retcode: 0x1
Jun 17 09:39:23 09:39:23.426968:CID-0:RT:pak_for_self : proto 6, dst port 53542, action 0x0
Jun 17 09:39:23 09:39:23.426968:CID-0:RT:  flow_first_create_session
Jun 17 09:39:23 09:39:23.426968:CID-0:RT:(flow_first_create_session) usp_tagged set session as mng session
Jun 17 09:39:23 09:39:23.426968:CID-0:RT:  flow_first_in_dst_nat: in <lt-0/0/0.1>, out <N/A> dst_adr 10.20.30.1, sp 179, dp 53542
Jun 17 09:39:23 09:39:23.426968:CID-0:RT:  chose interface lt-0/0/0.1 as incoming nat if.
Jun 17 09:39:23 09:39:23.426968:CID-0:RT:  packet dropped: for self but not interested
Jun 17 09:39:23 09:39:23.426968:CID-0:RT:  packet dropped, packet dropped: for self but not interested.
Jun 17 09:39:23 09:39:23.426968:CID-0:RT:  flow find session returns error.
Jun 17 09:39:23 09:39:23.426968:CID-0:RT: ----- flow_process_pkt rc 0x7 (fp rc -1)
This is because of flow bypassing (per current design) if the destination interface is lt-0/0/0.x

 
Below is the sequence of events:
  1. RE initiates with the following TCP SYN packet:
    <10.20.30.1/53542->10.20.30.2/179;6>
  2. Flow is bypassed and packet is sent through lt-0/0/0.1 to lt-0/0/0.2, so there is no flow trace.
  3. Flow receives packet on interface lt-0/0/0.2 and creates session to .local..4
    lt-0/0/0.2:10.20.30.1/53542->10.20.30.2/179, tcp, flag 2 syn
  4. Packet is forwarded to RE.
  5. RE responds with the following TCP SYN-ACK packet:
    <10.20.30.2/179->10.20.30.1/53542;6>
  6. Flow is bypassed and packet is sent through lt-0/0/0.2 to lt-0/0/0.1, so there is no flow trace.
    lt-0/0/0.1:10.20.30.2/179->10.20.30.1/53542, tcp, flag 12 syn ack
  7. Flow receives packet on interface lt-0/0/0.1 and tries to find a matching session.
    But it cannot match the reverse wing of the session created in step #3, because the incoming IFP does not match (.local..4 in the session, but flow has lt-0/0/0.1).
    So flow tries to create a new session. 10.20.30.1 is identified as self IP, but no application is interested in port 53542, so flow drops the packet.
    packet dropped: for self but not interested

Solution

If the flow bypassing did not happen when the destination interface is lt-0/0/0.x, this problem would not have been there.

As a workaround, create a firewall filter to bypass flow mode and apply it on lt-0/0/0.x interface. 

[edit]
root@jtac# show firewall 
filter lt-flowbypass {
    term 1 {
        then {
            packet-mode;
            accept;
        }
    }
}

[edit]
root@jtac# show interfaces lt-0/0/0    
unit 1 {
    encapsulation ethernet;
    peer-unit 2;
    family inet {
        filter {
            input lt-flowbypass;
        }
        address 10.20.30.1/30;
    }
}
unit 2 {
    encapsulation ethernet;
    peer-unit 1;
    family inet {
        filter {
            input lt-flowbypass;
        }
        address 10.20.30.2/30;
    }
}

[edit]
root@jtac#

Modification History

2020-07-08: Article reviewed for accuracy; no changes required.