Description

Junos does not support the function similar to 'unset flow reverse-route clear-text' in ScreenOS

Symptoms

SRX has two ISPs, topology and configuration as follows:

ISP-B------(ge-8/0/0.0) SRX5800 (ae0.0)------host(10.1.1.10)
ISP-A------(ge-8/0/4.0)
 
ge-8/0/0.0 is in ISP-B zone;
ge-8/0/4.0 is in ISP-A zone;
Static NAT is configured to map 192.168.38.143 to 10.1.1.10;
 

The issue is that traffic with source/destination address pair 172.16.67.130/192.168.38.143 comes into the SRX ge-8/0/4.0 interface within zone ISP-A, a new session is established and the packet is forwarded out of ae0.0. However, the reverse route lookup for source address 172.16.67.130 is pointing to ge-8/0/0 within ISP-B zone.

When the return packet comes back into ae0 and matches the existing session, the route lookup for 172.16.67.130 is pointing to ge-8/0/0.0 within ISP-B zone instead of the ge-8/0/4.0 within ISP-A zone, so the packet is dropped due to re-route failed .

Below is the flow traceoptions of the return packet being dropped:

Nov 18 18:56:19 03:34:52.1332242:CID-00:FPC-11:PIC-01:THREAD_ID-10:RT:<10.1.1.10/0->172.16.67.130/39087;1> matched filter b:
Nov 18 18:56:19 03:34:52.1332278:CID-00:FPC-11:PIC-01:THREAD_ID-10:RT:packet [60] ipid = 28663, @7d6668e4
Nov 18 18:56:19 03:34:52.1332298:CID-00:FPC-11:PIC-01:THREAD_ID-10:RT:---- flow_process_pkt: (thd 10): flow_ctxt type 13, common flag 0x0, mbuf 0xe90a200
Nov 18 18:56:19 03:34:52.1332321:CID-00:FPC-11:PIC-01:THREAD_ID-10:RT: flow process pak fast ifl 69 in_ifp ae0.0
Nov 18 18:56:19 03:34:52.1332333:CID-00:FPC-11:PIC-01:THREAD_ID-10:RT:flow_np_session_id2nsp: NP hdr: session id - 621247072, Flag - 0
Nov 18 18:56:19 03:34:52.1332353:CID-00:FPC-11:PIC-01:THREAD_ID-10:RT:  flow session id 490080
Nov 18 18:56:19 03:34:52.1332383:CID-00:FPC-11:PIC-01:THREAD_ID-10:RT:  route lookup failed : dest-ip 172.16.67.130 orig ifp ge-8/0/4.0 output_ifp ge-8/0/0.0 fto 0x136f4d48 orig-zone 11 out-zone 10 vsd 0
Nov 18 18:56:19 03:34:52.1332426:CID-00:FPC-11:PIC-01:THREAD_ID-10:RT:  packet dropped,   pak dropped since re-route failed

Solution

1. In ScreenOS, if the command " unset flow reverse-route clear-text " is set, the reverse packet to the initiator will be forwarded out exactly via the interface in the session table, so the symmetric path is followed.

2. On a SRX devices, no such function exists in ScreenOS, so the asymmetric patch is followed. The work-around is to set the two interfaces on the asymmetric side in the same zone, then the C2S traffic will go over a different path with S2C traffic. Below is the flow traceoptions output before and after implementing the work-around.

Initial packet flow traceoptions:

Nov 19 03:33:13 09:58:51.702239:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:SPU received an event, type 80
Nov 19 03:33:13 09:58:51.702253:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:SPU received pak with event message from CP, cp_sess_id=0001cd22 a
Nov 19 03:33:13 09:58:51.702298:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:<172.16.67.130/32795->192.168.38.143/7;6> matched filter a:
Nov 19 03:33:13 09:58:51.702333:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:packet [60] ipid = 40929, @7de4f910
Nov 19 03:33:13 09:58:51.702353:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:---- flow_process_pkt: (thd 19): flow_ctxt type 17, common flag 0x0, mbuf 0xe96f600
Nov 19 03:33:13 09:58:51.702379:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:flow process pak, mbuf e96f600, ifl 71, ctxt_type 17 inq type 1
Nov 19 03:33:13 09:58:51.702402:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT: in_ifp <untrust:ge-8/0/4.0>
Nov 19 03:33:13 09:58:51.702414:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:lpak_init: lpak 250d4358, paksize 60, machdr 0, iphdr 0x7de4f910
Nov 19 03:33:13 09:58:51.702440:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:flow_process_pkt_exception: setting rtt in lpak to 24d3e568
Nov 19 03:33:13 09:58:51.702458:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:inq_type 0x1
Nov 19 03:33:13 09:58:51.702468:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:flow_process_pkt_exception: local_flag: 0x00000100
Nov 19 03:33:13 09:58:51.702487:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:  ge-8/0/4.0:172.16.67.130/32795->192.168.38.143/7, tcp, flag 2 syn
Nov 19 03:33:13 09:58:51.702524:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT: find flow: table 0x54914e60, hash 364268(0x7ffff), sa 172.16.67.130, da 192.168.38.143, sp 32795, dp 7, proto 6, tok 448
Nov 19 03:33:13 09:58:51.702580:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:  no session found, start first path. in_tunnel - 0, from_cp_flag - 2048
Nov 19 03:33:13 09:58:51.702609:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:  flow_first_create_session
Nov 19 03:33:13 09:58:51.702628:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:Installing pending sess (118007) in ager
Nov 19 03:33:13 09:58:51.702644:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:First path alloc and instl pending session, natp=0x35e2bb58, id=118007
Nov 19 03:33:13 09:58:51.702669:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:  flow_first_in_dst_nat: in <ge-8/0/4.0> , out <N/A> dst_adr 192.168.38.143, sp 32795, dp 7
Nov 19 03:33:13 09:58:51.702702:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:  chose interface ge-8/0/4.0 as incoming nat if.
Nov 19 03:33:13 09:58:51.702720:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:nat_get_mapped_ip: zone untrust found if_mip 0x5d31a480.
Nov 19 03:33:13 09:58:51.702743:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:nat_get_mapped_ip_if_mip: IPv4 to IPv4: v4 192.168.38.143 -> v4 10.1.1.10
Nov 19 03:33:13 09:58:51.702777:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:nat_get_mapped_or_incoming_dip: map_index 1 for dst/new IP 192.168.38.143/10.1.1.10 on interface ge-8/0/4.0(root)
Nov 19 03:33:13 09:58:51.702817:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:nat_get_mapped_or_incoming_dip: diff 0, calls 0
Nov 19 03:33:13 09:58:51.702832:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:flow_first_rule_dst_xlate: packet 172.16.67.130->192.168.38.143 nsp2 0.0.0.0->10.1.1.10.
Nov 19 03:33:13 09:58:51.702881:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:flow_first_routing: call flow_route_lookup(): src_ip 172.16.67.130, x_dst_ip 10.1.1.10, in ifp ge-8/0/4.0, out ifp N/A sp 32795, dp 7, ip_proto 6, tos 10
Nov 19 03:33:13 09:58:51.702927:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:Doing DESTINATION addr route-lookup
Nov 19 03:33:13 09:58:51.702939:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:flow_ipv4_rt_lkup in VR-id: 0
Nov 19 03:33:13 09:58:51.702955:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:flow_ipv4_rt_lkup: Found route entry 0x0x5d56ad38,nh id 0x22a, out if 0x40
Nov 19 03:33:13 09:58:51.702981:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:flow_ipv4_rt_lkup: nh word 0x50010
Nov 19 03:33:13 09:58:51.702995:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:flow_ipv4_rt_lkup success 10.1.1.10, iifl 0x47, oifl 0x40
Nov 19 03:33:13 09:58:51.703022:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:  routed (x_dst_ip 10.1.1.10) from untrust (ge-8/0/4.0 in 0) to ae0.0, Next-hop: 10.1.1.10
Nov 19 03:33:13 09:58:51.703061:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:  policy search from zone untrust-> zone trust
Nov 19 03:33:13 09:58:51.703126:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:  app 0, timeout 1800s, curr ageout 20s
Nov 19 03:33:13 09:58:51.703143:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:Permitted by policy 5
Nov 19 03:33:13 09:58:51.703162:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:nat_vsys_hip_search: No Host found for static nat 172.16.67.130 on ifp ae0.0
Nov 19 03:33:13 09:58:51.703193:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:flow_first_src_xlate: 172.16.67.130/32795 -> 192.168.38.143/7 | 10.1.1.10/7 -> 0.0.0.0/32795: nat_src_xlated: False, nat_src_xlate_failed: False
Nov 19 03:33:13 09:58:51.703260:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:pst_nat_proc_from_internal: failed to get binding, lsys_id: 0, ip/port: 172.16.67.130/32795
Nov 19 03:33:13 09:58:51.703295:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:flow_first_src_xlate: src nat 0.0.0.0(32795) to 10.1.1.10(7) returns status: 0, rule/pool id: 0/0, pst_nat: False.
Nov 19 03:33:13 09:58:51.703336:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:  dip id = 0/0, 172.16.67.130/32795->172.16.67.130/32795
Nov 19 03:33:13 09:58:51.703374:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:flow_first_get_out_ifp: 1000 -> cone nat test
Nov 19 03:33:13 09:58:51.703386:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:  choose interface ae0.0 as outgoing phy if
Nov 19 03:33:13 09:58:51.703400:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:is_loop_pak: No loop: on ifp: ae0.0, addr: 10.1.1.10, rtt_idx:0
Nov 19 03:33:13 09:58:51.703429:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:policy is NULL (wx/pim scenario)
Nov 19 03:33:13 09:58:51.703444:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:sm_flow_interest_check: app_id 0, policy 5, app_svc_en 0, flags 0x2. not interested
Nov 19 03:33:13 09:58:51.703465:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:flow_first_service_lookup(): natp(0x35e2bb58): app_id, 0(0).
Nov 19 03:33:13 09:58:51.703487:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:  service lookup identified service 0.
Nov 19 03:33:13 09:58:51.703498:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:  flow_first_final_check: in <ge-8/0/4.0> , out <ae0.0>
Nov 19 03:33:13 09:58:51.703514:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:flow_first_final_check: flow_set_xlate_vector.
Nov 19 03:33:13 09:58:51.703525:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:In flow_first_complete_session
Nov 19 03:33:13 09:58:51.703535:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:flow_first_complete_session: pak_ptr is xlated packet
Nov 19 03:33:13 09:58:51.703553:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:  existing vector list 1002-13619e28.
Nov 19 03:33:13 09:58:51.703571:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:  Session (id:118007) created for first pak 1002
Nov 19 03:33:13 09:58:51.703590:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:first pak processing successful
Nov 19 03:33:13 09:58:51.703599:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:  flow_first_install_session======> 0x35e2bb58
Nov 19 03:33:13 09:58:51.703615:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT: nsp 0x35e2bb58, nsp2 0x35e2bbd4
Nov 19 03:33:13 09:58:51.703637:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:  make_nsp_ready_no_resolve()
Nov 19 03:33:13 09:58:51.703652:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:flow_ipv4_rt_lkup in VR-id: 0
Nov 19 03:33:13 09:58:51.703664:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:flow_ipv4_rt_lkup: Found route entry 0x0x5d31b088,nh id 0x230, out if 0x48
Nov 19 03:33:13 09:58:51.703687:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:flow_ipv4_rt_lkup: nh word 0x97b0728
Nov 19 03:33:13 09:58:51.703704:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:flow_ipv4_rt_lkup success 172.16.67.130, iifl 0x0, oifl 0x48
Nov 19 03:33:13 09:58:51.703731:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:  route lookup: dest-ip 172.16.67.130 orig ifp ge-8/0/4.0 output_ifp ge-8/0/0.0 orig-zone 7 out-zone 7 vsd 0
Nov 19 03:33:13 09:58:51.703762:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:  reroute handling for tunnel 0       
Nov 19 03:33:13 09:58:51.703772:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:  clearing tunnel since the routed interface is ge-8/0/0.0
Nov 19 03:33:13 09:58:51.703784:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:new output if ge-8/0/0.0 <<< the reverse route lookup is done, reverse route interface is ge-8/0/0.0, the response packet from 10.1.1.10 will be forwarded out via ge-8/0/0.0 instead of ge-8/0/4.0
Nov 19 03:33:13 09:58:51.703793:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:  route to 172.16.67.130
Nov 19 03:33:13 09:58:51.703826:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:queue pak for pending session 118007, natp=0x35e2bb58, paks queued 1
Nov 19 03:33:13 09:58:51.703857:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:SPU send install sess to CP cp_sess_id=0001cd22, spu_sess_id=0001ccf7, natp=0x35e2bb58
Nov 19 03:33:13 09:58:51.703891:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:first path session installation succeeded
Nov 19 03:33:13 09:58:51.703903:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:  flow found or created a pending session.
Nov 19 03:33:13 09:58:51.703915:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT:flow_process_pkt_exception: Freeing lpak 250d4358 associated with mbuf 0xe96f600
Nov 19 03:33:13 09:58:51.703941:CID-00:FPC-07:PIC-00:THREAD_ID-19:RT: ----- flow_process_pkt rc 0xf (fp rc 0)



Response packet flow traceoptions after work-around:

When both ge-8/0/0.0 and ge-8/0/4.0 are put in the same zone, e.g, ISP-A, the return packet from 10.1.1.10 will be forwarded out via ge-8/0/0.0, as the destination address of return packet is routed via ISP-B.

Nov 19 03:33:16 09:58:54.1106055:CID-00:FPC-07:PIC-00:THREAD_ID-29:RT:packet [60] ipid = 0, @7b1f20e4
Nov 19 03:33:16 09:58:54.1106075:CID-00:FPC-07:PIC-00:THREAD_ID-29:RT:---- flow_process_pkt: (thd 29): flow_ctxt type 13, common flag 0x0, mbuf 0xe737800
Nov 19 03:33:16 09:58:54.1106100:CID-00:FPC-07:PIC-00:THREAD_ID-29:RT: flow process pak fast ifl 64 in_ifp ae0.0
Nov 19 03:33:16 09:58:54.1106115:CID-00:FPC-07:PIC-00:THREAD_ID-29:RT:flow_np_session_id2nsp: NP hdr: session id - 118007, Flag - 0
Nov 19 03:33:16 09:58:54.1106133:CID-00:FPC-07:PIC-00:THREAD_ID-29:RT:NP session id - 118007 returns Non-Init side nsp -0x35e2bbd4
Nov 19 03:33:16 09:58:54.1106158:CID-00:FPC-07:PIC-00:THREAD_ID-29:RT:  flow session id 118007
Nov 19 03:33:16 09:58:54.1106177:CID-00:FPC-07:PIC-00:THREAD_ID-29:RT:flow_tcp_wsf_update: wsf 2
Nov 19 03:33:16 09:58:54.1106189:CID-00:FPC-07:PIC-00:THREAD_ID-29:RT:  tcp flags 0x12, flag 0x12
Nov 19 03:33:16 09:58:54.1106202:CID-00:FPC-07:PIC-00:THREAD_ID-29:RT:  Got syn_ack, 10.1.1.10(7)->172.16.67.130(32795), nspflag 0x1020, 0x1021
Nov 19 03:33:16 09:58:54.1106249:CID-00:FPC-07:PIC-00:THREAD_ID-29:RT:  post addr xlation: 23.23.23.200->172.16.67.130.
Nov 19 03:33:16 09:58:54.1106280:CID-00:FPC-07:PIC-00:THREAD_ID-29:RT:mbuf 0xe737800, exit nh 0x97b0728 <<< the return packet from 10.1.1.10 is forwarded via ge-8/0/0.0 based on the reverse route done at session setup
Nov 19 03:33:16 09:58:54.1106299:CID-00:FPC-07:PIC-00:THREAD_ID-29:RT: ----- flow_process_pkt rc 0x0 (fp rc 0)

Modification History

2019-08-23: Updated overall explanations based on customer feedback.