This article addresses troubleshooting traffic flows and session establishment on all SRX devices. It also addresses packet capture and analysis on SRX Branch devices.
This article contains the following sections for troubleshooting traffic through SRX devices. Click the section title below to jump directly to that section:
192.168.224.30 (Server in Trust zone) ---------SRX-------- 1.1.1.100 (client in Untrust zone) SRX interfaces: ge-0/0/0.0: IP Address 192.168.224.3/24 (Trust) fe-0/0/7.0: IP Address 1.1.1.50/24 (Untrust) Static NAT on SRX for server: 1.1.1.30 (public IP address for server) -> 192.168.224.30 (internal Server IP address) Source NAT on SRX for incoming clients: (for this scenario because 192.168.244.30 network does not have knowledge of 1.1.1.0/24 network) 1.1.1.0/24 -> 192.168.224.3
192.168.224.30 (Server in Trust zone) ---------SRX-------- 1.1.1.100 (client in Untrust zone)
set security flow traceoptions file DebugTraffic set security flow traceoptions flag basic-datapath # define filter to capture traffic from client to Server's public IP address 1.1.1.30 set security flow traceoptions packet-filter MatchTraffic source-prefix 1.1.1.100/32 destination-prefix 1.1.1.30/32 (Note: You may also specify a subnet for the prefix, 1.1.1.0/24, instead of the client's /32 IP address) #define filter to capture traffic from Server in Trust to Client's NAT'd address set security flow traceoptions packet-filter MatchTrafficReverse source-prefix 192.168.224.30/32 destination-prefix 192.168.224.3/32
set security flow traceoptions file DebugTraffic set security flow traceoptions flag basic-datapath
set security flow traceoptions packet-filter MatchTraffic source-prefix 1.1.1.100/32 destination-prefix 1.1.1.30/32
set security flow traceoptions packet-filter MatchTrafficReverse source-prefix 192.168.224.30/32 destination-prefix 192.168.224.3/32
Aug 12 16:22:22 16:22:21.1100315:CID-0:RT:< 1.1.1.100/57650 ->1.1.1.30/3389;6 > matched filter MatchTraffic:
)
Aug 12 16:22:22 16:22:21.1100315:CID-0:RT:packet [48] ipid = 885 , @422ec91e Aug 12 16:22:22 16:22:21.1100315:CID-0:RT:---- flow_process_pkt: (thd 1): flow_ctxt type 13, common flag 0x0, mbuf 0x422ec780 Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: flow process pak fast ifl 69 in_ifp fe-0/0/7.0
Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: fe-0/0/7.0 :1.1.1.100/57650->1.1.1.30/3389, tcp, flag 2 syn
Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: find flow: table 0x4dd31658, hash 32955(0xffff), sa 1.1.1.100, da 1.1.1.30, sp 57650, dp 3389, proto 6, tok 448
Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: no session found, start first path. in_tunnel - 0, from_cp_flag - 0 Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: flow_first_create_session
Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: flow_first_in_dst_nat: in <fe-0/0/7.0> , out <N/A> dst_adr 1.1.1.30, sp 57650, dp 3389 Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: chose interface fe-0/0/7.0 as incoming nat if.
Aug 12 16:22:22 16:22:21.1100315:CID-0:RT:flow_first_rule_dst_xlate: packet 1.1.1.100->1.1.1.30 nsp2 0.0.0.0->192.168.224.30.
Aug 12 16:22:22 16:22:21.1100315:CID-0:RT:flow_first_routing: call flow_route_lookup(): src_ip 1.1.1.100, x_dst_ip 192.168.224.30, in ifp fe-0/0/7.0, out ifp N/A sp 57650, dp 3389, ip_proto 6, tos 0 Aug 12 16:22:22 16:22:21.1100315:CID-0:RT:Doing DESTINATION addr route-lookup
Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: routed (x_dst_ip 192.168.224.30) from untrust (fe-0/0/7.0 in 0) to ge-0/0/0.0, Next-hop: 192.168.224.30
Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: policy search from zone untrust-> zone trust Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: app 0, timeout 1800s, curr ageout 20s
Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: packet dropped, denied by policy Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: packet dropped, policy deny. Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: flow find session returns error. Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: ----- flow_process_pkt rc 0x7 (fp rc -1)
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: <1.1.1.100/51303->1.1.1.30/3389;6 > matched filter MatchTraffic:
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:packet [48] ipid = 5015 , @423d7e9e Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:---- flow_process_pkt: (thd 1): flow_ctxt type 13, common flag 0x0, mbuf 0x423d7d00
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: flow process pak fast ifl 72 in_ifp fe-0/0/7.0
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: fe-0/0/7.0:1.1.1.100/51303->1.1.1.30/3389, tcp, flag 2 syn
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: find flow: table 0x5258d7b0, hash 17008(0xffff), sa 1.1.1.100, da 1.1.1.30, sp 51303, dp 3389, proto 6 , tok 448
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: no session found , start first path. in_tunnel - 0, from_cp_flag – 0
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: flow_first_create_session Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: flow_first_in_dst_nat: in <fe-0/0/7.0> , out <N/A> dst_adr 1.1.1.30, sp 51303, dp 3389
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: chose interface fe-0/0/7.0 as incoming nat if. Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:flow_first_rule_dst_xlate: packet 1.1.1.100->1.1.1.30 nsp2 0.0.0.0->192.168.224.30.
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:flow_first_routing: call flow_route_lookup(): src_ip 1.1.1.100, x_dst_ip 192.168.224.30, in ifp fe-0/0/7.0, out ifp N/A sp 51303, dp 3389, ip_proto 6, tos 0
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:Doing DESTINATION addr route-lookup
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: routed (x_dst_ip 192.168.224.30) from untrust (fe-0/0/7.0 in 0) to ge-0/0/0.0, Next-hop: 192.168.224.30
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: policy search from zone untrust-> zone trust Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: policy has timeout 900
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: app 0, timeout 1800s, curr ageout 20s
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:flow_first_src_xlate: src nat 0.0.0.0(51303) to 192.168.224.30(3389) returns status 1, rule/pool id 1/2.
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: dip id = 2/0, 1.1.1.100/51303->192.168.224.3/48810
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: choose interface ge-0/0/0.0 as outgoing phy if Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:is_loop_pak: No loop: on ifp: ge-0/0/0.0, addr: 192.168.224.30, rtt_idx:0
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:policy is NULL (wx/pim scenario) Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:sm_flow_interest_check: app_id 0, policy 9, app_svc_en 0, flags 0x2. not interested
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:sm_flow_interest_check: app_id 1, policy 9 , app_svc_en 0, flags 0x2. not interested
## TIP: The policy 9 content can be checked by looking through the policy list with the command “show security policies” or with the command “show security policies | find "Index\: 9"” which uses the regular expression to search for Index: 9 in the policy and will show the first instance. Visually make sure that the traffic is matching the correct rule .
.
root@SRX210# run show security policies | find "Index\: 9" Policy: Allow-RDP-Server, State: enabled, Index: 9, Sequence number: 1 Source addresses: 1.1.1.0/24 Destination addresses: 192.168.224.30 Applications: RDP Action: permit, log
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:flow_first_service_lookup(): natp(0x51ee4680): app_id, 0(0). Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: service lookup identified service 0.
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: flow_first_final_check: in <fe-0/0/7.0> , out <ge-0/0/0.0>
Aug 14 19:11:40 19:11:40.192283:CID-0:RT:< 192.168.224.30/3389 ->192.168.224.3/48810;6 > matched filter MatchTrafficReverse: Aug 14 19:11:40 19:11:40.192283:CID-0:RT:packet [40] ipid = 8174, @42308b9e Aug 14 19:11:40 19:11:40.192283:CID-0:RT:---- flow_process_pkt: (thd 1): flow_ctxt type 13, common flag 0x0, mbuf 0x42308a00 Aug 14 19:11:40 19:11:40.192283:CID-0:RT: flow process pak fast ifl 67 in_ifp ge-0/0/0.0 Aug 14 19:11:40 19:11:40.192283:CID-0:RT: ge-0/0/0.0:192.168.224.30/3389->192.168.224.3/48810, tcp, flag 10
Aug 14 19:11:40 19:11:40.192283:CID-0:RT: find flow: table 0x4dd31658, hash 44831(0xffff), sa 192.168.224.30, da 192.168.224.3, sp 3389, dp 48810, proto 6, tok 384 Aug 14 19:11:40 19:11:40.192283:CID-0:RT: flow got session.
Aug 14 19:11:40 19:11:40.192283:CID-0:RT: flow session id 12535 Aug 14 19:11:40 19:11:40.192283:CID-0:RT: tcp seq check.
Aug 14 19:11:40 19:11:40.192283:CID-0:RT: post addr xlation: 1.1.1.30->1.1.1.100.
Aug 14 19:11:40 19:11:40.192283:CID-0:RT:mbuf 0x42308a00, exit nh 0x80010 Aug 14 19:11:40 19:11:40.192283:CID-0:RT: ----- flow_process_pkt rc 0x0 (fp rc 0)
datapath-debug
monitor traffic interface
root@SRX210# set forwarding-options packet-capture file filename pcap files 10 size 10000 root@SRX210# set forwarding-options packet-capture maximum-capture-size 1500
root@SRX210# set firewall filter PCAP term capture from source-address 192.168.224.0/24 root@SRX210# set firewall filter PCAP term capture from destination-address 192.168.224.3/32 root@SRX210# set firewall filter PCAP term capture from protocol ospf root@SRX210# set firewall filter PCAP term capture from protocol esp root@SRX210# set firewall filter PCAP term capture then sample root@SRX210# set firewall filter PCAP term capture then accept root@SRX210# set firewall filter PCAP term allow-all-else then accept
root@SRX210# set interfaces fe-0/0/7 unit 0 family inet filter input PCAP root@SRX210# commit and-quit
First, configure the packet capture file information, including the file name, size, copies, and the amount of traffic to capture per packet. The minimum amount of traffic that must be captured will be 68 Bytes (Layer 2-4) but a maximum of 1500 Bytes can be captured.
Next, a packet capture filter will be applied to select what packets should be captured in this configuration. Technically the SRX does not need to filter the traffic, however it is highly recommended to filter only the desired traffic, which is what is shown in this example. This is done by configuring the firewall filter which defines what to match. You can select the source-address, destination-address, source-port, destination-port, and protocol. The following packet capture is going to collect traffic that arrives on the interface with a source address in the prefix 192.168.224.0/24 AND destination-address 192.168.224.3 AND protocol OSPF AND protocol ESP. Also the action allow-all-else is used to make sure that the SRX does not drop any other traffic, but do not sample it either. It is important to remember that the packet capture will match the traffic as it leaves or arrives on the interface based on the Layer 3 or Layer 4 properties in the packet itself. So you need to take this into account when NAT is being performed on the packet, and match the traffic as is.
Lastly, apply the filter to the interface to define that it should capture the traffic. Then commit the configuration. In this example, only the traffic arriving on the input direction is captured, but the outbound direction can be set as well in another statement.
Disabling Packet Captures: To stop the packet capture, either deactivate or delete the configuration “interfaces fe-0/0/7 unit 0 family inet filter input PCAP” and optionally, remove the configuration “firewall filter PCAP” and “forwarding-options packet-capture” as shown below:
root@SRX210# delete interfaces fe-0/0/7 unit 0 family inet filter input PCAP (deactivates packet capture) root@SRX210# delete firewall filter PCAP (optional: delete filter if not using it again) root@SRX210# delete forward-options packet-capture
root@SRX210# delete interfaces fe-0/0/7 unit 0 family inet filter input PCAP
root@SRX210# delete firewall filter PCAP
root@SRX210# delete forward-options packet-capture
root@SRX210> start shell
root@SRX210% cd /var/tmp
Either login to the SRX with the root account, or use the “start shell” command with a super-user class account permission.
Change directories to the /var/tmp directory
Issue the tcpdump command with the –r option to read the pcap formatted file (in this case the name pcap is used but that name is arbitrary). The –v option gives additional verboseness to the output and shows some additional info, and finally pcap.fe-0.0.0 is the name of the capture. There are other options that can be explored by viewing the tcpdump manual page at http://www.tcpdump.org/tcpdump_man.html .
root@SRX210% tcpdump –r –v pcap.fe-0.0.7 21:05:19.974301 In IP (tos 0xc0, ttl 64, id 3464, offset 0, flags [none], proto: ESP (50), length: 120) 192.168.224.1 > 192.168.224.3: ESP(spi=493530462,seq=0x324) 21:05:24.979969 In IP (tos 0xc0, ttl 1, id 31089, offset 0, flags [none], proto: OSPF (89), length: 72) 192.168.224.6 > ospf-all.mcast.net: OSPFv2, Hello, length 52 Router-ID 192.168.225.1, Backbone Area, Authentication Type: none (0) Options [External] Hello Timer 10s, Dead Timer 40s, Mask 255.255.255.0, Priority 1 Designated Router 192.168.224.3, Backup Designated Router 192.168.224.1 Neighbor List: Juniper2350 192.168.230.1 21:05:24.980113 In IP (tos 0xc0, ttl 64, id 3500, offset 0, flags [none], proto: ESP (50), length: 120) 192.168.224.1 > 192.168.224.3: ESP(spi=493530462,seq=0x325) 21:05:26.977403 In IP (tos 0x0, ttl 64, id 3513, offset 0, flags [none], proto: ESP (50), length: 104) 192.168.224.1 > 192.168.224.3: ESP(spi=493530462,seq=0x326) 21:05:27.985879 In IP (tos 0xc0, ttl 64, id 3515, offset 0, flags [none], proto: ESP (50), length: 136) 192.168.224.1 > 192.168.224.3: ESP(spi=493530462,seq=0x327) 21:05:28.976092 In IP (tos 0xc0, ttl 1, id 3521, offset 0, flags [none], proto: OSPF (89), length: 72) 192.168.224.1 > ospf-all.mcast.net: OSPFv2, Hello, length 52 Router-ID 192.168.230.1, Backbone Area, Authentication Type: none (0) Options [External] Hello Timer 10s, Dead Timer 40s, Mask 255.255.255.0, Priority 1 Designated Router 192.168.224.3, Backup Designated Router 192.168.224.1 Neighbor List: Juniper2350 192.168.225.1
2020-04-17: Article reviewed for accuracy; no changes required.