Description

This article describes the issue of SYN packets being dropped, as the session is not correctly aged out after the 4-way FIN close.

It is also applicable to UDP sessions, when sessions are not correctly aged out.

Symptoms

From the session output, you can see that the FIN 4-way close has been completed:

admin@fw1> show security flow session session-identifier 130758
Session ID: 130758, Status: Normal, State: Active
Flag: 0x88000040
Policy name: web/47
Source NAT pool: Null
Maximum timeout: 10, Current timeout: 102712
Session State: Valid
Start time: 16097417, Duration: 28362
  In: 198.12.13.10/42280 --> 198.12.10.12/11350;tcp,
     Interface: reth5.0,
     Session token: 0x9, Flag: 0x0x21
     Route: 0x66e3c2, Gateway: 198.12.13.10, Tunnel: 0
     Port sequence: 0, FIN sequence: 0,
     FIN state: 2, <
     Pkts: 29, Bytes: 2244
  Out: 198.12.10.12/11350 --> 198.12.13.10/42280;tcp,
     Interface: reth3.0,
     Session token: 0x7, Flag: 0x0x20
     Route: 0x4403c2, Gateway: 198.12.10.12, Tunnel: 0
     Port sequence: 0, FIN sequence: 0,
     FIN state: 2, --> Pkts: 6, Bytes: 380
Total sessions: 1

Note : For more information about session FIN state, refer to KB22738 - [SRX] SYN packet gets dropped in the TCP session [juniper.net] .

The FIN state is 2 for both of the session wings. In such cases, the session timeout should be set to 2 seconds and subsequently aged out. However, you can see from the above Current timeout that this is not the case.

In this situation, if a new SYN packet which matches this session hits the SRX, the SYN packet will be dropped by TCP sequence checking (which is enabled by default). To check if new SYN packets are being dropped by the TCP sequence check, you can check the following interface counters:

root@SRX3400-51# run show interfaces reth0 extensive | match seq
TCP sequence number out of window: 1

root@SRX3400-51# run show interfaces reth0 extensive | match seq
TCP sequence number out of window: 2

Another side effect of this issue is that you may see invalidated sessions not getting cleared from the session counters. For example:

{primary:node0}
root@-fw1> show security flow session summary
node0:
---------------------------------------------
Unicast-sessions: 0
Multicast-sessions: 0
Failed-sessions: 0
Sessions-in-use: 2
Valid sessions: 0
Pending sessions: 0
Invalidated sessions: 2 <
Sessions in other states: 0
Maximum-sessions: 524288

node1:
----------------------------------------------
Unicast-sessions: 0
Multicast-sessions: 0
Failed-sessions: 0
Sessions-in-use: 3
Valid sessions: 0
Pending sessions: 0
Invalidated sessions: 3 <
Sessions in other states: 0
Maximum-sessions: 524288

The Invalidated session counter is a real time counter. It counts sessions which have reached timeout 0 and are to be cleared and eventually cleared from the session table. Hence, the journey of a session goes from Timeout to Invalidated to deleted. (Note: In a normal state, this counter value should go UP and DOWN showing invalidation and correct deletion.) Since there are no longer any sessions on the firewall, there should not be any sessions shown in the invalidated session counter as well.

Solution

  • This was a software issue due to multi-threaded infrastructure. At times, the session ager may miss aging out sessions; especially when the traffic rate is high.
  • This issue affects both TCP and UDP sessions
  • This was initially reported on SRX650 and SRX240 platforms i.e. Branch, but could also impact High End on the affected versions.

Note : In case of UDP traffic being affected by this issue (such as DNS traffic), there is a high chance that short-lived UDP sessions not being aged out appropriately could cause traffic stop issues or session table getting full on the SRX device.

The fix for this issue was included in the following releases:

  • 10.4R9
  • 11.2R2
  • 11.3R1 or later

Further, there could be other reasons in later versions when the said counter " Invalidated sessions " get stuck or reach the PFE's max limit.

In that case,
We can collect the invalid sessions to understand and confirm what is stuck and if the same issue matches the traffic in question.
But since the same information needs to be collected from PFE of the box, it is highly recommended to open a JTAC case and have a JTAC Engineer collect and confirm the issue for you in a controlled environment.

Modification History

2019-07-24: Added more clarity about the invalidated session and how to monitor whether the counter is good/problematic in production.