On all SRX platforms using IKED, some IPSec tunnels may fail to form after recovering from the session table is full or under temporary stress conditions.
The issue occurs because IKED does not handle notifications from the PFE properly during tunnel installation failures, leading to stale sessions and failure for new tunnel installations due to lack of available space.
Firewall has the session table full because of traffic burst:
root@fw1> show security monitoring
Flow session Flow session CP session CP session
FPC PIC CPU Mem current maximum current maximum
10 0 49 40 13106372 13107200 15728639 15728640
10 1 99 66 26214399 26214400 31457278 31457280
Total Sessions: 39320771 39321600 47185917 47185920 <<<< Flow session and CP sessions are maxed out
root@fw1> show log messages.2.gz | match "Alarm set"
Jan 22 08:52:54 fw1 alarmd[33371]: Alarm set: SESSION TABLE id=-2147483647, color=YELLOW, class=ETHER, reason=Flow session table is almost full on FPC 10 PIC 0
Jan 22 08:52:54 fw1 craftd[32572]: Minor alarm set, Flow session table is almost full on FPC 10 PIC 0
Jan 22 08:52:59 fw1 alarmd[33371]: Alarm set: SESSION TABLE id=-2147483646, color=YELLOW, class=ETHER, reason=DCP session table is almost full on FPC 10 PIC 0
Jan 22 08:52:59 fw1 craftd[32572]: Minor alarm set, DCP session table is almost full on FPC 10 PIC 0
Jan 22 08:53:20 fw1 alarmd[33371]: Alarm set: SESSION TABLE id=-2147483631, color=YELLOW, class=ETHER, reason=Flow session table is almost full on FPC 10 PIC 1
Jan 22 08:53:20 fw1 craftd[32572]: Minor alarm set, Flow session table is almost full on FPC 10 PIC 1
Jan 22 08:53:20 fw1 alarmd[33371]: Alarm set: SESSION TABLE id=-2147483630, color=YELLOW, class=ETHER, reason=DCP session table is almost full on FPC 10 PIC 1
Jan 22 08:53:20 fw1 craftd[32572]: Minor alarm set, DCP session table is almost full on FPC 10 PIC 1
Jan 22 08:54:00 fw1 alarmd[33371]: Alarm set: SESSION TABLE id=-2147483646, color=YELLOW, class=ETHER, reason=DCP session table is almost full on FPC 10 PIC 0
Jan 22 08:54:00 fw1 craftd[32572]: Minor alarm set, DCP session table is almost full on FPC 10 PIC 0
After traffic burst is done and session table is no longer full, some tunnels are inactive on the PFE side even though they show in up state on the routing engine.
root@fw1> show security ike sa
Index State Initiator cookie Responder cookie Mode Remote Address
1280 UP d9b66d653128d9cb 7f75fedda0b42107 IKEv2 1.1.1.65
1281 UP 2d9115a4f4015646 25b0ee662109f7c1 IKEv2 1.1.1.144
1028 UP 5251770c247a90be bcb3ad1a4af33428 IKEv2 1.1.1.110
=== SPU10.1, fpc10.pic1 flowd> show usp ipsec sa
Tunnel | Crypto/Hash | | |Auto|Dyna|Anti|VPN |Bind ifp/| Bytes | Bytes |Invalid|TS check|Inactive| Local | Remote |Thd |
id | Algo |Proto| Status |Key | EP |Rply|Mon |Policy id| In | Out | SPI | FAIL | SA |Address | Address|ID |
--------+---------------+-----+---------+----+----+----+----|---------|-------------|-------------|-------+--------|--------|--------|-----------|
500562 | NULL/NULL |ESP |Inactive |Manu|Disa|Coun|VD |st0.28 | 0| 0|0 |0 |0 |1.1.1.1 |1.1.1.110|3| <<<<<<<<<<<<<<<< Tunnel/BGP is down:Full packet loss
500743 | AES256/SHA256|ESP |Active |Auto|Disa|Coun|VD |st0.26 | 282941| 564432|34 |0 |77 |1.1.1.1 |1.1.1.144|12|<<<<<<<<<<<<<<<< Partial packet loss
501032 | AES256/SHA256|ESP |Active |Auto|Disa|Coun|VD |st0.178 | 509341| 967604|0 |0 |0 |1.1.1.1 |1.1.1.65|15|<<<<<<<<<<<<<<<<< No packet loss
The inactive tunnels will affect BGP connections that are established on top of these tunnels
root@fw1> show bgp summary | refresh | match groups
---(refreshed at 2025-01-22 08:53:12 PST)---
Groups: 4 Peers: 330 Down peers: 330 <<<<<<<<<<<<<<<<<<<<<< All BGP connections are down due to traffic burst and session full
---(refreshed at 2025-01-22 08:06:17 PST)---
...
Groups: 4 Peers: 330 Down peers: 297
---(refreshed at 2025-01-22 12:15:22 PST)---
Groups: 4 Peers: 330 Down peers: 16 <<<<<<<<<<<<<<<<<<<<<< Some BGP connections are still down even after 4 hours after session table recovered due to inactive tunnel entries on the PFE
Other tunnels are showing active on PFE side however partial packet loss is observed for some traffic.
If ping is originated from the firewall (st0.26) or generated from the peer destined to the firewall st0.26 , it shows partial loss in both scenarios
set interfaces st0 unit 26 family inet address 10.255.184.254/31
root@fw1> ping 10.255.184.255
PING 10.255.184.255 (10.255.184.255): 56 data bytes
64 bytes from 10.255.184.255: icmp_seq=13 ttl=64 time=6.300 ms
64 bytes from 10.255.184.255: icmp_seq=14 ttl=64 time=6.294 ms
64 bytes from 10.255.184.255: icmp_seq=15 ttl=64 time=6.212 ms
64 bytes from 10.255.184.255: icmp_seq=24 ttl=64 time=6.293 ms <<<<<<<<<< packet loss for ping originated from firewall to the peer
64 bytes from 10.255.184.255: icmp_seq=25 ttl=64 time=6.263 ms
64 bytes from 10.255.184.255: icmp_seq=26 ttl=64 time=6.198 ms
root@peer> ping 10.255.184.254
PING 10.255.184.254 (10.255.184.254): 56 data bytes
64 bytes from 10.255.184.254: icmp_seq=38 ttl=64 time=8.133 ms
64 bytes from 10.255.184.254: icmp_seq=39 ttl=64 time=6.139 ms
64 bytes from 10.255.184.254: icmp_seq=48 ttl=64 time=6.233 ms <<<<<<<<<< packet loss for ping originated from peer to the firewall
64 bytes from 10.255.184.254: icmp_seq=49 ttl=64 time=6.279 ms
root@fw1> ping 10.255.184.255 rapid count 10
....!.!!..
--- 10.253.32.7 ping statistics ---
10 packets transmitted, 3 packets received, 70% packet loss
round-trip min/avg/max/stddev = 5.939/6.389/7.193/0.570 ms
root@fw1> show security packet-drop records | match 10.255.184.255 | match icmp
Jan 17 07:05:51
06:47:03.462715:FPC10PIC0: LSYS-ID-00 10.255.184.255/29293-->10.255.184.254/43;icmp,ipid-44604,st0.26,Dropped by FLOW:pkt drop invalid tunnel <<<<<<error for incoming ping
07:05:27.737211:FPC10PIC0: LSYS-ID-00 10.255.184.254/28616-->10.255.184.255/7;icmp,ipid-45722,.local..0,Dropped by FLOW:First path Out ifp error <<<<<<error for outgoing ping
Jan 17 05:52:50 05:52:50.679171:CID-00:FPC-10:PIC-00:THREAD_ID-26:LSYS_ID-00:RT: get_nsp_tunnel - Tunnel not found. if st0.222, nexthop ip 0x7020fd0a, policy id 1
Jan 17 05:52:50 05:52:50.679172:CID-00:FPC-10:PIC-00:THREAD_ID-26:LSYS_ID-00:RT: packet dropped, outgoing tunnel missing in tunnel-if
No nsp tunnel installed on the SPC so partial traffic loss is happening on non-anchor PIC
From start shell mode issue the below command
cprod -A fpc10.pic0 -c show usp ipsec nsp-tunnel-table | grep 500743
(no output) <----------no nsp tunnel found.
The mini-SA for non working tunnel shows 0x0 for session:
cprod -A fpc10.pic0 -c show usp ipsec mini-sa | grep 500743
Tunnel | Anchor SPU | Session | NATT |Hash Bckt| DF bit |ADVPN|Multi-SA| Inactive|
id |(tid:spu:ike )| | Session | | | | | SA |
--------+------------- +----------+----------+---------|--------|-----|--------|---------|
500743 | 19:41:1| 0x0|0x0|2894 | clear|FALSE|FALSE |FALSE |0
Session is 0x0.
For working tunnel (st0.178) the mini-SA and nsp tunnel match:
root@fw1# cprod -A fpc10.pic1 -c show usp ipsec mini-sa | grep 503336
501032 | 15: 40: 1|0xc25f8d000|0x0|1375 | clear|FALSE|FALSE|FALSE|0 |
Session = 0xc25f8d000
root@fw1# cprod -A fpc10.pic1 -c show usp ipsec nsp-tunnel-table | grep 501032
[44584][1]: tunnel_id 501032, nsp_tunnel 0xc25f8d000, gen_id 0
In summary:
1. If tunnel fails on anchor PIC then full traffic loss is observed.2. If tunnel fails on non-anchor PIC then partial traffic loss will be seen as anchor PIC will keep on processing the traffic.
The issue is fixed via this PR
https://prsearch.juniper.net/PR1757178
If upgrade is not an option, then you can manually clear sessions and tunnel to recover from problematic state.
clear security flow session all
clear security ike security-association"