A MultiServices Dense Port Concentrator (MS-DPC) PIC that is used for Carrier-Grade Network Address Translation (CGNAT) may stop responding or may restart due to memory exhaustion when it has reached the limit on flows. Reducing the inactivity-timeout for HTTP and HTTPS applications will reduce the number of flows and free up memory.
This article describes how this can be done.
The main symptom is traffic reduction per service set. Consider the following MS-DPC:
FPC 3 REV 25 750-024064 CADY2028 MS-DPC CPU REV 12 710-013713 CADY2229 DPC PMB PIC 0 BUILTIN BUILTIN MS-DPC PIC PIC 1 BUILTIN BUILTIN MS-DPC PIC
Before the memory limit is reached, the following messages are seen in logs, which show the Physical Interface Card (PIC) transitioning to yellow or even red memory zones:
Jul 29 19:04:48 CGNAT-mx960-re0 (FPC Slot 3, PIC Slot 0) Entered green memory zone 900 ms ago Jul 29 19:04:48 CGNAT-mx960-re0 (FPC Slot 3, PIC Slot 0) Entered yellow memory zone 700 ms ago Jul 29 19:04:53 CGNAT-mx960-re0 (FPC Slot 3, PIC Slot 0) Entered yellow memory zone 900 ms ago Jul 29 19:04:53 CGNAT-mx960-re0 (FPC Slot 3, PIC Slot 0) Entered green memory zone 900 ms ago Jul 29 19:05:09 CGNAT-mx960-re0 (FPC Slot 3, PIC Slot 0) Entered yellow memory zone 700 ms ago Jul 29 19:05:09 CGNAT-mx960-re0 (FPC Slot 3, PIC Slot 0) Entered green memory zone 700 ms ago Jul 29 19:05:35 CGNAT-mx960-re0 (FPC Slot 3, PIC Slot 0) Entered yellow memory zone 100 ms ago
When memory is under stress and the number of flows per PIC is checked, the values will be high, as shown below:
root@CGNAT-mx960-re0> show services stateful-firewall flows count |no-more Interface Service set Flow count rsp0 NAT_Group_0 6023088 rsp1 NAT_Group_1 6148033 rsp2 NAT_Group_2 5785188 rsp3 NAT_Group_3 6119330
Subsequently, new flow creation will fail because no memory is available:
root@CGNAT-mx960-re0> request pfe execute target fpc3 pic-slot 0 command "show services fwnat counters" | match "out of" GOT: Flow create out of memory 99891376 GOT: EIM out of memory errors 0 GOT: EIF out of memory errors 0 GOT: EIM nat ports out of order 0 GOT: Det-nat out of ports error 0 {MASTER} root@CGNAT-mx960-re0> request pfe execute target fpc3 pic-slot 0 command "show services fwnat counters" | match "out of" GOT: Flow create out of memory 99923453 <<< This counter increasing!!! GOT: EIM out of memory errors 0 GOT: EIF out of memory errors 0 GOT: EIM nat ports out of order 0 GOT: Det-nat out of ports error 0
The outputs that have been included mean that memory is exhausted and the number of active flows need to be reduced. In testing, a number of approximately seven million flows was seen to be the limit that could be reached before memory was exhausted.
root@Lab-re0> show services stateful-firewall flows count | no-more Interface Service set Flow count rsp0 NAT_Group_0 6916047 rsp1 NAT_Group_1 7376324 rsp2 NAT_Group_2 7462848 rsp3 NAT_Group_3 7468552
With ~7M flows, memory used per PIC is about 97% as shown below:
root@Lab-re0> request pfe execute target fpc3 pic-slot 0 command "show arena" SENT: Ukern command: show arena GOT: GOT: ID Base Total(b) Free(b) Used(b) % Name GOT: -- -------- --------- --------- --------- --- ---- GOT: 0 c00001b0 939523664 608 939523056 99 shared memory GOT: 1 40001b0 1946156624 5 1946156568 99 shared memory+ GOT: 2 f8000030 134201297 449832 133751465 99 PACKET memory GOT: 3 c486560 1240076 1238092 1984 0 ipsec eventq GOT: 4 dcba4d0 16336 14352 1984 12 PFEMAN msgpool GOT: 5 4c814e0 78643200 78567840 75360 0 Services control arena GOT: 6 979dc38 1048528 1046840 1688 0 RPC portmap GOT: 7 1f1ab080 536870864 32 536870832 100 CGN Arena #3 GOT: 8 80200030 266338256 23571880 242766376 91 Kseg0 shared memory CALCULATED 3904038845 104889532 3799149313 97% used (Used/Total)
One solution may be to reduce the number of flows that are active in CGNAT by configuring an inactivity timeout for some key applications. Normally, HTTP and HTTPS will be the top consumers of flows and memory. So, having these flows time out faster will often be sufficient to relieve the memory stress on the card.
In the following example, 450 seconds was used to reduce the flows. After reconfiguring the timeout value, allow enough time (15 min) to observe the results of the change. It should reduce the number of active flows and free up memory on the card:
applications { application junos-http { inactivity-timeout 450; } application junos-https { inactivity-timeout 450; } }
If a given CGNAT system is seeing other applications consume the majority of the flows, the timeouts can be applied there. Note that if the traffic is constant and not ephemeral (temporary), changing the timeout will have no impact.