Customer uses MX-SPC3 to provide CGNAT for DHCP end clients. Those DHCP end clients having their corresponding static gateway interfaces configured on the MX.
They faced an issue with the SPC3 card where its stopped being able to do translations and is dropping traffic after migrating more than 60000 end clients.
From syslog, below can be seen:
Jul 18 21:20:14.449 re0 (FPC Slot 2, PIC Slot 0) usp_ipc_client_open: failed to create client socket: Resource temporarily unavailable (11)
Jul 18 21:20:16.282 re0 (FPC Slot 2, PIC Slot 0) usp_ipc_client_open: failed to create client socket: Resource temporarily unavailable (11)
Jul 18 21:23:16.785 re0 (FPC Slot 2, PIC Slot 0) usp_ipc_client_open: failed to create client socket: Connection refused (111)
Jul 18 21:23:16.785 re0 (FPC Slot 2, PIC Slot 0) usp_ipc_client_open: failed to create client socket: Operation now in progress (115)
MX-SPC3 PICs failed to carry traffic and syslog like "(FPC Slot 8, PIC Slot 1) usp_ipc_client_open: failed to create client socket: Resource temporarily unavailable (11)" can be seen
Customer has static interfaces configured to provide the gateway for the end DHCP clients. Example of interface configure as this:
ae10 {
unit 10000 {
description STAG3206CTAG1;
vlan-tags outer 3206 inner 1;
family inet {
mtu 1600;
address 10.187.160.1/30;
}
set interfaces ge-0/0/0 gigether-options 802.3ad ae10
So, for each subinerface, there will have 2 ifls being installed in the PFE ukernel, ie, ae10.10000 and ge-0/0/0.10000.
When in problem state, either PIC0 or PIC1 will not be accessible, and the kernel memory is nearly 100%.
{master}
labroot@re0> start shell
% vty fpc2.pic0
<<<<<<<<<<<<<<<<<< pic0 no response now
% vty fpc2.pic1
TOR platform (2299 Mhz Intel(R) Xeon(R) processor, 0MB memory, 16384KB flash)
FLOWD_SPC3-2.1(vty)# show heap
ID Base Total(b) Free(b) Used(b) % Name
-- ---------- ----------- ----------- ----------- --- -----------
0 7f63c0000000 1073741824 10343552 1063398272 99 Kernel
^^^^^^^^ Only 1G MB was assiged, which is too less
1 7f6a06620000 28883943424 7163653056 21720290368 75 User
FLOWD_SPC3-2.1(vty)# show ifl sum
IFL subsystem information
ifl count = 64344
ifl up count = 64278
Here is the same show output from MPC7E in the same box:
SMPC10(re0 vty)# show ifl summary
ifl count = 64358
ifl up count = 64292
SMPC10(re0 vty)# show heap
0 31338000 3321884648 2617089960 704794688 21 Kernel
<<<<<<<<<<<< 3.3GB, and 21% load only
1 1c000000 234881024 183254640 51626384 21 DMA
2 18dfffe0 52428784 52428784 0 0 Blob
3 1d3e1000 21221376 21219880 1496 0 Turbotx
This is a day-1 design issue, Juniper engineering team is working on the solution.
As a wokaround, customer can either reduce the static routing interface scale, or change to follow the Junos Subscriber-Management way to use dynamic interfaces for those end clients.