When an ACX device is configured to serve as a dhcp-relay agent, memory leaks may happen during the dhcpd process, which can result in exhaustion of resident memory, and utilization and possible exhaustion of swap memory as well.

This article gives the reason for the memory leak to occur on these devices and also the solution using which the memory leak can be prevented.

The following messages may be seen in the logs on these devices when the memory leak occurs:

Oct 11 12:22:03.918  ACX-LAB /kernel: Process (75055,jdhcpd) has exceeded 85% of RLIMIT_DATA: used 523504 KB Max 524288 KB
Oct 11 12:27:04.020  ACX-LAB /kernel: Process (75055,jdhcpd) has exceeded 85% of RLIMIT_DATA: used 524148 KB Max 524288 KB
Oct 11 12:28:18.856  ACX-LAB /kernel: Process (75055,jdhcpd) attempted to exceed RLIMIT_DATA: attempted 524304 KB Max 524288 KB
Oct 11 12:33:18.898  ACX-LAB /kernel: Process (75055,jdhcpd) attempted to exceed RLIMIT_DATA: attempted 524296 KB Max 524288 KB

When the show system process extensive command output is examined, the resident memory (RES) or the memory utilized by the process will be seen to steadily increase over time until the utilization reaches 100%, at which point swap memory would be utilized.

The following sample outputs for the show system process extensive command (entries shown only for jdhcpd for brevity) show the steady increase in RES memory for the dhcpd process.

PID USERNAME         PRI NICE   SIZE    RES       STATE    TIME   WCPU COMMAND
2984 root               1  97    0   152M 73380K select  14:52  2.34% jdhcpd
2984 root               1  99    0   152M 73380K select  14:52  2.39% jdhcpd
2984 root               1  99    0   152M 73396K RUN     14:52  2.20% jdhcpd
2984 root               1  99    0   152M 73400K select  14:52  2.05% jdhcpd
2984 root               1 104    0   153M 74376K select  15:17  4.25% jdhcpd
2984 root               1  96    0   153M 75396K RUN     15:17  4.25% jdhcpd
2984 root               1  96    0   153M 77356K select  15:17  4.05% jdhcpd
2984 root               1  96    0   153M 79200K select  15:18  4.05% jdhcpd
2984 root               1  96    0   153M 81224K select  15:18  4.35% jdhcpd
2984 root               1  96    0   153M 84436K select  15:18  4.35% jdhcpd

Subsequently, if the router runs out of swap memory, the device may become unreachable and stop forwarding traffic. At this point, the following messages may be seen on the console connection to the device:

vnode_pager_putpages: residual I/O 65536 at 36
vnode_pager_putpages: I/O error 28
vnode_pager_putpages: residual I/O 65536 at 36
vnode_pager_putpages: I/O error 28
vnode_pager_putpages: residual I/O 65536 at 36
vnode_pager_putpages: I/O error 28
vnode_pager_putpages: residual I/O 65536 at 36
vnode_pager_putpages: I/O error 28

Typically, the relay agent maintains a DHCP relay binding database that has a mapping of the client MAC address with the IP address assigned to the MAC address by DHCP, based on the DHCP transactions between the DHCP clients and the DHCP server via the DHCP relay agent.

On the ACX platform, each time a new binding happens or an existing binding breaks, several write operations are performed into NAND flash. In scaled networks, frequent writes to NAND flash would result in memory leaks and in the reduction of life of NAND flash.

However, when the forward-only knob is used, the DHCP relay functions are carried out but the database containing the MAC address to IP address binding is not maintained, thus preventing the frequent NAND writes.

Note: This database is maintained by the DHCP server in either case.

This is a product limitation on ACX devices.

To avoid the issue, configure the forward-only knob, which prevents the frequent NAND writes and the resulting memory loss. For configuring the forward-only knob, refer to forward-only (DHCP Relay Agent) .