Alert Type

PSN - Product Support Notification

Product Affected

All JUNOS platforms using "protocol mpls sensor-based-stat" and "ldp-tunneling"

Alert Description

When using "set protocol mpls sensor-based-stats" and "ldp-tunneling", every time an LSP is added or changed, part of its data structure is not freed which will cause the resources to be exhausted. The relevant data structure is called "Generic config" or "gencfg". Junos software imposes a maximum limit of 2000000 entries for the gencfg data structure. Once the system reaches the limit, the system can no longer add a new LSP to the PFE causing transit packets being dropped.
The following are syslog entries indicating that the routing engine is running out of resources

<syslog>
Jul 1 11:32:00 router1 rpd[2595]: %DAEMON-3-RPD_KRT_Q_RETRIES: Stats Id Group ADD: Cannot allocate memory
Jul 1 11:32:00 router1 /kernel: %KERN-3: gencfg_peer_update Cannot retrieve gencfg node from ifstate
Jul 1 11:32:00 router1 last message repeated 3931 times
Jul 1 11:32:00 router1 agentd[2636]: %DAEMON-3: failed to add sensor 281475513713550

To check if the number of "Generic config" entries in used, you can use the following command "ifsmon -Id" - these are a capital "i" and a lowercase "d" in the argument of "ifsmon". In this example, the number of "Generic config" entries is at the maximum and no more entry can be allocated. 

cli> start shell user root
# ifsmon -Id | grep "Generic config"

Generic config : 2000027 1​

Rate of accumulation of the Generic config entries can be determined by collecting the information multiple times. During a router's normal operation, the number of "Generic config" counters can vary depending on the number of LSPs, LSP activities, and how often configuration changed. However, the number of entries should not be trending upward.

In a large network, it can take approximately 30 weeks for the "Generic config" entries to reach its limit.

Solution

If you are using "set protocol mpls sensor-based-stats" and "ldp-tunneling" configuration, you need to monitor the "Generic config" resource.

If there is a tendency for the "Generic config" to accumulate, we recommend that you take corrective actions as follow:
1. Determine the amount of "Generic config" being used by the system and the rate of accumulation
2. If the number of "Generic config" is more than 1000000 (one million entries) and the rate of accumulation is positive

Disable the "sensor-based-stats" configuration to stop the accumulation of "Generic config" entries and plan for a maintenance window to restart RPD to reclaim the entries.

This issue is being tracked via PR1447170. For latest update and availability of the fix, please consult PR1447170 at https://prsearch.juniper.net/PR1447170

Modification History

First publication date 2019-08-02

Related Information