We observed a protocol restart with rpd crash, after we started seeing dynamic db size exceeded alarms were seen.
mgd[95509]: UI_DBASE_EXTEND_FAILED: Unable to extend configuration database file '/var/run/db/juniper.dyn' to size 0x51c00000: size exceeds limit
show system core-dumps
-rw-rw---- 1 root wheel 1492101219 Jul 5 00:53 /var/tmp/rpd.core-tarball.4.tgz
The dynamic database has reached its maximum size limit.
As a result load operations on this database is failing and the database has gone to an inconsistent state.
The same database is being read by RPD which causes this crash because the database is in bad state.
> The dynamic config db uses memory fragmentation while add/delete actions keep happening.
> When we delete a set of config lines from dyn DB & then add a set of new config lines with higher size, the earlier freed up memory will not accommodate the new set there.
> Hence it uses new memory fragment, there by creating a void or gap in memory & the total memory looks to be utilised, including these gaps.
> This continues with further when we keep continuing add/delete in same config dyn db.
Here customer needs to check the scale of configs in the dynamic database and try to reduce the size according to the database size limit.
In short:
> Customer appears to have a scaled config in dynamic database.
> Please suggest to reduce the scale of config in dynamic database and consider alternatives.
file list /var/rundb/ detail
<snip>
-rw-r----- 1 root config 1365770240 Jul 5 00:51 juniper.dyn
Below is the workaround, which can be used temporarily to reduce the size.
- Ensure that the dynamic-db config is saved to a file (this is very much important).
- Delete the juniper.dyn file in /var/run/db.
- Load the config back to the dynamic database.
CLI Steps:
edit dynamic
show | save /var/tmp/dyn_bkp.txt
exit
(ensure the dyn_bkp.txt file is generated & it has config)
file list /var/tmp/ | grep dyn_bkp.txt
file delete /var/rundb/juniper.dyn
load merge /var/tmp/dyn_bkp.txt
commit
The suggested option is to optimise the CLI config lines.
If this is not possible, they can explore below option & this needs to be discussed with accounts team, by customer.
The dynamic config file has only one instance, which can extend max upto 1.3GB in size.
Other option available to customer is go for ephemeral db, which can have upto 8 instances.