Router-re0> configure exclusive error: configuration database modified -------------------------->>> Problem logs # commit confirmed will be rolled back in less than 1 minute Router-re0# show | compare # commit confirmed will be rolled back in less than 1 minute {master}[edit] Router-re0# commit and-quit re0: configuration check succeeds re1: commit complete error: session failure: unexpected termination -------------------------->>> Problem logs error: remote side unexpectedly closed connection -------------------------->>> Problem logs Router-re0> show system rollback compare 1 2 error: session failure: unexpected termination -------------------------->>> Problem logs error: remote side unexpectedly closed connection -------------------------->>> Problem logs
>> Please go through the RSI and Varlogs from the problem time.>> After above, "commit and-quit" and "show system rollback compare 1 2" both terminated the ssh session forcing the user to log out.>> We can see multiple cores on the router for commitd and mgd.>> Investigated the varlogs but did not find anything that could point to the cause of the issue.
Router-re0> show system core-dumps no-forwarding /var/crash/*core*: No such file or directory -rw------- 1 root wheel 20450177 Jul 8 14:52 /var/tmp/commitd.core.0.gz -rw------- 1 root wheel 20365906 Jul 8 15:01 /var/tmp/commitd.core.1.gz -rw------- 1 root wheel 20402478 Jul 8 15:24 /var/tmp/commitd.core.2.gz -rw-r--r-- 1 root wheel 6843360 Apr 3 2023 /var/tmp/fpc-3.J-UKERN.core.0.tgz -rw------- 1 root wheel 13732975 Jul 8 15:06 /var/tmp/mgd.core.0.gz -rw------- 1 root wheel 13780381 Jul 8 15:28 /var/tmp/mgd.core.1.gz -rw------- 1 root wheel 13778579 Jul 8 17:01 /var/tmp/mgd.core.2.gz
>> I also went through the BT of one core of commitd and mgd.>> I could see Error on commitd core: <error: Cannot access memory at address 0xf>
Below is the understanding from the available logs.
The customer confirmed they saw the issue after committing changes related to ACL and when we checked the same with Engineering, they confirmed that in some rare cases, some commits may cause database corruption.In the problem scenario, Juniper.db is not in a sane state. The core that got generated is not the culprit; instead, it is a victim core, and collateral of some previous commit in which the sanity of the database went wrong.Because of this problem, mgd is trying to access a corrupted memory location, which is creating a problem and throwing an error.This is a rare condition that may be seen."mgd -I" can be used as a workaround as it rebuilds the juniper.db again.A memory location that is being accessed or referenced by multiple processes, and during that time configurations are being added or deleted, can cause memory corruption, though this is not possible for a specific commit and is more of a timing issue that happens in a very rare case.This router might have multiple configurations being pushed, polling of data, route processing, and multiple other things happening on the router, but you would not see similar issues happening in the past. If there had been any specific commit, then you would have seen the problem more often.We consider this situation as data corruption and recover it with "mgd -I".