May 21 08:40:45.253 mgd[93657]: %INTERACT-4-UI_EPHEMERAL_COMMIT: User 'root' has requested commit on 'junos-analytics' ephemeral database May 21 08:40:46.149 jsd[90389]: %DAEMON-3-UI_CHILD_SIGNALED: Child received signal: PID 93666, signal Terminated, command='/usr/libexec/jade' May 21 08:40:46.332 agentd[93661]: %DAEMON-3-UI_DBASE_OPEN_FAILED: Database open failed for file '/var/run/db/juniper.data+': No such file or directory May 21 08:40:46.357 na-grpcd[91849]: %DAEMON-3-NA_GRPCD_CONFIG_EDIT_FAILURE: Ephemeral DB Edit config: error_code=1, error_message='Invalid Request'
>> Due to configuration lock, user was unable to commit anything on the router. User decided to restart MGD and JSD for recovery.>> After restarting of MGD and JSD, it corrupted the juniper.data and ephemeral file in /var/run/db location of RE0 and after reffect of this was JTI unavailability.>> This broke JTI and their customer were unable to get the metric information.>> After going through the logs, we could see JSD was rebooted twice
May 21 06:39:25.272 mgd[64099]: %INTERACT-4-UI_RESTART_EVENT: User 'lab-user' restarting daemon 'JET Services Daemon' immediately May 21 06:39:25.687 na-grpcd[30843]: %DAEMON-3-NA_GRPCD_CONFIG_EDIT_FAILURE: Ephemeral DB Edit config: error_code=4, error_message='Invalid Configuration' May 21 06:43:24.678 mgd[65396]: %INTERACT-4-UI_RESTART_EVENT: User 'lab-user' restarting daemon 'JET Services Daemon' immediately
>> Below logs are seen in na-grpcd continuously.
May 21 08:57:28 #8578# <configuration-json>{"configuration":{"services":{"analytics":{"export-profile":[{"name":"export_1000","transport":"grpc"},{"name":"export_1001","transport":"grpc"},{"name":"export_1002","transport":"grpc"}],"sensor":[{"name":"sensor_1000","export-name":"export_1000","life-time":"long-lived","reporting-rate":"6","resource":"/junos/system/cmerror/configuration/","subscription-id":"1000"},{"name":"sensor_1001","export-name":"export_1001","life-time":"long-lived","reporting-rate":"6","resource":"/junos/system/cmerror/counters/","subscription-id":"1001"},{"name":"sensor_1002","export-name":"export_1002","life-time":"long-lived","reporting-rate":"6","resource":"/components/component/properties/property[name='state']/","subscription-id":"1002"}]}}}}</configuration-json> May 21 08:57:28 #8579# Closing the isync pipe for ID =2 May 21 08:57:28 #8580# Closed the isync pipe for ID =2 May 21 08:57:28 #8581# Destroyed the isync ctx for ID =2 May 21 08:57:30 #8582# MGD command failed (cmd = 0). Error code: 1. Error message: Opening configuration database: Could not open configuration database ; configuration check-out failed ; May 21 08:57:30 #8583# ID = 1: Commit in systemAdd took 2294.123668 milliseconds. May 21 08:57:30 #8584# ConsolidatorSystemHandle create failed for 1000 May 21 08:57:30 #8585# Unassigning Internal Id 1000 May 21 08:57:30 #8586# Deleted Internal Id = 1000 May 21 08:57:30 #8587# Failed to add/update internal subscription 1000 ref count = 0 May 21 08:57:30 #8588# ConsolidatorSystemHandle create failed for 1001 May 21 08:57:30 #8589# Unassigning Internal Id 1001 May 21 08:57:30 #8590# Deleted Internal Id = 1001 May 21 08:57:30 #8591# Failed to add/update internal subscription 1001 ref count = 0 May 21 08:57:30 #8592# ConsolidatorSystemHandle create failed for 1002 May 21 08:57:30 #8593# Unassigning Internal Id 1002 May 21 08:57:30 #8594# Deleted Internal Id = 1002 May 21 08:57:30 #8595# Failed to add/update internal subscription 1002 ref count = 0 May 21 08:57:30 #8596# Failed to add Request ID: 1, system_accepted_request paths size: 0
>> In messages file we could see below logs continuously populated. We can see there is some commit request coming in but later it is failing with "%DAEMON-3-NA_GRPCD_CONFIG_EDIT_FAILURE: Ephemeral DB Edit config: error_code=1, error_message='Invalid Request'".
>> After going through the file in /var/run/db on RE0 and RE1, we could see below difference.
++ RE0 is missing with below
-rw-r----- 1 root config 34078720 May 21 11:41 juniper.data+ -rw-r----- 1 root config 1572864 May 11 10:48 juniper.eph -rw-r----- 1 root config 1572864 May 11 10:48 juniper.eph.data -rw-r----- 1 root config 0 May 11 10:48 juniper.eph.lck
++ RE0 is having this extra
-rw-r----- 1 root config 1572864 May 21 11:32 juniper.eph_junos-analytics -rw-r----- 1 root config 0 Jan 24 07:08 .lock_rollback_juniper.db
++++++ Below is the procedure used for recover ++++++
1. Perform mastership switch over to RE1 2. After switch over is done and both devices are negotiating properly, we can reboot RE0 3. After RE0 is UP and normalized, we can check the path of RE0 /var/run/db if we have juniper.data+ and other emphemeral files. Once JTAC confirms, you can switchover to RE0 back.
>> The user performed step 1 and JTI services were back to normal when RE1 became master.>> After RE0 reboot, we were able to confirm RE0 was healthy and all the missing files were back again on RE0.>> Later mastership was changed to RE0 and JTI / other services were working fine.