If a Junos Space deployment has been running on the same system with only upgrades and no rebuild for five years , the MySQL certificate has likely expired, preventing replication from working and reporting error 2026 (database out of sync) in the show slave status output.
show slave status
Note: This article applies only for the error 2026 that is found in the show slave status output.
The article explains how to check whether the MySQL certificate has expired, regenerate the certificates, and restart the services.
After seeing "Out Of Sync" for the database node, perform the following:
Use the following command on the database nodes:
mysql -ujboss -p$(grep mysql.jboss /etc/sysconfig/JunosSpace/pwd | awk -F= '{print $2}') -e "show slave status\G show master status\G"
Note: This article applies only for the following error:
Last_IO_Errno: 2026 Last_IO_Error: error connecting to master 'repUser@REMOVED:3306' - retry-time: 10 retries: 5
To confirm whether the certificate is the problem, use the following command on all database nodes:
openssl x509 -text -inform PEM -in /var/chroot/mysql/etc/pki/mysql/ca-cert.pem | grep After
openssl x509 -text -inform PEM -in /var/chroot/mysql/etc/pki/mysql/server-cert.pem | grep After
Output example: (This date should have passed.)
Not After : May 19 20:03:55 2019 GMT
If this date is NOT in the past. This KB will NOT correct the problem.
The MySQL certificates are generated for a period of five years. If your Junos Space system is upgraded across multiple versions without being rebuilt, the certificate will expire and go out of sync in the UI.
Note: This process should be followed with JTAC assistance. Note: Do not allow the VIP to move. Ensure VIP has not moved nodes since this error started appearing
Stop these processes on all Junos Space nodes by using the following commands:
service jmp-watchdog stop service jboss stop service jboss-dc stop service mysql stop
Note: jboss-dc is active only on the VIP node, but the command will not do any harm.
jboss-dc
Confirm whether JBoss has been turned off.
service jboss status service mysql status
The output should show a "not running" or stopped result.
Make a note of the current expiration and file information to ensure that they have updated (all Junos Space database nodes).
openssl x509 -text -inform PEM -in /var/chroot/mysql/etc/pki/mysql/ca-cert.pem | grep After openssl x509 -text -inform PEM -in /var/chroot/mysql/etc/pki/mysql/server-cert.pem | grep After ls -l /var/chroot/mysql/etc/pki/mysql/
Create a new root certificate [(only the first node (VIP)].
/var/www/cgi-bin/generateCACert.sh
Confirm that new ca-cert.pem and ca-key.pem files have been generated. Compare to the output from the earlier step [only the first node (VIP)].
ca-cert.pem
ca-key.pem
ls -l /var/chroot/mysql/etc/pki/mysql/
Transfer the keypair to the other database node:
From the VIP node, transfer the file as shown.
scp /var/chroot/mysql/etc/pki/mysql/ca-cert.pem admin@ <other mysql node> :/home/admin/ scp /var/chroot/mysql/etc/pki/mysql/ca-key.pem admin@ <other mysql node> :/home/admin/
On the destination database node, move the file into place and fix permissions.
mv /home/admin/ca-cert.pem /home/admin/ca-key.pem /var/chroot/mysql/etc/pki/mysql/ chown root:mysql /var/chroot/mysql/etc/pki/mysql/ca-cert.pem /var/chroot/mysql/etc/pki/mysql/ca-key.pem chmod 640 /var/chroot/mysql/etc/pki/mysql/ca-cert.pem /var/chroot/mysql/etc/pki/mysql/ca-key.pem
Confirm that the new files ca-cert.pem and ca-key.pem exist on both nodes, and have the same size/timestamp.
Create new server keys based on the root certificate on both database nodes:
cd /var/www/cgi-bin/secure perl setupDbSSL operation=generateServerCert ls -l /var/chroot/mysql/etc/pki/mysql/
Check the expiration time on the certificates on all Junos Space database nodes:
openssl x509 -text -inform PEM -in /var/chroot/mysql/etc/pki/mysql/ca-cert.pem | grep After openssl x509 -text -inform PEM -in /var/chroot/mysql/etc/pki/mysql/server-cert.pem | grep After
The dates should now be updated on all nodes. ca-cert will match all database nodes. server-cert will be different between both nodes.
ca-cert
server-cert
Now start services on the VIP node.
service jmp-watchdog start
Note: Watchdog starts the other processes.
Note: Additional database replication errors may be encountered after clearing this specific error. Database backup/restore, or other manual database resync processes will likely be needed to ensure proper replication. Database errors may not be seen for a few days until a conflicting data change has been made.”
After you can log in to the WebUI from the VIP node, start all other Junos Space nodes.
Check MySQL status on both nodes to confirm that the error is no longer seen (can take some time after startup of the second node for the message to clear). IF the error has been occurring for more than 24 hrs, perform a database backup and restore. To completely sync the data. If the error has only existed for 1 day or less, you can wait for the system to catch up on replication. (non-replicated bin-files are likely lost if more than 24hrs)
2020-02-13: Added some clarification/warnings notes.