Description

When MYSQL DB is corrupted on Qfabric, it should be replaced with a backed up MYSQL DB.

This article explains how to replace a corrupted MYSQL DB using a backup DB on Qfabric.

Symptoms

If MySQL does not start due to DB corruption, perform the steps provided in the solution on the primary DG to restore the backup.

[root@dg0~]# cli
DBI connect('sfcdb;host=db','root',....) failed: can't connect to MySQL server on 'db' (111) at /opt/demo/mgd/bin/map_ip_to_id.pl line 13 sfc-cli: CLI can not start. Could not retrieve LF id  

Solution

  1. Rename MySQL script to disable auto-start: 

    [root@dg0 ~]# ls -lah /opt/dcf/scripts/*mysql-5.5.28
-rwxr-xr-x 1 root 13K Aug 28  2020 /opt/dcf/scripts/mysql-5.5.28*

[root@dg0]# date;mv /opt/dcf/scripts/mysql-5.5.28 /opt/dcf/scripts/0mysql-5.5.28 
Wed Apr  7 16:43:39 UTC 2021

[root@dg0 ~]# ls -lah /opt/dcf/scripts/*mysql-5.5.28
-rwxr-xr-x 1 root 13K Aug 28  2020 /opt/dcf/scripts/0mysql-5.5.28*

  2. Stop MySQL service:

    [root@dg0 ~]# date; /opt/dcf/scripts/0mysql-5.5.28 stop                       
    Wed Apr  7 16:44:50 UTC 2021
    Shutting down MySQL.... SUCCESS!  <-- MySQL service may not be running in an error situation so this may return an error.

  3. Remove MySQL data directory:

    [root@dg0 ~]# date; rm -rf /pbdata/usr/mysql-5.5.28/data/*
    Wed Apr  7 16:46:17 UTC 2021

  4. Restore MySQL data directory from backup:

    [root@dg0]# date; scp -rp root@dg1:/mysql-backup/data.tgz /var/tmp/
    Wed Apr  7 19:30:27 UTC 2021
    root@dg1's password: 
    data.tgz
     
    [root@dg0]# date; tar -xzvf /var/tmp/data.tgz -C /var/tmp 
                               
    [root@dg0]# cp -a -R /var/tmp/pbdata/usr/mysql-5.5.28/data/* /pbdata/usr/mysql-5.5.28/data/
                                
    Note : Wait 5 minutes after DB restore before going to the next step.

  5. Start MySQL service.

    [root@dg0]# date; /opt/dcf/scripts/0mysql-5.5.28 start
    Wed Apr  7 20:00:38 UTC 2021
    Starting MySQL.. SUCCESS! 

  6.  Rename MySQL script.

    [root@dg0 ]# date;mv /opt/dcf/scripts/0mysql-5.5.28 /opt/dcf/scripts/mysql-5.5.28
    Wed Apr  7 20:02:31 UTC 2021

  7. Check MySQL and Cluster services are running.

    [root@dg0 ~]# date; service mysql status
    Wed Apr  7 20:02:51 UTC 2021
    SUCCESS! MySQL running (9964)
    [root@dg0 ~]# date;clustat
    Wed Apr  7 20:03:24 UTC 2021
    Member Status: Quorate
    Member Name                             ID   Status
    ------ ----                             ---- ------
    dg0                                         1 Online, Local, rgmanager
    dg1                                         2 Online, rgmanager
    Service Name                   Owner (Last)                   State         
    ------- ----                   ----- ------                   -----         
    dcfservice:dcf_svc             dg0                            started  <-- Indicates MySQL is started       
    service:pbccif_svc0            dg0                            started       
    service:pbccif_svc1            dg1                            started       
    service:pbgfs_svc0             dg0                            started       
    service:pbgfs_svc1             dg1                            started       
    service:pblb_dhcp_svc0         dg0                            started       
    service:pblb_dhcp_svc1         dg1                            started       
    service:pbnfs_svc0             dg0                            started       
    service:pbnfs_svc1             dg1                            started      

  8. Delete MySQL backup data to conserve space:

    [root@dg0 ~]# date; rm -rf /var/tmp/pbdata*
    [root@dg0 ~]# rm  /var/tmp/data.tgz 
    rm: remove regular file `/var/tmp/data.tgz'? y
     
    [root@dg1 ~]# date; rm -rf /mysql-backup

Related Information