The upgrade from 7.5.0 UP7 to 7.5.0 UP8 version failed for the console. The primary console is rebooted after the patch was aborted. Post the reboot, the WebGUI is available however, the cluster state is unstable.

The /var/log/setup-x.x.x.x/patches.log reports the below errors.

[2024-05-27 19:51:18] [RHEL8MIGRATION] [Mon May 27 19:50:35 IST 2024] [ERROR] [LEAPP_UPGRADE] Upgrade failed. Please check the logs to see what needs to be fixed. (rc=1)

[2024-05-27 19:51:18] [RHEL8MIGRATION] [Mon May 27 19:50:35 IST 2024] [INFO] Performing cleanup ..

The /var/log/leapp/leapp-upgrade.log reports the below RPMs due to which the leapp upgrade fails and in turn the patch fails.

2024-05-27 19:48:50.117 DEBUG  PID: 31445 leapp.workflow.Download.dnf_package_download: RPM: warning: Generating 6 missing index(es), please wait...

2024-05-27 19:48:50.119 DEBUG  PID: 31445 leapp.workflow.Download.dnf_package_download: The downloaded packages were saved in cache until the next successful transaction.

2024-05-27 19:48:50.120 DEBUG  PID: 31445 leapp.workflow.Download.dnf_package_download: You can remove cached packages by executing 'dnf clean packages'.

2024-05-27 19:48:50.159 DEBUG  PID: 31445 leapp.workflow.Download.dnf_package_download: Error: Transaction test error:

2024-05-27 19:48:50.161 DEBUG  PID: 31445 leapp.workflow.Download.dnf_package_download:  file /usr/sbin/pidof from install of procps-ng-3.3.15-14.el8.x86_64 conflicts with file from package sysvinit-tools-2.88-14.dsf.el7.x86_64

2024-05-27 19:48:50.163 DEBUG  PID: 31445 leapp.workflow.Download.dnf_package_download:  file /usr/bin/last from install of util-linux-2.32.1-41.el8.x86_64 conflicts with file from package sysvinit-tools-2.88-14.dsf.el7.x86_64

2024-05-27 19:48:50.164 DEBUG  PID: 31445 leapp.workflow.Download.dnf_package_download:  file /usr/bin/mesg from install of util-linux-2.32.1-41.el8.x86_64 conflicts with file from package sysvinit-tools-2.88-14.dsf.el7.x86_64

2024-05-27 19:48:50.166 DEBUG  PID: 31445 leapp.workflow.Download.dnf_package_download:  file /usr/bin/wall from install of util-linux-2.32.1-41.el8.x86_64 conflicts with file from package sysvinit-tools-2.88-14.dsf.el7.x86_64

2024-05-27 19:48:50.167 DEBUG  PID: 31445 leapp.workflow.Download.dnf_package_download:  file /usr/share/man/man1/last.1.gz from install of util-linux-2.32.1-41.el8.x86_64 conflicts with file from package sysvinit-tools-2.88-14.dsf.el7.x86_64

2024-05-27 19:48:50.168 DEBUG  PID: 31445 leapp.workflow.Download.dnf_package_download:  file /usr/share/man/man1/mesg.1.gz from install of util-linux-2.32.1-41.el8.x86_64 conflicts with file from package sysvinit-tools-2.88-14.dsf.el7.x86_64

2024-05-27 19:48:50.169 DEBUG  PID: 31445 leapp.workflow.Download.dnf_package_download:  file /usr/share/man/man1/wall.1.gz from install of util-linux-2.32.1-41.el8.x86_64 conflicts with file from package sysvinit-tools-2.88-14.dsf.el7.x86_64

2024-05-27 19:48:50.171 DEBUG  PID: 31445 leapp.workflow.Download.dnf_package_download:  file /usr/bin/pyinotify from install of python3-inotify-0.9.6-13.el8.noarch conflicts with file from package python-inotify-0.9.4-4.el7.noarch

2024-05-27 19:48:50.172 DEBUG  PID: 31445 leapp.workflow.Download.dnf_package_download:  file /usr/lib/python2.7/site-packages/backports/__init__.pyc from install of python2-backports-1.0-16.module+el8.4.0+9193+f3daf6ef.x86_64 conflicts with file from package python-backports-1.0-8.el7.x86_64

2024-05-27 19:48:50.173 DEBUG  PID: 31445 leapp.workflow.Download.dnf_package_download:  file /usr/lib/python2.7/site-packages/backports/__init__.pyo from install of python2-backports-1.0-16.module+el8.4.0+9193+f3daf6ef.x86_64 conflicts with file from package python-backports-1.0-8.el7.x86_64

2024-05-27 19:48:50.174 DEBUG  PID: 31445 leapp.workflow.Download.dnf_package_download:  file /usr/lib64/python2.7/site-packages/backports/__init__.pyc from install of python2-backports-1.0-16.module+el8.4.0+9193+f3daf6ef.x86_64 conflicts with file from package python-backports-1.0-8.el7.x86_64

2024-05-27 19:48:50.176 DEBUG  PID: 31445 leapp.workflow.Download.dnf_package_download:  file /usr/lib64/python2.7/site-packages/backports/__init__.pyo from install of python2-backports-1.0-16.module+el8.4.0+9193+f3daf6ef.x86_64 conflicts with file from package python-backports-1.0-8.el7.x86_64

2024-05-27 19:48:50.177 DEBUG  PID: 31445 leapp.workflow.Download.dnf_package_download:  file /usr/share/man/man8/yum-rhn-plugin.8.gz from install of dnf-plugin-spacewalk-2.8.5-11.module+el8.1.0+3455+3ddf2832.noarch conflicts with file from package yum-rhn-plugin-2.0.1-10.el7.noarch

Red Hat Enterprise Linux ships with Leapp as the supported upgrade management tool which delivers a single path to upgrading to the next major version of Red Hat Enterprise Linux. Leapp is included with Red Hat Enterprise Linux 7 and Red Hat Enterprise Linux 8, allowing you to upgrade from Red Hat Enterprise Linux 7.9 to Red Hat Enterprise Linux 8.

If the leapp upgrade during the patch process fails due to the errors mentioned in the symptoms, please break the HA between the primary and secondary console by running the command on the primary console : /opt/qradar/bin/ha_setup.sh --uninstall

Note : Running the uninstall command will cause the primary console to reboot to remove the HA related configuration from the appliance. The above steps need to be performed in the presence of a JTAC engineer if you are unsure about the current status of HA between the primary and secondary JSA console.

Once the primary console has rebooted and the SSH access to the appliance is available, run the below commands to remove the RPMs which are causing a problem during the leapp upgrade.

RPMs removal:
--------------------------------------------------------------------------------------------------------------------------
rpm -e python-chardet-2.2.1-3.el7.noarch --nodeps
rpm -e yum-rhn-plugin-2.0.1-10.el7.noarch --nodeps
rpm -e python-inotify-0.9.4-4.el7.noarch --nodeps
rpm -e python-backports-1.0-8.el7.x86_64 --nodeps
rpm -e sysvinit-tools-2.88-14.dsf.el7.x86_64 --nodeps
rpm -e python-ethtool-0.8-8.el7.x86_64 --nodeps
rpm -e python-linux-procfs-0.4.11-4.el7.noarch --nodeps
--------------------------------------------------------------------------------------------------------------------------

Manual RHEL migration:
--------------------------------------------------------------------------------------------------------------------------
systemctl stop hostcontext tomcat hostservices
mkdir /var/log/qradar_patch/
mkdir /leapp_scratch
ln -s /leapp_scratch /var/lib/leapp
mv /storetmp/<UP8>.sfs /var/log/qradar_patch/
mkdir -p /media/updates
mount -o loop /var/log/qradar_patch/<UP8>.sfs /media/updates/
cp /media/updates/tmp/migrate_to_rhel8/migrate-to-rhel8-bundle-1.0.31.bin /root/
scp /root/migrate-to-rhel8-bundle-1.0.31.bin /usr/bin/migrate-to-rhel8-bundle
chmod +x /usr/bin/migrate-to-rhel8-bundle
touch /tmp/patch_skip_patch_service
touch /tmp/patch_qroc_system
yum install /media/updates/repo/python2-chardet-3.0.4-10.module+el8.1.0+3111+de3f2d8e.noarch.rpm
migrate-to-rhel8-bundle performUpgrade
-------------------------------------------------------------------------------------------------------------------------

Once the above process completes successfully, reboot the system.

--------------------------------------------------------------------------------------------------------------------------
reboot
--------------------------------------------------------------------------------------------------------------------------


Post the reboot, you can proceed with the rest of the patch to 7.5.0 UP8 by running the following commands:

JSA Patch upgrade:
--------------------------------------------------------------------------------------------------------------------------
mount -o loop /var/log/qradar_patch/<UP8>.sfs /media/updates/
touch /tmp/patch.skip_undeployed_check
touch /var/log/leapp/patch_migration_pretest_passed
/media/updates/installer
--------------------------------------------------------------------------------------------------------------------------

Once this is done for both the servers,  please re-add the servers in HA.

Upgrade failing for Primary Managed hosts: 

In case of managed hosts in the deployment, the commands related to RPM removal and Manual RHEL migration need to be run on all the managed hosts before proceeding with the upgrade, otherwise those managed hosts will fail with the same error.

Upgrade failing for Secondary Managed hosts:

If you have High Availability configured for the managed hosts, you will need to run the below commands to correct the HA state between the primary and secondary managed hosts. The bad HA state between the primary and secondary managed hosts
is due to the post migration RHEL scripts not being run on the secondary hosts. In order to run the post migration RHEL scripts, please run the below commands on the primary managed host.

--------------------------------------------------------------------------------------------------------------------------
systemctl stop ha_manager
systemctl stop drbd
/media/updates/supplementary_scripts/ha_patch_helper.sh --post_rhel8_setup
--------------------------------------------------------------------------------------------------------------------------
 

Once the above commands have been successfully executed, the HA will go back to a stable state and we need to wait for the DRBD synchronization to finish between the primary and secondary hosts and post that, we can proceed with the
JSA patch upgrade commands mentioned above. 

NOTE : If you are unsure about running the above commands, please reach out to JTAC so that they can assist you further.

After upgrading all the hosts along with the console in the deployment, you can perform a reinstall of 7.5.0 UP8 on the secondary console and add it back to the HA with the primary console.