This KB explains a scenario where a vSRX may lose rollback configuration files after an unexpected KVM host shutdown, resulting in configuration recovery from the peer node. It also provides recommendations to reduce the risk of similar issues.
- Unexpected power loss or KVM host crash/reboot.
- vSRX boots up with missing, corrupted, or unavailable rollback configuration files.
- File system recovery messages may be seen during boot.
- In chassis cluster deployments, the affected node may recover configuration from the peer node after boot.
- Configuration may appear older than expected if the latest files were not successfully written to disk before the outage.
- Only one vSRX can be affected even when multiple VMs experience the same power event.
Cause:
- The issue is related to disk/file-system corruption caused by an unclean shutdown of the KVM host.
- When power is lost unexpectedly, pending disk writes may not complete successfully.
- During reboot, the file system performs recovery and any incomplete or corrupted files may be lost.
- If configuration or rollback files are affected, vSRX may not be able to read them after boot.
- In a chassis cluster, the affected node can automatically synchronize configuration from the peer node once communication is restored.
Why only one/specific vSRX could be affected:
- During an unexpected power loss, file corruption depends on what data is actively being written to disk at that specific moment.
- Different VMs and even different vSRX instances may be performing different disk operations when the interruption occurs.
- As a result, specific vSRX may experience configuration or rollback file corruption while other vSRX instances or VMs recover normally.
- The impact is determined by the timing of the disk write activity rather than the type of VM or application running on the system.
Recommendations:
- Investigate storage, file-system, and disk errors on the hypervisor with the virtualization or storage team.
- Ensure proper power protection and graceful shutdown procedures for virtualization hosts.
- Maintain regular configuration backups.
- Verify storage health and file-system integrity on the hypervisor.
- Where supported by the virtualization platform, use storage and cache settings that prioritize data integrity during unexpected power interruptions.
Example:
<driver name='qemu' type='qcow2' cache='writethrough'/>
Using writethrough cache mode may help reduce the risk of data loss from pending disk writes during an unexpected shutdown.