#3288 Issue open: Can't Restore ReaR backup on RHEL9 (ISO image got corrupted)¶
Labels: support / question, ready-to-close?
msharif4 opened issue at 2024-07-20 16:13:¶
Hi ReaR team,
I installed ReaR in a rhel9 server and configured it. I took a full
backup last week, and now the server is not working after patching it. I
am trying to restore the server from the ReaR backup. However, I am
getting an error "failed to load ldlinux.c32". The backup was stored
in an NFS share, and I am using the Iventoy tool to boot the system from
the network. I have both the iso file and the zip file with full backup.
Did misconfigure the /etc/local.conf file? Now, can I restore the backup
when the system is only starting on rescue mode? I am stressed out and
desperate; please help me out. Below is the configuration file content,
and the second one is the backup directory files list.
OUTPUT_URL=nfs://192.168.1.118/mnt/backup/rhel
BACKUP=NETFS
BACKUP_URL=nfs://192.168.1.118/mnt/backup/rhel
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/media' '/var/tmp' '/var/crash')
#NETFS_KEEP_OLD_BACKUP_COPY=y
BACKUP_TYPE=incremental
FULLBACKUPDAY=( Sun )
[root@pxe iso]# ll rhel-server-02/
total 4135152
-rw-------. 1 root root 3592429931 Jul 19 18:24 2024-07-19-0000-F.tar.gz
-rw-------. 1 root root 9372601 Jul 19 18:24 backup.log
-rw-------. 1 root root 202 Jul 19 18:24 README
-rw-------. 1 root root 631203840 Jul 19 18:24 rear-rhel-server-02.iso
-rw-------. 1 root root 169446 Jul 19 18:24 rear-rhel-server-02.log
-rw-------. 1 root root 0 Jul 19 18:24 selinux.autorelabel
-rw-------. 1 root root 287 Jul 19 18:24 VERSION
I appreciate your help.
abbbi commented at 2024-07-20 17:03:¶
i dont know about iventoy, you may consider booting the iso in a different way. I guess the iso image thats been created may not be compatible to PXE boot.
To be able to boot the generated ISO Image via PXE you would have
neededOUTPUT=PXE in your configuration file (which is not present,
see:
https://relax-and-recover.org/rear-user-guide/basics/configuration.html)
The generated ISO image may only boot via USB or using an emulated ISO image forward via the servers remote console (or other ways to attach the iso image virtually, if not possible on site)
msharif4 commented at 2024-07-20 18:32:¶
I included OUTPUT=PXE and ran the command rear -v mkbackup, and it ran successfully. However, it didn't include an iso file. Just curious that without an iso, how can we restore the backup through PXE? Updated /etc/rear/local.conf file contents and backup file's content. Is there anything else that needs to be done?
OUTPUT=PXE
OUTPUT_URL=nfs://192.168.1.118/mnt/backup/rhel
BACKUP=NETFS
BACKUP_URL=nfs://192.168.1.118/mnt/backup/rhel
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/media' '/var/tmp' '/var/crash')
#NETFS_KEEP_OLD_BACKUP_COPY=y
BACKUP_TYPE=incremental
FULLBACKUPDAY=( Sun )
root@backup:/mnt/backup/rhel-server-03# ll
-rw------- 1 root root 1675321694 Jul 20 11:00 2024-07-20-1053-F.tar.gz
-rw------- 1 root root 5005087 Jul 20 11:00 backup.log
-rw------- 1 root root 516 Jul 20 10:56 README
-rw------- 1 root root 489 Jul 20 10:56 rear-rhel-server-03
-rw------- 1 root root 155071 Jul 20 10:56 rear-rhel-server-03.log
-rw------- 1 root root 0 Jul 20 11:00 selinux.autorelabel
-r-------- 1 root root 615039052 Jul 20 10:56 rhel-server-03.initrd.cgz
-r-------- 1 root root 13609080 Jul 20 10:56 rhel-server-03.kernel
-r-------- 1 root root 282 Jul 20 10:56 rhel-server-03.message
-rw------- 1 root root 282 Jul 20 10:56 VERSION
Thanks,
msharif4 commented at 2024-07-20 19:37:¶
It didn't allow me to attach it as an attachment so it got uploaded to
the
Google drive. It's a virtual machine that created Linux KVM. Please let
me
know if anything else is needed. Thanks
Edit: I removed the ISO link; I wasn't sure if it's publicly
available.
On Sat, Jul 20, 2024 at 10:04 AM Michael Ablassmeier <
***@***.***> wrote:
more info required.
you may want to share the iso image youve created. Could it be that the
system in question is an UEFI system and youre attempting to boot via
regular BIOS and SYSLINUX while the iso image is an UEFI bootable?—
Reply to this email directly, view it on GitHub
https://github.com/rear/rear/issues/3288#issuecomment-2241212973, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/BGF7YZTTFMVFHQWV37V2UZLZNKKABAVCNFSM6AAAAABLGBSWSGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBRGIYTEOJXGM
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
abbbi commented at 2024-07-20 20:32:¶
I included OUTPUT=PXE
yes, in this situation an initrd an kernel image are stored with the backups instead of an ISO image.
-r-------- 1 root root 615039052 Jul 20 10:56 rhel-server-03.initrd.cgz
-r-------- 1 root root 13609080 Jul 20 10:56 rhel-server-03.kernel
which can then be booted via PXE, depending on the environment you need
to place these files on an PXE boot
server which serves these via TFTP to boot.
The main problem however is why the generated iso image does not boot.
abbbi commented at 2024-07-20 20:40:¶
rear-bitwarden.iso
i loop mounted the iso image on my system and it seems like most files within the generated iso image are corrupted. The iso images contents dont look sane to me. So either there has been a problem during creation of the iso image or it has been corrupted by a third party (like storage, even some anti-virus software are known to alter files they think they are suspicious to them).
The syslinux files and especially the file which cant be loaded during boot are usually ELF executable files:
file /usr/lib/syslinux/modules/bios/ldlinux.c32
/usr/lib/syslinux/modules/bios/ldlinux.c32: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, stripped
but within your iso image most of them are only "data" files containing basically only zeroes, even the kernel and initrd:
/mnt/isolinux/
hexdump -C kernel
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00cf9870 00 00 00 00 00 00 00 00 |........|
some files are OK, but most of them are corrupt:
mount -o loop rear-bitwarden.iso /mnt
file /mnt/isolinux/*
/mnt/isolinux/boot.cat: data
/mnt/isolinux/chain.c32: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, stripped
/mnt/isolinux/hdt.c32: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, stripped
/mnt/isolinux/initrd.cgz: data
/mnt/isolinux/isolinux.bin: data
/mnt/isolinux/isolinux.cfg: data
/mnt/isolinux/kernel: data
/mnt/isolinux/ldlinux.c32: data
/mnt/isolinux/libcom32.c32: data
/mnt/isolinux/libgpl.c32: data
/mnt/isolinux/libmenu.c32: data
/mnt/isolinux/libutil.c32: data
/mnt/isolinux/menu.c32: data
/mnt/isolinux/message: data
/mnt/isolinux/pci.ids: data
/mnt/isolinux/poweroff.c32: data
/mnt/isolinux/rear.help: data
/mnt/isolinux/reboot.c32: data
/mnt/isolinux/vesamenu.c32: data
This explains why the ISO image does not boot. Either there was an broken iso image created (which i doubt) or the image was corrupted due to other issues.
Does the backup log where the ISO Image was created (rear-rhel-server-02.log) still exist?
msharif4 commented at 2024-07-20 21:57:¶
Can you provide me with all the contents that need to be added to the
local
.conf file? I added all the content to the ticket. I will be using
another
server via NFS to store the backup files. I don’t want to have the
same
issue in the future. If you can update the existing content with the
correct parameters, I can use it.
I appreciate your help.
On Sat, Jul 20, 2024 at 1:40 PM Michael Ablassmeier <
***@***.***> wrote:
rear-bitwarden.iso
i loop mounted the iso image on my system and it seems like all files
within the generated iso image are somewhat
broken. The iso images contents dont look sane to me. So either there has
been a problem during creation of the iso image or the storage where the
iso image comes from is broken..—
Reply to this email directly, view it on GitHub
https://github.com/rear/rear/issues/3288#issuecomment-2241288862, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/BGF7YZX3Y5EVEO77PO634JDZNLDMNAVCNFSM6AAAAABLGBSWSGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBRGI4DQOBWGI
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
msharif4 commented at 2024-07-21 03:13:¶
No, the log file doesn't exist; I had two cron jobs run an hour apart, the first one ran to take backup, and the second one deleted the log file. That log file is removed, and the server is not booting up; that's why I wanted to restore the backup. Thanks
msharif4 commented at 2024-07-21 22:15:¶
Hi, I found the backup.log file. I am attaching it if that helps.
Thanks,
[
backup.log
](url)
abbbi commented at 2024-07-22 08:39:¶
Hi, I found the backup.log file. I am attaching it if that helps. Thanks, [ backup.log ](url)
unfortunately not, this is the log of the backup command (tar) not the log of the mkrescue step that generated the image.
msharif4 commented at 2024-07-23 01:08:¶
When the mkbackup job ran, it created a full backup; I mean, with iso, there is also a tar file 2024-07-19-0000-F.tar.gz, which contains all of the directories such as / var, etc. boot, usr, etc. Is there a way to restore the server using the .tar.gz file? Thanks
abbbi commented at 2024-07-23 05:59:¶
When the mkbackup job ran, it created a full backup; I mean, with iso, there is also a tar file 2024-07-19-0000-F.tar.gz, which contains all of the directories such as / var, etc. boot, usr, etc. Is there a way to restore the server using the .tar.gz file? Thanks
without a working iso image it will be complicated. You can only setup a system (re-install) with similar layout and restore the contents from the file backup (2024-07-19-0000-F.tar.gz) manually as required.
msharif4 commented at 2024-07-23 18:13:¶
Understood. I will reinstall the OS and test the file copying process. To prevent a similar issue in the future, I believe it's necessary to re-configure the local.conf file with the correct parameters. I've added the following content to the file and executed the 'rear -v mkbackup' command. This command successfully created a full backup, but it did not generate a .iso file. Could it be that the 'pxe' value for output field in local.conf file is preventing the creation of an iso file? My aim is to have a backup that can be restored via network boot.
OUTPUT=PXE
OUTPUT_URL=nfs://192.168.1.118/mnt/backup/rhel
BACKUP=NETFS
BACKUP_URL=nfs://192.168.1.118/mnt/backup/rhel
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/media' '/var/tmp' '/var/crash')
#NETFS_KEEP_OLD_BACKUP_COPY=y
BACKUP_TYPE=incremental
FULLBACKUPDAY=( Sun )
Thanks,
abbbi commented at 2024-07-23 20:10:¶
OUTPUT=PXE will create two files as explained before. The linux kernel image and the initrd (rhel-server-XX[kernel,].initrd.cgz) Both can be booted via PXE. Complete ISO images are usually not booted via PXE, the PXE boot solution (iventoy) you are using seems to extract required contents from the ISO Image to make them boot via PXE.
On a regular PXE server you only need to place these two files and serve them via TFtP to boot from it.
You can also try with OUTPUT=ISO and create an iso image as before (which hopefully is not corrupted this time) and attempt to configure this one in your iventoy server, but i dont know if this is going to work. It seems iventoy has a curated list of supported ISO images:
https://www.iventoy.com/en/isolist.html
it can well be that iventoy cannot handle the REAR iso image due to its specific folder layout. Best would probably be to create the iso image and test booting it with QEMU/virt-manager, to rule out an general issue.
jsmeix commented at 2024-08-27 13:45:¶
@abbbi
thank you for your help, in particular for
your debugging what the root cause was, cf.
https://github.com/rear/rear/issues/3288#issuecomment-2241288862
[Export of Github issue for rear/rear.]