#2450 Issue closed: Download latest backup file from network (if there is newer) to USB Stick before restore process

Labels: enhancement, support / question, no-issue-activity

matekubi opened issue at 2020-07-02 22:16:

Hello,

  1. Is that possible to boot REAR from only rescue ISO and after running download backup file from HTTP Server in external network?

  2. I was looking for the first option but it would be great to support AWS S3 buckets and target and source place for backup. What do you think?

Relax-and-Recover (ReaR) Issue Template

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

  • ReaR version ("/usr/sbin/rear -V"):
    Latest

  • OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):
    Centos 8.1

  • System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
    X64

  • Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
    Bios and UEFI

  • Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
    Local

jsmeix commented at 2020-07-03 06:51:

Regarding using Amazon S3 like cloud storage buckets we had
https://github.com/rear/rear/issues/2400

Currently there is no support for AWS S3 buckets in ReaR
and according to https://github.com/rear/rear/issues/2400
it seems it is not easily possible to successfully
mount Amazon S3 like cloud storage buckets
from within the ReaR recovery system.

I am not an Amazon Web Services (AWS) user and in particular
I am not an Amazon Simple Storage Service (Amazon S3) user
so I cannot actually help with "Simple Storage Service" issues.
But as its name tells I suggest to "simply" ask the makers of
that "Simple Storage Service" how to "simply" mount their
"Simple Storage Service" from within a simple system
that simply runs inside a simple ramdisk ;-)

Seriously:
According to
https://github.com/rear/rear/issues/2400#issuecomment-631280520
it seems the actual problem with mounting Amazon S3 buckets
from within the ReaR recovery system is to somehow get
"enough disk space for used as cache or (temporary) directory for s3fs"
available inside the recovery system.

jsmeix commented at 2020-07-03 07:01:

In general regarding downloading a backup file from the network
there is the problem where to store the dowloaded huge backup file
inside the ReaR recovery system that runs only in a ramdisk.

The current methods that are implemented in ReaR do not download
the whole backup file.
Instead a "network storage thingy" (e.g. a NFS share) is mounted
at a mountpoint directory inside the ReaR recovery system
so that the actual backup file remains at its network storage place
and becomes only accessible from within the ReaR recovery system.

This way during backup restore the backup file data is only streamed
from its network storage place down into the ReaR recovery system
so on the ReaR recovery system only some small buffer space is needed.

gozora commented at 2020-07-03 07:06:

there is an option to create single iso that can boot and also contains your data ...

V.

jsmeix commented at 2020-07-03 07:10:

I think the ReaR man page is a bit confusing about
what http actually means in ReaR.

https://github.com/rear/rear/blob/master/doc/rear.8.adoc
reads (excerpts)

Currently Relax-and-Recover supports ... a variety of network protocols
(incl. sftp, ftp, http, nfs, cifs) for storage and backup

that looks as if both "OUTPUT_URL=http" and "BACKUP_URL=http"
would be supported but "man rear" only tells about

OUTPUT_URL=http://
    Write the image using lftp and the HTTP (PUT) procotol.
OUTPUT_URL=https://
    Write the image using lftp and the HTTPS (PUT) protocol.

but nothing about "BACKUP_URL=http" which shows that
only "OUTPUT_URL=http" but not "BACKUP_URL=http"
is supported.

matekubi commented at 2020-07-03 07:14:

Thank you very much for your comments.

@jsmeix OK. S3 doesn't work. Http download is not possible due to small ramdisk.
I was wondering if there is any other options to download backup from WAN network and use it in rear.
Or, only way is create some local storage like CIFS or NFS.

@gozora Yes, i used this option and it was good but in my case if I boot iso without backup everything is fine, but if I boot ISO with backup I've got error - Failed to load ldlinux.c32
I guess the reason of this error is that I boot using iPXE (sanboot method there I use ISO as source) and maybe ISO is too large.

  1. My general use case is to restore backup on machine without any storage in any network - that's why I wanted to use HTTP or S3 because I was able to boot REAR Rescue CD with iPXE via WAN network and now I need backup - but it looks like that this is not possible without any local storage :(

  2. What about BORG? I can use there some SSH connection - Can I use BORG in public network and somehow it mount backup? I don't know - just thinking loudly.

  3. My backup is like 2GB- maybe it will be 3-4. Machine has 8gb ram - is that possible to use bigger ramdisk and somehow WGET backup to machine and restore?

jsmeix commented at 2020-07-03 07:15:

I vaguely remember there is some sophisticated method
by using an intermediate filesystem (I think it was FUSE)
how one could mount things on a usual HTTP server
provided the HTTP server supports certain HTTP methods.

jsmeix commented at 2020-07-03 07:21:

Hmmm...
I wonder why in general a simple pipe like

HTTP_DOWNLOAD_COMMAND | BACKUP_RESTORE_COMMAND

should not work?

gozora commented at 2020-07-03 07:23:

What about BORG? I can use there some SSH connection - Can I use BORG in public network and somehow it mount backup? I don't know - just thinking loudly.

If SSH is an option for you (you have SSH connection available from AWS to sever holding your ReaR backup), you can certainly use Borg which works primarily over SSH. But be aware that when rear recover is triggered, it does not mount your data, but uses borg extract ... instead.

V.

matekubi commented at 2020-07-03 08:25:

Ok. Thanks, guys for your input.

Based on you comments I'm going to change a little bit my approach.

I will restore from USB. So, I will create bootable USB with REAR Rescue (without backup). Then, I have USB Storage for backup, right? I'd like to create bash script with get the latest backup via HTTP from server and save it USB and then REAR restore it

Is that possible? Do you see any obstacles?

  1. Can I add some custom script to execute during start to REAR somehow?

  2. USB Storage is available in REAR for download backup from HTTP server?

  3. Can I add some additional software to rear? How? I guess, I need WGET etc... maybe even python.

  4. Is there any package manager in REAR?

jsmeix commented at 2020-07-03 08:33:

@matekubi
regarding my https://github.com/rear/rear/issues/2450#issuecomment-653394904
I meant something like hacking as an initial experimental test
usr/share/rear/restore/NETFS/default/400_restore_backup.sh
and replace therein in the tar case the hardcoded dd if=$restore_input
by a HTTP download program call (e.g. curl) as appropriate in your case
(provided you use BACKUP=NETFS with its default tar).

jsmeix commented at 2020-07-03 09:10:

@matekubi
when you use OUTPUT=USB
why don't you then also use BACKUP_URL=usb://...
which is the usual and recommended way how to use USB, cf.
http://relax-and-recover.org/documentation/getting-started

But using a bootable disk device for the ReaR recovery system
has a generic drawback, see
https://github.com/rear/rear/issues/1854

By default "BACKUP_URL=usb" does not behave compatible
with how "BACKUP_URL=nfs" behaves, see
https://github.com/rear/rear/issues/1164

And there are other special issues with USB,
for example see https://github.com/rear/rear/issues/2171
and https://github.com/rear/rear/issues/1738 and others.

The root cause is that the whole USB stuff is basically
some kind of "add-on hack" for a special use case
that is not well integrated with how the rest of ReaR works,
see in particular https://github.com/rear/rear/issues/1164 and
https://github.com/rear/rear/issues/1166#issuecomment-272868388

jsmeix commented at 2020-07-03 09:14:

@matekubi
you can copy what you need into the ReaR recovery system
provided it exists on your original system, see the user config variables
COPY_AS_IS, REQUIRED_PROGS, and LIBS in default.conf
https://github.com/rear/rear/blob/master/usr/share/rear/conf/default.conf#L1340

matekubi commented at 2020-07-03 09:18:

Yes, it can use USB as Backup but my goal is more complex.

I have to create flow (bash script) which check backup on USB (if this is latest) and in case if this is not the latest then I will check backup on HTTP server, download it to USB STICK and then restore this latest backup. I will be creating backups in other places and restore in other place. That's why I need to get latest version from server.

What do you think? Is that possible?

That's why I want to use USB now, and need possibility to download latest and place in USB. Do you see any issues?

jsmeix commented at 2020-07-03 09:21:

@matekubi
there is no package manager in ReaR.
The only thing is ReaR's RequiredSharedObjects function (cf. default.conf).
But when you need to copy a runtime environment for an interpreded programming language
into the ReaR recovery system you need to find the needed files on your own.
On an RPM based Linux distribution rpm -q --requires could help a bit.

jsmeix commented at 2020-07-03 09:27:

@matekubi
offhandedly I see no issue when you replace an existing backup file
with another one (e.g. a newer one) provided it is the same file name
(and of course provided it is the same backup file type, e.g. a tar.gz),
cf. https://github.com/rear/rear/issues/2440

jsmeix commented at 2020-07-03 09:37:

@matekubi
to automate such things we have PRE_RECOVERY_SCRIPT
and PRE_BACKUP_SCRIPT (cf. default.conf).
I would make an appropriate script and put it e.g. into /etc/rear
so that the script gets automatically copied into the recovery system
(cf. usr/share/rear/build/GNU/Linux/100_copy_as_is.sh)
or place the script elsewhere and specify it via COPY_AS_IS
and call such a script via PRE_RECOVERY_SCRIPT
and/or PRE_BACKUP_SCRIPT as appropriate.

matekubi commented at 2020-07-03 12:06:

Thanks for your help. I will back to this topic once I will prepare USB and will have some questions - working on it now

matekubi commented at 2020-07-03 12:55:

  1. Regarding issues mentioned by you with USB:

When I will use:
OUTPUT=USB
BACKUP_URL=usb://...
Still, I will face those issues with changing letters for drives etc?

  1. So, maybe still is the better option boot rescue cd by PXE, run REAR, then attach USB and download backup from WWW server?

  2. Apart from HTTP there is also info about FTP support - maybe it is better option to download backup?
    But still I don't have place in memdisk to extract it and restore - also here I can use USB.

I was wondering what can be the easiest and safe way to achieve my goals....

jsmeix commented at 2020-07-03 14:04:

@matekubi
in any case (i.e. regardless where the backup of your files is)
when you boot the ReaR recovery system from a disk device
you could be hit by https://github.com/rear/rear/issues/1854
in particular with newest kernels because of
https://github.com/rear/rear/issues/2254

I myself was never hit by https://github.com/rear/rear/issues/1854
but we do have evidence that a USB device can become /dev/sda
on the replacement hardware, see
https://github.com/rear/rear/issues/1854#issuecomment-403015226
that points to
https://github.com/rear/rear/issues/1271

Therefore to be more on the safe side against https://github.com/rear/rear/issues/1854
I think it is better to boot the ReaR recovery system not from a disk device
but plug in a (removable / USB) disk after the ReaR recovery system was booted
as temporary storage where the backup could be downloaded to.

But when you have now your backup on a /dev/sdc USB disk
there perhaps on the first partition /dev/sdc1 as backup.tar.gz
you need to mount that in the ReaR recovery system
e.g. mount /dev/sdc1 at /current_backup/$HOSTNAME
(you may need a $HOSTNAME sub-directory because
the backup_path function in usr/share/rear/lib/global-functions.sh
indicates that NETFS_PREFIX="$HOSTNAME" is appended)
and then you need to modify /etc/rear/local.conf in the ReaR recovery system
so that the variables that specify where the actual backup is
(e.g. you may set BACKUP_URL=file:///current_backup)
because otherwise 'rear recover' would try to get the backup
from the old/outdated/unaccessible loaction.

I verified right now that with BACKUP_URL=file:///other/backup
(I have a separated filesystem mounted at /other) one gets

root@linux-h9wr # usr/sbin/rear -D mkbackup
...
Using backup archive '/other/backup/linux-h9wr/backup.tar.gz'

so a $HOSTNAME sub-directory is automatically appended
and during "rear recover" with BACKUP_URL=file:///other/backup
a file /other/backup/linux-h9wr/backup.tar.gz must be there
so if backup.tar.gz is directly on a USB disk partition /dev/sdc1
that partition would have to be mounted at /other/backup/linux-h9wr/
or as in my case (I have a separated filesystem mounted at /other)
the backup is created in backup/linux-h9wr/backup.tar.gz
in that filesystem so that during "rear recover" the same mount
(i.e. that filesystem mounted at /other) is expected.

jsmeix commented at 2020-07-03 15:30:

When you have sufficient RAM you may mount a tmpfs.
(tmpfs is a filesystem in RAM).

For example as a test I did right now on my original system

# mkdir /mnt/mytmpfs

# mount -o size=123M -t tmpfs none /mnt/mytmpfs/

# df -h
Filesystem     Size  Used Avail Use% Mounted on
...
none           123M     0  123M   0% /mnt/mytmpfs

# dd if=/dev/urandom of=/mnt/mytmpfs/urandom.stuff bs=1M status=progress
67108864 bytes (67 MB, 64 MiB) copied, 1 s, 66.7 MB/s
dd: error writing '/mnt/mytmpfs/urandom.stuff': No space left on device
124+0 records in
123+0 records out
128974848 bytes (129 MB, 123 MiB) copied, 1.94111 s, 66.4 MB/s

# ls -lh /mnt/mytmpfs/
total 123M
-rw-r--r-- 1 root root 123M Jul  3 17:21 urandom.stuff

# umount /mnt/mytmpfs/

I assume the same should also work inside the ReaR recovery system.

matekubi commented at 2020-07-10 07:26:

Thanks for your help.

I was working on the implementation above cases - we will which one is better for me.

I have a side question:

GOAL: Restore backup created on AWS EC2 to a physical machine.

  1. I've created REAR RESCUE with the backup inside (USB) on a physical machine.
  2. I've restored this backup on the same machine from USB - everything works fine - no issues witch changing drive letters.
  3. Now - my goal will be building backup zip on cloud machine - AWS EC2 and restore it on physical.
  4. So, I've created ONLY BACKUP on AWS EC2 and then I've replaced backup zip on my USB - this Rescue USB has been created on a physical machine.
  5. After replacing, I've started restore using USB stick created on a physical machine with backup zip create in the cloud.
  6. Restore process was fine - no questions, no errors.
  7. After reboot, I cannot boot machine. Even I don't see my CentOS in EFI list of bootable drives.

What was wrong? Rescue CD also contains any system-specific data? Or maybe during restore some changes in the backup are needed? Am I able to fix it somehow?

My another approach will be create full-back with restore iso in EC2 and then to restore in in physical machine. I hope it helps.

jsmeix commented at 2020-07-10 12:25:

@matekubi
does your USB stick with the ReaR recovery system on it boot in UEFI mode?

I.e. did you "rear format" your USB stick for booting in UEFI mode
whit a command of the form

# rear -D format -- -efi /dev/sdX

?

If you did not "rear format" your USB stick for booting in UEFI mode:

As far as I know it is in general not possible to boot an installation system
in BIOS mode (the ReaR recovery system is an installation system)
and then install an UEFI bootloader for the target system.

I think the reason behind is that for a system that was booted in BIOS mode
UEFI related stuff (e.g. EFI variables and things like that) is not accessible.

I assume in this case it did not work at the end of "rear recover" during
its "finalize" stage to let ReaR install an UEFI bootloader
for the recreated target system.

It might work to chroot /mnt/local into the recreated target system
and manually install its UEFI bootloader from within the target system.
I don't know if it can work to manually install an UEFI bootloader this way.
I think when an installation system was booted in BIOS mode
it is impossible to install an UEFI bootloader for the target system.

matekubi commented at 2020-07-10 12:47:

Yes, I did format with EFI mode.

I'm not sure if I was able to describe it cleary in above post:

But the problem I guess is: Amazon EC2 doesn't use EFI and there I've created "only backup" zip, and now I'm trying to restore by using EFI USB Stick (this stick was created on hardware, not Amazon EC2), before I replace backup from EC2 with backup on local USB Stick.

So, backup.tar.gz was prepared on a non EFI system (Amazon EC2)
Rescue USB was created by rear as EFI USB Stick, rescue bootable drive was created also on EFI hardware BUT backup comes from EC2. I've replaced it.

My goal will be to create a backup of a file system in the cloud and restore it on the physical machine.

jsmeix commented at 2020-07-10 14:32:

@matekubi

at the end of "rear recover" during its "finalize" stage
when ReaR installs a bootloader for the recreated target system
it is done via chroot /mnt/local into the recreated target system
so that the bootloader is installed by calling the bootloader installation tool
of the restored target system with the restored bootloader config files of the target system
i.e. with the bootloader installation tool and its config files from the original system,
e.g. for GRUB2 BIOS bootloader installation see
https://github.com/rear/rear/blob/master/usr/share/rear/finalize/Linux-i386/660_install_grub2.sh
versus for UEFI bootloader installation see
https://github.com/rear/rear/blob/master/usr/share/rear/finalize/Linux-i386/670_run_efibootmgr.sh
plus - as needed - e.g. additional UEFI related things like
https://github.com/rear/rear/blob/master/usr/share/rear/finalize/SUSE_LINUX/i386/675_install_shim.sh

So when you don't have already UEFI bootloader installation tools and config files
on your original system you can't get an UEFI bootloader installed on the target system.

Switching from BIOS to UEFI is not something that "works out of the box" in ReaR,
cf. the section "Fully compatible replacement hardware is needed" in
https://en.opensuse.org/SDB:Disaster_Recovery

What you intend to do is a major system migration.

In general regarding system migration see
http://lists.relax-and-recover.org/pipermail/rear-users/2018-November/003626.html

I never migrated from BIOS to UEFI so the following is only an offhanded guess:

First you need UEFI bootloader installation tools and config files on your original system
to be able to manually install a UEFI bootloader after "rear recover" finished
but still inside the running ReaR recovery system via chroot /mnt/local into
the recreated target system and then adapt the target system bootloader config files
as needed to migrate the recreated target system from BIOS to UEFI and
afterwards install the target system bootloader.

But that would be only the bootloader migration from BIOS to UEFI.

There is also disk layout migration needed from BIOS to UEFI:

Usually a UEFI system requires an EFI System Partition (ESP)
so your disklayout.conf from your original BIOS system won't have that
and a UEFI system requires GPT partitioning but your original BIOS system
may still use MBR (a.k.a. MSDOS) partitioning (inspect your disklayout.conf
whether or not it contains disk ... gpt or still msdos) so you may have to
manually adapt your disklayout.conf to create GPT partitioning with an ESP.

matekubi commented at 2020-07-15 19:33:

Hello,

Thanks again for your comments. I was able to achieve another goal.

  • In physical PC and AWS EC2 I use legacy bios - not UEFI.
  • I have created USB stick on physical PC
  • I have created backup zip on AWS EC2 machine.
  • From USB Stick with Rear I've booted system and replaced backup zip with backup zip from cloud.
  • Started rear recover
  • and.....

Everything was able to restore backup but there is a problem with the bootloader.
The system is able to load bootloader after the restart but I can see some errors:
https://ibb.co/JshhyJ5

Log from rear recovery: https://pastebin.com/zenyTr8N

FSTAB doesn't look good: https://ibb.co/5TcMSXs

It looks like that I'm close to the final solution - I hope.... :)

jsmeix commented at 2020-07-16 08:52:

@matekubi
in your initial
https://github.com/rear/rear/issues/2450#issue-650231346
you wrote

ReaR version ("/usr/sbin/rear -V"):
Latest

but your recent
https://pastebin.com/zenyTr8N
contradicts

2020-07-15 16:34:39.138267461 Relax-and-Recover 2.4 / Git

So first and foremost actually use our latest ReaR upstream version
which is ReaR version 2.6, cf.
http://relax-and-recover.org/download/
or preferably use our current ReaR upstream GitHub master code
cf. "Testing current ReaR upstream GitHub master code" in
https://en.opensuse.org/SDB:Disaster_Recover

In general we at ReaR upstream do not support older ReaR versions, cf.
http://relax-and-recover.org/support/
and in particular for SUSE Linux Enterprise see the section
"SUSE support for Relax-and-Recover" in
https://en.opensuse.org/SDB:Disaster_Recovery
(I don't know about other Linux distributions).

We at ReaR upstream do not plain reject issues with older ReaR versions
(e.g. we may answer easy to solve questions also for older ReaR versions)
but we do not spend much time on issues with older ReaR versions because
we do not (and cannot) fix issues in released ReaR versions.

I recommend to try out our current ReaR upstream GitHub master code
because that is the only place where we at ReaR upstream fix issues.

Issues in released ReaR versions are not fixed by us (by ReaR upstream).
Issues in released ReaR versions that got fixed in current ReaR upstream
GitHub master code might be fixed (if the fix can be backported with
reasonable effort) by the Linux distributor wherefrom you got ReaR.

In your
https://pastebin.com/zenyTr8N
there is (excerpt)

2020-07-15 16:40:52.500272377 Including finalize/Linux-i386/620_install_grub2.sh
2020-07-15 16:40:52.504701734 Installing GRUB2 boot loader
Generating grub configuration file ...
device node not found
...
device node not found
Cannot find list of partitions!  (Try mounting /sys.)
done
Installing for i386-pc platform.
Installation finished. No error reported.
Installing for i386-pc platform.
Installation finished. No error reported.

therein in particular the Cannot find list of partitions! (Try mounting /sys.)
does not look right and seems to point to the root cause why GRUB2
was actually not correctly installed regardless that it does not report an error.

Since ReaR version 2.5

...  /proc /sys /dev and /run are bind-mounted into TARGET_FS_ROOT at the
    beginning of the finalize stage via the new
    110_bind_mount_proc_sys_dev_run.sh script ...

cf.
https://github.com/rear/rear/blob/master/doc/rear-release-notes.txt#L1181
and
https://raw.githubusercontent.com/rear/rear/master/usr/share/rear/finalize/default/110_bind_mount_proc_sys_dev_run.sh

github-actions commented at 2020-09-15 01:34:

Stale issue message

[Export of Github issue for rear/rear.]