#3464 PR open: Add initial Cove support

Labels: enhancement

svlv opened issue at 2025-04-28 13:24:

Pull Request Details:

The Backup Manager integration was tested manually by the Cove team. Tests were performed on VMware virtual machines for the following distributions:

  • Debian 12
  • Ubuntu 22.04, 24.04
  • CentOS 7, 8, 9
  • RHEL 7, 8, 9

The ISO rescue image created by ReaR for each distribution was used to recover or migrate the source system.

  • Description of the changes in this pull request:

The backup process requires the Backup Manager to be installed in order to include the necessary shared libraries in the rescue media. It is supposed that rear will be launched by the Backup Manager as an external program, providing the COVE_TIMESTAMP environment variable, which is necessary to find the appropriate backup session for files and folders.

The Backup Manager provides the required configuration parameters in the site.conf file, which is installed by the Backup Manager installer. The Backup Manager supports by default OUTPUT=ISO.

After ReaR successfully creates the ISO rescue image, it is backed up as a regular file by the Backup Manager.

At recovery time, ReaR downloads and installs the Backup Manager on the partitioned target disks and launches the Backup Manager, which restores the backed up files and folders.

gdha commented at 2025-05-05 13:52:

@svlv General remark - please read and adapt the https://github.com/rear/rear/wiki/Coding-Style rules.

gdha commented at 2025-05-09 09:12:

@rear/contributors Assigned this to all rear contributors so we are all aware of this new external backup/restore software

svlv commented at 2025-05-14 09:02:

Hi @rear/contributors,

What can I do to promote this PR to be reviewed/approved?

jsmeix commented at 2025-05-14 11:19:

@svlv
FYI: I won't find time for a review this week.
Hopefully - as time permits - next week...

svlv commented at 2025-05-14 15:45:

Hi @schlomo,

Thank you for your comments. Could you please check this comment? It will likely clarify the integration between the Backup Manager and ReaR.

However, if you still have questions, I’d be happy to set up a call to discuss them.

schlomo commented at 2025-05-14 17:11:

I see, thank you. I didn't see that earlier comment and it explains some things.

I'd like to suggest speaking in person for efficiency. After reading through all of this I'm not sure that you'd actually benefit from that kind of rest integration over using ReaR as an embedded tool without modification.

I'm thinking of using PORTABLE output and EXTERNAL backup, together with a generic other rescue system line Systemrescuecd or such.

I think it might be much simpler and closer to your use case than teaching ReaR to do your bidding.

My reasoning:

  • adding a backup method that works only with ISO output goes massively against the ReaR philosophy
  • ReaR is made to be in control and instrument the backup software to recover the files, not the other way around. That is the reason why we need integrations. If ReaR is not in control you don't really need an integration, just embed it
  • ReaR collects required shared libraries, so no need to implement that again. Easier to use rear as intended.
    ReaR is designed for crying personalized rescue images with a very high degree of reliability to successfully perform a recovery. We added the portable mode for all use cases that need a generic rescue image and recommend against using ReaR to create that (as we don't develop and test for this).
  • in professional (data center) environments typically all servers with persistent data (pets, not cattle) have lots of RAM and GB sized rescue images are unfortunately the norm with most backup software (not like the puny 70MB when I started nearly 20 years ago). With modern backup systems that is typically not a problem to add. Therefore it might be a bit of premature optimization adding complexity, or maybe you have a special environment in mind.
  • putting the backup software onto the target disc calls for potential conflicts with restoring files onto that
  • if the backup agent evolves quickly we recommend to create a new rescue image with every backup run, as a pre task to it. Much more reliable than dynamic installations from the Internet during recovery IMHO

Out of curiosity, did you try the traditional way of ReaR integration? Why wouldn't that work? Including auto discovery for the installation paths etc.?

Anyhow, happy to chat about that in a meeting and I'm sure we'll quickly figure out what's best for our combined users.

Maybe I don't have the full picture yet and we are committed to make ReaR the boat useful Linux disaster recovery tool for all.

Kind regards,
Schlomo

jsmeix commented at 2025-05-19 12:55:

@svlv
FYI a side note regarding

we recommend to create a new rescue image
with every backup run

in https://github.com/rear/rear/pull/3464#issuecomment-2880963296
see the section
"Relax-and-Recover versus backup and restore"
https://en.opensuse.org/SDB:Disaster_Recovery#Relax-and-Recover_versus_backup_and_restore
therein in particular the part about
(excerpts)

It is your task to ensure your backup is consistent.

First and foremost the files in your backup must be consistent
with the data that is stored in your ReaR recovery system
...
(e.g. the contents of the restored etc/fstab must match
the actually recreated disk layout).
Therefore ... each change of the basic system
... needs ... to create a new ReaR recovery system
together with a matching new backup of the files

svlv commented at 2025-05-19 14:13:

Hello @jsmeix,

we recommend to create a new rescue image
with every backup run

We actually do this. The recovery system matches the backup files and folders (it is done via embedding timestamp to find the corresponding backup session for files). But the recovery system does not include the backup software, and it downloads it from the Internet, I guess it is the main concern.

The motivation for this approach is that users have backup sessions that are 1 month or even 1 year old, and they want to use this backups to restore the system layout. If the rescue ISO image contains the Backup Manager, it is very likely that the included version will be outdated and may not work properly. It is not hypothetical example, it is our real cases.

IMHO downloading the installer from the Internet is at least not bad idea because

  1. You can update the backup software during the recovery process
  2. The rescue system does not depend on the backup software version

The backup software restore files and folders via Internet why the installer couldn't be downloaded.

Any way, I understand you requirements for the integration. I will update the PR to satisfy ReaR standards and requirements regarding the integration. It will take some time but the logic will be much simpler.

Additionally, I am checking the possibility to use PORTABLE output with the generic rescue ISO image, it looks nice.

schlomo commented at 2025-05-19 14:33:

@svlv could you share more context for the 1 month or 1 year old backups that need to be recovered with a newer version of Backup Manager? Where is it enough to have only seldom backups? Are these special systems that kind of never change?

Just to understand what is going on: The online download of the backup client is a workaround for a tight version coupling between backup server and backup client that would be broken by "older" rescue images containing an older version of the backup agent?

I'm asking because this scenario didn't come up with any of the previous backup software integrations...

svlv commented at 2025-05-19 15:23:

Hello @schlomo,

could you share more context for the 1 month or 1 year old backups that need to be recovered with a newer version of Backup Manager? Where is it enough to have only seldom backups? Are these special systems that kind of never change?

These are users decision. Users can configure how often they want to create backups (once per day or even every 15 mins) and how long these sessions have to be stored (one month, one year, etc.)

Just to understand what is going on: The online download of the backup client is a workaround for a tight version coupling between backup server and backup client that would be broken by "older" rescue images containing an older version of the backup agent?

Yes, I meant that the included backup agent into the recovery system might not be backward compatible with the backup server.

Additionally, we are solving issue with outdated certificates for the backup agent that it is very likely appear. We would need to have a special tool to somehow update the certificate safely. But I think we won't have such a tool in the near future.

Additionally, the backup agent uses the database that is used to build files and folders from data chunks. This database is not supposed to be included in the rescue system. It might have several GBs of size if users have a lot of sessions, but it is downloading from the network if it is not available locally. That is why we decided to move the backup agent to partitioned disks. In that case the DB is located on the disk, not in RAM. Thanks to that the recovery process consumes much less memory (4GB vs 8Gb). RAM consuming is important for us because users have also quite small VMs (e.g., 2GB of RAM). Why not support them if we found a way?

I hope, now it is a bit more clear

schlomo commented at 2025-05-19 16:15:

We are talking about your story here amongst the ReaR maintainers and got some thoughts:

Is Cove a cloud-only backup tool? Especially in that case I'd like to suggest a more streamlined approach, but that should also work well with a local backup server:

  • boot the bare metal recovery system from a generic boot loader (somehow, ISO, floppy disk, PXE, whatever)
  • boot the actual rescue system via HTTP(S) over the network. This way the client doesn't need to have actual network boot support but you can still benefit from the flexibility of it.
  • the backup server / cloud service provisions an up-to-date rescue system (with latest certificates, backup agents etc.) on the fly to boot the recovery hardware from. You can use initramfs chaining for this purpose :-) This skips the update of the rescue image from inside and ensures that the resuce image is up to date when it boots.
  • This chainloading can be also used to inject credentials or other private data
  • Adding CA certificates and the backup manager software can be done also via a public URL
  • This way you can guarantee much better that the rescue system is truly up-to-date with all what you need
  • This approach will also work well with booting a generic rescue system and "adding" ReaR and the required configuration as additional initramfs images

About the local disk:

  • Yes, use the local disk for your database if you need to, even though it "breaks" the ReaR design philosophy.

However, the ReaR rescue system is till a "personal" rescue system for a single server. If you don't want that then with the OUTPUT=PORTABLE and BACKUP=EXTERNAL methods you should IMHO also be able to simply integrate ReaR within Cove and use ReaR as a BMR tool.

If you want to continue with the personalised ReaR rescue system, then I'd like to suggest that you also adopt the proven ReaR concepts like:

  • bundle the backup client and let ReaR handle the library dependencies
  • ensure that users create a new ReaR rescue system whenever something changes, e.g. the certificates or the backup client. That way the ReaR rescue system is guaranteed to 1) be compatible with your backup server and 2) be up-to-date and actually function
  • optimise the whole system to have less things that can go wrong during recovery, as ReaR does with all the other backup tools

[Export of Github issue for rear/rear.]