#2246 Issue `closed`: Integration REAR and NetBackup. I cannot restore differentials and fulls backups with NetBackup backup types standard, BMR or True Image Restore and "ALL_LOCAL_DRIVES" selection¶

Labels: enhancement, needs sponsorship, external tool, no-issue-activity

cge77 opened issue at 2019-10-04 14:39:¶

Relax-and-Recover (ReaR) Issue Template¶

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

ReaR version ("/usr/sbin/rear -V"): 2.4.
OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"): RHEL 7.7.
ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

# Default is to create Relax-and-Recover rescue media as ISO image
# set OUTPUT to change that
# set BACKUP to activate an automated (backup and) restore of your data
# Possible configuration values can be found in /usr/share/rear/conf/default.conf
#
# This file (local.conf) is intended for manual configuration. For configuration
# through packages and other automated means we recommend creating a new
# file named site.conf next to this file and to leave the local.conf as it is.
# Our packages will never ship with a site.conf.

# To restore servers using multipath or not
AUTOEXCLUDE_MULTIPATH=y

# Recover data output to a ISO 
OUTPUT=ISO

# Location of the ISO
ISO_DIR=/tmp

# Backup type NetBackup
BACKUP=NBU

# Time synchronization
TIMESYNC=NTP

# Add programs to the iso image
PROGS=( ${PROGS[@]}
telnet )

# Add NetBackup Client files to the iso image
COPY_AS_IS=( ${COPY_AS_IS[@]}
/opt/Symantec
/opt/veritas
/opt/VRTSpbx
/etc/vx
/usr/openv
)

# Password to access using SSH to the rescue system
SSH_ROOT_PASSWORD=

Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR): virtual machine.
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device): x86.
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot): BIOS and GRUB.
Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe): SAN FC.
Description of the issue (ideally so that others can reproduce it):

I am using REAR with NetBackup 8.1.2.

If I restore BMR backups with "ALL_LOCAL_DRIVES" selection, I can only restore full backups, Differential backups are not restored OK because NetBackup restores all the data of the server from the date and time answered in the REAR process question to the past: mix policies and do not take in account the relation between the differentials and full backups, etc. Basically, NetBackup restores all the data about the server he finds. The server boots OK but the restore is not consistent.

If I restore Standard or TIR (True Image Restore) backups with "ALL_LOCAL_DRIVES" selection the data restored it seems perfect but the server does not boot OK, the server boots in the Emergency Mode and with the dracut prompt (see the screens attached).

If I restore Standard or TIR (True Image Restore) backups selecting/mark all the folders of the server but not with "ALL_LOCAL_DRIVES" option and full and differential backups, the data restored is perfect and the server boots OK. The problem is I cannot use the "ALL_LOCAL_DRIVES" option and this is not operative.

I do not know what is the difference for REAR or NetBackup between use the "ALL_LOCAL_DRIVES" option or mark all the folders of the servers but it seems this is the key.

The ideal scenario for me it would be NetBackup backup type standard, BMR or TIR and the "ALL_LOCAL_DRIVES" selection.

I have tried everything... Mainly in the "bprestore" command in the "400_restore_with_nbu.sh” but I had not luck... I have tried in the command the parameters TIR (-T), end time date (-e), start time date (-s), policy (-p), etc.

I run out of ideas... I do not know what I can do more...

Many thanks.

Workaround, if any:
Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):

To paste verbatim text like command output or file content,
include it between a leading and a closing line of three backticks like

verbatim content

jsmeix commented at 2019-10-05 12:44:¶

@cge77
not that it actually helps you with this particular issue
but in general see the section
"Relax-and-Recover versus backup and restore" in
https://en.opensuse.org/SDB:Disaster_Recovery
that reads in particular (excerpt)

... Usually only basic support for the various backup tools
is implemented in ReaR (i.e. plain making a backup
during "rear mkbackup" and plain restore during "rear recover").
It is very different to what extent support for each individual
backup tool is implemented in ReaR because support for each
individual backup tool is implemented separated from each other.
Therefore for some particular backup tools the current support
in ReaR could be even only "very basic" ...

I guess what you ask for here

NetBackup ... restore ... differentials and fulls backups
with NetBackup backup types standard, BMR
or True Image Restore and "ALL_LOCAL_DRIVES" selection

is perhaps not (yet) supported by the NetBackup code in ReaR.

In general regarding issues with third party backup tools:
Usually there is nothing at all what we at ReaR upstream
could do in case of issues with third-party backup tools
because usually we do not have such software
(in particular not when it is proprietary software
according to https://en.wikipedia.org/wiki/NetBackup)
so that we can neither test nor reproduce anything.
Therefore for issues with third party backup tools
you shoud also ask for help and support where
you got your particular third party backup tool.

cge77 commented at 2019-10-07 07:39:¶

Hi jsmeix.

Thanks for the fast response.

Yes, I know and understand the support for third party tools as NetBackup is limited here. I exposed my case here because I understand some people is treating the integration of REAR and NetBackup and it is possible they have encountered the same or similar problems.

In general, I see the integration of REAR and NetBackup is complex. As you said, the basic functionalities work but if you want to do something more advanced and professional, is difficult.

For example, you can restore the last full backup of a server perfectly if the last backup of your server is a backup of all the data of the server, if the last full backup of your server is a backup only of the "/var" folder, REAR and NetBackup only will restore the "/var" folder. Other topic is one of I mentioned in my first post, in many occasions the restore does not take in account the relation between the differentials and full backups and restore all the data about the server he finds. Gives the impression you have to configure all in a determined way to restore the server consistent.

Well, I continue with the fight :)

Thanks and regards.

cge77 commented at 2019-10-07 12:22:¶

Hi.

I am still in the fight :)

The data restore is OK but the server only boots with the rescue boot. If I try a normal boot the server shows the message "/dev/mapper/vg00-root does not exist" and the dracut shell. I am investigating if it is possible to do any action in the rescue boot to fix the normal boot of the server. Any help is appreciated :D

Thanks and regards.

cge77 commented at 2019-10-08 11:08:¶

Hi.

I got it :) We are able to restore BMR full and differential incremental backups. The key is the "bprestore" command has to be as follows:

bprestore_args="-B -H -L $TMP_DIR/bplog.restore -8 -R $TMP_DIR/nbu_change_file -t 0 -w 0 -e ${edate} -p <policy_name> -T -C ${NBU_CLIENT_SOURCE} -D ${NBU_CLIENT_NAME} -f $TMP_DIR/restore_fs_list"

The most important parameters of the "bprestore" command are:

-e ${edate}: date and time of the backup we want to restore. The parameter “-e” means "end date and time of the backup".

-p: policy name of the backup we want to restore. This parameter is useful to specify and control only we restore backups of this policy. It is essential because without this parameter NetBackup restores other policies that backup the server to restore.

-T: restore a BMR backup. This parameter controls the relation between the differential incremental backups and the full backups. As the parameter "-p", the parameter "-T" is a control parameter to avoid NetBackup restores all the data about the server he finds, without this parameter, NetBackup do not take in account the relation between the differential incremental backups and the full backups.

With this parameters, REAR and NetBackup restore full and differential incremental backups in a consistent way.

One important topic, the NetBackup backups has to be BMR backups. The backup data selection we use is "ALL_LOCAL_DRIVES".

I wanted to share with the community in case someone is fighting with the integration of NetBackup and REAR and can help :D

Thanks and regards.

cge77 commented at 2019-10-08 16:27:¶

Hi again.

One essential topic I forgot comment in the last post. If the servers does not boot in the Normal Mode, boot the server in the Rescue Mode and execute this command:

dracut --regenerate-all -f && grub2-mkconfig -o /boot/grub2/grub.cfg

And then, boot the server in Normal Mode and the server is recovered. My opinion is REAR and NetBackup touch the boot configurations as MBR, GRUB, boot, kernels, etc; and we have to use the "dracut" command to fix these configurations.

I have used Red Hat to do the tests.

Thanks and regards.

jsmeix commented at 2019-10-09 14:59:¶

@cge77
can you post or attach your whole ReaR script here
that contains your modified "bprestore" command
so that I (as non NetBackup user) have at least a chance
to understand what you are talking about.
Thank you in advance!

cge77 commented at 2019-10-10 06:53:¶

Hi Jsmeix.

Yes, sure.

The script I edited is the "/usr/share/rear/restore/NBU/default/400_restore_with_nbu.sh". Basically, is the original script but with the first "bprestore" command modified with these parameters: "-p <nbu_backup_policy_name> -T".

# 400_restore_with_nbu.sh
# restore files with NBU

LogPrint "NetBackup: restoring / into $TARGET_FS_ROOT"

echo "change / to $TARGET_FS_ROOT" > $TMP_DIR/nbu_change_file

# Do not use ARGS here because that is readonly in the rear main script.
# $TMP_DIR/restore_fs_list was made by 300_create_nbu_restore_fs_list.sh
if [ ${#NBU_ENDTIME[@]} -gt 0 ] ; then
    edate="${NBU_ENDTIME[@]}"
    bprestore_args="-B -H -L $TMP_DIR/bplog.restore -8 -R $TMP_DIR/nbu_change_file -t 0 -w 0 -e ${edate} **-p <nbu_backup_policy_name> -T** -C ${NBU_CLIENT_SOURCE} -D ${NBU_CLIENT_NAME} -f $TMP_DIR/restore_fs_list"
else
    bprestore_args="-B -H -L $TMP_DIR/bplog.restore -8 -R $TMP_DIR/nbu_change_file -t 0 -w 0 -C ${NBU_CLIENT_SOURCE} -D ${NBU_CLIENT_NAME} -f $TMP_DIR/restore_fs_list"
fi

LogPrint "RUN: /usr/openv/netbackup/bin/bprestore $bprestore_args"
LogPrint "Restore progress: see $TMP_DIR/bplog.restore"
LANG=C /usr/openv/netbackup/bin/bprestore $bprestore_args
rc=$?
if (( $rc > 1 )) ; then
    Error "bprestore failed (return code = $rc)"
fi

With these 2 parameters and the parameter "-e ${edate}" you get the next:

-e ${edate}: date and time of the backup we want to restore. The parameter “-e” means "end date and time of the backup".

-p: policy name of the backup we want to restore. This parameter is useful to specify and control only we restore backups of this policy. It is essential because without this parameter NetBackup restores other policies that backup the server to restore.

-T: restore a BMR backup. This parameter controls the relation between the differential incremental backups and the full backups. As the parameter "-p", the parameter "-T" is a control parameter to avoid NetBackup restores all the data about the server he finds, without this parameter, NetBackup do not take in account the relation between the differential incremental backups and the full backups.

With these parameters, REAR and NetBackup restore full and differential incremental backups in a consistent way.