#2367 Issue closed: NETWORKER backing up only the ISO file?¶
Labels: support / question, fixed / solved / done,
external tool
IT-Guy-1973 opened issue at 2020-04-15 00:34:¶
-
ReaR version ("/usr/sbin/rear -V"): 1.17.2
-
OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"):
-
ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
-
Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR):
-
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
-
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
-
Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
-
Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT" or "lsblk" as makeshift):
-
Description of the issue (ideally so that others can reproduce it):
I am able to run "rear -vD mkbackup" and see the following messages.
# rear -vD mkbackup
Relax-and-Recover 1.17.2 / Git
Using log file: /var/log/rear/rear-lnxjump01.log
Creating disk layout
Excluding mountpoint /nmon_data. (MANUAL_INCLUDE mode)
Creating root filesystem layout
EMC Networker will recover these filesystems: / /nmon_data /boot
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Creating initramfs
Making ISO image
Wrote ISO image: /var/lib/rear/output/rear-lnxjump01.iso (141M)
Saving result files with NSR (EMC NetWorker)
If the NSR_RETENTION_TIME="3650 days" is too low please add NSR_RETENTION_TIME variable in /etc/rear/local.conf
pool retent name
============================
#
To me, this looks like it is backing up the "/var/lib/rear/output/rear-lnxjump01.iso " file only. According to my site.conf file I should be backing up '/boot' '/' 'swap' '/dev' as well.
# cat site.conf | grep -v "^#"
BACKUP=NSR
OUTPUT=ISO
MODULES_LOAD=(vmxnet)
BACKUP_PROG_INCLUDE=( '/boot' '/' 'swap' '/dev')
MANUAL_INCLUDE=YES
Why are those file systems not been backed up?
-
Workaround, if any:
-
Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):
IT-Guy-1973 commented at 2020-04-15 00:36:¶
This is the output I see on the Networker logs.
174411:save: Step (1 of 7) for PID-17953: save has been started on the client 'lnxjump01.c1shrd.com'.
175313:save: Step (2 of 7) for PID-17953: Running the backup on the client 'lnxjump01' for the selected save sets.
175318:save: Identifed a save for the backup with PID-17953 on the client 'lnxjump01'. Updating the total number of steps from 7 to 6.
174920:save: Step (3 of 6) for PID-17953: Contacting the NetWorker server through the nsrd process to obtain a handle to the target media device through the nsrmmd process for the save set '/var/lib/rear/output/rear-lnxjump01.iso'.
174908:save: Saving the backup data in the pool 'A0C3-S'.
175019:save: Received the media management binding information on the host 'a0c3us034nve001.usp01.xstream360.cloud'.
174910:save: Connected to the nsrmmd process on the host 'a0c3us034nve001.usp01.xstream360.cloud'.
175295:save: Successfully connected to the Data Domain device.
129292:save: Successfully established Client direct save session for save-set ID '3902163720' (lnxjump01:/var/lib/rear/output/rear-lnxjump01.iso) with Data Domain volume 'A0C3S.010'.
174922:save: Step (4 of 6) for PID-17953: Successfully connected to the target media device through the nsrmmd process on the host 'a0c3us034nve001.usp01.xstream360.cloud' for the save set '/var/lib/rear/output/rear-lnxjump01.iso'.
174422:save: Step (5 of 6) for PID-17953: Reading the save sets and writing to the target device.
/var/lib/rear/output/rear-lnxjump01.iso
174917:save: Step (6 of 6) for PID-17953: Backup has succeeded. Save is exiting.
/var/lib/rear/output/
/var/lib/rear/
/var/lib/
/var/
/
save: /var/lib/rear/output/rear-lnxjump01.iso 143 MB 00:00:03 6 files
94694:save: The backup of save set '/var/lib/rear/output/rear-lnxjump01.iso' succeeded.
Thank you,
Regards,
Niranjan
IT-Guy-1973 commented at 2020-04-15 00:51:¶
Hello Support,
So with the same setting when I do a REAR backup to an NFS server it backs up the system files not only the ISO image.
'/boot' '/' 'swap' '/dev'
Why is Networker backing up these files?
Regards,
Niranjan
gdha commented at 2020-04-15 07:04:¶
@IT-Guy-1973 First of all 1.17.2 is really old! Please upgrade ReaR to
2.4 or 2.5.
And, yes ReaR will only save the ISO file after the mkbackup and it is
designed in this way (was a sponsored project). With EMC Networker (and
this is also true for all EXTERNAL backup programs) you are responsible
of making your own full backup of the Linux system.
IT-Guy-1973 commented at 2020-04-15 15:08:¶
@gdha When I ran "yum in rear" on my RedHat v6.10 system it installed
this version. I will look to upgrade it soon. But so far it has been
working great.
When I backed up my VM to an NFS file system and restored it - REAR did
all the work. It created the disk to restore and created the MBR and
everything. Can I expect the same experience when I integrate it with
Networker? Networker currently takes backup of all my file systems on
that server on a daily basis. I am going to try the restore soon but I
just wanted to ask you first.
Some background:
Our larger goal is to implement rear on our 23000 VM's and physical
servers we support. They all run either RedHat (v6 and v7) and SUSE (v11
and v12). So the support you are providing me now by answering all my
questions is really appreciated. Thank you, again.
Regards,
Niranjan
IT-Guy-1973 commented at 2020-04-16 14:23:¶
Hello Support,
I was able to restore using NSR to another new system (VM) with the same hard disk sizes. However, when restoring to the new system REAR identified that the IP was different but it said "Disks are identical" and started the restore without prompting me. I had two disks on this system one 80GB and the other 40GB. Both belong to different volume groups. But REAR restored all data to one disk and (80GB) and kept the other disk free. The system after restore went to maintenance mode and I had to manually comment out the file system that belongs to the second VG to bring the system up.
Regards,
Niranjan
jsmeix commented at 2020-04-16 15:04:¶
@IT-Guy-1973
see in default.conf the section about
MIGRATION_MODE recovery during "rear recover"
how ReaR determines if "Disks are identical"
https://github.com/rear/rear/blob/master/usr/share/rear/conf/default.conf#L351
and also the subsequent section about
Resizing partitions in MIGRATION_MODE during "rear recover"
You don't have that in your old ReaR 1.17.2 version.
But you really must have that newer and much more fail-safe behaviour.
In particular when you migrate systems.
See the ReaR release notes for some more info
about that newer and much more fail-safe behaviour,
in particular follow the links to some of the scaring issues
that had led to the newer and much more fail-safe behaviour
https://raw.githubusercontent.com/rear/rear/master/doc/rear-release-notes.txt
ReaR version 2.3 has an improved MIGRATION_MODE autodetection
when the disk layout looks ambiguous.
ReaR version 2.4 has major rework and changed default behaviour
how ReaR behaves in migration mode when partitions can or must
be resized to fit on replacement disks with different size.
ReaR version 2.5 has lots of other improvements that are in particular
useful in migration mode i.e. when the replacement system has
non-identical hardware (e.g. MODULES=( 'all_modules' ) and
a new config variable GRUB2_INSTALL_DEVICES and so on...).
Newer ReaR will never start the restore without prompting you.
There is always a confirmation dialog with 30 seconds timeout, cf.
https://github.com/rear/rear/issues/1271#issuecomment-346788244
and
https://github.com/rear/rear/blob/master/usr/share/rear/layout/prepare/default/250_compare_disks.sh#L136
Only if you do nothing it will start the restore after the timeout.
This is to be backward compatible with automated restore behaviour.
Note that resizing only happens for plain disk partitions
but not for higher level storage objects like logical volumes.
So in general when you have things like LVs it is recommended
to recreate the system without resizing - or you need to manually
and carefully adapt all the related values in disklayout.conf.
Furthermore see the section
"Testing current ReaR upstream GitHub master code" in
https://en.opensuse.org/SDB:Disaster_Recovery
Or just better:
Complelety read through that whole long "SDB:Disaster Recovery"
article
(you may leave out YaST and other SUSE specific things ;-)
IT-Guy-1973 commented at 2020-04-17 04:25:¶
Hello @jsmeix ,
Thank you as always.
I updated the rear to v2.4.
The first thing I noticed with the new version is that it does not prompt for the network card. The IP is already there of the source system.
One reason to use REAR for us is to upgrade systems by building another system. Therefore, we need to restore the system onto another system while the source system is in production. We cannot afford to have an IP conflict. Luckily the v2.4 of REAR stops at the prompt. I changed the original IP that was there with REAR to a different IP so that no IP conflict happens.
The other option that I am looking for is the list of disks or file systems that I could select for the restore. We almost all of the time need only the root disk to be restored. The root disk would be about 30 GB. We have databases on other disks that run to TB's and we should not restore those with REAR. I do not see the option presented in REAR with Networker. How can I select what disks o file systems to restore? Because Networker anyway backs up everything and I cannot stop Networker from backing all the mounted file systems. How can we select what to restore when restoring through REAR v2.4?
Thank you,
Regards,
Niranjan
hpannenb commented at 2020-04-17 07:46:¶
Hi @IT-Guy-1973.
Within Legato Networker configuration there are IMHO possibilities to
exclude files/directories based on patterns and wildcards (IMHO named
backup directives) or determine the backup filesystems in the saveset.
So it is - as @gdha pointed to - always the responsibility of the
EXTERNAL backup tool what files will be recovered to the system.
ReaR's part is to "prepare" the to be recovered system to its initial
state (i.e. setup IPs, partitions, filesystems, start the required
daemons etc) and thereafter starts i.e. the EXTERNAL recovery or waits
for the EXTERNAL recovery to do its part. What is included or not is up
to the EXTERNAL backup/recovery tool only.
Cheers,
Holger.
jsmeix commented at 2020-04-17 10:14:¶
@IT-Guy-1973
regarding
we need to restore the system onto another system
while the source system is in production.
We cannot afford to have an IP conflict.
see "Some general information FYI" in
https://github.com/rear/rear/issues/2355#issuecomment-611408518
Additionally see
In general regarding "changing IP address" and ReaR
there are two different things
in
https://github.com/rear/rear/issues/2344#issuecomment-601132633
jsmeix commented at 2020-04-17 10:23:¶
@IT-Guy-1973
regarding
The other option that I am looking for is the list of disks
or file systems that I could select for the restore
Offhandedly I think the currently best way to do that is
to manually adapt disklayout.conf as you need it
before you run "rear recover"
or
enforce MIGRATION_MODE and manually adapt disklayout.conf
at the matching dialog that appears while running "rear recover"
or
use RECOVERY_UPDATE_URL (cf. below).
What is currently not well supported in ReaR is that the user can
specify exactly what he wants to get included in disklayout.conf
during "rear mkrescue/mkbackup", see
https://github.com/rear/rear/issues/2229
In general regarding system migration with ReaR see
https://github.com/rear/rear/issues/2339#issuecomment-598067616
and follow the links therein in particular see
http://lists.relax-and-recover.org/pipermail/rear-users/2018-November/003626.html
and also follow the links therein in particular read through
https://github.com/rear/rear/issues/943
which is a bit older so some things are a bit different now
(e.g. RECOVERY_UPDATE is now called RECOVERY_UPDATE_URL)
but the general topics therein are still relevant, in particular see
https://github.com/rear/rear/issues/943#issuecomment-237544630
how certain directories are different in the ReaR recovery system
when you run usr/sbin/rear from within a sub-directory
e.g. when you try out a GitHub clone via
"Testing current ReaR upstream GitHub master code" in
https://en.opensuse.org/SDB:Disaster_Recovery
jsmeix commented at 2020-04-17 11:16:¶
Only as a side note FYI about my "future dream"
you may have a look at
https://github.com/rear/rear/issues/1085
that got a child BACKUP=YUM which already has grown up
bigger than its parent BACKUP=ZYPPER still is, see also
https://github.com/rear/rear/blob/master/doc/user-guide/14-ZYPPER-and-YUM.adoc
IT-Guy-1973 commented at 2020-04-17 12:43:¶
@hpannenb
Hello Holger,
Thank you.
Our systems would normally have a "rootvg" volume group and then data volume groups that include files systems related to applications and/or databases. All file systems are backed up by Networker. The goal is to recover any file or file system if needed.
As a system admin I may have to recover only the rootvg. So I will have a system where many disks are attached and the ISO image from REAR is mounted. I boot off the ISO image and when I run "rear recover" I should ONLY restore everything in rootvg and NOTHING in datavgs.
Can I use this parameter to do it with REAR?
ONLY_INCLUDE_VG=()
Networker will have all the data backed up as their job is to take care of any data on that system.
Regards,
Niranjan
jsmeix commented at 2020-04-17 13:08:¶
@IT-Guy-1973
FYI a general method how to get an idea for what a particular
ReaR config variable is actually used:
# find usr/share/rear -type f | xargs grep 'ONLY_INCLUDE_VG'
usr/share/rear/conf/default.conf:
# syntax : e.g. ONLY_INCLUDE_VG=( "vg00" "vg01" )
ONLY_INCLUDE_VG=()
usr/share/rear/conf/examples/borg-example.conf:
ONLY_INCLUDE_VG=( "vg00" )
usr/share/rear/conf/examples/RHEL7-ISO-Oracle-example.conf:
#ONLY_INCLUDE_VG=( "vg00" )
usr/share/rear/layout/save/default/310_include_exclude.sh:
if [ ${#ONLY_INCLUDE_VG[@]} -gt 0 ] ; then
usr/share/rear/layout/save/default/310_include_exclude.sh:
if ! IsInArray "${name#/dev/}" "${ONLY_INCLUDE_VG[@]}" ; then
usr/share/rear/layout/save/default/335_remove_excluded_multipath_vgs.sh:
# If AUTOEXCLUDE_MULTIPATH=n is used in combination with ONLY_INCLUDE_VG or
# to further investigate if EXCLUDE_VG or ONLY_INCLUDE_VG was respected for multipath devices:
So there is no code about ONLY_INCLUDE_VG
that is related to Legato Networker i.e. BACKUP=NSR
i.e. no file with /NSR/ in its path.
Accordingly it seems ONLY_INCLUDE_VG is only used
for what gets included in disklayout.conf
but what is included in the Legato Networker backup
seems to be totally unrelated to that - in particular because
there is no code in ReaR that makes a Legato Networker backup
(there is no usr/share/rear/backup/NSR/ directory).
Therefore I think if your Legato Networker backup contains
for example all files of all VGs or LVs of a system
but you use ONLY_INCLUDE_VG to only get the
"rootvg" volume group included in disklayout.conf then
I would assume you get by "rear recover" a disk layout
recreated that contains only the "rootvg" volume group and
when then the full-system Legato Networker backup gets restored
you get all the files restored into the "rootvg" volume group.
But that is only what I imagine off the top of my head.
I am not a Legato Networker user.
I don't use third-party backup tools (no time for that)
and in particular I cannot use proprietary backup tools.
IT-Guy-1973 commented at 2020-04-17 13:28:¶
Hello @jsmeix ,
Thank you for your reply above regarding the "ONLY_INCLUDE_VG". I was thinking of it in terms of restore. Like in - only include vgxx when restoring. But as you pointed out this parameter is not related to NSR.
Also, I am still going through your earlier responses to my question.
It looks like the options I have is to edit the disk layout files during restore. But let me carefully read all you posted here before I raise another question.
Thank you,
Regards,
Niranjan
IT-Guy-1973 commented at 2020-04-17 17:39:¶
Hello @jsmeix ,
When using Networker with REAR there is absolutely no way to control what is restored on the VM. With REAR you can select what disks you want etc ... But when the restore happens it is going to restore everything Networker backed up.
Example: I have a volume group on disk "/dev/sdb" named nmonvg and a file system in it name /nmon_data. If I disable the "/dev/sdb" using REAR the "/nmon_data" will anyway get restored on a directory under /.
@jsmeix Let me know if my understanding of this is correct.
Regards,
Niranjan
IT-Guy-1973 commented at 2020-04-18 20:10:¶
Hi @IT-Guy-1973.
Within Legato Networker configuration there are IMHO possibilities to exclude files/directories based on patterns and wildcards (IMHO named backup directives) or determine the backup filesystems in the saveset. So it is - as @gdha pointed to - always the responsibility of the EXTERNAL backup tool what files will be recovered to the system.
ReaR's part is to "prepare" the to be recovered system to its initial state (i.e. setup IPs, partitions, filesystems, start the required daemons etc) and thereafter starts i.e. the EXTERNAL recovery or waits for the EXTERNAL recovery to do its part. What is included or not is up to the EXTERNAL backup/recovery tool only.Cheers,
Holger.
Yes, you are absolutely correct. I tried to change what needs to be restored but that cannot be done. Even if I change the disk layout hoping it will not restore certain volume groups the restore happens on the rootvg as directories.
Regards,
Niranjan
hpannenb commented at 2020-04-19 07:58:¶
Hi, @IT-Guy-1973.
In my use case EMC Networker backups all data of a (single) server. The setup backups the dedicated filesystems /, /var/log, etc. instead of ALL. Furthermore it is possible to exclude all unwanted data i.e. the database files of a MySQL in /data/mysqldata/ or all logfiles in /var/log/ and so on.
When it comes to a disaster recovery ReaR prepares my (single) server with all required filesystems and afterwards EMC Networker is triggered to restore all the backuped files into them.
Using NSR_CLIENT_MODE=YES it is possible to i.e. execute EMC Networker's "recovery" tool manually to recover data of dedicated filessystems or manually select all the data required. NSR_CLIENT_MODE=YES waits until someone recovered the data with EMC Networker.
Regards,
Holger.
IT-Guy-1973 commented at 2020-04-19 17:08:¶
Hello @hpannenb ,
I added NSR_CLIENT_MODE=YES to the site.conf file. Now, as you mention correctly the restore stops at the "rear>" prompt.
At this point do I need to inform the backup team to initiate a restore from their end or can I run the "recover" command at my end to perform the restore?
I have 3 file systems on this system.
/, /boot and /nmon_data
I need to avoid installing /nmon_data. I only need / and /boot only.
On the restore system I see:
/mnt/local/
/mnt/local/boot/
file systems.
I am not very familiar with NSR commands. Please let me know what I need to do at this point.
Thank you,
Regards,
Niranjan
hpannenb commented at 2020-04-20 07:00:¶
Hi, @IT-Guy-1973
You can handle it as You prefer. Either execute the recovery on Your own or with the help of Your backup team.
Quiet some time ago I found the required EMC Networker commands at this page here. You can look for "recover".
With this and the knowledge from the ReaR sources in https://github.com/rear/rear/blob/c5f61051d017b20149041f9376d549e3d311af4c/usr/share/rear/restore/NSR/default/400_restore_with_nsr.sh#L26 you should be able to (batch) recover the required filesystems i.e.in Your case with
export VAR_DIR=/var/lib/rear/
recover -s $(cat $VAR_DIR/recovery/nsr_server) -c $(hostname) -d /mnt/local -a / /boot
I propose You check Your backup strategy if /nmon_data needs to be
backed up for each of the nodes or if it is of no use for a the generic
recovery. It might also be required the filesystem exists so check if
You really need to exclude it in the ReaR config or exclude it in the
external backup tool EMC Networker.
Sincerely,
Holger.
IT-Guy-1973 commented at 2020-04-23 04:23:¶
Hello @hpannenb ,
Thank you for the information. I have not tried this yet. But hopefully, I will try this tomorrow. Thank you for sharing the Networker command-line help page as well. Networker commands are pretty hard to assemble.
Regards,
Niranjan
IT-Guy-1973 commented at 2020-04-23 16:02:¶
Hello @hpannenb ,
I ran the commands you had provided. I had to manually add the hostname and the Networker server name. It worked. However, I did not see the boot image been created at the end. It is at the "rear>" prompt now after the restore.
Do I need to run any manual commands at this point to get the boot image created before I reboot the newly restored system? Please see the attachment.
@jsmeix @hpannenb
hpannenb commented at 2020-04-24 06:42:¶
Hi, @IT-Guy-1973.
You should have seen a prompt beforehand
Please let the restore process start on Your backup server i.e. $(cat $VAR_DIR/recovery/nsr_server).
Make sure all required data is restored to /mnt/local .
When the restore is finished type 'exit' to continue the recovery.
Info: You can check the recovery process i.e. with the command 'df'.
I suppose at the prompt You entered the "recover" command. When Your manual EMC recovery has been completed type "exit"(+ENTER) to continue the restore process.
Regards,
Holger.
IT-Guy-1973 commented at 2020-04-29 14:21:¶
@hpannenb Thank you for your valuable input. It all worked perfectly! I did restore multiple times to get a feel of the process.
Thank you again,
Regards,
Niranjan
jsmeix commented at 2020-04-29 14:25:¶
@IT-Guy-1973
can I conclude from your
https://github.com/rear/rear/issues/2367#issuecomment-621243091
that this issue which is primarily about NETWORKER backup usage
is now sufficiently solved for you?
IT-Guy-1973 commented at 2020-04-30 02:01:¶
@jsmeix Yes sir, you can. I also like to thank you for providing me with answers to the questions that I have raised with this request as well as other requests in the past.
Regards,
Niranjan
jsmeix commented at 2020-04-30 08:52:¶
@IT-Guy-1973
thank you for your positive feedback!
[Export of Github issue for rear/rear.]