#2752 Issue `closed`: ReaR backup error¶

Labels: support / question, fixed / solved / done

exfarmer opened issue at 2022-02-03 03:18:¶

Relax-and-Recover (ReaR) Issue Template¶

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

ReaR version ("/usr/sbin/rear -V"): Relax-and-Recover 2.6 / 2020-06-17
OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"): Red Hat Enterprise Linux Server release 6.10 (Santiago)
ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
OUTPUT=ISO
OUTPUT_URL=nfs://ussydd6300/data/col1/sylmar_linux_bu_image
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://ussydd6300/data/col1/sylmar_linux_bu_image
ONLY_INCLUDE_VG=( 'VolGroup00' 'vg01' )
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/media' '/var/tmp' '/var/crash' '/srv')
NETFS_KEEP_OLD_BACKUP_COPY=y
Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR):
VM
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
x86_64
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
SAN
Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"):
Description of the issue (ideally so that others can reproduce it):
Info 02/02/2022 21:35:11 Relax-and-Recover 2.6 / 2020-06-17
Info 02/02/2022 21:35:11 Running rear mkbackup (PID 7507)
Info 02/02/2022 21:35:11 Using log file: /var/log/rear/rear-ussy-messtgora20.log
Error 02/02/2022 21:35:12 Script 'default/01_set_drlm_env.sh' without leading 3-digit number 'NNN_' is likely run in wrong order
Info 02/02/2022 21:35:12 Running workflow mkbackup on the normal/original system
Error 02/02/2022 21:35:12 Script 'GNU/Linux/20_include_agetty.sh' without leading 3-digit number 'NNN_' is likely run in wrong order
Error 02/02/2022 21:35:12 speed 9600 baud; line = 0;
Error 02/02/2022 21:35:12 -brkint -imaxbel
Error 02/02/2022 21:35:12 speed 9600 baud; line = 0;
Error 02/02/2022 21:35:12 -brkint -imaxbel
Error 02/02/2022 21:35:12 stty: /dev/ttyS2: Input/output error
Error 02/02/2022 21:35:12 stty: /dev/ttyS3: Input/output error
Error 02/02/2022 21:35:12 Script 'GNU/Linux/21_include_dhclient.sh' without leading 3-digit number 'NNN_' is likely run in wrong order
Error 02/02/2022 21:35:12 Script 'GNU/Linux/22_include_lvm_tools.sh' without leading 3-digit number 'NNN_' is likely run in wrong order
Error 02/02/2022 21:35:12 Script 'GNU/Linux/23_include_md_tools.sh' without leading 3-digit number 'NNN_' is likely run in wrong order
Error 02/02/2022 21:35:12 Script 'GNU/Linux/24_include_multipath_tools.sh' without leading 3-digit number 'NNN_' is likely run in wrong order
Error 02/02/2022 21:35:13 Script 'GNU/Linux/28_include_systemd.sh' without leading 3-digit number 'NNN_' is likely run in wrong order
Error 02/02/2022 21:35:13 Script 'GNU/Linux/28_include_vmware_tools.sh' without leading 3-digit number 'NNN_' is likely run in wrong order
...
...
...
Error 02/02/2022 21:40:01 /usr/share/rear/verify/NSR/default/
Error 02/02/2022 21:40:01 /usr/share/rear/verify/NSR/default/410_verify_nsr_paths.sh
Error 02/02/2022 21:40:01 /usr/share/rear/verify/NSR/default/400_verify_nsr.sh
Error 02/02/2022 21:40:01 /usr/share/rear/verify/NSR/default/41_verify_nsr_paths.sh
Error 02/02/2022 21:40:01 /usr/share/rear/verify/NSR/default/40_verify_nsr.sh
Error 02/02/2022 21:40:01 /usr/share/rear/verify/default/
Error 02/02/2022 21:40:01 /usr/share/rear/verify/default/04_validate_variables.sh
Error 02/02/2022 21:40:01 /usr/share/rear/verify/default/03_translate_tape.sh
Error 02/02/2022 21:40:01 /usr/share/rear/verify/default/020_translate_url.sh
Error 02/02/2022 21:40:01 /usr/share/rear/verify/default/020_cciss_scsi_engage.sh
Error 02/02/2022 21:40:01 /usr/share/rear/verify/default/02_translate_url.sh
Error 02/02/2022 21:40:01 /usr/share/rear/verify/default/040_validate_variables.sh
Error 02/02/2022 21:40:01 /usr/share/rear/verify/default/050_create_mappings_dir.sh
Error 02/02/2022 21:40:01 /usr/share/rear/verify/default/02_cciss_scsi_engage.sh
Error 02/02/2022 21:40:01 /usr/share/rear/verify/default/030_translate_tape.sh
Error 02/02/2022 21:40:01 /usr/share/rear/verify/GALAXY10/
Error 02/02/2022 21:40:01 /usr/share/rear/verify/GALAXY10/default/
Error 02/02/2022 21:40:01 /usr/share/rear/verify/GALAXY10/default/550_request_point_in_time_restore_parameters.sh
Error 02/02/2022 21:40:01 /usr/share/rear/verify/GALAXY10/default/39_create_ramdisk.sh
Info 02/02/2022 21:40:01 You have reached logging limit
Workaround, if any: NONE
Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):

To paste verbatim text like command output or file content,
include it between a leading and a closing line of three backticks like

```
verbatim content
```

rear-ussy-messtgora20.log

gdha commented at 2022-02-03 06:47:¶

@exfarmer you are mixing two ReaR versions I'm afraid. To fix this remove the current rear package and then remove the directory /usr/share/rear, then you can re-install the rear package. It will work better by doing this.

jsmeix commented at 2022-02-03 08:44:¶

@exfarmer
alternatively see the section
"Testing current ReaR upstream GitHub master code" in
https://en.opensuse.org/SDB:Disaster_Recovery

I recommend to try out our latest GitHub master code because
the GitHub master code is the only place where we fix things
and if there are issues it helps when you use exactly the code
where we could fix things.

exfarmer commented at 2022-02-03 13:06:¶

I didn’t notice the 2 versions installed. I removed them both and re-installed the 2.6 version.
Looks like it is working again.
Thank you

What’s odd is that we made no changes on this server in over a year and it just started failing.
It’s working, I’m happy, Thank you again

Abbott
Gary Hess
Administrator, Sr - Unix
Abbott

253 Financial Blvd.
Liberty, SC 29657 USA

O:
+1 864-843-8352
M:
+1 864-546-8921

exfarmer commented at 2022-02-03 13:08:¶

I spoke too soon, the last run also failed:

2022-02-03 05:05:50.219788567 Including build/GNU/Linux/620_verify_os_release_file.sh 2022-02-03 05:05:50.257947938 Including build/GNU/Linux/630_simplify_systemd_reboot_halt_poweroff_shutdown.sh 2022-02-03 05:05:50.296919756 Including build/GNU/Linux/630_verify_resolv_conf_file.sh 2022-02-03 05:05:50.337276462 Supposedly valid nameserver '10.94.231.24' in /tmp/rear.PjwQLI3KhCMZyog/rootfs/etc/resolv.conf 2022-02-03 05:05:50.371171479 Including build/GNU/Linux/640_verify_lvm_conf.sh 2022-02-03 05:05:50.469609420 Including build/default/950_check_missing_programs.sh 2022-02-03 05:05:50.532596894 Including build/default/960_remove_encryption_keys.sh 2022-02-03 05:05:50.570725235 Including build/default/970_add_rear_release.sh 2022-02-03 05:05:50.610752041 Including build/default/975_update_os_conf.sh 2022-02-03 05:05:50.651349708 Including build/default/990_verify_rootfs.sh 2022-02-03 05:05:50.667381525 Testing that the recovery system in /tmp/rear.PjwQLI3KhCMZyog/rootfs contains a usable system 2022-02-03 05:05:50.731705432 Testing 'ldd /bin/bash' to ensure 'ldd' works for the subsequent 'ldd' tests within the recovery system chroot: failed to run command `/bin/ldd': Operation not permitted 2022-02-03 05:05:50.754237421 Build area kept for investigation in /tmp/rear.PjwQLI3KhCMZyog, remove it when not needed 2022-02-03 05:05:50.807535723 ERROR:¶

BUG in /usr/share/rear/build/default/990_verify_rootfs.sh line 57: 'ReaR recovery system in '/tmp/rear.PjwQLI3KhCMZyog/rootfs' is broken: 'ldd /bin/bash' failed'¶

Please report this issue at https://github.com/rear/rear/issues and include the relevant parts from /var/log/rear/rear-ussy-messtgora20.log preferably with full debug information via 'rear -D mkbackup'¶

===== Stack trace =====
Trace 0: /usr/sbin/rear:541 main
Trace 1: /usr/share/rear/lib/mkbackup-workflow.sh:15 WORKFLOW_mkbackup
Trace 2: /usr/share/rear/lib/framework-functions.sh:116 SourceStage
Trace 3: /usr/share/rear/lib/framework-functions.sh:56 Source
Trace 4: /usr/share/rear/build/default/990_verify_rootfs.sh:57 source
Trace 5: /usr/share/rear/lib/_input-output-functions.sh:708 BugError
=== End stack trace ===
2022-02-03 05:05:50.857382731 Exiting rear mkbackup (PID 48779) and its descendant processes ...
2022-02-03 05:05:54.331122101 rear,48779 /usr/sbin/rear -d -v mkbackup
-rear,21043 /usr/sbin/rear -d -v mkbackup-pstree,21044 -Aplau 48779
/usr/share/rear/lib/_input-output-functions.sh: line 151: kill: (21049) - No such process
2022-02-03 05:05:54.645582279 Running exit tasks
2022-02-03 05:05:54.657714555 Exit task 'cleanup_build_area_and_end_program'
2022-02-03 05:05:54.668878451 Finished in 186 seconds
2022-02-03 05:05:54.680255903 You should also rm -Rf /tmp/rear.PjwQLI3KhCMZyog
2022-02-03 05:05:54.691697218 End of program reached
2022-02-03 05:05:54.703186174 Exit task '(( EXIT_FAIL_MESSAGE )) && echo 'rear mkbackup failed, check /var/log/rear/rear-ussy-messtgora20.log for details' 1>&8'
2022-02-03 05:05:54.714727499 Exit task 'exec 8>&-'
2022-02-03 05:05:54.726234596 Exit task 'exec 7>&-'
2022-02-03 05:05:54.737597558 Exit task 'exec 6<&-'
2022-02-03 05:05:54.749008405 Exit task ''

Version: Relax-and-Recover 2.6 / 2020-06-17

Thank you
Abbott

pcahyna commented at 2022-02-03 13:12:¶

"chroot: failed to run command `/bin/ldd': Operation not permitted" that's weird, do you get some SELinux errors when this happens?

exfarmer commented at 2022-02-03 13:27:¶

Not that I know of, but I can ask someone in about an hour.

exfarmer commented at 2022-02-03 13:35:¶

I couldn’t find any messages containing “ldd” with the ausearch.

exfarmer commented at 2022-02-03 16:31:¶

Hi, Any idea where this is coming from?
Like I said, the backup has been working for weeks, now it fails.

exfarmer commented at 2022-02-03 20:07:¶

I found the issue, it was flagged by Carbon Black.
I’ll follow up once that is resolved.

exfarmer commented at 2022-02-03 21:37:¶

The issue with the CarbonBlack blocking the backups is fixed, but the issue I originally started with is not.

***@***.*** ~]# rear -V
Relax-and-Recover 2.6 / 2020-06-17

This is in the local.conf file: BACKUP_OPTIONS="nfsvers=3,nolock"

I can mount the NFS disk with “mount -v disk directory -t nfs -o vers=3”
but the mount command isn’t using that:
2022-02-03 13:27:20.222760885 Finished running 'build' stage in 133 seconds
2022-02-03 13:27:20.229017175 ======================
2022-02-03 13:27:20.235056067 Running 'pack' stage
2022-02-03 13:27:20.241021287 ======================
2022-02-03 13:27:20.275957757 Including pack/GNU/Linux/900_create_initramfs.sh
2022-02-03 13:27:20.299573307 Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
2022-02-03 13:27:41.934086644 Created initrd.cgz with gzip default compression (121700972 bytes) in 21 seconds
2022-02-03 13:27:41.944672771 Finished running 'pack' stage in 21 seconds
2022-02-03 13:27:41.953245896 ======================
2022-02-03 13:27:41.960995281 Running 'output' stage
2022-02-03 13:27:41.969726719 ======================
2022-02-03 13:27:42.013064627 Including output/default/010_set_umask.sh
2022-02-03 13:27:42.026260058 Setting umask to 077
2022-02-03 13:27:42.053480325 Including output/default/100_mount_output_path.sh
mkdir: created directory /tmp/rear.XOaHbxxa0dHb2MK/outputfs' 2022-02-03 13:27:42.076297839 Added 'rm -Rf -v /tmp/rear.XOaHbxxa0dHb2MK/outputfs >&2' as an exit task 2022-02-03 13:27:42.102478106 Mounting with 'mount -v -t nfs -o rw,noatime ussydd6300:/data/col1/sylmar_linux_bu_image /tmp/rear.XOaHbxxa0dHb2MK/outputfs' mount.nfs: mount(2): Permission denied mount.nfs: access denied by server while mounting ussydd6300:/data/col1/sylmar_linux_bu_image mount.nfs: timeout set for Thu Feb 3 13:29:42 2022 mount.nfs: trying text-based options 'vers=4,addr=10.94.101.36,clientaddr=10.94.30.85' 2022-02-03 13:27:44.529708643 ERROR: Mount command 'mount -v -t nfs -o rw,noatime ussydd6300:/data/col1/sylmar_linux_bu_image /tmp/rear.XOaHbxxa0dHb2MK/outputfs' failed. ===== Stack trace ===== Trace 0: /usr/sbin/rear:541 main Trace 1: /usr/share/rear/lib/mkbackup-workflow.sh:24 WORKFLOW_mkbackup Trace 2: /usr/share/rear/lib/framework-functions.sh:116 SourceStage Trace 3: /usr/share/rear/lib/framework-functions.sh:56 Source Trace 4: /usr/share/rear/output/default/100_mount_output_path.sh:15 source Trace 5: /usr/share/rear/lib/global-functions.sh:563 mount_url === End stack trace === 2022-02-03 13:27:44.563688165 Exiting rear mkbackup (PID 18120) and its descendant processes ... 2022-02-03 13:27:47.865810778 rear,18120 /usr/sbin/rear -d -v mkbackup-rear,48523 /usr/sbin/rear -d -v mkbackup
-pstree,48524 -Aplau 18120 /usr/share/rear/lib/_input-output-functions.sh: line 151: kill: (48527) - No such process 2022-02-03 13:27:48.218938568 Running exit tasks 2022-02-03 13:27:48.232315318 Exit task 'rm -Rf -v /tmp/rear.XOaHbxxa0dHb2MK/outputfs >&2' removed directory:/tmp/rear.XOaHbxxa0dHb2MK/outputfs'
2022-02-03 13:27:48.254961580 Exit task 'cleanup_build_area_and_end_program'
2022-02-03 13:27:48.267087127 Finished in 287 seconds
2022-02-03 13:27:48.279517964 You should also rm -Rf /tmp/rear.XOaHbxxa0dHb2MK
2022-02-03 13:27:48.292739083 End of program reached
2022-02-03 13:27:48.305676846 Exit task '(( EXIT_FAIL_MESSAGE )) && echo 'rear mkbackup failed, check /var/log/rear/rear-ussy-messtgora20.log for details' 1>&8'
2022-02-03 13:27:48.318083562 Exit task 'exec 8>&-'
2022-02-03 13:27:48.330518689 Exit task 'exec 7>&-'
2022-02-03 13:27:48.342243680 Exit task 'exec 6<&-'
2022-02-03 13:27:48.355757205 Exit task ''

I have a work-around, but I don’t want to hard code BACKUP_OPTIONS= and OUTPUT_OPTIONS= in /usr/share/rear/conf/default.conf
The whole point of this was to not do that.

Thank you
Abbott

pcahyna commented at 2022-02-03 22:26:¶

I found the issue, it was flagged by Carbon Black.

you already found that once half a year ago, #2624 :-)

The issue with the CarbonBlack blocking the backups is fixed, but the issue I originally started with is not.

you originally started with script numbering issue, now you have a nfs issue. I think you should file a separate issue here, as the original one seems to be solved.

This is in the local.conf file: BACKUP_OPTIONS="nfsvers=3,nolock" I can mount the NFS disk with “mount -v disk directory -t nfs -o vers=3” but the mount command isn’t using that

exfarmer commented at 2022-02-03 23:39:¶

I can open a new case if needed, thank you

Please close this one

jsmeix commented at 2022-02-04 13:16:¶

The subsequent issue is https://github.com/rear/rear/issues/2753
"When OUTPUT_URL is set OUTPUT_OPTIONS does not inherit BACKUP_OPTIONS"

[Export of Github issue for rear/rear.]

#2752 Issue closed: ReaR backup error¶