#3425 Issue `open`: Unable to backup with RAID¶

Labels: support / question, fixed / solved / done, minor bug

Immick opened issue at 2025-03-12 19:45:¶

ReaR version¶

Relax-and-Recover 2.9-git.5586.e89f78ab.release / 2025-01-31

Describe the ReaR bug in detail¶

2025-03-12 22:34:58.831486403 Relax-and-Recover 2.9-git.5586.e89f78ab.release / 2025-01-31
2025-03-12 22:34:58.835054220 Running rear mkbackup (PID 3861 date 2025-03-12 22:34:57)
2025-03-12 22:34:58.838908429 Using log file: /var/log/rear/rear-afmgtu.log
2025-03-12 22:34:58.857716709 Including conf/Linux-i386.conf
2025-03-12 22:34:58.874181233 Including conf/GNU/Linux.conf
2025-03-12 22:34:58.995835400 Including /etc/rear/site.conf
2025-03-12 22:34:59.006699171 Including /etc/rear/local.conf
2025-03-12 22:34:59.011674408 ======================
2025-03-12 22:34:59.015360647 Running 'init' stage
2025-03-12 22:34:59.019024800 ======================
2025-03-12 22:34:59.043857152 Including init/default/001_verify_config_arrays.sh
2025-03-12 22:34:59.475238055 Including init/default/002_check_rear_recover_mode.sh
2025-03-12 22:34:59.477669302 Running workflow mkbackup on the normal/original system
2025-03-12 22:34:59.480993211 Including init/default/010_EFISTUB_check.sh
2025-03-12 22:34:59.484488661 Including init/default/010_set_drlm_env.sh
2025-03-12 22:34:59.487989602 Including init/default/030_update_recovery_system.sh
2025-03-12 22:34:59.491454375 Including init/default/100_check_stale_nfs_mounts.sh
2025-03-12 22:34:59.498571765 Including init/default/950_check_missing_programs.sh
2025-03-12 22:34:59.504733292 Including init/default/998_dump_variables.sh
2025-03-12 22:34:59.519243864 Finished running 'init' stage in 0 seconds
2025-03-12 22:34:59.522579645 Running mkbackup workflow
2025-03-12 22:34:59.525693854 ======================
2025-03-12 22:34:59.527457157 Running 'prep' stage
2025-03-12 22:34:59.529145490 ======================
2025-03-12 22:34:59.586875522 Including prep/default/005_remove_workflow_conf.sh
2025-03-12 22:34:59.598081382 Including prep/default/020_translate_url.sh
2025-03-12 22:34:59.603781636 Including prep/default/030_translate_tape.sh
2025-03-12 22:34:59.611629898 Including prep/default/035_valid_backup_methods.sh
2025-03-12 22:34:59.617219610 Including prep/default/036_valid_output_methods.sh
2025-03-12 22:34:59.622839258 Including prep/default/040_check_backup_and_output_scheme.sh
2025-03-12 22:34:59.632750676 Including prep/NETFS/default/040_inspect_configuration_files.sh
2025-03-12 22:34:59.638248972 Including prep/NETFS/default/050_check_NETFS_requirements.sh
2025-03-12 22:34:59.644780219 Skipping 'ping' test for host '' in BACKUP_URL 'iso:///backup'
2025-03-12 22:34:59.652481650 Including prep/default/050_check_keep_old_output_copy_var.sh
2025-03-12 22:34:59.657880881 Including prep/NETFS/default/060_mount_NETFS_path.sh
2025-03-12 22:34:59.664931730 Including prep/NETFS/default/070_set_backup_archive.sh
2025-03-12 22:34:59.671461704 Using backup archive '/var/tmp/rear.847PuuWOmYgyCOY/tmp/isofs/backup/backup.tar.gz'
2025-03-12 22:34:59.676564494 Including prep/NETFS/default/090_check_encrypted_backup.sh
2025-03-12 22:34:59.681480525 Including prep/NETFS/default/100_check_nfs_version.sh
2025-03-12 22:34:59.725513902 Including prep/default/100_init_workflow_conf.sh
2025-03-12 22:34:59.732131708 Including prep/NETFS/default/150_save_rsync_version.sh
2025-03-12 22:34:59.739864977 Including prep/GNU/Linux/200_include_getty.sh
2025-03-12 22:34:59.777241684 Including prep/GNU/Linux/200_include_serial_console.sh
2025-03-12 22:34:59.786292396 Including prep/NETFS/GNU/Linux/200_selinux_in_use.sh
2025-03-12 22:34:59.797990966 Including prep/NETFS/GNU/Linux/205_inspect_tar_capabilities.sh
2025-03-12 22:34:59.820466654 Including prep/GNU/Linux/210_include_dhclient.sh
2025-03-12 22:34:59.936713512 No DHCP client could be auto-detected: DHCP will not be enabled on the rescue system
2025-03-12 22:34:59.944585783 Including prep/GNU/Linux/220_include_lvm_tools.sh
2025-03-12 22:34:59.950075290 Device mapper found enabled. Including LVM tools.
2025-03-12 22:35:00.574372062 Including prep/GNU/Linux/230_include_md_tools.sh
2025-03-12 22:35:00.580174165 Software RAID detected. Including mdadm tools.
2025-03-12 22:35:00.587334109 Including prep/GNU/Linux/240_include_multipath_tools.sh
2025-03-12 22:35:00.594186861 Including prep/GNU/Linux/280_include_systemd.sh
2025-03-12 22:35:00.612797505 Including systemd (init replacement) tool-set to bootstrap Relax-and-Recover
2025-03-12 22:35:00.619348820 Including prep/GNU/Linux/280_include_virtualbox.sh
2025-03-12 22:35:00.631941298 Including prep/GNU/Linux/280_include_vmware_tools.sh
2025-03-12 22:35:00.643705495 Including prep/GNU/Linux/290_include_drbd.sh
2025-03-12 22:35:00.654983978 Including prep/GNU/Linux/300_check_backup_and_output_url.sh
2025-03-12 22:35:00.667990210 Including prep/ISO/default/300_check_iso_dir.sh
2025-03-12 22:35:00.673773753 Including prep/GNU/Linux/300_include_grub_tools.sh
2025-03-12 22:35:00.703184363 Including prep/GNU/Linux/310_include_cap_utils.sh
2025-03-12 22:35:00.710186212 Including prep/ISO/default/320_check_cdrom_size.sh
2025-03-12 22:35:00.715783771 ISO Directory '/var/lib/rear/output' [/dev/md0] has 3970 MB free space
2025-03-12 22:35:00.722667568 Including prep/default/320_include_uefi_env.sh
2025-03-12 22:35:00.744832030 Including prep/ISO/GNU/Linux/320_verify_mkisofs.sh
2025-03-12 22:35:00.810127290 Including prep/default/321_EFISTUB_check_uefi_env.sh
2025-03-12 22:35:00.817728806 Including prep/ISO/Linux-i386/330_find_isolinux.sh
2025-03-12 22:35:01.348359419 Including prep/default/330_include_uefi_tools.sh
2025-03-12 22:35:01.355725300 Including prep/Linux-i386/330_set_efi_arch.sh
2025-03-12 22:35:01.362765306 Including prep/ISO/GNU/Linux/340_add_isofs_module.sh
2025-03-12 22:35:01.380052248 Including prep/default/340_include_password_tools.sh
2025-03-12 22:35:01.386652045 Including prep/ISO/GNU/Linux/360_EFISTUB_prechecks.sh
2025-03-12 22:35:01.392894358 Including prep/Linux-i386/370_detect_secure_boot.sh
2025-03-12 22:35:01.399497128 Including prep/default/380_include_opal_tools.sh
2025-03-12 22:35:01.405668418 Including prep/default/390_include_python.sh
2025-03-12 22:35:01.411755199 Including prep/NETFS/default/400_automatic_exclude_recreate.sh
2025-03-12 22:35:01.420842368 Including prep/GNU/Linux/400_guess_kernel.sh
2025-03-12 22:35:01.425251309 Using autodetected kernel '/boot/vmlinuz-4.2.5-300.fc23.i686+PAE' as kernel in the recovery system
2025-03-12 22:35:01.430979195 Including prep/default/400_save_directories.sh
2025-03-12 22:35:01.434478682 Creating recovery directory (when not existing)
2025-03-12 22:35:01.544918618 FHS directory /etc/xml does not exist
2025-03-12 22:35:01.617187700 FHS directory /usr/X11R6 does not exist
2025-03-12 22:35:01.656439130 Including prep/default/490_store_write_protect_settings.sh
2025-03-12 22:35:01.662535023 Including prep/GNU/Linux/500_EFISTUB_check_kernel.sh
2025-03-12 22:35:01.679083866 Including prep/NETFS/default/980_umount_NETFS_dir.sh
2025-03-12 22:35:01.691158757 Including prep/default/989_check_for_restore_esp.sh
2025-03-12 22:35:01.697281014 Including prep/default/990_verify_empty_rootfs.sh
2025-03-12 22:35:01.706399872 Finished running 'prep' stage in 2 seconds
2025-03-12 22:35:01.709779229 ======================
2025-03-12 22:35:01.712871898 Running 'layout/save' stage
2025-03-12 22:35:01.716005711 ======================
2025-03-12 22:35:01.741568206 Including layout/save/GNU/Linux/100_create_layout_file.sh
2025-03-12 22:35:01.745197981 Creating disk layout
2025-03-12 22:35:01.748614963 Creating layout directories (when not existing)
2025-03-12 22:35:01.757609324 Overwriting existing disk layout file /var/lib/rear/layout/disklayout.conf
2025-03-12 22:35:01.927650723 Including layout/save/GNU/Linux/150_save_diskbyid_mappings.sh
2025-03-12 22:35:02.424291105 Saved diskbyid_mappings
2025-03-12 22:35:02.430097030 Including layout/save/GNU/Linux/190_opaldisk_layout.sh
2025-03-12 22:35:02.436650394 Including layout/save/GNU/Linux/200_partition_layout.sh
2025-03-12 22:35:02.455779780 Saving disks and their partitions
2025-03-12 22:35:05.385162631 Including layout/save/GNU/Linux/210_raid_layout.sh
2025-03-12 22:35:05.392564013 Saving Software RAID configuration
2025-03-12 22:35:06.752245580 Some messages from /var/tmp/rear.847PuuWOmYgyCOY/tmp/rear.mkbackup.stdout_stderr since the last called script 210_raid_layout.sh:
  /usr/share/rear/layout/save/GNU/Linux/210_raid_layout.sh: line 298: test: -gt: unary operator expected
  /usr/share/rear/layout/save/GNU/Linux/210_raid_layout.sh: line 298: test: -gt: unary operator expected
2025-03-12 22:35:06.859386750 ERROR: RAID /dev/md3 level '' is not a single word
2025-03-12 22:35:06.950913182 Error exit of rear mkbackup (PID 3861) and its descendant processes
2025-03-12 22:35:06.990065094 rear,3861 /usr/sbin/rear mkbackup
                                `-rear,7253 /usr/sbin/rear mkbackup
                                    `-rear,7438 /usr/sbin/rear mkbackup
                                        `-pstree,7439 -Aplau 3861
2025-03-12 22:35:08.049543932 Exiting subshell 1 (where the actual error happened)
2025-03-12 22:35:08.056257306 Exiting rear mkbackup (PID 3861) and its descendant processes ...
2025-03-12 22:35:08.086511033 rear,3861 /usr/sbin/rear mkbackup
                                `-rear,7468 /usr/sbin/rear mkbackup
                                    `-pstree,7469 -Aplau 3861
2025-03-12 22:35:08.119383483 Running exit tasks
2025-03-12 22:35:08.126009074 Finished rear mkbackup in 11 seconds
2025-03-12 22:35:08.130422080 Removing build area /var/tmp/rear.847PuuWOmYgyCOY
2025-03-12 22:35:08.243409636 End of program 'rear' reached

Platform¶

Linux i686

OS version¶

NAME=Fedora VERSION="23 (Twenty Three)" ID=fedora VERSION_ID=23 PRETTY_NAME="Fedora 23 (Twenty Three)"

Backup¶

NETFS

Storage layout¶

NAME    MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda       8:0    0   2.7T  0 disk
├─sda1    8:1    0    10M  0 part
├─sda2    8:2    0   200M  0 part
├─sda3    8:3    0    25G  0 part
│ └─md0   9:0    0    50G  0 raid5 /
├─sda4    8:4    0   500M  0 part
├─sda5    8:5    0    30G  0 part
│ └─md1   9:1    0    60G  0 raid5 /var/log
├─sda6    8:6    0    75G  0 part
│ └─md2   9:2    0 149.9G  0 raid5 /backup
├─sda7    8:7    0   200G  0 part
│ └─md3   9:3    0 399.8G  0 raid5 /var/lib/mysql
├─sda8    8:8    0   900G  0 part
│ └─md4   9:4    0   1.8T  0 raid5 /home
└─sda9    8:9    0   1.5T  0 part
  └─md5   9:5    0   3.1T  0 raid5 /var/samba
sdb       8:16   0   2.7T  0 disk
├─sdb1    8:17   0    10M  0 part
├─sdb2    8:18   0   200M  0 part
├─sdb3    8:19   0    25G  0 part
│ └─md0   9:0    0    50G  0 raid5 /
├─sdb4    8:20   0   500M  0 part
├─sdb5    8:21   0    30G  0 part
│ └─md1   9:1    0    60G  0 raid5 /var/log
├─sdb6    8:22   0    75G  0 part
│ └─md2   9:2    0 149.9G  0 raid5 /backup
├─sdb7    8:23   0   200G  0 part
│ └─md3   9:3    0 399.8G  0 raid5 /var/lib/mysql
├─sdb8    8:24   0   900G  0 part
│ └─md4   9:4    0   1.8T  0 raid5 /home
└─sdb9    8:25   0   1.5T  0 part
  └─md5   9:5    0   3.1T  0 raid5 /var/samba
sdc       8:32   0   2.7T  0 disk
├─sdc1    8:33   0    10M  0 part
├─sdc2    8:34   0   200M  0 part  /boot
├─sdc3    8:35   0    25G  0 part
│ └─md0   9:0    0    50G  0 raid5 /
├─sdc4    8:36   0   500M  0 part
├─sdc5    8:37   0    30G  0 part
│ └─md1   9:1    0    60G  0 raid5 /var/log
├─sdc6    8:38   0    75G  0 part
│ └─md2   9:2    0 149.9G  0 raid5 /backup
├─sdc7    8:39   0   200G  0 part
│ └─md3   9:3    0 399.8G  0 raid5 /var/lib/mysql
├─sdc8    8:40   0   900G  0 part
│ └─md4   9:4    0   1.8T  0 raid5 /home
└─sdc9    8:41   0   1.5T  0 part
  └─md5   9:5    0   3.1T  0 raid5 /var/samba

What steps will reproduce the bug?¶

rear mkbackup

Workaround, if any¶

Additional information¶

The current repository of my OS provides an old version:

# dnf info rear
Last metadata expiration check: 1:09:17 ago on Wed Mar 12 21:45:57 2025.
Available Packages
Name        : rear
Arch        : i686
Epoch       : 0
Version     : 1.19
Release     : 1.fc23
Size        : 415 k
Repo        : updates

I'm not sure 1.19 will work correctly so I decided to build my own package with ver 2.9 but it has some problems with detecting /dev/md3. My RAID has no problem by the way.

Also I have a question. Why my site.conf file is ignoring:

# cat /etc/rear/site.conf
OUTPUT=ISO
OUTPUT_URL=cifs://192.168.33.85/Server.Backup/
OUTPUT_OPTIONS="username=ksc,password=pass"
BACKUP=NETFS
BACKUP_URL=iso:///backup
BACKUP_OPTIONS="username=ksc,password=pass"

According to the log it creates a local backup archive in '/var/tmp/rear.847PuuWOmYgyCOY/tmp/isofs/backup/backup.tar.gz' instead of writing to smb-share =/

Thank you!

P.S. And I'm sorry, I set wrong label... It should be "support / question" I guess =/
And I know my OS is too old but before upgrading it I'm trying to backup everyting via ReaR but got failed :(

gdha commented at 2025-03-13 15:36:¶

@Immick Run 'rear -d savelayout' and attach the log file as it contains the debugging output.

Immick commented at 2025-03-13 21:58:¶

@gdha
rear-afmgtu.log

gdha commented at 2025-03-14 07:50:¶

@Immick

According to the log it creates a local backup archive in '/var/tmp/rear.847PuuWOmYgyCOY/tmp/isofs/backup/backup.tar.gz' instead of writing to smb-share =/

In the ReaR log you should see that it does mount the CIFS share.

Run 'rear -D savelayout' and attach the log file as it contains the script debugging out as I would like to see why:
/usr/share/rear/layout/save/GNU/Linux/210_raid_layout.sh: line 298: test: -gt: unary operator expected
fails?
Perhaps also attach the /var/lib/rear/layout/disklayout.conf file to view the result.

Immick commented at 2025-03-14 13:35:¶

@gdha

# rear -D savelayout
Relax-and-Recover 2.9-git.5586.e89f78ab.release / 2025-01-31
Running rear savelayout (PID 15440 date 2025-03-14 16:27:49)
Command line options: /usr/sbin/rear -D savelayout
Using log file: /var/log/rear/rear-afmgtu.log
Using build area: /var/tmp/rear.l7ROxGqzZGx7Ul6
Setting TMPDIR to ReaR's '/var/tmp/rear.l7ROxGqzZGx7Ul6/tmp' (was unset when ReaR was launched)
Running 'init' stage ======================
Running workflow savelayout on the normal/original system
Running 'layout/save' stage ======================
Creating disk layout
Disabling excluded components in /var/lib/rear/layout/disklayout.conf
GRUB found in first bytes on /dev/sdc and GRUB 2 is installed, using GRUB2 as a guessed bootloader for 'rear recover'
Skip saving storage layout as 'barrel' devicegraph (no 'barrel' command)
Verifying that the entries in /var/lib/rear/layout/disklayout.conf are correct
Created disk layout (check the results in /var/lib/rear/layout/disklayout.conf)
Exiting rear savelayout (PID 15440) and its descendant processes ...
Running exit tasks
To remove the build area you may use (with caution): rm -Rf --one-file-system /var/tmp/rear.l7ROxGqzZGx7Ul6

files.zip

In the ReaR log you should see that it does mount the CIFS share

I see some CIFS references in log file but don't understand does it actually mount or not...

gdha commented at 2025-03-14 14:13:¶

@Immick Hum, seems the error is coming from the line:
container=$( grep "Container" $mdadm_details | tr -d " " | cut -d ":" -f "2" | cut -d "," -f "1")
further in the script the line test $container_size -gt 0 && raid_layout_entry+=" size=$container_size" will give an error:
210_raid_layout.sh: line 298: test: -gt: unary operator expected

Your mdadm output doesn't contain a Container line, but that is a minor issue, which doesn't has an impact of the disklayout.conf file (this one looks okay to me).

Be aware that rear savelayout doesn't require CIFS as it writes it localy into /var/lib/rear/layout directory.

If you run rear -v mkbackup it should write the ISO image to the CIFS share. Unless it has an issue with the account used for CIFS, but that should be visible in the rear logfile.

Immick commented at 2025-03-14 14:41:¶

@gdha
Thank you so much for observing my logs!

If you run rear -v mkbackup it should write the ISO image to the CIFS share. Unless it has an issue with the account used for CIFS, but that should be visible in the rear logfile

I started rear -v mkbackup and interrupted it as it still makes backup on /var/tmp/. And I see no CIFS-references in the rear log file :(
Could you look into it please?
rear-afmgtu.log
What am I doing wrong?

And one more thing I don't understand. Does this backup mean that I needed the exact drives structure to restore my backup? I mean, this OS is on 3x3TB disks with RAID 5, usable storage is 6 TB. Will I need also 3x3TB disks to restore or can I have only one disk with 6TB on it? Actually I need to migrate that OS to one 6 TB (regardless of original RAID) disk for experiments with further OS upgrade. Is it possible with ReaR?

gdha commented at 2025-03-17 07:17:¶

@gdha Thank you so much for observing my logs!

If you run rear -v mkbackup it should write the ISO image to the CIFS share. Unless it has an issue with the account used for CIFS, but that should be visible in the rear logfile

I started rear -v mkbackup and interrupted it as it still makes backup on /var/tmp/. And I see no CIFS-references in the rear log file :( Could you look into it please? rear-afmgtu.log What am I doing wrong?

when you see 2025-03-14 17:27:35.901589753 Using backup archive '/var/tmp/rear.KmGf8ZyKq9ik2h8/tmp/isofs/backup/backup.tar.gz' then it means it mounted the CIFS share. Do'nt you see anything appearing on the share itself?

And one more thing I don't understand. Does this backup mean that I needed the exact drives structure to restore my backup? I mean, this OS is on 3x3TB disks with RAID 5, usable storage is 6 TB. Will I need also 3x3TB disks to restore or can I have only one disk with 6TB on it? Actually I need to migrate that OS to one 6 TB (regardless of original RAID) disk for experiments with further OS upgrade. Is it possible with ReaR?

Rear is a DR solution to recover on the same or similar HW. Therefore, going from a 3x3TB to 1 6TB disk is something that ReaR will not cover for you. However, what you can do is install on OS on the new system and create the exact same mount points and then extract the tar file created by ReaR (or only the mount points you need) onto the new system.

Immick commented at 2025-03-17 13:58:¶

then it means it mounted the CIFS share. Do'nt you see anything appearing on the share itself?

Yep, nothing is appearing on the cifs-share. It writes /var/tmp/... decresing free space on / mount but I don't have such space on / to get that backup =/

gdha commented at 2025-03-17 14:18:¶

@Immick Did you ever try to manual mount the CIFS share on your Linux system?

Immick commented at 2025-03-17 15:18:¶

@gdha Yes, I can succesfully mount this share via:
mount -t cifs //192.168.33.85/AFMGTU.Server.Backup /mnt/cifs -o username=KSC,password=123456,vers=2.1
And able to write files here.

So changed my site.conf to:

OUTPUT=ISO
OUTPUT_URL=cifs://192.168.33.85/AFMGTU.Server.Backup/
OUTPUT_OPTIONS="username=ksc,password=123456,vers=2.1"
BACKUP=NETFS
BACKUP_URL=iso:///backup
BACKUP_OPTIONS="username=ksc,password=123456,vers=2.1"

Is my configuration file correct?

But it still writes to /. And I don't understand why nothing can be found in the log file regarding issues with mounting my CIFS share:
rear-afmgtu.zip

+ source /usr/share/rear/prep/NETFS/default/060_mount_NETFS_path.sh
++ [[ -n '' ]]
++ mount_url iso:///backup /var/tmp/rear.O2O6PgG97EHS6yv/outputfs username=ksc,password=123456,vers=2.1
++ local url=iso:///backup
++ local mountpoint=/var/tmp/rear.O2O6PgG97EHS6yv/outputfs
++ local defaultoptions=rw,noatime
++ local options=username=ksc,password=123456,vers=2.1
++ local scheme
+++ url_scheme iso:///backup
+++ local url=iso:///backup
+++ local scheme=iso
+++ echo iso
+++ grep -q :
+++ echo iso
++ scheme=iso
++ scheme_supports_filesystem iso
++ local scheme=iso
++ test iso
++ case "$scheme" in
++ return 0
++ local mount_cmd
++ case "$scheme" in
++ test recover = mkbackup
++ return 0
+ source_return_code=0

As can I see, no issues here, right?

Is there a way to debug this mounting somehow manually?

gdha commented at 2025-03-17 15:22:¶

@Immick try commenting these 2 lines:

OUTPUT_URL=cifs://192.168.33.85/AFMGTU.Server.Backup/
OUTPUT_OPTIONS="username=ksc,password=123456,vers=2.1"

and see what happens next?

Or, as iso: means creating local ISO image (including backups). you could also change the OUTPUT lines into BACKUP lines and comment the BACKUP_URL line.

Immick commented at 2025-03-17 17:08:¶