#2770 Issue closed: Ubuntu 20.04: mkrescue Fails w/ Permission Error¶
Labels: support / question, fixed / solved / done
paulra1 opened issue at 2022-03-11 04:17:¶
Relax-and-Recover (ReaR) Issue Template¶
Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):
-
ReaR version ("/usr/sbin/rear -V"): 2.5 / Git
-
OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"): Ubuntu 20.04 (LTS)
-
OS_VENDOR=Ubuntu
OS_VERSION=20.04 -
ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"): local.conf
OUTPUT=USB
BACKUP=NETFS
BACKUP_URL=usb:///dev/disk/by-label/REAR-000
USB_DEVICE=/dev/sda
USB_DEVICE_PARTED_LABEL=gpt
USB_DEVICE_FILESYSTEM=ext4
-
Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR): System75
-
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device): AMD64
-
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot): EFI
-
Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe): Local Disk SSD
-
Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"):
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
/dev/loop0 7:0 0 938.5M 1 loop /snap/android-studio/118
/dev/loop1 7:1 0 61.9M 1 loop /snap/core20/1361
/dev/loop2 7:2 0 55.5M 1 loop /snap/core18/2284
/dev/loop3 7:3 0 4K 1 loop /snap/bare/5
/dev/loop4 7:4 0 938.5M 1 loop /snap/android-studio/119
/dev/loop5 7:5 0 55M 1 loop /snap/core18/1880
/dev/loop6 7:6 0 248.8M 1 loop /snap/gnome-3-38-2004/99
/dev/loop7 7:7 0 62.1M 1 loop /snap/gtk-common-themes/1506
/dev/loop8 7:8 0 391.3M 1 loop /snap/gimp/383
/dev/loop9 7:9 0 219M 1 loop /snap/gnome-3-34-1804/77
/dev/loop10 7:10 0 49.8M 1 loop /snap/snap-store/467
/dev/loop11 7:11 0 255.6M 1 loop /snap/gnome-3-34-1804/36
/dev/loop12 7:12 0 61.9M 1 loop /snap/core20/1376
/dev/loop13 7:13 0 164.8M 1 loop /snap/gnome-3-28-1804/161
/dev/loop14 7:14 0 65.2M 1 loop /snap/gtk-common-themes/1519
/dev/loop15 7:15 0 43.6M 1 loop /snap/snapd/14978
/dev/loop16 7:16 0 247.9M 1 loop /snap/gnome-3-38-2004/87
/dev/loop17 7:17 0 54.2M 1 loop /snap/snap-store/558
/dev/sda 8:0 0 1.8T 0 disk
`-/dev/sda1 8:1 0 1.8T 0 part
/dev/nvme0n1 259:0 0 931.5G 0 disk
|-/dev/nvme0n1p1 259:1 0 512M 0 part /boot/efi
|-/dev/nvme0n1p2 259:2 0 927G 0 part /
`-/dev/nvme0n1p3 259:3 0 4G 0 part [SWAP]
-
Description of the issue (ideally so that others can reproduce it):
-
Permission Problen when copying /usr/bin/mount
-
Workaround, if any:
-
Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):
-
rear-mobius2.log
rear-mobius2.log
jsmeix commented at 2022-03-11 08:16:¶
Excerpts from
https://github.com/rear/rear/files/8229000/rear-mobius2.log
2022-03-10 19:32:41.830201530 Copying binaries and libraries
...
2022-03-10 19:32:42.132201976 Binaries being copied: ... /usr/bin/mount ...
cp: preserving permissions for '/tmp/rear.d6ugzJTqfE6vm1w/rootfs/bin/mount': Operation not permitted
2022-03-10 19:32:42.431725484 ERROR: Failed to copy '/usr/bin/mount' to '/tmp/rear.d6ugzJTqfE6vm1w/rootfs/bin'
I have no idea what the reason is why "Operation not permitted"
happens.
Because "rear mkrescue" is run as 'root' it is "something" that even
prohibits
'root' to do things. Usually such "something" is things like SELinux or
AppArmor.
The matching code is in
build/GNU/Linux/390_copy_binaries_libraries.sh
For ReaR 2.5 that code was
function copy_binaries () {
...
if ! cp $verbose --archive --dereference --force "$binary" "$destdir" 1>&2 ; then
Error "Failed to copy '$binary' to '$destdir'"
fi
...
}
In our current ReaR GitHub master code that code is meanwhile
function copy_binaries () {
...
if ! cp $verbose --archive --dereference --force "$binary" "$destdir" 1>&2 ; then
# When a binary should be copied where its target already exists as dangling symlink
# e.g. when /sbin/lvcreate should be copied to /tmp/rear.XXX/rootfs/bin/lvcreate
# but there is already the dangling symlink /tmp/rear.XXX/rootfs/bin/lvcreate -> lvm
# because its link target was not yet copied into the recovery system
# cf. "create LVM symlinks" in build/GNU/Linux/005_create_symlinks.sh
# then cp fails (regardless of the --force option) with an error message like
# cp: not writing through dangling symlink '/tmp/rear.XXX/rootfs/bin/lvcreate'
# so we silently skip cp errors here regardless what the reason is why cp failed here
# and add it to REQUIRED_PROGS to error out later if it is actually missing in the recovery system
# (for binaries in PROGS copy_binaries is only called when it exists in the original system)
# and show the issue in debug mode so the user could see why it errors out at REQUIRED_PROGS
# cf. https://github.com/rear/rear/pull/2643
DebugPrint "copy_binaries skipped '$binary' (it gets verified later via REQUIRED_PROGS)"
REQUIRED_PROGS+=( $( basename "$binary" ) )
fi
...
}
So we do no longer error out here when cp fails
because cp fails too often regardless of the --force option
and as far as I remember from my tests during
https://github.com/rear/rear/pull/2643
in such cases cp fails with the generic exit code 1
so one cannot distinguish fatal errors where cp cannot continue
from issues where cp could continue (in particular with --force).
@paulra1
please test if it works for you when you use
our current ReaR GitHub master code.
See the section
"Testing current ReaR upstream GitHub master code" in
https://en.opensuse.org/SDB:Disaster_Recovery
how you can try out our current ReaR GitHub master code
without conflicts with your already installed ReaR version.
In general:
I recommend to try out our latest GitHub master code because
the GitHub master code is the only place where we fix things
and if there are issues it helps when you use exactly the code
where we could fix things.
In general we at ReaR upstream do not support older ReaR versions.
We at ReaR upstream do not plain reject issues with older ReaR
versions
(e.g. we may answer easy to solve questions also for older ReaR
versions)
but we do not spend much time on issues with older ReaR versions
because
we do not (and cannot) fix issues in released ReaR versions.
Issues in released ReaR versions are not fixed by us (by ReaR
upstream).
Issues in released ReaR versions that got fixed in current ReaR
upstream
GitHub master code might be fixed (if the fix can be backported with
reasonable effort) by the Linux distributor wherefrom you got ReaR.
By the way:
In
https://github.com/rear/rear/files/8229000/rear-mobius2.log
there is at the beginning
2022-03-10 19:32:25.592829276 Relax-and-Recover 2.5 / Git
2022-03-10 19:32:25.593740935 Running rear udev (PID 9101)
2022-03-10 19:32:25.594647731 Command line options: /usr/sbin/rear udev
Running "rear mkrescue" via udev is problematic, see
https://github.com/rear/rear/issues/840
See also
https://github.com/rear/rear/issues/838
and
https://github.com/rear/rear/issues/2180
paulra1 commented at 2022-03-11 16:12:¶
Hi,
I obtaiined rear version 2.5 with apt from the current Ubuntu repositories which are as follows:
http://archive.ubuntu.com/ubuntu/
focal main restricted universe multiverse
http://security.ubuntu.com/ubuntu/
focal-security main restricted universe multiverse
http://archive.ubuntu.com/ubuntu/
focal-updates main restricted universe multiverse
The following Linux command was used.
sudo apt install rear extlinux
NOTE: extlinux is the bootloaders package.
I assume you need to uninstall the current version of rear before
installing the master version.
Also, where can I find the master version ?
In iissue #838 I noticed the following configration on local.conf.
USB_DEVICE=/dev/disk/by-label/REAR-000
In my local.conf it is set as follows which I believe is correct. Is this a problem ?
USB_DEVICE=/dev/sda
paulra1 commented at 2022-03-11 18:33:¶
Hi,
I downloaded the master branch of rear version 2.6 and created the .deb
file,
rear_2.6-git.0.0.master.changed_amd64.deb, as follows.
make deb
Then I installed it and ran rear as before and it failed. There are
several errors in the log
file which is attached to this message. The short version of the
relevant diagnostics
are as follows:
++ PrintError 'ERROR: Failed to copy initrd to /var/tmp/rear-efi.YTjBXHAvlN//EFI/BOOT/initrd.cgz'
++ Log 'ERROR: Failed to copy initrd to /var/tmp/rear-efi.YTjBXHAvlN//EFI/BOOT/initrd.cgz'
++ echo '2022-03-11 10:15:27.121827028 ERROR: Failed to copy initrd to /var/tmp/rear-efi.YTjBXHAvlN//EFI/BOOT/initrd.cgz'
2022-03-11 10:15:27.121827028 ERROR: Failed to copy initrd to /var/tmp/rear-efi.YTjBXHAvlN//EFI/BOOT/initrd.cgz
++ LogToSyslog 'ERROR: Failed to copy initrd to /var/tmp/rear-efi.YTjBXHAvlN//EFI/BOOT/initrd.cgz
++ logger -t rear -i 'ERROR: Failed to copy initrd to /var/tmp/rear-efi.YTjBXHAvlN//EFI/BOOT/initrd.cgz'
NOTE: When I ran "rear dump", the following diagnostics were displayed at the end of the output.
# Validation status:
# /usr/share/rear/lib/validated/Ubuntu/20.04/i386.txt : missing/empty
# Your system is not yet validated. Please carefully check all functions
# and create a validation record with 'rear validate'. This will help others
# to know about the validation status of Relax-and-Recover on this system.
# End of dump configuration and system information
paulra1 commented at 2022-03-11 18:54:¶
Hi,
I didn't notice that there is an obvious space problem on the external
USB disk. The relevant
diagnostics are as follows.
cp: error writing '/var/tmp/rear-efi.YTjBXHAvlN//EFI/BOOT/initrd.cgz':
No space left on device
cp: error writing
'''/var/tmp/rear-efi.YTjBXHAvlN//EFI/BOOT/initrd.cgz''': No space left
on device'
cp: error writing
'''/var/tmp/rear-efi.YTjBXHAvlN//EFI/BOOT/initrd.cgz''': No space left
on devicea'
When you execute "df -h", you will notice that /dev/sda2 has no space,
below. It is the
partition with the EFI sector and must have been created with "rear -v
format /dev/sda".
/dev/sda2 511M 511M 0 100% /var/tmp/rear-efi.YTjBXHAvlN
/dev/sdb3 1.8T 77M 1.7T 1% /media/paulr/REAR-000
Could this be fixed by modifying or adding something to the local.conf
configuration file ?
For convenience, this is my /etc/rear/local.conf configuration.
OUTPUT=USB
BACKUP=NETFS
BACKUP_URL=usb:///dev/disk/by-label/REAR-000
USB_DEVICE=/dev/sda
USB_DEVICE_PARTED_LABEL=gpt
USB_DEVICE_FILESYSTEM=ext4
paulra1 commented at 2022-03-11 19:58:¶
One more possibly relevant detail. The external USB hard driive model
is
a Segate STGX2000400.
gdha commented at 2022-03-14 09:39:¶
@paulra1 See https://github.com/rear/rear/blob/master/usr/share/rear/conf/default.conf#L1017
pcahyna commented at 2022-03-14 10:08:¶
I have no idea what the reason is why "Operation not permitted" happens.
Because "rear mkrescue" is run as 'root' it is "something" that even prohibits
'root' to do things. Usually such "something" is things like SELinux or AppArmor.
@paulra1 what are the permissions and other attributes of the original
/usr/bin/mount binary (including ACLs)? (Try
getfattr -d -m - /usr/bin/mount and getfacl /usr/bin/mount.) I would
like to know if this can be solved in a proper way instead of ignoring
cp errors.
jsmeix commented at 2022-03-14 10:11:¶
@paulra1
could you show us the contents of the ESP on your ReaR USB disk
with file sizes of all files there?
I am interested to see why 512 MiB are not sufficient in your case
because I like to understand if our current default.conf setting
USB_UEFI_PART_SIZE="512"
has become again too small in general so we should increase it again
cf.
https://github.com/rear/rear/pull/1205
or if your case is an exceptional case.
paulra1 commented at 2022-03-14 21:09:¶
Hi,
I formatted the external USB hard disk as follows, set
USB_EFI_PART_SIZE to 1024 in
/etc/rear/local.conf and tried mkresue as before. It failed again with
similar diagnostics.
sudo rear -v format -- --efi /dev/sda
Incidentally, the size of the initrd.cgz file is 581 MB.
The results of "rear dump" are in the attached file, settings.txt, the
log file is in
the attacched file, rear-mobius2.log.
The command getfattr -d -m - /usr/bin/mount produces no output.
This is the ouput from getfacl /usr/bin/mount
# file: usr/bin/mount
# owner: root
# group: root
# flags: s--
user::rwx
group::r-x
other::r-x
The mount point for the USB disk is as follows:
/dev/sda1 on /var/tmp/rear-efi.EjXDiN9RdG type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
And the contents are as follows:
/var/tmp/rear-efi.EjXDiN9RdG:
total 4.0K
drwx------ 3 root root 4.0K Mar 14 13:31 EFI
/var/tmp/rear-efi.EjXDiN9RdG/EFI:
total 4.0K
drwx------ 2 root root 4.0K Mar 14 13:31 BOOT
/var/tmp/rear-efi.EjXDiN9RdG/EFI/BOOT:
total 511M
-rwx------ 1 root root 1.7M Mar 14 13:31 BOOTX64.efi
-rwx------ 1 root root 499M Mar 14 13:31 initrd.cgz
-rwx------ 1 root root 11M Mar 14 13:31 kernel
The temporary rear working directory is as follows:
/var/tmp/rear.NHWxofJikkqIV06/tmp/
And the contents of the directory are as follows:
/var/tmp/rear.NHWxofJikkqIV06/tmp/:
total 581M
-rw-r--r-- 1 root root 9 Mar 14 13:30 bootloader_area_strings
-rw-r--r-- 1 root root 193 Mar 14 13:30 copy-as-is-exclude
-rw-r--r-- 1 root root 247K Mar 14 13:30 copy-as-is-filelist
-rw-r--r-- 1 root root 581M Mar 14 13:31 initrd.cgz
drwxr-xr-x 2 root root 4.0K Mar 14 13:30 mappings
-rw-r--r-- 1 root root 0 Mar 14 13:30 nfs.mount.info
-rw-r--r-- 1 root root 152 Mar 14 13:30 parted
-rw-r--r-- 1 root root 83 Mar 14 13:30 partitions
-rw-r--r-- 1 root root 46 Mar 14 13:30 partitions-data
-rw-r--r-- 1 root root 46 Mar 14 13:30 partitions_unsorted
/var/tmp/rear.NHWxofJikkqIV06/tmp/mappings:
total 0
-rw-r--r-- 1 root root 0 Mar 14 13:30 routes
The contents of my local.conf file are as follows:
#
# Linux rear, Relax-and-Recover, local.conf configuration file. This version
# is for an Ubuntu 20.4 OS and version 2.6-git.0.0 of rear.
#
# These settings are for creating an image an an external USB hard disk that
# includes both a boot partition and ext4 file system partition and can be
# used for bare metal recovery.
#
# Setting USB indicates the external USB hard disk is a USB device.
#
OUTPUT=USB
#
# Using NETFS indicates the USB disk is an external device.
#
BACKUP=NETFS
#
# REAR-000 is the label of the ext4 partition on the USB disk.
#
BACKUP_URL=usb:///dev/disk/by-label/REAR-000
#
# /dev/sda is the default mount point for the USB disk.
#
USB_DEVICE=/dev/sda
#
# The maximum size of the EFI partition in megabytes.
#
USB_EFI_PART_SIZE=1024
#
# This indicates EFI style booting is used.
#
USB_DEVICE_PARTED_LABEL=gpt
#
# This indicates the data is stored with an ext4 file system.
#
USB_DEVICE_FILESYSTEM=ext4
#
# This file (local.conf) is intended for manual configuration. For configuration
# through packages and other automated means we recommend creating a new
# file named site.conf next to this file and to leave the local.conf as it is.
# Our packages will never ship with a site.conf.
#
paulra1 commented at 2022-03-14 23:20:¶
Hi,
I apologize, I accidentally misspelled USB_UEFI_PART_SIZE before
which yield invalid results.
Now it is spelled correctly in /etc/read/local.conf and set to 1024.
This time I used a
clean disk and formatted it as follows:
sudo rear -v format /dev/sda
I think "sudo rear mkrescue -v -D /dev/sda" still failed but in a much
more benign way.
This is the problem description from the end of the log. It suggests I
need to use
"rear validate". Is this true ? There is certainly not 20.05
subdirectory.
2022-03-14 15:40:18.551495210 # Validation status:
2022-03-14 15:40:18.567138736 # /usr/share/rear/lib/validated/Ubuntu/20.04/i386.txt : missing/empty
2022-03-14 15:40:18.571653115 # Your system is not yet validated. Please carefully check all functions
2022-03-14 15:40:18.576018915 # and create a validation record with 'rear validate'. This will help others
2022-03-14 15:40:18.580386759 # to know about the validation status of Relax-and-Recover on this system.
2022-03-14 15:40:18.584970984 # End of dump configuration and system information.
2022-03-14 15:40:18.589581899 Finished running dump workflow
2022-03-14 15:40:18.598299615 Saving /var/log/rear/rear-mobius2.log.lockless as /var/log/rear/rear-mobius2.log
Also, the new rear-mobius2.log is attached to this message.
The EFI partition is no longer mounted and only mount point I see now is as follows:
/dev/sda3 on /media/paulr/REAR-000 type ext4 (rw,nosuid,nodev,relatime,uhelper=udisks2
The attached file, disk.txt, shows its contents.
But, perhaps, the EFI partition is still there because doing an ls on /dev/sda yields the following.
brw-rw---- 1 root disk 8, 3 Mar 14 16:04 /dev/sda3
brw-rw---- 1 root disk 8, 2 Mar 14 16:04 /dev/sda2
brw-rw---- 1 root disk 8, 1 Mar 14 16:04 /dev/sda1
brw-rw---- 1 root disk 8, 0 Mar 14 16:04 /dev/sda
paulra1 commented at 2022-03-15 01:59:¶
Somehow, I attached the wrong log file. This is the right one.
PR
paulra1 commented at 2022-03-15 04:54:¶
Hi,
On my last attempt to run mkrescue, I failed to attach the right
rear-mobius2.log file.
This time I am confident I attached the right one. The previous EFI
partition space
problem is solved but mkrescue still fails.
Again I formatted the disk as follows.
sudo rear -v format /dev/sda
After running "sudo rear -v -D mkrescue, the following failure
diagnostics were displayed when rear
exited.
Configuring GRUB2 initrd /EFI/BOOT/initrd.cgz
Configuring GRUB2 root device as 'search --no-floppy --set=root --label REAR-EFI'
/dev/ttyS4 may not work as serial console for GRUB (only /dev/ttyS0 up to /dev/ttyS3 should work)
GRUB2 modules to load: ext2 fat part_gpt part_msdos
ERROR: Mount command 'mount -v -o rw,noatime /dev/sda /var/tmp/rear.POHWwUm5DhJzOUO/outputfs' failed.
Some latest log messages since the last called script 100_mount_output_path.sh:
2022-03-14 21:21:53.745237976 Entering debugscript mode via 'set -x'.
mkdir: created directory '/var/tmp/rear.POHWwUm5DhJzOUO/outputfs'
2022-03-14 21:21:53.764727451 Added 'remove_temporary_mountpoint '/var/tmp/rear.POHWwUm5DhJzOUO/outputfs'' as an exit task
2022-03-14 21:21:53.769383883 Mounting with 'mount -v -o rw,noatime /dev/sda /var/tmp/rear.POHWwUm5DhJzOUO/outputfs'
mount: /var/tmp/rear.POHWwUm5DhJzOUO/outputfs: wrong fs type, bad option, bad superblock on /dev/sda, missing codepage or helper program, or other error.
Aborting due to an error, check /var/log/rear/rear-mobius2.log for details
Exiting rear mkrescue (PID 4144) and its descendant processes ...
Running exit tasks
To remove the build area you may use (with caution): rm -Rf --one-file-system /var/tmp/rear.POHWwUm5DhJzOUO
Terminated
The obvious ERROR messages in the log file are as follows but there are
also others that
mention a "bad superblock".
+ PrintError 'ERROR: Mount command '\''mount -v -o rw,noatime /dev/sda /var/tmp/rear.POHWwUm5DhJzOUO/outputfs
'\'' failed.'
++ Log 'ERROR: Mount command '\''mount -v -o rw,noatime /dev/sda /var/tmp/rear.POHWwUm5DhJzOUO/outputfs'\'' fa
iled.'
++ echo '2022-03-14 21:21:53.850424907 ERROR: Mount command '\''mount -v -o rw,noatime /dev/sda /var/tmp/rear.
POHWwUm5DhJzOUO/outputfs'\'' failed.'
2022-03-14 21:21:53.850424907 ERROR: Mount command 'mount -v -o rw,noatime /dev/sda /var/tmp/rear.POHWwUm5DhJz
OUO/outputfs' failed.
++ LogToSyslog 'ERROR: Mount command '\''mount -v -o rw,noatime /dev/sda /var/tmp/rear.POHWwUm5DhJzOUO/outputf
s'\'' failed.'
++ logger -t rear -i 'ERROR: Mount command '\''mount -v -o rw,noatime /dev/sda /var/tmp/rear.POHWwUm5DhJzOUO/o
utputfs'\'' failed.'
I think the following diagnostics are especially important.
++ eval mount -v -o rw,noatime /dev/sda /var/tmp/rear.POHWwUm5DhJzOUO/outputfs
+++ mount -v -o rw,noatime /dev/sda /var/tmp/rear.POHWwUm5DhJzOUO/outputfs
mount: /var/tmp/rear.POHWwUm5DhJzOUO/outputfs: wrong fs type, bad option, bad superblock on /dev/sda, missing codepage or helper program, or other error.
++ Error 'Mount command '\''mount -v -o rw,noatime /dev/sda /var/tmp/rear.POHWwUm5DhJzOUO/outputfs'\'' failed.'
pcahyna commented at 2022-03-15 10:07:¶
# file: usr/bin/mount
# owner: root
# group: root
# flags: s--
user::rwx
group::r-x
other::r-x
I suspect that the suid bit is the problem: something prevents setting
it on the copy in /tmp.
jsmeix commented at 2022-03-15 12:44:¶
For comparison on my openSUSE Leap 15.3 system
where "rear mkrescue" works without issues for me:
On the original system:
# ls -l /usr/bin/mount
-rwsr-xr-x 1 root root 51456 Feb 23 13:11 /usr/bin/mount
# getfattr -d -m - /usr/bin/mount
[no output]
# getfacl /usr/bin/mount
getfacl: Removing leading '/' from absolute path names
# file: usr/bin/mount
# owner: root
# group: root
# flags: s--
user::rwx
group::r-x
other::r-x
In the recovery system:
# chroot /var/tmp/rear.bT5gNCVMilwwdLr/rootfs
bash-4.4# ls -l /bin/mount
-rwsr-xr-x 1 root root 51456 Feb 23 13:11 /bin/mount
bash-4.4# getfattr -d -m - /bin/mount
[no output]
bash-4.4# getfacl /bin/mount
getfacl: Removing leading '/' from absolute path names
# file: bin/mount
# owner: root
# group: root
# flags: s--
user::rwx
group::r-x
other::r-x
"rear --debugscripts x mkrescue" terminal output excerpts:
# usr/sbin/rear --debugscripts x mkrescue
...
Copying binaries and libraries
Copying all kernel modules in ...
so no messages related to "Copying binaries and libraries".
"rear --debugscripts x mkrescue" log file excerpts:
++ copy_binaries ... /usr/bin/mount ...
...
++ (( 86 > 1 ))
++ shift
++ binary=/usr/bin/mount
++ contains_visible_char /usr/bin/mount
+++ tr -d -c '[:graph:]'
++ test /usr/bin/mount
++ cp --verbose --archive --dereference --force /usr/bin/mount /var/tmp/rear.bT5gNCVMilwwdLr/rootfs/bin
'/usr/bin/mount' -> '/var/tmp/rear.bT5gNCVMilwwdLr/rootfs/bin/mount'
++ (( 85 > 1 ))
++ shift
so no errors or warnings from cp here.
jsmeix commented at 2022-03-15 14:02:¶
The mount error message
... wrong fs type, bad option, bad superblock on /dev/sda, missing codepage or helper program, or other error
lists all those as possible reasons (i.e. as an OR clause, not a AND
clause)
so its actual meaning in practice is basically only "something went
wrong".
jsmeix commented at 2022-03-15 14:07:¶
With current GitHub master code I get
# usr/sbin/rear format -- --help
Use 'rear format [ -- OPTIONS ] DEVICE' like 'rear -v format -- --efi /dev/sdX'
Valid format workflow options are: -b/--bios -e/--efi -f/--force -y/--yes
so I think one must use something like
# rear -v format -- --efi /dev/sdX
to get it formatted for [U]EFI.
pcahyna commented at 2022-03-15 14:16:¶
The mount error message
... wrong fs type, bad option, bad superblock on /dev/sda, missing codepage or helper program, or other errorlists all those as possible reasons (i.e. as an OR clause, not a AND clause) so its actual meaning in practice is basically only "something went wrong".
This "something" must have been to use /dev/sda instead of some
partition /dev/sda<n>, so "bad superblock on /dev/sda" was correct.
Why it decided to use the whole disk /dev/sda instead of the
appropriate partition is the key question.
paulra1 commented at 2022-03-15 19:20:¶
Hi,
The mount error with /dev/sda still occurs and what I consider the most
relevant diagnostics
follow below. (Also the log file is attached to this message.) The kinds
of diagnostics shown below
are as follows:
- rear -v format -- --efi /dev/sda: Output.
- sudo rear -v -D mkrescue: Output
- Last diagnostics from /var/log/rear-mobius2.log
- Subset of Diagnostics relevant to /dev/sda* from /var/log/rear-mobius2.log
Note that I commented out the following line in local.conf and it had
no material impact.
USB_DEVICE_PARTED_LABEL=gpt
As psahyna mentioned the problem revolves around the use of /dev/sda to
find the superblock
rather than something like /dev/sda1 to find the superblock.
Also, the following diagnostic is output but I do not think it is a technical problem.
# /usr/share/rear/lib/validated/Ubuntu/20.04/i386.txt : missing/empty
sudo rear -v format -- --efi /dev/sda
Relax-and-Recover 2.6-git.0.0.master.changed / 2022-03-11
Running rear format (PID 4194 date 2022-03-15 11:35:14)
Using log file: /var/log/rear/rear-mobius2.log
Running workflow format on the normal/original system
USB or disk device /dev/sda is not formatted with ext2/3/4 or btrfs filesystem
Formatting /dev/sda will remove all currently existing data on that whole device
Type exactly 'Yes' to format /dev/sda with ext4 filesystem
(default 'No' timeout 300 seconds)
Yes
Repartitioning /dev/sda
Creating partition table of type msdos on /dev/sda
Making an EFI bootable device /dev/sda
Creating EFI system partition /dev/sda1 with size 1024 MiB aligned at 8 MiB
Setting 'esp' flag on EFI partition /dev/sda1
Creating ReaR data partition /dev/sda2 up to 100% of /dev/sda
Setting 'boot' flag on ReaR data partition /dev/sda2
Creating vfat filesystem on EFI system partition on /dev/sda1
Creating ext4 filesystem with label 'REAR-000' on ReaR data partition /dev/sda2
Adjusting filesystem parameters on ReaR data partition /dev/sda2
Exiting rear format (PID 4194) and its descendant processes ...
Running exit tasks
sudo rear -v -D mkrescue
Relax-and-Recover 2.6-git.0.0.master.changed / 2022-03-11
Running rear mkrescue (PID 4785 date 2022-03-15 11:36:57)
Command line options: /usr/sbin/rear -v -D mkrescue
Using log file: /var/log/rear/rear-mobius2.log
Using build area: /var/tmp/rear.is3sBT79jG9RucF
Running 'init' stage ======================
Running workflow mkrescue on the normal/original system
Running 'prep' stage ======================
Found EFI system partition /dev/nvme0n1p1 on /boot/efi type vfat
Using UEFI Boot Loader for Linux (USING_UEFI_BOOTLOADER=1)
Using autodetected kernel '/boot/vmlinuz-5.15.15-76051515-generic' as kernel in the recovery system
USB disk IDs of '/dev/sda' added to WRITE_PROTECTED_IDS
File system label of '/dev/sda' added to WRITE_PROTECTED_FS_LABEL_PATTERNS
Running 'layout/save' stage ======================
Creating disk layout
Overwriting existing disk layout file /var/lib/rear/layout/disklayout.conf
Automatically excluding disk /dev/sda (not used by any mounted filesystem)
Marking component '/dev/sda' as done in /var/lib/rear/layout/disktodo.conf
Dependant component /dev/sda1 is a child of component /dev/sda
Marking component '/dev/sda1' as done in /var/lib/rear/layout/disktodo.conf
Dependant component /dev/sda2 is a child of component /dev/sda
Marking component '/dev/sda2' as done in /var/lib/rear/layout/disktodo.conf
Disabling excluded components in /var/lib/rear/layout/disklayout.conf
Disabling component 'disk /dev/sda' in /var/lib/rear/layout/disklayout.conf
Disabling component 'part /dev/sda' in /var/lib/rear/layout/disklayout.conf
Component 'part /dev/sda' is disabled in /var/lib/rear/layout/disklayout.conf
Using guessed bootloader 'EFI' for 'rear recover' (found in first bytes on /dev/nvme0n1)
Verifying that the entries in /var/lib/rear/layout/disklayout.conf are correct
Created disk layout (check the results in /var/lib/rear/layout/disklayout.conf)
Running 'rescue' stage ======================
Creating recovery system root filesystem skeleton layout
Handling network interface 'enp3s0'
enp3s0 is a physical device
Handled network interface 'enp3s0'
Handling network interface 'wlp0s20f3'
wlp0s20f3 is a physical device
Handled network interface 'wlp0s20f3'
Included current keyboard mapping (via 'dumpkeys -f')
No default US keyboard mapping included (no KEYMAPS_DEFAULT_DIRECTORY specified)
No support for different keyboard layouts (neither KEYMAPS_DEFAULT_DIRECTORY nor KEYMAPS_DIRECTORIES specified)
To log into the recovery system via ssh set up /root/.ssh/authorized_keys or specify SSH_ROOT_PASSWORD
Trying to find what to use as UEFI bootloader...
Trying to find a 'well known file' to be used as UEFI bootloader...
Using '/boot/efi/EFI/Ubuntu/grubx64.efi' as UEFI bootloader file
Copying logfile /var/log/rear/rear-mobius2.log into initramfs as '/tmp/rear-mobius2-partial-2022-03-15T11:37:04-07:00.log'
Running 'build' stage ======================
Copying files and directories
Copying binaries and libraries
Copying all kernel modules in /lib/modules/5.15.15-76051515-generic (MODULES contains 'all_modules')
Copying all files in /lib*/firmware/
Skip copying broken symlink '/etc/mtab' target '/proc/17528/mounts' on /proc/ /sys/ /dev/ or /run/
Skip copying broken symlink '/etc/resolv.conf' target '/run/systemd/resolve/stub-resolv.conf' on /proc/ /sys/ /dev/ or /run/
Testing that the recovery system in /var/tmp/rear.is3sBT79jG9RucF/rootfs contains a usable system
Testing each binary with 'ldd' and look for 'not found' libraries within the recovery system
Testing that the existing programs in the PROGS array can be found as executable command within the recovery system
Testing that each program in the REQUIRED_PROGS array can be found as executable command within the recovery system
Running 'pack' stage ======================
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (608688691 bytes) in 39 seconds
Running 'output' stage ======================
Configuring EFI partition '/dev/disk/by-label/REAR-EFI' for EFI boot with 'grubx64.efi'
Configuring GRUB2 for EFI boot
Configuring GRUB2 kernel /EFI/BOOT/kernel
Configuring GRUB2 initrd /EFI/BOOT/initrd.cgz
Configuring GRUB2 root device as 'search --no-floppy --set=root --label REAR-EFI'
/dev/ttyS4 may not work as serial console for GRUB (only /dev/ttyS0 up to /dev/ttyS3 should work)
GRUB2 modules to load: ext2 fat part_gpt part_msdos
ERROR: Mount command 'mount -v -o rw,noatime /dev/sda /var/tmp/rear.is3sBT79jG9RucF/outputfs' failed.
Some latest log messages since the last called script 100_mount_output_path.sh:
2022-03-15 11:38:19.116308548 Entering debugscript mode via 'set -x'.
mkdir: created directory '/var/tmp/rear.is3sBT79jG9RucF/outputfs'
2022-03-15 11:38:19.135540184 Added 'remove_temporary_mountpoint '/var/tmp/rear.is3sBT79jG9RucF/outputfs'' as an exit task
2022-03-15 11:38:19.140368501 Mounting with 'mount -v -o rw,noatime /dev/sda /var/tmp/rear.is3sBT79jG9RucF/outputfs'
mount: /var/tmp/rear.is3sBT79jG9RucF/outputfs: wrong fs type, bad option, bad superblock on /dev/sda, missing codepage or helper program, or other error.
Aborting due to an error, check /var/log/rear/rear-mobius2.log for details
Exiting rear mkrescue (PID 4785) and its descendant processes ...
Running exit tasks
To remove the build area you may use (with caution): rm -Rf --one-file-system /var/tmp/rear.is3sBT79jG9RucF
Terminated
Last Signifcant Diagnostics from rear-mobius2.log
++ LogToSyslog 'ERROR: Mount command '\''mount -v -o rw,noatime /dev/sda /var/tmp/rear.is3sBT79jG9RucF/outputfs'\'' failed.'
++ logger -t rear -i 'ERROR: Mount command '\''mount -v -o rw,noatime /dev/sda /var/tmp/rear.is3sBT79jG9RucF/outputfs'\'' failed.'
++ echo '===== Stack trace ====='
===== Stack trace =====
++ local c=0
++ caller 0
++ :
++ caller 1
++ :
++ caller 2
++ :
++ awk ' { l[NR]=$3":"$1" "$2 }
END { for (i=NR; i>0;) print "Trace "NR-i": "l[i--] }
++ caller 3
'
++ :
++ caller 4
++ :
++ caller 5
++ :
++ caller 6
Trace 0: /usr/sbin/rear:619 main
Trace 1: /usr/share/rear/lib/mkrescue-workflow.sh:22 WORKFLOW_mkrescue
Trace 2: /usr/share/rear/lib/framework-functions.sh:129 SourceStage
Trace 3: /usr/share/rear/lib/framework-functions.sh:59 Source
Trace 4: /usr/share/rear/output/default/100_mount_output_path.sh:9 source
Trace 5: /usr/share/rear/lib/global-functions.sh:644 mount_url
++ echo '=== End stack trace ==='
=== End stack trace ===
++ kill -USR1 4785
+++ EXIT_FAIL_MESSAGE=0
+++ echo 'Aborting due to an error, check /var/log/rear/rear-mobius2.log for details'
Subset of Diagnostics Relevant to /dev/sda* from rear-mobius2.log
+++ local url_without_scheme=/dev/sda
++ echo usb-Seagate_Portable_NACAQQYH-0:0-part2 /dev/sda2
+++++ my_udevinfo -q path -n /dev/sda
++ Log 'Having USB Device /dev/sda filesystem ext2 kernel module in MODULES and MODULES_LOAD'
+++ lsblk -ino UUID /dev/sda
+++ lsblk -ino PARTUUID /dev/sda
++ echo usb-Seagate_Portable_NACAQQYH-0:0 /dev/sda
++ echo usb-Seagate_Portable_NACAQQYH-0:0-part1 /dev/sda1
+++ get_device_name /sys/block/sda
+++ blockdev --getsize64 /dev/sda
++ declare sysfs_name=sda
++ [[ /sys/block/sda/sda1 = *\/\m\m\c\b\l\k+([0-9])\r\p\m\b ]]
++ [[ /sys/block/sda/sda1 = *\/\m\m\c\b\l\k+([0-9])\b\o\o\t+([0-9]) ]]
++ [[ /sys/block/sda/sda2 = *\/\m\m\c\b\l\k+([0-9])\r\p\m\b ]]
++ [[ /sys/block/sda/sda2 = *\/\m\m\c\b\l\k+([0-9])\b\o\o\t+([0-9]) ]]
+++ get_device_name sda1
+++ local name=sda1
+++ name=sda1
++ partition_name=/dev/sda1
++ partition_name=sda1
++ partition_name=sda1
++ partition_name=/dev/sda2
++ partition_name=sda2
++ partition_name=sda2
++ echo 'part /dev/sda 1999316713472 1082130432 primary boot /dev/sda2'
+++ echo '/dev/sda 2000398933504 msdos'
++ name=/dev/sda
++ add_component /dev/sda disk
++ echo 'todo /dev/sda disk'
+++ echo '/dev/sda 1073741824 8388608 primary esp /dev/sda1'
++ disk=/dev/sda
+++ echo '/dev/sda 1073741824 8388608 primary esp /dev/sda1'
++ name=/dev/sda1
++ add_dependency /dev/sda1 /dev/sda
++ echo '/dev/sda1 /dev/sda'
+++ echo '/dev/sda 1999316713472 1082130432 primary boot /dev/sda2'
++ REAL_USB_DEVICE=/dev/sda
2022-03-15 11:37:02.037098527 Automatically excluding disk /dev/sda (not used by any mounted filesystem)
++ Print 'Automatically excluding disk /dev/sda (not used by any mounted filesystem)'
++ echo '2022-03-15 11:37:02.044708703 Marking component '\''/dev/sda'\'' as done in /var/lib/rear/layout/disktodo.conf'
2022-03-15 11:37:02.044708703 Marking component '/dev/sda' as done in /var/lib/rear/layout/disktodo.conf
++ Debug 'Cannot mark component '\''opaldisk:/dev/sda'\'' as done because there is no '\''todo opaldisk:/dev/sda '\'' in /var/lib/rear/layout/disktodo.conf'
++ echo '2022-03-15 11:37:02.066459165 Dependant component /dev/sda1 is a child of component /dev/sda'
++ Print 'Marking component '\''/dev/sda1'\'' as done in /var/lib/rear/layout/disktodo.conf'
++ mount_cmd='mount -v -o rw,noatime /dev/sda /var/tmp/rear.is3sBT79jG9RucF/outputfs'
++ Log 'Mounting with '\''mount -v -o rw,noatime /dev/sda /var/tmp/rear.is3sBT79jG9RucF/outputfs'\'''
++ echo '2022-03-15 11:38:19.140368501 Mounting with '\''mount -v -o rw,noatime /dev/sda /var/tmp/rear.is3sBT79jG9RucF/outputfs'\'''
2022-03-15 11:38:19.140368501 Mounting with 'mount -v -o rw,noatime /dev/sda /var/tmp/rear.is3sBT79jG9RucF/outputfs'
++ eval mount -v -o rw,noatime /dev/sda /var/tmp/rear.is3sBT79jG9RucF/outputfs
+++ mount -v -o rw,noatime /dev/sda /var/tmp/rear.is3sBT79jG9RucF/outputfs
mount: /var/tmp/rear.is3sBT79jG9RucF/outputfs: wrong fs type, bad option, bad superblock on /dev/sda, missing codepage or helper program, or other error.
++ Error 'Mount command '\''mount -v -o rw,noatime /dev/sda /var/tmp/rear.is3sBT79jG9RucF/outputfs'\'' failed.'
++ PrintError 'ERROR: Mount command '\''mount -v -o rw,noatime /dev/sda /var/tmp/rear.is3sBT79jG9RucF/outputfs'\'' failed.'
2022-03-15 11:38:19.140368501 Mounting with '\''mount -v -o rw,noatime /dev/sda /var/tmp/rear.is3sBT79jG9RucF/outputfs'\''
jsmeix commented at 2022-03-16 07:28:¶
What does not match is that UEFI requires GPT partitioning
so rear -v format -- --efi /dev/sda should not result
Creating partition table of type msdos on /dev/sda
There have been recent enhancements in GitHub master code
regarding USB formatting, see
https://github.com/rear/rear/issues/2698
and
https://github.com/rear/rear/pull/2705
In general regarding OUTPUT=USB we have currently
https://github.com/rear/rear/issues/2648
So perhaps there is currently a dilemma that on the one hand the
GitHub master code has a more robust copy_binaries() function
but on the other hand there are curently issues with OUTPUT=USB
in particular with "rear format" for an EFI USB disk.
Currently I am busy with other things and
I cannot easily test OUTPUT=USB on a UEFI system.
jsmeix commented at 2022-03-16 07:50:¶
https://github.com/rear/rear/files/8256469/rear-mobius2.log
shows (excerpts)
+ source /usr/share/rear/prep/default/020_translate_url.sh
...
++ OUTPUT_URL=usb:///dev/sda
.
.
.
+ source /usr/share/rear/prep/NETFS/default/050_check_NETFS_requirements.sh
...
+++ mount -v -o rw,noatime /dev/disk/by-label/REAR-000 /var/tmp/rear.is3sBT79jG9RucF/outputfs
.
.
.
+ source /usr/share/rear/output/default/100_mount_output_path.sh
...
+++ mount -v -o rw,noatime /dev/sda /var/tmp/rear.is3sBT79jG9RucF/outputfs
++ Error 'Mount command '\''mount -v -o rw,noatime /dev/sda /var/tmp/rear.is3sBT79jG9RucF/outputfs'\'' failed.'
I assume the reason why /dev/sda is tried to be mounted is in
https://github.com/rear/rear/issues/2770#issuecomment-1067290080
(excerpts)
The contents of my local.conf file are as follows:
...
# REAR-000 is the label of the ext4 partition on the USB disk.
#
BACKUP_URL=usb:///dev/disk/by-label/REAR-000
#
# /dev/sda is the default mount point for the USB disk.
#
USB_DEVICE=/dev/sda
but usr/share/rear/conf/default.conf reads
# The device to use, set automatically by BACKUP=NETFS and BACKUP_URL=usb:///dev/sdb1
USB_DEVICE=
so USB_DEVICE should not be explicitly specified
(unless really needed for some special case)
because it gets set automatically via BACKUP_URL
because OUTPUT_URL inherits from BACKUP_URL
see usr/share/rear/conf/default.conf
jsmeix commented at 2022-03-16 08:16:¶
https://github.com/rear/rear/commit/d615581577676d2e9201f1278d02e357273115f9
should now better describe USB_DEVICE in default.conf
in particular that normally it gets set automatically
and if it is set then it must be a partition device like /dev/sdb1
pcahyna commented at 2022-03-16 09:35:¶
@jsmeix good catch! But I suspect that there is ambiguous documentation
about USB_DEVICE in other places, like the man page:
`OUTPUT=USB` Create a bootable USB disk (using extlinux). Specify the USB storage device by using `USB_DEVICE`.
@paulra1 why did you think that you needed to set USB_DEVICE this way
and which place of documentation should we improve to avoid making the
impression that USB_DEVICE needs to be set as you did?
pcahyna commented at 2022-03-16 09:51:¶
What does not match is that UEFI requires GPT partitioning
so
rear -v format -- --efi /dev/sdashould not resultCreating partition table of type msdos on /dev/sda... there are curently issues with OUTPUT=USB in particular with "rear format" for an EFI USB disk.
Currently I am busy with other things and I cannot easily test OUTPUT=USB on a UEFI system.
I can test OUTPUT=USB on a UEFI system, although I am also busy with other things (testing RAID10 for example).
jsmeix commented at 2022-03-16 09:54:¶
https://github.com/rear/rear/issues/2770#issuecomment-1068372382
shows (excerpts):
Copying all kernel modules in /lib/modules/5.15.15-76051515-generic (MODULES contains 'all_modules')
Copying all files in /lib*/firmware/
...
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (608688691 bytes) in 39 seconds
608688691 bytes are 581 MiB for ReaR's initrd on Ubuntu 20.04
where the default USB_UEFI_PART_SIZE="512" is too small
On my openSUSE Leap 15.3 system I get
Copying all kernel modules in /lib/modules/5.3.18-150300.59.54-default (MODULES contains 'all_modules')
Copying all files in /lib*/firmware/
...
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (344177688 bytes) in 24 seconds
344177688 bytes are 329 MiB for ReaR's initrd on openSUSE Leap 15.3
where the default USB_UEFI_PART_SIZE="512" is just sufficient
but there is not much free space left.
I think the main difference in size is that I don't have Nvidia stuff
but on the Ubuntu 20.04 system here there is Nvidia stuff
according to
https://github.com/rear/rear/files/8249023/disk.txt
(excerpts - only what is listed as megabytes with 'M'):
/var/tmp/rear.2BiUEOChkGL5nbG/rootfs/lib/firmware/nvidia/470.103.01:
total 35M
...
-rw-r--r-- 1 root root 41M Feb 28 06:51 nvidia.ko
so Nvidia stuff adds about 100MiB to ReaR's initrd
which would make the initrd about 450 MiB on openSUSE Leap 15.3
where the default USB_UEFI_PART_SIZE="512" is hardly sufficient.
Therefore I will change the default to USB_UEFI_PART_SIZE="1024"
so that we become somewhat future proof (hopefully at least for some
time).
jsmeix commented at 2022-03-16 10:25:¶
As always our documentation could be better
but as far as I see there is at least nothing badly wrong regarding
USB_DEVICE
# find doc/ -type f | xargs grep 'USB_DEVICE'
doc/rear-release-notes.txt: o Implemented USB_DEVICE_FILESYSTEM_LABEL (issue #1535)
doc/rear-release-notes.txt: o New USB_PARTITION_ALIGN_BLOCK_SIZE and USB_DEVICE_FILESYSTEM_PARAMS
doc/rear-release-notes.txt: o The USB_DEVICE_PARTED_LABEL=gpt setting is now honered while formatting
doc/rear-release-notes.txt: o Fixed a bug around USB_DEVICE and OUTPUT_URL mis-match (issue #579)
doc/rear-presentation.adoc:USB_DEVICE=/dev/disk/by-label/REAR-000
doc/rear-presentation.adoc:USB_DEVICE=/dev/disk/by-label/REAR-000
doc/rear.8.adoc:device by using +USB_DEVICE+.
doc/rear.8.adoc:can specify the device using +USB_DEVICE=/dev/disk/by-path/REAR-000+.
doc/user-guide/03-configuration.adoc:using +USB_DEVICE+.
doc/user-guide/03-configuration.adoc:can specify the device using +USB_DEVICE=/dev/disk/by-label/REAR-000+.
doc/user-guide/04-scenarios.adoc:USB_DEVICE=/dev/disk/by-label/REAR-000
doc/user-guide/04-scenarios.adoc:USB_DEVICE=/dev/disk/by-label/REAR-000
doc/user-guide/04-scenarios.adoc:USB_DEVICE=/dev/disk/by-label/REAR-000
doc/user-guide/04-scenarios.adoc:USB_DEVICE=/dev/disk/by-label/REAR-000
doc/user-guide/04-scenarios.adoc:for the USB device is what is configured for +USB_DEVICE+.
doc/user-guide/04-scenarios.adoc:USB_DEVICE=/dev/disk/by-label/REAR-000
doc/user-guide/12-BLOCKCLONE.adoc:USB_DEVICE=/dev/disk/by-label/REAR-000
doc/user-guide/12-BLOCKCLONE.adoc:USB_DEVICE=/dev/disk/by-label/REAR-000
doc/rear.8:USB_DEVICE\&.
doc/rear.8:USB_DEVICE=/dev/disk/by\-path/REAR\-000\&.
I will fix those two typos right now:
doc/rear.8.adoc:can specify the device using +USB_DEVICE=/dev/disk/by-path/REAR-000+.
doc/rear.8:USB_DEVICE=/dev/disk/by\-path/REAR\-000\&.
jsmeix commented at 2022-03-16 10:38:¶
https://github.com/rear/rear/commit/2e9b1d34afafeb5b305f254bdc954b0be23e97e9
Update doc/rear.8.adoc
Fixed typo '/dev/disk/by-path/REAR-000' ->
'/dev/disk/by-label/REAR-000'
https://github.com/rear/rear/commit/84df1b88ef0000c8906ad6d7e27ff66ddb72acc2
Update doc/rear.8
Fixed typo '/dev/disk/by-path/REAR-000' ->
'/dev/disk/by-label/REAR-000'
jsmeix commented at 2022-03-16 11:54:¶
@pcahyna
regarding RAID[10] testing versus testing OUTPUT=USB with UEFI
my selfish preference would be RAID testing because
we have the pending RAID issues
https://github.com/rear/rear/pull/2768
and
https://github.com/rear/rear/issues/2759
where I think RAID is more important for our business customers
than ReaR with USB disks
and I assume that OUTPUT=USB with UEFI will work
for @paulra1 with right formatting of his USB disk
and right settings in his etc/rear/local.conf
@paulra1
in general regarding ReaR see in particular the sections
"Inappropriate expectations" and
"Disaster recovery with Relax-and-Recover (ReaR)" in
https://en.opensuse.org/SDB:Disaster_Recovery
paulra1 commented at 2022-03-16 20:49:¶
After eliminating the manual USB_DEVICE setting from the local.conf
file mkrescue appears
to have succeeded and the log file, rear.log, is attached to this
message. The only
commands I used are as follows. Note that I insured the /dev/sda and all
its child devices
were not mounted before executing the commands.
sudo rear -v format -- --efi /dev/sda
sudo rear -v -D mkrescue
What do you recommend doing to verify the backup is correct ?
Note that mkrescue output included the following diagnostic and I am not sure of its significance
/dev/ttyS4 may not work as serial console for GRUB (only /dev/ttyS0 up to /dev/ttyS3 should work)
GRUB2 modules to load: ext2 fat part_gpt part_msdos
It is believed the partition problem we discussed is resolved. It is
that setting USB_DEVICE
to /dev/sda or setting it manually is wrong in the case OUTPUT=USB. When
USB_DEVICE is not
manually set, it is automatically set to something like usb:///dev/sdb1,
which is a partition device.
(Generally, this applies to the case where an external USB hard drive is
used as the backup media
for a single Linux system.)
In Johannes Meixner's explanation of the problem he notes that
BACKUP_URL=usb:///dev/sdb1
in the default.conf file documentation and I assume his intention is to
identify the relevant
section of the file. (On my system the parent device for the external
USB disk is /dev/sda.)
In my local.conf file BACKUP_URL=usb:///dev/disk/by-label/REAR-000 as
recommended by
section "Using NETFS as backup strategy (internal archive method)" of
the document referenced
by the following URL.
https://relax-and-recover.org/rear-user-guide/basics/configuration.html
Is this correct ? Also, please note that the "Rescue media (OUTPUT)"
section of the same
document suggests manually setting USB_DEVICE.
Is there some section in the documentation referenced by the following
URL that documents
USB_DEVICE and BACKUP_URL, or is it only covered in the default.conf
file ?
http://relax-and-recover.org/documentation/
In any case, I think the documentation needs to explicitly address the
situation
where the backup media is an external USB device. In particular, I think
it should
explain how to set the following configuration variables in the case
OUTPUT=USB.
- BACKUP: Explain why NETFS is correct and does not only apply to network backups.
- BACKUP_URL: Clarify when usb:///dev/disk/by-label/REAR-000 is appropriate.
- USB_DEVICE: IMPORTANT - Emphasize that is automatically set and
should
be manually set only in special cases.
My current local.conf file, and the dump, format and mkrescue output follow.
Current /etc/rear/local.conf
OUTPUT=USB
BACKUP=NETFS
USB_UEFI_PART_SIZE=1024
BACKUP_URL=usb:///dev/disk/by-label/REAR-000
Output of "rear -v format -- --efi /dev/sda"
Relax-and-Recover 2.6-git.0.0.master.changed / 2022-03-11
Running rear format (PID 7789 date 2022-03-16 12:58:21)
Using log file: /var/log/rear/rear-mobius2.log
Running workflow format on the normal/original system
USB or disk device /dev/sda is not formatted with ext2/3/4 or btrfs filesystem
Formatting /dev/sda will remove all currently existing data on that whole device
Type exactly 'Yes' to format /dev/sda with ext4 filesystem
(default 'No' timeout 300 seconds)
Yes
Repartitioning /dev/sda
Creating partition table of type msdos on /dev/sda
Making an EFI bootable device /dev/sda
Creating EFI system partition /dev/sda1 with size 1024 MiB aligned at 8 MiB
Setting 'esp' flag on EFI partition /dev/sda1
Creating ReaR data partition /dev/sda2 up to 100% of /dev/sda
Setting 'boot' flag on ReaR data partition /dev/sda2
Creating vfat filesystem on EFI system partition on /dev/sda1
Creating ext4 filesystem with label 'REAR-000' on ReaR data partition /dev/sda2
Adjusting filesystem parameters on ReaR data partition /dev/sda2
Exiting rear format (PID 7789) and its descendant processes ...
Running exit tasks
Output of rear -v -D mkrescue
Relax-and-Recover 2.6-git.0.0.master.changed / 2022-03-11
Running rear mkrescue (PID 8522 date 2022-03-16 13:01:03)
Command line options: /usr/sbin/rear -v -D mkrescue
Using log file: /var/log/rear/rear-mobius2.log
Using build area: /var/tmp/rear.w27yTmR1l6kdphJ
Running 'init' stage ======================
Running workflow mkrescue on the normal/original system
Running 'prep' stage ======================
Found EFI system partition /dev/nvme0n1p1 on /boot/efi type vfat
Using UEFI Boot Loader for Linux (USING_UEFI_BOOTLOADER=1)
Using autodetected kernel '/boot/vmlinuz-5.15.15-76051515-generic' as kernel in the recovery system
USB disk IDs of '/dev/disk/by-label/REAR-000' added to WRITE_PROTECTED_IDS
File system label of '/dev/disk/by-label/REAR-000' added to WRITE_PROTECTED_FS_LABEL_PATTERNS
Running 'layout/save' stage ======================
Creating disk layout
Overwriting existing disk layout file /var/lib/rear/layout/disklayout.conf
Automatically excluding disk /dev/sda (not used by any mounted filesystem)
Marking component '/dev/sda' as done in /var/lib/rear/layout/disktodo.conf
Dependant component /dev/sda1 is a child of component /dev/sda
Marking component '/dev/sda1' as done in /var/lib/rear/layout/disktodo.conf
Dependant component /dev/sda2 is a child of component /dev/sda
Marking component '/dev/sda2' as done in /var/lib/rear/layout/disktodo.conf
Disabling excluded components in /var/lib/rear/layout/disklayout.conf
Disabling component 'disk /dev/sda' in /var/lib/rear/layout/disklayout.conf
Disabling component 'part /dev/sda' in /var/lib/rear/layout/disklayout.conf
Component 'part /dev/sda' is disabled in /var/lib/rear/layout/disklayout.conf
Using guessed bootloader 'EFI' for 'rear recover' (found in first bytes on /dev/nvme0n1)
Verifying that the entries in /var/lib/rear/layout/disklayout.conf are correct
Created disk layout (check the results in /var/lib/rear/layout/disklayout.conf)
Running 'rescue' stage ======================
Creating recovery system root filesystem skeleton layout
Handling network interface 'enp3s0'
enp3s0 is a physical device
Handled network interface 'enp3s0'
Handling network interface 'wlp0s20f3'
wlp0s20f3 is a physical device
Handled network interface 'wlp0s20f3'
Included current keyboard mapping (via 'dumpkeys -f')
No default US keyboard mapping included (no KEYMAPS_DEFAULT_DIRECTORY specified)
No support for different keyboard layouts (neither KEYMAPS_DEFAULT_DIRECTORY nor KEYMAPS_DIRECTORIES specified)
To log into the recovery system via ssh set up /root/.ssh/authorized_keys or specify SSH_ROOT_PASSWORD
Trying to find what to use as UEFI bootloader...
Trying to find a 'well known file' to be used as UEFI bootloader...
Using '/boot/efi/EFI/Ubuntu/grubx64.efi' as UEFI bootloader file
Copying logfile /var/log/rear/rear-mobius2.log into initramfs as '/tmp/rear-mobius2-partial-2022-03-16T13:01:10-07:00.log'
Running 'build' stage ======================
Copying files and directories
Copying binaries and libraries
Copying all kernel modules in /lib/modules/5.15.15-76051515-generic (MODULES contains 'all_modules')
Copying all files in /lib*/firmware/
Skip copying broken symlink '/etc/mtab' target '/proc/21268/mounts' on /proc/ /sys/ /dev/ or /run/
Skip copying broken symlink '/etc/resolv.conf' target '/run/systemd/resolve/stub-resolv.conf' on /proc/ /sys/ /dev/ or /run/
Testing that the recovery system in /var/tmp/rear.w27yTmR1l6kdphJ/rootfs contains a usable system
Testing each binary with 'ldd' and look for 'not found' libraries within the recovery system
Testing that the existing programs in the PROGS array can be found as executable command within the recovery system
Testing that each program in the REQUIRED_PROGS array can be found as executable command within the recovery system
Running 'pack' stage ======================
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (608687936 bytes) in 38 seconds
Running 'output' stage ======================
Configuring EFI partition '/dev/disk/by-label/REAR-EFI' for EFI boot with 'grubx64.efi'
Configuring GRUB2 for EFI boot
Configuring GRUB2 kernel /EFI/BOOT/kernel
Configuring GRUB2 initrd /EFI/BOOT/initrd.cgz
Configuring GRUB2 root device as 'search --no-floppy --set=root --label REAR-EFI'
/dev/ttyS4 may not work as serial console for GRUB (only /dev/ttyS0 up to /dev/ttyS3 should work)
GRUB2 modules to load: ext2 fat part_gpt part_msdos
/dev/ttyS4 may not work as serial console for SYSLINUX (only /dev/ttyS0 up to /dev/ttyS3 should work)
Saved /var/log/rear/rear-mobius2.log as rear/mobius2/20220316.1301/rear-mobius2.log
Making /dev/sda bootable with syslinux/extlinux
Writing syslinux MBR /usr/lib/EXTLINUX/mbr.bin of type msdos to /dev/sda
Exiting rear mkrescue (PID 8522) and its descendant processes ...
Running exit tasks
To remove the build area you may use (with caution): rm -Rf --one-file-system /var/tmp/rear.w27yTmR1l6kdphJ
Output of "rear dump"
# Begin dumping out configuration and system information:
# This is a 'Linux-x86_64' system, compatible with 'Linux-i386'.
# Configuration tree:
# Linux-i386.conf : OK
# GNU/Linux.conf : OK
# Debian.conf : missing/empty
# Debian/i386.conf : missing/empty
# Debian/20.04.conf : missing/empty
# Debian/20.04/i386.conf : missing/empty
# Ubuntu.conf : OK
# Ubuntu/i386.conf : missing/empty
# Ubuntu/20.04.conf : missing/empty
# Ubuntu/20.04/i386.conf : missing/empty
# site.conf : missing/empty
# local.conf : OK
# System definition:
ARCH="Linux-i386"
OS="GNU/Linux"
OS_MASTER_VENDOR="Debian"
OS_MASTER_VERSION="20.04"
OS_MASTER_VENDOR_ARCH="Debian/i386"
OS_MASTER_VENDOR_VERSION="Debian/20.04"
OS_MASTER_VENDOR_VERSION_ARCH="Debian/20.04/i386"
OS_VENDOR="Ubuntu"
OS_VERSION="20.04"
OS_VENDOR_ARCH="Ubuntu/i386"
OS_VENDOR_VERSION="Ubuntu/20.04"
OS_VENDOR_VERSION_ARCH="Ubuntu/20.04/i386"
# Backup with NETFS:
NETFS_KEEP_OLD_BACKUP_COPY=""
NETFS_PREFIX="mobius2"
NETFS_RESTORE_CAPABILITIES=("No")
BACKUP_DUPLICITY_NAME="rear-backup"
BACKUP_INTEGRITY_CHECK=""
BACKUP_MOUNTCMD=""
BACKUP_ONLY_EXCLUDE="no"
BACKUP_ONLY_INCLUDE="no"
BACKUP_OPTIONS=""
BACKUP_RESTORE_MOVE_AWAY_DIRECTORY="/var/lib/rear/moved_away_after_backup_restore/"
BACKUP_RESTORE_MOVE_AWAY_FILES=("/boot/grub/grubenv" "/boot/grub2/grubenv")
BACKUP_RSYNC_OPTIONS=("--sparse" "--archive" "--hard-links" "--numeric-ids" "--stats")
BACKUP_SELINUX_DISABLE="1"
BACKUP_TYPE=""
BACKUP_UMOUNTCMD=""
BACKUP_URL="usb:///dev/disk/by-label/REAR-000"
# Backup program is 'tar':
BACKUP_PROG_ARCHIVE="backup"
BACKUP_PROG_COMPRESS_OPTIONS=("--gzip")
BACKUP_PROG_COMPRESS_SUFFIX=".gz"
BACKUP_PROG_CRYPT_ENABLED="false"
BACKUP_PROG_CRYPT_KEY=""
BACKUP_PROG_CRYPT_OPTIONS="/usr/bin/openssl des3 -salt -k "
BACKUP_PROG_DECRYPT_OPTIONS="/usr/bin/openssl des3 -d -k "
BACKUP_PROG_EXCLUDE=("/tmp/*" "/dev/shm/*" "/var/lib/rear/output/*" "/var/tmp/rear.r0JkYusgvEcA1sL")
BACKUP_PROG_OPTIONS=("--anchored")
BACKUP_PROG_OPTIONS_CREATE_ARCHIVE=""
BACKUP_PROG_OPTIONS_RESTORE_ARCHIVE=""
BACKUP_PROG_SUFFIX=".tar"
BACKUP_PROG_WARN_PARTIAL_TRANSFER="1"
# Output to USB:
USB_BIOS_BOOT_DEFAULT=""
USB_BOOTLOADER=""
USB_DEVICE=""
USB_DEVICE_FILESYSTEM="ext4"
USB_DEVICE_FILESYSTEM_LABEL="REAR-000"
USB_DEVICE_FILESYSTEM_PARAMS=""
USB_DEVICE_FILESYSTEM_PERCENTAGE="100"
USB_DEVICE_PARTED_LABEL="msdos"
USB_PARTITION_ALIGN_BLOCK_SIZE="8"
USB_RETAIN_BACKUP_NR="2"
USB_SUFFIX=""
USB_UEFI_PART_SIZE="1024"
OUTPUT_EFISTUB_SYSTEMD_BOOTLOADER="/usr/lib/systemd/boot/efi/systemd-bootx64.efi"
OUTPUT_LFTP_OPTIONS=""
OUTPUT_MOUNTCMD=""
OUTPUT_OPTIONS=""
OUTPUT_PREFIX="mobius2"
OUTPUT_PREFIX_PXE=""
OUTPUT_UMOUNTCMD=""
OUTPUT_URL=""
# Validation status:
# /usr/share/rear/lib/validated/Ubuntu/20.04/i386.txt : missing/empty
# Your system is not yet validated. Please carefully check all functions
# and create a validation record with 'rear validate'. This will help others
# to know about the validation status of Relax-and-Recover on this system.
# End of dump configuration and system information.
paulra1 commented at 2022-03-17 02:33:¶
The complete backup hangs but there is still occasional USB disk
activity.
NOTE: I formatted the USB disk as before.
"sudo rear -d -V mkbackup" progress to the following point and then
hangs with the output shown
below. Just prior to that /dev/sda2 is automatically mounted as follows.
/dev/sda2 on /var/tmp/rear.vAabUCiIQKWKEBU/outputfs type ext4 (ro,noatime)
The man page states the "mkbackup" is for internal methods only. Is this
part of
the problem ?
Running 'output' stage ======================
Configuring EFI partition '/dev/disk/by-label/REAR-EFI' for EFI boot with 'grubx64.efi'
Configuring GRUB2 for EFI boot
Configuring GRUB2 kernel /EFI/BOOT/kernel
Configuring GRUB2 initrd /EFI/BOOT/initrd.cgz
Configuring GRUB2 root device as 'search --no-floppy --set=root --label REAR-EFI'
/dev/ttyS4 may not work as serial console for GRUB (only /dev/ttyS0 up to /dev/ttyS3 should work)
GRUB2 modules to load: ext2 fat part_gpt part_msdos
/dev/ttyS4 may not work as serial console for SYSLINUX (only /dev/ttyS0 up to /dev/ttyS3 should work)
Saved /var/log/rear/rear-mobius2.log as rear/mobius2/20220316.1858/rear-mobius2.log
Making /dev/sda bootable with syslinux/extlinux
Writing syslinux MBR /usr/lib/EXTLINUX/mbr.bin of type msdos to /dev/sda
Running 'backup' stage ======================
Making backup (using backup method NETFS)
Creating tar archive '/var/tmp/rear.vAabUCiIQKWKEBU/outputfs/rear/mobius2/20220316.1858/backup.tar.gz'
Archived 0 MiB [avg 0 KiB/sec]
This is the last meaningful thing I can see in the log before it goes
into a loop.
and rear seems to have trouble killing process 31223.
++ ProgressStart 'Preparing archive operation'
++ echo -en '\e[2K\rPreparing archive operation\e7'
++ BackupPID=31223
++ starttime=98
++ sleep 1
2022-03-16 19:00:04.515448066 tar --warning=no-xdev --sparse --block-number --totals --verbose --no-wildcards-match-slash --one-file-system --ignore-failed-read --anchored --xattrs --xattrs-include=security.capability --xattrs-include=security.selinux --acls --gzip -X /var/tmp/rear.vAabUCiIQKWKEBU/tmp/backup-exclude.txt -C / -c -f - /boot/efi / /var/log/rear/rear-mobius2.log | dd of=/var/tmp/rear.vAabUCiIQKWKEBU/outputfs/rear/mobius2/20220316.1858/backup.tar.gz bs=1M
++ test 1
++ case "$( basename $BACKUP_PROG )" in
+++ basename tar
++ sleep 1
++ kill -0 31223
+++ awk '{s+=$1} END {print s}'
+++ stat -c %s /var/tmp/rear.vAabUCiIQKWKEBU/outputfs/rear/mobius2/20220316.1858/backup.tar.gz
++ size=27787264
++ ProgressInfo 'Archived 26 MiB [avg 13568 KiB/sec] '
++ echo -en '\e[2K\rArchived 26 MiB [avg 13568 KiB/sec] \e7'
++ sleep 1
++ kill -0 31223
+++ awk '{s+=$1} END {print s}'
paulra1 commented at 2022-03-17 03:24:¶
The complete backup hangs but there is still occasional USB disk
activity.
NOTE: I formatted the USB disk as before.
"sudo rear -d -V mkbackup" progress to the following point and then
hangs with the output shown
below. Just prior to that /dev/sda2 is automatically mounted as follows.
/dev/sda2 on /var/tmp/rear.vAabUCiIQKWKEBU/outputfs type ext4 (ro,noatime)
The man page states the "mkbackup" is for internal methods only. Is this
part of
the problem ?
Running 'output' stage ======================
Configuring EFI partition '/dev/disk/by-label/REAR-EFI' for EFI boot
with 'grubx64.efi'
Configuring GRUB2 for EFI boot
Configuring GRUB2 kernel /EFI/BOOT/kernel
Configuring GRUB2 initrd /EFI/BOOT/initrd.cgz
Configuring GRUB2 root device as 'search --no-floppy --set=root --label
REAR-EFI'
/dev/ttyS4 may not work as serial console for GRUB (only /dev/ttyS0 up
to /dev/ttyS3 should work)
GRUB2 modules to load: ext2 fat part_gpt part_msdos
/dev/ttyS4 may not work as serial console for SYSLINUX (only /dev/ttyS0
up to /dev/ttyS3 should work)
Saved /var/log/rear/rear-mobius2.log as
rear/mobius2/20220316.1858/rear-mobius2.log
Making /dev/sda bootable with syslinux/extlinux
Writing syslinux MBR /usr/lib/EXTLINUX/mbr.bin of type msdos to
/dev/sda
Running 'backup' stage ======================
Making backup (using backup method NETFS)
Creating tar archive
'/var/tmp/rear.vAabUCiIQKWKEBU/outputfs/rear/mobius2/20220316.1858/backup.tar.gz'
Archived 0 MiB [avg 0 KiB/sec]
This is the last meaningful thing I can see in the log before it goes
into a loop.
and rear seems to have trouble killing process 31223.
++ ProgressStart 'Preparing archive operation'
++ echo -en '\e[2K\rPreparing archive operation\e7'
++ BackupPID=31223
++ starttime=98
++ sleep 1
2022-03-16 19:00:04.515448066 tar --warning=no-xdev --sparse
--block-number --totals --verbose --no-wildcards-match-slash
--one-file-system --ignore-failed-read --anchored --xattrs
--xattrs-include=security.capability --xattrs-include=security.selinux
--acls --gzip -X /var/tmp/rear.vAabUCiIQKWKEBU/tmp/backup-exclude.txt -C
/ -c -f - /boot/efi / /var/log/rear/rear-mobius2.log | dd
of=/var/tmp/rear.vAabUCiIQKWKEBU/outputfs/rear/mobius2/20220316.1858/backup.tar.gz
bs=1M
++ test 1
++ case "$( basename $BACKUP_PROG )" in
+++ basename tar
++ sleep 1
++ kill -0 31223
+++ awk '{s+=$1} END {print s}'
+++ stat -c %s
/var/tmp/rear.vAabUCiIQKWKEBU/outputfs/rear/mobius2/20220316.1858/backup.tar.gz
++ size=27787264
++ ProgressInfo 'Archived 26 MiB [avg 13568 KiB/sec] '
++ echo -en '\e[2K\rArchived 26 MiB [avg 13568 KiB/sec] \e7'
++ sleep 1
++ kill -0 31223
+++ awk '{s+=$1} END {print s}'
jsmeix commented at 2022-03-17 14:32:¶
@paulra1
regarding "What do you recommend doing to verify the backup is correct
?"
See in particular the sections
"Fully compatible replacement hardware is needed" and
"No disaster recovery without testing and continuous validation" in
https://en.opensuse.org/SDB:Disaster_Recovery
Note that "rear mkrescue" does not create any backup of the files.
When you do not have fully compatible replacement hardware
to test and verify that "rear recover" actually works for you
(e.g. when you use ReaR for your personal computer at home
and you don't have a second same model - of course not)
you could try if the USB disk with the ReaR recovery system
at least boots on that same hardware.
Of course you would not run "rear recover" on that same hardware
because "rear recover" would overwrite all what there is on its disk
by reinstalling the complete system anew from scratch
(which is the whole purpose of disaster recovery).
But there should be no harm when you only boot the ReaR recovery
system
and afterwards only shut it down again (log in as 'root' and type
"poweroff").
jsmeix commented at 2022-03-17 14:54:¶
kill -0 31223 is not meant for killing process 31223,
cf. "man kill" that reads (excerpt)
If signal is 0, then no actual signal is sent, but error checking is still performed.
which describes rather indirectly that kill -0 PID can be used
to check whether or not a process with that PID exists
i.e. whether or not a process with that PID is running,
for example like
# ( for i in $( seq 9 ) ; do echo $i ; sleep 1 ; done ) & pid=$! ; while kill -0 $pid ; do echo $pid still running ; sleep 2 ; done
[1] 9988
9988 still running
1
2
9988 still running
3
4
9988 still running
5
6
9988 still running
7
8
9988 still running
9
[1]+ Done ( for i in $( seq 9 );
do
echo $i; sleep 1;
done )
-bash: kill: (9988) - No such process
jsmeix commented at 2022-03-17 15:09:¶
By the way:
Personally I prefer in my etc/rear/local.conf
PROGRESS_MODE="plain"
PROGRESS_WAIT_SECONDS="3"
which results simpler progress reporting via successive traditional
plain text lines
without the "fancy" (but possibly terminal dependant) ANSI escape
sequences,
cf. "PROGRESS_MODE" and "PROGRESS_WAIT_SECONDS" in default.conf
paulra1 commented at 2022-03-17 16:20:¶
Thank you, the formatting configuration you gave was very helpful.
One paranoid question: Is mkbackup known to work as documented with USB
devices ?
rear -v -D mkbackup still fails in a similar way.
The tail of the rear output is as follows and the log file is attached
to this message.
It appears archiving has stopped, but I suppose there is a chance it is
some kind of
timing anomaly, and archiving would continue if you waited long enough.
Archived 5270 MiB [avg 27963 KiB/sec]
Archived 5301 MiB [avg 27695 KiB/sec]
Archived 0 MiB [avg 0 KiB/sec]
Archived 0 MiB [avg 0 KiB/sec]
Archived 0 MiB [avg 0 KiB/sec]
... Forever ...
The corresponding segment of the log file is as follows:
++ kill -0 39497
+++ awk '{s+=$1} END {print s}'
+++ stat -c %s /var/tmp/rear.6Db2Qcbw72vSfRY/outputfs/rear/mobius2/20220317.0855/backup.tar.gz
++ size=5558501376
++ ProgressInfo 'Archived 5301 MiB [avg 27695 KiB/sec] '
++ echo 'Archived 5301 MiB [avg 27695 KiB/sec] '
++ sleep 3
++ kill -0 39497
+++ awk '{s+=$1} END {print s}'
+++ stat -c %s
stat: missing operand
Try 'stat --help' for more information.
++ size=
++ ProgressInfo 'Archived 0 MiB [avg 0 KiB/sec] '
++ echo 'Archived 0 MiB [avg 0 KiB/sec] '
++ sleep 3
++ kill -0 39497
+++ awk '{s+=$1} END {print s}'
+++ stat -c %s
stat: missing operand
Try 'stat --help' for more information.
++ size=
++ ProgressInfo 'Archived 0 MiB [avg 0 KiB/sec] '
++ echo 'Archived 0 MiB [avg 0 KiB/sec] '
++ sleep 3
jsmeix commented at 2022-03-18 08:28:¶
@paulra1
offhandedly I have no idea why the backup does not finish
in your particular case.
The progress info
Archived 5270 MiB [avg 27963 KiB/sec]
Archived 5301 MiB [avg 27695 KiB/sec]
Archived 0 MiB [avg 0 KiB/sec]
Archived 0 MiB [avg 0 KiB/sec]
... Forever .....
looks as if the backup could be even complete
because 5270 MiB is a bit more than 5 GiB
which looks like a reasonable backup size
of a normal system but then the backup process
(that runs the actual backup program - by default 'tar')
does somehow not finish.
There is a separated log file 'backup.log' in the directory
where the backup 'backup.tar.gz' is stored (i.e. on your USB drive)
where the backup process logs in particular
what the backup program 'tar' reports
so inspecting that backup.log file may tell
why the backup program does not proceed and finish.
What I do not have in my "rear -D mkbackup" log file
is something like you get (i.e. the sudden 'stat -c %s' without a file
name)
+++ stat -c %s /var/tmp/rear.6Db2Qcbw72vSfRY/outputfs/rear/mobius2/20220317.0855/backup.tar.gz
++ size=5558501376
++ ProgressInfo 'Archived 5301 MiB [avg 27695 KiB/sec] '
++ echo 'Archived 5301 MiB [avg 27695 KiB/sec] '
++ sleep 3
++ kill -0 39497
+++ awk '{s+=$1} END {print s}'
+++ stat -c %s
stat: missing operand
Try 'stat --help' for more information.
The matching code is in
usr/share/rear/backup/NETFS/default/500_make_backup.sh
while sleep $PROGRESS_WAIT_SECONDS ; kill -0 $BackupPID 2>/dev/null; do
#blocks="$(stat -c %b ${backuparchive})"
#size="$((blocks*512))"
size="$(stat -c %s ${backuparchive}* | awk '{s+=$1} END {print s}')"
ProgressInfo "Archived $((size/1024/1024)) MiB [avg $((size/1024/(SECONDS-starttime))) KiB/sec] "
done
so stat -c is only called as long as the backup program is running.
In our master code this is currently at
https://github.com/rear/rear/blob/master/usr/share/rear/backup/NETFS/default/500_make_backup.sh#L256
I have in my log file when the backup program 'tar' has finished
(right now I did a small testing backup onto my USB disk):
+++ stat -c %s /var/tmp/rear.vGaIHBRascMILNG/outputfs/rear/linux-h9wr/20220318.0923/backup.tar.gz
++ size=463667200
++ ProgressInfo 'Archived 442 MiB [avg 23831 KiB/sec] '
++ echo 'Archived 442 MiB [avg 23831 KiB/sec] '
++ sleep 3
++ kill -0 11063
++ ProgressStop
++ echo OK
So it looks as if somehow your backup file
/var/tmp/rear.6Db2Qcbw72vSfRY/outputfs/rear/mobius2/20220317.0855/backup.tar.gz
suddelnly disappears while the backup program 'tar' is still running?
Regarding "Is mkbackup known to work as documented with USB devices ?":
Yes, "rear mkbackup" is known to work with USB devices.
Many users use it (but not so many in enterprise environments).
It is described as an initial "Getting Started" method on
http://relax-and-recover.org/documentation/getting-started
I presented this live on FOSDEM 2020
(the good old times when we could freely meet in person)
as live demo without safety net on real hardware how ReaR on USB
(both "rear mkbackup" and then "rear recover" with a USB disk)
works to recover my laptop from an intentional fatal soft error
(I deleted essential files via "# rm -r /lib.*" so nothing works
then).
Of course I did not physically destroy my laptop for a presentation.
pcahyna commented at 2022-03-18 09:10:¶
The man page states the "mkbackup" is for internal methods only. Is this part of
the problem
No, because NETFS is an internal backup method.
I have been using backups to USB successfully and never saw the backup problem that you are encountering. Weird.
pcahyna commented at 2022-03-18 09:15:¶
So it looks as if somehow your backup file
/var/tmp/rear.6Db2Qcbw72vSfRY/outputfs/rear/mobius2/20220317.0855/backup.tar.gz
suddelnly disappears while the backup program 'tar' is still running?
Yes, I would check in another terminal whether the file mentioned in the log still exists, and if not, whether at least the directory exists, whether the file got renamed or the filesystem unmounted or something....
jsmeix commented at 2022-03-18 09:42:¶
@paulra1
do you perhaps use an unusual 'tar' program?
What results do you get on your system for
# type -a tar
# tar --version
For comparison what I get on openSUSE Leap 15.3
# tar --version
tar (GNU tar) 1.30
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by John Gilmore and Jay Fenlason.
# type -a tar
tar is /usr/bin/tar
tar is /bin/tar
# file /bin/tar /usr/bin/tar
/bin/tar: symbolic link to /usr/bin/tar
/usr/bin/tar: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=11d3fd55d64c10d4517aab1d623b10e6bac511d5, for GNU/Linux 3.2.0, stripped
paulra1 commented at 2022-03-18 18:00:¶
I succeeded in making "rear -v -D mkbackup" work correctly and did not
find any tar problems.
I believe the previous failure was do the the following operational
error on my part.
After formatting the USB disk, /dev/sda on my system, I unmounted its
child device,
/dev/sda2 on my system. What appears to have happened, is that for some
reason I don't
understand, after starting mkbackup, /dev/sda2 was mounted and then, at
some point,
automatically unmounted, which resulted in the loop I previously
mentioned.
After formatting the USB disk with the same procedures as before and
starting
mkbackup without performing any unmounts, the complete backup succeeded
and the
tail of the output was as follows. (The log file is attached for your
convenience.) The only
problem I noticed was a tar warning and I am sure of its significance.
Please inform
me if you think it is a problem.
rear -v -D mkbackup
... Lots of output ...
Archived 61798 MiB [avg 34712 KiB/sec]
Archived 61929 MiB [avg 34729 KiB/sec]
Archived 62069 MiB [avg 34750 KiB/sec]
Archived 62198 MiB [avg 34765 KiB/sec]
Archived 62262 MiB [avg 34744 KiB/sec]
Archived 62343 MiB [avg 34733 KiB/sec]
Archived 62422 MiB [avg 34720 KiB/sec]
OK
WARNING: tar ended with return code 1 and below output (last 5 lines):
---snip---
tar: 119: Warning: Cannot flistxattr: Operation not supported
tar: 118: Warning: Cannot flistxattr: Operation not supported
tar: 2284: Warning: Cannot flistxattr: Operation not supported
tar: 2344: Warning: Cannot flistxattr: Operation not supported
tar: /sys: file changed as we read it
----------
This means that files have been modified during the archiving
process. As a result the backup may not be completely consistent
or may not be a perfect copy of the system. Relax-and-Recover
will continue, however it is highly advisable to verify the
backup in order to be sure to safely recover this system.
Archived 62422 MiB in 1844 seconds [avg 34663 KiB/sec]
Exiting rear mkbackup (PID 8477) and its descendant processes ...
Running exit tasks
To remove the build area you may use (with caution): rm -Rf --one-file-system /var/tmp/rear.YKpGz6ooxVLBLh6
jsmeix commented at 2022-03-21 07:55:¶
@paulra1
thank you for your feedback what caused such weird behaviour in your
case.
Such feedback helps us a lot because now we know that there is not
some obscure thing lurking in ReaR that sometimes goes wrong.
I regularly experience similar issues with my USB disk:
When I plug in my USB disk its partitions get automatically mounted
by some automated medium mounting functionality
(I think it is systemd's udisks2.service)
and then "rear mkrescue/mkbackup" fails with an error
because the USB disk partitions are already mounted
so I manually unmount automatically mounted USB disk partitions
before I could run "rear mkrescue/mkbackup" successfully.
I assume that also in your case some automated
mounting/umounting functionality gets into your way.
It looks like a bug in automated mounting/umounting functionality
if it "just unmounts" something regardless that it is currently in use
(in this case unmount it regardless that 'tar' has open files on it)
because this will lead to wrong data and loss of data
in particular when data write operations get disrupted.
What is unexpected is that 'tar' does not exit with an error
when the partition with its backup.tar.gz gets unmounted
but perhaps 'tar' cannot do anything in this case
because it simply hangs in a write() call.
jsmeix commented at 2022-03-21 08:26:¶
I also notice since some time 'tar' messages like
tar: /sys: file changed as we read it
I ignore it because files under '/sys' should be irrelevant in a
backup
but perhaps things could go wrong if files under '/sys' get restored.
I do not understand why 'tar' tries to read files under '/sys'
because 'tar' is called by ReaR with the '--one-file-system' option
so 'tar' should not read files under '/sys'
because /sys is a separated filesystem
# mount | grep 'on /sys '
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
Currently I fail to see why 'tar' tries to read files under '/sys'
because I have in my "rear -D mkbackup" log file
2022-03-21 09:10:26.370401573 Making backup (using backup method NETFS)
2022-03-21 09:10:26.375792216 Backup include list (backup-include.txt contents):
2022-03-21 09:10:26.381591846 /
2022-03-21 09:10:26.388275901 /nfs
2022-03-21 09:10:26.394027532 /var/lib/libvirt
2022-03-21 09:10:26.398662998 Backup exclude list (backup-exclude.txt contents):
2022-03-21 09:10:26.403629654 /tmp/*
2022-03-21 09:10:26.410193691 /dev/shm/*
2022-03-21 09:10:26.414531494 /root/rear.github.master.oldest/rear.github.master.old/var/lib/rear/output/*
2022-03-21 09:10:26.419307937 /var/tmp/rear.OkpDycGr90ada7W
2022-03-21 09:10:26.426056553 Creating tar archive '/var/tmp/rear.OkpDycGr90ada7W/outputfs/rear/linux-h9wr/20220321.0909/backup.tar.gz'
2022-03-21 09:10:26.584877893 tar --warning=no-xdev --sparse --block-number --totals --verbose --no-wildcards-match-slash --one-file-system --ignore-failed-read --anchored --xattrs --xattrs-include=security.capability --xattrs-include=security.selinux --acls --gzip -X /var/tmp/rear.OkpDycGr90ada7W/tmp/backup-exclude.txt -C / -c -f - / /nfs /var/lib/libvirt /root/rear.github.master.oldest/rear.github.master.old/var/log/rear/rear-linux-h9wr.log | dd of=/var/tmp/rear.OkpDycGr90ada7W/outputfs/rear/linux-h9wr/20220321.0909/backup.tar.gz bs=1M
where the "Backup include list" matches my mounted "normal filesystems"
# df -h --output=source,target | grep -v tmpfs
Filesystem Mounted on
/dev/mapper/cr_ata-TOSHIBA_MQ01ABF050_Y2PLP02CT-part3 /
/dev/sda5 /var/lib/libvirt
/dev/sda4 /nfs
pcahyna commented at 2022-03-21 09:12:¶
I do not understand why 'tar' tries to read files under '/sys'
because 'tar' is called by ReaR with the '--one-file-system' option
so 'tar' should not read files under '/sys'
because /sys is a separated filesystem
Is it possible that tar is not actually reading files under /sys but
only the /sys directory itself? Can you please check your backup
whether it contains anything under /sys and whether it contains the
empty /sys mountpoint when this warning occurs?
jsmeix commented at 2022-03-21 10:45:¶
I did a test:
My disk layout (/dev/sda is my system disk and /dev/sdb is my USB disk):
# lsblk -ipo NAME,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT
NAME TYPE FSTYPE LABEL SIZE MOUNTPOINT
/dev/sda disk 465.8G
|-/dev/sda1 part 8M
|-/dev/sda2 part crypto_LUKS 4G
| `-/dev/mapper/cr_ata-TOSHIBA_MQ01ABF050_Y2PLP02CT-part2 crypt swap 4G [SWAP]
|-/dev/sda3 part crypto_LUKS 200G
| `-/dev/mapper/cr_ata-TOSHIBA_MQ01ABF050_Y2PLP02CT-part3 crypt ext4 200G /
|-/dev/sda4 part ext4 100G /nfs
|-/dev/sda5 part ext4 150G /var/lib/libvirt
|-/dev/sda6 part ext2 8G
|-/dev/sda7 part crypto_LUKS 1G
`-/dev/sda8 part crypto_LUKS 1G
/dev/sdb disk 465.8G
|-/dev/sdb1 part 8M
|-/dev/sdb2 part vfat REAR-EFI 512M
|-/dev/sdb3 part ext2 MY-BOOT 1G
`-/dev/sdb4 part ext3 MY-DATA 45.1G
/dev/sr0 rom 1024M
My etc/rear/local.conf
FIRMWARE_FILES=( 'no' )
MODULES=( 'loaded_modules' )
PROGRESS_MODE="plain"
PROGRESS_WAIT_SECONDS="3"
SSH_ROOT_PASSWORD="rear"
OUTPUT=USB
USB_DEVICE_FILESYSTEM_PERCENTAGE=10
USB_DEVICE_FILESYSTEM_LABEL='MY-DATA'
USB_BOOT_PART_SIZE=1024
USB_BOOTLOADER="grub"
USB_DEVICE_BOOT_LABEL=MY-BOOT
OUTPUT_URL=usb:///dev/disk/by-label/MY-BOOT
USB_DEVICE_PARTED_LABEL=gpt
USB_BOOTLOADER="grub"
BACKUP=NETFS
BACKUP_URL=usb:///dev/disk/by-label/MY-DATA
BACKUP_ONLY_INCLUDE="yes"
BACKUP_PROG_INCLUDE=( / )
My "rear -D mkbackup" terminal output (excerpts)
# usr/sbin/rear -D mkbackup
...
Running 'backup' stage ======================
Making backup (using backup method NETFS)
Creating tar archive '/var/tmp/rear.wp8twDARZQkDzR6/outputfs/rear/linux-h9wr/20220321.0936/backup.tar.gz'
Preparing archive operation
Archived 91 MiB [avg 23368 KiB/sec]
Archived 125 MiB [avg 18406 KiB/sec]
...
Archived 18689 MiB [avg 6736 KiB/sec]
OK
WARNING: tar ended with return code 1 and below output (last 5 lines):
---snip---
tar: lxcfs: Warning: Cannot flistxattr: Operation not supported
tar: /var/lib/lxcfs: file changed as we read it
tar: /sys: file changed as we read it
----------
This means that files have been modified during the archiving
process. As a result the backup may not be completely consistent
or may not be a perfect copy of the system. Relax-and-Recover
will continue, however it is highly advisable to verify the
backup in order to be sure to safely recover this system.
Archived 18689 MiB in 2844 seconds [avg 6729 KiB/sec]
Exiting rear mkbackup (PID 32275) and its descendant processes ...
My "rear -D mkbackup" log file output (excerpts)
2022-03-21 09:37:09.838210458 Making backup (using backup method NETFS)
2022-03-21 09:37:09.843082957 Backup include list (backup-include.txt contents):
2022-03-21 09:37:09.847944145 /
2022-03-21 09:37:09.852543277 Backup exclude list (backup-exclude.txt contents):
2022-03-21 09:37:09.856941896 /tmp/*
2022-03-21 09:37:09.861237587 /dev/shm/*
2022-03-21 09:37:09.865453749 /root/rear.github.master.oldest/rear.github.master.old/var/lib/rear/output/*
2022-03-21 09:37:09.869515175 /var/tmp/rear.wp8twDARZQkDzR6
2022-03-21 09:37:09.873983163 Creating tar archive '/var/tmp/rear.wp8twDARZQkDzR6/outputfs/rear/linux-h9wr/20220321.0936/backup.tar.gz'
2022-03-21 09:37:09.889544438 tar --warning=no-xdev --sparse --block-number --totals --verbose --no-wildcards-match-slash --one-file-system --ignore-failed-read --anchored --xattrs --xattrs-include=security.capability --xattrs-include=security.selinux --acls --gzip -X /var/tmp/rear.wp8twDARZQkDzR6/tmp/backup-exclude.txt -C / -c -f - / /root/rear.github.master.oldest/rear.github.master.old/var/log/rear/rear-linux-h9wr.log | dd of=/var/tmp/rear.wp8twDARZQkDzR6/outputfs/rear/linux-h9wr/20220321.0936/backup.tar.gz bs=1M
---snip---
tar: lxcfs: Warning: Cannot flistxattr: Operation not supported
tar: /var/lib/lxcfs: file changed as we read it
tar: /sys: file changed as we read it
----------
This means that files have been modified during the archiving
process. As a result the backup may not be completely consistent
or may not be a perfect copy of the system. Relax-and-Recover
will continue, however it is highly advisable to verify the
backup in order to be sure to safely recover this system.
...
2022-03-21 10:24:35.096579036 Archived 18689 MiB in 2844 seconds [avg 6729 KiB/sec]
'/var/tmp/rear.wp8twDARZQkDzR6/tmp/backup.log' -> '/var/tmp/rear.wp8twDARZQkDzR6/outputfs/rear/linux-h9wr/20220321.0936/backup.log'
And here we have the reason for the
'tar' message "tar: /sys: file changed as we read it"
# grep -A1 ': /sys/' /var/tmp/rear.wp8twDARZQkDzR6/tmp/backup.log
block 8194644: /sys/
tar: /sys: file changed as we read it
# mount /dev/sdb4 /tmp/sdb4
# tar -tzvf /tmp/sdb4/rear/linux-h9wr/20220321.0936/backup.tar.gz | grep ' sys/'
dr-xr-xr-x root/root 0 2022-03-21 06:59 sys/
@pcahyna
thank you for your helpful idea what the root cause is!
jsmeix commented at 2022-03-21 10:58:¶
Regarding
tar: lxcfs: Warning: Cannot flistxattr: Operation not supported
tar: /var/lib/lxcfs: file changed as we read it
excerpts from my backup.log
# grep lxcfs /var/tmp/rear.wp8twDARZQkDzR6/tmp/backup.log
tar: lxcfs: Warning: Cannot flistxattr: Operation not supported
block 8188058: /var/lib/lxcfs/
tar: /var/lib/lxcfs: file changed as we read it
How that file looks on my original system
# ls -ld /var/lib/lxcfs
drwxr-xr-x 2 root root 0 Mar 21 11:49 /var/lib/lxcfs
# getfacl /var/lib/lxcfs
getfacl: Removing leading '/' from absolute path names
# file: var/lib/lxcfs
# owner: root
# group: root
user::rwx
group::r-x
other::r-x
# getfattr -d /var/lib/lxcfs
getfattr: /var/lib/lxcfs: Operation not supported
# mount | grep lxcfs
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
pcahyna commented at 2022-03-21 11:05:¶
@jsmeix you are welcome! You see, the problem is, the backup program
should back up the /sys mount point in /, otherwise the /sys file
system would not be mountable in restored system, so the behavior is
correct. But it can not really back up the real /sys directory in the
root filesystem, because when /sys is mounted, the original /sys
directory is shadowed by the root directory of the /sys filesystem,
and thus inaccessible. And apparently the root directory of /sys keeps
changing constantly.
jsmeix commented at 2022-03-21 12:00:¶
@paulra1
according to our findings above the 'tar' message
tar: /sys: file changed as we read it
is harmless but it cannot be avoided
cf.
https://github.com/rear/rear/issues/2770#issuecomment-1073762225
Only all the 'tar' messages of the form "file changed as we read it"
could be suppressed via --warning=no-file-changed
but I think we should not suppress all those 'tar' messages
because it is the user who must decide in each particular case
if such a message indicates a real problem for him like
tar: /path/to/database: file changed as we read it
or if it is harmless.
Regarding your 'tar' messages
tar: 119: Warning: Cannot flistxattr: Operation not supported
tar: 118: Warning: Cannot flistxattr: Operation not supported
tar: 2284: Warning: Cannot flistxattr: Operation not supported
tar: 2344: Warning: Cannot flistxattr: Operation not supported
You have to check your backup.log file about those messages
in your particular system.
When you run "rear -D mkbackup" ReaR's temporary working area is kept
(as in my case, see above) so you have your backup.log file not only
on your USB disk (which is unmounted after "rear rmkbackup" finished)
but also still in ReaR's temporary working area
usually as /var/tmp/rear.*/tmp/backup.log
I think "tar: ...: Warning: Cannot flistxattr: Operation not
supported"
is normally harmless and it also cannot be avoided because
we need to call 'tar' with "-xattrs" to have extended attributes
support
but then files where extended attributes are not supported
(e.g. files on filesystems that do not support extended attributes)
will cause such 'tar' warnings and - as far as I see - there is no
matching --warning=no-... option to suppress that in general.
paulra1 commented at 2022-03-21 22:12:¶
I am confident that I understand how to create a rescue disk now but not
sure of
the recovery details with the current version rear. (Version 2.6)
There are several instructional tutorials and documents on how to
recover after creating a
rescue disk with "rear mkbackup". All the instructions I have seen
specify the same initial set
of 3 steps which are as follows.
- Boot from the backup media. (An exxternal USB disk in my case.)
- Login as root. (Relax-and-Recover 2.6-git.0.0.master.changed / 2022-03-11)
- Enter the recover command. (rear -v -D recover)
After that point the instructions differ. Some specify that you will be
prompted to verify
the disk layout and restore script, others specify unattaching from the
ISO, and others that
you don't need to take further action. Which is correct for the current
version and do you have
a link to a correct set of instructions ?
jsmeix commented at 2022-04-04 11:49:¶
@paulra1
the different instructions describe different behaviour
what could happen during "rear recover".
In the best case "rear recover" can complete successfully
without any needed action by the user so a whole recovery
is only those 4 steps:
- Boot the ReaR recovery system
- Log in as 'root'
- Run "rear recover"
- Reboot
But when "rear recover" cannot do its job without help of the user
there will be user dialogs where the user must make decisions.
This happens in particular when recreating on not 100% compatible
hardware where the most important point are differences regarding
the storage (i.e. different disks) and then recreating the system
becomes what we call a MIGRATION, cf. what I wrote at
https://github.com/rear/rear/issues/2782#issuecomment-1085852827
You can try out what user dialogs are shown during "rear recover"
by forcing ReaR into its MOGRATION_MODE via
# export MIGRATION_MODE='true'
directly before calling "rear recover".
E.g. you may do it this on your Dell machine
when you use it as playground to try out things with ReaR.
In the worst case "rear recover" will fail and then the user needs
sufficient expert knowledge to have a chance to get things to work
only with what there is in the rather limited ReaR recovery system
(i.e. only with low level basic tools on plain commandline),
cf. the part "Be prepared for the worst case"
in the section "Recommendations"
and the part "Hopeless in retrospect"
in the section "Help and Support" in
https://en.opensuse.org/SDB:Disaster_Recover
[Export of Github issue for rear/rear.]