#1571 Issue closed: OUTPUT_URL=null conflicts with OUTPUT=USB but ReaR does not error out appropriately¶
Labels: enhancement, fixed / solved / done, minor bug
prophet60091 opened issue at 2017-11-09 00:52:¶
Version:
2.2-git.0.0 /2017-10-30
OS
centOS Linux release 7.2.1511
site.conf:
OUTPUT=USB
OUTPUT_URL=null
BACKUP=NETFS
BACKUP_PROG=rsync
BACKUP_URL=usb:///dev/disk/by-label/REAR-000
export "TMPDIR=/home/backup/tmp"
UEFI boot
Brief description of the issue:
after running
rear format /dev/sdc -- --efi
and then
rear mkbackup
It errors out after after the initrd.cgz is created with
BUG in /usr/share/rear/output/USB/Linux-i386/850_make_USB_bootable.sh line 40 Filesystem for device ' dev/sdc2' could not be found
I think this is the relevant portion of the log:
2017-11-08 17:06:10.542111909 Entering debugscripts mode via 'set -x'.
+ source /usr/share/rear/output/USB/Linux-i386/850_make_USB_bootable.sh
++ FEATURE_DD_OFLAG=
+++ get_version 'dd --version'
+++ TERM=dumb
+++ dd --version
+++ sed -rn 's/^[^0-9\.]*([0-9]+\.[-0-9a-z\.]+).*$/\1/p'
+++ head -1
++ dd_version=8.22
++ version_newer 8.22 5.3.0
++ v1list=(${1//[-.]/ })
++ local v1list
++ v2list=(${2//[-.]/ })
++ local v2list
++ local max=2
++ (( 2 < 3 ))
++ max=3
++ local pos
+++ seq 0 2
++ for pos in '$(seq 0 $(( max -1 )))'
++ (( 10#08 < 10#05 ))
++ (( 10#08 > 10#05 ))
++ return 0
++ FEATURE_DD_OFLAG=y
++ '[' /dev/sdc -a /dev/sdc2 ']'
++ BugIfError 'RAW_USB_DEVICE and REAL_USB_DEVICE should be already set'
++ (( 0 != 0 ))
+++ get_usb_syslinux_version
+++ for file in '$BUILD_DIR/outputfs/{boot/syslinux,}/{ld,ext}linux.sys'
+++ [[ -s /home/backup/tmp/rear.4t1uO1nBlTaGUrP/outputfs/boot/syslinux/ldlinux.sys ]]
+++ for file in '$BUILD_DIR/outputfs/{boot/syslinux,}/{ld,ext}linux.sys'
+++ [[ -s /home/backup/tmp/rear.4t1uO1nBlTaGUrP/outputfs/boot/syslinux/extlinux.sys ]]
+++ for file in '$BUILD_DIR/outputfs/{boot/syslinux,}/{ld,ext}linux.sys'
+++ [[ -s /home/backup/tmp/rear.4t1uO1nBlTaGUrP/outputfs//ldlinux.sys ]]
+++ for file in '$BUILD_DIR/outputfs/{boot/syslinux,}/{ld,ext}linux.sys'
+++ [[ -s /home/backup/tmp/rear.4t1uO1nBlTaGUrP/outputfs//extlinux.sys ]]
+++ return 1
++ usb_syslinux_version=
+++ get_syslinux_version
+++ local syslinux_version
++++ get_version extlinux --version
++++ TERM=dumb
++++ extlinux --version
++++ sed -rn 's/^[^0-9\.]*([0-9]+\.[-0-9a-z\.]+).*$/\1/p'
++++ head -1
+++ syslinux_version=4.05
+++ [[ -z 4.05 ]]
+++ [[ -z 4.05 ]]
+++ [[ -z 4.05 ]]
+++
tried manually setting usb_filesystem in the site.conf, but didn't
work
tried reformatting
prophet60091 commented at 2017-11-09 01:16:¶
I should add that yes /proc/mounts was not listed the device, however, If unplug it and plug it back in, it automounts and I run rear mkbackup, it errors with USB device /dev/sdc2 is already mounted on '/run/media/root/rear-000
It seems to be a kind of catch22 proc mounts wont have the info unless it's mounted, but I cannot mount it and start the backup.
gozora commented at 2017-11-09 08:05:¶
Hi,
Please attach FULL logfile created by rear -d -D mkrescue
thx
V.
gozora commented at 2017-11-09 08:17:¶
thinking of it, try to disable automount of rear ubs disk or try to
umount the disk manually, prior rear mkrescue
jsmeix commented at 2017-11-09 10:22:¶
As far as I understand it there is not much what ReaR could do.
During "rear mkrescue/mkbackup" ReaR needs to mount
the USB device at a ReaR-specific mount point but that
could conflict with whatever existing USB automount magic
that may have already mounted that USB device elsewhere.
I think ReaR should never ever try to mess around with
the existing system setup (like changing how an existing
USB automount magic behaves) because the setup
of the original system is sacrosanct for ReaR
(i.e. ReaR should never change the original system).
Instead the user of ReaR has to setup his system
so that his system setup does not conflict with ReaR.
jsmeix commented at 2017-11-09 10:46:¶
@prophet60091
as far as I see the script that mounts the USB device
at the ReaR-specific mount point is
usr/share/rear/output/default/100_mount_output_path.sh
which calls
mount_url $OUTPUT_URL $BUILD_DIR/outputfs $OUTPUT_OPTIONS
and the mount_url function is defined in
usr/share/rear/lib/global-functions.sh
For me on my SLES12 test system the mount_url function
is called like (from my "rear -d -D mkbackup" log file):
+ source /root/rear.master/usr/share/rear/output/default/100_mount_output_path.sh ... ++ mount_url usb:///dev/disk/by-label/MyBackup /tmp/rear.Gql4esSs7sKmOdb/outputfs
according to what I have in my test system
# ls -l /dev/disk/by-label/MyBackup lrwxrwxrwx ... /dev/disk/by-label/MyBackup -> ../../sdb1
If you like you could perhaps do a GitHub pull request
that enhances ReaR so that when the USB device
that is configured in ReaR is already mounted
could be automatically unmounted to that then
it works to mount it at the ReaR-specific mount point
$BUILD_DIR/outputfs = /tmp/rear.XXX.../outputfs
gozora commented at 2017-11-09 11:35:¶
If you like you could perhaps do a GitHub pull request
that enhances ReaR so that when the USB device
that is configured in ReaR is already mounted
could be automatically unmounted to that then
it works to mount it at the ReaR-specific mount point
$BUILD_DIR/outputfs = /tmp/rear.XXX.../outputfs
I'd disagree here, as we don't know neither reason nor method how particular USB disk was mounted, ReaR should not interfere here but rather let user to disable automatic mounting of USB disk or solve such situation by him self.
V.
jsmeix commented at 2017-11-09 12:28:¶
I think I should have better explained why I thought
it could be o.k. to let ReaR automatically umount it:
My reasoning was that I assumed that when the user
(who must be 'root') runs "rear mkrescue/mkbackup"
then he knows what that results and he really wants it
which means he wants that ReaR writes onto the USB device
that he has configured in ReaR to be used by ReaR.
On the other hand:
When the user accidentally misconfigured a wrong
USB device in ReaR e.g. 'sdb1' instead of 'sdd1' like
USB_DEVICE=/dev/sdb1
or
BACKUP_URL=usb:///dev/sdb1
where /dev/sdb1 is one of his system filesystems,
then it is probably not so much o.k. to let ReaR
"just automatically umount it" ;-)
Bottom line:
I agree with
https://github.com/rear/rear/issues/1571#issuecomment-343128675
that ReaR should not automatically umount it.
jsmeix commented at 2017-11-09 12:31:¶
Accordingly I close the issue as "invalid" for ReaR
because the user must solve such situations by himself
because ReaR must never ever try to mess around
with the existing system.
jsmeix commented at 2017-11-09 12:40:¶
@prophet60091
you could add a script that implements
what you need for your particular case
to the 'init' stage.
I.e. add such a script to the
usr/share/rear/init/default/
directory.
ReaR scripts must be named
with an exactly three digits prefix like
usr/share/rear/init/default/123_my_special_stuff.sh
prophet60091 commented at 2017-11-09 16:04:¶
Well, I've tried not auto mounting, and I've tried also just unmounting
before running the script. In this case it gives the error that I
originally described.
Here the full log:
https://github.com/prophet60091/rearlogs/blob/master/FSnotFound.log
If it's mounted as it would be after running the format for example then
get error:**
Message: USB device '/dev/sdb2' is already mounted on '/run/media/root/REAR-000'
full log https://github.com/prophet60091/rearlogs/blob/master/mounted.log
The problem seems to be that it looking in /proc/mounts isn't going to work when the device isn't mounted. But mounting it will cause complaints of being already mounted.
I tried hardcoding the file system to usb_filesystem=ext4 (I've used 3 as well no difference there)
I already noticed in the log that there is another complaint that states RAW_USB_DEVICE and REAL_USB_DEVICE, should already be set.
I think I can track it down, but what should the values of those two variables be ideally set to? The script mentioned in this issue, says it's assuming they are set, but I don't know where.
gozora commented at 2017-11-09 16:20:¶
@prophet60091 just for fun of it, can you remove OUTPUT_URL=null from
your config file ?
V.
prophet60091 commented at 2017-11-09 16:29:¶
wow. that solved it.
I had left it part of the config from doing ISOs earlier in the day and didn't think it would be a culprit. Good call, and thank you!
gozora commented at 2017-11-09 16:34:¶
Anytime! 😃
jsmeix commented at 2017-11-10 08:44:¶
Wow!
Now we have a real minor bug/enhancement here because
ReaR should error out for known not working settings,
cf. "Try to care about possible errors" at
https://github.com/rear/rear/wiki/Coding-Style
jsmeix commented at 2017-11-10 08:46:¶
I will do a pull request...
jsmeix commented at 2017-11-10 11:14:¶
I can reproduce it with "rear mkrescue" with
OUTPUT=USB OUTPUT_URL=null BACKUP=NETFS BACKUP_URL=usb:///dev/sdb1
# usr/sbin/rear -d -D mkrescue ... Saved /root/rear.master/var/log/rear/rear-e205.log as rear/e205/20171110.1210/rear-e205.log ERROR: ==================== BUG in /root/rear.master/usr/share/rear/output/USB/Linux-i386/850_make_USB_bootable.sh line 40: 'Filesystem for device '/dev/sdb1' could not be found'
jsmeix commented at 2017-11-10 12:28:¶
Seems to be much more complicated than expected.
That USB stuff was added a longer time ago
as a dirty hack for a very specific use case and
that results and endless sequence of weird issues
like this one.
By chance I found while debugging this script:
build/USB/default/800_enforce_usb_output.sh
https://github.com/rear/rear/blob/master/usr/share/rear/build/USB/default/800_enforce_usb_output.sh
Its code does not make any sense for me.
WHY could OUTPUT=... be different in
$ROOTFS_DIR/etc/rear/local.conf
compared to what the user had set
in /etc/rear/local.conf on the original system
that justifies to enforce OUTPUT=USB in
$ROOTFS_DIR/etc/rear/local.conf
and WHY does OUTPUT=... matter at all
in the ReaR recovery system?
I would expect that OUTPUT is only used for
"rear mkrescue/mkbackup" but not for "rear recover".
@gdha
do you perhaps have any idea what the reason is/was
for this script?
It this script stays inexplicable for us I would suggest
to "just blindly remove it right now" and then wait
and see if any user reports any regression.
Or we may only disable it with an added initial
# No idea what the reason behind is for the code below # so that it is - at least for now - just disabled: return 0
jsmeix commented at 2017-11-10 13:02:¶
O.k. seems I found how it fails:
In my log I get (excerpts):
+ source /root/rear.master/usr/share/rear/output/default/100_mount_output_path.sh ++ mount_url null /tmp/rear.8n2C189DVGc6flF/outputfs ++ local url=null ++ case $(url_scheme $url) in +++ url_scheme null ++ return 0 ... + source /root/rear.master/usr/share/rear/output/USB/Linux-i386/850_make_USB_bootable.sh +++ grep -E '^/dev/sdb1 |^/dev/sdb1 ' /proc/mounts ++ usb_filesystem= ++ case "$usb_filesystem" in ++ BugError 'Filesystem for device '\''/dev/sdb1'\'' could not be found'
i.e. 850_make_USB_bootable.sh expects that the USB device
was already mounted by 100_mount_output_path.sh
but because of OUTPUT_URL=null the 'mount_url null ...'
function call did nothing (and 'return 0') so that
the 'grep ... /proc/mounts' in 850_make_USB_bootable.sh
cannot find a mounted USB device which results the BugError.
Accordingly I think now that the actual conflict is
"OUTPUT=USB" together with "OUTPUT_URL=null"
and ReaR should detect this and error out.
jsmeix commented at 2017-11-10 13:19:¶
With the change in
https://github.com/rear/rear/pull/1572
it errors out instantly in this case:
# usr/sbin/rear -d -D mkrescue Relax-and-Recover 2.2 / Git Using log file: /root/rear.master/var/log/rear/rear-e205.log ERROR: 'OUTPUT_URL=null' conflicts with 'OUTPUT=USB' Aborting due to an error, check /root/rear.master/var/log/rear/rear-e205.log for details Terminated
jsmeix commented at 2017-11-10 13:22:¶
With
https://github.com/rear/rear/pull/1572
merged
I consider this issue to be sufficiently avoided/fixed.
jsmeix commented at 2017-11-10 13:28:¶
FWIW:
With
OUTPUT=USB BACKUP=NETFS BACKUP_URL=usb:///dev/sdb1
"rear mkrescue" and "rear mkbackup"
and then also "rear recover"
still "just work" for me.
gdha commented at 2017-11-10 14:52:¶
If OUTPUT=USB the OUTPUT_URL with USB may not be null IMHO as
otherwise it is a complete useless definition as we must be able to boot
from the USB drive, right. That is the purpose of OUTPUT_URL. However,
in recover mode the OUTPUT_URL is not required as the BACKUP_URL is
important there.
I agree USB part was a quick and dirty hack by Jeroen and Dag, but we
may not complain as it is used a lot and seems to be functional most of
times.
jsmeix commented at 2017-11-10 15:09:¶
@gdha
many thanks for the info.
Accordingly
https://github.com/rear/rear/pull/1572
is the right fix.
@gdha
do you perhaps also have an idea what the reason behind
the 800_enforce_usb_output.sh script could be, cf. my
https://github.com/rear/rear/issues/1571#issuecomment-343461088
Only a side note for the fun of it:
I am allowed to complain (at least a bit from time to time)
because I fixed so much with it ;-)
Have a nice weekend!
gdha commented at 2017-11-10 16:14:¶
@jsmeix concerning:
build/USB/default/800_enforce_usb_output.sh
https://github.com/rear/rear/blob/master/usr/share/rear/build/USB/default/800_enforce_usb_output.sh
Its code does not make any sense for me.
WHY could OUTPUT=... be different in
$ROOTFS_DIR/etc/rear/local.conf
I seem to remember (and that is already an achievement!) that that was a hack when Dag plugged in the USB drive into a Proliant server it automagically started doing things with ReaR. However, I might be completely wrong here - if @dagwieers could bring some light that would be even better.
[Export of Github issue for rear/rear.]