#326 Issue closed: ERROR: Mount command 'mount /dev/disk/by-label/RELAXRECOVER /tmp/rear.fbhHW5ocbWdLhET/outputfs' failed¶
Labels: enhancement, bug, cleanup, support / question,
fixed / solved / done
sirtow opened issue at 2013-11-09 22:08:¶
Hi , i have 3.10.9 kernel running with udev 204.
While running rear recover im getting the following error :
ERROR: Mount command 'mount /dev/disk/by-label/RELAXRECOVER /tmp/rear.fbhHW5ocbWdLhET/outputfs' failed
i have 2 block devices detected :
ls -l /sys/block/ |grep -v "ram\|loop"
total 0
lrwxrwxrwx 1 root root 0 Nov 9 21:58 sda -> ../devices/pci0000:00/0000:00:0d.0/ata1/host0/target0:0:0/0:0:0:0/block/sda
lrwxrwxrwx 1 root root 0 Nov 9 21:58 sr0 -> ../devices/pci0000:00/0000:00:01.1/ata4/host3/target3:0:0/3:0:0:0/block/sr0
so it seems kernel was able to detect my virtual disk and cdrom(virtualbox)
the disk/by-lable is empty :
ls -l /dev/disk/by-label/
total 0
Any suggestions on how to solve and where is there is the problem?
Edit:
Actually no entries were created under /dev for block devices.
No /dev/sda or /dev/sr0 exists.
I manually created sr0 (mknod b 11 0) and added a link under /dev/disk/by-lable but i think its shouldn't be like that...
gdha commented at 2013-11-12 12:54:¶
Could you share your /etc/rear/local.conf file and on which linux
version you're working?
sirtow commented at 2013-11-12 18:22:¶
local.conf:
# Create Relax-and-Recover rescue media as ISO image
OUTPUT=ISO
BACKUP=NETFS
#BACKUP_URL=file:///mnt/Pluto_Rescue
BACKUP_URL=iso://backup
ISO_MAX_SIZE=4500
OUTPUT_URL=file:///mnt/Pluto_Rescue/
MODULES=(${MODULES[@]} configs )
EXCLUDE_MD=(${EXCLUDE_MD[@]} /dev/md0)
EXCLUDE_MOUNTPOINTS=( ${EXCLUDE_MOUNTPOINTS[@]} /mnt/vol1 /mnt/mirror /mnt/Pluto_Rescue )
COPY_AS_IS_EXCLUDE=( ${COPY_AS_IS_EXCLUDE[@]} /opt/mkcdrec /opt/ut/settings/torrents /usr/portage /opt/ut/settings/settings )
SSH_ROOT_PASSWORD=123456
os.conf 👍
KERNEL_FILE=/boot/3.10.9-LUN-NODEP
relevant portion of rear dump:
elax-and-Recover 1.15 / Git
Using log file: /opt/rear/var/log/rear/rear-pluto.log.lockless
Dumping out configuration and system information
This is a 'Linux-x86_64' system, compatible with 'Linux-i386'.
System definition:
ARCH = Linux-i386
OS = GNU/Linux
OS_MASTER_VENDOR =
OS_MASTER_VERSION =
OS_MASTER_VENDOR_ARCH =
OS_MASTER_VENDOR_VERSION =
OS_MASTER_VENDOR_VERSION_ARCH =
OS_VENDOR = Gentoo
OS_VERSION = n/a
OS_VENDOR_ARCH = Gentoo/i386
OS_VENDOR_VERSION = Gentoo/n/a
OS_VENDOR_VERSION_ARCH = Gentoo/n/a/i386
gdha commented at 2013-11-13 08:27:¶
@sirtow Ok I understand the issue now. Will try it myself.
gdha commented at 2013-12-03 15:24:¶
@sirtow if you could try the latest snapshot and give me some feedback on it. I guess it will work better, but need your input to confirm or decline my statement. thx.
sirtow commented at 2013-12-03 22:21:¶
Tried it. now rear recover fails with error :
ERROR: The LSB package is not installed
Running lsb_release on my system produce :
lsb_release -a
LSB Version: n/a
Distributor ID: Gentoo
Description: NAME=Gentoo
Release: n/a
Codename: n/a
There are no sdX or hdX entries under /dev
gdha commented at 2013-12-04 14:55:¶
the library script ./lib/config-functions.sh contains a function
called SetOSVendorAndVersion which bails out because it doesn't find
lsb_release executable. You could try to create /etc/rear/os.conf
file containing something like:
OS_VENDOR="Gentoo"
OS_VERSION="(no idea what we need to fill in here)"
If lsb_release executable is absent we try to pre-built this via RPM
installation, but Gentoo is working differently, right. Tips are
welcome...
sirtow commented at 2013-12-04 15:59:¶
I do have lsb_release installed in my system :
pluto rear # type lsb_release
lsb_release is /usr/bin/lsb_release
pluto rear # which lsb_release
/usr/bin/lsb_release
So do i have a problem with packaging this binary while creating a
backup?
Also for
If lsb_release executable is absent we try to pre-built this via RPM installation
do you mean installing it on host,if absent, during backup or creating a package that can be later used during recovery(without installing)?
gdha commented at 2013-12-05 06:27:¶
What I meant is that by creating a /etc/rear/os.conf file you don't need lsb_release anymore. The packaging tries to do this for you. I have no clue what Gentoo requires to achieve the same?
Verstuurd vanaf mijn iPhone
Op 4-dec.-2013 om 16:59 heeft sirtow notifications@github.com het volgende geschreven:
I do have lsb_release installed in my system :
pluto rear # type lsb_release
lsb_release is /usr/bin/lsb_release
pluto rear # which lsb_release
/usr/bin/lsb_releaseSo do i have a problem with packaging this binary while creating a backup?
Also forIf lsb_release executable is absent we try to pre-built this via RPM installation
do you mean installing it on host,if absent, during backup or creating a package that can be later used during recovery(without installing)?
—
Reply to this email directly or view it on GitHub.
sirtow commented at 2013-12-06 21:28:¶
I managed to solve the missing lsb_release problem by adding it :
PROGS=(${PROGS[@]} /usr/bin/lsb_release)
to local.conf
After that i'm getting same error:
ERROR: Mount command 'mount /dev/disk/by-label/RELAXRECOVER /tmp/rear.fbhHW5ocbWdLhET/outputfs' failed
gdha commented at 2013-12-10 09:07:¶
@sirtow could you run rear -vD mkrescue and put the log on github gist
system for inspection? I wonder if the isofs kernel module is
available or loaded when trying to mount the ISO image.
sirtow commented at 2013-12-10 15:42:¶
I can but i dont think this is the issue. Once i manually populate /dev as :
mknod /dev/sda b 8 0
mknod /dev/sr0 b 11 0
ln -s /dev/sr0 /dev/disk/by-label/RELAXRECOVER
Im able to run the recover process. So it seems my kernel knows how to
mount iso image.
Do you think i still need to test your last suggestion?
gdha commented at 2013-12-11 07:37:¶
@sirtow is udevd running in recovery mode?
sirtow commented at 2013-12-11 14:34:¶
Yes it is :
udevd --daemon
On Wed, Dec 11, 2013 at 2:37 AM, gdha notifications@github.com wrote:
@sirtow https://github.com/sirtow is udevd running in recovery mode?
gdha commented at 2013-12-11 15:56:¶
If that is the case then the devices should have been created automatically by udevd. Somehow it must fail, are you sure you didn't see any error messages while booting the rescue image?
gdha commented at 2013-12-20 08:45:¶
@sirtow would be nice to see a debug session from a recovery
rear -vD recover (add this to gisthub)? We have no clue for the
moment, sorry.
astafyev commented at 2014-01-23 21:07:¶
It might be related to #222. I left a comment there but it seems that
notifications stop working after ticket is closed. So adding it here:
"The problem is that ext3 module is not loaded if udev is present.
Despite the fact that ext3 module is present in /etc/modules, it's not
loaded in 40-start-udev-or-load-modules.sh because udev is present. But
udev doesn't load ext3 either."
gdha commented at 2014-02-11 13:02:¶
@astafyev did you found an error about ext3 via e.g. dmesg output?
astafyev commented at 2014-02-11 13:57:¶
No, there is only 'mount' error. If I load ext3 manually then I can
mount manually too. Also if I change 40-start-udev-or-load-modules.sh
script to always load modules regardless udev presence then this issue
goes away.
This issue is reproducible on CentOS as ext3 is compiled as a module.
Fedora doesn't have this issue as ext3 is compiled into kernel.
sirtow commented at 2014-02-11 17:02:¶
In my case, the FS is part of the kernel.
I tend to agree that udevd does not create entires under /dev but the
question is why? Same kernel is being used in my system and the entries
are there.
gdha commented at 2014-05-29 19:14:¶
@sirtow @astafyev have a look at the script
./build/GNU/Linux/60_verify_and_adjust_udev.sh - it might be that some
slight adjustment is required for your environment?
sirtow commented at 2014-05-29 22:12:¶
Thanks, ill check that out. Might take some time.
On Thu, May 29, 2014 at 3:14 PM, gdha notifications@github.com wrote:
@sirtow https://github.com/sirtow @astafyev
https://github.com/astafyev have a look at the script
./build/GNU/Linux/60_verify_and_adjust_udev.sh - it might be that some
slight adjustment is required for your environment?—
Reply to this email directly or view it on GitHub
https://github.com/rear/rear/issues/326#issuecomment-44572288.
gdha commented at 2014-06-20 14:05:¶
@sirtow @astafyev fix in usr/share/rear/build/GNU/Linux/60_verify_and_adjust_udev.sh for systemd-udevd daemon present or not. See also issue #405
jezzaaa commented at 2014-06-27 03:53:¶
Even after patching 60_verify_and_adjust_udev.sh, I get the same error during restore. I have a device in /dev/disk/by-label/ called boot, but not RELAXRECOVER. So udev seems to be working just fine, but the mount fails. If I create the symlink to /dev/sr1 (my CD device) the mount works and the recovery completes. On RHELv5.
jezzaaa commented at 2014-06-30 06:21:¶
Perhaps this is a different problem. Is there something I can do to confirm that it's the same fault? Or perhaps should I open a new issue ticket?
gdha commented at 2015-02-03 11:54:¶
@jezzaaa sorry for the extremely late reply - did you try the latest unstable rear repo yet? As many fixes were added around udev. Would be nice to know if it is any better now?
jezzaaa commented at 2015-02-03 20:04:¶
@gdha Sorry, I'm not able to test any more. The host is now in production.
williat commented at 2015-02-07 18:18:¶
I have just started looking at "rear" and am testing with a centos 5
64-bit vm on vmware esxi. I just replaced the version from EPEL with
rear-1.16.1-121.git201502062115.el5.noarch.rpm, ran rear -v mkbackup,
copied the resulting ISO to esxi and attached it to a second (empty) vm.
I booted from that ISO and ran rear -v recover. It failed. "ERROR: Mount
command 'mount /dev/disk/by-label/RELAXRECOVER /tmp/rear.blah/outputfs'
failed". /dev/disk/by-label is empty. Also, in
/var/log/rear/rear-
OUTPUT=ISO
OUTPUT_URL=nfs://a.b.c.d/some/path/backup
BACKUP=NETFS
BACKUP_URL=iso:///backup
This does appear to add my data to the ISO and then copies the ISO to the nfs server. And, as noted above, if I create the symlink, "ln -s /dev/hdc /dev/disk/by-label/RELACRECOVER", I can then run "rear recover" successfully.
gdha commented at 2015-06-03 11:22:¶
This issue is related to issue #363 as well
hpannenb commented at 2015-09-22 10:18:¶
Hello. I am facing the same described behaviour from above when trying to recover a virtual machine (KVM) on Ubuntu 14.04.3. I am using the current Relax-and-Recover 1.17.2-git201509191126 / 2015-09-19. Maybe I can help in debugging the situation a bit further.
gdha commented at 2015-09-22 18:35:¶
@hpannenb That would be appreciated as this issue is open much too long already!
hpannenb commented at 2015-11-07 16:33:¶
I tested ReaR 1.17.2 with different Ubuntu versions (all with latest updates) and backup to ISO as follows:
- 12.04 => Recovery works fine.
- 14.04 => Fails as mentioned (no /dev/disk/by-label/RELAXRECOVER found)
- 15.04 => Recovery works fine.
That is my state. What can I do to debug the situation under 14.04 a bit
better?
Might this
https://wiki.ubuntu.com/DebuggingUdev
help?
gdha commented at 2016-02-18 17:28:¶
usr/share/rear/skel/default/etc/scripts/40-start-udev-or-load-modules.sh was recently modified for module loading (issue #766). Perhaps it could fix the issue with Ubuntu 14.04 as well?
DaGSUMan commented at 2016-03-01 14:46:¶
Where should the symbolic link be placed? I cannot get the iso to mount during recovery.
hardrain47 commented at 2016-03-01 21:54:¶
Just ran into this same problem on Redhat 5.11 on 1.17.2 RPM install. The issue comes because the ISO label isn't being read and aliased into the /dev/disk/by-name/ directory
I fix it by running
ln -s /dev/cdrom /dev/disk/by-label/RELAXRECOVER
Isn't there a simple way to fix this by putting an option in to specify the iso device label in the configuration file, or to specify it at recover time?
gdha commented at 2016-03-02 08:47:¶
@hardrain47 Could you try the latest snapshot as several modifications were made around udev? - see http://download.opensuse.org/repositories/Archiving:/Backup:/Rear:/Snapshot/CentOS_CentOS-5/x86_64/
DaGSUMan commented at 2016-03-02 16:17:¶
My original local.conf
OUTPUT=ISO
BACKUP=NETFS
BACKUP_URL=iso://backup
ISO_MAX_SIZE=3500
OUTPUT_URL="nfs://server_name/mnt/array1/CMISLEMS/"
EOF
*Was getting this during recover *
Please insert the media called REARRECOVER in your CD-ROM drive...
RELAXRECOVER detected in /dev/sr0 ...
Please insert the media called REARRECOVER in your CD-ROM drive...
RELAXRECOVER detected in /dev/sr0 ...
Please insert the media called REARRECOVER in your CD-ROM drive...
RELAXRECOVER detected in /dev/sr0 ...
Please insert the media called REARRECOVER in your CD-ROM drive...
RELAXRECOVER detected in /dev/sr0 ...
The new local.conf
OUTPUT=ISO
BACKUP=NETFS
BACKUP_URL=iso:///dev/disk/by-label/REARRECOVER/
ISO_MAX_SIZE=3500MB
OUTPUT_URL="nfs://10.10.10.10/mnt/array1/server_name/"
EOF
Backup
[root@server_name ~]# rear -v mkbackup
Relax-and-Recover 1.17.2 / Git
Using log file: /var/log/rear/rear-server_name.log
Creating disk layout
Creating root filesystem layout
TIP: To login as root via ssh you need to set up
/root/.ssh/authorized_keys or SSH_ROOT_PASSWORD in your configuration
file
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Creating initramfs
Encrypting disabled
Creating tar archive
'/tmp/rear.MmfKBLuMwAsre2x/tmp/isofs/dev/disk/by-label/REARRECOVER/backup.tar.gz'
Archived 10573 MiB in 1430 seconds [avg 7571 KiB/sec]
Preparing 4 ISO images ...
Making additionnal ISO image : rear-server_name_01.iso
Wrote ISO image: /var/lib/rear/output/rear-server_name_01.iso (380K)
Making additionnal ISO image : rear-server_name_02.iso
Wrote ISO image: /var/lib/rear/output/rear-server_name_02.iso (380K)
Making additionnal ISO image : rear-server_name_03.iso
Wrote ISO image: /var/lib/rear/output/rear-server_name_03.iso (380K)
Making ISO image
Wrote ISO image: /var/lib/rear/output/rear-server_name.iso (11G)
Copying resulting files to nfs location
Recover sitll gets
Please insert the media called REARRECOVER in your CD-ROM drive...
RELAXRECOVER detected in /dev/sr0 ...
Please insert the media called REARRECOVER in your CD-ROM drive...
RELAXRECOVER detected in /dev/sr0 ...
Please insert the media called REARRECOVER in your CD-ROM drive...
RELAXRECOVER detected in /dev/sr0 ...
Please insert the media called REARRECOVER in your CD-ROM drive...
RELAXRECOVER detected in /dev/sr0 ...
Can anyone tell me what I am doing wrong on the backup or the recover?
hardrain47 commented at 2016-03-03 16:03:¶
Tried the snapshot. Same issue. There's no link for it in the /dev/disk/by-label directory.
ln -s /dev/cdrom /dev/disk/by-label/RELAXRECOVER
fixes the problem
rear-1.17.2-170.git201603021132.el5.x86_64.rpm was the RPM I used.
DaGSUMan commented at 2016-03-03 16:08:¶
@hardrain47 At what point are you making the symbolic link?
hardrain47 commented at 2016-03-03 16:34:¶
I made the symlink after booting to recovery and opening the shell.
in your case, you'll want to link to sr0 instead of /dev/cdrom. (I assume you knew that). We're having separate issues possibly because you're dealing with multiple dvds.
hardrain47 commented at 2016-03-04 15:19:¶
@GDHA did you make a code change on this? I saw one on udev.
DaGSUMan commented at 2016-03-04 15:34:¶
@hardrain47 i am having no luck. Can you send me a sample local.conf file to see the iso backup format. FYI when my backup was less that 4GB and a single iso. the restore worked just fine. its only since it has grown over 4GB that it splits into two and the restore gives the RELAXRECOVER issue.
gdha commented at 2016-03-04 15:34:¶
@hardrain47 added some extra sleep time...
hardrain47 commented at 2016-03-04 16:05:¶
What OS you on @DaGSUMan? I'll see if I can test multiple disks.
DaGSUMan commented at 2016-03-04 16:28:¶
@hardrain47 I'm on Red Hat 6.7
hardrain47 commented at 2016-03-04 18:01:¶
here's my local.conf. I'm going to drop my iso size to 1GB and test.
OUTPUT=ISO
OUTPUT_URL=null
ISO_MAX_SIZE=4500
BACKUP=NETFS
BACKUP_URL=iso://backup
ISO_DEFAULT=manual
hardrain47 commented at 2016-03-04 19:01:¶
Well, on 5u11, multiple DVDs appears to be working when I do
ln -s /dev/cdrom /dev/disk/by-label/RELAXRECOVER
before running >rear recover
I also just use the default output location: /var/lib/rear/output
I'll test on 6.7
DaGSUMan commented at 2016-03-04 20:57:¶
Please see the screen shots
REARError.docx
hardrain47 commented at 2016-03-04 21:20:¶
Something already exists in your /dev/disk/by-label directory.
What does > ls -al /dev/disk/by-label report?
DaGSUMan commented at 2016-03-04 21:24:¶
RELAXRECOVER -> ../../sr0
.
hardrain47 commented at 2016-03-04 21:34:¶
what happens when you do ls -al /dev/sr0 ?
DaGSUMan commented at 2016-03-04 21:40:¶
hpannenb commented at 2016-03-05 13:19:¶
@gdha I retested with Relax-and-Recover 1.17.2-git201603021132 / 2016-03-02 on Ubuntu 14.04.4 in a KVM. Still missing the /dev/disk/by-label/RELAXRECOVER.
hardrain47 commented at 2016-03-07 17:09:¶
@DaGSUMan Couldn't test over the weekend with 6.7, but I have some results for you.
I am on an extremely similar RHEL release 6.7.2.6-32-573.e16.x86_64
I did a test setup with a 700MB max iso size which split my system into two isos and tried a restore. I had no problem restoring with multiple isos. Worked flawlessly. Do you think perhaps it could be your CD drive, or how you burned your isos? I am using a USB DVD drive with isos burnt to DVD-RW on windows.
Another thing I thought you might be able to do is do a ctrl-z when it's asking for that CD and try to mount the filesystem manually. That may be easier said than done.
Still stuck on 5u11, but I can work around that issue. Going to try it on RHEL 4
DaGSUMan commented at 2016-03-07 17:14:¶
THank you @hardrain47 I'll try it again. Although, fyi I am using VM to test the restoring and mounting the ISO. It is creating three iso's one that is 7GB and the other two are like 300K
hardrain47 commented at 2016-03-07 17:26:¶
Oh, this could totally a virtual device issue. How are you "loading" these?
DaGSUMan commented at 2016-03-07 17:28:¶
I was trying to burn them to an DVD but it was too large. It's not breaking the DVD size equally. My is 7GB from the physical server. That is why I was trying to restore to VM.
jsmeix commented at 2018-08-10 10:47:¶
With
https://github.com/rear/rear/pull/1894
merged
this issue should be now fixed in a (hopefully) better way.
See
https://github.com/rear/rear/pull/1894#issuecomment-411766357
for some examples how it behaves now.
[Export of Github issue for rear/rear.]