#1253 Issue closed: FATAL: Module fuse not found -- in RESCUE shell¶
Labels: support / question, fixed / solved / done
John-Leone opened issue at 2017-03-18 20:04:¶
Relax-and-Recover (rear) Issue Template¶
RESCUE xrearm1d:~ # rear -V Relax-and-Recover 2.00 / Git
RESCUE xrearm1d:~ # cat /etc/rear/os.conf OS_VENDOR=RedHatEnterpriseServer OS_VERSION=6 # # WARNING ! This information was added automatically by the mkbackup workflow !!! ARCH='Linux-i386' OS='GNU/Linux' OS_VERSION='6' OS_VENDOR='RedHatEnterpriseServer' OS_VENDOR_VERSION='RedHatEnterpriseServer/6' OS_VENDOR_ARCH='RedHatEnterpriseServer/i386'
RESCUE xrearm1d:~ # cat /etc/rear/local.conf
# John L. added the lines below.
OUTPUT=ISO
BACKUP=NETFS
OUTPUT_URL=file:///midrange/backups
BACKUP_URL=sshfs://root@xxx.xxx.xxx.xxx/dbar/
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/u01' '/u02' '/u03')
- Are you using legacy BIOS or UEFI boot? BIOS
- Brief description of the issue:
We use sshfs for backups and recovery on internal networks.. For servers behind firewalls we use BACKUP_URL=file:/// option, then we copy the files to a DBAR server for save keeping.
To recover DMZ servers we edit the BACKUP_URL= to include the correct IP for recovery.
In the past "modprobe fuse" would load the kernel module but now it fails.
RESCUE xrearm1d:~ # modprobe fuse **FATAL: Module fuse not found.**
- Work-around, if any: None.
Is there anything I can do to get fuse to load in the rescue shell?
Thanks for your help.
John L.
gdha commented at 2017-03-19 09:14:¶
@John-Leone Have a look at
https://github.com/rear/rear/commit/f6e54dd02899b0ea4c111ea42f653cc7bd8db121
- it should have been working. After running the mkbackup flow did
you check the log file on fuse module being not found?
Perhaps, it has another name? Check with lsmod | grep fuse
John-Leone commented at 2017-03-19 23:19:¶
@gdha When BACKUP_URL=file:///midrange/backups this is all that's found in the mkbackup flow log.
[root@xrearm1d xrearm1d]# grep fuse rear-xrearm1d.log
2017-03-18 13:31:52 Cloning groups: group disk cdrom floppy tape audio
video lp tty dialout kmem uucp ssh_keys plugdev tty usbmuxd usbmux fuse
kvm oinstall 74 2 32 69 99 29
When BACKUP_URL=sshfs://root@xxx.xxx.xxx.xxx/dbar/ there is much more
found in the mkbackup flow log. See attached file.
rear-xrearm1d-log.txt
gdha commented at 2017-03-20 06:58:¶
@John-Leone the kernel module will only be loaded and copied to initrd
of ReaR when you specify BACKUP_URL=sshfs://root@xxx.xxx.xxx.xxx/dbar/
and that also happened I saw in the latter logfile. Are you sure the
module was not found nor loaded in the recover then?
John-Leone commented at 2017-03-20 11:39:¶
@gdha when BACKUP_URL=sshfs://root@xxx.xxx.xxx.xxx/dbar/ backup &
recovery works fine. But, in the DMZ network we code
BACKUP_URL=file:///midrange/backups for mkbackup, then for recovery we
cd to /etc/rear in the ReaR rescue shell and update local.conf to
BACKUP_URL=sshfs://root@xxx.xxx.xxx.xxx/dbar/ so we can recover the
server. This is when we encounter this error:
RESCUE xrearm1d:~ # modprobe fuse
FATAL: Module fuse not found.
It looks like when mkbackup with BACKUP_URL=file:/// we can't change
and use BACKUP_URL=sshfs://root@xxx.xxx.xxx.xxx/dbar/ for recovery.
I'm not sure if this is by design or if there is a problem?
Let me know if I can provide more information.
gdha commented at 2017-03-20 12:12:¶
@John-Leone OK I understand your problem now - be sure you add
MODULES_LOAD=${MODULES_LOAD[@] fuse } in the /etc/rear/local.conf
file when you run rear mkbackup when the BACKUP_URL variable starts
with file:// as kernel module fuse will only be copied when the
BACKUP_URL variable starts with sshfs://
John-Leone commented at 2017-03-21 01:39:¶
@gdha I tried MODULES_LOAD=${MODULES_LOAD[@] fuse } but got the
following error: "/usr/share/rear/lib/global-functions.sh: line 285:
sshfs: command not found "
Then I updated local.conf to include:
REQUIRED_PROGS=("${REQUIRED_PROGS[@]}" 'sshfs') and I got this
error: "ERROR: Cannot run 'sshfs' because 'fuse' kernel module is not
loadable"
Do you have anything else I can try?
gdha commented at 2017-03-21 07:39:¶
@John-Leone check the rear log to see whether sshfs executable and
fuse modules were copied to the initrd image. Did you try
modprobe fuse?
John-Leone commented at 2017-03-21 09:57:¶
@gdha modprobe fuse worked when just MODULES_LOAD=${MODULES_LOAD[@]
fuse } was in local.conf.
Where do I check the initrd image?
gdha commented at 2017-03-21 10:15:¶
@John-Leone Sorry I meant check the rear logging if it was copied, but you already answered it yourself - yes it was. Does a recover work now?
jsmeix commented at 2017-03-21 10:39:¶
Only side notes FYI (I did not yet read all the detail above):
@gdha
only FYI there was a similar issue
https://github.com/rear/rear/issues/1181
@John-Leone
in general regarding possibly missing modules
in the recovery system see the related issue
https://github.com/rear/rear/issues/1202
Whenever a kernel module is missing in the recovery system
the "big hammer" to get them all included is
COPY_AS_IS=( "${COPY_AS_IS[@]:-}" /lib/modules )
or wherever all the kernel modules are located
for your particular Linux distribution.
John-Leone commented at 2017-03-21 12:04:¶
@gdha @jsmeix I tried the big hammer, "COPY_AS_IS=( "${COPY_AS_IS[@]:-}" /lib/modules )" in the local.conf file and got this error:
From /var/log/rear/rear-xrearm1d.log:
cp: `/lib/modules/2.6.32-642.13.1.el6.x86_64/weak-updates/cnic/cnic.ko' and `/tmp/rear.qgoq0Blk6OqdKfi/rootfs//lib/modules/2.6.32-642.13.1.el6.x86_64/weak-updates/cnic/cnic.ko' are the same file 2017-03-21 07:47:15 ERROR: [ModulesCopyTo] Could not copy '/lib/modules/2.6.32-642.13.1.el6.x86_64/weak-updates/cnic/cnic.ko' to '/tmp/rear.qgoq0Blk6OqdKfi/rootfs' ==== Stack trace ==== Trace 0: /usr/sbin/rear:504 main Trace 1: /usr/share/rear/lib/mkbackup-workflow.sh:15 WORKFLOW_mkbackup Trace 2: /usr/share/rear/lib/framework-functions.sh:85 SourceStage Trace 3: /usr/share/rear/lib/framework-functions.sh:45 Source Trace 4: /usr/share/rear/build/GNU/Linux/400_copy_modules.sh:16 source Trace 5: /usr/share/rear/lib/linux-functions.sh:131 ModulesCopyTo Trace 6: /usr/share/rear/lib/_input-output-functions.sh:132 StopIfError Message: [ModulesCopyTo] Could not copy '/lib/modules/2.6.32-642.13.1.el6.x86_64/weak-updates/cnic/cnic.ko' to '/tmp/rear.qgoq0Blk6OqdKfi/rootfs' == End stack trace == 2017-03-21 07:47:15 Running exit tasks. 2017-03-21 07:47:15 Finished in 243 seconds 2017-03-21 07:47:15 Removing build area /tmp/rear.qgoq0Blk6OqdKfi removed directory: `/tmp/rear.qgoq0Blk6OqdKfi' 2017-03-21 07:47:15 End of program reached
jsmeix commented at 2017-03-21 15:16:¶
Too bad!
Seems a rare case where ReaR errors out too picky.
I will have a look tomorrorw (if time permits).
In the future this should behave (hopefully) better,
i.e. when I have implemented
https://github.com/rear/rear/issues/1202#issuecomment-284377443
gozora commented at 2017-03-21 15:54:¶
@John-Leone
I don't like this:
cp: `/lib/modules/2.6.32-642.13.1.el6.x86_64/weak-updates/cnic/cnic.ko' and `/tmp/rear.qgoq0Blk6OqdKfi/rootfs//lib/modules/2.6.32-642.13.1.el6.x86_64/weak-updates/cnic/cnic.ko' are the same file
Maybe some broken/relative symlinks?
how does
ls -al /lib/modules/2.6.32-642.13.1.el6.x86_64/weak-updates/cnic/cnic.ko
output looks like on your system?
V.
John-Leone commented at 2017-03-21 16:12:¶
@gozora
Here is the output:
[root@xrearm1d ~]# ls -al /lib/modules/2.6.32-642.13.1.el6.x86_64/weak-updates/cnic/cnic.ko lrwxrwxrwx 1 root root 58 Mar 18 13:25 /lib/modules/2.6.32-642.13.1.el6.x86_64/weak-updates/cnic/cnic.ko -> /lib/modules/2.6.32-220.13.1.el6.x86_64/extra/cnic/cnic.ko
gozora commented at 2017-03-21 16:25:¶
Hmm, nothing strange about that ...
can you run rear -d -D mkrescue and post created logfile from
/var/log/rear here?
John-Leone commented at 2017-03-21 16:46:¶
@gozora
Sure, I have attached the file.
rear-xrearm1d-log.txt
gozora commented at 2017-03-21 17:35:¶
Thanks!
Can you please show output of
ls -al /lib/modules/2.6.32-220.13.1.el6.x86_64/extra/cnic/cnic.ko ?
John-Leone commented at 2017-03-21 17:39:¶
@gozora
Thanks for your help.
[root@xrearm1d ~]# ls -al /lib/modules/2.6.32-220.13.1.el6.x86_64/extra/cnic/cnic.ko -rw-r--r-- 1 root root 1065518 May 9 2012 /lib/modules/2.6.32-220.13.1.el6.x86_64/extra/cnic/cnic.ko
gozora commented at 2017-03-21 19:34:¶
Really strange, I can't reproduce this error even with symlinks :-(.
Could you provide outputs from:
df -h /tmp
ls -ald /tmp/rear.*/rootfs//lib/modules/2.6.32-642.13.1.el6.x86_64/weak-updates/cnic
la -al /tmp/rear.*/rootfs//lib/modules/2.6.32-642.13.1.el6.x86_64/weak-updates/cnic
Thanks
V.
John-Leone commented at 2017-03-21 20:17:¶
@gozora Thanks again.
[root@xrearm1d tmp]# df -h /tmp
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rootvg-tmp
7.8G 510M 7.3G 7% /tmp
[root@xrearm1d tmp]# ls -ald /tmp/rear.*/rootfs//lib/modules/2.6.32-642.13.1.el6.x86_64/weak-updates/cnic
drwxr-xr-x 2 root root 4096 Mar 6 17:16 /tmp/rear.DSMokqrzrKyZRrO/rootfs/lib/modules/2.6.32-642.13.1.el6.x86_64/weak-updates/cnic
[root@xrearm1d tmp]# ls -al /tmp/rear.*/rootfs//lib/modules/2.6.32-642.13.1.el6.x86_64/weak-updates/cnic
total 8
drwxr-xr-x 2 root root 4096 Mar 6 17:16 .
drwxr-xr-x 8 root root 4096 Mar 6 17:16 ..
lrwxrwxrwx 1 root root 58 Mar 21 12:35 cnic.ko -> /lib/modules/2.6.32-220.13.1.el6.x86_64/extra/cnic/cnic.ko
jsmeix commented at 2017-03-22 09:10:¶
In the ModulesCopyTo function in lib/linux-functions.sh
I wonder about why there is "cp -a" used
cp $v -a -L "$1" "$dest/$dir" >&2
because '-a' is 'same as -dR --preserve=all'
and '-d' is '--no-dereference --preserve=links'
but '-L' is '--dereference' which contradicts '-d'
and '-R' copies directories recursively
where I wonder why that is needed?
I have the dim feeling the whole modules copying code
may deserve some revision - at least I do not understand it.
@gozora
do you understand the modules copying code?
gozora commented at 2017-03-22 11:16:¶
@jsmeix all I was trying yesterday was to find out why we have
cp: are the same file error, without actually looking at module
copying code.
But looking at
https://github.com/rear/rear/issues/1253#issuecomment-288204867
it looks like that some ReaR code did copy symlink
/lib/modules/2.6.32-220.13.1.el6.x86_64/extra/cnic/cnic.ko to ReaR
build area and some other code try to copy cnic.ko in next step which
caused cp: are the same file.
I'll try to find guilty code later today.
V.
John-Leone commented at 2017-03-23 14:06:¶
@gdha @jsmeix @gozora I was able to successfully complete a recovery in
the DMZ with local.conf using BACKUP_URL=file:///midrange/backups. As
recommended by gdha I added MODULES_LOAD=${MODULES_LOAD[@] fuse } to
local.conf but this doesn't work unless you "modprobe fuse" before you
run "rear mkbackup" then for sshfs to work I added
REQUIRED_PROGS=("${REQUIRED_PROGS[@]}" 'sshfs')
Like I said before on the internal network we use
BACKUP_URL=sshfs://root@xxx.xxx.xxx.xxx/dbar/ and never have problems
with missing modules so I think it would be good if all the modules are
available in the recovery shell when BACKUP_URL=file is used in
local.conf
Thanks to everyone for helping.
[root@xrearm1d ~]# cat /etc/rear/local.conf
OUTPUT=ISO
BACKUP=NETFS
OUTPUT_URL=file:///midrange/backups
BACKUP_URL=file:///midrange/backups
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/u01' '/u02'
'/u03')
MODULES_LOAD=${MODULES_LOAD[@] fuse }
REQUIRED_PROGS=("${REQUIRED_PROGS[@]}" 'sshfs')
[Export of Github issue for rear/rear.]