#2414 Issue `closed`: fedora 32 mkbackup fails due to missing module file¶

Labels: enhancement, support / question, fixed / solved / done

jholland-vin-dit opened issue at 2020-05-30 16:48:¶

Relax-and-Recover (ReaR) Issue Template¶

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

ReaR version ("/usr/sbin/rear -V"):Relax-and-Recover 2.4 / 2018-06-21
OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):ID=fedora
VERSION_ID=32
ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

KEEP_BUILD_DIR="yes"
OUTPUT=ISO
OUTPUT_URL=file:///home/rear
BACKUP=NETFS
BACKUP_URL="cifs://192.168.61.122/FED5"
#BACKUP_URL="file:///mnt/backup"
BACKUP_OPTIONS="cred=/etc/rear/cifs"
BACKUP_PROG_EXCLUDE=( "${BACKUP_PROG_EXCLUDE[@]}" '/home/jholland/rhel8.0'  )

Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR):
PC
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
x86_64
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
UEFI
Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
SSD
Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT" or "lsblk" as makeshift):

NAME                          KNAME PKNAME TRAN   TYPE  FSTYPE   SIZE MOUNTPOINT
/dev/sda                      /dev/sda
|                                          sata   disk         465.8G 
|-/dev/sda1                   /dev/sda1
|                                   /dev/sda
|                                                 part  vfat     600M /boot/efi
|-/dev/sda2                   /dev/sda2
|                                   /dev/sda
|                                                 part  ext4       1G /boot
`-/dev/sda3                   /dev/sda3
                                    /dev/sda
                                                  part  crypto 464.2G 
  `-/dev/mapper/luks-4d4441c7-3072-4ce7-9d44-4fe0c5891200
                              /dev/dm-0
                                    /dev/sda3
                                                  crypt LVM2_m 464.2G 
    |-/dev/mapper/fedora_localhost--live-root
    |                         /dev/dm-1
    |                               /dev/dm-0
    |                                             lvm   ext4      70G /
    |-/dev/mapper/fedora_localhost--live-swap
    |                         /dev/dm-2
    |                               /dev/dm-0
    |                                             lvm   swap    23.8G [SWAP]
    `-/dev/mapper/fedora_localhost--live-home
                              /dev/dm-3
                                    /dev/dm-0
                                                  lvm   ext4   370.4G /home
/dev/sr0                      /dev/sr0
                                           sata   rom           1024M

Description of the issue (ideally so that others can reproduce it):

Copying kernel modules
ERROR: nls_cp437 exists but no module file?

Workaround, if any:
Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):

To paste verbatim text like command output or file content,
include it between a leading and a closing line of three backticks like

```
verbatim content
```

jsmeix commented at 2020-06-02 07:23:¶

@jholland-vin-dit
the error exit happens in
usr/share/rear/build/GNU/Linux/400_copy_modules.sh
https://github.com/rear/rear/blob/master/usr/share/rear/build/GNU/Linux/400_copy_modules.sh#L156

I am not a Fedora user so I cannot reproduce what goes wrong
on Fedora 32 that a kernel module nls_cp437 seems to exist
but without a file that contains the module.

E.g. on my openSUSE Leap 15.1 system things look normal:

# modinfo -F filename nls_cp437
/lib/modules/4.12.14-lp151.28.48-default/kernel/fs/nls/nls_cp437.ko

# KERNEL_VERSION="$( uname -r )"
# modinfo -k $KERNEL_VERSION -F filename nls_cp437
/lib/modules/4.12.14-lp151.28.48-default/kernel/fs/nls/nls_cp437.ko

You could inspect the ReaR log file
when you run it in debug mode like "rear -D mkrescue/mkbackup"
what happend during build/GNU/Linux/400_copy_modules.sh
regarding the nls_cp437 module, cf.
"Debugging issues with Relax-and-Recover" in
https://en.opensuse.org/SDB:Disaster_Recovery

As a workaround try in etc/rear/local.conf

MODULES=( 'all_modules' )

which runs simpler code in build/GNU/Linux/400_copy_modules.sh
that should not fail because of whatever inconsistencies of kernel
module names versus kernel module files.

Read the MODULES description in your
usr/share/rear/conf/default.conf file.

jholland-vin-dit commented at 2020-06-02 10:29:¶

Thanks, that got me past that hurdle!

jholland-vin-dit commented at 2020-06-02 11:14:¶

Thanks, that got me past that hurdle!

On Jun 2, 2020, at 3:23 AM, Johannes Meixner notifications@github.com wrote:

@jholland-vin-dit https://github.com/jholland-vin-dit
the error exit happens in
usr/share/rear/build/GNU/Linux/400_copy_modules.sh
https://github.com/rear/rear/blob/master/usr/share/rear/build/GNU/Linux/400_copy_modules.sh#L156 https://github.com/rear/rear/blob/master/usr/share/rear/build/GNU/Linux/400_copy_modules.sh#L156
I am not a Fedora user so I cannot reproduce what goes wrong
on Fedora 32 that a kernel module nls_cp437 seems to exist
but without a file that contains the module
like on my openSUSE Leap 15.1 system:

modinfo -F filename nls_cp437¶

/lib/modules/4.12.14-lp151.28.48-default/kernel/fs/nls/nls_cp437.ko

KERNEL_VERSION="$( uname -r )"¶

modinfo -k $KERNEL_VERSION -F filename nls_cp437¶

/lib/modules/4.12.14-lp151.28.48-default/kernel/fs/nls/nls_cp437.ko
You could inspect the ReaR log file
when you run it in debug mode like "rear -D mkrescue/mkbackup"
what happend during build/GNU/Linux/400_copy_modules.sh
regarding the nls_cp437 module, cf.
"Debugging issues with Relax-and-Recover" in
https://en.opensuse.org/SDB:Disaster_Recovery https://en.opensuse.org/SDB:Disaster_Recovery
As a workaround try in etc/rear/local.conf

MODULES=( 'all_modules' )
which runs simpler code in build/GNU/Linux/400_copy_modules.sh
that should not fail because of whatever inconsistencies of kernel
module names versuse kernel module files.

Read the MODULES description in your
usr/share/rear/conf/default.conf file.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/rear/rear/issues/2414#issuecomment-637344902, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAYEXR6PJ2KR6AFMAFH5S6LRUSSIVANCNFSM4NOYO2TQ.

jsmeix commented at 2020-06-02 11:21:¶

@jholland-vin-dit
thanks for your feedback that things work now for you.
It is important for us at ReaR upstream to get such explicit feedback
so we know whether or not there could be a hidden issue inside ReaR.
FYI:
Since ReaR 2.5 MODULES=( 'all_modules' ) is the default, cf.
https://github.com/rear/rear/blob/master/doc/rear-release-notes.txt#L211
so this issue should no longer happen with ReaR 2.5 default behaviour.

jholland-vin-dit commented at 2020-06-02 11:30:¶

Well, that got me past that hurdle. Then I ran into another one with https://github.com/rear/rear/issues/1996 I followed the advice near the bottom of that issue page to¶

remove the linuxefi module which seems to not exist anymore,
ending up with having same line as for slackware.

I have followed your sugestion and also removed multiboot (it was causing the same problems as linuxefi)¶

Now my backup has completed and created the big tarball and iso.
I am going to try a restore when I get a spare HD to restore onto.

It's great to get the help, thank you!

jsmeix commented at 2020-06-02 11:40:¶

@jholland-vin-dit
in our current ReaR upstream master code we have substantial changes
how UEFI booting of the ReaR recovery system ISO is implemented,
in particular see https://github.com/rear/rear/pull/2293

So you may like to try out our current ReaR upstream GitHub master code
from within a separated directory as a test to find out if things work better
with current ReaR upstream master code compared to your installed
ReaR version, see the section
"Testing current ReaR upstream GitHub master code" in
https://en.opensuse.org/SDB:Disaster_Recovery

Of course I am totally selfish when I suggest you should test
our current ReaR upstream GitHub master code because
we at ReaR upstream need tests with recent Fedora versions ;-)

jholland-vin-dit commented at 2020-06-02 11:46:¶

Sorry, but I probably won’t do that very soon. Maybe sometime in the future.

On Jun 2, 2020, at 7:40 AM, Johannes Meixner notifications@github.com wrote:

@jholland-vin-dit https://github.com/jholland-vin-dit
in our current ReaR upstream master code we have substantial changes
how UEFI booting of the ReaR recovery system is implemented, in
particular see #2293 https://github.com/rear/rear/pull/2293
So you may like to try out our current ReaR upstream GitHub master code
from within a separated directory as a test to find out if things work better
with current ReaR upstream master code compared to your installed
ReaR version, see the section
"Testing current ReaR upstream GitHub master code" in
https://en.opensuse.org/SDB:Disaster_Recovery https://en.opensuse.org/SDB:Disaster_Recovery
Of course I am totally selfish when I suggest you should test
our current ReaR upstream GitHub master code because
we at ReaR upstream need tests with recent Fedora versions ;-)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/rear/rear/issues/2414#issuecomment-637483457, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAYEXRZ3KBCQKC76JNPRC73RUTQLBANCNFSM4NOYO2TQ.

jholland-vin-dit commented at 2020-06-03 22:23:¶

So I got a spare hard drive and tried to restore my fedora 32 backup. It seemed to be having problems with the source disk being encrypted? It looked like it was trying to prompt for a password but it wasn’t letting me enter it. I tried going in the rear shell but didn’t see anything that looked helpful.

On Jun 2, 2020, at 7:46 AM, John Holland johnbholland@icloud.com wrote:

Sorry, but I probably won’t do that very soon. Maybe sometime in the future.

On Jun 2, 2020, at 7:40 AM, Johannes Meixner notifications@github.com wrote:

@jholland-vin-dit
in our current ReaR upstream master code we have substantial changes
how UEFI booting of the ReaR recovery system is implemented, in
particular see #2293

So you may like to try out our current ReaR upstream GitHub master code
from within a separated directory as a test to find out if things work better
with current ReaR upstream master code compared to your installed
ReaR version, see the section
"Testing current ReaR upstream GitHub master code" in
https://en.opensuse.org/SDB:Disaster_Recovery

Of course I am totally selfish when I suggest you should test
our current ReaR upstream GitHub master code because
we at ReaR upstream need tests with recent Fedora versions ;-)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

jsmeix commented at 2020-06-04 07:28:¶

@jholland-vin-dit
I am not a CIFS user so I won't reproduce CIFS issues
or could even fix possible bugs with CIFS in ReaR.

From plain looking at the ReaR documentation that
mentiones cifs I see in "man rear"

BACKUP_URL=cifs://
To backup to a Samba share (CIFS), use
BACKUP_URL=cifs://cifs-server-name/share/path.
To provide credentials for CIFS mounting use a
/etc/rear/cifs credentials file and define
BACKUP_OPTIONS="cred=/etc/rear/cifs"
and pass along:

username=_username_
password=_secret password_
domain=_domain_

and in usr/share/rear/conf/examples/USB-and-Samba-example.conf

# Backup to a network share:
BACKUP=NETFS
# When using CIFS as backup/output destination make sure that cifs-utils package is installed
# Path to the cifs share:
BACKUP_URL=cifs://server/share
# Username and password of a valid Samba/Active Directory user:
BACKUP_OPTIONS="username=sambauser,pass=password"

so in both cases username and the secret password
must be provided either in a /etc/rear/cifs file
or directly in the BACKUP_OPTIONS config variable
which means the ReaR recovery system contains
the secret password.

From plain looking at the ReaR code that is related to cifs I see
in usr/share/rear/backup/NETFS/default/100_mount_NETFS_path.sh

mount_url $BACKUP_URL $BUILD_DIR/outputfs $BACKUP_OPTIONS

that leads to usr/share/rear/lib/global-functions.sh

### Mount URL $1 at mountpoint $2[, with options $3]
mount_url() {
...
    case $(url_scheme $url) in
    ....
        (cifs)
            if [ x"$options" = x"$defaultoptions" ];then
                # defaultoptions contains noatime which is not valid for cifs (issue #752)
                mount_cmd="mount $v -o rw,guest //$(url_host $url)$(url_path $url) $mountpoint"
            else
                mount_cmd="mount $v -o $options //$(url_host $url)$(url_path $url) $mountpoint"
            fi
            ;;

so your BACKUP_OPTIONS="cred=/etc/rear/cifs"
should result a mount command of the form

mount -v -o cred=/etc/rear/cifs ...

to mount the CIFS share at a local mointpoint.

As far as I see there is currently no way in ReaR how to provide
CIFS username and the secret password in an interactive way
via some kind of user dialog in ReaR.

In "man mount.cifs" I see

OPTIONS
...
password=arg|pass=arg
specifies the CIFS password. If this option is not given
then the environment variable PASSWD is used.
If the password is not specified directly or indirectly
via an argument to mount,
mount.cifs will prompt for a password,
unless the guest option is specified.
Note that a password which contains the delimiter
character (i.e. a comma ',') will fail to be parsed correctly
on the command line. However, the same password defined
in the PASSWD environment variable or via a credentials
file (see below) or entered at the password prompt
will be read correctly.

credentials=filename|cred=filename
specifies a file that contains a username and/or password
and optionally the name of the workgroup.
The format of the file is:
    username=value
    password=value
    domain=value
This is preferred over having passwords in plaintext
in a shared file, such as /etc/fstab .
Be sure to protect any credentials file properly.

Accordingly it seems in your case it is mount.cifs that
prompts for a password because you did not provide
it via ReaR options or via the environment variable PASSWD
before you launched "rear recover".

But you won't see when mount.cifs prompts for a password
because while usr/sbin/rear is running both stdout and stderr
are redirected into ReaR's log file, cf.
"What to do with stdin, stdout, and stderr" in
https://github.com/rear/rear/wiki/Coding-Style

jsmeix commented at 2020-06-04 07:42:¶

@jholland-vin-dit
according to your lsblk output you use
LVM on LUKS encrypted volume.

So when you made the CIFS access working for you
the next possible issue could be already waiting for you:
https://github.com/rear/rear/issues/2389

jholland-vin-dit commented at 2020-06-04 12:45:¶

Thanks for looking into it,
I think I will wait a bit before trying REAR on Fedora 32 again. I like REAR when it works as it did on RHEL 8 for me. It’s too disruptive for me to test/troubleshoot it on Fedora though.

On Jun 4, 2020, at 3:42 AM, Johannes Meixner notifications@github.com wrote:

@jholland-vin-dit
according to your lsblk output you use
LVM on LUKS encrypted volume.

So when you made the CIFS access working for you
the next possible issue could be already waiting for you:
#2389

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

z1atk0 commented at 2020-08-04 14:33:¶

Just for kicks (I only upgraded to FC32 yesterday and got bitten by this problem today), adding the following line to usr/share/rear/build/GNU/Linux/400_copy_modules.sh at line 108 right after the line containing

modinfo $module 1>/dev/null || continue

fixes the problem for me:

grep -q $(echo $module | tr '_-' '..') /lib/modules/$KERNEL_VERSION/modules.builtin && continue

Feel free to use this workaround if you want. :slightly_smiling_face:

Bye,
Thomas

jholland-vin-dit commented at 2020-08-04 20:17:¶

This is interesting. I don’t think I’m going to continue trying REAR for Fedora as doing full backups of Fedora seems silly since it is such a moving target. Congratulations on solving it though!

On Aug 4, 2020, at 10:33 AM, Thomas Zajic notifications@github.com wrote:

Just for kicks (I only upgraded to FC32 yesterday and got bitten by this problem today), adding the following line to usr/share/rear/build/GNU/Linux/400_copy_modules.sh at line 108 (right after the line containing modinfo $module 1>/dev/null || continue) fixes the problem for me:
grep -q $(echo $module | tr '_-' '..') /lib/modules/$KERNEL_VERSION/modules.builtin && continue
Feel free to use this workaround if you want. 🙂

Bye,
Thomas

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/rear/rear/issues/2414#issuecomment-668632798, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAYEXR6IOWOCCSCZJP67FLDR7AL4NANCNFSM4NOYO2TQ.

jsmeix commented at 2020-08-05 10:29:¶

@z1atk0
thank you for your
https://github.com/rear/rear/issues/2414#issuecomment-668632798

I implemented it for current GitHub master code via
https://github.com/rear/rear/pull/2469

FYI:
I am not a Fedora user so I cannot reproduce Fedora specific issues
but on my openSUSE Leap 15.1 system I noticed that there are modules
that are listed in /lib/modules/*/modules.builtin and also exist as a file:

# uname -a
Linux linux-h9wr 4.12.14-lp151.28.52-default ...

# for builtin_module in $( cat /lib/modules/4.12.14-lp151.28.52-default/modules.builtin ) ; do \
 module_name=$( basename $builtin_module | cut -d '.' -f1 ) ; \
 modinfo $module_name &>/dev/null && echo builtin module $builtin_module found by modinfo ; done
builtin module kernel/lib/crc32.ko found by modinfo

# grep crc32 /lib/modules/4.12.14-lp151.28.52-default/modules.builtin
kernel/crypto/crc32c_generic.ko
kernel/lib/crc32.ko

# modinfo crc32 -F filename
/lib/modules/4.12.14-lp151.28.52-default/kernel/arch/x86/crypto/crc32-pclmul.ko
/lib/modules/4.12.14-lp151.28.52-default/kernel/crypto/crc32_generic.ko

With https://github.com/rear/rear/pull/2469
the new behaviour is that when modules are listed in modules.builtin
and are shown by modinfo then those modules are skipped.
Before https://github.com/rear/rear/pull/2469 for such modules
the modules file(s) would have been included in the recovery system.
This would be a backward incompatible change if we had not already
MODULES=( 'all_modules' ) by default so the different behaviour
with https://github.com/rear/rear/pull/2469 should affect only exceptional cases.

jsmeix commented at 2020-08-10 14:24:¶

With https://github.com/rear/rear/pull/2469 merged
the initially described issue "... fails due to missing module file"
should be avoided.

[Export of Github issue for rear/rear.]

#2414 Issue closed: fedora 32 mkbackup fails due to missing module file¶