#2601 Issue `closed`: KERNEL_CMDLINE="netdev=eno1" ignored. Unattended recovery stopped at: original network interface eno1 [...] is not available¶

Labels: support / question, fixed / solved / done

3id0 opened issue at 2021-04-15 12:49:¶

Relax-and-Recover (ReaR) Issue Template¶

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

ReaR version ("/usr/sbin/rear -V"):

Relax-and-Recover 2.4 / Git

OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):

OS_VENDOR=Debian
OS_VERSION=10

ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

OUTPUT=USB
USB_DEVICE=/dev/disk/by-label/REAR-000
GRUB_RESCUE=n
EXCLUDE_BACKUP+=( fs:/media fs:/mnt fs:/dev fs:/proc fs:/sys fs:/tmp fs:/run )

ISO_RECOVER_MODE="unattended"
PXE_RECOVER_MODE="unattended"
USER_INPUT_LAYOUT_MIGRATION_CONFIRM_MAPPINGS=1
USER_INPUT_DISK_LAYOUT_PROCEED_RECOVERY='yes'
USER_INPUT_BORGBACKUP_ARCHIVE_TO_RECOVER=1

KERNEL_CMDLINE="netdev=eno1"
#KERNEL_CMDLINE="net.ifnames=0 netdev=eth0"

POST_RECOVERY_SCRIPT="reboot"

RESULT_FILES=()
RESULT_MAILTO=( '[redacted]' )
RESULT_MAILFROM=root
RESULT_MAILSUBJECT="ReaR recovery results"
RESULT_SENDMAIL="$( type -p sendmail || echo /usr/lib/sendmail )"
RESULT_SENDMAIL_OPTIONS=( -oi -t )

BACKUP=BORG
BORGBACKUP_HOST="192.168.10.10"
BORGBACKUP_USERNAME="backupborg"
BORGBACKUP_REPO="/~/symlink-borg-repos/${HOSTNAME}"
BORGBACKUP_REMOTE_PATH="/usr/local/bin/borg"
BORGBACKUP_PRUNE_KEEP_HOURLY=5
BORGBACKUP_PRUNE_KEEP_WEEKLY=2
BORGBACKUP_COMPRESSION="zlib,9"
BORGBACKUP_ENC_TYPE="repokey-blake2"
export BORG_PASSPHRASE="[redacted]"
export BORG_RELOCATED_REPO_ACCESS_IS_OK="yes"
export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK="yes"
export BORG_RSH="ssh -i /root/.ssh/id_rsa_[redacted]"
SSH_FILES=( '/root/.ssh/id_rsa_[redacted]' '/root/.ssh/known_hosts' )
SSH_UNPROTECTED_PRIVATE_KEYS='yes'

Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR):

PC

System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):

x86_64

Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):

BIOS and GRUB

Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):

local disk

Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT" or "lsblk" as makeshift):

NAME                                   KNAME     PKNAME    TRAN TYPE FSTYPE        SIZE MOUNTPOINT
/dev/sda                               /dev/sda            ata  disk             931.5G
|-/dev/sda1                            /dev/sda1 /dev/sda       part ext2          487M /boot
|-/dev/sda2                            /dev/sda2 /dev/sda       part                 1K
`-/dev/sda5                            /dev/sda5 /dev/sda       part LVM2_member   931G
  |-/dev/mapper/linuxhost17--vg-root   /dev/dm-0 /dev/sda5      lvm  ext4        930.1G /
  `-/dev/mapper/linuxhost17--vg-swap_1 /dev/dm-1 /dev/sda5      lvm  swap          976M [SWAP]

Description of the issue (ideally so that others can reproduce it):

Hello, I'm trying to setup a fully automated/unattended recovery/migration to a another bare metal server (thus, with different mac addresses) from ReaR USB. But before "rear recover" can start, the process stops and asks the following:

The original network interface eno1 <old-mac-address> is not available
1) eno1 <new-mac-address-1> igb
2) eno2 <new-mac-address-2> igb
3) Skip replacing eno1 <old-mac-address>
Choose replacement for eno1 <old-mac-address>

I would like the first answer to always be chosen and never needing user interaction.
I have tried configurations with this line:
KERNEL_CMDLINE="netdev=eno1"
and this line:
KERNEL_CMDLINE="net.ifnames=0 netdev=eth0"
but, even though in the second case, the net.ifnames is correctly taken into account since the prompt displays eth0 instead of eno1, I am still prompted to manually select a network interface.
How to automatically choose the first available network interface?

Workaround, if any:

None

gdha commented at 2021-04-15 14:11:¶

@3id0 Try adding USER_INPUT_TIMEOUT=3 to the local.conf file.

3id0 commented at 2021-04-15 14:17:¶

Thank you for your quick response. I do not have access to the servers right now but I will try that and keep you updated.

jsmeix commented at 2021-04-16 07:23:¶

I think USER_INPUT_TIMEOUT won't help because the code
that asks for the right network interface is in
skel/default/etc/scripts/system-setup.d/55-migrate-network-devices.sh
https://github.com/rear/rear/blob/master/usr/share/rear/skel/default/etc/scripts/system-setup.d/55-migrate-network-devices.sh

echo "The original network interface $old_dev $old_mac is not available"
PS3="Choose replacement for $old_dev $old_mac "
skip_choice="Skip replacing $old_dev $old_mac"
select choice in "${NEW_DEVICES[@]}" "$skip_choice" ; do

and select does not have a timeout or so.

@3id0
attach your "rear -D mkrescue/mkbackup" debug log file cf.
https://github.com/rear/rear/blob/master/.github/ISSUE_TEMPLATE.md
so we could see what goes on on your system.

3id0 commented at 2021-04-23 18:43:¶

Here you go.
I hope I'm not publishing anything important in this log -_-'

rear.log

By the way, is there a best pratice for doing a fully automated/unattended restore with borg via ssh without leaving an unprotected private key everywhere? (current host, rescue image, next host...)
I've used the restrict directive for ssh but still it feels weird leaving this key everywhere without password.

EDIT: Oh yes, and adding the USER_INPUT_TIMEOUT=3 did not prevent the interface prompt to interrupt the unatteded recovery.

jsmeix commented at 2021-04-26 14:25:¶

When I run "rear -D mkrescue" with this etc/rear/local.conf (excerpts)

OUTPUT=ISO
KERNEL_CMDLINE+=" qqqqkey=qqqqvalue"

I get in var/log/rear/rear-linux-h9wr.log (excerpts)

+ source /root/rear.github.master/usr/share/rear/output/ISO/Linux-i386/300_create_isolinux.sh
++ echo 'append initrd=initrd.cgz root=/dev/ram0 vga=normal rw  selinux=0 qqqqkey=qqqqvalue'
++ echo 'append initrd=initrd.cgz root=/dev/ram0 vga=normal rw  selinux=0 qqqqkey=qqqqvalue auto_recover '

@3id0
in contrast in your case I don't see how your KERNEL_CMDLINE is ever
added to the bootloader of the bootable medium via OUTPUT=USB because your
https://github.com/rear/rear/files/6367471/rear-linuxhost17.log
only contains (excerpts)

+ source /usr/share/rear/rescue/GNU/Linux/400_use_serial_console.sh
...
... Modified kernel commandline to: 'netdev=eno1 console=ttyS0,9600 console=ttyS1,9600 console=tty0'
...

In current ReaR code KERNEL_CMDLINE is only used in the following USB scripts:

usr/share/rear/output/USB/Linux-i386/300_create_extlinux.sh
usr/share/rear/output/USB/Linux-i386/100_create_efiboot.sh

where usr/share/rear/output/USB/Linux-i386/300_create_extlinux.sh is actually run
and usr/share/rear/output/USB/Linux-i386/100_create_efiboot.sh is basically skipped
because USB should not do UEFI boot but only traditional BIOS boot.

In usr/share/rear/output/USB/Linux-i386/300_create_extlinux.sh
KERNEL_CMDLINE is in those lines

    append initrd=/$USB_PREFIX/$REAR_INITRD_FILENAME root=/dev/ram0 vga=normal rw $KERNEL_CMDLINE
    append initrd=/$USB_PREFIX/$REAR_INITRD_FILENAME root=/dev/ram0 vga=normal rw $KERNEL_CMDLINE auto_recover

but currently I don't understand what actually happens while
usr/share/rear/output/USB/Linux-i386/300_create_extlinux.sh
is running from what I see in
https://github.com/rear/rear/files/6367471/rear-linuxhost17.log

That obfuscated ReaR USB code always drives me nuts...

3id0 commented at 2021-04-26 17:24:¶

Is there anything I can do to get you a better log of the "syslinux.cfg" step?
Though if I remember correctly, on the ReaR USB boot menu, I once tried editing the boot options and I could see netdev=eno1 in there along with the other options.

jsmeix commented at 2021-04-28 12:38:¶

I tested KERNEL_CMDLINE+="..." with OUTPUT=USB
on a USB disk that is connected as /dev/sdb
to my homeoffice laptop with hostname linux-h9wr
with the following settings in etc/rear/local.conf

OUTPUT=USB
BACKUP=NETFS
BACKUP_URL=usb:///dev/disk/by-label/REAR-000
USB_DEVICE=/dev/disk/by-label/REAR-000
USB_SUFFIX="ReaRtestBackup2"
KERNEL_CMDLINE+=" QQQQQQ=qqqqqq"

I am used to use USB_SUFFIX because I prefer to have things stored on USB
in the same way as via NFS, see the comment about USB_SUFFIX in default.conf
but that should not make a real difference regarding kernel command line options.

I got the kernel command line option added to the bootloader config on the USB:

# mount /dev/sdb1 /mnt

# find /mnt -type f -name 'syslinux.cfg' | xargs grep QQQQQQ
/mnt/rear/linux-h9wr/ReaRtestBackup2/syslinux.cfg:
    append initrd=/rear/linux-h9wr/ReaRtestBackup2/initrd.cgz root=/dev/ram0 vga=normal rw  selinux=0 QQQQQQ=qqqqqq
/mnt/rear/linux-h9wr/ReaRtestBackup2/syslinux.cfg:
    append initrd=/rear/linux-h9wr/ReaRtestBackup2/initrd.cgz root=/dev/ram0 vga=normal rw  selinux=0 QQQQQQ=qqqqqq auto_recover

The rear/linux-h9wr/ReaRtestBackup2/syslinux.cfg
contents get included by the rear/syslinux.cfg file via

# less /mnt/rear/syslinux.cfg
...
    include /rear/linux-h9wr/ReaRtestBackup2/syslinux.cfg
...

so that the additional kernel command line option should be passed to the kernel
when booting the ReaR recovery system from that USB disk but I did not test it
because I won't boot my homeoffice laptop now from USB (I need it to work on it).

@3id0
to verify whether or not your additional kernel command line option
was actually passed to the kernel of your ReaR recovery system
boot your ReaR recovery system, log in as root, and check the output
of cat /proc/cmdline and the output of dmesg if that kernel command
line option is shown.

3id0 commented at 2021-04-29 14:16:¶

# cat /proc/cmdline
BOOT_IMAGE=/rear/linuxhost17/20210429.1535/kernel initrd=/rear/linuxhost17/20210429.1535/initrd.cgz root=/dev/ram0 vga=normal rw netdev=eno1 console=ttyS0,9600

So yes, netdev option is indeed passed to the kernel of the recovery system.
(Copied from a picture, please forgive the typos if any.)
What's next?

jsmeix commented at 2021-04-29 15:24:¶

@3id0
So your additional kernel command line option does not result
what you expect it to result so the root cause seems to be now
that something while starting up the ReaR recovery system
even with that kernel command line option does not result
the ReaR recovery system networking setup as you want it to be.

In general reegarding ReaR recovery system networking setup we have
the special ip= nm= netdev= gw= kernel command line options, see the section
"RESCUE IMAGE KERNEL COMMAND LINE OPTIONS" in man rear
https://github.com/rear/rear/blob/master/doc/rear.8.adoc
and we also have NETWORKING_PREPARATION_COMMANDS
see its description in default.conf
https://github.com/rear/rear/blob/master/usr/share/rear/conf/default.conf

Perhaps those could help you to get the ReaR recovery system
networking setup as you want it to be in your particular case?

But I fear those won't help because I fear the recovery system setup script
etc/scripts/system-setup.d/55-migrate-network-devices.sh
which shows the The original network interface ... is not available input prompt
runs before the other setup scripts that do the actual networking setup.

3id0 commented at 2021-04-29 16:26:¶

I don't remember where I found out about KERNEL_CMDLINE but I'm pretty sure there were no mention of the "unattended" option...
I just tried KERNEL_CMDLINE="netdev=eno1 unattended" after checking your links and it works ! (no prompt about the network interface)
Thanks a lot @jsmeix !

jsmeix commented at 2021-04-30 11:59:¶

@3id0
thank you for your feedback what helped in your case!

The `KERNEL_CMDLINE="... unattended" points to
this code in system-setup.d/55-migrate-network-devices.sh

if unattended_recovery ; then
    # we gonna cheat a bit and say we have map made (but we did not and just hope that the interfaces
    # will be in the same order on the recover vm as on the client vm)
    # For some background info see https://github.com/gdha/rear-automated-testing/issues/36
    test $MANUAL_MAC_MAPPING || MANUAL_MAC_MAPPING=unattended
fi

and - voila! - the right ReaR upstream issue was finally found
https://github.com/gdha/rear-automated-testing/issues/36
that shows more background information and details behind.

3id0 commented at 2021-04-30 12:31:¶

It's too bad the line "the original network interface [...] is not available" in issue #36 was in another repo and in a picture (not in text) because I searched for this specific string both in the issues of the rear repo and in DuckDuckGo before posting. If I had found out about issue #36 this would have saved all of us a lot of time ^^' well I did learn a lot about the internals of ReaR along the way so it's not time completely wasted for me at least. And I guess the next person who searches this string in a search engine should find this issue.

A suggestion:
I looked around and I'm pretty sure I read about KERNEL_CMDLINE in default.conf And, as you can see, in this file there is no mention of unattended or netdev options, these are only mentionned in the manpage.
So, I would suggest copying the RESCUE IMAGE KERNEL COMMAND LINE OPTIONS part from the manpage to default.conf for better documentation (or at least mentioning this section of the manpage in default.conf).

jsmeix commented at 2021-04-30 12:58:¶

Improved description of the ReaR specific special KERNEL_CMDLINE settings in default.conf via
https://github.com/rear/rear/commit/7b1354d6f3be145620e371a43da4affa29df5924

3id0 commented at 2021-04-30 13:02:¶

Thank you!

jsmeix commented at 2021-04-30 13:04:¶

@3id0
you are welcome!

FYI
what I basically always do to get an idea how things are in ReaR
when I have only some ReaR string as entry point is

# cd usr/share/rear/

# find . -type f | xargs grep '<some_ReaR_string>'

and then I inspect the found code parts.

3id0 commented at 2021-04-30 13:07:¶

I feel a bit stupid for never thinking of that but that's a very good suggestion. Will do next time, thank you.

jsmeix commented at 2021-04-30 13:28:¶

Don't worry,
I feel somewhat stuipd all the time when I deal with weird ReaR issues ;-)

I wish you a relaxed and recovering weekend!

jsmeix commented at 2021-05-03 11:13:¶

I wonder if we may also need similar (or perhaps even same) code in
etc/scripts/system-setup.d/55-migrate-network-devices.sh
for the automatic_recovery case
(i.e. when kernel command line contains automatic or auto_recover)
as we already have there for the unattended_recovery case
(i.e. when kernel command line contains unattended)
?

3id0 commented at 2021-06-02 16:56:¶