#2992 Issue `open`: Rear not honoring kernel parameter net.ifnames=0 and renaming interfaces (question)¶

Labels: support / question

ccamacho opened issue at 2023-05-25 08:29:¶

Hi,

We are facing an issue after running Rear in some of our testing environments.

We have a Jenkins pipeline that runs Rear in order to test that we are able to do some backups of some guest VMs, after restoring the backup images we noticed that the guests are unresponsive and this is because the network interfaces are renamed from eth0 to ens3, eth1 to ens4 and eth2 to ens5.

This is something that should be fixed as we should be using predictable naming for all the network interfaces.

The initial state of the VMs is (net.ifnames=0)
Before restoring the images:

[root@undercloud-0 stack]# cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt3)/vmlinuz-5.14.0-283.el9.x86_64 root=UUID=62b51192-13b0-4838-a267-e410f86ee01e console=tty0 console=ttyS0,115200n8 no_timer_check net.ifnames=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M

After restoring the images:

[root@undercloud-0 stack]# cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt3)/vmlinuz-5.14.0-283.el9.x86_64 root=UUID=62b51192-13b0-4838-a267-e410f86ee01e ro

Not having 'net.ifnames=0' anymore.

We played with the following configuration parameters in the local.conf without any additional change.

Default values.
COPY_KERNEL_PARAMETERS=( 'net.ifnames')
GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0"

If we reboot the guest VMs without running Rear there is no change in the interface naming.

When creating the backups of the images we see these lines:

2023-05-09 13:59:33.470942363 Adding net.ifnames=0 to KERNEL_CMDLINE
2023-05-09 13:59:33.711087009 Modified kernel commandline to: 'selinux=0 net.ifnames=0 console=ttyS0,115200 console=tty0

And when running the restore we have this:

2023-05-11 11:30:21.811225857 Including finalize/GNU/Linux/300_create_mac_mapping.sh
2023-05-11 11:30:21.822929609 Including finalize/GNU/Linux/310_migrate_udev_rules.sh
2023-05-11 11:30:21.827879926 Including finalize/GNU/Linux/320_migrate_network_configuration_files.sh
2023-05-11 11:30:21.830587664 Patching /mnt/local/etc/sysconfig/network-scripts/ifcfg-br-ctlplane
2023-05-11 11:30:21.833158409 Patching /mnt/local/etc/sysconfig/network-scripts/ifcfg-ens3
2023-05-11 11:30:21.835679147 Patching /mnt/local/etc/sysconfig/network-scripts/ifcfg-eth0
2023-05-11 11:30:21.838289485 Patching /mnt/local/etc/sysconfig/network-scripts/ifcfg-eth1
2023-05-11 11:30:21.840801955 Patching /mnt/local/etc/sysconfig/network-scripts/ifcfg-eth2
2023-05-11 11:30:21.843472096 Patching /mnt/local/etc/sysconfig/network-scripts/ifcfg-lo

We would like to understand what change is introducing Rear that is triggering the renaming of the network interfaces, we are using Relax-and-Recover 2.6 / 2020-06-17.

Is there a BACKUP_RESTORE_MOVE_AWAY_FILES, POST_RECOVERY_SCRIPT or PRE_RECOVERY_SCRIPT configuration parameter we could use to avoid having udevadm trigger renaming all the network interfaces?

There is also another reference #1818 where we might have a similar situation, that is, Rear is not able to restore the previous network configuration and the init scripts are regenerated.

Thanks,
@fdiazbra and @ccamacho

jsmeix commented at 2023-05-25 09:14:¶

@ccamacho
as far as I know the "Modified kernel commandline" stuff only belongs
to the kernel commandline of the ReaR recovery system,
see usr/share/rear/conf/default.conf where
KERNEL_CMDLINE and COPY_KERNEL_PARAMETERS are described
only for the "rescue/recovery system" / "rescue image"
(i.e. the ReaR recovery system).

Off the top of my head I think how the kernel commandline
of the original system is saved and recovered happens as
follows:
In the "finalize" stage the bootloader gets installed
via 'chroot' inside the recreated target system
after the backup was restored so the bootloader gets installed
with the restored bootloader config files from the backup.
For example on my current openSUSE Leap 15.4 system I have
in /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="resume=... mitigations=..."

so this kernel commandline parameters are used when GRUB2
is installed and this kernel commandline parameters appear
in my /boot/grub2/grub.cfg like

linux   /boot/vmlinuz... root=... resume=... mitigations=...

Accordingly I assume that in your case your bootloader config files
on your original system may not contain the kernel commandline
parameters which you are missing.

Regarding finalize/GNU/Linux/320_migrate_network_configuration_files.sh
see the comments in this script how it works.
When those automated more or less hardcoded modifications
of the restored files get in your way the direct workaround
is to skip those automated modifications by skipping the whole
finalize/GNU/Linux/320_migrate_network_configuration_files.sh
by inserting an additional

return 0

line at the beginning of this script - also as needed for
other "finalize" scripts that modify restored files.
You can add that 'return 0' line
either
when you are logged in as 'root' inside the ReaR recovery system
before you run "rear recover"
or
you add that line on your original system and recreate the
ReaR recovery system via "rear mkrescue" or "rear mkbackup".

jsmeix commented at 2023-05-25 09:16:¶

@pcahyna
do you know @ccamacho ?
I ask because
https://github.com/ccamacho
shows him as belonging to RedHatOfficial
https://github.com/RedHatOfficial

pcahyna commented at 2023-05-25 09:45:¶

@jsmeix thanks for noticing, I contacted them

jsmeix commented at 2023-05-25 09:58:¶

@pcahyna
thank you for having a look here!

pcahyna commented at 2023-05-25 11:01:¶

2023-05-11 11:30:21.830587664 Patching /mnt/local/etc/sysconfig/network-scripts/ifcfg-br-ctlplane
2023-05-11 11:30:21.833158409 Patching /mnt/local/etc/sysconfig/network-scripts/ifcfg-ens3
2023-05-11 11:30:21.835679147 Patching /mnt/local/etc/sysconfig/network-scripts/ifcfg-eth0
2023-05-11 11:30:21.838289485 Patching /mnt/local/etc/sysconfig/network-scripts/ifcfg-eth1
2023-05-11 11:30:21.840801955 Patching /mnt/local/etc/sysconfig/network-scripts/ifcfg-eth2
2023-05-11 11:30:21.843472096 Patching /mnt/local/etc/sysconfig/network-scripts/ifcfg-lo

it would be interesting to know what changes (if any) have been made by the patching process. @ccamacho can you please diff these files after recovery process and before the recovery process?
Also, how are the interfaces called in the rescue system? And what is the kernel command line in the rescue system? I suppose it has net.ifnames=0 because of your KERNEL_CMDLINE and/or COPY_KERNEL_PARAMETERS settings, and interfaces are called the same as in the original install - eth1, eth2, eth3, but I would like to confirm that.
Also, are you restoring to exactly the same hardware as the backup was made on, or to some replacement hardware? (Are the network interfaces and their MAC addresses the same in the original and recovered system?)

fdiazbra commented at 2023-05-25 12:32:¶

The files: ifcfg-eth0, ifcfg-eth1, ifcfg-eth2, ifcfg-br-ctlplane, and ifcfg-lo identical in the original and rescued system. The file ifcfg-ens3 only exists in the rescued machine.
The interface names are called "ethX" in the rescue system, as in the original deployment.
About the hardware, there is no replacement. We have both vms over the same hypervisor.

pcahyna commented at 2023-05-25 13:36:¶

after some discussion it looks that re-adding net.ifnames=0 to the recovered system makes this work again. So the only question is why is the whole console=tty0 console=ttyS0,115200n8 no_timer_check net.ifnames=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M part missing from the restored kernel command line, and I smell some misconfiguration here, aka GIGO (garbage in, garbage out).

pcahyna commented at 2023-05-25 16:21:¶

We played with the following configuration parameters in the local.conf without any additional change.
* Default values.

* COPY_KERNEL_PARAMETERS=( 'net.ifnames')

* GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0"

Those additions shouldn't be necessary, default.conf contains this:

# - Check net.ifnames and biosdevname kernel parameter as it may impact the network interface name during recovery/migration.
COPY_KERNEL_PARAMETERS=( 'net.ifnames' 'biosdevname' )

[Export of Github issue for rear/rear.]

#2992 Issue open: Rear not honoring kernel parameter net.ifnames=0 and renaming interfaces (question)¶