#3270 Issue closed: reduzing the initrd size?

Labels: support / question

damm620 opened issue at 2024-07-09 09:03:

With the option FIRMWARE_FILES one can minimize the firmwares that are being loaded into the ReaR initrd.

We currently have the issue, that on CentOS 9 servers we do have big initrds (like 500-650MB). We try to reduze that, but the FIRMWARE_FILES parameter only allows a "whitelist" on what firmware will be loaded.

Is it possible to blacklist firmware?
I am asking this because than I would go though the lib*/firmware folder and blacklist firmwares that are pretty big in size but will probably never used within the rear environment.

If that is not possible, can this feature please be developed?

Is there any other option that minimize the firmwares that are being loaded into the ReaR initrd to reduze the initrd size?
The only other option I see is the REAR_INITRD_COMPRESSION which we already discovered. But the lzma compression only shrinks the size about 10%.

gdha commented at 2024-07-09 09:20:

Did you try FIRMWARE_FILES=( 'no' ) already?

damm620 commented at 2024-07-09 09:37:

As I understand, that means ReaR does not load any firmware, correct?
Can you please explain for what ReaR needs firmwares?

jsmeix commented at 2024-07-09 09:39:

@damm620
I think right now the most simple and straightforward way
to get what you need is to add a new script like
usr/share/rear/build/default/980_remove_needless.sh
where you do something like

pushd $ROOTFS_DIR 1>&2
  rm relative_path/to/files
popd 1>&2

to remove what you don't need from within
your particular ReaR recovery system.

jsmeix commented at 2024-07-09 09:45:

@damm620
ReaR does not need any firmware nor does ReaR load any.
Certain hardware needs firmware and the kernel loads it.
For example NICs and GPUs (GPU firmware is often big).
Without proper firmware such hardware won't work.
Bad when the kernel booted fine (except firmware loading)
and the ReaR recovery system is up and running
but you sit in front of a black screen without network access ;-)
In general fully virtualized VMs usually don't need firmware.
It depends on your hardware and/or on the VM if firmware is needed.
So try out if the ReaR recovery system works on your hardware
with FIRMWARE_FILES=( 'no' ) and if yes all is fine.
Otherwise - provided you know what firmware files
your hardware needs - specify the needed firmware
via FIRMWARE_FILES.

damm620 commented at 2024-07-09 10:14:

So as long as I can boot into rear (grub or ISO), see all disk, have network connectivity and the tsm backup client works, the test should be sufficient right?

jsmeix commented at 2024-07-09 10:24:

Yes, this should be sufficient to check
that the ReaR recovery system works for your needs.

But in general see
https://en.opensuse.org/SDB:Disaster_Recovery#No_disaster_recovery_without_testing_and_continuous_validation

jsmeix commented at 2024-07-09 13:09:

An addedum regarding

... provided you know what firmware files your hardware
needs - specify the needed firmware via FIRMWARE_FILES

see
https://github.com/rear/rear/issues/2681#issuecomment-930879162
why it is not easy to know or to find out what
firmware files get used on one's own hardware.
There is nothing we can do in ReaR against that
(i.e. we cannot determine what firmware files are used)
so to be on the safe side we include by default
all firmware files in the ReaR recovery system
(except on architectures like ppc64 or ppc64le).

schlomo commented at 2024-07-11 12:46:

See also https://youtu.be/QN6vk5DfzAk?si=GTj74bC98dtduC26&t=1464 ...

I use this configuration to reduce the rescue image size and build times:

FIRMWARE_FILES=('no')
MODULES=('loaded_modules')
EXCLUDE_MD5SUM_VERIFICATION=all

About modules and firmware files you need to understand where you might want to boot a rescue image. With these options limiting the modules and firmware files your rescue image will only work on identical or extremely similar hardware, e.g. perfect for recovering a virtual machine.

For recovering on physical hardware, that could be very different from the original, I'd recommend to include more or all modules and firmware files. The more you test and know your recovery environment, the more you can narrow it down. If you don't know then take it all, just to be sure.

If you need to reduce the rescue image even further and want to use ReaR for recovery without the ReaR rescue system, then you might take a look at the new PORTABLE output mode: https://github.com/rear/rear/blob/master/doc/user-guide/17-Portable-Mode.adoc

@damm620 please kindly close the issue when appropriate.

damm620 commented at 2024-07-11 13:07:

@schlomo

Thanks for your answer.

With these options limiting the modules and firmware files your rescue image will only work on identical or extremely similar hardware, e.g. perfect for recovering a virtual machine.

What exacltly do you mean with this? In most cases we want to recover on the same or very similar hardware. Does this work when I remove all firmware? Why is this?
When I remove all firmware but the ReaR(-kernel) needs some firmware for the network hardware for example to get it running, then I am stuck when recovering because the network in the ReaR does not work, right?

Can you please clearify? We will scale this out to a few thousands servers and I just want to be sure that I understood correctly.

by the way: nice presentation :)

schlomo commented at 2024-07-11 13:44:

@damm620 the dependency goes like this: kernel -> modules -> firmware.

So you would need to know the modules that your hardware requires and then you can check with those modules for required firmware. We could, TBH, actually determine this automatically because modinfo reports the potential firmware files used ba a module.

In my experience you don't need any firmware on virtual machines. On physical hardware you might have drivers that require firmware files.

The reason that ReaR takes along all drivers and all firmware is so that you could recover even a very old system on brand new hardware, as long as the kernel of that old system supports the new hardware.

So to answer your question:

  1. know your potential recovery hardware
  2. determine which modules are required for that, and use either MODULES=('loaded_modules') or specify the modules you need
  3. determine if any of those modules require firmware and add that to FIRMWARE_FILES, e.g. as a pattern.
  4. Repeat if your potential recovery hardware changes

And yes, my suggestion from above might be just what you need - but please validate.

BTW, if you plan to roll out ReaR to thousands of systems then I'd like to suggest for you to reach out to us for a little bit of consulting around design and architecture. For such a large installation there is usually a lot of stuff that ReaR can do for you that is not so obvious and where our experience from other large installations can be useful.

damm620 commented at 2024-07-11 14:31:

@schlomo

So ReaR per default when setting FIRMWARE_FILES=( 'no' ) is going to load all modules (incl. their firmware if there are dependencies), but non "additional" firmware right? So all dependend firmware is going to be loaded but no extra ones?

second questions: So only modules can load firmware, not the kernel itself? Because I did check via lsmod -> modinfo $module|grep -i firmware if there are any firmware loaded on our servers. I turns out none of them does.

schlomo commented at 2024-07-11 14:44:

@damm620 no, sorry. FIRMWARE_FILES=( 'no' ) simply doesn't take any firmware files, no automagic here (yet). I just realised that we could do some automagic here by running modprobe against all modules that we include.

Kernel can load firmware if something that needs firmware is compiled into the kernel. Sorry for being imprecise here. Additionally, there could be firmware compiled into the kernel, but then ReaR wouldn't have to deal with that.

IIRC there was also some kind of kernel magic to match firmware device IDs or such to filenames to auto-load firmware, but maybe I remember that wrong.

In the end, testing is the best way. If you can't find any firmware that plays a role on your systems then I'd think it to be safe to disable firmware inclusion in the rescue system. Worst case, you'll have to add another initramfs file to the boot media to add the missing firmware files as a hot patch for an existing ReaR rescue image.

damm620 commented at 2024-07-11 14:52:

@schlomo

okay, we use only official centos, rhel and suse distros. So I doubt that there are any modules backed into the kernel, that will load firmware, right? Or is this assumption incorrect?

Can you show me the part of the code in ReaR (or file/folder) so that I can maybe develop an "automagic" doing something like:

- get all kernel modules with lsmod
- modprobe each of them and grep after "firmware"
- if s.th. is found include this firmware into the initrd

schlomo commented at 2024-07-11 15:01:

usr/share/rear/build/GNU/Linux/420_copy_firmware_files.sh would be your friend. It runs already after the modules so that you can rely on the previous script that copied the modules.

For a PR I'd suggest to give the FIRMWARE_FILES variable a new special value of auto which would trigger the autodetect. We could potentially set FIRMWARE_FILES=auto if MODULES=loaded_modules if we assume that this would be typically used together

Please make sure that you also handle the CPU microcode updates, I don't know if and how that works and if ReaR rescue systems that include all firmware files would update CPU microcode correctly.

gdha commented at 2024-07-15 15:51:

@damm620 Are your questions sufficient answered?

damm620 commented at 2024-07-16 07:05:

Yes thank you guys very much :)

[Export of Github issue for rear/rear.]