#3553 PR open: Automatically include firmware files for the kernel modules in the recovery system

Labels: enhancement

jsmeix opened issue at 2026-01-12 17:25:

In build/GNU/Linux/420_copy_firmware_files.sh
automatically include firmware files
for the kernel modules in the recovery system
see https://github.com/rear/rear/issues/3551

pcahyna commented at 2026-01-13 13:04:

Thanks @jsmeix, this sounds good, but as a user I don't understand the details of how it works. The documentation for FIRMWARE_FILES is here : https://github.com/rear/rear/blob/96aae1e7b504e069d3f07641f05874ed0bb3eaf4/usr/share/rear/conf/default.conf#L1914 , does it need updating? (Has the interpretation of the special values changed? I see you are deleting a comment in the code about the meaning of empty FIRMWARE_FILES ...)

jsmeix commented at 2026-01-13 13:33:

Of course default.conf gets adapted accordingly.
I thought I could do this today morning
but "something else" got in my way :-(

jsmeix commented at 2026-01-13 15:07:

Yesterday I tested
MODULES=( 'loaded_modules' )
with empty FIRMWARE_FILES
and for me things look good so far.

On my openSUSE Leap 15.6 workstation with AMD graphics
(built-in in CPU "Ryzen 3 4300G with Radeon Graphics")
the amdgpu kernel module is loaded which needs firmware:

# usr/sbin/rear -D mkrescue
...
Using build area: /var/tmp/rear.9vpz6XhzEPiqZGk
...
Copying only currently loaded kernel modules (MODULES contains 'loaded_modules') and those in MODULES_LOAD
Copying firmware files that belong to the copied kernel modules (FIRMWARE_FILES not specified)
...

# du -hs /var/tmp/rear.9vpz6XhzEPiqZGk/rootfs/
246M    /var/tmp/rear.9vpz6XhzEPiqZGk/rootfs/

For comparison with

FIRMWARE_FILES=( 'yes' )
MODULES=( 'loaded_modules' )

what one would normally do when one does not know
which firmware files are needed

# usr/sbin/rear -D mkrescue
...
Using build area: /var/tmp/rear.JB1vAflGYIR1Del
...
Copying only currently loaded kernel modules (MODULES contains 'loaded_modules') and those in MODULES_LOAD
Copying all files in /lib*/firmware/
...

# du -hs /var/tmp/rear.JB1vAflGYIR1Del/rootfs/
784M    /var/tmp/rear.JB1vAflGYIR1Del/rootfs/

and with

FIRMWARE_FILES=( 'no' )
MODULES=( 'loaded_modules' )

what results a recovery system where graphical display may not work
depending on whether or not the used AMD graphics hardware
can work in standard (VGA) modes without firmware:

# usr/sbin/rear -D mkrescue
...
Using build area: /var/tmp/rear.AR44mXniirFRtbv
...
Copying only currently loaded kernel modules (MODULES contains 'loaded_modules') and those in MODULES_LOAD
Omit copying files in /lib*/firmware/ (FIRMWARE_FILES='no')
...

# du -hs /var/tmp/rear.AR44mXniirFRtbv/rootfs/
221M    /var/tmp/rear.AR44mXniirFRtbv/rootfs/

The amdgpu firmware files are not so much:

# du -hs /lib/firmware/amdgpu/
31M     /lib/firmware/amdgpu/

jsmeix commented at 2026-01-13 15:15:

It is questionable how far the "modinfo -F firmware" method
actually works reliably in practice because:

On my original system (where "rear mkrescue" is run):

# find /lib/firmware/amdgpu/ | wc -l
639

# modinfo -F firmware amdgpu | wc -l
576

With

FIRMWARE_FILES=( 'yes' )
MODULES=( 'loaded_modules' )

# find /var/tmp/rear.JB1vAflGYIR1Del/rootfs/lib/firmware/amdgpu/ | wc -l
639

versus with

FIRMWARE_FILES=()
MODULES=( 'loaded_modules' )

# find /var/tmp/rear.9vpz6XhzEPiqZGk/rootfs/lib/firmware/amdgpu/ | wc -l
566

pcahyna commented at 2026-01-13 15:43:

@jsmeix

It is questionable how far the "modinfo -F firmware" method actually works reliable in practice because: (...)

As I am not familiar with this area, I am curious what problem do you see there. Could you please be more specific?

jsmeix commented at 2026-01-13 15:50:

I am also not familiar with this area.
What I am doing here are my first steps in this area.

I mean the non-matching numbers of the firmware files
indicate things could be inconsistent- in particular
needed firmware files could be missing because they
might not be listed by "modinfo -F firmware"?

See also in my initial desctiption
https://github.com/rear/rear/issues/3551#issue-3796994085

... it is normal that some firmware files
which are listed by modinfo
may not exist on the system.

I.e. currently the whole "modinfo -F firmware" stuff
looks somewhat inconsistent/unreliable to me.

Tomorrow I will investigate more (as time permits).

jsmeix commented at 2026-01-16 13:34:

With the new DebugPrint when no file is found in /lib*/firmware
that matches the partial firmware filename which is reported
by modinfo for a kernel module
those numbers make more sense to me (at least in my specific case).

I got

# usr/sbin/rear -D mkrescue
...
Using build area: /var/tmp/rear.CIShROSO7xxV54s
...
Copying only currently loaded kernel modules (MODULES contains 'loaded_modules') and those in MODULES_LOAD
Copying firmware files that belong to the copied kernel modules (FIRMWARE_FILES not specified)
No file in /lib*/firmware matching 'amdgpu/cyan_skillfish_gpu_info.bin' (reported by modinfo for 'amdgpu')
No file in /lib*/firmware matching 'amdgpu/ip_discovery.bin' (reported by modinfo for 'amdgpu')
No file in /lib*/firmware matching 'amdgpu/vega10_cap.bin' (reported by modinfo for 'amdgpu')
No file in /lib*/firmware matching 'amdgpu/sienna_cichlid_cap.bin' (reported by modinfo for 'amdgpu')
No file in /lib*/firmware matching 'amdgpu/navi12_cap.bin' (reported by modinfo for 'amdgpu')
No file in /lib*/firmware matching 'amdgpu/aldebaran_cap.bin' (reported by modinfo for 'amdgpu')
No file in /lib*/firmware matching 'amdgpu/gc_11_0_0_toc.bin' (reported by modinfo for 'amdgpu')
No file in /lib*/firmware matching 'amdgpu/sienna_cichlid_mes1.bin' (reported by modinfo for 'amdgpu')
No file in /lib*/firmware matching 'amdgpu/sienna_cichlid_mes.bin' (reported by modinfo for 'amdgpu')
No file in /lib*/firmware matching 'amdgpu/navi10_mes.bin' (reported by modinfo for 'amdgpu')
No file in /lib*/firmware matching 'amdgpu/gc_11_0_3_mes.bin' (reported by modinfo for 'amdgpu')
No file in /lib*/firmware matching 'renesas_usb_fw.mem' (reported by modinfo for 'xhci_pci')

and /var/tmp/rear.CIShROSO7xxV54s/rootfs/lib/firmware/amdgpu
contains all firmware files which are reported by modinfo
in particular for the 'amdgpu' kernel module
except those in the DebugPrint message
i.e. those which do not exist on my system.

jsmeix commented at 2026-01-16 13:39:

I think my current implementation is all what I can do
in practice to automatically include firmware files
for the kernel modules in the recovery system
so I would like to merge it in its current state.

@rear/contributors
please have a look (as time permits)
and ideally try it out how it behaves on your systems
in particular with MODULES=( 'loaded_modules' )
and the default empty FIRMWARE_FILES=()
because this is the main case where I would like
to get things automatically working reasonably well.

pcahyna commented at 2026-01-26 13:41:

Hi @jsmeix , can you give me a few more days, please? I would like to have a look but have not had time for it lately.

jsmeix commented at 2026-01-26 14:18:

@gdha
thank you for testing it!

@pcahyna
yes of course - there is nothing urgent here.
I explicitly requested a review from you now.

pcahyna commented at 2026-02-09 13:13:

I found out that the size of firmware and consequently of the initramfs keeps increasing, which makes this kind of improvements more and more valuable. Let me try on RHEL to see the effect...

jsmeix commented at 2026-03-23 12:46:

@rear/contributors
I would like to merge it on Thursday afternoon
unless there are objections.

We are still in ReaR 3.0 development phase
so if this changes cause more problems than they solve
we can try to improve things to make things work
or I can simply revert things to what it was before.

In any case the default behaviour did not change
(excerpts from the default.conf description)

The by default empty FIRMWARE_FILES array
together with the default MODULES=( 'all_modules' )
results that normally all files in the /lib*/firmware/
directories get included in the rescue/recovery system.

and the user can explicity specify MODULES and FIRMWARE_FILES
according to what he needs in his particular environment
regardless what our new default behaviour is which is now

When MODULES does not contain 'all_modules'
(e.g. when MODULES=( 'loaded_modules' ) is specified),
then an empty FIRMWARE_FILES array results that
firmware files which belong to those kernel modules
which get included in the rescue/recovery system
get also included in the rescue/recovery system

pcahyna commented at 2026-03-25 19:16:

Hello @jsmeix , thank you for the reminder. I researched the topic in detail and found that it is actually quite complicated (as it is often the case). I see two main problems with the current state of the PR:

  1. Regarding what the does: it looks incomplete to me. When MODULES=( 'loaded_modules' ) and FIRMWARE_FILES is empty, we should copy firmware for all loaded modules, but some modules can be built-in and those can also have firmware. (On a RHEL kernel, most of them don't, but I saw at least one having a firmware, and this can change if you build your own kernel and decide that some hardware drivers should be compiled in.) Here's a WIP fix of this issue, using the list /lib/modules/$KERNEL_VERSION/modules.builtin that newer kernels provide, and the ability of sufficiently new modinfo to query even builtin modules:
diff --git a/usr/share/rear/build/GNU/Linux/420_copy_firmware_files.sh b/usr/share/rear/build/GNU/Linux/420_copy_firmware_files.sh
index 462fc7ad3..9b9a9493c 100644
--- a/usr/share/rear/build/GNU/Linux/420_copy_firmware_files.sh
+++ b/usr/share/rear/build/GNU/Linux/420_copy_firmware_files.sh
@@ -61,9 +61,42 @@ fi
 # Automatically also copy the matching firmware files here, see
 # https://github.com/rear/rear/issues/3551
 LogPrint "Copying firmware files that belong to the copied kernel modules (FIRMWARE_FILES not specified)"
-for module in "${COPY_MODULES[@]}" ; do
-    firmware_partial_filenames=$( modinfo -k $KERNEL_VERSION -F firmware "$module" ) || continue
-    for firmware_partial_filename in $firmware_partial_filenames ; do
+local builtin_list="/lib/modules/$KERNEL_VERSION/modules.builtin"
+local firmware_partial_filenames=()
+
+local modinfo_error=0
+if [ -f "$builtin_list" ] ; then
+    local builtin_path
+    local builtin_modules=(
+        $(
+            while read -r builtin_path; do
+                builtin_mod="$(basename "$builtin_path")"
+                # remove the .ko (and possibly other) suffix
+                echo "${builtin_mod%%.*}"
+            done < "$builtin_list"
+        )
+    )
+    for module in "${COPY_MODULES[@]}" "${builtin_modules[@]}" ; do
+        # grep needs to be there as some versions of modinfo print "name: <module name>" for builtin modules
+        # even if not asked for it.
+        if ! firmware_partial_filenames+=( $( modinfo -k $KERNEL_VERSION -F firmware "$module" | grep -v '^[[:alnum:]]*:[[:blank:]]'; exit ${PIPESTATUS[0]} ) ); then
+            # This may happen when $builtin_list is present but
+            # /lib/modules/$KERNEL_VERSION/modules.builtin.modinfo is not.
+            # Only sufficiently new kernels provide modules.builtin.modinfo.
+            # The RHEL 8 kernel, for example, does not.
+            modinfo_error=$?
+            LogPrintError "Error determining firmware files for kernel module $module"
+            LogPrint "The command modinfo -k $KERNEL_VERSION -F firmware $module returned $modinfo_error"
+            break
+        fi
+    done
+else
+    LogPrintError "Unable to determine builtin module list to list firmware in use ($builtin_list not found)"
+    modinfo_error=1
+fi
+
+if [[ "$modinfo_error" == 0 ]] ; then
+    for firmware_partial_filename in "${firmware_partial_filenames[@]}" ; do
         # For example the command "modinfo -F firmware amdgpu" may show
         # amdgpu/cyan_skillfish_gpu_info.bin
         # amdgpu/navi12_gpu_info.bin
@@ -81,4 +114,13 @@ for module in "${COPY_MODULES[@]}" ; do
         fi
         cp $verbose -t $ROOTFS_DIR -p -L --parents $firmware_complete_filename
     done
-done
+else
+    LogPrintError "Defaulting to copy all firmware files - determining firmware in use not possible"
+    # TODO: copy of a block above - unify
+    LogPrint "Copying all files in /lib*/firmware/"
+    # Use a simple 'cp -a' for this case to be safe against possible issues
+    # with the more complicated 'find ... | xargs cp' method below.
+    # The '--parents' is needed to get the '/lib*/' directory in the copy.
+    # It is crucial to append to /dev/$DISPENSABLE_OUTPUT_DEV (cf. 'Print' in lib/_framework-setup-and-functions.sh):
+    cp $verbose -t $ROOTFS_DIR -a --parents /lib*/firmware 2>>/dev/$DISPENSABLE_OUTPUT_DEV 1>&2
+fi
  1. Regarding the interface. The problem with the (by default) empty FIRMWARE_FILES is that the meaning changes according to the value of MODULES. When MODULES=( 'all_modules' ), it means a different thing than if MODULES has any other value. I don't like this kind of long-lange interaction. (Of course I know that there are long-range interactions in the code, expressed for example in global variables, but here I mean the interaction between the value of the variables.) Not only it is inelegant and hard for the user to keep track of (one must be aware of the value of one variable in order to understand the meaning of the value of another variable), it also prevents a valuable use case: copying all the firmware that belongs to any of the available modules (because to copy all the available modules, one uses MODULES=( 'all_modules' ) , but this changes the meaning of FIRMWARE_FILES=() from "copy the firmware of all the modules that get copied", which we would like here, to "copy all firmware files").
    Why do I care about this? I believe that copying all the modules (not just the loaded ones) and only the firmware files that belong to these modules (instead of all) is the most valuable use case for these features, actually. It produces an image that can be used on the widest range of hardware, while reducing the initramfs size by omitting the firmware files that are not used by any module, and thus presumably useless.
    Now the questions are of course, how to specify it and then how to implement it. Regarding the specification, perhaps a special value would be the best option, something like FIRMWARE_FILES=( 'copied_modules' ) or perhaps a scalar FIRMWARE_FILES='copied_modules'. I don't like special values, especially as array elements, but we have been using it for modules as well. Alternatively, a special scalar variable like ADD_MODULE_FIRMWARE=y/n could be used.
    Regarding the implementation, this is of course made difficult by the need of listing all the copied modules, which we don't know if we copy everything. For this, the modules.dep file can be used as it lists all the modules. I started looking at this, but then I got distracted by other things. Gemini suggested this code (would need to be improved, I suppose - see the patch above that implements the part with builtin modules a bit differently):
KERNEL_VER=$(uname -r)
MOD_DIR="/lib/modules/${KERNEL_VER}"

get_all_firmware() {
    # =========================================================
    # PIPELINE 1: Loadable Modules (via modules.dep)
    # =========================================================
    # 1. awk -F: -> Extract the relative path (left side of the colon).
    # 2. sed -> Prepend the base directory to make absolute paths.
    # 3. xargs -> Pass physical files directly to modinfo.

    if [[ -f "${MOD_DIR}/modules.dep" ]]; then
        awk -F: '{print $1}' "${MOD_DIR}/modules.dep" \
            | sed "s|^|${MOD_DIR}/|" \
            | xargs -r modinfo -F firmware 2>/dev/null
    fi

    # =========================================================
    # PIPELINE 2: Built-in Modules (via modules.builtin)
    # =========================================================
    # 1. awk -F/ -> Strip the directory paths.
    # 2. sed -> Strip .ko and compression extensions to get pure names.
    # 3. xargs -> Pass NAMES to modinfo (using -k to force the lookup).

    if [[ -f "${MOD_DIR}/modules.builtin" ]]; then
        awk -F/ '{print $NF}' "${MOD_DIR}/modules.builtin" \
            | sed -E 's/\.ko(\..*)?$//' \
            | xargs -r modinfo -k "${KERNEL_VER}" -F firmware 2>/dev/null
    fi
}

In order to not block the merging for too long, I suggest that we agree on the proper interface of specifying only the copied modules and do the actual implementation of the combination of this with MODULES=( 'all_modules' ) separately (until it is implemented, we can error out when this combination is provided).

jsmeix commented at 2026-03-31 13:11:

@pcahyna
I may have a closer look at your above
https://github.com/rear/rear/pull/3553#issuecomment-4129123318
at the earliest after Easter public holidays
(in Germany until including 06. April)
perhaps later as other (non-ReaR) issues may permit.

A question:
Is there something actually wrong
with the changes as this pull request currently is
so it must not be merged in its current state
because this will cause real bugs
(cf. "bug" in https://github.com/rear/rear/labels)
OR
would the current state be acceptable to be merged
as a first step into the right direction
which should be further adaptend and enhanced
as you described in your above
https://github.com/rear/rear/pull/3553#issuecomment-4129123318
?

The thing is:
If the current state would be an acceptable first step
we could move forward step by step as neded and as time permits
because further adaptions and enhancements could be done
via subsequent pull requests as long as we are
in the ReaR 3.0 development phase.
But currently with
https://github.com/rear/rear/pull/3553#issuecomment-4129123318
moving forward within the ReaR 3.0 development phase
is blocked for an unpredictable period of time.

In general:
When something in a pull request is actually wrong
so it must not be merged in its current state,
then the reviewer should set the review state to
Request changes
to make this clear.

jsmeix commented at 2026-04-07 09:12:

As usual:
Security issues elsewhere got in my way to proceed here.

[Export of Github issue for rear/rear.]