#3189 Issue open: booting rescue system (ISO) of SLES on POWER fails

Labels: support / question

kai-uwe-rommel opened issue at 2024-04-02 13:17:

  • ReaR version ("/usr/sbin/rear -V"):

  • If your ReaR version is not the current version, explain why you can't upgrade:
    comes from SLES repository

  • OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):
    PRETTY_NAME="SUSE Linux Enterprise Server 15 SP5"

  • ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

(The ISO file is then copied from the NFS server into the VIOS media library and mounted into the new PowerVM LPAR as a virtual optical drive.)

  • Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR):
    IBM PowerVM LPAR

  • System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):

  • Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
    SAN w/multipath


  • Description of the issue (ideally so that others can reproduce it):
    The rear invocation creates an ISO file and that looks fine to me.
    When booting it in a new/blank PowerVM LPAR, it ends up with a kernel panic.
    To me it looks like it does not load the initrd.
    But the initrd file is in the ISO and seems to be referenced properly in the grub.cfg file.
    Please see content of the boot log file.

  • Workaround, if any:
    unfortunately, none - need help

  • Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):

jsmeix commented at 2024-04-02 14:46:

please attach a "rear -D mkrescue" (or "rear -D mkbackup")
debug log file.

In general:
Caution with possible secrets in a debug log file:
When 'rear' is run via '-D' in debugscript mode
it logs executed commands via the bash command 'set -x'
that print commands and their arguments as they are executed.
So in particular when arguments contain secret values
(e.g. something like a password or whatever else)
such secret values may appear in the log file.
Also secrets may be stored in some other files
like /var/lib/rear/layout/disklayout.conf
or /var/lib/rear/layout/diskrestore.sh
cf. [password=<password>] in the section
"Disk layout file syntax" in
online at
So before you attach your debug log file or other files here
(GitHub is a public accessible place), inspect your files
and verify that they do not accidentally cointain secrets.

kai-uwe-rommel commented at 2024-04-03 16:31:

The mkrescue debug log:

kai-uwe-rommel commented at 2024-04-03 16:32:

Also, let me know if you need access to the generated ISO file.

jsmeix commented at 2024-04-08 07:34:

it seems your
is only the terminal output of "rear -D mkrescue" (excerpts)

hermes:~ # rear -D mkrescue
Relax-and-Recover 2.7 / 2022-07-13
Running rear mkrescue (PID 11791 date 2024-04-03 18:22:27)
Command line options: /usr/sbin/rear -D mkrescue
Using log file: /var/log/rear/rear-hermes.log
Running 'output' stage ======================
Making ISO image
Making ISO image
Wrote ISO image: /var/lib/rear/output/rear-hermes.iso (319M)

but I liked to get the debug log file i.e. the
log file: /var/log/rear/rear-hermes.log
(except secret values if any).

Additionally please attach the terminal output of

# rear -s mkrescue

because in your
I am missing messages that tell a bootloader gets installed
before the ISO is made.

For comparison on my x86_64 UEFI homeoffice workstation

# grep -v '^#' etc/rear/local.conf 

# usr/sbin/rear -s mkrescue
Source output/ISO/Linux-i386/250_populate_efibootimg.sh
Source output/ISO/Linux-i386/260_EFISTUB_populate.sh
Source output/ISO/Linux-i386/300_create_isolinux.sh
Source output/default/400_copy_disk_struct_files.sh
Source output/ISO/Linux-i386/700_create_efibootimg.sh
Source output/ISO/Linux-i386/800_create_isofs.sh
Source output/ISO/Linux-i386/810_prepare_multiple_iso.sh
Source output/ISO/Linux-i386/820_create_iso_image.sh
Source output/ISO/Linux-i386/830_create_iso_image_EFISTUB.sh
Source output/ISO/Linux-i386/850_check_for_errors.sh

For PPC64LE we have

# ls -1 usr/share/rear/output/ISO/Linux-ppc64le

but usr/share/rear/output/ISO/Linux-ppc64le/300_create_grub2.sh
does no output on the terminal and it does not error out when

grub2-mkimage -O powerpc-ieee1275 ...

fails - at least this needs to be fixed, cf.

So I need your debug log file /var/log/rear/rear-hermes.log
to see in particular what goes on on your particular system
while the GRUB2 bootloader is installed for the iso image.

jsmeix commented at 2024-04-08 07:43:

in your initial description you wrote

But the initrd file is in the ISO and
seems to be referenced properly in the grub.cfg file.
Please see content of the boot log file.

but I fail to see in your
that the initrd file is in the ISO and
that it is referenced properly in the grub.cfg file,
in particular I fail to see the grub.cfg file?

jsmeix commented at 2024-04-08 08:05:

see my "FYI (2)" part in
how you may use KEEP_BUILD_DIR="yes"
to inspect the recovery system contents after "rear mkrescue"
so you could inspect in particular your grub.cfg file
in your ReaR recovery system.

kai-uwe-rommel commented at 2024-04-08 12:41:

This is the output of "rear -s mkrescue":

hermes:~ # rear -s mkrescue
Relax-and-Recover 2.7 / 2022-07-13
Running rear mkrescue (PID 8073 date 2024-04-08 14:39:53)
Using log file: /var/log/rear/rear-hermes.log
Simulation mode activated, Relax-and-Recover base directory: /usr/share/rear
Source conf/Linux-ppc64le.conf
Source conf/GNU/Linux.conf
Source conf/SUSE_LINUX.conf
Source init/default/005_verify_os_conf.sh
Source init/default/010_EFISTUB_check.sh
Source init/default/010_set_drlm_env.sh
Source init/default/030_update_recovery_system.sh
Source init/default/050_check_rear_recover_mode.sh
Source init/default/950_check_missing_programs.sh
Source prep/default/005_remove_workflow_conf.sh
Source prep/default/020_translate_url.sh
Source prep/default/030_translate_tape.sh
Source prep/default/035_valid_backup_methods.sh
Source prep/default/036_valid_output_methods.sh
Source prep/default/040_check_backup_and_output_scheme.sh
Source prep/default/050_check_keep_old_output_copy_var.sh
Source prep/default/100_init_workflow_conf.sh
Source prep/GNU/Linux/200_include_getty.sh
Source prep/GNU/Linux/200_include_serial_console.sh
Source prep/GNU/Linux/210_include_dhclient.sh
Source prep/GNU/Linux/220_include_lvm_tools.sh
Source prep/GNU/Linux/230_include_md_tools.sh
Source prep/GNU/Linux/240_include_multipath_tools.sh
Source prep/GNU/Linux/280_include_systemd.sh
Source prep/GNU/Linux/280_include_virtualbox.sh
Source prep/GNU/Linux/280_include_vmware_tools.sh
Source prep/GNU/Linux/290_include_drbd.sh
Source prep/GNU/Linux/300_check_backup_and_output_url.sh
Source prep/ISO/default/300_check_iso_dir.sh
Source prep/GNU/Linux/300_include_grub_tools.sh
Source prep/GNU/Linux/310_include_cap_utils.sh
Source prep/ISO/default/320_check_cdrom_size.sh
Source prep/default/320_include_uefi_env.sh
Source prep/ISO/GNU/Linux/320_verify_mkisofs.sh
Source prep/default/321_EFISTUB_check_uefi_env.sh
Source prep/default/330_include_uefi_tools.sh
Source prep/ISO/GNU/Linux/340_add_isofs_module.sh
Source prep/default/340_include_password_tools.sh
Source prep/ISO/GNU/Linux/360_EFISTUB_prechecks.sh
Source prep/default/380_include_opal_tools.sh
Source prep/GNU/Linux/400_guess_kernel.sh
Source prep/TSM/default/400_prep_tsm.sh
Source prep/default/400_save_directories.sh
Source prep/default/490_store_write_protect_settings.sh
Source prep/GNU/Linux/500_EFISTUB_check_kernel.sh
Source layout/save/GNU/Linux/100_create_layout_file.sh
Source layout/save/GNU/Linux/150_save_diskbyid_mappings.sh
Source layout/save/GNU/Linux/190_opaldisk_layout.sh
Source layout/save/GNU/Linux/200_partition_layout.sh
Source layout/save/GNU/Linux/210_raid_layout.sh
Source layout/save/GNU/Linux/220_lvm_layout.sh
Source layout/save/GNU/Linux/230_filesystem_layout.sh
Source layout/save/GNU/Linux/240_swaps_layout.sh
Source layout/save/GNU/Linux/250_drbd_layout.sh
Source layout/save/GNU/Linux/260_crypt_layout.sh
Source layout/save/GNU/Linux/270_hpraid_layout.sh
Source layout/save/GNU/Linux/280_multipath_layout.sh
Source layout/save/default/300_list_dependencies.sh
Source layout/save/default/310_autoexclude_usb.sh
Source layout/save/default/310_include_exclude.sh
Source layout/save/default/320_autoexclude.sh
Source layout/save/default/330_remove_exclusions.sh
Source layout/save/default/335_remove_excluded_multipath_vgs.sh
Source layout/save/GNU/Linux/340_false_blacklisted.sh
Source layout/save/default/340_generate_mountpoint_device.sh
Source layout/save/GNU/Linux/350_copy_drbdtab.sh
Source layout/save/default/350_save_partitions.sh
Source layout/save/default/400_check_backup_special_files.sh
Source layout/save/default/445_guess_bootloader.sh
Source layout/save/default/450_check_bootloader_files.sh
Source layout/save/default/450_check_network_files.sh
Source layout/save/default/490_check_files_to_patch.sh
Source layout/save/GNU/Linux/500_extract_vgcfg.sh
Source layout/save/GNU/Linux/510_current_disk_usage.sh
Source layout/save/default/600_snapshot_files.sh
Source layout/save/default/950_verify_disklayout_file.sh
Source rescue/default/010_merge_skeletons.sh
Source rescue/default/100_hostname.sh
Source rescue/default/200_etc_issue.sh
Source rescue/GNU/Linux/220_load_modules_from_initrd.sh
Source rescue/GNU/Linux/230_storage_and_network_modules.sh
Source rescue/GNU/Linux/240_kernel_modules.sh
Source rescue/GNU/Linux/250_udev.sh
Source rescue/GNU/Linux/260_collect_initrd_modules.sh
Source rescue/GNU/Linux/260_storage_drivers.sh
Source rescue/GNU/Linux/290_kernel_cmdline.sh
Source rescue/GNU/Linux/300_dns.sh
Source rescue/default/300_patch_root_home.sh
Source rescue/GNU/Linux/310_network_devices.sh
Source rescue/GNU/Linux/320_inet6.sh
Source rescue/GNU/Linux/350_routing.sh
Source rescue/GNU/Linux/390_check_usb_modules.sh
Source rescue/GNU/Linux/400_use_serial_console.sh
Source rescue/GNU/Linux/410_use_xen_console.sh
Source rescue/default/430_prepare_timesync.sh
Source rescue/GNU/Linux/500_clone_keyboard_mappings.sh
Source rescue/default/500_ssh.sh
Source rescue/GNU/Linux/550_copy_ldconfig.sh
Source rescue/default/550_vagrant.sh
Source rescue/default/850_save_sysfs_uefi_vars.sh
Source rescue/default/860_set_uefi_vars.sh
Source rescue/default/900_clone_users_and_groups.sh
Source rescue/default/910_copy_logfile.sh
Source rescue/GNU/Linux/950_cfg2html.sh
Source rescue/GNU/Linux/960_collect_MC_serviceguard_infos.sh
Source rescue/GNU/Linux/990_sysreqs.sh
Source build/GNU/Linux/005_create_symlinks.sh
Source build/GNU/Linux/090_create_lib_directories_and_symlinks.sh
Source build/GNU/Linux/100_copy_as_is.sh
Source build/GNU/Linux/110_touch_empty_files.sh
Source build/GNU/Linux/130_create_dotfiles.sh
Source build/GNU/Linux/150_adjust_permissions.sh
Source build/GNU/Linux/390_copy_binaries_libraries.sh
Source build/GNU/Linux/400_copy_modules.sh
Source build/GNU/Linux/420_copy_firmware_files.sh
Source build/GNU/Linux/450_symlink_mingetty.sh
Source build/default/490_fix_broken_links.sh
Source build/default/500_ssh_setup.sh
Source build/default/501_check_ssh_keys.sh
Source build/default/502_include_mdadm_conf.sh
Source build/default/503_store_tty_root_password.sh
Source build/GNU/Linux/600_verify_and_adjust_udev.sh
Source build/SUSE_LINUX/610_link_systemd_lib.sh
Source build/GNU/Linux/610_verify_and_adjust_udev_systemd.sh
Source build/GNU/Linux/620_verify_os_release_file.sh
Source build/GNU/Linux/630_simplify_systemd_reboot_halt_poweroff_shutdown.sh
Source build/GNU/Linux/630_verify_resolv_conf_file.sh
Source build/GNU/Linux/640_verify_lvm_conf.sh
Source build/default/950_check_missing_programs.sh
Source build/default/960_remove_encryption_keys.sh
Source build/default/970_add_rear_release.sh
Source build/default/975_update_os_conf.sh
Source build/default/990_verify_rootfs.sh
Source build/default/995_md5sums_rootfs.sh
Source pack/GNU/Linux/900_create_initramfs.sh
Source output/default/010_set_umask.sh
Source output/default/100_mount_output_path.sh
Source output/default/150_save_copy_of_prefix_dir.sh
Source output/default/200_make_boot_dir.sh
Source output/default/200_make_prefix_dir.sh
Source output/default/250_create_lock.sh
Source output/ISO/Linux-ppc64le/300_create_grub2.sh
Source output/default/400_copy_disk_struct_files.sh
Source output/ISO/Linux-ppc64le/800_create_isofs.sh
Source output/ISO/Linux-ppc64le/810_prepare_multiple_iso.sh
Source output/ISO/Linux-ppc64le/820_create_iso_image.sh
Source output/default/940_grub2_rescue.sh
Source output/default/940_grub_rescue.sh
Source output/default/950_copy_result_files.sh
Source output/TSM/default/950_dsmc_save_result_files.sh
Source output/default/950_email_result_files.sh
Source output/TSM/default/960_dsmc_verify_isofile.sh
Source output/default/970_remove_lock.sh
Source output/default/980_umount_output_dir.sh
Exiting rear mkrescue (PID 8073) and its descendant processes ...
Running exit tasks

kai-uwe-rommel commented at 2024-04-08 12:42:

And this is the actual debug log from /var/log/rear:

kai-uwe-rommel commented at 2024-04-08 12:44:

After the "rear mkrescue" and after the failed boot I inspected the ISO file:

C:\IBM\Temp>7z l rear-hermes.iso                                                     
7-Zip 22.00 (x64) : Copyright (c) 1999-2022 Igor Pavlov : 2022-06-15                 
Scanning the drive for archives:                                                     
1 file, 333985792 bytes (319 MiB)                                                    
Listing archive: rear-hermes.iso                                                     
Path = rear-hermes.iso                                                               
Type = Iso                                                                           
Physical Size = 333985792                                                            
Created = 2024-04-06 14:01:04.00                                                     
Modified = 2024-04-06 14:01:04.00                                                    
   Date      Time    Attr         Size   Compressed  Name                            
------------------- ----- ------------ ------------  ------------------------        
2024-04-06 14:01:04 D....                            boot                            
2024-04-06 14:01:03 D....                            boot\grub                       
2024-04-06 14:01:03 .....          115          115  boot\grub\grub.cfg              
2024-04-06 14:01:03 .....       383756       383756  boot\grub\powerpc.elf           
2024-04-06 14:01:00 .....    285125611    285125611  initrd.cgz                      
2024-02-12 13:14:40 .....     48086576     48086576  kernel                          
2024-04-06 14:01:04 D....                            ppc                             
2024-04-06 14:01:03 .....          160          160  ppc\bootinfo.txt                
------------------- ----- ------------ ------------  ------------------------        
2024-04-06 14:01:04          333596218    333596218  5 files, 3 folders

So there is both the initrd.cgz and the grub.cfg files.

kai-uwe-rommel commented at 2024-04-08 12:45:

And the grub.cfg file in the ISO contains:

set timeout=100

menuentry "Relax-and-Recover" {
    linux   /kernel root=/dev/ram0  selinux=0
    initrd  /initrd.cgz

jsmeix commented at 2024-04-08 14:27:

thank you for your prompt replies!

Now I am stuck because everything looks OK
(as far as I can see) but things are not OK.

I am not a booting expert and I know basically nothing
about POWER architecure specific things - I only know
that low-level things when booting on POWER architecture
are rather different compared to booting on x86 architecture.

As far as I know currently we do not have
an active ReaR upstream maintainer who is an
expert in POWER architecure specific things.

The /etc/os-release in your initial comment shows
you have "SUSE Linux Enterprise Server 15 SP5".

Is this exactly what you actually have or do you perhaps have
a SUSE product/extension where SUSE officially supports ReaR?

Because in your initial comment you wrote (excerpts):

ReaR version 2.7
comes from SLES repository

it indicates that you have a SUSE product/extension
where SUSE officially supports ReaR because otherwise
I assume you won't have ReaR from a SLES repository
(I mean from an official SUSE SLES repository).

See the section "SUSE support for Relax-and-Recover" in

For example - as far as I know (no warranty) - the
SUSE Linux Enterprise High Availability Extension
is included in
SUSE Linux Enterprise Server for SAP Applications.

If you have a SUSE product/extension
where SUSE officially supports ReaR
I would recommend that you file
an official support request at SUSE
via your official support contact at SUSE.
If you do that please provide an URL to this issue
in your SUSE support request.

kai-uwe-rommel commented at 2024-04-09 14:36:

It is indeed a SLES 15 and might even be a SLES 15 for SAP, I will have to check.
When I started this, I just looked and found that SUSE made ReaR available through their SLES 15 repository so this made things easier as I did not have to add some other repo or install it manually.
When it comes from the standard SLES 15 repo, will SUSE then support it?
I will open a ticket and try to get support ...

jsmeix commented at 2024-04-09 15:19:

As far as I know - at least in the usual cases -
SUSE provides Relax-and-Recover (ReaR) only via the
SUSE Linux Enterprise High Availability Extension (SLE-HA)
which is the only SUSE product/extension
where SUSE officially supports ReaR.

But what matters in the end is what your particular
support contract states.

It cannot be wrong to open a ticket at SUSE.
In the worst case you get notified that ReaR is not
supported with your current support contract.

shows "/hana/" mountpoints which indicate you run SAP HANA
which further indicates you have SLES for SAP Applications
which is rather common for SAP HANA on POWER architecture.

kai-uwe-rommel commented at 2024-04-09 15:20:

I have checked and we do have

  • High Availability Extension
  • SLES for SAP Applications
    So we should be covered.

kai-uwe-rommel commented at 2024-04-09 15:28:

The strange thing is that the SUSE customer portal does not let me create a support case and insists that "You are not currently entitled to use the support system" - we will have te check what is missing there.

jsmeix commented at 2024-04-12 12:38:


an offhanded idea based on what I vaguely remember
from the past about booting issues which we had
here at ReaR upstream with POWER architecture:

You use 'BACKUP=TSM' which means
all what TSM needs to restore a TSM backup
gets included in the ReaR recovery system.

As far as I vaguely remember those TSM restore parts
are relatively big so the initrd which contains the
ReaR recovery system becomes relatively big.
In your case according to your

   Date      Time    Attr         Size   Compressed  Name
2024-04-06 14:01:00 .....    285125611    285125611  initrd.cgz

your initrd is about 272 MiB big
(285125611 / 1024 / 1024 = 271.9).

As far as I vaguely remember there can be
weird (inexplicable) booting issues on POWER architecture
when the initrd is "somehow too big".
As far as I vaguely remember in such cases
there are no helpful (error) messages during booting
but is only "just somehow does not work".

To find out if the initrd size causes this issue here
I would like to suggest that you replace


then make a new ReaR recovery system ISO image with

# rear -D mkrescue

and inspected the new ISO same as what you did for your
and post the numbers for the new ISO here.

Also try out if that boots the ReaR recovery system
i.e. if it works that you can log in as 'root'
in the booted ReaR recovery system.

You cannot use that new ISO to recreate a system
no backup restore software is included
in the ReaR recovery system, see the
'BACKUP=REQUESTRESTORE' description in "man rear"
e.g. online at

That new ISO is only used here to find out
if the initrd size causes this issue here.

jsmeix commented at 2024-04-12 12:50:

Only as a side note FYI:
I found an older issue about initrd size on POWER architecture
but that one is about booting on ppc64 via yaboot
while here GRUB2 is used as bootloader.

jsmeix commented at 2024-04-12 13:08:

another offhanded idea what might help:

In your
I see

Using '/usr/bin/mkisofs' to create ISO filesystem images

Nowadays 'xorrisofs' is in general considered "better"
than the traditional 'mkisofs' (I know basically nothing
about the details of making ISO images so I cannot explain
why 'xorrisofs' is considered "better" than 'mkisofs').
Using 'xorrisofs' preferred was added via
that points to
but there is no explanation WHY 'xorrisofs' is preferred.
I guess at that time it was preferred just because
'xorrisofs' worked for Debian where the other ones
had issues.
But at least
shows a case where 'xorrisofs' is better on UEFI systems
where otherwise 'ebiso' was needed as a workaround, cf.

By default on SLES 15 you get '/usr/bin/mkisofs' installed
but 'xorrisofs' (in RPM package 'xorriso') should be also
available for SLES 15, cf.

When you install the RPM package 'xorriso' (this is possible
in addition to an already installed RPM package 'mkisofs')
then ReaR will automatically use 'xorrisofs' via how
ISO_MKISOFS_BIN is set in usr/share/rear/conf/default.conf
and you will see that during "rear -D mkrescue" as

Using '/usr/bin/xorrisofs' to create ISO filesystem images

So please also try out if it helps in your case
when you use 'xorrisofs' to make the ISO image.

[Export of Github issue for rear/rear.]