#3189 Issue `closed`: BACKUP=TSM booting rescue system (ISO) on POWER fails (initrd bigger than 123 MiB)¶

Labels: support / question, fixed / solved / done, external tool, special hardware or VM

kai-uwe-rommel opened issue at 2024-04-02 13:17:¶

ReaR version ("/usr/sbin/rear -V"):
2.7
If your ReaR version is not the current version, explain why you can't upgrade:
comes from SLES repository
OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):
NAME="SLES"
VERSION="15-SP5"
VERSION_ID="15.5"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP5"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15:sp5"
ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
OUTPUT=ISO
OUTPUT_URL=nfs://prometheus/nim/rear
BACKUP=TSM

(The ISO file is then copied from the NFS server into the VIOS media library and mounted into the new PowerVM LPAR as a virtual optical drive.)

Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR):
IBM PowerVM LPAR
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
PPC64LE
Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
SAN w/multipath
Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"):
lsblk.txt
Description of the issue (ideally so that others can reproduce it):
The rear invocation creates an ISO file and that looks fine to me.
When booting it in a new/blank PowerVM LPAR, it ends up with a kernel panic.
To me it looks like it does not load the initrd.
But the initrd file is in the ISO and seems to be referenced properly in the grub.cfg file.
boot.log
Please see content of the boot log file.
Workaround, if any:
unfortunately, none - need help
Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):

jsmeix commented at 2024-04-02 14:46:¶

@kai-uwe-rommel
please attach a "rear -D mkrescue" (or "rear -D mkbackup")
debug log file.

In general:
Caution with possible secrets in a debug log file:
When 'rear' is run via '-D' in debugscript mode
it logs executed commands via the bash command 'set -x'
that print commands and their arguments as they are executed.
So in particular when arguments contain secret values
(e.g. something like a password or whatever else)
such secret values may appear in the log file.
Also secrets may be stored in some other files
like /var/lib/rear/layout/disklayout.conf
or /var/lib/rear/layout/diskrestore.sh
cf. [password=<password>] in the section
"Disk layout file syntax" in
doc/user-guide/06-layout-configuration.adoc
online at
https://github.com/rear/rear/blob/rear-2.7/doc/user-guide/06-layout-configuration.adoc
So before you attach your debug log file or other files here
(GitHub is a public accessible place), inspect your files
and verify that they do not accidentally cointain secrets.

kai-uwe-rommel commented at 2024-04-03 16:31:¶

The mkrescue debug log:
rear-mkrescue-debug.log

kai-uwe-rommel commented at 2024-04-03 16:32:¶

Also, let me know if you need access to the generated ISO file.

jsmeix commented at 2024-04-08 07:34:¶

@kai-uwe-rommel
it seems your
https://github.com/rear/rear/files/14855435/rear-mkrescue-debug.log
is only the terminal output of "rear -D mkrescue" (excerpts)

hermes:~ # rear -D mkrescue
Relax-and-Recover 2.7 / 2022-07-13
Running rear mkrescue (PID 11791 date 2024-04-03 18:22:27)
Command line options: /usr/sbin/rear -D mkrescue
Using log file: /var/log/rear/rear-hermes.log
...
Running 'output' stage ======================
Making ISO image
Making ISO image
Wrote ISO image: /var/lib/rear/output/rear-hermes.iso (319M)

but I liked to get the debug log file i.e. the
log file: /var/log/rear/rear-hermes.log
(except secret values if any).

Additionally please attach the terminal output of

# rear -s mkrescue

because in your
https://github.com/rear/rear/files/14855435/rear-mkrescue-debug.log
I am missing messages that tell a bootloader gets installed
before the ISO is made.

For comparison on my x86_64 UEFI homeoffice workstation
(excerpts):

# grep -v '^#' etc/rear/local.conf 
OUTPUT=ISO
BACKUP=NETFS
BACKUP_URL=file:///other/
...

# usr/sbin/rear -s mkrescue
...
Source output/ISO/Linux-i386/250_populate_efibootimg.sh
Source output/ISO/Linux-i386/260_EFISTUB_populate.sh
Source output/ISO/Linux-i386/300_create_isolinux.sh
Source output/default/400_copy_disk_struct_files.sh
Source output/ISO/Linux-i386/700_create_efibootimg.sh
Source output/ISO/Linux-i386/800_create_isofs.sh
Source output/ISO/Linux-i386/810_prepare_multiple_iso.sh
Source output/ISO/Linux-i386/820_create_iso_image.sh
Source output/ISO/Linux-i386/830_create_iso_image_EFISTUB.sh
Source output/ISO/Linux-i386/850_check_for_errors.sh

For PPC64LE we have

# ls -1 usr/share/rear/output/ISO/Linux-ppc64le
300_create_grub2.sh
800_create_isofs.sh
810_prepare_multiple_iso.sh
820_create_iso_image.sh

but usr/share/rear/output/ISO/Linux-ppc64le/300_create_grub2.sh
does no output on the terminal and it does not error out when

grub2-mkimage -O powerpc-ieee1275 ...

fails - at least this needs to be fixed, cf.
https://github.com/rear/rear/wiki/Coding-Style#try-hard-to-care-about-possible-errors

So I need your debug log file /var/log/rear/rear-hermes.log
to see in particular what goes on on your particular system
while the GRUB2 bootloader is installed for the iso image.

jsmeix commented at 2024-04-08 07:43:¶

@kai-uwe-rommel
in your initial description you wrote

But the initrd file is in the ISO and
seems to be referenced properly in the grub.cfg file.
https://github.com/rear/rear/files/14837101/boot.log
Please see content of the boot log file.

but I fail to see in your
https://github.com/rear/rear/files/14837101/boot.log
that the initrd file is in the ISO and
that it is referenced properly in the grub.cfg file,
in particular I fail to see the grub.cfg file?

jsmeix commented at 2024-04-08 08:05:¶

@kai-uwe-rommel
see my "FYI (2)" part in
https://github.com/rear/rear/discussions/3184#discussioncomment-8902863
how you may use KEEP_BUILD_DIR="yes"
to inspect the recovery system contents after "rear mkrescue"
so you could inspect in particular your grub.cfg file
in your ReaR recovery system.

kai-uwe-rommel commented at 2024-04-08 12:41:¶

This is the output of "rear -s mkrescue":

hermes:~ # rear -s mkrescue
Relax-and-Recover 2.7 / 2022-07-13
Running rear mkrescue (PID 8073 date 2024-04-08 14:39:53)
Using log file: /var/log/rear/rear-hermes.log
Simulation mode activated, Relax-and-Recover base directory: /usr/share/rear
Source conf/Linux-ppc64le.conf
Source conf/GNU/Linux.conf
Source conf/SUSE_LINUX.conf
Source init/default/005_verify_os_conf.sh
Source init/default/010_EFISTUB_check.sh
Source init/default/010_set_drlm_env.sh
Source init/default/030_update_recovery_system.sh
Source init/default/050_check_rear_recover_mode.sh
Source init/default/950_check_missing_programs.sh
Source prep/default/005_remove_workflow_conf.sh
Source prep/default/020_translate_url.sh
Source prep/default/030_translate_tape.sh
Source prep/default/035_valid_backup_methods.sh
Source prep/default/036_valid_output_methods.sh
Source prep/default/040_check_backup_and_output_scheme.sh
Source prep/default/050_check_keep_old_output_copy_var.sh
Source prep/default/100_init_workflow_conf.sh
Source prep/GNU/Linux/200_include_getty.sh
Source prep/GNU/Linux/200_include_serial_console.sh
Source prep/GNU/Linux/210_include_dhclient.sh
Source prep/GNU/Linux/220_include_lvm_tools.sh
Source prep/GNU/Linux/230_include_md_tools.sh
Source prep/GNU/Linux/240_include_multipath_tools.sh
Source prep/GNU/Linux/280_include_systemd.sh
Source prep/GNU/Linux/280_include_virtualbox.sh
Source prep/GNU/Linux/280_include_vmware_tools.sh
Source prep/GNU/Linux/290_include_drbd.sh
Source prep/GNU/Linux/300_check_backup_and_output_url.sh
Source prep/ISO/default/300_check_iso_dir.sh
Source prep/GNU/Linux/300_include_grub_tools.sh
Source prep/GNU/Linux/310_include_cap_utils.sh
Source prep/ISO/default/320_check_cdrom_size.sh
Source prep/default/320_include_uefi_env.sh
Source prep/ISO/GNU/Linux/320_verify_mkisofs.sh
Source prep/default/321_EFISTUB_check_uefi_env.sh
Source prep/default/330_include_uefi_tools.sh
Source prep/ISO/GNU/Linux/340_add_isofs_module.sh
Source prep/default/340_include_password_tools.sh
Source prep/ISO/GNU/Linux/360_EFISTUB_prechecks.sh
Source prep/default/380_include_opal_tools.sh
Source prep/GNU/Linux/400_guess_kernel.sh
Source prep/TSM/default/400_prep_tsm.sh
Source prep/default/400_save_directories.sh
Source prep/default/490_store_write_protect_settings.sh
Source prep/GNU/Linux/500_EFISTUB_check_kernel.sh
Source layout/save/GNU/Linux/100_create_layout_file.sh
Source layout/save/GNU/Linux/150_save_diskbyid_mappings.sh
Source layout/save/GNU/Linux/190_opaldisk_layout.sh
Source layout/save/GNU/Linux/200_partition_layout.sh
Source layout/save/GNU/Linux/210_raid_layout.sh
Source layout/save/GNU/Linux/220_lvm_layout.sh
Source layout/save/GNU/Linux/230_filesystem_layout.sh
Source layout/save/GNU/Linux/240_swaps_layout.sh
Source layout/save/GNU/Linux/250_drbd_layout.sh
Source layout/save/GNU/Linux/260_crypt_layout.sh
Source layout/save/GNU/Linux/270_hpraid_layout.sh
Source layout/save/GNU/Linux/280_multipath_layout.sh
Source layout/save/default/300_list_dependencies.sh
Source layout/save/default/310_autoexclude_usb.sh
Source layout/save/default/310_include_exclude.sh
Source layout/save/default/320_autoexclude.sh
Source layout/save/default/330_remove_exclusions.sh
Source layout/save/default/335_remove_excluded_multipath_vgs.sh
Source layout/save/GNU/Linux/340_false_blacklisted.sh
Source layout/save/default/340_generate_mountpoint_device.sh
Source layout/save/GNU/Linux/350_copy_drbdtab.sh
Source layout/save/default/350_save_partitions.sh
Source layout/save/default/400_check_backup_special_files.sh
Source layout/save/default/445_guess_bootloader.sh
Source layout/save/default/450_check_bootloader_files.sh
Source layout/save/default/450_check_network_files.sh
Source layout/save/default/490_check_files_to_patch.sh
Source layout/save/GNU/Linux/500_extract_vgcfg.sh
Source layout/save/GNU/Linux/510_current_disk_usage.sh
Source layout/save/default/600_snapshot_files.sh
Source layout/save/default/950_verify_disklayout_file.sh
Source rescue/default/010_merge_skeletons.sh
Source rescue/default/100_hostname.sh
Source rescue/default/200_etc_issue.sh
Source rescue/GNU/Linux/220_load_modules_from_initrd.sh
Source rescue/GNU/Linux/230_storage_and_network_modules.sh
Source rescue/GNU/Linux/240_kernel_modules.sh
Source rescue/GNU/Linux/250_udev.sh
Source rescue/GNU/Linux/260_collect_initrd_modules.sh
Source rescue/GNU/Linux/260_storage_drivers.sh
Source rescue/GNU/Linux/290_kernel_cmdline.sh
Source rescue/GNU/Linux/300_dns.sh
Source rescue/default/300_patch_root_home.sh
Source rescue/GNU/Linux/310_network_devices.sh
Source rescue/GNU/Linux/320_inet6.sh
Source rescue/GNU/Linux/350_routing.sh
Source rescue/GNU/Linux/390_check_usb_modules.sh
Source rescue/GNU/Linux/400_use_serial_console.sh
Source rescue/GNU/Linux/410_use_xen_console.sh
Source rescue/default/430_prepare_timesync.sh
Source rescue/GNU/Linux/500_clone_keyboard_mappings.sh
Source rescue/default/500_ssh.sh
Source rescue/GNU/Linux/550_copy_ldconfig.sh
Source rescue/default/550_vagrant.sh
Source rescue/default/850_save_sysfs_uefi_vars.sh
Source rescue/default/860_set_uefi_vars.sh
Source rescue/default/900_clone_users_and_groups.sh
Source rescue/default/910_copy_logfile.sh
Source rescue/GNU/Linux/950_cfg2html.sh
Source rescue/GNU/Linux/960_collect_MC_serviceguard_infos.sh
Source rescue/GNU/Linux/990_sysreqs.sh
Source build/GNU/Linux/005_create_symlinks.sh
Source build/GNU/Linux/090_create_lib_directories_and_symlinks.sh
Source build/GNU/Linux/100_copy_as_is.sh
Source build/GNU/Linux/110_touch_empty_files.sh
Source build/GNU/Linux/130_create_dotfiles.sh
Source build/GNU/Linux/150_adjust_permissions.sh
Source build/GNU/Linux/390_copy_binaries_libraries.sh
Source build/GNU/Linux/400_copy_modules.sh
Source build/GNU/Linux/420_copy_firmware_files.sh
Source build/GNU/Linux/450_symlink_mingetty.sh
Source build/default/490_fix_broken_links.sh
Source build/default/500_ssh_setup.sh
Source build/default/501_check_ssh_keys.sh
Source build/default/502_include_mdadm_conf.sh
Source build/default/503_store_tty_root_password.sh
Source build/GNU/Linux/600_verify_and_adjust_udev.sh
Source build/SUSE_LINUX/610_link_systemd_lib.sh
Source build/GNU/Linux/610_verify_and_adjust_udev_systemd.sh
Source build/GNU/Linux/620_verify_os_release_file.sh
Source build/GNU/Linux/630_simplify_systemd_reboot_halt_poweroff_shutdown.sh
Source build/GNU/Linux/630_verify_resolv_conf_file.sh
Source build/GNU/Linux/640_verify_lvm_conf.sh
Source build/default/950_check_missing_programs.sh
Source build/default/960_remove_encryption_keys.sh
Source build/default/970_add_rear_release.sh
Source build/default/975_update_os_conf.sh
Source build/default/990_verify_rootfs.sh
Source build/default/995_md5sums_rootfs.sh
Source pack/GNU/Linux/900_create_initramfs.sh
Source output/default/010_set_umask.sh
Source output/default/100_mount_output_path.sh
Source output/default/150_save_copy_of_prefix_dir.sh
Source output/default/200_make_boot_dir.sh
Source output/default/200_make_prefix_dir.sh
Source output/default/250_create_lock.sh
Source output/ISO/Linux-ppc64le/300_create_grub2.sh
Source output/default/400_copy_disk_struct_files.sh
Source output/ISO/Linux-ppc64le/800_create_isofs.sh
Source output/ISO/Linux-ppc64le/810_prepare_multiple_iso.sh
Source output/ISO/Linux-ppc64le/820_create_iso_image.sh
Source output/default/940_grub2_rescue.sh
Source output/default/940_grub_rescue.sh
Source output/default/950_copy_result_files.sh
Source output/TSM/default/950_dsmc_save_result_files.sh
Source output/default/950_email_result_files.sh
Source output/TSM/default/960_dsmc_verify_isofile.sh
Source output/default/970_remove_lock.sh
Source output/default/980_umount_output_dir.sh
Exiting rear mkrescue (PID 8073) and its descendant processes ...
Running exit tasks

kai-uwe-rommel commented at 2024-04-08 12:42:¶

And this is the actual debug log from /var/log/rear:
rear-mkrescue-debug.log

kai-uwe-rommel commented at 2024-04-08 12:44:¶

After the "rear mkrescue" and after the failed boot I inspected the ISO file:

C:\IBM\Temp>7z l rear-hermes.iso                                                     
7-Zip 22.00 (x64) : Copyright (c) 1999-2022 Igor Pavlov : 2022-06-15                 
Scanning the drive for archives:                                                     
1 file, 333985792 bytes (319 MiB)                                                    
Listing archive: rear-hermes.iso                                                     
--                                                                                   
Path = rear-hermes.iso                                                               
Type = Iso                                                                           
Physical Size = 333985792                                                            
Created = 2024-04-06 14:01:04.00                                                     
Modified = 2024-04-06 14:01:04.00                                                    
   Date      Time    Attr         Size   Compressed  Name                            
------------------- ----- ------------ ------------  ------------------------        
2024-04-06 14:01:04 D....                            boot                            
2024-04-06 14:01:03 D....                            boot\grub                       
2024-04-06 14:01:03 .....          115          115  boot\grub\grub.cfg              
2024-04-06 14:01:03 .....       383756       383756  boot\grub\powerpc.elf           
2024-04-06 14:01:00 .....    285125611    285125611  initrd.cgz                      
2024-02-12 13:14:40 .....     48086576     48086576  kernel                          
2024-04-06 14:01:04 D....                            ppc                             
2024-04-06 14:01:03 .....          160          160  ppc\bootinfo.txt                
------------------- ----- ------------ ------------  ------------------------        
2024-04-06 14:01:04          333596218    333596218  5 files, 3 folders

So there is both the initrd.cgz and the grub.cfg files.

kai-uwe-rommel commented at 2024-04-08 12:45:¶

And the grub.cfg file in the ISO contains:

set timeout=100

menuentry "Relax-and-Recover" {
    linux   /kernel root=/dev/ram0  selinux=0
    initrd  /initrd.cgz
}

jsmeix commented at 2024-04-08 14:27:¶

@kai-uwe-rommel
thank you for your prompt replies!

Now I am stuck because everything looks OK
(as far as I can see) but things are not OK.

I am not a booting expert and I know basically nothing
about POWER architecure specific things - I only know
that low-level things when booting on POWER architecture
are rather different compared to booting on x86 architecture.

As far as I know currently we do not have
an active ReaR upstream maintainer who is an
expert in POWER architecure specific things.

The /etc/os-release in your initial comment shows
you have "SUSE Linux Enterprise Server 15 SP5".

Is this exactly what you actually have or do you perhaps have
a SUSE product/extension where SUSE officially supports ReaR?

Because in your initial comment you wrote (excerpts):

ReaR version 2.7
...
comes from SLES repository

it indicates that you have a SUSE product/extension
where SUSE officially supports ReaR because otherwise
I assume you won't have ReaR from a SLES repository
(I mean from an official SUSE SLES repository).

See the section "SUSE support for Relax-and-Recover" in
https://en.opensuse.org/SDB:Disaster_Recovery

For example - as far as I know (no warranty) - the
SUSE Linux Enterprise High Availability Extension
is included in
SUSE Linux Enterprise Server for SAP Applications.

If you have a SUSE product/extension
where SUSE officially supports ReaR
I would recommend that you file
an official support request at SUSE
via your official support contact at SUSE.
If you do that please provide an URL to this issue
in your SUSE support request.

kai-uwe-rommel commented at 2024-04-09 14:36:¶

It is indeed a SLES 15 and might even be a SLES 15 for SAP, I will have to check.
When I started this, I just looked and found that SUSE made ReaR available through their SLES 15 repository so this made things easier as I did not have to add some other repo or install it manually.
When it comes from the standard SLES 15 repo, will SUSE then support it?
I will open a ticket and try to get support ...

jsmeix commented at 2024-04-09 15:19:¶

As far as I know - at least in the usual cases -
SUSE provides Relax-and-Recover (ReaR) only via the
SUSE Linux Enterprise High Availability Extension (SLE-HA)
which is the only SUSE product/extension
where SUSE officially supports ReaR.

But what matters in the end is what your particular
support contract states.

It cannot be wrong to open a ticket at SUSE.
In the worst case you get notified that ReaR is not
supported with your current support contract.

Your
https://github.com/rear/rear/files/14837097/lsblk.txt
shows "/hana/" mountpoints which indicate you run SAP HANA
which further indicates you have SLES for SAP Applications
which is rather common for SAP HANA on POWER architecture.

kai-uwe-rommel commented at 2024-04-09 15:20:¶

I have checked and we do have

High Availability Extension
SLES for SAP Applications
So we should be covered.

kai-uwe-rommel commented at 2024-04-09 15:28:¶

The strange thing is that the SUSE customer portal does not let me create a support case and insists that "You are not currently entitled to use the support system" - we will have te check what is missing there.

jsmeix commented at 2024-04-12 12:38:¶

@kai-uwe-rommel

an offhanded idea based on what I vaguely remember
from the past about booting issues which we had
here at ReaR upstream with POWER architecture:

You use 'BACKUP=TSM' which means
all what TSM needs to restore a TSM backup
gets included in the ReaR recovery system.

As far as I vaguely remember those TSM restore parts
are relatively big so the initrd which contains the
ReaR recovery system becomes relatively big.
In your case according to your
https://github.com/rear/rear/issues/3189#issuecomment-2042658547

   Date      Time    Attr         Size   Compressed  Name
...
2024-04-06 14:01:00 .....    285125611    285125611  initrd.cgz

your initrd is about 272 MiB big
(285125611 / 1024 / 1024 = 271.9).

As far as I vaguely remember there can be
weird (inexplicable) booting issues on POWER architecture
when the initrd is "somehow too big".
As far as I vaguely remember in such cases
there are no helpful (error) messages during booting
but it only "just somehow does not work".

To find out if the initrd size causes this issue here
I would like to suggest that you replace
'BACKUP=TSM' with 'BACKUP=REQUESTRESTORE' i.e.

OUTPUT=ISO
OUTPUT_URL=nfs://prometheus/nim/rear
BACKUP=REQUESTRESTORE

then make a new ReaR recovery system ISO image with

# rear -D mkrescue

and inspect the new ISO same as what you did for your
https://github.com/rear/rear/issues/3189#issuecomment-2042658547
and post the numbers for the new ISO here.

Also try out if that boots the ReaR recovery system
i.e. if it works that you can log in as 'root'
in the booted ReaR recovery system.

You cannot use that new ISO to recreate a system
because with 'BACKUP=REQUESTRESTORE'
no backup restore software is included
in the ReaR recovery system, see the
'BACKUP=REQUESTRESTORE' description in "man rear"
e.g. online at
https://github.com/rear/rear/blob/master/doc/rear.8.adoc

That new ISO is only used here to find out
if the initrd size causes this issue here.

jsmeix commented at 2024-04-12 12:50:¶

Only as a side note FYI:
I found an older issue about initrd size on POWER architecture
https://github.com/rear/rear/issues/1142
but that one is about booting on ppc64 via yaboot
while here GRUB2 is used as bootloader.

jsmeix commented at 2024-04-12 13:08:¶

@kai-uwe-rommel
another offhanded idea what might help:

In your
https://github.com/rear/rear/files/14855435/rear-mkrescue-debug.log
I see

Using '/usr/bin/mkisofs' to create ISO filesystem images

Nowadays 'xorrisofs' is in general considered "better"
than the traditional 'mkisofs' (I know basically nothing
about the details of making ISO images so I cannot explain
why 'xorrisofs' is considered "better" than 'mkisofs').
Using 'xorrisofs' preferred was added via
https://github.com/rear/rear/commit/fb5bebca167ce5e4ca151292c00958f41f4d9ba4
that points to
https://github.com/rear/rear/pull/962
but there is no explanation WHY 'xorrisofs' is preferred.
I guess at that time it was preferred just because
'xorrisofs' worked for Debian where the other ones
had issues.
But at least
https://github.com/rear/rear/issues/3084
shows a case where 'xorrisofs' is better on UEFI systems
where otherwise 'ebiso' was needed as a workaround, cf.
https://github.com/rear/rear/issues/3084#issuecomment-1835840904

By default on SLES 15 you get '/usr/bin/mkisofs' installed
but 'xorrisofs' (in RPM package 'xorriso') should be also
available for SLES 15, cf.
https://github.com/rear/rear/issues/3084#issuecomment-1833496190

When you install the RPM package 'xorriso' (this is possible
in addition to an already installed RPM package 'mkisofs')
then ReaR will automatically use 'xorrisofs' via how
ISO_MKISOFS_BIN is set in usr/share/rear/conf/default.conf
see
https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/conf/default.conf#L893
and you will see that during "rear -D mkrescue" as

Using '/usr/bin/xorrisofs' to create ISO filesystem images

So please also try out if it helps in your case
when you use 'xorrisofs' to make the ISO image.

kai-uwe-rommel commented at 2024-04-24 15:14:¶

Sorry for being quiet for two weeks ...

I just tested with BACKUP=REQUESTRESTORE and this results in a initrd of size 116 MB compared to the 278 MB with TSM client. The resulting ISO booted successfully and I end up with a root shell of the rescue system.

Next I changed back to BACKUP=TSM, installed the xorriso package and got a new ISO with a 278 MB sized initrd again. Booting this ISO led to the same kernel panic (initrd not loaded) like the original one.

So regardless of mkisofs or xorrisofs this seems to be a problem of initrd size ... which options do we have now? Where to report the problem so that it gets a chance of being fixed?

kai-uwe-rommel commented at 2024-04-24 15:25:¶

Since we use TSM here I thought this would really have been a very convenient solution.
For the time being I will look for other options. So I just tried with BACKUP=NETFS to the same NFS location.
Strangely, a "rear mkbackup" resulted in an empty backup.tar.gz file being created there except for a log file in it:

root@prometheus:/nim/rear/hermes # gzip -dc backup.tar.gz |tar tvf -
-rw------- 0   0    50348 Apr 24 17:17:47 2024 var/log/rear/rear-hermes.8241.log

Probably I somewhere have to define some include/exclude list (such as of filesystems) what to back up?

Does rear create the backup.tar.gz really first completely on local disk and copies it to NFS afterwards?
So always need to have enough local space but currently there definitely is.

jsmeix commented at 2024-04-24 15:51:¶

Regarding compression of the initramfs/initrd
see REAR_INITRD_COMPRESSION in default.conf
https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/conf/default.conf#L3425
so perhaps

REAR_INITRD_COMPRESSION="lzma"

may help to get a sufficiently small initrd, see also
https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/conf/examples/RHEL7-PPC64LE-Multipath-PXE-GRUB.conf#L42

I suggest to report the problem with initrd size
on POWER architecture to SUSE because we have people at SUSE
who know much more than I about POWER architecture.
Alternatively or additionally I suggest to also report
that generic problem with initrd size on POWER architecture
to IBM because they should know better than anyone else
what one could do to boot a POWER machine
with a relatively big initrd.

Regarding BACKUP=NETFS:

Please provide a "rear -D mkbackup" debug log file
for the BACKUP=NETFS case so that I could (hopefully)
understand why you get an empty backup.tar.gz.

You may have a look at my examples in
https://github.com/rear/rear/wiki/Test-Matrix-ReaR-2.7
how I used BACKUP=NETFS for my tests.
In most of my cases I used the SUSE default btrfs structure
so I needed some special settings in etc/rear/local.conf
according to
usr/share/rear/conf/examples/SLE12-SP2-btrfs-example.conf
https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/conf/examples/SLE12-SP2-btrfs-example.conf

If you use the SUSE default btrfs structure without that
special settings your backup.tar.gz would be incomplete
but (as far as I currently imagine) you should not get an
empty backup.tar.gz (something should be directly in '/').

kai-uwe-rommel commented at 2024-04-24 16:35:¶

Using the compression got the initrd down to 211 MB but it still does not load.

kai-uwe-rommel commented at 2024-04-24 16:54:¶

Here is the log file from a "rear -D mkbackup".
rear-hermes.log
I would like to get just the entire root file system into the backup.
We have installed everything into one / file system except the actual SAP installation and data.
And the root file system is the only logical volume in the "system" volume group.

kai-uwe-rommel commented at 2024-04-24 17:16:¶

I have created a SUSE support case.

jsmeix commented at 2024-04-25 11:13:¶

Regarding initrd size I see now
that also on POWER the newer default

MODULES=( 'all_modules' )

is used and accordingly your
https://github.com/rear/rear/files/15097745/rear-hermes.log
contains (excerpts)

+ source /usr/share/rear/build/GNU/Linux/400_copy_modules.sh
...
++ LogPrint 'Copying all kernel modules in /lib/modules/5.14.21-150500.55.49-default (MODULES contains '\''all_modules'\'')'

so it should help to get a noticeably smaller initrd
when you use something like

MODULES=( 'loaded_modules' )

see the MODULES description in
usr/share/rear/conf/default.conf
https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/conf/default.conf#L1540

MODULES=( 'all_modules' ) was introduced
in ReaR version 2.5 (May 2019), see
https://github.com/rear/rear/blob/rear-2.7/doc/rear-release-notes.txt#L2242
for the reasoning behind why that was set as default.

jsmeix commented at 2024-04-25 11:26:¶

FYI
regarding firmware files which make the initrd really big,
see my numbers on x86
https://github.com/rear/rear/discussions/2640#discussioncomment-908335

On my openSUSE Leap 15.2 system
I get this ReaR recovery system sizes:
By default:
-----------------------------------------------
# du -hs /tmp/rear.EHYoS4gnFtyPRE4/rootfs/
953M    /tmp/rear.EHYoS4gnFtyPRE4/rootfs/
-----------------------------------------------

With FIRMWARE_FILES=( 'no' ):
-----------------------------------------------
# du -hs /tmp/rear.GuUugJjsXSfC3G2/rootfs/
454M    /tmp/rear.GuUugJjsXSfC3G2/rootfs/
-----------------------------------------------

With FIRMWARE_FILES=( 'no' ) and MODULES=( 'loaded_modules' ):
-----------------------------------------------
# du -hs /tmp/rear.OF6uW70Cp1l2yv1/rootfs/
201M    /tmp/rear.OF6uW70Cp1l2yv1/rootfs/
-----------------------------------------------

So I get about 200M for the bare ReaR recovery system
plus about 250M for all kernel modules
plus about 500M for the firmware files.

Those numbers are the uncompressed ReaR recovery system
files in /tmp/rear.XXXXXXXXXXXXXXX/rootfs/

On POWER this is different by default
because we set FIRMWARE_FILES=( 'no' ) in
usr/share/rear/conf/Linux-ppc64.conf and
usr/share/rear/conf/Linux-ppc64le.conf
when the default 'FIRMWARE_FILES=()' is used
so on POWER by default no firmware files are included
in the initrd and accordingly there is in your
https://github.com/rear/rear/files/15097745/rear-hermes.log

+ source /usr/share/rear/build/GNU/Linux/420_copy_firmware_files.sh
...
++ LogPrint 'Omit copying files in /lib*/firmware/ (FIRMWARE_FILES='\''no'\'')'

jsmeix commented at 2024-04-25 11:57:¶

I see that the NETFS backup.tar.gz is empty
but currently I do not yet understand why...

kai-uwe-rommel commented at 2024-04-25 12:24:¶

When I look at the backup.log I can see that tar is called explicitly with (only) the log file path/name as the argument of what to store. Instead, / should have been passed. It also already gets --one-file-system passed. Based on that, it is no wonder that a backup.tar.gz is produced with only the log file included. If / would have been passed to it (with the --one-file-system) it would have been correct. The question is, why is the tar command constructed this way? Also, there is some strange include/exclude file magic. The log shows a command to "cat" a backup-include.txt but the tar command gets a backup-exclude.txt file passed with the -X option.
backup.log

kai-uwe-rommel commented at 2024-04-25 12:45:¶

Using MODULES=( 'loaded_modules' ) (and compression) gets the initrd down to 162 MB with BACKUP=TSM. Unfortunately, this still does not load but ends up in a kernel panic. We know (see above) that a 116 MB one would work ... but I guess we do not have any more options to further reduce the size.

kai-uwe-rommel commented at 2024-04-25 12:52:¶

The question is now what the actual limit of initrd size is. And why grub (is it grub's problem?) imposes it.

jsmeix commented at 2024-04-25 13:06:¶

Regarding the size of what is included
in the ReaR recovery system for TSM
see in usr/share/rear/conf/default.conf
the things about COPY_AS_IS_TSM
https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/conf/default.conf#L2087

That part originated at
https://github.com/rear/rear/commit/1e5236dc99d223861c6cca17fc98a8bf469b3a9a
which points to
https://github.com/rear/rear/pull/1566
from @schabrolles who contributed so much
for POWER support in ReaR but he is no longer
an active ReaR upstream maintainer.

It depends on what you actually need from TSM
during "rear recover" to restore a TSM backup.

kai-uwe-rommel commented at 2024-04-25 13:23:¶

Thanks for the pointer. See my comment on https://github.com/rear/rear/pull/1566.

kai-uwe-rommel commented at 2024-04-25 14:37:¶

Sounds like getting POWER and grub to load a larger initrd is not an option?

jsmeix commented at 2024-04-26 06:56:¶

I am not a booting expert but I think GRUB is not the reason.
I think GRUB can load the initrd because I assume otherwise
there would be some GRUB or kernel error message, e.g. as in
https://www.ibm.com/support/pages/system-boot-ends-grub-out-memory-oom

kai-uwe-rommel commented at 2024-04-26 17:25:¶

I finally got a rescue image with TSM client small enough. From the SUSE support case I got the information that the initrd image inside the ISO needs to be smaller than the limit of 124 MB for POWER.

I decided to not use an explicit file list in COPY_AS_IS_TSM but stick with the default values for it and instead specifically exclude large but not needed files (most of them foreign language files) from the image. This approach should be safer for future changes of the TSM client.

That saved a LOT and got the initrd down to 120 MB even with default gzip compression. With lzma it is only 77 MB. Of course, MODULES=('loaded_modules') is also needed. But when using lzma compression, the "rear mkrescue" runs 3 instead of 1 minute.

kai-uwe-rommel commented at 2024-04-26 17:26:¶

The values used now are:

COPY_AS_IS_TSM=( /etc/adsm
                 /opt/tivoli/tsm/client
                 /usr/local/ibm/gsk8* )
COPY_AS_IS_EXCLUDE_TSM=( /opt/tivoli/tsm/client/ba/bin/*.jar
                         /opt/tivoli/tsm/client/ba/bin/*.log*
                         /opt/tivoli/tsm/client/ba/bin/CS_CZ
                         /opt/tivoli/tsm/client/ba/bin/DE_DE
                         /opt/tivoli/tsm/client/ba/bin/ES_ES
                         /opt/tivoli/tsm/client/ba/bin/FR_FR
                         /opt/tivoli/tsm/client/ba/bin/HU_HU
                         /opt/tivoli/tsm/client/ba/bin/IT_IT
                         /opt/tivoli/tsm/client/ba/bin/JA_JP
                         /opt/tivoli/tsm/client/ba/bin/KO_KR
                         /opt/tivoli/tsm/client/ba/bin/PL_PL
                         /opt/tivoli/tsm/client/ba/bin/PT_BR
                         /opt/tivoli/tsm/client/ba/bin/RU_RU
                         /opt/tivoli/tsm/client/ba/bin/ZH_CN
                         /opt/tivoli/tsm/client/ba/bin/ZH_TW
                         /opt/tivoli/tsm/client/api/bin64/CS_CZ
                         /opt/tivoli/tsm/client/api/bin64/DE_DE
                         /opt/tivoli/tsm/client/api/bin64/ES_ES
                         /opt/tivoli/tsm/client/api/bin64/FR_FR
                         /opt/tivoli/tsm/client/api/bin64/HU_HU
                         /opt/tivoli/tsm/client/api/bin64/IT_IT
                         /opt/tivoli/tsm/client/api/bin64/JA_JP
                         /opt/tivoli/tsm/client/api/bin64/KO_KR
                         /opt/tivoli/tsm/client/api/bin64/PL_PL
                         /opt/tivoli/tsm/client/api/bin64/PT_BR
                         /opt/tivoli/tsm/client/api/bin64/RU_RU
                         /opt/tivoli/tsm/client/api/bin64/ZH_CN
                         /opt/tivoli/tsm/client/api/bin64/ZH_TW )

Yes, the exclude line is very long ...

kai-uwe-rommel commented at 2024-04-26 17:38:¶

And with some brainstorming about a smarter wildcard, I got that to collapse quite neatly into a very short list:

COPY_AS_IS_TSM=( /etc/adsm
                 /opt/tivoli/tsm/client
                 /usr/local/ibm/gsk8* )
COPY_AS_IS_EXCLUDE_TSM=( /opt/tivoli/tsm/client/ba/bin/*.jar
                         /opt/tivoli/tsm/client/ba/bin/*.log*
                         /opt/tivoli/tsm/client/*/*/[A-Z][!N]_?? )

kai-uwe-rommel commented at 2024-04-26 17:43:¶

Ok, problem solved. What's left:

Shouldn't we solve the BACKUP=NETFS issue with the empty backup.tar.gz file? Probably I should open a separate issue?

jsmeix commented at 2024-04-29 07:25:¶

@kai-uwe-rommel
thank you so much for the information how one can get
the TSM client in the ReaR recovery system smaller.
I will enhance things accordingly.

Usually we at ReaR upstream do not have or use
third-party backup tools (in particular not
if a third-party backup tool is proprietary software)
so usually we cannot reproduce issues
with third-party backup tools.
Therefore we totally depend on contributions and
help from users who use and know about a specific
third-party backup tool.

Regarding your
https://github.com/rear/rear/issues/3189#issuecomment-2079811746

COPY_AS_IS_EXCLUDE_TSM=( ...
                         /opt/tivoli/tsm/client/*/*/[A-Z][!N]_?? )

I assume [A-Z][!N]_ is to exclude EN_
to keep English language files?

Regarding the empty NETFS backup:
I did not further analyze its root cause
because I assumed TSM is more important for you.
Will analyze the empty NETFS backup root cause.

kai-uwe-rommel commented at 2024-04-29 07:30:¶

Yes, the wildard excludes all but the EN_US language subdirectory.

Sure the route using TSM is my primary goal. But since I discovered the NETFS issue as a collateral, I think it should be fixed as well. Also, I'm not done with a complete recovery test with TSM yet. I may encounter other issues, even showstoppers. So I think that NETFS may be important as a fallback for us.

jsmeix commented at 2024-04-29 07:34:¶

Yes of course NETFS is the ReaR standard method
so the "empty NETFS backup" root cause needs to be found.
It was only what to do with highest priority in your case.

kai-uwe-rommel commented at 2024-04-29 09:06:¶

If you touch this: by default, it should exclude /var/lib/rear/output.

jsmeix commented at 2024-04-29 09:13:¶

As far as I can see /var/lib/rear/output
is excluded from the ReaR recovery system via
those settings in default.conf:

COPY_AS_IS=( $SHARE_DIR $VAR_DIR )

COPY_AS_IS_EXCLUDE=( $VAR_DIR/output/\* dev/.udev dev/shm dev/shm/\* dev/oracleasm dev/mapper dev/watchdog\* )

where VAR_DIR is set in usr/sbin/rear

VAR_DIR="$REAR_DIR_PREFIX/var/lib/rear"

kai-uwe-rommel commented at 2024-04-29 09:15:¶

I may have overlooked it but I thought I did not see it in the actually generated tar command in the NETFS backup log.

jsmeix commented at 2024-04-29 12:34:¶

Ah! I misunderstood.
You meant to exclude /var/lib/rear/output
by default from the NETFS backup.

For this we have in default.conf

# BACKUP_PROG_EXCLUDE is an array of strings that get written into a backup-exclude.txt file
# that is used e.g. in 'tar -X backup-exclude.txt' to get things excluded from the backup.
...
BACKUP_PROG_EXCLUDE=( '/tmp/*' '/dev/shm/*' "$VAR_DIR/output/*" )

so var/lib/rear/output is excluded by default
from the NETFS backup.

You won't see that exclusion directly in the tar command
but only indirecly via tar ... -X backup-exclude.txt.

The backup-exclude.txt content is shown in the
normal ReaR log file (not the backup log file) via
usr/share/rear/backup/NETFS/default/500_make_backup.sh

# Log what is included in the backup and what is excluded from the backup
# cf. backup/NETFS/default/400_create_include_exclude_files.sh
Log "Backup include list (backup-include.txt contents):"
while read -r backup_include_item ; do
    test "$backup_include_item" && Log "  $backup_include_item"
done < $TMP_DIR/backup-include.txt
Log "Backup exclude list (backup-exclude.txt contents):"
while read -r backup_exclude_item ; do
    test "$backup_exclude_item" && Log "  $backup_exclude_item"
done < $TMP_DIR/backup-exclude.txt

Accordingly your
https://github.com/rear/rear/files/15097745/rear-hermes.log
contains (excerpts - only actual 'Log()' messages):

2024-04-24 18:38:20.254914145 Using build area: /var/tmp/rear.aQ2zh33mXgievJm
...
2024-04-24 18:39:07.171830692 Backup include list (backup-include.txt contents):
2024-04-24 18:39:07.175749525 Backup exclude list (backup-exclude.txt contents):
2024-04-24 18:39:07.179574293   /tmp/*
2024-04-24 18:39:07.183286446   /dev/shm/*
2024-04-24 18:39:07.186897500   /var/lib/rear/output/*
2024-04-24 18:39:07.190769423   /var/tmp/rear.aQ2zh33mXgievJm
...
2024-04-24 18:39:07.209802648 tar --warning=no-xdev --sparse --block-number --totals --verbose --no-wildcards-match-slash --one-file-system --ignore-failed-read --anchored --xattrs --xattrs-include=security.capability --xattrs-include=security.selinux --acls --gzip -X /var/tmp/rear.aQ2zh33mXgievJm/tmp/backup-exclude.txt -C / -c -f - /var/log/rear/rear-hermes.log | dd of=/var/tmp/rear.aQ2zh33mXgievJm/outputfs/hermes/backup.tar.gz bs=1M

So your backup-include.txt is empty which is why I wrote above
https://github.com/rear/rear/issues/3189#issuecomment-2077011476

For comparison:

How it looks on one of my test VMs with ReaR 2.7
on x86_64 with the SUSE default btrfs structure:

# lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINTS /dev/sda
NAME        KNAME     PKNAME   TRAN TYPE FSTYPE SIZE MOUNTPOINTS
/dev/sda    /dev/sda           ata  disk         15G 
|-/dev/sda1 /dev/sda1 /dev/sda      part          8M 
|-/dev/sda2 /dev/sda2 /dev/sda      part btrfs   13G /var
|                                                    /root
|                                                    /tmp
|                                                    /usr/local
|                                                    /boot/grub2/i386-pc
|                                                    /srv
|                                                    /opt
|                                                    /boot/grub2/x86_64-efi
|                                                    /home
|                                                    /.snapshots
|                                                    /
`-/dev/sda3 /dev/sda3 /dev/sda      part swap     2G [SWAP]

Using this ReaR config:

OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://192.168.178.66/nfs
REQUIRED_PROGS+=( snapper chattr )
PROGS+=( lsattr )
COPY_AS_IS+=( /usr/lib/snapper/installation-helper /etc/snapper/config-templates/default )
BACKUP_PROG_INCLUDE=( /boot/grub2/i386-pc /boot/grub2/x86_64-efi /home /opt /root /srv /tmp /usr/local /var )
POST_RECOVERY_SCRIPT=( 'if snapper --no-dbus -r $TARGET_FS_ROOT get-config | grep -q "^QGROUP.*[0-9]/[0-9]" ; then snapper --no-dbus -r $TARGET_FS_ROOT set-config QGROUP= ; snapper --no-dbus -r $TARGET_FS_ROOT setup-quota && echo snapper setup-quota done || echo snapper setup-quota failed ; else echo snapper setup-quota not used ; fi' )
SSH_ROOT_PASSWORD='rear'
USE_DHCLIENT="yes"
PROGRESS_MODE="plain"
PROGRESS_WAIT_SECONDS="5"
MODULES=( loaded_modules )
FIRMWARE_FILES=( no )

"rear -D mkbackup" log excerpts - only actual 'Log()' messages:

2024-04-29 14:45:49.329755032 Including backup/NETFS/default/500_make_backup.sh
2024-04-29 14:45:49.331440517 Entering debugscript mode via 'set -x'.
2024-04-29 14:45:49.339585602 Making backup (using backup method NETFS)
2024-04-29 14:45:49.347139698 Backup include list (backup-include.txt contents):
2024-04-29 14:45:49.349245934   /boot/grub2/i386-pc
2024-04-29 14:45:49.351251309   /boot/grub2/x86_64-efi
2024-04-29 14:45:49.353379145   /home
2024-04-29 14:45:49.355323537   /opt
2024-04-29 14:45:49.357239244   /root
2024-04-29 14:45:49.359060043   /srv
2024-04-29 14:45:49.360984958   /tmp
2024-04-29 14:45:49.362798965   /usr/local
2024-04-29 14:45:49.364743496   /var
2024-04-29 14:45:49.366581308   /
2024-04-29 14:45:49.368497576 Backup exclude list (backup-exclude.txt contents):
2024-04-29 14:45:49.370376595   /tmp/*
2024-04-29 14:45:49.372303543   /dev/shm/*
2024-04-29 14:45:49.374119013   /root/rear.2.7/var/lib/rear/output/*
2024-04-29 14:45:49.375999263   /var/tmp/rear.1HOOD1P1tTy6Eyn
2024-04-29 14:45:49.378041589 Creating tar archive '/var/tmp/rear.1HOOD1P1tTy6Eyn/outputfs/localhost/backup.tar.gz'
2024-04-29 14:45:49.387479061 tar --warning=no-xdev --sparse --block-number --totals --verbose --no-wildcards-match-slash --one-file-system --ignore-failed-read --anchored --xattrs --xattrs-include=security.capability --xattrs-include=security.selinux --acls --gzip -X /var/tmp/rear.1HOOD1P1tTy6Eyn/tmp/backup-exclude.txt -C / -c -f - /boot/grub2/i386-pc /boot/grub2/x86_64-efi /home /opt /root /srv /tmp /usr/local /var / /root/rear.2.7/var/log/rear/rear-localhost.log | dd of=/var/tmp/rear.1HOOD1P1tTy6Eyn/outputfs/localhost/backup.tar.gz bs=1M

And without BACKUP_PROG_INCLUDE in local.conf

OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://192.168.178.66/nfs
REQUIRED_PROGS+=( snapper chattr )
PROGS+=( lsattr )
COPY_AS_IS+=( /usr/lib/snapper/installation-helper /etc/snapper/config-templates/default )
POST_RECOVERY_SCRIPT=( 'if snapper --no-dbus -r $TARGET_FS_ROOT get-config | grep -q "^QGROUP.*[0-9]/[0-9]" ; then snapper --no-dbus -r $TARGET_FS_ROOT set-config QGROUP= ; snapper --no-dbus -r $TARGET_FS_ROOT setup-quota && echo snapper setup-quota done || echo snapper setup-quota failed ; else echo snapper setup-quota not used ; fi' )
SSH_ROOT_PASSWORD='rear'
USE_DHCLIENT="yes"
PROGRESS_MODE="plain"
PROGRESS_WAIT_SECONDS="5"
MODULES=( loaded_modules )
FIRMWARE_FILES=( no )
USER_INPUT_UNATTENDED_TIMEOUT=1

I get in the "rear -D mkbackup" log
(excerpts - only actual 'Log()' messages):

2024-04-29 15:07:39.125849969 Including backup/NETFS/default/500_make_backup.sh
2024-04-29 15:07:39.127581460 Entering debugscript mode via 'set -x'.
2024-04-29 15:07:39.135772001 Making backup (using backup method NETFS)
2024-04-29 15:07:39.138247680 Backup include list (backup-include.txt contents):
2024-04-29 15:07:39.140505651   /
2024-04-29 15:07:39.142441997 Backup exclude list (backup-exclude.txt contents):
2024-04-29 15:07:39.144371891   /tmp/*
2024-04-29 15:07:39.146287468   /dev/shm/*
2024-04-29 15:07:39.148164082   /root/rear.2.7/var/lib/rear/output/*
2024-04-29 15:07:39.150046748   /var/tmp/rear.M11HPUBIGuYyivX
2024-04-29 15:07:39.152145038 Creating tar archive '/var/tmp/rear.M11HPUBIGuYyivX/outputfs/localhost/backup.tar.gz'
2024-04-29 15:07:39.167986757 tar --warning=no-xdev --sparse --block-number --totals --verbose --no-wildcards-match-slash --one-file-system --ignore-failed-read --anchored --xattrs --xattrs-include=security.capability --xattrs-include=security.selinux --acls --gzip -X /var/tmp/rear.M11HPUBIGuYyivX/tmp/backup-exclude.txt -C / -c -f - / /root/rear.2.7/var/log/rear/rear-localhost.log | dd of=/var/tmp/rear.M11HPUBIGuYyivX/outputfs/localhost/backup.tar.gz bs=1M

So normally at least '/' is in backup-include.txt

jsmeix commented at 2024-04-29 13:26:¶

Your
https://github.com/rear/rear/files/15097745/rear-hermes.log
contains (excerpt):

+ source /usr/share/rear/backup/NETFS/default/400_create_include_exclude_files.sh
++ is_true no
++ case "$1" in
++ return 1
++ '[' NO '!=' YES ']'
++ read mountpoint device junk
++ for backup_exclude_item in "${BACKUP_PROG_EXCLUDE[@]}"

while mine (without BACKUP_PROG_INCLUDE in local.conf)
contains (excerpt):

+ source /root/rear.2.7/usr/share/rear/backup/NETFS/default/400_create_include_exclude_files.sh
++ is_true no
++ case "$1" in
++ return 1
++ '[' NO '!=' YES ']'
++ read mountpoint device junk
++ IsInArray /
++ return 1
++ echo /
++ read mountpoint device junk
++ for backup_exclude_item in "${BACKUP_PROG_EXCLUDE[@]}"

The matching code in
usr/share/rear/backup/NETFS/default/400_create_include_exclude_files.sh
is

# Backup all that is explicitly specified in BACKUP_PROG_INCLUDE:
for backup_include_item in "${BACKUP_PROG_INCLUDE[@]}" ; do
    test "$backup_include_item" && echo "$backup_include_item"
done > $TMP_DIR/backup-include.txt

# Implicitly also backup all local filesystems as defined in mountpoint_device
# except BACKUP_ONLY_INCLUDE or MANUAL_INCLUDE is set:
if ! is_true "$BACKUP_ONLY_INCLUDE" ; then
    if [ "${MANUAL_INCLUDE:-NO}" != "YES" ] ; then
        # Add the mountpoints that will be recovered to the backup include list
        # unless a mountpoint is excluded:
        while read mountpoint device junk ; do
            if ! IsInArray "$mountpoint" "${EXCLUDE_MOUNTPOINTS[@]}" ; then
                echo "$mountpoint"
            fi
        done <"$VAR_DIR/recovery/mountpoint_device" >> $TMP_DIR/backup-include.txt
    fi
fi

# Exclude all that is explicitly specified in BACKUP_PROG_EXCLUDE:
for backup_exclude_item in "${BACKUP_PROG_EXCLUDE[@]}" ; do

so it seems your "$VAR_DIR/recovery/mountpoint_device" is empty
while mine contains

# cat var/lib/rear/recovery/mountpoint_device
/ /dev/sda2

@kai-uwe-rommel
is your var/lib/rear/recovery/mountpoint_device empty?
If not, what does it contain?

Because in your
https://github.com/rear/rear/files/15097745/rear-hermes.log
there is only

++ '[' NO '!=' YES ']'
++ read mountpoint device junk
++ for backup_exclude_item in "${BACKUP_PROG_EXCLUDE[@]}"

I assume your var/lib/rear/recovery/mountpoint_device is empty.

So my next step towards the root cause is
to find out why it is empty...

jsmeix commented at 2024-04-29 13:29:¶

@kai-uwe-rommel
a possible way to enforce having '/' in the NETFS backup is

BACKUP_ONLY_INCLUDE="yes"
BACKUP_PROG_INCLUDE=( / )

cf. the BACKUP_ONLY_INCLUDE description in default.conf

jsmeix commented at 2024-04-29 13:44:¶

Your
https://github.com/rear/rear/files/15097745/rear-hermes.log
contains:

+ source /usr/share/rear/layout/save/default/340_generate_mountpoint_device.sh
++ excluded_mountpoints=()
++ read fs device mountpoint junk
+++ grep '^fs' /var/lib/rear/layout/disklayout.conf
++ read fs device mountpoint junk
+++ grep '^fs' /var/lib/rear/layout/disklayout.conf
+ source_return_code=0

The matching code in
usr/share/rear/layout/save/default/340_generate_mountpoint_device.sh
is

excluded_mountpoints=()
while read fs device mountpoint junk ; do
    if IsInArray "fs:$mountpoint" "${EXCLUDE_BACKUP[@]}" ; then
        excluded_mountpoints+=( $mountpoint )
    fi
    for component in $(get_parent_components "fs:$mountpoint" | sort -u) ; do
        if IsInArray "$component" "${EXCLUDE_BACKUP[@]}" ; then
            excluded_mountpoints+=( $mountpoint )
        fi
    done
done < <(grep ^fs $LAYOUT_FILE)
...
while read fs device mountpoint junk ; do
    if IsInArray "$mountpoint" "${excluded_mountpoints[@]}" ; then
        continue
    fi
    echo "$mountpoint $device"
done < <(grep '^fs' $LAYOUT_FILE) > $VAR_DIR/recovery/mountpoint_device

By the way:
The second comment therein is plain wrong in ReaR 2.7.
I fixed it in current master code via
https://github.com/rear/rear/commit/341435c25cfb0072084fcd524519825dfe239e03
see also
https://github.com/rear/rear/issues/2906#issuecomment-1369919522

So it seems 'grep ^fs $LAYOUT_FILE' i.e.

# grep ^fs var/lib/rear/layout/disklayout.conf

results nothing which normally means
that there was no mounted filesystem
while "rear mkrescue/mkbackup" was run,
in particular that nothing was mounted at '/'
which looks inexplicable to me because according to your
https://github.com/rear/rear/files/14837097/lsblk.txt

NAME                          ... TYPE  FSTYPE ... SIZE MOUNTPOINT
    |-/dev/mapper/system-root     lvm   xfs        60G /
    |-/dev/mapper/system-root     lvm   xfs        60G /

an XFS on /dev/mapper/system-root is mounted at '/'.

@kai-uwe-rommel
please attach your var/lib/rear/layout/disklayout.conf file
that matches your
https://github.com/rear/rear/files/15097745/rear-hermes.log
or redo what you did for
https://github.com/rear/rear/files/15097745/rear-hermes.log
and attach a new "rear -D mkbackup" log file
plus the new var/lib/rear/layout/disklayout.conf file.

jsmeix commented at 2024-04-29 14:40:¶

Your
https://github.com/rear/rear/files/15097745/rear-hermes.log
contains (excerpt):

# cut -b-150 rear-hermes.log | egrep '+ source |echo .*fs '
...
+ source /usr/share/rear/layout/save/GNU/Linux/230_filesystem_layout.sh
++ echo '# Filesystems (only ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs are supported).'
++ echo '# Format: fs <device> <mountpoint> <fstype> [uuid=<uuid>] [label=<label>] [<attributes>]'
++ echo -n 'fs /dev/mapper/system-root / xfs'
++ echo -n 'fs /dev/mapper/vg_HSM_backup-lv_HSM_backup /hana/shared/HSM/HDB00/backup xfs'
++ echo -n 'fs /dev/mapper/vg_HSM_data-lv_HSM_data /hana/data/HSM xfs'
++ echo -n 'fs /dev/mapper/vg_HSM_log-lv_HSM_log /hana/log/HSM xfs'
++ echo -n 'fs /dev/mapper/vg_HSM_shrd-lv_HSM_sap /usr/sap xfs'
++ echo -n 'fs /dev/mapper/vg_HSM_shrd-lv_HSM_shrd /hana/shared/HSM xfs'

so your disklayout.conf should contain a line

fs /dev/mapper/system-root / xfs

But later basically all in disklayout.conf gets disabled:

# egrep ' source /usr/share/rear/layout/save/default/330_remove_exclusions.sh| Disabling component ' rear-hermes.log | egrep ' source |^2024-04-24 '

+ source /usr/share/rear/layout/save/default/330_remove_exclusions.sh
2024-04-24 18:38:26.240582346 Disabling component 'lvmgrp /dev/system' in /var/lib/rear/layout/disklayout.conf
2024-04-24 18:38:26.247981103 Disabling component 'lvmgrp /dev/vg_HSM_log' in /var/lib/rear/layout/disklayout.conf
2024-04-24 18:38:26.255566589 Disabling component 'lvmgrp /dev/vg_HSM_shrd' in /var/lib/rear/layout/disklayout.conf
2024-04-24 18:38:26.262875234 Disabling component 'lvmgrp /dev/vg_HSM_data' in /var/lib/rear/layout/disklayout.conf
2024-04-24 18:38:26.270048847 Disabling component 'lvmgrp /dev/vg_HSM_backup' in /var/lib/rear/layout/disklayout.conf
2024-04-24 18:38:26.321328596 Disabling component 'fs ... /' in /var/lib/rear/layout/disklayout.conf
2024-04-24 18:38:26.328537547 Disabling component 'fs ... /hana/shared/HSM/HDB00/backup' in /var/lib/rear/layout/disklayout.conf
2024-04-24 18:38:26.335848129 Disabling component 'fs ... /hana/data/HSM' in /var/lib/rear/layout/disklayout.conf
2024-04-24 18:38:26.343437911 Disabling component 'fs ... /hana/log/HSM' in /var/lib/rear/layout/disklayout.conf
2024-04-24 18:38:26.350743767 Disabling component 'fs ... /usr/sap' in /var/lib/rear/layout/disklayout.conf
2024-04-24 18:38:26.357935255 Disabling component 'fs ... /hana/shared/HSM' in /var/lib/rear/layout/disklayout.conf
2024-04-24 18:38:26.364964298 Disabling component 'swap /dev/mapper/system-swap' in /var/lib/rear/layout/disklayout.conf
2024-04-24 18:38:26.372457798 Disabling component 'multipath /dev/mapper/36005076801b901fc6800000000000276' in /var/lib/rear/layout/disklayout.conf
2024-04-24 18:38:26.381346007 Disabling component 'part /dev/mapper/36005076801b901fc6800000000000276' in /var/lib/rear/layout/disklayout.conf
2024-04-24 18:38:26.394557309 Disabling component 'multipath /dev/mapper/36005076801b901fc680000000000027a' in /var/lib/rear/layout/disklayout.conf
2024-04-24 18:38:26.401261285 Disabling component 'multipath /dev/mapper/36005076801b901fc680000000000027b' in /var/lib/rear/layout/disklayout.conf
2024-04-24 18:38:26.408074822 Disabling component 'multipath /dev/mapper/36005076801b901fc680000000000027c' in /var/lib/rear/layout/disklayout.conf
2024-04-24 18:38:26.415338362 Disabling component 'multipath /dev/mapper/36005076801b901fc680000000000027e' in /var/lib/rear/layout/disklayout.conf
2024-04-24 18:38:26.424136533 Disabling component 'lvmdev /dev/system' in /var/lib/rear/layout/disklayout.conf
2024-04-24 18:38:26.431613783 Disabling component 'lvmdev /dev/vg_HSM_log' in /var/lib/rear/layout/disklayout.conf
2024-04-24 18:38:26.438902571 Disabling component 'lvmdev /dev/vg_HSM_shrd' in /var/lib/rear/layout/disklayout.conf
2024-04-24 18:38:26.446098694 Disabling component 'lvmdev /dev/vg_HSM_data' in /var/lib/rear/layout/disklayout.conf
2024-04-24 18:38:26.453141357 Disabling component 'lvmdev /dev/vg_HSM_backup' in /var/lib/rear/layout/disklayout.conf

So my next step towards the root cause is
to find out why all that gets disabled...

jsmeix commented at 2024-04-29 15:02:¶

It seems I got it:

2024-04-24 18:38:25.226539584 Including layout/save/default/320_autoexclude.sh
2024-04-24 18:38:25.229744397 Entering debugscript mode via 'set -x'.
2024-04-24 18:38:25.238001570 Automatically excluding multipath device /dev/mapper/36005076801b901fc6800000000000276
2024-04-24 18:38:25.243734169 Marking component '/dev/mapper/36005076801b901fc6800000000000276' as done in /var/lib/rear/layout/disktodo.conf
...

Very many of those kind of messages.

I think this is because your '/' is on a multipath device

NAME                                                    KNAME      PKNAME    TRAN TYPE  FSTYPE       LABEL                             SIZE MOUNTPOINT
...
|-/dev/sda2                                             /dev/sda2  /dev/sda       part  LVM2_member                                     82G
`-/dev/mapper/36005076801b901fc6800000000000276         /dev/dm-1  /dev/sda       mpath                                                100G
  |-/dev/mapper/36005076801b901fc6800000000000276-part1 /dev/dm-4  /dev/dm-1      part                                                   4M
  `-/dev/mapper/36005076801b901fc6800000000000276-part2 /dev/dm-5  /dev/dm-1      part  LVM2_member                                     82G
    |-/dev/mapper/system-root                           /dev/dm-6  /dev/dm-5      lvm   xfs                                             60G /
...
|-/dev/sde2                                             /dev/sde2  /dev/sde       part  LVM2_member                                     82G
`-/dev/mapper/36005076801b901fc6800000000000276         /dev/dm-1  /dev/sde       mpath                                                100G
  |-/dev/mapper/36005076801b901fc6800000000000276-part1 /dev/dm-4  /dev/dm-1      part                                                   4M
  `-/dev/mapper/36005076801b901fc6800000000000276-part2 /dev/dm-5  /dev/dm-1      part  LVM2_member                                     82G
    |-/dev/mapper/system-root                           /dev/dm-6  /dev/dm-5      lvm   xfs                                             60G /

but by default (i.e. in default.conf) we have

# Automatically exclude multipath disks and their dependent components
AUTOEXCLUDE_MULTIPATH=y

cf.
usr/share/rear/conf/examples/RHEL7-PPC64LE-Multipath-PXE-GRUB.conf
(excerpt):

# SAN and multipath and boot on SAN specific part on PPC64/PPC64LE:
# There is no direct link between multipath/boot_on_SAN and PXE (even on POWER).
# Multipath is used in this example because PowerVM LPARs usually need multipath.
# There is nothing special on PowerVM LPAR with single VIOS but that is not commonly used.
# In contrast PowerVM LPAR with DUAL VIOS (HA configuration) needs multipath
# for dual disk access (vscsi over 2 VIOS) or NPIV (Fiber Channel virtualization).
# And NPIV (which is quite popular) needs also BOOT_ON_SAN.
AUTOEXCLUDE_MULTIPATH=n
BOOT_OVER_SAN=y

so you need some specific settings in etc/rear/local.conf
for your specific hardware.

jsmeix commented at 2024-04-30 07:43:¶

'BOOT_ON_SAN' in the comment in
usr/share/rear/conf/examples/RHEL7-PPC64LE-Multipath-PXE-GRUB.conf
is a typo that will get fixed via
https://github.com/rear/rear/pull/3211

kai-uwe-rommel commented at 2024-04-30 13:30:¶

Yes, the /var/lib/rear/recovery/mountpoint_device is empty.
disklayout.conf
Yes, / is on a multipath device.

kai-uwe-rommel commented at 2024-04-30 13:53:¶

I have now set

AUTOEXCLUDE_MULTIPATH=n
ONLY_INCLUDE_VG=( system )

and NETFS works fine that way.

jsmeix commented at 2024-04-30 14:06:¶

I think you may also need

BOOT_OVER_SAN=y

Did you try out that your ReaR recovery system
can actually boot on your replacement hardware (or VM)
and that the ReaR recovery system seems to work?
I.e. log in as 'root' and call for example 'lsblk'.
Does it list the expected block devices?

kai-uwe-rommel commented at 2024-04-30 14:13:¶

I have not used BOOT_OVER_SAN=y yet. But will add it.
When I boot the rescue ISO, I see the disk configured for the LPAR.
I have not yet attempted "rear recover" and won't have time to start this today.
But I already wondered if it would also restore the multipath config and what I need to do to get it do that.
The lsblk shows the (single) multipathed disk currently as two separate disks.

kai-uwe-rommel commented at 2024-04-30 14:54:¶

Something missing here? (BOOT_FROM_SAN=yes is set)
Can this be caused by the "loaded_modules"?

RESCUE hermes:~ # multipath
1730.984273 | DM multipath kernel driver not loaded

jsmeix commented at 2024-05-03 12:09:¶

Via
https://github.com/rear/rear/commit/7b2354e8a91306adeaf9228d79d13b23b5426011
I describe in usr/share/rear/conf/default.conf
our findings so far from this issue here
how one could reduce the size of ReaR's initrd
in case of BACKUP=TSM in particular on POWER architecture
via COPY_AS_IS_TSM and COPY_AS_IS_EXCLUDE_TSM, see
https://github.com/rear/rear/issues/3189#issuecomment-2079811746

I also describe the about 123 MiB initrd size limit on POWER
(128 MiB minus about 4 MiB for additional prep boot data), see
https://github.com/rear/rear/pull/3212#issuecomment-2092481874

I also tell about generic methods to reduce initrd size
like MODULES=( 'loaded_modules' ), see
https://github.com/rear/rear/issues/3189#issuecomment-2076939562
and REAR_INITRD_COMPRESSION="lzma", see
https://github.com/rear/rear/issues/3189#issuecomment-2079794186
and I mention that on POWER by default
FIRMWARE_FILES=( 'no' ) is used, see
https://github.com/rear/rear/issues/3189#issuecomment-2076960341

kai-uwe-rommel commented at 2024-05-03 12:22:¶

The size reduction of initrd for BACKUP=TSM is not platform specific for POWER. It applies to the usual x64 platforms as well. It may not be needed there to reduce initrd to avoid boot failures but it speeds things up and reduces the storage footprint.

You may also want to put in a warning about systems using multipathing that they may end up with empty NETFS backups and need to use suitable explicit include statements.

jsmeix commented at 2024-05-03 12:41:¶

@kai-uwe-rommel
regarding BOOT_OVER_SAN=y and multipath:

In your
https://github.com/rear/rear/issues/3189#issuecomment-2085562002
you wrote BOOT_FROM_SAN=yes is set.
There is no 'BOOT_FROM_SAN', there is only 'BOOT_OVER_SAN', see
https://github.com/rear/rear/issues/3189#issuecomment-2084609577
and https://github.com/rear/rear/pull/3211

'BOOT_OVER_SAN' is only used in
usr/share/rear/prep/GNU/Linux/240_include_multipath_tools.sh
which is run during "rear mkrescue/mkbackup" and in
usr/share/rear/layout/prepare/GNU/Linux/210_load_multipath.sh
which is run during "rear recover".

The former includes multipath tools into the recovery system
that are needed there during "rear recover" when
usr/share/rear/layout/prepare/GNU/Linux/210_load_multipath.sh
is run which activates multipath if BOOT_OVER_SAN is true.

In particular with BOOT_OVER_SAN=y multipath gets not activated
during recovery system startup via some system-setup script in
usr/share/rear/skel/default/etc/scripts/system-setup.d/
which appear under /etc/scripts/system-setup.d/
in the running recovery system.
So when one logs in as 'root' in the booted recovery system
multipath is not yet activated with BOOT_OVER_SAN=y.

With BOOT_OVER_SAN=y multipath gets activated when
usr/share/rear/layout/prepare/GNU/Linux/210_load_multipath.sh
is run during "rear recover".

I cannot tell why multipath gets not activated during
recovery system startup but later during "rear recover" via
usr/share/rear/layout/prepare/GNU/Linux/210_load_multipath.sh
It originated in
https://github.com/rear/rear/commit/ff19120671a5571a4be354b05d4b45a843933ec8
as usr/share/rear/layout/prepare/GNU/Linux/21_load_multipath.sh

I have zero personal experience with multipath
so I need to somehow find my way through here and
when I get again stuck now with multipath like above
https://github.com/rear/rear/issues/3189#issuecomment-2042901448
another SUSE support case would be needed to get
multipath issues on POWER architecture properly sorted out.

To find out whether or not "rear recover" actually works
on your POWER system with or without 'BOOT_OVER_SAN=y'
you need a fully separated and fully compatible
replacement system where you could try out "rear recover",
see in parrticular the sections
"Fully compatible replacement hardware is needed"
and
"No disaster recovery without testing and continuous validation"
in
https://en.opensuse.org/SDB:Disaster_Recovery

jsmeix commented at 2024-05-03 13:02:¶

@kai-uwe-rommel
regarding initrd size reduction for BACKUP=TSM
on any platform via a reduced/minimal TSM client see my
https://github.com/rear/rear/pull/3212#issuecomment-2092481874
(excerpt)

Same basic reasoning (i.e. "it depends")
for including a reduced/minimal TSM client:
When the initrd size does not matter in some cases
then the full TSM client should normally be included
to have all TSM client functionality available
during ReaR recovery system runtime.

In the past we had way too many various kind of issues
here at ReaR upstream because ReaR tried by default
too much to keep its recovery system small.

When something is needed in this or that specific case
in the ReaR recovery system but it is not there,
then "rear recover" is usually a dead end
and instead of a working disaster recovery method
ReaR becomes a final disaster (in some specific cases).

My main example is my change to the new default
MODULES=( 'all_modules' ) in ReaR 2.5, cf.
https://github.com/rear/rear/blob/master/doc/rear-release-notes.txt
because we had way too many issues where crucial things
did not work in the ReaR recovery system because some
"special" additionally needed kernel module was missing
(not much fun when e.g. a USB keyboard does not work)
see my description in default.conf about MODULES
and see the therein mentioned ReaR upstream issues.

There is a crucial point:
By default things should work reasonably well.
But there must be also "final power to the user"
which means that the user can specify things
to be different than the defaults as the user wants.

jsmeix commented at 2024-05-03 13:11:¶

@kai-uwe-rommel
regarding "warning about ... empty NETFS backups":

Yes, I am already thinking about a generic check for
an empty NETFS backup (regardless of multipath or whatever).

My point is that I assume at least '/' must be included
in any backup because it is the basic functionality
that "rear mkbackup" makes a backup of the files
of the basic system so at least '/' must be
backed up (with '--one-file-system').

So in case of a NETFS backup at least '/'
should normally be in backup-include.txt
and - as far as I can imagine - at least an
empty backup-include.txt is always an error.

kai-uwe-rommel commented at 2024-05-03 13:32:¶

Thanks for the multipath stuff explanation.
Will test as time permits.

Regarding TSM client size reduction:
what was previously in the comments was indeed
a "stripped down" client e.g. which could have
functionality missing.
After looking at what makes it so fat
I found that I would get most reduction
by removing all the foreign language files.
This way I got a slim TSM client in the rescue image
without (!) reducing functionality.
Since that was not yet sufficient for the POWER 123 MB limit,
I decided to also remove *.jar files which are only needed
for the Java based GUI (X11) client - but X11 is not available
in rescue image anyway so this is no real restriction either.

jsmeix commented at 2024-05-03 15:06:¶

@kai-uwe-rommel
thank you for your explanation why those *.jar files are not needed.
Now it is much more clear to me that what you excluded
did not result a TSM client with less functionality,
at least not for the TSM version you have.

I will further enhance the comment in default.conf
to better explain that crucial difference compared to
the minimal COPY_AS_IS_TSM example.

What still worries me a bit is that

/opt/tivoli/tsm/client/*/*/[A-Z][!N]_??

may accidentally match too many files
(i.e. more than only language files)
because of too unspecific /*/*/ and _??
so it may accidentally result a crippled TSM client
for some other/future TSM versions.
Perhaps something more specific like

COPY_AS_IS_EXCLUDE_TSM=( /opt/tivoli/tsm/client/ba/bin/*.jar
                         /opt/tivoli/tsm/client/ba/bin/*.log*
                         /opt/tivoli/tsm/client/ba/bin/[A-Z][!N]_[!U][A-Z]
                         /opt/tivoli/tsm/client/api/bin64/[A-Z][!N]_[!U][A-Z] )

could be better as an example to make it more clear
what files are meant to be excluded.

Did you try out if the EN_US language files are really needed
to normally use the TSM client (in English)?

I think that usually the strings in the source code
are in English so I think that usually no translation
into English is required (i.e. the user would then see
the original strings "as is" from the source code).

As far as I know EN_US or EN_GB language files are
usually onyl there to provide US or GB English style
for arbitrary English strings in the source code
or some style or spelling fixes for English strings
in the source code (e.g. for English mistakes
from non-native English source code developers).

I.e. could you (as time permits) please test if
you can normally use the TSM client in English
even without EN_US language files with

COPY_AS_IS_EXCLUDE_TSM=( /opt/tivoli/tsm/client/ba/bin/*.jar
                         /opt/tivoli/tsm/client/ba/bin/*.log*
                         /opt/tivoli/tsm/client/ba/bin/[A-Z][A-Z]_[A-Z][A-Z]
                         /opt/tivoli/tsm/client/api/bin64/[A-Z][A-Z]_[A-Z][A-Z] )

kai-uwe-rommel commented at 2024-05-03 15:56:¶

The wildcard [A-Z][!N]_?? only matches files (in this case, directories) that look line X?_?? but not XN_?? (X = any upper case letter). So it will not match EN_US and there are actually no other such directories with N as the second. There is IMO almost no risk that there will be other X?_?? files other than the typically named language file subdirectories. Probably the wildcard can be refined to more restrictively match what I intended but then it will become quite long and "ugly" to read. Also, I was not sure, if more sophisticated regular expressions are accepted and properly interpreted/matched here.

The /*/*/ means it matches any such file/directory under .../tsm/client/ba/bin/ and .../tsm/client/api/bin64/ directories. There are no other directories under .../tsm/client than ba and api.

And yes, some language files are needed and EN_US is used per default.

kai-uwe-rommel commented at 2024-05-06 13:28:¶

Did the complete recovery test in a new/separate LPAR right now. Works.
Multipathing is initiated properly and root partition and filesystem created, too.
TSM restore into this new / also worked.
Could then modify IP address before reboot.
Reboot of the new LPAR worked.
Of course, then the usual stuff that needs to be done when booting with all the other LVM disks and filesystems missing e.g. emergency mode, changing /etc/fstab (could have done this before the first boot if I had remembered it).

jsmeix commented at 2024-05-13 15:07:¶

@kai-uwe-rommel
thank you for the feedback how far it worked
in your case.

Can this issue be closed from your point of view?

FYI only an offhanded side note regarding your

... the usual stuff ... with all the other LVM disks
and filesystems missing ... changing /etc/fstab
... before the first boot

You may have a look at
https://github.com/rear/rear/issues/2812#issuecomment-1251096019

https://github.com/rear/rear/commit/e822ad69a8ce8dec6132741806008db9c6c3b429
shows an example how an /etc/rear/lun_wwid_mapping.conf
may look like.

As far as I imagine
with such an /etc/rear/lun_wwid_mapping.conf
usr/share/rear/finalize/GNU/Linux/250_migrate_lun_wwid.sh
should at the end of "rear recover"
adapt only etc/elilo.conf and etc/fstab
in the restored system (i.e. within /mnt/local).

I assume etc/elilo.conf is from older times
when (only?) elilo was used as bootloader on POWER.

Currently grub.cfg is not adapted by
usr/share/rear/finalize/GNU/Linux/250_migrate_lun_wwid.sh
so for GRUB this needs to be enhanced.

I think what you may try out is:
As root in the booted ReaR recovery system
before calling "rear recover"
manually create an /etc/rear/lun_wwid_mapping.conf file
according to what 'lsblk' shows in the ReaR recovery system
for the "new_lun_wwid" values, e.g. use something like

# lsblk -ipo NAME,KNAME,TYPE,SIZE,UUID,WWN

The "original_lun_wwid" values should be stored in
your var/lib/rear/layout/disklayout.conf
at least in the initial comment that shows the

lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT,UUID,WWN

output since ReaR 2.7 via
https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/layout/save/GNU/Linux/100_create_layout_file.sh#L28

But having only an /etc/rear/lun_wwid_mapping.conf
(without also having an /etc/rear/disklayout.conf)
my not work because of the code in
usr/share/rear/layout/prepare/default/010_prepare_files.sh
see the comments in that code
https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/layout/prepare/default/010_prepare_files.sh#L28

kai-uwe-rommel commented at 2024-05-13 15:30:¶

Thanks for the explanation.
The case can be closed now. All problems solved.

jsmeix commented at 2024-05-15 15:45:¶

With https://github.com/rear/rear/pull/3219
and https://github.com/rear/rear/pull/3221
merged (the latter solves https://github.com/rear/rear/issues/3217)
all subsequent issues from this one should be fixed
so that this one should be done.

[Export of Github issue for rear/rear.]

#3189 Issue closed: BACKUP=TSM booting rescue system (ISO) on POWER fails (initrd bigger than 123 MiB)¶