#3149 Issue open: OS vendor and version autodetection fails on Fedora¶
Labels: bug, cleanup
lzaoral opened issue at 2024-02-08 12:26:¶
-
ReaR version ("/usr/sbin/rear -V"): latest
HEAD -
OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):
$ cat /etc/os-release
NAME="Fedora Linux"
VERSION="39 (Thirty Nine)"
ID=fedora
VERSION_ID=39
VERSION_CODENAME=""
PLATFORM_ID="platform:f39"
PRETTY_NAME="Fedora Linux 39 (Thirty Nine)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:39"
DEFAULT_HOSTNAME="fedora"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f39/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=39
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=39
SUPPORT_END=2024-11-12
- Description of the issue (ideally so that others can reproduce it):
ReaR does not properly detect OS vendor and version on Fedora:
# /usr/sbin/rear dump
...
# System definition:
ARCH="Linux-arm"
OS="GNU/Linux"
OS_MASTER_VENDOR="Fedora"
OS_MASTER_VERSION="VERSION_ID=39"
OS_MASTER_VENDOR_ARCH="Fedora/arm"
OS_MASTER_VENDOR_VERSION="Fedora/VERSION_ID=39"
OS_MASTER_VENDOR_VERSION_ARCH="Fedora/VERSION_ID=39/arm"
OS_VENDOR="RedHatEnterpriseServer"
OS_VERSION="VERSION_ID=39"
OS_VENDOR_ARCH="RedHatEnterpriseServer/arm"
OS_VENDOR_VERSION="RedHatEnterpriseServer/VERSION_ID=39"
OS_VENDOR_VERSION_ARCH="RedHatEnterpriseServer/VERSION_ID=39/arm"
...
The version should be only 39 and not VERSION_ID=39 and the vendor
should be Fedora. Therefore, both the OS_VENDOR_* and
OS_MASTER_VENDOR_* should be equal.
cc: @pcahyna
jsmeix commented at 2024-02-08 15:36:¶
As far as I know this variables are set by the function
SetOSVendorAndVersion
in lib/config-functions.sh
I think this function is rather fragile
(some time ago I have had "not so much fun" with it).
I assume most of its values are not actually used in ReaR scripts
so perhaps things could be simplified to what is actually used?
jsmeix commented at 2024-02-08 15:43:¶
FYI:
What variables are set by SetOSVendorAndVersion:
# grep -o '[A-Z][A-Z_]*=' usr/share/rear/lib/config-functions.sh | cut -d '=' -f1 | sort -u
OS_MASTER_VENDOR
OS_MASTER_VENDOR_ARCH
OS_MASTER_VENDOR_VERSION
OS_MASTER_VENDOR_VERSION_ARCH
OS_MASTER_VERSION
OS_VENDOR
OS_VENDOR_ARCH
OS_VENDOR_VERSION
OS_VENDOR_VERSION_ARCH
OS_VERSION
VERSION_ID
Where those variables are used in ReaR scripts:
Here I excluded ReaR scripts that use those variables
in a generic way which are
usr/share/rear/lib/config-functions.sh
usr/share/rear/lib/dump-workflow.sh
usr/share/rear/lib/framework-functions.sh
usr/share/rear/lib/validate-workflow.sh
usr/sbin/rear
usr/share/rear/build/default/975_update_os_conf.sh
usr/share/rear/rescue/default/010_merge_skeletons.sh
usr/share/rear/finalize/default/890_finish_checks.sh
usr/share/rear/init/default/005_verify_os_conf.sh
so only those ReaR scripts should be left
where the behaviour of ReaR depends on those variables:
# for v in $( grep -o '[A-Z][A-Z_]*=' usr/share/rear/lib/config-functions.sh | cut -d '=' -f1 | sort -u ) ; \
do echo $v ; \
find usr/sbin/rear usr/share/rear -type f | xargs grep "$v" | grep -v ': *#' ; \
echo ======================== ; \
done \
| egrep -v 'usr/share/rear/lib/config-functions.sh|usr/share/rear/lib/dump-workflow.sh|usr/share/rear/lib/framework-functions.sh|usr/share/rear/lib/validate-workflow.sh|usr/sbin/rear|usr/share/rear/build/default/975_update_os_conf.sh|usr/share/rear/rescue/default/010_merge_skeletons.sh|usr/share/rear/finalize/default/890_finish_checks.sh|usr/share/rear/init/default/005_verify_os_conf.sh
OS_MASTER_VENDOR
usr/share/rear/layout/prepare/Linux-s390/205_s390_enable_disk.sh:case $OS_MASTER_VENDOR in
usr/share/rear/layout/prepare/Linux-s390/205_s390_enable_disk.sh: LogPrintError "No code for DASD disk device enablement on $OS_MASTER_VENDOR"
usr/share/rear/rescue/GNU/Linux/310_network_devices.sh:if [[ "$ARCH" == "Linux-s390" && "$OS_MASTER_VENDOR" != "SUSE_LINUX" ]] ; then
usr/share/rear/lib/layout-functions.sh: case $OS_MASTER_VENDOR in
========================
OS_MASTER_VENDOR_ARCH
========================
OS_MASTER_VENDOR_VERSION
========================
OS_MASTER_VENDOR_VERSION_ARCH
========================
OS_MASTER_VERSION
usr/share/rear/lib/layout-functions.sh: if (( $OS_MASTER_VERSION < 12 )) ; then
usr/share/rear/lib/layout-functions.sh: if (( $OS_MASTER_VERSION < 7 )) ; then
========================
OS_VENDOR
usr/share/rear/conf/default.conf:OS_VENDOR=generic
usr/share/rear/rescue/GNU/Linux/990_sysreqs.sh: echo "${OS_VENDOR} ${OS_VERSION}"
usr/share/rear/lib/bootloader-functions.sh: echo "echo \"build from host: $HOSTNAME ($OS_VENDOR $OS_VERSION $ARCH)\""
usr/share/rear/finalize/Linux-i386/610_EFISTUB_run_efibootmgr.sh:--label ${OS_VENDOR} --loader "$loader" \
usr/share/rear/finalize/Linux-i386/670_run_efibootmgr.sh: LogPrint "Creating EFI Boot Manager entry '$OS_VENDOR $OS_VERSION' for '$bootloader' (UEFI_BOOTLOADER='$UEFI_BOOTLOADER') "
usr/share/rear/finalize/Linux-i386/670_run_efibootmgr.sh: Log efibootmgr --create --gpt --disk $disk --part $partition_number --write-signature --label \"${OS_VENDOR} ${OS_VERSION}\" --loader \"\\${bootloader}\"
usr/share/rear/finalize/Linux-i386/670_run_efibootmgr.sh: if efibootmgr --create --gpt --disk $disk --part $partition_number --write-signature --label "${OS_VENDOR} ${OS_VERSION}" --loader "\\${bootloader}" ; then
========================
OS_VENDOR_ARCH
========================
OS_VENDOR_VERSION
========================
OS_VENDOR_VERSION_ARCH
========================
OS_VERSION
usr/share/rear/conf/default.conf:OS_VERSION=none
usr/share/rear/rescue/GNU/Linux/990_sysreqs.sh: echo "${OS_VENDOR} ${OS_VERSION}"
usr/share/rear/lib/bootloader-functions.sh: echo "echo \"build from host: $HOSTNAME ($OS_VENDOR $OS_VERSION $ARCH)\""
usr/share/rear/finalize/Linux-i386/670_run_efibootmgr.sh: LogPrint "Creating EFI Boot Manager entry '$OS_VENDOR $OS_VERSION' for '$bootloader' (UEFI_BOOTLOADER='$UEFI_BOOTLOADER') "
usr/share/rear/finalize/Linux-i386/670_run_efibootmgr.sh: Log efibootmgr --create --gpt --disk $disk --part $partition_number --write-signature --label \"${OS_VENDOR} ${OS_VERSION}\" --loader \"\\${bootloader}\"
usr/share/rear/finalize/Linux-i386/670_run_efibootmgr.sh: if efibootmgr --create --gpt --disk $disk --part $partition_number --write-signature --label "${OS_VENDOR} ${OS_VERSION}" --loader "\\${bootloader}" ; then
========================
VERSION_ID
========================
So it seems (according to that quick and dirty test)
those variables are not actually used in ReaR scripts
OS_MASTER_VENDOR_ARCH
OS_MASTER_VENDOR_VERSION
OS_MASTER_VENDOR_VERSION_ARCH
OS_VENDOR_ARCH
OS_VENDOR_VERSION
OS_VENDOR_VERSION_ARCH
VERSION_ID
jsmeix commented at 2024-02-08 16:19:¶
What directories we have where the value of such variables
is used as directory name (also a quick and dirty test):
I exluded some generic directories
and those where the OUTPUT value
# grep -o 'OUTPUT=[A-Z]*' usr/share/rear/conf/default.conf | sort -u
OUTPUT=ISO
OUTPUT=OBDR
OUTPUT=PXE
OUTPUT=RAMDISK
OUTPUT=RAWDISK
OUTPUT=USB
or the BACKUP value
# grep -o 'BACKUP=[A-Z][A-Z0-9]*' usr/share/rear/conf/default.conf | sort -u
BACKUP=AVA
BACKUP=BACULA
BACKUP=BAREOS
BACKUP=BLOCKCLONE
BACKUP=BORG
BACKUP=CDM
BACKUP=DP
BACKUP=DUPLICITY
BACKUP=EXTERNAL
BACKUP=FDRUPSTREAM
BACKUP=GALAXY
BACKUP=GALAXY10
BACKUP=GALAXY11
BACKUP=GALAXY7
BACKUP=NBKDC
BACKUP=NBU
BACKUP=NETFS
BACKUP=NFS4SERVER
BACKUP=NSR
BACKUP=RBME
BACKUP=REQUESTRESTORE
BACKUP=RSYNC
BACKUP=SESAM
BACKUP=TSM
BACKUP=YUM
BACKUP=ZYPPER
and where GNU and Linux
is the last directory name
# pushd usr/share/rear/ ; \
find . \
| sed -e 's|/[0-9][0-9][0-9].*.sh||g' \
| sort -u \
| egrep -v '/conf/examples|/conf/templates|/lib|/skel|/default|readme|README|shellcheckrc' \
| egrep -v '/ISO$|/OBDR$|/PXE$|/RAMDISK$|/RAWDISK$|/USB$' \
| egrep -v '/AVA$|/BACULA$|/BAREOS$|/BLOCKCLONE$|/BORG$|/CDM$|/DP$|/DUPLICITY$|/EXTERNAL$|/FDRUPSTREAM$|/GALAXY$|/GALAXY10$|/GALAXY11$|/GALAXY7$|/NBKDC$|/NBU$|/NETFS$|/NFS4SERVER$|/NSR$|/RBME$|/REQUESTRESTORE$|/RSYNC$|/SESAM$|/TSM$|/YUM$|/ZYPPER$' \
| egrep -v '/GNU$|/Linux$' ; \
popd
.
./backup
./build
./build/Debian
./build/Debian/i386
./build/OPALPBA
./build/OPALPBA/Linux-i386
./build/SUSE_LINUX
./check
./conf
./conf/Debian
./conf/Debian/ia64.conf
./conf/GNU/Linux.conf
./conf/Linux-i386.conf
./conf/Linux-ia64.conf
./conf/Linux-ppc64.conf
./conf/Linux-ppc64le.conf
./conf/SUSE_LINUX.conf
./conf/Ubuntu
./conf/Ubuntu.conf
./conf/Ubuntu/7.10.conf
./final-mount
./finalize
./finalize/Debian
./finalize/Debian/i386
./finalize/Debian/ppc64le
./finalize/Fedora
./finalize/Fedora/s390
./finalize/Linux-arm
./finalize/Linux-i386
./finalize/Linux-ia64
./finalize/Linux-ppc64
./finalize/Linux-ppc64le
./finalize/SUSE_LINUX
./finalize/SUSE_LINUX/i386
./finalize/SUSE_LINUX/ppc64
./finalize/SUSE_LINUX/ppc64le
./finalize/SUSE_LINUX/s390
./format
./init
./layout
./layout/compare
./layout/do-mount
./layout/precompare
./layout/prep-for-mount
./layout/prep-for-mount/Linux-s390
./layout/prepare
./layout/prepare/Linux-s390
./layout/recreate
./layout/save
./layout/save/FDRUPSTREAM/Linux-s390
./layout/save/Linux-arm
./output
./output/IPL
./output/IPL/Linux-s390
./output/ISO/Linux-i386
./output/ISO/Linux-ia64
./output/ISO/Linux-ppc64
./output/ISO/Linux-ppc64le
./output/OBDR/Linux-i386
./output/OBDR/Linux-ia64
./output/OBDR/Linux-ppc64
./output/OBDR/Linux-ppc64le
./output/RAWDISK/Linux-i386
./output/USB/Linux-i386
./pack
./prep
./prep/ISO/Linux-i386
./prep/ISO/Linux-ia64
./prep/Linux-arm
./prep/Linux-s390
./prep/OBDR/Linux-i386
./prep/OBDR/Linux-ia64
./prep/OPALPBA
./prep/OPALPBA/Linux-i386
./prep/RAWDISK/Linux-i386
./prep/TAPE
./prep/USB/Linux-arm
./prep/USB/Linux-i386
./prep/USB/Linux-ppc64
./prep/USB/Linux-ppc64le
./prep/USB/Linux-s390
./rescue
./restore
./restore/Fedora
./restore/NETFS/Linux-i386
./restore/OPALPBA
./restore/SUSE_LINUX
./setup
./verify
./wrapup
So it seems only those values are actually used
as directory names:
Debian
Fedora
IPL
Linux-arm
Linux-i386
Linux-ia64
Linux-ppc64
Linux-ppc64le
Linux-s390
SUSE_LINUX
Ubuntu
i386
ppc64
ppc64le
s390
By the way:
Currently I do not understand how
usr/share/rear/output/IPL/Linux-s390/800_create_ipl.sh
gets called because I fail to see
where the value IPL is set in ReaR.
lzaoral commented at 2024-02-09 09:24:¶
I have also found a related bug. If I create a custom
/etc/rear/os.conf with correct values for Fedora, the OS_* and
OS_MASTER_* variables will match. Therefore, the scripts array in
the SourceStage function will contain some Fedora-specific entries
twice because duplicates will not be discarded:
https://github.com/rear/rear/blob/096bfde5e234f5a803bae74f24e3821798022c7c/usr/share/rear/lib/framework-functions.sh#L120-L125
For example, the initrd regeneration is then executed twice
(rear recover log on s390x):
...
2024-02-06 10:56:46.249979866 ======================
2024-02-06 10:56:46.251775722 Running 'restore' stage
2024-02-06 10:56:46.253627052 ======================
2024-02-06 10:56:46.259645721 Including restore/Fedora/050_copy_dev_files.sh
2024-02-06 10:56:46.270044331 Including restore/Fedora/050_copy_dev_files.sh <-- DUPLICATE
2024-02-06 10:56:46.278551895 Including restore/default/050_remount_async.sh
2024-02-06 10:56:46.281950834 Including restore/NETFS/default/100_mount_NETFS_path.s
...
2024-02-06 10:57:07.404822080 Including finalize/default/520_confirm_finalize.sh
2024-02-06 10:57:07.408345447 Including finalize/Fedora/550_rebuild_initramfs.sh
2024-02-06 10:57:07.411208165 Original OLD_INITRD_MODULES=( )
2024-02-06 10:57:07.417129059 New INITRD_MODULES=' dasd_eckd_mod dasd_fba_mod dasd_mod'
2024-02-06 10:57:07.425685641 Running dracut...
2024-02-06 10:57:17.506420149 Updated initrd with new drivers for kernel 6.8.0-0.rc0.20240112git70d201a40823.5.fc40.s390x.
2024-02-06 10:57:17.511338220 Including finalize/Fedora/550_rebuild_initramfs.sh <-- DUPLICATE
2024-02-06 10:57:17.514697050 Original OLD_INITRD_MODULES=( )
2024-02-06 10:57:17.519743182 New INITRD_MODULES=' dasd_eckd_mod dasd_fba_mod dasd_mod'
2024-02-06 10:57:17.529820643 Running dracut...
2024-02-06 10:57:26.938005410 Updated initrd with new drivers for kernel 6.8.0-0.rc0.20240112git70d201a40823.5.fc40.s390x.
2024-02-06 10:57:26.944130040 Including finalize/Fedora/s390/660_install_zipl.sh
2024-02-06 10:57:26.946353158 Installing boot loader ZIPL...
2024-02-06 10:57:28.017620673 Including finalize/Fedora/s390/660_install_zipl.sh <-- DUPLICATE
2024-02-06 10:57:28.021901091 Including finalize/default/880_check_for_mount_by_id.sh
...
lzaoral commented at 2024-02-09 09:26:¶
By the way:
Currently I do not understand how
usr/share/rear/output/IPL/Linux-s390/800_create_ipl.sh
gets called because I fail to see
where the value IPL is set in ReaR.
It is a supported OUTPUT value on s390x. It just seems that the
documentation in default.conf is missing the OUTPUT=IPL entry.
jsmeix commented at 2024-02-09 09:44:¶
Regarding scripts get called twice
when different variables have same value:
Ugh!
It seems whenever I look at ReaR code what I see is a mess
that has grown over longer times :-(
jsmeix commented at 2024-02-09 09:52:¶
Regarding missing "OUTPUT=IPL" documentation in default.conf:
@lzaoral
could you please make a pull request with the missing
"OUTPUT=IPL" documentation in default.conf
because you actually use ReaR on s390x so you know better
than me what to tell our users about "OUTPUT=IPL".
I do no longer test ReaR on IBM Z and I never managed to boot the
ReaR recovery system via the initial program loader on IBM Z 'zipl'.
Thank you in advance!
lzaoral commented at 2024-02-12 12:27:¶
@jsmeix Hopefully, I'll get to the documentation of the IPL output
option this week. Right now, me and @pcahyna are fixing the packaging of
ReaR in Fedora (which we finally co-maintain) in the preparation of the
initial branching of CentOS Stream 10.
lzaoral commented at 2024-02-13 14:19:¶
@jsmeix I've found out, that the initial PR with s390x support in ReaR
was a bit messy. The IPL option is completely redundant because it
does the exact same thing as the already existing and documented
RAMDISK option:
https://github.com/rear/rear/blob/d48d071ddb6e8d70a42cbc3e6ffec0cb037e1ba1/usr/share/rear/output/IPL/Linux-s390/800_create_ipl.sh#L1-L4
vs.
Therefore, I suggest to just remove the IPL subtree and to add a
fallback that will replace OUTPUT=IPL with OUTPUT=RAMDISK during the
prep phase to be backwards compatible with existing local.conf
files. Unless there are objections, I'll prepare this clean-up PR
tomorrow.
jsmeix commented at 2024-02-14 09:27:¶
@lzaoral
thank you for having a look at our s390 code!
That IPL is redundant (same as RAMDISK)
is what I also thought but I was unsure
because I could no longer test ReaR on IBM Z
and because your above
https://github.com/rear/rear/issues/3149#issuecomment-1935589747
indicated that IPL is actually used.
I fully agree with your proposed clean-up
and I look forward to your PR!
By the way regarding documentation about
ReaR on IBM Z:
Could you write some initial documentation about
"ReaR on IBM Z"?
Only something short so that we have a starting point
which then can get enhanced step by step as needed?
In particular I would be much interested in
some documentation how to use ReaR on IBM Z
because I never tested what @mutable-dan had
implemented.
But I would like to try that out (as time permits)
to get at least an initial basic experience
how ReaR can be used on IBM Z.
In particular my main point where I know basically
nothing is how to make some "IPL-able thingy"
from ReaR's initrd and the kernel and how to
actually "IPL" that on a replacement VM on IBM Z.
I got lots of IBM documenation also about booting
on IBM Z - but that is a so much different world
that I gave up trying to understand all that
(hundreds of pages of IBM Z domain specific language).
So what I would like to get is some real example
how to make some real "IPL-able thingy" and
how to actually "IPL" that on one specific
kind of IBM Z VM.
[Export of Github issue for rear/rear.]