#1142 Issue `closed`: use better compression for initrd which must be less than 32MB when booting on ppc64 via yaboot¶

Labels: enhancement, fixed / solved / done

abbahag opened issue at 2016-12-29 09:16:¶

Relax-and-Recover (rear) Issue Template¶

Please fill in the following items before submitting a new issue (quick response is not guaranteed with free support):

rear version (/usr/sbin/rear -V):Relax-and-Recover 1.19-git201612131449 / 2016-12-13
OS version (cat /etc/rear/os.conf or lsb_release -a):

LSB Version:    core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-ppc64:core-3.2-ppc64:core-4.0-ppc64:desktop-4.0-noarch:desktop-4.0-ppc32:desktop-4.0-ppc64:graphics-2.0-noarch:graphics-2.0-ppc32:graphics-2.0-ppc64:graphics-3.2-noarch:graphics-3.2-ppc32:graphics-3.2-ppc64:graphics-4.0-noarch:graphics-4.0-ppc32:graphics-4.0-ppc64
Distributor ID: SUSE LINUX
Description:    SUSE Linux Enterprise Server 11 (ppc64)
Release:    11
Codename:   n/a

rear configuration files (cat /etc/rear/site.conf or cat /etc/rear/local.conf):

cat /etc/rear/local.conf

# sample local configuration
OUTPUT=PXE
# optionally define (non-default) backup software, e.g. TSM, NBU, DP, BACULA
## Backup with NFS the root VG
BACKUP=NETFS
BACKUP_URL=nfs://nim-v/NIM/REAR/node
NETFS_KEEP_OLD_BACKUP_COPY=1
ONLY_INCLUDE_VG=( system )

Brief description of the issue
If you want to boot SLES11 SP4 ppc64 to an lpar on IBM 9119-MME (P8 880) with firmware> = 840 via a TFTP server (NIM), the initrd must be less than or equal to 32 MB (with yaboot). Unfortunately, a standard HANA on Power (with SLES HA ) installation is larger than 32 MB.
Work-around, if any
The solution to the problem of switching from cpio/gzip to cpio/xz-lzma. This reduces the generated initrd by approximately 13 MB. With gzip, the initrd is 34.1 MB with xz and lzma 21.4 MB. For this purpose the file 900_create_initramfs.sh in /usr/share/rear/pack and the File 800_copy_to_tftp.sh in /usr/share/rear/output/ must be adapted.

900_create_initramfs.sh:

# 100_create_initramfs.sh
#
# create initramfs for Relax-and-Recover
#
# This file is part of Relax-and-Recover, licensed under the GNU General
# Public License. Refer to the included COPYING for full text of license.

LogPrint "Creating initramfs"

pushd "$ROOTFS_DIR" >&8
find . ! -name "*~"  |\
    tee /dev/fd/8  |\
    cpio -H newc --create --quiet  |\
    xz --format=lzma --compress --stdout > "$TMP_DIR/initrd.xz" # changed from gzip to xz lzma
StopIfError "Could not create initramfs archive"
popd >&8

800_copy_to_tftp.sh:
Change .cgz with .xz

jsmeix commented at 2016-12-30 14:12:¶

@abbahag
many thanks for your descriptive issue report
and your problem analysis plus a solution!

Regarding switching from cpio/gzip to cpio/xz-lzma
we need to check if cpio/xz-lzma is supported on
all platforms (even older ones) where ReaR runs,
cf. "Maintain backward compatibility" at
https://github.com/rear/rear/wiki/Coding-Style

Next Monday I will have a closer look.

For now:
Happy New Year!

jsmeix commented at 2017-01-02 16:09:¶

Not for the right now upcoming ReaR 2.0 but after that.

schabrolles commented at 2017-01-25 18:37:¶

Could it be specified via a VARIABLE OPTION in /etc/local.conf ? (INITRD_CPIO_COMPRESSION)
Default could stay unchanged "cpio/gzip", but we could change it to "cpio/xz-lzma" (if available)
The idea is similar to BACKUP_PROG_COMPRESS_OPTIONS or BORGBACKUP_COMPRESSION options.

abbahag commented at 2017-01-26 07:46:¶

I think your suggestion is the more elegant and more flexible. It should
be implemented via the local.conf.

jsmeix commented at 2017-01-26 11:23:¶

@schabrolles
I do fully agree.

In general:

I like it very much when "all and everything"
can be adjusted via configuration variables.

The less is hardcoded in the actual code and
the more is specified via configuration variables
the better it is.

When "all and everything" can be adjusted
via configuration variables the crucial word
is "can".

This means that usually the user should not need
to manually specifiy "all and everything".

For "all and everything" there should be a reasonable
fail-safe default setting where the primary intent behind is
to make things "just work" in the usual use cases (where
it is of course the question what "the usual use cases" are).

Furthermore for "all and everything" there should be a reasonable
fallback behaviour when there is no default setting
(e.g. when the user has specified VAR="")
or when it does not work with the default setting.

In this particular case we may even be able
to "automatically do the right thing" via the
"Dirty hacks welcome" method as described in
https://github.com/rear/rear/wiki/Coding-Style
something like the following:

# 900_create_initramfs.sh
#
# create initramfs for Relax-and-Recover
#
# This file is part of Relax-and-Recover, licensed under the GNU General
# Public License. Refer to the included COPYING for full text of license.

LogPrint "Creating initramfs"

pushd "$ROOTFS_DIR" >/dev/null
# First try to create initrd.xz with the newer xz-lzma compression,
# see https://github.com/rear/rear/issues/1142
if find . ! -name "*~" | cpio -H newc --create --quiet | xz --format=lzma --compress --stdout > "$TMP_DIR/initrd.xz" ; then
    LogPrint "Created initrd.xz with xz-lzma compression ($( stat -c%s $TMP_DIR/initrd.xz ) bytes)"
else 
    # If it fails to create initrd.xz with xz-lzma compression
    # fall back to the traditional way and create initrd.cgz with gzip compression,
    # cf. "Dirty hacks welcome" at https://github.com/rear/rear/wiki/Coding-Style
    if find . ! -name "*~" | cpio -H newc --create --quiet | gzip > "$TMP_DIR/initrd.cgz" ; then
        LogPrint "Created initrd.cgz with gzip compression ($( stat -c%s $TMP_DIR/initrd.cgz ) bytes)"
    else
        Error "Failed to create initramfs"
    fi
fi
popd >/dev/null

As a consequence all scripts where initrd.cgz is useed
must be adapted accordingly to try first initrd.xz
and fall back to initrd.cgz if initrd.xz does not exist like
in 800_copy_to_tftp.sh at the beginning

# In 900_create_initramfs.sh the Relax-and-Recover initrd is created
# either as initrd.xz with the newer xz-lzma compression
# or as initrd.cgz with gzip compression
# see https://github.com/rear/rear/issues/1142
test -s "$TMP_DIR/initrd.cgz" && initrd_filename="initrd.cgz"
test -s "$TMP_DIR/initrd.xz" && initrd_filename="initrd.xz"
test "$initrd_filename" || Error "Neither initrd.cgz nor initrd.xz found"

and then using $initrd_filename everywhere.

I will do a pull request to make it more clear
what I propose to change...

jsmeix commented at 2017-01-26 11:59:¶

I fear my "automatically do the right thing" attempt
is not sufficiently fail-safe in practice because
the ReaR initrd is loaded by possibly various
bootloaders (normally syslinux - but I assume
any bootloader blindly loads any initrd file)
but currently I do not know if a xz-lzma compressed
initrd will always work, in particular with older kernels
that may not support a xz-lzma compressed initrd?

In other words:
To "automatically do the right thing" when
compressing the initrd does not imply that
later that initrd can be used (decomressed)
by whatever kernel that will be booted later.

abbahag commented at 2017-01-26 13:42:¶

The solution with xz-lzma is necessary for me to boot under IBM Power8
SLES 11 with a HANA installation.
Unfortunately yaboot only has a maximum size of 32 MB. I have the initrd
with cpio / gz not get below the limit of 32 MB.
I found the idea good and flexible with an option in the local.conf to
determine whether one would want a different compression than the
standard.

jsmeix commented at 2017-01-26 13:59:¶

I am currently testing
https://github.com/rear/rear/pull/1182

I noticed a big disadvantage of xz-lzma:
I takes "ages" while xz runs at 99% CPU usage
(the remaining 1% CPU is used by cpio).
From the "rear -d -D mkbackup" log file:

2017-01-26 14:34:42.321885598 Creating recovery/rescue system initramfs/initrd
...
2017-01-26 14:36:37.937206502 Created initrd.xz with xz-lzma compression (91401572 bytes)

I.e. 116 seconds (almost two minutes)
to create initrd.xz (about 87 MB)

In contrast with 'gzip':

2017-01-26 14:51:32.230285525 Creating recovery/rescue system initramfs/initrd
...
2017-01-26 14:51:48.339397663 Created initrd.cgz with gzip compression (149199959 bytes)

I.e. 16 seconds (more than 7 times faster as xz-lzma) to create
initrd.cgz (about 142 MB about 1.6 times bigger than xz-lzma).

This matters because "rear mkbackup" is meant to run
while the system is in use - probably not while the system
runs under full load - nevertheless we should try to avoid
stuff that needs much ressources unless it is really needed.

Accordingly I will change my current implementation
from "automatically do the right thing"
to use by default gzip and have a config variable
so that the user can if needed specify xz-lzma.

This way it is even safe against regerssions because
I do not change the current behaviour by default.

jsmeix commented at 2017-01-26 14:00:¶

@abbahag
I know what you had already reported.
I am only thinking about the right way how to implement it.

jsmeix commented at 2017-01-26 15:09:¶

In https://github.com/rear/rear/pull/1182
I implemented now the new config variable
REAR_INITRD_COMPRESSION
so that the user can if needed specify
the compression as described in default.conf:

# REAR_INITRD_COMPRESSION
#
# What compression to use when creating the initramfs/initrd for Relax-and-Recover.
# By default and also as fallback an initrd.cgz with gzip default compression is created.
# The default is known to work well in practice.
# The ReaR initrd is loaded by possibly various bootloaders (normally syslinux).
# Usually any bootloader should be able to load any initrd file but then
# the initrd must be usable by the kernel which means a specially compressed initrd
# may not always work, in particular not with older kernels.
# With REAR_INITRD_COMPRESSION="fast"
# an initrd.cgz with gzip --fast compression is created (fast speed but less compression).
# With REAR_INITRD_COMPRESSION="best"
# an initrd.cgz with gzip --best compression is created (best compression but slow speed).
# With REAR_INITRD_COMPRESSION="lzma"
# an initrd.xz with xz using the lzma compression is created
# (see https://github.com/rear/rear/issues/1142).
REAR_INITRD_COMPRESSION=""

Results on my SLES12 test system

"rear mkrescue" with the default:

Created initrd.cgz with gzip default compression (149198234 bytes)

about 142 MiB.

"rear mkrescue" with REAR_INITRD_COMPRESSION="fast"

Created initrd.cgz with gzip fast compression (159204253 bytes)

about 152 MiB.

"rear mkrescue" with REAR_INITRD_COMPRESSION="best"

Created initrd.cgz with gzip best compression (148502717 bytes)

about 142 MiB.

"rear mkrescue" with REAR_INITRD_COMPRESSION="lzma"

Created initrd.xz with xz lzma compression (91405354 bytes)

about 87 MiB.

I found an advantage of using a high compression:
The smaller initrd is loaded noticeable faster by the bootloader.

This matters also because system recovery time is very
important to get back the system as fast as possible.

ReaR is known to be fast so that this way we could
make it even a bit faster!

jsmeix commented at 2017-01-27 12:03:¶

With https://github.com/rear/rear/pull/1182
merged I consider this issue to be fixed.

@abbahag
Please try out the newest rear GitHub master code
when you set in local.conf

REAR_INITRD_COMPRESSION="lzma"

and provide feedback whether or not
that works for you.

How to test the currently newest rear GitHub master code
independent of an already installed ReaR software:

Basically "git clone" it into a directory and then
configre and run it from within that directory like:

# git clone https://github.com/rear/rear.git

# cd rear

# vi etc/rear/local.conf

# usr/sbin/rear -d -D mkbackup

(note the relative paths "etc/rear/" and "usr/sbin/").

abbahag commented at 2017-01-27 16:43:¶

@jsmeix Option is working fine in my environment.

jsmeix commented at 2017-01-30 08:56:¶

@abbahag
many thanks for your prompt feedback!
It helps us a lot when if we get prompt feedback
whether or not things also work o.k. for others.

jsmeix commented at 2017-01-30 09:30:¶

I have a general question about ReaR's initrd size:

On my x86_64 system (using BIOS) the ReaR initrd.xz
with xz lzma compression has a size of 87 MiB, see
https://github.com/rear/rear/issues/1142#issuecomment-275412073

In contrast
https://github.com/rear/rear/issues/1142#issue-197976731
reads that on ppc64 with yaboot only 32MiB are possible
and that on ppc64 "With gzip, the initrd is 34.1 MB
with xz and lzma 21.4 MB".

I think independent of the architecture the ReaR initrd should
cointain basically the same set of files because I assume
the ReaR rescue/recovery system on x86_64 and on ppc64
should be basically the same.

Therefore I wonder why the ReaR initrd.xz on ppc64
is less than half of the size compared to x86_64.

Can someone explain to me what the reason
for this big difference is?

FYI:
On my x86_64 system (using BIOS) my ReaR ROOTFS_DIR
/tmp/rear.BBbYtTeC5sJIVFt/rootfs (i.e. the uncompressed
content of the ReaR initrd) has a disk usage of 338M:

e205:/tmp/rear.BBbYtTeC5sJIVFt/rootfs # du -shc *
37M     bin
7.5M    boot
132K    dev
9.2M    etc
0       init
196M    lib
9.5M    lib64
20K     mnt
8.0K    proc
5.2M    root
28K     run
0       sbin
8.0K    selinux
8.0K    sys
1.4M    tmp
72M     usr
136K    var
338M    total

e205:/tmp/rear.BBbYtTeC5sJIVFt/rootfs # du -shc lib/*
143M    lib/firmware
54M     lib/modules
0       lib/udev
196M    total

It seems more than the half of ReaR's initrd
on my x86_64 system is used for firmware.

Perhaps on ppc64 there is much less firmware?

schabrolles commented at 2017-01-30 11:53:¶

Hi @jsmeix,
I think it is because /lib/firmware is not included in the initrd when the Linux OS is running Virtualized under PowerVM (which is always the case in ppc64 arch, but not in ppc64le).

Linux-ppc64.conf file contain the following variable:

COPY_AS_IS_EXCLUDE=( ${COPY_AS_IS_EXCLUDE[@]} /lib*/firmware )

jsmeix commented at 2017-01-30 12:29:¶

@schabrolles
many thanks for the information!
Now it is clear.
Those commits implemeted it for ppc64 and ppc64le:
https://github.com/rear/rear/commit/7ca70cb3675c3d8d93a3d72ad34f3786364b9ec1
and
https://github.com/rear/rear/commit/5d4af5137155b052ec80f326f98e095d65059119
both from Masanori Mitsugi mitsugi@vnet.linux.ibm.com
and then
https://github.com/rear/rear/commit/89ddb9fc17adb022ce9c10be0c3e5b835ba139d7
from Sebastien Chabrolles s.chabrolles@fr.ibm.com
where in particular your explanatory commit log messages
makes things clear.

I assume on ppc64le in BareMetal Mode (PowerNV)
it does not work with yaboot.

ProBackup-nl commented at 2017-02-28 22:05:¶

Sharing my results using Arch Linux 4.9.11-1 x86_64:
Created initrd.cgz with gzip default compression (156865102 bytes)
Created initrd.xz with xz lzma compression (90740089 bytes)

Now it's time to clean up /usr/lib/firmware.

jsmeix commented at 2017-03-01 08:57:¶

@ProBackup-nl
many thanks for sharing your results on your particular system.
Such information helps a lot to get a better overview
and better general understanding how things are
on various Linux distributions.

ProBackup-nl commented at 2017-03-01 12:15:¶

For the bare minimum Arch Linux system installers, cleaning up /usr/lib/modules might not be such a good idea. It will generate errors when rebuilding your initramfs in the fallback section /etc/mkinitcpio.d/linux.preset: 'fallback'.

After removing /usr/lib/firmware contents except for /usb/lib/firmware/rtl_nic (# find /usr/lib/firmware/* -maxdepth 0 -name 'rtl_nic' -prune -o -exec rm -rf '{}' ';'), I ended up with:

Created initrd.xz with xz lzma compression (48327540 bytes)

After removing a load of /usr/lib/modules contents, like bluetooth, pata, pcmcia, infiniband, parport, Wi-Fi, snd, wimax, etcetera, 83M /usr/lib/modules shrank to 48M, resulting in:

Created initrd.xz with xz lzma compression (35513549 bytes)

PS The beauty of Arch Linux is that you can prevent /usr/lib/firmware re-creation upon firmware package upgrades by adding a line to /etc/pacman.conf:

NoExtract = usr/lib/firmware/* !usr/lib/firmware/rtl_nic/*

ProBackup-nl commented at 2017-03-01 15:01:¶

@jsmeix Is it really necessary to create an initrd file when the kernel of Arch Linux runs with a 6M/11M initramfs (instead of 35M/150M generated initrd)?

# ls -lAh /boot  | grep '^-'
-rwxr-xr-x 1 root root  11M Mar  1 12:50 initramfs-linux-fallback.img
-rwxr-xr-x 1 root root 5.6M Mar  1 12:50 initramfs-linux.img
-rwxr-xr-x 1 root root 961K Nov 15 18:43 intel-ucode.img
-rwxr-xr-x 1 root root 4.7M Feb 19 14:49 vmlinuz-linux

jsmeix commented at 2017-03-02 08:28:¶

@ProBackup-nl
I think you misunderstand something here.
ReaR's own initrd that is created here contains first and foremost
the whole ReaR recovery system, cf. my above
https://github.com/rear/rear/issues/1142#issuecomment-276015345
To inspect what during "rear mkrescue/mkbackup" ReaR's initrd
contains set KEEP_BUILD_DIR="yes" in etc/rear/local.conf
and then after "rear mkrescue/mkbackup" check what
/tmp/rear.XXXXXXXXXXXXXXX/rootfs
contains. In particular it contains all the ReaR scripts plus all
binaries (like parted, mkfs.*, mount, tar, ...) and all other files
(like kernel modules, systemd stuff, system startup scripts, ...)
that are needed so that "rear recover" can run sucessfully
when it is run in the ReaR recovery system.

jsmeix commented at 2017-03-03 14:44:¶

@ProBackup-nl
regarding "firmware" in your above
https://github.com/rear/rear/issues/1142#issuecomment-283324917
my new enhancement idea in
https://github.com/rear/rear/issues/1216
is probably of interest for you.

jsmeix commented at 2017-03-03 14:50:¶

@schabrolles
regarding my above
https://github.com/rear/rear/issues/1142#issuecomment-276050900

I assume on ppc64le in BareMetal Mode (PowerNV)
it does not work with yaboot.

see in my new enhancement idea in particular
https://github.com/rear/rear/issues/1216#issuecomment-283972392

schabrolles commented at 2017-03-03 14:56:¶

Yes, you are right.

In BareMetal mode (PowerNV), only recent Linux distro are supported (rhel7.2+ and ubuntu16.04)

But we can have older distro in PowerVM or PowerKVM (sles11, rhel6) => They still uses yaboot/lilo

ProBackup-nl commented at 2017-03-05 15:19:¶

@jsmeix After the modules and firmware cleanup in my x86-64 UEFI booted Arch Linux, these (manual) folder traverse size sorted listings of ROOTFS are potential suspects for further size reduction.

Additionally, for the cases where you don't need networking to recover (as is the goal of my USB setup), removing all network related stuff (sshd, ssh, ip, rsync, kernel/net, kernel/drivers/net) could save a considerable amount of space and time to create the initrd.

[rootfs]
98M usr
18M bin
9.5M    etc
3.3M    root
…
91M usr/lib
7.1M    usr/share
…
26M usr/lib/libicudata.so.58.2
14M usr/lib/udev
11M usr/lib/modules
11M usr/lib/libstdc++.so.6.0.22
4.7M    usr/lib/systemd
…
8.1M    usr/lib/udev/hwdb.bin
5.6M    usr/lib/udev/hwdb.d
…
9.7M    usr/lib/modules/4.9.11-1-ARCH/kernel
200K    usr/lib/modules/4.9.11-1-ARCH/modules.alias
196K    usr/lib/modules/4.9.11-1-ARCH/modules.alias.bin
…
7.1M    usr/lib/modules/4.9.11-1-ARCH/kernel/drivers
1.2M    usr/lib/modules/4.9.11-1-ARCH/kernel/fs
576K    usr/lib/modules/4.9.11-1-ARCH/kernel/crypto
400K    usr/lib/modules/4.9.11-1-ARCH/kernel/arch
352K    usr/lib/modules/4.9.11-1-ARCH/kernel/net
128K    usr/lib/modules/4.9.11-1-ARCH/kernel/lib
…
5.7M    usr/lib/modules/4.9.11-1-ARCH/kernel/drivers/net
484K    usr/lib/modules/4.9.11-1-ARCH/kernel/drivers/usb
328K    usr/lib/modules/4.9.11-1-ARCH/kernel/drivers/ata
152K    usr/lib/modules/4.9.11-1-ARCH/kernel/drivers/input
…
2.1M    usr/lib/systemd/libsystemd-shared-232.so
448K    usr/lib/systemd/systemd-udevd
…
6.7M    usr/share/terminfo
412K    usr/share/systemd
…
1.1M    bin/init
864K    bin/ldconfig
812K    bin/bash
752K    bin/sshd
700K    bin/ssh
684K    bin/btrfsck
684K    bin/btrfs
628K    bin/awk
456K    bin/ip
452K    bin/rsync
…
8.1M    etc/udev
552K    etc/ssh
…
8.1M    etc/udev/hwdb.bin
4.0K    etc/udev/udev.conf
…
3.2M    root/rear

[Export of Github issue for rear/rear.]

#1142 Issue closed: use better compression for initrd which must be less than 32MB when booting on ppc64 via yaboot¶