#1218 Issue `closed`: Add lz4 initrd compression support for even faster mkrescue¶

Labels: enhancement, fixed / solved / done, minor bug

ProBackup-nl opened issue at 2017-03-06 00:22:¶

Even with REAR_INITRD_COMPRESSION="fast", rear mkrescue does spend significant time in compressing rootfs into a compressed initrd file. After inserting some bash timer code in 900_create_initramfs.sh, it seems that the creation of the 51M file takes 9 seconds.

Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip fast compression
Created initrd.cgz with gzip fast compression (53040694 bytes) in 9 seconds

I'd wish support for the even faster lz4 codec, which seems understood by the Arch Linux kernel by default according to:

$ cat /etc/mkinitcpio.conf | grep COMPRESSION=
#COMPRESSION="gzip"
#COMPRESSION="bzip2"
#COMPRESSION="lzma"
#COMPRESSION="xz"
#COMPRESSION="lzop"
#COMPRESSION="lz4"

ProBackup-nl commented at 2017-03-06 00:33:¶

Apparently I did something wrong. The commit that was intended for a new pull request was merged with my earlier pull request. I don't know (yet) how to fix this. My uneducated guess is that I need to make a git branch somewhere in the process.

Note: with REAR_INITRD_COMPRESSION="lz4" the 64M initrd file creation is now done in 4 till 5 seconds:

Creating recovery/rescue system initramfs/initrd initrd.lz4 with lz4 compression
Created initrd.lz4 with lz4 compression (66863824 bytes) in 4 seconds
...
Creating recovery/rescue system initramfs/initrd initrd.lz4 with lz4 compression
Created initrd.lz4 with lz4 compression (66864362 bytes) in 5 seconds

I can't test whether it boots correctly because I have an issue with:

BUG in /root/rear/usr/share/rear/output/USB/Linux-i386/100_create_efiboot.sh:
'Unknown EFI bootloader'

jsmeix commented at 2017-03-06 10:19:¶

@ProBackup-nl
the way how I do GitHub pull requests is basically
(on my local git copy):

$ git checkout master

$ git checkout -b implement_new_lz4_initrd_compression_issue1218
Switched to a new branch 'implement_new_lz4_initrd_compression_issue1218'

code hacking

$ git commit -am "implemented new lz4 initrd compression (issue 1218)"

$ git push origin implement_new_lz4_initrd_compression_issue1218

Now my new branch
'implement_new_lz4_initrd_compression_issue1218'
exists on GitHub.
Then I use the GitHub web interface
to create a new pull request from that new branch
'implement_new_lz4_initrd_compression_issue1218'
to the current ReaR upstream repository.

In general see
http://relax-and-recover.org/development/

jsmeix commented at 2017-03-06 10:24:¶

@ProBackup-nl
assume you created a new pull request from your branch
and now I request changes, then you can push more commits
to that same branch and all your commits get automatically
included in your existing pull request of that branch, e.g.:
(on yor local git copy):

$ git checkout master

$ git checkout implement_new_lz4_initrd_compression_issue1218
Switched to branch 'implement_new_lz4_initrd_compression_issue1218'

further code hacking

$ git commit -am "improved lz4 initrd compression (issue 1218)"

$ git push origin implement_new_lz4_initrd_compression_issue1218

jsmeix commented at 2017-03-06 10:53:¶

With
https://github.com/rear/rear/pull/1217
merged, this issue should be fixed.

ProBackup-nl commented at 2017-03-10 23:19:¶

There is a bug in the code for this lz4 initramfs feature. When the initramfs of Arch Linux itself is build using mkinitcpio using lz4 compression, the kernel will boot ok.
REAR_INITRD_COMPRESSION="fast" boots OK too.
REAR_INITRD_COMPRESSION="lz4" results in error:

initramfs unpacking failed: junk in compressed archive

@jsmeix Should this issue be re-opened or a new issue created?

ProBackup-nl commented at 2017-03-10 23:30:¶

$ lsinitcpio -a /tmp/sdb1/initrd.cgz
/usr/bin/lsinitcpio: line 78: warning: command substitution: ignored null byte in input
/usr/bin/lsinitcpio: line 90: warning: command substitution: ignored null byte in input
/usr/bin/lsinitcpio: line 175: /tmp/lsinitcpio.Ru0WEp/VERSION: No such file or directory
/usr/bin/lsinitcpio: line 178: /tmp/lsinitcpio.Ru0WEp/config: No such file or directory
==> Image: /tmp/sdb1/initrd.cgz
==> Kernel: 4.9.11-1-ARCH
==> Size: 50.6 MiB
==> Compressed with: gzip
  -> Uncompressed size: 120.06 MiB (.421 ratio)
  -> Estimated extraction time: 3.126s

==> Included binaries:

$ lsinitcpio -a /tmp/sdb1/initrd.lz4
==> ERROR: Newer lz4 stream format detected! This may not boot!
/usr/bin/lsinitcpio: line 175: /tmp/lsinitcpio.RbVERF/VERSION: No such file or directory
/usr/bin/lsinitcpio: line 178: /tmp/lsinitcpio.RbVERF/config: No such file or directory
==> Image: /tmp/sdb1/initrd.lz4
==> Kernel: 4.9.11-1-ARCH
==> Size: 63.78 MiB
==> Compressed with: lz4
  -> Uncompressed size: 120.06 MiB (.531 ratio)
  -> Estimated extraction time: 3.503s

==> Included binaries:

$ lsinitcpio -a /boot/initramfs-linux.img
==> Image: /boot/initramfs-linux.img
==> Created with mkinitcpio 22
==> Kernel: 4.9.11-1-ARCH
==> Size: 6.41 MiB
==> Compressed with: lz4 -l
  -> Uncompressed size: 14.4 MiB (.445 ratio)
  -> Estimated extraction time: 0.248s
...

$ lz4 --help
...
 -l     : compress using Legacy format (Linux kernel compression)
...

ProBackup-nl commented at 2017-03-10 23:56:¶

Adding Linux legacy `-l` to lz4 results¶

Size increased from 66864362 to 66871053: negligible factor 1.0001
Estimated extraction time reduced from 3.503s to 0.239s: 14.6 times faster
Creating the compressed initramfs file is still in the 4-5 seconds range
The lsinitcpio "ERROR: Newer lz4 stream format detected! This may not boot!" is gone

$ lsinitcpio -a /tmp/sdb1/initrd.lz4
/usr/bin/lsinitcpio: line 175: /tmp/lsinitcpio.22JAqI/VERSION: No such file or directory
/usr/bin/lsinitcpio: line 178: /tmp/lsinitcpio.22JAqI/config: No such file or directory
==> Image: /tmp/sdb1/initrd.lz4
==> Kernel: 4.9.11-1-ARCH
==> Size: 63.77 MiB
==> Compressed with: lz4 -l
  -> Uncompressed size: 120.06 MiB (.531 ratio)
  -> Estimated extraction time: 0.239s

==> Included binaries:

And last but not least: it boots to ReaR

jsmeix commented at 2017-03-13 09:41:¶

Reopening because it needs some enhancement, see
https://github.com/rear/rear/issues/1218#issuecomment-285817529
and
https://github.com/rear/rear/pull/1235
and
https://github.com/rear/rear/pull/1236

ProBackup-nl commented at 2017-03-14 16:43:¶

The strange thing here is that Arch Linux on the source system boots even with an lz4 -9 compressed initrd file. No need for lz4 -l on the source system.

jsmeix commented at 2017-03-16 10:07:¶

With
https://github.com/rear/rear/pull/1235
and
https://github.com/rear/rear/pull/1236
merged
this issue schould be again fixed/solved/done
(and if not we can reopen it again).

jsmeix commented at 2017-03-16 10:39:¶

@ProBackup-nl
regardless that I am not at all a Linux booting expert
here an offhanded vague idea regarding why
an lz4 -9 compressed initrd file works on the original system
but not in the ReaR recovery system:

As far as I understand it
it is the kernel that uncompresses the initrd
and the kernel must do that on its own "as is"
without any additional stuff like kernel modules
because as long as the contents of the initrd are not
accessible, the kernel cannot load a module.

Because the ReaR recovery system uses the exact same kernel
as the original system (provided the KERNEL_FILE value is really
the kernel that runs in the original system - check that if several
kernel files exist in /boot/) the difference that lets it fail versus
that makes it work cannot be in the kernel.

Because at the time when the kernel uncompresses the initrd
there is only the kernel and the initrd, the difference
that lets it fail versus that makes it work
must be in the initrd of the original system
versus the initrd of the ReaR recovery system.

That the difference must be in the initrds matches the fact
that the initrd of the original system is created differently
compared to the initrd of the ReaR recovery system.

A colleague told me that he thinks to remember in the past
for special initrd compressions one had to do special
additional stuff with the initrd to ensure the kernel
could later successfuly uncompress it - he was vaguely
talking about manually adjusting magic numbers at the
beginning of the initrd to help the kernel to correctly
autodetect the actual initrd compression or things like that.

[Export of Github issue for rear/rear.]

#1218 Issue closed: Add lz4 initrd compression support for even faster mkrescue¶