#2525 PR `merged`: Error out when files greater or equal ISO_FILE_SIZE_LIMIT should be included in the ISO¶

Labels: enhancement, fixed / solved / done, critical / security / legal

jsmeix opened issue at 2020-11-24 15:30:¶

Type: Bug Fix / Enhancement
Impact: Low / Critical

Low impact (actually there should be no impact) under normal circumstances.

In contrast critical when the backup is included in the ISO with BACKUP_URL=iso://
where the backup archive file (e.g. backup.tar.gz) could be greater or equal 2GiB.
When there is a 2GiB file size limit a backup.tar.gz that is greater than 2GiB
will get corrupted in the ISO so backup restore via "rear recover" would fail
but then it is too late and the backup in the ISO is unusable.

Therefore such a test is critical because without such a test
for the user it appears af if everything went OK during "rear mkbackup"
until he finds out later during "rear recover" that his backup got corrupted.

Reference to related issue (URL):
https://github.com/gozora/ebiso/issues/12
How was this pull request tested?
"rear mkrescue" still works well for me.
I did not yes test with a file greater or equal 2GiB for the ISO.
Brief description of the changes in this pull request:

There is a limit of the ISO 9660 file system that is 2GiB or 4GiB according to
https://en.wikipedia.org/wiki/ISO_9660#The_2/4_GiB_file_size_limit

To be on the safe side we use by default a 2GiB limit.
2GiB is at least the actual limit when ebiso is used, cf.
https://github.com/gozora/ebiso/issues/12

Under normal circumstances files greater or equal 2GiB should not appear in the ISO.

An exception is when the backup is included in the ISO with BACKUP_URL=iso://
where the backup archive file (e.g. backup.tar.gz) could be greater or equal 2GiB.

The user might adapt ISO_FILE_SIZE_LIMIT provided he verified that "rear recover"
actually works in his particular environment even when there are files in his ISO
(in particular backup.tar.gz) that are actually greater than the default 2GiB limit.
When there is a 2GiB file size limit a backup.tar.gz that is greater than 2GiB
will get corrupted in the ISO so backup restore via "rear recover" would fail.

Also the ReaR recovery system initrd could become greater or equal 2GiB
(e.g. because of accidentally too much in COPY_AS_IS) which is very likely an error
so we error out when files greater or equal ISO_FILE_SIZE_LIMIT should be included in the ISO.

jsmeix commented at 2020-11-24 15:38:¶

There are various scripts that make an ISO image
so this test should probably become a function.

The scripts that are suspect are

# find usr/sbin/rear usr/share/rear/ -type f | xargs grep -l 'ISO_MKISOFS_BIN' | grep -v '.conf$'

usr/share/rear/output/ISO/Linux-i386/800_create_isofs.sh
usr/share/rear/output/ISO/Linux-i386/810_prepare_multiple_iso.sh
usr/share/rear/output/ISO/Linux-i386/820_create_iso_image.sh
usr/share/rear/output/ISO/Linux-i386/830_create_iso_image_EFISTUB.sh
usr/share/rear/output/ISO/Linux-i386/250_populate_efibootimg.sh
usr/share/rear/output/ISO/Linux-ia64/800_create_isofs.sh
usr/share/rear/output/ISO/Linux-ppc64le/800_create_isofs.sh
usr/share/rear/output/ISO/Linux-ppc64le/820_create_iso_image.sh
usr/share/rear/prep/ISO/GNU/Linux/320_verify_mkisofs.sh

and from those scripts the most suspect are

# for f in $( find usr/sbin/rear usr/share/rear/ -type f | xargs grep -l 'ISO_MKISOFS_BIN' | grep -v '.conf$' ) ; \
 do grep -H 'pushd ' $f ; done

usr/share/rear/output/ISO/Linux-i386/810_prepare_multiple_iso.sh:    pushd $TEMP_ISO_DIR 1>/dev/null
usr/share/rear/output/ISO/Linux-i386/820_create_iso_image.sh:pushd $TMP_DIR/isofs >/dev/null
usr/share/rear/output/ISO/Linux-i386/830_create_iso_image_EFISTUB.sh:pushd $TMP_DIR/isofs >/dev/null
usr/share/rear/output/ISO/Linux-ia64/800_create_isofs.sh:pushd $TMP_DIR/isofs # so that relative paths will work
usr/share/rear/output/ISO/Linux-ppc64le/820_create_iso_image.sh:pushd $TMP_DIR

I will check them all...

jsmeix commented at 2020-11-24 15:57:¶

With artificial too small ISO_FILE_SIZE_LIMIT=12345 in etc/rear/local.conf
"rear -v mkresue" fails for me with (excerpt)

Making ISO image
ERROR: File for ISO chain.c32 size 20192 greater or equal ISO_FILE_SIZE_LIMIT=12345

jsmeix commented at 2020-11-25 15:23:¶

It seems with /usr/bin/mkisofs
at least the one on SLES12-SP5 from the cdrkit-cdrtools-compat RPM
it works for files greater 2GiB in the ISO, see
https://github.com/gozora/ebiso/issues/12#issuecomment-733766152

Perhaps with /usr/bin/mkisofs another ISO file size limit appears
later (e.g. 4GiB or something like that).

I think it is good to have a default ISO_FILE_SIZE_LIMIT of 2 GiB in ReaR
so users who exceed that limit in exceptional cases get a hard info
to have a closer look whether or not things work for them
with files greater or equal 2GiB in their ISO

jsmeix commented at 2020-11-26 11:50:¶

According to my reasoning in default.conf
why the default ISO_FILE_SIZE_LIMIT is 2GiB and why we error out
when files greater or equal ISO_FILE_SIZE_LIMIT should be included in the ISO
I think meanwhile
it is really needed to have a default ISO_FILE_SIZE_LIMIT of 2 GiB
so users who exceed that limit in exceptional cases get a hard info
to thoroughly test and verify whether or not things actually work for them
in their particular environment with files greater or equal 2GiB in their ISO.

@rear/contributors
to get user feedback from users who use current GitHub master code
I would like to merge this pull request early (i.e. tomorrow afternoon)
if there are no objections.

jsmeix commented at 2020-11-27 12:28:¶

Now things work well for me as far as I tested it
so I would merge it in about 30 minutes
unless there are objections.

jsmeix commented at 2020-11-27 14:00:¶

Now let's wait and see
how many users hit the default 2GiB ISO_FILE_SIZE_LIMIT
that results now an error exit like

ERROR: File for ISO backup.tar.gz size 2345678901 greater or equal ISO_FILE_SIZE_LIMIT=2147483648

[Export of Github issue for rear/rear.]

#2525 PR merged: Error out when files greater or equal ISO_FILE_SIZE_LIMIT should be included in the ISO¶