#755 Issue `closed`: rear does not work for very minimal systems¶

Labels: enhancement, documentation, discuss / RFC, won't fix / can't fix / obsolete

jsmeix opened issue at 2016-01-14 10:17:¶

This issue is primarily intended to document and discuss
a very special use case.
This issue is no bug report.
It might become a valid enhancement request for the future.

Some users strip down their systems to only
what is really needed to run the system.

On such systems a lot of "usually installed" stuff
that is needed by rear could be missing.

In particular all what is only needed to install the system
but not to run it.
For example what is needed for disk partitioning (e.g. parted)
and for creating filesystems (e.g. mkfs.*) could be missing.

Or "usually installed" standard tools that are not needed
for a particular very specific use case.

E.g. from @mbrookhuis I got a report about a system
where 'less' is not installed.

To find out what programs rear usually needs,
inspect the files in /usr/share/rear/conf/
for the contents of the variables REQUIRED_PROGS
and PROGS and for documentation
see /usr/share/rear/conf/default.conf

Currently a lot of needed programs are defined via PROGS
where /usr/share/rear/conf/default.conf reads:

These progs are optional, if they are missing, nothing happens.

The "nothing happens" is not exactly true because
if they are missing they are silently not included
in the ISO image of the recovery system but
if they are actually called in the recovery system
then an error happens during recovery.

As a first step it would be nice to avoid in advance
such errors during recovery.

My fist idea is moving more of them from PROGS
to REQUIRED_PROGS so that the user gets
an early error during "rear mkbackup".

But in practice this is complicated because e.g.
on nowadays systems mkfs.btrfs is usually needed
but on older systems that may not exist.

A better approach would be to replace step by step
the current static lists of PROGS and REQUIRED_PROGS
by generated lists during "rear mkbackup" depending
on what is actually needed to recreate a particular system.

E.g. when a btrfs filesystem is to be recreated
then mkfs.btrfs is added to REQUIRED_PROGS
otherwise it is left as is in PROGS.

This approach would be fully backward compatible.

But this approach would not really help on very minimal systems
where neither parted nor mkfs.* is installed.

One way for such very minimal systems is to install them
as usual (i.e. with all usual tools), then let rear create
its recovery system ISO image and afterwards remove
what is not needed to run the system.

For the future an alternative and more generic way
for very minimal systems could be a prebuilt static
rear recovery system that is relatively big because
it contains basically everything that is ever needed
to recreate any system, cf.
"generic 'static/fixed' rear installation system" in
https://github.com/rear/rear/issues/732#issuecomment-163196707

mbrookhuis commented at 2016-01-14 10:35:¶

I agree with @jsmeix
The problem is that there is an error and you get an option to read the file and then get the error.

Some remarks:

I suspect that when a progs is in REQUIRED_PROGS the mkrescue will fail. This is OK when some really needed programs are missing. eg mkfs.* in the example.
for less, I would suggest that we check if less is there, and if not change to the use of: cat |more. These are part of bash and should always be present.

jsmeix commented at 2016-01-14 10:36:¶

Only as a side note - perhaps I detected a bug:
In /etc/rear/local.conf I have

REQUIRED_PROGS=( "${REQUIRED_PROGS[@]}" qqq )

but I got no error during "rear mkrescue"

# rear -d -D mkrescue && echo OK || echo FAIL 
Relax-and-Recover 1.17.2 / Git
Using log file: /var/log/rear/rear-d186.log
Creating disk layout
Creating root filesystem layout
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Creating initramfs
Making ISO image
Wrote ISO image: /var/lib/rear/output/rear-d186.iso (107M)
Copying resulting files to nfs location
You should also rm -Rf /tmp/rear.QZaXSbSbyMWDGyz
OK
# grep qqq /var/log/rear/rear-d186.log
++ REQUIRED_PROGS=("${REQUIRED_PROGS[@]}" qqq)
++ has_binary qqq
++ type qqq
++++ get_path qqq
++++ type -P qqq
# type -P qqq && echo OK || echo FAIL
FAIL

jsmeix commented at 2016-01-14 10:40:¶

@gdha
what is the intended behaviour when someting
in REQUIRED_PROGS is missing?

usr/share/rear/conf/default.conf reads

# program files (find them in the path). These progs are optional,
# if they are missing, nothing happens
PROGS=( )
# required programs. Same as above, but if they are missing, we abort.
REQUIRED_PROGS=(

Under what exact circumstances should rear abort
if something in REQUIRED_PROGS is missing?

BobBagwill commented at 2016-02-10 15:13:¶

Perhaps a custom busybox would be a solution.

schlomo commented at 2016-02-10 16:11:¶

Careful! Most of our code relies on advanced features of the GNU utilities. Busybox with its reduced featureset will definitively break a lot of code.

I think the really actionable item from this issue is cleaning up the PROGS / REQUIRED_PROGS relationship. And making it fail if a REQUIRED_PROGS entry is missing.

Maybe we can automatically scan rear code during build time to fill these variables?

gdha commented at 2016-02-10 17:03:¶

We should start by defining minimal system?
When rear detects an executable is missing in array REQUIRED_PROGS it will fail with an error.
OTOH lots of executables listed in PROGS are assumed to be available by default (part of core GNU/Linux system). This could lead to migrating lots of executables listed in PROGS to REQUIRED_PROGS.

jsmeix commented at 2016-02-11 10:57:¶

@gdha
see my above
https://github.com/rear/rear/issues/755#issuecomment-171603580
and
https://github.com/rear/rear/issues/755#issuecomment-171607112

For me "rear mkrescue" did not fail with an error.

Under what exact circumstances should rear "fail with an error"
if something in REQUIRED_PROGS is missing?

In general regarding
https://github.com/rear/rear/issues/755#issuecomment-171603294
"check if a standard tool is there, and if not work around it":

Theoretically it is doable but in practice nobody does it
i.e. nobody implements that consistently everywhere
so that one could really rely on it.

Therefore I think leit's Keep It Simple and Straightforward
(cf. https://en.wikipedia.org/wiki/KISS_principle):

Move everything where usual system admins assume
it is available from PROGS to REQUIRED_PROGS
and let "rear mkrecsue"/"rear mkbackup" fail if something
in REQUIRED_PROGS is actually not available.

This way running "rear mkrecsue"/"rear mkbackup" on
very minimal systems would fail which tells the user/admin
about the issue.

jsmeix commented at 2016-02-11 11:03:¶

@schlomo regarding
"automatically scan rear code for required programs":

I had the same issue some time ago when I liked to specify
RPM package requirements for what is needed by rear.

As far as I understand it the basic problem is to scan bash scripts
for what programs are called.

I searched the Internet a lot to find out if that is possible
(to some reasonable extent) but I failed to find a solution.
I only found comments why it is not possible.
At that time I gave up.

jsmeix commented at 2018-11-14 09:24:¶

Since https://github.com/rear/rear/issues/892 is done
via https://github.com/rear/rear/pull/1961
and https://github.com/rear/rear/pull/1962
"rear mkrescue/mkbackup" should reliably error out when
a program in REQUIRED_PROGS is missing in the recovery system.

But this is not sufficient because specific mandatrory programs
get not added to REQUIRED_PROGS, see
https://github.com/rear/rear/issues/1963

[Export of Github issue for rear/rear.]

#755 Issue closed: rear does not work for very minimal systems¶