#1923 Issue `closed`: mkrescue doesn't contain valid .conf files when files are symlinks¶

Labels: bug, fixed / solved / done

rmetrich opened issue at 2018-10-08 14:42:¶

ReaR version ("/usr/sbin/rear -V"): All, including latest
OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"): RHEL7 and RHEL6
ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"): Default one
Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR): ALL
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device): ALL
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot): ALL
Storage (lokal disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe): ALL
Description of the issue (ideally so that others can reproduce it):

When not using the standard configuration files /etc/rear/*.conf but its own configuration directory, e.g. /etc/rear/myconf and having the conf files be symlinks, the configuration files are not included properly in the ISO image:

Create configuration¶

# mkdir /etc/rear/myconf
# touch /etc/rear/site.conf
# ln -s /etc/rear/os.conf /etc/rear/myconf/
# ln -s /etc/rear/site.conf /etc/rear/myconf/

Create ISO¶

# rear -dD mkrescue

Check ISO content (broken)¶

# ls -l /tmp/rear.*/rootfs/etc/rear
-rw-------. 1 root root  534 Oct  8 16:15 local.conf
lrwxrwxrwx. 1 root root   10 Oct  8 16:15 os.conf -> ../os.conf
-rw-r--r--. 1 root root 1039 Oct  8 16:15 rescue.conf
lrwxrwxrwx. 1 root root   12 Oct  8 16:15 site.conf -> ../site.conf

We have here, os.conf point to /etc/os.conf and site.conf point to non-existing file /etc/site.conf.

This is due to using cp $v -r command in /usr/share/rear/build/GNU/Linux/100_copy_as_is.sh on line 83:

 76 Log "Copying ReaR configuration directory"
 77 # Copy ReaR configuration directory:
 78 mkdir $v -p $ROOTFS_DIR/etc/rear
 79 # This will do same job as lines below.
 80 # On top of that, it does not throw log warning like:
 81 # "cp: missing destination file operand after"
 82 # if hidden file (.<filename>) is missing in $CONFIG_DIR
 83 cp $v -r $CONFIG_DIR/. $ROOTFS_DIR/etc/rear/ 1>&2
 84

With this cp, symlinks are kept instead of content copied. This ends up not having os.conf and site.conf be created.

Additionallly /etc/os.conf is only having the content generated by /usr/share/rear/build/default/990_update_os_conf.sh. This is due to having the symlink /etc/rear/os.conf point to non-existing /etc/os.conf, but that latter file gets populated by /usr/share/rear/build/default/990_update_os_conf.sh echoes.

Work-around, if any: Use hard links instead of soft links for the configuration files

rmetrich commented at 2018-10-08 14:43:¶

The cp $v -r should be replaced by cp $v -r -L to dereference links.

jsmeix commented at 2018-10-08 15:24:¶

@rmetrich
you may also have a look at
usr/share/rear/build/GNU/Linux/390_copy_binaries_libraries.sh
because there is already code that copies symlink targets
and all link target components.

FYI:
On my personal "secret future stuff" TODO list is
to add a check for dangling symlinks in the recovery system
that is run during "rear mkrescue/mkbackup" via
usr/share/rear/build/default/980_verify_rootfs.sh

jsmeix commented at 2018-10-08 15:41:¶

I wonder if cp -r -L is really "the right thing" because it replaces
what was originally a symbolic link by the content of the link target
so that afterwards that content may exist in two independent regular files
i.e. the meaning of the symlink (same conent under different file names)
was destroyed which might have unwanted bad consequences in some cases.

rmetrich commented at 2018-10-08 15:43:¶

Well, the copy is used to copy files from local system to recovery system (in the tmp build dir), so I don't see why this would be an issue.

schlomo commented at 2018-10-08 15:54:¶

@jsmeix I remember us introducing the -L to be sure not to have dangling symlinks via a "brute force" copy at the potential waste of some storage space. This was to solve various really stupid cases where OS files of some support packages where symlinks on some OS and regular files on others. update-alternatives is another candidate that can create symlinks at unexpected places.

👍 for the idea to also copy the ReaR config with -L

jsmeix commented at 2018-10-08 15:58:¶

Only an offhanded guess (I did not check the details):
One might get the same config file content then in two separated regular files
so that one would have to find and edit both of them in the recovery system
to get a consistent change (e.g. before one runs rear recover).

In general I think when cp -r is used the intent is to copy whole trees of files
and in this case I assume the usual intent is to keep the copied tree
as much as possible the same as the original tree was and I think this
is the reason why When copying from a symbolic link, cp normally follows the link only when not copying recursively cf.
https://www.gnu.org/software/coreutils/manual/html_node/cp-invocation.html

But perhaps I am overpedantic in this particular case here.

@schlomo
I do not mind a bit of wasted storage space.
I do mind sudden unexpected duplicates.

FYI:
Another one my personal "secret future stuff" TODO list is
to use (if available) fdupes in the recovery system
to find duplicate content files.

rmetrich commented at 2018-10-08 16:04:¶

For rear configuration files, there is no other option than copying dereferenced files, since rear modifies files specified with -c /path into /etc/rear forcibly.

jsmeix commented at 2018-10-09 07:05:¶

@rmetrich
as always thank you so much for your essential explanation in
https://github.com/rear/rear/issues/1923#issuecomment-427890470

Now even I understand why cp -r -L is "the right thing" in this particular case.

I fixed it directly via
https://github.com/rear/rear/commit/2a17522668105a1cf4448865f1b69135d32f049e

jsmeix commented at 2018-10-09 07:30:¶

By the way I was also looking at the additional config files
in $CONFIG_APPEND_FILES (specified via '-C') and I did a usability fix via
https://github.com/rear/rear/commit/839796e18f7c8825200dce93547ad3b673a8a8d0

jsmeix commented at 2018-10-09 08:10:¶

Now I even found the reason why I am in general cautious
when the idea is to "just copy symlink targets":

See
https://github.com/rear/rear/pull/1636#issuecomment-350258366
and the other comments related to it.

We must not "just copy symlink targets" into the recovery system
unless in exceptional cases where we know it won't hurt too much
(hopefully nobody has symlinks in $CONFIG_DIR that point to huge files ;-)

jsmeix commented at 2018-10-09 08:34:¶

And finally - so that it is documented here -
(probably a somewhat artificial example)
why copying symlink targets in $CONFIG_DIR
could lead to unexpected behaviour for the user:

Assume /etc/rear/local.conf is a symbolic link that
points to one of several predefined system-specific config
files like /etc/rear/host_foo.conf /etc/rear/host_bar.conf
/etc/rear/web_server.conf /etc/rear/mail_server.conf ...
(the user may like to have all of them on his systems
and let his local.conf symlink point to the right one).

In the recovery system that symlink is replaced by what its target
was during "rear mkrescue/mkbackup" so that the config files structure
is different in the recovery system which could be unexpected for the user
(regardless that "rear recover" should usually still work).

We can enhance copying config files when such kind of issues
actually get reported to us by real users.

At least for now the current cp -r -L should be "good enough".

[Export of Github issue for rear/rear.]

#1923 Issue closed: mkrescue doesn't contain valid .conf files when files are symlinks¶