#1946 Issue closed: System won't boot after restore - symlink for /etc/mtab missing¶
Labels: support / question, won't fix / can't fix / obsolete,
not ReaR / invalid
jonathankeuser opened issue at 2018-10-31 10:52:¶
I'm using rear-2.4-1 on RHEL7.4:
After a system restore my server won't boot anymore.
The reason for this is the following:
On RHEL7-Systems - also CentOS - the file "/etc/mtab" is a symlink to
"/proc/self/mount". At a restore this file will not be created, empty
created or with an unusable content created. So my server will hang up
while booting:
[FAILED] Failed to mount /. See 'systemctl status -.mount' for details.
After booting the rescue image again, create the symlink for /etc/mtab and boot the OS again, my server will boot successfully.
jsmeix commented at 2018-10-31 11:09:¶
See
https://raw.githubusercontent.com/rear/rear/master/.github/ISSUE_TEMPLATE.md
How things get restored depends on what you have in your backup,
cf. "Relax-and-Recover versus backup and restore" in
https://en.opensuse.org/SDB:Disaster_Recovery
Perhaps BACKUP_RESTORE_MOVE_AWAY_FILES
or POST_RECOVERY_SCRIPT may help in your particular
case, see usr/share/rear/conf/default.conf
gdha commented at 2018-10-31 11:16:¶
@jonathankeuser Did you check your rear log file for something like:
'/tmp/rear.g7ZIwz3UWiFCXP1/rootfs/etc/mtab' -> '/proc/self/mounts'?
I never had issues with RHEL 7 (or CenOS 7) so far with this.
gdha commented at 2018-10-31 12:21:¶
Is related to #1734
jonathankeuser commented at 2018-10-31 12:51:¶
@jsmeix I created a workaroud using POST_RECOVERY_SCRIPT. The variable BACKUP_RESTORE_MOVE_AWAY_FILES is empty. So it has no effect on this behavior.
@gdha In the rear log file i found something like
'/tmp/rear.g7ZIwz3UWiFCXP1/rootfs/etc/mtab' -> '/proc/self/mounts'
This link is also included in backup.tar.gz, but at a recover it is not created. Also in the rear recovery log located at /var/lib/rear/restore/ is no entry to find.
jsmeix commented at 2018-10-31 13:04:¶
@jonathankeuser
FYI what I have in my 'backup.tar.gz' and on my original system:
# tar -tvf backup.tar.gz | grep etc/mtab lrwxrwxrwx root/root 0 2018-09-13 09:42 etc/mtab -> ../proc/self/mounts # file /etc/mtab /etc/mtab: symbolic link to ../proc/self/mounts
@gdha
I do not understand how /etc/mtab in the recovery system is related
to etc/mtab in the restored system (which is /mnt/local/etc/mtab in the
recovery system)
at least I do not see a relationship in the code.
gdha commented at 2018-10-31 13:22:¶
@jsmeix We should not worry about ROOTFS/etc/mtab as it supposed to be a
symbolic link to /proc/self/mounts on RHEL 7.
On RHEL 5 anf 6: /etc/mtab is the file that mount and umount maintains
for a list of currently mounted filesystems. While, /proc/mounts is
the kernel's view of the mount.
/proc/mounts is currently a link to /proc/self/mounts which lists
mount points of the processes's mount namespace. The VFS is responsible
to add to this file VFS options found in binary flags interface of each
filesystem, but, any specific filesystem option should be added by the
filesystem code using the superblock method show_options()
On RHEL7, various command and libraries have been updated to use
libmount to read mount information and /etc/mtab is now a symbolic
link to /proc/self/mounts
Info pasted from https://access.redhat.com/solutions/65949
jonathankeuser commented at 2018-10-31 13:42:¶
@jsmeix Sorry, I expressed myself a little bit incomprehensible:
After booting the rescue image again, create the symlink for /etc/mtab and boot the OS again, my server will boot successfully.
I did not mean the /etc/mtab of the recovery system, but the mtab of the system being restored which is located at /mnt/local/etc/mtab.
FYI what I have in my 'backup.tar.gz' and on my original system:
'# tar -tvf backup.tar.gz | grep etc/mtab
lrwxrwxrwx root/root 0 2018-09-13 09:42 etc/mtab -> ../proc/self/mounts'
'# file /etc/mtab
/etc/mtab: symbolic link to ../proc/self/mounts'
That's about what I find in my original system, as described above.
jsmeix commented at 2018-10-31 13:48:¶
@gdha
I know about /etc/mtab which was traditionally a file
versus a symbolic link on nowadays systems.
My question was because your
https://github.com/rear/rear/issues/1946#issuecomment-434649350
is about etc/mtab in the recovery system
while @jonathankeuser issue here (if I understand it correctly)
is about the restored etc/mtab in the recreated system.
@jonathankeuser
I did a "rear recover" (but I run SLES12) and for me etc/mtab gets
correctly
restored and is still correct on the rebooted recreated system
(excerpts):
RESCUE d67:~ # export MIGRATION_MODE=1 RESCUE d67:~ # rear -D recover Relax-and-Recover 2.4 / Git Running rear recover (PID 1234) ... Restoring from '/tmp/rear.IoLcmPctGOwFZtD/outputfs/d67/backup.tar.gz' (restore log in /var/lib/rear/restore/recover.backup.tar.gz.1234.restore.log) ... ... Confirm restored config files or edit them 1) Confirm it is OK to recreate initrd and reinstall bootloader and continue 'rear recover' 2) Edit restored etc/fstab (/mnt/local/etc/fstab) 3) View restored etc/fstab (/mnt/local/etc/fstab) 4) Use Relax-and-Recover shell and return back to here 5) Abort 'rear recover' (default '1' timeout 300 seconds) 4 ... rear> grep etc/mtab /var/lib/rear/restore/recover.backup.tar.gz.1234.restore.log block 132653: etc/mtab rear> ls -l /mnt/local/etc/mtab lrwxrwxrwx 1 root root 19 Sep 27 09:40 /mnt/local/etc/mtab -> ../proc/self/mounts rear> exit ... Finished recovering your system. You can explore it under '/mnt/local'. RESCUE d67:~ # ls -l /mnt/local/etc/mtab lrwxrwxrwx 1 root root 19 Sep 27 09:40 /mnt/local/etc/mtab -> ../proc/self/mounts RESCUE d67:~ # reboot f186:~ # ls -l /etc/mtab lrwxrwxrwx 1 root root 19 Sep 27 11:40 /etc/mtab -> ../proc/self/mounts
gdha commented at 2018-10-31 13:57:¶
@jsmeix The restoration of /etc/mtab is not the issue here, seems that
the symbolic link itself is not restored. However, I always thought that
everything under /proc file system is under control of the linux
kernel itself. Seems to be proven wrong?
jsmeix commented at 2018-10-31 16:14:¶
@gdha
I am afraid - in this issue I do somehow not understand you.
I would think restoration of /etc/mtab means that the symbolic link itself is restored?
Cf.
https://github.com/rear/rear/issues/1946#issuecomment-434674774
This link is also included in backup.tar.gz, but at a recover it is not created.
I.e. somehow for @jonathankeuser the link is in his 'tar' backup
but it does not get restored (in contrast to what happens on my system).
Because we had in the past already some weird issues with 'tar' on
RHEL
I guess this one could be perhaps another one of those kind of issues?
gdha commented at 2018-11-05 15:39:¶
I've performed a test on CentOS 7.5 with BACKUP=NETFS and tar. The tar
archive contained the following after grepping on:
# tar ztf backup.tar.gz | grep -E '(mtab|mounts)'
etc/mtab
usr/share/man/man8/mountstats.8.gz
usr/lib/python2.7/site-packages/tuned/plugins/plugin_mounts.py
usr/lib/python2.7/site-packages/tuned/plugins/plugin_mounts.pyo
usr/lib/python2.7/site-packages/tuned/plugins/plugin_mounts.pyc
usr/sbin/mountstats
usr/lib64/python2.7/symtable.pyc
usr/lib64/python2.7/symtable.pyo
usr/lib64/python2.7/symtable.py
var/lib/nfs/rmtab
After doing a recovery test I still have:
[root@client ~]# ls -l /etc/mtab /proc/self/mounts
lrwxrwxrwx. 1 root root 17 Sep 30 21:58 /etc/mtab -> /proc/self/mounts
-r--r--r--. 1 root root 0 Nov 5 16:22 /proc/self/mounts
So far, nothing has changed for me (before and after the modification
done at build/default/985_fix_broken_links.sh)
jsmeix commented at 2019-04-26 12:58:¶
Because we can neither reproduce nor imagine what goes actually wrong
and because we had in the past already some weird issues with 'tar' on
RHEL
I assume this one is another one of those kind of issues.
[Export of Github issue for rear/rear.]