#1989 Issue closed: Ubuntu 14.04 mkbackup does not backup anything¶

Labels: bug, support / question, fixed / solved / done

Pablohn26 opened issue at 2018-11-30 23:40:¶

Relax-and-Recover (ReaR) Issue Template¶

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

• ReaR version ("/usr/sbin/rear -V"): Relax-and-Recover 2.4 / 2018-06-21

• OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"):

OS_VENDOR=Ubuntu
OS_VERSION=14.04

• ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

OUTPUT=ISO
BACKUP=NETFS
BACKUP_URL=nfs://213.251.184.109/

• Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR): KVM guest

• System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device): x86_64

• Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot): GRUB

• Storage (lokal disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe): virtual local disk

• Description of the issue (ideally so that others can reproduce it):

When I try to create a backup of my server, there is no information saved on backup.tar.gz. Just a log file named /var/log/rear/rear-hostname.log, that I have attached it.

backup.tar.gz

I am not sure if it is related to #446

There is no error that suggests the problem.

• Workaround, if any: no workaround

• Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):

rear -D mkbackup
Relax-and-Recover 2.4 / 2018-06-21
Using log file: /var/log/rear/rear-comoreclamar.log
Using backup archive '/tmp/rear.TkIQFqVP3JleVJU/outputfs/comoreclamar/backup.tar.gz'
Creating disk layout
Using guessed bootloader 'GRUB' (found in first bytes on /dev/vda)
Creating root filesystem layout
Skipping 'br-ee7d1d7e64de': not bound to any physical interface.
Skipping 'docker0': not bound to any physical interface.
Handling network interface 'eth0'
eth0 is a physical device
Handled network interface 'eth0'
Skipping 'tap0': not bound to any physical interface.
Cannot include keyboard mappings (no keymaps default directory '')
Copying logfile /var/log/rear/rear-comoreclamar.log into initramfs as '/tmp/rear-comoreclamar-partial-2018-12-01T00:36:30+0100.log'
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Copying all files in /lib*/firmware/
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (115543053 bytes) in 31 seconds
Making ISO image
Wrote ISO image: /var/lib/rear/output/rear-comoreclamar.iso (117M)
Copying resulting files to nfs location
Saving /var/log/rear/rear-comoreclamar.log as rear-comoreclamar.log to nfs location
Creating tar archive '/tmp/rear.TkIQFqVP3JleVJU/outputfs/comoreclamar/backup.tar.gz'
Preparing archive operationOK
Exiting rear mkbackup (PID 9947) and its descendant processes
Running exit tasks
You should also rm -Rf /tmp/rear.TkIQFqVP3JleVJU

jsmeix commented at 2018-12-03 08:38:¶

@Pablohn26
where is the "rear -D mkbackup" debug log file?

Your
https://github.com/rear/rear/files/2635326/backup.tar.gz
contains var/log/rear/rear-comoreclamar.log
which contains (excerpts)

2018-11-30 22:28:40.091296354 Relax-and-Recover 2.4 / 2018-06-21
2018-11-30 22:28:40.095342864 Command line options: /usr/sbin/rear -d -v mkbackup
...
2018-11-30 22:31:10.529187514 Running 'backup' stage
...
2018-11-30 22:31:11.294714606 Including backup/NETFS/default/400_create_include_exclude_files.sh
2018-11-30 22:31:11.307264238 Including backup/NETFS/default/500_make_backup.sh
2018-11-30 22:31:11.311758099 Include list:
2018-11-30 22:31:11.317633013 Exclude list:
2018-11-30 22:31:11.323896985  /tmp/*
2018-11-30 22:31:11.329709331  /dev/shm/*
2018-11-30 22:31:11.335236477  /var/lib/rear/output/*
2018-11-30 22:31:11.388136406 Encrypting backup archive is disabled

so nothing gets included in your backup (Include list is empty).

Do you use btrfs as filesystem?
If yes, do you use btrfs subvolumes?
If yes, you may have to specify them in BACKUP_PROG_INCLUDE, cf.
https://github.com/rear/rear/blob/master/usr/share/rear/conf/examples/SLE12-btrfs-example.conf

jsmeix commented at 2018-12-07 15:39:¶

No news is good news.

Pablohn26 commented at 2019-01-18 14:57:¶

No, i do not have btrfs

root@comoreclamar:~# cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    nodev,noexec,nosuid 0       0
/dev/vda1       /               ext3    errors=remount-ro 0       1
/dev/vdb        none            swap    sw              0       0


root@comoreclamar:~# blkid 
/dev/vda1: UUID="b9734409-df40-4819-8aa5-7dc376bf87c2" TYPE="ext3" 
/dev/vdb: UUID="55cc02c1-ef0e-4a77-9f9f-6edf3ce09fdf" TYPE="swap" 
/dev/fd0: LABEL="grub" UUID="fe4a3e31-2fee-402a-b9c6-586807ae3462" TYPE="ext2"


root@comoreclamar:~# lsblk 
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
fd0      2:0    1   1.4M  0 disk 
sr0     11:0    1  1024M  0 rom  
vda    253:0    0    99G  0 disk 
`-vda1 253:1    0    98G  0 part /
vdb    253:16   0     1G  0 disk [SWAP]

root@comoreclamar:~# lsblk -f
NAME   FSTYPE LABEL MOUNTPOINT
fd0    ext2   grub  
sr0                 
vda                 
`-vda1 ext3         /
vdb    swap         [SWAP]


root@comoreclamar:~# df -Th
Filesystem     Type      Size  Used Avail Use% Mounted on
udev           devtmpfs  3.0G  8.0K  3.0G   1% /dev
tmpfs          tmpfs     597M  1.2M  596M   1% /run
/dev/vda1      ext3       98G   47G   46G  51% /
none           tmpfs     4.0K     0  4.0K   0% /sys/fs/cgroup
none           tmpfs     5.0M     0  5.0M   0% /run/lock
none           tmpfs     3.0G  2.1M  3.0G   1% /run/shm
none           tmpfs     100M     0  100M   0% /run/user
none           aufs       98G   47G   46G  51% /var/lib/docker/aufs/mnt/65a3fa760f7110d8afb2e16b55af50f1970e852853bda2fc79d6610a0cf6c638
none           aufs       98G   47G   46G  51% /var/lib/docker/aufs/mnt/bd155644a030b211628db27a4c379423227cc9953562b8ac00ad9af39614a11f
none           aufs       98G   47G   46G  51% /var/lib/docker/aufs/mnt/22cd0f89e367b23525a18ce2bb217043d9e71487be1b5c8cde4fcb90c24d12d2
none           aufs       98G   47G   46G  51% /var/lib/docker/aufs/mnt/bebb89c1e24badf67ee52cd3a22899faf6945c7a3a1dfadda83fc1b9286f704c
none           aufs       98G   47G   46G  51% /var/lib/docker/aufs/mnt/4a26e4e6383ba8ae34121bfa3108d710af7458dd30639776d4254befc74f931c
none           aufs       98G   47G   46G  51% /var/lib/docker/aufs/mnt/f613d0edfec6197885d41ac4623b949d811abf7f34fdfd2445cb7d03c6aa5dea
none           aufs       98G   47G   46G  51% /var/lib/docker/aufs/mnt/2f2c24947d488e053988d793e783fffa9aa6ace889f6e58b1be1aab3a96b37a0
none           aufs       98G   47G   46G  51% /var/lib/docker/aufs/mnt/7bde49124b5497185a10060e819f0e605e254db4166c20fa659176b49d671c46
shm            tmpfs      64M     0   64M   0% /var/lib/docker/containers/82b98d1f6db03914fa7f4662a6e7c6393c7fc683602c3541766080acc607800d/shm
shm            tmpfs      64M     0   64M   0% /var/lib/docker/containers/f810b09edb6eb4236e5c6fccaa0a3501b3cc1359af14811448ead1f04680170d/shm
shm            tmpfs      64M     0   64M   0% /var/lib/docker/containers/a31797f2600e126d46f2cedf76960f058f44af6724e33b174734d9d2492c9c27/shm
shm            tmpfs      64M  4.0K   64M   1% /var/lib/docker/containers/a81659e133827925e341df4230ef394c7f38df31c3d72a7f9a0ebed8a864d317/shm
shm            tmpfs      64M     0   64M   0% /var/lib/docker/containers/b07f82d38edcb27e5d68847121a13710e4363173faf9f955f81a40c02bb72b2b/shm
shm            tmpfs      64M     0   64M   0% /var/lib/docker/containers/dcf540ad2f2c2453d013a26adb99586e8d89cbc4dfdb9941e95b84a6e57eb63b/shm
shm            tmpfs      64M     0   64M   0% /var/lib/docker/containers/e60fa65d8a6c642c39b2caa68d29ca2c73b79f5be74f1249d65d873c3d9d6b18/shm
shm            tmpfs      64M     0   64M   0% /var/lib/docker/containers/fc63c1b757cec63b0863d6e43f73911637d562ee35c20f2c61a658878f5cab77/shm


fsck -N /dev/sda1
fsck from util-linux 2.20.1
[/sbin/fsck.ext2 (1) -- /dev/sda1] fsck.ext2 /dev/sda1


root@comoreclamar:~# mount
/dev/vda1 on / type ext3 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/cgroup type tmpfs (rw)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
none on /run/user type tmpfs (rw,noexec,nosuid,nodev,size=104857600,mode=0755)
none on /sys/fs/pstore type pstore (rw)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu type cgroup (rw,relatime,cpu)
cgroup on /sys/fs/cgroup/cpuacct type cgroup (rw,relatime,cpuacct)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,relatime,freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,relatime,perf_event)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,relatime,hugetlb)
rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
systemd on /sys/fs/cgroup/systemd type cgroup (rw,noexec,nosuid,nodev,none,name=systemd)




root@comoreclamar:~# rear -D mkbackup
Relax-and-Recover 2.4 / 2018-06-21
Using log file: /var/log/rear/rear-comoreclamar.log
Using backup archive '/tmp/rear.2SUZpKoG9AOYP8s/outputfs/comoreclamar/backup.tar.gz'
Creating disk layout
Using guessed bootloader 'GRUB' (found in first bytes on /dev/vda)
Creating root filesystem layout
Skipping 'br-ee7d1d7e64de': not bound to any physical interface.
Skipping 'docker0': not bound to any physical interface.
Handling network interface 'eth0'
eth0 is a physical device
Handled network interface 'eth0'
Skipping 'tap0': not bound to any physical interface.
Cannot include keyboard mappings (no keymaps default directory '')
Copying logfile /var/log/rear/rear-comoreclamar.log into initramfs as '/tmp/rear-comoreclamar-partial-2019-01-18T15:51:14+0100.log'
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Copying all files in /lib*/firmware/
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (115546198 bytes) in 28 seconds
Making ISO image
Wrote ISO image: /var/lib/rear/output/rear-comoreclamar.iso (117M)
Copying resulting files to nfs location
Saving /var/log/rear/rear-comoreclamar.log as rear-comoreclamar.log to nfs location
Creating tar archive '/tmp/rear.2SUZpKoG9AOYP8s/outputfs/comoreclamar/backup.tar.gz'
Preparing archive operationOK
Exiting rear mkbackup (PID 4262) and its descendant processes
Running exit tasks
You should also rm -Rf /tmp/rear.2SUZpKoG9AOYP8s

rear-comoreclamar.log

Sorry about my late response :(

jsmeix commented at 2019-01-18 16:01:¶

@Pablohn26

your rear-comoreclamar.log contains (except)

2019-01-18 15:53:35.912133471 Including backup/NETFS/default/400_create_include_exclude_files.sh
2019-01-18 15:53:35.915736961 Entering debugscripts mode via 'set -x'.
+ source /usr/share/rear/backup/NETFS/default/400_create_include_exclude_files.sh
++ is_true no
++ case "$1" in
++ return 1
++ '[' NO '!=' YES ']'

which does not match our source code in
https://raw.githubusercontent.com/rear/rear/master/usr/share/rear/backup/NETFS/default/400_create_include_exclude_files.sh
that starts with

# Backup all that is explicitly specified in BACKUP_PROG_INCLUDE:
for backup_include_item in "${BACKUP_PROG_INCLUDE[@]}" ; do
    test "$backup_include_item" && echo "$backup_include_item"
done > $TMP_DIR/backup-include.txt

# Implicitly also backup all local filesystems as defined in mountpoint_device
# except BACKUP_ONLY_INCLUDE or MANUAL_INCLUDE is set:
if ! is_true "$BACKUP_ONLY_INCLUDE" ; then
    if [ "${MANUAL_INCLUDE:-NO}" != "YES" ] ; then

so that it seems your
usr/share/rear/backup/NETFS/default/400_create_include_exclude_files.sh
is somehow broken.

When one of your ReaR scripts is broken you cannot know
what else could be also broken so that I suggest
to re-install ReaR completely anew.

I would also recommend to try out our current
ReaR upstream GitHub master code as follows:

Basically "git clone" our current ReaR upstream GitHub master code
into a separated directory and then configure and run ReaR
from within that directory like:

# git clone https://github.com/rear/rear.git

# mv rear rear.github.master

# cd rear.github.master

# vi etc/rear/local.conf

# usr/sbin/rear -D mkbackup

Note the relative paths "etc/rear/" and "usr/sbin/".

I recommend to try out our current ReaR upstream GitHub master code
because that is the only place where we fix bugs - i.e. bugs in older
ReaR versions are not fixed by us (i.e. by ReaR upstream).
If it works with our current ReaR upstream GitHub master code
we would appreciate an explicit positive feedback.

jsmeix commented at 2019-01-18 16:18:¶

@Pablohn26
my previous comment
https://github.com/rear/rear/issues/1989#issuecomment-455596494
is probably wrong because by default in usr/share/rear/conf/default.conf

BACKUP_PROG_INCLUDE=( )

and set -x shows nothing when a for loop is run with empty argument

# BACKUP_PROG_INCLUDE=( )
# set -x
# for backup_include_item in "${BACKUP_PROG_INCLUDE[@]}" ; do echo "'$backup_include_item'" ; done )

so that I need to do some more analysis
what the actual root cause could be...

jsmeix commented at 2019-01-21 12:14:¶

Found it:

@Pablohn26
your rear-comoreclamar.log contains (excepts)

+ source /usr/share/rear/layout/save/GNU/Linux/230_filesystem_layout.sh
...
++ docker_is_running=
++ service docker status
++ docker_is_running=yes
...
++ is_true yes
++ case "$1" in
++ return 0
+++ docker info
+++ grep 'Docker Root Dir'
+++ awk '{print $4}'
++ docker_root_dir=
++ Log '/dev/vda1 is mounted below  (mount point / is under docker control), skipping.'
+++ date '+%Y-%m-%d %H:%M:%S.%N '
++ local 'timestamp=2019-01-18 15:51:06.300003384 '
++ test 1 -gt 0
++ echo '2019-01-18 15:51:06.300003384 /dev/vda1 is mounted below  (mount point / is under docker control), skipping.'
2019-01-18 15:51:06.300003384 /dev/vda1 is mounted below  (mount point / is under docker control), skipping.
++ echo /
++ grep -q '^'
++ continue

The matching code is in
usr/share/rear/layout/save/GNU/Linux/230_filesystem_layout.sh
here from our current one in
https://raw.githubusercontent.com/rear/rear/master/usr/share/rear/layout/save/GNU/Linux/230_filesystem_layout.sh
the code excerpts that match the above log lines

# docker daemon mounts file systems for its docker containers
# see also https://docs.docker.com/storage/storagedriver/device-mapper-driver/#configure-direct-lvm-mode-for-production
# As it is for container usage only we do not to backup these up or recreate as this disk device is completely under
# control by docker itself (even the recreation of it incl, the creation of the volume group). Usually this is
# done via a kind of cookbook (Chef, puppet or ansible)
docker_is_running=""
service docker status >/dev/null 2>&1 && docker_is_running="yes"
...
        # docker specific exclude part
        if is_true $docker_is_running ; then
            # docker daemon/service is running
            docker_root_dir=$( docker info 2>/dev/null | grep 'Docker Root Dir' | awk '{print $4}' )
            # If $docker_root_dir is in the beginning of the $mountpoint string then FS is under docker control
            # and we better exclude from saving the layout,
            # see https://github.com/rear/rear/issues/1749
            Log "$device is mounted below $docker_root_dir (mount point $mountpoint is under docker control), skipping."
            echo "$mountpoint" | grep -q "^$docker_root_dir" && continue
        fi

What looks suspicious in your case is that
it seems ReaR fails to get the docker_root_dir
because that is empty in your case.

As a consequence the filesystem that is mounted at the '/' directoy
gets excluded from var/lib/rear/layout/disklayout.conf
so that you have no entry for your root filesystem in your disklayout.conf
which would let rear recover fail anyway.

Another consequence is that you don't get automatically the directory
where your root filesystem is mounted (i.e. the / directory) included
in your TMP_DIR/backup-include.txt file via
usr/share/rear/backup/NETFS/default/400_create_include_exclude_files.sh
so that subsequently nothing appears in your backup via
usr/share/rear/backup/NETFS/default/500_make_backup.sh
that basically calls tar -c $(cat backup-include.txt)
to get things included in the backup, cf. the BACKUP_PROG_INCLUDE
description in usr/share/rear/conf/default.conf

@Pablohn26
do you use docker?

jsmeix commented at 2019-01-21 13:20:¶

The changes in
https://github.com/rear/rear/pull/2021/files
may fix it.
At least they make that code behave fail-safe.
But I do not use docker so that I cannot test
how things behave when docker is used.

jsmeix commented at 2019-01-21 13:22:¶

@Pablohn26
what does the command docker info (run it as root)
output on your system (wee need the whole exact output)?

jsmeix commented at 2019-01-22 12:27:¶

With https://github.com/rear/rear/pull/2021
ReaR should now behave in a more obvious way
when there is a possibly crippled Docker installation.
In particular when Docker is used but its 'Docker Root Dir'
cannot be determined it shows now

Cannot determine Docker Root Dir - things may go
wrong - check /var/lib/rear/layout/disklayout.conf

and in this case it does no longer skip all filesystems.

But I use neither Docker nor Ubuntu so that I cannot test
how things behave when Docker is used in Ubuntu.

@Pablohn26
please test our current ReaR upstream GitHub master code
as described in https://github.com/rear/rear/issues/1989#issuecomment-455596494

gdha commented at 2019-02-18 14:25:¶

@jsmeix IMHO it is fine to close this issue - the fix will work

---

[Export of Github issue for rear/rear.]