#1749 Issue `closed`: docker hosts would better exclude the file systems under control by docker¶

Labels: enhancement, bug, fixed / solved / done

gdha opened issue at 2018-03-08 08:49:¶

rear version (/usr/sbin/rear -V): rear-2.00 and higher
OS version (cat /etc/rear/os.conf or lsb_release -a): rhel 7 (but probably others as well)
rear configuration files (cat /etc/rear/site.conf or cat /etc/rear/local.conf): empty
Are you using legacy BIOS or UEFI boot? BIOS
Brief description of the issue: checking the savelayout only

#-> df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/xvda2      16764908 4288520  12476388  26% /
devtmpfs        32871068       0  32871068   0% /dev
tmpfs           32896424       8  32896416   1% /dev/shm
tmpfs           32896424  239080  32657344   1% /run
tmpfs           32896424       0  32896424   0% /sys/fs/cgroup
/dev/xvdf      103081248 2790172  96015008   3% /app
/dev/dm-3       10473472  260024  10213448   3% /app/docker/devicemapper/mnt/d21a02d860b705dccf3b79c57ffcf0a863d13ddfe061d40a0d967a4f5bc2169e
shm                65536       0     65536   0% /app/docker/containers/4c2331930fcefd69f5c9dd7e401cd678c24c89450ac6d9185135dfc40d747e37/shm
shm                65536       0     65536   0% /app/docker/containers/99a4b36771e0db20fc1360da3072d826e9f78faaa640720c85b90c0a6c1db33e/shm
/dev/dm-5       10473472  278428  10195044   3% /app/docker/devicemapper/mnt/c369cf523f69144ec83d35ed9ae49f02cb088927a5d19824cd7733706199d4ea
shm                65536       0     65536   0% /app/docker/containers/1a0e1c0452ff3762c5c2d3daca9a744f154a91b49884c774eb978b22b7707e76/shm
/dev/dm-7       10473472  294072  10179400   3% /app/docker/devicemapper/mnt/903969c07c44bbf949ef7b4b24fbd6fcf1ebbcfbf798c79ad415b03bd1eb1a27
shm                65536       0     65536   0% /app/docker/containers/e253881cf82cde743cb13653de1a886bdfe1b5830feeb1793694ea5fa41a1469/shm
/dev/dm-8       10473472  410340  10063132   4% /app/docker/devicemapper/mnt/a238e6a06ab4820134e95a6924eb5a8ff93fbdc1be9cd7bb9f01b6814f147c6d
shm                65536       0     65536   0% /app/docker/containers/bc25d0d82a98ad55a04d9d409fdd7c1ab271f010ecec6261c1fba5fee553836c/shm
/dev/dm-9       10473472  267064  10206408   3% /app/docker/devicemapper/mnt/731c75288780b5cd959925096d8cfe2114029091637266b39cabe170605c7223
shm                65536       0     65536   0% /app/docker/containers/8855132be26f5043630d073e64938330340e22060a2dd45e8d2fadee44d0b067/shm
/dev/dm-10      10473472  275788  10197684   3% /app/docker/devicemapper/mnt/029be07c5533645e1d2177b32383386b8a418947cf2f577183e5cd8fbf3e6811
shm                65536       0     65536   0% /app/docker/containers/ec5abed6e23f3b05c6d10cb26e2d9e132ec4b16cf374ad45a82a71163d38f22b/shm
/dev/dm-11      10473472  294332  10179140   3% /app/docker/devicemapper/mnt/73dc5c60598e25b2e13539e816fe747ef18b7797fe0187e5049a9fea68ab90ac
shm                65536       0     65536   0% /app/docker/containers/d704d9391a673166819f27bd8b9e294697c9dd8283f60035919f1fa2fda50d54/shm
/dev/dm-14      10473472   81860  10391612   1% /app/docker/devicemapper/mnt/3d40fc33d3ab1fb42c2f8df9f74b5154951d765e8ba070e654b01c7f1bc656db
shm                65536       0     65536   0% /app/docker/containers/ccb098bab9cb8a5907785bfe741b79445dae5113ee61cc3d339e4d07d43fa761/shm
tmpfs            6579288       0   6579288   0% /run/user/0
/dev/dm-6       10473472 2626616   7846856  26% /app/docker/devicemapper/mnt/3da983fbe101aa59943b9c2d69c28538a08dcb886b8fd779852910e7d5404e0a
shm                65536       0     65536   0% /app/docker/containers/be09c2017aa71a6bda025e4709751958c42abce165624f4a30c723632600ec64/shm
/dev/dm-12      10473472  193356  10280116   2% /app/docker/devicemapper/mnt/3d527d49b0b344c26f38af5c567e01678493b2936ed71be66e12d23383bc150c
shm                65536       0     65536   0% /app/docker/containers/a1c56fa2ca4779529765f3a703264240e898b87b5ca02a5717abf72be819db15/shm
/dev/dm-13      10473472 1007124   9466348  10% /app/docker/devicemapper/mnt/b23bc01d6b87dfe523aa92cbf822e9c111f1610665b122291b3f884ef6a2f4e8
shm                65536       0     65536   0% /app/docker/containers/2af9689dfec03b79f2697b42de58cd34cd760d6ef1141d6af0e7e65657f8df51/shm
tmpfs            6579288       0   6579288   0% /run/user/17010
tmpfs            6579288       0   6579288   0% /run/user/11787

#-> lsblk
NAME                                                                                        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
xvda                                                                                        202:0    0    16G  0 disk
├─xvda1                                                                                     202:1    0     1M  0 part
└─xvda2                                                                                     202:2    0    16G  0 part /
xvdb                                                                                        202:16   0   200G  0 disk
├─vg--docker-vg--thinpool_tmeta                                                             253:0    0   100M  0 lvm
│ └─vg--docker-vg--thinpool                                                                 253:2    0 199.8G  0 lvm
│   ├─docker-202:80-720905-d21a02d860b705dccf3b79c57ffcf0a863d13ddfe061d40a0d967a4f5bc2169e 253:3    0    10G  0 dm   /app/docker/devicemapper/mnt/d21
│   ├─docker-202:80-720905-8f2d7f8d1d402ec145dc9dbe6d9b38b137f3b16c8b3dc2ba54c978e8ec46452c 253:4    0    10G  0 dm
│   ├─docker-202:80-720905-c369cf523f69144ec83d35ed9ae49f02cb088927a5d19824cd7733706199d4ea 253:5    0    10G  0 dm   /app/docker/devicemapper/mnt/c36
│   ├─docker-202:80-720905-3da983fbe101aa59943b9c2d69c28538a08dcb886b8fd779852910e7d5404e0a 253:6    0    10G  0 dm   /app/docker/devicemapper/mnt/3da
│   ├─docker-202:80-720905-903969c07c44bbf949ef7b4b24fbd6fcf1ebbcfbf798c79ad415b03bd1eb1a27 253:7    0    10G  0 dm   /app/docker/devicemapper/mnt/903
│   ├─docker-202:80-720905-a238e6a06ab4820134e95a6924eb5a8ff93fbdc1be9cd7bb9f01b6814f147c6d 253:8    0    10G  0 dm   /app/docker/devicemapper/mnt/a23
│   ├─docker-202:80-720905-731c75288780b5cd959925096d8cfe2114029091637266b39cabe170605c7223 253:9    0    10G  0 dm   /app/docker/devicemapper/mnt/731
│   ├─docker-202:80-720905-029be07c5533645e1d2177b32383386b8a418947cf2f577183e5cd8fbf3e6811 253:10   0    10G  0 dm   /app/docker/devicemapper/mnt/029
│   ├─docker-202:80-720905-73dc5c60598e25b2e13539e816fe747ef18b7797fe0187e5049a9fea68ab90ac 253:11   0    10G  0 dm   /app/docker/devicemapper/mnt/73d
│   ├─docker-202:80-720905-3d527d49b0b344c26f38af5c567e01678493b2936ed71be66e12d23383bc150c 253:12   0    10G  0 dm   /app/docker/devicemapper/mnt/3d5
│   ├─docker-202:80-720905-b23bc01d6b87dfe523aa92cbf822e9c111f1610665b122291b3f884ef6a2f4e8 253:13   0    10G  0 dm   /app/docker/devicemapper/mnt/b23
│   └─docker-202:80-720905-3d40fc33d3ab1fb42c2f8df9f74b5154951d765e8ba070e654b01c7f1bc656db 253:14   0    10G  0 dm   /app/docker/devicemapper/mnt/3d4
└─vg--docker-vg--thinpool_tdata                                                             253:1    0 199.8G  0 lvm
  └─vg--docker-vg--thinpool                                                                 253:2    0 199.8G  0 lvm
    ├─docker-202:80-720905-d21a02d860b705dccf3b79c57ffcf0a863d13ddfe061d40a0d967a4f5bc2169e 253:3    0    10G  0 dm   /app/docker/devicemapper/mnt/d21
    ├─docker-202:80-720905-8f2d7f8d1d402ec145dc9dbe6d9b38b137f3b16c8b3dc2ba54c978e8ec46452c 253:4    0    10G  0 dm
    ├─docker-202:80-720905-c369cf523f69144ec83d35ed9ae49f02cb088927a5d19824cd7733706199d4ea 253:5    0    10G  0 dm   /app/docker/devicemapper/mnt/c36
    ├─docker-202:80-720905-3da983fbe101aa59943b9c2d69c28538a08dcb886b8fd779852910e7d5404e0a 253:6    0    10G  0 dm   /app/docker/devicemapper/mnt/3da
    ├─docker-202:80-720905-903969c07c44bbf949ef7b4b24fbd6fcf1ebbcfbf798c79ad415b03bd1eb1a27 253:7    0    10G  0 dm   /app/docker/devicemapper/mnt/903
    ├─docker-202:80-720905-a238e6a06ab4820134e95a6924eb5a8ff93fbdc1be9cd7bb9f01b6814f147c6d 253:8    0    10G  0 dm   /app/docker/devicemapper/mnt/a23
    ├─docker-202:80-720905-731c75288780b5cd959925096d8cfe2114029091637266b39cabe170605c7223 253:9    0    10G  0 dm   /app/docker/devicemapper/mnt/731
    ├─docker-202:80-720905-029be07c5533645e1d2177b32383386b8a418947cf2f577183e5cd8fbf3e6811 253:10   0    10G  0 dm   /app/docker/devicemapper/mnt/029
    ├─docker-202:80-720905-73dc5c60598e25b2e13539e816fe747ef18b7797fe0187e5049a9fea68ab90ac 253:11   0    10G  0 dm   /app/docker/devicemapper/mnt/73d
    ├─docker-202:80-720905-3d527d49b0b344c26f38af5c567e01678493b2936ed71be66e12d23383bc150c 253:12   0    10G  0 dm   /app/docker/devicemapper/mnt/3d5
    ├─docker-202:80-720905-b23bc01d6b87dfe523aa92cbf822e9c111f1610665b122291b3f884ef6a2f4e8 253:13   0    10G  0 dm   /app/docker/devicemapper/mnt/b23
    └─docker-202:80-720905-3d40fc33d3ab1fb42c2f8df9f74b5154951d765e8ba070e654b01c7f1bc656db 253:14   0    10G  0 dm   /app/docker/devicemapper/mnt/3d4
xvdf                                                                                        202:80   0   100G  0 disk /app

The docker device /dev/xvdb is not mounted by the system, but is under control by the docker application. Therefore all thin-pool mounted file systems should not be included in savelayout.conf file:

#-> cat /var/lib/rear/layout/disklayout.conf
# Disk /dev/xvda
# Format: disk <devname> <size(bytes)> <partition label type>
disk /dev/xvda 17179869184 gpt
# Partitions on /dev/xvda
# Format: part <device> <partition size(bytes)> <partition start(bytes)> <partition type|name> <flags> /dev/<partition>
part /dev/xvda 1048576 1048576 rear-noname bios_grub /dev/xvda1
part /dev/xvda 17177755136 2097152 rear-noname none /dev/xvda2
# Disk /dev/xvdb
# Format: disk <devname> <size(bytes)> <partition label type>
#disk /dev/xvdb 214748364800 unknown
# Partitions on /dev/xvdb
# Format: part <device> <partition size(bytes)> <partition start(bytes)> <partition type|name> <flags> /dev/<partition>
# Disk /dev/xvdf
# Format: disk <devname> <size(bytes)> <partition label type>
disk /dev/xvdf 107374182400 loop
# Partitions on /dev/xvdf
# Format: part <device> <partition size(bytes)> <partition start(bytes)> <partition type|name> <flags> /dev/<partition>
#lvmdev /dev/vg-docker /dev/xvdb qVorYA-oIev-qunB-hscU-nJH4-t2c5-8zvfK5 419430400
#lvmgrp /dev/vg-docker 4096 51199 209711104
#lvmvol /dev/vg-docker vg-thinpool 51149 419012608
# Filesystems (only ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs are supported).
# Format: fs <device> <mountpoint> <fstype> [uuid=<uuid>] [label=<label>] [<attributes>]
fs /dev/mapper/docker-202:80-720905-029be07c5533645e1d2177b32383386b8a418947cf2f577183e5cd8fbf3e6811 /app/docker/devicemapper/mnt/029be07c5533645e1d2177b32383386b8a418947cf2f577183e5cd8fbf3e6811 xfs uuid=b382d6a5-9147-47b2-8c3a-6ef7c33751dc label=  options=rw,relatime,nouuid,attr2,inode64,logbsize=128k,sunit=256,swidth=256,noquota
fs /dev/mapper/docker-202:80-720905-3d40fc33d3ab1fb42c2f8df9f74b5154951d765e8ba070e654b01c7f1bc656db /app/docker/devicemapper/mnt/3d40fc33d3ab1fb42c2f8df9f74b5154951d765e8ba070e654b01c7f1bc656db xfs uuid=b382d6a5-9147-47b2-8c3a-6ef7c33751dc label=  options=rw,relatime,nouuid,attr2,inode64,logbsize=128k,sunit=256,swidth=256,noquota
fs /dev/mapper/docker-202:80-720905-3d527d49b0b344c26f38af5c567e01678493b2936ed71be66e12d23383bc150c /app/docker/devicemapper/mnt/3d527d49b0b344c26f38af5c567e01678493b2936ed71be66e12d23383bc150c xfs uuid=b382d6a5-9147-47b2-8c3a-6ef7c33751dc label=  options=rw,relatime,nouuid,attr2,inode64,logbsize=128k,sunit=256,swidth=256,noquota
fs /dev/mapper/docker-202:80-720905-3da983fbe101aa59943b9c2d69c28538a08dcb886b8fd779852910e7d5404e0a /app/docker/devicemapper/mnt/3da983fbe101aa59943b9c2d69c28538a08dcb886b8fd779852910e7d5404e0a xfs uuid=b382d6a5-9147-47b2-8c3a-6ef7c33751dc label=  options=rw,relatime,nouuid,attr2,inode64,logbsize=128k,sunit=256,swidth=256,noquota
fs /dev/mapper/docker-202:80-720905-731c75288780b5cd959925096d8cfe2114029091637266b39cabe170605c7223 /app/docker/devicemapper/mnt/731c75288780b5cd959925096d8cfe2114029091637266b39cabe170605c7223 xfs uuid=b382d6a5-9147-47b2-8c3a-6ef7c33751dc label=  options=rw,relatime,nouuid,attr2,inode64,logbsize=128k,sunit=256,swidth=256,noquota
fs /dev/mapper/docker-202:80-720905-73dc5c60598e25b2e13539e816fe747ef18b7797fe0187e5049a9fea68ab90ac /app/docker/devicemapper/mnt/73dc5c60598e25b2e13539e816fe747ef18b7797fe0187e5049a9fea68ab90ac xfs uuid=b382d6a5-9147-47b2-8c3a-6ef7c33751dc label=  options=rw,relatime,nouuid,attr2,inode64,logbsize=128k,sunit=256,swidth=256,noquota
fs /dev/mapper/docker-202:80-720905-903969c07c44bbf949ef7b4b24fbd6fcf1ebbcfbf798c79ad415b03bd1eb1a27 /app/docker/devicemapper/mnt/903969c07c44bbf949ef7b4b24fbd6fcf1ebbcfbf798c79ad415b03bd1eb1a27 xfs uuid=b382d6a5-9147-47b2-8c3a-6ef7c33751dc label=  options=rw,relatime,nouuid,attr2,inode64,logbsize=128k,sunit=256,swidth=256,noquota
fs /dev/mapper/docker-202:80-720905-a238e6a06ab4820134e95a6924eb5a8ff93fbdc1be9cd7bb9f01b6814f147c6d /app/docker/devicemapper/mnt/a238e6a06ab4820134e95a6924eb5a8ff93fbdc1be9cd7bb9f01b6814f147c6d xfs uuid=b382d6a5-9147-47b2-8c3a-6ef7c33751dc label=  options=rw,relatime,nouuid,attr2,inode64,logbsize=128k,sunit=256,swidth=256,noquota
fs /dev/mapper/docker-202:80-720905-b23bc01d6b87dfe523aa92cbf822e9c111f1610665b122291b3f884ef6a2f4e8 /app/docker/devicemapper/mnt/b23bc01d6b87dfe523aa92cbf822e9c111f1610665b122291b3f884ef6a2f4e8 xfs uuid=b382d6a5-9147-47b2-8c3a-6ef7c33751dc label=  options=rw,relatime,nouuid,attr2,inode64,logbsize=128k,sunit=256,swidth=256,noquota
fs /dev/mapper/docker-202:80-720905-c369cf523f69144ec83d35ed9ae49f02cb088927a5d19824cd7733706199d4ea /app/docker/devicemapper/mnt/c369cf523f69144ec83d35ed9ae49f02cb088927a5d19824cd7733706199d4ea xfs uuid=b382d6a5-9147-47b2-8c3a-6ef7c33751dc label=  options=rw,relatime,nouuid,attr2,inode64,logbsize=128k,sunit=256,swidth=256,noquota
fs /dev/mapper/docker-202:80-720905-d21a02d860b705dccf3b79c57ffcf0a863d13ddfe061d40a0d967a4f5bc2169e /app/docker/devicemapper/mnt/d21a02d860b705dccf3b79c57ffcf0a863d13ddfe061d40a0d967a4f5bc2169e xfs uuid=b382d6a5-9147-47b2-8c3a-6ef7c33751dc label=  options=rw,relatime,nouuid,attr2,inode64,logbsize=128k,sunit=256,swidth=256,noquota
fs /dev/xvda2 / xfs uuid=de4def96-ff72-4eb9-ad5e-0847257d1866 label=  options=rw,relatime,attr2,inode64,noquota
fs /dev/xvdf /app ext3 uuid=86eaf035-485c-483d-a44e-cf8c4c8a0868 label= blocksize=4096 reserved_blocks=4% max_mounts=-1 check_interval=0d bytes_per_inode=16384 default_mount_options=user_xattr,acl options=rw,relatime,data=ordered
# Swap partitions or swap files
# Format: swap <filename> uuid=<uuid> label=<label>

Work-around, if any:

jsmeix commented at 2018-03-08 10:24:¶

Only FYI:
We had already https://github.com/rear/rear/issues/1496
Error during layout recreation with btrfs and docker
where docker stuff appears as lots of btrfs subvolumes
under /var/lib/docker/btrfs/subvolumes/...
which was fixed by @OliverO2 via
https://github.com/rear/rear/pull/1497

jsmeix commented at 2018-03-19 09:05:¶

I assume since https://github.com/rear/rear/pull/1757 is merged
this issue is "fixed/solved/done" and can be closed.
@gdha if not just reopen it.

gdha commented at 2020-04-14 15:19:¶

We still see messages like:

2020-04-14 16:21:23.170257675 /dev/mapper/vg00-lv_audit is mounted below /app/gtsc/docker (mount point /var/log/audit is under docker control), skipping.

but which are incorrect. The savelayout does contain:

lvmvol /dev/vg00 lv_audit 4294967296b linear
fs /dev/mapper/vg00-lv_audit /var/log/audit ext3 uuid=8e92b10f-1065-46ff-ae41-eb9dcf594a25 label= blocksize=4096 reserved_blocks=4% max_mounts=-1 check_interval=0d bytes_per_inode=16384 default_mount_options=user_xattr,acl options=rw,relatime,data=ordered

The wrong assumption happens in script /usr/share/rear/layout/save/GNU/Linux/230_filesystem_layout.sh

        if is_true $docker_is_running ; then
            # docker daemon/service is running
            docker_root_dir=$( docker info 2>/dev/null | grep 'Docker Root Dir' | awk '{print $4}' )
            # If $docker_root_dir is in the beginning of the $mountpoint string then FS is under docker control
            # and we better exclude from saving the layout,
            # see https://github.com/rear/rear/issues/1749
            Log "$device is mounted below $docker_root_dir (mount point $mountpoint is under docker control), skipping."
            echo "$mountpoint" | grep -q "^$docker_root_dir" && continue
        fi

The Log should only happen when we find a hit on docker_root_dir.
ReaR 2.5 has the issue, but also the official RHEL rear-2.4-10.el7_7.x86_64 RPM has been bitten by it.
@rmetrich is this something you could fix in the rear-2.4 branch of RHEL?

rmetrich commented at 2020-04-14 15:46:¶

@pcahyna I'm sorry I'm under the sea since a few months. Could you have a look at what is reported here?

gdha commented at 2020-04-14 16:00:¶

@rmetrich Under the sea? What do you mean (just for my knowledge if I can still bother you or not)?

rmetrich commented at 2020-04-14 16:04:¶

Well I'm in a deep dive under the water :-) and there is lots of water

pcahyna commented at 2020-04-14 16:40:¶

Is it related to #1989 (fixed by #2021)? A fix for that one was released with RHEL 7.8 two weeks ago. @yontalcar can you please have a look?

gdha commented at 2020-04-16 08:56:¶

Code is fixed in master branch.
@pcahyna @yontalcar The only remaining question is now will RedHat back-port this to RHEL 7.7 as well?

gdha commented at 2020-04-20 10:25:¶

@pcahyna @yontalcar Found a minor bug in the script 230_filesystem_layout.sh which also exclude the mount point docker_root_dir itself, which should not be the case but only the mountpoints beneath docker_root_dir.
I think this fix should be inlcuded in your update to the rear package in rhel 7.7, 7.8 and 8.x

jsmeix commented at 2020-04-29 15:46:¶

I think this isue is fixed by
https://github.com/rear/rear/pull/2373
so that it can be closed.
Feel free to reopen it if I am wrong.

[Export of Github issue for rear/rear.]

#1749 Issue closed: docker hosts would better exclude the file systems under control by docker¶