#2365 Issue closed: Longhorn iscsi PVC devices should be excluded from savelayout¶
Labels: enhancement, bug, fixed / solved / done
gdha opened issue at 2020-04-14 15:56:¶
-
ReaR version ("/usr/sbin/rear -V"): 2.4 and higher
-
OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"): RHEL 7.7 (rear-2.4-10.el7_7.x86_64)
-
ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
BACKUP=NETFS
BACKUP_URL=nfs://NAS/vol/NAS_linux/linux_images_1/itsgbhhlsp01720
BACKUP_PROG_EXCLUDE=( ${BACKUP_PROG_EXCLUDE[@]} '/app/gtsc/docker/*' )
NETFS_PREFIX=image
NETFS_KEEP_OLD_BACKUP_COPY=yes
AUTOEXCLUDE_DISKS=no
-
Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR):
VMware vSphere -
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
x86_64 -
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot): BIOS
-
Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe): scsi
-
Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT" or "lsblk" as makeshift):
#-> lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT
NAME KNAME PKNAME TRAN TYPE FSTYPE SIZE MOUNTPOINT
/dev/sda /dev/sda disk 45G
|-/dev/sda1 /dev/sda1 /dev/sda part ext3 512M /boot
`-/dev/sda2 /dev/sda2 /dev/sda part LVM2_member 44.5G
|-/dev/mapper/vg00-lv_root /dev/dm-0 /dev/sda2 lvm ext3 8G /
|-/dev/mapper/vg00-swap /dev/dm-1 /dev/sda2 lvm swap 4G [SWAP]
|-/dev/mapper/vg00-lv_home /dev/dm-4 /dev/sda2 lvm ext3 4G /home
|-/dev/mapper/vg00-lv_audit /dev/dm-5 /dev/sda2 lvm ext3 4G /var/log/audit
|-/dev/mapper/vg00-lv_log /dev/dm-6 /dev/sda2 lvm ext3 4G /var/log
|-/dev/mapper/vg00-lv_var /dev/dm-7 /dev/sda2 lvm ext3 8G /var
|-/dev/mapper/vg00-lv_tmp /dev/dm-8 /dev/sda2 lvm ext3 2G /var/tmp
`-/dev/mapper/vg00-lv_openv /dev/dm-9 /dev/sda2 lvm ext3 5.2G /usr/openv
/dev/sdb /dev/sdb disk LVM2_member 250G
`-/dev/mapper/vg01-lv00 /dev/dm-2 /dev/sdb lvm ext3 250G /app/util
/dev/sdc /dev/sdc disk LVM2_member 260G
`-/dev/mapper/vg02-lv00 /dev/dm-3 /dev/sdc lvm ext3 254G /app/gtsc
/dev/sdd /dev/sdd disk xfs 400G /app/gtsc/docker
/dev/sde /dev/sde iscsi disk ext4 10G /var/lib/kubelet/pods/61ed399a-d51b-40b8-8fe8-a78e84a1dd0b/volumes/kubernetes.io~csi/pvc-c65df331-f1c5-466a-9731-b2aa5e6da714/mount
/dev/sr0 /dev/sr0 sata rom 1024M
- Description of the issue (ideally so that others can reproduce it):
Kubernetes/Rancher 2 Longhorn iscsi devices are not real devices, but
pseudo-devices created on a file system (chosen via configuration) on
top of e.g. /app/gtsc/longhorn/ (in our case).
Sounds complicated, but these kubernetes PVC devices should not be saved
by ReaR.
#-> parted /dev/longhorn/pvc-c65df331-f1c5-466a-9731-b2aa5e6da714 print
Model: IET VIRTUAL-DISK (scsi)
Disk /dev/longhorn/pvc-c65df331-f1c5-466a-9731-b2aa5e6da714: 10.7GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 10.7GB 10.7GB ext4
[root@ITSGBHHLSP01720:/var/lib/rear/layout]#
#-> grep long disk*
diskdeps.conf:fs:/var/lib/kubelet/pods/61ed399a-d51b-40b8-8fe8-a78e84a1dd0b/volumes/kubernetes.io~csi/pvc-c65df331-f1c5-466a-9731-b2aa5e6da714/mount /dev/longhorn/pvc-c65df331-f1c5-466a-9731-b2aa5e6da714
disklayout.conf:fs /dev/longhorn/pvc-c65df331-f1c5-466a-9731-b2aa5e6da714 /var/lib/kubelet/pods/61ed399a-d51b-40b8-8fe8-a78e84a1dd0b/volumes/kubernetes.io~csi/pvc-c65df331-f1c5-466a-9731-b2aa5e6da714/mount ext4 uuid=4fafdd40-a9ae-4b62-8bfb-f29036dbe3b9 label= blocksize=4096 reserved_blocks=0% max_mounts=-1 check_interval=0d bytes_per_inode=16384 default_mount_options=user_xattr,acl options=rw,relatime,data=ordered
- Workaround, if any: none so far
gdha commented at 2020-04-16 08:50:¶
@pcahyna @yontalcar Could you also have a look at this issue please?
jsmeix commented at 2020-04-20 12:23:¶
@gdha
do I understand your initial comment that reads (excerpts)
lsblk ...
...
/dev/sde ... iscsi ... ext4 ... /var/lib/kubelet/pods/61ed399a-d51b-40b8-8fe8-a78e84a1dd0b/volumes/kubernetes.io~csi/pvc-c65df331-f1c5-466a-9731-b2aa5e6da714/mount
...
disklayout.conf:fs /dev/longhorn/pvc-c65df331-f1c5-466a-9731-b2aa5e6da714 /var/lib/kubelet/pods/61ed399a-d51b-40b8-8fe8-a78e84a1dd0b/volumes/kubernetes.io~csi/pvc-c65df331-f1c5-466a-9731-b2aa5e6da714/mount ext4 ...
correctly that the kernel device node /dev/sde is the target of a
symbolic link
named /dev/longhorn/pvc-c65df331-f1c5-466a-9731-b2aa5e6da714
and that thingy is listed in disklayout.conf with its symbolic link
name?
jsmeix commented at 2020-04-21 14:56:¶
For the log the answer to
https://github.com/rear/rear/issues/2365#issuecomment-616518450
is now explained in the code
https://github.com/rear/rear/blob/master/usr/share/rear/layout/save/GNU/Linux/230_filesystem_layout.sh#L118
that reads (excerpts):
In case Longhorn is rebuilding a replica device
it will show up as a pseudo-device and when that is the
case then you would find traces of it in the
/var/lib/rear/layout/disklayout.conf file, which would
break the recovery as Longhorn Engine replica's are
under control of Rancher Longhorn software and these are
rebuild automatically via kubernetes longhorn-engine pods.
...
In normal situations you will find traces of longhorn in the log
saying skipping non-block devices.
For example an output of the 'df' command:
/dev/longhorn/pvc-ed09c0f2-c086-41c8-a38a-76ee8c289792 82045336 4500292 77528660 6% /var/lib/kubelet/pods/7f47aa55-30e2-4e7b-8fec-ec9a1e761352/volumes/kubernetes.io~csi/pvc-ed09c0f2-c086-41c8-a38a-76ee8c289792/mount
lsscsi shows it as:
[34:0:0:0] storage IET Controller 0001 -
[34:0:0:1] disk IET VIRTUAL-DISK 0001 /dev/sdf
ls -l /dev/sdf /dev/longhorn/pvc-ed09c0f2-c086-41c8-a38a-76ee8c289792
brw-rw---- 1 root disk 8, 80 Apr 17 12:02 /dev/sdf
brw-rw---- 1 root root 8, 64 Apr 17 10:36 /dev/longhorn/pvc-ed09c0f2-c086-41c8-a38a-76ee8c289792
and parted says:
parted /dev/longhorn/pvc-ed09c0f2-c086-41c8-a38a-76ee8c289792 print
Model: IET VIRTUAL-DISK (scsi)
Disk /dev/longhorn/pvc-ed09c0f2-c086-41c8-a38a-76ee8c289792: 85.9GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 85.9GB 85.9GB ext4
as result ... we would end up with an entry in the disklayout.conf file:
fs /dev/longhorn/pvc-ed09c0f2-c086-41c8-a38a-76ee8c289792 /var/lib/kubelet/pods/61ed399a-d51b-40b8-8fe8-a78e84a1dd0b/volumes/kubernetes.io~csi/pvc-c65df331-f1c5-466a-9731-b2aa5e6da714/mount ext4 uuid=4fafdd40-a9ae-4b62-8bfb-f29036dbe3b9 label= blocksize=4096 reserved_blocks=0% max_mounts=-1 check_interval=0d bytes_per_inode=16384 default_mount_options=user_xattr,acl options=rw,relatime,data=ordered
jsmeix commented at 2020-04-29 15:47:¶
I think this isue is fixed by
https://github.com/rear/rear/pull/2373
so that it can be closed.
Feel free to reopen it if I am wrong.
[Export of Github issue for rear/rear.]