#2621 Issue closed: BTRFS RAID1 backup issues¶
Labels: enhancement, support / question, needs sponsorship,
no-issue-activity
MVKozlov opened issue at 2021-06-02 11:41:¶
Relax-and-Recover 2.6 / 2020-06-17
os
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
site.coinf
OUTPUT=ISO
OUTPUT_URL=cifs://1.2.3.4/Backup$
OUTPUT_OPTIONS="cred=/etc/rear/cifs_credentials"
REQUIRED_PROGS+=( lsblk efibootmgr unshare tune2fs resize2fs nslookup grub-install update-grub ldconfig.real )
BACKUP=NETFS
BACKUP_URL=cifs://1.2.3.4/Backup$
BACKUP_PROG_EXCLUDE+=( '/var/lib/pgpro/main/data/base' '/var/lib/pgpro/main_base' )
BACKUP_PROG_INCLUDE+=( '/var/lib/pgpro/main_base/1' )
BACKUP_OPTIONS="cred=/etc/rear/cifs_credentials"
AUTOSHRINK_DISK_SIZE_LIMIT_PERCENTAGE=10
AUTORESIZE_PARTITIONS=""
UEFI
lsblk
NAME KNAME PKNAME TRAN TYPE FSTYPE SIZE MOUNTPOINT /dev/sda /dev/sda sata disk 894.3G `-/dev/sda1 /dev/sda1 /dev/sda part btrfs 894.3G /btrfs/ssd /dev/sdb /dev/sdb sata disk 894.3G `-/dev/sdb1 /dev/sdb1 /dev/sdb part btrfs 894.3G /dev/sdc /dev/sdc sata disk 5.5T |-/dev/sdc1 /dev/sdc1 /dev/sdc part vfat 512M |-/dev/sdc2 /dev/sdc2 /dev/sdc part linux_raid_member 197G | `-/dev/md0 /dev/md0 /dev/sdc2 raid1 196.9G | |-/dev/md0p1 /dev/md0p1 /dev/md0 part ext4 1G /boot | |-/dev/md0p2 /dev/md0p2 /dev/md0 part ext4 64G / | `-/dev/md0p3 /dev/md0p3 /dev/md0 part swap 131.9G [SWAP] `-/dev/sdc3 /dev/sdc3 /dev/sdc part btrfs 5T /btrfs/hdd /dev/sdd /dev/sdd sata disk 5.5T |-/dev/sdd1 /dev/sdd1 /dev/sdd part vfat 512M /boot/efi |-/dev/sdd2 /dev/sdd2 /dev/sdd part linux_raid_member 197G | `-/dev/md0 /dev/md0 /dev/sdd2 raid1 196.9G | |-/dev/md0p1 /dev/md0p1 /dev/md0 part ext4 1G /boot | |-/dev/md0p2 /dev/md0p2 /dev/md0 part ext4 64G / | `-/dev/md0p3 /dev/md0p3 /dev/md0 part swap 131.9G [SWAP] `-/dev/sdd3 /dev/sdd3 /dev/sdd part btrfs 5T /dev/sr0 /dev/sr0 sata rom 1024M /dev/nvme0n1 /dev/nvme0n1 nvme disk 931.5G `-/dev/nvme0n1p1 /dev/nvme0n1p1 /dev/nvme0n1 nvme part btrfs 931.5G /btrfs/nvme /dev/nvme1n1 /dev/nvme1n1 nvme disk 931.5G `-/dev/nvme1n1p1 /dev/nvme1n1p1 /dev/nvme1n1 nvme part btrfs 931.5G
/etc/fstab:
LABEL=raid1nvme /mnt/btrfs/nvme btrfs rw,noatime,defaults 0 0
LABEL=raid1ssd /mnt/btrfs/ssd btrfs rw,noatime,defaults 0 0
LABEL=raid1hdd /mnt/btrfs/hdd btrfs rw,noatime,defaults 0 0
LABEL=raid1ssd /var/lib/pgpro/main btrfs subvol=postgres,rw,noatime,defaults 0 0
LABEL=raid1nvme /var/lib/pgpro/pg_wal btrfs subvol=pg_wal,rw,noatime,defaults 0 0
LABEL=raid1nvme /var/lib/pgpro/main_base btrfs subvol=main_base,rw,noatime,defaults 0 0
LABEL=raid1hdd /opt btrfs subvol=opt,rw,noatime,defaults 0 0
LABEL=raid1hdd /media/backup btrfs subvol=backup,rw,noatime,defaults 0 0
LABEL=raid1hdd /home/usr1 btrfs subvol=usr1,rw,noatime,defaults 0 0
when default btrfs volumes mounted as showed in fstab (under
/mnt/btrfs/xxx)
then ALL brtfs volumes skipped according to rear.log
example:
Disk /dev/nvme0n1 is not used by any mounted filesystem. Excluding.
Disk /dev/nvme1n1 is not used by any mounted filesystem. Excluding.
Disk /dev/sda is not used by any mounted filesystem. Excluding.
Disk /dev/sdb is not used by any mounted filesystem. Excluding.
when default volumes mounted as /btrfs/xxx
then only mirror disk volumes skipped in logs (sdb and nvme1n1)
but anyway nothing from any btrfs fs path go into backup
when I try to restore backup to test VM (add 6 drives sda...sdf)
only sdc and sdd volumes asks for remapping. so any btrfs data not
restored at all
I try to manually add some volumes to remap and layout and restore fails
on disk layout creation
(may be nvme - sdX remap is another problem)
I understand that btrfs support limited but is there any workarounds for full backup/restoration ?
jsmeix commented at 2021-06-02 12:15:¶
A quick reply as far as I know off the top of my head:
Real hardware RAID is supported by ReaR
because underlying lower level things below the RAID do not appear.
Traditional RAID1 via MD devices aka Linux Software RAID is supported by ReaR.
Other forms of RAID like so called "Firmware RAID"
cf.
https://github.com/rear/rear/issues/1540
or using one single btrfs filesystem on multiple devices
cf.
https://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices
are currently not supported by ReaR.
jsmeix commented at 2021-06-02 12:22:¶
Regarding btrfs subvolumes and the backup of the files see
https://github.com/rear/rear/issues/2604#issuecomment-824848943
In general including/excluding components like
disks/partitions/filesystems
is different from what gets included/excluded regarding the backup.
There are some relationships but in general both are different things,
cf.
https://github.com/rear/rear/issues/2586#issuecomment-805601063
MVKozlov commented at 2021-06-02 12:23:¶
disklayout.conf contain uncommented btrfs drives, partitions, fs,
subvolumes and mounted subvolumes and my config contain
BACKUP_PROG_INCLUDE+=( '/var/lib/pgpro/main_base/1' )
if at least the data got into the backup, this would not be a big problem, but they are silently skipped.
I feel it as a bug
jsmeix commented at 2021-06-02 13:05:¶
Regarding the backup with BACKUP=NETFS we have
BACKUP_ONLY_INCLUDE and BACKUP_ONLY_EXCLUDE
that should help to specify explicitly what gets included in the
backup.
See its documentation in the default.conf file
https://github.com/rear/rear/blob/master/usr/share/rear/conf/default.conf
The script that by default (i.e. unless BACKUP_ONLY_EXCLUDE is
'true')
implicitly also adds excluded mountpoints to the backup exclude list
is
backup/NETFS/default/400_create_include_exclude_files.sh
https://github.com/rear/rear/blob/master/usr/share/rear/backup/NETFS/default/400_create_include_exclude_files.sh
and indeed it does that silently.
But what is plobably worse is that the actual backup script
backup/NETFS/default/500_make_backup.sh
https://github.com/rear/rear/blob/master/usr/share/rear/backup/NETFS/default/500_make_backup.sh
only writes the "Backup include list" and the "Backup exclude list"
to the log file so that the user won't see anything on his terminal
also not in verbose -v mode (or in debug modes -d or -D).
MVKozlov commented at 2021-06-02 13:59:¶
thanks, a careful reading of the entire thread # 2604 helped to add all
the data into backup
but it seems simultaneous
BACKUP_PROG_EXCLUDE+=( '/somedirectory' )
BACKUP_PROG_INCLUDE+=( '/somedirectory/subdirectory' )
cannot include /somedirectory/subdirectory' in backup. it not clear from
the doc
for brtfs recreation I try to use
PRE_RECOVERY_SCRIPT=
POST_RECOVERY_SCRIPT=
github-actions commented at 2021-08-02 02:10:¶
Stale issue message
[Export of Github issue for rear/rear.]