#2700 Issue closed: /var/tmp running out of space during iso creation¶
Labels: support / question, fixed / solved / done
humis988 opened issue at 2021-10-20 20:19:¶
gdha commented at 2021-10-21 04:50:¶
@humis988 please fill in the template, otherwise, we need to close this issue.
humis988 commented at 2021-10-21 13:19:¶
Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):
ReaR version ("/usr/sbin/rear -V"):
Relax-and-Recover 2.6-git.4538.7c42e2b.master
OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):
Oracle Linux Server release 7.9
ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
cat /etc/rear/site.conf
# local rear /etc/rear/site.conf file
# recommended not to change local.conf and to crate site.conf
# when using package management
# see /etc/rear/local.conf
# configured for Rubrik BMR
# iso written to /var/lib/rear/output/
# and also copied to OUTPUT_URL as specified below
OUTPUT=ISO
BACKUP=CDM
OUTPUT_URL=file:///root/restoreinfo
Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM
guest or PowerVM LPAR):
ovm / kvm / hp physical hardware - current testing with kvm
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
x86
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
test vm is bios
Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
kvm host uses fc
Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT"):
** /var below was expanded to facilitate testing
lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT
NAME KNAME PKNAME TRAN TYPE FSTYPE SIZE MOUNTPOINT
/dev/sr0 /dev/sr0 ata rom 1024M
/dev/sda /dev/sda disk 150G
|-/dev/sda2 /dev/sda2 /dev/sda part LVM2_member 96G
| |-/dev/mapper/ol-swap /dev/dm-1 /dev/sda2 lvm swap 16G [SWAP]
| |-/dev/mapper/ol-usr_openv /dev/dm-8 /dev/sda2 lvm xfs 5G /usr/openv
| |-/dev/mapper/ol-tmp /dev/dm-6 /dev/sda2 lvm xfs 16G /tmp
| |-/dev/mapper/ol-var_log /dev/dm-4 /dev/sda2 lvm xfs 5G /var/log
| |-/dev/mapper/ol-home /dev/dm-2 /dev/sda2 lvm xfs 16G /home
| |-/dev/mapper/ol-root /dev/dm-0 /dev/sda2 lvm xfs 16G /
| |-/dev/mapper/ol-opt /dev/dm-7 /dev/sda2 lvm xfs 16G /opt
| |-/dev/mapper/ol-var_log_audit /dev/dm-5 /dev/sda2 lvm xfs 1G /var/log/audit
| `-/dev/mapper/ol-var /dev/dm-3 /dev/sda2 lvm xfs 50G /var
|-/dev/sda3 /dev/sda3 /dev/sda part LVM2_member 53G
| `-/dev/mapper/ol-var /dev/dm-3 /dev/sda3 lvm xfs 50G /var
`-/dev/sda1 /dev/sda1 /dev/sda part xfs 1G /boot
Description of the issue (ideally so that others can reproduce it):
/var running out of space during iso creation
Workaround, if any:
looking for options?
Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):
To paste verbatim text like command output or file content,
include it between a leading and a closing line of three backticks like
jsmeix commented at 2021-10-21 13:30:¶
See "TMPDIR" in usr/share/rear/conf/default.conf that is currently at
https://github.com/rear/rear/blob/master/usr/share/rear/conf/default.conf#L46
jsmeix commented at 2021-10-21 13:33:¶
In general how big the ISO with the ReaR recovery system could get see
https://github.com/rear/rear/discussions/2640#discussioncomment-908335
FYI:
I am not a BACKUP=CDM user
so I cannot reproduce issues that are specific for BACKUP=CDM
nor do I know how things should behave with BACKUP=CDM.
pcahyna commented at 2021-10-21 13:42:¶
/dev/mapper/ol-var /dev/dm-3 /dev/sda3 lvm xfs 50G /var
How much free space do you have on /var? ISOs can be big, but 50G
should be IMO enough.
Concerning the ISO size, I see that you are using Rubrik CDM. This seems
to be particularly prone to generating large rescue images, see
https://github.com/rear/rear/issues/2685#issuecomment-934639011
humis988 commented at 2021-10-21 13:44:¶
Thanks, in that output, /var was expanded for testing.
Servers in question have /var of 5 GB; in that environment /var/tmp grew to ~ 6GB
Sorry for the confusion.
pcahyna commented at 2021-10-21 13:54:¶
in that environment /var/tmp grew to ~ 6GB
I see, 6GB, so that's apparenly not enough. I also think that ReaR
stores the rescue image files to /var multiple times, once the
unpacked content of the ISO (kernel+initrd), and then the resulting ISO
image under /var/lib/rear. Maybe even before this it holds a copy of
the whole unpacked rescue system tree (that the initrd is created from)
under TMPDIR - not sure about this now. Combined with the propensity of
BACKUP=CDM to create big images, it does not seem unlikely to overflow
6GB.
You can store the ISO directly to another location by setting ISO_DIR
in the config file (by default it is set to $VAR_DIR/output). See the
comment /usr/share/rear/conf/default.conf. This will save at least
some space in /var by avoiding a copy of the ISO image there.
humis988 commented at 2021-10-21 14:08:¶
Tmpdir is looking promising - will test with that and return with the result.
Thank you.
humis988 commented at 2021-10-21 20:58:¶
Looks good - quick and a little dirty approximate results below. This provides an answer to my question and will avoid having to expand /var for the time being.
Thank you!
OUTPUT_URL=file:///root/restoreinfo
export TMPDIR="${TMPDIR-/opt/reartmp}"
~ 1 GB on /var
~ 4 GB on /opt
iso ~ 1 GB
with ISO_DIR=:/root/restoreinfo
/var ~ 0 GB
/opt ~ 3.3 GB
iso ~ 1 GB
jsmeix commented at 2021-10-22 09:29:¶
@humis988
thank you for your feedback how you made it work in your specific case.
jsmeix commented at 2022-01-26 13:03:¶
This issue is likely fixed via https://github.com/rear/rear/pull/2747
[Export of Github issue for rear/rear.]