#2983 Issue open: remote webdav cloud storage not applicable for TMPDIR on low HD space?¶
Labels: support / question, special hardware or VM
andreasberner opened issue at 2023-05-09 07:49:¶
-
ReaR version ("/usr/sbin/rear -V"):
2.7 -
OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):
Debian 11.13 -
ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
TMPDIR=/mnt/hidrive/tmp
OUTPUT=ISO
BACKUP=NETFS
// OUTPUT_URL=sshfs://andreas@berner-telecom.de/home/andreas/share/
OUTPUT_URL=file:///mnt/hidrive/primus0
# Integrate Backup into the ISO
BACKUP_URL=iso:///backup/
# Wegen Restore Problem auf Virtueller Maschine bei Netcup
AUTOEXCLUDE_MULTIPATH=n
# BOOT_OVER_SAN=y
-
Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR):
VM from netcup, germany -
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
x86 -
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
I guess BIOS -
Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
local HD and remote webdav-Drive from IONOS, Germany, supposed for TMPFILE and -
Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"):
NAME KNAME PKNAME TRAN TYPE FSTYPE LABEL SIZE MOUNTPOINT
/dev/sr0 /dev/sr0 ata rom 1024M
/dev/vda /dev/vda disk 39G
|-/dev/vda1 /dev/vda1 /dev/vda part ext4 38.6G /
`-/dev/vda2 /dev/vda2 /dev/vda part swap 467M [SWAP]
- Description of the issue (ideally so that others can reproduce it):
I would like to make an ISO file with a complete backup over the net to
a remote clould storage.
The cloud is accessable via webdav protocoll.
During the backup process the internal HD runs out of space because the
iso file is created locally and subsequently copied to the backup
target.
I tried to overcome this shortage by changing the TMPDIR to the webdav
drive by using
TMPDIR=/mnt/hidrive/tmp
but
# rear -v mkbackup
leads to errors like
Script 'default/005_verify_os_conf.sh' without leading 3-digit number 'NNN_' is likely run in wrong order
Script 'default/010_EFISTUB_check.sh' without leading 3-digit number 'NNN_' is l ikely run in wrong order
Script 'default/010_set_drlm_env.sh' without leading 3-digit number 'NNN_' is li kely run in wrong order
and then the backup aborts.
What can I do to solve this problem of low local HD-space?
- Workaround, if any:
jsmeix commented at 2023-05-11 10:32:¶
@andreasberner
TMPDIR=/mnt/hidrive/tmp
and also
export TMPDIR=/mnt/hidrive/tmp
in etc/rear/local.conf do not work because usr/sbin/rear
creates its temporary working area (BUILD_DIR) at
https://github.com/rear/rear/blob/master/usr/sbin/rear#L505
before the config files get sourced later at
https://github.com/rear/rear/blob/master/usr/sbin/rear#L598
How to set TMPDIR properly see the description in
usr/share/rear/conf/default.conf
(excerpts)
# TMPDIR
#
# Relax-and-Recover needs a (temporary) working area where it builds in particular
# the rescue/recovery system ISO image (and perhaps even stores the backup archive).
...
# To have a specific working area directory prefix for Relax-and-Recover call
# export TMPDIR="/prefix/for/rear/working/directory"
# before calling 'rear' (/prefix/for/rear/working/directory must already exist).
For me it works as described:
# mount -v /dev/sda6 /other
# mkdir /other/tmp
# export TMPDIR=/other/tmp
# usr/sbin/rear -D mkrescue
Relax-and-Recover 2.7 / Git
Running rear mkrescue (PID 26919 date 2023-05-11 12:19:23)
Command line options: usr/sbin/rear -D mkrescue
Using log file: /root/rear.github.master/var/log/rear/rear-linux-h9wr.log
Using build area: /other/tmp/rear.TxZuJW4iSV3Cwpd
...
But I do not have a "remote webdav-Drive from IONOS"
to test how ReaR behaves with that.
The error messages you get like
Script 'default/005_verify_os_conf.sh' without leading 3-digit number 'NNN_' ...
do not make sense because the script '005_verify_os_conf.sh'
has a leading 3-digit number 'NNN_'.
The code that does this is in
usr/share/rear/lib/framework-functions.sh
https://github.com/rear/rear/blob/master/usr/share/rear/lib/framework-functions.sh#L128
For me this code works in ReaR and also on command line
# script='default/005_verify_os_conf.sh'
# grep -q '^[0-9][0-9][0-9]_' <<< $( basename $script ) || echo "Script '$script' without leading 3-digit number 'NNN_'"
[no output]
# script='default/05_verify_os_conf.sh'
# grep -q '^[0-9][0-9][0-9]_' <<< $( basename $script ) || echo "Script '$script' without leading 3-digit number 'NNN_'"
Script 'default/05_verify_os_conf.sh' without leading 3-digit number 'NNN_'
# script='default/0500_verify_os_conf.sh'
# grep -q '^[0-9][0-9][0-9]_' <<< $( basename $script ) || echo "Script '$script' without leading 3-digit number 'NNN_'"
Script 'default/0500_verify_os_conf.sh' without leading 3-digit number 'NNN_'
Usually such inexplicable error messages indicate
that the ReaR installation might be somehow messed up
or that the environment where ReaR is run
is not sufficiently standards compliant.
You may try out the current ReaR upstream GitHub master code
from within a separated directory as a test to find out
if things work better then, see the section
"Testing current ReaR upstream GitHub master code" in
https://en.opensuse.org/SDB:Disaster_Recovery
andreasberner commented at 2023-05-11 16:55:¶
@Johannes
In general, the workaround to set the TMPDIR outside the rear
configuration seems to work.
I also tried to use the most updated code from GITHIB for rear with
the
following results:
-
Without changing the TMPDIR to the mounted partiton, the system begins
to run (and later on runs out of disk space as expected) -
With the TMPDIR pointing to the mounted partition, all kinds of errors
occur and the system aborts
I do suspect that the webdav mounting is not compatible with the rear
software - if any remote mounted at all.
I guess I need to change at least the protocoll to SSHFS but this is
no
current option of the mounted partition.
Thanks at this time for your really professional answer.
Best
Andreas
Am 11.05.2023 um 12:32 schrieb Johannes Meixner:
@andreasberner https://github.com/andreasberner
|TMPDIR=/mnt/hidrive/tmp |
and also
|export TMPDIR=/mnt/hidrive/tmp |
in etc/rear/local.conf do not work because usr/sbin/rear
creates its temporary working area (BUILD_DIR) at
https://github.com/rear/rear/blob/master/usr/sbin/rear#L505
before the config files get sourced later at
https://github.com/rear/rear/blob/master/usr/sbin/rear#L598How to set TMPDIR properly see the description in
usr/share/rear/conf/default.conf
(excerpts)|# TMPDIR # # Relax-and-Recover needs a (temporary) working area where
it builds in particular # the rescue/recovery system ISO image (and
perhaps even stores the backup archive). ... # To have a specific
working area directory prefix for Relax-and-Recover call # export
TMPDIR="/prefix/for/rear/working/directory" # before calling 'rear'
(/prefix/for/rear/working/directory must already exist). |For me it works as described:
|# mount -v /dev/sda6 /other # mkdir /other/tmp # export
TMPDIR=/other/tmp # usr/sbin/rear -D mkrescue Relax-and-Recover 2.7 /
Git Running rear mkrescue (PID 26919 date 2023-05-11 12:19:23) Command
line options: usr/sbin/rear -D mkrescue Using log file:
/root/rear.github.master/var/log/rear/rear-linux-h9wr.log Using build
area: /other/tmp/rear.TxZuJW4iSV3Cwpd ... |But I do not have a "remote webdav-Drive from IONOS"
to test how ReaR behaves with that.The error messages you get like
|Script 'default/005_verify_os_conf.sh' without leading 3-digit number
'NNN_' ... |do not make sense because the script '005_verify_os_conf.sh'
has a leading 3-digit number 'NNN_'.The code that does this is in usr/share/rear/lib/framework-functions.sh
https://github.com/rear/rear/blob/master/usr/share/rear/lib/framework-functions.sh#L128For me this code works in ReaR and also on command line
|# script='default/005_verify_os_conf.sh' # grep -q
'^[0-9][0-9][0-9]' <<< $( basename $script ) || echo "Script
'$script' without leading 3-digit number 'NNN'" [no output] #
script='default/05_verify_os_conf.sh' # grep -q '^[0-9][0-9][0-9]'
<<< $( basename $script ) || echo "Script '$script' without leading
3-digit number 'NNN'" Script 'default/05_verify_os_conf.sh' without
leading 3-digit number 'NNN_' #
script='default/0500_verify_os_conf.sh' # grep -q '^[0-9][0-9][0-9]'
<<< $( basename $script ) || echo "Script '$script' without leading
3-digit number 'NNN'" Script 'default/0500_verify_os_conf.sh' without
leading 3-digit number 'NNN_' |Usually such inexplicable error messages indicate
that the ReaR installation might be somehow messed up
or that the environment where ReaR is run
is not sufficiently standards compliant.You may try out the current ReaR upstream GitHub master code
from within a separated directory as a test to find out
if things work better then, see the section
"Testing current ReaR upstream GitHub master code" in
https://en.opensuse.org/SDB:Disaster_Recovery—
Reply to this email directly, view it on GitHub
https://github.com/rear/rear/issues/2983#issuecomment-1543746833, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AEA7Q5WFSXKWZPNN4POZNQLXFS53ZANCNFSM6AAAAAAX3AKDUA.
You are receiving this because you were mentioned.Message ID:
***@***.***>
jsmeix commented at 2023-05-12 06:49:¶
@andreasberner
because you wrote
"the webdav mounting is not compatible with the rear software"
I think there could be a misunderstanding what I meant with
"the environment where ReaR is run
is not sufficiently standards compliant".
The ReaR software is only bash scripts (and documentation)
so what ReaR (i.e. usr/sbin/rear) does is to run bash.
What bash that is run by ReaR does is to call various
GNU/Linux standard programs as specified in ReaR's bash scripts.
In the end what is done is that various GNU/Linux standard
programs are called (in particular with BACKUP=NETFS where
GNU/Linux standard programs are used for backup and restore
and no special third party backup software is involved).
So when things work without errors (up to "out of space")
with the default TMPDIR '/tmp'
but with the "webdav TMPDIR" all kinds of errors occur,
then it means that with the "webdav TMPDIR"
various GNU/Linux standard programs (including bash)
fail with "all kinds of errors".
You should get the same failures when you call those programs
manually on bash command line when you use the "webdav TMPDIR".
So - as far as I understand it - it is the "webdav TMPDIR"
which is not sufficiently GNU/Linux standards compliant.
Perhaps you can change something how that "remote webdav-Drive"
is mounted (i.e. use some appropriate 'mount' options) to make
the mounted thing sufficiently GNU/Linux standards compliant
to be used as TMPDIR?
A different idea:
What exactly did run out of space with the default TMPDIR?
Relly out of disk space (i.e. physical or virtual harddisk)?
I.e. in your case really out of space on '/dev/vda1'?
Or is it perhaps actually out of memory (RAM plus swap)?
Because nowadays /tmp is often mounted as 'tmpfs'
so /tmp is in RAM (plus swap - '/dev/vda2' in your case).
Check the output of 'df' and 'free' what actually runs out of space.
Because ReaR 2.7 uses by default /var/tmp for its
temporary working area (BUILD_DIR) and because /var/tmp
is normally on the harddisk, it is expected
that really '/dev/vda1' runs out of space.
But if one of the programs that are called by ReaR
uses the default TMPDIR /tmp it may run "out of memory".
When you run ReaR in debug modes (via -d or -D)
then KEEP_BUILD_DIR is automatically set to true,
(see its description in usr/share/rear/conf/default.conf)
which means old ReaR working areas /var/tmp/rear.*
are not removed so you have to manually clean it up.
andreasberner commented at 2023-05-24 05:33:¶
Hi Johannes,
thanks for your response. There seems to be still a problem in using
remote filesystems for the TMPDIR.
While the webdav protocoll sucks with the below given error messages,
a
sshfs-mounted TMPDIR fails with other error messages like described in
When you mention your working example in your reply, can you please
specify details about the mounted filesystem /other?
Especially, is it a local filesystem or a remote filesystem? If
remote,
which protocoll is used for the mounting? NFS, SSHFS ?
Thanks in advance and best regards
Andreas
Am 11.05.2023 um 12:32 schrieb Johannes Meixner:
@andreasberner https://github.com/andreasberner
|TMPDIR=/mnt/hidrive/tmp |
and also
|export TMPDIR=/mnt/hidrive/tmp |
in etc/rear/local.conf do not work because usr/sbin/rear
creates its temporary working area (BUILD_DIR) at
https://github.com/rear/rear/blob/master/usr/sbin/rear#L505
before the config files get sourced later at
https://github.com/rear/rear/blob/master/usr/sbin/rear#L598How to set TMPDIR properly see the description in
usr/share/rear/conf/default.conf
(excerpts)|# TMPDIR # # Relax-and-Recover needs a (temporary) working area where
it builds in particular # the rescue/recovery system ISO image (and
perhaps even stores the backup archive). ... # To have a specific
working area directory prefix for Relax-and-Recover call # export
TMPDIR="/prefix/for/rear/working/directory" # before calling 'rear'
(/prefix/for/rear/working/directory must already exist). |For me it works as described:
|# mount -v /dev/sda6 /other # mkdir /other/tmp # export
TMPDIR=/other/tmp # usr/sbin/rear -D mkrescue Relax-and-Recover 2.7 /
Git Running rear mkrescue (PID 26919 date 2023-05-11 12:19:23) Command
line options: usr/sbin/rear -D mkrescue Using log file:
/root/rear.github.master/var/log/rear/rear-linux-h9wr.log Using build
area: /other/tmp/rear.TxZuJW4iSV3Cwpd ... |But I do not have a "remote webdav-Drive from IONOS"
to test how ReaR behaves with that.The error messages you get like
|Script 'default/005_verify_os_conf.sh' without leading 3-digit number
'NNN_' ... |do not make sense because the script '005_verify_os_conf.sh'
has a leading 3-digit number 'NNN_'.The code that does this is in usr/share/rear/lib/framework-functions.sh
https://github.com/rear/rear/blob/master/usr/share/rear/lib/framework-functions.sh#L128For me this code works in ReaR and also on command line
|# script='default/005_verify_os_conf.sh' # grep -q
'^[0-9][0-9][0-9]' <<< $( basename $script ) || echo "Script
'$script' without leading 3-digit number 'NNN'" [no output] #
script='default/05_verify_os_conf.sh' # grep -q '^[0-9][0-9][0-9]'
<<< $( basename $script ) || echo "Script '$script' without leading
3-digit number 'NNN'" Script 'default/05_verify_os_conf.sh' without
leading 3-digit number 'NNN_' #
script='default/0500_verify_os_conf.sh' # grep -q '^[0-9][0-9][0-9]'
<<< $( basename $script ) || echo "Script '$script' without leading
3-digit number 'NNN'" Script 'default/0500_verify_os_conf.sh' without
leading 3-digit number 'NNN_' |Usually such inexplicable error messages indicate
that the ReaR installation might be somehow messed up
or that the environment where ReaR is run
is not sufficiently standards compliant.You may try out the current ReaR upstream GitHub master code
from within a separated directory as a test to find out
if things work better then, see the section
"Testing current ReaR upstream GitHub master code" in
https://en.opensuse.org/SDB:Disaster_Recovery—
Reply to this email directly, view it on GitHub
https://github.com/rear/rear/issues/2983#issuecomment-1543746833, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AEA7Q5WFSXKWZPNN4POZNQLXFS53ZANCNFSM6AAAAAAX3AKDUA.
You are receiving this because you were mentioned.Message ID:
***@***.***>
jsmeix commented at 2023-05-24 06:43:¶
@andreasberner
what is mounted at /other is
a normal local harddisk partition
with an ext2 filesystem:
# lsblk -io NAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT
NAME TRAN TYPE FSTYPE SIZE MOUNTPOINT
sda sata disk 465.8G
...
|-sda6 part ext2 8G /other
I never tried a remote filesystem for TMPDIR.
In
https://github.com/rear/rear/issues/2983#issuecomment-1543746833
I only liked to show how ReaR works with a different TMPDIR.
andreasberner commented at 2023-05-24 06:48:¶
I see. My problem is that the server to be backed up is a hosted server and the internal HDD is almost full or at least not sufficient remaining capacity to back up the system before copying the result to an other server.
That is why I am looking for an alternate solution to back up directly on a remote HDD.
Any idea how to achieve that?
jsmeix commented at 2023-05-24 07:21:¶
@andreasberner
I know basically nothing about
remote storage and remote filesystems
so I cannot help in this area.
For my tests with ReaR I use almost always
OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://NFS.server.IP.address/path/to/rear/backup
which results on 'NFS.server.IP.address'
in a 'backup/HOSTNAME' directory a 'rear-HOSTNAME.iso'
plus a separated 'backup.tar.gz' which has the advantage
that the backup of the files is directly accessible
and not somehow "hidden" inside the ISO image.
In general I prefer to keep separated things separated,
cf. RFC 1925 item (5).
For ReaR this means I prefer to keep the ReaR recovery system
(i.e. its ISO image) separated from the backup of the files,
cf. "Relax-and-Recover versus backup and restore" in
https://en.opensuse.org/SDB:Disaster_Recovery
[Export of Github issue for rear/rear.]