#2400 Issue closed: Include mounting nfs capability in rescue ISO to mount Amazon S3 like cloud storage buckets¶
Labels: support / question, fixed / solved / done
ase7en34 opened issue at 2020-05-18 06:50:¶
Relax-and-Recover (ReaR) Issue Template¶
Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):
-
ReaR version ("/usr/sbin/rear -V"):
Relax-and-Recover 2.5 / Git -
OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"):
RedHat 7 -
ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
OUTPUT=ISO
BACKUP=NETFS
OUTPUT_URL=file:///s3fs
BACKUP_URL=file:///s3fs
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/tmp/*' '/var/tmp/*' )
USE_DHCLIENT=yes
COPY_AS_IS=( "${COPY_AS_IS[@]}" /root/.passwd-s3fs )
PROGS=( "${PROGS[@]}" s3fs )
Hello all,
first I would like to tank you for your efforts and for this great piece of code.
I have a following question regarding backups on Amazon S3 like bucked cloud provider.
We have access to cloud storage via buckets. For integrating with ReaR we have installed back-end ( f3fs utility ) which can mount such bucket as local directory. Creating backups are working for now - tar archives created without any errors, however we have difficulties mounting this file system in rescue mode. When trying to mount with s3fs we have got following error:
s3fs: There is no enough disk space for used as cache or (temporary) directory for s3fs
We have tried some methods to solve this like creating of ramfs or tmpfs and mounting in /tmp/ramfs, but without success. Also options like nocache, use_cachedir did not helped us, error remains same.
We know that you are not responsible for s3fs code, but we are wondering if we include mount.nfs in PROGS[@] variable we will be able to mount remote NFS server which already serve mounted via S3FS bucket and this way to complete the recover.
I need to mention that this NFS server is not always switched on and will be started only when recover/migration/newmachine tasks are needed. Also for now there is no way to create template from existing machine and deploy it as new one - therefore we need to use tool like ReaR.
Many thanks in advance.
jsmeix commented at 2020-05-18 15:22:¶
I know nothing about Amazon S3 like cloud storage via buckets.
The ReaR recovery system is minimal so it contains by default
only those programs that are needed to run "rear recover" with the
same etc/rear/local.conf that was used for "rear mkrescue/mkbackup".
The ReaR recovery system runs in main memory via a ramdisk
so it should not be bigger than what is actually needed
to not waste main memory space.
Furthermore the ReaR recovery system is specific for each host
where "rear mkrescue/mkbackup" was run so you need to
store a separated ReaR recovery system for each host
(together with the backup of the files of each host)
so it should not be bigger than what is actually needed
to not waste backup storage space.
With BACKUP_URL=file:// no NFS mount tools get
automatically included in the ReaR recovery system.
In contrast with BACKUP_URL=nfs:// NFS mount tools get
automatically included in the ReaR recovery system.
So I think the simplest and most straightforward way is to
use BACKUP_URL=nfs:// and have your NFS server also
switched on when you run "rear mkrescue/mkbackup".
Then things are consistent during "rear mkrescue/mkbackup"
with what will happen later during "rear recover".
If you really must use "rear mkrescue/mkbackup" in an
environment that is inconsistent with the one where
"rear recover" will happen later, then
you must manually work around those inconsistencies
by manually specifying the additionally needed programs
for your "rear recover" environment via the generic config arrays
COPY_AS_IS, LIBS, PROGS, and REQUIRED_PROGS,
see their description in default.conf in particular
https://github.com/rear/rear/blob/master/usr/share/rear/conf/default.conf#L1334
gdha commented at 2020-05-19 09:40:¶
@ase7en34 Why not prepare a pull request for mounting s3fs buckets? It
can be done similar as we did for sshfs.
Start with the script
prep/NETFS/default/050_check_NETFS_requirements.sh and
lib/global-functions.sh
It is not complicated - just tested it out to mount an AWS s3 bucket
with s3fs.
ase7en34 commented at 2020-05-19 19:32:¶
Hello all,
many thanks for your answers.
@gdha honestly I'm not a script guy - it is too complicated to me with
all that functions and variables. I need to satisfy with what I have.
Meanwhile I managed to include mount.nfs binary in rescue media and
successfully completed the restore.
jsmeix commented at 2020-05-20 07:06:¶
@ase7en34
according to your
https://github.com/rear/rear/issues/2400#issuecomment-631036075
I assume the issue is solved for you so that I can close it hereby.
@gdha
according to what @ase7en34 wrote in his initial comment (excerpts)
PROGS=( "${PROGS[@]}" s3fs )
...
we have difficulties mounting this file system
in rescue mode. When trying to mount
with s3fs we have got following error:
s3fs: There is no enough disk space for used
as cache or (temporary) directory for s3fs
We have tried some methods to solve this
like creating of ramfs or tmpfs and mounting
in /tmp/ramfs, but without success.
Also options like nocache, use_cachedir
did not helped us, error remains same.
it seems it could become a rather tricky task
to prepare the recovery system to successfully
mount Amazon S3 like cloud storage buckets
from within the recovery system.
jsmeix commented at 2020-07-24 11:51:¶
Regarding
"there is no enough disk space for used as cache or (temporary)
directory"
in the ReaR recovery system:
I noticed by chance
restore/GALAXY/default/390_create_ramdisk.sh
https://github.com/rear/rear/blob/master/usr/share/rear/restore/GALAXY/default/390_create_ramdisk.sh
and the two identical
verify/GALAXY7/default/390_create_ramdisk.sh
verify/GALAXY10/default/390_create_ramdisk.sh
https://github.com/rear/rear/blob/master/usr/share/rear/verify/GALAXY10/default/390_create_ramdisk.sh
which are old code (that was not actually maintained since more than 10
years)
but it may at least indicate what one might do to get
"enough disk space for used as cache or (temporary) directory"
in the ReaR recovery system.
There is also
verify/DUPLICITY/default/200_check_tmpfs.sh
https://github.com/rear/rear/blob/master/usr/share/rear/verify/DUPLICITY/default/200_check_tmpfs.sh
ase7en34 commented at 2020-07-24 12:08:¶
Regarding
"there is no enough disk space for used as cache or (temporary) directory"
in the ReaR recovery system:I noticed by chance
restore/GALAXY/default/390_create_ramdisk.sh
https://github.com/rear/rear/blob/master/usr/share/rear/restore/GALAXY/default/390_create_ramdisk.sh
and the two identical
verify/GALAXY7/default/390_create_ramdisk.sh
verify/GALAXY10/default/390_create_ramdisk.sh
https://github.com/rear/rear/blob/master/usr/share/rear/verify/GALAXY10/default/390_create_ramdisk.sh
which are old code (that was not actually maintained since more than 10 years)
but it may at least indicate what one might do to get
"enough disk space for used as cache or (temporary) directory"
in the ReaR recovery system.There is also
verify/DUPLICITY/default/200_check_tmpfs.sh
https://github.com/rear/rear/blob/master/usr/share/rear/verify/DUPLICITY/default/200_check_tmpfs.sh
Hello Johannes,
thank you for this update. However I mentioned in my first post that creating /ramfs or /tmpfs in memory did not solve the problem:
----Quote-----
We have tried some methods to solve this like creating of ramfs or tmpfs and mounting in /tmp/ramfs, but without success. Also options like nocache, use_cachedir did not helped us, error remains same.
----EndQuote-----
jsmeix commented at 2020-07-24 12:15:¶
@ase7en34
ah - yes,
thank you to remind my lossy mind ;-)
[Export of Github issue for rear/rear.]