#1532 Issue closed: Cannot create USB ReaR dir: no space on device¶
Labels: support / question, fixed / solved / done
GlenHenshaw opened issue at 2017-10-12 01:12:¶
Relax-and-Recover (ReaR) Issue Template¶
Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):
-
rear version (/usr/sbin/rear -V):
Relax-and-Recover 2.2 / 2017-07-20
-
OS version (cat /etc/rear/os.conf or lsb_release -a):
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 9.2 (stretch)
Release: 9.2
Codename: stretch -
rear configuration files (cat /etc/rear/site.conf or cat /etc/rear/local.conf):
cat /etc/rear/local.conf:
BACKUP=NETFS OUTPUT=USB USB_DEVICE=/dev/disk/by-uuid/cb17c608-fbf4-4c75-b230-9eaafa372b14
cat /etc/rear/site.conf:
export TMPDIR=/media/bighdd/tmp/rear
-
Are you using legacy BIOS or UEFI boot?
UEFI -
Brief description of the issue:
$ sudo rear mkbackup ERROR: Could not create USB ReaR dir [/media/bighdd/tmp/rear/rear.sEQIyr5CRMAMnod/outputfs/rear/glen-server/20171011.2108] ! Aborting due to an error, check /var/log/rear/rear-glen-server.log for details
Looking at the end of the log file reports:
mkdir: cannot create directory '/media/bighdd/tmp/rear/rear.sEQIyr5CRMAMnod/outputfs/rear/glen-server/20171011.2108': No space left on device 2017-10-11 21:08:58.350146600 ERROR: Could not create USB ReaR dir [/media/bighdd/tmp/rear/rear.sEQIyr5CRMAMnod/outputfs/rear/glen-server/20171011.2108] !
...but /media/bighdd has plenty of space:
$ df /media/bighdd Filesystem 1K-blocks Used Available Use% Mounted on /dev/sdb1 961302560 77093604 835354496 9% /media/bighdd
and it also has plenty of inodes:
$ df -i /media/bighdd Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sdb1 61054976 217236 60837740 1% /media/bighdd
So what gives?
BTW,
I can change TMPDIR to any folder on any storage
device I have, and I get exactly the same error.
jsmeix commented at 2017-10-12 07:48:¶
The code that errors out in ReaR is in
usr/share/rear/output/USB/Linux-i386/300_create_extlinux.sh
USB_REAR_DIR="$BUILD_DIR/outputfs/$USB_PREFIX"
if [ ! -d "$USB_REAR_DIR" ]; then
mkdir -p $v "$USB_REAR_DIR" >/dev/null
StopIfError "Could not create USB ReaR dir [$USB_REAR_DIR] !"
fi
Right now I cannot see how it could be an issue inside ReaR
when the 'mkdir' fails so that from my current point of view
as far as I currently understand the issue, the root cause
is not inside ReaR so that I close the issue accordingly.
Nevertheless you could still add comments to this issue
to provide more information that indicates why in this case
the root cause is actually inside ReaR.
In general:
When a software itself reports an "Error" it shows that
this condition is already handled within the software
and that it is intentionally regarded to be an "Error".
Usually "Error" messages from a software do not mean
a bug inside the software but the opposite:
Something outside of the software is wrong and
that is reported by the software as an "Error".
I.e. usually an issue like "foo aborts with error 'bar'"
is not a bug within foo but something in the environment
where foo runs is wrong or insufficient to run foo so that
all what foo can do is to abort with an error message.
For example think about "foo aborts with 'out of disk space'"
then it is initially unlikely that the root cause is in foo itself.
E.g. the root cause could be too litte space in /tmp/
(or some obscure quota limitation or whatever else) but
foo must create a temporary file there to be able to run
so that all what foo can do in this case is to abort.
Nevertheless an in-depth analysis might later show the root cause
is actually in foo (e.g. if foo runs mad and fills up the disk).
GlenHenshaw commented at 2017-10-12 13:43:¶
I hear what you're saying, but here's the thing: when I run , e.g., a sudo mktemp /media/bighdd manually from the terminal, it succeeds.
If there was some external issue that was causing straightforward errors when attempting to create files on all of my hard drives, I would expect my entire system to exhibit very strange behaviors. But it doesn't. It's a brand-new Debian install, literally less than a week old, and everything else is working fine. The only program that fails to run the way I expect is ReaR.
And, as I indicated, 90% of the space on my 1 Tb hard drive is free, as are 99% of the inodes. And also, as I said above: I have tried to move the ReaR tmp directory around to several different storage devices, and it fails in exactly the same way on all of them. Both hard drives and SSDs.
If you can give me any other ideas for why this might be failing, I'd love to hear them because I'm at a loss.
gozora commented at 2017-10-12 13:53:¶
Hello @GlenHenshaw,
Did you format your USB device ?
I can't tell you from top of my head exact command for UEFI, but it can
be something like this:
rear -v format -- --efi </dev/disk>.
If you did format your disk, can you please attach your ReaR log output
from rear -d -D mkbackup.
Thanks
V.
GlenHenshaw commented at 2017-10-12 14:08:¶
I did format it, and in fact I successfully ran ReaR using the same USB device three days ago. So something changed between Monday and today.
I assume you don't have to reformat the USB device every time you run ReaR using it, right?
The only other thing that has changed on my system is the installation of some packages that, as far as I know, have nothing to do with ReaR: netatalk, python3, some julia packages (DifferentialEquations and NLOpt), some octave packages, and some docs packages.
I'm at work, and can't get the ReaR output file. I will post that this evening.
gozora commented at 2017-10-12 14:13:¶
I assume you don't have to reformat the USB device every time you run ReaR using it, right?
Yes, you need to format it just once ...
If ReaR was working for you before, then @jsmeix was most probably right and this is not a ReaR issue.
Anyhow, I'll take a look on your debug output once you've uploaded it, then hopefully we will know for sure what is actually wrong.
V.
GlenHenshaw commented at 2017-10-12 23:35:¶
OK, here is the command line output:
$ sudo rear -v mkbackup
Relax-and-Recover 2.2 / 2017-07-20
Using log file: /var/log/rear/rear-glen-server.log
Using backup archive '/media/bighdd/tmp/rear.1nvAk65Zt0gsoEc/outputfs/rear/glen-server/20171012.1932/backup.tar.gz'
Creating disk layout
Using guessed bootloader 'GRUB'
Creating root filesystem layout
Copying logfile /var/log/rear/rear-glen-server.log into initramfs as '/tmp/rear-glen-server-partial-2017-10-12T19:33:00-04:00.log'
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Copying all files in /lib*/firmware/
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (76440726 bytes) in 8 seconds
ERROR: Could not create USB ReaR dir [/media/bighdd/tmp/rear.1nvAk65Zt0gsoEc/outputfs/rear/glen-server/20171012.1932] !
Aborting due to an error, check /var/log/rear/rear-glen-server.log for details
The log file is attached.
GlenHenshaw commented at 2017-10-12 23:35:¶
Log file
rear-glen-server.log
GlenHenshaw commented at 2017-10-12 23:42:¶
So it appears to be choking on
mkdir /media/bighdd/tmp/rear.1nvAk65Zt0gsoEc/outputfs/rear/glen-server/20171012.1932
but if I execute that command manually, it succeeds. So I'm kind of stumped about what the problem might be. My best guess at this point is that even though the logfile shows the tmp directory is on my 1 Tb hard drive (/media/bighdd/tmp), maybe that is erroneous and it's actually trying to create the system image on my root partition, in my normal /tmp directory. That volume has only 12 Gb of 30 Gb free.
GlenHenshaw commented at 2017-10-13 00:36:¶
...I took a shot and reformatted my USB drive, and it appears to be working now.
Which confuses me - this drive had already been formatted, and I had used ReaR to make a backup volume on it earlier this week. And the error message also confuses me - even if the USB drive wasn't formatted, why would ReaR hang on trying to make a directory on a completely different drive?
But anyway - it's working now.
gozora commented at 2017-10-13 06:09:¶
Really strange,
Currently there two possible causes on my mind that can cause this.
- Some HW problem with your "bighdd"
- Location of your TMPDIR
To eliminate option no. 1 just try to run backup on some other USB device if issue occurs again.
For option no. 2 I'm not sure how ReaR behaves if TMPDIR is located at destination backup medium. So again if this issue should raise again, just try to change TMPDIR to some location on your local disk.
V.
jsmeix commented at 2017-10-13 07:27:¶
@gozora
many thanks for your valuable help here!
Frankly I had hoped you would have a look here because
you often help much with such "mission impossible" issues :-)
jsmeix commented at 2017-10-13 08:15:¶
@GlenHenshaw
as always sleeping over it helped so that I have now at least
an idea how one could debug such issues in general.
The problem here is that within a "rear mkbackup" run
a command fails that "just works" outside of ReaR
on plain command line.
This behaviour indicates that within the "rear mkbackup" run
ReaR itself changes its environment into something where
that command no longer works which could further indicate
a bug inside ReaR - or the root cause is still outside of ReaR
and the isssue gets only triggered by something in ReaR.
How to debug such issues in general:
Find the ReaR script where it fails, e.g. use the debug options
like "rear -d -D mkbackup" and inspect the log file or
search all ReaR scripts for the error message like
find usr/sbin/rear usr/share/rear/ | xargs grep -l 'error message'
In this case it is the 'mkdir' in
usr/share/rear/output/USB/Linux-i386/300_create_extlinux.sh
USB_REAR_DIR="$BUILD_DIR/outputfs/$USB_PREFIX"
if [ ! -d "$USB_REAR_DIR" ]; then
mkdir -p $v "$USB_REAR_DIR" >/dev/null
StopIfError "Could not create USB ReaR dir [$USB_REAR_DIR] !"
fi
In the ReaR script where the command fails add a
rear_shell
call directly before the command that fails
which would be in this case
USB_REAR_DIR="$BUILD_DIR/outputfs/$USB_PREFIX"
if [ ! -d "$USB_REAR_DIR" ]; then
rear_shell
mkdir -p $v "$USB_REAR_DIR" >/dev/null
StopIfError "Could not create USB ReaR dir [$USB_REAR_DIR] !"
fi
For details about 'rear_shell' see usr/share/rear/lib/linux-functions.sh
Now run "rear -d -D mkbackup" again and it will run a shell
inside the ReaR script environment where the command fails
directly before that command.
In this shell you can now inspect the environment and try
to find out why that command fails in the ReaR script environment.
jsmeix commented at 2017-10-13 08:50:¶
@GlenHenshaw
some blind guesses:
Do you perhaps use btrfs as filesystem for that USB drive?
I ask because btrfs is know to run out of disk space
in traditionally unexpected ways (the reason is the
different way how btrfs actually stores things which
leads to false output of traditional commands that
show free space - one must use btrfs tools for that).
Could it be that you happen to have a 4k
drive?
Cf.
https://github.com/rear/rear/issues/1439
GlenHenshaw commented at 2017-10-13 23:26:¶
Do you perhaps use btrfs as filesystem for that USB drive?
No, in fact it was ext4 (the first time). I will admit that I didn't format it using ReaR, I used GParted. THe second time around I formatted it as ext3 using ReaR.
Could it be that you happen to have a 4k drive?
Well, not that I know of... I have two storage devices. The main device, where my normal /tmp folder resides, is a Samsung EVO3 SSD. The other device, the one that I was trying to use as shown above in the log files, is a WD Gold 1 Tb hard drive I just bought. Both are formatted ext4.
As for the USB drive I'm backing up to, I have no idea. How can you tell?
GlenHenshaw commented at 2017-10-13 23:29:¶
OK, this is getting strange again. I just re-ran ReaR, again making a backup volume on my USB drive. The same drive as yesterday. Backing up using the exact same configuration, of the exact same system - I have literally done nothing to this computer since the last time I used ReaR. And now it fails, giving me the original error message.
It seems that ReaR, in this case, can only back up to a pristine, newly formatted USB drive. Is this expected behavior?
GlenHenshaw commented at 2017-10-13 23:41:¶
So I took jsmeix's advice and inserted a rear_shell command in the script, where it fails. Then, I do:
rear> df /media/bighdd/tmp/rear.jY5PjRNzPMZrcrO/outputfs
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sde1 30788604 30788604 0 100% /media/bighdd/tmp/rear.jY5PjRNzPMZrcrO/outputfs
My HDD isn't on /dev/sde1. It's on /dev/sdb1. The USB I'm backing up to is what is on /dev/sde1. So "outputfs" is a mount point for the target USB drive, which ReaR then uses as the rsync target (or something), yes?
Is this telling me is that ReaR thinks it can't backup over an old existing backup?
jsmeix commented at 2017-10-14 11:31:¶
@GlenHenshaw
without a detailed analsysis from plain looking at your
https://github.com/rear/rear/issues/1532#issuecomment-336591267
I understand it that your original system is on /dev/sdb1
and you like to get the backup written onto /dev/sde1
but on /dev/sde1 there is no space left on device.
If yes, everything works as it should.
You need to use something with already sufficient free space
whereto ReaR can write its data (the ISO image and the backup).
ReaR will not delete any old backup automatically.
You must delete no longer needed old backups on your own,
cf.
https://github.com/rear/rear/issues/800#issuecomment-297126780
and see "Relax-and-Recover versus backup and restore" in
https://en.opensuse.org/SDB:Disaster_Recovery
I would recommend for backup on USB to use something like
BACKUP_URL=usb:///dev/disk/by-label/REAR-000
or
USB_DEVICE=/dev/disk/by-label/REAR-000
to ensure the specific "rear format" formatted device is used
(and not accidentally anything else).
jsmeix commented at 2017-10-14 11:39:¶
@GlenHenshaw
see also
https://github.com/rear/rear/issues/1164
and
https://github.com/rear/rear/pull/1165
and see USB_SUFFIX in default.conf
https://raw.githubusercontent.com/rear/rear/master/usr/share/rear/conf/default.conf
I guess with an appropriate USB_SUFFIX setting
things will work for you as you expect.
GlenHenshaw commented at 2017-10-14 14:03:¶
OK. This makes sense now. I will set USB_SUFFIX in my local.conf to a fixed value, so I only ever get one copy of the restore volume, and a new version will overwrite the old one. That is exactly what I want.
I would recommend for backup on USB to use something like
BACKUP_URL=usb:///dev/disk/by-label/REAR-000
Yes. I had been doing it by UUID, but that changes every time you reformat the device, and is an opportunity for error.
Thanks for the help guys.
GlenHenshaw commented at 2017-10-14 16:19:¶
With that change, it seems to be working as expected. Again, thanks for all the help.
gozora commented at 2017-10-14 17:42:¶
Just a short add from my site.
When using USB backup with EFI, rear format (by default) creates 2
partitions.
- EFI system partition (ESP) for booting (I guess you've wrongly used this partition for storing backups)
- for backup storage
You can change default configuration options (filesystem type, partition sizes, ESP size ...) by overriding appropriate values in default.conf
V.
GlenHenshaw commented at 2017-10-14 20:14:¶
Well, at this point all I'm doing is a rear format followed by a
rear mkbackup. My local.conf is:
BACKUP=NETFS
OUTPUT=USB
USB_DEVICE=/dev/disk/by-label/REAR-000
USB_SUFFIX=CURRENT
Am I using the partitions wrong?
gozora commented at 2017-10-14 20:19:¶
Looks good (more/less) ;-)
But try to replace
USB_DEVICE=/dev/disk/by-label/REAR-000
with
BACKUP_URL=usb:///dev/disk/by-label/REAR-000
V.
GlenHenshaw commented at 2017-10-14 23:12:¶
OK. Looking at the local.conf documentation, there doesn't seem to be a difference. What's the difference?
gozora commented at 2017-10-15 07:40:¶
Reading default.conf
# The device to use, set automatically by BACKUP=NETFS and BACKUP_URL=usb:///dev/sdb1
USB_DEVICE=
and I'm not 100% sure, but BACKUP_URL is mandatory for rear recover
V.
jsmeix commented at 2017-10-16 07:51:¶
@GlenHenshaw
FYI an addedum regarding your
https://github.com/rear/rear/issues/1532#issuecomment-336591267
where it seems you use a disk that is already 100% full.
When a disk is 100% full, I would assume
something that was run before had badly failed
because I think in practice it does not happen
that a disk gets exactly 100% full and everything
is stored correctly on that disk.
I mean when a disk is 100% full, it indicates that
something was not completely stored on that disk.
In particular when you do backups on a disk that gets 100% full
I would assume some backup data is incomplete on that disk.
Bottom line:
Check what went wrong so that your disk got 100% filled up.
jsmeix commented at 2017-10-16 08:03:¶
@gozora
regarding your
https://github.com/rear/rear/issues/1532#issuecomment-336692657
"BACKUP_URL is mandatory"
It is not mandatory in general, see "man rear":
An example to use TSM for backup and ISO for output
would be to add these lines to /etc/rear/local.conf
(no need to define a BACKUP_URL when using
an external backup solution):
BACKUP=TSM
OUTPUT=ISO
But I think BACKUP_URL is "more or less mandatory"
for BACKUP=NETFS, see "man rear":
When using BACKUP=NETFS you should provide the backup target location through the BACKUP_URL variable.
where "should provide" equals "more or less mandatory" ;-)
In general for testing mandatory variables cf.
https://github.com/rear/rear/pull/1440#issuecomment-323736213
GlenHenshaw commented at 2017-10-16 22:56:¶
Bottom line:
Check what went wrong so that your disk got 100% filled up.
The size of my root partition is 30 Gb of which 12 are free. The size of
the USB device I am backing up to is 32 Gb. After a (supposedly)
successful rear mkbackup, the USB target has 31.5 Gb used.
ReaR does not complain about the disk being full, if I run it as described above. But I assume that I should probably invest in a bigger thumb drive...
jsmeix commented at 2017-10-26 10:39:¶
FYI:
With only that in local.conf
OUTPUT=USB USB_DEVICE=/dev/sdb1 BACKUP=NETFS
I get on /dev/sdb1
# mount /dev/sdb1 /tmp/sdb1 # find /tmp/sdb1 /tmp/sdb1 /tmp/sdb1/rear /tmp/sdb1/rear/syslinux.cfg /tmp/sdb1/rear/e205 /tmp/sdb1/rear/e205/20171026.1217 /tmp/sdb1/rear/e205/20171026.1217/syslinux.cfg /tmp/sdb1/rear/e205/20171026.1217/kernel /tmp/sdb1/rear/e205/20171026.1217/rear-e205.log /tmp/sdb1/rear/e205/20171026.1217/initrd.cgz /tmp/sdb1/rear/e205/20171026.1217/backup.log /tmp/sdb1/rear/e205/20171026.1217/backup.tar.gz /tmp/sdb1/e205 /tmp/sdb1/e205/rear-e205.log /tmp/sdb1/e205/VERSION /tmp/sdb1/e205/.lockfile /tmp/sdb1/e205/README /tmp/sdb1/boot /tmp/sdb1/boot/syslinux /tmp/sdb1/boot/syslinux/pci.ids /tmp/sdb1/boot/syslinux/ldlinux.sys /tmp/sdb1/boot/syslinux/config.c32 /tmp/sdb1/boot/syslinux/extlinux.conf /tmp/sdb1/boot/syslinux/poweroff.com /tmp/sdb1/boot/syslinux/disk.c32 /tmp/sdb1/boot/syslinux/cmd.c32 /tmp/sdb1/boot/syslinux/sysdump.c32 /tmp/sdb1/boot/syslinux/chain.c32 /tmp/sdb1/boot/syslinux/hdt.c32 /tmp/sdb1/boot/syslinux/cat.c32 /tmp/sdb1/boot/syslinux/reboot.c32 /tmp/sdb1/boot/syslinux/cpuid.c32 /tmp/sdb1/boot/syslinux/menu.c32 /tmp/sdb1/boot/syslinux/rosh.c32 /tmp/sdb1/boot/syslinux/ls.c32 /tmp/sdb1/boot/syslinux/host.c32 /tmp/sdb1/boot/syslinux/rear.help /tmp/sdb1/boot/syslinux/message /tmp/sdb1/boot/syslinux/vesamenu.c32 /tmp/sdb1/boot/syslinux/lua.c32 /tmp/sdb1/boot/syslinux/kbdmap.c32
i.e. it seems one does not need a BACKUP_URL
in case of BACKUP=NETFS together with OUTPUT=USB
but in this case "rear recover" fails:
RESCUE e205:~ # rear -d -D recover Relax-and-Recover 2.2 / Git Using log file: /var/log/rear/rear-e205.log Running workflow recover within the ReaR rescue/recovery system ERROR: You must specify either BACKUP_URL or BACKUP_MOUNTCMD and BACKUP_UMOUNTCMD ! Aborting due to an error, check /var/log/rear/rear-e205.log for details
I changed in the recovery sytem /etc/rear/local.conf to
OUTPUT=USB USB_DEVICE=/dev/sdb1 BACKUP=NETFS BACKUP_URL=usb:///dev/sdb1
and now "rear recover" works so that in practice
a BACKUP_URL is actually required.
gdha commented at 2017-10-26 11:23:¶
@jsmeix the man page of rear mentions that BACKUP=NETFS requires a
backup target location through variable BACKUP_URL. I would say the
odd variable is USB_DEVICE=/dev/sdb1, but that had probably historical
reasons (implemented by Dag and Jeroen).
jsmeix commented at 2017-10-26 11:26:¶
@gdha
the man page of rear does not "require"
a BACKUP_URL for BACKUP=NETFS,
it only reads "should provide".
I will correct that.
jsmeix commented at 2017-10-26 11:32:¶
Fixed via
https://github.com/rear/rear/pull/1549/commits/49b534ebb177b7c9a49731b2535bdb001a9b0085
in
https://github.com/rear/rear/pull/1549
jsmeix commented at 2017-10-26 11:50:¶
I verified that with only
OUTPUT=USB BACKUP=NETFS BACKUP_URL=usb:///dev/sdb1
both "rear mkbackup" and "rear recover" work well for me.
With only
OUTPUT=USB USB_DEVICE=/dev/sdb1 BACKUP=NETFS
it errors out now:
# usr/sbin/rear -d -D mkbackup ... ERROR: BACKUP=NETFS requires a BACKUP_URL backup target location Aborting due to an error, check /root/rear.master/var/log/rear/rear-e205.log for details
jsmeix commented at 2017-10-26 11:59:¶
With
https://github.com/rear/rear/pull/1549
merged
this issue should now be completely fixed.
[Export of Github issue for rear/rear.]