#2767 Issue `closed`: Suddenly "no space left on device" where EFI partition is¶

Labels: support / question, fixed / solved / done

LiamFry opened issue at 2022-03-08 01:27:¶

Relax-and-Recover 2.6 / 2020-06-17
Zorin OS 15.3 (bionic)

Config:
OUTPUT=USB
BACKUP=NETFS
BACKUP_URL=usb:///dev/disk/by-label/REAR-000
PC hardware, x86 compatible, UEFI
grub-install (GRUB) 2.02-2ubuntu8.21+zorin1

My system's primary (and only) HD is a Samsung EVO 500GB NVMe

I successfully perform at format:

rear -v format -- --efi /dev/sdb

The process fails when I try to create a rescue image:
rear -v mkrescue

Created initrd.cgz with gzip default compression (434040431 bytes) in 33 seconds
ERROR: Failed to copy initrd to /tmp/rear-efi.A0Iu8RqU9F//EFI/BOOT/initrd.cgz
Some latest log messages since the last called script 100_create_efiboot.sh:
  2022-03-07 19:59:58.841068953 Including output/USB/Linux-i386/100_create_efiboot.sh
  2022-03-07 19:59:58.842115763 Configuring device for EFI boot
  '/boot/efi/EFI/ubuntu/grubx64.efi' -> '/tmp/rear-efi.A0Iu8RqU9F//EFI/BOOT/BOOTX64.efi'
  '/boot/vmlinuz-5.4.0-100-generic' -> '/tmp/rear-efi.A0Iu8RqU9F//EFI/BOOT/kernel'
  '/tmp/rear.mZskxippnJnLTyE/tmp/initrd.cgz' -> '/tmp/rear-efi.A0Iu8RqU9F//EFI/BOOT/initrd.cgz'
  cp: error writing '/tmp/rear-efi.A0Iu8RqU9F//EFI/BOOT/initrd.cgz': No space left on device
Aborting due to an error, check /var/log/rear/rear-murkor.log for details
Exiting rear mkrescue (PID 10445) and its descendant processes ...
Running exit tasks
Terminated

The log was of no additional help:

2022-03-07 19:59:23.999291962 Including build/default/995_md5sums_rootfs.sh
2022-03-07 19:59:24.004090170 Creating md5sums for regular files in /tmp/rear.mZskxippnJnLTyE/rootfs
/tmp/rear.mZskxippnJnLTyE/rootfs /tmp/rear.mZskxippnJnLTyE/rootfs/usr/lib/systemd
/tmp/rear.mZskxippnJnLTyE/rootfs/usr/lib/systemd
2022-03-07 19:59:25.535852330 Finished running 'build' stage in 40 seconds
2022-03-07 19:59:25.536779503 ======================
2022-03-07 19:59:25.537621873 Running 'pack' stage
2022-03-07 19:59:25.538423269 ======================
2022-03-07 19:59:25.543198416 Including pack/GNU/Linux/900_create_initramfs.sh
2022-03-07 19:59:25.545746981 Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
2022-03-07 19:59:58.828180659 Created initrd.cgz with gzip default compression (434040431 bytes) in 33 seconds
2022-03-07 19:59:58.829437108 Finished running 'pack' stage in 33 seconds
2022-03-07 19:59:58.830315365 ======================
2022-03-07 19:59:58.831124122 Running 'output' stage
2022-03-07 19:59:58.831996511 ======================
2022-03-07 19:59:58.837008292 Including output/default/010_set_umask.sh
2022-03-07 19:59:58.837971296 Setting umask to 077
2022-03-07 19:59:58.841068953 Including output/USB/Linux-i386/100_create_efiboot.sh
2022-03-07 19:59:58.842115763 Configuring device for EFI boot
'/boot/efi/EFI/ubuntu/grubx64.efi' -> '/tmp/rear-efi.A0Iu8RqU9F//EFI/BOOT/BOOTX64.efi'
'/boot/vmlinuz-5.4.0-100-generic' -> '/tmp/rear-efi.A0Iu8RqU9F//EFI/BOOT/kernel'
'/tmp/rear.mZskxippnJnLTyE/tmp/initrd.cgz' -> '/tmp/rear-efi.A0Iu8RqU9F//EFI/BOOT/initrd.cgz'
cp: error writing '/tmp/rear-efi.A0Iu8RqU9F//EFI/BOOT/initrd.cgz': No space left on device
2022-03-07 19:59:59.156683030 ERROR: Failed to copy initrd to /tmp/rear-efi.A0Iu8RqU9F//EFI/BOOT/initrd.cgz
===== Stack trace =====
Trace 0: /usr/sbin/rear:541 main
Trace 1: /usr/share/rear/lib/mkrescue-workflow.sh:22 WORKFLOW_mkrescue
Trace 2: /usr/share/rear/lib/framework-functions.sh:116 SourceStage
Trace 3: /usr/share/rear/lib/framework-functions.sh:56 Source
Trace 4: /usr/share/rear/output/USB/Linux-i386/100_create_efiboot.sh:38 source
=== End stack trace ===
2022-03-07 19:59:59.160985476 Exiting rear mkrescue (PID 10445) and its descendant processes ...
2022-03-07 20:00:02.186226445 rear,10445 /usr/sbin/rear -v mkrescue
  `-rear,29102 /usr/sbin/rear -v mkrescue
      `-pstree,29103 -Aplau 10445
/usr/share/rear/lib/_input-output-functions.sh: line 151: kill: (29106) - No such process
2022-03-07 20:00:02.200332479 Running exit tasks
2022-03-07 20:00:02.201570407 Finished in 82 seconds
2022-03-07 20:00:02.202672075 Removing build area /tmp/rear.mZskxippnJnLTyE
removed directory '/tmp/rear.mZskxippnJnLTyE'
2022-03-07 20:00:02.470578795 End of program reached

I succeeded in the past when the destination disk was a Crucial MX500 SSD
I tried it today (with that SSD) and it failed with the "no space" error.
I re-ran the format process, tried again, same error.
I then switched to a brand new, never used WD BLUE 1TB SSD
Same error: ... no space

jsmeix commented at 2022-03-08 07:35:¶

The crucial part is

'/tmp/rear.mZskxippnJnLTyE/tmp/initrd.cgz' -> '/tmp/rear-efi.A0Iu8RqU9F//EFI/BOOT/initrd.cgz'
  cp: error writing '/tmp/rear-efi.A0Iu8RqU9F//EFI/BOOT/initrd.cgz': No space left on device

so there is no space left on those device where initrd.cgz should be copied to
and that is where the /tmp/rear-efi.A0Iu8RqU9F//EFI/BOOT/ directory is
which is the device where /tmp/ is.

Nowadays /tmp/ is often on a tmpfs which is usually on RAM,
cf. https://wiki.archlinux.org/title/tmpfs
so no space left on a tmpfs is related to no RAM left.
To check how much free space there is where /tmp/ is you may call

# df -h /tmp

By default up to ReaR 2.6 it uses the traditional Unix/Linux
directory for temporary files which is /tmp/
see the description about TMPDIR in your
usr/share/rear/conf/default.conf file
or in our rear-2.6 branch here online at
https://github.com/rear/rear/blob/rear-2.6/usr/share/rear/conf/default.conf#L44
how you could specify a different directory for temporary files
where there is more space available by setting TMPDIR.

In current ReaR master code (i.e. what will be released as ReaR 2.7)
we use now by default /var/tmp/ which is usually not a tmpfs
but on the system disk where sufficient space should be available,
cf. the current ReaR master code default.conf file online currently at
https://github.com/rear/rear/blob/master/usr/share/rear/conf/default.conf#L44

LiamFry commented at 2022-03-23 22:59:¶

Nope. Doesn't fix it. Exact same error. I've added export TMPDIR="${TMPDIR-/var/tmp}" to my /etc/rear/local.conf file and tried again. Same error at same point in the process. Well, more precisely ...

Testing that the recovery system in /var/tmp/rear.nDmaK1UpGTWmoiL/rootfs contains a usable system
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (434054757 bytes) in 33 seconds
ERROR: Failed to copy initrd to /tmp/rear-efi.gmmioOCmly//EFI/BOOT/initrd.cgz
Some latest log messages since the last called script 100_create_efiboot.sh:
  2022-03-23 18:54:12.418427497 Including output/USB/Linux-i386/100_create_efiboot.sh
  2022-03-23 18:54:12.419493637 Configuring device for EFI boot
  '/boot/efi/EFI/ubuntu/grubx64.efi' -> '/tmp/rear-efi.gmmioOCmly//EFI/BOOT/BOOTX64.efi'
  '/boot/vmlinuz-5.4.0-105-generic' -> '/tmp/rear-efi.gmmioOCmly//EFI/BOOT/kernel'
  '/var/tmp/rear.nDmaK1UpGTWmoiL/tmp/initrd.cgz' -> '/tmp/rear-efi.gmmioOCmly//EFI/BOOT/initrd.cgz'
  cp: error writing '/tmp/rear-efi.gmmioOCmly//EFI/BOOT/initrd.cgz': No space left on device
Aborting due to an error, check /var/log/rear/rear-murkor.log for details
Exiting rear mkrescue (PID 5482) and its descendant processes ...
Running exit tasks
Terminated

Notice how /var/tmp is now used but also notice the copy from /var/tmp to /tmp

jsmeix commented at 2022-03-24 08:10:¶

Ah!
The code where it fails is in
usr/share/rear/output/USB/Linux-i386/100_create_efiboot.sh
(excerpts)

local efi_mpt
efi_mpt=$( mktemp -d $TMPDIR/rear-efi.XXXXXXXXXX ) || Error "mktemp failed to create mount point '$TMPDIR/rear-efi.XXXXXXXXXX' for EFI partition '$efi_part'"

local efi_dir="/EFI/BOOT"
local efi_dst="$efi_mpt/$efi_dir"
.
.
.
# Copy initrd:
cp -L $v "$TMP_DIR/$REAR_INITRD_FILENAME" "$efi_dst/$REAR_INITRD_FILENAME" || Error "Failed to copy initrd to $efi_dst/$REAR_INITRD_FILENAME"

This is from our current GitHub master code.
In ReaR 2.6. it was rather different (in particular hardcoded '/tmp/').

So where it copies to is the mountpoint of an EFI partition
so there is "No space left on device" on the EFI partition on your USB disk.

See in usr/share/rear/conf/default.conf
the descriptions about USB_... config variables in particular USB_UEFI_PART_SIZE
that is currently by default 512 MiB USB_UEFI_PART_SIZE="512"

You could specify e.g. 1 GiB with USB_UEFI_PART_SIZE="1024"
cf. https://github.com/rear/rear/pull/2774
and format again your USB disk with rear -v format -- --efi /dev/sdb
which is the clean way.

Alternatively you may try to reduce the size of the ReaR initrd
(the ReaR initrd contains the ReaR recovery system)
with things like

FIRMWARE_FILES=( 'no' )
MODULES=( 'loaded_modules' )

(see the descriptions in usr/share/rear/conf/default.conf)
but that makes your ReaR recovery system less generally usable
up to completely unusable when mandatory firmware files are missing
that are needed on your particular hardware or when you try to
recreate your system on compatible but not identical replacement hardware
then mandatory kernel modules for the replacement hardware could be missing
cf. https://github.com/rear/rear/discussions/2640#discussioncomment-908335

[Export of Github issue for rear/rear.]

#2767 Issue closed: Suddenly "no space left on device" where EFI partition is¶

LiamFry opened issue at 2022-03-08 01:27:¶

jsmeix commented at 2022-03-08 07:35:¶

LiamFry commented at 2022-03-23 22:59:¶

jsmeix commented at 2022-03-24 08:10:¶

#2767 Issue `closed`: Suddenly "no space left on device" where EFI partition is¶