#2526 Issue closed: Failed to create recovery system (too small ESP on USB device - no space for .../EFI/BOOT/initrd.cgz)

Labels: support / question, fixed / solved / done

gaia opened issue at 2020-11-24 22:03:

Relax-and-Recover (ReaR) Issue Template

  • ReaR version:
    Relax-and-Recover 2.4

  • OS version:
    Debian 10 (buster)

  • ReaR configuration:

OUTPUT=USB
BACKUP=BORG
USB_DEVICE=/dev/disk/by-label/REAR-000
###
USB_DEVICE_PARTED_LABEL=gpt
USB_DEVICE_FILESYSTEM=ext4
USING_UEFI_BOOTLOADER=1
UEFI_BOOTLOADER="/boot/efi/EFI/grubx64.efi"
###
SSH_UNPROTECTED_PRIVATE_KEYS="no"
SSH_FILES="avoid_sensitive_files"
###
EXCLUDE_MOUNTPOINTS=( '/zfs_containers2' '/mnt/storage' '/mnt/containers' '/mnt/usb' '/mnt/bindmounts' '/mnt/storage/swapfile' )
EXCLUDE_MD=( '/dev/md1' )
EXCLUDE_VG=( '/dev/mapper/lvmthin0' '/dev/mapper/lvmthin1' )
###
BORGBACKUP_REPO="/my_borg_backup"
BORGBACKUP_UMASK="0002"
BORGBACKUP_PRUNE_DAILY=7
BORGBACKUP_ENC_TYPE="keyfile"
BORGBACKUP_COMPRESSION="lz4"
BORGBACKUP_SHOW_PROGRESS="yes"
BORGBACKUP_SHOW_STATS="yes"
export BORG_PASSPHRASE='__________'
export BORG_RELOCATED_REPO_ACCESS_IS_OK="yes"
export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK="yes"
###
PROGS_BORG=( )
BACKUP_PROG_INCLUDE=('/etc/pve/' )
COPY_AS_IS_BORG=( '/root/.config/borg/keys/' )
COPY_AS_IS_EXCLUDE=( "${COPY_AS_IS_EXCLUDE[@]}" 'home/*/.cache/*' 'var/cache/*' 'var/tmp/*' 'dev/*' 'media/*' 'proc/*' 'sys/*' 'tmp/*' 'var/run/*' 'var/lock/*' 'var/lib/apt/lists/*' '/var/cache/apt/*' )
COPY_AS_IS=( $SHARE_DIR $VAR_DIR )

  • Hardware:
    Bare Metal Xeon 9th gen

  • System architecture:
    x64

  • Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
    UEFI + Grub2

  • Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
    Spinning Rust + NVME disks

  • Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT" or "lsblk" as makeshift):

  • Description of the issue:

2020-11-24 04:30:34.623833405 ======================
2020-11-24 04:30:34.624841108 Running 'backup' stage
2020-11-24 04:30:34.626411395 ======================
2020-11-24 04:30:35.002765598 Including backup/default/005_valid_backup_methods.sh
2020-11-24 04:30:35.210903671 Including backup/default/010_pre_backup_script.sh
2020-11-24 04:30:35.261015771 Including backup/BORG/default/100_get_suffix.sh
2020-11-24 04:30:35.321998787 Including backup/BORG/default/250_mount_usb.sh
2020-11-24 04:30:35.335243934 Mounting with 'mount -v -o rw,noatime /dev/disk/by-label/REAR-000 /tmp/rear.60xzwC41QJxGTjm/borg_backup'
mount: /dev/sde2 mounted on /tmp/rear.60xzwC41QJxGTjm/borg_backup.
2020-11-24 04:30:35.654866407 Including backup/BORG/default/400_create_include_exclude_files.sh
2020-11-24 04:30:35.839167065 Including backup/BORG/default/500_make_backup.sh
2020-11-24 04:30:35.887096791 Creating archive rear_270 in repository /my_borg_backup
2020-11-24 04:44:10.968795129 ERROR: Failed to create backup
==== Stack trace ====
Trace 0: /usr/sbin/rear:543 main
Trace 1: /usr/share/rear/lib/mkbackuponly-workflow.sh:16 WORKFLOW_mkbackuponly
Trace 2: /usr/share/rear/lib/framework-functions.sh:101 SourceStage
Trace 3: /usr/share/rear/lib/framework-functions.sh:49 Source
Trace 4: /usr/share/rear/backup/BORG/default/500_make_backup.sh:34 source
Trace 5: /usr/share/rear/lib/_input-output-functions.sh:371 StopIfError
Message: Failed to create backup

  • Workaround, if any:
    N/A, or at least IDH. It worked under this config for a while, then it stopped.

  • Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):
    N/A

gozora commented at 2020-11-25 07:51:

From the logs you've provided I can see that your USB Borg repository already contains some previous backups (archives), What I mean is that you've already run some successful backups prior this error. Is that correct?
If so, can it be that your USB disk is experiencing some IO errors/HW problems?
If not, can you please provide full log from rear -d -D mkbackup session ?

Thanks

V.

gaia commented at 2020-11-25 21:52:

Relax-and-Recover 2.4 / Git
Using log file: /var/log/rear/rear.log
Using UEFI Boot Loader for Linux (USING_UEFI_BOOTLOADER=1)
Creating disk layout
Excluding RAID /dev/md1.
Excluding Volume Group /dev/mapper/____.
Excluding Volume Group /dev/mapper/____
Using guessed bootloader 'EFI' (found in first bytes on /dev/sda)
Creating root filesystem layout
Skipping 'vmbr1': not bound to any physical interface.
Cannot include keyboard mappings (no keymaps default directory '')
Using '/boot/efi/EFI/proxmox/grubx64.efi' as UEFI bootloader file
Copying logfile /var/log/rear/rear.log into initramfs as '/tmp/rear-partial-2020-11-25T13:39:29-08:00.log'
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Copying all files in /lib*/firmware/
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (205115772 bytes) in 24 seconds
ERROR: Could not copy /tmp/rear.JsEmyhSq6taUJk5/tmp/initrd.cgz to /tmp/rear-efi.B2USe//EFI/BOOT/initrd.cgz
Aborting due to an error, check /var/log/rear/rear.log for details
Exiting rear mkbackup (PID 12639) and its descendant processes
Running exit tasks
You should also rm -Rf /tmp/rear.JsEmyhSq6taUJk5
Terminated

Note I altered the actual block device names in EXCLUDE_VG, that's why it does not match the other logs.

gozora commented at 2020-11-25 22:04:

Can you attach /var/log/rear/rear-node2.log ?

gaia commented at 2020-11-26 03:15:

the pword to the zip file is your username

thanks

gozora commented at 2020-11-26 07:32:

You need bigger ESP on your USB device:

Excerpt from the log:

++ cp -p -v /tmp/rear.JsEmyhSq6taUJk5/tmp/initrd.cgz /tmp/rear-efi.B2USe//EFI/BOOT/initrd.cgz
'/tmp/rear.JsEmyhSq6taUJk5/tmp/initrd.cgz' -> '/tmp/rear-efi.B2USe//EFI/BOOT/initrd.cgz'
cp: error writing '/tmp/rear-efi.B2USe//EFI/BOOT/initrd.cgz': No space left on device
...

ReaR controls size of ESP partition by USB_UEFI_PART_SIZE variable. Be aware that after updating of USB_UEFI_PART_SIZE, you need to reformat your USB storage (rear format ...), which will obviously cause DATA LOSS.

V.

gdha commented at 2020-11-26 08:08:

@gozora @jsmeix perhaps it would be a nice idea to have something like https://github.com/gdha/upgrade-ux/blob/master/opt/upgrade-ux/scripts/postexecute/default/96_call_for_action_after_preview.sh to scan for errors?

gozora commented at 2020-11-26 08:19:

Hello @gdha,

I personally don't think it is hard to locate error in ReaR log file, but if you think that others might benefit, I don't mind.
Just be aware that running code from 96_call_for_action_after_preview.sh on ReaR debug log still returns lot of errors:

$ ./check_error.sh rear-node2.log | wc -l
236

V.

gdha commented at 2020-11-26 08:22:

Hello @gdha,

I personally don't think it is hard to locate error in ReaR log file, but if you think that others might benefit, I don't mind.
Just be aware that running code from 96_call_for_action_after_preview.sh on ReaR debug log still returns lot of errors:

$ ./check_error.sh rear-node2.log | wc -l
236

V.

That is indeed true, but out of space can be search out of array of the most common errors for example.

jsmeix commented at 2020-11-26 09:17:

ReaR 2.4 is rather old and the terminal output in
https://github.com/rear/rear/issues/2526#issuecomment-733962038
shows that ReaR 2.4. does not have our current ReaR
enhanced Error function behaviour that always shows
Some latest log messages since the last called script ..., cf.
https://github.com/rear/rear/blob/master/usr/share/rear/lib/_input-output-functions.sh#L557

I implemented this enhanced Error function behaviour via
https://github.com/rear/rear/commit/2420688530e84cf5fbee37b715d9e22bc8ba7f2e
and
https://github.com/rear/rear/pull/1877
in ReaR 2.5, see doc/rear-release-notes.txt

Now the Error function shows some last messages of the last sourced
script to the user (issues #1877 #1875)

https://github.com/rear/rear/blob/master/doc/rear-release-notes.txt#L1419

For an example how current ReaR Error function behaviour
looks like on the user's terminal see the rear -v mkrescue output at
https://github.com/rear/rear/issues/2524#issue-749678000

I assume if @gaia would have used current ReaR
there would have been some more useful output about the actual error
on the terminal where "rear mkrescue/mkbackup" was run
and then this issue report would perhaps have not been needed
because the root cause would have been directly visible to the user.

I think this issue can be closed and no further attention is needed.

jsmeix commented at 2020-11-26 10:19:

@gdha
only for the fun of it:
Your script
https://github.com/gdha/upgrade-ux/blob/master/opt/upgrade-ux/scripts/postexecute/default/96_call_for_action_after_preview.sh
is a "no-go" because it contains the forbidden word WARNING, cf.
https://schlomo.schapiro.org/2015/04/warning-is-waste-of-my-time.html
;-))

jsmeix commented at 2020-11-26 12:45:

@gozora
thank you for your time requesting ReaR debug log files
and inspecting them to show the user the root cause!

[Export of Github issue for rear/rear.]