#2631 Issue closed: rear mkbackup "ERROR: dd failed with return code 1" because of too little quota on NFS share¶
Labels: support / question, fixed / solved / done
exfarmer opened issue at 2021-06-10 13:11:¶
Relax-and-Recover (ReaR) Issue Template¶
Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):
-
ReaR version ("/usr/sbin/rear -V"):
Relax-and-Recover 2.4 / Git -
OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):
LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: RedHatEnterpriseServer
Description: Red Hat Enterprise Linux Server release 7.8 (Maipo)
Release: 7.8
Codename: Maipo
- ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
OUTPUT=ISO
OUTPUT_URL=nfs://ussvdd3300/data/col1/ussv_linux_bu_image
BACKUP_OPTIONS="nfsvers=3,nolock"
OUTPUT_OPTIONS="nfsvers=3,nolock"
BACKUP=NETFS
BACKUP_URL=nfs://ussvdd3300/data/col1/ussv_linux_bu_image
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/media' '/var/tmp' '/var/crash' '/var/log' '/usr/openv')
ONLY_INCLUDE_VG=( 'VolGroup00' 'rhel' )
NETFS_KEEP_OLD_BACKUP_COPY=y
-
Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR):
Dell Inc. PowerEdge R640 -
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
-
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
3.10.0-1160.25.1.el7.x86_64 -
Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
SAN -
Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT" or "lsblk" as makeshift):
NAME KNAME PKNAME TRAN TYPE FSTYPE SIZE MOUNTPOINT
/dev/sda /dev/sda disk 278G
|-/dev/sda1 /dev/sda1 /dev/sda part vfat 200M /boot/efi
|-/dev/sda2 /dev/sda2 /dev/sda part xfs 1G /boot
`-/dev/sda3 /dev/sda3 /dev/sda part LVM2_member 276.8G
|-/dev/mapper/rhel-root /dev/dm-0 /dev/sda3 lvm xfs 50G /
|-/dev/mapper/rhel-swap /dev/dm-1 /dev/sda3 lvm swap 24G [SWAP]
|-/dev/mapper/rhel-lvdata /dev/dm-2 /dev/sda3 lvm xfs 100G /ichome
`-/dev/mapper/rhel-lvopenv /dev/dm-3 /dev/sda3 lvm xfs 12G /usr/openv
- Description of the issue (ideally so that others can reproduce it):
Creating tar archive '/tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz'
Archived 4694 MiB [avg 5071 KiB/sec] OK
ERROR: dd failed with return code 1 and below output:
---snip---
dd: writing to '/tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz': Input/output error
----------
This means that the archiving process ended prematurely, or did
not even start. As a result it is unlikely you can recover this
system properly. Relax-and-Recover is therefore aborting execution.
Aborting due to an error, check /var/log/rear/rear-ussv-rdhpapp02.log for details
Exiting rear mkbackup (PID 236532) and its descendant processes
Running exit tasks
-
Workaround, if any:
-
Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):
To paste verbatim text like command output or file content,
include it between a leading and a closing line of three backticks like
```
verbatim content
```
pcahyna commented at 2021-06-10 13:25:¶
From a quick look it seems that something was wrong with the NFS mount
(from ussvdd3300).
exfarmer commented at 2021-06-10 13:31:¶
I agree, but the backup starts, the /tmp/rear.KCXi6xCey8b60rU is created: Creating tar archive '/tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz' Archived 4694 MiB [avg 5071 KiB/sec] OK ERROR: dd failed with return code 1 and below output: ---snip--- dd: writing to '/tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz': Input/output error¶
I ran this # rear -D -d mkbackup
Can I run it with more debug information?
I haven’t deleted the /tmp/rear.KCXi6xCey8b60rU file yet, and I can see
the ISO and other files on the target NFS disk:
//ussvdd3300/data/col1/ussv_linux_bu_image/ussv-rdhpapp02.
Any ideas why this would fail?
pcahyna commented at 2021-06-10 13:53:¶
rear -D -d mkbackup
This is enough, is there anything more interesting in the log
(/var/log/rear/rear*.log) before the error message?
Any ideas why this would fail?
No concrete idea, perhaps network problem, or lack of space on the NFS server? Please check the system logs both on the system that you are backing up and on the NFS server, especially for NFS and filesystem related messages.
pcahyna commented at 2021-06-10 14:02:¶
also, you say that you see other files on the target. How big is the (unfinished) backup.tar.gz ?
exfarmer commented at 2021-06-10 14:18:¶
Hi, there’s nothing in dmesg or var/log/messages related to NFS or the
rear backup.
It looks like it finishes the backup, then fails.
See attached
Abbott
[image001]
Gary Hess
Administrator, Sr - Unix
Abbott
335 Financial Blvd.
Liberty, SC 29657 USA
O:
+1 864-843-8352
M:
+1 864-546-8921
E:
***@***.***
From: pcahyna ***@***.***>
Sent: Thursday, June 10, 2021 9:53 AM
To: rear/rear ***@***.***>
Cc: Hess, Gary ***@***.***>; Author
***@***.***>
Subject: Re: [rear/rear] ERROR: dd failed with return code 1 (#2631)
EXTERNAL EMAIL: Only click links or open attachments if you recognize the sender and know the content is safe.
rear -D -d mkbackup
This is enough, is there anything more interesting in the log (`/var/log/rear/rear*.log) before the error message?
Any ideas why this would fail?
No concrete idea, perhaps network problem, or lack of space on the NFS server? Please check the system logs both on the system that you are backing up and on the NFS server, especially for NFS and filesystem related messages.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on
GitHubhttps://urldefense.com/v3/__https:/github.com/rear/rear/issues/2631*issuecomment-858642620__;Iw!!BBM_p3AAtQ!csdBL1Lgt0KXaBQ3qC2IbzjxPKxfYHBk7SktPNlXu2G5syvYXZmTacnRokenEPcJ$,
or
unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/ASRDKRLKXVS2EJNSQLPCP4DTSC7VHANCNFSM46OPZ3TQ__;!!BBM_p3AAtQ!csdBL1Lgt0KXaBQ3qC2IbzjxPKxfYHBk7SktPNlXu2G5syvYXZmTacnRoixJPsZ8$.
+++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4647452672 ++ ProgressInfo 'Archived 4432 MiB [avg 5014 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4432 MiB [avg 5014 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4652204032 ++ ProgressInfo 'Archived 4436 MiB [avg 5014 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4436 MiB [avg 5014 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4656807936 ++ ProgressInfo 'Archived 4441 MiB [avg 5013 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4441 MiB [avg 5013 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4661510144 ++ ProgressInfo 'Archived 4445 MiB [avg 5013 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4445 MiB [avg 5013 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4666081280 ++ ProgressInfo 'Archived 4449 MiB [avg 5012 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4449 MiB [avg 5012 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4670881792 ++ ProgressInfo 'Archived 4454 MiB [avg 5007 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4454 MiB [avg 5007 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4675239936 ++ ProgressInfo 'Archived 4458 MiB [avg 5006 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4458 MiB [avg 5006 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4677025792 ++ ProgressInfo 'Archived 4460 MiB [avg 5002 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4460 MiB [avg 5002 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4680384512 ++ ProgressInfo 'Archived 4463 MiB [avg 5000 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4463 MiB [avg 5000 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4683497472 ++ ProgressInfo 'Archived 4466 MiB [avg 4998 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4466 MiB [avg 4998 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4686413824 ++ ProgressInfo 'Archived 4469 MiB [avg 4996 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4469 MiB [avg 4996 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4689166336 ++ ProgressInfo 'Archived 4471 MiB [avg 4993 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4471 MiB [avg 4993 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4693049344 ++ ProgressInfo 'Archived 4475 MiB [avg 4992 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4475 MiB [avg 4992 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4697653248 ++ ProgressInfo 'Archived 4480 MiB [avg 4991 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4480 MiB [avg 4991 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4702273536 ++ ProgressInfo 'Archived 4484 MiB [avg 4991 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4484 MiB [avg 4991 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4706795520 ++ ProgressInfo 'Archived 4488 MiB [avg 4990 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4488 MiB [avg 4990 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4710203392 ++ ProgressInfo 'Archived 4492 MiB [avg 4988 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4492 MiB [avg 4988 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4713250816 ++ ProgressInfo 'Archived 4494 MiB [avg 4986 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4494 MiB [avg 4986 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4716707840 ++ ProgressInfo 'Archived 4498 MiB [avg 4985 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4498 MiB [avg 4985 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4720214016 ++ ProgressInfo 'Archived 4501 MiB [avg 4983 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4501 MiB [avg 4983 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4723605504 ++ ProgressInfo 'Archived 4504 MiB [avg 4981 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4504 MiB [avg 4981 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4728258560 ++ ProgressInfo 'Archived 4509 MiB [avg 4981 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4509 MiB [avg 4981 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4731846656 ++ ProgressInfo 'Archived 4512 MiB [avg 4974 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4512 MiB [avg 4974 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4734844928 ++ ProgressInfo 'Archived 4515 MiB [avg 4971 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4515 MiB [avg 4971 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4737630208 ++ ProgressInfo 'Archived 4518 MiB [avg 4969 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4518 MiB [avg 4969 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4740464640 ++ ProgressInfo 'Archived 4520 MiB [avg 4967 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4520 MiB [avg 4967 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4743426560 ++ ProgressInfo 'Archived 4523 MiB [avg 4964 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4523 MiB [avg 4964 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4750950400 ++ ProgressInfo 'Archived 4530 MiB [avg 4967 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4530 MiB [avg 4967 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4767531008 ++ ProgressInfo 'Archived 4546 MiB [avg 4979 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4546 MiB [avg 4979 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4783915008 ++ ProgressInfo 'Archived 4562 MiB [avg 4991 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4562 MiB [avg 4991 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4799758336 ++ ProgressInfo 'Archived 4577 MiB [avg 5002 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4577 MiB [avg 5002 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4815666688 ++ ProgressInfo 'Archived 4592 MiB [avg 5013 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4592 MiB [avg 5013 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4829069312 ++ ProgressInfo 'Archived 4605 MiB [avg 5016 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4605 MiB [avg 5016 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4845336576 ++ ProgressInfo 'Archived 4620 MiB [avg 5028 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4620 MiB [avg 5028 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4859641856 ++ ProgressInfo 'Archived 4634 MiB [avg 5037 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4634 MiB [avg 5037 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4874843136 ++ ProgressInfo 'Archived 4649 MiB [avg 5048 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4649 MiB [avg 5048 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4890804224 ++ ProgressInfo 'Archived 4664 MiB [avg 5059 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4664 MiB [avg 5059 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4907055616 ++ ProgressInfo 'Archived 4679 MiB [avg 5070 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4679 MiB [avg 5070 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 +++ awk '{s+=$1} END {print s}' +++ stat -c %s /tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz ++ size=4922940416 ++ ProgressInfo 'Archived 4694 MiB [avg 5071 KiB/sec] ' ++ echo -en '\e[2K\rArchived 4694 MiB [avg 5071 KiB/sec] \e7' ++ sleep 1 ++ kill -0 20200 ++ ProgressStop ++ echo -e '\e8\e[KOK' ++ transfertime=949 ++ wait 20200 ++ backup_prog_rc=2 ++ [[ '' =~ ^[yY1] ]] ++ sleep 1 ++ case "$(basename $BACKUP_PROG)" in +++ basename tar ++ (( 2 != 0 )) +++ cat /tmp/rear.KCXi6xCey8b60rU/tmp/failing_backup_prog ++ prog=dd ++ (( 2 == 1 )) +++ cat /tmp/rear.KCXi6xCey8b60rU/tmp/failing_backup_prog_rc ++ rc=1 +++ grep '^dd: ' /tmp/rear.KCXi6xCey8b60rU/tmp/backup.log +++ sed -e 's/^/ /' +++ tail -n3 ++ Error 'dd failed with return code 1 and below output: ---snip--- dd: writing to '''/tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz''': Input/output error¶
This means that the archiving process ended prematurely, or did not even start. As a result it is unlikely you can recover this system properly. Relax-and-Recover is therefore aborting execution. ' ++ LogPrintError 'ERROR: dd failed with return code 1 and below output: ---snip--- dd: writing to '''/tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz''': Input/output error¶
This means that the archiving process ended prematurely, or did not even start. As a result it is unlikely you can recover this system properly. Relax-and-Recover is therefore aborting execution. ' ++ Log 'ERROR: dd failed with return code 1 and below output: ---snip--- dd: writing to '''/tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz''': Input/output error¶
This means that the archiving process ended prematurely, or did not even start. As a result it is unlikely you can recover this system properly. Relax-and-Recover is therefore aborting execution. ' +++ date '+%Y-%m-%d %H:%M:%S.%N ' ++ local 'timestamp=2021-06-10 05:48:39.222516820 ' ++ test 1 -gt 0 ++ echo '2021-06-10 05:48:39.222516820 ERROR: dd failed with return code 1 and below output: ---snip--- dd: writing to '''/tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz''': Input/output error¶
This means that the archiving process ended prematurely, or did not even start. As a result it is unlikely you can recover this system properly. Relax-and-Recover is therefore aborting execution. ' 2021-06-10 05:48:39.222516820 ERROR: dd failed with return code 1 and below output: ---snip--- dd: writing to '/tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz': Input/output error¶
This means that the archiving process ended prematurely, or did
not even start. As a result it is unlikely you can recover this
system properly. Relax-and-Recover is therefore aborting execution.
++ PrintError 'ERROR: dd failed with return code 1 and below output: ---snip--- dd: writing to '''/tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz''': Input/output error¶
This means that the archiving process ended prematurely, or did not even start. As a result it is unlikely you can recover this system properly. Relax-and-Recover is therefore aborting execution. ' ++ echo -e 'ERROR: dd failed with return code 1 and below output: ---snip--- dd: writing to '''/tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz''': Input/output error¶
This means that the archiving process ended prematurely, or did not even start. As a result it is unlikely you can recover this system properly. Relax-and-Recover is therefore aborting execution. ' ++ LogToSyslog 'ERROR: dd failed with return code 1 and below output: ---snip--- dd: writing to '''/tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz''': Input/output error¶
This means that the archiving process ended prematurely, or did not even start. As a result it is unlikely you can recover this system properly. Relax-and-Recover is therefore aborting execution. ' ++ logger -t rear -i 'ERROR: dd failed with return code 1 and below output: ---snip--- dd: writing to '''/tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz''': Input/output error¶
This means that the archiving process ended prematurely, or did not even start. As a result it is unlikely you can recover this system properly. Relax-and-Recover is therefore aborting execution. ' ++ has_binary caller ++ for bin in '$@' ++ type caller ++ return 0 ++ echo '==== Stack trace ====' ==== Stack trace ==== ++ local c=0 ++ caller 0 ++ awk ' { l[NR]=$3":"$1" "$2 } END { for (i=NR; i>0;) print "Trace "NR-i": "l[i--] } ' ++ : ++ caller 1 ++ : ++ caller 2 ++ : ++ caller 3 ++ : ++ caller 4 ++ : ++ caller 5 Trace 0: /usr/sbin/rear:547 main Trace 1: /usr/share/rear/lib/mkbackup-workflow.sh:26 WORKFLOW_mkbackup Trace 2: /usr/share/rear/lib/framework-functions.sh:101 SourceStage Trace 3: /usr/share/rear/lib/framework-functions.sh:49 Source Trace 4: /usr/share/rear/backup/NETFS/default/500_make_backup.sh:252 source ++ echo 'Message: dd failed with return code 1 and below output: ---snip--- dd: writing to '''/tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz''': Input/output error¶
This means that the archiving process ended prematurely, or did not even start. As a result it is unlikely you can recover this system properly. Relax-and-Recover is therefore aborting execution. ' Message: dd failed with return code 1 and below output: ---snip--- dd: writing to '/tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz': Input/output error¶
This means that the archiving process ended prematurely, or did
not even start. As a result it is unlikely you can recover this
system properly. Relax-and-Recover is therefore aborting execution.
++ echo '== End stack trace =='
== End stack trace ==
++ kill -USR1 236532
+++ EXIT_FAIL_MESSAGE=0
+++ echo 'Aborting due to an error, check
/var/log/rear/rear-ussv-rdhpapp02.log for details'
+++ kill 236532
+++ DoExitTasks
+++ apply_bash_flags_and_options_commands 'set +o xtrace;set +o
vi;set +o verbose;set +o privileged;set +o posix;set +o pipefail;set +o
physical;set +o onecmd;set +o nounset;set +o notify;set +o nolog;set +o
noglob;set +o noexec;set +o noclobber;set +o monitor;set +o keyword;set
-o interactive-comments;set +o ignoreeof;set +o history;set +o
histexpand;set -o hashall;set +o functrace;set +o errtrace;set +o
errexit;set +o emacs;set -o braceexpand;set +o allexport;shopt -u
autocd;shopt -u cdable_vars;shopt -u cdspell;shopt -u checkhash;shopt
-u checkjobs;shopt -u checkwinsize;shopt -s cmdhist;shopt -u
compat31;shopt -u compat32;shopt -u compat40;shopt -u compat41;shopt -u
direxpand;shopt -u dirspell;shopt -u dotglob;shopt -u execfail;shopt -u
expand_aliases;shopt -u extdebug;shopt -s extglob;shopt -s
extquote;shopt -u failglob;shopt -s force_fignore;shopt -u
globstar;shopt -u gnu_errfmt;shopt -u histappend;shopt -u
histreedit;shopt -u histverify;shopt -s hostcomplete;shopt -u
huponexit;shopt -s interactive_comments;shopt -u lastpipe;shopt -u
lithist;shopt -u login_shell;shopt -u mailwarn;shopt -u
no_empty_cmd_completion;shopt -u nocaseglob;shopt -u
nocasematch;shopt -s nullglob;shopt -s progcomp;shopt -s
promptvars;shopt -u restricted_shell;shopt -u shift_verbose;shopt -s
sourcepath;shopt -u syslog_history;shopt -u xpg_echo;'
+++ eval 'set +o xtrace;set +o vi;set +o verbose;set +o privileged;set
+o posix;set +o pipefail;set +o physical;set +o onecmd;set +o
nounset;set +o notify;set +o nolog;set +o noglob;set +o noexec;set +o
noclobber;set +o monitor;set +o keyword;set -o interactive-comments;set
+o ignoreeof;set +o history;set +o histexpand;set -o hashall;set +o
functrace;set +o errtrace;set +o errexit;set +o emacs;set -o
braceexpand;set +o allexport;shopt -u autocd;shopt -u cdable_vars;shopt
-u cdspell;shopt -u checkhash;shopt -u checkjobs;shopt -u
checkwinsize;shopt -s cmdhist;shopt -u compat31;shopt -u compat32;shopt
-u compat40;shopt -u compat41;shopt -u direxpand;shopt -u dirspell;shopt
-u dotglob;shopt -u execfail;shopt -u expand_aliases;shopt -u
extdebug;shopt -s extglob;shopt -s extquote;shopt -u failglob;shopt -s
force_fignore;shopt -u globstar;shopt -u gnu_errfmt;shopt -u
histappend;shopt -u histreedit;shopt -u histverify;shopt -s
hostcomplete;shopt -u huponexit;shopt -s interactive_comments;shopt -u
lastpipe;shopt -u lithist;shopt -u login_shell;shopt -u mailwarn;shopt
-u no_empty_cmd_completion;shopt -u nocaseglob;shopt -u
nocasematch;shopt -s nullglob;shopt -s progcomp;shopt -s
promptvars;shopt -u restricted_shell;shopt -u shift_verbose;shopt -s
sourcepath;shopt -u syslog_history;shopt -u xpg_echo;'
++++ set +o xtrace
+++ LogPrint 'Exiting rear mkbackup (PID 236532) and its descendant
processes'
+++ Log 'Exiting rear mkbackup (PID 236532) and its descendant
processes'
++++ date '+%Y-%m-%d %H:%M:%S.%N '
+++ local 'timestamp=2021-06-10 05:48:39.247117472 '
+++ test 1 -gt 0
+++ echo '2021-06-10 05:48:39.247117472 Exiting rear mkbackup (PID
236532) and its descendant processes'
2021-06-10 05:48:39.247117472 Exiting rear mkbackup (PID 236532) and its
descendant processes
+++ Print 'Exiting rear mkbackup (PID 236532) and its descendant
processes'
+++ test 1
+++ echo -e 'Exiting rear mkbackup (PID 236532) and its descendant
processes'
+++ sleep 1
++++ pstree -Aplau 236532
+++ Log 'rear,236532 /usr/sbin/rear -D -d mkbackup
-rear,30262 /usr/sbin/rear -D -d mkbackup-pstree,30263 -Aplau
236532'
++++ date '+%Y-%m-%d %H:%M:%S.%N '
+++ local 'timestamp=2021-06-10 05:48:40.323787378 '
+++ test 1 -gt 0
+++ echo '2021-06-10 05:48:40.323787378 rear,236532 /usr/sbin/rear -D -d
mkbackup
-rear,30262 /usr/sbin/rear -D -d mkbackup-pstree,30263 -Aplau
236532'
2021-06-10 05:48:40.323787378 rear,236532 /usr/sbin/rear -D -d
mkbackup
-rear,30262 /usr/sbin/rear -D -d mkbackup-pstree,30263 -Aplau
236532
+++ local remaining_columns=-40
+++ test -40 -ge 40
+++ remaining_columns=40
+++ local descendant_pid=
+++ local not_yet_terminated_pids=
++++ descendants_pids 236532
++++ local parent_pid=236532
++++ kill -0 236532
++++ local child_pid=
+++++ ps --ppid 236532 -o pid=
++++ for child_pid in '$( ps --ppid $parent_pid -o pid= )'
++++ kill -0 30266
++++ descendants_pids 30266
++++ local parent_pid=30266
++++ kill -0 30266
++++ local child_pid=
+++++ ps --ppid 30266 -o pid=
++++ for child_pid in '$( ps --ppid $parent_pid -o pid= )'
++++ kill -0 30269
++++ kill -0 30266
++++ echo 30266
++++ kill -0 236532
++++ echo 236532
+++ for descendant_pid in '$( descendants_pids $MASTER_PID )'
+++ test 236532 -eq 30266
+++ kill -0 30266
/usr/share/rear/lib/_input-output-functions.sh: line 157: kill: (30266)
- No such process
+++ continue
+++ for descendant_pid in '$( descendants_pids $MASTER_PID )'
+++ test 236532 -eq 236532
+++ continue
+++ test ''
+++ LogPrint 'Running exit tasks'
+++ Log 'Running exit tasks'
++++ date '+%Y-%m-%d %H:%M:%S.%N '
+++ local 'timestamp=2021-06-10 05:48:40.409647045 '
+++ test 1 -gt 0
+++ echo '2021-06-10 05:48:40.409647045 Running exit tasks'
2021-06-10 05:48:40.409647045 Running exit tasks
+++ Print 'Running exit tasks'
+++ test 1
+++ echo -e 'Running exit tasks'
+++ local exit_task=
+++ for exit_task in '"${EXIT_TASKS[@]}"'
+++ Debug 'Exit task '''umount -f -v
'''/tmp/rear.KCXi6xCey8b60rU/outputfs''' >&2''''
+++ test 1
+++ Log 'Exit task '''umount -f -v
'''/tmp/rear.KCXi6xCey8b60rU/outputfs''' >&2''''
++++ date '+%Y-%m-%d %H:%M:%S.%N '
+++ local 'timestamp=2021-06-10 05:48:40.416158078 '
+++ test 1 -gt 0
+++ echo '2021-06-10 05:48:40.416158078 Exit task '''umount -f -v
'''/tmp/rear.KCXi6xCey8b60rU/outputfs''' >&2''''
2021-06-10 05:48:40.416158078 Exit task 'umount -f -v
'/tmp/rear.KCXi6xCey8b60rU/outputfs' >&2'
+++ eval 'umount -f -v '''/tmp/rear.KCXi6xCey8b60rU/outputfs'''
>&2'
++++ umount -f -v /tmp/rear.KCXi6xCey8b60rU/outputfs
/tmp/rear.KCXi6xCey8b60rU/outputfs: nfs mount point detected
/tmp/rear.KCXi6xCey8b60rU/outputfs: umounted
+++ for exit_task in '"${EXIT_TASKS[@]}"'
+++ Debug 'Exit task '''rmdir -v /tmp/rear.KCXi6xCey8b60rU/outputfs
>&2''''
+++ test 1
+++ Log 'Exit task '''rmdir -v /tmp/rear.KCXi6xCey8b60rU/outputfs
>&2''''
++++ date '+%Y-%m-%d %H:%M:%S.%N '
+++ local 'timestamp=2021-06-10 05:48:41.269866202 '
+++ test 1 -gt 0
+++ echo '2021-06-10 05:48:41.269866202 Exit task '''rmdir -v
/tmp/rear.KCXi6xCey8b60rU/outputfs >&2''''
2021-06-10 05:48:41.269866202 Exit task 'rmdir -v
/tmp/rear.KCXi6xCey8b60rU/outputfs >&2'
+++ eval 'rmdir -v /tmp/rear.KCXi6xCey8b60rU/outputfs >&2'
++++ rmdir -v /tmp/rear.KCXi6xCey8b60rU/outputfs
rmdir: removing directory, '/tmp/rear.KCXi6xCey8b60rU/outputfs'
+++ for exit_task in '"${EXIT_TASKS[@]}"'
+++ Debug 'Exit task '''cleanup_build_area_and_end_program''''
+++ test 1
+++ Log 'Exit task '''cleanup_build_area_and_end_program''''
++++ date '+%Y-%m-%d %H:%M:%S.%N '
+++ local 'timestamp=2021-06-10 05:48:41.279237906 '
+++ test 1 -gt 0
+++ echo '2021-06-10 05:48:41.279237906 Exit task
'''cleanup_build_area_and_end_program''''
2021-06-10 05:48:41.279237906 Exit task
'cleanup_build_area_and_end_program'
+++ eval cleanup_build_area_and_end_program
++++ cleanup_build_area_and_end_program
++++ Log 'Finished in 1096 seconds'
+++++ date '+%Y-%m-%d %H:%M:%S.%N '
++++ local 'timestamp=2021-06-10 05:48:41.285377708 '
++++ test 1 -gt 0
++++ echo '2021-06-10 05:48:41.285377708 Finished in 1096 seconds'
2021-06-10 05:48:41.285377708 Finished in 1096 seconds
++++ is_true 1
++++ case "$1" in
++++ return 0
++++ LogPrint 'You should also rm -Rf /tmp/rear.KCXi6xCey8b60rU'
++++ Log 'You should also rm -Rf /tmp/rear.KCXi6xCey8b60rU'
+++++ date '+%Y-%m-%d %H:%M:%S.%N '
++++ local 'timestamp=2021-06-10 05:48:41.291332095 '
++++ test 1 -gt 0
++++ echo '2021-06-10 05:48:41.291332095 You should also rm -Rf
/tmp/rear.KCXi6xCey8b60rU'
2021-06-10 05:48:41.291332095 You should also rm -Rf
/tmp/rear.KCXi6xCey8b60rU
++++ Print 'You should also rm -Rf /tmp/rear.KCXi6xCey8b60rU'
++++ test 1
++++ echo -e 'You should also rm -Rf /tmp/rear.KCXi6xCey8b60rU'
++++ Log 'End of program reached'
+++++ date '+%Y-%m-%d %H:%M:%S.%N '
++++ local 'timestamp=2021-06-10 05:48:41.297223479 '
++++ test 1 -gt 0
++++ echo '2021-06-10 05:48:41.297223479 End of program reached'
2021-06-10 05:48:41.297223479 End of program reached
+++ for exit_task in '"${EXIT_TASKS[@]}"'
+++ Debug 'Exit task '''(( EXIT_FAIL_MESSAGE )) && echo '''rear
mkbackup failed, check /var/log/rear/rear-ussv-rdhpapp02.log for
details''' 1>&8''''
+++ test 1
+++ Log 'Exit task '''(( EXIT_FAIL_MESSAGE )) && echo '''rear mkbackup
failed, check /var/log/rear/rear-ussv-rdhpapp02.log for details'''
1>&8''''
++++ date '+%Y-%m-%d %H:%M:%S.%N '
+++ local 'timestamp=2021-06-10 05:48:41.303283797 '
+++ test 1 -gt 0
+++ echo '2021-06-10 05:48:41.303283797 Exit task '''((
EXIT_FAIL_MESSAGE )) && echo '''rear mkbackup failed, check
/var/log/rear/rear-ussv-rdhpapp02.log for details''' 1>&8''''
2021-06-10 05:48:41.303283797 Exit task '(( EXIT_FAIL_MESSAGE )) &&
echo 'rear mkbackup failed, check /var/log/rear/rear-ussv-rdhpapp02.log
for details' 1>&8'
+++ eval '(( EXIT_FAIL_MESSAGE )) && echo '''rear mkbackup failed,
check /var/log/rear/rear-ussv-rdhpapp02.log for details''' 1>&8'
++++ (( EXIT_FAIL_MESSAGE ))
+++ for exit_task in '"${EXIT_TASKS[@]}"'
+++ Debug 'Exit task '''exec 8>&-''''
+++ test 1
+++ Log 'Exit task '''exec 8>&-''''
++++ date '+%Y-%m-%d %H:%M:%S.%N '
+++ local 'timestamp=2021-06-10 05:48:41.309498693 '
+++ test 1 -gt 0
+++ echo '2021-06-10 05:48:41.309498693 Exit task '''exec 8>&-''''
2021-06-10 05:48:41.309498693 Exit task 'exec 8>&-'
+++ eval 'exec 8>&-'
++++ exec
+++ for exit_task in '"${EXIT_TASKS[@]}"'
+++ Debug 'Exit task '''exec 7>&-''''
+++ test 1
+++ Log 'Exit task '''exec 7>&-''''
++++ date '+%Y-%m-%d %H:%M:%S.%N '
+++ local 'timestamp=2021-06-10 05:48:41.315422836 '
+++ test 1 -gt 0
+++ echo '2021-06-10 05:48:41.315422836 Exit task '''exec 7>&-''''
2021-06-10 05:48:41.315422836 Exit task 'exec 7>&-'
+++ eval 'exec 7>&-'
++++ exec
+++ for exit_task in '"${EXIT_TASKS[@]}"'
+++ Debug 'Exit task '''exec 6<&-''''
+++ test 1
+++ Log 'Exit task '''exec 6<&-''''
++++ date '+%Y-%m-%d %H:%M:%S.%N '
+++ local 'timestamp=2021-06-10 05:48:41.320910611 '
+++ test 1 -gt 0
+++ echo '2021-06-10 05:48:41.320910611 Exit task '''exec 6<&-''''
2021-06-10 05:48:41.320910611 Exit task 'exec 6<&-'
+++ eval 'exec 6<&-'
++++ exec
+++ for exit_task in '"${EXIT_TASKS[@]}"'
+++ Debug 'Exit task '''''''
+++ test 1
+++ Log 'Exit task '''''''
++++ date '+%Y-%m-%d %H:%M:%S.%N '
+++ local 'timestamp=2021-06-10 05:48:41.325896808 '
+++ test 1 -gt 0
+++ echo '2021-06-10 05:48:41.325896808 Exit task '''''''
2021-06-10 05:48:41.325896808 Exit task ''
+++ eval ''
exfarmer commented at 2021-06-10 14:19:¶
***@***.*** ~]# ls -ltr /gsh_rear/ussv-rdhpapp02
total 5135487
-rw------- 1 root root 336216064 Jun 10 05:32 rear-ussv-rdhpapp02.iso
-rw------- 1 root root 271 Jun 10 05:32 VERSION
-rw------- 1 root root 202 Jun 10 05:32 README
-rw------- 1 root root 7079148 Jun 10 05:32 rear-ussv-rdhpapp02.log
-rw------- 1 root root 4915441664 Jun 10 05:48 backup.tar.gz
pcahyna commented at 2021-06-10 14:31:¶
And is there enough space on the server to write more data to backup.tar.gz? Is there anything else in /tmp/rear.KCXi6xCey8b60rU/tmp/backup.log ?
exfarmer commented at 2021-06-10 14:49:¶
Hi,
block 29610321: /boot/initramfs-3.10.0-1160.25.1.el7.x86_64.img
block 29674835: /boot/initramfs-3.10.0-1160.25.1.el7.x86_64kdump.img
block 29704307: /boot/config-3.10.0-1160.21.1.el7.x86_64
block 29704610: /boot/vmlinuz-3.10.0-1160.21.1.el7.x86_64
block 29717843: /boot/initramfs-3.10.0-1160.21.1.el7.x86_64kdump.img
dd: writing to
'/tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz':
Input/output error
9615118+0 records in
9615117+0 records out
4922939904 bytes (4.9 GB) copied, 947.122 s, 5.2 MB/s
***@***.*** ~]# df -H /tmp
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rhel-root 54G 18G 37G 32% /
***@***.*** ~]# df -H /gsh_rear/
Filesystem Size Used Avail Use% Mounted on
ussvdd3300:/data/col1/ussv_linux_bu_image 17T 2.7T 14T 17% /gsh_rear
Although 4.9G is large, it isn’t close to the limit.
Other file systems on the NFS share are twice that size.
***@**.*** ~]# du -h /gsh_rear/|sort
3.7G /gsh_rear/ussv-cad01
4.3G /gsh_rear/ussv-rdhpapp01
4.8G /gsh_rear/ussv-rdhpapp01.old
4.9G /gsh_rear/ussv-rdhpapp02
4.9G /gsh_rear/ussv-rdhpapp02.old
5.5G /gsh_rear/ussv-cad02.old
6.0G /gsh_rear/ussv-cad02
6.2G /gsh_rear/ussv-tvup01
6.2G /gsh_rear/ussv-tvup01.old
9.1G /gsh_rear/ussv-rdhpapp00.old
9.4G /gsh_rear/ussv-rdhpapp00
pcahyna commented at 2021-06-10 14:51:¶
Is the error reproducible? Have you tried the backup multiple times?
exfarmer commented at 2021-06-10 14:58:¶
Yes, the backup runs through Blade Logic, but I also ran it manually and each time, twice, it failed with the same error.
/var/log/rear/rear-ussv-rdhpapp02.log ==== Stack trace ==== Trace 0: /usr/sbin/rear:547 main Trace 1: /usr/share/rear/lib/mkbackup-workflow.sh:26 WORKFLOW_mkbackup Trace 2: /usr/share/rear/lib/framework-functions.sh:101 SourceStage Trace 3: /usr/share/rear/lib/framework-functions.sh:49 Source Trace 4: /usr/share/rear/backup/NETFS/default/500_make_backup.sh:252 source ++ echo 'Message: dd failed with return code 1 and below output: ---snip--- dd: writing to '''/tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz''': Input/output error¶
This means that the archiving process ended prematurely, or did not even start. As a result it is unlikely you can recover this system properly. Relax-and-Recover is therefore aborting execution. ' Message: dd failed with return code 1 and below output: ---snip--- dd: writing to '/tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz': Input/output error¶
This means that the archiving process ended prematurely, or did
not even start. As a result it is unlikely you can recover this
system properly. Relax-and-Recover is therefore aborting execution.
++ echo '== End stack trace =='
== End stack trace ==
++ kill -USR1 236532
+++ EXIT_FAIL_MESSAGE=0
+++ echo 'Aborting due to an error, check
/var/log/rear/rear-ussv-rdhpapp02.log for details'
/var/log/rear/rear-ussv-rdhpapp02.log.old ==== Stack trace ==== Trace 0: /usr/sbin/rear:547 main Trace 1: /usr/share/rear/lib/mkbackup-workflow.sh:26 WORKFLOW_mkbackup Trace 2: /usr/share/rear/lib/framework-functions.sh:101 SourceStage Trace 3: /usr/share/rear/lib/framework-functions.sh:49 Source Trace 4: /usr/share/rear/backup/NETFS/default/500_make_backup.sh:252 source Message: dd failed with return code 1 and below output: ---snip--- dd: writing to '/tmp/rear.Qhi8DRaSmBo7kPc/outputfs/ussv-rdhpapp02/backup.tar.gz': Input/output error¶
This means that the archiving process ended prematurely, or did
not even start. As a result it is unlikely you can recover this
system properly. Relax-and-Recover is therefore aborting execution.
== End stack trace ==
2021-06-10 05:21:44.728908362 Exiting rear mkbackup (PID 199284) and its
descendant processes
2021-06-10 05:21:45.803894935 rear,199284 /usr/sbin/rear -d -v
mkbackup
-rear,232834 /usr/sbin/rear -d -v mkbackup-pstree,232835 -Aplau
199284
/usr/share/rear/lib/_input-output-functions.sh: line 157: kill:
(232838) - No such process
2021-06-10 05:21:45.890447727 Running exit tasks
I can run it again and see what is in the
/tmp.rear.whatever/tmp/backup.log, without deleting the one that is
there now.
Let me know
pcahyna commented at 2021-06-10 15:02:¶
Does it abort at the same size of output file every time (4694 MiB)? Do I understand correctly that you are saving backups from multiple machines to the same NFS server and only one of them has the problem? If so, do you know what might make the problematic machine different?
exfarmer commented at 2021-06-10 15:21:¶
Hi,
Just ran it again:
Archived 4680 MiB [avg 5056 KiB/sec] OK ERROR: dd failed with return code 1 and below output: ---snip--- dd: writing to '/tmp/rear.oqOJpPF14XJAVjF/outputfs/ussv-rdhpapp02/backup.tar.gz': Input/output error¶
This means that the archiving process ended prematurely, or did
not even start. As a result it is unlikely you can recover this
system properly. Relax-and-Recover is therefore aborting execution.
From /tmp/rear.oqOJpPF14XJAVjF/tmp/backup.log
block 29606323: /boot/symvers-3.10.0-1160.25.1.el7.x86_64.gz
block 29606953: /boot/vmlinuz-3.10.0-1160.25.1.el7.x86_64
block 29620186: /boot/initramfs-3.10.0-1160.25.1.el7.x86_64.img
block 29684700: /boot/initramfs-3.10.0-1160.25.1.el7.x86_64kdump.img
dd: writing to
'/tmp/rear.oqOJpPF14XJAVjF/outputfs/ussv-rdhpapp02/backup.tar.gz':
Input/output error
9586305+0 records in
9586304+0 records out
4908187648 bytes (4.9 GB) copied, 948.811 s, 5.2 MB/s
++ pipes_rc=(${PIPESTATUS[@]})
++ let index=3-1
From the previous one, /tmp/rear.KCXi6xCey8b60rU/tmp/backup.log
block 29674835: /boot/initramfs-3.10.0-1160.25.1.el7.x86_64kdump.img
block 29704307: /boot/config-3.10.0-1160.21.1.el7.x86_64
block 29704610: /boot/vmlinuz-3.10.0-1160.21.1.el7.x86_64
block 29717843: /boot/initramfs-3.10.0-1160.21.1.el7.x86_64kdump.img
dd: writing to
'/tmp/rear.KCXi6xCey8b60rU/outputfs/ussv-rdhpapp02/backup.tar.gz':
Input/output error
9615118+0 records in
9615117+0 records out
4922939904 bytes (4.9 GB) copied, 947.122 s, 5.2 MB/s
++ pipes_rc=(${PIPESTATUS[@]})
++ let index=3-1
It’s close, but not exact
exfarmer commented at 2021-06-10 16:55:¶
Hi,
The storage team increased the quota on the NFS share, that fixed the
issue.
It would have been nice to see that there was a quota issue in the log
file, but all’s good now.
Thank you
pcahyna commented at 2021-06-11 07:45:¶
It would have been nice to see that there was a quota issue in the log file, but all’s good now.
Glad to hear that. ReaR itself can't report a quota issue, it merely
runs tools like dd and reports the result. Why dd was not able to report
the quota error is a valid question, it seems that it merely reports the
error from the kernel. "Input/output error" is EIO (see the errno(3)
man page). I would expect EDQUOT (Disk quota exceeded) instead. I
suggest to ask your storage administrators whether they know how one can
make NFS to report quota errors properly (maybe there is a way), or
contact Red Hat support.
Anyway, this is not the right place for solving this problem, my NFS-fu is not strong enough for this.
exfarmer commented at 2021-06-11 12:36:¶
Hi,
I meant that the dd statement would return different codes, not that I
expected ReaR to do it.
I think we’re good, the error was from a Wednesday night server, the
Thursday night servers had no issues.
The failed server also worked.
Close this case, if something comes up next Wednesday, I’ll email you
Thank you
jsmeix commented at 2021-06-18 10:10:¶
@pcahyna
thank you for your analysis that revealed the root cause in the user's
environment!
[Export of Github issue for rear/rear.]