#2054 Issue closed: USB mount error Filesystem for device '''/dev/sdb1''' could not be found

Labels: support / question, won't fix / can't fix / obsolete, not ReaR / invalid, special hardware or VM

Ed746 opened issue at 2019-02-26 12:44:

Relax-and-Recover (ReaR) Issue Template

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

  • ReaR version ("/usr/sbin/rear -V"): Relax-and-Recover 2.4 / Git

  • OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"):

NAME="Oracle Linux Server"
VERSION="7.6"
ID="ol"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.6"
PRETTY_NAME="Oracle Linux Server 7.6"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:7:6:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://bugzilla.oracle.com/"

ORACLE_BUGZILLA_PRODUCT="Oracle Linux 7"
ORACLE_BUGZILLA_PRODUCT_VERSION=7.6
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=7.6
  • ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
OUTPUT=USB 
BACKUP=NETFS 
#BACKUP_URL=usb:///dev/disk/by-label/REAR-000 
BACKUP_URL=usb:///dev/sdb1
  • Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR):
System Information
    Manufacturer: HP
    Product Name: ProLiant DL360p Gen8
    Version: Not Specified
  • System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian

  • Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):

  • Storage (lokal disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
    Local disk

  • Description of the issue (ideally so that others can reproduce it):
    USB disk formats fine but mkrestore/mkbackup fails:

2019-02-26 07:39:04.686963495 Mounting with 'mount -v -o rw,noatime /dev/sdb1 /tmp/rear.ftm9sh6Aw5OcNEw/outputfs'
++ eval mount -v -o rw,noatime /dev/sdb1 /tmp/rear.ftm9sh6Aw5OcNEw/outputfs
+++ mount -v -o rw,noatime /dev/sdb1 /tmp/rear.ftm9sh6Aw5OcNEw/outputfs
mount: wrong fs type, bad option, bad superblock on /dev/sdb1,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
++ StopIfError 'Mount command '\''mount -v -o rw,noatime /dev/sdb1 /tmp/rear.ftm9sh6Aw5OcNEw/outputfs'\'' failed.'
++ ((  32 != 0  ))
++ Error 'Mount command '\''mount -v -o rw,noatime /dev/sdb1 /tmp/rear.ftm9sh6Aw5OcNEw/outputfs'\'' failed.'
++ LogPrintError 'ERROR: Mount command '\''mount -v -o rw,noatime /dev/sdb1 /tmp/rear.ftm9sh6Aw5OcNEw/outputfs'\'' failed.'

  • Workaround, if any:

  • Attachments, as applicable ("rear -D mkrescue

rear-localhost.log

/mkbackup/recover" debug log files):

gozora commented at 2019-02-26 13:44:

@Ed746
Do you have any particular reason for using
BACKUP_URL=usb:///dev/sdb1 instead of BACKUP_URL=usb:///dev/disk/by-label/REAR-000 ?

V.

Ed746 commented at 2019-02-26 13:52:

I've tried both, same result. I saw that option might be a possible work around, but no.

gozora commented at 2019-02-26 14:01:

@Ed746
Are you using UEFI or legacy boot ?
Is your USB device unmounted when running rear mkbackup ?
Can you post here log from running rear -d -D mkbackup with BACKUP_URL=usb:///dev/disk/by-label/REAR-000

Can you try maybe with

BACKUP=NETFS
OUTPUT=USB
USB_DEVICE=/dev/disk/by-label/REAR-000

?

V.

Ed746 commented at 2019-02-26 14:54:

Legacy BIOS boot
The device /dev/sdb1 shows unmounted:
umount: /dev/sdb1: not mounted
The log requested is attached
rear-localhost.log

gozora commented at 2019-02-26 15:05:

@Ed746

Can you post here log from running rear -d -D mkbackup with BACKUP_URL=usb:///dev/disk/by-label/REAR-000

Sorry for confusion ;-).

Ed746 commented at 2019-02-26 15:15:

Here you are.
rear-localhost.log

local.conf:

BACKUP=NETFS
OUTPUT=USB
BACKUP_URL=usb:///dev/disk/by-label/REAR-000

gozora commented at 2019-02-26 15:37:

@Ed746 thanks for provided logs.
It looks that something is wrong with your /dev/disk/by-label/REAR-000.
Are you able to mount it manually using e.g. mkdir -p /mnt/rear_test && mount -v -o rw,noatime /dev/disk/by-label/REAR-000 /mnt/rear_test ?

Can you show output from blkid and ls -al /dev/sdb*?

V.

Ed746 commented at 2019-02-26 15:55:

From the mkdir test:

mount: wrong fs type, bad option, bad superblock on /dev/sdb1,
       missing codepage or helper program, or other error

And:

[root@localhost firmware]# dmesg | tail
[33012.510717] JBD: no valid journal superblock found
[33012.510727] EXT3-fs (sdb1): error loading journal
[37633.241273] JBD: no valid journal superblock found
[37633.241277] EXT3-fs (sdb1): error loading journal
[37851.244733] JBD: no valid journal superblock found
[37851.244744] EXT3-fs (sdb1): error loading journal
[46863.096390] JBD: no valid journal superblock found
[46863.096400] EXT3-fs (sdb1): error loading journal
[49235.151605] JBD: no valid journal superblock found
[49235.151610] EXT3-fs (sdb1): error loading journal

blkid:

/dev/mapper/ol-root: UUID="04926f0f-f4a8-424b-8d9c-a5e96bf59ff3" TYPE="xfs" 
/dev/sda2: UUID="b8kGy8-cmi6-Czjf-RDdk-3yok-JNyb-3daoI2" TYPE="LVM2_member" 
/dev/sda1: UUID="1e95822d-4aee-4f2b-95f5-e455a13a6559" TYPE="xfs" 
/dev/mapper/ol-swap: UUID="0bfc5dc3-7a93-498e-9dc8-79eb9cce0557" TYPE="swap" 
/dev/mapper/ol-home: UUID="f93bc91b-1672-4a20-883d-a2c342b8b4cd" TYPE="xfs" 
/dev/sdb1: LABEL="REAR-000" UUID="bdbf4483-e0f9-4bcb-877e-be98805eb950" SEC_TYPE="ext2" TYPE="ext3"

ls -al /dev/sdb* :

brw-rw---- 1 root disk 8, 16 Feb 26 06:06 /dev/sdb
brw-rw---- 1 root disk 8, 17 Feb 26 06:06 /dev/sdb1

gozora commented at 2019-02-26 16:10:

yes, looks like something went wrong with your USB drive between rear format and rear mkbackup/mkrescue

You can try to format your USB drive again with something like mkfs.ext3 -v -L REAR-000 /dev/sdb1 or using ReaR format rear format -v -- /dev/sdb, once formatting is done try to mount your device with: mkdir -p /mnt/rear_test && mount -v -o rw,noatime /dev/disk/by-label/REAR-000 /mnt/rear_test if mount succeeds, run umount /mnt/rear_test and try rear mkbackup again ...

V.

Ed746 commented at 2019-02-26 16:34:

Ughh, same thing, what's confusing is it seems the disk gets formatted OK:

mke2fs 1.42.9 (28-Dec-2013)
fs_types for mke2fs.conf resolution: 'ext3'
Filesystem label=REAR-000
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
8192000 inodes, 32765952 blocks
1638297 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
1000 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
    4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done     ```

# mkdir -p /mnt/rear_test && mount -v -o rw,noatime /dev/disk/by-label/REAR-000 /mnt/rear_test
mount: wrong fs type, bad option, bad superblock on /dev/sdb1,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

#### <img src="https://avatars.githubusercontent.com/u/12116358?u=1c5ba9dcee5ca3082f03029a7fbe647efd30eb49&v=4" width="50">[gozora](https://github.com/gozora) commented at [2019-02-26 16:50](https://github.com/rear/rear/issues/2054#issuecomment-467518150):

@Ed746 can it be that you are somehow missing ext3 support in your kernel ?
Can you try `modprobe ext3` ?

V.

#### <img src="https://avatars.githubusercontent.com/u/12116358?u=1c5ba9dcee5ca3082f03029a7fbe647efd30eb49&v=4" width="50">[gozora](https://github.com/gozora) commented at [2019-02-26 16:56](https://github.com/rear/rear/issues/2054#issuecomment-467520896):

If you are missing ext3 support in your kernel, you might try with ext4. For this to work, you need to set `USB_DEVICE_FILESYSTEM=ext4` in your `/etc/rear/local.conf` (and format you device again).

V.

#### <img src="https://avatars.githubusercontent.com/u/12116358?u=1c5ba9dcee5ca3082f03029a7fbe647efd30eb49&v=4" width="50">[gozora](https://github.com/gozora) commented at [2019-02-26 17:00](https://github.com/rear/rear/issues/2054#issuecomment-467522397):

Btw. `modprobe ext4` might work as well ;-)

V.

#### <img src="https://avatars.githubusercontent.com/u/48016290?v=4" width="50">[Ed746](https://github.com/Ed746) commented at [2019-02-26 17:07](https://github.com/rear/rear/issues/2054#issuecomment-467525115):

both modprobe ext3 and ext4 command returns quietly, I believe that is a good thing.

#### <img src="https://avatars.githubusercontent.com/u/48016290?v=4" width="50">[Ed746](https://github.com/Ed746) commented at [2019-02-26 17:08](https://github.com/rear/rear/issues/2054#issuecomment-467525800):

I'm about to try a new USB disk

#### <img src="https://avatars.githubusercontent.com/u/12116358?u=1c5ba9dcee5ca3082f03029a7fbe647efd30eb49&v=4" width="50">[gozora](https://github.com/gozora) commented at [2019-02-26 17:32](https://github.com/rear/rear/issues/2054#issuecomment-467535027):

Exactly, no error is a good error ;-).

Yes, trying different HW is a good option too ...

V.

#### <img src="https://avatars.githubusercontent.com/u/48016290?v=4" width="50">[Ed746](https://github.com/Ed746) commented at [2019-02-26 19:07](https://github.com/rear/rear/issues/2054#issuecomment-467570203):

Progress:

[54023.406692] FAT-fs (sdb1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[55743.324225] JBD: no valid journal superblock found
[55743.324235] EXT3-fs (sdb1): error loading journal

fsck /dev/sdb1

fsck from util-linux 2.23.2
e2fsck 1.42.9 (28-Dec-2013)
Superblock has an invalid journal (inode 8).
Clear? yes
*** ext3 journal has been deleted - filesystem is now ext2 only ***

REAR-000 was not cleanly unmounted, check forced.
Pass 1: Checking inodes, blocks, and sizes
Journal inode is not in use, but contains data. Clear? yes

Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences: +(4292608--4293121) +(4325376--4325889) +(4358144--4358657) +(4390912--4391425) +(4423680--4424193) ...............................
Fix? yes
Free blocks count wrong for group #498 (0, counted=32254).
Fix? yes
Free blocks count wrong for group #499 (31707, counted=32254).
Fix? yes
Free blocks count wrong (32205774, counted=32238575).
Fix? yes
Recreate journal? yes
Creating journal (32768 blocks): Done.

*** journal has been re-created - filesystem is now ext3 again ***

REAR-000: ***** FILE SYSTEM WAS MODIFIED *****
REAR-000: 11/8192000 files (0.0% non-contiguous), 562182/32767956 blocks

umount /dev/sdb1

umount: /dev/sdb1: not mounted

umount /dev/sdb

umount: /dev/sdb: not mounted

mke2fs -t ext3 -O ^has_journal /dev/sdb1

mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
8192000 inodes, 32767956 blocks
1638397 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
1000 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: done
Writing inode tables: done
Writing superblocks and filesystem accounting information: done

mkdir -p /mnt/rear_test && mount -v -o rw,noatime /dev/disk/by-label/REAR-000 /mnt/rear_test

mount: special device /dev/disk/by-label/REAR-000 does not exist

mkdir -p /mnt/rear_test && mount -v -o rw,noatime /dev/sdb1 /mnt/rear_test

mount: /dev/sdb1 mounted on /mnt/rear_test.

#### <img src="https://avatars.githubusercontent.com/u/48016290?v=4" width="50">[Ed746](https://github.com/Ed746) commented at [2019-02-26 19:10](https://github.com/rear/rear/issues/2054#issuecomment-467571566):

sorry about those bolds.. but the command `# mke2fs -t ext3 -O ^has_journal /dev/sdb1`
creates the missing journal. This is rapidly over my head but it seems that some OS issue makes the existing rear format incomplete???
I have no idea, but I can mount the disk now, so I should label it and test a mkbackup??

#### <img src="https://avatars.githubusercontent.com/u/12116358?u=1c5ba9dcee5ca3082f03029a7fbe647efd30eb49&v=4" width="50">[gozora](https://github.com/gozora) commented at [2019-02-26 19:26](https://github.com/rear/rear/issues/2054#issuecomment-467577364):

Yes, you can label your disk _REAR-000_ and try to run `rear mkbackup`
But in either case you should investigate closer what those "some OS issue" is because having corrupted filesytem might be undesirable in the future ...

> but the command # mke2fs -t ext3 -O ^has_journal /dev/sdb1
> creates the missing journal.

I've never used mke2fs, but reading `man mke2fs` says, excerpt:

-O feature[,...]
...
To disable a feature, simply prefix the feature name with
a caret ('^') or a minus ('-') character.
...

So I'm guessing here that you've disabled filesystem journal instead ...

V.

#### <img src="https://avatars.githubusercontent.com/u/48016290?v=4" width="50">[Ed746](https://github.com/Ed746) commented at [2019-02-26 20:16](https://github.com/rear/rear/issues/2054#issuecomment-467596364):

Not surprising, I found that mke2fs on google and tried it :-)
Anyhow, it seemed to make progress with an error at the end. What to make of that?

e2label /dev/sdb1 REAR-000

rear -d -D mkbackup

Relax-and-Recover 2.4 / Git
Using log file: /var/log/rear/rear-localhost.log
Using backup archive '/tmp/rear.VkPdIO4qmC7oN19/outputfs/rear/localhost/20190226.1459/backup.tar.gz'
Creating disk layout
Using guessed bootloader 'GRUB' (found in first bytes on /dev/sda)
Creating root filesystem layout
Handling network interface 'bond0'
bond0 is a bond
bond0 has lower interface eno1
eno1 is a physical device
bond0 has lower interface eno2
Handled network interface 'bond0'
To log into the recovery system via ssh set up /root/.ssh/authorized_keys or specify SSH_ROOT_PASSWORD
Copying logfile /var/log/rear/rear-localhost.log into initramfs as '/tmp/rear-localhost-partial-2019-02-26T14:59:40-0500.log'
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Copying all files in /lib*/firmware/
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (251337928 bytes) in 31 seconds
Saved /var/log/rear/rear-localhost.log as rear/localhost/20190226.1459/rear-localhost.log
Writing MBR of type msdos to /dev/sdb
Copying resulting files to usb location
Saving /var/log/rear/rear-localhost.log as rear-localhost.log to usb location
ERROR: Could not mkdir '/tmp/rear.VkPdIO4qmC7oN19/outputfs/rear/localhost/20190226.1459'
Aborting due to an error, check /var/log/rear/rear-localhost.log for details
Exiting rear mkbackup (PID 23315) and its descendant processes
Running exit tasks
You should also rm -Rf /tmp/rear.VkPdIO4qmC7oN19
Terminated

#### <img src="https://avatars.githubusercontent.com/u/12116358?u=1c5ba9dcee5ca3082f03029a7fbe647efd30eb49&v=4" width="50">[gozora](https://github.com/gozora) commented at [2019-02-26 20:23](https://github.com/rear/rear/issues/2054#issuecomment-467598824):

@Ed746 :-)

Did you check _/var/log/rear/rear-localhost.log_ ?

V.

#### <img src="https://avatars.githubusercontent.com/u/48016290?v=4" width="50">[Ed746](https://github.com/Ed746) commented at [2019-02-26 21:00](https://github.com/rear/rear/issues/2054#issuecomment-467611673):

Yep & tried again:

rear -d -D mkbackup
Relax-and-Recover 2.4 / Git
Using log file: /var/log/rear/rear-localhost.log
Using backup archive '/tmp/rear.g4IvV6O9NnTsrtR/outputfs/rear/localhost/20190226.1554/backup.tar.gz'
Creating disk layout
Using guessed bootloader 'GRUB' (found in first bytes on /dev/sda)
Creating root filesystem layout
Handling network interface 'bond0'
bond0 is a bond
bond0 has lower interface eno1
eno1 is a physical device
bond0 has lower interface eno2
Handled network interface 'bond0'
To log into the recovery system via ssh set up /root/.ssh/authorized_keys or specify SSH_ROOT_PASSWORD
Copying logfile /var/log/rear/rear-localhost.log into initramfs as '/tmp/rear-localhost-partial-2019-02-26T15:54:20-0500.log'
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Copying all files in /lib*/firmware/
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (251332621 bytes) in 31 seconds
ERROR: Could not mkdir '/tmp/rear.g4IvV6O9NnTsrtR/outputfs/localhost'
Aborting due to an error, check /var/log/rear/rear-localhost.log for details
Exiting rear mkbackup (PID 51491) and its descendant processes
Running exit tasks
You should also rm -Rf /tmp/rear.g4IvV6O9NnTsrtR
Terminated

mkdir: cannot create directory '/tmp/rear.g4IvV6O9NnTsrtR/outputfs/localhost': Input/output error
++ StopIfError 'Could not mkdir '''/tmp/rear.g4IvV6O9NnTsrtR/outputfs/localhost''''
++ (( 1 != 0 ))

I can mkdir that directory from the command line btw.
Attached is the full log
[rear-localhost.log](https://github.com/rear/rear/files/2907503/rear-localhost.log)

#### <img src="https://avatars.githubusercontent.com/u/12116358?u=1c5ba9dcee5ca3082f03029a7fbe647efd30eb49&v=4" width="50">[gozora](https://github.com/gozora) commented at [2019-02-27 06:49](https://github.com/rear/rear/issues/2054#issuecomment-467745092):

Assuming you did not yet removed _/tmp/rear.g4IvV6O9NnTsrtR_ ...

can you try following:

mount -v -o rw,noatime /dev/disk/by-label/REAR-000 /tmp/rear.g4IvV6O9NnTsrtR/outputfs

mkdir -p -v -m0750 /tmp/rear.g4IvV6O9NnTsrtR/outputfs/localhost

?

V.

#### <img src="https://avatars.githubusercontent.com/u/1788608?u=925fc54e2ce01551392622446ece427f51e2f0ce&v=4" width="50">[jsmeix](https://github.com/jsmeix) commented at [2019-02-27 09:29](https://github.com/rear/rear/issues/2054#issuecomment-467789635):

I think by default ReaR removes its build area in `$TMPDIR/rear.XXXXXXXXXXXXXXX`
(at least when ReaR finishes normally) but with

KEEP_BUILD_DIR="yes"

in etc/rear/local conf one can enforce to always keep it,
cf. usr/share/rear/conf/default.conf

#### <img src="https://avatars.githubusercontent.com/u/12116358?u=1c5ba9dcee5ca3082f03029a7fbe647efd30eb49&v=4" width="50">[gozora](https://github.com/gozora) commented at [2019-02-27 09:31](https://github.com/rear/rear/issues/2054#issuecomment-467790373):

@jsmeix this on was run with `rear -d -D mkbackup` so the build area should remain ...

V.

#### <img src="https://avatars.githubusercontent.com/u/1788608?u=925fc54e2ce01551392622446ece427f51e2f0ce&v=4" width="50">[jsmeix](https://github.com/jsmeix) commented at [2019-02-27 09:35](https://github.com/rear/rear/issues/2054#issuecomment-467791481):

Oh! - so many cases we have in ReaR that make me confused ...

#### <img src="https://avatars.githubusercontent.com/u/48016290?v=4" width="50">[Ed746](https://github.com/Ed746) commented at [2019-02-27 14:35](https://github.com/rear/rear/issues/2054#issuecomment-467884612):

Yes those commands worked:

mkdir -p -v -m0750 /tmp/rear.g4IvV6O9NnTsrtR/outputfs/localhost
mkdir: created directory ‘/tmp/rear.g4IvV6O9NnTsrtR/outputfs’
mkdir: created directory ‘/tmp/rear.g4IvV6O9NnTsrtR/outputfs/localhost’

mount -v -o rw,noatime /dev/disk/by-label/REAR-000 /tmp/rear.g4IvV6O9NnTsrtR/outputfs
mount: /dev/sdb1 mounted on /tmp/rear.g4IvV6O9NnTsrtR/outputfs.

#### <img src="https://avatars.githubusercontent.com/u/1788608?u=925fc54e2ce01551392622446ece427f51e2f0ce&v=4" width="50">[jsmeix](https://github.com/jsmeix) commented at [2019-02-27 15:23](https://github.com/rear/rear/issues/2054#issuecomment-467904358):

@Ed746 
please - see what @gozora actually had you asked to do in his
https://github.com/rear/rear/issues/2054#issuecomment-467745092

In case of doubt inspect your log file
https://github.com/rear/rear/files/2907503/rear-localhost.log
that contains (excerpts)

+++ mount -v -o rw,noatime /dev/disk/by-label/REAR-000 /tmp/rear.g4IvV6O9NnTsrtR/outputfs
mount: /dev/sdb1 mounted on /tmp/rear.g4IvV6O9NnTsrtR/outputfs.
...
++ mkdir -p -v -m0750 /tmp/rear.g4IvV6O9NnTsrtR/outputfs/localhost
mkdir: cannot create directory '/tmp/rear.g4IvV6O9NnTsrtR/outputfs/localhost': Input/output error

The idea is to try to do that on command line.

#### <img src="https://avatars.githubusercontent.com/u/48016290?v=4" width="50">[Ed746](https://github.com/Ed746) commented at [2019-02-27 15:51](https://github.com/rear/rear/issues/2054#issuecomment-467916222):

From the command line in the order provided:

[root@localhost tmp]# umount /dev/sdb1
[root@localhost tmp]# rm -rf rear.*

[root@localhost tmp]# mount -v -o rw,noatime /dev/disk/by-label/REAR-000 /tmp/rear.g4IvV6O9NnTsrtR/outputfs
mount: mount point /tmp/rear.g4IvV6O9NnTsrtR/outputfs does not exist
[root@localhost tmp]# mkdir -p -v -m0750 /tmp/rear.g4IvV6O9NnTsrtR/outputfs/localhost
mkdir: created directory ‘/tmp/rear.g4IvV6O9NnTsrtR’
mkdir: created directory ‘/tmp/rear.g4IvV6O9NnTsrtR/outputfs’
mkdir: created directory ‘/tmp/rear.g4IvV6O9NnTsrtR/outputfs/localhost’
[root@localhost tmp]# mount -v -o rw,noatime /dev/disk/by-label/REAR-000 /tmp/rear.g4IvV6O9NnTsrtR/outputfs
mount: /dev/sdb1 mounted on /tmp/rear.g4IvV6O9NnTsrtR/outputfs.

#### <img src="https://avatars.githubusercontent.com/u/12116358?u=1c5ba9dcee5ca3082f03029a7fbe647efd30eb49&v=4" width="50">[gozora](https://github.com/gozora) commented at [2019-02-27 15:59](https://github.com/rear/rear/issues/2054#issuecomment-467919529):

@Ed746 nearly perfect :-) if you'd omit `rm -rf rear.*` it would be perfect ...

Now try following:
1. run `rear -D -d mkbackup` (I'm assuming that it will fail again)
2. locate what is your CURRENT build directory. It will be something like _/tmp/rear.somerandomstring_ but it is different at every evocation
3. run

mount -v -o rw,noatime /dev/disk/by-label/REAR-000 /tmp/rear.somerandomstring/outputfs

mkdir -p -v -m0750 /tmp/rear.somerandomstring/outputfs/localhost

4. report back here with results

V.

#### <img src="https://avatars.githubusercontent.com/u/48016290?v=4" width="50">[Ed746](https://github.com/Ed746) commented at [2019-02-27 16:56](https://github.com/rear/rear/issues/2054#issuecomment-467942553):

Here you are:

[root@localhost tmp]# rm -rf rear.*
[root@localhost tmp]# umount /dev/sdb1
umount: /dev/sdb1: not mounted
[root@localhost tmp]# rear -D -d mkbackup
Relax-and-Recover 2.4 / Git
Using log file: /var/log/rear/rear-localhost.log
Using backup archive '/tmp/rear.crhGMlqQjR99EiC/outputfs/rear/localhost/20190227.1148/backup.tar.gz'
Creating disk layout
Using guessed bootloader 'GRUB' (found in first bytes on /dev/sda)
Creating root filesystem layout
Handling network interface 'bond0'
bond0 is a bond
bond0 has lower interface eno1
eno1 is a physical device
bond0 has lower interface eno2
Handled network interface 'bond0'
To log into the recovery system via ssh set up /root/.ssh/authorized_keys or specify SSH_ROOT_PASSWORD
Copying logfile /var/log/rear/rear-localhost.log into initramfs as '/tmp/rear-localhost-partial-2019-02-27T11:49:11-0500.log'
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Copying all files in /lib*/firmware/
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (251335115 bytes) in 31 seconds
ERROR: Could not mkdir '/tmp/rear.crhGMlqQjR99EiC/outputfs/localhost'
Aborting due to an error, check /var/log/rear/rear-localhost.log for details
Exiting rear mkbackup (PID 58403) and its descendant processes
Running exit tasks
[root@localhost tmp]# mount -v -o rw,noatime /dev/disk/by-label/REAR-000 /tmp/rear.crhGMlqQjR99EiC/outputfs
mount: mount point /tmp/rear.crhGMlqQjR99EiC/outputfs does not exist
[root@localhost tmp]# mkdir -p -v -m0750 /tmp/rear.crhGMlqQjR99EiC/outputfs/localhost
mkdir: created directory ‘/tmp/rear.crhGMlqQjR99EiC/outputfs’
mkdir: created directory ‘/tmp/rear.crhGMlqQjR99EiC/outputfs/localhost’
[root@localhost tmp]#

#### <img src="https://avatars.githubusercontent.com/u/12116358?u=1c5ba9dcee5ca3082f03029a7fbe647efd30eb49&v=4" width="50">[gozora](https://github.com/gozora) commented at [2019-02-27 17:28](https://github.com/rear/rear/issues/2054#issuecomment-467954617):

Can you attach logfile from this session?

V.

#### <img src="https://avatars.githubusercontent.com/u/12116358?u=1c5ba9dcee5ca3082f03029a7fbe647efd30eb49&v=4" width="50">[gozora](https://github.com/gozora) commented at [2019-02-27 17:31](https://github.com/rear/rear/issues/2054#issuecomment-467955762):

and maybe output from `ls -alR /tmp/rear.crhGMlqQjR99EiC`

V.

#### <img src="https://avatars.githubusercontent.com/u/48016290?v=4" width="50">[Ed746](https://github.com/Ed746) commented at [2019-02-27 18:19](https://github.com/rear/rear/issues/2054#issuecomment-467972987):

Both attached.
[rear-localhost.log](https://github.com/rear/rear/files/2911705/rear-localhost.log)
[output.txt](https://github.com/rear/rear/files/2911715/output.txt)

#### <img src="https://avatars.githubusercontent.com/u/12116358?u=1c5ba9dcee5ca3082f03029a7fbe647efd30eb49&v=4" width="50">[gozora](https://github.com/gozora) commented at [2019-02-27 18:52](https://github.com/rear/rear/issues/2054#issuecomment-467984545):

Ccan you try following:

mkdir -p -v /tmp/rear.crhGMlqQjR99EiC/outputfs

mount -v -o rw,noatime /dev/disk/by-label/REAR-000 /tmp/rear.crhGMlqQjR99EiC/outputfs

mkdir -p -v -m0750 /tmp/rear.crhGMlqQjR99EiC/outputfs/localhost

ls -alR /tmp/rear.crhGMlqQjR99EiC

And I'm slowly running out of ideas :-) ...

V.

#### <img src="https://avatars.githubusercontent.com/u/12116358?u=1c5ba9dcee5ca3082f03029a7fbe647efd30eb49&v=4" width="50">[gozora](https://github.com/gozora) commented at [2019-02-27 18:57](https://github.com/rear/rear/issues/2054#issuecomment-467986726):

In your log I've noticed:

mkdir: cannot create directory '/tmp/rear.crhGMlqQjR99EiC/outputfs/localhost': Input/output error

Can it be that your USB disk is not that reliable after all ?

V.

#### <img src="https://avatars.githubusercontent.com/u/48016290?v=4" width="50">[Ed746](https://github.com/Ed746) commented at [2019-02-27 22:48](https://github.com/rear/rear/issues/2054#issuecomment-468062347):

I'm getting the same from the command line...

[root@localhost /]# mkdir -p -v /tmp/rear.crhGMlqQjR99EiC/outputfs
[root@localhost /]# mount -v -o rw,noatime /dev/disk/by-label/REAR-000 /tmp/rear.crhGMlqQjR99EiC/outputfs
mount: /dev/sdb1 mounted on /tmp/rear.crhGMlqQjR99EiC/outputfs.
[root@localhost /]# mkdir -p -v -m0750 /tmp/rear.crhGMlqQjR99EiC/outputfs/localhost
mkdir: cannot create directory ‘/tmp/rear.crhGMlqQjR99EiC/outputfs/localhost’: Input/output error

These are brand new SanDisk 128 GB USB 2.0 thumb drives

#### <img src="https://avatars.githubusercontent.com/u/48016290?v=4" width="50">[Ed746](https://github.com/Ed746) commented at [2019-02-27 22:49](https://github.com/rear/rear/issues/2054#issuecomment-468062522):

I'll try a different brand

#### <img src="https://avatars.githubusercontent.com/u/1788608?u=925fc54e2ce01551392622446ece427f51e2f0ce&v=4" width="50">[jsmeix](https://github.com/jsmeix) commented at [2019-02-28 13:59](https://github.com/rear/rear/issues/2054#issuecomment-468282242):

@Ed746 
good that finally you were able to reproduce on command line
what ReaR does which indicates the root cause is not inside ReaR.

Out of curiousity I would like to know how the partitioning on that
`brand new SanDisk 128 GB USB 2.0 thumb drive`
looks like.

When `/dev/sdb` is your USB drive what does

parted -s /dev/sdb unit MiB print

show?

#### <img src="https://avatars.githubusercontent.com/u/48016290?v=4" width="50">[Ed746](https://github.com/Ed746) commented at [2019-02-28 22:49](https://github.com/rear/rear/issues/2054#issuecomment-468470857):

The one I got to work farther is now ext2 because I ran fsck.

parted -s /dev/sdb unit MiB print
Model: Generic Flash Disk (scsi)
Disk /dev/sdb: 128000MiB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number Start End Size Type File system Flags
1 0.17MiB 128000MiB 128000MiB primary ext2
The new one I tried shows the bad super block:
parted -s /dev/sdb unit MiB print
Model: Generic Flash Disk (scsi)
Disk /dev/sdb: 128000MiB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number Start End Size Type File system Flags
1 8.00MiB 128000MiB 127992MiB primary ext3 boot
fsck /dev/sdb1
fsck from util-linux 2.23.2
e2fsck 1.42.9 (28-Dec-2013)
Superblock has an invalid journal (inode 8).
Clear? yes
*** ext3 journal has been deleted - filesystem is now ext2 only ***

The third USB I tried does the same thing.

Is there a different format command I can try? I'm on OL (RHEL) 7.6

#### <img src="https://avatars.githubusercontent.com/u/48016290?v=4" width="50">[Ed746](https://github.com/Ed746) commented at [2019-02-28 23:16](https://github.com/rear/rear/issues/2054#issuecomment-468478002):

After fsck completed fixing bad block counts and stuff the parted command shows this:

parted -s /dev/sdb unit MiB print
Model: Generic Flash Disk (scsi)
Disk /dev/sdb: 128000MiB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number Start End Size Type File system Flags
1 8.00MiB 128000MiB 127992MiB primary ext3 boot

That looked good so I tried rear mkbackup.
No Joy
Same log errors.

[root@localhost ~]# dmesg | tail
[ 960.937201] [] worker_thread+0x112/0x540
[ 960.937203] [] ? rescuer_thread+0x3f0/0x3f0
[ 960.937205] [] kthread+0xda/0xf0
[ 960.937207] [] ? __schedule+0x23e/0x850
[ 960.937209] [] ? kthread_create_on_node+0x1b0/0x1b0
[ 960.937211] [] ret_from_fork+0x61/0x90
[ 960.937213] [] ? kthread_create_on_node+0x1b0/0x1b0
[ 1564.301306] sdb: sdb1
[ 1759.577426] JBD: no valid journal superblock found
[ 1759.577436] EXT3-fs (sdb1): error loading journal

#### <img src="https://avatars.githubusercontent.com/u/12116358?u=1c5ba9dcee5ca3082f03029a7fbe647efd30eb49&v=4" width="50">[gozora](https://github.com/gozora) commented at [2019-03-01 08:18](https://github.com/rear/rear/issues/2054#issuecomment-468582337):

@Ed746 you should not run `fsck` immediately after you've created a filesystem. Something is obviously wrong with your HW of FW :-(.

Maybe you should open support case to HPE that you can't reliably write to UBS devices ...

V.

#### <img src="https://avatars.githubusercontent.com/u/12116358?u=1c5ba9dcee5ca3082f03029a7fbe647efd30eb49&v=4" width="50">[gozora](https://github.com/gozora) commented at [2019-03-01 08:27](https://github.com/rear/rear/issues/2054#issuecomment-468584601):

An interesting test would be to observe logs (_dmesg & syslog_) and output, when writing zeros to your USB device with `dd`

`dd if=/dev/zero of=/dev/<your_usb_device> bs=1M`

V.

#### <img src="https://avatars.githubusercontent.com/u/48016290?v=4" width="50">[Ed746](https://github.com/Ed746) commented at [2019-03-05 16:47](https://github.com/rear/rear/issues/2054#issuecomment-469756728):

I did the dd and there were no errors logged

#### <img src="https://avatars.githubusercontent.com/u/1788608?u=925fc54e2ce01551392622446ece427f51e2f0ce&v=4" width="50">[jsmeix](https://github.com/jsmeix) commented at [2019-04-26 13:01](https://github.com/rear/rear/issues/2054#issuecomment-487048473):

Regardless what exactly the root cause is, the root cause it not in ReaR.


-------------------------------------------------------------------------------



[Export of Github issue for [rear/rear](https://github.com/rear/rear).]