#1953 Issue closed: ERROR: No filesystem mounted on '/mnt/local'. Stopping.¶

Labels: support / question, fixed / solved / done

pdanek opened issue at 2018-11-06 11:42:¶

Relax-and-Recover (ReaR) Issue Template¶

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

• ReaR version ("/usr/sbin/rear -V"): Relax-and-Recover 2.00 / Git (rpm package rear-2.00-7.el7_5.x86_64)

• OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"):
  Red Hat Enterprise Linux Server release 7.4, kernel 3.10.0-693.5.2.el7.x86_64

• ReaR configuration file cat /etc/rear/local.conf:

OUTPUT=ISO
BACKUP=NBU
SSH_ROOT_PASSWORD='...'
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/media' '/var/tmp' '/var/crash')
ONLY_INCLUDE_VG=( "rootvg" )
KERNEL_CMDLINE="$KERNEL_CMDLINE net.ifnames=0"
AUTOEXCLUDE_MULTIPATH=y

• Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR):
  Cisco UCS

• System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
  x86

• Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
  BIOS

• Storage (lokal disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
  FC

• Description of the issue (ideally so that others can reproduce it):

RESCUE sop00dbhh04t:~ # rear recover
Relax-and-Recover 2.00 / Git
Using log file: /var/log/rear/rear-sop00dbhh04t.log
..........
Enter date (mm/dd/yyyy) or date/time (mm/dd/yyyy HH:MM:SS) or press ENTER [30 secs]:
Skipping Point-In-Time Restore, will restore most recent data.
Comparing disks.
Disk configuration is identical, proceeding with restore.
Start system layout restoration.
Disk layout created.
ERROR: No filesystem mounted on '/mnt/local'. Stopping.
Aborting due to an error, check /var/log/rear/rear-sop00dbhh04t.log for details
Terminated

Log file attached.
rear-sop00dbhh04t_AUTOEXCLUDE_MULTIPATH=y.log

• Workaround, if any:
  Using AUTOEXCLUDE_MULTIPATH=n, we get different error - log file attached.

rear-sop00dbhh04t_AUTOEXCLUDE_MULTIPATH=n.log

• Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):
  rear-sop00dbhh04t_AUTOEXCLUDE_MULTIPATH=y_DEBUG.log

gdha commented at 2018-11-06 12:51:¶

+++ parted -s /dev/mapper/3600507680c8082f60000000000000015 mkpart '"primary"' 1048576B 4398045462527B
Error: partition length of 8589930496 sectors exceeds the msdos-partition-table-imposed maximum of 4294967295

It struck me that the partition table is of msdos type instead of gpt.
Can you check the output of parted /dev/sdd print?
I see you are using the RHEL rear-2.00 version - so if you could open a support case at RedHat for quick assistance.

pdanek commented at 2018-11-06 13:45:¶

I already opened Red Hat case in parallel, but you guys are most of the time way more effective in finding root cause. :)

RESCUE sop00dbhh04t:~ # parted /dev/sdd print
Model: IBM 2145 (scsi)
Disk /dev/sdd: 1100GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  1075MB  1074MB  primary  ext3         boot
 2      1075MB  1100GB  1098GB  primary               lvm

Does it mean I should focus on workaround scenario with AUTOEXCLUDE_MULTIPATH=y?

Thanks,
Peter

gdha commented at 2018-11-06 14:32:¶

@pdanek Start with adding the following 2 lines in local.conf

AUTOEXCLUDE_MULTIPATH=n
BOOT_OVER_SAN=y

However, what worries me is the partition table label msdos. How can a multipath disk:

create: 3600507680c8082f60000000000000015 undef IBM     ,2145            
size=4.0T features='0' hwhandler='0' wp=undef
|-+- policy='service-time 0' prio=50 status=undef
| |- 3:0:3:13  sdav 66:240 undef ready running
| `- 13:0:3:13 sdcy 70:96  undef ready running
`-+- policy='service-time 0' prio=10 status=undef
  |- 3:0:0:13  sds  65:32  undef ready running
  `- 13:0:0:13 sdbs 68:96  undef ready running

of size 4TB be using a msdos partition table instead of gpt. That does not make sense at all.
@schabrolles Have you seen this before?

pdanek commented at 2018-11-06 15:15:¶

1. Tested with BOOT_OVER_SAN=y, still same error (partition table label msdos).
2. I also downloaded latest rear-2.4-1.el7.x86_64.rpm from http://relax-and-recover.org/download/, but there we face same issue as with latest rear package in Red Hat resositories:
   rear_-v_mkrescue.txt

--

jsmeix commented at 2018-11-07 11:16:¶

@gdha
I have a question as multipath noob - perhaps I can learn a bit:

In https://github.com/rear/rear/issues/1953#issuecomment-436241774
you wrote about /dev/sdd.

How did you find out in the attached logs form @pdanek
that /dev/mapper/3600507680c8082f60000000000000015
belongs to /dev/sdd?

pdanek commented at 2018-12-18 11:33:¶

With latest ReaR 2.4 version and AUTOEXCLUDE_MULTIPATH=n + BOOT_OVER_SAN=y we get following after rear recover:

Ambiguous possible target disks need manual configuration (more than one with same size found)
Switching to manual disk layout configuration
Using /dev/mapper/3600507680c8082f60000000000000002 (same name and same size) for recreating /dev/mapper/3600507680c8082f60000000000000002
Current disk mapping table (source -> target):
/dev/mapper/3600507680c8082f60000000000000002 /dev/mapper/3600507680c8082f60000000000000002
Confirm or edit the disk mapping

1. Confirm disk mapping and continue 'rear recover'
2. Edit disk mapping (/var/lib/rear/layout/disk_mappings)
3. Use Relax-and-Recover shell and return back to here
4. Abort 'rear recover'

Also:

RESCUE sop00dbhh03t:~ # grep -v "^#" /var/lib/rear/layout/disklayout.conf | grep part
part /dev/mapper/3600507680c8082f60000000000000002 1073741824 1048576 primary boot /dev/mapper/3600507680c8082f60000000000000002p1
part /dev/mapper/3600507680c8082f60000000000000002 1098436837376 1074790400 primary lvm /dev/mapper/3600507680c8082f60000000000000002p2
RESCUE sop00dbhh03t:~ #

Do you have an idea what could be wrong please?

gdha commented at 2018-12-18 12:31:¶

@pdanek Was the restore (https://github.com/rear/rear/issues/1953#issuecomment-448190719) successful that way? What is the content of /var/lib/rear/layout/disklayout.conf?

pdanek commented at 2018-12-18 12:38:¶

I didn't try the option 1 yet as I was assumed "rear recover" should automatically know which disk is correct.
This is physical machine so I didn't want to break it as we can't easily revert to VM snapshot.

disklayout.conf attached - disklayout.conf.txt
Should I try restoring with option 1?

Thanks.

gdha commented at 2018-12-18 13:08:¶

@pdanek what you can try to do is the following:

1. download script from https://github.com/gdha/mismas/blob/master/make_rear_diskrestore_script.sh
2. the usage is explained at http://www.it3.be/2016/06/08/rear-diskrestore/
3. post the diskrestore.sh script for review

jsmeix commented at 2018-12-18 15:10:¶

@pdanek
because your assumption that "rear recover" should automatically know
which disk is correct does not hold in general I added this additional test
to be on the safe side against a total disaster in case of ambiguous
possible target disks.

For the reason behind see the MIGRATION_MODE documentation
in the current default.conf file and in
http://relax-and-recover.org/documentation/release-notes-2-4
the part about 'Improved MIGRATION_MODE autodetection
when the disk layout looks ambiguous.'

If you know what you do you can specify MIGRATION_MODE='false'
to let "rear recover" blindly proceed ( as it did before ;-)

jsmeix commented at 2018-12-18 15:14:¶

@pdanek
could yo provide a "rear -D recover" log file because
I would like to understand how that identical disk mapping from
/dev/mapper/3600507680c8082f60000000000000002 to the same
/dev/mapper/3600507680c8082f60000000000000002
happened in your particular case.

gdha commented at 2018-12-27 09:03:¶

@pdanek could you try out the snapshot version from http://download.opensuse.org/repositories/Archiving:/Backup:/Rear:/Snapshot/RHEL_7/x86_64/ ? The issue with missing NBU libraries should be fixed in there.

gdha commented at 2019-01-19 17:43:¶

@pdanek are your questions sufficient answered?

pdanek commented at 2019-02-25 09:16:¶

Thanks everyone for the help here.
Also disk restore script I have used it already for another issue, useful.

In regards to the above issue with MIGRATION_MODE, already being resolved in #2050.

---

[Export of Github issue for rear/rear.]