#2720 Issue closed: disk reformatting issue during BMR on linux 7.9 physical server

Labels: support / question, fixed / solved / done, external tool, not ReaR / invalid

nmaheve opened issue at 2021-11-22 18:06:

we are facing issue while doing BMR recovery .

disk reformatting issue during the time of recovery , i have attached the snapshot of same as well

error

errror1

jsmeix commented at 2021-11-23 07:16:

https://raw.githubusercontent.com/rear/rear/master/.github/ISSUE_TEMPLATE.md

nmaheve commented at 2021-11-23 15:18:

Relax-and-Recover (ReaR) Issue Template

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

  • ReaR version ("/usr/sbin/rear -V"):

  • OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"): RHEL7.9

  • ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

  • Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR): BareMetal

  • System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device): x86_64

  • Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot): UEFI/grub2

  • Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):DM

  • Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT"):

  • Description of the issue (ideally so that others can reproduce it): file system is not getting restored

  • Workaround, if any:

  • Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):

suraj952 commented at 2021-11-29 17:35:

Requested details have been sent out over the email as well. do we have any update on this please.

pcahyna commented at 2021-11-29 17:44:

Can you please attach the log of the failing restore, and also the disklayout.conf file (/var/lib/rear/layout/disklayout.conf)?

pcahyna commented at 2021-11-29 17:51:

and do you restore to the same or identical hardware as was the original system, or is the hardware different? I am especially interested in your disk sizes, are they same as on the original hardware?

suraj952 commented at 2021-11-29 18:07:

We are trying to recover data on same machine. so there is no hardware mismatch issue should be occur and the logs we have already provided what we have handy and currently server is crashed.

pcahyna commented at 2021-11-29 18:15:

@suraj952 I am interested in the log file offered in entry 2 on the last screen (/var/log/rear/rear-test something .log). /var/lib/rear/layout/disklayout.conf is also present in the rescue system. The script /var/lib/rear/layout/diskrestore.sh would also be interesting to have. Please fetch them from the rescue system (it should be possible to ssh/sftp to it, if the network is configured) and attach them here.

pcahyna commented at 2021-11-29 18:24:

oh and please try to rerun recover with the -D parameter (rear -D recover) and attach also a log file from this attempt, it will contain much more detail.

suraj952 commented at 2021-11-30 15:23:

Hi Team,

Please find the attached log files which were requested to debug the issue.

pcahyna commented at 2021-11-30 15:41:

Hello Suraj, I don't see any files here. I think you should reply using the web interface instead of e-mail. In the web interface there is a bar titled "Attach files by dragging & dropping, selecting or pasting them." just below the text comment box. Click on it and upload your files.

suraj952 commented at 2021-11-30 15:50:

rear-testl07dc.log

BMR log file.zip

PFA the requested log files in zipped folder.
.

suraj952 commented at 2021-11-30 15:52:

Hi,

All the three files have been attached in zipped folder.

Regards,

Suraj Rawat | Storage Services
Office: +1(408) 722-9387 Extn 0 | Cell: +91-8742939172

pcahyna commented at 2021-11-30 16:09:

Thanks, from the log I see that the problem most likely is "A volume group called rhel already exists." and is due to restoring to the disks with your original system, so the original volume group is still there. I can suggest wiping the old data first using wipefs, as mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1919989#c5. (that is, you boot the rescue system, and first you wipe data the old disks and only then you start rear recover. Or, you can restore to clean disks, if you have some spare disks with the same size as the original disks, and keep the old disks as backup.)

I also suggest to edit /etc/rear/local.conf in the rescue system and add the line MIGRATION_MODE=n. It can be seen from your screenshots that due to the presence of two disks of the same size, ReaR has switched to the so-called migration mode, which is usually used for migrating to a different hardware, and the restoration of LVM is then less faithful. As you are restoring to the same hardware, you don't need migration mode.

suraj952 commented at 2021-11-30 16:46:

latest-log-after_wipefs.zip

We have wiped out the data in same way as given in Bugzilla link and did “rear recover” But still no luck.

Please find the latest logs for the same and let me know if you have any question related to this issue.

pcahyna commented at 2021-11-30 16:55:

Now the layout recreation succeeds and your problem seems to be

Including restore/REQUESTRESTORE/default/200_prompt_user_to_start_restore.sh
2021-11-30 11:30:52.689499158 Please start the restore process on your backup host.

Make sure that you restore the data into /mnt/local (by default '/mnt/local')
instead of '/' because the hard disks of the recovered system are mounted there.

2021-11-30 11:30:52.733135532 Please restore your backup in the provided shell and, when finished, type exit
in the shell to continue recovery.

Have you restored your backup to the recreated filesystems? It seems that you have not and your filesystems are empty. What backup software are you using (what is the value of the BACKUP configuration variable)?

suraj952 commented at 2021-11-30 18:01:

We are using Rubrik backup software.

suraj952 commented at 2021-11-30 19:40:

yes we have tried to restore backup by running “rear recover” and but there was issue related to ownership of filesystem. I believe, that error can be found in log files which we had shared earlier.

DamaniN commented at 2021-11-30 20:28:

@suraj952 and @nmaheve, rear recover only prepares the system for data recovery. You then have to go back to Rubrik and recover the actual data. The Rubrik agent should now be running on the rescue system, and you should be able to access it from Rubrik. Remember that when you recover the actual data from Rubrik to redirect the recover to /mnt/local instead of /.

See Chapter 16 of the ReaR user guide for more details. You should be around step 11 now.

pcahyna commented at 2021-12-01 09:16:

@suraj952 note that Rubrik CDM support was added to ReaR in version 2.6 by @DamaniN in PR #2249, but according to the logs and screenshot, you are using ReaR 2.4, so you must restore the data manually (not sure how, I have never used Rubrik CDM).
Concerning

there was issue related to ownership of filesystem. I believe, that error can be found in log files

-if you mean these lines:

2021-11-30 11:31:42.229193058 Recreating directories (with permissions) from /var/lib/rear/recovery/directories_permissions_owner_group
'bin' -> 'usr/bin'
mkdir: created directory 'etc'
mode of 'etc' retained as 0755 (rwxr-xr-x)
chroot: failed to run command '/bin/bash': No such file or directory
2021-11-30 11:31:42.256494605 Failed to 'chown root:root etc'

then this is not an "issue related to ownership of filesystem", but the fact that the recreated system is empty, there has been no backup restored to the reformatted disks. I see this in the log earlier:

Please start the restore process on your backup host.

Make sure that you restore the data into /mnt/local (by default '/mnt/local')
instead of '/' because the hard disks of the recovered system are mounted there.

2021-11-30 11:30:52.733135532 Please restore your backup in the provided shell and, when finished, type exit
in the shell to continue recovery.

Have you started the restore process when prompted? I suppose this prompt was shown on the screen as well.

(Anyway, this issue has digressed from the original disk reformatting issue, which is a known problem when using disks with prior data, and I believe it is solved already in the development version. For older versions, wiping the disks manually is an easy workaround, as shown by your latest log, which records that the disks were reformatted just fine.)

suraj952 commented at 2021-12-01 10:28:

@pcahyna and @DamaniN Currently server is crashed and as we have ran Wipe on the system so i think backup services also get wiped from the host and Rubrik CDM is also not able to communicate with the Host as RBS status showing disconnected. Can you please share POA per our current scenario.

pcahyna commented at 2021-12-01 14:27:

Sure, everything gets wiped from the host, simply because ReaR reformats all the disks, as the title of the issue ("disk reformatting issue") shows. You need to have the software needed for backup restoration in the rescue system. You haven't added the Rubrik CDM client there when creating the rescue media (ISO or USB)? If not, I suppose it will be posible to copy it to the rescue system from somewhere else and thus resolve your problem, but I can't help you with this, I know nothing about CDM. By the way, can you paste your /etc/rear/local.conf file here?

suraj952 commented at 2021-12-01 14:58:

we are not sure about adding the Rubrik CDM client there when creating the rescue media (ISO or USB) but yes we are trying to install Rubrik CDM services on rescue system at the moment and will share you the /etc/rear/local.conf soon.

suraj952 commented at 2021-12-01 15:33:

Hi,

Please see the output below of /etc/rear/local.conf file here.

***@***.***

Regards,

Suraj Rawat | Storage Services
Office: +1(408) 722-9387 Extn 0 | Cell: +91-8742939172

suraj952 commented at 2021-12-01 16:13:

Is it possible if you can join us on virtual session. our linux team can able to see the file system and you were saying that file syste m is empty so that would be great help if we can work together on the VR.

DamaniN commented at 2021-12-01 16:30:

@suraj952, If you have re-installed the Rubrik CDM agent on the rescue system, you can register the rescue system with your Rubrik server. Assuming that you backed up the server in the past, you will be able to recover the data to it. The rescue system will be registered as a new host, so you will need to perform a redirected recovery of the data. If you need help with registration and restoring the data from Rubrik at this point, please contact Rubrik support.

You may run into a conflict where Rubrik thinks that the rescue system is the original system because it has the same IP address. Normally, the CDM integration with ReaR takes care of this. Since the proper version of ReaR wasn't used, this conflict will have to be taken care of manually. Rubrik Support can help with that.

As listed in the link I provided above, be sure to recover the data in / to /mnt/local. After the data recovery, be sure to exit the ReaR shell prompt and answer the last question. This will allow ReaR to finalize the rescue, and you will be able to reboot the rescue host properly.

gdha commented at 2021-12-01 16:32:

@DamaniN Thank you for your wise words.

suraj952 commented at 2021-12-01 16:42:

No @DaminiN we are not able to install rubrik services into server as server got crashed when we tried to perform recovery from ISO backed up file. We tried to install Rubrik backup agent in crashed server but it's not working.

So if you guys can help over the virtual remote session that would be great. Also see the below snapshot and let us know what next step we need to perform from here.

***@***.***

Regards,

Suraj Rawat | Storage Services
Office: +1(408) 722-9387 Extn 0 | Cell: +91-8742939172
Email: ***@***.******@***.***> | Web: http://www.aarp.orghttp://www.aarp.org/
***@***.***

From: Damani ***@***.***>
Sent: Wednesday, December 1, 2021 10:01 PM
To: rear/rear ***@***.***>
Cc: Rawat, Suraj (TMP) ***@***.***>; Mention ***@***.***>
Subject: Re: [rear/rear] disk reformatting issue during BMR on linux 7.9 physical server (Issue #2720)

@suraj952https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsuraj952&data=04%7C01%7Csrawat%40aarp.org%7Ce2727ec02b1f4a60e65708d9b4e7f214%7Ca395e38b4b754e4493499a37de460a33%7C0%7C0%7C637739730555715255%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qZ%2BIGY7JZ2Q51A1GIc9fHN%2Fq2v6P9q%2BROoW7wlcDfc8%3D&reserved=0, If you have re-installed the Rubrik CDM agent on the rescue system, you can register the rescue system with your Rubrik server. Assuming that you backed up the server in the past, you will be able to recover the data to it. The rescue system will be registered as a new host, so you will need to perform a redirected recovery of the data. If you need help with registration and restoring the data from Rubrik at this point, please contact Rubrik support.

You may run into a conflict where Rubrik thinks that the rescue system is the original system because it has the same IP address. Normally, the CDM integration with ReaR takes care of this. Since the proper version of ReaR wasn't used, this conflict will have to be taken care of manually. Rubrik Support can help with that.

As listed in the link I provided above, be sure to recover the data in / to /mnt/local. After the data recovery, be sure to exit the ReaR shell prompt and answer the last question. This will allow ReaR to finalize the rescue, and you will be able to reboot the rescue host properly.

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Frear%2Frear%2Fissues%2F2720%23issuecomment-983814120&data=04%7C01%7Csrawat%40aarp.org%7Ce2727ec02b1f4a60e65708d9b4e7f214%7Ca395e38b4b754e4493499a37de460a33%7C0%7C0%7C637739730555725214%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=bjmR%2Fxhobdwy38%2BYZyRcfjxqI2XlBCnJCZrNu2CccEs%3D&reserved=0, or unsubscribehttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAWGV3USPQLSZWCW2DHZIANLUOZESRANCNFSM5IRUBLZA&data=04%7C01%7Csrawat%40aarp.org%7Ce2727ec02b1f4a60e65708d9b4e7f214%7Ca395e38b4b754e4493499a37de460a33%7C0%7C0%7C637739730555725214%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=fkq4UH5E9o7UAV5M1090zg2vT6aTAgB%2B9I5TjByfAyk%3D&reserved=0.
Triage notifications on the go with GitHub Mobile for iOShttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Csrawat%40aarp.org%7Ce2727ec02b1f4a60e65708d9b4e7f214%7Ca395e38b4b754e4493499a37de460a33%7C0%7C0%7C637739730555735166%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2FEl3ClxsGRJfqS0EaNn39J3kIehP0kHYzE6xbIvf0mc%3D&reserved=0 or Androidhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Csrawat%40aarp.org%7Ce2727ec02b1f4a60e65708d9b4e7f214%7Ca395e38b4b754e4493499a37de460a33%7C0%7C0%7C637739730555735166%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=uFq3Cz5N626kKXChGK6%2BnNbC%2BvLDdvtWy5lWr3Ddw58%3D&reserved=0.

DamaniN commented at 2021-12-01 18:09:

@suraj952, please create a case with Rubrik support for further help with re-installing the Rubrik agent on the rescue host.

suraj952 commented at 2021-12-01 18:22:

@DamaniN per your suggestion, Case #00286029 have been opened with Rubrik support for the same.

pcahyna commented at 2021-12-02 11:54:

@suraj952

Hi, Please see the output below of /etc/rear/local.conf file here. ***@***.***

I don't see anything unfortunately, I suggest to use the web interface. Anyway, I hope that Rubrik support will be able to help you with CDM. I have a suggestion for you and maybe the Rubrik support: instead of trying to get the Rubrik client into the booted rescue system, wouldn't it be better to restore the entire backup to a different server and then copy the whole subdirectory to the /mnt/local directory at the rescue system during the rescue process, when ReaR asks to restore files? This could be done e.g. via rsync or via tar and ssh, something like tar cf - . | ssh root@crashed_system "cd /mnt/local; tar xpf -" (of course tar will need some more parameters to restore all the attributes properly, but that's the general idea). @DamaniN what do you think?

pcahyna commented at 2021-12-02 13:46:

@DamaniN thanks a lot for looking into the issue.

jsmeix commented at 2022-01-31 11:20:

I think this is meanwhile fixed or done outside of ReaR.

[Export of Github issue for rear/rear.]