#1065 Issue `closed`: ReaR does not support OS booting from CRC enabled XFS (e.g. on Ubuntu MATE 16.04)¶

Labels: enhancement, support / question, fixed / solved / done

danboid opened issue at 2016-11-11 09:42:¶

Relax-and-Recover (rear) Issue Template¶

Please fill in the following items before submitting a new issue:

rear version (/usr/sbin/rear -V): 1.19
OS version (cat /etc/rear/os.conf or lsb_release -a): Ubuntu MATE 16.04 x64
rear configuration files (cat /etc/rear/site.conf or cat /etc/rear/local.conf):

OUTPUT=USB
USB_DEVICE=/dev/sdb1
BACKUP=NETFS
BACKUP_URL=cifs://192.168.1.4/sys
BACKUP_OPTIONS="username=dan,pass=xxxxxxxxxxxx"

Brief description of the issue

I tried rear for the first time last night. Everything seemed to be working until rear had finished recovering my MATE 16.04 install and it said it would not boot because it hadn't installed a bootloader because there was no code for installing GRUB for Ubuntu 16.04 i386.

The main problem here is that rear incorrectly detected a 64 bit Ubuntu distro as being i386 but I'm surprised rear doesn't support installing GRUB for Ubuntu 16.04 i386 / 32 bit! If rear supported any 32 bit distros at all I expected Ubuntu 16.04 i386 would be supported. Does this mean rear has dropped support for 32 bit distros?

I was disappointed with MATE 16.04 so I'm going back to Arch. I hope rear supports installing grub / doing a complete recovery under both 32 and 64 bit Arch?

gozora commented at 2016-11-11 11:08:¶

Hello @danboid
Ubuntu MATE 16.04 x64 is kind of new processor type for me (never had MATE installed ...),

Could you please provide output from getconf LONG_BIT, just to be sure ...

Thanks

V.

danboid commented at 2016-11-11 11:35:¶

Hi Gozora

MATE is my fave Linux desktop - its an updated GNOME 2 that now has GTK3 support etc.

I'm not quite sure what you're asking me to do. Is getconf a script that comes with rear? I've already installed Arch on the laptop I was testing rear / Ubuntu MATE with but I'm willing to give it another go if you think we can fix it, even if I don't want to use Ubuntu (MATE) 16.04 any more. If anything should work with rear, it's Ubuntu.

I think you want me to restore my MATE recovery, manually fix grub to get it to boot and then run getconf LONG_BIT, right?

You didn't answer my question about 32 bit / i386 support. Does rear still support 32 bit distros? If so, how come it doesn't support Ubuntu 16.04 i386? I thought the grub install procedure was the same on 64 bit vs 32 bit anyway?

gozora commented at 2016-11-11 11:44:¶

Well, so far I've dealt mostly with 64-bit Linux version which identified with processor type string x86_64 (not x64). That's why I've mentioned that this kind of string is new for me.
if you execute getconf LONG_BIT on your normally running MATE, it should return integer 32 or 64, which should definitively answer the question if you have 32-bit or 64-bit version of CPU.

Regarding your question about support, yes I'd say code for restoring 32-bit and 64-bit should be same and ReaR should not have any trouble restoring it. My best guess would be that this just some minor issue coming from your lsb_release output and string x64.

danboid commented at 2016-11-11 11:54:¶

Yes, I was running Ubunbtu MATE x86_64 on an i7 laptop. x86_64, x64 and amd64 are interchangeable terms. I just use x64 because its quicker to type.

If the code is the same for restoring 32 and 64 bit (which I believe it should be. despite not having looked) then I should've never got this error. If there was a need for such an error, I would've preferred to get told about it before starting the backup rather than after recovery.

gozora commented at 2016-11-11 11:59:¶

Yes, I agree you should be warned.
But again, what I see in your output in ReaR issue template is confusing me:

OS version (cat /etc/rear/os.conf or lsb_release -a): Ubuntu MATE 16.04 x64

So before diving any deeper into this, I'd like to see what would be output of getconf LONG_BIT

danboid commented at 2016-11-11 12:22:¶

Seeing as I don't have any Ubuntu MATE installs now I went into #ubuntu-mate on Freenode and asked someone runnng Ubuntu MATE 16.04 x86_64 to run getconf LONG_BIT and they got 64

gozora commented at 2016-11-11 12:26:¶

nice, thanks,
I'm about to install 16.4 MATE now. And try ReaR backup restore.

Will keep you posted.

gozora commented at 2016-11-11 12:31:¶

@danboid maybe one further question.
Did you use UEFI or legacy boot?

danboid commented at 2016-11-11 12:37:¶

Legacy / BIOS and I installed to a single XFS partition on sda1 with no swap, if that matters.

gozora commented at 2016-11-11 12:43:¶

thx for info again!

gdha commented at 2016-11-11 13:36:¶

Within rear x86_64 and i386 are the same and rear uses internally i386 as identifier.

gdha commented at 2016-11-11 13:38:¶

@danboid For Arch rear has no up-to-date package so far, perhaps, you can help us with this?

gozora commented at 2016-11-11 13:43:¶

hmm, looks like I've (again) misunderstood something :-).
Just installed Mate and lsb_release -a is showing:

sodoma@mate:~$  lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.1 LTS
Release:    16.04
Codename:   xenial

So this makes sense now:

Yes, I was running Ubunbtu MATE x86_64 on an i7 laptop. x86_64, x64 and amd64 are interchangeable terms. I just use x64 because its quicker to type.

ok, ok, ok
So string x64 is fully irrelevant!

Will try backup restore now.

danboid commented at 2016-11-11 14:22:¶

gdha:

That's strange that rear should see x86_64 and i386 as being the same - I would've thought that was a bug unless you told me it was intentional. Why does it do this?

If they are the same as far as rear is concerned then the error I got would mean rear has no code for installing GRUB for Ubuntu 16.04 x86_64 which is very likely the most popular Linux distro and platform right now.

I've not tried rear under Arch yet but it is in the AUR and it has been updated recently:

https://aur.archlinux.org/packages/relax-and-recover-git/

I will be trying it under Arch soon, this weekend probably. I will let you know if I have any issues.

gozora commented at 2016-11-11 16:07:¶

Ok, I have my first findings.
rear recover ended up with error (I guess) similar to yours:

2016-11-11 15:36:39 Installing GRUB2 boot loader
Installing for i386-pc platform.
grub-install: warning: cannot open directory `/usr/share/locale': No such file or directory.
grub-install: error: unknown filesystem.
chroot: failed to run command 'grub2-install': No such file or directory
...
--------------------  ATTENTION ATTENTION ATTENTION -------------------
|                                                                     |
|          IF YOU DO NOT INSTALL A BOOT LOADER MANUALLY,              |
|                                                                     |
|          THEN YOUR SYSTEM WILL N O T BE ABLE TO BOOT !              |
|                                                                     |
-----------------------------------------------------------------------

You can use 'chroot /mnt/local bash --login' to access the recovered system.
Please remember to mount /proc before trying to install a boot loader.

The main problem here is however not missing ReaR code for i386 platforms.
Message Installing for i386-pc platform is coming from grub-install. Meaning that if you would run grub-install /dev/sda on your running system, you'd get very same message.

The problem you are experiencing is that grub-install can't detect XFS filesystem on /dev/sda. I'm currently trying to find out why ...

danboid commented at 2016-11-11 17:37:¶

Hi Gozora!

Yes, I think you're getting the same error I was. Sounds like not many rear users use XFS if nobody else has reported this. I thought it might be more popular thanks to it being the default in RHEL/CentOS 7 and its much faster fsck times vs ext4 but I suppose most distros default to ext4 still and most people stick with the defaults.

gozora commented at 2016-11-11 18:23:¶

Now I'm not (again) sure if XFS is the real troublemaker here.
Once rear recover finished, I've try manual grub-install /dev/sda but if failed with:
grub-install: error: unknown filesystem..
Then I try MATEs live CD and it failed with same message :-(.

That concludes that mkfs.xfs during rear recover must have done something which is not fully OK.
Anyhow, I'd need to dig a bit deeper into XFS and its options as that would be probably the key.

But as today is Friday evening I'd need to finish a beer or two first ;-) and will continue to work on this tomorrow.

Will keep you posted.

V.

gozora commented at 2016-11-12 21:13:¶

Hello @danboid,

After doing a bit of research on XFS I've concluded following:

You should not use XFS for booting
You should not use XFS for booting
see pt 1. and 2
;-)

But seriously now. I've setup similar configuration on Ubuntu Mate like you had, but with one small change. I've created separate /boot partition with ext3 and ReaR worked as charm.
I've managed to write a patch so if you don't like idea with separate /boot you can git clone https://github.com/gozora/rear and it should work just fine.

The problem you've encountered is quite tricky one. During rear recover following code is executed:

mkfs.xfs -f $device
if [ -n "$uuid" ] ; then
   xfs_admin -U $uuid $device >&2
fi

In case of Ubuntu MATE $uuid is really used, so xfs_admin sets UUID right after filesystem is created.
So far so good. However once you have XFS Self-Describing Metadata enabled (crc=1), you will get incompatible flag set (this should in general avoid older kernels to mount this filesystem). With this flag set grub-install will not be able to detect filesystem type and fail.

A small example:

### Use xfs_admin to set uuid
# xfs_admin -U 2d65defa-2593-4542-b293-26cd93f82711 /dev/sda1
Clearing log and setting UUID
writing all SBs
new UUID = 2d65defa-2593-4542-b293-26cd93f82711

### Flag is now set and grub-probe (resp grub-install) can't recognize filesystem type
# grub-probe -d /dev/sda1
grub-probe: error: unknown filesystem.

### Revert uuid back to original 
# xfs_admin -U restore /dev/sda1
Clearing log and setting UUID
writing all SBs
new UUID = f6cffcf4-49bb-40ed-ad2c-eb754e576e76

### All works fine now
RESCUE mate:~ # grub-probe -d /dev/sda1
xfs

The patch I've created just removes code where xfs_admin changes uuid and delegated uuid creation to mkfs.xfs. As I'm quite new to XFS so I can't tell if if is good or not. It might be that set uuid option was introduced just recently to mkfs.xfs so it might not work with older Linux flavors.

I'd like to ask users in ReaR community who are actively using XFS, if mkfs.xfs -m uuid="<uuid>" was introduced just recently, or it is present longer.

Thanks for any answer in advance!

V.

gozora commented at 2016-11-12 21:27:¶

It was not so hard after all to find information when was option -m uuid=<uuid> added to mkfs.xfs. Funny thing is that I've discovered it short after I click comment :-).
Anyhow, according following link it was added just recently in mkfs.xfs 4.3.0.
So I'll need to modify patch i've wrote.

@gdha @jsmeix or anybody else out there, I'd suggest following:
Try mkfs.xfs -m uuid=<uuid> as first option and once it fails fallback to xfs_admin -U <uuid> would would you say?

V.

danboid commented at 2016-11-13 10:38:¶

Hi Gozora

Thanks for investigating this. I've never been a fan of boot partitions so I've always avoided them where possible. I don't really know much about XFS myself - all I know is that it fscks much faster than ext3/4 and supports larger filesystems so thats why I use it, not that I really need the latter advantage.

I'm hoping we can find a solution to this that doesn't require me having to patch my rear installs to run with XFS without an ext /boot partition. Otherwise a new error message will need to be added to warn users about this issue.

I'm surprised XFS is still getting new features as it must be 20 years old now.

gozora commented at 2016-11-13 11:11:¶

hello @danboid
Well, migrating existing installations to /boot can be indeed painful. Like I already said, you can use https://github.com/gozora/rear, I'm pretty sure it will work just fine with your current MATE setup (it will however fail if you use it with older XFS version resp. kernels).
I can't tell you if this exact code will (and when) be pushed to upstream. See comment, but I'm currently working on XFS code that should work with old and new XFS versions just fine.

I never followed XFS development but it looks like they are still adding new featured and what is (for me) more concerning, they are changing defaults.
A small comparison on SLES11 SP3, Centos 7.2 and Ubuntu Mate 16.4

OS	Self-Describing Metadata (CRC)	Change uuid with mkfs.xfs
SLES11 SP3	NO	NO
Centos 7.2	YES (not default)	NO
Mate 16.4	YES (default)	YES

I can say now with certainty is that current ReaR code will not restore (without user interaction) OS that is booting from CRC enabled XFS.

gozora commented at 2016-11-13 13:45:¶

@danboid
I've managed to put together patch which does not (hopefully) breaks backward compatibility.
Is there a chance for you to test it?
git clone -b xfs https://github.com/gozora/rear.git

danboid commented at 2016-11-13 13:56:¶

I'll try your fork/patch under Arch (installed on XFS w/ no boot partition) soon if not later today. Arch is the OS I mainly want to use with rear.

gozora commented at 2016-11-13 14:02:¶

Good luck! ;-)

danboid commented at 2016-11-13 14:09:¶

Will I need to install your version or can I run it (under Arch but with the same local.conf) without installation? Is rear installation optional? The local.conf would have to be in place, of course.

gozora commented at 2016-11-13 14:16:¶

@danboid
This patch updates just one file, so all you need is download usr/share/rear/layout/prepare/GNU/Linux/130_include_filesystem_code.sh from my repo, and replace your local copy. All other files (including local.conf) can remain unchanged.

danboid commented at 2016-11-13 14:16:¶

Going OT briefly - does incremental backup to a samba share work? Does rear backup to a flat, uncompressed filesystem to do this?

Is there a panel applet to show the status of rear syncs available? grsync might be able to do this.

Finally, what is the deal with restoring multi-boot configs? Can rear restore, or does it plan to ever be able to restore Windows, OSX, FreeBSD and Haiku etc? I presume it can handle restoring multi-boot Linux configs already ie triple booting Arch, CentOS and Debian so long as they don't use ZFS?

gozora commented at 2016-11-13 14:28:¶

I'm not sure how exactly internal incremental backups works in ReaR, I saw some issues that they are not 100% working but don't follow details exactly (see e.g. #1062). We however have support for Borg backup already implemented so if you want to save some disk space, be sure to check scemarios and examples.

Linux multi boot should be working, I however never someone with such request so I'm not entirely sure.
Regarding other OS restore, I'll let ReaR elders to respond ;-), because I really don't have any insides about this.

jsmeix commented at 2016-11-14 10:26:¶

@gozora regarding your
https://github.com/rear/rear/issues/1065#issuecomment-260149486
where you propose basically something like

# try 'mkfs.xfs -m uuid=...' and if that does not work fall back
# to plain 'mkfs.xfs' plus an additional 'xfs_admin -U ...' call:
if ! mkfs.xfs -m uuid=$uuid ... ; then
    mkfs.xfs ...
    xfs_admin -U $uuid ...
fi

plus your very valuable comparison in your
https://github.com/rear/rear/issues/1065#issuecomment-260179601
match perfectly what I wrote about
"Maintain backward compatibility" and
"Dirty hacks welcome" in
https://github.com/rear/rear/wiki/Coding-Style

We already had a similar issue, see
https://github.com/rear/rear/issues/890
in particular therein see
https://github.com/rear/rear/issues/890#issuecomment-228276737
and how I finally implemented it, see
https://github.com/rear/rear/pull/894

That was basically the same kind of issue as this one.

jsmeix commented at 2016-11-14 10:51:¶

I had to fix a typo in my
https://github.com/rear/rear/issues/1065#issuecomment-260299845

I had two times 'mkfs.xfs -m uuid=...'
but this way it should be right:

if ! mkfs.xfs -m uuid=$uuid ... ; then
    mkfs.xfs ...
    xfs_admin -U $uuid ...
fi

@danboid
better wait until I fixed incremental backup sufficiently,
see https://github.com/rear/rear/issues/1062
and https://github.com/rear/rear/pull/1066

danboid commented at 2016-11-14 11:21:¶

gozora:

I didn't have time to test your patch yesterday and today looks unlikely too. Tomorrow night is more likely now.

jsmeix:

It sounds like incremental backups aren't fully implemented yet so I'll hold out on testing that. It is something I'd definitely want to see added to rear but I need to see regular, full backup and restore working first.

It sounds like incremental backup / restore will work with two tarballs. I don't suppose that rear will have uncompress the main recovery tarball to disk when creating new incremental tarballs to work out whats changed will it?

gozora commented at 2016-11-14 11:46:¶

Hello @danboid
No problem with that, code will be probably adapted soon ;-). So I guess it would be better to wait for final patch version.

jsmeix commented at 2016-11-14 12:13:¶

@danboid
in general please keep separated issues separated, i.e. in case of
incremental (actually differential - see the ReaR documentation)
backup / restore issues please submit new separated GitHub issues
so that this issue here is kept only about what its subject tells:
"booting from CRC enabled XFS".

gozora commented at 2016-11-15 07:45:¶

Hello @danboid @jsmeix @gdha,
Sorry I did not had time yesterday to work on #1067 (I had to do some unexpected HW maintenance home ...).
I'll work on it today afternoon.

V.

jsmeix commented at 2016-11-15 13:41:¶

@danboid
regarding your
https://github.com/rear/rear/issues/1065#issuecomment-260188833
"does incremental backup to a samba share work?"
see
https://github.com/rear/rear/issues/1062#issuecomment-260643526

danboid commented at 2016-11-15 14:12:¶

I could test this tonight if the patch is considered ready? Otherwise, please let me know when its ready to test.

Because I'll be testing this with Arch, I formatted the drive myself (via the Arch install ISO) this time with:

mkfs.xfs -f /dev/sda1

Hence this might work around the CRC problem I was having with Ubuntu MATE? I would expect the other flavours of Ubuntu 16.04+ are also affected, if the user chooses XFS at install time.

gozora commented at 2016-11-15 14:16:¶

I think it would be better to wait. I'll correct the patch today and update pull request.
I will drop you a message once it is merged to main ReaR code.

V.

danboid commented at 2016-11-15 14:19:¶

OK - thanks Vlad!

jsmeix commented at 2016-11-16 10:22:¶

With
https://github.com/rear/rear/pull/1067
merged, I consider this issue to be fixed.

danboid commented at 2016-11-16 10:29:¶

I should be able to install the latest git rear and restore XFS installs in a partition-miser friendly way now right?

I'll test it tonight.

gozora commented at 2016-11-16 11:14:¶

@danboid yes, all should be OK now ...

gdha commented at 2017-01-19 08:03:¶

@phracek Is this issue related to https://access.redhat.com/errata/RHBA-2017:0090 and https://bugzilla.redhat.com/show_bug.cgi?id=1399487 ?
Any chance you will be looking at rear-2.00 to replace rear-1.17.2?

sandeen commented at 2017-06-19 15:41:¶

Just a note - in the future, if you have any questions about xfs behavior or tools, please don't hesitate to ask on the xfs mailing list - we're always happy to help. ;)

gozora commented at 2017-06-19 15:51:¶

Hello @sandeen,

Thanks for your kind offer!
I somehow feel that I'll use it in the future ;-), because (as you might been able to read) most thing I've learned about XFS were by "try - fail - learn - repeat". So asking someone who actually knows what is going on, would be a nice change!

V.

[Export of Github issue for rear/rear.]

#1065 Issue closed: ReaR does not support OS booting from CRC enabled XFS (e.g. on Ubuntu MATE 16.04)¶