#1078 Issue closed: NTFS support (via BLOCKCLONE backup method)¶
Labels: enhancement, documentation, fixed / solved / done
danboid opened issue at 2016-11-22 12:45:¶
It would be very useful if rear supported the backup and recovery of ntfs / Windows OS partitions for people who multiboot Linux with Windows, like I do in work.
I'm only expecting rear backup and recovery to work from under Linux and I'd be happy enough if that's the only way it ever worked but now that Ubuntu officially runs under Windows (10) there is maybe scope in the future for having rear work under the Windows 10 bash shell? I'm not really bothered if that is not possible but I do feel the ability to backup and restore NTFS is a bit of a must. Windows support has been the first question I have been asked with 2 of the first 3 people I have told about rear.
I can't remember the details now but I know that if you install Windows and leave it to its own devices it creates at least one boot / recovery partition as well as the OS partition but if you manually install it it can live happily in one NTFS partition, or at least that was the case with Windows 7. I mention this as it could be we can only get this working if the user has a simple config like that ie no fancy Windows boot partitions etc.
I don't know what rear does when it backs up a multiboot Linux system (ie dual booting Debian and OpenSUSE for example) yet but if I was backing up a Linux/Windows dual-boot machine I'd be expecting two separate archives to be produced.
gozora commented at 2016-11-22 13:34:¶
Functionality you are describing here, could be done by simple mount of NTFS volume + backup its content (e.g. using tar). I'm not sure if ReaR is needed for such operation.
danboid commented at 2016-11-22 14:28:¶
The same could be said for Linux. I know how to manually back up and restore Linux and Windows but rear is supposed to make the process easier and that's why I'm interested. More people will be interested in rear if it handles more use cases, the major missing one being Windows.
gozora commented at 2016-11-22 15:44:¶
So what you are proposing is "just" a NTFS support in ReaR?
danboid commented at 2016-11-22 15:55:¶
Yes, or I'd like to know why it has been decided not to support NTFS / Windows?
As it stands, I can only recommend rear to people who only use Linux but I'd like to be able to recommend it to people who multiboot with Windows too. As it stands I'd have to tell them they have to stick with Clonezilla unless they want to backup Windows separately and at that point you may as well do both with Clonezilla or fsarchiver or whatever.
It's not as easy to use as a pre-configured rear setup but at least Clonezilla handles Windows as well as Linux.
jsmeix commented at 2016-11-22 16:00:¶
Let's keep
"having rear work under the Windows 10 bash shell"
out of this issue and focus on
"backup and restore NTFS".
In general regarding backup and restore see
https://github.com/rear/rear/issues/769
and therein see in particular the initial description
https://github.com/rear/rear/issues/769#issue-133203316
that reads (excerpt):
By design ReaR operates on the filesystem level. That means that anything not mounted is out of scope for ReaR. The design goal of ReaR was to take a backup / layout snapshot of a running system
Regarding backup of files on a NTFS filesystem:
It should "just work" to mount the NTFS filesystem
somewhere and mention that "somewhere" in
the BACKUP_PROG_INCLUDE array.
I think the current missing part is the restore of
files on a NTFS filesystem during "rear recover".
To do that one needs to:
"Simply" create a NTFS filesystem under Linux and mount it so that one can "just" write (i.e. restore) files there.
gozora commented at 2016-11-22 16:06:¶
Hello @jsmeix
Reading code in
usr/share/rear/layout/save/GNU/Linux/230_filesystem_layout.sh:
# Comma separated list of filesystems that is used for "mount/findmnt -t <list,of,filesystems>" below:
supported_filesystems="ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs"
# Read filesystem information from the system by default using the traditional mount command
# limited to only the supported filesystems which results output lines of the form
# device mountpoint filesystem (list,of,options)
# for example
# /dev/sda2 / btrfs (rw,relatime,space_cache)
# /dev/sda2 /.snapshots btrfs (rw,relatime,space_cache)
# /dev/sda2 /var/tmp btrfs (rw,relatime,space_cache)
read_filesystems_command="mount -t $supported_filesystems | cut -d ' ' -f 1,3,5,6"
Isn't this a showstopper for NTFS?
gozora commented at 2016-11-22 16:27:¶
I unfortunately don't know exact reasons behind not including NTFS, but
I'm personally not brave enough to read NTFS partitions directly from
Linux (mainly because I don't like messing up with anything related to
Windows ;-) ).
I can however take this challenge and try to recover WIndows with ReaR ,
after all it might be fun ...
jsmeix commented at 2016-11-22 16:35:¶
I think the code in
usr/share/rear/layout/save/GNU/Linux/230_filesystem_layout.sh
is an intentional showstopper for NTFS filesystem recreation
because currently there is no such thing in ReaR.
But I guess what goes into the backup should be
independent of filesystem layout recreation code.
I.e. I think one can get any mounted (i.e. accessible) files
into the backup regardless whether or not ReaR could
recreate the matching filesystems.
But I never ever tested anything with NTFS
because I do not at all use that filesystem
(in particular I do not use Windows).
danboid commented at 2016-11-22 17:11:¶
Good to hear you guys aren't opposed to NTFS support although its clearly not something you have much personal interest in, not being Windows users.
It'll be interesting to hear how you get on gozora. As I said, I think you'll find it much easier to backup and restore Windows if you manually create an NTFS partition first and then install Windows into it as this circumvents the Windows boot partition. Ideally though ReaR would be able to handle backing up and restoring stock Windows (7/8/10) installs which involve more than one partition.
gozora commented at 2016-11-22 17:46:¶
@danboid Yeah, instant regret :-), my MSDN account looks to have some authentication trouble (most probably due lack of use), I'll need to run tests with Windows 7 ... Will keep you posted ...
V.
danboid commented at 2016-11-22 18:37:¶
Thanks Vlad! Sorry to have to inflict a lesser OS on you! :)
So what does rear do when you are backing up a multiboot machine? Does it pack every distro/OS into one tarball or does each OS get archived separately? I ask because in a multiboot config I would like to have the option to only restore specific OSs/partitions instead of all or nothing. I'd expect auto recover to restore them all but hopefully manual recovery would let you choose.
gozora commented at 2016-11-22 19:57:¶
Lesser :-), I'd not say lesser :-), It is different and (for me) very
hard to control.
But I'm certain it have some bright sites which I was unfortunately not
able to discover yet.
Anyhow, I was able to recover my MSDN account so I can run tests with
Windows 10 now (assuming I'll be able to install it)
gozora commented at 2016-11-22 20:02:¶
@danboid and to answer your question. I'd guess that all OSes will be
packed to single archive.
Assuming that I'll be able to recover Windows with ReaR during my tests,
and we have at least small chance that something like this is possible,
we can start discussion what would be the best way to implement it.
Until than it is just wase of time ;-)
jsmeix commented at 2016-11-23 08:43:¶
Current Relax-and-Recover does not support
more than one single operating system, namely
the currently running operating system, cf.
https://github.com/rear/rear/issues/769
I never tried but I assume current Relax-and-Recover
does not support two Linux systems on one harddrive.
Current Relax-and-Recover does not support
more than one single backup method, cf.
https://github.com/rear/rear/issues/769
and only one single backup archive type, cf.
https://github.com/rear/rear/issues/1069
with usually only one single backup archive (e.g. for 'tar'), cf.
https://github.com/rear/rear/issues/1069
where the latter depends on the backup method, cf.
https://en.opensuse.org/SDB:Disaster_Recovery#Relax-and-Recover_.28rear.29_versus_backup_and_restore
I have no personal experience to what extent mkfs.ntfs
actually works to make a NTFS filesystem so that it
actually works with current Windows versions.
I have no personal experience to what extent usual
Linux backup programy like 'tar' actually work to
backup and restore files on a NTFS filesystem so that
after restore it actually works again to be used by
current Windows versions.
jsmeix commented at 2016-11-23 08:46:¶
@gdha
@schlomo
I would like to get some comments from you
what you think in general about adding
NTFS support in ReaR.
gozora commented at 2016-11-23 09:00:¶
I had a small talk with my NT colleagues this morning, and so far it looks like there might be a problem with permissions and unmovable files. I'll however continue with test today afternoon. (at least for my personal curiosity).
danboid commented at 2016-11-23 09:13:¶
Thanks for highlighting all the obstacles between the current state of rear and what would be required to get individual OSs recovered jsmeix - seems there is quite a lot to do to get to that stage.
In the past I've successfully used Clonezilla, ntfsclone and fsarchiver to backup and restore NTFS partitions (mainly Windows 7 and XP) under Linux but I understand the need to reduce deps and I've not tried cloning Windows 10 yet.
gdha commented at 2016-11-23 09:30:¶
In the past (during my mkCDrec times) I first zero-ed the unused space
in NTFS volumes and just used dd + compress to dump the NTFS
volumes. That worked quite well, but I must admit I still believe
ReaR i meant for Linux FS (to recover Linux servers).
gozora commented at 2016-11-23 09:33:¶
@danboid ntfsclone looks like way to go, thanks for the hint.
@gdha fully agree, Windows don't deserve to be restored!!! :-)
jsmeix commented at 2016-11-23 09:35:¶
I have a private secret ultimate plan
( who will attend FOSDEM 2017? ;-)
why I probably would like to implement support for
"Multiple Backup Methods" at some time in the future.
For a possible use-case for "Multiple Backup Methods" see
https://github.com/rear/rear/issues/769#issuecomment-183310560
Now the comment of @danboid here
https://github.com/rear/rear/issues/1078#issuecomment-262463839
and the commenf of @gdha here
https://github.com/rear/rear/issues/1078#issuecomment-262467224
may show another possible use-case for "Multiple Backup Methods":
Use a generic backup method (e.g. 'tar') to backup and restore
files on usual Linux filesystems and use another specific backup
method to backup and restore a Windows NTFS filesystem.
jsmeix commented at 2016-11-23 09:41:¶
I also agree that Windows NTFS filesystems
should just be dumped ( into the dumping ground ;-)
danboid commented at 2016-11-23 09:41:¶
gozora:
I forgot to mention partclone.ntfs. I think thats got more features than ntfsclone. IIRC partclone is what clonezilla uses. The advantage fsarchiver has over these is that it can restore images onto smaller target partitions than they were cloned from, provided there is enough space for the files.
gozora commented at 2016-11-23 09:49:¶
@danboid that one look interesting as well, but I'll try ntfsclone first, as it looks to be part of ntfs-3g package.
jsmeix commented at 2016-11-23 09:50:¶
@gdha regarding your
https://github.com/rear/rear/issues/1078#issuecomment-262467224
"zero-ed the unused space ... and just used dd + compress"
Perhaps this might be a generic way to backup a whole
btrfs filesystem that contains snapshot subvolumes
so that after restore the btrfs snapshots are still
real btrfs snapshots and not duplicated files, cf.
https://en.opensuse.org/SDB:Disaster_Recovery#btrfs
that reads (excerpt and bottom line):
"btrfs snapshot subvolumes cannot be backed up and restored
as usual with file-based backup software"
gozora commented at 2016-11-23 10:08:¶
@jsmeix but that would mean that we avoid user to restore a single file/directory from archive in simple way.
jsmeix commented at 2016-11-23 10:20:¶
One cannot have all:
Unusual filesystems plus all features of usual filesystems.
In general using 'dd' for disk partitions:
This could be a nice generic way how ReaR could
backup and restore any "foreign" filesystems
(even filesystems that cannot be mounted under Linux).
Of course on a multi-boot system "rear recover" can probably
not recover "foreign" bootloaders (except all systems are booted
by a Linux bootloader that is supported by ReaR - i.e. Grub).
But at least after "rear recover" the user would have all his files
back.
Hopefully he can boot his "foreign" systems manually via
appropriate "foreign" external boot media.
danboid commented at 2016-11-24 11:35:¶
Sorry for being OT but I've been wondering about two things related to rear I hope one of you will kindly answer:
-
Can I use the same USB drive, once it has been formatted and used as a rear recovery drive for one host, to recover multiple different host or is it one recovery drive per host?
-
If I lose / accidentally format or whatever my recovery stick whilst I still have the backup on a network share but I don't have a working machine just like the one I'm attempting to recover currently running to do a
rear mkrescue, is there a way to create a working recovery USB drive or ISO that would be capable of restoring such a rear backup using any old Linux box?
I'm asking these question here because I seem to be having trouble subscribing to your mailing list you don't seem to have an IRC channel.
Thanks
jsmeix commented at 2016-11-24 13:19:¶
@danboid
in any case use new separated GitHub issues
for new separated isuses and/or questions.
Otherwise things get messed up and finally
all may end in the middle of nowhere.
jsmeix commented at 2016-11-24 13:26:¶
I asked colleagues about their experience with
ntfsprogs (e.g. mkfs.ntfs) in particular
if it actually works reliably in practice.
I got only very little response.
I assume only a few use NTFS filesystems.
So far what I got as replies:
It works fine for me, and has done so for some time. The only caveat I have for ntfs-3g is that it's handling of 'unclean' NTFS partitions is very conservative, probably with good reason. It will not allow RW mounting of an unclean ntfs filesystem, which means you might need to use a Windows machine to run a filesystem check before you can use it fully again with ntfs-3g. So I wouldn't use ntfs as a filesystem of choice, but it no longer gets in my way when I need to use it.
plus a subsequent reply in particular regarding
handling of 'unclean' NTFS partitions:
This causes small problem with dualboot machine from Win8+. Default powerof for this systems are Hybrid Sleep and NTFS partitions are everytime flagged as unclean + some data can be wrongly journaled. So this feature needs to be disabled on dualboot systems with shared NTFS partition.
danboid commented at 2016-11-24 14:32:¶
The unclean mounting through having hybrid sleep / Fast Startup enabled also applies to Windows 10 as well as W8. Fast Startup is enabled by default but its easy to disable:
- Open Control Panel in the small icons view and click on Power Options.
- Click on Choose what the power buttons do.
- Click on Change settings that are currently unavailable.
- Uncheck Turn on fast startup (recommended).
If you'd prefer to answer my questions via email I'd be happy with that. I think opening a ticket for a couple of small questions seems like overkill.
gozora commented at 2016-11-24 17:59:¶
Just a short ping that I'm still alive and working hard on this topic.
Unfortunately I don't have so much spare time as I would like and things
are progressing quite slow as Windows is not so happy with
(unintentional) damages I'm causing. I never thought how fragile it is
when handled from inside Linux.
I was however able to use ntfsclone to make working copy of my
Windows 10.
The question if and how it should be implemented into ReaR is misery to
me.
@danboid @jsmeix thanks for your inputs on NTFS!
@jsmeix do you want to share with me your private secret ultimate plan? Maybe that could put me to right direction
V.
jsmeix commented at 2016-11-24 18:10:¶
@gozora
I was wondering all the time why nobody asked me
what my "private secret ultimate plan" is regardless
that I tried so much to make it look irresistibly interesting ;-)
But today is a bit too late for such a great announcement
so that you must be patient until tomorrow (and sleep well ;-)
Hint: It is related to
https://github.com/rear/rear/issues/732
jsmeix commented at 2016-11-24 18:17:¶
@gozora
only an untested idea regarding experiments with NTFS:
Perhaps for the very first experiments with NTFS
you can leave out Windows and use it only on Linux,
i.e. create a NTFS with mkfs.ntfs, mount it, put some
files therein, and enhance ReaR to work with that
(first all only on Linux). When this works, one can
try out how far it also works with a NTFS filesystem
that was created and is used by Windows.
gozora commented at 2016-11-24 18:22:¶
@jsmeix Yep, that is stage 3 of my tests (stage 1 install windows :-)
and stage 2 restore it without ReaR intervention just to get some first
slaps from this technology).
First problem I need to solve is that ntfs does not identify as NTFS
but as fuseblk (which I guess covers all possible fuse file systems)
schlomo commented at 2016-11-24 19:04:¶
@jsmeix my thoughts are the usual: Pull requests are welcome as long as they don't break ReaR for non-NTFS users. Changing the ReaR architecture to accommodate the new features will require a lost of discussions. The goal of ReaR is to be the best DR solution for Linux users, whatever their use case might be.
That aside, I would strongly recommend to integrate another backup tool like clonezilla into ReaR and to have that tool take care of all the non-Linux stuff. AFAIK that already even handles partition resizing and other useful things that you probably don't want to invent again.
danboid commented at 2016-11-25 08:10:¶
Thanks for the update Vlad - its good to hear you have cloned W10 OK with ntfsclone. That's a solid start but I'd like to remind you all that I would be hoping for the ability to restore partitions onto smaller ones that have enough space to hold the files which, last I checked, is a feature that fsarchiver allows that the other tools do not. IIRC fsarchiver is written by the same guy who wrote partclone, which is what CZ uses hence this is a limitation of CZ I'd like to not inherit.
If ntfsclone can do this now then great!
jsmeix commented at 2016-11-25 09:50:¶
Regarding my "private secret ultimate plan", see
https://github.com/rear/rear/issues/1085
jsmeix commented at 2016-11-25 10:04:¶
@schlomo regarding your
https://github.com/rear/rear/issues/1078#issuecomment-262832981
In general "don't break ReaR" is mandatory, cf.
"Maintain backward compatibility" in
https://github.com/rear/rear/wiki/Coding-Style
Regarding
"I would strongly recommend to integrate another backup tool
like clonezilla into ReaR and to have that tool take care
of all the non-Linux stuff":
I do very much appreciate this!
When ReaR should also support non-Linux stuff
via other backup tools, it means support for
"Multiple Backup Methods" is a precondition
so that users can use one backup tool for their
Linux stuff (e.g. 'tar') and additionally another
backup tool (e.g. Clonezilla or even dumb 'dd')
for their non-Linux stuff.
For my "private secret ultimate plan" I also need
support for "Multiple Backup Methods" so that
users can use one kind of "backup" tool like 'zypper'
to get RPM package installed and additionally another
backup tool like 'tar' to get afterwards config files
and/or whatever data files installed so that the
final result is a ready-to-use system.
Therefore I will try to implement support for
"Multiple Backup Methods" in ReaR.
But currently I have no good idea how I could cleanly
implement that without changing too much in ReaR.
gdha commented at 2016-11-25 10:17:¶
We may not forget ReaR is used by thousands of users in Linux
production environments, which do not care about Windows (as these run
on separate servers or VMs). Therefore, I strongly would avoid
backup/restore of NTFS file systems as we cannot guarantee the data
integrity at all (it would only be a best effort).
Believe me - I've seen it before with mkCDrec - and I would like to
avoid that history repeats ifself.
jsmeix commented at 2016-11-25 10:29:¶
I fully agree that we must first and foremost care
about current ReaR users and their use cases.
On the other hand NTFS support might be also
of interest for business production environments
in particular for company laptops and workstations
with Linux/Windows dual-boot installations?
But I have zero personal experience with such use-cases.
gozora commented at 2016-11-25 11:48:¶
Hello guys,
I don't think that anybody here really wishes to break current ReaR
code!
I however think that @jsmeix have a good idea with support of multiple
backup methods. If this is done carefully, it would be good start for
implementation of alien file systems like NTFS (or anything else Linux
can handle). Of course it can go either way, but I guess it is worth to
try it.So if you need some help on this topic, count me in!
@gdha
Believe me - I've seen it before with mkCDrec - and I would like to avoid that history repeats ifself.
I'd really like to know more about this, maybe on FOSDEM?
V.
jsmeix commented at 2016-11-25 13:39:¶
I also think it is worth to at least try to go this way in general.
If we find out one cannot guarantee data integrity for NTFS,
that would be only bad luck for one particular alien filesystem
but it would not make the whole idea behind invalid.
@gozora
I look forward to seeing you at FOSDEM 2017!
gozora commented at 2016-11-25 15:19:¶
Hello @danboid,
I basically know what I've needed to know about restoring WIndows (NTFS)
under Linux.
Currently it looks like (at least from my point of view) that we can't
put something like this into ReaR without lots of changes which could
potentially affect lots of users which are running highly production
Linux environments (as mentioned by @gdha in
comment
with which I fully agree btw.).
"Multiple Backup Methods" mentioned by @jsmeix could be a way how NTFS could find its place in ReaR. As mentioned earlier, it can be a dead end, but it is worth a try.
V.
gozora commented at 2016-11-25 16:04:¶
@schlomo I just had a quick look on clonezilla (I never used i before),
and at first glace it looks like it can't make backups of running
system. You need to boot clonezilla live medium in order to make magic
happen.
So inmplementing clonezilla into ReaR would basically mean "just"
putting all the clonezilla binaries into ReaR rescue/recover medium.
Which can be easily done by user with help of arrays REQUIRED_PROGS
or PROGS.
danboid commented at 2016-11-26 08:40:¶
I hope an easy-ish way to implement multiple backup methods is found then.
I get the impression this will be some way off being implemented, maybe a year or two, if it happens at all.
schlomo commented at 2016-11-26 16:38:¶
@gozora you precisely describe the situation with regard to ReaR and non-Linux stuff. Obviously the non-Linux stuff is not active while Linux runs. So using tools from clonezilla should work perfectly well.
You are right that clonezilla does not take backups of running systems which is exactly why ReaR (and previously mkCDrec) was invented. ReaR takes care of the running Linux and out-tasks everything else to clonezilla or another tool.
@gdha you are right with your experience. I would hope that ReaR's modular design is sufficiently robust to give existing users stable use while still allowing new use cases to be added.
gozora commented at 2016-11-27 09:53:¶
Hello @danboid
"easy-ish way" is very relative. We need to look here for a compromise
(which is at least for me quite a challenge) between not breaking
current ReaR mission and implementing something new. Once I have idea
how to do this, coding would be piece of cake.
I'm not abandoning your idea and still continue working on it, because
as I stated couple of times, it is worth trying.
V.
gozora commented at 2016-12-01 20:48:¶
Hello @danboid,
I've tested fsarchiver as you suggested, but somehow I don't have a good feeling about making backups of NTFS on filesystem level. The thing is that I was able to restore my Windows 10 with fsarchiver, I've however stumbled across hundreds of messages like follows during backup:
[errno=31, Too many links]: oper_save.c#349,createar_item_winattr():winattr:lgetxattr(/Windows/WinSxS/x86_microsoft-windows-i..mentation-migration_31bf3856ad364e35_11.0.10240.16384_none_facb7050c43c1570/WininetPlugin.dll,system.ntfs_dos_name): returned negative attribute size
According to my quick research, this problem have something to do with
hardlinks and old DOS names. The reason why you didn't had trouble so
far was that you've backed up Win7.
I've found following
thread describing
all this in more depth.
A quote from mentioned discussion:
WinXP did not use hard links, so there should be no problem.
Win7 has begun to use hard links gradually,
and Win8 uses a lot of hard links.
and another even more relevant
DOS names are an obsolescent feature,
only useful for running very old applications,
but the major issue is that fsarchiver does not
restore files which have dos name errors.
So as a result I was able to restore my OS and it even booted, I was however missing ~800 files. This proves to me that fsarchiver should not be used for OS backup/restores and if ReaR have to be anyhow involved in NTFS backups/restores, I'd much rather avoid interaction with filesystem layer and use ntfsclone.
Any feedback is more then welcome ;-)
V.
gdha commented at 2016-12-02 07:33:¶
@gozora I would avoid NTFS altogether ;=)
gozora commented at 2016-12-02 08:04:¶
Yes agree here @gdha, however ntfsclone looks to me just like a bit smarted brother of dd. And while NTFS will be handled on block level, we will not deal with NTFS :-)
gozora commented at 2016-12-02 08:13:¶
My idea to implement NTFS support would be just to introduce another
backup method (e.g. NTFSCLONE), and use "Multiple backups" feature
recently implemented by @jsmeix.
Doing NTFS support this way, should not have any bad effects on existing
ReaR code.
jsmeix commented at 2016-12-02 08:57:¶
@gozora
what about dumb 'dd' as a simple generic "backup" method
because I think that should work for any unmounted filesystem
(strictly speaking for any block device that is not in use)
in particular it should work for any alien filesystem
and probably even for so called "raw" devices.
Of course 'dd' is dumb and slow and whatever else,
but I think it would at least "simply work" where real
backup methods do not or cannot work.
gozora commented at 2016-12-02 09:04:¶
@jsmeix in general I have nothing against dd and it should work
perfectly fine.
I don't care about slowness until tool is reliable. But that one thing
that ntfsclone knows better is "where NTFS data ends" (that is mine:
"however ntfsclone looks to me just like a bit smarted brother of dd").
Hence ntfsclone will not copy whole block device 1:1 but stops after
there are no more data to backup.
gozora commented at 2016-12-02 09:08:¶
Hmm, thinking of it now, input parameters for dd and ntfsclone should be quite same. So maybe we could do something like BACKUP=BLOCK_CLONE with some configuration options for tool to be used, like:
BACKUP=BLOCK_CLONE
BLOCK_CLONE_BIN=dd | ntfsclone
danboid commented at 2016-12-02 09:18:¶
Thanks for the update Vlad!
I don't have any comment regarding those errors you are getting from fsarchiver because I haven't tried it with Windows 8 or 10 yet. I'll try to have a go with it later.
Going off what you say it sounds that if you can reliably detect NTFS partitions then rear should use ntfsclone and maybe use dd as a NTFS fallback and for alien filesystems.
jsmeix commented at 2016-12-02 10:14:¶
@gozora
I look forward to your pull request :-)
I think the user needs to specify the block device
he wants to BLOCK_CLONE, e.g. something like
the following:
/etc/rear/sda3.conf
BACKUP=BLOCK_CLONE BLOCK_CLONE_DEVICE="/dev/sda3" BLOCK_CLONE_TOOL="dd ibs=2M obs=2M"
/etc/rear/sdb2.conf
BACKUP=BLOCK_CLONE BLOCK_CLONE_DEVICE="/dev/sdb2" BLOCK_CLONE_TOOL="ntfsclone -f --ignore-fs-check"
/etc/rear/sdc.conf
BACKUP=BLOCK_CLONE BLOCK_CLONE_DEVICE="/dev/sdc" BLOCK_CLONE_TOOL="dd ibs=8M obs=8M"
where e.g. on /dev/sda3 there is an alien filesystem,
on /dev/sdb2 there is NTFS, and /dev/sdc is a raw device.
And finally there is the "tiny complication" ;-)
that ReaR must do the partitioning, i.e.
ReaR would have to create at least the right
block devices (i.e. partitions) for the alien filesystems...
jsmeix commented at 2016-12-02 10:21:¶
Regarding partitioning for alien filesystems:
As an initial step let the user specify what to do
e.g. something like /etc/rear/sdb2.conf
BACKUP=BLOCK_CLONE BLOCK_CLONE_DEVICE="/dev/sdb2" BLOCK_CLONE_DEVICE_SETUP="parted -s /dev/sdb mklabel ... ; parted -s /dev/sdb mkpart ... ; parted -s /dev/sdb set ... type ... ; ..." BLOCK_CLONE_TOOL="ntfsclone -f --ignore-fs-check"
and later implement the automatism.
jsmeix commented at 2016-12-02 10:28:¶
I have the gut feeling that partitioning for alien filesystems
should not be done with the current ReaR layout code
but kept strictly separated from it.
I think during "rear -C base_system recover"
nothing about alien stuff should happen.
I think any alien stuff should happen strictly separated
from recreating the Linux base system.
Simply put:
I would prefer that any alien stuff can never ever cause issues
for recreating regular Linux systems.
gozora commented at 2016-12-02 11:08:¶
hello @jsmeix exactly this can't be done with parted.
WIndows used MBR code that is hard to recreate. To my opinion best way
is just to copy MBR 1:1 like:
dd if=/dev/sdb of=/dev/sdc bs=512 count=1
This will copy bootloader code + create partitions.
gozora commented at 2016-12-02 11:09:¶
And of course this partition code will be part of BLOCK_CLONE so indeed that should not cause any trouble.
jsmeix commented at 2016-12-02 11:31:¶
I humbly apologize for my uninformed example
but I am just a poor boy who cannot afford to pay
for non free 'stuff' (I have another word in my mind ;-)
that does not provide what free men need ;-)
Seriously:
I look even more forward to your pull request!
This could be a very interesting template case for the future
how one could cleanly provide support even for problematic
alien stuff in ReaR.
gozora commented at 2016-12-04 16:08:¶
Well I'm poor as well, but some time ago I've exchanged my soul for MSDN
membership :-).
This is what I have so far:
https://github.com/gozora/rear/commit/1db133f0a74bf3094f5196fe874543c50d3bed5a
It is surprisingly small amount of code compared how much time it took
me to make it work :-).
I want to use existing ReaR code used for NETFS backups (at least for mounting and unmounting) without creating even a minimal chance for breaking something that works. So far this is just quick and dirty as I needed to get mount functionality on board as quick as possible, so I'm fully aware that some protos might not even work (so far I've done tests only with nfs://) so I rely on comments from you guys about possible implications of such approach ;-)
I've used following MO for Ubuntu Mate / Windows 10 dual boot:
-
This is most simple scenario when every OS is located on single physical HDD
-
HDD with Windows 10 have 2 partitions (1. boot; 2. c:\)
-
3 config files in /etc/rear:
root@mate:/etc/rear# cat base_os.conf
BACKUP=NETFS
BACKUP_URL=nfs://beta.virtual.sk/mnt/rear
BACKUP_PROG_EXCLUDE=( ${BACKUP_PROG_EXCLUDE[@]} '/media/*' )
root@mate:/etc/rear# cat windows_boot.conf
BACKUP=BLOCKCLONE
BACKUP_URL="nfs://beta.virtual.sk/mnt/rear"
BACKUP_PROG_ARCHIVE="windows_boot"
BACKUP_PROG_SUFFIX=".img"
BACKUP_PROG_COMPRESS_SUFFIX=""
BLOCKCLONE_PROG=ntfsclone
BLOCKCLONE_SOURCE_DEV="/dev/sdb1"
BLOCKCLONE_SAVE_MBR_DEV="/dev/sdb"
root@mate:/etc/rear# cat windows_data.conf
BACKUP=BLOCKCLONE
BACKUP_URL="nfs://beta.virtual.sk/mnt/rear"
BACKUP_PROG_ARCHIVE="windows_data"
BACKUP_PROG_SUFFIX=".img"
BACKUP_PROG_COMPRESS_SUFFIX=""
BLOCKCLONE_PROG=ntfsclone
BLOCKCLONE_SOURCE_DEV="/dev/sdb2"
Now:
-
Backup Ubuntu Mate
rear -C base_os mkbackup -
Backup Windows 10 partitions
rear -C windows_boot mkbackuponly
rear -C windows_data mkbackuponly
Restore part (which does not exist currently) could look like:
rear -C base_os recover
rear -C windows_boot restoreonly
rear -C windows_data restoreonly
jsmeix commented at 2016-12-05 09:15:¶
Wow!
I am deeply impressed!
Dear "MSDN member": Vade retro satana!
cf.
https://en.wikipedia.org/wiki/Vade_retro_satana
;-)
Seriously:
I do appreciate it very much that you know about the
Windows environment because I know basically nothing
in this area.
FYI:
This week I will commit enhancements of the logging
cf.
https://github.com/rear/rear/issues/1096
that is needed to make output useful when running
multiple backups/restores in parallel and I will commit
support for running multiple "mkbackuponly/restoreonly"
in parallel, cf.
https://github.com/rear/rear/issues/1088
jsmeix commented at 2016-12-05 09:25:¶
@gozora
regarding "every OS is located on single physical HDD":
I think that explains the difference between my proposal
using parted to create a partition for Windows and your way
using 'dd' for the Windows MBR.
What I had in mind was a dual boot system on a single disk
(i.e. what I think most normal users have e.g. on their laptops)
where GRUB is used as bootloader for Linux and Windows.
In this case I think it should work to use parted to create
a partition for Windows and afterwards that partition
is filled up with a 'dd' image of Windows.
What I do not know if after such a recovery
GRUB can still boot Windows.
gozora commented at 2016-12-05 09:30:¶
This is something I must check, Windows have some limitations in regards
of on which partition it is installed on ...
I have a foggy feeling for an example that you can't install windows on
other that first partition.
In any case parted will still be useless.
gozora commented at 2016-12-05 09:33:¶
@danboid can you share with us what disk layout do you use?
danboid commented at 2016-12-05 10:11:¶
Hi Vlad
My work drive looks like this:
Disk /dev/sda: 238.5 GiB, 256060514304 bytes, 500118192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xb73b7001
Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 718847 716800 350M 7 HPFS/NTFS/exFAT
/dev/sda2 718848 247420927 246702080 117.7G 7 HPFS/NTFS/exFAT
/dev/sda3 247420928 500118191 252697264 120.5G 83 Linux
sda1/2 = Windows 8
sda3 = Arch
IIRC you don't have to install Windows in the first partition but it does always replace GRUB or whatever bootloader you are using with the Windows bootloader at install time, hence why you should install Windows first then Linux or other operating systems, if you are multi-booting.
Are you saying your current alien fs rear implementation doesn't support Windows being installed on a separate drive? I'd agree that having both installed on the same HD is more common but having Windows and Linux installed on separate drives is also a common config.
gozora commented at 2016-12-05 11:16:¶
@danboid thanks for info, could you do one more thing for me?
dd if=/dev/sda of=/tmp/sda_mbr.img bs=446 count=1 and upload
/tmp/sda_mbr.img somewhere?
I'd like to see bootstrap code area of yours.
Are you saying your current alien fs rear implementation doesn't support Windows being installed on a separate drive?
Fist, this is not "implementation", let's say it is just a "concept"
:-)
Second, just opposite, I just tested situation when each OS was
installed on separate disk. Having all this mess on one disk would be
certainly more challenging. ;-)
danboid commented at 2016-12-05 11:45:¶
MBR attached
gozora commented at 2016-12-05 11:50:¶
Thanks @danboid,
BTW, check sizes ;-)
-rw-rw-r-- 1 sodoma sodoma 606 Dec 5 12:48 mbr.zip
-rw-r--r-- 1 sodoma sodoma 446 Dec 5 11:40 sda_mbr.img
danboid commented at 2016-12-05 11:56:¶
Yes, I noticed that. I was in the mood for some anti-compression ;)
jsmeix commented at 2016-12-08 09:59:¶
@danboid
regarding your
https://github.com/rear/rear/issues/1078#issuecomment-262756069
I seem to be having trouble subscribing to your mailing list
cf.
https://github.com/rear/rear/issues/1113
perhaps that one is related or even the root cause?
gozora commented at 2016-12-09 19:23:¶
Hello all,
Just a short update on my "progress".
I've managed to restore that parody of OS, in spite of its vital
resistance... (It looks to me that MS guys wants reinstall to be only
option how to bring it back to life.)
The biggest problem tends to be partitioning.
Windows looks to need exactly same partitioning layout to be able to
boot. ReaR however seems to create partitions on approximate basis.
Small example:
Original OS
root@onepart:~# parted /dev/sda unit s print
Model: ATA VBOX HARDDISK (scsi)
Disk /dev/sda: 73400320s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
1 2048s 1026047s 1024000s primary ntfs boot
2 1026048s 36386815s 35360768s primary ntfs
3 36388862s 73398271s 37009410s extended
5 36388864s 65011711s 28622848s logical ext4
6 65013760s 73398271s 8384512s logical linux-swap(v1)
After recovery
RESCUE onepart:~ # parted /dev/sda unit s print
Model: ATA VBOX HARDDISK (scsi)
Disk /dev/sda: 104857600s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
1 4096s 1523740s 1519645s primary ntfs boot
2 1523744s 54000093s 52476350s primary ntfs
3 54000096s 104857599s 50857504s extended
5 54000104s 96477195s 42477092s logical ext4
6 96477200s 104857599s 8380400s logical linux-swap(v1)
I'll try to solve this problem without touching original partitioning code. So far only clumsy solutions are on my mind, hope it will get better.
V.
gozora commented at 2016-12-09 20:55:¶
Hmm, I just found #492 talking something about shifting partitions when recovering to disks of different sizes. Exactly my case. I'll try to restore to same sized disk ...
gozora commented at 2016-12-10 10:53:¶
With same sized disks situation looks a bit better. However extended
partition is not correctly created even for same sized disk. It appears
that Linux simply doesn't care, but Windows (again) refused to boot.
Extended partition was create as follows by ReaR rescue/recovery media:
/dev/sda3 36388862 73400319 37011458 17.7G f W95 Ext'd (LBA)
But it should look something like
/dev/sda3 36388862 73398271 37009410 17.7G 5 Extended
So two thing went wrong
- End of extended partition is shifted by 2048 sectors
- LBA flag was incorrectly set for extended partition
As this could affect ReaR regular (Linux restore) operations as well, I'll try to find code responsible for this and correct it. I somehow have the feeling that shift by 2048 will not by coincidence.
gozora commented at 2016-12-11 15:05:¶
Hello @danboid ,
Today I've managed to make successful restore in both discussed
scenarios (having Windows & Linux on same and on separate disks).
I'd very much appreciate if you could give it a try. You can download
code from my Github page:
git clone -b disk_clone https://github.com/gozora/rear.git
During testing, please bare in mind that code is certainly not perfect
and it might happen that you will not be able to recover your OSes at
all! So don't do tests on systems/data you will need ;-).
Here is my configuration that worked for me:
General settings for ReaR
root@onepart:/etc/rear# cat local.conf
OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://beta.virtual.sk/mnt/rear
GRUB_RESCUE=n
REQUIRED_PROGS=( "${REQUIRED_PROGS[@]}" ntfsclone )
BLOCKCLONE_STRICT_PARTITIONING="yes"
BLOCKCLONE_SAVE_MBR_DEV="/dev/sda"
Backup of LInux
root@onepart:/etc/rear# cat base_os.conf
this_file_name=$( basename ${BASH_SOURCE[0]} )
LOGFILE="$LOG_DIR/rear-$HOSTNAME-$WORKFLOW-${this_file_name%.*}.log"
BACKUP_PROG_ARCHIVE="backup-${this_file_name%.*}"
BACKUP_PROG_EXCLUDE=( ${BACKUP_PROG_EXCLUDE[@]} '/media/*' )
Backup of Windows boot partition
root@onepart:/etc/rear# cat windows_boot.conf
BACKUP=BLOCKCLONE
BACKUP_PROG_ARCHIVE="windows_boot"
BACKUP_PROG_SUFFIX=".img"
BACKUP_PROG_COMPRESS_SUFFIX=""
BLOCKCLONE_PROG=ntfsclone
BLOCKCLONE_SOURCE_DEV="/dev/sda1"
Backup of Windows OS partition
root@onepart:/etc/rear# cat windows_data.conf
BACKUP=BLOCKCLONE
BACKUP_PROG_ARCHIVE="windows_data"
BACKUP_PROG_SUFFIX=".img"
BACKUP_PROG_COMPRESS_SUFFIX=""
BLOCKCLONE_PROG=ntfsclone
BLOCKCLONE_SOURCE_DEV="/dev/sda2"
To backup
rear -C base_os mkbackup
rear -C windows_boot mkbackuponly
rear -C windows_data mkbackuponly
To restore
rear -C base_os recover
rear -C windows_boot restoreonly
rear -C windows_data restoreonly
jsmeix commented at 2016-12-12 09:19:¶
In general on "same hardware" replacement disks
(in particular same disk size, ideally same disk model
basically only the disk UUID is different)
ReaR should recreate identical partitioning.
Reasoning as far as I understand it:
In disklayout.conf ReaR stores the values in bytes.
During "rear recover" parted should be called with
that byte values so that one should get the partitions
recreated with the same byte value positions on the
replacement disk. This should result an indentically
recreated partitioning - as far as I can imagine.
Regarding "extended partition is not correctly created
even for same sized disk":
I never used ReaR with extended partitions.
Perhaps there is a bug in ReaR or in parted
or whatever else?
Regarding parted:
There is the alignment-type option for parted.
Perhaps it is parted's alignment that results that
recreated partitions can be a bit shifted?
gozora commented at 2016-12-12 09:53:¶
Hello @jsmeix
Honestly, I did not study layout code as there were no complaints about
it and like said before, I don't want to break existing code.
To overcome this I've put following
code
into layout/prepare/GNU/Linux/100_include_partition_code.sh, which
will (re)create partitioning on disk where Windows resides, 1:1 with
original (only limitation here is that target disk must be same size or
larger than original).
I know that in #1107 you've started campaign against use of sfdisk,
but I think, due my recent experience, that once we use sfdisk to get
data from msdos and sgdisk for gpt, all should be OK.
V.
danboid commented at 2016-12-12 10:12:¶
Hi Vlad
Great news! I doubt I'll have time to test it today but I'll try it as soon as I get the chance.
I realise you say its very experimental so I just wanted to confirm:
-
In your testing, did you try totally removing all partitions (esp. the Windows ones) before attempting a recovery?
-
Should this work with both BIOS and EFI Windows installs? Maybe it makes no difference?
-
Would it be possible to backup and restore with one config file, one backup comand and one restore command? Seeing as the two Windows configs look very similar apart from the partition, it would be nice if this could be made less redundant. Maybe this is just purely prototypical and not what it'll work like when finished?
-
Which version of Windows have you been testing with? Windows 10 Home 64-bit?
Thanks
jsmeix commented at 2016-12-12 10:14:¶
I have no problem with any kind of dirty hacks
that are needed to make things work for now
as log as it is well documented in the code
so that later it could be cleaned up if possible, cf.
https://github.com/rear/rear/wiki/Coding-Style
As far as I know sfdisk does not work with GPT.
When you need sfdisk can you implement a test
that errors out if a GPT partitioning is there?
Something like
is_true "$BLOCKCLONE_STRICT_PARTITIONING" && [[ "$label" = "gpt" || "$label" == "gpt_sync_mbr" ]] && Error "BLOCKCLONE_STRICT_PARTITIONING does not work with GPT"
Such a test should happen during "rear mkbackup/mkrescue"
and not when it is too late for the user during "rear recover".
jsmeix commented at 2016-12-12 10:18:¶
@danboid
baciacally I insist on keeping any alien stuff strictly
separated from recreating the Linux base system, cf.
https://github.com/rear/rear/issues/1078#issuecomment-264422896
gozora commented at 2016-12-12 11:12:¶
Hello @danboid,
In your testing, did you try totally removing all partitions (esp. the Windows ones) before attempting a recovery?
YES
Should this work with both BIOS and EFI Windows installs? Maybe it makes no difference?
I was doing testing with legacy boot. ( I can add EFI code as well, should not be a problem )
Would it be possible to backup and restore with one config file, one backup comand and one restore command? Seeing as the two Windows configs look very similar apart from the partition, it would be nice if this could be made less redundant. Maybe this is just purely prototypical and not what it'll work like when finished?
You can always create your own custom shell script, which will do all work for you. And yes this is just a prototype, all fancy staff can be considered once we have basic concept working ;-)
Which version of Windows have you been testing with? Windows 10 Home 64-bit?
It was 64-bit Windows maybe Enterprise (but I'm not sure)
V.
gozora commented at 2016-12-12 11:15:¶
@jsmeix
As far as I know sfdisk does not work with GPT.
When you need sfdisk can you implement a test
that errors out if a GPT partitioning is there?
Something like
This is what I had on my mind, unfortunately once I've installed Windows
it created msdos type of partitioning, so all current code is related to
msdos.
I guess that Windows will use GPT once I switch for EFI, then I'll adapt
code accordingly.
But first I need to know that it works for somebody else (except me) as
well ;-)
V.
danboid commented at 2016-12-14 11:05:¶
I'm just about to try Vlads NTFS code but it has made me wonder - is there a preferred distro for testing rear with? Preferred for me would mean its quick and easy to install both the OS and rear so ideally a bare bones distro that includes all the rear deps as standard.
gozora commented at 2016-12-14 11:12:¶
Hello @danboid,
I was testing with MATE, but it should work fine with any distro you
like.
danboid commented at 2016-12-14 18:06:¶
Hi Vlad!
I'm most of the way to creating a suitable test env for this now, I
think. I cloned a Win 10 VM, resized its OS partition and installed
CentOS 7 (minimal) into a third (XFS) partition. CentOS seems like a
good distro for testing rear because rear is already in the CentOS repos
so I installed rear with yum to get the extra deps then removed it
before cloning your repo and doing a make rpm and installing the rear
rpm.
The only thing left to do now before I test (I think) is to put the four rear config files into place but I wanted to make sure there aren't any extra options I may need? My plan is to create a samba share on my VM host and have rear backup to/ restore from that. That should be possible, right?
I expect I'll be able to get away with using almost exactly the same config files as you used Vlad but I'm wondering why the following lines in local.conf are required?
GRUB_RESCUE=n
BLOCKCLONE_STRICT_PARTITIONING="yes"
BLOCKCLONE_SAVE_MBR_DEV="/dev/sda"
I thought rear always backed up the MBR, or that was the impression I got seeing as I didn't use any of these options previously and GRUB recovered fine?
Are there any other options you might suggest I use given my test plan? I'm hoping I'll only have to do this once! :)
gozora commented at 2016-12-14 18:52:¶
Hello @danboid
My plan is to create a samba share on my VM host and have rear backup to/ restore from that. That should be possible, right?
Yes that should be fully OK, you just need to adapt following config options (in local.conf) to match your backup location.
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://beta.virtual.sk/mnt/rear
I'm sure you can adapt them to suit your needs ;-)
I thought rear always backed up the MBR, or that was the impression I got seeing as I didn't use any of these options previously and GRUB recovered fine?
You are totally right here, ReaR does all that partitioning and savig
stuff just right, for Windows however we need a bit more to make the
magic happen.
You can forget about GRUB_RESCUE ( I should remove it long time ago ),
but
BLOCKCLONE_STRICT_PARTITIONING and BLOCKCLONE_SAVE_MBR_DEV are
important.
-
BLOCKCLONE_STRICT_PARTITIONING
As Windows uses some strict boot code (see https://github.com/rear/rear/issues/1078#issuecomment-266099227), I was not able to make it boot with ReaR default partitioning code. Once you use this option, partitions will be created EXACTLY same as on original OS. This however have some limits. Biggest limit is that you must restore to same or greater disk than original. Restoring to smaller disk will simply fail with some error message (some off topic) that I've probably did not changed yet, sorry! -
BLOCKCLONE_SAVE_MBR_DEV
Tells ReaR where to look for partitioning data. In other words you put here disk (without partition number) where is your WIndows located. If you have just one disk sharing WIndows and Linux it will be most probably /dev/sda.
The configuration is a bit clumsy but I'm working on it to make it better and maybe even simpler.
Thanks for your help with testing!
V.
jsmeix commented at 2016-12-15 10:32:¶
ReaR does not backup the MBR.
During "rear mkbackup/mkrescue" it remembers
the partitioning in disklayout.conf and
during "rear recover" it recreates the partitioning
from scratch (via diskrestore.sh).
I.e. during "rear recover" one gets a new
from scratch written MBR or GPT via
"parted mklabel msdos" or "pated mklabel gpt".
For alien systems this recreating the partitioning
from scratch via parted does not always work,
cf. the above comments from @gozora and me.
danboid commented at 2016-12-15 13:12:¶
After jumping through hundreds of hoops, I've backed up my dual-boot Win10/CentOS 7 VM as you described and I'm currently cloning my test VM just in case recovery fails.
I presume my only way to recover is to use those three separate commands ie auto recovery currently won't auto restore all three partitions for me, yet? Manual recovery is the only option for recovering NTFS right now?
Should NTFS support get merged, would I be able to use auto recovery and have NTFS partitions auto-recovered?
Back soon with the results...
gozora commented at 2016-12-15 13:20:¶
Hello @danboid,
What is your idea, about how recovery should work?
danboid commented at 2016-12-15 13:26:¶
When I boot the recovery iso, I would expect auto-recovery to restore all partitions (on all drives) by default. Is this not the default behaviour?
gozora commented at 2016-12-15 13:32:¶
Honestly, I don't know if ReaR can do something like this. I saw some
"automatic recover" option in ReaR rescue/recovery system but I've never
used it.
I think it is a matter of taste, I for example don't think starting
recovery automatically is a good idea and prefer more control over it.
I'm not saying that this is final stage how this patch will behave, but cosmetics can be done after we have something working ;-)
danboid commented at 2016-12-15 13:41:¶
I've just tried the automatic recovery option but, as I expected from your response, it didn't work:
ERROR: Backup archive 'backup.tar.gz' not found.
If you are recovering a multiboot system, recovering all partitions in auto-recovery mode seems like the only sensible default to me.
I'll try a manual recovery now...
gozora commented at 2016-12-15 13:55:¶
If you are recovering a multiboot system, recovering all partitions in auto-recovery mode seems like the only sensible default to me.
I would not agree here, Imagine that only your Linux installation is broken (Windows is running just file). In such case restoring Windows can be useless ...
danboid commented at 2016-12-15 14:02:¶
What should auto-recovery do then, if it doesn't restore all partitions?
IMO you should only need to do a manual recovery if you DON'T want to restore all partitions - but that's just me.
jsmeix commented at 2016-12-15 15:48:¶
FYI regarding "automated recovery" see
https://github.com/rear/rear/issues/915
gozora commented at 2016-12-15 15:53:¶
@jsmeix will it work with "Multiple backup methods"?
Like I said before, I've never tried it out, but my idea is that
"automatic recover" just runs rear recover after boot.
We would need
rear -C base_os recover; rear -C windows_boot restoreonly; rear -C windows_data restoreonly
instead.
Maybe another variable in default.conf (AUTOEXEC) would be handy ;-) ?
V.
jsmeix commented at 2016-12-15 15:59:¶
@gozora
I agree with you point of view in your
https://github.com/rear/rear/issues/1078#issuecomment-267332645
cf. what I wrote in
https://github.com/rear/rear/issues/987#issue-173428037
clean up the currently single-big-and-fat "recover" workflow ... Currently "rear recover" is not prepared to skip only certain parts...
Regarding
will it work with "Multiple backup methods"?
As far as I understand it for NTFS backup and restore
using "Multiple backup methods" is mandatory because
the NTFS backup and restore is a separated
and totally different backup method compared to the
backup method that is used for the basic Linux system.
jsmeix commented at 2016-12-15 16:04:¶
@gozora I think I misunderstood your
https://github.com/rear/rear/issues/1078#issuecomment-267362763
Currently multiple backups only works as documented
in doc/user-guide/11-multiple-backups.adoc
i.e. nothing "automated" there - one must manually
specify the right rear commands.
For the future see
https://github.com/rear/rear/issues/1088#issuecomment-263887450
but that will not happen for ReaR 2.0 - but later...
gozora commented at 2016-12-15 16:18:¶
Good, because I've started to to think that error is in my receiver
:-)
No rush though, important thing is that it is on your radar.
danboid commented at 2016-12-15 16:19:¶
@gozora
Unfortunately, whilst both backup and recovery seemed to proceed without errors, recovery failed. Surprisingly, neither Windows nor CentOS would boot after recovery although at least GRUB seemed to get restored OK.
Whilst CentOS (7 minimal install iso) seems like a good choice for testing rear to me, its not perfect for testing NTFS support as you have to add the EPEL repo to get ntfs-3g and ntfsprogs and EPEL doesn't work until you disable SSL for its repo. Also, the COS installer didn't detect my Windows install so I had to manually add it to the GRUB config.
Another hiccup on my way to creating the backup was that I was unable to disable fast boot under Windows 10 when running it under QEMU/KVM, which prevented me cloning the Windows data partition straight away. Rather than forcing ntfsclone to ignore the unclean partition, I rebooted into Windows and then shut it down using the command:
shutdown /s /f /t 0
After booting back into CentOS, I was able to backup the Windows data partition without using the force switch.
My config local.conf looked like:
OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="username=dan,pass=MyAmazingPassword"
BACKUP_URL=cifs://192.168.35.5/rearshare
GRUB_RESCUE=n
REQUIRED_PROGS=( "${REQUIRED_PROGS[@]}" ntfsclone )
BLOCKCLONE_STRICT_PARTITIONING="yes"
BLOCKCLONE_SAVE_MBR_DEV="/dev/sda"
USE_DHCLIENT=yes
The other config files I kept identical to the Vlad's but I also left /etc/rear/os.conf in-place as it seemed harmless to me:
OS_VENDOR=RedHatEnterpriseServer
OS_VERSION=7
I will attach screenshots of the errors displayed when I try to boot
Windows. Push Enter, F8 or F9 does nothing (in true Windows stylee).
danboid commented at 2016-12-15 16:26:¶
One other thing I should've mentioned - after backing up my test VM with
rear and before recovery I booted systemrescuecd on my test VM and nuked
its partition table using the o option in fdisk.
gozora commented at 2016-12-15 16:29:¶
Hello @danboid,
I know these screens!!! (maybe I should start to consider to switch to
NT support :-) ). I've got exactly these screens when partitions had
different parameters!
Could you please do following:
-
on your source OS
Get partition parameters (and put them here) of your disk where Linux and Windows are installed like:
parted /dev/sda unit s print -
boot ReaR rescue/recovery system and restore Linux only with:
rear -C base_os recover
Once done trigger same command (and post result here)
parted /dev/sda unit s print
Thanks
V.
gozora commented at 2016-12-15 16:29:¶
One other thing I should've mentioned - after backing up my test VM with rear and before recovery I booted systemrescuecd on my test VM and nuked its partition table using the o option in fdisk.
This should not be a problem
gozora commented at 2016-12-15 16:44:¶
One more question @danboid, your NTFS partition(s) were mounted or unmounted during backup?
danboid commented at 2016-12-15 16:46:¶
Hi Vlad
By source OS I presume you mean my pre-rear'd VM, right? (ie the clone of my test VM that still boots CentOS and Windows fine):
[root@localhost rear]# parted /dev/sda unit s print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sda: 130023424s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
1 2048s 1026047s 1024000s primary ntfs boot
2 1026048s 89001983s 87975936s primary ntfs
3 89001984s 130023423s 41021440s primary xfs
I then booted the rear recovery iso on my test VM that doesn't boot and ran those two commands, the output of which you can see in the attached screenshot:
danboid commented at 2016-12-15 16:50:¶
My NTFS partitions were unmounted during backup and Windows still fails to boot after re-recovering base_os.
gozora commented at 2016-12-15 16:53:¶
By source OS I presume you mean my pre-rear'd VM, right? (ie the clone of my test VM that still boots CentOS and Windows fine):
Yep
Partitioning looks good, despite my previous suspicion.
I assume that you've restored to same virtual HW as you've made backup
on, is that correct?
danboid commented at 2016-12-15 16:56:¶
Yes, I restored on the same VM as I backed up on. The parted text output in my last but one post was took from the clone of my test VM, which still boots OK.
jsmeix commented at 2016-12-15 16:57:¶
@danboid
only FYI regarding screenshots from a ReaR recovery system:
You may have a look at
https://en.opensuse.org/SDB:Disaster_Recovery
therein the section about
"First steps with Relax-and-Recover"
and therein the part about
"run 'rear recover' from remote via ssh"
that reads (excerpts):
In /etc/rear/local.conf set USE_DHCLIENT="yes" SSH_ROOT_PASSWORD="rear"
This way you can access a running ReaR recovery system
from remote via ssh and then it is usally easier to get
logs and things like that from the ReaR recovery system.
danboid commented at 2016-12-15 16:59:¶
@gozora
You've prob noticed in my last attached screenshot that sda1 was aligned differently after the original recovery than when I restored base_os a second time. After restoring a second time, the sectors match those on my cloned VM but this didn't make it boot.
danboid commented at 2016-12-15 17:12:¶
@jsmeix
Thanks, I'll be sure to add that into my config next time!
gozora commented at 2016-12-15 17:14:¶
@danboid this just hit me in the head, you wrote:
Unfortunately, whilst both backup and recovery seemed to proceed without errors, recovery failed. Surprisingly, neither Windows nor CentOS would boot after recovery.
I'm interested in unbootable Centos, do you know what was the problem there? Did you saw at least Grub2 menu?
gozora commented at 2016-12-15 17:25:¶
You've prob noticed in my last attached screenshot that sda1 was aligned differently after the original recovery than when I restored base_os a second time. After restoring a second time, the sectors match those on my cloned VM but this didn't make it boot.
I must be blind or something ..
Your OS where you've run rear -C base_os mkbackup have this partition
layout:
Number Start End Size Type File system Flags
1 2048s 1026047s 1024000s primary ntfs boot
2 1026048s 89001983s 87975936s primary ntfs
3 89001984s 130023423s 41021440s primary xfs
and same numbers are shown on your screenshot after you've done
rear -C base_os recover
danboid commented at 2016-12-15 17:37:¶
Yes, I see the GRUB2 menu fine but all 3 Centos options fail to boot. The CentOS progress bar reaches the top, turns white and then booting stops.
If I edit the GRUB entries for CentOS to remove rhgb quiet so that it
shows the boot messages you can see it only gets as far as loading
Network Manager before it gives up.
danboid commented at 2016-12-15 17:40:¶
@gozora
Look near the top of my last-but-one screenshot where it says "Old situation:" and you'll see sda1 was aligned differently after the first restore.
I'm off home now so any more questions will have to wait until tomorrow.
Thanks Vlad!
gozora commented at 2016-12-15 17:59:¶
Look near the top of my last-but-one screenshot where it says "Old situation:" and you'll see sda1 was aligned differently after the first restore.
That is fully OK, that is effect of BLOCKCLONE_STRICT_PARTITIONING. I was describing this earlier. ReaR does not create exactly same partitioning with original. And in order not to break current code, I just let let ReaR to run original partitioning code and once it is finished, I just rewrite it again so it matches original. (this might change in the future).
I'm a bit concerned with your last screenshot. In general Linux should be able to boot regardless if you try to restore windows or not. This might be some problem not related to NTFS topic. I'll try to reproduce it somehow. :-).
Just for summary, can you please describe exact commands you've used for backup and for restore + add content of files in /etc/rear directory?
Thanks
V.
gozora commented at 2016-12-15 18:26:¶
@danboid , one more idea. maybe you could do this test with some more "pop" distro ;-). Something like Ubuntu, Fedora, OpenSuSE ?
danboid commented at 2016-12-15 19:13:¶
I'll attach my exact config files tomorrow but I believe I've already given you enough info for you to recreate my rear config exactly - at least as far as the rear confg files are concerned.
I had the same thoughts as you wrt Linux not booting. Does rear do any kind of checksumming for creating the backup archives? I know clonezilla asks if you want to perform checks before and after you clone when you use its advanced mode. Maybe I can force rear to do checksums with an additional option? As I said, no errors were reported during the backup process for all three partitions, same goes for when I was restoring them.
CentOS, probably because its the most popular RHEL derivative, is THE most popular (Linux or Windows) server OS in the UK, by a long shot so I suspect its one of the most popular server OSs worldwide alongside Debian, RHEL and Ubuntu. CentOS pretty much is Linux as far as the datacentres in the UK are concerned. rear HAS to work with CentOS, if nothing else.
It would be interesting to try it with CentOS 6 too as that prob has more installations than C7 presently.
gozora commented at 2016-12-15 19:37:¶
I'll attach my exact config files tomorrow but I believe I've already given you enough info for you to recreate my rear config exactly - at least as far as the rear confg files are concerned.
Wrong, you never can have enough info ;-)
I'm not suspecting that you had corrupted archive. But that something
else went wrong.
I have my dualboot Centos 7 up and running now, and I'll try to
reproduce soon.
ReaR is working just fine with Centos, I think that you can try it even with Centos 5 ;-)
danboid commented at 2016-12-15 20:18:¶
Some other things I should mention:
I installed CentOS 7 minimal x86-64 (64-bit) to a single XFS partition with no swap and no boot partition. The installer does nag at you for not creating a swap partition but I've been swapless for years now and happy! I realise this isn't always an option such as people who use suspend to disk - they need swap.
I used gparted on systemrescuecd to shrink my Windows data partition to make way for the CentOS partition. Prob doesn't matter but...
I used the virtio network adapter under QEMU for my test VM. I gave it 2GB RAM but other than that, it is virt-manager defaults for the VM hardware.
My samba server was running on the same (Arch Linux) box as the VM but I still had to punch holes in the firewalld config for smb to work. Again, this is likely unrelated but worth noting if you wanted to replicate my testing method.
I made sure I installed ntfs-3g, ntfsprogs and all the usual samba packages under C7.
The KVM VMs are stored in qcow2 files.
I used the C7 installer defaults as far as filesystem options are concerned.
I can't think of anything else worth mentioning right now...
danboid commented at 2016-12-15 20:44:¶
I did a full yum update after installing C7, rebooted the VM then
installed rear, added the EPEL repo and the aforemtioned extra deps of
samba and ntfsprogs etc.
gozora commented at 2016-12-15 20:49:¶
Thanks!
I think I have all I needed. After installation of Centos7 and Windows10
I'm in same situation as you are.
Neither Linux nor Windows are booting.
For Linux I'm suspecting selinux, and Windows heaven knows ...
I'll update you once I know more.
V.
danboid commented at 2016-12-16 00:27:¶
Good to hear its not just me then!
Thanks V!
jsmeix commented at 2016-12-16 08:36:¶
@danboid
only a side note FYI:
If you ever test with openSUSE do yourself and me a favour and
do not use SUSE's complicated default btrfs structure for your
test systems - unless you like to test especially btrfs issues, cf.
https://en.opensuse.org/SDB:Disaster_Recovery#First_steps_with_Relax-and-Recover
danboid commented at 2016-12-16 09:14:¶
@jsmeix
I presume you are mainly (only) a Suse user? Do you work for them? I've not tried (open)suse in years so its time I gave it another go really.
I decided some time ago to never use BTRFS. It's just a mess. I run FreeBSD 11 on my home server because FreeBSD has more mature ZFS support than Linux. I'd like to see a Linux distro that supports ZFS as well as FreeBSD does but licensing makes that tricky or impossible.
I prefer to use XFS under Linux but in some cases ext4 is the only option or a better fit. In kinda related news Wayland is finally coming of age but I just have no hope for BTRFS. It seems fundamentally flawed and poorly designed from everything I hear about it. Maybe bcachefs will bring a good, modern filesystem to Linux that distros can actually use?
jsmeix commented at 2016-12-16 10:40:¶
I work for SUSE, see
https://en.opensuse.org/User:Jsmeix
I cannot recommend what filesystem(s) to use
for production systems (I am not a filesystem expert).
I only recommend what to use for testing ReaR
to avoid issues with filesystems - unless the goal is
to verity that ReaR works with particular filesystems.
In contrast testing how ReaR works with the filesystems
that are used on production systems is mandatory before
ReaR is used for disaster recovery on those systems, cf.
"No disaster recovery without testing and continuous validation"
at
https://en.opensuse.org/SDB:Disaster_Recovery
To not use btrfs on openSUSE in the YaST installer's
"Suggested Partitioning" dialog click on
"Edit Proposal Settings" and then select
"Ext4" as "File System for Root Partition".
gozora commented at 2016-12-19 17:34:¶
Looks that I was right about Linux part. (in
https://github.com/rear/rear/issues/1078#issuecomment-267440144).
Once I've disabled SELinux and rerun the backup, Centos 7 booted just
fine after recovery.
I'll try to have a look on it once I have a bit of spare time.
Now I need to figure out what is wrong with Windows part ...
gozora commented at 2016-12-19 18:36:¶
Hello @danboid,
Hope you still remember me ;-). I was off during weekend playing Kerbal Space Program, somehow missing mood to solve problems, but now I'm back :-).
In
https://github.com/rear/rear/issues/1078#issuecomment-268026797
I've found reason behind unbootable Linux. I don't have patched it yet,
but if you disabled SELinux and rerun backup, your Linux should boot
just fine.
For a case you are not familiar with SELinux you need to edit
/etc/sysconfig/selinux as follows:
- SELINUX=enforcing
+ SELINUX=disabled
and
reboot
rear -C base_os mkbackup
Now for Windows part. Once you've finished with recovering your Windows data partition try to poke a bit your MBR like this AFTER last part of NTFS restore is done:
- while in ReaR rescue/recovery system you should have file sda_mbr.img (or simmilar) in /var/lib/rear/layout/
- execute
dd if=/var/lib/rear/layout/sda_mbr.img of=/dev/sda bs=446 bs=1andreboot - now Windows should boot just fine
Hope this time it will work for you just fine. I'll of course implement this (somehow) into ReaR so user will not need any "hacking" in future.
danboid commented at 2016-12-20 09:35:¶
Hi Vlad
No probs about the wait! I do realise you have a life outside of developing ReaR! I've not played Kerbal Space but one of my co-workers at my last job was a fan of that.
As for the CentOS boot prob with SELinux, it seemed obvious when you suggested it. I was surprised I must've never tried cloning a box that had SELinux enabled before as otherwise I would've disabled it, which is something I normally do on a COS/RHEL install I'm actually using.
More importantly for this ticket, I just ran:
dd if=/var/lib/rear/layout/sda_mbr.img of=/dev/sda bs=446 bs=1
On my rear test VM and as you say, Windows 10 boots fine now! Yippee!
Good work Mr G!
gozora commented at 2016-12-26 16:46:¶
Hello @danboid,
I've updated my ReaR
fork, with patched
code according latest findings we've made.
I've also made successful test with restoring Windows 7, so the basic
idea looks to be working.
Now I'll take care about cloning of disks using dd. Hope it will be
same fun ...
Happy holidays everybody!
danboid commented at 2016-12-28 10:21:¶
Thanks for the update Vlad!
Happy holidays to you too!
gozora commented at 2017-01-19 09:15:¶
Pull request https://github.com/rear/rear/pull/1172 was created with initial BLOCKCLONE code
danboid commented at 2017-01-19 09:41:¶
Thanks for the update Vlad!
Has anyone else tried the ntfs-cloning yet apart from me and you?
Do you plan to streamline or improve it any more or is it pretty much done now?
gozora commented at 2017-01-19 09:50:¶
Hello @danboid,
As far as I know you are that only privileged tester so far (apart from me ;-) ).
No it is not yet done, I'll try to add at least GPT/UEFI support. Except that, I think that anybody can write improvements to code so only time will show how all this would develop.
V.
gozora commented at 2017-01-26 16:19:¶
Hello @danboid,
Documentation for BLOCKCLONE backup method was merged today to upstream
code.
As only known user who actually tested BLOCKCLONE, I'd appreciate if you
could have a look on
documentation
and tell me your thoughts on it.
I've tried to write is as easy as possible, but it is quite hard to keep
a distance especially if you are documenting your own code.
Thanks
V.
jsmeix commented at 2017-01-26 16:24:¶
With
https://github.com/rear/rear/pull/1172
"BLOCKCLONE backup method"
and
https://github.com/rear/rear/pull/1180
"BLOCKCLONE documentation"
merged
there is now NTFS support in ReaR
so that I consider this issue to be solved.
Again @gozora contributed a very valuable part to ReaR.
Many thanks for that!
@danboid
now it is your turn to test if the
BLOCKCLONE backup method
also works for you.
In case of issues:
For each separated issue a separated GitHub issue
should be created to keep separated things separated.
jsmeix commented at 2017-01-26 16:34:¶
@danboid
for using the BLOCKCLONE backup method
a prerequirement is understanding how
"Using Multiple Backups for Relax-and-Recover"
works, see
https://github.com/rear/rear/blob/master/doc/user-guide/11-multiple-backups.adoc
danboid commented at 2017-01-27 09:21:¶
Great to see NTFS support for rear! Thanks for all your hard work @gozora ! I will never actually use BLOCKCLONE mode for NTFS but I understand the need for a fallback.
I will try to give blockclone mode a test soon but I'm short of free time now until I get a new place to live sorted. I got told last weekend I have to move out of my house by the end of Feb.
[Export of Github issue for rear/rear.]