#943 Issue closed: P2V HP microserver to VmWare

Labels: support / question, fixed / solved / done

mzlumin opened issue at 2016-07-26 14:16:

Hey Guys,

first of all, thank you for this epic and outstanding fabulous tool! I love using ReaR at my home environment.

However i am a bit lost and wanted to ask if you may can help me out to get my physical Server backed up.

  • rear version :
    Relax-and-Recover 1.18-git201607251102 / 2016-07-25
  • OS version :
    Distributor ID: Ubuntu Description: Ubuntu 15.10 Release: 15.10 Codename: wily
  • rear configuration file (local.conf):
    OUTPUT=ISO BACKUP=NETFS BACKUP_URL="cifs://192.168.0.13/g/BACKUPS/rear" BACKUP_OPTIONS="cred=/etc/rear/creds" BACKUP_PROG_EXCLUDE=( '/proc/kcore' '/home/zuli12/downloads/*' '/dev/shm/*' '/mnt/*' '/media/*' )

Some Information on the original System as rear dump output:
https://gist.github.com/mzlumin/6d12b93b2a90f0c66d2eddeeed302ffb

rear savelayout output:
https://gist.github.com/mzlumin/ee5c04923a1a3ee7171a0554121ccd10

gdha's disklayout build script output:
https://gist.github.com/mzlumin/66a2daf960eba5832cc077d6c2b701c8

I am running LVM over an 2TB HDD, i have build this using the default Ubuntu Server install proccess and i am trying to recover this one on an VmWare 12 Workstation Pro ( i have been able to restore 2 VM's without Issues )

Phsysikal CPU Info:
https://gist.github.com/mzlumin/db241e02f73fdb5b0ceb0b4e34829536

VmWare Host CPU is a Intel Core I5
Prozessor Intel(R) Core(TM) i5-3570 CPU @ 3.40GHz, 3401 MHz, 4 Kern(e), 4 logische(r) Prozessor(en)

OK so if i try to recover from the saved backup i run through the interactive menu until i face some parted faults, please see here:

unbenannt1

unbenannt

I have also tried to modify the script and layout, however it seem not to work for me, i would ask you for some help in this Case :(

Thanks,
Mario

gdha commented at 2016-07-29 08:25:

See also the discussion of issue #102 - it is about the same

jsmeix commented at 2016-07-29 08:56:

@gdha
I think the parted "Warning ... not properly aligned ..."
is https://github.com/rear/rear/issues/102
but that does not let "rear recover" fail.

I think the subsequent parted "Error" is a different issue
and that one lets "rear recover" fail.

@mzlumin
I have no personal experience in using rear for migration.
I think in general using rear for migrating a system
onto different kind of "hardware" is tricky and depending
on the exact difference it may become a complicated task.
I think in your case you need to manually adapt the
var/lib/rear/layout/disklayout.conf file into something
that matches the new "hardware", cf. the section
"Restore to different hardware" in
doc/user-guide/06-layout-configuration.adoc
Again:
I have no personal experience in using rear for migration
so that my above information could be totally wrong.

jsmeix commented at 2016-07-29 09:17:

Perhaps my curently first steps in https://github.com/rear/rear/issues/841
to implement rear recovery system update support
could be already useful here:

Instead of manually changing files like
var/lib/rear/layout/disklayout.conf
in the running rear recovery system
my currently implemented first steps in
https://github.com/rear/rear/issues/841#issuecomment-229069740
to download and install files into
the running rear recovery system
could make this task much easier for the user.

@gdha @mzlumin
if you are interested I could do a pull request and merge it
to have its current state available in rear master code
so that you could test it and try out if it is useful for you
and provide feedback and issue reports if it does not work.

jsmeix commented at 2016-07-29 09:28:

I created https://github.com/rear/rear/pull/945

@gdha
if you like it, please merge it.
It is a pure additional functionality so that
it cannot cause backward incompatible regressions.

mzlumin commented at 2016-07-29 10:12:

@jsmeix @gdha
Thanks for the heads up on this and the answers.
I will look into #841 , i will surely try it out on my site if you want to.

I consider this also might help a few other people out there in such a Case ? :)

If this one is merged i need to get the newest revision, correct ?
Is there anything i need to look at afterwards, or should it work out of the Box ? Just some silly enduser questions :)

jsmeix commented at 2016-07-29 10:47:

@mzlumin
in general regarding how to test the currently
newest rear GitHub master code:

Basically "git clone" it into a directory and
then run rear from within that directory.

# git clone https://github.com/rear/rear.git
# cd rear
# vi etc/rear/local.conf
# usr/sbin/rear -d -D mkbackup

mzlumin commented at 2016-07-29 11:52:

Thanks @jsmeix will do that.
Is it okay if i do the clone now and try it out ? Or do i need to wait for the merge ?

Do i need to run anything advanced then or is trying and report if that works enough for you ?

Thanks

jsmeix commented at 2016-07-29 12:27:

As long as it is not merged into rear master
cloning the rear master code does not contain it.

When it is merged clone it and then read my comments
how to use it (in particular my comments in default.conf)
and for an example how it I used it see
https://github.com/rear/rear/issues/841#issuecomment-229069740

mzlumin commented at 2016-07-29 13:18:

Thanks will do that as soon as possible after the merge ! :)

jsmeix commented at 2016-08-01 10:41:

Right now I merged https://github.com/rear/rear/pull/945
so that you can try it out.

See default.conf how RECOVERY_UPDATE_URL works.

Note that this new RECOVERY_UPDATE_URL
functionality does not at all do anything to make
your initial problem P2V migration go away.

The RECOVERY_UPDATE_URL functionality
only could help you to adapt the rear recovery system
more easily on the new hardware so that "rear recover"
works on the new hardware.

In short what I think you should do is:

On the old system update rear to the newest rear GitHub
master code as described above
in https://github.com/rear/rear/issues/943#issuecomment-236151041
and have in your etc/rear/local.conf (in the directory
whereto you "git cloned" the newest rear GitHub master code)

RECOVERY_UPDATE_URL="http://your_HTTP_server/$HOSTNAME.rear_config.tar.gz"
REQUIRED_PROGS=( "${REQUIRED_PROGS[@]}" curl )

and then run "rear mkbackup" there.

Now you have a rear recovery system with the
new RECOVERY_UPDATE_URL functionality.

When you boot the new hardware with the above created
new rear recovery system and run "rear recover" there,
it will download

http://your_HTTP_server/$HOSTNAME.rear_config.tar.gz

from your HTTP server and install it into the
rear recovery system before it proceeds.

This way you can provide for example an adapted
var/lib/rear/layout/disklayout.conf to be installed
into the rear recovery system before it
creates partitions, filesystems, and so on
so that you can get partitions, filesystems, and so on
created by "rear rcover" on the new hardware
as you need it there.

Probably your first experiments with an adapted
var/lib/rear/layout/disklayout.conf will fail but
with the RECOVERY_UPDATE_URL functionality
it should be much easier for you to do several
trial and error attempts until you got it right
because now you can update the disklayout.conf file
on your HTTP server, pack it into a *.tar.gz and
reboot the new hardware with the same above created
new rear recovery system but now it will install
the updated disklayout.conf file from your HTTP server.

Bottom line:
With the RECOVERY_UPDATE_URL functionality
you get shorter/easier trial and error cycles when you
need adaptions in the rear recovery system to make
"rear recover" work.

mzlumin commented at 2016-08-01 11:31:

Thanks @jsmeix i am on it to create the backup at this time, and i have build up an simple WebServer to server the config Files.

This will really help a lot to get things changed quicker! Thanks for the merge.

However i consider i will struggle to get this working with my experience of partitions and stuff though.

But i will give it a try and report back for sure.

jsmeix commented at 2016-08-01 11:59:

@mzlumin
many thanks in advance for your feedback about
the RECOVERY_UPDATE_URL functionality,
see https://github.com/rear/rear/pull/945#discussion_r72958496

The new RECOVERY_UPDATE_URL functionality
will require further enhancements...

At least it should help you to struggle faster ;-)

jsmeix commented at 2016-08-01 12:10:

@mzlumin
I think for your tests you could simplify it and
leave out the $HOSTNAME indirection and
use a fixed tar.gz so that you can use a fixed URL like:

RECOVERY_UPDATE_URL="http://your_HTTP_server/P2Vstuff.tar.gz"
REQUIRED_PROGS=( "${REQUIRED_PROGS[@]}" curl )

mzlumin commented at 2016-08-01 12:36:

No worries @jsmeix i think i will get the Update_URL functionality to work, also i have added ssh access, seems to work better on VmWare though :)

Let's see how far i come with this one.

Thanks!

jsmeix commented at 2016-08-01 13:12:

@mzlumin
if you need scp support for automated update
via RECOVERY_UPDATE_URL like

RECOVERY_UPDATE_URL="scp://user:password@host/path/to/file.tar.gz"

I think I could add this relatively easily.

mzlumin commented at 2016-08-01 13:44:

Wow, basically i think this would also help some people out there, in my particular case i don't need scp, but thanks for the information!

I consider i might need more help with the actual recovery as i am somehow too silly to get this working.

I will post the results with the new git version soon, will get this done in the next 2 hours i think.
A quick view showed it does download the file just fine, i will need to look into it to say something.

Will keep you posted - Thanks @jsmeix

mzlumin commented at 2016-08-01 15:16:

Hi @jsmeix

ok i can confirm that it is way easier to get things modified using the newest git and the UPDATE_URL attribute - Thanks again for that.

However i seem to be to silly to get things working, it has not magically fixed my Issues as you stated :(
Here is my latest log from the recover test:
https://gist.github.com/mzlumin/80258ba5ce094aae0153882eeb8a1c47

Here is the disklayout File:
https://gist.github.com/mzlumin/0ed1fa5c2720de5dc1901c833af19f55

I tried my best, but as said, i'm not to deep with partitions and stuff unfortunately :&

What i am wondering about is the last lines in the Log File:

+++ parted -s /dev/sda mkpart '"logical"' 256909312B 2147483647999B parted: invalid token: logical Error: Expecting a partition type.

I am pretty sure that a somehow equal config works on another Box, also with those logical partition ?

mzlumin commented at 2016-08-01 15:21:

Ah forgot, this one is the script which is getting built on this box which fails:

https://gist.github.com/mzlumin/0460c2bcaf60979beab663866d7fc7f8

mzlumin commented at 2016-08-01 15:24:

Also the Links directly, seems the referral points to an 404:

Log FIle: https://gist.github.com/mzlumin/80258ba5ce094aae0153882eeb8a1c47

Disklayout: https://gist.github.com/mzlumin/0ed1fa5c2720de5dc1901c833af19f55

Script: https://gist.github.com/mzlumin/0460c2bcaf60979beab663866d7fc7f8

jsmeix commented at 2016-08-02 12:55:

Welcome to parted!

In general regarding parted see
https://www.gnu.org/software/parted/manual/parted.html

I can reproduce your parted behaviour
on my SLES12-SP1 test system:

# parted -s /dev/sdb mklabel msdos
# parted -s /dev/sdb print        
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sdb: 4295MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 
Number  Start  End  Size  Type  File system  Flags
# parted -s /dev/sdb mkpart primary 4MiB 8MiB
# parted -s /dev/sdb print
...
Number  Start   End     Size    Type     File system  Flags
 1      4194kB  8389kB  4194kB  primary               type=83
# parted -s /dev/sdb set 1 boot on
# parted -s /dev/sdb print
...
Number  Start   End     Size    Type     File system  Flags
 1      4194kB  8389kB  4194kB  primary               boot, type=83

# parted -s /dev/sdb mkpart primary 8MiB 16MiB
# parted -s /dev/sdb print
...
Number  Start   End     Size    Type     File system  Flags
 1      4194kB  8389kB  4194kB  primary               boot, type=83
 2      8389kB  16.8MB  8389kB  primary               type=83
# parted -s /dev/sdb mkpart logical 16MiB 32MiB
parted: invalid token: logical
Error: Expecting a partition type.

The reason is that you cannot have a logical partition
directly on the disk.

You first need an extended partition - a container partition
that usually goes to the end of the whole disk,
then you can add logical partitions inside the extended
container partition:

# parted -s /dev/sdb mkpart extended 16MiB 128MiB
# parted -s /dev/sdb print
...
Number  Start   End     Size    Type      File system  Flags
 1      4194kB  8389kB  4194kB  primary                boot, type=83
 2      8389kB  16.8MB  8389kB  primary                type=83
 3      16.8MB  134MB   117MB   extended               lba, type=0f
# parted -s /dev/sdb mkpart logical 16MiB 32MiB
Error: You requested a partition from 16.8MB to 33.6MB (sectors 32768..65535).
The closest location we can manage is 16.8MB to 33.6MB (sectors 32769..65535).
# parted -s /dev/sdb mkpart logical 17MiB 34MiB
# parted -s /dev/sdb print
...
Number  Start   End     Size    Type      File system  Flags
 1      4194kB  8389kB  4194kB  primary                boot, type=83
 2      8389kB  16.8MB  8389kB  primary                type=83
 3      16.8MB  134MB   117MB   extended               lba, type=0f
 5      17.8MB  35.7MB  17.8MB  logical                type=83
# parted -s /dev/sdb mkpart logical 34MiB 64MiB
Error: You requested a partition from 35.7MB to 67.1MB (sectors 69632..131071).
The closest location we can manage is 35.7MB to 67.1MB (sectors 69633..131071).
# parted -s /dev/sdb mkpart logical 36MiB 68MiB
# parted -s /dev/sdb print
...
Number  Start   End     Size    Type      File system  Flags
 1      4194kB  8389kB  4194kB  primary                boot, type=83
 2      8389kB  16.8MB  8389kB  primary                type=83
 3      16.8MB  134MB   117MB   extended               lba, type=0f
 5      17.8MB  35.7MB  17.8MB  logical                type=83
 6      37.7MB  71.3MB  33.6MB  logical                type=83

I suggest that you boot the rear recovery system on the
new hardware, log in as root but do not run "rear recover".
Instead do the parted calls manually until you know what
exact parted calls will work so that you can make the
disklayout.conf according to the exact parted calls
that had actually worked.

mzlumin commented at 2016-08-04 11:24:

Thanks @jsmeix i think with that last information i have been able to get this working!

Just one last thing i have seen while testing these, i have tried it using the latest git, it also downloaded the file from the WebServer:

2016-08-04 15:02:17 Relax-and-Recover 1.18 / Git
2016-08-04 15:02:17 Command line options: /bin/rear recover
2016-08-04 15:02:17 Using log file: /var/log/rear/rear-Stark.log
2016-08-04 15:02:17 Including /etc/rear/os.conf
2016-08-04 15:02:17 Including conf/Linux-i386.conf
2016-08-04 15:02:17 Including conf/GNU/Linux.conf
2016-08-04 15:02:17 Including conf/Ubuntu.conf
2016-08-04 15:02:17 Including /etc/rear/local.conf
2016-08-04 15:02:17 Including /etc/rear/rescue.conf
2016-08-04 15:02:17 Running 'init' stage
2016-08-04 15:02:17 Including init/default/01_set_drlm_env.sh
2016-08-04 15:02:17 Including init/default/03_update_recovery_system.sh
2016-08-04 15:02:17 Updating recovery system with the content from 'http://192.168.0.13/Stark.rear_config.tar.gz':

  • Trying 192.168.0.13...
  • Connected to 192.168.0.13 (192.168.0.13) port 80 (#0)
    GET /Stark.rear_config.tar.gz HTTP/1.1
    Host: 192.168.0.13
    User-Agent: curl/7.43.0
    Accept: /

< HTTP/1.1 200 OK
< Date: Thu, 04 Aug 2016 11:02:17 GMT
< Server: Apache/2.4.18 (Win32) OpenSSL/1.0.2e PHP/7.0.8
< Last-Modified: Thu, 04 Aug 2016 11:02:07 GMT
< ETag: "d92-5393ce17b2a22"
< Accept-Ranges: bytes
< Content-Length: 3474
< Content-Type: application/x-gzip
<
{ [3474 bytes data]

  • Connection #0 to host 192.168.0.13 left intact

The only thing was that it has not used those modified files, if i e.g. looked at the disklayout file, although modified on the webserver and packed again to the same name...

I just give it another shot with a fresh backup and recovery..

Thanks anyways for your great help!

jsmeix commented at 2016-08-04 12:57:

@mzlumin
when using the latest rear master code
from within a sub-directory "rear" as in

# git clone https://github.com/rear/rear.git
# cd rear
# vi etc/rear/local.conf
# usr/sbin/rear -d -D mkbackup

the disklayout.conf file is not in
/var/lib/rear/layout/disklayout.conf
but in that "rear" sub-directory
/.../rear/var/lib/rear/layout/disklayout.conf
below whatever path of parent directories.

Accordingly in the rear recovery system you must
install an updated disklayout.conf into the same
/.../rear/var/lib/rear/layout/disklayout.conf

For example on my SLES12-SP1 test system
I run the newest GitHub master code in /root/rear
with /root/rear/etc/rear/local.conf

RECOVERY_UPDATE_URL="http://10.160.4.244/$HOSTNAME.rear-update.tar.gz"
REQUIRED_PROGS=( "${REQUIRED_PROGS[@]}" curl )

where 10.160.4.244 is my HTTP server
and "g215" is the hostname of the system
where I run "usr/sbin/rear -d -D mkbackup".

When I change /root/rear/var/lib/rear/layout/disklayout.conf
I make a tar.gz as

tar -czvf g215.rear-update.tar.gz /root/rear/var/lib/rear/layout/disklayout.conf

and copy that into /srv/www/htdocs/ on my HTTP server.

In the rear recovery system it looks then as follows:

RESCUE g215:~ # rear -d -D recover
...
Updating recovery system with the content from 'http://10.160.4.244/g215.rear-update.tar.gz':
/ ~
root/rear/var/lib/rear/layout/disklayout.conf
~
Updated recovery system.
...

This way I get the new disklayout.conf available
in the rear recovery system at
/root/rear/var/lib/rear/layout/disklayout.conf

I guess you get your modified disklayout.conf
installed at the wrong place in your rear recovery system.

Note that the RECOVERY_UPDATE functionality is meant
to be usable in a generic way which is the reason why
it does not have sophisticated built-in "intelligence"
whereto stuff might be "best" installed. It simply installs
any content of the tar.gz in the root '/' of the recovery system.

mzlumin commented at 2016-08-08 09:45:

Thanks @jsmeix i consider this one can be set to solved or fixed :)

Thanks for all your help and for getting this amazing project running!

jsmeix commented at 2016-08-08 10:03:

@mzlumin
I understand your latest comment that
with the right content in the update tar.gz
it also works for you.

[Export of Github issue for rear/rear.]