#2876 PR merged: Implement basic 'barrel' support¶
Labels: fixed / solved / done, severe improvement
jsmeix opened issue at 2022-10-04 10:04:¶
-
Type: New Feature
-
Impact: High
High impact only on systems where 'barrel' exists.
There should be no change (and no impact) on other systems
regarding the actual disk layout recreation via diskrestore.sh
(i.e. nothing of the actual disk layout recreation
via diskrestore.sh is changed).
There are visible changes on all systems
regarding the choices in the user dialogs
during "rear recover" that are improved
so that users have now more options what to do
and what should happen during "rear recover".
There was no change what the default choice is
so the default behaviour of "rear recover" is same as before
unless BARREL_DEVICEGRAPH is explicitly set to true by the user.
In particular users can now additionally choose:
"Skip wiping disks"
Before there was no way to do that without changing
/etc/rear/local.conf and restarting "rear recover".
"Again wipe the disks in DISKS_TO_BE_WIPED"
Before there was no way to do that during "rear recover".
The only way was to abort "rear recover" and restart it.
"Show what is currently on the disks ('lsblk' block devices list)"
Before there was no easy way to see what is currently on the disks.
The only way was to use the ReaR shell and manually type
e.g. an appropriate 'lsblk' command.
"Confirm what is currently on the disks and continue 'rear recover'"
Before there was no way to do that.
The only way was to fix diskrestore.sh and re-run it until all is OK
or do something via ReaR shell and re-run barrel until all is OK.
But the user might also fix things (in particular smaller things)
by only using the ReaR shell and then confirm what he created
on his disks and continue with restoring the backup.
Those additional options during "rear recover" became useful
and needed by me while I did my various tests with 'barrel'
so those additional options during "rear recover" are a direct
consequence of the added support for a second method how to
recreate the disk layout - in particular when I first used
'barrel' and then switched back to 'diskrastore.sh' where
I needed to "Again wipe the disks in DISKS_TO_BE_WIPED"
and where I liked to see what is currently on the disks.
The BARREL_DEVICEGRAPH and BARREL_MAPPING_FILE config variables
are curently not documented in default.conf because the names
and semantics of 'barrel' config variables may change during
the currently ongoing development of 'barrel' support.
- Reference to related issue (URL):
https://github.com/rear/rear/issues/2863
- How was this pull request tested?
Several tests on my SLES15 SP4 test VM
with the default SUSE storage structure:
# lsblk -ipo NAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINTS
NAME TRAN TYPE FSTYPE SIZE MOUNTPOINTS
/dev/sda ata disk 20G
|-/dev/sda1 part 8M
|-/dev/sda2 part btrfs 12.6G /var
| /root
| /usr/local
| /tmp
| /srv
| /boot/grub2/x86_64-efi
| /opt
| /boot/grub2/i386-pc
| /.snapshots
| /
|-/dev/sda3 part xfs 5.4G /home
`-/dev/sda4 part swap 2G [SWAP]
- Brief description of the changes in this pull request:
In layout/recreate/default/200_run_layout_code.sh
added the new storage layout recreation method
Recreating storage layout with 'barrel' devicegraph
as optional alternative to recreating with diskrestore.sh
when the config variable BARREL_DEVICEGRAPH is 'true'
plus an option to switch from 'barrel' back to diskrestore.sh.
Additionally there are the new options
Show what is currently on the disks ('lsblk' block devices list)
and
Again wipe those disks ... in DISKS_TO_BE_WIPED
to check after a failed attempt what was recreated and
to redo the recreation from scratch on clean disks.
jsmeix commented at 2022-10-04 10:40:¶
Currently this implements only very basic 'barrel' support.
In particular support for 'barrel' device mapping
during "rear recover" via user dialogs in MIGRATION_MODE
is currently missing.
Current 'barrel' device mapping must be done manually via
BARREL_MAPPING_FILE="/etc/barrel-mapping.json"
COPY_AS_IS+=( $BARREL_MAPPING_FILE )
in etc/rear/local.conf with an appropriate manually created
/etc/barrel-mapping.json file with content for example like
{
"mapping" : {
"/dev/sda" : "/dev/disk/by-id/ata-QEMU_HARDDISK_QM00001"
}
}
which works for me for my KVM/QEMU test VMs because
all my VMs have /dev/disk/by-id/ata-QEMU_HARDDISK_QM00001
as a symbolic link that points to ../../sda
jsmeix commented at 2022-10-05 12:19:¶
I also tested on my above SLES15 SP4 test VM
that things behave backward compatible
when there is no 'barrel' program.
jsmeix commented at 2022-10-05 12:23:¶
@gdha @pcahyna @rear/contributors
I would appreciate it very much
if you could have a look at my code changes
whether or not they look backward compatible to you.
I will do some more tests tomorrow
and when there are no objections
I would like to merge it on Friday afternoon
to get some initial very basic 'barrel' support
as a first step - further enhancements will happen
afterwards via separated pull requests as needed.
pcahyna commented at 2022-10-05 16:32:¶
@jsmeix I will not be able to review the change this week, as I am leaving now. But according to the description, the "barrel" tool sounds very promising for use in ReaR.
jsmeix commented at 2022-10-06 08:11:¶
@pcahyna
thank you for your prompt reply.
Merging it can wait until end of next week.
gdha commented at 2022-10-06 09:20:¶
@jsmeix I'm not familiar with "barrel", so, not much input from my side. However, if you feel okay with it please proceed.
jsmeix commented at 2022-10-06 10:10:¶
@gdha
thank you for your feedback.
I do not expect you or other ReaR contributors (except me)
to know about 'barrel' or to even learn what 'barrel' is.
I only liked to get feedback from a general point of view.
pcahyna commented at 2022-10-10 15:31:¶
There should be no change (and no impact) on other systems.
I had only a quick look at the change and I see some code changes in
places that are not protected by if is_true "$BARREL_DEVICEGRAPH".
Especially in
usr/share/rear/layout/recreate/default/120_confirm_wipedisk_disks.sh
and usr/share/rear/layout/recreate/default/200_run_layout_code.sh .
It may be the case that the changes do not change anything when barrel
is not in use, but it is far from obvious and
Additionally there are the new options
Show what is currently on the disks ('lsblk' block devices list)
and
Again wipe those disks ... in DISKS_TO_BE_WIPED
to check after a failed attempt what was recreated and
to redo the recreation from scratch on clean disks.
seems to contradict it.
Would it be possible to separate the barrel-related changes from the disk-wiping changes?
jsmeix commented at 2022-10-11 06:50:¶
@pcahyna
thank you for having a look!
I enhanced my initial description of this pull request
https://github.com/rear/rear/pull/2876#issue-1395991143
that now explains why those additional user dialog choices
got added together with the actual 'barrel' support.
In theory I could separate those additional user dialog choices
from the actual 'barrel' support but in practice I needed
those additional user dialog choices while I was using
'barrel' and 'diskrestore.sh' alternately during one same
'rear recover' run so in practice all that belongs together.
jsmeix commented at 2022-10-13 07:24:¶
If there are no strong objections against the
new additional choices during "rear recover"
I would like to merge this pull request as is
today afternoon.
When issues appear with the new additional choices
during "rear recover" I will of course fix them.
I already know one but that really belongs
to a separated pull request:
Currently it does not work to run diskrestore.sh
then "Again wipe the disks in DISKS_TO_BE_WIPED"
and then re-run diskrestore.sh
because the second run of diskrestore.sh does nothing
because all is marked as already done by the first run.
The reason why things get marked as done can be seen
how it is meant to behave in the section "The Ad-Hoc Way" in
https://github.com/rear/rear/blob/master/doc/user-guide/06-layout-configuration.adoc
so one can modify diskrestore.sh step by step and re-run it
and it only does the not yet done things.
What currently "just works" (at least for me) is
to run 'barrel'
then "Again wipe the disks in DISKS_TO_BE_WIPED"
and re-run 'barrel'
and so on as often as one likes.
It also works for me to just re-run 'barrel'
even without "Again wipe the disks in DISKS_TO_BE_WIPED"
in between because 'barrel' does a lot to be able to run
on already used disks where the same layout already exists
because this is how 'barrel' gets usually tested
(again and again via AutoYaST with already used disks).
Of course there is no guarantee that one can re-run 'barrel'
again and again as often as one likes and all goes well.
During my tests I managed to mess up things so much
that 'barrel' failed to recreate things even with
"Again wipe the disks in DISKS_TO_BE_WIPED" before.
But in this case it "just worked" (at least for me)
to abort the current "rear recover" run and
re-boot the ReaR recovery system and let
"rear recover" run again from scratch.
jsmeix commented at 2022-10-13 13:23:¶
@pcahyna
again thank you for checking my code!
It is much appreciated.
Feel free to add further comments about code places
that look "fishy" to you regardless that
this pull request is merged.
pcahyna commented at 2022-10-14 11:43:¶
It also works for me to just re-run 'barrel'
even without "Again wipe the disks in DISKS_TO_BE_WIPED"
in between because 'barrel' does a lot to be able to run
on already used disks where the same layout already exists
@jsmeix does it mean that in practice barrel supports to do "only the not yet done things" natively without marking things as done? That's how I understand it and would be good for consistency between barrel and the usual layout code.
jsmeix commented at 2022-10-14 12:42:¶
@pcahyna
no, you misunderstood.
'barrel' does not support to do "only the not yet done things".
It is the opposite:
Running 'barrel' always recreates all from scratch.
Running 'barrel' results that what is specified
as so called 'devicegraph' via an XML file
will get (basically mercilessly) setup on the disks
(regardless what there already is on the disks),
cf. at the end of
https://github.com/rear/rear/issues/2863#issue-1375652428
Its name is 'barrel' because it even supports
"shooting yourself in the foot" when used without care ;-)
So with 'barrel' one cannot recreate step by step
as with disklayout.conf and diskrestore.sh.
When recreating with 'barrel' plus devicegraph fails
it is a dead end in practice because manually editing
the devicegraph XML is likely not what one wants to do.
BUT
One can run barrel also interactively to manually
setup storage things, cf.
https://github.com/openSUSE/barrel/blob/master/doc/tutorial.md
So when recreating with 'barrel' plus devicegraph fails
the user may manually setup his storage things with 'barrel'
interactively via the ReaR shell
or the user can switch back to ReaR's native recreating
method via disklayout.conf and diskrestore.sh
or perhaps even a combination of both and finally he can
"Confirm what is currently on the disks and continue 'rear recover'"
to restore his backup into what he manually created.
I assume this way will not be done very often in practice.
But when all seems lost this could be the only way out
(except reinstalling from scratch without ReaR).
With "'barrel' does a lot to be able to run on already
used disks where the same layout already exists"
I meant that 'barrel' does a lot to deal with unwanted
remainders of what there already is on a disk on its own.
So 'barrel' does not need a separated "wipe disks" step
before - at least in usual cases a.k.a. in the already known
cases where the 'libstorage-ng' developer(s) implemented
"the right things" how to deal with remainders on disks.
'barrel' is a command line frontend for 'libstorage-ng'
https://github.com/aschnell/libstorage-ng
so the actual work is implemented in 'libstorage-ng'
(as far as I understood what I was told about it).
jsmeix commented at 2022-10-14 12:54:¶
@pcahyna
thank you for your questions!
They help me to think in more detail about how 'barrel' works
and how it can and should be well integrated into ReaR.
jsmeix commented at 2022-10-14 12:54:¶
@pcahyna @gdha @rear/contributors
I wish you all a relaxed and recovering weekend!
[Export of Github issue for rear/rear.]