...¶

Labels: enhancement, fixed / solved / done, severe improvement

jsmeix opened issue at 2016-03-15 15:09:¶

Hereby I propose to let a "cleanupdisk" script run early
(i.e. before anything is done with the harddisk,
in particular before a "parted" command is run).

The purpose of the "cleanupdisk" script is to wipe any
possibly remainders of various kind of metadata information
from the harddisk that could belong to various higher layers
of storage objects.

Currently (cf. https://github.com/rear/rear/issues/540) "wipefs" is run in
130_include_filesystem_code.sh for each partition device node
before a filesystem is created on that partition device node.

But after I wrote https://github.com/rear/rear/issues/791#issuecomment-196212960 I noticed that running "wipefs" before filesystems are created is probably too late.

I had this "too late" problem already recognized in https://github.com/rear/rear/issues/540#issuecomment-71814659 (there "it failed for RHEL6 at the partitioning level because of old data of the MD level so that before partitioning the MD tool would have to be run to clean up old MD data") but unfortunately that had slipped my lossy mind :-(

See https://github.com/rear/rear/issues/791#issuecomment-196212960 for the reason why "wipefs" musts probably be run before anything is done with the harddisk, in particular before a "parted" command is run (excerpt):

For example if you re-use a harddisk that
had before LVM on it, it may happen that
after creating partitions from scatch on
that hsrddisk, udev may also trigger
to run LVM tools ...
When those LVM tools detect remaining
old/outdated LVM metadata information
on the harddisk, there could be arbitrarily
unexpected results (e.g. all of a sudden
LVM issues may get in your way regardless
that you only had called parted to create
partitions).

Here what I get in the ReaR recovery system
directly after login as root
on pristine new hardware
(where "pristine new hardware" is a new
from scratch created QEMU/KVM virtual
machine with full hardware virtualization):

 # ls -l /dev/sd*
brw-rw---- 1 root disk 8, 0 Mar 15 14:36 /dev/sda

# parted /dev/sda print
Error: /dev/sda: unrecognised disk label
Model: ATA QEMU HARDDISK (scsi)                                           
Disk /dev/sda: 21.5GB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:

And now ( tada - surprise! - not really ;-)
what I get in the ReaR recovery system
directly after login as root
on same kind of a machine where I already
had done a "rear recover" some time before
(i.e. where a subsequent "rear recover" would run
on a system where the harddisk was already in use):

# ls -l /dev/sd*
brw-rw---- 1 root disk 8, 0 Mar 15 14:31 /dev/sda
brw-rw---- 1 root disk 8, 1 Mar 15 14:31 /dev/sda1
brw-rw---- 1 root disk 8, 2 Mar 15 14:31 /dev/sda2

# parted /dev/sda print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sda: 21.5GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system     Flags
 1      1049kB  1571MB  1570MB  primary  linux-swap(v1)  type=83
 2      1571MB  21.5GB  19.9GB  primary  ext4            boot, type=83

Accordingly I think ReaR should run something like

wipefs -a -f /dev/sda2
wipefs -a -f /dev/sda1
wipefs -a -f /dev/sda

to fully clean up the used harddisk before doing anything with it.

Regarding the '-f' option see https://github.com/rear/rear/issues/540#issuecomment-163241543

jsmeix commented at 2016-03-15 15:11:¶

@gdha @schlomo @thefrenchone @tbsky

I ask you all for feedback what you think about it.

jsmeix commented at 2016-03-16 12:04:¶

What wipefs results in the rear recovery system
directly after login as root
on the above mentioned (second) machine
where the harddisk was already in use:

Welcome to Relax and Recover. Run "rear recover" to restore your system !

RESCUE f197:~ # ls -l /dev/sd*
brw-rw---- 1 root disk 8, 0 Mar 16 11:02 /dev/sda
brw-rw---- 1 root disk 8, 1 Mar 16 11:02 /dev/sda1
brw-rw---- 1 root disk 8, 2 Mar 16 11:02 /dev/sda2

RESCUE f197:~ # parted /dev/sda print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sda: 21.5GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 
Number  Start   End     Size    Type     File system     Flags
 1      1049kB  1571MB  1570MB  primary  linux-swap(v1)  type=83
 2      1571MB  21.5GB  19.9GB  primary  ext4            boot, type=83

RESCUE f197:~ # wipefs -a -f /dev/sda2
/dev/sda2: 2 bytes were erased at offset 0x00000438 (ext4): 53 ef

RESCUE f197:~ # ls -l /dev/sd*
brw-rw---- 1 root disk 8, 0 Mar 16 11:42 /dev/sda
brw-rw---- 1 root disk 8, 1 Mar 16 11:02 /dev/sda1
brw-rw---- 1 root disk 8, 2 Mar 16 11:42 /dev/sda2

RESCUE f197:~ # parted /dev/sda print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sda: 21.5GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 
Number  Start   End     Size    Type     File system     Flags
 1      1049kB  1571MB  1570MB  primary  linux-swap(v1)  type=83
 2      1571MB  21.5GB  19.9GB  primary                  boot, type=83

RESCUE f197:~ # wipefs -a -f /dev/sda1
/dev/sda1: 10 bytes were erased at offset 0x00000ff6 (swap): 53 57 41 50 53 50 41 43 45 32

RESCUE f197:~ # parted /dev/sda print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sda: 21.5GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 
Number  Start   End     Size    Type     File system  Flags
 1      1049kB  1571MB  1570MB  primary               type=83
 2      1571MB  21.5GB  19.9GB  primary               boot, type=83

RESCUE f197:~ # ls -l /dev/sd*
brw-rw---- 1 root disk 8, 0 Mar 16 11:42 /dev/sda
brw-rw---- 1 root disk 8, 1 Mar 16 11:42 /dev/sda1
brw-rw---- 1 root disk 8, 2 Mar 16 11:42 /dev/sda2

RESCUE f197:~ # wipefs -a -f /dev/sda
/dev/sda: 2 bytes were erased at offset 0x000001fe (dos): 55 aa

RESCUE f197:~ # parted /dev/sda print
Error: /dev/sda: unrecognised disk label
Model: ATA QEMU HARDDISK (scsi)                                           
Disk /dev/sda: 21.5GB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags: 

RESCUE f197:~ # ls -l /dev/sd*
brw-rw---- 1 root disk 8, 0 Mar 16 11:42 /dev/sda
brw-rw---- 1 root disk 8, 1 Mar 16 11:42 /dev/sda1
brw-rw---- 1 root disk 8, 2 Mar 16 11:42 /dev/sda2

RESCUE f197:~ # partprobe -s /dev/sda

RESCUE f197:~ # ls -l /dev/sd*
brw-rw---- 1 root disk 8, 0 Mar 16 11:47 /dev/sda
brw-rw---- 1 root disk 8, 1 Mar 16 11:42 /dev/sda1
brw-rw---- 1 root disk 8, 2 Mar 16 11:42 /dev/sda2

RESCUE f197:~ # parted /dev/sda mklabel msdos
Information: You may need to update /etc/fstab.

RESCUE f197:~ # ls -l /dev/sd*
brw-rw---- 1 root disk 8, 0 Mar 16 11:51 /dev/sda

RESCUE f197:~ # partprobe -s /dev/sda
/dev/sda: msdos partitions

RESCUE f197:~ # parted /dev/sda print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sda: 21.5GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 
Number  Start  End  Size  Type  File system  Flags

RESCUE f197:~ # parted /dev/sda mklabel gpt
Warning: The existing disk label on /dev/sda will be destroyed
and all data on this disk will be lost.
Do you want to continue?
Yes/No? y                                                                 
Information: You may need to update /etc/fstab.

RESCUE f197:~ # parted /dev/sda print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sda: 21.5GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 
Number  Start  End  Size  File system  Name  Flags

RESCUE f197:~ # ls -l /dev/sd*
brw-rw---- 1 root disk 8, 0 Mar 16 11:55 /dev/sda

Summary:

After wipefs the harddisk /dev/sda looks empty for parted
but nevertheless regardless of a partprobe call
the old/outdated partition device nodes (/dev/sda1and /dev/sda2)
do not go away.

To make the old/outdated partition device nodes go away
an explicit "parted /dev/sda mklabel" command is needed.

It seems to be fail-safe to set a hardcoded "msdos" dummy label
in such a "cleanupdisk" script via "parted /dev/... mklabel msdos"
to make old/outdated partition device nodes go away because
later the actual disk label (e.g. "gpt") will be set
in 10_include_partition_code.sh via

parted -s $device mklabel $label >&2

thefrenchone commented at 2016-03-16 21:01:¶

wipefs would be great to have run earlier. It's code generated from 10_include_partition_code.sh that would occasionally fail or require extra time for me. Just removing LVM information was usually enough.

At the moment I can't test this but hopefully soon I'll be able to. I'll try adding the wipefs command to clear the disk earlier.

jsmeix commented at 2016-03-18 14:01:¶

@mattihautameki
I remembered your issue https://github.com/rear/rear/issues/649 that also seems to be related to disk cleanup (cf. https://github.com/rear/rear/issues/649#issuecomment-148710502) so that this issue here could be also of interest for you.

jsmeix commented at 2016-05-23 09:23:¶

For the fun of it:
It seems SUSE implements a new parted option --wipesignatures,
see
https://bugzilla.opensuse.org/show_bug.cgi?id=980834

For the current sources see the
"parted-implement-wipesignatures-option.patch" at
https://build.opensuse.org/package/show/Base:System/parted

But it seems that fails curently in the same way
as the initial comment in https://github.com/rear/rear/issues/793#issue-139355299

It seems currently "everybody" has issues with "udev vs. parted",
cf. https://github.com/rear/rear/issues/791#issuecomment-220923467

jsmeix commented at 2016-05-23 09:25:¶

Because the more I learn about it
the more I get confused how to make it working
I postpone this issue to any later unspecified
"future rear version".

jsmeix commented at 2016-05-30 09:56:¶

An addedum regarding a higher stack of storage objects
in particular regarding MD devices (aka Linux Software RAID):

One can try to remove mdadm superblocks from hardrives by

mdadm --zero-superblock /dev/sd{a,b,c,d}

to avoid hdds to be detected as mdadm devices.

Ideally only calling the generic "wipefs" tool
would do all what is needed but perhaps
we may have to also call the individual tools
of each particular higher level storage object
to really clean up everything properly.

jsmeix commented at 2016-10-04 12:43:¶

FYI regarding
"How do I delete a RAID volume that was created with mdadm?"

One may have a look at
https://forums.opensuse.org/showthread.php/489778-How-do-I-delete-a-RAID-volume-that-was-created-with-mdadm

Key steps:

Stop the RAID
```
mdadm --stop /dev/md0¶
```
Remove the RAID signatures from every (!) part of the RAID
```
mdadm --zero-superblock /dev/sd??¶
```
It should also work to use "wipefs".

gdha commented at 2017-01-21 11:34:¶

@jsmeix perhaps shred utility could also be useful (http://www.computerhope.com/unix/shred.htm). I noticed that RH engineers are using this command to wipe a disk (before doing rear recover test) - e.g. shred /dev/sda

schlomo commented at 2017-05-12 05:23:¶

Let's just make sure that we don't touch the hard disks before

The user actually starts rear recover.
The verify stage was run successfully.

jsmeix commented at 2017-05-12 08:32:¶

I think to have such a "cleanupdisk" script behave
in compliance with how "rear recover" currently works
the script must not work directly on the harddisk(s)
but instead output commands that clean up disks
into the LAYOUT_CODE script (i.e. diskrestore.sh).

Such an early "cleanupdisk" script would have to run
before scripts like
layout/prepare/GNU/Linux/100_include_partition_code.sh
...
layout/prepare/GNU/Linux/130_include_filesystem_code.sh
to get the disk cleanup commands are at the beinning
of the diskrestore.sh script, cf. my initial
https://github.com/rear/rear/issues/799#issue-141001306

... "wipefs" musts probably be run before anything
is done with the harddisk, in particular before
a "parted" command is run ...

schlomo commented at 2017-05-12 09:01:¶

@jsmeix very good point. Yes, of course the cleanup stuff should be added to the beginning of the diskrestore.sh and not run directly.

jsmeix commented at 2018-06-27 12:01:¶

Right now I had an issue because of remaining LVM metadata on a used disk:

0.)
On the original system there is only /dev/sda with LVM on it.
1.)
I recovered that on replacement system with only /dev/sda without problems.
2.)
I added a /dev/sdb to the replacement system (same size as /dev/sda).
3.)
I recovered the original system again on the replacement system
but now I selected to map /dev/sda to /dev/sdb (in MIGRATION_MODE).
4.)
This lets LVM setup in diskrestore.sh fail because an LVM tool got confused
that a UUID which it should create on /dev/sdb already exists on /dev/sda.
5.)
The manual solution was to go during "rear recover" into the ReaR shell
and do wipefs -af /dev/sda1 (that was used for LVM before)
and wipefs -af /dev/sda (to be on the safe side).
6.)
Afterwards the (unmodified) diskrestore.sh works.

gdha commented at 2019-09-13 07:37:¶

@jsmeix Any plans to go forward with this enhancement as it is a long outstanding item? I guess there are no volunteers to write this :-(

jsmeix commented at 2019-09-13 13:56:¶

@gdha
if time permits my plan would be to try out something
and if that works reasonably well for me I would add
an initial version of such a script that could be further
improved step by step as needed
BUT
"time permits" never happens in practice because of various
kind of annoying issues elsewhere, in particular an endless
sequence of security issues in printing related bloatware
(my other working area - I just hate that stuff)
SO
in practice never time for long term improvements in ReaR
only time for some quick fix here some quick fix there
ad nauseam...
cf. "ad nauseam" in
https://en.opensuse.org/SDB:CUPS_and_SANE_Firewall_settings

jsmeix commented at 2020-03-13 09:19:¶

Since my SUSE Hack Week 19 continuing experiments
https://github.com/rear/rear/issues/2254#issuecomment-595678350
I have good and bad news:

The good news is that I could experience myself
a few different kind of "horrible" consequences
of remainder meta data on an already used disk
in particular remainders of RAID and LVM metadata.

The bad news is that wipefs alone is insufficient,
i.e. something like

wipefs -a -f /dev/sdb3
wipefs -a -f /dev/sdb2
wipefs -a -f /dev/sdb1
wipefs -a -f /dev/sdb

wipefs -a -f /dev/sda3
wipefs -a -f /dev/sda2
wipefs -a -f /dev/sda1
wipefs -a -f /dev/sda

is insufficient in general, cf.
https://github.com/rear/rear/issues/2019#issuecomment-476598723
and the subsequent comments therein.

According to my personal (very limited) experience
wipefs is only sufficient if LVM PVs are plain partitions
but when the PVs are higher level storage objects
like RAID1 block devices then wipefs of disk partitions
is insufficient because it does not remove LVM metadata.

The current generic way how to deal with it is to
"Prepare replacement hardware for disaster recovery", cf.
https://en.opensuse.org/SDB:Disaster_Recovery
therein in particular the part about
"you must completely zero out your replacement storage"
BUT
that workaround is not possible within reasonable time
when one needs to recover from soft errors on the original machine
i.e. when the exact same original disks are re-used "as is".

To recover from "minor" soft errors (i.e. when only files had been
destroyed or messed up so that a backup restore is sufficient)
the new mountonly workflow provides a solution.

To recover from arbitrary soft errors on the original machine
it is mandatory to have an automated way to clean up
all remainder meta data on the disks.

Details:

I experimented with a test virtual machine that has two disks
with same partitioning where RAID1 is used for the partitions
and the RAID1 block devices are used as PVs for LVM.

I did "rear mkbackup" on that machine and then "rear recover"
on a second identical new virtual machine which "just worked" for me.

Then I used only the second machine for my actual experiments
(to keep the first one as an untouched pristine original system).

On the second machine I implemented completely new experimental
scripts from scratch to recreate the disk layout.
My primary intent was to experiment how to recreate the disk layout
without relying on kernel device names, cf.
https://github.com/rear/rear/issues/2254

But my primary finding was how badly things fail
when there is remainder meta data on the disk
in particular remainders of RAID and LVM metadata.

When one recreates the disk layout (exactly as it was before)
on the exact same disks that had been used before
things fail badly, in particular LVM setup fails badly
because of LVM metadata remainders on the disk.

As soon as partitioning and RAID1 block devices had been
recreated by the recreation script, the LVM metadata remainders
become "re-visible" again on the disk so that the LVM magic
is finding again its old LVM thingies (PVs, VGs, LVs)
and is building up the old LVM while at the same time
the recreation script (that runs in parallel) tries to set up
that LVM stuff anew which fails in unpredictable ways
with arbitrary weird error messages because what exactly
happens depends on random timing constraints.

github-actions commented at 2020-07-02 01:33:¶

Stale issue message

gdha commented at 2020-12-09 15:01:¶

As the PR https://github.com/rear/rear/pull/2514 is in progress we better re-open this issue

github-actions commented at 2021-02-08 02:01:¶

Stale issue message

jsmeix commented at 2021-03-16 15:13:¶

I like to merge https://github.com/rear/rear/pull/2514
tomorrow afternoon unless there are objections
to have at least a first step that can be used in ReaR.
Then we can improve things as needed step by step.

jsmeix commented at 2021-03-17 13:24:¶

With https://github.com/rear/rear/pull/2514 merged
disks can be wiped before recreating partitions/volumes/filesystems/...

There is now the new DISKS_TO_BE_WIPED in default.conf
and for details and some background information see
usr/share/rear/layout/recreate/default/README.wipe_disks
https://github.com/rear/rear/blob/master/usr/share/rear/layout/recreate/default/README.wipe_disks

This is currently new and experimental functionality so that
currently by default via DISKS_TO_BE_WIPED='false' no disk is wiped
to avoid possible regressions until this new feature was more tested
by interested users via explicit DISKS_TO_BE_WIPED='' in local.conf

jsmeix commented at 2021-03-17 14:10:¶

Phew!
I created this issue on Mar 15 2016
and closed it today on Mar 17 2021
which is 5 years and two days later
and it is still not completely solved, cf.
https://github.com/rear/rear/pull/2514#issuecomment-801097878

[Export of Github issue for rear/rear.]

#799 Issue closed: Clean up disks before recreating partitions/volumes/filesystems/...¶