#2955 Issue closed: Devuan 4 + ReaR Not Able to Recover¶
Labels: enhancement, support / question, needs sponsorship,
no-issue-activity
3t0n1c opened issue at 2023-03-07 20:15:¶
-
ReaR version ("/usr/sbin/rear -V"): 2.6 or 2.7
-
If your ReaR version is not the current version, explain why you can't upgrade:
I tried 2.6 that is available through my distro, and when it did not work,
I manually installed 2.7 - which also fails similarly. -
OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):
PRETTY_NAME="Devuan GNU/Linux 4 (chimaera)"
NAME="Devuan GNU/Linux"
VERSION_ID="4"
VERSION="4 (chimaera)"
VERSION_CODENAME="chimaera"
ID=devuan
ID_LIKE=debian
HOME_URL="https://www.devuan.org/"
SUPPORT_URL="https://devuan.org/os/community"
BUG_REPORT_URL="https://bugs.devuan.org/"
- ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
OUTPUT=ISO
OUTPUT_URL="nfs://192.168.1.100/md0/REAR"
BACKUP=NETFS
BACKUP_URL=iso:///backup
-
Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR):
Dell PowerEdge R210 -
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
Intel Xeon 64 Bit -
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
Bios with Grub2 -
Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
Local SSD -
Storage layout (
lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT):
/dev/sda /dev/sda sata disk 14.9G
`-/dev/sda1 /dev/sda1 /dev/sda part ext4 14.9G /
- Description of the issue (ideally so that others can reproduce it):
Because this machine does not use very much storage
I opted to include the backup within the ISO
as you can see in my site.conf file listed above.
ReaR created the ISO and stores it on my NFS share without issues.
The problem arises when I try to recover.
So I boot the ISO, I get to the login, type root and go in.
After I type 'rear recover' I get the following:
ERROR: Cannot mount ISO because there is no /dev/disk/by-label/RELAXRECOVER
Also if I do ifconfig there are no ethernet adapters available
other than lo. That is strange to me.
One thing that I found odd is that it tries to run systemd-udevd
(and giving a countdown) and it fails to start it because
it doesn't exist on my OS.
I was unable to get it to run any further...
-
Workaround, if any:
None -
Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):
TBD
You can drag-drop log files into this editor to create an attachment or
paste verbatim text
like command output or file content by including it between a leading
and a closing line of
three backticks like this:
verbatim content
jsmeix commented at 2023-03-08 09:04:¶
Right now I didn't do an analysis.
From plain looking at
https://github.com/rear/rear/issues/2955#issue-1614143309
the ISO label 'RELAXRECOVER' puzzles me
because there is no longer 'RELAXRECOVER'
in our ReaR 2.7 code since
https://github.com/rear/rear/pull/2814
was merged, see also our ReaR 2.7 release notes at
https://github.com/rear/rear/blob/rear-2.7/doc/rear-release-notes.txt#L319
But 'RELAXRECOVER' was in ReaR 2.6
Perhaps something mixed up while you
tried both ReaR 2.6 and ReaR 2.7 on your system?
See the section
"Testing current ReaR upstream GitHub master code"
in
https://en.opensuse.org/SDB:Disaster_Recovery
how you can have several ReaR versions in parallel
each one in its own separated directory
without conflicts between each other.
In general I recommend to try out our latest GitHub master code
because the GitHub master code is the only place where we fix things
and if there are issues it helps when you use exactly the code
where we could fix things.
In general for an analysis we need full debug ReaR log files.
See the section
"Debugging issues with Relax-and-Recover" in
https://en.opensuse.org/SDB:Disaster_Recovery
jsmeix commented at 2023-03-08 09:20:¶
In particular regarding Devuan and
in general regarding Debian based Linux distributions:
On
https://en.wikipedia.org/wiki/Devuan
I read that
Devuan is a fork of the Debian Linux distribution
that uses sysvinit, runit or OpenRC instead of systemd
At ReaR upstream we neither have a maintainer for Devuan
nor do we have a maintainer for Debian or for
Debian based Linux distributions (like Ubuntu).
So ReaR support for Debian based Linux distributions
and in particular ReaR support for Devuan
with its nowadays unusual init system
can be only as good as voluntary contributors
who use such Linux distributions contribute
to ReaR upstream - which is much appreciated!
3t0n1c commented at 2023-03-08 13:19:¶
RELAXRECOVER' was in ReaR 2.6
I apologize for the confusion. You are correct in stating that
RELAXRECOVER' was in ReaR 2.6 but not in 2.7.
The error I get in 2.7 is different, though I tried so many things I
forgot to take a screenshot of the exact error for 2.7.
It very much stops in the same spot.
I will take your advise and try the latest from Master and see where
that takes me.
I don't expect anyone to go out of their way to make this work with
Devuan if it's that much of a hassle. I was more or less looking for
advise on how to make it work (if possible) perhaps from someone who got
it working on Devuan.
Since Devuan is a fork of Debian without systemD, and also because ReaR
works just fine on Debian 11, it made sense to me to point fingers at
systemD not being there as being the culprit.
I've attached 2 screenshots that may show more relevant information.
I appreciate all your efforts and I will report back using the latest
version from Master.
Thank you!
3t0n1c commented at 2023-03-08 14:55:¶
So I grabbed the latest from github and compiled my own .deb file.
I ran into the ISO size limitation while running "rear mkbackup", so I
decided to place the backup.tar.gz file on the nfs share instead. That
worked well to create the ISO and backup file on the nfs server.
All went well, until I needed to restore.
I have taken screenshots to show everything. Hopefully it can shed some
light on what is happening.
No network interfaces are available. Right after I logged in with root,
I ran ifconfig. One of the screenshots shows that only lo interface is
available. No wonder it can't get to the nfs server during restore.
See screenshots below for details.
Thank you
3t0n1c commented at 2023-03-09 22:40:¶
If anyone has any suggestions I am willing to try just about anything. I would love to be able to use ReaR on Devuan...
Thanks everyone!
jsmeix commented at 2023-03-10 08:58:¶
@3t0n1c
sorry for possibly longer delays, you are not forgotten.
But other stuff came in at work which has higher priority.
In general SUSE customer issues have higher priority for me
(and for my employer and our customers) than ReaR upstream issues.
gdha commented at 2023-03-10 09:13:¶
@3t0n1c if you do not have active network interface then mounting an ISO
image over NFS is not possible.
Try to identify how the network interfaces are started on a normal
system as perhaps we are missing the required init script on the rescue
image as this Devuan is not systemd minded.
jsmeix commented at 2023-03-10 09:13:¶
@3t0n1c
a general info:
After you logged in as 'root' into the ReaR recovery system
you can do therein what you want before you start "rear -D recover".
In case of issues I recommend to run 'rear' with '-D' to get
debug messages on the terminal and full 'set -x' debugscript
messages in the ReaR log file.
So before you run "rear -D recover" you can do what is needed
to manually set up networking as you need it in your case.
When there is no network interface like 'eth0' perhaps some
kernel modules for the network interface hardware may not
have been loaded automatically in Devuan so you may have
to load them manually.
When you know what commands make networking work in the
ReaR recovery system as you need it in your case
you can specify them in NETWORKING_PREPARATION_COMMANDS
to get that automated, see usr/share/rear/conf/default.conf
https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/conf/default.conf#L3185
and to get kernel module loading done use MODULES_LOAD
https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/conf/default.conf#L1598
To get all modules of the original system in the recovery system
with the same module loading ordering as in the original system
you may use the output of a command like
# lsmod | tail -n +2 | cut -d ' ' -f 1 | tac | tr -s '[:space:]' ' '
as value for MODULES_LOAD, cf.
https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/conf/examples/SLE12-SP2-btrfs-example.conf#L38
You can also do what is needed (e.g. create a symlink)
so that /dev/disk/by-label/RELAXRECOVER points to
the actual kernel device node where your ISO medium is.
E.g. use 'lsblk' to see what block devices there are
in the ReaR recovery system - one of them should be
the one where your ISO medium is.
3t0n1c commented at 2023-03-10 11:49:¶
@3t0n1c if you do not have active network interface then mounting an ISO image over NFS is not possible. Try to identify how the network interfaces are started on a normal system as perhaps we are missing the required init script on the rescue image as this Devuan is not systemd minded.
As you can see from my reply, I am aware there are no network interfaces
available thus NFS is not going to work.
The question is what fails to run and how do I get it going?
I don't mind modifying some scripts to get things running. I feel pretty
comfortable in bash.
I just don't want to start having to learn what each script does and how
they all interact in order to reinvent the wheel if I don't have to.
Thank you for your input!
3t0n1c commented at 2023-03-10 11:55:¶
@3t0n1c a general info:
After you logged in as 'root' into the ReaR recovery system you can do therein what you want before you start "rear -D recover".
In case of issues I recommend to run 'rear' with '-D' to get debug messages on the terminal and full 'set -x' debugscript messages in the ReaR log file.
So before you run "rear -D recover" you can do what is needed to manually set up networking as you need it in your case. When there is no network interface like 'eth0' perhaps some kernel modules for the network interface hardware may not have been loaded automatically in Devuan so you may have to load them manually. When you know what commands make networking work in the ReaR recovery system as you need it in your case you can specify them in NETWORKING_PREPARATION_COMMANDS to get that automated, see usr/share/rear/conf/default.conf https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/conf/default.conf#L3185 and to get kernel module loading done use MODULES_LOAD https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/conf/default.conf#L1598 To get all modules of the original system in the recovery system with the same module loading ordering as in the original system you may use the output of a command like
# lsmod | tail -n +2 | cut -d ' ' -f 1 | tac | tr -s '[:space:]' ' 'as value for MODULES_LOAD, cf. https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/conf/examples/SLE12-SP2-btrfs-example.conf#L38
You can also do what is needed (e.g. create a symlink) so that /dev/disk/by-label/RELAXRECOVER points to the actual kernel device node where your ISO medium is. E.g. use 'lsblk' to see what block devices there are in the ReaR recovery system - one of them should be the one where your ISO medium is.
I will look into what you suggested. Sounds like I have some work to
do.
As you can see from my screenshots, there are no network interfaces
available. Also unde /dev/ there is no sr0 or sr1, or even cdrom. No
ethX interfaces exist. Like you said, and I agree the drivers are not
loaded.
That leads me again to believing without systemd modules are not loaded
thus things just don't work.
I am not sure how easy it is/will be modify any of the scripts to make
this work. I am beginning to be more and more pessimistic about this
ever working properly on Devuan without a tone of work or modifications.
Thank you for your advice!
3t0n1c commented at 2023-03-10 14:03:¶
I just went and tried a few things just to see if we can turn the
spotlight on the problem... So far no go.
So first I tried to just run lsblk to see the drives. It returned
nothing... See picture.
The I did the lsmod to list the loaded modules.
See output here:
I don't know if that tells you anything, but I don't see any network
modules listed, or drives, or cdrom or anything like that.
There is barely anything there. The original system has a boatload of
modules loaded. Shouldn't they all be present here as well?
Should is the key word here. Is there a way to force-load all the
modules that are loaded on the original system on here and see what
happens?
As suggested, I will try to run it using the -D flag and report back.
Thank you all. I really appreciate the efforts.
jsmeix commented at 2023-03-10 14:36:¶
@3t0n1c
regarding
Is there a way to force-load all the modules
that are loaded on the original system
on here and see what happens?
In my
https://github.com/rear/rear/issues/2955#issuecomment-1463501314
I meant that you shoud run on your original system
a command like
# lsmod | tail -n +2 | cut -d ' ' -f 1 | tac | tr -s '[:space:]' ' '
and use its output as value
for MODULES_LOAD in your etc/rear/local.conf
and then recreate the ReaR recovery system and the backup
anew on your original system via
# rear -D mkbackup
Then retry if this works better to recreate your original system
on your replacement hardware or your replacement VM.
3t0n1c commented at 2023-03-10 15:00:¶
I apologize, I misunderstood.
I will try that are report back.
Thank you!
3t0n1c commented at 2023-03-10 16:05:¶
Quick update!
So I added the list of modules to my /etc/rear/local.conf file.
Ran rear -D mkbackup. Worked fine. No issues.
Then, proceeded to boot the system from the fresh ISO.
It appears to get stuck at the following screen, though the kernel is
not locked. I can hit Numlock and the led on my keyboard flashes, so
she's not dead or panicked. I've waited 30 minutes, never made it past
this point. See picture below:
Also, just in case this may shed some light on stuff, here's the output from rear -D mkbackup
rear -D mkbackup
Relax-and-Recover 2.7-git.4996.ff64d61e.master / 2023-03-07
Running rear mkbackup (PID 6479 date 2023-03-10 10:39:20)
Command line options: /usr/sbin/rear -D mkbackup
Using log file: /var/log/rear/rear-router01.log
Using build area: /var/tmp/rear.LbktYX8daUnl7m7
Running 'init' stage ======================
Running workflow mkbackup on the normal/original system
Running 'prep' stage ======================
Using backup archive '/var/tmp/rear.LbktYX8daUnl7m7/outputfs/router01/backup.tar.gz'
Using '/usr/bin/xorrisofs' to create ISO filesystem images
Using autodetected kernel '/boot/vmlinuz-5.10.0-17-amd64' as kernel in the recovery system
Modified ReaR recovery system area after 'prep' stage (/var/tmp/rear.LbktYX8daUnl7m7/rootfs not empty)
Running 'layout/save' stage ======================
Creating disk layout
Overwriting existing disk layout file /var/lib/rear/layout/disklayout.conf
Disabling excluded components in /var/lib/rear/layout/disklayout.conf
Using guessed bootloader 'GRUB' for 'rear recover' (found in first bytes on /dev/sda)
Skip saving storage layout as 'barrel' devicegraph (no 'barrel' command)
Verifying that the entries in /var/lib/rear/layout/disklayout.conf are correct
Created disk layout (check the results in /var/lib/rear/layout/disklayout.conf)
Running 'rescue' stage ======================
Creating recovery system root filesystem skeleton layout
Handling network interface 'eth0'
eth0 is a physical device
Handled network interface 'eth0'
Handling network interface 'eth0.2'
eth0.2 is a vlan
eth0.2 has eth0 as parent
eth0 already handled...
Handled network interface 'eth0.2'
Handling network interface 'eth0.3'
eth0.3 is a vlan
eth0.3 has eth0 as parent
eth0 already handled...
Handled network interface 'eth0.3'
Handling network interface 'eth0.5'
eth0.5 is a vlan
eth0.5 has eth0 as parent
eth0 already handled...
Handled network interface 'eth0.5'
Handling network interface 'eth1'
eth1 is a physical device
Handled network interface 'eth1'
Handling network interface 'eth1.147'
eth1.147 is a vlan
eth1.147 has eth1 as parent
eth1 already handled...
Handled network interface 'eth1.147'
Handling network interface 'eth1.148'
eth1.148 is a vlan
eth1.148 has eth1 as parent
eth1 already handled...
Handled network interface 'eth1.148'
Handling network interface 'eth1.149'
eth1.149 is a vlan
eth1.149 has eth1 as parent
eth1 already handled...
Handled network interface 'eth1.149'
Handling network interface 'eth1.150'
eth1.150 is a vlan
eth1.150 has eth1 as parent
eth1 already handled...
Handled network interface 'eth1.150'
Skipping 'proton0': not bound to any physical interface.
Skipping 'tun0': not bound to any physical interface.
Included current keyboard mapping (via 'dumpkeys -f')
No default US keyboard mapping included (no KEYMAPS_DEFAULT_DIRECTORY specified)
No support for different keyboard layouts (neither KEYMAPS_DEFAULT_DIRECTORY nor KEYMAPS_DIRECTORIES specified)
Copying logfile /var/log/rear/rear-router01.log into initramfs as '/tmp/rear-router01-partial-2023-03-10T10:39:24-05:00.log'
Running 'build' stage ======================
Copying files and directories
Copying binaries and libraries
Copying all kernel modules in /lib/modules/5.10.0-17-amd64 (MODULES contains 'all_modules')
Copying all files in /lib*/firmware/
Symlink '/usr/share/misc/magic' -> '/usr/share/file/magic' refers to a non-existing directory on the recovery system.
It will not be copied by default. You can include '/usr/share/file/magic' via the 'COPY_AS_IS' configuration variable.
Skip copying broken symlink '/etc/mtab' target '/proc/16146/mounts' on /proc/ /sys/ /dev/ or /run/
Testing that the recovery system in /var/tmp/rear.LbktYX8daUnl7m7/rootfs contains a usable system
Testing each binary with 'ldd' and look for 'not found' libraries within the recovery system
Testing that the existing programs in the PROGS array can be found as executable command within the recovery system
Testing that each program in the REQUIRED_PROGS array can be found as executable command within the recovery system
Running 'pack' stage ======================
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (166591322 bytes) in 25 seconds
Running 'output' stage ======================
Making ISO image
Wrote ISO image: /var/lib/rear/output/rear-router01.iso (167M)
Copying resulting files to nfs location
Saving /var/log/rear/rear-router01.log as rear-router01.log to nfs location
Copying result files '/var/lib/rear/output/rear-router01.iso /var/tmp/rear.LbktYX8daUnl7m7/tmp/VERSION /var/tmp/rear.LbktYX8daUnl7m7/tmp/README /var/tmp/rear.LbktYX8daUnl7m7/tmp/rear-router01.log' to /var/tmp/rear.LbktYX8daUnl7m7/outputfs/router01 at nfs location
Running 'backup' stage ======================
Making backup (using backup method NETFS)
Creating tar archive '/var/tmp/rear.LbktYX8daUnl7m7/outputfs/router01/backup.tar.gz'
Archived 2972 MiB [avg 10283 KiB/sec] OK
WARNING: tar ended with return code 1 and below output (last 5 lines):
---snip---
tar: /var/tmp/rear.LbktYX8daUnl7m7/tmp/backup.log: file changed as we read it
----------
This means that files have been modified during the archiving
process. As a result the backup may not be completely consistent
or may not be a perfect copy of the system. Relax-and-Recover
will continue, however it is highly advisable to verify the
backup in order to be sure to safely recover this system.
Archived 2972 MiB in 297 seconds [avg 10248 KiB/sec]
Exiting rear mkbackup (PID 6479) and its descendant processes ...
Running exit tasks
To remove the build area you may use (with caution): rm -Rf --one-file-system /var/tmp/rear.LbktYX8daUnl7m7
As always thank you for your help!
3t0n1c commented at 2023-03-10 19:11:¶
Enjoy your weekend everyone!
jsmeix commented at 2023-03-13 07:32:¶
Regarding no further output after
Probing EDD (edd=off to disable)... ok
In ReaR 2.7 we have a new issue because
we fixed some serial console code in ReaR
but we did not see how all that serial console code in ReaR
(partially falsely) worked so that in the end it mostly worked.
In
https://github.com/rear/rear/issues/2843
see in particular
https://github.com/rear/rear/issues/2843#issuecomment-1196337765
for the reason (as far as I understand it) and
https://github.com/rear/rear/issues/2843#issuecomment-1196588256
https://github.com/rear/rear/issues/2843#issuecomment-1196592017
for possible solutions.
For the latter solution ReaR version 2.7 is needed.
When I boot the ReaR recovery system on my KVM VMs
without USE_SERIAL_CONSOLE='no' in my
etc/rear/local.conf (during "rear mkrescue/mkbackup)
I get
...
Loading kernel.....................
Loading initrd.cgz..........................ready.
Probing EDD (edd=off to disable)... ok
and then no further kernel boot messages and
no ReaR recovery system startup messages are shown.
But after a shorter time (about half a minute)
the screen blanks and shows the recovery system login screen
where then all works normal.
It never happened to me that after
Probing EDD (edd=off to disable)... ok
no longer any output is shown
but that happened to another ReaR user, see
https://github.com/rear/rear/issues/2933#issuecomment-1432684472
Now I see that
https://github.com/rear/rear/issues/2843#issuecomment-1196556049
shows how it can happen that with a wrong serial console setting
it can look as if the ReaR recovery system had hung up after
Probing EDD (edd=off to disable)... ok
(excerpt from that @hpannenb issue comment)
During the startup I will be asked in 55-migrate-network-devices.sh
about to provide the proper network mapping.
With ReaR 2.7 this dialog is present on e.g. ttyS0 (serial) only;
with ReaR 2.6 this dialog was shown on tty0 (console).
In
skel/default/etc/scripts/system-setup.d/55-migrate-network-devices.sh
there is
echo "The original network interface $old_dev $old_mac is not available"
PS3="Choose replacement for $old_dev $old_mac "
skip_choice="Skip replacing $old_dev $old_mac"
select choice in "${NEW_DEVICES[@]}" "$skip_choice" ; do
But 'select' does not support a timeout
so 'select' waits endlessly and for the user
it looks as if the ReaR recovery system had hung up.
By "googling" for 'bash select timeout' I found my
https://github.com/rear/rear/issues/1366
and the related
https://github.com/rear/rear/issues/1399
Those resulted the new UserInput() function
and its usage during "rear recover".
But we cannot 'source lib/_input-output-functions.sh'
during ReaR recovery system startup in
skel/default/etc/scripts/system-setup
because lib/_input-output-functions.sh also does special stuff
like stdin stdout redirections and exit task settings.
I think this should not be in a script that defines functions
but it was always therein and I do not understand
why this is in lib/_input-output-functions.sh and
not in sbin/rear (where I would expect it)
so I left it as is because it works as is.
But in the end it means the 'select' and 'read' calls in the
ReaR recovery system startup scripts in skel/default/etc/scripts
were not changed so there are not yet timeouts for
user dialogs during ReaR recovery system startup
which contradicts the initial intent of
https://github.com/rear/rear/issues/1366
3t0n1c commented at 2023-03-13 19:52:¶
Hey all. I'm back with updates.
So what you suggested worked getting me further, and I was able to get
it to load past Probing EDD (edd=off to disable)... ok.
Since I use 2.7 I added USE_SERIAL_CONSOLE='no' to my local.conf file,
so that indeed helped.
Now, the system comes up to the login screen. I have networking and I
can ping my nfs server. This is awesome!
Though the saga continues. For some odd reason it does not detect any
storage. The storage is a single SSD drive attached to sata0 and it is
functional as the machine can boot from it.
I ran lsblk and fdisk -l. Both return nothing.
Just an educated guess, perhaps udev is not running or not starting,
hence can't detect storage? Is it due to systemd not being present?
So of course further down if I run "rear recover", it fails stating ERROR: No filesystem mounted on /mnt/local.
Thoughts?
Thank you for all the help and effort guys!!!
3t0n1c commented at 2023-03-13 23:37:¶
I just took a few screenshots for you guys to see details.
I hope this helps:
Thank you
jsmeix commented at 2023-03-14 13:59:¶
I don't know why there are no disk / block device nodes
in the ReaR recovery system for Devuan.
I guess it is because the Devuan init system is too different
and because there is no Devuan specific support in ReaR
"rear mkrescue" fails to make "the right things" in the
ReaR recovery system for Devuan.
In general regarding debugging issues during
the startup of the ReaR recovery system:
The basic idea behind is to not let the startup scripts
that are run initially in the ReaR recovery system are
run automatically and mostly silently but instead run
them manually one after the other with 'set -x'
bash debugging mode.
To do that add 'debug' to the ReaR kernel command line
when booting the ReaR recovery system.
In the ReaR recovery system boot menue select
the topmost enty of the form "Recover HOSTNAME" and
press the [Tab] key to edit the boot command line
and append the word ' debug' at its end and boot that.
You might then find yourself stopped at a blank screen.
In this case press [Enter] which runs the very first of the
startup scripts (/etc/scripts/system-setup.d/00functions.sh).
There is or was sometimes the issue that the initial message
is not always printed so you may need to type the very first
[Enter] blindly.
The further startup scripts are run one by one
each one after pressing [Enter].
In 'debug' mode the startup scripts are run with 'set -x'
so that this way you can better see what actually goes on
in each of the startup scripts.
Perhaps this might indicate where things do not work
during ReaR recovery system startup for Devuan.
3t0n1c commented at 2023-03-14 14:28:¶
I will give that a shot and report back.
Thank you for all your support, it is much appreciated!
3t0n1c commented at 2023-03-14 18:04:¶
So before I dug any deeper, I looked again at the list of kernel modules
and realized I was missing the sata related module. I've added it to the
list, and made another backup.
Things have improved but we're not out of the woods yet. So, lsblk now
shows both my sata drive and the USB thumb drive that rear boots from,
however fdisk -l still returns no output.
I am not sure if this might help in any way shape or form, but I thought
it was worth mentioning.
Rear proceeds to error out stating that /dev/sda is not a block device.
Something is still broken...
See screenshots:
See BUG in the middle of screen below. Perhaps it's not a bug but yet
erroring out due to other unknown issues.
I will have to run in debug mode as suggested, but if any other suggestions come to mind please don't hesitate to reply.
Thank you
jsmeix commented at 2023-03-15 11:41:¶
@3t0n1c
I wonder how you could "missing the sata related module"?
Did a command like
# lsmod | tail -n +2 | cut -d ' ' -f 1 | tac | tr -s '[:space:]' ' '
and using its output as value
for MODULES_LOAD in your etc/rear/local.conf
somehow not work for you?
Module dependencies can be rather tricky and non-obvious, cf.
https://github.com/rear/rear/issues/1355
and see the description about 'MODULES' in default.conf
https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/conf/default.conf#L1540
So ensure to have all modules loaded inside the ReaR recovery system
that are loaded on your original system.
How does
# parted -s /dev/sda unit GiB print
show your disk when called inside the ReaR recovery system?
Does it work to manually use 'parted' in the recovery system like
# parted -s /dev/sda mklabel gpt
and if yes, what does then
# parted -s /dev/sda unit GiB print
show?
As far as I see ReaR does not call 'fdisk' but 'parted' and
'sfdisk' only in the special case of BLOCKCLONE_STRICT_PARTITIONING.
jsmeix commented at 2023-03-15 11:48:¶
@3t0n1c
to reduce testing time while you try out this and that
with the ReaR recovery system:
To only create a new ReaR recovery system
without a backup of the files use
# rear -D mkrescue
This may result errors when the backup is included in the ISO
but it works when the backup is e.g. on an NFS server.
jsmeix commented at 2023-03-15 12:01:¶
The above BUG ERROR is not a bug in ReaR.
It is because of the missing support in ReaR
for the rather special Devuan init system.
I had tested ReaR 2.7 with old SLES11 that has no systemd
and I had no issues with the recovery system startup, cf.
https://github.com/rear/rear/wiki/Test-Matrix-ReaR-2.7#sles11-sp4-with-lvm-that-is-luks-encrypted
So usual traditional SysVinit is supported in ReaR 2.7
at least SysVinit as used in SLES and RHEL.
3t0n1c commented at 2023-03-15 12:40:¶
@3t0n1c I wonder how you could "missing the sata related module"?
The command worked fine, when I copied and pasted the output, I missed
some stuff. The line was very long.
Error on my part...
Did a command like
# lsmod | tail -n +2 | cut -d ' ' -f 1 | tac | tr -s '[:space:]' ' 'and using its output as value for MODULES_LOAD in your etc/rear/local.conf somehow not work for you?
Module dependencies can be rather tricky and non-obvious, cf. #1355 and see the description about 'MODULES' in default.conf https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/conf/default.conf#L1540
So ensure to have all modules loaded inside the ReaR recovery system that are loaded on your original system.
Well, I tried parted as per your instructions and no go.
See screenshot below:
How does
# parted -s /dev/sda unit GiB printshow your disk when called inside the ReaR recovery system?
Does it work to manually use 'parted' in the recovery system like
# parted -s /dev/sda mklabel gptand if yes, what does then
# parted -s /dev/sda unit GiB printshow?
As far as I see ReaR does not call 'fdisk' but 'parted' and 'sfdisk' only in the special case of BLOCKCLONE_STRICT_PARTITIONING.
jsmeix commented at 2023-03-15 13:53:¶
On my openSUSE Leap 15.4 homeoffice laptop I get
# man lsblk
...
DESCRIPTION
lsblk lists information about all available
or the specified block devices.
The lsblk command reads the sysfs filesystem
and udev db to gather information.
If the udev db is not available
or lsblk is compiled without udev support,
then it tries to read LABELs, UUIDs
and filesystem types from the block device.
In this case root permissions are necessary.
...
# mount
...
sysfs on /sys type sysfs ...
devtmpfs on /dev type devtmpfs ...
# ls -l /sys/dev/block/ | grep sda
lrwxrwxrwx ... 8:0 -> ../../devices/pci0000:00/0000:00:1f.2/ata5/host4/target4:0:0/4:0:0:0/block/sda
lrwxrwxrwx ... 8:1 -> ../../devices/pci0000:00/0000:00:1f.2/ata5/host4/target4:0:0/4:0:0:0/block/sda/sda1
lrwxrwxrwx ... 8:2 -> ../../devices/pci0000:00/0000:00:1f.2/ata5/host4/target4:0:0/4:0:0:0/block/sda/sda2
...
# ls -l /dev/sda*
brw-rw---- 1 root disk 8, 0 ... /dev/sda
brw-rw---- 1 root disk 8, 1 ... /dev/sda1
brw-rw---- 1 root disk 8, 2 ... /dev/sda2
...
So it seems you have things only in the
kernel's own (virtual) sysfs filesystem, cf.
https://en.wikipedia.org/wiki/Sysfs
but no device nodes in /dev
(which is in my case also a virtual devtmpfs filesystem).
Because usually udev cares about the device nodes in /dev
it seems udev (or whatever else is used for the device nodes in /dev)
does not work or is missing in the ReaR recovery system for Devuan.
I neither know what is used in Devuan for the device nodes in /dev
nor how that is used nor how to enable/start it in Devuan.
3t0n1c commented at 2023-03-16 13:06:¶
I appreciate all the help and input.
I am trying to figure out what handles /dev nodes in Devuan.
I believe it actually uses eudev, hence the presence of
/etc/init.d/eudev .
Then, under /etc/rc0.d I have a file called K02eudev.
Obviously that file is a link to /etc/init.d/eudev.
These are the contents.
Hopefully this might shed some light on what I could add/modify
to the ReaR udev scripts to get this going.
#!/bin/sh -e
### BEGIN INIT INFO
# Provides: eudev udev
# Required-Start: mountkernfs
# Required-Stop: umountroot
# Default-Start: S
# Default-Stop: 0 6
# Short-Description: Start udevd, populate /dev and load drivers.
### END INIT INFO
PATH="/lib/udev:/sbin:/bin"
NAME="udevd"
DAEMON="/sbin/udevd"
DAEMON_ARGS="--daemon"
DESC="hot-plug events dispatcher"
# we need to unmount /dev/pts/ and remount it later over the devtmpfs
unmount_devpts() {
if mountpoint -q /dev/pts/; then
umount -n -l /dev/pts/
fi
if mountpoint -q /dev/shm/; then
umount -n -l /dev/shm/
fi
}
# mount a devtmpfs over /dev, if somebody did not already do it
mount_devtmpfs() {
if grep -E -q "^[^[:space:]]+ /dev devtmpfs" /proc/mounts; then
mount -n -o remount,nosuid,size=$tmpfs_size,mode=0755 -t devtmpfs devtmpfs /dev
return
fi
if ! mount -n -o nosuid,size=$tmpfs_size,mode=0755 -t devtmpfs devtmpfs /dev; then
log_failure_msg "eudev requires devtmpfs support, $NAME not started"
log_end_msg 1
fi
return 0
}
create_dev_makedev() {
if [ -e /sbin/MAKEDEV ]; then
ln -sf /sbin/MAKEDEV /dev/MAKEDEV
else
ln -sf /bin/true /dev/MAKEDEV
fi
}
# shell version of /usr/bin/tty
my_tty() {
[ -x /bin/readlink ] || return 0
[ -e /proc/self/fd/0 ] || return 0
readlink --silent /proc/self/fd/0 || true
}
warn_if_interactive() {
if [ "$RUNLEVEL" = "S" -a "$PREVLEVEL" = "N" ]; then
return
fi
TTY=$(my_tty)
if [ -z "$TTY" -o "$TTY" = "/dev/console" -o "$TTY" = "/dev/null" ]; then
return
fi
printf "\n\n\nIt has been detected that the command\n\n\t$0 $*\n\n"
printf "has been run from an interactive shell.\n"
printf "It will probably not do what you expect, so this script will wait\n"
printf "60 seconds before continuing. Press ^C to stop it.\n"
printf "RUNNING THIS COMMAND IS HIGHLY DISCOURAGED!\n\n\n\n"
sleep 60
}
make_static_nodes() {
[ -e /lib/modules/$(uname -r)/modules.devname ] || return 0
[ -x /bin/kmod ] || return 0
/bin/kmod static-nodes --format=tmpfiles --output=/proc/self/fd/1 | \
while read type name mode uid gid age arg; do
[ -e $name ] && continue
case "$type" in
c|b|c!|b!) mknod -m $mode $name $type $(echo $arg | sed 's/:/ /') ;;
d|d!) mkdir $name ;;
*) echo "unparseable line ($type $name $mode $uid $gid $age $arg)" >&2 ;;
esac
if [ -x /sbin/restorecon ]; then
/sbin/restorecon $name
fi
done
}
##############################################################################
[ -x $DAEMON ] || exit 0
# defaults
tmpfs_size="10M"
if [ -e /etc/udev/udev.conf ]; then
. /etc/udev/udev.conf
fi
. /lib/lsb/init-functions
if [ ! -e /proc/filesystems ]; then
log_failure_msg "eudev requires a mounted procfs, $NAME not started"
log_end_msg 1
fi
if ! grep -q '[[:space:]]devtmpfs$' /proc/filesystems; then
log_failure_msg "eudev requires devtmpfs support, $NAME not started"
log_end_msg 1
fi
if [ ! -d /sys/class/ ]; then
log_failure_msg "eudev requires a mounted sysfs, $NAME not started"
log_end_msg 1
fi
# System processes and/or kernel threads are surrounded by brackets: [...]
if ! ps --no-headers --format args ax | egrep -q '^\['; then
log_warning_msg "eudev does not support containers, $NAME not started"
exit 0
fi
if [ -d /sys/class/mem/null -a ! -L /sys/class/mem/null ] || \
[ -e /sys/block -a ! -e /sys/class/block ]; then
log_warning_msg "CONFIG_SYSFS_DEPRECATED must not be selected"
log_warning_msg "Booting will continue in 30 seconds but many things will be broken"
sleep 30
fi
# When modifying this script, do not forget that between the time that the
# new /dev has been mounted and udevadm trigger has been run there will be
# no /dev/null. This also means that you cannot use the "&" shell command.
case "$1" in
start)
if [ ! -e "/run/udev/" ]; then
warn_if_interactive
fi
if [ -w /sys/kernel/uevent_helper ]; then
echo > /sys/kernel/uevent_helper
fi
if ! mountpoint -q /dev/; then
unmount_devpts
mount_devtmpfs
[ -d /proc/1 ] || mount -n /proc
fi
make_static_nodes
# clean up parts of the database created by the initramfs udev
udevadm info --cleanup-db
# set the SELinux context for devices created in the initramfs
[ -x /sbin/restorecon ] && /sbin/restorecon -R /dev
log_daemon_msg "Starting $DESC" "$NAME"
if start-stop-daemon --start --name $NAME --exec $DAEMON --quiet \
-- $DAEMON_ARGS; then
log_end_msg $?
else
log_warning_msg $?
log_warning_msg "Waiting 15 seconds and trying to continue anyway"
sleep 15
fi
log_action_begin_msg "Synthesizing the initial hotplug events (subsystems)"
if udevadm trigger --type=subsystems --action=add; then
log_action_end_msg $?
else
log_action_end_msg $?
fi
log_action_begin_msg "Synthesizing the initial hotplug events (devices)"
if udevadm trigger --type=devices --action=add; then
log_action_end_msg $?
else
log_action_end_msg $?
fi
create_dev_makedev
# wait for the udevd childs to finish
log_action_begin_msg "Waiting for /dev to be fully populated"
if udevadm settle; then
log_action_end_msg 0
else
log_action_end_msg 0 'timeout'
fi
;;
stop)
log_daemon_msg "Stopping $DESC" "$NAME"
if start-stop-daemon --stop --name $NAME --quiet --oknodo --retry 5; then
log_end_msg $?
else
log_end_msg $?
fi
;;
restart)
log_daemon_msg "Stopping $DESC" "$NAME"
if start-stop-daemon --stop --name $NAME --quiet --oknodo --retry 5; then
log_end_msg $?
else
log_end_msg $? || true
fi
log_daemon_msg "Starting $DESC" "$NAME"
if start-stop-daemon --start --name $NAME --exec $DAEMON --quiet \
-- $DAEMON_ARGS; then
log_end_msg $?
else
log_end_msg $?
fi
;;
reload|force-reload)
udevadm control --reload-rules
;;
status)
status_of_proc $DAEMON $NAME && exit 0 || exit $?
;;
*)
echo "Usage: /etc/init.d/eudev {start|stop|restart|reload|force-reload|status}" >&2
exit 1
;;
esac
exit 0
3t0n1c commented at 2023-03-16 13:11:¶
I apologize about the way the code pasted. I used the "add code" tag but
it didn't help much for obvious reasons...
Perhaps I'm not doing it right.
Sorry!
jsmeix commented at 2023-03-16 13:57:¶
See
https://github.com/rear/rear/blob/master/.github/ISSUE_TEMPLATE.md
how to paste verbatim text like command output or file content
by including it between a leading and a closing line
of three backticks (othing else on those lines) like:
```
verbatim text
```
3t0n1c commented at 2023-03-16 14:02:¶
See https://github.com/rear/rear/blob/master/.github/ISSUE_TEMPLATE.md how to paste verbatim text like command output or file content by including it between a leading and a closing line of three backticks (othing else on those lines) like:
verbatim text
Good to know. Thank you sir!
3t0n1c commented at 2023-03-16 14:07:¶
This might also help understand how Devuan uses udev, or eudev rather.
Is it running?
-->
someuser@mybox:~$ /etc/init.d/eudev status
or
-->
someuser@mybox:~$ ps -aef | grep udev
restart it:
-->
someuser@mybox:~$ sudo /etc/init.d/eudev restart
eudev is a replacement for udev obviously.
Udev depends on and is integrated with systemd,
so regular udev won't work in Devuan for that reason.
Thanks
jsmeix commented at 2023-03-16 14:11:¶
See in default.conf the descriptions about COPY_AS_IS
and the LIBS PROGS and REQUIRED_PROGS arrays
how to get additional things as needed
included in the ReaR recovery system.
This is only a first step to get some initial support
in ReaR for the rather special Devuan init system
for your specific case.
When you got the needed things of the Devuan init system
included in the ReaR recovery system, you can then
boot the ReaR recovery system, log in as root
and - as a first trial and error step - manually
do what is needed get the needed device nodes
setup in /dev inside the ReaR recovery system,
e.g. by running that script in
https://github.com/rear/rear/issues/2955#issuecomment-1471921526
Check what programs are called by that script because
you also need all that in your recovery system.
You may add a line set -x at the beginning of that script
to print commands and their arguments as they are executed
for debugging.
When the needed device nodes in /dev are there
then you can as the next trial and error step
run "rear -D recover" and see how far that goes.
And then step by step repeat the above procedure
again and again until "rear recover" works
for your specific case.
It is likely much more work to get general support
in ReaR for the rather special Devuan init system.
jsmeix commented at 2023-03-16 14:13:¶
Perhaps you could get in contact with Devuan developers
and make them aware of ReaR?
Perhaps they get interested so that
they may even contribute to ReaR
to get general support in ReaR
for the Devuan init system?
jsmeix commented at 2023-03-16 14:26:¶
FYI
what I found regarding 'init.d' in the ReaR sources (excerpt)
localhost:~/rear.github.master # find usr/sbin/rear usr/share/rear -type f | xargs grep 'init\.d'
...
usr/share/rear/conf/Linux-ia64.conf:
COPY_AS_IS+=( /etc/sysconfig /etc/dev.d /etc/udev /etc/init.d /etc/modprobe.d /etc/security /etc/netplug /etc/netplug.d )
Because IA64 support is rather old I guess it is from times
when SysVinit was in use, so e.g. something like
COPY_AS_IS+=( ... /etc/init.d ... )
could be needed (and likely much more).
E.g. in the script in
https://github.com/rear/rear/issues/2955#issuecomment-1471921526
I see things like 'log_failure_msg' and others are called
so you need to get all that in your recovery system.
3t0n1c commented at 2023-03-16 14:33:¶
I appreciate your help.
It sounds like it's becoming exponentially more complicated as we move
along with this.
I posted on the Devuan help forum about ReaR.
Almost 100 views, no replies as of yet.
I am guessing there are not many folks that use or contribute to Devuan
that are even aware or ReaR.
Though this may be a dumb question.
Wasn't there a version of ReaR that existed pre-systemD?
If so, what are the chances it may work on Devuan?
I will post here if I hear back from Devuan folks.
Thank you
jsmeix commented at 2023-03-16 14:40:¶
Because the Devuan init system uses 'eudev'
it seems the Devuan init system is rather
different compared to traditional SysVinit
so I think an old ReaR version won't work.
I know nothing at all about 'eudev'.
I notice
https://wiki.gentoo.org/wiki/Eudev
for the very first time.
3t0n1c commented at 2023-03-16 18:45:¶
Quick update.
So I struggled with it for a bit today.
I believe I managed to get eudev to run or at least seem to run on ReaR
before the restore.
So when I execute
/etc/init.d/eudev restart
it restarts with no errors.
Looking at the processes I can see the PID for udev change, so that
tells me it actually restarts udev.
My storage still does not seem to appear in /dev/.
I don't know if anyone has any other suggestions I can try, I'd be happy
to give it another shot.
If not, I will reluctantly stop trying at this time in hopes that in the
near future Devuan will be supported.
I will leave this issue open for the time being.
Thank you all!!!
Regards
github-actions commented at 2023-05-20 02:17:¶
Stale issue message
[Export of Github issue for rear/rear.]