#2137 Issue closed: Support IBM Z architecture "s390x" (a.k.a. "z Systems" formerly "System/390")

Labels: enhancement, needs sponsorship, fixed / solved / done, sponsored

jsmeix opened issue at 2019-05-07 07:34:

Current ReaR does not support IBM Z architecture
regardless that some Z architecture specific things
like IBM S390 DASD drive dasd appear in some scripts
but that is only a very first (incomplete) step, cf.
https://github.com/rear/rear/issues/783
and
https://github.com/rear/rear/commit/f719fee860f656859ec12838b0c5b36d32532d6a

In general see
http://relax-and-recover.org/documentation/release-notes-2-4

Supported Architectures
...
ReaR-2.4 does not support:
 * s390x type of processors

and the section "Non PC compatible architectures" at
https://en.opensuse.org/SDB:Disaster_Recovery

See also issues
like https://github.com/rear/rear/issues/1261
and https://github.com/rear/rear/issues/781

What is different on IBM Z s390x architecture compared to x86
is basically everything that is crucial for ReaR:

  • storage
  • bootloader
  • how to make and boot the ReaR recovery system
  • and likely many little other things (system console, network devices, udev rules, ...)

schabrolles commented at 2019-05-07 07:47:

@jsmeix, I cannot really help here as I don't know how IBM Z is working.

jsmeix commented at 2019-05-07 07:49:

@rmetrich
I also assigned you here because of the issues
https://github.com/rear/rear/issues/1261 and
https://github.com/rear/rear/issues/781 from Red Hat.

jsmeix commented at 2019-05-07 07:50:

@schabrolles
I also assigned you here because you are from IBM
so you may perhaps know someone else from IBM
who can actually help here.

Don't worry, I also don't know how IBM Z is working, cf.
https://github.com/rear/rear/commit/f719fee860f656859ec12838b0c5b36d32532d6a#commitcomment-33438908

jsmeix commented at 2019-05-07 08:38:

For comparison what ReaR scripts are not run during "rear mkbackup"
on s390x compared to x86 with the on x86 usually used
OUTPUT=ISO in /etc/rear/local.conf
(the diff of usr/sbin/rear -s mkbackup on s390x compared to x86):

+Source conf/Linux-i386.conf
+Source prep/ISO/Linux-i386/330_find_isolinux.sh
+Source output/ISO/Linux-i386/250_populate_efibootimg.sh
+Source output/ISO/Linux-i386/260_EFISTUB_populate.sh
+Source output/ISO/Linux-i386/300_create_isolinux.sh
+Source output/ISO/Linux-i386/700_create_efibootimg.sh
+Source output/ISO/Linux-i386/800_create_isofs.sh
+Source output/ISO/Linux-i386/810_prepare_multiple_iso.sh
+Source output/ISO/Linux-i386/820_create_iso_image.sh
+Source output/ISO/Linux-i386/830_create_iso_image_EFISTUB.sh
+Source output/ISO/Linux-i386/850_check_for_errors.sh

i.e. those scripts are run on x86 but not on s390x
but I guess OUTPUT=ISO is useless anyway on s390x.

jsmeix commented at 2019-05-07 09:11:

I got a ready-to-use s390x system where I can login via ssh.

On that system I use our latest GitHub ReaR master code as follows:
I "git clone" our current ReaR upstream GitHub master code
into a separated directory and then configure and run ReaR
from within that directory like:

# git clone https://github.com/rear/rear.git

# mv rear rear.github.master

# cd rear.github.master

# vi etc/rear/local.conf

# usr/sbin/rear -D mkbackup

(note the relative paths "etc/rear/" and "usr/sbin/")

To make "rear mkbackup" running there to its end without an error
(but that does not at all mean what it did can be used for "rear recover")
I needed some adaptions on s390x, see the three 'diff' output below.

On my s390x system there is SLES 12 SP4
with its default btrfs structure installed.

I have

# lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT
NAME          KNAME       PKNAME     TRAN TYPE FSTYPE  SIZE MOUNTPOINT
/dev/dasda    /dev/dasda                  disk         6.9G 
|-/dev/dasda1 /dev/dasda1 /dev/dasda      part ext2    200M /boot/zipl
|-/dev/dasda2 /dev/dasda2 /dev/dasda      part swap    730M [SWAP]
`-/dev/dasda3 /dev/dasda3 /dev/dasda      part btrfs     6G /

# grep -v ^# etc/rear/local.conf 
OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://10.160.67.243/nfs
REQUIRED_PROGS=( "${REQUIRED_PROGS[@]}" snapper chattr lsattr )
COPY_AS_IS=( "${COPY_AS_IS[@]}" /usr/lib/snapper/installation-helper /etc/snapper/config-templates/default )
BACKUP_PROG_INCLUDE=( /usr/local /tmp /var/lib/mailman /var/lib/libvirt/images /var/lib/pgsql /var/tmp /var/spool /var/lib/machines /opt /var/lib/mysql /var/lib/named /home /var/lib/mariadb /var/log /srv /boot/grub2/s390x-emu /var/opt /var/cache )
POST_RECOVERY_SCRIPT=( 'if snapper --no-dbus -r $TARGET_FS_ROOT get-config | grep -q "^QGROUP.*[0-9]/[0-9]" ; then snapper --no-dbus -r $TARGET_FS_ROOT set-config QGROUP= ; snapper --no-dbus -r $TARGET_FS_ROOT setup-quota && echo snapper setup-quota done || echo snapper setup-quota failed ; else echo snapper setup-quota not used ; fi' )
SSH_ROOT_PASSWORD="rear"
USE_DHCLIENT="yes"
KEEP_BUILD_DIR="yes"
FIRMWARE_FILES=( 'no' )
PROGRESS_MODE="plain"
PROGRESS_WAIT_SECONDS="5"

cf. https://github.com/rear/rear/blob/master/usr/share/rear/conf/examples/SLE12-SP2-btrfs-example.conf

My terminal output of my "rear mkbackup" run:

# usr/sbin/rear -D mkbackup
Relax-and-Recover 2.4 / Git
Running rear mkbackup (PID 23459)
Using log file: /root/rear.github.master/var/log/rear/rear-s390vsl202.log
Using backup archive '/tmp/rear.ZrVnTvcYrRz4l5u/outputfs/s390vsl202/backup.tar.gz'
Using autodetected kernel '/boot/image-4.12.14-95.13-default' as kernel in the recovery system
Creating disk layout
Using sysconfig bootloader 'grub2'
Verifying that the entries in /root/rear.github.master/var/lib/rear/layout/disklayout.conf are correct ...
Creating root filesystem layout
Handling network interface 'eth0'
eth0 is a physical device
Handled network interface 'eth0'
Handling network interface 'vlan209'
vlan209 is a vlan
vlan209 has eth0 as parent
eth0 already handled...
Handled network interface 'vlan209'
Copying logfile /root/rear.github.master/var/log/rear/rear-s390vsl202.log into initramfs as '/tmp/rear-s390vsl202-partial-2019-05-07T04:48:23-04:00.log'
Copying files and directories
Copying binaries and libraries
Copying all kernel modules in /lib/modules/4.12.14-95.13-default (MODULES contains 'all_modules')
Omit copying files in /lib*/firmware/ (FIRMWARE_FILES='no')
Skip copying broken symlink './etc/mtab' target '/proc/33152/mounts' on /proc/ /sys/ /dev/ or /run/
Testing that the recovery system in /tmp/rear.ZrVnTvcYrRz4l5u/rootfs contains a usable system
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (81290193 bytes) in 14 seconds
Copying resulting files to nfs location
Saving /root/rear.github.master/var/log/rear/rear-s390vsl202.log as rear-s390vsl202.log to nfs location
Copying result files '/tmp/rear.ZrVnTvcYrRz4l5u/tmp/VERSION /tmp/rear.ZrVnTvcYrRz4l5u/tmp/README /tmp/rear.ZrVnTvcYrRz4l5u/tmp/rear-s390vsl202.log' to /tmp/rear.ZrVnTvcYrRz4l5u/outputfs/s390vsl202 at nfs location
Creating tar archive '/tmp/rear.ZrVnTvcYrRz4l5u/outputfs/s390vsl202/backup.tar.gz'
Preparing archive operation
Archived 101 MiB [avg 14900 KiB/sec] 
...
Archived 2093 MiB [avg 5989 KiB/sec] 
OK
Archived 2093 MiB in 363 seconds [avg 5906 KiB/sec]
Exiting rear mkbackup (PID 23459) and its descendant processes ...
Running exit tasks
You should also rm -Rf /tmp/rear.ZrVnTvcYrRz4l5u

That results on my NFS server:

# ls -lhtr /nfs/s390vsl202/
total 2.1G
-rw-rw-rw- 1 nobody nobody  267 May  7 10:49 VERSION
-rw-rw-rw- 1 nobody nobody  202 May  7 10:49 README
-rw-rw-rw- 1 nobody nobody 3.1M May  7 10:49 rear-s390vsl202.log
-rw-rw-rw- 1 nobody nobody 2.1G May  7 10:55 backup.tar.gz
-rw-rw-rw- 1 nobody nobody  12M May  7 10:55 backup.log

According to https://github.com/rear/rear/issues/2137#issuecomment-489989425
what is missing is the ISO image, for comparison what one gets on x86

# ls -lhtr /nfs/linux-52m8
total 1.4G
-rw-rw-rw- 1 nobody nobody  85M May  7 11:01 rear-linux-52m8.iso
-rw-rw-rw- 1 nobody nobody  267 May  7 11:01 VERSION
-rw-rw-rw- 1 nobody nobody  202 May  7 11:01 README
-rw-rw-rw- 1 nobody nobody 4.6M May  7 11:01 rear-linux-52m8.log
-rw-rw-rw- 1 nobody nobody 1.3G May  7 11:04 backup.tar.gz
-rw-rw-rw- 1 nobody nobody  11M May  7 11:04 backup.log

My adaptions to make "rear mkbackup" run on s390x:

--- usr/share/rear/layout/prepare/GNU/Linux/100_include_partition_code.sh.original      2019-04-29 11:42:53.039143520 -0400
+++ usr/share/rear/layout/prepare/GNU/Linux/100_include_partition_code.sh       2019-04-29 12:36:16.459143520 -0400
@@ -132,7 +132,7 @@
             ### GPT disks need 33 LBA blocks at the end of the disk
             # For the SUSE specific gpt_sync_mbr partitioning scheme
             # see https://github.com/rear/rear/issues/544
-            if [[ "$label" == "gpt" || "$label" == "gpt_sync_mbr" ]] ; then
+            if [[ "$label" == "gpt" || "$label" == "gpt_sync_mbr" || "$label" == "dasd" ]] ; then
                 device_size=$( mathlib_calculate "$device_size - 33*$block_size" )
                 # Only if resizing all partitions is explicity wanted
                 # resizing of arbitrary partitions may also happen via the code below

--- usr/share/rear/layout/save/GNU/Linux/200_partition_layout.sh.original       2019-04-29 11:42:53.039143520 -0400
+++ usr/share/rear/layout/save/GNU/Linux/200_partition_layout.sh        2019-04-29 12:21:13.289143520 -0400
@@ -140,7 +140,7 @@
     ### Find partition name for GPT disks.
     # For the SUSE specific gpt_sync_mbr partitioning scheme
     # see https://github.com/rear/rear/issues/544
-    if [[ "$disk_label" = "gpt" || "$disk_label" == "gpt_sync_mbr" ]] ; then
+    if [[ "$disk_label" = "gpt" || "$disk_label" == "gpt_sync_mbr" || "$disk_label" == "dasd" ]] ; then
         if [[ "$FEATURE_PARTED_MACHINEREADABLE" ]] ; then
             while read partition_nr size start junk ; do
                 # In case of GPT the 'type' field contains actually the GPT partition name.

--- usr/share/rear/layout/save/default/950_verify_disklayout_file.sh.original   2019-04-29 11:42:53.049143520 -0400
+++ usr/share/rear/layout/save/default/950_verify_disklayout_file.sh    2019-04-29 12:25:16.329143520 -0400
@@ -81,7 +81,7 @@
     # because that indicates partitions of another form than /dev/sdX1 /dev/sdX2 /dev/sdX3 are used:
     if test $highest_used_part_num -gt 0 ; then
         case $parted_mklabel in
-            (gpt)
+            (gpt|dasd)
                 # For the GPT partitioning scheme the partitions must have consecutive numbers 1 2 3 ..
                 non_consecutive_part_found=""
                 unused_part_nums=()

rmetrich commented at 2019-05-07 09:14:

Hello,
We (Red Hat) were supporting S390x with rear-1.17.2 but apparently the code changed with rear-2.0 and support has been dropped.
I would suggest @pcahyna talks about opportunity to support recent rear to his management.

mutable-dan commented at 2019-05-07 17:51:

I see in your output:

My terminal output of my "rear mkbackup" run

Creating disk layout
Using sysconfig bootloader 'grub2'
Verifying that the entries in /root/rear.github.master/var/lib/rear/layout/disklayout.conf are correct ...
Creating root filesystem layout

Does suse use grub on s390 for the bootloader instead of zIPL?

mutable-dan commented at 2019-05-07 18:18:

Interesting... I opened a zlinux instance running suse 12

the mbr of the dasd boot device refs zIPL

dd if=/dev/dasda bs=4096 count=4|strings
4+0 records in
4+0 records out
16384 bytes (16 kB, 16 KiB) copied, 0.000983469 s, 16.7 MB/s
zIPL

yet the boot dir lists grub2 and zIPL. rhel on the other hand has no references to grub and the guess_bootloader fails. This has been the area I was working on

jsmeix commented at 2019-05-08 09:38:

I learned right now how booting IBM Z works in general on SLES12 and SLES15
(the following describes it as far as I understand but there could be errors
because I am not an IBM Z expert):

The idea behind is to get during booting as soon as possible
to a common state where things behave same for the user
on all architectures where SLES is supported.

That common state is GRUB2.
The user should see GRUB2 on all architectures where SLES is supported.
I think this is in particular useful to boot a snapper btrfs snapshot via
one same user interface (i.e. the GRUB2 boot menue).

But GRUB2 has no support to access files on the disk on IBM Z.
There was an attempt to add that support into GRUB2
but that became too big and too complicated and
a nightmare to maintain it for all future times.

On the other hand the kernel has "ready-to-use" support
to access files on the disk on IBM Z and it turned out
that it is relatively easy to run GRUB2 like a normal
user process by the kernel.

So on SLES12 and SLES15 booting IBM Z basically works this way:

Initially zipl loads a kernel and
that kernel runs GRUB2 and
GRUB2 loads the actual kernel and does a kexec.

To re-install that during "rear recover" all what is basically needed
is to call grub2-install because that is appropriately enhanced
to do "the right thing" on IBM Z (i.e. it also does the zipl setup
for the initial zipl kernel load).

This could mean that in the end during "rear recover"
on SLES12/15 basically the same as in
usr/share/rear/finalize/Linux-i386/660_install_grub2.sh
needs to be done on IBM Z.

What is a different and completely separated thing is
how the ReaR recovery system is booted on IBM Z.

On x86 architecture the ReaR recovery system is usually
a bootable ISO image where the SYSLINUX/ISOLINUX bootloader
is used.

I think for booting the ReaR recovery system on IBM Z
the simplest way that works should be sufficient because
I think on IBM Z the system is not booted accidentally
by an unexperienced normal user so that I think
there is no need for a sophisticated boot menue
when booting the ReaR recovery system on IBM Z.

Accordingly I think plain direct kernel and initrd loading via zipl
should be sufficient for booting the ReaR recovery system on IBM Z
(at least this should be initially sufficient for ReaR support on IBM Z).

jsmeix commented at 2019-05-08 13:36:

I found
https://www.ibm.com/support/knowledgecenter/en/linuxonibm/com.ibm.linux.z.lhdd/lhdd_c_ipl_vs_boot.html
that describes

On Z ...
... the main steps of booting SUSE Linux Enterprise Server 12 SP4

mutable-dan commented at 2019-05-09 15:53:

I found
https://www.ibm.com/support/knowledgecenter/en/linuxonibm/com.ibm.linux.z.lhdd/lhdd_c_ipl_vs_boot.html
that describes

On Z ...
... the main steps of booting SUSE Linux Enterprise Server 12 SP4

It looks like suse differs from rhel and ubuntu.
https://www.ibm.com/support/knowledgecenter/en/linuxonibm/com.ibm.linux.z.lkdd/lkdd_c_ipl_vs_boot.html

The aux kernel image with grub allows rear to complete the boot verification process in suse. On rhel and ubuntu, it will fail with an unknown boot loader. The push req I submitted gets rhel past the boot verification process on rhel 7.2

jsmeix commented at 2019-05-10 08:34:

@rmetrich @pcahyna
regardless of what Red Hat management may later decide
could you have a look here, in particular also have a look at
https://github.com/rear/rear/pull/2142

Perhaps by plain looking at the code changes you may already see
things that are not right or where @mutable-dan could need help
to avoid that he does much coding work mostly on his own but later
we may tell him about a lot of things that he should have done differently.

Many thanks in advance for your help!

jsmeix commented at 2019-12-16 14:07:

With https://github.com/rear/rear/pull/2142 merged we have now in ReaR
initial preliminary first basic support for IBM Z architecture "s390x"
so that interested users can try out how far things work for them, cf.
https://github.com/rear/rear/commit/2e3ff0fbacaf15055ebf4fcfab75f29ee58fe838

jsmeix commented at 2019-12-16 14:13:

@rmetrich @pcahyna @tcerna
could you test (or do you know one at Red Hat who could test)
how far our current preliminary first basic support for IBM Z
actually works in your particular IBM Z environments at Red Hat
and provide feedback - preferably plus needed fixes, adaptions,
and enhancements?

[Export of Github issue for rear/rear.]