#1840 Issue closed: Booting RHEL 7.3 LPAR on Power stops at Starting Multi-User.

Labels: support / question, no-issue-activity

lniesz opened issue at 2018-06-22 10:26:

Relax-and-Recover (ReaR) Issue Template

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

  • ReaR version ("/usr/sbin/rear -V"):
    Relax-and-Recover 2.4-git.3020.aa7b197.master.changed / 2018-06-21

  • OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"):
    LSB Version: :core-4.1-noarch:core-4.1-ppc64
    Distributor ID: RedHatEnterpriseServer
    Description: Red Hat Enterprise Linux Server release 7.3 (Maipo)
    Release: 7.3
    Codename: Maipo

  • ReaR configuration files ("cat /etc/rear/site.conf" or "cat /etc/rear/local.conf"):

OUTPUT=ISO
BACKUP=NETFS
BACKUP_URL=nfs://9.157.244.148/QOpenSys/Niesz
BACKUP_OPTIONS="nfsvers=3,nolock"
REAR_INITRD_COMPRESSION=lzma
USE_STATIC_NETWORKING=y
AUTOEXCLUDE_MULTIPATH=n
BOOT_OVER_SAN=y

REQUIRED_PROGS=(
"${REQUIRED_PROGS[@]}"
parted
sfdisk
)

PROGS=(
"${PROGS[@]}"
partprobe
fdisk
cfdisk
mkofboot
ofpath
ybin
yabootconfig
bootlist
pseries_platform
nvram
ofpathname
bc
agetty
)

COPY_AS_IS=(
"${COPY_AS_IS[@]}"
/usr/lib/yaboot/yaboot
/usr/lib/yaboot/ofboot
/usr/lib/yaboot/yaboot.debug
/usr/lib/yaboot/addnote
)

KERNEL_CMDLINE="LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=hvc0"

# If not specified have no firmware files in the recovery system on Linux-ppc64 architecture
# because only ppc64le Linux can run in BareMetal Mode (PowerNV) where firmware files are needed
# cf. https://github.com/rear/rear/issues/1142#issuecomment-276050900
# test for "${name[*]}" because FIRMWARE_FILES is an array and the test should succeed
# when there is any non-empty array member, (not necessarily the first one):
test "${FIRMWARE_FILES[*]}" || FIRMWARE_FILES=( 'no' )

  • System architecture (x86 compatible or POWER and/or what kind of virtual machine):
    POWER

  • Are you using BIOS or UEFI or another way to boot?

  • Brief description of the issue:
    Booting from REAR image stop at Starting Multi-User.

Log:

OF stdout device is: /vdevice/vty@30000000
Preparing to boot Linux version 3.10.0-514.el7.ppc64 (mockbuild@ppc-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Wed Oct 19 11:30:41 EDT 2016
Detected machine type: 0000000000000101
Max number of cores passed to firmware: 512 (NR_CPUS = 2048)
Calling ibm,client-architecture-support... done
command line: BOOT_IMAGE=/kernel root=/dev/ram0 LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=hvc0
memory layout at init:
  memory_limit : 0000000000000000 (16 MB aligned)
  alloc_bottom : 0000000006280000
  alloc_top    : 0000000010000000
  alloc_top_hi : 0000000010000000
  rmo_top      : 0000000010000000
  ram_top      : 0000000010000000
instantiating rtas at 0x000000000ee80000... done
prom_hold_cpus: skipped
copying OF device tree...
Building dt strings...
Building dt structure...
Device tree strings 0x0000000006290000 -> 0x0000000006291688
Device tree struct  0x00000000062a0000 -> 0x00000000062d0000
Calling quiesce...
returning from prom_init
[    0.000000] Using pSeries machine description
[    0.000000] Page sizes from device-tree:
[    0.000000] base_shift=12: shift=12, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=0
[    0.000000] base_shift=12: shift=16, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=7
[    0.000000] base_shift=12: shift=24, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=56
[    0.000000] base_shift=16: shift=16, sllp=0x0110, avpnm=0x00000000, tlbiel=1, penc=1
[    0.000000] base_shift=16: shift=24, sllp=0x0110, avpnm=0x00000000, tlbiel=1, penc=8
[    0.000000] base_shift=24: shift=24, sllp=0x0100, avpnm=0x00000001, tlbiel=0, penc=0
[    0.000000] base_shift=34: shift=34, sllp=0x0120, avpnm=0x000007ff, tlbiel=0, penc=3
[    0.000000] Using 1TB segments
[    0.000000] Found initrd at 0xc000000003a00000:0xc000000006276728
[    0.000000] bootconsole [udbg0] enabled
[    0.000000] Partition configured for 4 cpus.
[    0.000000] CPU maps initialized for 4 threads per core
[    0.000000] Starting Linux PPC64 #1 SMP Wed Oct 19 11:30:41 EDT 2016
[    0.000000] -----------------------------------------------------
[    0.000000] ppc64_pft_size                = 0x1b
[    0.000000] physicalMemorySize            = 0x200000000
[    0.000000] htab_hash_mask                = 0xfffff
[    0.000000] -----------------------------------------------------
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.10.0-514.el7.ppc64 (mockbuild@ppc-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Wed Oct 19 11:30:41 EDT 2016
[    0.000000] [boot]0012 Setup Arch
[    0.000000] PPC64 nvram contains 15360 bytes
[    0.000000] Unable to enable relocation on exceptions: -67
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x00000000-0x1ffffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node  14: [mem 0x00000000-0x1ffffffff]
[    0.000000] Could not find start_pfn for node 0
[    0.000000] [boot]0015 Setup Done
[    0.000000] PERCPU: Embedded 3 pages/cpu @c000000001900000 s124952 r0 d71656 u262144
[    0.000000] Built 2 zonelists in Node order, mobility grouping on.  Total pages: 130960
[    0.000000] Policy zone: DMA
[    0.000000] Kernel command line: BOOT_IMAGE=/kernel root=/dev/ram0 LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=hvc0
[    0.000000] PID hash table entries: 4096 (order: -1, 32768 bytes)
[    0.000000] Sorting __ex_table...
[    0.000000] freeing bootmem node 14
[    0.000000] Memory: 8219328k/8388608k available (19328k kernel code, 0k reserved, 1728k data, 3657k bss, 6080k init)
[    0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=4, Nodes=256
[    0.000000] Hierarchical RCU implementation.
[    0.000000]  RCU restricting CPUs from NR_CPUS=2048 to nr_cpu_ids=4.
[    0.000000] NR_IRQS:512 nr_irqs:512 16
[    0.000002] clocksource: timebase mult[1f40000] shift[24] registered
[    0.000075] Console: colour dummy device 80x25
[    0.000092] console [hvc0] enabled, bootconsole disabled
[    0.000092] console [hvc0] enabled, bootconsole disabled
[    0.003136] allocated 2097152 bytes of page_cgroup
[    0.003153] please try 'cgroup_disable=memory' option if you don't want memory cgroups
[    0.003197] Enabling automatic NUMA balancing. Configure with numa_balancing= or the kernel.numa_balancing sysctl
[    0.003204] pid_max: default: 32768 minimum: 301
[    0.003237] Security Framework initialized
[    0.003251] SELinux:  Initializing.
[    0.003396] Dentry cache hash table entries: 1048576 (order: 7, 8388608 bytes)
[    0.005121] Inode-cache hash table entries: 524288 (order: 6, 4194304 bytes)
[    0.005988] Mount-cache hash table entries: 4096
[    0.006616] Initializing cgroup subsys memory
[    0.006713] Initializing cgroup subsys devices
[    0.006720] Initializing cgroup subsys freezer
[    0.006723] Initializing cgroup subsys net_cls
[    0.006726] Initializing cgroup subsys blkio
[    0.006729] Initializing cgroup subsys perf_event
[    0.006732] Initializing cgroup subsys hugetlb
[    0.006735] Initializing cgroup subsys pids
[    0.006737] Initializing cgroup subsys net_prio
[    0.006878] EEH: pSeries platform initialized
[    0.006890] POWER7 performance monitor hardware support registered
[    0.008694] Brought up 4 CPUs
[    0.011374] Enabling Asymmetric SMT scheduling
[    0.011976] devtmpfs: initialized
[    0.019824] EEH: devices created
[    0.020716] atomic64 test passed
[    0.021160] NET: Registered protocol family 16
[    0.021219] EEH: No capable adapters found
[    0.021247] IBM eBus Device Driver
[    0.021462] pstore: Registered nvram as persistent store backend
[    0.022669] PCI: Probing PCI hardware
[    0.022776] pseries-rng: Registering arch random hook.
[    0.050751] vgaarb: loaded
[    0.050807] SCSI subsystem initialized
[    0.050850] usbcore: registered new interface driver usbfs
[    0.050859] usbcore: registered new interface driver hub
[    0.050883] usbcore: registered new device driver usb
[    0.051063] NetLabel: Initializing
[    0.051067] NetLabel:  domain hash size = 128
[    0.051069] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.051081] NetLabel:  unlabeled traffic allowed by default
[    0.051125] Switched to clocksource timebase
[    0.059325] NET: Registered protocol family 2
[    0.059510] TCP established hash table entries: 65536 (order: 3, 524288 bytes)
[    0.059634] TCP bind hash table entries: 65536 (order: 4, 1048576 bytes)
[    0.059846] TCP: Hash tables configured (established 65536 bind 65536)
[    0.059861] TCP: reno registered
[    0.059866] UDP hash table entries: 4096 (order: 1, 131072 bytes)
[    0.059889] UDP-Lite hash table entries: 4096 (order: 1, 131072 bytes)
[    0.059945] NET: Registered protocol family 1
[    0.059991] Unpacking initramfs...
[    9.510607] Freeing initrd memory: 41408K (c000000003a00000 - c000000006270000)
[    9.512176] IOMMU table initialized, virtual merging enabled
[    9.561357] futex hash table entries: 1024 (order: 1, 131072 bytes)
[    9.561398] Initialise system trusted keyring
[    9.561422] audit: initializing netlink socket (disabled)
[    9.561436] type=2000 audit(1529662537.560:1): initialized
[    9.629777] HugeTLB registered 16 MB page size, pre-allocated 0 pages
[    9.629787] HugeTLB registered 16 GB page size, pre-allocated 0 pages
[    9.630991] zpool: loaded
[    9.631001] zbud: loaded
[    9.631218] VFS: Disk quotas dquot_6.5.2
[    9.631260] Dquot-cache hash table entries: 8192 (order 0, 65536 bytes)
[    9.631467] msgmni has been set to 16264
[    9.631516] Key type big_key registered
[    9.632222] NET: Registered protocol family 38
[    9.632231] Key type asymmetric registered
[    9.632235] Asymmetric key parser 'x509' registered
[    9.632264] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
[    9.632316] io scheduler noop registered
[    9.632322] io scheduler deadline registered (default)
[    9.632376] io scheduler cfq registered
[    9.632435] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    9.632811] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    9.633140] Linux agpgart interface v0.103
[    9.633173] crash memory driver: version 1.1
[    9.633212] rdac: device handler registered
[    9.633280] hp_sw: device handler registered
[    9.633287] emc: device handler registered
[    9.633290] alua: device handler registered
[    9.633321] libphy: Fixed MDIO Bus: probed
[    9.633370] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    9.633381] ehci-pci: EHCI PCI platform driver
[    9.633389] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    9.633399] ohci-pci: OHCI PCI platform driver
[    9.633407] uhci_hcd: USB Universal Host Controller Interface driver
[    9.633447] usbcore: registered new interface driver usbserial
[    9.633454] usbcore: registered new interface driver usbserial_generic
[    9.633460] usbserial: USB Serial support registered for generic
[    9.633521] mousedev: PS/2 mouse device common for all mice
[    9.633691] rtc-generic rtc-generic: rtc core: registered rtc-generic as rtc0
[    9.633717] cpuidle: using governor menu
[    9.661147] nx_compress_pseries ibm,compression-v1: nx842_OF_upd: max_sync_size new:4096 old:0
[    9.661163] nx_compress_pseries ibm,compression-v1: nx842_OF_upd: max_sync_sg new:510 old:0
[    9.661167] nx_compress_pseries ibm,compression-v1: nx842_OF_upd: max_sg_len new:4080 old:0
[    9.691188] alg: No test for 842 (842-nx)
[    9.691234] hidraw: raw HID events driver (C) Jiri Kosina
[    9.691323] usbcore: registered new interface driver usbhid
[    9.691326] usbhid: USB HID core driver
[    9.691335] drop_monitor: Initializing network drop monitor service
[    9.691395] TCP: cubic registered
[    9.691400] Initializing XFRM netlink socket
[    9.691494] NET: Registered protocol family 10
[    9.691703] NET: Registered protocol family 17
[    9.691881] Loading compiled-in X.509 certificates
[    9.692309] Loaded X.509 cert 'Red Hat Enterprise Linux Driver Update Program (key 3): bf57f3e87362bc7229d9f465321773dfd1f77a80'
[    9.692717] Loaded X.509 cert 'Red Hat Enterprise Linux kpatch signing key: 4d38fd864ebe18c5f0b72e3852e2014c3a676fc8'
[    9.693121] Loaded X.509 cert 'Red Hat Enterprise Linux kernel signing key: 3cfc88215d1f3bc9e3592dad239c05c31d3cf80f'
[    9.693162] registered taskstats version 1
[    9.693376] rtc-generic rtc-generic: setting system clock to 2018-06-22 10:15:38 UTC (1529662538)
[    9.694974] Freeing unused kernel memory: 6080K (c000000000cf0000 - c0000000012e0000)
[    9.700055] ip_tables: (C) 2000-2006 Netfilter Core Team
[    9.700078] systemd[1]: Inserted module 'ip_tables'
[    9.704486] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
[    9.704582] systemd[1]: Detected architecture ppc64.
[    9.704590] systemd[1]: Running with unpopulated /etc.

Welcome to Red Hat Enterprise Linux Server 7.3 (Maipo)!

[    9.705367] systemd[1]: Set hostname to .
[    9.705401] systemd[1]: Initializing machine ID from random generator.
[    9.708094] systemd[1]: Failed to populate /etc with preset unit settings, ignoring: Invalid argument
[  OK  ] Reached target System Initialization.
[    9.713194] systemd[1]: Reached target System Initialization.
[    9.713206] systemd[1]: Starting System Initialization.
[  OK  ] Reached target Multi-User.
[    9.713910] systemd[1]: Reached target Multi-User.
[    9.713953] systemd[1]: Startup finished in 9.696s (kernel) + 16ms (userspace) = 9.712s.
[    9.713966] systemd[1]: Starting Multi-User.
  • Work-around, if any:

schabrolles commented at 2018-06-22 10:46:

@lniesz

I don't have any issues when running ReaR on my PowerLPAR with rhel7.4 or rhel7.5.
I don't think rhel7.3 is different and could explain why you can't boot on the ReaR rescue media ISO.

could you pleas make another try with only those parameters:

OUTPUT=ISO
BACKUP=NETFS
BACKUP_URL=nfs://9.157.244.148/QOpenSys/Niesz
BACKUP_OPTIONS="nfsvers=3,nolock"
REAR_INITRD_COMPRESSION=lzma
USE_STATIC_NETWORKING=y
AUTOEXCLUDE_MULTIPATH=n
BOOT_OVER_SAN=y

lniesz commented at 2018-06-22 11:57:

Hi,

Thanks for the quick reply.

I've changed the local.conf file as you suggested, but got the same result. Boot still stops at starting multi-user.

After booting a menu comes up:

Relax-and-Recover

I've checked the parameters, which has really changed as a result of the new config file:

setparams 'Relax-and-Recover'

    linux   /kernel root=/dev/ram0  selinux=0                              
    initrd  /initrd.xz

If I accept this then boot stops as I wrote above.

schabrolles commented at 2018-06-22 11:59:

What kind of virtualization are you using:

  • Power BareMetal (non virtualized)
  • PowerKVM guest
  • PowerVM LPAR

lniesz commented at 2018-06-22 12:03:

Backup is from a PowerVM LPAR and target would be the same.
I'm trying the restore on the same machine.

lniesz commented at 2018-06-22 12:30:

I've found the following differences in multi-user.target on source LPAR, but I do not know if this can cause the problem:

[root@mlnx rear]# cat /usr/lib/systemd/system/multi-user.target
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Multi-User System
Documentation=man:systemd.special(7)
Requires=basic.target
Conflicts=rescue.service rescue.target
After=basic.target rescue.service rescue.target
AllowIsolate=yes

[root@mlnx rear]# cat /usr/share/rear/skel/default/usr/lib/systemd/system/multi-user.target
#  This file is part of systemd.
#
# See systemd.special(7) for details

[Unit]
Description=Multi-User
Requires=sysinit.target
AllowIsolate=yes

[Install]
Alias=default.target

lniesz commented at 2018-06-22 13:00:

I'm not sure but is that normal, that the following directory is empty:
/tmp/rear.AIjNOOTddKIIANl/rootfs/etc/systemd/system

I think this is where the ISO image is populated from.

Also the boot log tells:
[ 9.708094] systemd[1]: Failed to populate /etc with preset unit settings, ignoring: Invalid argument

lniesz commented at 2018-06-26 09:33:

@schabrolles Could you check if that is normal, that /etc is not populated, and "target" files for systemd are different than on source LPAR?

schabrolles commented at 2018-06-26 12:49:

@lniesz,

I'm currently not in the office and can't test right now. I should be able to try end of this week.

gdha commented at 2018-10-02 16:25:

@schabrolles I made you the owner of this issue as you are the man ;-) with the appropriate LPAR knowledge

github-actions commented at 2020-06-30 01:33:

Stale issue message

[Export of Github issue for rear/rear.]