#1653 Issue closed: xfs_admin: xfs_db/xfs_growfs command not found (on SLES 12 SP2 ppc64le)¶
Labels: support / question, fixed / solved / done,
external tool, not ReaR / invalid
sebaobi opened issue at 2017-12-20 13:40:¶
-
rear version (/usr/sbin/rear -V): Relax-and-Recover 2.2-git.0.3ad23a0.unknown / 2017-12-19
-
OS version (cat /etc/rear/os.conf or lsb_release -a): SUSE Linux Enterprise Server 12 (ppc64le)
-
rear configuration files (cat /etc/rear/site.conf or cat /etc/rear/local.conf):
OS_VENDOR=SUSE_LINUX
OS_VERSION=12
BACKUP=NSR
OUTPUT=ISO
BACKUP_URL=nfs://192.168.251.12/data/col1/rear
BACKUP_OPTIONS="hard,intr,nolock,nfsvers=3,tcp,timeo=1200,rsize=1048600,wsize=1048600,bg"
NSR_SERVER=bckserver.org
POOLNAME=dd_os
NSR_RETENTION_TIME=Month
USE_CFG2HTML=y
AUTOEXCLUDE_MULTIPATH=n
BOOT_OVER_SAN=y -
Are you using legacy BIOS or UEFI boot? BIOS
-
Brief description of the issue:
We are trying to make initial backup a SLES 12 SP2 PPC running on Power using Networker. During initial backup we get error with xfs:
2017-12-20 15:15:02.096993678 Including
layout/save/GNU/Linux/220_lvm_layout.sh
2017-12-20 15:15:02.098098179 Saving LVM layout.
2017-12-20 15:15:02.394626014 Including
layout/save/GNU/Linux/230_filesystem_layout.sh
2017-12-20 15:15:02.395608007 Begin saving filesystem layout
2017-12-20 15:15:02.397477953 Saving filesystem layout (using the
findmnt command).
/usr/sbin/xfs_admin: line 37: xfs_db: command not found
/usr/sbin/xfs_admin: line 37: xfs_db: command not found
/usr/sbin/xfs_info: line 25: xfs_growfs: command not found
- Work-around, if any:
jsmeix commented at 2017-12-20 13:44:¶
On my SLES12 x86_64 system:
# type -a xfs_db xfs_db is /usr/sbin/xfs_db # rpm -qf /usr/sbin/xfs_db xfsprogs-4.3.0-8.8.x86_64 # type -a xfs_growfs xfs_growfs is /usr/sbin/xfs_growfs # rpm -qf /usr/sbin/xfs_growfs xfsprogs-4.3.0-8.8.x86_64
All those programs are also available in the
xfsprogs RPM package for SLE12 ppc64le
both for SLE-12-GA (a.k.a. SP0)
# rpm -qlvp xfsprogs-3.2.1-1.3.ppc64le.rpm | egrep 'xfs_db|xfs_growfs|xfs_admin|xfs_info' | grep bin | tr -s ' ' -rwxr-xr-x 1 root root 1380 Sep 21 2014 /usr/sbin/xfs_admin -rwxr-xr-x 1 root root 626336 Sep 21 2014 /usr/sbin/xfs_db -rwxr-xr-x 1 root root 362912 Sep 21 2014 /usr/sbin/xfs_growfs -rwxr-xr-x 1 root root 472 Sep 21 2014 /usr/sbin/xfs_info
and for SLE-12-SP2:
$ rpm -qlvp xfsprogs-4.3.0-8.8.ppc64le.rpm | egrep 'xfs_db|xfs_growfs|xfs_admin|xfs_info' | grep bin | tr -s ' ' -rwxr-xr-x 1 root root 1380 Oct 25 2016 /usr/sbin/xfs_admin -rwxr-xr-x 1 root root 733496 Oct 25 2016 /usr/sbin/xfs_db -rwxr-xr-x 1 root root 404448 Oct 25 2016 /usr/sbin/xfs_growfs -rwxr-xr-x 1 root root 472 Oct 25 2016 /usr/sbin/xfs_info
sebaobi commented at 2017-12-20 13:56:¶
Sure, and ofc i have thist packages:
test:/var/log/rear # type -a xfs_db xfs_db is /usr/sbin/xfs_db
test:/var/log/rear # rpm -qf /usr/sbin/xfs_db xfsprogs-4.3.0-12.1.ppc64le
test:/var/log/rear # type -a xfs_growfs xfs_growfs is /usr/sbin/xfs_growfs
test:/var/log/rear # rpm -qf /usr/sbin/xfs_growfs xfsprogs-4.3.0-12.1.ppc64le
jsmeix commented at 2017-12-20 14:04:¶
When xfs_admin errors out with
"xfs_db/xfs_growfs command not found"
the root cause is (with probability one) not inside ReaR
but in xfs_admin or elsewhere that causes the failure of xfs_admin.
At least for now I set the "not ReaR" and "external tool" labels
until there is evidence that the root cause is inside ReaR.
sebaobi commented at 2017-12-20 14:04:¶
I think this is linked with this:
test:/var/log/rear # vim +37 /usr/sbin/xfs_admin
eval xfs_db -x -p xfs_admin $DB_OPTS $1test:/var/log/rear # vim +25 /usr/sbin/xfs_info
1) xfs_growfs -p xfs_info -n $OPTS "$1"
jsmeix commented at 2017-12-20 14:13:¶
It seems your /usr/sbin/xfs_admin is different to mine
regardless that we both have xfsprogs-4.3.0:
$ cat -n /usr/sbin/xfs_admin | egrep -1 'xfs_db|xfs_growfs'
22 U) DB_OPTS=$DB_OPTS" -c 'uuid "$OPTARG"'";;
23 V) xfs_db -p xfs_admin -V
24 status=$?
--
36 then
37 eval xfs_db -x -p xfs_admin $DB_OPTS $1
38 status=$?
jsmeix commented at 2017-12-20 14:18:¶
Forget my last comment - I confused things.
Actually I have the same as you:
# cat -n /usr/sbin/xfs_admin | grep -1 'xfs_db'
22 U) DB_OPTS=$DB_OPTS" -c 'uuid "$OPTARG"'";;
23 V) xfs_db -p xfs_admin -V
24 status=$?
--
36 then
37 eval xfs_db -x -p xfs_admin $DB_OPTS $1
38 status=$?
# cat -n /usr/sbin/xfs_info | grep -1 'xfs_growfs'
12 t) OPTS="-t $OPTARG" ;;
13 V) xfs_growfs -p xfs_info -V
14 status=$?
--
24 case $# in
25 1) xfs_growfs -p xfs_info -n $OPTS "$1"
26 status=$?
sebaobi commented at 2017-12-21 09:37:¶
Ok, so what can we do with that ?,
sebaobi commented at 2017-12-21 09:53:¶
I just did backup to nfs (rear -v mkbackup) and it creates all without problem, additionally I checked thist with strace and rear using the same script 230_filesystem_layout.sh, this is so strange.
jsmeix commented at 2017-12-21 10:19:¶
@sebaobi
to only see which scripts e.g. 'rear mkbackup' runs
(depending on the particular BACKUP=... method
and the particular architectue like ppc64le) call it
in simulation mode '-s':
rear -s mkbackup
To see what each script during 'rear mkbackup' actually does
run it in debugscripts mode '-D' (this is no simulation)
rear -D mkbackup
and inspect the log file.
Cf. "Debugging issues with Relax-and-Recover" at
https://en.opensuse.org/SDB:Disaster_Recovery
In general I cannot help with issues with third-party backup tools
like BACKUP=NSR (EMC Networker; Legato) because I neither
use third-party backup tools nor do I have such programs so that
I cannot reproduce anything with third-party backup tools.
I cannot imagine how BACKUP=NSR versus
BACKUP=NETFS could make any difference
whether or not xfs_admin or xfs_info fail with
xfs_db or xfs_growfs command not found.
@gozora
because this issue is somehow related to XFS
could you have a look - perhaps you even have
any idea what might go on here?
sebaobi commented at 2017-12-21 10:40:¶
Here you are some more logs, tests with NSR:
2017-12-21 12:25:51.193822426 Saving filesystem layout (using the findmnt command).
++ read_filesystems_command='/usr/bin/findmnt -mnrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs | sort -t '\'' '\'' -k 1,1 -u'
++ echo '# Filesystems (only ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs are supported).'
++ echo '# Format: fs <device> <mountpoint> <fstype> [uuid=<uuid>] [label=<label>] [<attributes>]'
++ read device mountpoint fstype options junk
+++ eval /usr/bin/findmnt -mnrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs '|' sort -t \' \' -k 1,1 -u
++++ /usr/bin/findmnt -mnrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs
++++ sort -t ' ' -k 1,1 -u
++ test -z /dev/mapper/HANADATA-data -o -z /hana/data -o -z xfs
++ '[' dev/mapper/HANADATA-data = /dev/mapper/HANADATA-data ']'
++ '[' '!' -b /dev/mapper/HANADATA-data ']'
++ '[' xfs = iso9660 ']'
++ [[ /dev/mapper/HANADATA-data == /dev/disk/by-uuid* ]]
+++ get_device_mapping /dev/mapper/HANADATA-data
+++ [[ ! -s /var/lib/rear/recovery/diskbyid_mappings ]]
+++ local name=HANADATA-data
++++ grep -w '^HANADATA-data' /var/lib/rear/recovery/diskbyid_mappings
++++ awk '{print $2}'
+++ local disk_name=
+++ [[ -z '' ]]
+++ echo /dev/mapper/HANADATA-data
++ device=/dev/mapper/HANADATA-data
+++ get_device_name /dev/mapper/HANADATA-data
+++ local name=mapper/HANADATA-data
+++ name=mapper/HANADATA-data
+++ contains_visible_char mapper/HANADATA-data
++++ tr -d -c '[:graph:]'
+++ test mapper/HANADATA-data
+++ local device dev_number mapper_number
+++ [[ -d /sys/block/mapper/HANADATA-data ]]
+++ [[ mapper/HANADATA-data =~ ^mapper/ ]]
+++ echo /dev/mapper/HANADATA-data
+++ return 0
++ device=/dev/mapper/HANADATA-data
++ echo -n 'fs /dev/mapper/HANADATA-data /hana/data xfs'
++ case "$fstype" in
+++ xfs_admin -u /dev/mapper/HANADATA-data
+++ cut -d= -f 2
+++ tr -d ' '
/usr/sbin/xfs_admin: line 37: xfs_db: command not found
++ uuid=
+++ xfs_admin -l /dev/mapper/HANADATA-data
+++ cut '-d"' -f 2
/usr/sbin/xfs_admin: line 37: xfs_db: command not found
++ label=
++ echo -n ' uuid= label= '
++ xfs_info /hana/data
+++ basename /dev/mapper/HANADATA-data.xfs
/usr/sbin/xfs_info: line 25: xfs_growfs: command not found
++ StopIfError 'Failed to save XFS options of /dev/mapper/HANADATA-data'
++ (( 127 != 0 ))
++ Error 'Failed to save XFS options of /dev/mapper/HANADATA-data'
++ LogPrintError 'ERROR: Failed to save XFS options of /dev/mapper/HANADATA-data'
++ Log 'ERROR: Failed to save XFS options of /dev/mapper/HANADATA-data'
+++ date '+%Y-%m-%d %H:%M:%S.%N '
++ local 'timestamp=2017-12-21 12:25:51.214664410 '
++ test 1 -gt 0
++ echo '2017-12-21 12:25:51.214664410 ERROR: Failed to save XFS options of /dev/mapper/HANADATA-data'
2017-12-21 12:25:51.214664410 ERROR: Failed to save XFS options of /dev/mapper/HANADATA-data
++ PrintError 'ERROR: Failed to save XFS options of /dev/mapper/HANADATA-data'
++ echo -e 'ERROR: Failed to save XFS options of /dev/mapper/HANADATA-data'
++ LogToSyslog 'ERROR: Failed to save XFS options of /dev/mapper/HANADATA-data'
++ logger -t rear -i 'ERROR: Failed to save XFS options of /dev/mapper/HANADATA-data'
++ has_binary caller
++ for bin in '$@'
++ type caller
++ return 0
++ echo '==== Stack trace ===='
==== Stack trace ====
++ local c=0
++ caller 0
++ :
++ caller 1
++ :
++ caller 2
++ awk ' { l[NR]=$3":"$1" "$2 }
END { for (i=NR; i>0;) print "Trace "NR-i": "l[i--] }
++ :
++ caller 3
'
++ :
++ caller 4
++ :
++ caller 5
++ :
++ caller 6
Trace 0: /usr/sbin/nsr_rear:543 main
Trace 1: /usr/share/rear/lib/mkbackup-workflow.sh:11 WORKFLOW_mkbackup
Trace 2: /usr/share/rear/lib/framework-functions.sh:101 SourceStage
Trace 3: /usr/share/rear/lib/framework-functions.sh:49 Source
Trace 4: /usr/share/rear/layout/save/GNU/Linux/230_filesystem_layout.sh:156 source
Trace 5: /usr/share/rear/lib/_input-output-functions.sh:251 StopIfError
++ echo 'Message: Failed to save XFS options of /dev/mapper/HANADATA-data'
Message: Failed to save XFS options of /dev/mapper/HANADATA-data
++ echo '== End stack trace =='
jsmeix commented at 2017-12-21 10:58:¶
@sebaobi
ony a blind idea regarding further debugging
how BACKUP=NSR versus BACKUP=NETFS
could make any difference whether or not
xfs_admin or xfs_info fail with
xfs_db or xfs_growfs command not found:
With BACKUP=NETFS I get
# usr/sbin/rear -s mkbackup | egrep 'NETFS|230_filesystem_layout' Source prep/NETFS/default/040_inspect_configuration_files.sh Source prep/NETFS/default/050_check_NETFS_requirements.sh Source prep/NETFS/default/060_mount_NETFS_path.sh Source prep/NETFS/default/070_set_backup_archive.sh Source prep/NETFS/default/090_check_encrypted_backup.sh Source prep/NETFS/default/100_check_nfs_version.sh Source prep/NETFS/default/150_save_rsync_version.sh Source prep/NETFS/GNU/Linux/200_selinux_in_use.sh Source prep/NETFS/GNU/Linux/205_inspect_tar_capabilities.sh Source prep/NETFS/default/400_automatic_exclude_recreate.sh Source prep/NETFS/default/980_umount_NETFS_dir.sh Source layout/save/GNU/Linux/230_filesystem_layout.sh ...
With BACKUP=NSR I get
# usr/sbin/rear -s mkbackup | egrep 'NSR|230_filesystem_layout' Source prep/NSR/default/400_prep_nsr.sh Source prep/NSR/default/450_check_nsr_client.sh Source layout/save/GNU/Linux/230_filesystem_layout.sh ...
When BACKUP=NSR versus BACKUP=NETFS
really makes a difference whether or not
xfs_admin or xfs_info fail with
xfs_db or xfs_growfs command not found
the scripts
usr/share/rear/prep/NSR/default/400_prep_nsr.sh
usr/share/rear/prep/NSR/default/450_check_nsr_client.sh
are "the usual suspects" because they run before
usr/share/rear/layout/save/GNU/Linux/230_filesystem_layout.sh
but
neither prep/NSR/default/400_prep_nsr.sh
nor prep/NSR/default/450_check_nsr_client.sh
looks as if the code therein could have any influence on
whether or not xfs_admin or xfs_info fail with
xfs_db or xfs_growfs command not found.
sebaobi commented at 2017-12-21 11:12:¶
Hmm oki, i just checked this scripts and don't see xfs related things:
test:/usr/sbin # cat -n /usr/share/rear/prep/NSR/default/400_prep_nsr.sh
1 #
2 # prepare stuff for NSR
3 #
4
5 COPY_AS_IS=( "${COPY_AS_IS[@]}" "${COPY_AS_IS_NSR[@]}" )
6 COPY_AS_IS_EXCLUDE=( "${COPY_AS_IS_EXCLUDE[@]}" "${COPY_AS_IS_EXCLUDE_NSR[@]}" )
7 PROGS=( "${PROGS[@]}" "${PROGS_NSR[@]}" fmt )
test:/usr/sbin # cat -n /usr/share/rear/prep/NSR/default/450_check_nsr_client.sh
1 # 450_check_nsr_client.sh
2 #
3 # This script checks if a EMC Legato client is installed and running
4 #
5
6 Log "Backup method is NetWorker (NSR): check nsrexecd"
7 if [ ! -x /usr/sbin/nsrexecd ] \
8 && [ ! -x /opt/networker/sbin/nsrexecd ]; then
9 Error "Please install EMC NetWorker (Legato) client software."
10 fi
11
12 ps ax | grep nsrexecd | grep -v grep 1>/dev/null
13 StopIfError $? "EMC NetWorker (Legato) nsrexecd was not running on this client."
jsmeix commented at 2017-12-21 11:32:¶
There is nothing that looks suspicious about 'xfs'
in any of the ReaR scripts that are run
before 230_filesystem_layout.sh
# for s in $( usr/sbin/rear -s mkbackup | grep -B100 230_filesystem_layout | sed -e 's|Source |usr/share/rear/|' ) ; do grep -i 'xfs' $s 2>/dev/null && echo -e "found in $s \n" ; done
mkfs.xfs
fsck.xfs
xfs_admin
xfs_db
xfs_repair
xfs_info
xfs_growfs
found in usr/share/rear/conf/GNU/Linux.conf
# We need directory for XFS options only if XFS is in use
if [ -n "$(mount -t xfs)" ]; then
LAYOUT_XFS_OPT_DIR="$VAR_DIR/layout/xfs"
mkdir -p $v $LAYOUT_XFS_OPT_DIR >&2
found in usr/share/rear/layout/save/GNU/Linux/100_create_layout_file.sh
supported_filesystems="ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs"
(xfs)
uuid=$(xfs_admin -u $device | cut -d'=' -f 2 | tr -d " ")
label=$(xfs_admin -l $device | cut -d'"' -f 2)
# Save current XFS file system options.
xfs_info $mountpoint > $LAYOUT_XFS_OPT_DIR/$(basename ${device}.xfs)
StopIfError "Failed to save XFS options of $device"
found in usr/share/rear/layout/save/GNU/Linux/230_filesystem_layout.sh
Currently I have no idea how to further debug it.
From my current point of view there is nothing in ReaR
which looks suspicious to be the cause of this issue.
sebaobi commented at 2017-12-21 11:39:¶
Ok I understand, thank you Johannes for your time.
[Export of Github issue for rear/rear.]