#1534 Issue `closed`: ReaR with TSM does not ask for TSM passwords¶

Labels: support / question, fixed / solved / done, external tool

dcz01 opened issue at 2017-10-13 09:40:¶

BUG: ReaR with TSM don't asks for passwords¶

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

rear version (/usr/sbin/rear -V): Relax-and-Recover 2.2 / 2017-07-20
OS version (cat /etc/rear/os.conf or lsb_release -a):

OS_VENDOR=RedHatEnterpriseServer
OS_VERSION=6

rear configuration files (cat /etc/rear/site.conf or cat /etc/rear/local.conf):

OUTPUT=ISO
OUTPUT_URL=file:///tmp/rear
#BACKUP=NETFS
BACKUP=TSM
#BACKUP_PROG=tar
#BACKUP_PROG_CRYPT_ENABLED=1
#BACKUP_PROG_CRYPT_KEY=<Verschluesselungskennwort>
#BACKUP_PROG_CRYPT_OPTIONS="/usr/bin/openssl aes256 -salt -k"
#BACKUP_PROG_DECRYPT_OPTIONS="/usr/bin/openssl aes256 -d -k"
#BACKUP_URL=nfs://<IP-Adresse oder DNS-Name>/<Freigabepfad>
#BACKUP_URL=cifs://<Server>/<Freigabe>
#BACKUP_OPTIONS="cred=/etc/rear/cifs,sec=ntlmsspi"
#BACKUP_TYPE=incremental
#FULLBACKUPDAY="Sat"
#BACKUP_PROG_EXCLUDE=( '/tmp/*' '/dev/shm/*' $VAR_DIR/output/\* '/opt/tivoli/tsm/rear/*' '/mnt/*' '/media/*' '/var/lib/pgsql/*/data/base/*' '/var/lib/pgsql/*/data/global/*' '/var/lib/pgsql/*/data/pg*/*' )
SSH_ROOT_PASSWORD='$1$HGjk3XUV$lid3Nd3k01Kht1mpMscLw1'

Are you using legacy BIOS or UEFI boot?
BIOS.
Brief description of the issue:
The ReaR command "rear recover" starts in the recovery environment normally.
But then when no /etc/adsm/TSM.PWD existed in the server, ReaR doesn't show up an correct prompt for the password for the TSM client.
Work-around, if any:
Type in the password without knowing about it.

Full detailed log is attached.

By this step of the "rear recover" workflow, ReaR should ask the user for the communication password of the TSM client but it doesn't (and so the workflow hangs here forever in a loop):

RESCUE brp-server:~ # rear recover
Relax-and-Recover 2.2 / 2017-07-20
Using log file: /var/log/rear/rear-brp-server.log
Running workflow recover within the ReaR rescue/recovery system

TSM restores by default the latest backup data. Alternatively you can specify
a different date and time to enable Point-In-Time Restore. Press ENTER to
use the most recent available backup
Enter date/time (YYYY-MM-DD HH:mm:ss) or press ENTER [30 secs]:
Skipping Point-In-Time Restore, will restore most recent data.
^CERROR: 'dsmc query filespace' failed !
Aborting due to an error, check /var/log/rear/rear-brp-server.log for details
Terminated

And by this input the workflow waits for user input but without showing the output from TSM (for encryption/decryption):

RESCUE brp-server:~ # rear recover
Relax-and-Recover 2.2 / 2017-07-20
Using log file: /var/log/rear/rear-brp-server.log
Running workflow recover within the ReaR rescue/recovery system

TSM restores by default the latest backup data. Alternatively you can specify
a different date and time to enable Point-In-Time Restore. Press ENTER to
use the most recent available backup
Enter date/time (YYYY-MM-DD HH:mm:ss) or press ENTER [30 secs]:
Skipping Point-In-Time Restore, will restore most recent data.

The TSM Server reports the following for this node:
                  #     Last Incr Date          Type    File Space Name
                --------------------------------------------------------------------------------
                  1     13-10-2017 10:37:03     EXT4    /
                  2     13-10-2017 10:36:58     EXT4    /boot
Please enter the numbers of the filespaces we should restore.
Pay attention to enter the filesystems in the correct order
(like restore / before /var/log) !
(default: 1 2): [30 secs]
We will now restore the following filesystems:
/
/boot
Is this selection correct ? (Y|n) [30 secs] y
Comparing disks.
Disk configuration is identical, proceeding with restore.
Start system layout restoration.
Creating partitions for disk /dev/sda (msdos)
Creating filesystem of type ext4 with mount point / on /dev/sda3.
2 bytes [53 ef] erased at offset 0x438 (ext4)
Mounting filesystem /
Creating filesystem of type ext4 with mount point /boot on /dev/sda1.
2 bytes [53 ef] erased at offset 0x438 (ext4)
Mounting filesystem /boot
Creating swap on /dev/sda2
Disk layout created.
Restoring /
IBM Tivoli Storage Manager
Command Line Backup-Archive Client Interface
Client Version 7, Release 1, Level 3.0
Client date/time: 10/13/17   11:19:19
(c) Copyright by IBM Corporation and other(s) 1990, 2015. All Rights Reserved.

Node Name: BRP-SERVER
Session established with server TSMFAG1: Windows
Server Version 7, Release 1, Level 5.200
Data compression forced on by the server
Server date/time: 10/13/17   11:19:23  Last access: 10/13/17   11:18:29

Restore function invoked.

ANS1247I Waiting for files from the server...
* Processed    20,500 files * \
--- User Action is Required ---
File: /mnt/local/opt/brp/uploads/eliste requires an encryption key.


Select an appropriate action
1. Prompt for encrypt key password
2. Skip this object from decryption
3. Skip all objects that are encrypted
A. Abort this operation
1

rear-brp-server.log

jsmeix commented at 2017-10-13 10:09:¶

@dcz01
many thanks for testing the current ReaR master code
together with TSM!
We really need users who test the current ReaR master code
with their particular third-party backup software.
Such tests help us very much to get issues fixed in advance
before ReaR v 2.3 is released.

I am not a TSM user so that I cannot reproduce
anything that is specific for TSM.

In general when an intercative program waits for user input
but without showing any output to the user (e.g. no prompt)
that program call must be fixed so that it is run with the
original file descriptors when usr/sbin/rear was launched like

COMMAND ... 0<&6 1>&7 2>&8

see "What to do with stdout and stderr" at
https://github.com/rear/rear/wiki/Coding-Style

Since
https://github.com/rear/rear/commit/6ab3c4173fe425bb018880af4a8461bf575ba1e0
STDOUT is redirected into the log file
see also https://github.com/rear/rear/issues/1399

When STDOUT is redirected into the log file
there could be regressions, see
https://github.com/rear/rear/issues/1398

jsmeix commented at 2017-10-13 10:51:¶

I know nothing about TSM so that I could be totally wrong here
but in
https://github.com/rear/rear/files/1382135/rear-brp-server.log
I found the following suspiciously looking message
"Please enter your user id :"
that is not in the ReaR code so that I guess it
comes from TSM:

+ source /usr/share/rear/verify/TSM/default/400_verify_tsm.sh
...
++ dsmc query filespace -date=2 -time=1 -scrollprompt=no
++ grep -A 10000 File
Please enter your user id : ++ '[' 12 -eq 0 -o 12 -eq 8 ']'
++ StopIfError ''\''dsmc query filespace'\'' failed !'
++ ((  1 != 0  ))
++ Error ''\''dsmc query filespace'\'' failed !'

If my guess is right the "dsmc query filespace" command in
verify/TSM/default/400_verify_tsm.sh
is (or sometimes could be) an interactively running command
so that it requires to be called with appropriate file descriptors
but because it is used in a pipe

LC_ALL=${LANG_RECOVER} dsmc query filespace -date=2 -time=1 -scrollprompt=no | grep -A 10000 'File' >$TMP_DIR/tsm_filespaces

its stdout file descriptor cannot be changed.

jsmeix commented at 2017-10-13 12:58:¶

It also seems that the "dsmc restore" command in
restore/TSM/default/400_restore_with_tsm.sh
is (or sometimes could be) an interactively running command
so that it requires to be called with appropriate file descriptors
but because it is also used in a pipe its stdout file descriptor
also cannot be changed.

jsmeix commented at 2017-10-13 13:09:¶

Regarding the oversophisticated "animated_cursor" functionality
(why the "dsmc restore" command is run in a pipe), see
https://github.com/rear/rear/issues/1398#issuecomment-313097020
and subsequent comments.

@dcz01
only a blind proposal because I do not have TSM:
Does it perhaps work better when you simplify
usr/share/rear/restore/TSM/default/400_restore_with_tsm.sh
to something like

for num in $TSM_RESTORE_FILESPACE_NUMS ; do
    filespace="${TSM_FILESPACES[$num]}"
    # make sure FileSpace has a trailing / (for dsmc)
    test "${filespace:0-1}" == "/" || filespace="$filespace/"
    LogPrint "Restoring ${filespace}"
    TsmProcessed=""
    Log "Running 'dsmc restore ${filespace}* $TARGET_FS_ROOT/$filespace -verbose -subdir=yes -replace=all -tapeprompt=no ${TSM_DSMC_RESTORE_OPTIONS[@]}'"
    LC_ALL=${LANG_RECOVER} dsmc restore \""${filespace}"\" \""$TARGET_FS_ROOT/${filespace}/"\" \
        -verbose -subdir=yes -replace=all \
        -tapeprompt=no "${TSM_DSMC_RESTORE_OPTIONS[@]}" \
        0<&6 1>&7 2>&8
done

jsmeix commented at 2017-10-13 13:11:¶

@schabrolles
I do not understand the overcomplicated looking quoting in

... dsmc restore \""${filespace}"\" \""$TARGET_FS_ROOT/${filespace}/"\"

Can you provide background information what the reason behind is?

schabrolles commented at 2017-10-13 15:50:¶

@jsmeix I'm not a TSM expert, but I would say it is to protect against space or special characters.
dsmc restore "/my filesystem" "/target/my filesystem"

jsmeix commented at 2017-10-14 10:42:¶

I think usually double quotes do not sufficiently protect (e.g. not against '$')
but perhaps the intent is that $var gets evaluated like in

dsmc restore "/$myfilesystem" "/target/$myfilesystem"

Normally would use single quotes to avoid any shell evaluation e.g. like

#  foo=' $( echo FOO ) '

# echo $foo
$( echo FOO )

# echo "$foo"
 $( echo FOO ) 

# echo "'$foo'"
' $( echo FOO ) '

# eval echo "'$foo'"
 $( echo FOO ) 

# eval echo "$foo"
FOO

# eval echo \""$foo"\"
 FOO

schabrolles commented at 2017-10-22 14:04:¶

@jsmeix
1- I made several test by simplified quoting... Seems to work even if there is "spaces" in the Path.
2- I had some difficulties to use the 0<&6 1>&7 2>&8 redirection on the first dsmc call because this one is used with PIPE.

dsmc query filespace -date=2 -time=1 -scrollprompt=no | grep -A 10000 'File' >$TMP_DIR/tsm_filespaces

I decided then to add a NEW first connection before this line (dsmc query mgmt).
It just print some information about the server. If the TSM passwd file is missing, it will ask the user to enter login/passwd to the prompt.

dsmc query mgmt 0<&6 1>&7 2>&8

schabrolles commented at 2017-10-25 12:55:¶

@dcz01 with #1539 merged, TSM should ask for ID/passwd on the prompt when TSM passwd is not part of the rescue image.
Waiting for your confirmation before closing this one.

dcz01 commented at 2017-10-26 09:11:¶

@schabrolles Well i would like to test your changes but i can't because the other bug isn't fixed yet with the missing librarys -> #1533
I need to build an new rescue image but this fails...
But when the other bug is fixed i test it and give you feedback.

schabrolles commented at 2017-11-06 14:03:¶

@dcz01, #1562 is now merged and should solve the TSM libraries issue.
Could you please test the master branch to confirm that TSM is now asking for password when needed.
Thanks

schabrolles commented at 2017-11-25 12:47:¶

@dcz01, did you find time to test if your issues regarding TSM are now solved by #1539 and #1562.
I would like to close this bug. thanks.

dcz01 commented at 2017-11-27 07:44:¶

@schabrolles I can't build a new image because of the missing two librarys from the other issue.
https://github.com/rear/rear/issues/1533#issuecomment-343067688

dcz01 commented at 2017-11-29 10:58:¶

@schabrolles Now i builded a new image with the new variable from @jsmeix
NON_FATAL_BINARIES_WITH_MISSING_LIBRARY='/opt/tivoli/tsm/client/ba/bin/libvixMntapi.so.1.1.0'

But now the output isn't enough to see when to enter the encryption password:
grafik
grafik

And other unnecessary outputs on the console from the TSM client:
grafik
grafik
grafik

gdha commented at 2017-12-11 08:52:¶

@jsmeix @schabrolles shall we push this to 2.4?

dcz01 commented at 2017-12-11 09:21:¶

@gdha If this bug is moved to ReaR 2.4 the client displays too much and you should rollback all changes made in this issue.

schabrolles commented at 2017-12-11 11:09:¶

@dcz01,

I'm made a small change in the code to ask TSM password sooner.
Could you give it a try and tell me if it is better :

git clone http://github.com/schabrolles/rear
cd rear
git checkout TSM_ask_passwd_2
make install

If you prefer this one, I'll create a Pull Request.

dcz01 commented at 2017-12-11 11:14:¶

@schabrolles I'm sorry, i cannot test this little change of the code because our servers to test cannot reach the internet above the proxy of my business.
And i can't change the proxy settings myself.

jsmeix commented at 2017-12-11 11:36:¶

@dcz01
you can see the needed changes in
https://github.com/rear/rear/pull/1641/files

schabrolles commented at 2017-12-11 11:53:¶

@dcz01

you can also create a file names /tmp/patch_TSM
copy the content of https://patch-diff.githubusercontent.com/raw/rear/rear/pull/1641.patch in /tmp/patch_TSM
cd /
patch -p1 < /tmp/patch_TSM

dcz01 commented at 2017-12-11 13:41:¶

@schabrolles How can i do the patch command?
My system doesn't find this command right now.
I installed git but it doesn't work.

schabrolles commented at 2017-12-11 14:14:¶

@dcz01,

yum install patch on a rhel6 should be enough to get access to the patch command.

dcz01 commented at 2017-12-11 14:18:¶

@schabrolles Now i worked to patch, thanks.
I'll test the changes now.

dcz01 commented at 2017-12-11 14:35:¶

@schabrolles There is now an error in the recovery:
grafik

I think we can cancel this issue here and restore the old defaults from the files in ReaR like it was in ReaR 2.0 to 2.2.
This is because an TSM client with ReaR recovery should always have an password file to be copied to the rescue image.

jsmeix commented at 2017-12-12 12:56:¶

Via https://github.com/rear/rear/commit/d9a8e2bcaa693bfb3e49eaaee667b4bc7360f9cb
the TSM first connection via "dsmc query mgmt"
that will prompt for TSM password via a separated script
as in https://github.com/rear/rear/pull/1641
was merged into the ReaR master code.

@dcz01
this is probably not what you want according to your
https://github.com/rear/rear/issues/1534#issuecomment-350741413
but I think it is at least a step into the right direction
to get the TSM first connection via "dsmc query mgmt"
out of usr/share/rear/verify/TSM/default/400_verify_tsm.sh
into a separated script
usr/share/rear/verify/TSM/default/389_check_TSM_connexion.sh
because now you could simply remove that separated script
usr/share/rear/verify/TSM/default/389_check_TSM_connexion.sh
if it causes problems for you.

Later that separated script could be adapted and enhanced
it to make TSM work even without a TSM password file
in the ReaR recovery system because in general the
recovery system should not contain secrets.

According to what I read in
https://github.com/rear/rear/issues/1533
it seems such a TSM password file is something like
/etc/adsm/TSM.PWD or /etc/opt/tb017/security/TSM.PWD
and I assume that file contains secrets.

@dcz01
I would like to know if TSM restore works for you even with
usr/share/rear/verify/TSM/default/389_check_TSM_connexion.sh
when you have your TSM password file in your recovery system.

dcz01 commented at 2017-12-12 13:45:¶

@jsmeix @schabrolles Yes, the restore with an TSM password file (TSM.PWD) works fine.
But the first connection to test if password is available should be a litte bit shorter.
We could maybe use the dsmc query session command instead of dsmc query mgmt?

jsmeix commented at 2017-12-12 14:19:¶

As far as I (as non-TSM user) understand the issue now,
it is currently no actual bug in ReaR that TSM backup restore
requires a 'TSM.PWD' file in the recovery system because
currently this is how TSM support is implemented in ReaR.

But TSM support should be enhanced in a future ReaR release
so that TSM backup restore also works without 'TSM.PWD'
the recovery system and therefore I made the new
https://github.com/rear/rear/issues/1642

jsmeix commented at 2017-12-12 14:26:¶

@dcz01
I cannot decide if for the connection test

dsmc query session

could also be used because (as non-TSM user) I don't know
if that command would be a valid connection test in any case
so that I like to leave that part to @schabrolles

I have closed this issue for ReaR 2.3 because
as far as I understand it TSM works as currently intended.

dcz01 commented at 2017-12-12 14:30:¶

@jsmeix I am an TSM user and TSM administrator.
This command connects once to the server and displays some server info.
This is also enough for the prompt of the password.

dcz01 commented at 2017-12-12 14:33:¶

@jsmeix @schabrolles Well now i tested another time and there is an problem...
It hangs after entering the password and doesn't restore anything else.
grafik
grafik

This are the two screens where the restore is hanging.

dcz01 commented at 2017-12-12 14:40:¶

Well the first hang was my fault but it doesn't display all password inputs right now...

grafik
grafik

jsmeix commented at 2017-12-12 15:31:¶

Via https://github.com/rear/rear/pull/1643
I tried to improve the TSM connection test and
I renamed the misspelled 389_check_TSM_connexion.sh
into 389_check_TSM_connection.sh

jsmeix commented at 2017-12-12 15:40:¶

@dcz01
in your
https://github.com/rear/rear/issues/1534#issuecomment-351070598
what does "display all password inputs" mean?
Does it mean there are several prompts from TSM for passwords?

Cf. my
https://github.com/rear/rear/issues/1534#issuecomment-336410596

I guess the root cause of all evil could be the overcomplicated
way how the "dsmc restore" command is called in
usr/share/rear/restore/TSM/default/400_restore_with_tsm.sh
because it is currently not called as

dsmc restore ... 0<&6 1>&7 2>&8

If the "dsmc restore" command shows a password prompt
via STDERR that password prompt is currently redirected
into the ReaR log file so that you won't see it on your terminal.

jsmeix commented at 2017-12-12 16:02:¶

Via https://github.com/rear/rear/pull/1645
I tried to simplify how the TSM 'dsmc restore' command is called.

@dcz01
could you try out if it works for you when you replace your
usr/share/rear/restore/TSM/default/400_restore_with_tsm.sh
with the code in
https://raw.githubusercontent.com/jsmeix/rear/bdb2a27f3d4b4d755f800d9963f13c8c3b09eabe/usr/share/rear/restore/TSM/default/400_restore_with_tsm.sh
Careful!
Long lines therein might be shown wrapped in the browser
(depending on how your browser shows plain text files).

dcz01 commented at 2017-12-13 08:04:¶

@jsmeix With "doesn't display all password inputs" i mean that some inputs of the password like of the encryption or decryption of files is not shown in the command line because it is redirected to the ReaR log.
I think we should change the wohle dsmc command from ReaR...
Is the -verbose option important for ReaR or could it be the -quiet option instead?

jsmeix commented at 2017-12-13 11:15:¶

@dcz01
in the current https://github.com/rear/rear/pull/1645
I call now 'dsmc restore' without '-verbose', cf.
https://github.com/rear/rear/pull/1645#discussion_r156612705
and I added a user information (but no "rear recover" abort,
see my comments in the code) when 'dsmc restore'
finished with non-zero exit code.

jsmeix commented at 2017-12-13 11:20:¶

@dcz01 @schabrolles
is it possible to provide passwords via command line options
to 'dsmc restore' i.e. via TSM_DSMC_RESTORE_OPTIONS
because conf/default.conf reads:

# Additional dsmc options for restore. Point-in-time read from user
# input is also added to this array.
TSM_DSMC_RESTORE_OPTIONS=( )

and I wonder what "Point-in-time read from user input" means?

dcz01 commented at 2017-12-13 11:46:¶

@jsmeix Yes, it is possible to deliver the TSM client directly the password with an extra option.
But i think this would not be enough...
The best solution is to check if an password file or (in TSM 8.1) many password files exist and if not then offer the chance to enter an administrator id and the password of that id.
Also should an extra option in TSM_DSMC_RESTORE_OPTIONS be used to avoid the automatic password change on the TSM server with the -virtualnodename option.

Or ReaR deletes by default the entry PASSWORDACCESS GENERATE in the /opt/tivoli/tsm/client/ba/bin/dsm.sys file when creating the rescue image.
Then the TSM client always wants the input of an password for each session and we don't need the -virtualnodename option.

schabrolles commented at 2017-12-13 11:59:¶

@jsmeix, for security, the best would be to not include /etc/adsm directory by default in COPY_AS_IS_TSM array.
and create an option like ADD_TSM_PASSWD_IN_RESCUE in default.conf file to activate the TSM password inclusion into the rear rescue image.

jsmeix commented at 2017-12-13 13:55:¶

I think the real solution how to implement it properly
should be done via https://github.com/rear/rear/issues/1642

What I currently do are only some quick fixes for now.

jsmeix commented at 2017-12-13 14:09:¶

I cannot test if or how it works in practice
but with https://github.com/rear/rear/pull/1646
it should now at least be technically possible to also specify TSM_DSMC_RESTORE_OPTIONS on command line.
The idea behind is that this way the user may not need
TSM password files in the recovery system but he could
specify the needed TSM passwords via

export TSM_DSMC_RESTORE_OPTIONS=( ... )

directly before he calls "rear recover".
If there is TSM_DSMC_RESTORE_OPTIONS
set in /etc/rear/local.conf the user could alternatively
add the TSM password options there as usual.
The way via 'export' on command line is just for
convenience.
In both cases the dscm options (i.e. also password options)
will appear in the ReaR log file and the ReaR log file is
copied into the recreated system so that one must
care about possible secrets in the ReaR log file,
cf. my code comments about "security relevant information" in
usr/share/rear/wrapup/default/990_copy_logfile.sh

jsmeix commented at 2017-12-13 17:14:¶

Forget https://github.com/rear/rear/pull/1646 see
https://github.com/rear/rear/pull/1646#issuecomment-351458486

jsmeix commented at 2018-01-09 16:05:¶

With https://github.com/rear/rear/pull/1645 merged
I assume/hope this issue is sufficiently fixed.

[Export of Github issue for rear/rear.]

#1534 Issue closed: ReaR with TSM does not ask for TSM passwords¶