#3006 PR merged: New --expose-secrets option plus SECRET_OUTPUT_DEV¶

Labels: enhancement, fixed / solved / done, critical / security / legal

jsmeix opened issue at 2023-06-06 08:34:¶

• Type: Critical Fix / Enhancement

• Impact: Critical

• Reference to related issue (URL):
  https://github.com/rear/rear/issues/2967

• How was this pull request tested?

• Brief description of the changes in this pull request:

In sbin/rear added --expose-secrets option
and SECRET_OUTPUT_DEV

jsmeix commented at 2023-06-06 08:45:¶

For a first quick test I added at the beginning of
init/default/030_update_recovery_system.sh

if { grep --Q=SECRET FOO /etc/issue ; } 2>>/dev/$SECRET_OUTPUT_DEV ; then
    Log "'grep --Q=... FOO /etc/issue' succeeded"
else
    { LogSecret "'grep --Q=SECRET FOO /etc/issue' failed with exit code $?" ; } 2>>/dev/$SECRET_OUTPUT_DEV
    Error "'grep --Q=... FOO /etc/issue' failed"
fi
Error "END"

With that I get:

# usr/sbin/rear help
ERROR: 'grep --Q=... FOO /etc/issue' failed
Use debug mode '-d' for some debug messages or debugscript mode '-D' for full debug messages with 'set -x' output
Aborting due to an error, check /root/rear.github.master/var/log/rear/rear-linux-h9wr.log.lockless for details
Terminated

# cat var/log/rear/rear-linux-h9wr.log.lockless
2023-06-06 10:39:40.921634121 Including /root/rear.github.master/etc/rear/os.conf
2023-06-06 10:39:40.927053310 Including conf/Linux-i386.conf
2023-06-06 10:39:40.930470239 Including conf/GNU/Linux.conf
2023-06-06 10:39:42.483603709 Including conf/SUSE_LINUX.conf
2023-06-06 10:39:42.491462279 Including /root/rear.github.master/etc/rear/local.conf
2023-06-06 10:39:42.500051376 ======================
2023-06-06 10:39:42.505623913 Running 'init' stage
2023-06-06 10:39:42.509317818 ======================
2023-06-06 10:39:42.521962978 Including init/default/001_verify_config_arrays.sh
2023-06-06 10:39:42.897170714 Including init/default/005_verify_os_conf.sh
2023-06-06 10:39:42.903739112 Including init/default/010_EFISTUB_check.sh
2023-06-06 10:39:42.910867106 Including init/default/010_set_drlm_env.sh
2023-06-06 10:39:42.917733749 Including init/default/030_update_recovery_system.sh
2023-06-06 10:39:42.932570308 ERROR: 'grep --Q=... FOO /etc/issue' failed
2023-06-06 10:39:42.943469639 Exiting rear help (PID 18409) and its descendant processes ...
2023-06-06 10:39:45.991156739 rear,18409 usr/sbin/rear help
                                `-rear,19037 usr/sbin/rear help
                                    `-pstree,19038 -Aplau 18409
2023-06-06 10:39:46.058937907 Running exit tasks

# grep SECRET var/log/rear/rear-linux-h9wr.log.lockless
[no output]

# usr/sbin/rear -e help
ERROR: 'grep --Q=... FOO /etc/issue' failed
Some latest log messages since the last called script 030_update_recovery_system.sh:
  2023-06-06 10:40:25.837571143 'grep --Q=SECRET FOO /etc/issue' failed with exit code 2
Use debug mode '-d' for some debug messages or debugscript mode '-D' for full debug messages with 'set -x' output
Aborting due to an error, check /root/rear.github.master/var/log/rear/rear-linux-h9wr.log.lockless for details
Terminated

# cat var/log/rear/rear-linux-h9wr.log.lockless
2023-06-06 10:40:23.820080914 Including /root/rear.github.master/etc/rear/os.conf
2023-06-06 10:40:23.827143367 Including conf/Linux-i386.conf
2023-06-06 10:40:23.831520599 Including conf/GNU/Linux.conf
2023-06-06 10:40:25.428790063 Including conf/SUSE_LINUX.conf
2023-06-06 10:40:25.434168151 Including /root/rear.github.master/etc/rear/local.conf
2023-06-06 10:40:25.438380487 ======================
2023-06-06 10:40:25.441715924 Running 'init' stage
2023-06-06 10:40:25.445781793 ======================
2023-06-06 10:40:25.455339793 Including init/default/001_verify_config_arrays.sh
2023-06-06 10:40:25.811407644 Including init/default/005_verify_os_conf.sh
2023-06-06 10:40:25.818007982 Including init/default/010_EFISTUB_check.sh
2023-06-06 10:40:25.824585919 Including init/default/010_set_drlm_env.sh
2023-06-06 10:40:25.832158164 Including init/default/030_update_recovery_system.sh
2023-06-06 10:40:25.837571143 'grep --Q=SECRET FOO /etc/issue' failed with exit code 2
2023-06-06 10:40:25.853775614 ERROR: 'grep --Q=... FOO /etc/issue' failed
2023-06-06 10:40:25.865643238 Exiting rear help (PID 19076) and its descendant processes ...
2023-06-06 10:40:28.914081578 rear,19076 usr/sbin/rear -e help
                                `-rear,19708 usr/sbin/rear -e help
                                    `-pstree,19709 -Aplau 19076
2023-06-06 10:40:28.943383213 Running exit tasks

# grep SECRET var/log/rear/rear-linux-h9wr.log.lockless
2023-06-06 10:46:13.196338241 'grep --Q=SECRET FOO /etc/issue' failed with exit code 2

# usr/sbin/rear -D help
Running 'init' stage ======================
ERROR: 'grep --Q=... FOO /etc/issue' failed
Some latest log messages since the last called script 030_update_recovery_system.sh:
  2023-06-06 10:41:07.679328205 Entering debugscript mode via 'set -x'.
Aborting due to an error, check /root/rear.github.master/var/log/rear/rear-linux-h9wr.log.lockless for details
Exiting rear help (PID 19744) and its descendant processes ...
Running exit tasks
Terminated

# cat var/log/rear/rear-linux-h9wr.log.lockless
...
+ source /root/rear.github.master/usr/share/rear/init/default/030_update_recovery_system.sh
++ Error ''\''grep --Q=... FOO /etc/issue'\'' failed'
++ test -s /root/rear.github.master/var/log/rear/rear-linux-h9wr.log.lockless
++ PrintError 'ERROR: '\''grep --Q=... FOO /etc/issue'\'' failed'
++ test -s /root/rear.github.master/var/log/rear/rear-linux-h9wr.log.lockless
++ test '  2023-06-06 10:41:07.679328205 Entering debugscript mode via '\''set -x'\''.'
++ PrintError 'Some latest log messages since the last called script 030_update_recovery_system.sh:'
++ PrintError '  2023-06-06 10:41:07.679328205 Entering debugscript mode via '\''set -x'\''.'
++ test -f /dev/null
++ test 1
++ test 1
++ Log 'ERROR: '\''grep --Q=... FOO /etc/issue'\'' failed'
...

# grep SECRET var/log/rear/rear-linux-h9wr.log.lockless
[no output]

# usr/sbin/rear -e -D help
Running 'init' stage ======================
ERROR: 'grep --Q=... FOO /etc/issue' failed
Some latest log messages since the last called script 030_update_recovery_system.sh:
  2023-06-06 10:43:23.114764758 Entering debugscript mode via 'set -x'.
  grep: unrecognized option '--Q=SECRET'
  Usage: grep [OPTION]... PATTERN [FILE]...
  Try 'grep --help' for more information.
  2023-06-06 10:43:23.126348711 'grep --Q=SECRET FOO /etc/issue' failed with exit code 2
Aborting due to an error, check /root/rear.github.master/var/log/rear/rear-linux-h9wr.log.lockless for details
Exiting rear help (PID 20612) and its descendant processes ...
Running exit tasks
Terminated

# cat var/log/rear/rear-linux-h9wr.log.lockless
...
+ source /root/rear.github.master/usr/share/rear/init/default/030_update_recovery_system.sh
++ grep --Q=SECRET FOO /etc/issue
grep: unrecognized option '--Q=SECRET'
Usage: grep [OPTION]... PATTERN [FILE]...
Try 'grep --help' for more information.
++ LogSecret ''\''grep --Q=SECRET FOO /etc/issue'\'' failed with exit code 2'
++ test 1
++ Log ''\''grep --Q=SECRET FOO /etc/issue'\'' failed with exit code 2'
++ test -w /root/rear.github.master/var/log/rear/rear-linux-h9wr.log.lockless
++ echo '2023-06-06 10:43:23.126348711 '\''grep --Q=SECRET FOO /etc/issue'\'' failed with exit code 2'
2023-06-06 10:43:23.126348711 'grep --Q=SECRET FOO /etc/issue' failed with exit code 2
++ Error ''\''grep --Q=... FOO /etc/issue'\'' failed'
++ test -s /root/rear.github.master/var/log/rear/rear-linux-h9wr.log.lockless
++ PrintError 'ERROR: '\''grep --Q=... FOO /etc/issue'\'' failed'
++ test -s /root/rear.github.master/var/log/rear/rear-linux-h9wr.log.lockless
++ test '  2023-06-06 10:43:23.114764758 Entering debugscript mode via '\''set -x'\''.
  grep: unrecognized option '\''--Q=SECRET'\''
  Usage: grep [OPTION]... PATTERN [FILE]...
  Try '\''grep --help'\'' for more information.
  2023-06-06 10:43:23.126348711 '\''grep --Q=SECRET FOO /etc/issue'\'' failed with exit code 2'
++ PrintError 'Some latest log messages since the last called script 030_update_recovery_system.sh:'
++ PrintError '  2023-06-06 10:43:23.114764758 Entering debugscript mode via '\''set -x'\''.
  grep: unrecognized option '\''--Q=SECRET'\''
  Usage: grep [OPTION]... PATTERN [FILE]...
  Try '\''grep --help'\'' for more information.
  2023-06-06 10:43:23.126348711 '\''grep --Q=SECRET FOO /etc/issue'\'' failed with exit code 2'
++ test -f /dev/null
++ test 1
++ test 1
++ Log 'ERROR: '\''grep --Q=... FOO /etc/issue'\'' failed'
...

# grep SECRET var/log/rear/rear-linux-h9wr.log.lockless
++ grep --Q=SECRET FOO /etc/issue
grep: unrecognized option '--Q=SECRET'
++ LogSecret ''\''grep --Q=SECRET FOO /etc/issue'\'' failed with exit code 2'
++ Log ''\''grep --Q=SECRET FOO /etc/issue'\'' failed with exit code 2'
++ echo '2023-06-06 10:48:31.745727366 '\''grep --Q=SECRET FOO /etc/issue'\'' failed with exit code 2'
2023-06-06 10:48:31.745727366 'grep --Q=SECRET FOO /etc/issue' failed with exit code 2
  grep: unrecognized option '\''--Q=SECRET'\''
  2023-06-06 10:48:31.745727366 '\''grep --Q=SECRET FOO /etc/issue'\'' failed with exit code 2'
  grep: unrecognized option '\''--Q=SECRET'\''
  2023-06-06 10:48:31.745727366 '\''grep --Q=SECRET FOO /etc/issue'\'' failed with exit code 2'

jsmeix commented at 2023-06-06 11:38:¶

For a test when a secret command succeeded
I added at the beginning of
init/default/030_update_recovery_system.sh

if { grep --Q=SECRET FOO /etc/issue || true ; } 2>>/dev/$SECRET_OUTPUT_DEV ; then
    { LogSecret "'grep --Q=SECRET FOO /etc/issue' succeded" || Log "'grep --Q=... FOO /etc/issue' succeeded" ; } 2>>/dev/$SECRET_OUTPUT_DEV
else
    { LogSecret "'grep --Q=SECRET FOO /etc/issue' failed with exit code $?" ; } 2>>/dev/$SECRET_OUTPUT_DEV
    Error "'grep --Q=... FOO /etc/issue' failed"
fi
Error "END"

and
I modified the LogSecret function to

function LogSecret () {
    test "$EXPOSE_SECRETS" && Log "$@"
}

so the LogSecret function returns a non-zero exit code
(the exit code of test "$EXPOSE_SECRETS")
when EXPOSE_SECRETS is not set
so that one can use it like

{ LogSecret "message with SECRET" || Log "generic message" ; } 2>>/dev/$SECRET_OUTPUT_DEV

With that I get

# usr/sbin/rear help
ERROR: END
Some latest log messages since the last called script 030_update_recovery_system.sh:
  2023-06-06 13:20:51.241569690 'grep --Q=... FOO /etc/issue' succeeded
Use debug mode '-d' for some debug messages or debugscript mode '-D' for full debug messages with 'set -x' output
Aborting due to an error, check /root/rear.github.master/var/log/rear/rear-linux-h9wr.log.lockless for details
Terminated

# cat var/log/rear/rear-linux-h9wr.log.lockless
2023-06-06 13:20:49.074055476 Including /root/rear.github.master/etc/rear/os.conf
2023-06-06 13:20:49.089350470 Including conf/Linux-i386.conf
2023-06-06 13:20:49.113310625 Including conf/GNU/Linux.conf
2023-06-06 13:20:50.789315284 Including conf/SUSE_LINUX.conf
2023-06-06 13:20:50.819178586 Including /root/rear.github.master/etc/rear/local.conf
2023-06-06 13:20:50.823482258 ======================
2023-06-06 13:20:50.826955042 Running 'init' stage
2023-06-06 13:20:50.830641076 ======================
2023-06-06 13:20:50.840225392 Including init/default/001_verify_config_arrays.sh
2023-06-06 13:20:51.215990184 Including init/default/005_verify_os_conf.sh
2023-06-06 13:20:51.222860082 Including init/default/010_EFISTUB_check.sh
2023-06-06 13:20:51.229854971 Including init/default/010_set_drlm_env.sh
2023-06-06 13:20:51.236528057 Including init/default/030_update_recovery_system.sh
2023-06-06 13:20:51.241569690 'grep --Q=... FOO /etc/issue' succeeded
2023-06-06 13:20:51.267955873 ERROR: END
2023-06-06 13:20:51.278484974 Exiting rear help (PID 5133) and its descendant processes ...
2023-06-06 13:20:54.348362381 rear,5133 usr/sbin/rear help
                                `-rear,5765 usr/sbin/rear help
                                    `-pstree,5766 -Aplau 5133
2023-06-06 13:20:54.407733190 Running exit tasks

# grep SECRET var/log/rear/rear-linux-h9wr.log.lockless
[no output]

# usr/sbin/rear -e help
ERROR: END
Some latest log messages since the last called script 030_update_recovery_system.sh:
  2023-06-06 13:21:15.056481900 'grep --Q=SECRET FOO /etc/issue' succeded
Use debug mode '-d' for some debug messages or debugscript mode '-D' for full debug messages with 'set -x' output
Aborting due to an error, check /root/rear.github.master/var/log/rear/rear-linux-h9wr.log.lockless for details
Terminated

# cat var/log/rear/rear-linux-h9wr.log.lockless
2023-06-06 13:21:13.118314154 Including /root/rear.github.master/etc/rear/os.conf
2023-06-06 13:21:13.123719808 Including conf/Linux-i386.conf
2023-06-06 13:21:13.127143425 Including conf/GNU/Linux.conf
2023-06-06 13:21:14.688056849 Including conf/SUSE_LINUX.conf
2023-06-06 13:21:14.693378519 Including /root/rear.github.master/etc/rear/local.conf
2023-06-06 13:21:14.697579236 ======================
2023-06-06 13:21:14.700833935 Running 'init' stage
2023-06-06 13:21:14.704080975 ======================
2023-06-06 13:21:14.713392404 Including init/default/001_verify_config_arrays.sh
2023-06-06 13:21:15.030903839 Including init/default/005_verify_os_conf.sh
2023-06-06 13:21:15.037883004 Including init/default/010_EFISTUB_check.sh
2023-06-06 13:21:15.044286030 Including init/default/010_set_drlm_env.sh
2023-06-06 13:21:15.050971066 Including init/default/030_update_recovery_system.sh
2023-06-06 13:21:15.056481900 'grep --Q=SECRET FOO /etc/issue' succeded
2023-06-06 13:21:15.071431761 ERROR: END
2023-06-06 13:21:15.081573904 Exiting rear help (PID 5799) and its descendant processes ...
2023-06-06 13:21:18.123068405 rear,5799 usr/sbin/rear -e help
                                `-rear,6432 usr/sbin/rear -e help
                                    `-pstree,6433 -Aplau 5799
2023-06-06 13:21:18.148469551 Running exit tasks

# grep SECRET var/log/rear/rear-linux-h9wr.log.lockless
2023-06-06 13:21:15.056481900 'grep --Q=SECRET FOO /etc/issue' succeded

# usr/sbin/rear -D help
Running 'init' stage ======================
ERROR: END
Some latest log messages since the last called script 030_update_recovery_system.sh:
  2023-06-06 13:21:40.512275229 Entering debugscript mode via 'set -x'.
  2023-06-06 13:21:40.521085733 'grep --Q=... FOO /etc/issue' succeeded
Aborting due to an error, check /root/rear.github.master/var/log/rear/rear-linux-h9wr.log.lockless for details
Exiting rear help (PID 6466) and its descendant processes ...
Running exit tasks
Terminated

# cat var/log/rear/rear-linux-h9wr.log.lockless
...
+ source /root/rear.github.master/usr/share/rear/init/default/030_update_recovery_system.sh
2023-06-06 13:21:40.521085733 'grep --Q=... FOO /etc/issue' succeeded
++ Error END
++ test -s /root/rear.github.master/var/log/rear/rear-linux-h9wr.log.lockless
++ PrintError 'ERROR: END'
++ test -s /root/rear.github.master/var/log/rear/rear-linux-h9wr.log.lockless
++ test '  2023-06-06 13:21:40.512275229 Entering debugscript mode via '\''set -x'\''.
  2023-06-06 13:21:40.521085733 '\''grep --Q=... FOO /etc/issue'\'' succeeded'
++ PrintError 'Some latest log messages since the last called script 030_update_recovery_system.sh:'
++ PrintError '  2023-06-06 13:21:40.512275229 Entering debugscript mode via '\''set -x'\''.
  2023-06-06 13:21:40.521085733 '\''grep --Q=... FOO /etc/issue'\'' succeeded'
++ test -f /dev/null
++ test 1
++ test 1
++ Log 'ERROR: END'
...

# grep SECRET var/log/rear/rear-linux-h9wr.log.lockless
[no output]

# usr/sbin/rear -e -D help
Running 'init' stage ======================
ERROR: END
Some latest log messages since the last called script 030_update_recovery_system.sh:
  2023-06-06 13:31:30.203456832 Entering debugscript mode via 'set -x'.
  grep: unrecognized option '--Q=SECRET'
  Usage: grep [OPTION]... PATTERN [FILE]...
  Try 'grep --help' for more information.
  2023-06-06 13:31:30.215654735 'grep --Q=SECRET FOO /etc/issue' succeded
Aborting due to an error, check /root/rear.github.master/var/log/rear/rear-linux-h9wr.log.lockless for details
Exiting rear help (PID 7510) and its descendant processes ...
Running exit tasks
Terminated

# cat var/log/rear/rear-linux-h9wr.log.lockless
...
+ source /root/rear.github.master/usr/share/rear/init/default/030_update_recovery_system.sh
++ grep --Q=SECRET FOO /etc/issue
grep: unrecognized option '--Q=SECRET'
Usage: grep [OPTION]... PATTERN [FILE]...
Try 'grep --help' for more information.
++ true
++ LogSecret ''\''grep --Q=SECRET FOO /etc/issue'\'' succeded'
++ test 1
++ Log ''\''grep --Q=SECRET FOO /etc/issue'\'' succeded'
++ test -w /root/rear.github.master/var/log/rear/rear-linux-h9wr.log.lockless
++ echo '2023-06-06 13:31:30.215654735 '\''grep --Q=SECRET FOO /etc/issue'\'' succeded'
2023-06-06 13:31:30.215654735 'grep --Q=SECRET FOO /etc/issue' succeded
++ Error END
++ test -s /root/rear.github.master/var/log/rear/rear-linux-h9wr.log.lockless
++ PrintError 'ERROR: END'
++ test -s /root/rear.github.master/var/log/rear/rear-linux-h9wr.log.lockless
++ test '  2023-06-06 13:31:30.203456832 Entering debugscript mode via '\''set -x'\''.
  grep: unrecognized option '\''--Q=SECRET'\''
  Usage: grep [OPTION]... PATTERN [FILE]...
  Try '\''grep --help'\'' for more information.
  2023-06-06 13:31:30.215654735 '\''grep --Q=SECRET FOO /etc/issue'\'' succeded'
++ PrintError 'Some latest log messages since the last called script 030_update_recovery_system.sh:'
++ PrintError '  2023-06-06 13:31:30.203456832 Entering debugscript mode via '\''set -x'\''.
  grep: unrecognized option '\''--Q=SECRET'\''
  Usage: grep [OPTION]... PATTERN [FILE]...
  Try '\''grep --help'\'' for more information.
  2023-06-06 13:31:30.215654735 '\''grep --Q=SECRET FOO /etc/issue'\'' succeded'
++ test -f /dev/null
++ test 1
++ test 1
++ Log 'ERROR: END'
...

# grep SECRET var/log/rear/rear-linux-h9wr.log.lockless
++ grep --Q=SECRET FOO /etc/issue
grep: unrecognized option '--Q=SECRET'
++ LogSecret ''\''grep --Q=SECRET FOO /etc/issue'\'' succeded'
++ Log ''\''grep --Q=SECRET FOO /etc/issue'\'' succeded'
++ echo '2023-06-06 13:31:30.215654735 '\''grep --Q=SECRET FOO /etc/issue'\'' succeded'
2023-06-06 13:31:30.215654735 'grep --Q=SECRET FOO /etc/issue' succeded
  grep: unrecognized option '\''--Q=SECRET'\''
  2023-06-06 13:31:30.215654735 '\''grep --Q=SECRET FOO /etc/issue'\'' succeded'
  grep: unrecognized option '\''--Q=SECRET'\''
  2023-06-06 13:31:30.215654735 '\''grep --Q=SECRET FOO /etc/issue'\'' succeded'

jsmeix commented at 2023-06-06 12:23:¶

@codefritzel @rear/contributors
could you please have a look here and
provide feedback what you think about it
and ideally also how it behaves for you.

Nothing is documented yet in "rear help" or "man rear"
until I did some more tests and until I got some feedback
what others think about it and how it behaves for them.

jsmeix commented at 2023-06-14 08:57:¶

@codefritzel @rear/contributors
I would like to merge it tomorrow afternoon
unless there are objections.

After the merge of this one I will replace our current

{ SECRET STUFF ; } 2>/dev/null

code with the new

{ SECRET STUFF ; } 2>>/dev/$SECRET_OUTPUT_DEV

method that is implemented by this pull request
via one or more separated pull request(s) as needed.

---

[Export of Github issue for rear/rear.]