#1512 Issue closed: Security Vulnerability (Gain Information): SSH private host key disclosure

Labels: fixed / solved / done, critical / security / legal

OliverO2 opened issue at 2017-09-22 19:12:

Relax-and-Recover 2.2 (ReaR) copies SSH private host keys from /etc/ssh to the rescue medium. These keys protect incoming SSH connections against man-in-the-middle attacks.

On the original system, the above private keys are only accessible with root user privileges. SSH host keys are never encrypted with a passphrase.

The ReaR rescue medium, which may be a DVD or a USB stick, grants complete unauthenticated access to all files, including copied private key files. Unless changed by some action outside of ReaR, these private keys remain valid for the original system, during recovery and on the restored system.

Attack Vector

An attacker with physical access to the rescue medium can steal the private host keys.

If the attacker obtains network access, he can then use a man-in-the-middle attack to read and alter SSH traffic between a network client and the compromised host.

Consequences

A successful attack allows an attacker to steal passwords and to read and alter sensitive information.

OliverO2 commented at 2017-09-23 14:07:

Suggestions

For ReaR Users

Make sure that each ReaR-generated rescue medium is stored safely and is accessible to authorized persons only. Guard rescue media like you would guard any non-encrypted backup medium.

Note: This applies to all rescue media previously generated by ReaR even when the issue will eventually be addressed in ReaR.

You can harden current Rear installations by using this line in your configuration file,

COPY_AS_IS_EXCLUDE=( "${COPY_AS_IS_EXCLUDE[@]}" /etc/ssh/ssh_host_* /root/.ssh/id_* )

but

  • make sure that it works as expected on your installation, and
  • be aware that you cannot login into the rescue system via SSH unless SSH host keys will be regenerated as shown in the developer suggestions below, and
  • be aware that you might be required to enter passwords for SSH connections during restore:

Once ReaR has been hardened as shown above or if the issue has been addressed in ReaR and the corresponding ReaR version has been installed: Safely overwrite all previously generated rescue media, e.g. by using

dd bs=1M if=/dev/urandom of="<path-to-rescue-medium>" status=progress

If a ReaR rescue medium has potentially been accessible to unauthorized persons:

  • For each host potentially compromised by ReaR rescue media having been accessible
    • On the compromised host
      • Change SSH host keys (/etc/ssh/ssh_host_*_key)
      • Change SSH root user keys (/root/.ssh/id_*) on the compromised host
    • On each network node
      • Delete the compromised host's entries from /etc/ssh/ssh_known_hosts and each */.ssh/known_hosts file
      • Delete the compromised host's root user entries from each */.ssh/authorized_keys file
      • Re-establish known host identification and root user authorization where required

For ReaR Developers

Note that implementing these suggestions will have the following consequences:

  • Initially connecting to the rescue system via SSH login wiill show a warning like this:
The authenticity of host 'server (192.168.1.5)' can't be established.
ECDSA key fingerprint is b5:0e:ec:b7:16:06:e6:24:a6:39:18:58:4e:ec:3b:d1.
Are you sure you want to continue connecting (yes/no)?
  • Connecting via SSH from the rescue system to other hosts (e.g. for restoring a backup) will generally require the user to enter a password, as SSH public/private key authentication will not be established. (Public/private key authentication is more suited for long-term trust relations between secure systems. It is ill-suited for temporary trust relations like the one involving a rescue system as public/private key authentication has to be explicitly revoked, which can easily fail or be forgotten.)

Suggestions:

  1. Avoid copying sensitive SSH files:
    • From /etc/ssh, copy only
      • moduli
      • ssh_config
      • sshd_config
      • ssh_known_hosts
    • From /root/.ssh, copy only
      • authorized_keys
      • known_hosts
  2. Re-generate SSH host keys for each rescue medium in the build stage, for example:
Log "Generating SSH host key"
ssh-keygen -t ed25519 -N '' -f "$ROOTFS_DIR/etc/ssh/ssh_host_ed25519_key"

jsmeix commented at 2017-09-25 08:02:

We (i.e the ReaR upstream authors and maintainers)
need to get this security issue fixed before the ReaR 2.3 release.

jsmeix commented at 2017-09-25 08:16:

@OliverO2
many thanks for your careful study how ReaR works
that reveals such security issues.

I am not at all an expert in this area but I think the
following code pieces cause such insecure default behaviour:

rear.master # find usr/sbin/rear usr/share/rear/ -type f | xargs grep 'COPY_AS_IS' | grep ssh | tr ':' '\n'

usr/share/rear/rescue/default/550_vagrant.sh
    COPY_AS_IS=( ${COPY_AS_IS[@]} $homedir/.ssh ) 

usr/share/rear/rescue/default/500_ssh.sh
    COPY_AS_IS=( "${COPY_AS_IS[@]}" /etc/ssh* /root/.s[s]h /root/.shos[t]s )

usr/share/rear/verify/YUM/default/050_check_YUM_requirements.sh
        COPY_AS_IS=( "${COPY_AS_IS[@]}" $HOME/.ssh )

usr/share/rear/prep/NETFS/default/050_check_NETFS_requirements.sh
        COPY_AS_IS=( "${COPY_AS_IS[@]}" $HOME/.ssh )

schlomo commented at 2017-09-25 08:52:

@jsmeix vagrant is only used for test and development, I don't see any issue with that ($homedir means the vagrant user).

I hope that #1513 solves this issue sufficiently well. If somebody has an idea about adding there more SSH keys to inspect then please let me know.

Can we all agree that in order to address this vulnerability it is enough to change the default behavior of ReaR so that it won't copy unprotected SSH keys in typical locations?

schlomo commented at 2017-09-25 09:42:

Concerning usr/share/rear/prep/NETFS/default/050_check_NETFS_requirements.sh I would say that this is just wrong because it copies the .ssh folder from the user running ReaR which is always root. Since we handle that already in usr/share/rear/rescue/default/500_ssh.sh I would suggest to remove that line altogether.

I'll add those to my PR.

schlomo commented at 2017-09-25 10:38:

@gozora suggests to change our definition of the rescue image to say that it contains secrets and that is must be protected like the backup itself.

Consequences of such a decision are amongst others that this PR is only a nice feature and doesn't address the vulnerability. The actual "solution" to the vulnerability is simply declaring the rescue image as a secret.

This approach would of course also solve #1512

jsmeix commented at 2017-09-25 11:04:

I agree that it is enough to change the default behavior of ReaR
so that it won't copy unprotected SSH keys in typical locations.
The user can still copy what he likes via COPY_AS_IS.

In contrast to @gozora I would prefer to keep
the default recovery system free of secrets.

But I think it should be documented that it can easily happen
that the recovery system contains (unexpectedly) some secrets
if the user is not careful so that it is recommended
to protect a recovery system ISO image or medium
to be on the safe side.

I think normally this should be a matter of course
but to be on the safe side it should be documented.

Don't take things carelessly out of your datacenter into your next pub:
https://github.com/rear/rear/pull/1472#issuecomment-328459748

schlomo commented at 2017-09-25 12:56:

I asked the rear-users mailing list: http://lists.relax-and-recover.org/pipermail/rear-users/2017-September/003474.html

OliverO2 commented at 2017-09-25 12:58:

I did not have sufficient time today to look closely at everything, but some thoughts on dealing with the issue via rescue media handling recommendations:

  • I agree that one should never restore from a rescue medium, which might have been compromised. So it's a must to protect rescue media.
  • Rescue media can be used for testing recovery, testing migration, actual migration and to restore production systems. In some of these scenarios, administrators might be under time pressure and could just forget a rescue medium.
  • A rescue medium cannot be made as safe as backup storage, because at some point the rescue medium must travel to the point of restoration in bootable form. In contrast, a backup might reside safely on an encrypted disk or a secure SSH-accessible network server at all times.

So I'd argue that making the rescue medium not leak secrets is necessary even if administrators do their best to keep those safe and ensure that they have not been tampered with before recovery.

OliverO2 commented at 2017-09-25 13:13:

When I re-read my argumentation above, I think it might not be clear enough:

  • A rescue medium must be guarded from the time it is created on insecure storage until the time is is used. This makes sure the system is restored to a safe state and the restoration has not been tampered with.
  • A rescue medium might be forgotten afterwards. Even if is is never used for restoration again, it still presents a threat if it contains secrets for anyone to copy.

jsmeix commented at 2017-10-09 12:05:

With https://github.com/rear/rear/pull/1513 merged
I assume that also this issue is sufficiently fixed.
Regarding what "sufficiently" means, cf.
https://github.com/rear/rear/pull/1513#issuecomment-332413380

But I am not an expert here so that I could be wrong.
@OliverO2 in this case please re-open this issue.

OliverO2 commented at 2017-10-09 13:18:

@jsmeix
Having looked at the changes, #1513 does not address private host keys. So this issue should be re-opened. Looks like I don't have sufficient privileges to do that, so @jsmeix, would you please step in?

jsmeix commented at 2017-10-10 07:45:

Reopening according to
https://github.com/rear/rear/issues/1512#issuecomment-335155392

jsmeix commented at 2017-10-10 09:50:

I think with https://github.com/rear/rear/pull/1513 merged
this issue here got much simplified compared to
https://github.com/rear/rear/issues/1512#issuecomment-331809122
I get now:

# find usr/sbin/rear usr/share/rear/ -type f | xargs grep 'COPY_AS_IS' | grep -i 'ssh' | tr ':' '\n'

usr/share/rear/rescue/default/500_ssh.sh
COPY_AS_IS=( "${COPY_AS_IS[@]}" /etc/ssh* /etc/openssh* /etc/centrifydc/ssh* /root/.s[s]h /root/.shos[t]s )

Accordingly I think an enhancement in rescue/default/500_ssh.sh
so that the user can specify what ssh stuff should be copied
would fix (or at least sufficiently mitigate) this issue here.

As far as I can understand it (but I am not at all an expert
in this area) it seems this means I need to implement
at least in some initial basic way a SSH_KEYS config
variable like described in
https://github.com/rear/rear/pull/1513#issuecomment-332123766

OliverO2 commented at 2017-10-10 12:16:

@jsmeix See section For ReaR Developers in https://github.com/rear/rear/issues/1512#issuecomment-331638066. Note that

  • host keys (/etc/ssh/ssh_host_*) are never password protected,
  • if host keys are missing, you cannot log into the rescue system via ssh.

So if the original system's host keys are not copied to the rescue system and ssh login is desired, a host key must be generated for the rescue system.

A host key can be generated

  1. during the rescue system boot (e.g. when sshd is started), which would be the most secure way possible (an attacker cannot know the key in advance), or
  2. by rear mkrescue (in which case an attacker getting hold of the rescue medium can still attack the rescue operation).

Both cases ensure that an attacker with access to the rescue medium can not attack the original system.

jsmeix commented at 2017-10-10 14:11:

@OliverO2 I saw your
https://github.com/rear/rear/issues/1512#issuecomment-331638066
and I tried to implement it in
https://github.com/rear/rear/pull/1530
where I create host keys anew during "rear mkrescue" in
build/default/500_patch_sshd_config.sh
(probably that script should be renamed if we keep it there).

jsmeix commented at 2017-10-17 13:57:

With https://github.com/rear/rear/pull/1530 merged
I think this issue is sufficiently fixed.

Now by default only with a

SSH_ROOT_PASSWORD="rear"

in local.conf one gets a recovery system without
(unprotected) keys and suring startup of the recovery system
an rsa SSH host key is generated so that remote access to
the recovery system via SSH (sshd) works.

What I have on the original system:

# find /etc/ssh* /root/.ssh/

/etc/ssh
/etc/ssh/ssh_host_rsa_key
/etc/ssh/ssh_host_dsa_key.pub
/etc/ssh/moduli
/etc/ssh/ssh_host_ecdsa_key
/etc/ssh/ssh_host_key
/etc/ssh/ldap.conf
/etc/ssh/ssh_config
/etc/ssh/ssh_host_dsa_key
/etc/ssh/ssh_host_key.pub
/etc/ssh/ssh_host_ecdsa_key.pub
/etc/ssh/sshd_config
/etc/ssh/ssh_host_ed25519_key.pub
/etc/ssh/ssh_host_rsa_key.pub
/etc/ssh/ssh_host_ed25519_key
/root/.ssh/
/root/.ssh/id_rsa.pub
/root/.ssh/id_dsa.pub
/root/.ssh/id_rsa
/root/.ssh/id_dsa
/root/.ssh/known_hosts

What there is in the recovery system ISO image in this case:

# find /tmp/rear.9eLkQrXtl5KvogM/rootfs/etc/ssh* /tmp/rear.9eLkQrXtl5KvogM/rootfs/root/.ssh/

/tmp/rear.9eLkQrXtl5KvogM/rootfs/etc/ssh
/tmp/rear.9eLkQrXtl5KvogM/rootfs/etc/ssh/moduli
/tmp/rear.9eLkQrXtl5KvogM/rootfs/etc/ssh/ssh_config
/tmp/rear.9eLkQrXtl5KvogM/rootfs/etc/ssh/sshd_config
/tmp/rear.9eLkQrXtl5KvogM/rootfs/root/.ssh/
/tmp/rear.9eLkQrXtl5KvogM/rootfs/root/.ssh/known_hosts

What there is in the booted recovery system in this case:

RESCUE e205:~ # find /etc/ssh* /root/.ssh/

/etc/ssh
/etc/ssh/sshd_config
/etc/ssh/ssh_host_rsa_key.pub
/etc/ssh/ssh_host_rsa_key
/etc/ssh/ssh_config
/etc/ssh/moduli
/root/.ssh/
/root/.ssh/known_hosts

jsmeix commented at 2017-11-29 16:17:

FYI:

Interestingly on my SLES12 test system only

SSH_ROOT_PASSWORD="rear"

"just works" to get ssh access to the recovery system
via a new from scratch generated host key there.

In contrast on my SLES11 test system that alone
does not work and also

SSH_ROOT_PASSWORD="rear"
SSH_FILES='yes'

does not work and the recovery system login screen shows

Could not load host key: /etc/ssh/ssh_host_rsa_key
Could not load host key: /etc/ssh/ssh_host_dsa_key
Could not load host key: /etc/ssh/ssh_host_ecdsa_key
Could not load host key: /etc/ssh/ssh_host_ed25519_key

On SLES11 I must use

SSH_ROOT_PASSWORD="rear"
SSH_UNPROTECTED_PRIVATE_KEYS='yes'

(SSH_FILES='yes' is not needed)
to get ssh access to the recovery system.

I.e. on SLES11 only the old ReaR behaviour works were
the keys from the original system are in the recovery system.

jsmeix commented at 2017-11-29 16:38:

It seems on SLES11 the script
/usr/share/rear/skel/default/etc/scripts/run-sshd
is not run automatically during recovery system startup.
When I run it manually in the recovery system
I get ssh access to the recovery system same as in SLES12
via a the generated /etc/ssh/ssh_host_rsa_key
by /usr/share/rear/skel/default/etc/scripts/run-sshd
and in this case /bin/sshd only reports

Could not load host key: /etc/ssh/ssh_host_dsa_key
Could not load host key: /etc/ssh/ssh_host_ecdsa_key
Could not load host key: /etc/ssh/ssh_host_ed25519_key

OliverO2 commented at 2017-11-29 16:41:

@jsmeix I wonder why setting SSH_UNPROTECTED_PRIVATE_KEYS would change whether the startup script runs or not...

If there is indeed an issue with an older sshd, could you replace line 9 in usr/share/rear/skel/default/etc/scripts/run-sshd, which contains

        ssh-keygen -t rsa -N '' -f /etc/ssh/ssh_host_rsa_key

with this content

        ssh-keygen -A

and try again if that works on both of your systems?

Maybe your older sshd is a bit too disappointed if there are some host keys configured in /etc/ssh/sshd_config without their key files being present.

jsmeix commented at 2017-11-29 16:45:

I will try that but when I run
/usr/share/rear/skel/default/etc/scripts/run-sshd
manually the sshd only reports the other missing keys but
works with the only generated /etc/ssh/ssh_host_rsa_key

OliverO2 commented at 2017-11-29 17:03:

Ah, seems like SLES11 is init-based while SLES12 is systemd-based. On init-based systems, ReaR will run sshd directly from the inittab while run-sshd will only be launched via systemd. This could be unified:

You could try to

  • delete the if grep ... and fi lines in run-sshd and
  • change line 85 in usr/share/rear/rescue/default/500_ssh.sh to:
echo "ssh:23:respawn:/etc/scripts/run-sshd" >>$ROOTFS_DIR/etc/inittab

Would that help?

jsmeix commented at 2017-11-30 14:05:

@OliverO2
late reply because I was busy with other stuff:
Yesterday when leaving the office I came to the same conclusion.
Setting SSH_UNPROTECTED_PRIVATE_KEYS does not change
whether /etc/scripts/run-sshd runs or not but with keys in the
recovery system starting plain sshd via /etc/inittab is sufficient
to get ssh access.
Strictly speaking my changes in this pull request did not actually
break something that had worked before (before it had worked
more or less "by accident" in an insecure way).
All what is needed is another small adjustment as you described in
https://github.com/rear/rear/issues/1512#issuecomment-347927227
to make the new rsa key generation in the recovery system
also work in case of traditional SysVinit.
I principle your
https://github.com/rear/rear/issues/1512#issuecomment-347927227
works and what I currently do is to make the recovery system
login screen look nice again even in case of rsa key generation.

As always many thanks for your prompt help.
It is much appreciated!

jsmeix commented at 2017-11-30 14:15:

@OliverO2
I have a question:
I wonder why sshd is started in /etc/scripts/run-sshd as

exec /bin/sshd -D

i.e. sshd will not detach and does not become a daemon.
I think this was because /etc/scripts/run-sshd was
up to now used only for systemd.
I guess when we use now in /etc/inittab for SysVinit

ssh:23:respawn:/etc/scripts/run-sshd

then sshd should probably better detach and become a daemon.
I hope that it does not really matter for systemd
where the same /etc/scripts/run-sshd is run
via /usr/lib/systemd/system/sshd.service
when sshd also detaches and becomes a daemon.

jsmeix commented at 2017-11-30 14:23:

Strange:
Without '-D' I gets at least on SLES11 with SysVinit

INIT: Id "ssh" respawning too fast: disabled for 5 minutes

so that I "just keep" the '-D' because this way it works for me.

jsmeix commented at 2017-11-30 15:00:

https://github.com/rear/rear/pull/1612 should fix it
and I even found an explanation what init in SysVinit actually does
for inittab entries with 'respawn' when that thingy daemoinzes itself:
https://www.linuxquestions.org/questions/linux-newbie-8/starting-daemon-via-inittab-respawn-causes-respawning-too-fast-disabled-for-5-minutes-4175579263/

OliverO2 commented at 2017-11-30 15:20:

@jsmeix No problem with taking your time for replies. Sure we all have other stuff going on...

The respawn directive in the inittab entry tells init to monitor the process and restart it if it should terminate. Init can only monitor the process it has started, as init has no way of knowing which one of possibly multiple child processes would be the "right" one to monitor.

So the process started by init's respawn directive must avoid forking (damonize) as init will have taken care of this. Otherwise init would recognize a process which is quickly terminating (after forking) and will try to restart it immediately. This is what happened when you saw respawning too fast. For details, see inittab (5).

Systemd can be configured to behave similarly via Type=simple (or its variants) and Restart=on-failure in systemd.service. But systemd has way more capabilities than init to monitor processes.

I was aware that you had already found an explanation, but I'm posting this in the hope that this shorter one might benefit others researching the topic.

OliverO2 commented at 2017-11-30 15:49:

BTW, I'm writing this from a perfectly recovered and migrated development system. Tested with Relax-and-Recover 2.2 / Git as of commit a0dcacf173349f6516bf59b5c6bd908e6c27044f (Nov 29, 2017) on Ubuntu 16.04.3 LTS/i386 with UEFI boot and upcoming RAWDISK output method.

Lots of thanks to everyone contributing to this useful project!

jsmeix commented at 2017-11-30 16:20:

@OliverO2
you do no longer have to use any special adaptions to ReaR
(i.e. just the current master code as it falls out of Git)
except what is needed for the RAWDISK output method?

OliverO2 commented at 2017-11-30 17:13:

@jsmeix I'm using this configuration:

OUTPUT=RAWDISK
OUTPUT_URL="file:///mnt/reserve/Transfer/Rear.nobackup/RAWDISK"

OPALPBA_URL="file:///mnt/reserve/Transfer/Rear.nobackup/OPALPBA"

# To generate a password, use mkpasswd -m sha-512
SSH_ROOT_PASSWORD='...'

BACKUP=INTERNAL

Plus some extra configuration lines and code for my INTERNAL backup method (not requiring changes to other ReaR code).

The git commit I mentioned was off by one. I actually had been using d798fdb39bb6d0a111d4655283ad28d91286222e. I'm using unmodified ReaR code from this repository with two exceptions (very small changes in two files related to enabling another workflow for TCG Opal support). On top of this I'm using extra code for the RAWDISK output method and TCG Opal support.

Are you referring to something specific, e.g. is there something that you'd expect not to work currently?

jsmeix commented at 2017-12-01 08:31:

@OliverO2
I only liked to get your confirmation that my SSH related
changes, adaptions, and enhancements "just work" for you.
My selfish resoning behind was to make myself a bit proud
of what I (still mostly a SSH noob) implemented ;-)
Seriously: Without all your patient and explanatory help
this would not have been possible so that actually
I am much more thankful than proud of myself.

[Export of Github issue for rear/rear.]