At my previous work, we had that centralized bastion for the engineers (we were 5, and it might be harder with a bigger team). Only ssh with rsa, and the rsa key was generated with yubico tools, putting the private key inside our yubikey, and protecting access to said key with a password.

Then basic RBAC with sudoer file according rights depending on your role. The only root account was accessible through two locked up yubikeys, and the passwords of those were in a password manager owned by the architect and manager.

When i left, we were starting a V2 on this with internal LDAP for server/proxy access, first for us then for our clients.

Have a look at https://docs.aws.amazon.com/systems-manager/latest/userguide... - in a lot of cases removes the need for a bastion and SSH keys at all

I have been looking into SSM recently and I was a little confused by the setup instructions but after seeing your comment I read them again and I think I understand more now. I was trying to see how SSM could be used to eliminate the need for engineers to have SSH keys set up with instances.

  > Who should use Session Manager?  
  > ...  
  > Users who want to connect to an instance with just one click from the browser or AWS CLI without having to provide SSH keys.

Perfect, how do I get started?

  > Step 8: (Optional) Enabling and controlling permissions for SSH connections through Session Manager  
  > ...  
  > Create or verify that you have a Privacy Enhanced Mail certificate (a PEM file), or at minimum a public key, to use when establishing connections to managed instances. This must be a key that is already associated with the instance.

Uh oh, what's going on?

Well, I think I understand now that using SSH with SSM is optional. You can use the AWS Console or the AWS CLI to connect to a shell on the instance without using SSH. Then you don't need keys and even grants additional benefits like the ability to shell into instances without a public IP and without opening any ports. However, SSM+SSH has some advantages like SCP which I don't believe is supported by SSM alone.

Another thing I've been looking into is AWS CloudShell which is free. When opening CloudShell essentially a virtual instance is created within your VPC with Amazon Linux installed. You can then use this shell to SSH into instances. I haven't looked into it much because I would rather use SSM but I believe this could be used to essentially be an ephemeral bastion that is secure in that it doesn't accept public requests, it can only be accessed via the Console.

The trick is to mix SSM with EC2 Instance Connect using the `aws ec2-instance-connect send-ssh-public-key` command.

We use bastions to connect to RDS instances. The bastions aren't accessible from the internet; only via SSM. You can wrap up all of the steps in a shell script that calls `ssh`, or with a bit more effort, concoct a ProxyCommand script that does everything for you and makes e.g. `ssh aws-bastion` just work.

We have a script used as an SSH ProxyCommand that:

1) queries EC2 to find a bastion host based on tags (the bastions are in an ASG and can change)

2) generates an SSH key

3) adds the generated private key to ssh-agent temporarily (using the `-t` parameter to `ssh-add`)

4) sends the generated public key to the selected host using ec2-instance-connect

5) starts an SSH session using `ssm start-session`

Then a `~/.ssh/config` entry that intercepts connections for host `aws-bastion` and specifies the ProxyCommand (as well as keepalive and ControlMaster to make subsequent connections fast).

Adding the key to the agent temporarily is a trick since there's no other way to pass information from a proxy command to the outer `ssh` process, and I couldn't find any other hook. I've found at least one instance where that trick doesn't work: when connecting to a database from within IntelliJ's database tools. For that, I added an option to the proxy command script to pick a key already registered in the agent rather than generating a new one (e.g. `ssh-add -L | head -1`).

You are starting an SSM session yet using SSH keys? Can you explain that more?

SSM has StartSession which drops you into a shell, and StartSSHSession which is the equivalent of opening a tcp connection to an ssh server. When used as a ProxyCommand, the latter lets you do everything you'd normally so with an ssh connection, including tunneling. But you still need keys or some other auth mechanism.

Using CloudShell / the AWS Console for anything involved sounds painful though, among other reasons due to the 12-hour console session timeout. I suppose you might be able to use screen to mitigate it, but getting logged out right in the middle of doing something important seems unpleasant. The same thing makes AWS Sagemaker Studio unusable.

You can use the CLI and your terminal directly, as well as use SSH over SSM for proxying and SCP.

I’d highly recommend using the ProxyJump directive in your ssh configuration file to avoid the need to track which host goes where manually.

You can try EC2 Instance Connect, but I remember there being some use cases it won't work for (but I don't remember what they are).

You can use SSH Certificates, but management is kind of a pain. Not only the certs, but to do quasi-RBAC you'd need to configure the host to accept/deny certain certs for certain users to certain things, but that's still not really RBAC.

You could set up your hosts to use an LDAP server, but the LDAP server would be the central attack vector.

Honestly, Teleport is probably the only thing that provides a complete solution for what you want and more. I have tried to get my company to pay for it because it would save us so much time trying to cobble together something that's almost the same, but they balked.

This whole situation is really interesting to me. Nobody has invented a piece of technology to solve the problem we really want solved. We don't really want to "port forward through a bastion host", we want to grant specific users limited access to a private network. I think what would solve this is an SSO-authenticated Wireguard that can forward to arbitrary Wireguards - does anyone know if that's a thing?

Since it’s the author of this original post, have you looked into Teleport? It’s a pretty slick solution for the bastion space.