If you're looking at going even deeper into SSH tunneling and port redirection, I recently made The Cyber Plumber's Handbook free: https://github.com/opsdisk/the_cyber_plumbers_handbook

I made it free to the HN community a few years back [1]. There is a paid interactive lab portion (details in the repo) if you are looking for hands-on experience.

Book Overview

This book is packed with practical and real world examples of SSH tunneling and port redirection in multiple realistic scenarios. It walks you through the basics of SSH tunneling (both local and remote port forwards), SOCKS proxies, port redirection, and how to utilize them with other tools like proxychains, nmap, Metasploit, and web browsers.

Advanced topics included SSHing through 4 jump boxes, throwing exploits through SSH tunnels, scanning assets using proxychains and Metasploit's Meterpreter, browsing the Internet through a SOCKS proxy, utilizing proxychains and nmap to scan targets, and leveraging Metasploit's Meterpreter portfwd command.

[1] https://news.ycombinator.com/item?id=19946941

One thing I’ve always tried to find is a method of forwarding a local port on server A to a public facing port on server B. Googling this is very difficult. Is that possible with ssh?

I’ve also had this issue for quite sometime and I’ve used a few approaches over the years. My most recent approach has been to connect server A to server B, via WireGuard. Then on Server B I use nginx to reverse proxy the WireGuard IP address of Server A. This works nicely when: 1) you have access to install software server A, but it’s stuck behind a bunch of firewalls. 2) You’re trying to expose HTTP services- it’s possible to “stream” non-HTTP with nginx, but I don’t know enough to recommend that.

You might also want to look at Apache Guacamole or Boring Proxy. I’d love other recommendations!

socat

Solid book, thanks for sharing.

You can also change/add them after the ssh session has started.

This uses the escape mechanism. For help on that, hit return, type "~?" and hit return again.

From there you will see that "~#" will list them. And "~C" takes you to a line-oriented command prompt where you can add them.

For example, type return "~C", then at the prompt type "-L1234:localhost:22". Then in another window, type "telnet localhost 1234". Then back in the ssh window, type return "~#" and you will see the tunnel being used.

Another trick: if you don't need a shell and only want to forward ports, run "ssh -N" instead of "ssh".

I'd never expected ssh to have an interactive shell. Thanks for sharing that.

Also want to add: you can type "help" at the "ssh>" prompt.

That is, hit enter, type "~C", then prompt appears, then "help" and hit enter again.

I have read far too many ssh tunneling articles and all of them either forget about the bind address or just pretend that the only host that can be tunneled is localhost.

The fact is that ssh tunnels are bound to an interface on one side and the other can be any (reachable) host:port. So if example.com:1234 isn’t reachable from your host network, you can do a local port forward to example.com:1234 from an intermediary host that can reach example.com:1234

https://sshuttle.readthedocs.io/ seems worth mentioning for convenient VPN-like tunnelling.

never knew ssh did tun/tap. live and learn.

my personal preference has been to use ssh -D and tsocks for this. it doesn't require root on either side and tsocks is elegant for inbueing just the processes you want with the ability to use the tunnel.

Proxychains is a good modern (maintained) alternative to tsocks

interesting. never realized that tsocks hasn't seen an update in nearly 20 years. i've never had an issue with it, although i suppose i haven't had to setup my own tunnels in quite some years now.

I wanted to build a TeamViewer-type system using reverse tunnels so that I could access my possibly NAT'd or dynamic IP machines from each other in a simple way. The typical use would be SSH control, copying files each way, VNC. I came up with something where each machine connects to a an always-on server with a domain name, and offers a reverse tunnel I can use to SSH down, but it occurs to me that there might be a better way, or even a tool someone has already build. Does anyone know of something like this?

(pwnat is pretty cool, but I don't think it does quite what I'm looking for)

I use WireGuard with a server in my pantry as a router. Dynamic IP of the server is handled by DuckDNS, and WireGuard gracefully handles client roaming e.g. I can switch from home wifi to mobile internet without interrupting my SSH sessions. Would recommend.

You should check https://sshreach.me - zero-configuration, remote-controlled secure tunnels to your computers. I am the author of the service.

Bookmarked your page. Am building embedded devices and thought about having them keep reverse tunnels to some gateway server. Cool, that there's a dedicated service for that! Have my own domain but am still eager to outsource this part.

Some things to note:

- The sshReach.me logo in the topbar links to https://sshreach.me/init/default/index.html rather than https://sshreach.me. Expected the latter to have a clean URL to bookmark.

- At the very bottom of https://sshreach.me/init/default/index.html, the links {debian,arch,mint,...} link to the distro homepages. I expected links going to distro-specific guides on how to set these distros up to use our service.

- The Yocto project is quite popular for building embedded devices. It would probably push your service, if you would provide yocto recipes which make the created systems maintain reverse ssh tunnels to your service.

Wireguard is my personal preference for this.

There are actually quite a few lovely tutorials for WireGuard out there and in my experience its performance is also far better than that of OpenVPN or other solutions, as well as the configuration is pretty easy.

I actually used it for working around NAT with a cheap VPS which now acts as an ingress and proxies the requests for all of the ports over to one of my homelab servers that i want to expose: https://blog.kronis.dev/tutorials/how-to-publicly-access-you...

(disclaimer: the above blog post is pretty simplistic and shouldn't necessarily be followed to the letter, in most cases you'll want to expose ports on a case by case basis)

For remote shell support tmate is great. You can run your own server or use theirs.

https://tmate.io/

That's where I make use of the "SSH Jump Host" capability. It's an incredible feature that is similarly not well known:

https://wiki.gentoo.org/wiki/SSH_jump_host

Not sure if it is exactly what you are looking for, but I use Remote Support Tool [1] as a free software TeamViewer replacement, and it works perfectly for me.

[1] https://github.com/OpenIndex/RemoteSupportTool

ZeroTier might be of interest.

Using Zerotier for this for quite a while now and can confirm it is working quite nicely.

TailScale might be an interesting alternative. Even more so as with headscale there is an Open Source server side implementation. Have not tested neither of those though.

The blog the article is on is from teleport (goteleport.com), which you might want to take a look at. I haven't used it myself but it could be used for this task.

Check out Tailscale. It's magic and free.

https://tailscale.com/

Tailscale?

Take a look at Cloudflare Tunnels.

Excellent article!

What’s also very interesting is that the article links to page from TrendMicro about malicious Android apps using Java’s version of SSH to infiltrate internal corporate networks.

TrendMicro’s own Android app ALSO contained the same Java SSH sdk.

For most use cases, you should also add a local bind, "-Llocalhost:5900:localhost:5900" or else everyone at the coffee shop will be able to access your insecure Vnc server

Seems to be the default for me.

The manual page says, "By default, the local port is bound in accordance with the GatewayPorts setting."

I don't see GatewayPorts in any of the config files, so the binary (at least mine) is probably built to have this off by default. Which seems like a sensible setting.

EDIT: It can't hurt to be sure, though. Especially if you're running ssh commands on different computers or writing a script that runs ssh.

At work, we have a handful of VPCs that we all work with. At the moment, we have a bastion host in every VPC. When something needs attention while on call, the engineer needs to first figure out which bastion host to ash into and then the actual work starts.

I was wondering if there is a better way to setup a central bastion host with RBAC such that the attack vector is also not centralised. Does anyone here have ideas?

At my previous work, we had that centralized bastion for the engineers (we were 5, and it might be harder with a bigger team). Only ssh with rsa, and the rsa key was generated with yubico tools, putting the private key inside our yubikey, and protecting access to said key with a password.

Then basic RBAC with sudoer file according rights depending on your role. The only root account was accessible through two locked up yubikeys, and the passwords of those were in a password manager owned by the architect and manager.

When i left, we were starting a V2 on this with internal LDAP for server/proxy access, first for us then for our clients.

Have a look at https://docs.aws.amazon.com/systems-manager/latest/userguide... - in a lot of cases removes the need for a bastion and SSH keys at all

I have been looking into SSM recently and I was a little confused by the setup instructions but after seeing your comment I read them again and I think I understand more now. I was trying to see how SSM could be used to eliminate the need for engineers to have SSH keys set up with instances.

  > Who should use Session Manager?  
  > ...  
  > Users who want to connect to an instance with just one click from the browser or AWS CLI without having to provide SSH keys.

Perfect, how do I get started?

  > Step 8: (Optional) Enabling and controlling permissions for SSH connections through Session Manager  
  > ...  
  > Create or verify that you have a Privacy Enhanced Mail certificate (a PEM file), or at minimum a public key, to use when establishing connections to managed instances. This must be a key that is already associated with the instance.

Uh oh, what's going on?

Well, I think I understand now that using SSH with SSM is optional. You can use the AWS Console or the AWS CLI to connect to a shell on the instance without using SSH. Then you don't need keys and even grants additional benefits like the ability to shell into instances without a public IP and without opening any ports. However, SSM+SSH has some advantages like SCP which I don't believe is supported by SSM alone.

Another thing I've been looking into is AWS CloudShell which is free. When opening CloudShell essentially a virtual instance is created within your VPC with Amazon Linux installed. You can then use this shell to SSH into instances. I haven't looked into it much because I would rather use SSM but I believe this could be used to essentially be an ephemeral bastion that is secure in that it doesn't accept public requests, it can only be accessed via the Console.

The trick is to mix SSM with EC2 Instance Connect using the `aws ec2-instance-connect send-ssh-public-key` command.

We use bastions to connect to RDS instances. The bastions aren't accessible from the internet; only via SSM. You can wrap up all of the steps in a shell script that calls `ssh`, or with a bit more effort, concoct a ProxyCommand script that does everything for you and makes e.g. `ssh aws-bastion` just work.

We have a script used as an SSH ProxyCommand that:

1) queries EC2 to find a bastion host based on tags (the bastions are in an ASG and can change)

2) generates an SSH key

3) adds the generated private key to ssh-agent temporarily (using the `-t` parameter to `ssh-add`)

4) sends the generated public key to the selected host using ec2-instance-connect

5) starts an SSH session using `ssm start-session`

Then a `~/.ssh/config` entry that intercepts connections for host `aws-bastion` and specifies the ProxyCommand (as well as keepalive and ControlMaster to make subsequent connections fast).

Adding the key to the agent temporarily is a trick since there's no other way to pass information from a proxy command to the outer `ssh` process, and I couldn't find any other hook. I've found at least one instance where that trick doesn't work: when connecting to a database from within IntelliJ's database tools. For that, I added an option to the proxy command script to pick a key already registered in the agent rather than generating a new one (e.g. `ssh-add -L | head -1`).

You are starting an SSM session yet using SSH keys? Can you explain that more?

SSM has StartSession which drops you into a shell, and StartSSHSession which is the equivalent of opening a tcp connection to an ssh server. When used as a ProxyCommand, the latter lets you do everything you'd normally so with an ssh connection, including tunneling. But you still need keys or some other auth mechanism.

Using CloudShell / the AWS Console for anything involved sounds painful though, among other reasons due to the 12-hour console session timeout. I suppose you might be able to use screen to mitigate it, but getting logged out right in the middle of doing something important seems unpleasant. The same thing makes AWS Sagemaker Studio unusable.

You can use the CLI and your terminal directly, as well as use SSH over SSM for proxying and SCP.

I’d highly recommend using the ProxyJump directive in your ssh configuration file to avoid the need to track which host goes where manually.

You can try EC2 Instance Connect, but I remember there being some use cases it won't work for (but I don't remember what they are).

You can use SSH Certificates, but management is kind of a pain. Not only the certs, but to do quasi-RBAC you'd need to configure the host to accept/deny certain certs for certain users to certain things, but that's still not really RBAC.

You could set up your hosts to use an LDAP server, but the LDAP server would be the central attack vector.

Honestly, Teleport is probably the only thing that provides a complete solution for what you want and more. I have tried to get my company to pay for it because it would save us so much time trying to cobble together something that's almost the same, but they balked.

This whole situation is really interesting to me. Nobody has invented a piece of technology to solve the problem we really want solved. We don't really want to "port forward through a bastion host", we want to grant specific users limited access to a private network. I think what would solve this is an SSO-authenticated Wireguard that can forward to arbitrary Wireguards - does anyone know if that's a thing?

Since it’s the author of this original post, have you looked into Teleport? It’s a pretty slick solution for the bastion space.

So, I’m kind of dumb and this article helped me in a tangible way. When I work with jupyter notebooks on an ec2 (where I already have ssh access), I go into terraform or the console and add my personal ip to allowed ips in the security group, and then access the notebook via EC2_PUBLIC_IP:JUPYTER_PORT in my browser. This is annoying in particular, in addition to needing to add an IP, in that when I move from the office to home or vice versa there’s always a brain fart where I cannot access because I’m on a new ip.

After reading this article I started a port forwarding session and can access the remote notebook without issue. All it took was the ssh port forwarding. Amazing.

>At first I opened a router port but the RPI showed access attempts every few minutes, which was troubling. Learned enough about ssh tunneling to setup a systemd service to auto-attach the RPI to an AWS instance.

What was the gain here exactly? The same port scanners are hitting your AWS instance now. If it concerned you before with the RPi (which it shouldn’t btw), I don’t see why it wouldn’t concern you with the AWS instance as well…

I’m currently in a low-user situation and can lock that down AWS instances by IP.

My concern is that I'm running an experiment and don't want things corrupted by some script kiddies simply because I'm not a great network/Linux admin. This merely reflects my lack of knowledge... hence my ask. (Which I guess people hate enough to downvote? haha ok)

That’s pretty simple to do with Linux too… Just install ufw and set it to deny by default, then open the SSH port to only your IP. No need to go through AWS for that.

Didn't know about ufw, but I'm not on the same network as the RPI and don't have a static IP. And the RPI was already uploading data to an AWS instance.

Sorry, then I don’t know what you’ve meant by “lock that down AWS instances by IP”.

Using an AWS security group. I'm a basic b** as the kids say.

And remove the word "that".

> I setup an RPI in a field at my dad’s w/ 900MHz internet bridge and wanted 24/7 remote access but didn’t want to pay for and configure VPN.

Hah, same. Didn't really go that deep into the failed login attempts though: private key authentication + I have a static IP so iptables is sufficient. In addition I have a script that sends a text to my phone each time someone logs into any of my computers/servers(even if it's me) and if not, I have one or two kill switches at my disposal.

Wireguard is great. Checkout Tailscale to manage key exchange for you automagically. I just set it up across my home network and devices and it’s shockingly easy. For personal/hobby projects it’s free.

Interesting read. Funny to see this on HN today while I’m reading up on Teleport anyway.

This is a great post, intriguing.