How to access a Private EKS Cluster from your local machine without a VPN

There are many ways to access a Kubernetes private cluster. My preferred way is the GKE one, on which the API endpoint is public but you whitelist your machine to be able to connect to it.
You can do it with a simple gcloud command and that is it. You can even automate your pipelines to do it, then remove the whitelisted IP when ready.

Both Azure and AWS do not do it this way when you create a private cluster inside a Private Network, it is really private… and it becomes painful. Hey, in Azure I even think their own Azure Portal cannot see the workloads. WTF.

The most preferred-by-infosec way of doing it is by using a VPN. Which yes, works, yet is another thing to take care of and managed VPNs are not cheap.

The general popular idea is to set up a Bastion Host (usually a VM) which is exposed to the internet (hey, this sounds great!) and then you can SSH into the Bastion to run kubectl commands in there:

PLEASE, DON’T DO THIS. Why would you create a private cluster if you are going to put a freaking VM with port 22 open in front of it…
::insert-facepalm-emoji-here:: It is nonsense!!!

Yet many people do it.

There are options like gototeleport.io which allow you to securely SSH into the Bastion host with MFA, or even kubectl directly to the cluster, skipping the Bastion at all. But it is really expensive. (I save you the Google search, $100/m per cluster)

Couldn’t be worse eh?

It gets even worse. I’ve seen some architectures that looks like this:

So now you have two SSH jumps. One into the “public” JumpBox, then from there, you SSH into a Bastion host located inside the private network, from which you can run kubectl commands.

This architecture is bad for many reasons. First, you are still exposing a VM with port 22 open. It also promotes using the JumpBox and the Bastion as “trash boxes” where your Kubernetes YAML files will lie forever.
Finally, I certainly believe this architecture should not exist. The proper way of having a truly private cluster is to make it private first, then only access it by using pipelines. If you really need to kubectl into it, then it is a DEVELOPMENT cluster, and whitelisting IPs or using a VPN are the best options.

How to jump over the crap

Now, let’s return to the issue, the architecture exists, it is a development cluster, it is real, and it is my problem now. I want to run kubectl from my machine, where I have my IDE and my stuff.

SSH and ~/.ssh/config to the rescue!

With SSH, you can create a proxy and forward all connections to a certain host, but in this case, we have a double SSH jump to do, so, in order to make it easy and readable, we can use the ~/.ssh/config file like this:

Host jumpbox
  HostName x.x.x.x    #[this is the public ip]   
  User myuser
  IdentityFile ~/.ssh/my-key.key

Host bastion
  HostName a.b.c.d    #[this is the private ip]
  User mybastionuser  #ec2-user in my case  
  ProxyJump jumpbox

Now, you can ssh bastion and you should be able to login to the Bastion server.

If you get a public_key denied message, make sure your machine’s public key is in the authorized_keys of the Bastion, not only on the JumpBox

How to make kubectl work over an SSH proxy?

In order to make kubectl work over a proxy, we need to do a few quick things:

• Copy the ~/.kube/config data you need from the Bastion to your local machine.
• If you are using AWS RBAC make sure aws cli is set up in your local machine, then follow this tutorial.
• Create an SSH proxy with ssh -D 1080 -f bastion -N
  • -f will send ssh to the background
• Force your current session to use that proxy for HTTPS with export HTTPS_PROXY=socks5://127.0.0.1:1080

Now, you can run kubectl get namespaces in your terminal and it will work!!!!

I know, it is not the easiest solution, but it is quite simple and transparent. I hope this will help someone!