Diego Sucaria

If, for some reason, you increased a GCP VM boot disk size following the standard procedure and then decided you didn’t need all that space, when you want to shrink the disk size, you find out that, well… it is not that easy as increasing it, but it is possible.

It is a tough and time-consuming process… 

TLDR

In short, we will need a temporary VM, attach the disk to it, resize the filesystem and partition, create a new disk with the desired size, and clone the contents.

Important assumptions:

• We will resize a 500G disk to 200G.
• We have already freed up space in the disk. 
• For this particular tutorial, we are using Debian 9

Step by Step Process

• First, stop the original VM

• Create a snapshot of the disk (for backup purposes)

• Create a new VM. We will name it the temp VM and use it to mount the disk and resize it.

• Detach the big disk from the original VM. Attach it to our new temp VM.

• Mount it

mkdir -p /mnt/disks/bigdisk
mount -o discard,defaults /dev/sdb1 /mnt/disks/bigdisk
ls -la /mnt/disks/bigdisk

• Check the available disk size

df -Th
Filesystem     Type      Size  Used Avail Use% Mounted on
udev           devtmpfs  2.0G     0  2.0G   0% /dev
tmpfs          tmpfs     393M  456K  393M   1% /run
/dev/sda1      ext4      9.7G  2.0G  7.2G  22% /
tmpfs          tmpfs     2.0G     0  2.0G   0% /dev/shm
tmpfs          tmpfs     5.0M     0  5.0M   0% /run/lock
/dev/sda15     vfat      124M   12M  113M  10% /boot/efi
tmpfs          tmpfs     393M     0  393M   0% /run/user/1002
/dev/sdb1      ext4      493G  133G  340G  29% /mnt/disks/bigdisk

Note that the partition is using 133G of space. We can shrink the disk to 200G and leave some room to grow.

NOTE: to avoid any issues and make things run faster, we will shrink the filesystem to 150G and partition to 160G, then expand everything to 200 in the new disk

• Run resize2fs to resize the filesystem

umount /dev/sdb1
e2fsck -f /dev/sdb1
resize2fs -p /dev/sdb1 150G
e2fsck -fy /dev/sdb1

The process will take some time, approximately 10 minutes in this case.

• After resizing the filesystem, we prepare to resize the partition

You can use fdisk or cfdisk. Note that to resize the partitions, we point to the disk, sdb instead of sdb1

• If fdisk is not installed, apt install fdisk

• Check partitions and get the first block size:

fdisk -l /dev/sdb
Disk /dev/sdb: 500 GiB, 536870912000 bytes, 1048576000 sectors
Disk model: PersistentDisk  
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x96c8eef6

Device     Boot Start       End   Sectors  Size Id Type
/dev/sdb1  *     4096 419434495 419430400  500G 83 Linux

The important number is 4096. We will use it later

• Resize the partition 

Keyboard inputs are in bold

fdisk /dev/sdb

Command (m for help): p
Disk /dev/sdb: 500 GiB, 536870912000 bytes, 1048576000 sectors
Disk model: PersistentDisk  
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x96c8eef6

Device     Boot Start       End   Sectors  Size Id Type
/dev/sdb1  *     2048 419432447 419430400  200G 83 Linux

Command (m for help): d
Selected partition 1
Partition 1 has been deleted.

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): *Enter*
First sector (2048-1048575999, default 2048): 4096
Last sector, +/-sectors or +/-size{K,M,G,T,P} (4096-1048575999, default 1048575999): +160G

Created a new partition 1 of type 'Linux' and of size 200 GiB.
Partition #1 contains a ext4 signature.

Do you want to remove the signature? [Y]es/[N]o: N

Command (m for help): a
Selected partition 1
The bootable flag on partition 1 is enabled now.

Command (m for help): p
Disk /dev/sdb: 500 GiB, 536870912000 bytes, 1048576000 sectors
Disk model: PersistentDisk  
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x96c8eef6

Device     Boot Start       End   Sectors  Size Id Type
/dev/sdb1  *     4096 419434495 419430400  160G 83 Linux

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

• Now we attach a new empty 200G disk to the temp VM. (it will be /dev/sdc)

• Clone the disk to the new one:

dd if=/dev/sdb of=/dev/sdc bs=8M status=progress

This operation will take some time, it took another 10 minutes

After 200G (the physical disk size), dd will fail, saying the destination disk is full (which is true!)

dd: error writing '/dev/sdc': No space left on device
25601+0 records in
25600+0 records out
214748364800 bytes (215 GB, 200 GiB) copied, 1028.88 s, 209 MB/s

• Mount to verify it is working, and run e2fsck just in case

mkdir -p /mnt/disks/shrinkeddisk
mount -o discard,defaults /dev/sdc1 /mnt/disks/shrinkeddisk
ls -la /mnt/disks/shrinkeddisk
umount /dev/sdc1
e2fsck -f /dev/sdc1

• Detach the new small disk from the temp VM and attach it to the original VM

And we are done!!!

There are many ways to access a Kubernetes private cluster. My preferred way is the GKE one, on which the API endpoint is public but you whitelist your machine to be able to connect to it.
You can do it with a simple gcloud command and that is it. You can even automate your pipelines to do it, then remove the whitelisted IP when ready.

Both Azure and AWS do not do it this way when you create a private cluster inside a Private Network, it is really private… and it becomes painful. Hey, in Azure I even think their own Azure Portal cannot see the workloads. WTF.

The most preferred-by-infosec way of doing it is by using a VPN. Which yes, works, yet is another thing to take care of and managed VPNs are not cheap.

The general popular idea is to set up a Bastion Host (usually a VM) which is exposed to the internet (hey, this sounds great!) and then you can SSH into the Bastion to run kubectl commands in there:

PLEASE, DON’T DO THIS. Why would you create a private cluster if you are going to put a freaking VM with port 22 open in front of it…
::insert-facepalm-emoji-here:: It is nonsense!!!

Yet many people do it.

There are options like gototeleport.io which allow you to securely SSH into the Bastion host with MFA, or even kubectl directly to the cluster, skipping the Bastion at all. But it is really expensive. (I save you the Google search, $100/m per cluster)

Couldn’t be worse eh?

It gets even worse. I’ve seen some architectures that looks like this:

So now you have two SSH jumps. One into the “public” JumpBox, then from there, you SSH into a Bastion host located inside the private network, from which you can run kubectl commands.

This architecture is bad for many reasons. First, you are still exposing a VM with port 22 open. It also promotes using the JumpBox and the Bastion as “trash boxes” where your Kubernetes YAML files will lie forever.
Finally, I certainly believe this architecture should not exist. The proper way of having a truly private cluster is to make it private first, then only access it by using pipelines. If you really need to kubectl into it, then it is a DEVELOPMENT cluster, and whitelisting IPs or using a VPN are the best options.

How to jump over the crap

Now, let’s return to the issue, the architecture exists, it is a development cluster, it is real, and it is my problem now. I want to run kubectl from my machine, where I have my IDE and my stuff.

SSH and ~/.ssh/config to the rescue!

With SSH, you can create a proxy and forward all connections to a certain host, but in this case, we have a double SSH jump to do, so, in order to make it easy and readable, we can use the ~/.ssh/config file like this:

Host jumpbox
  HostName x.x.x.x    #[this is the public ip]   
  User myuser
  IdentityFile ~/.ssh/my-key.key

Host bastion
  HostName a.b.c.d    #[this is the private ip]
  User mybastionuser  #ec2-user in my case  
  ProxyJump jumpbox

Now, you can ssh bastion and you should be able to login to the Bastion server.

If you get a public_key denied message, make sure your machine’s public key is in the authorized_keys of the Bastion, not only on the JumpBox

How to make kubectl work over an SSH proxy?

In order to make kubectl work over a proxy, we need to do a few quick things:

• Copy the ~/.kube/config data you need from the Bastion to your local machine.
• If you are using AWS RBAC make sure aws cli is set up in your local machine, then follow this tutorial.
• Create an SSH proxy with ssh -D 1080 -f bastion -N
  • -f will send ssh to the background
• Force your current session to use that proxy for HTTPS with export HTTPS_PROXY=socks5://127.0.0.1:1080

Now, you can run kubectl get namespaces in your terminal and it will work!!!!

I know, it is not the easiest solution, but it is quite simple and transparent. I hope this will help someone!