There are plenty of Kubernetes distributions available that can be used to install Kubernetes locally. They tend to use minimal resources which is ideal. Question is what’s the best Kubernetes distribution for local use?
Local Kubernetes Distributions
This post contains benchmarks and reviews of the following Kubernetes distributions:
minikube
Minikube is the oldest and most popular Kubernetes distribution for local environments. The project started in 2016. Its default configuration makes it very easy to get started. Most development tools support minikube out of the box. Minikube supports various “driver”s which allow you to run in different set-ups (virtualbox, docker, bare-metal, etc). For docker driver to work, docker has to be installed on the system and the current user has to have the right permissions.
microk8s
MicroK8s is Canonical‘s way of getting into Kubernetes market. It’s a single “snap” command to install on Ubuntu systems. Internally MicroK8s uses containerd as the container runtime which is bundled into the snap package.
During the benchmark microk8s did the worst. I am not sure what the slowness was caused by but my guess is its resource use.
Storage and DNS addons are disabled by default so they need to be enabled if persistence and DNS. Microk8s is the only distribution that disables DNS by default. During benchmarking I had issues with helm v3 packaged with microk8s because it was an early release which had a bug.
k3s
k3s is a distribution from Rancher and is a CNCF project. k3s binary contains all the necessary cluster components so the cluster setup is very quick (the quickest of the 5 distributions). Although it comes with both storage and DNS installed, it is also very light. k3s doesn’t have any dependencies.
k3s is very easy to use but as it’s not as widely used external tool support for it is not as good as minikube. It also lacks the shortcuts useful for beginners (e.g. helm works out of the box with minikube, but with k3s you have to extract kubeconfig from kubectl using “k3s kubectl config view –raw”).
k0s
k0s is similar to k3s in many ways. Like k3s, k0s also comes as a single binary which achieves a very quick setup times. k0s binary is massive (170MB) compared to k3s (50MB). You can add k0s to startup by using “k0s install controller” command or can run it quickly using “k0s controller” command. k0s, like k3s doesn’t have any dependencies.
Because of it’s low resource footprint I was surprised by the benchmark results. The average response time was similar to the response times achieved with microk8s.
k0s does not come with a storage handler. There isn’t even a command to install it. I had to set up helm and install openebs. For that reason it got the worst ease of use score.
kind
Similar to minikube, kind is also maintained by the Kubernetes team but unlike minikube it only works with docker. “kind” stands for “Kubernetes in Docker”. It hasn’t got a stable release yet so I wouldn’t switch to it just yet. kind requires docker to be installed on your system with the right permissions.
Resource-wise kind scored average and did great in the benchmark.
kind is quite easy to use. Cluster comes with everything you need but not used as widely as minikube.
Benchmark
During testing and benchmarks I made some choices to make it as fair as possible:
- minikube uses “docker” driver recommended in the documentation
- All tests and benchmarks done on a clean virtual machine snapshot with 2 cores
- No port-forwarding during benchmarks because port-forwarding was significantly slower
- Installed WordPress using Bitnami WordPress helm chart
- Used Apache Bench command “ab -n 1000 -c 5 http://localhost:31234/” to get the average response time on WordPress homepage
- Counted the number of processes on the clean snapshot and subtracted from the process count after setting up the cluster
- Cluster setup time includes binary downloads but excludes dependencies (e.g. docker)
Feature comparison
When comparing distributions I made the following assumptions:
- minikube can actually run at but only if you define the service manually. k0s has a command to define the service. k3s will run at boot if installed by following the quick start guide
- Subtracted ease of use points for each obstacle encountered during the installation of wordpress helm chart (missing addon, no default kubeconfig)
- Considered k3s and k0s as single binary because they have no dependencies and do no downloads during the set up process
Verdict
After spending hours testing and benchmarking these tools I’ll probably continue using minikube because it’s so easy to use. I will keep watching kind and k3s to see if they get better. Speed of k3s is its best advantage and I’m sure that alone will make lots of people use it.
What’s your Kubernetes distribution choice?
Just wanted to bring Talos (https://talos.dev) to your attention – runs a vanilla certified K8s distribution that is the same locally, in cloud, on virtual machines, and bare metal.
Also immutable and hardened.
Just FYI.