Background

I have been using Docker Desktop for while in macOS. The recent update of Docker Desktop was killing huge memory and storage in my machine. So I thought to give up on Docker Desktop and install Docker with Minikube and Hyperkit hypervisor. Minikube is used to run a Kubernetes cluster on local environment. But it also runs a Docker daemon that can be used to run containers. Hyperkit is an open-source hypervisor for macOS hypervisor, optimized for lightweight virtual machines and container deployment.

Docker Desktop

The docker engine which is the core software behind the docker only runs on Linux kernal(the engine can run on a physical or a virtual machine, but it can only run on top of a Linux kernel i.e. any OS that is flavour of Linux). Since Docker Engine only runs on Linux, developers who use Windows and macOS for software development cannot run the engine until they spin up a virtual machine (VM) that runs linux. That is where Docker Desktop comes in. Docker Desktop is a closed-source software that allows developers working on Windows/macOS to use container technology seamlessly on their development environment without needing to manage the complexity of operating a VM and all the dependencies that comes along with it (networking, virtualization, knowledge of linux etc).

Minikube Replacement

Docker Desktop is not the core technology that runs containers, it only aims to make it easier to develop software on Windows/macOS that runs in containers. So we can replace Docker Desktop with using native Linux VM(and Hypervisor). In here, I’m gonna use Minikube which designed to be used as a virtual machine(VM) as a replacement for Docker Desktop. Minikube required a hypervisor to run, so I’m using Hyperkit hypervisor. Following are the steps to follow.

1. Uninstall Docker Desktop

Following is the way to fully uninstall Docker Desktop and remove all it’s dependencies on local file system.

sudo rm -Rf /Applications/Docker.app
sudo rm -f /usr/local/bin/docker
sudo rm -f /usr/local/bin/docker-machine
sudo rm -f /usr/local/bin/docker-compose
sudo rm -f /usr/local/bin/docker-credential-desktop
sudo rm -f /usr/local/bin/docker-credential-ecr-login
sudo rm -f /usr/local/bin/docker-credential-osxkeychain
sudo rm -Rf ~/.docker
sudo rm -Rf ~/Library/Containers/com.docker.docker
sudo rm -Rf ~/Library/Application\ Support/Docker\ Desktop
sudo rm -Rf ~/Library/Group\ Containers/group.com.docker
sudo rm -f ~/Library/HTTPStorages/com.docker.docker.binarycookies
sudo rm -f /Library/PrivilegedHelperTools/com.docker.vmnetd
sudo rm -f /Library/LaunchDaemons/com.docker.vmnetd.plist
sudo rm -Rf ~/Library/Logs/Docker\ Desktop
sudo rm -Rf /usr/local/lib/docker
sudo rm -f ~/Library/Preferences/com.docker.docker.plist
sudo rm -Rf ~/Library/Saved\ Application\ State/com.electron.docker-frontend.savedState
sudo rm -f ~/Library/Preferences/com.electron.docker-frontend.plist

2. Install Minikube

Following are the required packages that needs to be installed with Minikube. I’m installing these packages with homebrew.

# intall hyperkit
brew install hyperkit

# install minikube
brew install minikube

# install docker cli
brew install docker

# install docker-compose
brew install docker-compose

3. Start Minikube

Following is the way to start Minikube. it starts a Kubernetes cluster with Docker daemon. When starting you may get an error. If error occurs try to clean Minikube and restart it as mentioned below.

# start minikube
minikube start

# if error occues when starting the minikube use bleow command to clean the minikube
# and then restart 
minikube delete --all --purge

# start minikube again
minikube start

# this will start single node kubernets cluster on minikube vm
# docker deamon run inside the minikube vm with k8s cluster
kubectl get nodes

NAME       STATUS   ROLES                  AGE     VERSION
minikube   Ready    control-plane,master   3h45m   v1.22.2

4. Configure Minikube

Then we need to configure docker-cli to use Minikube VM. Add following entry to ~/.bashrc or ~/.zshrc and then configure the Minikube VMs’ IP address.

# add this entry to ~/.bashrc or ~/.zshrc to work in all terminal sessions
eval $(minikube docker-env)

# find minikube vm ip address, this ip use as the docker host ip
minikube ip 
>> 192.168.64.3

# add minikube ip to /etc/hosts if want
echo "`minikube ip` docker.local" | sudo tee -a /etc/hosts > /dev/null

# minikube configurations will be stored in the following config file 
~/.minikube/machines/minikube/config.json

5. Docker Volume Mapping

The docker volumes of the container will be created inside Minikube VM. It cannot directly access by the host macOS. We need to ssh into Minikube VM to access them.

# run container with volument mapping
# volume will be created inside the minikube vm
docker run -d --name redis -p 6379:6379 -v /private/var/services/redis:/data redis:5.0.3
# ssh to minikube to view the volume
minikube ssh

# volume is on /private/var/services/redis directory
ls /private/var/services/redis
>> dump.rdb

To access these docker volumes inside the Minikume VM from host machine, we need to mount the volume folder inside the Minikube into host(macOS) machine. Following is the way to do that.

# stop minikube
minikube stop

# start minikube with volument mount
minikube start --mount --mount-string="/private/var/services/:/private/var/services/"

This will mount the /private/var/services directory inside the Minikube VM to /private/var/services directory in the host machine. Now if you run the container, the volume mappings will be exists in the host machine.

# stop minikube
minikube stop

# start minikube with volument mount
minikube start --mount --mount-string="/private/var/services/:/private/var/services/"

6. Configure Memory and CPU

By default Minikube allocates 2 CPUs and 2 GB memory. We can increase Minikube cpu and memory in following ways.

# by default minikube alloca 2 cpus and 2 gb memory
minikube start

# delete existing minikube instance and allocate more memory/cpu on start up
minikube stop
minikube delete
minikube start --memory 8192 --cpus 4
>> 🔥  Creating hyperkit VM (CPUs=4, Memory=8192MB, Disk=20000MB)

# keep existing minikube instance and allocate more memory/cpu
# please note that in some systems this method not working
minikube stop
minikube config set memory 8192
minikube config set cpus 4
minikube start

# view minikube config
kubectl get node minikube -o jsonpath='{.status.capacity}'
>>{"cpu":"2","ephemeral-storage":"17784752Ki","hugepages-2Mi":"0","memory":"3935188Ki","pods":"110"}
minikube config get memory
>> 8192
minikube config get cpus
>> 4

# increase virtual memory
# in some scenarios(e.g running elasticsearch) it required more virual memory
minikube ssh
sudo sysctl -w vm.max_map_count=262144

7. Configure Insecure Docker Registry

Minikube allows users to configure the docker engine’s --insecure-registry flag. We need to delete existing Minikube cluster before start the new cluster with --insecure-registry flag.

# delete existing cluster
minikube stop
minikube delete

# start new cluster with insecure registry
minikube start --insecure-registry="dockerregistry.rahasak.local"

# start new cluster with multiple configs
#   memory
#   cpus
#   insecure registry
minikube start --memory 8192 --cpus 4 --insecure-registry="dockerregistry.rahasak.local"

8. Configure Docker Bridge

When the Docker service is started, a linux bridge is created on the host machine. The interfaces on the containers talk to the bridge, and the bridge proxies to the external world. Multiple containers on the same host can talk to each other through this bridge. The default docker bridge(docker0) assigned a IP address 172.17.0.1. We can dynamically configure the docker’s default bridge(docker0) IP by using docker-opts=bip=<new ip/net mask> flag. It specifies the IP address and netmask to use for docker’s default bridge(docker0). Then the new docker containers will use IP addresses within this range.

# the default docker0 bridge inside minikube
minikube ssh
ifconfig

>> output
docker0   Link encap:Ethernet  HWaddr 02:42:E1:6D:15:33
          inet addr:172.17.0.1  Bcast:172.17.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:311191 errors:0 dropped:0 overruns:0 frame:0
          TX packets:307842 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:53946915 (51.4 MiB)  TX bytes:27206303 (25.9 MiB)
          
---

# delete existing cluster
minikube stop
minikube delete

# configure minikube cluster with new bridge ip
minikube start --docker-opt=bip=172.17.42.1/16

# new docker0 bridge 
minikube ssh
ifconfig

>> output
docker0   Link encap:Ethernet  HWaddr 02:42:0F:1B:F8:1B
          inet addr:172.17.42.1  Bcast:172.17.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:30 errors:0 dropped:0 overruns:0 frame:0
          TX packets:36 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:3907 (3.8 KiB)  TX bytes:12940 (12.6 KiB)

9. Expose Docker REST API

Docker daemon which resides in the Minikube listens to the /var/run/docker.sock which is a Unix socket(Unix Sockets use the local filesystem for communication, while IP Sockets use the network). We can bind this Unix socket to TCP socket and expose outside. In following example I’m binding the Unix socket to TCP port 2375 on Minikube host via socat. Then we can connect to this port from the host machine. This set up can be used to expose docker REST API(e.g via unencrypted channel) to outside. Then the REST clients can connects to the port 2375 to access the docker REST API.

# connect to minikube
minikube ssh

# bind docker unix socket to tcp port 2375
# -d -d - prints fatal, error, warning, and notice messages
# reuseaddr - allows an immediate restart of the server process
# fork - forks a new child process for each connection
# & - run in background 
socat -d -d TCP-LISTEN:2375,reuseaddr,fork UNIX-CONNECT:/var/run/docker.sock &

# connect to docker api
#   192.168.64.14 - minikube ip
#   2375 - unix socket binding port
curl 192.168.64.14:2375/version

# output
{
  "Platform": {
    "Name": "Docker Engine - Community"
  },
  "Components": [
    {
      "Name": "Engine",
      "Version": "20.10.8",
      "Details": {
        "ApiVersion": "1.41",
        "Arch": "amd64",
        "BuildTime": "2021-07-30T19:55:09.000000000+00:00",
        "Experimental": "false",
        "GitCommit": "75249d8",
        "GoVersion": "go1.16.6",
        "KernelVersion": "4.19.202",
        "MinAPIVersion": "1.12",
        "Os": "linux"
      }
    },
    {
      "Name": "containerd",
      "Version": "v1.4.9",
      "Details": {
        "GitCommit": "e25210fe30a0a703442421b0f60afac609f950a3"
      }
    },
    {
      "Name": "runc",
      "Version": "1.0.1",
      "Details": {
        "GitCommit": "4144b63817ebcc5b358fc2c8ef95f7cddd709aa7"
      }
    },
    {
      "Name": "docker-init",
      "Version": "0.19.0",
      "Details": {
        "GitCommit": "de40ad0"
      }
    }
  ],
  "Version": "20.10.8",
  "ApiVersion": "1.41",
  "MinAPIVersion": "1.12",
  "GitCommit": "75249d8",
  "GoVersion": "go1.16.6",
  "Os": "linux",
  "Arch": "amd64",
  "KernelVersion": "4.19.202",
  "BuildTime": "2021-07-30T19:55:09.000000000+00:00"
}

10. Minikube without Kubernetes

The Minikube recent update(v1.24.0) supports to start Minikube VM without starting any Kubernetes in it. This is nice features if you want to run only Docker without Kubernetes. It will save more CPU of the computer as well. In my scenario I have run Minikube only with Docker(without Kubernetes) and run the Kubernetes cluster with K3d.

# update minikube if install old version
brew upgrade minikube

# run minikube without kuberntes
# used the flag --no-kubernetes
minikube start --memory 6144 --cpus 4 --docker-opt=bip=172.17.42.1/16 --no-kubernetes

Replace Docker Desktop with Minikube and Hyperkit on macOS