Local Development Environment for Kubernetes using Minikube

Kubernetes can be an ultimate local development environment particularly if you are wrangling with a large number of microservices. In this post, we will cover how you can create a local development workflow using Minikube and tools such as Make to iterate fast without the wait imposed by your continuous integration pipeline. With this workflow, you can code and test changes immediately.

Traditionally, you can test your code changes by rebuilding the Docker images (either locally or via your continuous integration pipeline), then pushing the image to a Docker registry after the successful build, and then redeploy to your Kubernetes cluster.

Overall development workflow

Here is our simplified development workflow,

1. Make changes in your code on your local laptop
2. Build local Docker images and deploy them to Minikube
3. Test your code changes after deploying into Minikube
4. If changes are good, commit them to version control repository
5. Your version control system triggers continuous integration pipeline
6. Continuous integration builds Docker images and push it your registry

We are mainly interested in the fast iteration of step 1-3. In later sections, we will discuss how we can streamline step 1-3 using GNU Make and other tools.

Prerequisite

Before we get started, let’s have a look at various dependencies you need to install in order to make this workflow work for you.

Docker

As we are dealing with Docker images, I assume you have downloaded and installed Docker for your operating system. I use Docker for Mac.

VirtualBox

You will require VirtualBox (or a similar VM driver such as xhyve driver or VMware Fusion depending on your operating system) which enables Minikube to run a single-node Kubernetes cluster inside a VM. So make sure you have downloaded VirtualBox and installed it.

Kubectl

Kubectl is a command line utility (CLI) for running commands against Kubernetes clusters. You need kubectl to interact with your local Kubernetese cluster (i.e. Minikube) as well as remote clusters.

On Mac, you can install Kubectl using one of the following methods,

• Using the Google Cloud SDK: gcloud components install kubectl
• Using Homebrew package manager: brew install kubectl

For other operating systems please see detailed documentation on how to install kubectl.

Kubectl Basics

There are some basic kubectl commands and concepts you should get familiar with.

Using kubectl you can interact with various Kubernetes clusters by setting an appropriate context. Typically a kubeconfig file is used to configure access to one or more clusters and include details required for a context. On Mac, this can be found at ~/.kube/config. For more details please review this page on how to configure access to multiple clusters by using configuration files.

kubectl cluster-info
kubectl config --kubeconfig=config view
kubectl config use-context minikube
kubectl config --kubeconfig=config set-context minikube --cluster=minikube
kubectl config --help

Using kubectl you can interact with namespaces.Namespaces are virtual clusters and typically used to isolate projects deployed on Kubernetes cluster.

kubectl get namespace
kubectl create namespace dev

When interacting with different namespaces, you can set temporary namespace,

kubectl -n dev get pods

Or permanently for all subsequent kubectl commands in that context,

kubectl config set-context $(kubectl config current-context) --namespace=dev
kubectl get pods

You can also list and edit pods, deployments, services, ingress, and nodes.

kubectl get pods
kubectl get deployments
kubectl get services
kubectl get ingress
kubectl get nodes

Minikube

Depending on your operating system, you should download and install appropriate Minikube binaries.

On Mac, you can install latest stable version of Minikube using Homebrew,

brew cask install minikube

Alternatively, you can run the following command from your Mac terminal and it will install a specific version (v0.24.1) of Minikube.

curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.24.1/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/

Minikube Basics

Starting the Minikube will create a single-node Kubernetes cluster inside a VM. Minikube start command also creates minikube context and set it as default.

minikube start
minikube status

Kubernetes dashboard is a web-based user interface. You can create and modify individual Kubernetes resources from this UI. It also provides information on the state of Kubernetes resources in your cluster.

By default, Minikube comes with dashboard add-on enabled, and we can access the Kubernetes dashboard in the browser by typing following command.

minikube dashboard

Another way to access dashboard is to use kubectl. Run the following command in your terminal:

kubectl proxy

kubectl will handle authentication with API server and make Dashboard available at http://localhost:8001/ui.

You can always stop or delete minikube cluster,

minikube stop
minikube delete

In addition, Minikube provides a range of Kubernetes versions to run. You can print a list of available versions to use,

minikube get-k8s-versions

Then start Minikube by passing a specific version,

minikube start --kubernetes-version v1.7.0

In addition, you can run Minikube with more customised configurations such as you can specify a different network plugin than default or use a different virtual machine driver rather than VirtualBox. You can also change default allocation of memory and CPU.

minikube start --kubernetes-version v1.7.0 \
 --extra-config=proxy.IPTables.SyncPeriod.Duration=5000000000 \
 --extra-config=proxy.IPTables.MinSyncPeriod.Duration=3000000000 \

These additional start options can be viewed by running,

minikube start --help

Manage Minikube Add Ons

Minikube provides a set of built-in add-ons and some of them such as dashboard is enabled by default. These add-ons that can be enabled, disabled, or opened inside of the local Kubernetes environment.

minikube addons list

For development purpose, I highly recommend enabling heapster add-on. Heapster enables multi-level monitoring and performance analysis including pods, nodes, and cluster.

Under the hood, it is using InfluxDB as the storage backend for metric data and Grafana as visualization UI.

minikube addons enable heapster
minikube addons open heapster

Once enabled we can open Grafana UI (to interact with Heapster) in the browser.

I also highly recommend enabling Ingress functionality for your Minikube,

minikube addons enable ingress

Kubernetes Config Generator

When working with Kubernetes, creating and managing config files can be a really tedious job. I highly recommend kubegen - a tool which can simplify your day-to-day Kubernetes workflow, by automating the generation of boilerplate config codes for you and letting you focus on the important things.

Assuming you have Node JS and NPM already installed on your development machine, to install kubegen you need to execute the following command:

npm install -g generator-kubegen

Basically, kubegen is a Yeoman generator. When we run yo kubegen on terminal it ask few questions and based on your inputs it creates config files such as deployment.yml (deployment config), svc.yml (service config), ing.yml (ingress config).

Rapid Development Cycle

Now that we have setup our local Kubernetes development environment lets take it for a ride. We are going to build a Python application message-api. This application will be deployed inside our local Minikube cluster.

Make changes in your code on your local laptop

To start with, we are going to keep this application quite simple. It has

1. A Dockerfile
2. A requirements.txt with only one dependency flask
3. An app.py with a list of flask routes and view functions

Now let’s review the content of Dockerfile. Dockerfile has instructions to install requirements for the application and run an application server which enables this application to handle HTTP request/response on container port 5000.

FROM python:3-alpine

RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

COPY requirements.txt /usr/src/app/
RUN pip install --no-cache-dir -r requirements.txt

COPY . /usr/src/app

EXPOSE 5000

CMD [ "python", "./app.py" ]

Our app.py is also quite simple. We just added one flask GET request /api/v1.0/messages which returns a list of messages.

from flask import Flask, jsonify
app = Flask(__name__)

messages = [
    {
        'id': 1,
        'message': u'Hello World',
    },
    {
        'id': 2,
        'message': u'Hello Docker',
    },
]

@app.route('/api/v1.0/messages', methods=['GET'])
def get_tasks():
    return jsonify({'messages': messages})

if __name__ == '__main__':
    app.run(host='0.0.0.0')

Build local Docker images and deploy them to Minikube

A. Build Docker image message-api against the minikube

eval $(minikube docker-env)
docker build -t message-api:v1 .

Please note that eval $(minikube docker-env) enables us to interact and build against docker daemon inside the minikube VM.

B. Create a deployment message-api

kubectl run hello-python --image=message-api:v1 --port=5000

C. Expose deployment message-api as service

kubectl expose deployment hello-python --type=NodePort
kubectl get services

As you can see, everytime you make changes in your code you still have to repeat few things: build a new Docker images (Step-A) and then update our deployment to use updated Docker image (Step-D).

D. Update the deployment to use updated Docker image

kubectl set image deployment/message-api *=message-api:v1

To avoid this repetition we are going to add a Makefile in our project folder. In this Makefile we will add an update command which basically allows us to automate step-A and step-D.

.PHONY: update
update:  
    @eval $$(minikube docker-env) ;\
    docker image build -t message-api:v1 -f Dockerfile .
    kubectl set image deployment/message-api *=message-api:v1

Now all we need is to run make update command after every code change and we will see update results in Minikube.

Test your code changes after deploying into Minikube

Open or get URL for service message-api so we can validate the changes,

minikube service hello-python
minikube service --url hello-python

My workflow

I generally use a more evolved version of workflow described in the previous section. Using kubegen I create Kubernetes YAML files for deployment, service and ingress resources.

I use config files generated by kubegen in my configurable Makefile. On header section of the file, you can change the name of the repo (used for naming image, deployment, service, and ingress). You can edit namespace and path of config files along with version strings. You can an view example Makefile snippet below. As you can see, there are a few new make commands to streamline my Kubernetes development workflow,

• make create builds the local image, then uses that image to create a deployment which is exposed as service.
• make ingress creates an ingress using the config files specified in the header section (IFILE)
• make update creates a time stamped version of the image and update the deployment to use this new image

REPO=message-api
TIMESTAMP=tmp-$(shell date +%s )
NSPACE=dev
DFILE=message-api/deployment.yml
SFILE=message-api/svc.yml
IFILE=message-api/ing.yml
VERSION=v1

.PHONY: update
update:  
    @eval $$(minikube docker-env) ;\
    docker image build -t $(REPO):$(TIMESTAMP) -f Dockerfile .
    kubectl set image -n $(NSPACE) deployment/$(REPO) *=$(REPO):$(TIMESTAMP)

.PHONY: delete
delete:
    kubectl delete -n $(NSPACE) deployment,service $(REPO)

.PHONY: create
create:
    @eval $$(minikube docker-env) ;\
    docker image build -t $(REPO):v1 -f Dockerfile .
    kubectl create -f $(DFILE)
    kubectl create -f $(SFILE)

.PHONY: ingress
ingress:
    kubectl create -f $(IFILE)

.PHONY: push
push: build
    docker tag $(REPO):$(VERSION) $(REPO):latest
    docker push $(REPO):$(VERSION)
    docker push $(REPO):latest

build:
    @eval $$(minikube docker-env -u);\
    docker image build -t $(REPO):$(VERSION) -f Dockerfile .

The big picture

In this post, we mainly focused on the local development environment using Minikube. As illustrated below this is a part of the bigger picture i.e. overall end-to-end workflow. In upcoming posts, we will cover other tools with the potential to fast-track development activities utilizing Kubernetes cluster.