In this project we will explore:

  • Redis NoSQL database
  • Kubernetes orchestration
  • App development in Go

The code repository is available on GitHub.

1. Database

Redis (REmote DIctionary Server) is a type of non-relational database and serves as an in-memory key-value database that supports various simple data structures. Running in memory results in fast read and write operations. Redis can also perform disk persistence like traditional databases which store data on disk. Disk lookups are usually slower than in memory lookups but come at a cheaper cost per byte of storage. As a result, Redis can support more operations and have faster response times. A benefit of storing data persistently is that data will be available after a shutdown of the database. In a production environment, persistent data is definitely preferable over in memory storage.

Cache is a hardware or software component that stores data in fast memory so future requests for that data can be served faster. The data stored in a cache might be the result of an earlier computation or the duplicate of data stored elsewhere. Redis is a good fit for serving as a cache because of its performance. Implementing Redis into a website would require only a single fetching of data and save it to memory. The impact will not become visible until a certain point where the number of users would cause disk lookups to become very expensive.

Redis supports replication over multiple servers. This is useful if a Redis server disconnects as replication over the entire system architecture will ensure a better quality of service. The architecture of a Redis cluster with replication consists of a master controlling a set of slaves. The slaves act as copies of the master and synchronize regularly. Since Redis supports a clustered topology it is easy to scale up if we need more servers. Redis is aimed to be as simple and fast as possible and accepts various data structures.

Redis is a data structure server storing key-values of strings and other more complex data structures. The database focuses on readability where keys are not limited to a set size, though it is not recommended to use large keys. Strings are the most simple type in Redis and can be used as keys. If the value we associate with the string key is a string as well, we are mapping a string to another string. The following example sets a string variable in Redis to the value of “somevalue”.

   > set mykey somevalue
   OK
   > get mykey
   "somevalue"

SET and GET are the commands used with set and retrieve string values associated to a key. Redis is designed to be as easy as possible and allows values to be set to almost anything with a limited size of 512 MB.

In this project we will store key-values as strings in a single node redis database.

2. Kubernetes

Containers offer fast deployment and portability accross platforms. Kubernetes is a container orchestrating tool for automating deployment, scaling, and managing of containerized applications.

graph LR;
    subgraph Kubernetes
        redis-master
        go-app
    end
    go-app --- redis-master
    client  --- go-app

2.1. Deployments

Deployments allows you to describe an application’s life cycle, such as which images to use for the app, the number of pods there should be, and the way in which they should be updated.

2.1.1. redis

We create a redis deployment containing a single pod with redis version 6.0.9. Port 6379 is the default port for redis.

kind: Deployment
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: master
        image: redis:6.0.9
        ports:
        - containerPort: 6379

2.1.2. Webserver

We create a webserver deployment containing a single pod with go-web-app container image. The go application has been containerized, see 3.4. Containerizing.

kind: Deployment
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: webserver
        image: "erik/go-web-app"
        imagePullPolicy: Never
        ports:
        - containerPort: 8080

2.2. Services

Services is a way to expose an application running on a set of Pods as a network service.

2.2.1. redis

The redis service expose port 6379.

kind: Service
spec:
  ports:
  - name: redis
    port: 6379
    targetPort: 6379

2.2.2. Webserver

The webserver service expose port 8080.

kind: Service
spec:
  ports:
  - name: webserver
    port: 8080
    targetPort: 8080

3. Go

"Go is an open source programming language that makes it easy to build simple, reliable, and efficient software."

The Go application is separated into a main file used for starting the http server, a webserver file for http server config and a redis file for interacting with the database.

go/
┣ main.go
┣ redis.go
┗ webserver.go

3.1. main.go

main.go starts a http listener served on port 8080.

http.HandleFunc("/", handler)
http.ListenAndServe(":8080", nil)

3.2. webserver.go

webserver.go contains functions to handle and respond to incoming requests.

3.2.1. Handler

The handler parses incoming HTTP requests and forwards the request to correct method.

func handler(res http.ResponseWriter, req *http.Request) {

	req.ParseForm()

	switch req.Method {
	case "POST":
		post(res, req)
	case "GET":
		get(res, req)
	default:
		bad(res, req)
	}
}

3.2.2. Post

Post handles post requests and posts key value sets to the database.

// key value object stored in redis
type rObject struct {
	Key   string
	Value string
}

func post(res http.ResponseWriter, req *http.Request) {

	var r rObject

	// get key value in request
	body, _ := ioutil.ReadAll(req.Body)
	json.Unmarshal(body, &r)

	// write to database
	setValue(r.Key, r.Value)
	fmt.Fprintf(res, "Wrote %s:%s to database\n", r.Key, r.Value)
}

3.2.3. Get

Get handles get requests and returns value of key from the database.

// get value of key from redis
func get(res http.ResponseWriter, req *http.Request) {

	// trim url
	var trimPath = strings.Trim(req.URL.Path, "/")
	fmt.Fprintf(res, "Getting value of key: %s\n", trimPath)

	// get value
	var value = getValue(trimPath)
	fmt.Fprintf(res, "Value: %s\n", value)
}

3.3. redis.go

redis.go contains functions to integrate the go application with the redis database.

3.3.1. Database connection

To establish a connection to redis we call NewClient. Kubernetes resolves the DNS record of redis-master to the corresponding pod address. See K8s docs.

var client = rClient()
var ctx = context.Background()

// connect to redis database
func rClient() *redis.Client {

	client := redis.NewClient(&redis.Options{
		// redis-master endpoint is created
		Addr: "redis-master:6379",
	})

	return client
}

3.3.2. Set

setValue writes a new key value set to the database.

func setValue(key string, value string) {

	client.Set(ctx, key, value, 0)
}

3.3.3. Get

getValue returns a value of an existing key.

func getValue(key string) string {

	client.Get(ctx, key).Result()
	return val
}

3.4. Containerizing

To run the go application in Kubernetes we need to containerize it. In the Dockerfile we specify the base image we need for our go application. An /app directory is created within the image to hold the application source files and the root directory is copied into the directory. go build is executed in the /app directory to compile the binary executable, which is executed with CMD ["/app/main"].

FROM golang:1.15-alpine3.12
RUN mkdir /app
ADD . /app
WORKDIR /app
RUN go build -o main .
CMD ["/app/main"]

4. Usage

I run a single node Kubernetes cluster with minikube for developing simple Kubernetes applications locally.

4.1. Prerequisites

4.2. Makefile

Run make all to build and deploy the application. make build utilizes the docker environment of minikube to build the container image.

all: build deploy forward

build:
	@eval $$(minikube docker-env) ;\
	docker build -t erik/go-web-app go/
deploy:
	kubectl apply -f k8s/redis-master-deployment.yaml
	kubectl apply -f k8s/redis-master-service.yaml
	kubectl apply -f k8s/webserver-deployment.yaml
	kubectl apply -f k8s/webserver-service.yaml

To access pods on the minikube node we need to either forward the minikube port to localhost, or ssh into the minikube node to access the pods directly. Forward the webserver port to localhost with make forward.

forward:
	kubectl port-forward service/webserver 8080:webserver

4.3. Example requests

Sending a GET request to the server

curl localhost:8080/foo

Returns the value of the key.

Getting value of key: foo
Value: bar

5. References