Continuous delivery with GitOps

Continuing our discussion around the modernization of the Tidepool backend infrastructure, this post will share how GitOps enables continuous delivery of our code and services.

With the basic infrastructure in place, we needed a way to manage change to our running services. Traditionally, this has been performed by systems such as Ansible, Chef, or Puppet. These systems:

1. Define a language for communicating change requests or new desired state from clients.
2. Store such change requests or desired state centrally.
3. Distribute these requests/demands to the nodes they manage.
4. Effect the changes on each node.

Contrast that to Kubernetes, where most of these service are provided:

1. Kubernetes consists of a number of controllers that read a desired state (from an internal etcd server) and transform the state of the cluster to match the desired state.The language of Kubernetes are YAML files called manifests that define the desired state of the Kubernetes objects.
2. Kubernetes stores this state in a distributed and reliable key/value store called etcd.
3. One may push a new desired state to the Kubernetes server from outside the cluster using the Kubernetes CLI or run a program to modify the cluster via the Kubernetes API.
4. Kubernetes controllers make the changes to the nodes in the cluster.

With a declarative engine and an endogenous persistent store in the cluster, the only missing component is the persistent exogenous store outside the cluster. Enter GitHub!

GitHub runs a persistent store with immutable versions. It is an ideal place to store versions of desired cluster state. By running a program in the cluster that watches for changes in a GitHub repo, and using the Kubernetes API, one can make pull changes and make changes to a cluster. This approach is called GitOps.

Image via fluxcd.io

WeaveWorks has written a Kubernetes controller called Flux and a companion called Helm Operator that implement GitOps. The former is a Sandbox project of the Cloud Native Computing Foundation. We use both Flux and the Helm Operator to pull configuration information from GitHub repos, one each of the five Kubernetes clusters that we currently manage. With desired state stored in Git, one merely commits to the Git config repo to cause the cluster(s) watching that repo to change to that state.

The Flux operator actually performs two services for us.

First, it watches the configuration repo for changes. When it detects changes it updates the cluster accordingly.

Second, Flux watches a Docker Repo such as DockerHub for the publication of new Docker images. When it finds a Docker image of the same name as you select and whose tag matches a pattern provider, then that Flux edits manifests stores in your Git config repo to use those new tags.

In this way, Flux pulls changes from both GitHub and DockerHub.

The Flux Helm Operator uses the popular Helm tool to install software packages into the Kubernetes cluster that it runs in. To do so, the Helm Operator defines a new custom resource called the HelmRelease that describes the location of a helm chart and the values needed to evaluate a helm template. The Helm Operator watches for new or changed HelmRelease manifests via the Kubernetes Watch API. When it sees a new HelmRelease, it materializes the Kubernetes manifests and installs corresponding resources. When it sees a modified HelmRelease, it updates the existing release in place.

While Flux and the Helm Operator do the heavy lifting, we use two companion tools to communicate.

flux-recv listens for changes to our Docker Hub and GitHub repositories. When it receives a webhook from one of the repositories, it requests that Flux update the Kubernetes cluster to match the configuration repo.

fluxcloud posts notifications to a Slack channel whenever flux takes an action in the cluster. For example this message indicates which commits were installed from the config repo and which Kubernetes resources were modified:

Applied flux changes to cluster

Event: Sync: d820dda, flux:helmrelease/helm-operator, production:helmrelease/tidepool, production:virtualservice/http

Commits:

* d820dda: Merge pull request #78 from tidepool-org/tpctl-2020-02-13-14-51-48

* 60cadd4: Added config packages

Resources updated:

* production:helmrelease/tidepool

* production:virtualservice/http

* flux:helmrelease/helm-operator

Together, these tools provide Tidepool an elegant way to maintain (in Git) and update (in cluster) the state of the Kubernetes resources.

On the horizon

The landscape of tools for continuous integration and continuous deployment for Kubernetes is evolving fast. We are excited by the JenkinsX project and the partnership between Argo and Weave. Both of these efforts may simplify the process of CI/CD in Kubernetes.

Upcoming

In our next engineering blog post, we will discuss how we protect system secrets.