Simplifying Bundled Kubernetes Resource Deployments

Simplifying bundled Kubernetes resource deployment

Premise

I recently had a discussion on how to simplify deployment of common bundled modules so as to enable developers to roll out resources and services on test environments with similar base requirements all while encapsulating the complexity of the setup. All this in an attempt to left-shift deployment on test infrastructure, all while empowering the developers to have slight insight on the architecture underlying and supporting their code.

I have highlighted a number of simple implementations/suggestions that would assist in a similar situation. The goal is to simplify the deployment on K8s by encapsulation and abstraction. We shall be looking at using Kubernetes Custom Operator, using Helm with sub-charts, using Terraform sub-modules and finally using Ansible Operators.

Dive-in

For most of the solutions outlined here, the setup might seem a bit tedious at first,more-so for the Devops teams, but at the end of the day seems like a safe trade-off considering how much it simplifies the deployments by providing high level APIs. We aim to leverage the ease of IaC solutions.

Kubernetes Custom Operators (CRDs)

Most of my experience interacting with custom operators was in a bid to reliably setup databases or data stores, most recently when trying to deploy Elasticsearch and MongoDB without having to reinvent the wheel and all while using the expertise and experience (Domain Knowledge) others have had tuning and securing the deployment frameworks. Many a time operators are created and shared by a provider and all it takes to setup is creating a K8s yaml with the required Kind and API consuming the C RD that is installed. Generally speaking, most of the definitions deployed on K8s are a form of CRD but just that they are offered as part of the core K8s environment, this is evident when you observe how patches or new features like the vertical pod autoscaler are supported prior to being part of a major release.

The question then arises of how to create these Custom Operators; the most basic way to do this is to use the apiextensions API with the CustomResourceDefinition kind on Kubernetes. This would define the structure/content of the required yaml that would spawn the given resources when defined basically, a further customization of the CRD would go as far as to actually roll out the services and resources but working with yaml isn't always straight forward. This is where the Operator SDK and KOPF comes in.

The Operator SDK is a product of the Redhat team and according to their words:

The Operator SDK provides the tools to build, test, and package Operators. Initially, the SDK facilitates the marriage of an application’s business logic (for example, how to scale, upgrade, or backup) with the Kubernetes API to execute those operations. Over time, the SDK can allow engineers to make applications smarter and have the user experience of cloud services. Leading practices and code patterns that are shared across Operators are included in the SDK to help prevent reinventing the wheel.

It works well with Ansible (Of course!!), Go and Helm. They aim to provide a High-level API that utilizes the K8s controller runtime to simplify logic and code scaffolding. It is very efficient in code generation. You can find a guide on using the Operator SDK here: https://sdk.operatorframework.io/build/ .

KOPF is a similar product by the Zalando team but written and used in Python. Personally I tested and liked the simplicity of use and how seamless it interacts with the Kubernetes API. It provides the ability to interact with the cluster and inspect the events related to the services being deployed, thus extending and allowing for a greater number integrations attached to the running resources. It is good as it can utilize pre-built yaml files and modify them based on need or pythonically-generate new yaml definitions and inject it into the cluster. It can be deployed in-cluster as a deployment with a service role allowing access to the required Kubernetes resources or out-of-cluster with the Kubernetes service authenticated into the required cluster. This flexibility and ease of use,plus it is Python, is why it is a strong contender to support the generation of High-Level CRDs. Below is a short guide on how to deploy a simple operator using KOPF.

KOPF Mini-Guide

All the code is available on: https://github.com/ianmuge/kopf-test

Install the K8s service role to be used by the deployment and the CRD definition.

P.S. More stringent restriction should be applied when creating rbac access so as to provide least privileges

kubectl apply -f setup/crd.yml
kubectl apply -f setup/rbac.yml

You can find a sample of the CRD below:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: databases.muge.net
spec:
  scope: Namespaced
  group: muge.net
  versions:
    - name: v1
      served: true
      storage: true
  names:
    kind: Database
    plural: databases
    singular: database
    shortNames:
      - db
      - dbs
  additionalPrinterColumns:
    - name: Type
      type: string
      priority: 0
      JSONPath: .spec.type
      description: The type of the database that should be created
    - name: Tag
      type: string
      priority: 0
      JSONPath: .spec.tag
      description: Image tag

It is important to note that we define the scope, group and version all of which are referenced when calling the operator. For extensive setups, there would be multiple definitions covering multiple kinds .

We were working with GKE so we can build and push the docker image containing the code that will roll out the resources.

export PROJECT_ID="$(gcloud config get-value project -q)"
gcloud auth configure-docker
docker build --no-cache -t operator:latest ./operator
docker tag operator:latest "gcr.io/${PROJECT_ID}/operator:latest"
docker push "gcr.io/${PROJECT_ID}/operator:latest"

We can then use the image we have pushed and create a deployment in-cluster that will use KOPF to roll out the ‘Database’ resources.

envsubst < setup/operator.yml | kubectl apply -f -

And request for the creation of the resources

kubectl apply -f deploy/

Inspecting one of the resource files

apiVersion: muge.net/v1
kind: Database
metadata:
  name: mongo-db
spec:
  type: mongo
  tag: latest

We manage to drastically reduce the yaml definition size by interacting with the abstracted High-Level CRD. You can inspect the logs of the operator deployment to monitor the roll out of the resources.

Helm Sub-Charts

Helm is one of the most versatile and easily sharable form of K8s resource abstractions. It views itself as “The Package Manager for Kubernetes” and over the years it has become analogous with DNF or YUM for Cent-OS and apt for Debian. It holds a large stable repository of pre-packaged resources. Most of the time all you need to provide are variables to customize the resources, with most having defaults, hence simply applying it most of the time would work without modifications. Its underlying architecture is built with Go thus the constructs and templating used might seem similar if you are familiar with Go, it also becomes easier to extend Helm. Helm goes further by managing the life-cycle of the resources that it creates.

Helm’s superpowers comes from it’s easy templating, that is mostly based on yaml, so has a reduced learning-curve and thus easier to adopt. From the large number of contributors and supporters, there is also a large repository where you can find openly shared charts on: https://hub.helm.sh/ . You will find a range of charts covering most requirements and if that doesn’t float your boat, creating Helm charts is rather easy.

Helm sub-charts specifically aims to incorporate multiple charts providing individual resources as a single bundle. Most of the times the sub-charts can be deployed independently or might have slight coupling with the other charts by sharing variables. A useful use-case is deploying the ELK stack: each of the elements (Elasticsearch, Kibana, Logstash and Beats) may be deployed on their own but it generally makes more sense to deploy them together. Below is the structure of a simple ELK Helm Chart with sub-charts. It can be found on: https://github.com/ianmuge/helm-elasticsearch

Sample Structure of Helm Chart with sub-charts

Terraform Sub-Modules

Terraform might be an oddity in the options here but the beauty and extensibility of its sub-modules structure is unsurpassed, in my humble opinion. The fact that it not only supports Kubernetes but also a myriad of other providers makes it the leader in simplifying/abstracting resource deployment. If we concentrate just on Kubernetes, Terraform allows the use of the Kubernetes provider which we can use in a similar fashion to the Helm charts and only calling the modules required. The ability to store the sub-modules remotely, like on Git, away from the HCL requiring it allows the core modules to be maintained independently. There is sequential application of the modules and simpler sharing/coupling of data between the modules as compared to Helm.

Terraform maintains state so modifications over time would be standardized and can be shared among various users and teams. This might be important for common or shared debugging on similar infrastructure. There are a lot of articles on the ease of using Terraform therefore I shall refrain from echoing their opinions.

A good example of creating Terraform sub-modules can be found on: https://www.hashicorp.com/blog/structuring-hashicorp-terraform-configuration-for-production and https://www.terraform.io/docs/modules/composition.html.

Ansible Galaxy/Operators

Another outlier worth considering is Ansible. It is very flexible and easy to use. With the Galaxy hub, it seems like a good mesh between Helm and Terraform as it allows for simple templating and sharing of the services, contains a well-stocked repository of community-shared roles and collections though Ansible Galaxy and allows for the deployment of more resources other than Kubernetes. Sadly Ansible doesn’t maintain state so it might be a bit difficult to maintain some deployed resources. However, it is a good alternative if all else fails or even if your team opts for it and are more comfortable with it.

Conclusion

For SaaS organizations, this might be a good starting point when you need to analyze the alternatives that fit your organization. Some might argue that this should be wholly under the purview of the Infrastructure or Devops team, but at times, development teams might want to deploy software bundles locally or on independent infrastructure without affecting testing or staging environments. Each of the 4 alternatives has their own pros and cons but when all is said and done, it mostly comes down to preference, and requirements of the teams involved. Previously coming from a mostly-development background, this might be a good starting point prior to Devops teams providing self-service interfaces to further simplify and abstract resource deployments.

TL;DR

We compare Custom Operators (Operator SDK and KOPF), Helm with sub-charts , Terraform sub-modules and Ansible as ways to simplify and abstract Kubernetes resources in an attempt to provide High-Level APIs to ease bundled resource deployment.

References

• https://sdk.operatorframework.io/
• https://sdk.operatorframework.io/build/
• https://kopf.readthedocs.io/en/latest/
• https://github.com/ianmuge/kopf-test
• https://gitlab.com/lucj/example-kopf-operator
• https://helm.sh/
• https://github.com/ianmuge/helm-elasticsearch
• https://www.hashicorp.com/blog/structuring-hashicorp-terraform-configuration-for-production
• https://www.terraform.io/docs/modules/composition.html.
• https://galaxy.ansible.com/community/kubernetes

Stay Lazy