Using kubebuilder to make CRDs and clients
kubebuilder is a front-end for a variety of code generators for Kubernetes resources, primarily for the use of creating Custom Resource Definitions and implementing the Operator pattern. It’s a newer, fancier version of the generate-groups.sh script (with about 100% less cut and pasting, too).
It’s a great way to get started with programming in a Kubernetes environment. It incorporates all the tooling and architecture you need to make code generate properly in Kubernetes into your repository, taking over your Makefile
, Dockerfile
, etc to provide you with useful defaults.
Setup
First and foremost, kubebuilder will work, by default with the kubernetes cluster configured in ~/.kube/config
. It is advisable to install something like https://k3s.io to bootstrap something to play with, if you don’t already have a preferred method. We will be working with Kubernetes 1.18 in this post.
Second, the release of kubebuilder you fetch is important as well; we are using kubebuilder 2.3.1. You can find this release here. Kustomize is also necessary, we fetched 3.6.1.
kubebuilder
likes to be unpacked into /usr/local/kubebuilder
for the tests, specifically. You can accomplish this like so:
1 | $ sudo tar vxz --strip-components=1 -C /usr/local/kubebuilder -f kubebuilder.tar.gz |
This will unpack other binaries like a kube-apiserver
and etcd
as well. Put kustomize
somewhere in $PATH
.
The Plan
The plan is to update a CRD with a client we create; this client will simply insert a UUID (if possible) with some basic data. We will then explore this data from the kubectl command-line.
All examples will be in golang and use all the traditional k8s.io
libraries. You should not need to, but could paste a lot of this code if necssary.
Let’s Initialize Our Repository
First things first, understand that kubebuilder
wants to own every detail your repository from the build system, down to the license of the code it generates under. If you don’t want this behavior, creating outside of the standard tree is probably advisable, or outside of the repository entirely, and depending on it instead.
This example removes the license to avoid forcing you to license your own code a specific way, but there are a few options. Every kubebuilder
sub-command has a --help
option.
1 | # yes, you should run under $GOPATH. |
You will see some output like this:
1 | erikh/k8s-api% kubebuilder init --domain example.org --license none |
This sets up a domain (you will get a default if you don’t specify it) for your API which is encoded into the code generation.
If we look at the repository now, quite a bit has changed. We can see a Makefile
as well as a bunch of directories:
1 | drwxrwxr-x 2 erikh erikh 4096 Jun 28 11:15 bin/ |
Creating the API
This is the build system for your API; it hasn’t even arrived yet! We need to run another kubebuilder
command to create it. We need to pick an API group and kind first; we’ll use “apis” and “UUID” respectively.
We’ll need both the resource and the controller for publishing our resource changes in the client; if we don’t provide these options, you will be prompted for them.
1 | erikh/k8s-api% kubebuilder create api --version v1 --group apis --kind UUID --resource --controller |
Making the API do what we want
Our task is fairly simple here; we’re going to edit some struct properties, and move on to making our client.
Let’s first look at our above-noted api/v1/uuid_types.go
:
1 | package v1 |
This is what it should look like (roughly) when you view it. We’re going to make a slight modification to UUIDSpec
and leave the rest alone.
Let’s change the inner struct body to look something more resembling this:
1 | // UUIDSpec defines the desired state of UUID |
Once this is done, type make
at the root of the repository. Try make test
, too. This will generate your code and keep everything up to date.
To install your CRD on to the cluster, type make install
. Do this now, as it will help with the next step.
Finally, the Client
The client leverages the controller-runtime client to interact with your types in a way that makes Golang happy. The code has no idea about your type until the point they’re compiled together; this abstraction allows them to import and work with nearly any type and the same client.
1 | package main |
Walking through what’s happening here:
- We first get our Kubernetes configuration from
~/.kube/config
. If you don’t like this path, change it here, as credentials will be loaded and servers will be used from this configuration. However, if you have followed the steps so far, this is what you have already been using. - We generate a UUID. There are numerous packages for this; we are using
github.com/google/uuid
for our generation. - We construct our object with the UUID represented as string and a random integer because we can. It’s not very random.
- Next we set the namespace and name, two required arguments for any namespaced object in the Kubernetes ecosystem.
- We now take a
runtime.Scheme
, and append our API to it. We then use theruntime.Scheme
in our controller-runtime client. - Finally, we tell the client to create the object. Any error about the name of the object or contents will appear in this step.
Building and Running the Client
Try this:
1 | $ mkdir /tmp/k8s-client |
On success, no output is returned.
Validating we got our data in
To validate our data has indeed arrived, let’s check it out with the standard tools instead of building our own orchestration. CRDs have the added benefit of being integrated and controllable directly from the standard API, making them accessible with tools like kubectl
.
Let’s try that (I ran it a few times, you should see one for each time you ran it successfully):
1 | $ kubectl get uuid -n runs |
Let’s describe one to see if we got our random integer:
1 | $ kubectl describe uuid -n runs b97e07ab-2399-4100-879f-0e3049971552 |
1 | Name: b97e07ab-2399-4100-879f-0e3049971552 |
We can see that indeed, not only is our random integer there, but it is quite large too. What’s also important is that we can see the update manifest of the item, this would be useful for auditing a badly behaving application or user.
kubebuilder == rails new
kubebuilder, as we saw, is basically rails new
for Kubernetes. Those of you familiar with the ruby/rails ecosystem may be familiar with this being a single command to generate giant swaths of code to edit later. I imagine it’s scope will expand to handling other patterns in Kubernetes and I look forward to using it for future projects.
I’ve put the code I generated here, if you want to pull it down to play around. Enjoy!