Using Harbor and Kubeapps to Serve Custom Helm Charts

In my last post, I walked through the process of deploying Kubeapps in an Enterprise PKS Kubernetes cluster. In this post, I wanted to examine the workflow required for utilizing Harbor, an open source cloud native registry, as an option to serve out a curated set of Helm charts to developers in an organization. We’ll walk through a couple of scenarios, including configuring a “private” project in Harbor that houses Helm charts and container images for a specific group of developers. Building on my last post, we’ll also add this new Helm chart repository into our Kubeapps deployment to allow our developers to deploy our curated applications directly from the Kubeapps dashboard.

Harbor is an an open source trusted cloud native registry project that stores, signs, and scans content. Harbor extends the open source Docker Distribution by adding the functionalities usually required by users such as security, identity and management. Having a registry closer to the build and run environment can improve the image transfer efficiency. Harbor supports replication of images between registries, and also offers advanced security features such as user management, access control and activity auditing. Enterprise support for Harbor Container Registry is included with VMware Enterprise PKS.

Along with the ability to host container images, Harbor also recently added functionality to act as a Helm chart repository. Harbor admins create “projects” that are normally dedicated to certain teams or environments. These projects, public or private, house container images as well as Helm charts to allow our developers to easily deploy curated applications in their Kubernetes cluster(s).

We already have Harbor deployed in our environment as an OpsMan tile. For more information on installing Harbor in conjunction with Enterprise PKS, see documentation here. For instructions detailing the Harbor installation procedure outside of an Enterprise PKS deployment, see the community documentation here.

Let’s get started!!

Creating a Private Project in Harbor

The first thing we’ll need to do is create a new private project that we’ll use to store our container images and Helm charts for our group of developers.

Navigate to the Harbor web UI and login with the admin credentials defined on install. Once logged in, select the + New Project button above the list of existing projects:

Name the project (developers-private-project in our case) and leave the Public option unchecked, as we only want our specific developer group to have access to this project:

Select the newly created project from the list and note the different menus we have available to us regarding the project, including Repositories, which will house our container images, as well as Helm Charts, which will house our Helm charts. We can also add individual members to the project to allow them to authenticate to the project with a username/password combination when pulling/pushing images or Helm charts to the project. For now, let’s select the Configuration tab and select the Automatically scan images on push option. This will instruct Harbor to scan container images for possible CVEs when they are uploaded to the project. Select Save:

Now that we’ve configured our private project, we need to upload our container image that will serve as the basis for our app.

Upload Image to Private Harbor Project

Now that we’ve created our project, we need to populate the project with the container image we are going to use to power this application.

In this example, we are using a simple “To Do List” application. Additional details on the application can be found here.

You’ll need access to a server with docker installed to perform this workflow. I am using the same Linux server where my Helm client is installed.

First, pull the docker image from the public repository:

$ docker pull prydonius/todo

Verify the image has been pulled:

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
prydonius/todo      latest              4089c4ba4620        24 months ago       107MB

Since the project is private, we need to use docker login to authenticate against Harbor. Use the Harbor admin user credentials to authenticate:

$ docker login harbor.pks.zpod.io

Login Succeeded

Now we can tag the todo image with the Harbor url as well as the repo name and tag (which we define, v1 in this case) and push it to our private registry:

$ docker tag prydonius/todo:latest harbor.pks.zpod.io/developers-private-project/todo:v1


$ docker push harbor.pks.zpod.io/developers-private-project/todo:v1

Let’s head over to the Harbor web UI and ensure our image has been successfully uploaded. Navigate to the Projects tab in the left hand menu, select the developers-private-project project, and ensure the todo image is present:

While we here, let’s click on the link for the image and examine the vulnerabilities:

As we selected the option to scan all images on push, our todo container was automatically scanned when it was uploaded. There are a couple of vulnerabilities of “High” severity that we’d want to examine before pushing this app to production. Harbor also provides the ability to set rules in the configuration for each project to ensure containers with known vulnerabilities are not deployed in clusters. Our development environment is not exposed outside of our datacenter so we can let this slide…for now.

Now that we’ve uploaded our container image, we are ready to build our custom Helm chart that will utilize this image in our Harbor repository to build the application in our Kubernetes cluster.

Creating our Custom Helm Chart

As discussed in the last post, Helm uses charts, a collection of files that describe a related set of Kubernetes resources, to simplify the deployment of applications in a Kubernetes cluster. Today, we are going to build a simple Helm chart that deploys our todo app and exposes the app via a load balancer.

We’ll navigate to the server running the Helm client and issue the following command which will build out the scaffolding required for a Helm chart. We’ll call this chart dev-to-do-chart:

$ helm create dev-to-do-chart

The following directory structure will be created:

dev-to-do-chart
|-- Chart.yaml
|-- charts
|-- templates
|   |-- NOTES.txt
|   |-- _helpers.tpl
|   |-- deployment.yaml
|   |-- ingress.yaml
|   `-- service.yaml
`-- values.yaml

The templates/ directory is where Helm finds the YAML definitions for your Services, Deployments and other Kubernetes objects. We will define variables for our deployment in the values.yaml file. Values here can be dynamically set at deployment time to define things such as using an Ingress resource to expose the application or assigning persistent storage to the app.

Let’s edit the values.yaml file to add a couple of additional bits of information. We want to define the image that we will use to back our application deployment. We’ll use the todo container image that we just uploaded to our private project.

Also, since this project/repository is private, we need to create a Kubernetes secret that contains access information for the repository so Kubernetes (and docker) is allowed to pull the image. For additional information on this process, see the Kubernetes documentation here.

In our example, I have created the private-repo-sec secret that we will add to the values.yaml, along with the image name:

$ vi dev-to-do-chart/templates/values.yaml

---
image:
  repository: harbor.pks.zpod.io/developers-private-project/todo
  tag: v1
  pullPolicy: IfNotPresent
imagePullSecrets:
- name: private-repo-sec
___

This will instruct Helm to build a Kubernetes deployment that contains a pod comprised of our todo container from our developers-private-project repo and utilize the private-repo-sec secret to authenticate to the private project.

Let’s also create a README.md (in the dev-to-do-chart directory) file that will display information about the Helm chart that will be visible in our Kubeapps dashboard:

$ vi dev-to-do-chart/README.md 

___
This chart will deploy the "To Do" application. 

Set "Service" to type "LoadBalancer" in the values file to expose the application via an L4 NSX-T load balancer.
___

Now that we’ve configured our chart, we need to package it up so we can upload it to our Harbor chart repo to share with our developers. Navigate back to the parent directory and run the following command to package the chart:

$ helm package ./dev-to-do-chart
Successfully packaged chart and saved it to: /home/user/dev-to-do-chart-0.1.0.tgz

We’ve created and packaged our custom Helm chart for our developers, now we’re ready to upload the chart to Harbor so they can deploy the todo application!!

Uploading Custom Helm Chart to Harbor

There are two ways to upload a Helm chart to harbor:

  • Via the Harbor web UI
  • Via the Helm CLI tool

We are going to use the Helm CLI tool to push the chart to our private project. The first thing we’ll need to do is grab the ca.crt for our project which will allow us to add the chart repo from our Harbor project to our local Helm client.

Navigate back to the homepage for the developers-private-project and select the Registry Certificate link:

This will download the ca.crt that we can use in the following command to push our Helm chart to our project. Since the project is private, we will need to authenticate with the admin users credentials as well as the ca.crt when we add the repo to our Helm repo list:

Note: These commands should be run from the Linux server where the Helm client is installed.

helm repo add developers-private-project --ca-file=ca.crt --username=admin --password=<password> https://harbor.pks.zpod.io/chartrepo/developers-private-project

Let’s verify the repo was added to our Helm repo list:

$ helm repo list
NAME                        URL                                                                                               
developers-private-project  https://harbor.pks.zpod.io/chartrepo/developers-private-project

It should be noted, the native Helm CLI does not support pushing charts so we need to install the helm-push plugin:

$ helm plugin install https://github.com/chartmuseum/helm-push

Now we’re ready to push our chart to our Harbor project:

$ helm push --ca-file=ca.crt --username=admin --password=<password> dev-to-do-chart-0.1.0.tgz developers-private-project
Pushing dev-to-do-chart-0.1.0.tgz to developers-private-project...
Done.

Let’s update our helm repos and search for our chart via the Helm CLI to confirm it is available in our project’s chart repo:

$ helm repo update


$ helm search dev-to-do
NAME                                        CHART VERSION   APP VERSION DESCRIPTION                
developers-private-project/dev-to-do-chart  0.1.0           1.0         A Helm chart for Kubernetes
local/dev-to-do-chart                       0.1.0           1.0         A Helm chart for Kubernetes

Now let’s confirm we can see it in the Harbor web UI as well. Navigate back to the developers-private-project homepage and select the Helm Charts tab:

Awesome!! Now we’re finally ready to add our private chart repo into our Kubeapps deployment so our developers can deploy our to-do app via the Kubeapps dashboard.

Adding a Private Project Helm Chart Repo to Kubeapps

Now that we’ve created our private project, populated with our custom container image and helm chart, we are ready to add the Helm chart repo into our Kubeapps deployment so our developers can deploy the to-do application via the Kubeapps dashboard.

First, we to access our Kubeapps dashboard. Once we’ve authenticated with our token, hover over the Configuration button in the top right-hand corner and select the App Repositories option from the drop down:

Select the Add App Repository button and file in the required details. We are using basic authentication with the Harbor admin user’s credentials. We also will need to add our ca.crt file as well. When finished, select the Install Repo button:

If all the credentials have been populated correctly, we can click on the developers-private-project link and see our dev-to-do-cart Helm chart:

Now, our developers can log in to the Kubeapps dashboard, select the Catalog option, search for our dev-to-do-chart, click on the entry, and select the Deploy button on the subsequent browner page:

In order for our developers to expose this app to access outside of the Kubernetes cluster, we need to change the Service from ClusterIP to LoadBalancer:

Once they’ve made this change, they can select the Submit button to deploy the application in their Kubernetes cluster. The subsequent webpage will show us information about our deployment, including the URL (IP of the NSX-T load balancer that was automatically created, highlighted with a red box in the screenshot) as well as the current state of the deployment:

Note: The automatic creation of the LoadBalancer service is made possible by the integration between NSX-T and Enterprise PKS. These instructions will need to be augmented to provide this same functionality running on a different set of infrastructure.

Navigate to the IP address of the load balancer to test application access:

Boom!! There we have it, our application being served out via our NSX-T L4 load balancer resource.

Conclusion

In this post, we walked through the steps required to create a private Harbor project for our developers that will house custom container images and Helm charts as well as building a custom Helm chart and uploading our container image and custom Helm chart to that private project.

We also walked through the process of adding a private Helm chart repo, hosted by our Harbor deployment, in to our Kubeapps dashboard so our developers can deploy this custom application for testing in their Kubernetes clusters.