In my last post, I walked through the process of deploying Kubeapps in an Enterprise PKS Kubernetes cluster. In this post, I wanted to examine the workflow required for utilizing Harbor, an open source cloud native registry, as an option to serve out a curated set of Helm charts to developers in an organization. We’ll walk through a couple of scenarios, including configuring a “private” project in Harbor that houses Helm charts and container images for a specific group of developers. Building on my last post, we’ll also add this new Helm chart repository into our Kubeapps deployment to allow our developers to deploy our curated applications directly from the Kubeapps dashboard.
Harbor is an an open source trusted cloud native registry project that stores, signs, and scans content. Harbor extends the open source Docker Distribution by adding the functionalities usually required by users such as security, identity and management. Having a registry closer to the build and run environment can improve the image transfer efficiency. Harbor supports replication of images between registries, and also offers advanced security features such as user management, access control and activity auditing. Enterprise support for Harbor Container Registry is included with VMware Enterprise PKS.
Along with the ability to host container images, Harbor also recently added functionality to act as a Helm chart repository. Harbor admins create “projects” that are normally dedicated to certain teams or environments. These projects, public or private, house container images as well as Helm charts to allow our developers to easily deploy curated applications in their Kubernetes cluster(s).
We already have Harbor deployed in our environment as an OpsMan tile. For more information on installing Harbor in conjunction with Enterprise PKS, see documentation here. For instructions detailing the Harbor installation procedure outside of an Enterprise PKS deployment, see the community documentation here.
Let’s get started!!
Creating a Private Project in Harbor
The first thing we’ll need to do is create a new private project that we’ll use to store our container images and Helm charts for our group of developers.
Navigate to the Harbor web UI and login with the admin credentials defined on install. Once logged in, select the
+ New Project button above the list of existing projects:
Name the project (
developers-private-project in our case) and leave the
Public option unchecked, as we only want our specific developer group to have access to this project:
Select the newly created project from the list and note the different menus we have available to us regarding the project, including
Repositories, which will house our container images, as well as
Helm Charts, which will house our Helm charts. We can also add individual members to the project to allow them to authenticate to the project with a username/password combination when pulling/pushing images or Helm charts to the project. For now, let’s select the
Configuration tab and select the
Automatically scan images on push option. This will instruct Harbor to scan container images for possible CVEs when they are uploaded to the project. Select
Now that we’ve configured our private project, we need to upload our container image that will serve as the basis for our app.
Upload Image to Private Harbor Project
Now that we’ve created our project, we need to populate the project with the container image we are going to use to power this application.
In this example, we are using a simple “To Do List” application. Additional details on the application can be found here.
You’ll need access to a server with
docker installed to perform this workflow. I am using the same Linux server where my Helm client is installed.
First, pull the docker image from the public repository:
$ docker pull prydonius/todo
Verify the image has been pulled:
$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE prydonius/todo latest 4089c4ba4620 24 months ago 107MB
Since the project is private, we need to use
docker login to authenticate against Harbor. Use the Harbor admin user credentials to authenticate:
$ docker login harbor.pks.zpod.io Login Succeeded
Now we can tag the
todo image with the Harbor url as well as the repo name and tag (which we define,
v1 in this case) and push it to our private registry:
$ docker tag prydonius/todo:latest harbor.pks.zpod.io/developers-private-project/todo:v1 $ docker push harbor.pks.zpod.io/developers-private-project/todo:v1
Let’s head over to the Harbor web UI and ensure our image has been successfully uploaded. Navigate to the
Projects tab in the left hand menu, select the
developers-private-project project, and ensure the
todo image is present:
While we here, let’s click on the link for the image and examine the vulnerabilities:
As we selected the option to scan all images on push, our
todo container was automatically scanned when it was uploaded. There are a couple of vulnerabilities of “High” severity that we’d want to examine before pushing this app to production. Harbor also provides the ability to set rules in the configuration for each project to ensure containers with known vulnerabilities are not deployed in clusters. Our development environment is not exposed outside of our datacenter so we can let this slide…for now.
Now that we’ve uploaded our container image, we are ready to build our custom Helm chart that will utilize this image in our Harbor repository to build the application in our Kubernetes cluster.
Creating our Custom Helm Chart
As discussed in the last post, Helm uses charts, a collection of files that describe a related set of Kubernetes resources, to simplify the deployment of applications in a Kubernetes cluster. Today, we are going to build a simple Helm chart that deploys our
todo app and exposes the app via a load balancer.
We’ll navigate to the server running the Helm client and issue the following command which will build out the scaffolding required for a Helm chart. We’ll call this chart
$ helm create dev-to-do-chart
The following directory structure will be created:
dev-to-do-chart |-- Chart.yaml |-- charts |-- templates | |-- NOTES.txt | |-- _helpers.tpl | |-- deployment.yaml | |-- ingress.yaml | `-- service.yaml `-- values.yaml
templates/ directory is where Helm finds the YAML definitions for your Services, Deployments and other Kubernetes objects. We will define variables for our deployment in the
values.yaml file. Values here can be dynamically set at deployment time to define things such as using an Ingress resource to expose the application or assigning persistent storage to the app.
Let’s edit the
values.yaml file to add a couple of additional bits of information. We want to define the image that we will use to back our application deployment. We’ll use the
todo container image that we just uploaded to our private project.
Also, since this project/repository is private, we need to create a Kubernetes secret that contains access information for the repository so Kubernetes (and docker) is allowed to pull the image. For additional information on this process, see the Kubernetes documentation here.
In our example, I have created the
private-repo-sec secret that we will reference in the
values.yaml file, along with the image name:
Note: Currently this means that the images from the private repo will only be able to be deployed to the namespace (or namespaces) that contain this secret. The Bitnami team is currently look at a design change that would allow for a better workflow of Kubeapps chart installations from private repos.
$ vi dev-to-do-chart/templates/values.yaml --- image: repository: harbor.pks.zpod.io/developers-private-project/todo tag: v1 pullPolicy: IfNotPresent imagePullSecrets: - name: private-repo-sec ___
This will instruct Helm to build a Kubernetes deployment that contains a pod comprised of our
todo container from our
developers-private-project repo and utilize the
private-repo-sec secret to authenticate to the private project.
Let’s also create a
README.md (in the
dev-to-do-chart directory) file that will display information about the Helm chart that will be visible in our Kubeapps dashboard:
$ vi dev-to-do-chart/README.md ___ This chart will deploy the "To Do" application. Set "Service" to type "LoadBalancer" in the values file to expose the application via an L4 NSX-T load balancer. ___
Now that we’ve configured our chart, we need to package it up so we can upload it to our Harbor chart repo to share with our developers. Navigate back to the parent directory and run the following command to package the chart:
$ helm package ./dev-to-do-chart Successfully packaged chart and saved it to: /home/user/dev-to-do-chart-0.1.0.tgz
We’ve created and packaged our custom Helm chart for our developers, now we’re ready to upload the chart to Harbor so they can deploy the
Uploading Custom Helm Chart to Harbor
There are two ways to upload a Helm chart to harbor:
- Via the Harbor web UI
- Via the Helm CLI tool
We are going to use the Helm CLI tool to push the chart to our private project. The first thing we’ll need to do is grab the
ca.crt for our project which will allow us to add the chart repo from our Harbor project to our local Helm client.
Navigate back to the homepage for the
developers-private-project and select the
Registry Certificate link:
This will download the
ca.crt that we can use in the following command to push our Helm chart to our project. Since the project is private, we will need to authenticate with the admin users credentials as well as the
ca.crt when we add the repo to our Helm repo list:
Note: These commands should be run from the Linux server where the Helm client is installed.
helm repo add developers-private-project --ca-file=ca.crt --username=admin --password=<password> https://harbor.pks.zpod.io/chartrepo/developers-private-project
Let’s verify the repo was added to our Helm repo list:
$ helm repo list NAME URL developers-private-project https://harbor.pks.zpod.io/chartrepo/developers-private-project
It should be noted, the native Helm CLI does not support pushing charts so we need to install the
$ helm plugin install https://github.com/chartmuseum/helm-push
Now we’re ready to push our chart to our Harbor project:
$ helm push --ca-file=ca.crt --username=admin --password=<password> dev-to-do-chart-0.1.0.tgz developers-private-project Pushing dev-to-do-chart-0.1.0.tgz to developers-private-project... Done.
Let’s update our helm repos and search for our chart via the Helm CLI to confirm it is available in our project’s chart repo:
$ helm repo update $ helm search dev-to-do NAME CHART VERSION APP VERSION DESCRIPTION developers-private-project/dev-to-do-chart 0.1.0 1.0 A Helm chart for Kubernetes local/dev-to-do-chart 0.1.0 1.0 A Helm chart for Kubernetes
Now let’s confirm we can see it in the Harbor web UI as well. Navigate back to the
developers-private-project homepage and select the
Helm Charts tab:
Awesome!! Now we’re finally ready to add our private chart repo into our Kubeapps deployment so our developers can deploy our
to-do app via the Kubeapps dashboard.
Adding a Private Project Helm Chart Repo to Kubeapps
Now that we’ve created our private project, populated with our custom container image and helm chart, we are ready to add the Helm chart repo into our Kubeapps deployment so our developers can deploy the
to-do application via the Kubeapps dashboard.
First, we to access our Kubeapps dashboard. Once we’ve authenticated with our token, hover over the
Configuration button in the top right-hand corner and select the App Repositories option from the drop down:
Add App Repository button and file in the required details. We are using basic authentication with the Harbor admin user’s credentials. We also will need to add our
ca.crt file as well. When finished, select the
Install Repo button:
If all the credentials have been populated correctly, we can click on the
developers-private-project link and see our
dev-to-do-cart Helm chart:
Now, our developers can log in to the Kubeapps dashboard, select the
Catalog option, search for our
dev-to-do-chart, click on the entry, and select the
Deploy button on the subsequent browner page:
In order for our developers to expose this app to access outside of the Kubernetes cluster, we need to change the
Once they’ve made this change, they can select the
Submit button to deploy the application in their Kubernetes cluster. The subsequent webpage will show us information about our deployment, including the URL (IP of the NSX-T load balancer that was automatically created, highlighted with a red box in the screenshot) as well as the current state of the deployment:
Note: The automatic creation of the
LoadBalancer service is made possible by the integration between NSX-T and Enterprise PKS. These instructions will need to be augmented to provide this same functionality running on a different set of infrastructure.
Navigate to the IP address of the load balancer to test application access:
Boom!! There we have it, our application being served out via our NSX-T L4 load balancer resource.
In this post, we walked through the steps required to create a private Harbor project for our developers that will house custom container images and Helm charts as well as building a custom Helm chart and uploading our container image and custom Helm chart to that private project.
We also walked through the process of adding a private Helm chart repo, hosted by our Harbor deployment, in to our Kubeapps dashboard so our developers can deploy this custom application for testing in their Kubernetes clusters.