Kubernetes Cluster Creation in VMware Cloud on AWS with CAPV: Part 2

In Part 1 of my series on deploying Kubernetes clusters to VMware on AWS environments with ClusterAPI Provider vSphere, I detailed the processes required to stand up the CAPV management plane. After completing those steps, I am ready to provision a workload cluster to VMC using CAPV.

Creating Workload Clusters

The CAPV cluster is the brains of the operations but I still need to deploy some workload clusters for my teams of developers to deploy their applications onto. The management cluster helps automate the provisioning of all of the provider components to support my workload clusters as well as instantiating the VMs provisioned as a Kubernetes cluster. The basic use case here is that I, as the infrastructure admin, am responsible for utilizing the CAPV management cluster to provision multiple workload clusters that can support individual teams of developers, individual application deployments, etc. The CAPV management cluster allows me to easily deploy a consistent cluster in a repeatable fashion with very little manual effort. I can quickly deploy a test, dev, prod set of clusters for a team or deploy 5 different workload clusters for 5 different groups of developers.

Another usage pattern, and probably more aligned with the DevOps mentality, is to configure authentication to the Management cluster and use Kubernetes RBAC constructs to assign teams the ability to create workload clusters in their respective namespaces. This way, developers have full control over when and what they provision as long as it fits within the limitations established for them by the infrastructure team. True self service!!

In this exercise, I’m going to deploy a workload cluster named “prod-cluster”, composed of 1 master node and 4 worker nodes. I’ll start by using the same docker “manifests” image I used in Part 1 of the series, along with the same envvars.txt file, to create the .yaml file scaffolding for my workload cluster. Note, I’m using a different cluster name (prod-cluster) so all of my config files will be stored in a new directory:

# docker run --rm \
  -v "$(pwd)":/out \
  -v "$(pwd)/envvars.txt":/envvars.txt:ro \
  gcr.io/cluster-api-provider-vsphere/release/manifests:v0.5.4 \
  -c prod-cluster

Just as in the management cluster example, the “manifests” docker image creates the .yaml files I’ll need to create my workload cluster. The first thing I’ll do is create the Cluster resource using the cluster.yaml file:

# kubectl apply -f ./out/prod-cluster/cluster.yaml 

cluster.cluster.x-k8s.io/prod-cluster created
vspherecluster.infrastructure.cluster.x-k8s.io/prod-cluster created

In ClusterAPI terms, the Cluster resource defines cluster-wide configuration such as generic networking concepts like pod and service ranges or DNS domain.

Next, I am ready to create the control plane node with the controlplane.yaml file. This file defines the configuration of a vSphere virtual machine as well as a kubeadm bootstrap config that instantiates the VM as a Kubernetes master node:

# kubectl apply -f ./out/prod-cluster/controlplane.yaml 

kubeadmconfig.bootstrap.cluster.x-k8s.io/prod-cluster-controlplane-0 created
machine.cluster.x-k8s.io/prod-cluster-controlplane-0 created
vspheremachine.infrastructure.cluster.x-k8s.io/prod-cluster-controlplane-0 created

Note the output of the kubectl command, which informs me that a kubeadm bootstrap config as well as a machine (virtual machine) was created. If I navigate to the VMC console, I can observe my control plane VM being created, using the CentOS template defined in the envvars.txt file:

Now I’m ready to create my worker nodes for the cluster, which are defined by the machinedeployment.yaml file. In ClusterAPI terms, a MachineDeployment is analogues to a Deployment in the Kubernetes world. MachineDeployments manage the desired state of a group of Machines (VMs) just as Deployments manage the desired state of pods.

Before I deploy my worker nodes, I need to up the replica count to 4 in the .yaml file. This will ensure my MachineDeployment contains 4 worker nodes:

# vi ./out/prod-cluster/machinedeployment.yaml 

  replicas: 4
      cluster.x-k8s.io/cluster-name: prod-cluster

Now I’m ready to create my worker nodes!

# kubectl apply -f ./out/prod-cluster/machinedeployment.yaml 

kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/prod-cluster-md-0 created
machinedeployment.cluster.x-k8s.io/prod-cluster-md-0 created
vspheremachinetemplate.infrastructure.cluster.x-k8s.io/prod-cluster-md-0 created

Again, note the output of the kubectl command which confirms that a bootstrap config has been created to instantiate these VMs as worker nodes and “join” them under the control of the existing control plane node to form a cluster. We can also confirm the Machines (VMs) have been created in the VMC console:

At this point, I now have a workload cluster that consists of 1 master and 4 worker nodes deployed in my VMC environment.

Accessing the Workload Cluster

Now I’ll need to obtain the kubeconfig file that will allow me to interact with the workload cluster. The kubeconfig file used to access workload clusters is stored as a Kubernetes Secret on the management cluster. The secrets are stored as <cluster-name>-kubeconfig. I can confirm my prod-cluster kubeconfig is available with the following command:

kubectl get secrets

NAME                        TYPE       DATA   AGE
prod-cluster-kubeconfig     Opaque     1      20m

I can also use the following command to decode the secret and place the plain text at ./out/prod-cluster/kubeconfig for later use:

kubectl get secret prod-cluster-kubeconfig -o=jsonpath='{.data.value}' | \
{ base64 -d 2>/dev/null || base64 -D; } >./out/prod-cluster/kubeconfig

In order to start interacting with the prod-cluster workload cluster, I’ll reset my KUBECONFIG environment variable to the new kubeconfig I just pulled out of the management cluster:

# export KUBECONFIG="$(pwd)/out/prod-cluster/kubeconfig"

Now I’ll use kubectl to examine the nodes of my workload cluster:

# kubectl get nodes

NAME                                 STATUS     ROLES    AGE   VERSION
prod-cluster-controlplane-0          NotReady   master   25m   v1.16.3
prod-cluster-md-0-55f55ffdb9-4467d   NotReady   <none>   16m   v1.16.3
prod-cluster-md-0-55f55ffdb9-b9hv8   NotReady   <none>   16m   v1.16.3
prod-cluster-md-0-55f55ffdb9-x9v7z   NotReady   <none>   16m   v1.16.3
prod-cluster-md-0-55f55ffdb9-xljxw   NotReady   <none>   16m   v1.16.3

Notice that I am now receiving information about my workload cluster (1 master/4 workers) instead of my management cluster (1 master). Great!

But I’m not done yet… Notice that the nodes are all in NotReady state. This is because workload clusters do not have any add-ons applied aside from those added by kubeadm. Nodes in the workload clusters will be in the NotReady state until I apply a [Container Network Interface][25] (CNI) add-on.

The “manifests” docker images automatically creates an addon.yaml file that contains the configuration to instantiate [Calico] as the CNI for the workload cluster, but you can use any CNI you wish. For the sake of simplicity, I’m going to utilize the default Calico config provided:

kubectl apply -f ./out/prod-cluster/addons.yaml

configmap/calico-config created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created

Note the Calico resources defined in the addons.yaml file include some RBAC config as well as a Deployment and DaemonSet for the Calico components running on the cluster, among other things.

Now, I can verify my nodes have all transitioned into a Ready state:

# kubectl get nodes

NAME                                 STATUS   ROLES    AGE   VERSION
prod-cluster-controlplane-0          Ready    master   39m   v1.16.3
prod-cluster-md-0-55f55ffdb9-4467d   Ready    <none>   29m   v1.16.3
prod-cluster-md-0-55f55ffdb9-b9hv8   Ready    <none>   29m   v1.16.3
prod-cluster-md-0-55f55ffdb9-x9v7z   Ready    <none>   29m   v1.16.3
prod-cluster-md-0-55f55ffdb9-xljxw   Ready    <none>   29m   v1.16.3

VOILA!! Now I’m ready to start deploying my workloads to the cluster!!


This concludes Part 2 of my post on deploying Kubernetes Clusters to a VMware Cloud on AWS environment at scale with ClusterAPI Provider vSphere! CAPV is a very powerful tool that helps infrastructure admins provide their users with a consistent, scalable workflow for deploying and managing Kubernetes clusters on top of vSphere infrastructure.

Stay tuned to the blog for a more in depth look into how Cluster API is utilized in Project Pacific to provide self service Kubernetes cluster management natively within vSphere!

Leave a Reply

Your email address will not be published. Required fields are marked *