In Parts 1 and 2 of my series on installing and configuring the Container Service Extension for VMware Cloud Director, I focused on setting the CSE server up to support CSE Standard Kubernetes cluster creation.
- Part 1: CSE Software Installation, Server Configuration, and Deployment
- Part 2: CSE Client Setup and Kubernetes Cluster Deployment
- Part 3: Configuring CSE Enterprise with VMware Enterprise PKS
CSE Standard clusters are comprised of deployed vApps that utilize NSX-V networking resources, utilizing Weave as the Container Network Interface for the Kubernetes clusters. In Part 3 of my series, I wanted to take some time to look at configuring the CSE Server to support the creation of CSE Enterprise Kubernetes clusters. CSE Enterprise clusters are comprised of VMware Enterprise PKS Kubernetes clusters deployed on top of NSX-T networking resources, utilizing the NSX Container Plugin as a CNI. CSE Enterprise brings enterprise grade features and functionality to CSE that include, but are not limited to:
- HA, multi-master Kubernetes clusters
- Dynamic persistent storage provisioning with the vSphere Cloud Provider integration
- Automated Day 1 and Day 2 Kubernetes cluster management via Bosh Director
- Microsegmentation capability for Kubernentes resources via integration with NSX-T
- Automated creation of Kubernetes service type LoadBalancer and ingress resrouces via NSX-T L4/L7 load balancers
- Support for Harbor, an open source cloud native registry
How Does CSE Enterprise Work?
Before we get started on the configuration steps, I want to take some time to explore how the CSE Server communicates with both Enterprise PKS and NSX-T to automate the creation of Kubernetes clusters, all triggered by a request from an end user via the vcd-cli. When a user issues a vcd cse create cluster command, they are either utilizing an OrgVDC that has been enabled for CSE Standard, CSE Enterprise, or no Kubernetes provider at all. If their OvDC has been enabled to support CSE Enterprise cluster creation (and they have the correct permission to create clusters), the cse extension of the vcd-cli communicates directly with the PKS and NSX-T APIs to initiate the request to create an Enterprise PKS Kubernetes cluster as well as creating all of the NSX-T resources (T1 routers, Load Balancers, logical switches, etc.) to support the cluster. The CSE server also communicates with the NSX-T API to create DFW rules that isolate provisioned clusters from each other to support network resource isolation.
As opposed to CSE Standard, which creates a vApp from a vApp template and then instantiates the vApp as a Kubernetes cluster, the PKS control plane receives an API call from the CSE server and utilizes the plan assigned to the OrgVDC to create the cluster. There is not vApp or vApp template required to provision CSE Enterprise clusters.
CSE PKS Config File
Before we get started, I want to note that I have already deployed both VMware Enterprise PKS and VMware NSX-T Datacenter in my envrironment. CSE will need to hook into an existing deployment of PKS and NSX-T so these components must be configured before configuring CSE Enterprise.
As mentioned in Part 1 of the series, CSE uses a yaml config file that contains information about the vSphere/VCD environment(s) that CSE will need to interact with to deploy CSE Standard Kubernetes clusters. Similarly, we’ll need to utilize a second yaml file that will contain all of the information that CSE will need to communicate with the PKS/NSX-T APIs to orchestrate the provisioning of Enterprise PKS Kubernetes clusters. You can view a sample PKS config file on the CSE docs site but I will walk through through the required components below.
The first step is to create a pks-config.yaml file that I’ll populate with the values below. I’m going to create the file in the same directory as my existing config.yaml file:
$ vi ~/pks-config.yaml
Now I can populate the file with the required information, detailed below:
pks_api_servers section
This section of the config file will contain information that CSE will use to communicate with the PKS API server:
pks_api_servers:
- clusters:
- RegionA01-COMP01 <--- Cluster name as it appears in vCenter
cpi: 6bd3f75a31e5c3f2d65e <--- Bosh CPI value, see note below
datacenter: RegionA01 <--- Datacenter name as it appears in vCenter
host: pks.corp.local <--- FQDN of the PKS API server
name: PKS-1 <--- Name used to identity this particular PKS API server, user defined in this file
port: '9021' <--- Port used to for PKS API communication, default = 9021
uaac_port: '8443' <--- Port used to authenticate against the PKS UAAC service, default = 8443
vc: vcsa-01 <--- vCenter name as it appears in VCD
verify: false <--- Set to "true" to verify SSL certificates
Note: The cpi value above can be obtained by utilizing the Bosh CLI to run the following command on the Opsman VM in the PKS environment:
$ bosh cpi-config
Using environment '172.31.0.2' as client 'ops_manager'
cpis:
- migrated_from:
- name: ""
name: 6bd3f75a31e5c3f2d65e <--- cpi value
---output omitted---
pks_accounts section
This section defines the PKS UAAC credentials that CSE will use to authenticate against the PKS API when creating clusters:
pks_accounts:
- name: PKS1-admin
pks_api_server: PKS-1
secret: fhB-Z5hHcsXl_UnC86dkuYlTzQPoE3Yz
username: admin
nameis a user defined value that identifies the name of this credentials set. This is used in case there are multiple PKS API servers, which require their own set of credentialspks_api_serveris the user defined variable for the PKS API server defined by thenamevalue in thepks_api_serverssection (PKS-1in my example)usernameis the UAAC username that CSE will use to authenticate against the PKS API server (using theadminuser in my example)secretis the secret tied to the user defined in theusernamevalue. If using theadminuser, you can obtain the secret by navigating to the Opsman web UI, selecting the PKS tile, selecting the credentials tab, and then selecting Link to Credential next to thePks Uaa Management Admin Cliententry. See screenshot below:
pvdcs section
This section defines the PvDC(s) in VCD that support the Org and OrgVDCs that are enabled to support PKS cluster creation via CSE:
pvdcs:
- cluster: RegionA01-COMP01 <--- Cluster name as it appears in vCenter
name: prod-pvdc <--- PvDC name as it appears in VCD
pks_api_server: PKS-1 <--- user defined variable for the PKS API server defined by the `name` value in the `pks_api_servers` section
nsxt_servers section
This section defines information that CSE will need to communicate with the NSX-T API to create the NSX-T resources to back the Kubernetes clusters as well as create the DFW to isolate the clusters provisioned via CSE:
nsxt_servers:
- distributed_firewall_section_anchor_id: 9d6d2a5c-c32d-419c-ada8-e5208475ca88
host: nsxmgr-01a.corp.local
name: nsxt-server-1
nodes_ip_block_ids:
- eac47bea-5304-4a7b-8c10-9b16e62f1cda
password: MySuperSecretPassword!
pks_api_server: PKS-1
pods_ip_block_ids:
- 27d2d1c3-969a-46a5-84b9-db503ce2edd5
username: admin
verify: false
distributed_firewall_section_anchor_idis the UUID of the “Default Layer3 Section” DFW rule created by PKS during the installation of Enterprise PKS, see screenshot below:
hostis the FDQN of the NSX-T Management servernameis a user defined name for this particular NSX-T instancenodes_ip_block_idsis the UUID of the IP block created in NSX-T to be used by PKS for cluster nodes, see screenshot below:
passwordis the password for the NSX-T user defined in the config filepks_api_serveris the user defined variable for the PKS API server defined by thenamevalue in thepks_api_serverssection (PKS-1in my example)pods_ip_block_idsis the UUID of the IP block created in NSX-T to be used by PKS to assign IP addresses to the pods running in a cluster.
usernameis the NSX-T username CSE will use to authenticate against the APIverifycan be set to “true” to verify SSL certificates
There we have it! This is the bare minimun information required for CSE to deploy Enterprise PKS clusters via vcd-cli. Do note, in my example deployment above, I have a single PKS deployment, cluster, PvDC, and NSX-T instance that CSE is communicate with. You can also add additional of each resource as required for your deployment.
Enabling CSE Enterprise on the CSE Server
Now that I’ve built my PKS config file for CSE, I’m ready to enable CSE Enterprise for my tenants’ org(s). I’ve already deployed the CSE server only utilizing the CSE Standard methodology so the first thing I’ll need to do is update the config.yaml file to point to the new pks-config.yaml file:
$ vi ~/config.yaml
---output omitted---
pks_config: pks-config.yaml
---output omitted---
After updating the config.yaml file with the filename of the PKS config file, I’ll stop the CSE service on the CSE server and re-install CSE:
$ sudo systemctl stop cse
$ cse install -c config.yaml --skip-template-creation
Required Python version: >= 3.7.3
Installed Python version: 3.7.3 (default, Nov 13 2019, 16:41:06)
[GCC 7.4.0]
Validating config file 'config.yaml'
Connected to AMQP server (vcd.corp.local:5672)
InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised.
Connected to vCloud Director (vcd.corp.local:443)
Connected to vCenter Server 'vcsa-01' as 'administrator@corp.local' (vcsa-01a.corp.local:443)
Config file 'config.yaml' is valid
Validating PKS config file 'pks-config.yaml'
Connected to PKS server (PKS-1 : pks.corp.local)
Connected to NSX-T server (nsxmgr-01a.corp.local)
PKS Config file 'pks-config.yaml' is valid
Installing CSE on vCloud Director using config file 'config.yaml'
Connected to vCD as system administrator: vcd.corp.local:443
Checking for AMQP exchange 'cse-ext'
AMQP exchange 'cse-ext' is ready
Registered cse as an API extension in vCD
Registering Right: CSE NATIVE DEPLOY RIGHT in vCD
Registering Right: PKS DEPLOY RIGHT in vCD
Creating catalog 'cse'
Created catalog 'cse'
Skipping creation of templates.
Configuring NSX-T server (nsxt-server-1) for CSE. Please check install logs for details.
As you can see from the output of the cse install command, CSE was able to communicate with the PKS and NSX-T API. The CSE server also added a new rights bundle (PKS DEPLOY RIGHT) that I will use to grant users the ability to provision CSE Enterprise Kubernetes clusters.
After successfully installing CSE with the PKS config, I’ll restart the CSE service:
$ sudo systemctl start cse
and verify the service is running as expected (look for the MessageConsumer threads):
$ sudo systemctl status cse
[sudo] password for cse:
● cse.service
Loaded: loaded (/etc/systemd/system/cse.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2019-11-18 12:08:18 EST; 1 day 3h ago
Main PID: 911 (sh)
Tasks: 7
Memory: 3.6M
CGroup: /system.slice/cse.service
├─911 /bin/sh /home/cse/cse.sh
└─914 /home/cse/cse-env/bin/python3.7 /home/cse/cse-env/bin/cse run -c /home/cse/config.yaml
Nov 18 12:08:22 cse sh[911]: CSE installation is valid
Nov 18 12:08:23 cse sh[911]: Started thread 'MessageConsumer-0 (139897843828480)'
Nov 18 12:08:23 cse sh[911]: Started thread 'MessageConsumer-1 (139897746745088)'
Nov 18 12:08:24 cse sh[911]: Started thread 'MessageConsumer-2 (139897755137792)'
Nov 18 12:08:24 cse sh[911]: Started thread 'MessageConsumer-3 (139897835435776)'
Nov 18 12:08:24 cse sh[911]: Started thread 'MessageConsumer-4 (139897738352384)'
Nov 18 12:08:24 cse sh[911]: Container Service Extension for vCloud Director
Nov 18 12:08:24 cse sh[911]: Server running using config file: /home/cse/config.yaml
Nov 18 12:08:24 cse sh[911]: Log files: cse-logs/cse-server-info.log, cse-logs/cse-server-debug.log
Nov 18 12:08:24 cse sh[911]: waiting for requests (ctrl+c to close)
Voila!! I’ve successfully configured CSE to support Enterprise PKS cluster creation!! Now, let’s have a look at enabling a tenant to deploy CSE Enterprise clusters via the vcd-cli.
Onboarding Tenants and Provisioning CSE Enterprise Kubernetes Clusters
The first thing I’ll need to do is login to the VCD deployment via the vcd-cli as the System Admin user:
$ vcd login vcd.corp.local system admin -iw
Next, I can use the cse extension to view the existing OvDCs and what Kubernetes provider is enabled for each OvDC:
$ vcd cse ovdc list
name org k8s provider
--------- -------- --------------
base-ovdc base-org native
acme-ovdc AcmeCorp none
From the results above, note that the base-ovdc is enabled for the Kubernetes provider of native, which means users in that org with the correct permissions can create CSE Standard clusters. The acme-ovdc does not have any Kubernetes provider assigned yet, but I want to enable it to support CSE Enterprise cluster creation.
First, I’ll need to instruct the vcd-cli to “use” the AcmeCorp org:
$ vcd org use AcmeCorp
now using org: 'AcmeCorp', vdc: 'acme-ovdc', vApp: ''.
Then, I’ll add the new PKS rights bundle to the AcmeCorp org:
$ vcd right add "{cse}:PKS DEPLOY RIGHT" -o AcmeCorp
Rights added to the Org 'AcmeCorp'
Before moving on to to the next step, I created a new role in the VCD tenant portal by the name of “orgadmin-k8” which mimics the “Org Admin” permissions. I also created a new user named “pks-k8-admin” and assigned that user the “orgadmin-k8” role.
After creating the new role and user, I need to use the vcd-cli to add the PKS DEPLOY RIGHT rights bundle to the custom role:
$ vcd role add-right "orgadmin-k8" "{cse}:PKS DEPLOY RIGHT"
Rights added successfully to the role 'orgadmin-k8'
The last thing I need to do as the System admin is enable the OvDC to support CSE Enterprise cluster with the following command:
$ vcd cse ovdc enable "acme-ovdc" -o AcmeCorp -k ent-pks -p "xsmall" -d "corp.local"
metadataUpdate: Updating metadata for Virtual Datacenter acme-ovdc(07972d80-faf1-48c7-8f7e-cea92ce7cc6e)
task: 9fced925-4ec8-47c3-a45e-7903b14b0c8d, Updated metadata for Virtual Datacenter acme-ovdc(07972d80-faf1-48c7-8f7e-cea92ce7cc6e), result: success
where -k is the Kubernetes provider (PKS in this case), -p is the PKS plan we want clusters to use when provisioned from CSE for this OvDC, and -d is the domain name that PKS will use for the hostname of each cluster.
Note, CSE uses compute profiles to create Availability Zones for each OvDC that is enabled. When vce cse ovdc enable command above is issued, CSE talks to the PKS API to create a compute profile that defines the Availability Zones for all clusters created in this OvDC as the Resource Pool that is assigned to that OvDC. This ensures that users that provision clusters to this OvDC will have their compute isolated (via the Resource Pool) from other clusters provisioned in other OvDCs in the VCD environment.
Verify that I now see a Kubernetes provider for our acme-ovdc after enabling it:
$ vcd cse ovdc list
name org k8s provider
--------- -------- --------------
base-ovdc base-org native
acme-ovdc AcmeCorp ent-pks
There we have it! Now I can tell the pks-k8-admin user that they are ready to provision some CSE Enterprise clusters!!
Provisioning CSE Enterprise Clusters
Now that I, as the system admin, have done the pre-work to enable the Org to support CSE Enterprise clusters, I’m ready to turn the tenants loose and allow them to provision CSE Enterprise Kubernetes clusters.
First, the orgadmin-k8 user will log in to VCD via the vcd-cli:
$ vcd login vcd.corp.local AcmeCorp pks-k8-admin -iw
All they have to do now is run the vcd cse cluster create command and CSE will handle the rest:
$ vcd cse cluster create prod-1 --nodes 2
property value
--------------------------- -----------------
kubernetes_master_host prod-1.corp.local
kubernetes_master_ips In Progress
kubernetes_master_port 8443
kubernetes_worker_instances 2
last_action CREATE
last_action_description Creating cluster
last_action_state in progress
name prod-1
worker_haproxy_ip_addresses
where prod-1 is the name of our cluster and --nodes is the number of worker nodes assigned to the cluster. As you can see, the FQDN of the master host will be “cluster-name”.”domain” where “domain” was defined when we enabled the OvDC.
Once the cluster has finished provisioning, we can use the cse extension to gather information about the cluster:
$ vcd cse cluster info prod-1
property value
--------------------------- -------------------------------------------------
k8s_provider ent-pks
kubernetes_master_host prod-1.corp.local
kubernetes_master_ips 10.40.14.37
kubernetes_master_port 8443
kubernetes_worker_instances 2
last_action CREATE
last_action_description Instance provisioning completed
last_action_state succeeded
name prod-1
network_profile_name
nsxt_network_profile
pks_cluster_name prod-1---be6cc6cb-b4a3-4bab-8d6f-e6d1499485bd
plan_name small
uuid 7a1283da-d8b4-418c-a5ea-720810195d72
worker_haproxy_ip_addresses
Note that 10.40.14.37 is the IP address of the Kubernetes master node. If I navigate to the NSX-T Manager web UI, I can verify that a virtual server was automatically created within an L4 load balancer to allow external access to the Kubernetes cluster via kubectl.
Now, the tenant can use the cse extension to pull down the Kubernetes config file from the cluster and store it in the default config file location on my local workstation (~/.kube/config):
Note: The config file will use prod-1.corp.local as the Kubernetes master server name so I have added a DNS entry that maps prod-1.corp.local to the IP of my NSX-T virtual server that fronts the Kubernetes master(s).
$ vcd cse cluster config prod-1 > ~/.kube/config
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
82d3022e-9fbc-4a31-9be2-fecc80e2ab27 Ready <none> 2d17h v1.13.5
d35f7324-0f09-440e-81b0-af9ad26481a6 Ready <none> 2d17h v1.13.5
Now the pks-k8-admin user has full admin access to their Enterprise PKS Kubernetes cluster and can instantly begin deploying their workloads to the newly created cluster!!
Conclusion
This wraps up my 3 part series on installing and configuring the Container Service Extension to support both CSE Standard and CSE Enterprise cluster creation. Feel free to reach out to me in the comment section or on Twitter if you have any additional questions or comments. Thanks for the read!