Container Service Extension 2.5 Installation: Part 3

In Parts 1 and 2 of my series on installing and configuring the Container Service Extension for VMware Cloud Director, I focused on setting the CSE server up to support CSE Standard Kubernetes cluster creation.

CSE Standard clusters are comprised of deployed vApps that utilize NSX-V networking resources, utilizing Weave as the Container Network Interface for the Kubernetes clusters. In Part 3 of my series, I wanted to take some time to look at configuring the CSE Server to support the creation of CSE Enterprise Kubernetes clusters. CSE Enterprise clusters are comprised of VMware Enterprise PKS Kubernetes clusters deployed on top of NSX-T networking resources, utilizing the NSX Container Plugin as a CNI. CSE Enterprise brings enterprise grade features and functionality to CSE that include, but are not limited to:

  • HA, multi-master Kubernetes clusters
  • Dynamic persistent storage provisioning with the vSphere Cloud Provider integration
  • Automated Day 1 and Day 2 Kubernetes cluster management via Bosh Director
  • Microsegmentation capability for Kubernentes resources via integration with NSX-T
  • Automated creation of Kubernetes service type LoadBalancer and ingress resrouces via NSX-T L4/L7 load balancers
  • Support for Harbor, an open source cloud native registry

How Does CSE Enterprise Work?

Before we get started on the configuration steps, I want to take some time to explore how the CSE Server communicates with both Enterprise PKS and NSX-T to automate the creation of Kubernetes clusters, all triggered by a request from an end user via the vcd-cli. When a user issues a vcd cse create cluster command, they are either utilizing an OrgVDC that has been enabled for CSE Standard, CSE Enterprise, or no Kubernetes provider at all. If their OvDC has been enabled to support CSE Enterprise cluster creation (and they have the correct permission to create clusters), the cse extension of the vcd-cli communicates directly with the PKS and NSX-T APIs to initiate the request to create an Enterprise PKS Kubernetes cluster as well as creating all of the NSX-T resources (T1 routers, Load Balancers, logical switches, etc.) to support the cluster. The CSE server also communicates with the NSX-T API to create DFW rules that isolate provisioned clusters from each other to support network resource isolation.

As opposed to CSE Standard, which creates a vApp from a vApp template and then instantiates the vApp as a Kubernetes cluster, the PKS control plane receives an API call from the CSE server and utilizes the plan assigned to the OrgVDC to create the cluster. There is not vApp or vApp template required to provision CSE Enterprise clusters.

CSE PKS Config File

Before we get started, I want to note that I have already deployed both VMware Enterprise PKS and VMware NSX-T Datacenter in my envrironment. CSE will need to hook into an existing deployment of PKS and NSX-T so these components must be configured before configuring CSE Enterprise.

As mentioned in Part 1 of the series, CSE uses a yaml config file that contains information about the vSphere/VCD environment(s) that CSE will need to interact with to deploy CSE Standard Kubernetes clusters. Similarly, we’ll need to utilize a second yaml file that will contain all of the information that CSE will need to communicate with the PKS/NSX-T APIs to orchestrate the provisioning of Enterprise PKS Kubernetes clusters. You can view a sample PKS config file on the CSE docs site but I will walk through through the required components below.

The first step is to create a pks-config.yaml file that I’ll populate with the values below. I’m going to create the file in the same directory as my existing config.yaml file:

$ vi ~/pks-config.yaml

Now I can populate the file with the required information, detailed below:

pks_api_servers section

This section of the config file will contain information that CSE will use to communicate with the PKS API server:

pks_api_servers:
- clusters:
  - RegionA01-COMP01          <--- Cluster name as it appears in vCenter
  cpi: 6bd3f75a31e5c3f2d65e   <--- Bosh CPI value, see note below
  datacenter: RegionA01       <--- Datacenter name as it appears in vCenter
  host: pks.corp.local        <--- FQDN of the PKS API server
  name: PKS-1                 <--- Name used to identity this particular PKS API server, user defined in this file
  port: '9021'                <--- Port used to for PKS API communication, default = 9021
  uaac_port: '8443'           <--- Port used to authenticate against the PKS UAAC service, default = 8443
  vc: vcsa-01                 <--- vCenter name as it appears in VCD
  verify: false               <--- Set to "true" to verify SSL certificates

Note: The cpi value above can be obtained by utilizing the Bosh CLI to run the following command on the Opsman VM in the PKS environment:

$ bosh cpi-config

Using environment '172.31.0.2' as client 'ops_manager'

cpis:
- migrated_from:
  - name: ""
  name: 6bd3f75a31e5c3f2d65e    <--- cpi value 
---output omitted---

pks_accounts section

This section defines the PKS UAAC credentials that CSE will use to authenticate against the PKS API when creating clusters:

pks_accounts:
- name: PKS1-admin                    
  pks_api_server: PKS-1
  secret: fhB-Z5hHcsXl_UnC86dkuYlTzQPoE3Yz
  username: admin
  • name is a user defined value that identifies the name of this credentials set. This is used in case there are multiple PKS API servers, which require their own set of credentials
  • pks_api_server is the user defined variable for the PKS API server defined by the name value in the pks_api_servers section (PKS-1 in my example)
  • username is the UAAC username that CSE will use to authenticate against the PKS API server (using the admin user in my example)
  • secret is the secret tied to the user defined in the username value. If using the admin user, you can obtain the secret by navigating to the Opsman web UI, selecting the PKS tile, selecting the credentials tab, and then selecting Link to Credential next to the Pks Uaa Management Admin Client entry. See screenshot below:

pvdcs section

This section defines the PvDC(s) in VCD that support the Org and OrgVDCs that are enabled to support PKS cluster creation via CSE:

pvdcs:
- cluster: RegionA01-COMP01  <--- Cluster name as it appears in vCenter
  name: prod-pvdc            <--- PvDC name as it appears in VCD
  pks_api_server: PKS-1      <--- user defined variable for the PKS API server defined by the `name` value in the `pks_api_servers` section

nsxt_servers section

This section defines information that CSE will need to communicate with the NSX-T API to create the NSX-T resources to back the Kubernetes clusters as well as create the DFW to isolate the clusters provisioned via CSE:

nsxt_servers:
- distributed_firewall_section_anchor_id: 9d6d2a5c-c32d-419c-ada8-e5208475ca88
  host: nsxmgr-01a.corp.local
  name: nsxt-server-1
  nodes_ip_block_ids:
  - eac47bea-5304-4a7b-8c10-9b16e62f1cda
  password: MySuperSecretPassword!
  pks_api_server: PKS-1
  pods_ip_block_ids:
  - 27d2d1c3-969a-46a5-84b9-db503ce2edd5
  username: admin
  verify: false
  • distributed_firewall_section_anchor_id is the UUID of the “Default Layer3 Section” DFW rule created by PKS during the installation of Enterprise PKS, see screenshot below:

  • host is the FDQN of the NSX-T Management server
  • name is a user defined name for this particular NSX-T instance
  • nodes_ip_block_ids is the UUID of the IP block created in NSX-T to be used by PKS for cluster nodes, see screenshot below:

  • password is the password for the NSX-T user defined in the config file
  • pks_api_server is the user defined variable for the PKS API server defined by the name value in the pks_api_servers section (PKS-1 in my example)
  • pods_ip_block_ids is the UUID of the IP block created in NSX-T to be used by PKS to assign IP addresses to the pods running in a cluster.

  • username is the NSX-T username CSE will use to authenticate against the API
  • verify can be set to “true” to verify SSL certificates

There we have it! This is the bare minimun information required for CSE to deploy Enterprise PKS clusters via vcd-cli. Do note, in my example deployment above, I have a single PKS deployment, cluster, PvDC, and NSX-T instance that CSE is communicate with. You can also add additional of each resource as required for your deployment.

Enabling CSE Enterprise on the CSE Server

Now that I’ve built my PKS config file for CSE, I’m ready to enable CSE Enterprise for my tenants’ org(s). I’ve already deployed the CSE server only utilizing the CSE Standard methodology so the first thing I’ll need to do is update the config.yaml file to point to the new pks-config.yaml file:

$ vi ~/config.yaml

---output omitted---

pks_config: pks-config.yaml

---output omitted---

After updating the config.yaml file with the filename of the PKS config file, I’ll stop the CSE service on the CSE server and re-install CSE:

$ sudo systemctl stop cse

$ cse install -c config.yaml --skip-template-creation

Required Python version: >= 3.7.3
Installed Python version: 3.7.3 (default, Nov 13 2019, 16:41:06)
[GCC 7.4.0]
Validating config file 'config.yaml'
Connected to AMQP server (vcd.corp.local:5672)
InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised.
Connected to vCloud Director (vcd.corp.local:443)
Connected to vCenter Server 'vcsa-01' as 'administrator@corp.local' (vcsa-01a.corp.local:443)
Config file 'config.yaml' is valid
Validating PKS config file 'pks-config.yaml'
Connected to PKS server (PKS-1 : pks.corp.local)
Connected to NSX-T server (nsxmgr-01a.corp.local)
PKS Config file 'pks-config.yaml' is valid
Installing CSE on vCloud Director using config file 'config.yaml'
Connected to vCD as system administrator: vcd.corp.local:443
Checking for AMQP exchange 'cse-ext'
AMQP exchange 'cse-ext' is ready
Registered cse as an API extension in vCD
Registering Right: CSE NATIVE DEPLOY RIGHT in vCD
Registering Right: PKS DEPLOY RIGHT in vCD
Creating catalog 'cse'
Created catalog 'cse'
Skipping creation of templates.
Configuring NSX-T server (nsxt-server-1) for CSE. Please check install logs for details.

As you can see from the output of the cse install command, CSE was able to communicate with the PKS and NSX-T API. The CSE server also added a new rights bundle (PKS DEPLOY RIGHT) that I will use to grant users the ability to provision CSE Enterprise Kubernetes clusters.

After successfully installing CSE with the PKS config, I’ll restart the CSE service:

$ sudo systemctl start cse

and verify the service is running as expected (look for the MessageConsumer threads):

$ sudo systemctl status cse
[sudo] password for cse:
● cse.service
   Loaded: loaded (/etc/systemd/system/cse.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2019-11-18 12:08:18 EST; 1 day 3h ago
 Main PID: 911 (sh)
    Tasks: 7
   Memory: 3.6M
   CGroup: /system.slice/cse.service
           ├─911 /bin/sh /home/cse/cse.sh
           └─914 /home/cse/cse-env/bin/python3.7 /home/cse/cse-env/bin/cse run -c /home/cse/config.yaml

Nov 18 12:08:22 cse sh[911]: CSE installation is valid
Nov 18 12:08:23 cse sh[911]: Started thread 'MessageConsumer-0 (139897843828480)'
Nov 18 12:08:23 cse sh[911]: Started thread 'MessageConsumer-1 (139897746745088)'
Nov 18 12:08:24 cse sh[911]: Started thread 'MessageConsumer-2 (139897755137792)'
Nov 18 12:08:24 cse sh[911]: Started thread 'MessageConsumer-3 (139897835435776)'
Nov 18 12:08:24 cse sh[911]: Started thread 'MessageConsumer-4 (139897738352384)'
Nov 18 12:08:24 cse sh[911]: Container Service Extension for vCloud Director
Nov 18 12:08:24 cse sh[911]: Server running using config file: /home/cse/config.yaml
Nov 18 12:08:24 cse sh[911]: Log files: cse-logs/cse-server-info.log, cse-logs/cse-server-debug.log
Nov 18 12:08:24 cse sh[911]: waiting for requests (ctrl+c to close)

Voila!! I’ve successfully configured CSE to support Enterprise PKS cluster creation!! Now, let’s have a look at enabling a tenant to deploy CSE Enterprise clusters via the vcd-cli.

Onboarding Tenants and Provisioning CSE Enterprise Kubernetes Clusters

The first thing I’ll need to do is login to the VCD deployment via the vcd-cli as the System Admin user:

$ vcd login vcd.corp.local system admin -iw

Next, I can use the cse extension to view the existing OvDCs and what Kubernetes provider is enabled for each OvDC:

$ vcd cse ovdc list

name       org       k8s provider
---------  --------  --------------
base-ovdc  base-org  native
acme-ovdc  AcmeCorp  none

From the results above, note that the base-ovdc is enabled for the Kubernetes provider of native, which means users in that org with the correct permissions can create CSE Standard clusters. The acme-ovdc does not have any Kubernetes provider assigned yet, but I want to enable it to support CSE Enterprise cluster creation.

First, I’ll need to instruct the vcd-cli to “use” the AcmeCorp org:

$ vcd org use AcmeCorp
now using org: 'AcmeCorp', vdc: 'acme-ovdc', vApp: ''.

Then, I’ll add the new PKS rights bundle to the AcmeCorp org:

$ vcd right add "{cse}:PKS DEPLOY RIGHT" -o AcmeCorp
Rights added to the Org 'AcmeCorp'

Before moving on to to the next step, I created a new role in the VCD tenant portal by the name of “orgadmin-k8” which mimics the “Org Admin” permissions. I also created a new user named “pks-k8-admin” and assigned that user the “orgadmin-k8” role.

After creating the new role and user, I need to use the vcd-cli to add the PKS DEPLOY RIGHT rights bundle to the custom role:

$ vcd role add-right "orgadmin-k8" "{cse}:PKS DEPLOY RIGHT"
Rights added successfully to the role 'orgadmin-k8'

The last thing I need to do as the System admin is enable the OvDC to support CSE Enterprise cluster with the following command:

$ vcd cse ovdc enable "acme-ovdc" -o AcmeCorp -k ent-pks -p "xsmall" -d "corp.local"

metadataUpdate: Updating metadata for Virtual Datacenter acme-ovdc(07972d80-faf1-48c7-8f7e-cea92ce7cc6e)
task: 9fced925-4ec8-47c3-a45e-7903b14b0c8d, Updated metadata for Virtual Datacenter acme-ovdc(07972d80-faf1-48c7-8f7e-cea92ce7cc6e), result: success

where -k is the Kubernetes provider (PKS in this case), -p is the PKS plan we want clusters to use when provisioned from CSE for this OvDC, and -d is the domain name that PKS will use for the hostname of each cluster.

Note, CSE uses compute profiles to create Availability Zones for each OvDC that is enabled. When vce cse ovdc enable command above is issued, CSE talks to the PKS API to create a compute profile that defines the Availability Zones for all clusters created in this OvDC as the Resource Pool that is assigned to that OvDC. This ensures that users that provision clusters to this OvDC will have their compute isolated (via the Resource Pool) from other clusters provisioned in other OvDCs in the VCD environment.

Verify that I now see a Kubernetes provider for our acme-ovdc after enabling it:

$ vcd cse ovdc list

name       org       k8s provider
---------  --------  --------------
base-ovdc  base-org  native
acme-ovdc  AcmeCorp  ent-pks

There we have it! Now I can tell the pks-k8-admin user that they are ready to provision some CSE Enterprise clusters!!

Provisioning CSE Enterprise Clusters

Now that I, as the system admin, have done the pre-work to enable the Org to support CSE Enterprise clusters, I’m ready to turn the tenants loose and allow them to provision CSE Enterprise Kubernetes clusters.

First, the orgadmin-k8 user will log in to VCD via the vcd-cli:

$ vcd login vcd.corp.local AcmeCorp pks-k8-admin -iw

All they have to do now is run the vcd cse cluster create command and CSE will handle the rest:

$ vcd cse cluster create prod-1 --nodes 2

property                     value
---------------------------  -----------------
kubernetes_master_host       prod-1.corp.local
kubernetes_master_ips        In Progress
kubernetes_master_port       8443
kubernetes_worker_instances  2
last_action                  CREATE
last_action_description      Creating cluster
last_action_state            in progress
name                         prod-1
worker_haproxy_ip_addresses

where prod-1 is the name of our cluster and --nodes is the number of worker nodes assigned to the cluster. As you can see, the FQDN of the master host will be “cluster-name”.”domain” where “domain” was defined when we enabled the OvDC.

Once the cluster has finished provisioning, we can use the cse extension to gather information about the cluster:

$ vcd cse cluster info prod-1

property                     value
---------------------------  -------------------------------------------------
k8s_provider                 ent-pks
kubernetes_master_host       prod-1.corp.local
kubernetes_master_ips        10.40.14.37
kubernetes_master_port       8443
kubernetes_worker_instances  2
last_action                  CREATE
last_action_description      Instance provisioning completed
last_action_state            succeeded
name                         prod-1
network_profile_name
nsxt_network_profile
pks_cluster_name             prod-1---be6cc6cb-b4a3-4bab-8d6f-e6d1499485bd
plan_name                    small
uuid                         7a1283da-d8b4-418c-a5ea-720810195d72
worker_haproxy_ip_addresses

Note that 10.40.14.37 is the IP address of the Kubernetes master node. If I navigate to the NSX-T Manager web UI, I can verify that a virtual server was automatically created within an L4 load balancer to allow external access to the Kubernetes cluster via kubectl.

Now, the tenant can use the cse extension to pull down the Kubernetes config file from the cluster and store it in the default config file location on my local workstation (~/.kube/config):

Note: The config file will use prod-1.corp.local as the Kubernetes master server name so I have added a DNS entry that maps prod-1.corp.local to the IP of my NSX-T virtual server that fronts the Kubernetes master(s).

$ vcd cse cluster config prod-1 > ~/.kube/config

$ kubectl get nodes

NAME                                   STATUS   ROLES    AGE     VERSION
82d3022e-9fbc-4a31-9be2-fecc80e2ab27   Ready    <none>   2d17h   v1.13.5
d35f7324-0f09-440e-81b0-af9ad26481a6   Ready    <none>   2d17h   v1.13.5

Now the pks-k8-admin user has full admin access to their Enterprise PKS Kubernetes cluster and can instantly begin deploying their workloads to the newly created cluster!!

Conclusion

This wraps up my 3 part series on installing and configuring the Container Service Extension to support both CSE Standard and CSE Enterprise cluster creation. Feel free to reach out to me in the comment section or on Twitter if you have any additional questions or comments. Thanks for the read!

Leave a Reply

Your email address will not be published. Required fields are marked *