Setting up a Kubernetes cluster with Talos Linux on OpenStack

Overview

In a separate tutorial we imported a Talos Linux image and provisioned it into pluscloudopen. In this tutorial we want to use it to set up a Kubernetes cluster inside our pluscloudopen project. Plusserver offers Plusserver Kubernetes Engine (PSKE), which is a Managed Kubernetes offering, but there might be scenarios, in which a Kubernetes cluster based on Talos Linux is a better fit for the requirements. Here we have adapted the platform specific installation documentation, which is part of the Talos documentation, to pluscloudopen specifics.

Loadbalancer

The control plane of a Kubernetes cluster (for production use) is spread over at least three nodes (in order to be able to survive the outage of one node, without the cluster to stop working). These cluster nodes provide the Kubernetes-API, too. In order to make the API highly available it is recommended to access the three control plane nodes via a loadbalancer.

A loadbalancer can be configured using the OpenStack-Client in pluscloudopen like this:

# Create the loadbalancer objekt with a public IP-address
openstack loadbalancer create --name talos-control-plane --vip-network-id ext01

# Configure the external port and protocol
openstack loadbalancer listener create --name talos-control-plane-listener --protocol TCP --protocol-port 6443 talos-control-plane

# Add a loadbalancer pool to the listener and configuring a loadbalancing algorithm 
openstack loadbalancer pool create --name talos-control-plane-pool --lb-algorithm ROUND_ROBIN --listener talos-control-plane-listener --protocol TCP

# Add a healthcheck to the pool 
openstack loadbalancer healthmonitor create --delay 5 --max-retries 4 --timeout 10 --type TCP talos-control-plane-pool

Instead of creating the pool members already, we create only their network ports. The advantage is, that mac- and ip-adresses will not change afterwards - even if we have to re-create the attached instances. Don’t forget to use the network name of your OpenStack project (openstack network list) here.

openstack port create --network <project-network-name> talos-control-plane-1 
openstack port create --network <project-network-name> talos-control-plane-2 
openstack port create --network <project-network-name> talos-control-plane-3 

In this tutorial, we will access our Talos-VMs from our bastion-host inside our openstack project. Thus we do not create Floating-IP addresses for these VMs.

Next wie add the just created ports as members to our loadbalancing pool. Pleas use the id of the subnet of your OpenStack project (openstack subnet list) here. And you should use the private ip-adresses of the just created control plane node ports (openstack port list -fvalue -c'Name' -c'Fixed IP Addresses' |grep -i talos), too:

openstack loadbalancer member create --subnet-id <project-subnetwork-id> --address <private ip-adress of the talos-control-plane-1 port> --protocol-port 6443 talos-control-plane-pool
openstack loadbalancer member create --subnet-id <project-subnetwork-id> --address <private ip-adress of the talos-control-plane-2 port> --protocol-port 6443 talos-control-plane-pool
openstack loadbalancer member create --subnet-id <project-subnetwork-id> --address <private ip-adress of the talos-control-plane-3 port> --protocol-port 6443 talos-control-plane-pool

Security Rules

Talos’ documentation includes some hints regarding the “network connectivity”. As long as the control plane nodes as well as the worker nodes are on the same layer-2 netowrk, we don’t need to configure security rules between the nodes. The default security rules, which are configured for any project in pluscloud open allow external access to VMs only via ssh (as long as the VM gets a floating-ip assigned). In order to allow additional access via port 6443 (for the Kubernetes-API) and port 50000 (for the Talos-API), you can either extand the “default” security rule or add a separate one an assign it to the Talos VMs. For this tutorial we create a separate security rule for Talos:

# Create the security rule 
openstack security group create talos-cluster

# Allow access to the Kubernetes-API from everywhere
openstack security group rule create talos-cluster --protocol TCP --dst-port 6443 --remote-ip 0.0.0.0/0

# Allow access to the Talos-API only from our project network 
openstack security group rule create talos-cluster --protocol TCP --dst-port 50000 --remote-ip 192.168.0.0/24 
openstack security group rule create talos-cluster --protocol TCP --dst-port 50001 --remote-ip 192.168.0.0/24 

Configure the Talos cluster

In order to configure Talos, we need to download “talosctl”:

curl -sL https://talos.dev/install | sh

This will install the go-binary “talosctl” in /usr/local/bin.

After that we can create the configuration files for the Talos cluster:

talosctl gen config talos-k8s-openstack https://$(openstack loadbalancer show talos-control-plane -f value -c vip_address):6443

This creates the files “controlplane.yaml”, “worker.yaml” and “talosconfig”. Both of the yaml-files will be needed as “user-data” in the next step, to create the cluster nodes.

Create the cluster nodes

Finally we can create the three control-plane and the two worker VMs. Don’t forget to set the path to the controlplane.yaml file correctly. Control plane VMs first:

for i in 1 2 3 ; do
  openstack server create talos-control-plane-$i --flavor SCS-2V-2-20 --nic port-id=talos-control-plane-$i --image "Talos Linux v.1.12.6 openstack" --security-group talos-cluster --user-data /where/is/my/controlplane.yaml;
done

Next the Worker VMS. As we did not create their network ports previously those will be created automtically, during VM creation. Thus the VMs will get different mac- and ip-addresses everytime we re-create them. Don’t forget to set the right network name for your OpenStack project (openstack network list):

for i in 1 2 ; do
  openstack server create talos-worker-$î --flavor SCS-2V-2-20 --network <project-network> --image "Talos Linux v.1.12.6 openstack" --security-group talos-cluster --user-data /where/is/my/worker.yaml;
done

Cluster Bootstrap

In order to build the Kubernetes cluster, the etcds on all control plane VMs need to find each other and build an etcd-cluster. In order to execute the corresponding command, we need to set the values for “nodes” and “endpoint” in the “talosconfig” file like this:

talosctl --talosconfig talosconfig config endpoint <private ip-address of one of the three control plane vMs>
talosctl --talosconfig talosconfig config nodes <private ip-address of the same control plane VMs as above>

Talos VM Dashboard

Even now it should be possible to view the read-only dashboard of the above Talos VM using

talosctl -n <node-ip> dashboard

You can leave with <strg>-C again.

After this, it should be possible to bootstrap the Kubernetes cluster like this:

talosctl -n <node-ip> bootstrap

You can monitor the progress via the dashboard (s. a.). As soon as the cluster has configured itself, we should be able to create the corresponging kubeconfig:

talosctl --talosconfig talosconfig kubeconfig .

With that, we can access the Kubernetes api. For that we need another binary: “kubectl”. The documentation shows how to get it:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

Using “kubectl” together with the just created “kubeconfig” we can view the nodes and pods in our new Kubernetes cluster:

kubectl get nodes -A
kubectl get pods -A

Hint

For a tighter integration of your new Kubernetes cluster with OpenStack see the Cloud Provider Openstack which should let you access loadbalancing and storage features from OpenStack via the Kubernetes-API.