Multi-Cluster with Cilium on Civo cloud

Connecting two Kubernetes clusters can seem counterintuitive at first, as Kubernetes was designed to scale up and scale out workloads. In a previous post, we discussed multi-cluster deployments and how and why they might benefit you.

In this post, we will focus on a practical implementation of a multi-cluster Kubernetes deployment using Cilium and Civo’s Kubernetes service; for this implementation guide, we will be using ClusterMesh, a feature of Cilium that allows users to connect multiple clusters together.

Foreword by Head of Community at Isovalent (Creators of Cilium)

When it comes to building a reliable, multi-cluster environment, Cilium provides a unique advantage. This guide will walk you through deploying a multi-cluster setup with Cilium on Civo, showcasing the simplicity and efficiency that eBPF-based networking brings to Kubernetes. Whether you’re scaling applications across multiple data centers or connecting services in diverse environments, Cilium’s approach reduces complexity and resource overhead, allowing you to focus on what really matters: your workloads. Dive in and see how to take Kubernetes connectivity to the next level with Cilium on Civo. — Bill Mulligan.

Cilium; Isn’t it a CNI?

If you have looked into container network interface (CNI) in any capacity, you have likely come across Cilium. Created by parent company, Isovalent — now part of Cisco, Cilium is a high performance CNI. While we will be focusing on the networking features of Cilium it also provides a host of observability and security features as well.

What makes Cilium special is eBPF and the manner in which it has been leveraged to create a great networking solution. If you are unfamiliar with it, eBPF is a Linux kernel feature that can run sandboxed programs in a privileged context, such as the operating system kernel.

What this means for us is that Cilium can skip using a sidecar proxy and leverage Linux primitives to handle service-to-service communications, in our case, cluster-to-cluster communication. This is huge because traditionally, deploying a sidecar alongside your service meant you incurred additional resource overhead and operational costs, especially in environments with a large number of services.

Sidecar vs. sidecarless

If you've been following the service mesh debate, you might remember that in September 2022, Istio released a new operating mode called Ambient Mesh. This was a significant development because it allowed Istio to run without a sidecar. The Istio team introduced the concept of "ztunnel" (zero-trust tunnel) to replace traditional sidecars. Ztunnel acts as a node-wide proxy, handling the same responsibilities as sidecars but at the node level, reducing the per-pod overhead.

The Istio team highlighted a few problems with the traditional sidecar approach:

• Invasiveness: Sidecars require "injection" into applications by modifying their Kubernetes pod spec and redirecting traffic within the pod. This means that installing or upgrading sidecars required restarting the application pod, which can disrupt workloads and complicate operations.
• Traffic breaking: The traffic capture and HTTP processing typically performed by Istio's sidecars is computationally expensive and can potentially break applications with non-conformant HTTP implementations.

Sidecarless architectures, on the other hand, leverage tunneling techniques to address these issues. By moving the proxy functionality out of individual pods and into the node's networking layer, sidecarless approaches can mitigate the invasiveness and potential disruptions caused by sidecars.

Cilium ClusterMesh

In the context of Cilium and its ClusterMesh feature, eBPF is leveraged to create a high-performance, sidecarless network between Kubernetes clusters. This approach allows for multi-cluster communication without the need for per-pod proxies, reducing resource overhead and simplifying operations.

Beyond multi-cluster, ClusterMesh also enables sharing of services such as secrets management, logging or DNS between all clusters, this greatly cuts down on the operational overhead of maintaining services across clusters.

Demo environment setup

To follow along with the demo walkthrough, ensure you have the following prerequisites:

• Some experience with Kubernetes
• The kubectl command-line tool installed on your machine.
• A Civo account. You can create a free account if you don’t one. Upon sign up, Civo will give you $250 credit to use on it services.

What would our final deployment be?

Before diving into the demo, below is a quick overview of how our final infrastructure will look. We provision two clusters within the same virtual network and expose the clustermesh-apiserver using a loadbalancer.

Installing Civo CLI

You can use Civo cloud via its console interface just like other cloud service providers (CSPs), but for this demo, run the following command to install the Civo CLI:

curl -sL https://civo.com/get | sh

The above command will detect your operating system and install the latest version of the CLI.

Creating a virtual private network

We will begin by creating a private network to house both clusters. This is important because all nodes in each cluster require network connectivity:

civo network create --create-default-firewall deathstar

Using the --create-default-firewall flag we create a default firewall rules for the network deathstar. The output of the command would be similar to the image below.

Provisioning the Kubernetes clusters

Using the Civo CLI, we can create the two clusters using the following commands:

civo k8s create --create-firewall --network=deathstar --nodes 2 -m --save --switch --wait --cni-plugin=cilium deathstar1

The command above creates a two node cluster called deathstar1, save the kubeconfig and install Cilium as the default CNI.

Repeat for cluster two deathstar2 with the command below, and you will get an output similar to the image above.

civo k8s create --create-firewall --network=deathstar --nodes 2 -m --save --switch --wait --cni-plugin=cilium deathstar2

Setting up Cilium on the Kubernetes clusters

To setup Cilium of the two Kubernetes clusters, we need to first instal the Cilium CLI. The Cilium CLI provides useful commands enabling and managing Cilium features, the steps vary slightly depending on your operating system, however for Linux users can run the following commands.

export CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
export CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum} 

The above command downloads and installs the latest version of the Cilium CLI for your operating system.

For MacOS, using the homebrew package manager, you can install the Cilium

brew install cilium-cli

Install Cilium on both clusters
Before install Cilium on both clusters, lets first set up an alias for each cluster. We can do this by using the export command below. First get the name of the clusters.

kubectl config get-contexts

Once you have identified the names of both clusters, save the names of the clusters as environment variables with the following commands:

export CLUSTER1=deathstar1
export CLUSTER2=deathstar2

After that, install Cilium on each cluster using the commands below.

cilium upgrade --version v1.16.0 --set cluster.name=$CLUSTER1 --set cluster.id=1 --context $CLUSTER1


cilium upgrade --version v1.16.0 --set cluster.name=$CLUSTER2 --set cluster.id=2 --context $CLUSTER2

The output of the command would be similar to the image below.

Verify that Cilium is installed correctly with the following command:

cilium status --context $CLUSTER1

The output is similar to the image below.

Create a shared Certificate Authority (CA)
This will help enable mTLS across both clusters:

kubectl --context=$CLUSTER1 get secret -n kube-system cilium-ca -o yaml | \
kubectl --context $CLUSTER2 create -f -

Enable clustermesh
Remember the diagram from earlier? Here is where we actually enable clustermesh and expose it by specifying the service type as LoadBalancer.

cilium clustermesh enable --context $CLUSTER1 --service-type=LoadBalancer


cilium clustermesh enable --context $CLUSTER2 --service-type=LoadBalancer

Verify ClusterMesh was enabled correctly

cilium clustermesh status --context $CLUSTER1 --wait

The output will be similar to the imagine.

Connect both clusters
At long last, we can connect both clusters:

cilium clustermesh connect --context $CLUSTER1 --destination-context $CLUSTER2

The output is similar to the image below.

With both clusters connected, let's create a deployment to test the setup by deploying a shared service.

Deploy a shared service

The Cilium documetation provides a sample application “Rebel Base” that will be sufficient for this demo. On cluster 1, deploy the following application:

kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.16.2/examples/kubernetes/clustermesh/global-service-example/cluster1.yaml --context=$CLUSTER1

On cluster two:

kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.16.2/examples/kubernetes/clustermesh/global-service-example/cluster2.yaml --context=$CLUSTER2

And now for the moment of truth. From either cluster, run the following command:

kubectl exec -ti deployment/x-wing -- curl rebel-base

After a couple of tries, your output should be similar to:

What enables cilium to distribute traffic across clusters is the use of the service.cilium.io/shared="false" annotation. When set to true, cilium will treat the service as global and distribute traffic across both clusters.

Debugging and troubleshooting

If you are having trouble getting your set up to work, here are some things to try:

Check your networks!!
I’m certain the Cilium team is sick of asking if nodes can reach other and for good reason. If you are having trouble seeing replies from both clusters, double check your network (or VPC ) set up to ensure that nodes from either cluster can reach each other.

Pro tip: running the following will get you a shell into a node and you can continue from there:

kubectl debug node/yournodename -it --image=ubuntu

Also be sure you are using the correct set up for your cloud of choice. In this tutorial we used Civo but guides for AWS and Azure are available in the docs.

When in doubt run connectivity tests using the command:

cilium connectivity test --context $CLUSTER1 --multi-cluster $CLUSTER2

Will run a series of connectivity tests to asses your cluster connections.

And if all else fails? Ask, join the Cilium slack or ping me on twitter.

Clean up

Upon completing this tutorial you might want to delete some of the resources we provisioned.

Disconnect both clusters run:

cilium clustermesh disable

Delete the clusters:

civo k8s rm deathstar1
civo k8s rm deathstar2

Wrapping up

In this post we discussed Cilium, overviewed the service mesh landscape, and used the Cilium ClusterMesh to connect two Civo Kubernetes clusters.

Running multi-cluster setup can seem like a lot overhead below a certain scale, however as you application grows more important and much more users start to depend on it, the slightest of outages can spell big trouble.

To learn more about Cilium and its ClusterMesh feature, check out the following resources:

• https://cilium.io/use-cases/cluster-mesh/
• https://isovalent.com/labs/cilium-cluster-mesh/
• https://cilium.io/blog/2019/03/12/clustermesh/