WireGuard article series (eight): An introduction to the K8S CNI Kilo based on WireGuard

Synopsis of the series:

1. WireGuard Part 1: What is a VPN?
2. WireGuard Part 2: Introduction to WireGuard - Fast, Modern, Secure VPN Tunnels
3. WireGuard Part 3: WireGuard Installation
4. WireGuard article series (4): WireGuard is quick to get started
5. WireGuard Part 5: Introduction to Netmaker - A Platform for Creating and Managing WireGuard Networks
6. WireGuard article series (6): Netmaker installation
7. WireGuard Part 7: Creating Full Mesh Networks with WireGuard and Netmaker

Next, we’ll look at the integration of WireGuard and Kubernetes, a K8S network plugin based on WireGuard, Kilo.

Kilo is built on WireGuardcloudy overlay networking, designed for Kubernetes.

Kilo overview

Kilo is provided by providing an encrypted oneThree-tier networkto connect nodes in the clusterNetworks can span data centers and public clouds。 Pod networks created by Kilo are always fully connected, even if the nodes are in different networks or behind NAT. By allowing pools of nodes in different locations to communicate securely, Kilo enables the operation of multi-cloud clusters. Kilo’s design allows clients to connect to the cluster over VPN in order to securely access services running on the cluster. In addition to creating multi-cloud clusters (i.e., one cluster spans multiple public clouds), Kilo also supports the creation of multi-cluster services, that is, services that span different Kubernetes clusters.

• Kilo’s Git repository
• Kilo’s Gitee mirror repository

How Kilo works

Kilo uses WireGuard Create a mesh between different nodes in the cluster.

Kilo agent(kg) runs on each node in the cluster to set up the public and private keys for the VPN and the rules required to route packets between locations.

Kilo can be used both as a complete, stand-alone K8S network plug-in or as a complement to a cluster networking solution currently installed on the cluster. This means that if the cluster uses Flannel for networking, then the Kilo can be installed on top of the cluster to enable pools of nodes in different locations (such as different clouds) to connect; Kilo is responsible for the network between locations, and Flannel is responsible for the network within each location.

Kilo Annotions

After the kilo is installed in the K8S cluster, the corresponding configuration is implemented through the annotations of the K8S node.

The following annotations can be added to any Kubernetes Node object to configure the Kilo network.

Name	type	examples
kilo.squat.ai/force-endpoint	host:port	55.55.55.55:51820, example.com:1337
kilo.squat.ai/force-internal-ip	CIDR	55.55.55.55/32, "-",""
kilo.squat.ai/leader	string	"", true
kilo.squat.ai/location	string	gcp-east, lab
kilo.squat.ai/persistent-keepalive	uint	10
kilo.squat.ai/allowed-location-ips	CIDR	66.66.66.66/32

force-endpoint

In order to create links between locations, Kilo requires at least one node in each location to have an endpoint (that is, public network address + port), i.ehost:portCombined, it can be routed from other locations. If the location is in a different cloud provider or in a different private network, then the host portion of the endpoint should be a publicly accessible IP address, or a DNS name that resolves to a public IP so that other locations can route packets to it. The Kilo agent running on each node will use heuristics to automatically detect the node’s external IP address and configure its endpoints correctly; However, in some cases, you may need to explicitly configure endpoints to use, such as:

• There is no automatic public IP on the NIC: On some cloud providers, nodes are usually assigned a public IP address, but only a private network address is automatically configured for NICs; In this case, you should specify the assigned public IP address; (This is the case for domestic public clouds, so use Kilo on domestic public clouds must be configured.) force-endpoint）
• Multiple public IP addresses: If a node has multiple public IP addresses, but one is the preferred IP address, specify the preferred IP address;
• IPv6: If a node has a public IPv4 address and an IPv6 address, and the Kilo network should run on IPv6, then an IPv6 address should be specified;
• Dynamic IP address: If a node has a dynamically assigned public IP address, such as an IP leased from a network provider, then a dynamic DNS name can be given.
• Overlay ports: If a node should listen on a specific port that is different from the default WireGuard port of the mesh (e.g. 52181), then this comment can be used to override the port; This can be useful, for example, to ensure that two nodes running behind the same port forwarding NAT gateway can each be assigned a different port.

force-internal-ip

Kilo uses the internal IP address of the node to send packets to nodes in the same logical location. The Kilo proxy running on each node will use heuristics to automatically detect the node’s private IP address; However, in some cases, you may need to explicitly configure an IP address, such as:

• Multiple private IP addresses: If a node has multiple private IP addresses, but one is the preferred IP address, you need to specify the preferred IP address (typically, one private IP is used for business and one private IP is used for storage; Or in the local virtual machine environment, a private IP is Host and a private IP is NAT);
• IPv6: If a node has both private IPv4 and IPv6 addresses, and the Kilo network runs on IPv6, you should specify an IPv6 address.
• Use “-” or “” to disable private IP: The node has a private IP address and a public IP address, but the private IP address is ignored.

leader

By default, Kilo creates a network mesh at the data center granularity. This means selecting a leader node from each location as the Edge Server and acting as a gateway to other locations; The network topology will be Full Mesh between leaders. Kilo automatically selects a leader for each location in a stable and deterministic manner to avoid clutter in the network configuration, while prioritizing nodes that are known to have public IP addresses. In some cases, it may be desirable to manually select a leader in a location, such as:

• firewall: Kilo requires an open UDP port, which defaults to 51820, to communicate between locations; If only one node is configured to open the port, that node should be given the leader comment;
• bandwidth: If some nodes in the cluster have higher bandwidth or lower latency Internet connections, those nodes should be given the leader comment.

Note:

Multiple nodes within a location can be given leader comments; In this case, Kilo selects a leader from a set of comment nodes.

location

Kilo allows nodes in different logical or physical locations to route packets to each other. In order to know what connections to create, Kilo needs to know which nodes are at each location. Kilo will try to start from the topology topology.kubernetes.io/region The location of each node is inferred from the label. If the node should not have a label, such as running a bare metal cluster or on an unsupported cloud provider (or a K8S cluster built by yourself), you should specify a location comment.

Note:

All nodes that do not have a defined location are considered to be in the default position""。

persistent-keepalive

In some deployments, the cluster nodes may be located NAT or firewallBehind, such as an edge node located behind a normal router. In this case, the NAT translated node can send packets to nodes outside the NAT translated network, but the external node can send packets to the NAT translated network only if the NAT mapping is valid. In order for a post-NAT node to receive packets from nodes outside the post-NAT network, it must maintain a NAT mapping relationship by periodically sending packets to these nodes (that is, sending keepalive packets). The frequency of sending these keepalive packets can be controlled by setting persistent-keepalive annotations on the nodes after NAT. The annotated node will use the specified value as the persistent-keepalive interval for all its peers, see NAT and firewall traversal in the WireGuard documentation。

allowed-location-ips

You can add allowed-location-ips to a location by annotating any node in the location. Add allowed location IPs in one location so that they can also be routed from other locations.

In a Kilo deployment example with two locations A and B, printers at location A can be accessed from nodes and pods at location B. In addition, Kilo Peers can use a printer from position A.

Additional mode

Administrators of existing clusters who do not wish to change existing network solutions can run Kilo in add-on mode.

In this mode, Kilo adds advanced features such as VPN and multi-cluster services to the cluster, while delegating CNI management and local networks to the cluster’s current network provider. Kilo currently supports running on flannel.

For example, to run Kilo on a Typhoon cluster running Flanel:

kubectl apply -f https://raw.githubusercontent.com/squat/kilo/main/manifests/crds.yaml
kubectl apply -f https://raw.githubusercontent.com/squat/kilo/main/manifests/kilo-typhoon-flannel.yaml

VPN

Kilo also allows peers outside the Kubernetes cluster to connect to the VPN, allows cluster applications to securely access external services, and allows developers and helpdesk staff to securely debug cluster resources.

To declare a peer, first define a Kilo peer resource:

cat <<'EOF' | kubectl apply -f -
apiVersion: kilo.squat.ai/v1alpha1
kind: Peer
metadata:
  name: squat
spec:
  allowedIPs:
  - 10.5.0.1/32
  publicKey: GY5aT1N9dTR/nJnT1N2f4ClZWVj0jOAld0r8ysWLyjg=
  persistentKeepalive: 10
EOF

This configuration can be applied to the local WireGuard interface, for examplewg0, let it be inkgctlTools to access the cluster:

kgctl showconf peer squat > peer.ini
sudo wg setconf wg0 peer.ini

Kilo Network Analysis

Can be used kgctlThe command-line tool analyzes the topology and configuration of the Kilo network.

For example, you can usegraphcommand to generate a network diagram in Graphviz format:

kgctl graph | circo -Tsvg > cluster.svg

Kilo network topology

Kilo allows customization of the topology of encrypted networks. Cluster administrators can specify whether the encrypted network should be a full mesh between each node or a grid between different node pools that communicate directly with each other. This allows encrypted networks to serve several purposes, such as:

• On cloud providers with insecure private networks, a complete mesh can be created between nodes to protect all cluster traffic;
• Nodes running in different cloud providers can join a single cluster by creating a link between the two clouds(This is also the main content of the actual battle later);
• More generally, unsafe links can be encrypted, while secure links can remain fast and unencapsulated.

Logical Groups Logical groups

By default, Kilo creates a grid between different logical locations in the cluster, such as data centers, cloud providers, and so on. Kilo leverages the Kubernetes topology topology.kubernetes.io/region The label infers the location of the node. In addition, Kilo supports setting command-line flags--topology-label=<label>to use custom topology labels. If this tag is not set, it can be used kilo.squat.ai/location Marking. For example, to connect nodes in Google Cloud and AWS to a separate cluster, administrators can comment out all nodes with GCP in the name using the following code snippet:

for node in $(kubectl get nodes | grep -i gcp | awk '{print $1}'); do 
    kubectl annotate node $node kilo.squat.ai/location="gcp"; 
done

In this case, Kilo would do this:

• Bring all of themGCP Annotion nodes are grouped into a logical location;
• Grouping all nodes without labels will be grouped to the default position; and
• Pick a leader at each location and make connections between them.

usekgctlAnalyzing the cluster produces the following results:

kgctl graph | circo -Tsvg > cluster.svg

Full Mesh

Creating a full mesh is a logical simplification of a logical grid where each node is in its own group. Kilo provides a shortcut to this topology in the form of command-line flags:--mesh-granularity=full. When full mesh granularity is specified, Kilo configures the network to use WireGuard encryption for traffic between all nodes.

usekgctlAnalyzing the cluster produces the following results:

kgctl graph | circo -Tsvg > cluster.svg

mix

kilo.squat.ai/location Callouts can mix some Full Mesh nodes with some nodes grouped by logical location. For example, if a cluster contains a set of nodes in Google Cloud and a group of nodes without a secure private network, such as some bare metal nodes, then Google Cloud nodes can be placed in a logical group, while bare metal nodes can form a complete net. This can be achieved by running the following command:

for node in $(kubectl get nodes | grep -i gcp | awk '{print $1}'); do kubectl annotate node $node kilo.squat.ai/location="gcp"; done
for node in $(kubectl get nodes | tail -n +2 | grep -v gcp | awk '{print $1}'); do kubectl annotate node $node kilo.squat.ai/location="$node"; done

usekgctlAnalyzing the cluster produces the following results:

kgctl graph | circo -Tsvg > cluster.svg

If your cluster also has nodes in AWS, you can use the following code snippet:

for node in $(kubectl get nodes | grep -i aws | awk '{print $1}'); do kubectl annotate node $node kilo.squat.ai/location="aws"; done
for node in $(kubectl get nodes | grep -i gcp | awk '{print $1}'); do kubectl annotate node $node kilo.squat.ai/location="gcp"; done
for node in $(kubectl get nodes | tail -n +2 | grep -v aws | grep -v gcp | awk '{print $1}'); do kubectl annotate node $node kilo.squat.ai/location="$node"; done

This in turn generates a chart like this:

kgctl graph | circo -Tsvg > cluster.svg

summary

Kilo is built on WireGuardcloudy overlay networking, designed for Kubernetes. It makes it possible to set up a fast, modern, and secure K8S cluster on a multi-cloud. 🎉🎉🎉