🛠️ Practical Session! Build a unified K8S cluster across multiple clouds based on K3S + WireGuard + Kilo. 💪💪💪
steps
1. Premise
1.1 Multiple cloud hosts across clouds
Prepare at least 2 cloud hosts with different public clouds (configured at least 1C1G to run), here are 6 with different hostname requirements, namely:
Sky Wing Cloud:ty1（K3S Server）
Alibaba Cloud:ali（K3S Agent）
HUAWEI CLOUD:hw1（K3S Agent）
Baidu Cloud:bd1 and bd2（K3S Agent）
Tencent Cloud:tx1（K3S Agent）
1.2 Operating System
Operating system: Ubuntu 20.04 is recommended (as of 1/22/2022) because WireGuard can be easily installed.
1.3 WireGuard installed
And WireGuard is installed, see 👉️ the installation processOver here
1.4 Network
Agreement
Port
Source
Description
TCP
6443
K3s agent node
Kubernetes API Server
UDP
51820
K3s server and agent node
Kilo inter-network communication
TCP
10250
K3s server and agent node
Kubelet metrics
TCP
2379-2380
K3s server node
Typically, all outbound traffic is allowed.
1.5 Make sure that the cloud host has a public IP address
Each location has at least one node’s IP address that is reachable to other locations.
If the location is in a different cloud or private network, then this must be a public IP address. If the IP address is not automatically configured on the node’s Ethernet device, it can be used kilo.squat.ai/force-endpoint The comment mode is specified manually.
1.6 Cloud hosts enable IPv4 forwarding
at /etc/sysctl.conf There is a line like this:
net.ipv4.ip_forward=1
And take effect:systel -p
1.7 (Optional) Configure the image repository configuration
K3S_CLUSTER_INIT=true: cluster mode, which installs the built-in etcd instead of sqlite3;
--tls-san {{ server_public_ip }}: {{ server_public_ip }} Change to the k3s server public IP that you specify,--tls-san This option adds an additional hostname or IP as an alternate name to the TLS certificate, which you can specify multiple times if you want to access by IP and hostname.
--node-external-ip {{ server_public_ip }}: Specify the public IP address of the node
--flannel-backend none:K3S The default network plugin is flannel, which means that flannel is not used, and Kilo is installed separately.
(Optional):--kube-apiserver-arg feature-gates=EphemeralContainers=true Enable feature-gates:EphemeralContainers to facilitate sidecar attachment for debugging without restarting the pod.
❌ mistake：
If not specified --tls-san , may cause kubectl Unable to pass server_public_ip to access the cluster.
If not specified --node-external-ip , which may prevent k3s agents located in other clouds from connecting to the K3S Server’s APIs.
If not specified --kube-proxy-arg metrics-bind-address=0.0.0.0, which may result in the inability to fetch metrics.
2.2 View installation results
The details are as follows:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
❯ systemctl status k3s.service ● k3s.service - Lightweight Kubernetes Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2022-01-22 16:27:14 CST; 4h 5min ago Docs: https://k3s.io Process: 5757 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS) Process: 5758 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS) Main PID: 5759 (k3s-server) Tasks: 49 Memory: 926.0M CGroup: /system.slice/k3s.service ├─ 5759 /usr/local/bin/k3s server ├─ 5774 containerd ├─18561 /var/lib/rancher/k3s/data/2e877cf4762c3c7df37cc556de3e08890fbf450914bb3ec042ad4f36b5a2413a/bin/containerd-shim-runc-v2 -namespace k> ├─18579 /pause └─18745 /opt/bin/kg --kubeconfig=/etc/kubernetes/kubeconfig --hostname=ty1.k3s
Jan 22 17:28:41 ty1.k3s k3s[5759]: I0122 17:28:41.435542 5759 reconciler.go:319] "Volume detached for volume... ...
1 2 3 4 5 6 7 8 9 10 11 12 13
❯ journalctl -f -b -u k3s.service -- Logs begin at Fri 2021-03-26 09:47:06 CST, end at Sat 2022-01-22 20:35:12 CST. -- Jan 22 16:20:21 ty1.k3s systemd[1]: Starting Lightweight Kubernetes... Jan 22 16:20:22 ty1.k3s k3s[1660]: time="2022-01-22T16:20:22+08:00" level=info msg="Starting k3s v1.22.5+k3s1 (405bf79d)" Jan 22 16:20:22 ty1.k3s k3s[1660]: time="2022-01-22T16:20:22+08:00" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, max> Jan 22 16:20:22 ty1.k3s k3s[1660]: time="2022-01-22T16:20:22+08:00" level=info msg="Configuring database table schema and indexes, this may take a momen> Jan 22 16:20:22 ty1.k3s k3s[1660]: time="2022-01-22T16:20:22+08:00" level=info msg="Database tables and indexes are up to date" Jan 22 16:20:22 ty1.k3s k3s[1660]: time="2022-01-22T16:20:22+08:00" level=info msg="Kine available at unix://kine.sock" Jan 22 16:20:22 ty1.k3s k3s[1660]: time="2022-01-22T16:20:22+08:00" level=info msg="Reconciling bootstrap data between datastore and disk" Jan 22 16:20:22 ty1.k3s k3s[1660]: time="2022-01-22T16:20:22+08:00" level=info msg="Running kube-apiserver --advertise-address=x.x.x.x --adverti> Jan 22 16:20:22 ty1.k3s k3s[1660]: Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24. Jan 22 16:20:22 ty1.k3s k3s[1660]: I0122 16:20:22.307328 1660 server.go:581] external host was not specified, using x.x.x.x Jan 22 16:20:22 ty1.k3s k3s[1660]: I0122 16:20:22.308688 1660 server.go:175] Version: v1.22.5+k3s1
1 2 3 4
❯ k3s kubectl cluster-info Kubernetes control plane is running at https://127.0.0.1:6443 CoreDNS is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy Metrics-server is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
1 2 3
❯ k3s kubectl get node NAME STATUS ROLES AGE VERSION ty1.k3s NotReady control-plane,master 4h10m v1.22.5+k3s1
⚠️ note：
The status of the K3S Server at this point is NotReady, which is normal because the CNI network plug-in has not been installed.
3. Install Kilo
3.1 Specify the K3S Server topology
My K3S Server is on Tianyi Cloud, specify the topology:
/ # ping 10.42.2.7 PING 10.42.2.7 (10.42.2.7): 56 data bytes 64 bytes from 10.42.2.7: seq=0 ttl=62 time=6.604 ms 64 bytes from 10.42.2.7: seq=1 ttl=62 time=6.520 ms 64 bytes from 10.42.2.7: seq=2 ttl=62 time=6.412 ms 64 bytes from 10.42.2.7: seq=3 ttl=62 time=6.430 ms 64 bytes from 10.42.2.7: seq=4 ttl=62 time=6.487 ms ^C --- 10.42.2.7 ping statistics --- 5 packets transmitted, 5 packets received, 0% packet loss round-trip min/avg/max = 6.412/6.490/6.604 ms / # ping 10.42.1.3 PING 10.42.1.3 (10.42.1.3): 56 data bytes 64 bytes from 10.42.1.3: seq=0 ttl=62 time=7.426 ms 64 bytes from 10.42.1.3: seq=1 ttl=62 time=7.123 ms 64 bytes from 10.42.1.3: seq=2 ttl=62 time=7.109 ms 64 bytes from 10.42.1.3: seq=3 ttl=62 time=7.129 ms ^C --- 10.42.1.3 ping statistics --- 4 packets transmitted, 4 packets received, 0% packet loss round-trip min/avg/max = 7.109/7.196/7.426 m
Pinging the WireGuard address of other nodes, or pinging, is as follows:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
/ # ping 10.4.0.1 PING 10.4.0.1 (10.4.0.1): 56 data bytes 64 bytes from 10.4.0.1: seq=0 ttl=64 time=0.077 ms 64 bytes from 10.4.0.1: seq=1 ttl=64 time=0.099 ms ^C --- 10.4.0.1 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.077/0.088/0.099 ms / # ping 10.4.0.2 PING 10.4.0.2 (10.4.0.2): 56 data bytes 64 bytes from 10.4.0.2: seq=0 ttl=63 time=29.000 ms 64 bytes from 10.4.0.2: seq=1 ttl=63 time=28.939 ms ^C --- 10.4.0.2 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 28.939/28.969/29.000 ms
Ping the default private network address of other nodes, you can also ping, as follows:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
/ # ping 172.17.0.3 PING 172.17.0.3 (172.17.0.3): 56 data bytes 64 bytes from 172.17.0.3: seq=0 ttl=63 time=6.327 ms 64 bytes from 172.17.0.3: seq=1 ttl=63 time=6.350 ms ^C --- 172.17.0.3 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 6.327/6.338/6.350 ms / # ping 192.168.64.4 PING 192.168.64.4 (192.168.64.4): 56 data bytes 64 bytes from 192.168.64.4: seq=0 ttl=63 time=29.261 ms 64 bytes from 192.168.64.4: seq=1 ttl=63 time=29.015 ms ^C --- 192.168.64.4 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 29.015/29.138/29.261 ms
Proof network (pod-pod, pod-node) all-pass.
5.2 i️ Detailed Description
5.2.1 Network Card
Kilo has 3 network cards, namely:
kilo0: WireGuard VPN network, used to form VPN intranets between nodes; （bd2 There is no NIC because and bd1 It belongs to the same VPC intranet, and the connection in the intranet is not encrypted by WireGuard by default, and only the outbound VPC intranet will be encrypted by WireGuard)
kube-bridge: Bridging the network, so that the network card of the pod is connected with the network card of the cloud host, so as to realize the shared communication of the pod through the WireGuard VPN network;
tunl0: In Bridge mode, multi-host network communication requires additional configuration of host routing, or the use of overlay networking. Automatic configuration via Kilo. For example, the network structure in the case of overlay is:
thereintokube-bridge The configuration is as follows:
visit 10.42.4.0/24(pod on bd1) and 10.42.5.0/24(pod on bd2), SNAT all follow bd1’s WireGuard VPN IP: 10.4.0.2 (because bd1 and bd2 are on the same VPC intranet, and only bd1 has a WireGuard VPN IP, not on bd2)
ty1 accesses other nodes through the private IP address that comes with the cloud host kilo0 SNAT is followed by the corresponding WireGuard VPN IP
BD1 access 10.42.5.0/24(pod on bd2), pass tunl0.
bd2, just kube-bridge and tunl0
The same machine pod visits each other,kube-bridge
Access other machine pods, go tunl0 The VPC private IP address (192.168.64.4) is forwarded to bd and forwarded by bd1.
🗒️ Refine：
In the same node, take the bridge;
Same location, go to the private IP (VPC network) that comes with it;
Different locations, go WireGuard VPN;
kgctl
Kilo provides a command-line tool for inspecting and interacting with clusters:kgctl。
This tool can be used to understand the topology of a mesh, get the WireGuard configuration of peers, or plot a cluster diagram.
kgctlNeed to go through the setupKUBECONFIGEnvironment variables or provided--kubeconfigflags to provide a Kubernetes configuration file.
Installation
kgctlThe binaries are automatically compiled for Linux, macOS, and Windows, and every version of Kilo can be downloaded from GitHub release page Download.
graphcommand to generate a graph representing a Kilo grid in GraphViz format. The diagram is useful for understanding or debugging the network topology.
Example:
1
kgctl graph
This will produce some DOT graph description language output, such as:
To render a graph, use one of the GraphViz layout tools, for examplecirco:
1
kgctl graph | circo -Tsvg > cluster.svg
showconf
showconfThe command outputs the WireGuard configuration of a node or peer in the cluster, that is, the configuration that the node or peer needs to set on its local WireGuard interface in order to participate in the mesh. Example:
1 2
NODE=master # the name of a node kgctl showconf node $NODE
This will produce some output in INI format, for example.
