Rancher Series Article - K3s Traefik MiddleWare Error - Failed to create middleware keys

overview

The book continues from the previous time:Rancher series - K3S Cluster UpgradeWe mentioned that upgrading the K3S cluster through a one-click script has an error.

Next, start the analysis and repair of Traefik errors, the problem is:

• All of Traefik’s IngressRoute Access error 404 is reported

Description of the problem

The error is reported as follows:

time="2022-05-05T09:51:21Z" level=error msg="Failed to create middleware keys: middleware kube-system/hsts-header is not in the IngressRoute namespace cert-manager" namespace=cert-manager providerName=kubernetescrd ingress=grafana

That is, you cannot call Traefik MiddleWare across NameSpace.

Resolution process

First according to the official documentation:Kubernetes IngressRoute & Traefik CRD - Traefik

Can be configured allowCrossNamespace parameter, which defaults to falseif the parameter is set totrue, IngressRoutes can reference resources in other NameSpaces.

Basically, this is the root cause. Looking at the Traefik configuration of K3s v1.22.5+k3s2, there is indeed no such parameter, as follows:

...
containers:
  - name: traefik
    image: rancher/mirrored-library-traefik:2.5.0
    args:
      - '--entryPoints.metrics.address=:9100/tcp'
      - '--entryPoints.traefik.address=:9000/tcp'
      - '--entryPoints.web.address=:8000/tcp'
      - '--entryPoints.websecure.address=:8443/tcp'
      - '--api.dashboard=true'
      - '--ping=true'
      - '--metrics.prometheus=true'
      - '--metrics.prometheus.entrypoint=metrics'
      - '--providers.kubernetescrd'
      - '--providers.kubernetesingress'
      - >-
        --providers.kubernetesingress.ingressendpoint.publishedservice=kube-system/traefik
      - '--entrypoints.websecure.http.tls=true'
...      

So, initially, it was planned to add this parameter by editing Helm’s files.

Edit the Manifests Helm file for K3s

📚️ Reference:

• Automatic deployment of manifests and Helm charts
  at/var/lib/rancher/k3s/server/manifestsAny Kubernetes manifest found will look similarkubectl applyto automatically deploy to K3s. Manifests deployed in this way are managed as AddOn custom resources and can be run throughkubectl get addon -Ato check it out. You’ll find AddOns for packaged components such as CoreDNS, Local-Storage, Traefik, etc. AddOns are automatically created by the deployment controller and named after their filenames in the manifests directory.

The file is located at:/var/lib/rancher/k3s/server/manifests/traefik.yaml, which reads as follows:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: traefik-crd
  namespace: kube-system
spec:
  chart: https://%{KUBERNETES_API}%/static/charts/traefik-crd-10.3.001.tgz

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: traefik
  namespace: kube-system
spec:
  chart: https://%{KUBERNETES_API}%/static/charts/traefik-10.3.001.tgz
  set:
    global.systemDefaultRegistry: ""
  valuesContent: |-
    rbac:
      enabled: true
    ports:
      websecure:
        tls:
          enabled: true
    podAnnotations:
      prometheus.io/port: "8082"
      prometheus.io/scrape: "true"
    providers:
      kubernetesIngress:
        publishedService:
          enabled: true
    priorityClassName: "system-cluster-critical"
    image:
      name: "rancher/mirrored-library-traefik"
    tolerations:
    - key: "CriticalAddonsOnly"
      operator: "Exists"
    - key: "node-role.kubernetes.io/control-plane"
      operator: "Exists"
      effect: "NoSchedule"
    - key: "node-role.kubernetes.io/master"
      operator: "Exists"
      effect: "NoSchedule"

Add the following configuration to the above yaml:

...
    providers:
      kubernetesCRD:
        allowCrossNamespace: true

After it takes effect, it can indeed return to normal, but K3s will periodically reset the manifests to the original configuration, which will cause the problem to recur.

So the problem is not finally solved.

Use HelmChartConfig to customize packaged components

However, according to the official documentation, we can pass Use HelmChartConfig to customize packaged components way to override the value of packaged components deployed as HelmCharts (such as Traefik).

The specific configuration is as follows:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    globalArguments:
    - "--providers.kubernetescrd.allowcrossnamespace=true"

After taking effect, it returned to normal, and no rollback occurred.

Problem solving.

🎉🎉🎉

📚️ Reference documentation

• Automatic deployment of manifests and Helm charts
• Use HelmChartConfig to customize packaged components