Kustomize Production Combat - Injection Monitoring APM Agent

This article was last updated on: February 7, 2024 pm

Introduction to Kustomize

Kubernetes native configuration management toolIt customizes the use of out-of-the-box apps by introducing a template-free way to customize app configurations. Currently, inkubectlBuilt in, passed apply -k ready to use.

Kustomize traverses the Kubernetes manifest to add, remove, or update configuration options without forking. It can be used both as a standalone binary and askubectlis used by native features.

Kustomize advantage

  • 📢 Fully declarative approach to configuration customization
  • 🍸 Build in natively kubectl middle
  • 🎶 Manage any number of uniquely customized Kubernetes configurations
  • ☸ Supplied as a standalone binary for extension and integration into other services
  • 📃 Each artifact used by customization is pure YAML and can be validated and processed
  • 🔁 Kustomize supports fork/modify/rebase workflows
  • 🔧 It is perfectly supported by GitOps tools such as ArgoCD

What Kustomize can do

📚️ Reference:

👉️URL: https://mp.weixin.qq.com/s/gmwkoqZpKbq1hM0B8XxQNw
In Kubernetes we use YAML files to declare how our application should be deployed to the underlying cluster, these YAML files contain application definitions, tags for governance needs, logs, security context definitions, resource dependencies, etc., when our application scales to hundreds or thousands of pods, managing these YAML files will become a nightmare.

Typically, there are many projects to manage, and at the same time there are multiple different deployment environments: development, test, UAT, and production. There may even be different public cloud Kubernetes distributions. Then each environment requires a variety of YAML files, but they directly differ only in some details. For example: image tag, service name, label, whether there is storage, etc.

If it’s all manual, the maintenance is huge and error-prone.

Compared with Helm, Kustomize is more suitable for solving the pain points of this scenario: there is a base template to manage all the basic YAML of a project, and more advanced requirements are overlayed through overlays.

Another typical scenario is fusing configuration data from different users (e.g. from devops/SRE and developers).

It can also be combined with helm for some more advanced configurations.

Let’s take a typical scenario today: production deployment is automatically injected into commercial (e.g. AppDynamics) or open source (SkyWalking/pinpoint) out-of-the-box Java Agent.

Hands-on - Monitor APM Agents via Kustomize injection

Background overview

In Java, for example, the APM Agent package here is an out-of-the-box Agent jar package provided by commercial (e.g., AppDynamics) or open source (SkyWalking/pinpoint) products.

In the Kubernetes scenario, there are several considerations:

  1. Separation from application images;
  2. Reuse

The Agent jar package is made into a general image, copied to the running application container through the init container, and automatically set the parameters by configuring environment variables.

✍️ Author’s Note:

In fact, commercial APM has the function of Helm or Operator to implement automatic installation and configuration, but the experience in actual use is not good, which is not suitable for our actual scenarios.

Monitor APM Agent steps through Kustomize injection

The steps are briefly described below: (using the AppDynamics Java Agent as an example)

  1. Get the original Deployment yaml file (pass kubectl and kubectl neat plugins)
  2. Pass each deployment via Kustomize patches Implement the following steps:
    1. inject initContainers: appd-agent-attach-javashould initContainers Yes:
      1. volumeMounts: Mount the Java Agent JAR package to the volume implementation share;
    2. In the Deployment -> application’s own container, add the following parameters:
      1. volumeMounts: Mount the Java Agent JAR package to the volume implementation share;
      2. Add env information required by various AppD Java agents;
    3. volumes Sharing is achieved through tmpdir.

Kustomize directory structure

The directory structure is as follows:

1
2
3
4
5
6
7
inject-appd-agent/
├── base
│   └── kustomization.yaml      
├── overlays
    └── prod
        ├── appd_agent.yaml   
        └── kustomization.yaml

Among them, the deployment of all subsequent applications that need to be injected into the APM Agent is placed base directory.

base/kustomization.yaml

1
2
resources:
  - ./foo-deployment.yml

🐾Note:
Here’s a note, currently resources File glob matching is not supported, and the specific issue can be found here:

But there is a temporary solution, which is by executing the command:kustomize edit add resource base/*.yml After running, it will iterate through the file blob and add the results one by one kustomization.yaml Middle.

The file after running might look like this:

1
2
3
4
5
6
7
8
resources:
  - ./foo-deployment.yml
  - ./bar-deployment.yml
  - ./a-deployment.yml
  - ./b-deployment.yml
  - ./c-deployment.yml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

overlays/prod/kustomization.yaml

1
2
3
4
5
6
7
8
bases:
  - ../../base
# patchesStrategicMerge:
#   - appd_agent.yaml
patches:
  - path: appd_agent.yaml
    target:
      kind: Deployment

🐾 Note:

It’s useless here patchesStrategicMerge, but usedpatches. The purpose is to batch patches. The above YAML means, willappd_agent.yaml Applies to all Deployment manifests.

overlays/prod/appd_agent.yaml

The specific contents are as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
apiVersion: apps/v1
kind: Deployment
metadata:
  name: not-important
spec:
  template:
    spec:
      containers:
        - name: app
          volumeMounts:
            - mountPath: /opt/appdynamics-java
              name: appd-agent-repo-java
          env:
            - name: APPDYNAMICS_AGENT_ACCOUNT_ACCESS_KEY
              value: xxxxxxxxxxxxxxxxxxxxxxxxxxxx
            - name: APPDYNAMICS_CONTROLLER_HOST_NAME
              value: 192.168.1.10
            - name: APPDYNAMICS_CONTROLLER_PORT
              value: '8090'
            - name: APPDYNAMICS_CONTROLLER_SSL_ENABLED
              value: 'false'
            - name: APPDYNAMICS_AGENT_ACCOUNT_NAME
              value: my_account
            - name: APPDYNAMICS_AGENT_APPLICATION_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: APPDYNAMICS_AGENT_TIER_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: APPDYNAMICS_AGENT_REUSE_NODE_NAME_PREFIX
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: JAVA_TOOL_OPTIONS
              value:
                ' -Dappdynamics.agent.accountAccessKey=$(APPDYNAMICS_AGENT_ACCOUNT_ACCESS_KEY)
                -Dappdynamics.agent.reuse.nodeName=true -Dappdynamics.socket.collection.bci.enable=true
                -javaagent:/opt/appdynamics-java/javaagent.jar'
            - name: APPDYNAMICS_NETVIZ_AGENT_HOST
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.hostIP
            - name: APPDYNAMICS_NETVIZ_AGENT_PORT
              value: '3892'
      initContainers:
        - command:
            - cp
            - -r
            - /opt/appdynamics/.
            - /opt/appdynamics-java
          image: appdynamics/appdynamics-java-agent:1.8-21.11.3.33314
          imagePullPolicy: IfNotPresent
          name: appd-agent-attach-java
          resources:
            limits:
              cpu: 200m
              memory: 75M
            requests:
              cpu: 100m
              memory: 50M
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /opt/appdynamics-java
              name: appd-agent-repo-java
      volumes:
        - name: appd-agent-repo-java

The above content will not be detailed, you can refer to this article to understand several ideas of APM Agent injection:

🐾 Note:
In practice, there are several points to note in the above content:

  1. spec/template/spec/containers/name Here is often the name of the app, such asfooIf appd_agent.yaml To patch correctly, you also need to write the same container name; This is not a problem for an application, but it is very inconvenient for batch automation/GitOps.

    1. I wanted to use Kustomize before nameReference To achieve it, but I didn’t figure it out, there are those who know can teach me
    2. ✔️20220706 11:00 update: Kustomize is strictly followed for security reasons metadata.name merged, so it can’t be solved by Kustomize; But it can pass yq(similar to jq, but for yaml) to solve, the example command is: i=foo yq -i '.spec.template.spec.containers[0].name="strenv(i)"' appd_agent.yaml

  2. The previous environment variables, when deployed manually, are as follows:

    1
    2
    - name: APPDYNAMICS_AGENT_APPLICATION_NAME
      value: foo

    Write directly to death, which is not convenient for batch automation/GitOps. The solution is to take advantage of Kubernetes env valueFrom Ability. Change it to the following.

    1
    2
    3
    valueFrom:
      fieldRef:
        fieldPath: metadata.name

  3. appd_agent.yaml The deployment name does not matter, because it will still be the name of the patch list; Other than that appd_agent.yaml Don’t bring it either namespace, convenient batch automation/GitOps, can be adapted to all NameSpace.

Automation scripts

Finally, the automation script demo looks like this: (Assuming the script is located in /inject-appd-agent/scripts/ under the directory)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
#!/bin/bash

# USAGE:
# bash inject-appd-agent.sh default

# NAMESPACE=default
NAMESPACE=$1
# only get deploy name
deployments=$(kubectl get deploy --no-headers=true -o custom-columns=:.metadata.name -n "${NAMESPACE}")

for i in ${deployments}; do
    # get deploy yaml
    cd ../base && kubectl get deploy -n "${NAMESPACE}" "${i}" -o yaml | kubectl neat >./"${i}.yml"
    # add the deploy yaml to base
    kustomize edit add resource ./"${i}.yml"
    # change appd_agent.yaml .spec.template.spec.containers[0].name
    #   = <the deploy name>
    cd ../overlays/prod && DEPLOY="${i}" yq -i '.spec.template.spec.containers[0].name = strenv(DEPLOY)' ./appd_agent.yaml
    # dry run
    # kustomize build . >../../examples/output/"${i}.yml"

    # deploy rollout
    kubectl apply -k .

    # remove the deploy yaml resource from base
    cd ../../base && kustomize edit remove resource ./"${i}.yml"
done

# clean up
kustomize edit remove resource ./*.yml
cd ../overlays/prod && yq -i '.spec.template.spec.containers[0].name = "app"' ./appd_agent.yaml

Run the above script to automatically inject AppD Java Agent into all deployments and Rollout takes effect.

🎉🎉🎉

📚️ Reference

CloudNative
#K8S #CloudNative #APM #DevOps #Monitoring #Observability #Tracing #Kustomize #GitOps
Kustomize Production Combat - Injection Monitoring APM Agent
https://e-whisper.com/posts/7973/
Author
east4ming
Posted on
July 5, 2022
Licensed under