Best practices for production on containerized applications

This article was last updated on: February 7, 2024 pm

preface

Lately busy to death, 👻👻👻 . Last week came once more fierce than 996 907. This Tuesday was finally a little unbearable, take a day off, take a little rest.

At the same time, my hands itch can’t do it, and I sent up the heavy article that I have been preparing for a long time haha. 😆😆😆

However, the time is still a bit rushed, so this time I will start first, and then there will be time to refine it.

img

Best practices for production on containerized applications

  1. Check whether the image and container are usedrootLaunch as well as configure other privileges. If not necessary, always use ordinary users.
  2. Check the imageLANGDisposition: LANG = en_US.UTF-8. Purpose: To avoid problems such as garbled characters in production
  3. Check the mirror time zone configuration: TZ=Asia/Shanghai Purpose: To avoid time zone inconsistencies in production
  4. Configure externalization. There are several means of externalization:
  5. Same image, from test stream to production. Give the mirror a hit${version}or${gitCommitId}Labels for this category. Purpose: Ensure the correct version flow to production by version number or commit id
  6. Discusses each component
  7. Log output optimization:
  8. Optionally, install redis/kafka/rabbitmq clusters (and configure exporter monitoring) as needed
  9. Microservice parameter optimization:
  10. DevOps pipelines for DEV, TEST, UAT, PRE-PROD, PROD.
  11. Configure Readiness and Liveness probes.
  12. Added JMX-exporter monitoring and tracing monitoring.
  13. NGINX conf suggests adding: worker_processes 1; Then adjust the number of replicas as needed.
  14. Optionally, configure the PDB, specifying during the upgrade or reboot:maxUnavailable orminAvailable(Especially suitable for: stateful applications. typical examples: redis, kafka, zookeeper, etc.)
  15. Configure anti-affinitypodAntiAffinity. Ensure that the same set of microservices/applications/components is as scattered as possible across different nodes.

5-6 Operation steps:

1
2
nodeSelector:
  zone: internet

An example of the 11-step step is as follows:

1
2
3
4
5
6
7
8
9
10
livenessProbe:
  tcpSocket:
    port: 8080
  initialDelaySeconds: 60
readinessProbe:
  httpGet:
    path: /myapp/services/
    port: 8080
    scheme: HTTP
  initialDelaySeconds: 60

An example of the 14 steps is as follows: (Note the keywords: maxUnavailable and minAvailable)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
kind: PodDisruptionBudget
apiVersion: policy/v1beta1
metadata:
  name: kafka-prod-kafka
  labels:
    app.kubernetes.io/instance: kafka-prod
    app.kubernetes.io/managed-by: strimzi-cluster-operator
    app.kubernetes.io/name: strimzi
    strimzi.io/cluster: kafka-prod
    strimzi.io/kind: Kafka
    strimzi.io/name: kafka-prod-kafka
spec:
  selector:
    matchLabels:
      strimzi.io/cluster: kafka-prod
      strimzi.io/kind: Kafka
      strimzi.io/name: kafka-prod-kafka
  maxUnavailable: 1
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
kind: PodDisruptionBudget
apiVersion: policy/v1beta1
metadata:
  name: redis-cluster-redis
  namespace: myapp
  labels:
    app.kubernetes.io/component: redis
    app.kubernetes.io/managed-by: redis-operator
    app.kubernetes.io/name: redis
    app.kubernetes.io/part-of: redis-cluster
    redis.kun/v1beta1: myapp_redis
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app.kubernetes.io/component: redis
      app.kubernetes.io/managed-by: redis-operator
      app.kubernetes.io/name: redis
      app.kubernetes.io/part-of: redis-cluster
      redis.kun/v1beta1: myapp_redis

An example of the 15 steps is as follows: (Note the keywords: podAntiAffinity)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: redis-cluster-redis
  labels:
    app.kubernetes.io/component: redis
    app.kubernetes.io/managed-by: redis-operator
    app.kubernetes.io/name: redis
    app.kubernetes.io/part-of: redis-cluster
    redis.kun/v1beta1: myapp_redis
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/component: redis
      app.kubernetes.io/managed-by: redis-operator
      app.kubernetes.io/name: redis
      app.kubernetes.io/part-of: redis-cluster
      redis.kun/v1beta1: myapp_redis
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/component: redis
        app.kubernetes.io/managed-by: redis-operator
        app.kubernetes.io/name: redis
        app.kubernetes.io/part-of: redis-cluster
        redis.kun/v1beta1: myapp_redis
    spec:
      containers:
        <...>
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/component: redis
                    app.kubernetes.io/managed-by: redis-operator
                    app.kubernetes.io/name: redis
                    app.kubernetes.io/part-of: redis-cluster
                    redis.kun/v1beta1: myapp_redis
                topologyKey: kubernetes.io/hostname
...

finish

🎉🎉🎉