Prometheus Operator and kube-prometheus One - Introduction

Brief introduction

Prometheus Operator

Prometheus Operator: Manage Prometheus clusters on Kubernetes. The goal of the project is to simplify and automate the configuration of the Prometheus-based Kubernetes cluster monitoring stack.

kube-prometheus

The easiest way is to deploy Prometheus Operator as part of kube-prometheus. kube-prometheus has deployed Prometheus Operator and has arranged a prometheus called prometheus-k8s, with alerts and rules by default, and with other components that prometheus requires, such as:

• Grafana
• kube-state-metrics
• prometheus adapter
• node exporter
• …

Prometheus Operator vs. kube-prometheus vs. community helm chart

Prometheus Operator

Prometheus Operator uses Kubernetes custom resources to simplify the deployment and configuration of Prometheus, Alertmanager, and related monitoring components.

kube-prometheus

kube-prometheus Provides an example configuration of a complete cluster monitoring stack based on Prometheus and Prometheus Operator. This includes deploying multiple Prometheus and Alertmanager instances, metrics exporters such as node_exporters to collect node metrics, target configurations that link Prometheus to various metrics endpoints, and sample alert rules to notify of potential issues in the cluster.

helm chart

prometheus-community/kube-prometheus-stack Helm Chart provides a similar feature set to Kube-Prometheus. This chart is maintained by the Prometheus community.

Prometheus Operator features

CRD

A core feature of the Prometheus Operator is to watch changes to specific objects by the Kubernetes API server and ensure that the current Prometheus deployment matches those objects. The Operator operates on the following custom resource definitions (CRDs):

monitoring.coreos.com/v1:

• Prometheus: It defines the deployment that Prometheus expects.
• Alertmanager: It defines the deployment expected by AlertManager.
• ThanosRuler: It defines the deployment expected by ThanosRuler; If there are multiple instances of Prometheus, pass ThanosRuler Unified management of alarm rules.
• ServiceMonitor: Prometheus Operator passed PodMonitor and ServiceMonitor Implement monitoring of resources,ServiceMonitor It is recommended to monitor any resource in K8S through Service ServiceMonitor. It declaratively specifies how the Kubernetes service should be monitored. The Operator automatically generates a Prometheus scratch configuration based on the current state of the objects in the API server.
• PodMonitor: Prometheus Operator passed PodMonitor and ServiceMonitor Implement monitoring of resources,PodMonitor Used to monitor pods, recommended choice ServiceMonitor. PodMonitor Declaratively specifies how a group of pods should be monitored. The Operator automatically generates a Prometheus scratch configuration based on the current state of the objects in the API server.
• Probe: It declaratively specifies how ingress or static target groups should be monitored. The Operator automatically generates a Prometheus scratch configuration based on the definition.
• PrometheusRule: for managing Prometheus alert rules; It defines the required set of Prometheus alert and/or logging rules. Prometheus generates a rules file that can be used by Prometheus instances.
• AlertmanagerConfig: Used to manage AlertManager profiles, mainly to whom alerts are sent; It declaratively specifies subsections of the Alertmanager configuration, allows alerts to be routed to custom receivers, and sets suppression rules.

Prometheus Operator automatically detects changes made by the Kubernetes API server to any of the above objects and ensures that matching deployments and configurations are kept in sync.

Simplified deployment configuration

Configure the basics of Prometheus, such as versions, persistence, retention policies, and copies from native Kubernetes resources. The simplest persistent deployment of Prometheus only needs to create the following yaml:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: persisted
spec:
  storage:
    volumeClaimTemplate:
      spec:
        storageClassName: ssd
        resources:
          requests:
            storage: 40Gi

Prometheus target configuration

Automatically generate monitoring target configurations based on familiar Kubernetes label queries; There is no need to learn Prometheus-specific configuration language.

Big factory case

Which big manufacturers are using Prometheus Operator or kube-prometheus?

RedHat

It can also be seen from the API of Prometheus Operator that this Operator was originally developed and open sourced by CoreOS, and now CoreOS has been acquired by RedHat, so RedHat’s OpenShift 4 completely uses Prometheus Operator as its Metrics solution. A typical architecture is shown below:

You can see that both Prometheus and AlertManager are managed through the Prometheus Operator.

Rancher

Rancher-monitoring after Rancher 2 is also based on kube-prometheus to make further improvements, which is through the Rancher-monitoring helm chart after deployment, you can see that there are still many deployed components:

• Grafana
• Prometheus CRD
• Prometheus Operator
• Prometheus
• AlertManager
• kube-state-metrics
• prometheus adapter
• node exporter
• …

Why do I recommend using Prometheus Operator or kube-prometheus instead of native prometheus?

The reasons are as follows:

1. The choice of many large manufacturers;
2. Greatly simplifies the configuration complexity of Prometheus;
3. Tons out of the box:
   1. Monitor objects such as: K8S components - coredns, kubelet, controller manager, apiserver, etcd, scheduler, kube proxy; Monitoring component self-monitoring - grafana, AlertManager, prometheus, etc.;
   2. Dashboard, comes with 24 dashboards, very practical, covering: cluster/component/network/storage/node/pod and other dimensions;
   3. Alarm rules, which come with more than 100 alarm rules, covering all aspects of K8S;
4. Many popular open source products also come with support for Prometheus Operator by default, such as Loki has a related ServiceMonitor;
5. ServiceMonitor and the like actually have more flexibility than adding Prometheus Annotation; As in the example below
6. Highly available support, such as:
   1. Multiple Prometheus shards
   2. Multiple AlertManagers
   3. ThanosRuler
7. RBAC: By default, 3 monitoring roles can be created: admin/edit/viewer, which can correspond to the monitoring administrator, maintenance personnel and read-only users;

Example, flexibility:

spec:
  endpoints:
    - honorLabels: true
      params:
        _scheme:
          - https
      port: metrics
      proxyUrl: http://pushprox-k3s-server-proxy.cattle-monitoring-system.svc:8080
      relabelings:
        - sourceLabels:
            - __metrics_path__
          targetLabel: metrics_path
  jobLabel: component
  namespaceSelector:
    matchNames:
      - cattle-monitoring-system
  podTargetLabels:
    - component
    - pushprox-exporter
  selector:
    matchLabels:
      component: k3s-server
      k8s-app: pushprox-k3s-server-client
      provider: kubernetes
      release: rancher-monitoring

📚️ Reference documentation

• prometheus-operator/prometheus-operator: Prometheus Operator creates/configures/manages Prometheus clusters atop Kubernetes (github.com)
• Prometheus Operator - Running Prometheus on Kubernetes (prometheus-operator.dev)
• OperatorHub.io | The registry for Kubernetes Operators

above

EOF