How to monitor K8s out-of-cluster services with Prometheus Operator?

This article was last updated on: July 24, 2024 am

preface

In previous series of articles:

Prometheus Operator and one of kube-prometheus - Introduction - Dongfeng Weiming Technology Blog (e-whisper.com)
Three Scenarios for Monitoring Kubernetes Cluster Certificate Expiration - Dongfeng Weiming Technology Blog (e-whisper.com)

Some advantages of Prometheus Operator over native Prometheus are introduced, which have been widely adopted by major vendors and popular open source cloud components. Recommended use.

But in practice, not all components may be in the K8S cluster, such as: LB, DB, global DNS, cloud services…

How can I monitor them with Prometheus Operator? Here are the following scenarios (not a solution, just a little trick)

Use Prometheus Operator to monitor K8s out-of-cluster service scenarios

As mentioned above, the K8s extra-cluster services here refer to some such as LB, DB, global DNS, cloud services… Static services.

For such services, there are the following monitoring schemes:

Via Prometheus Operator CR - prometheus spec;
1. This scheme and other configurations of Prometheus have high coupling;
Via external name Service + ServiceMonitor
1. This scheme has a premise, that is: the monitored service is a domain name;
Pass Service + Endpoint + ServiceMonitor
1. This scheme has strong adaptability and low coupling. Recommend. 👍️
If it is the monitoring of the BlackboxProbe class, that is, monitor: various parameters of the Endpoint (HTTP/S, DNS, TCP, ICMP and grpc), including HTTP response time, DNS query latency, SSL certificate expiration information, TLS version, etc. It can be used directly Probe CR, above: How do I monitor URLs using Blackbox Exporter? - Dongfeng Weiming Technology Blog (e-whisper.com) It has already been mentioned, so I will not repeat it this time.

Option 1: `prometheus` spec

In short, it’s directly in prometheus Add a static configuration like this (static_configs) to the spec:

1
2
3

static_configs:
  - targets:
    - SERVICE-FQDN

The specific configuration example is as follows:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: monitor-kube-prometheus-st-prometheus
spec:
  additionalScrapeConfigs:
  - job_name: external
    metrics_path: /metrics
    static_configs:
      - targets:
        - <IP>:<PORT>

Solution 2: External name `Service` + `ServiceMonitor`

Take advantage of Kubernetes Externalname Serivce, mapping services to DNS names instead of typical selection operators such as my-service or cassandra.

Configure the Externalname Service:

apiVersion: v1
kind: Service
metadata:
  name: gpu-metrics-svc
  namespace: monitoring
  labels:
    k8s-app: gpu-metrics
spec:
  type: ExternalName
  externalName: <gpu-machine-ip>
  clusterIP: ''
  ports:
    - name: metrics
      port: 9100
      protocol: TCP
      targetPort: 9100

Configure a ServiceMonitor that points to the service:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: gpu-metrics-sm
  labels:
    k8s-app: gpu-metrics
    prometheus: kube-prometheus
spec:
  selector:
    matchLabels:
      k8s-app: gpu-metrics
    namespaceSelector:
      matchNames:
        - monitoring
  endpoints:
    - port: metrics
      interval: 10s
      honorLabels: true

Option 3: `Service` + `Endpoint` + `ServiceMonitor`

Pass Service + Endpoint way, explicitly map external services as internal services.

Examples are as follows:

kind: Service
apiVersion: v1
metadata:
  name: external-es-exporter
  labels:
    app: elasticsearch
  namespace: monitoring
spec:
  type: ClusterIP
  ports:
    - name: metrics
      port: 9114
      protocol: TCP
      targetPort: 9114

apiVersion: v1
kind: Endpoints
metadata:
  name: external-log-es-exporter
  labels:
    app: elasticsearch
  namespace: monitoring
subsets:
  - addresses:
      - ip: <elasticsearch_ip_1>
      - ip: <elasticsearch_ip_2>
      - ip: <elasticsearch_ip_3>
    ports:
      - name: metrics
        port: 9114
        protocol: TCP

Similar to the second solution, create the corresponding ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: elasticsearch
spec:
  selector:
    matchLabels:
      app: elasticsearch
    namespaceSelector:
      matchNames:
        - monitoring
    endpoints:
    - port: metrics
      path: /metrics
      interval: 30s

Although this bypasses some, it can be guaranteed that when modifying the monitoring of component A, it will not affect the configuration of component B at all; In addition, it will not affect other monitoring of Prometheus.

More precise configuration;
finer granularity;
Lower coupling.