Skip to content

Kubernetes Pod Annotations for Prometheus scrape configuration

Kubernetes Pod annotations are used to configure Prometheus to discover and scrape Metrics from application pods dynamically. By adding specific metadata to a Pod's definition, Prometheus can automatically identify the target endpoint, port, and path without requiring manual configuration in the prometheus.yml file for every single service[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md][400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md]^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md].

This mechanism relies on the Prometheus [[Service Discovery|Service Discovery]] feature, specifically the kubernetes-apiservers or kubernetes-pods roles, which watches the Kubernetes API for changes to Pod resources[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md][400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md].

Key Annotations

The following annotations are recognized by the standard Prometheus scrape configuration (relabel_configs) to control how Metrics are collected^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md]:

  • prometheus.io/scrape:
    • Description: Determines whether Prometheus should scrape this Pod.
    • Value: true to enable scraping. If not set or set to false, the Pod is ignored^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md].
  • prometheus.io/path:
    • Description: The path to the Metrics endpoint on the Pod.
    • Value: A string representing the path (e.g., /metrics, /actuator/prometheus). If omitted, Prometheus typically defaults to /metrics^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md].
  • prometheus.io/port:
    • Description: The port number where the Metrics endpoint is exposed.
    • Value: A valid port number exposed by the container (e.g., 8080, 9090)[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md][400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md].
  • prometheus.io/scheme (Optional):
    • Description: The scheme to use for scraping.
    • Value: http or https. Defaults to http if not specified^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md].

Example Configuration

To enable scraping for a standard HTTP application, add these annotations under the metadata section of your Pod or Deployment manifest^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md]:

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"
    prometheus.io/path: "/[Metrics](<./metrics.md>)"

For applications requiring specific schemes (like Traefik) or paths, adjust the values accordingly^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md]:

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "12346"
    prometheus.io/path: "/"
    prometheus.io/scheme: "http"

How It Works

When the Prometheus server runs a job with the Kubernetes service discovery role enabled (e.g., role: pod), it queries the API for all Pods. It then processes the list using relabel_configs^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md]:

  1. Keep/Drop: It checks the prometheus.io/scrape annotation. If it is not true, the Pod is dropped as a target^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md].
  2. Path Replacement: If prometheus.io/path is present, its value replaces the default __metrics_path__ label^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md].
  3. Address Construction: The __address__ label (which usually defaults to the Pod's IP and a default port) is replaced by the Pod's IP combined with the port specified in prometheus.io/port^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md].

This allows for a "code as configuration" approach where developers control how their applications are monitored directly in their deployment specifications, assuming the Prometheus cluster-wide configuration supports these standard relabeling rules^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md].

Sources

^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md]