Skip to content

Prometheus service discovery configurations

Prometheus service discovery configurations are the settings within Prometheus that define how the monitoring system dynamically identifies and connects to target endpoints (such as nodes, pods, or services) for Metrics collection^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md].

Configuration Overview

In the Prometheus configuration file (prometheus.yml), service discovery is defined under scrape_configs. Each job specifies a kubernetes_sd_configs section that determines the role (e.g., node, pod, service, endpoints) used to discover targets^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md].

Relabeling Mechanism

Service discovery often returns a large set of metadata (labels) for each target. To manage this and control which targets are actually scraped, Prometheus uses relabel_configs^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md].

The relabel_configs block allows users to: * Filter Targets: Keep or drop specific targets based on label values using the action: keep or action: drop directive^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md]. * Transform Labels: Rename existing metadata labels to standard Prometheus labels using action: replace. For example, __address__ can be constructed from multiple source labels^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md]. * Metadata Handling: Use action: labelmap to copy labels matching a regex (e.g., __meta_kubernetes_pod_label_(.+)) to the target, preserving Kubernetes labels as Prometheus labels^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md].

Common Discovery Roles

Prometheus supports several roles specific to Kubernetes environments^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md]:

  • role: pod: Discovers all Pods and utilizes their IP addresses and exposed ports.
  • role: node: Discovers all Nodes, typically used for scraping Node Exporter or kubelet metrics^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md].
  • role: endpoints: Discovers Endpoint objects, which is useful for scraping Service-backed targets^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md].
  • role: service: Discovers all Services.

Example: Pod Scrape with Annotations

To enable standard scraping for a Pod, specific annotations can be applied to the Pod metadata. Prometheus detects these annotations during the role: pod discovery process^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md].

Common annotations include: * prometheus_io_scrape: Set to true to enable scraping^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md]. * prometheus_io_path: Defines the Metrics path (e.g., /metrics). * prometheus_io_port: Specifies the port to scrape^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md].

Sources