Skip to content

Prometheus scrape configuration

Prometheus scrape configuration defines the targets that the Prometheus server monitors, how to connect to them, and how to process the collected metrics.^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md]

Configuration Structure

The configuration is typically stored in a file named prometheus.yml.^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md] The file contains a global configuration section and a list of scrape jobs.

Global Settings

Global settings apply to all scrape jobs unless overridden.^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md]

  • scrape_interval: How often to scrape targets (e.g., 15s).^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md]
  • evaluation_interval: How often to evaluate rules (e.g., 15s).^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md]
  • external_labels: Labels to add to any time series or alert when communicating with external systems.

Scrape Jobs

The scrape_configs section defines a list of jobs.^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md] Each job has the following structure:

  • job_name: A unique name for the job.
  • scrape_interval: Optional override of the global interval.
  • scrape_timeout: Optional timeout for the scrape.
  • metrics_path: The HTTP resource path on which to fetch metrics (defaults to /metrics).
  • scheme: The protocol scheme to use (http or https).
  • static_configs: A static list of targets and labels.
  • kubernetes_sd_configs: Configuration for Kubernetes-based service discovery.
  • relabel_configs: A list of relabeling rules to apply to the targets before scraping.

Static Configuration

Static configuration manually specifies the targets to scrape.

scrape_configs:
  - job_name: 'etcd'
    static_configs:
      - targets:
        - '10.4.7.12:2379'
        - '10.4.7.21:2379'
        - '10.4.7.22:2379'

^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md]

Kubernetes Service Discovery

Prometheus supports several roles for discovering targets within a Kubernetes cluster.

Roles

  • endpoints: Discovers targets from the listed endpoints of a service.
  • pod: Discovers all pods and exposes their containers as targets.
  • node: Discovers one target per cluster node.
  • service: Discovers targets for each service.

TLS and Authentication

When scraping Kubernetes resources that use TLS, such as the API server or ETCD, you can specify the configuration within the job.

  - job_name: 'kubernetes-apiservers'
    scheme: https
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    kubernetes_sd_configs:
    - role: endpoints

^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md]

Relabeling

Relabeling is a powerful tool to filter, modify, or add labels to targets before they are scraped.^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md] It uses a list of relabel_configs.

Common Actions

  • keep: Keeps the target if the regex matches.
  • replace: Replaces the value of the target label with a replacement string (default).
  • labelmap: Copies the value of the source label to the target label based on regex.

Example: Pod Annotation Scraping

A common pattern is to use Pod annotations to configure scraping for custom applications.

  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
    - role: pod
    relabel_configs:
    # Keep only pods with the annotation prometheus.io/scrape = true
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
      action: keep
      regex: true
    # Use the annotation prometheus.io/path as the metrics path
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
      action: replace
      target_label: __metrics_path__
      regex: (.+)
    # Use the annotation prometheus.io_port as the port
    - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
      action: replace
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      target_label: __address__

^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md]

Sources