Skip to content

logging-agent (DaemonSet) pattern

The logging-agent (DaemonSet) pattern is a cluster-level logging architecture designed to ensure logs are persisted regardless of the lifecycle of containers, Pods, or Nodes^[Kubernetes相关生态.md#L36-L37]. In this approach, a specialized logging agent is deployed on every node within the cluster to forward container logs to a backend storage system^[Kubernetes相关生态.md#L38-L41].

Architecture

The core component of this pattern is the logging agent, typically deployed as a Kubernetes [[DaemonSet]]^[Kubernetes相关生态.md#L39-L40]. This agent runs on each node and mounts the host's container log directory (such as /var/log/containers)^[Kubernetes相关生态.md#L39-L40]. It reads the log files generated by the applications running on that node and forwards them to a centralized backend, such as an Elasticsearch instance^[Kubernetes相关生态.md#L40-L41].

Key Characteristics

Cluster-Level Logging

This pattern implements cluster-level-logging, meaning the logging system functions independently of the applications and the underlying infrastructure^[Kubernetes相关生态.md#L36-L38]. This decoupling ensures that logs are preserved and accessible even if specific containers fail, Pods are deleted, or Nodes experience downtime^[Kubernetes相关生态.md#L36-L38].

Log Source Requirement

For the standard DaemonSet pattern to function effectively, applications are typically required to log to stdout and stderr^[Kubernetes相关生态.md#L44-L45]. The container runtime (e.g., Docker) and a log handler (like kubelet) then write these streams to log files on the host node, which the agent accesses^[Kubernetes相关生态.md#L44-L46].

Advantages and Disadvantages

Advantages

  • Resource Efficiency: A single logging agent instance runs per node, rather than one per Pod or container^[Kubernetes相关生态.md#L42-L43].
  • Non-Invasive: The pattern requires no changes to the application code or Pod specifications, as it relies on standard output streams and host volume mounts^[Kubernetes相关生态.md#L42-L43].

Disadvantages

  • Standard Output Dependency: The pattern relies on applications logging to stdout and stderr^[Kubernetes相关生态.md#L44-L45]. If an application generates a high volume of logs, it may exhaust the system's default log quotas configured for the container runtime, potentially causing logs to be dropped^[Kubernetes相关生态.md#L46-L48].
  • Mitigation Complexity: Preventing log loss due to quota limits requires either increasing system quotas or mounting persistent storage volumes for the containers^[Kubernetes相关生态.md#L49-L50].

Alternative patterns exist to address specific limitations of the standard DaemonSet approach, particularly when applications must write to arbitrary log files:

  • Sidecar with Stream Redirection: A sidecar container runs alongside the application, reads specific log files, and redirects them to its own stdout/stderr to be picked up by the host node agent^[Kubernetes相关生态.md#L52-L55].
  • Sidecar with Direct Forwarding: A sidecar container runs its own logging agent (e.g., fluentd) to send log files directly to remote storage, bypassing the node-level agent^[Kubernetes相关生态.md#L61-L63].

Sources

^[Kubernetes相关生态.md]