Skip to content

cluster-level-logging

Cluster-level-logging is a logging architecture design in Kubernetes where the log processing system is completely independent of the lifecycles of containers, Pods, and Nodes.^[原理及源码解析__Kubernetes相关生态.md] This decoupling ensures that application logs remain accessible and intact even if specific containers fail, Pods are deleted, or Nodes experience downtime.^[原理及源码解析__Kubernetes相关生态.md]

Architecture Models

There are three primary architectural patterns for implementing cluster-level-logging, each utilizing different deployment strategies for the logging agent.

Node-based Logging Agent

In this model, a logging agent is deployed on every Node within the cluster, typically as a [[DaemonSet]].^[原理及源码解析__Kubernetes相关生态.md] The agent mounts the host's container log directory and forwards log files to a backend storage system.^[原理及源码解析__Kubernetes相关生态.md]

  • Advantages: This approach is resource-efficient, requiring only a single agent per node, and is non-intrusive to applications or Pods.^[原理及源码解析__Kubernetes相关生态.md]
  • Requirements: Applications must output logs to stdout and stderr.^[原理及源码解析__Kubernetes相关生态.md]
  • Limitations: High-volume logging can exhaust system log quotas, leading to data loss. Mitigations include increasing quotas or mounting storage for containers.^[原理及源码解析__Kubernetes相关生态.md]

Sidecar with Stream Redirection

This pattern is used when applications write logs to specific files rather than standard output. A sidecar container runs alongside the application container, reads these log files, and redirects them to its own stdout and stderr streams.^[原理及源码解析__Kubernetes相关生态.md] This allows the standard node-based logging agent to capture the logs as if they came directly from the application.

  • Limitations: This results in duplicate log files on the host (the original application file and the sidecar's JSON file), which wastes disk space. It is generally recommended only when the application cannot be modified.^[原理及源码解析__Kubernetes相关生态.md]

Sidecar with Remote Logging Agent

In this variation, a sidecar container runs a dedicated logging agent that sends application logs directly to a remote backend storage (e.g., using fluentd to send data to Elasticsearch), rather than piping them to standard output.^[原理及源码解析__Kubernetes相关生态.md]

  • Advantages: It allows logs to be output to fixed files and simplifies deployment.
  • Limitations: The sidecar may consume significant resources, potentially impacting the performance of the application container. Additionally, logs are not visible via kubectl logs.^[原理及源码解析__Kubernetes相关生态.md]

Operational Considerations

A critical aspect of cluster-level-logging is log rotation and storage management.^[原理及源码解析__Kubernetes相关生态.md] Regardless of the architecture used, log files must be promptly cleaned from the host, or log directories must be mounted to high-capacity remote storage. Failure to manage log storage can result in the host's main disk partition filling up, potentially causing the entire system to crash.^[原理及源码解析__Kubernetes相关生态.md]

Sources

  • 原理及源码解析__Kubernetes相关生态.md