Skip to content

Fluentd log streaming with Docker Compose

Fluentd log streaming with Docker Compose refers to a logging architecture where the Fluentd log collector is configured to aggregate logs from other containers within a Docker Compose environment. This setup typically involves forwarding local container logs to a remote service, such as a hosted Datadog agent.^[400-devops__04-CI-CD-Pipelines__devops-cd__monitoring__README.md]

Architecture Overview

In this configuration, the application or logging driver defines Fluentd as the mechanism to handle log output.^[400-devops__04-CI-CD-Pipelines__devops-cd__monitoring__README.md] Instead of processing logs locally, the container forwards the log data to a specified Datadog agent endpoint.^[400-devops__04-CI-CD-Pipelines__devops-cd__monitoring__README.md]

Environment Variables

Configuration is managed via environment variables that define the connection details:

  • DD_AGENT_HOST: Specifies the hostname or IP address of the Datadog agent.^[400-devops__04-CI-CD-Pipelines__devops-cd__monitoring__README.md]
  • DD_TRACE_AGENT_PORT: Specifies the port number for the trace agent.^[400-devops__04-CI-CD-Pipelines__devops-cd__monitoring__README.md]

Docker Compose Configuration

Docker-from-Docker-Compose Pattern

A common pattern in development is to run the Fluentd container using a "docker-from-docker-compose" approach^[400-devops__04-CI-CD-Pipelines__devops-cd__monitoring__README.md]. In this setup, the docker-compose.yml file configures the logging driver for the service to use fluentd as the log target^[400-devops__04-CI-CD-Pipelines__devops-cd__monitoring__README.md]. The configuration typically points to a Fluentd instance that may be running locally or within a linked container.

Log Management

When services are running, logs can be monitored in real-time using standard Docker Compose commands^[400-devops__04-CI-CD-Pipelines__devops-cd__monitoring__README.md]. For example, to follow the log output of the Fluentd container specifically, one would execute:

bash docker-compose logs -f fluentd^[400-devops__04-CI-CD-Pipelines__devops-cd__monitoring__README.md]

Sources

^[400-devops__04-CI-CD-Pipelines__devops-cd__monitoring__README.md]