Skip to content

Delay Queue Pattern with TTL and dead letter exchange

The Delay queue pattern with TTL and dead letter exchange is a message queueing strategy used to schedule the processing of a message after a specified delay. If the processing fails or times out, the message is automatically routed to a secondary "Dead Letter" queue for error handling or logging^[001-todo-code-copy.md].

Architecture

This pattern typically utilizes RabbitMQ and involves a specific chain of queue exchanges and bindings^[001-todo-code-copy.md]:

  1. Direct Exchange: An entry point where messages are initially routed.
  2. Delay Queue: A durable queue (often a Quorum queue) configured with a Time-To-Live (TTL).^[001-todo-code-copy.md]
  3. Dead Letter Exchange (DLX): An exchange configured to handle messages that expire from the Delay Queue.
  4. Dead Letter Queue (DLQ): The final destination for messages, binding to the DLX.

Configuration

To implement this, the delay queue is configured with specific arguments: * x-dead-letter-exchange: Specifies the exchange to which messages are routed after the TTL expires^[001-todo-code-copy.md]. * x-dead-letter-routing-key: Specifies the routing key to be used when routing to the dead letter exchange^[001-todo-code-copy.md]. * x-message-ttl: Defines the delay time in milliseconds^[001-todo-code-copy.md].

In the provided implementation, the delay queue is defined as: plt.basic.report.delay.q It is bound to a Direct Exchange (plt.basic.report.delay.ex) and points to a Dead Letter Exchange (plt.basic.report.dead.ex)^[001-todo-code-copy.md].

Workflow

  1. Ingestion: A listener (e.g., registerReportFailListener) consumes a message from a primary processing queue^[001-todo-code-copy.md].
  2. Delay: If a task requires a delay or a retry mechanism, the listener sends the message payload to the Delay Queue^[001-todo-code-copy.md].
  3. Expiration: The message sits in the Delay Queue for the duration specified by the x-message-ttl^[001-todo-code-copy.md]. In the source example, this is calculated as REPORT_CACHE_MINUTE * 60 * 1000^[001-todo-code-copy.md].
  4. Routing: Once the TTL expires, the message expires and is automatically moved to the Dead Letter Exchange by the message broker.
  5. Final Processing: A dedicated listener (e.g., handleFailedRecord) attached to the Dead Letter Queue receives the message to perform final actions, such as updating the status to "FAIL"^[001-todo-code-copy.md].

Use Cases

  • Deferred Processing: Executing a task after a waiting period (e.g., a cache validation or time-delay trigger).
  • Retry Management: Implementing a "retry later" logic where a failed task is moved to a delay queue before being attempted again or marked as permanently failed.
  • Timeout Handling: Automatically handling tasks that have not been successfully processed within a specific window.
  • Message Queue
  • [[Dead Letter Queue]]
  • [[Message Exchange Pattern]]
  • [[TTL (Time To Live)]]

Sources

^[001-todo-code-copy.md]