Skip to content

Asynchronous report generation with RabbitMQ

Asynchronous report generation is a design pattern implemented to handle computationally intensive data export tasks (such as CSV or PDF generation) without blocking the main application thread.^[001-todo-code-copy.md] This architecture utilizes RabbitMQ to decouple the request for a report from the actual processing and file generation.

Architecture Overview

The system operates on a producer-consumer model where a report request is created via HTTP, persisted to a database, and then pushed to a Message Queue.^[001-todo-code-copy.md] Background workers consume these messages, generate the documents, and update the record status upon completion.^[001-todo-code-copy.md]

Data Model

The core of the system relies on the ReportDownloadRecord entity, which tracks the lifecycle of a report generation request.^[001-todo-code-copy.md]

Database Entity (ReportDownloadRecordEntity): * id: Unique identifier for the report record.^[001-todo-code-copy.md] * reportType: The specific type of report (e.g., Proxy, Proxy Domain) to be generated.^[001-todo-code-copy.md] * status: The current state of the job.^[001-todo-code-copy.md] * RUNNING: The report is currently being processed.^[001-todo-code-copy.md] * SUCCESS: The file has been successfully generated.^[001-todo-code-copy.md] * FAIL: The generation process failed.^[001-todo-code-copy.md] * searchParamHash: An MD5 hash of the query parameters, used to identify duplicate requests.^[001-todo-code-copy.md] * filePath: The storage path of the generated file.^[001-todo-code-copy.md] * errorMessage: Details logged if the status is FAIL.^[001-todo-code-copy.md]

Workflow

1. Report Creation

When a user requests a report, the ReportManageController handles the POST request.^[001-todo-code-copy.md] The system performs several checks before dispatching the job: * Duplicate Check: The system checks if a report with the same searchParamHash and creatorId already exists in a SUCCESS or RUNNING status within a specific time window.^[001-todo-code-copy.md] If a duplicate is found, the existing record is returned to the user. * Record Initialization: If no duplicate exists, a new ReportDownloadRecordEntity is created with the status RUNNING and inserted into the database.^[001-todo-code-copy.md] * Queue Dispatch: A ReportQryVo object is constructed and sent to a RabbitMQ topic exchange.^[001-todo-code-copy.md]

2. Message Processing (Consumer)

The ReportListener consumes messages from the queue to perform the generation logic.^[001-todo-code-copy.md]

Components: * Topic Exchange & Queue: Requests are routed to a queue (e.g., plt.basic.report.topic.q) based on routing key rules (e.g., plt.basic.report.topic.{type}.q).^[001-todo-code-copy.md] * Domain Service: The listener delegates the business logic to ReportDomainService.^[001-todo-code-copy.md]

3. File Generation and Storage

The actual document creation is handled by specific service implementations implementing the ReportDocumentService interface.^[001-todo-code-copy.md]

Supported Formats: * CSV: Implemented by CSVDocumentServiceImpl. It converts rows of data into comma-separated values and handles byte array generation.^[001-todo-code-copy.md] It also includes logic for combining multiple CSV documents.^[001-todo-code-copy.md] * PDF: Defined in PDFDocumentServiceImpl but currently throws an UnsupportedOperationException.^[001-todo-code-copy.md]

Storage: Generated files are uploaded to Google Cloud Storage (GCS) via GcsFileManageServiceImpl, which manages blobs and file paths.^[001-todo-code-copy.md]

4. Result Handling

Once the document is generated: * The database record is updated with the filePath and status set to SUCCESS.^[001-todo-code-copy.md] * The user can then download the file via a GET /download endpoint using the report ID.^[001-todo-code-copy.md]

Failure Handling and Retries

To manage reliability, the system implements a Dead Letter Queue (DLQ) and delay mechanism.^[001-todo-code-copy.md]

  • Delay Queue: If a report enters a failure state or requires monitoring, it may be sent to a delay queue (plt.basic.report.delay.q) with a configured TTL (Time To Live).^[001-todo-code-copy.md]
  • Dead Letter Queue: Messages that ultimately fail processing are routed to a Dead Letter Exchange (plt.basic.report.dead.ex) and queue (plt.basic.report.dead.q).^[001-todo-code-copy.md] A specific listener updates the report status to FAIL when messages land here.^[001-todo-code-copy.md]

Sources

  • 001-todo-code-copy.md