Skip to content

Redis Hash-Based Query Coordination

Redis Hash-Based Query Coordination is a mechanism used to manage asynchronous, distributed data retrieval tasks. It utilizes a Redis Hash structure to track the completion status of sub-tasks (queries) across multiple listeners, allowing a system to aggregate data from different sources or projects before proceeding to a final processing step, such as generating a CSV file for storage.^[001-TODO__報表下載.md]

Core Workflow

The coordination logic typically revolves around a counter stored within a Redis Hash to manage the lifecycle of a request (e.g., identified by a FileDownloadRecordEntity.id).^[001-TODO__報表下載.md]

1. Task Distribution (Fanout)

A primary Message Queue (MQ) broadcasts a task ID to multiple listeners or consumers via a Fanout Exchange^[001-TODO__報表下載.md]. Each listener is responsible for querying a specific data source or table (e.g., plt_user.vs_user_tag_relation or plt_fund.vs_withdraw)^[001-TODO__報表下載.md].

2. Status Tracking

Listeners write their retrieved data into a shared Redis Hash and increment a counter[1][001-TODO__報表下載.md]. The key for the Redis Hash is typically derived from the task ID, and it has a limited Time-To-Live (TTL), such as 10 minutes^[001-TODO__報表下載.md].

3. Coordination and Aggregation

A monitoring process checks the queryDoneCount field in the Redis Hash^[001-TODO__報表下載.md]. * If queryDoneCount >= Target: The system determines that all necessary sub-queries are complete and proceeds to the next step, such as aggregating the intermediate data and converting it to CSV for upload to Google Cloud Storage^[001-TODO__報表下載.md]. * If Incomplete: The task may be requeued for further processing (often with a retry limit, such as 6 times)^[001-TODO__報表下載.md].

Data Structure

The Redis Hash acts as a temporary, volatile store for intermediate query results and metadata^[001-TODO__報表下載.md].

  • Key Pattern: FileDownloadRecordEntity-{ID}
  • Fields:
    • queryDoneCount: An integer tracking the number of finished listeners.
    • query results: Fields containing the actual data arrays retrieved by specific listeners (e.g., plt_fund.vs_withdraw.user_id: [...]).
  • Message Queue
  • [[Fanout Exchange]]
  • [[Asynchronous Processing]]
  • [[Google Cloud Storage]]

Sources

  • 001-TODO__報表下載.md