Skip to content

mtime-based caching for file-based data

mtime-based caching is a strategy for improving data retrieval performance in systems that read from the file system, where the "modification time" (mtime) attribute of a file serves as the validator for cache freshness^[001-TODO__Hermes_HUD_Web_UI_-_Agent_意识监控仪表盘.md].

In this approach, the system caches the result of reading and processing a file. On subsequent access requests, the system compares the file's current mtime with the timestamp of the cached data. If the mtime has not changed, the system serves the cached data directly, bypassing the expensive I/O and parsing operations. This ensures that the cache is invalidated automatically whenever the file content is modified^[001-TODO__Hermes_HUD_Web_UI_-_Agent_意识监控仪表盘.md].

Implementation Patterns

Decorator Implementation

In Python-based architectures, mtime-based caching is commonly implemented as a function decorator^[001-TODO__Hermes_HUD_Web_UI_-_Agent_意识监控仪表盘.md]. This wraps data collection logic with a caching layer.

For example, in the Hermes HUD architecture, a @cache_with_mtime() decorator is applied to data collector functions^[001-TODO__Hermes_HUD_Web_UI_-Agent_意识监控仪表盘.md]. When a decorated function is called, the logic typically follows these steps: 1. Check if a valid entry exists in the in-memory cache. 2. Retrieve the current mtime from the file system metadata. 3. If the cached entry's timestamp matches the current file mtime, return the cached data. 4. If they differ (or no cache exists), execute the file read/parse operation, update the cache, and return the new data^[001-TODO__Hermes_HUD_Web_UI-_Agent_意识监控仪表盘.md].

Integration with File Watchers

This caching mechanism is often paired with active file system monitoring to support real-time updates^[001-TODO__Hermes_HUD_Web_UI_-_Agent_意识监控仪表盘.md].

A typical workflow involves: * Detection: A file watcher (e.g., utilizing watchfiles) detects modifications to the target data directory^[001-TODO__Hermes_HUD_Web_UI_-Agent_意识监控仪表盘.md]. * Invalidation: The watcher triggers an event that clears the specific cache entry. * Broadcast: The system notifies connected clients (e.g., via WebSocket) that new data is available^[001-TODO__Hermes_HUD_Web_UI-_Agent_意识监控仪表盘.md]. * Revalidation: Clients request the new data, and the backend serves it by utilizing the mtime-checking logic described above.

Advantages

  • Reduced I/O Overhead: By avoiding unnecessary disk reads for unchanged files, the system significantly lowers input/output load^[001-TODO__Hermes_HUD_Web_UI_-_Agent_意识监控仪表盘.md].
  • Lower CPU Usage: Expensive parsing operations (e.g., deserializing JSON or YAML) are skipped if the file has not been modified^[001-TODO__Hermes_HUD_Web_UI_-_Agent_意识监控仪表盘.md].
  • Automatic Consistency: Unlike time-to-live (TTL) caching which might serve stale data until expiration, mtime-based caching guarantees freshness immediately upon a file save operation^[001-TODO__Hermes_HUD_Web_UI_-_Agent_意识监控仪表盘.md].
  • [[Server-Sent Events]]
  • [[WebSocket]]
  • [[Memoization]]

Sources

  • 001-TODO__Hermes_HUD_Web_UI_-_Agent_意识监控仪表盘.md