Skip to content

Flask REST API Data Storage Migration

This page documents the process of migrating a [[Flask]] REST API from file-based data storage to a [[Redis]] backend. The migration involves modifying data access functions to interact with a Redis Sentinel cluster while maintaining the existing API contract^[400-devops__09-Scripting-Language__python__introduction__part-5.database.redis__README.md].

Initial State: File-Based Storage

The starting point is a Flask application that manages customer data using a local JSON file (customers.json).^[400-devops__09-Scripting-Language__python__introduction__part-5.database.redis__README.md]

Initial helper functions handle data persistence: * getCustomers(): Reads the entire JSON file and parses it^[400-devops__09-Scripting-Language__python__introduction__part-5.database.redis__README.md]. * getCustomer(id): Retrieves a specific customer dictionary from the loaded data^[400-devops__09-Scripting-Language__python__introduction__part-5.database.redis__README.md]. * updateCustomers(data): Serializes the customer dictionary to JSON and overwrites the file^[400-devops__09-Scripting-Language__python__introduction__part-5.database.redis__README.md].

The application exposes standard REST endpoints: * GET /: Returns all customers^[400-devops__09-Scripting-Language__python__introduction__part-5.database.redis__README.md]. * GET /get/<customerID>: Returns a specific customer or 404 if not found^[400-devops__09-Scripting-Language__python__introduction__part-5.database.redis__README.md]. * POST /set: Validates input JSON (checking for customerID, firstName, lastName) and adds/updates a customer^[400-devops__09-Scripting-Language__python__introduction__part-5.database.redis__README.md].

Redis Configuration and Connectivity

To migrate to Redis, the application connects to a highly available cluster using redis-py and Redis Sentinel^[400-devops__09-Scripting-Language__python__introduction__part-5.database.redis__README.md].

Connection parameters are sourced from environment variables to allow configuration without code changes^[400-devops__09-Scripting-Language__python__introduction__part-5.database.redis__README.md]: * REDIS_SENTINELS: Comma-separated list of host:port pairs (e.g., sentinel-0:5000,sentinel-1:5000). * REDIS_MASTER_NAME: The name of the master group (e.g., mymaster). * REDIS_PASSWORD: Authentication password.

A Sentinel client is initialized, and a master connection is established for write operations^[400-devops__09-Scripting-Language__python__introduction__part-5.database.redis__README.md].

Reliability: Retry Logic

Because Sentinel may promote a new slave to master during failovers, temporary connection errors can occur^[400-devops__09-Scripting-Language__python__introduction__part-5.database.redis__README.md].

A redis_command wrapper function is implemented to handle these transient failures^[400-devops__09-Scripting-Language__python__introduction__part-5.database.redis__README.md]. * It attempts to execute the passed Redis command. * If a ConnectionError or TimeoutError occurs, it waits for a backoffSeconds period and retries, up to max_retries^[400-devops__09-Scripting-Language__python__introduction__part-5.database.redis__README.md].

Refactoring Data Access

The core migration involves replacing the file I/O logic with Redis operations using the wrapper function^[400-devops__09-Scripting-Language__python__introduction__part-5.database.redis__README.md].

Retrieve Single Customer

The getCustomer function is updated to fetch a string value from Redis by customerID.^[400-devops__09-Scripting-Language__python__introduction__part-5.database.redis__README.md] * If the key does not exist (returns None), it returns an empty dictionary {}. * If it exists, it decodes the byte string to UTF-8 and parses the JSON back into a dictionary^[400-devops__09-Scripting-Language__python__introduction__part-5.database.redis__README.md].

Retrieve All Customers

The getCustomers function is rewritten to iterate over keys rather than reading a single file^[400-devops__09-Scripting-Language__python__introduction__part-5.database.redis__README.md]. * It uses scan_iter("*") to find all keys. * It loops through the IDs, calling getCustomer for each to rebuild the full customer list^[400-devops__09-Scripting-Language__python__introduction__part-5.database.redis__README.md].

Update Customer

The write logic is simplified to focus on a single record^[400-devops__09-Scripting-Language__python__introduction__part-5.database.redis__README.md]. * The new updateCustomer function takes a Customer object, serializes it to JSON, and sets the value in Redis using the customerID as the key^[400-devops__09-Scripting-Language__python__introduction__part-5.database.redis__README.md].

Flask Endpoint Updates

Finally, the Flask routes are updated to use the new updateCustomer function^[400-devops__09-Scripting-Language__python__introduction__part-5.database.redis__README.md].

The POST /set endpoint now validates the JSON request, instantiates a Customer object, and calls updateCustomer, persisting the data directly to Redis instead of a local file^[400-devops__09-Scripting-Language__python__introduction__part-5.database.redis__README.md].

Sources

^[400-devops__09-Scripting-Language__python__introduction__part-5.database.redis__README.md]

  • [[Flask]]
  • [[Redis]]
  • [[Sentinel]]
  • [[Data persistence]]
  • [[Docker]]