Skip to content

Python file-based data persistence patterns

Python file-based data persistence patterns refer to the strategies and implementations used to store and retrieve application data on a file system using the Python programming language^[400-devops__09-Scripting-Language__python__introduction__part-3.json__README.md]. These patterns enable long-term data storage by serializing in-memory objects into formats like JSON, allowing data to persist between program executions^[400-devops__09-Scripting-Language__python__introduction__part-3.json__README.md].

Data Interchange Formats

When implementing file-based persistence, applications typically manipulate data as native Python objects in memory but must serialize it into a standard string-based format for storage or transport^[400-devops__09-Scripting-Language__python__introduction__part-3.json__README.md].

A widely adopted standard for these data structures is [[json|JSON]] (JavaScript Object Notation)^[400-devops__09-Scripting-Language__python__introduction__part-3.json__README.md]. JSON is utilized across various domains, including:

  • Configuration Files: Storing application settings^[400-devops__09-Scripting-Language__python__introduction__part-3.json__README.md].
  • Data Storage: Maintaining records in databases or caches^[400-devops__09-Scripting-Language__python__introduction__part-3.json__README.md].
  • Inter-Process Communication: Transferring data between applications via HTTP Web APIs^[400-devops__09-Scripting-Language__python__introduction__part-3.json__README.md].
  • Infrastructure as Code: Managing configuration for DevOps resources^[400-devops__09-Scripting-Language__python__introduction__part-3.json__README.md].

Serialization Patterns

Object Serialization

Serialization is the process of converting an object into a format suitable for saving to a file or transmitting over a network^[400-devops__09-Scripting-Language__python__introduction__part-3.json__README.md]. Python's json library is commonly used for this purpose via the json.dumps() function^[400-devops__09-Scripting-Language__python__introduction__part-3.json__README.md].

A common challenge involves converting custom Class instances into JSON. The json.dumps() function cannot directly serialize custom classes; it typically requires the data to be in a pure dictionary form^[400-devops__09-Scripting-Language__python__introduction__part-3.json__README.md].

To handle this, implementations often utilize the __dict__ property of a class to transform the object into a dictionary^[400-devops__09-Scripting-Language__python__introduction__part-3.json__README.md]. For example, a class instance like Customer("h", "Marcel", "Dempers") can be converted to a dictionary structure: {"customerID": "h", "firstName": "Marcel", "lastName": "Dempers"}^[400-devops__09-Scripting-Language__python__introduction__part-3.json__README.md].

Writing Data

The standard pattern for writing data involves opening a file, serializing the data structure, and writing the content^[400-devops__09-Scripting-Language__python__introduction__part-3.json__README.md].

import json

def updateCustomers(customer):
  with open('customers.json', 'w', newline='') as customerFile:
    customerJSON = json.dumps(customer) 
    customerFile.write(customerJSON)
^[400-devops__09-Scripting-Language__python__introduction__part-3.json__README.md]

Reading Data

The complementary pattern for data retrieval involves checking for the file's existence, reading its content, and deserializing it back into a Python dictionary^[400-devops__09-Scripting-Language__python__introduction__part-3.json__README.md].

import os
import json

def getCustomers():
  if os.path.isfile("customers.json"):
    with open('customers.json', newline='') as customerFile:
      data = customerFile.read()
      customers = json.loads(data) 
      return customers
  else: 
    return {}
^[400-devops__09-Scripting-Language__python__introduction__part-3.json__README.md]

Migration from CSV

Prior patterns, such as those using the [[csv|CSV]] format, involved manually managing fields and iterating over rows to write data^[400-devops__09-Scripting-Language__python__introduction__part-3.json__README.md]. Migrating to a JSON-based pattern simplifies the persistence layer by relying on standard dictionary serialization rather than custom field mapping^[400-devops__09-Scripting-Language__python__introduction__part-3.json__README.md].

Sources