Mastering The Persistence Layer For Snapshot Management

Introduction to the Persistence Layer

In the realm of data management and version control, the persistence layer is the unsung hero, ensuring that your valuable data, particularly snapshots, remains intact even after unexpected server restarts. This article delves into the intricacies of a robust persistence layer, focusing on a Hybrid BlobStore approach with Fs default storage. Our goal is to achieve two critical objectives: first, to make snapshots and, optionally, leases survive server restarts, and second, to maintain strict determinism throughout the process. We’ll explore how a combination of blob-on-disk for raw byte storage and SQLite for metadata management forms the backbone of this system. For those seeking enhanced portability or working with smaller repositories, an opt-in blob-in-DB mode is also discussed. This detailed examination will guide you through the design decisions, data layouts, API considerations, and garbage collection strategies necessary to build a reliable and deterministic persistence layer.

The Core Goal: Surviving Restarts with Determinism

The primary goal of this persistence layer is to ensure that snapshots and optionally leases are not lost when the MCP server restarts. This means that all the essential metadata and the actual data blobs that constitute a snapshot must be stored in a way that can be reliably reloaded. Crucially, this persistence must be achieved without sacrificing strict determinism. In essence, if you create a snapshot today, and the server restarts, the exact same snapshot should be retrievable tomorrow, and its contents should be identical to what was originally stored, regardless of any internal storage mechanisms or server restarts. This strict determinism is vital for debugging, auditing, and ensuring the integrity of versioned data. The default storage strategy employs a hybrid model: blob-on-disk (FsBlobStore) is used for storing the actual bytes of the blobs, leveraging the filesystem for efficient storage and retrieval. Complementing this is SQLite, which serves as the central repository for all metadata, including snapshot details, manifest entries, and lease information. For scenarios where extreme portability is a requirement, such as sharing a small repository as a single file or for demonstration purposes, an optional blob-in-DB mode is available. This mode stores the blob data directly within SQLite's BLOB columns, offering a self-contained solution at the potential cost of performance for very large datasets. Both storage models are designed to adhere to the same BlobStore trait API, ensuring that the choice of backend storage is largely transparent to the rest of the system. The put(hash, bytes), get(hash), and has(hash) methods provide a consistent interface for interacting with blob data, regardless of whether it resides on the filesystem or within a database.

Defining the Scope: What's In and What's Out

When designing any complex system, clearly defining the scope is paramount to avoid feature creep and ensure timely delivery. For this persistence layer, we have a well-defined set of features that are in scope. The most fundamental is the Persistent BlobStore for snapshot file bytes. This addresses the core requirement of storing the actual content of your snapshots in a content-addressed manner. This means that each piece of data is identified by its hash, ensuring that identical data is stored only once. Alongside this, we have the Persistent Snapshot/Manifest metadata, which will be stored in SQLite. This includes all the information that defines a snapshot, such as its origin, base, and the list of files it contains. Furthermore, Persistent LeaseStore metadata is also in scope, with a specific emphasis on clear restart semantics. Leases, which often represent temporary locks or ownership of data, need to be handled carefully to ensure that the system behaves predictably after a restart. A crucial aspect of this persistence is ensuring deterministic listing and retrieval behaviors that match spec schemas. This means that when you ask for a list of files in a snapshot, or retrieve a specific file, the results should always be the same for a given snapshot, adhering strictly to predefined schema expectations. Finally, a robust GC strategy to keep disk usage bounded is included. Without effective garbage collection, storage space would quickly be exhausted by old or unreferenced data. This strategy ensures that only necessary data is retained.

On the flip side, several features are explicitly out of scope (for this plan) to maintain focus. UI-specific workflows (Tauri screens) are not part of this core persistence layer; the UI will interact with the persistence layer through its defined APIs. Similarly, remote syncing or multi-machine sharing are advanced features that require a different set of considerations and are beyond the current scope. Encryption at rest is another feature that, while valuable, can be added later as a distinct enhancement once the fundamental persistence mechanisms are in place. By clearly delineating these boundaries, we can concentrate our efforts on building a solid and reliable foundation for snapshot persistence.

Key Design Decisions for Flexibility and Scalability

Several crucial design decisions shape the architecture of this persistence layer, with a focus on balancing performance, portability, and maintainability. The cornerstone of our approach is the Storage model: hybrid, content-addressed. This means we don't force a single storage solution but offer flexibility based on use cases. The default configuration utilizes a combination of blob-on-disk (FsBlobStore) for storing the actual byte content of files and SQLite for managing all the associated metadata. This default setup is optimized for performance and scalability, especially for larger repositories where direct filesystem access for blobs can be significantly faster than database lookups.

Complementing the default is the optional blob-in-DB (SqliteBlobStore) mode. In this configuration, the byte content of blobs is stored directly within SQLite's BLOB columns. This mode is particularly useful for scenarios requiring extreme portability, such as creating self-contained backup files or for use in environments where a simple, single-file database is highly advantageous, like in demonstration setups or for very small repositories. Despite the difference in where the bytes are stored, both FsBlobStore and SqliteBlobStore implement the same BlobStore trait API. This consistent API, featuring methods like put(hash, bytes), get(hash), and has(hash), ensures that the rest of the system interacts with blob storage in a uniform manner, abstracting away the underlying storage details.

The rationale behind this hybrid approach is multifaceted. The blob-on-disk strategy generally scales better and avoids SQLite WAL bloat for large files, which can become a performance bottleneck in a purely database-centric approach. Conversely, the blob-in-DB mode truly enables “single file portability”, making it incredibly easy to share or back up entire datasets. Critically, the determinism is unaffected by storage location as long as the fundamental contract – that a given hash precisely maps to a specific set of bytes, and that write operations are atomic – is maintained. This design decision provides a robust and adaptable foundation for managing snapshot data effectively.

Understanding the Data Layout: Filesystem and SQLite

A clear understanding of the data layout is essential for managing and debugging the persistence layer. We employ a two-pronged approach, leveraging both the filesystem for blob storage and SQLite for metadata. The Filesystem blob layout (default) is designed for efficient content-addressable storage. The root directory for blobs is located within the <cortex_data_dir>/blobs/ path. Within this, blobs are organized using a hierarchical structure for better performance and to avoid excessively large directories. The typical path follows the pattern: blobs/<algo>/<prefix>/<hash>. For instance, a blob identified by a SHA256 hash might be found at blobs/sha256/ab/ab12...ff, where ab represents the first two hexadecimal characters of the hash, serving as a directory prefix. The hash itself forms the filename, containing the raw bytes of the blob. Importantly, the system supports optional compression (like zstd), which can be configured via a flag in the SQLite metadata. When writing files to the filesystem, an atomic write requirement is strictly enforced. This is typically achieved by writing the data to a temporary file in the same directory (e.g., .../<hash>.tmp.<random>) and then performing an atomic rename operation to move it to its final destination (.../<hash>). This ensures that the data is either fully written or not written at all, preventing corruption. The choice of fsync policy is crucial and should be documented based on the desired durability guarantees.

Complementing the filesystem storage is the SQLite database layout, which is always present. This database, typically located at <cortex_data_dir>/store.sqlite, acts as the central index and metadata repository. It holds information critical for managing snapshots and their associated data. The SQLite database contains tables for: snapshots, detailing each snapshot's origin and metadata; manifests and entries, listing the files within each snapshot; a blob index, mapping hashes to their metadata and storage location; and leases (which can be optionally persisted). For enhanced query performance, especially for operations like listing, grepping, or diffing within snapshots, an optional path index can also be maintained within SQLite. This dual-storage approach ensures that while large binary data is efficiently handled by the filesystem, all the relational metadata and indexing are managed robustly by SQLite.

SQLite Minimal Schema: Organizing Metadata

To support the persistence of snapshots and related data, a well-defined SQLite Minimal Schema is proposed. This schema is designed to store all necessary metadata efficiently and in a way that supports deterministic operations. The schema consists of several key tables:

Table: blobs This table tracks all known blobs, irrespective of whether their bytes are stored on the filesystem (fs) or within the database (db).

• hash TEXT PRIMARY KEY: The content hash of the blob (e.g., sha256:<hex>). This is the unique identifier.
• size_bytes INTEGER NOT NULL: The size of the blob in bytes.
• compression TEXT NOT NULL: Indicates the compression type used (e.g., none, zstd). This is crucial for correct decompression.
• storage TEXT NOT NULL: Specifies the storage backend (fs or db).
• refcount INTEGER NOT NULL DEFAULT 0: A reference counter to manage garbage collection. It tracks how many snapshots or other entities refer to this blob.
• created_at INTEGER NULL: An optional timestamp, not used for determinism, but can be helpful for auditing.

Table: snapshots This table stores metadata for each individual snapshot.

• snapshot_id TEXT PRIMARY KEY: The unique identifier for the snapshot, typically its hash.
• repo_root TEXT NOT NULL: The root path of the repository at the time the snapshot was taken.
• head_sha TEXT NOT NULL: The hash of the commit or head of the snapshot.
• base_ref TEXT NULL, base_sha TEXT NULL: Information about the base snapshot, if any, used for delta compression or historical tracking.
• branch TEXT NULL: The branch associated with the snapshot, if applicable.
• options_hash TEXT NULL: A hash representing the configuration options used for this snapshot.
• fingerprint_json TEXT NOT NULL: A canonical JSON string representing the snapshot's unique fingerprint, including metadata and potentially tree structure. This is key for determinism.
• manifest_hash TEXT NOT NULL: The hash of the manifest data (either the canonical manifest bytes or its corresponding blob hash).
• manifest_bytes BLOB NULL: Optionally, the canonical manifest bytes can be stored directly here for quicker access, though it's recommended to store them as a blob referenced by manifest_hash.
• created_at INTEGER NULL: An optional timestamp, not used for determinism.

Table: manifest_entries This table stores the contents of each snapshot's manifest as individual rows, enabling efficient listing and traversal.

• snapshot_id TEXT NOT NULL: Foreign key referencing the snapshots table.
• path TEXT NOT NULL: The repository-relative path of the file or directory entry, normalized to ensure consistency.
• blob_hash TEXT NOT NULL: The hash of the blob containing the file's content (foreign key to blobs.hash).
• size_bytes INTEGER NOT NULL: The size of the blob in bytes.
• PRIMARY KEY (snapshot_id, path): A composite primary key ensuring uniqueness for each path within a snapshot.

Table: leases (optional persistence) This table stores information about leases, which can be persisted or treated as ephemeral.

• lease_id TEXT PRIMARY KEY: The unique identifier for the lease.
• repo_root TEXT NOT NULL: The root of the repository the lease applies to.
• fingerprint_json TEXT NOT NULL: The canonical JSON string of the fingerprint associated with the lease.
• touched_json TEXT NOT NULL: A sorted JSON array of paths affected by the lease.
• issued_at INTEGER NULL: Optional timestamp when the lease was issued.
• expires_at INTEGER NULL: Optional timestamp for lease expiration (TTL).

This schema provides a robust foundation for managing snapshot data, ensuring that all critical information is stored and accessible in a deterministic and efficient manner.

API and Traits: Defining Interactions

To ensure a clean and modular design, the interaction with the persistence layer is defined through a set of API and Traits. These traits abstract the underlying storage mechanisms, allowing different implementations (like FsBlobStore and SqliteBlobStore) to be plugged in seamlessly.

T1. BlobStore trait This trait defines the fundamental operations for interacting with blob data, focusing on content-addressed storage. The core methods are:

• put(hash, bytes, compression): Stores a given byte slice, identified by its hash, and returns the stored blob's metadata (hash and size). It also accepts a compression type.
• get(hash): Retrieves the raw bytes associated with a given hash. If the blob was stored compressed, this method is responsible for decompression.
• has(hash): Checks for the existence of a blob identified by its hash.

This trait ensures that regardless of whether blobs are stored on the filesystem or in a database, the interface for putting, getting, and checking for blobs remains consistent.

T2. SnapshotStore (metadata) This trait defines the API for managing snapshot metadata and entries within the SQLite database.

• put_snapshot(snapshot_id, snapshot_meta, manifest_entries): Persists a new snapshot, including its unique ID, metadata, and all its manifest entries.
• get_snapshot(snapshot_id): Retrieves the metadata and manifest entries for a given snapshot ID.
• list_snapshot_entries(snapshot_id, prefix_path, depth, limit): Provides functionality to list entries within a snapshot, allowing filtering by path prefix, depth, and limiting the number of results. This operation must be deterministic.
• get_blob_hash(snapshot_id, path): Given a snapshot ID and a relative path, returns the blob_hash associated with that path within the snapshot.

T3. LeaseStore Handling leases requires careful consideration of their persistence across restarts. Two valid approaches are offered, allowing the system to choose based on requirements:

• Option A (simple): Leases are ephemeral, not restored after restart. In this approach, lease information is not persisted in the database. If the server restarts, any existing leases are considered invalid. The system would respond with STALE_LEASE or NOT_FOUND for operations that rely on a previously existing lease. While simpler to implement, this might lead to a less seamless user experience for long-running operations.
• Option B (full): Leases persist with fingerprint validation. This option involves persisting lease records in the SQLite database. Upon server restart, these lease records are restored. Crucially, every request involving a lease must perform a fingerprint validation to ensure that the lease is still valid for the current state of the system. This provides a more robust and user-friendly experience for long-running sessions, but adds complexity to the implementation.

The choice between Option A and Option B depends on the specific requirements of the application. Given the potential goal of UI goals, Option A is acceptable and simpler. However, Option B is nicer for long-running sessions and offers greater resilience.

Effective Garbage Collection Strategy

Managing storage space effectively is crucial for any persistent system. The GC Strategy outlined here ensures that disk usage is kept bounded by intelligently removing unreferenced data. We propose two main strategies: Reference Counting (preferred) and Mark-and-Sweep (fallback).

G1. Reference counting (preferred) This is the generally preferred method due to its efficiency and precise tracking of resource usage. The core idea is to maintain a reference count for each blob. The reference count indicates how many active snapshots or other entities are currently using that specific blob.

• On snapshot create: For every blob included in the new snapshot's manifest, the system must locate its corresponding row in the blobs table (creating it if it doesn't exist) and increment its refcount. This signifies that one more entity now depends on this blob.
• On snapshot delete: When a snapshot is deleted, the system must iterate through all the blobs that constituted that snapshot and decrement their refcount. If, after decrementing, a blob's refcount reaches zero, it means no active snapshot or entity refers to it anymore, and its data (either in the filesystem or the database) can be safely removed.

This strategy requires mechanisms to trigger snapshot deletion or to implement an internal retention policy for automatic cleanup. A snapshot.delete tool or a background cleanup process would be responsible for identifying and removing old snapshots, thereby driving the refcount decrements and subsequent blob garbage collection.

G2. Mark-and-sweep (fallback) This is a more traditional garbage collection approach, often simpler to implement initially, especially if snapshot deletion semantics are not yet finalized or are complex.

• Periodically: The system initiates a garbage collection cycle. This involves first marking all blobs that are reachable from currently active snapshots. This is typically done by traversing the manifests of all active snapshots and collecting the hashes of all blobs they reference.
• Delete unmarked blobs: After the marking phase, the system identifies all blobs that were not marked during the traversal. These unmarked blobs are considered garbage and can be safely deleted from both the filesystem (if FsBlobStore is used) and the blobs table in the SQLite database.

While mark-and-sweep can be easier to implement in the short term, it is generally slower than reference counting because it requires a full traversal of all active data. Reference counting, on the other hand, performs updates incrementally as snapshots are created or deleted. For a production system, reference counting is usually the more performant and scalable choice.

Guarantees: Ensuring Strict Determinism

The critical requirement for this persistence layer is that it must not introduce any nondeterminism. This means that the outputs and behaviors of the system should be exactly the same every time, regardless of the internal state, storage order, or server restarts. Several guarantees are put in place to ensure this:

1. Lexicographical Path Order: All manifest entries, whether stored in SQLite or retrieved, are always stored and retrieved in lexicographic path order. This is crucial because the order in which files are listed or processed can affect hashing and comparison operations. By enforcing a consistent, alphabetical ordering of paths, we eliminate variability that could arise from different internal data structures or database retrieval orders.

2. Consistent Snapshot Derivation: The process for deriving a snapshot's identifier remains consistent. The snapshot's hash is derived from a combination of its canonical fingerprint JSON (which includes essential metadata and configuration) and its manifest bytes. The newline character is used as a separator. This strict formula ensures that any change in fingerprint or manifest content directly results in a new, different hash, and conversely, identical content will always produce the same hash. No timestamps or DB insertion order are ever used in this derivation process, as these are inherently non-deterministic and can vary between runs or environments.

3. Storage Location Invariance: A fundamental guarantee is that the blob location (filesystem vs. database) never affects the bytes returned. The BlobStore trait ensures that whether a blob's data is stored in a file on disk or within a SQLite BLOB column, the get(hash) operation will always return the exact same byte content for a given hash. This abstraction is key to supporting the hybrid storage model without compromising data integrity or determinism.

These guarantees collectively ensure that the persistence layer functions as a reliable and predictable component of the system. Users can trust that snapshots will be consistent, retrievable, and reproducible, forming a solid foundation for version control and data management.

Phased Implementation Steps

To manage the complexity of building this persistence layer, a phased Implementation Steps approach is recommended. This breaks down the work into manageable chunks, allowing for iterative development and testing.

Phase 1: Foundations This phase focuses on setting up the basic configuration and core storage components.

1. Add StorageConfig: Introduce a configuration structure that includes essential settings like data_dir (the main directory for storing data), blob_backend (to specify fs or db), and compression (none or zstd).
2. Implement SQLite migrations and schema: Set up the necessary tools and logic for managing database schema versions and applying migrations. This ensures the database schema can evolve safely.
3. Implement FsBlobStore: Develop the FsBlobStore component. This includes handling atomic writes to the filesystem, supporting optional zstd compression, and implementing the core has/get/put methods defined by the BlobStore trait.
4. Implement SqliteBlobStore (optional, behind feature flag): Develop the alternative SqliteBlobStore. This component will store blob bytes in SQLite's blobs.bytes BLOB column. It must also adhere to the BlobStore trait behavior. This is initially behind a feature flag to allow development and testing without impacting the default behavior.

Phase 2: Persist Snapshots With the foundational storage components in place, this phase focuses on making snapshots persistent. 5. Persist snapshot creation: Implement the logic to save snapshots. This involves writing the actual blob data using the chosen BlobStore, persisting the manifest entries in the manifest_entries SQLite table, and saving the snapshot metadata in the snapshots table. 6. Update snapshot-mode reads: Modify existing read operations that are in snapshot mode to use the new persistent storage. This means implementing snapshot.list(mode=snapshot) by querying the manifest_entries table and snapshot.file(mode=snapshot) by looking up the path in manifest_entries to get the blob_hash, and then retrieving the bytes from the BlobStore.

Phase 3: Persist Leases (choose A or B) This phase addresses the persistence of leases, requiring a decision on the chosen strategy. 7. Option A Implementation: If Option A (ephemeral leases) is chosen, the implementation involves ensuring that upon server restart, all lease records are invalidated. The system should simply respond with STALE_LEASE or NOT_FOUND when a stale lease is referenced. 8. Option B Implementation: If Option B (full persistence) is chosen, implement the logic to persist lease records into the SQLite database. This also requires implementing the logic to restore lease records upon restart and, critically, to validate the fingerprint against the lease on every relevant operation.

Phase 4: Add GC Hooks Implementing garbage collection is vital for managing storage. 9. Add refcount maintenance: Integrate logic to update the refcount in the blobs table. This means incrementing the count when a blob is added to a snapshot and decrementing it when a snapshot is deleted. 10. Add snapshot deletion or retention policy hooks: Create the necessary interfaces or background processes to handle snapshot deletion. This is crucial for triggering the decrementing of reference counts. 11. Implement GC job: Develop the background garbage collection job. This job will be responsible for the safe deletion of unreferenced blobs from both the filesystem (if applicable) and the SQLite database, based on the refcount.

Phase 5: Verification The final phase is dedicated to thorough testing and validation. 12. Extend golden tests: Enhance existing golden tests to cover persistence scenarios. This includes creating a snapshot, restarting the server, and then verifying that reading the snapshot list and individual files still functions correctly. 13. Add corruption edge tests: Introduce tests that simulate data corruption, such as a missing blob file. The system should respond with a structured error (e.g., NOT_FOUND or INTERNAL, as specified) rather than crashing or producing incorrect results. 14. Ensure schema-lint and runtime cache-hint enforcement: Verify that existing checks, such as schema-lint and runtime enforcement of cache hints, continue to pass with the new persistence layer.

This phased approach ensures a structured and robust development process, leading to a reliable and deterministic persistence layer.

Deliverables: The Tangible Outcomes

Upon successful completion of the implementation phases, several key Deliverables will mark the culmination of this project. These tangible outcomes represent the functional components and verified capabilities of the new persistence layer.

• StorageConfig and Persistent Storage Directory Layout: A fully defined StorageConfig structure will be available, allowing users to specify data_dir, blob_backend, and compression settings. This will be accompanied by a clearly documented and implemented persistent storage directory layout, defining where store.sqlite and the blobs/ directory (if FsBlobStore is used) reside.

• FsBlobStore and SqliteBlobStore: The FsBlobStore will be implemented as the default backend, providing efficient, file-system-based blob storage. The SqliteBlobStore will also be available as an optional backend, activated via a feature flag, offering single-file portability by storing blob data within the SQLite database.

• SQLite Schema and Migrations: A finalized SQLite schema, comprising tables for snapshots, manifest entries, the blob index, and optional leases, will be in place. This schema will be accompanied by a robust set of migrations that allow for safe schema evolution and database setup.

• Garbage Collection Strategy: The chosen GC strategy will be fully implemented. The reference counting mechanism is the preferred outcome, ensuring efficient management of blob lifecycles. This includes the logic for incrementing and decrementing reference counts and a mechanism for safely deleting unreferenced blobs.

• Verified Persistence and Deterministic Outputs: Comprehensive Golden Tests will prove the core functionality. These tests will specifically verify that the persistence layer correctly handles server restarts (snapshots remain available) and that all operations adhere to the deterministic outputs required by the system's specification. Edge cases, such as data corruption, will also be tested, ensuring graceful error handling.

• Concrete Migration Files: The proposed SQLite tables will be translated into concrete migration files. These files will be executable and versioned, ensuring that the database schema can be reliably managed across different deployments and updates.

• Rust Module Layout and Error Mapping: A clean Rust module layout will be established, clearly separating traits, implementations, and related utilities. Furthermore, all error types generated by the persistence layer will be mapped to the corresponding spec/schemas/common.schema.json error codes, ensuring consistent error reporting across the system.

These deliverables collectively ensure that the persistence layer is not only functional but also robust, efficient, and seamlessly integrated into the broader system, meeting all the specified requirements for snapshot management.

Conclusion: A Resilient and Deterministic Future

By meticulously designing and implementing the persistence layer with a hybrid BlobStore approach and SQLite for metadata, we achieve the critical goals of surviving server restarts while maintaining strict determinism. The flexible FsBlobStore and SqliteBlobStore backends cater to different needs, from performance-intensive operations to single-file portability. The robust SQLite schema, coupled with atomic file operations and carefully managed reference counting for garbage collection, ensures data integrity and efficient storage utilization. The guarantees of lexicographical ordering and consistent snapshot derivation mean that users can rely on the reproducibility of their data, a cornerstone of effective version control. The phased implementation plan provides a clear roadmap for development and verification, culminating in a set of comprehensive deliverables that validate the system's resilience and performance. This robust persistence layer lays a solid foundation for future enhancements and ensures the long-term reliability of snapshot management. For deeper insights into related data management and version control concepts, exploring resources like Git Documentation can provide valuable context on how other systems handle similar challenges.