db/ourdb

OurDB

OurDB is a lightweight, efficient key-value database implementation that provides data persistence with history tracking capabilities. This Rust implementation offers a robust and performant solution for applications requiring simple but reliable data storage.

Features

• Simple key-value storage with history tracking
• Data integrity verification using CRC32
• Support for multiple backend files for large datasets
• Lookup table for fast data retrieval
• Incremental mode for auto-generated IDs
• Memory and disk-based lookup tables

Limitations

• Maximum data size per entry is 65,535 bytes (~64KB) due to the 2-byte size field in the record header

Usage

Basic Example

use ourdb::{OurDB, OurDBConfig, OurDBSetArgs};
use std::path::PathBuf;

fn main() -> Result<(), ourdb::Error> {
    // Create a new database
    let config = OurDBConfig {
        path: PathBuf::from("/tmp/ourdb"),
        incremental_mode: true,
        file_size: None, // Use default (500MB)
        keysize: None,   // Use default (4 bytes)
    };
    
    let mut db = OurDB::new(config)?;
    
    // Store data (with auto-generated ID in incremental mode)
    let data = b"Hello, OurDB!";
    let id = db.set(OurDBSetArgs { id: None, data })?;
    println!("Stored data with ID: {}", id);
    
    // Retrieve data
    let retrieved = db.get(id)?;
    println!("Retrieved: {}", String::from_utf8_lossy(&retrieved));
    
    // Update data
    let updated_data = b"Updated data";
    db.set(OurDBSetArgs { id: Some(id), data: updated_data })?;
    
    // Get history (returns most recent first)
    let history = db.get_history(id, 2)?;
    for (i, entry) in history.iter().enumerate() {
        println!("History {}: {}", i, String::from_utf8_lossy(entry));
    }
    
    // Delete data
    db.delete(id)?;
    
    // Close the database
    db.close()?;
    
    Ok(())
}

Key-Value Mode vs Incremental Mode

OurDB supports two operating modes:

1. Key-Value Mode (incremental_mode: false): You must provide IDs explicitly when storing data.
2. Incremental Mode (incremental_mode: true): IDs are auto-generated when not provided.

Configuration Options

• path: Directory for database storage
• incremental_mode: Whether to use auto-increment mode
• file_size: Maximum file size (default: 500MB)
• keysize: Size of lookup table entries (2-6 bytes)
  • 2: For databases with < 65,536 records
  • 3: For databases with < 16,777,216 records
  • 4: For databases with < 4,294,967,296 records (default)
  • 6: For large databases requiring multiple files

Architecture

OurDB consists of three main components:

1. Frontend API: Provides the public interface for database operations
2. Lookup Table: Maps keys to physical locations in the backend storage
3. Backend Storage: Manages the actual data persistence in files

Record Format

Each record in the backend storage includes:

• 2 bytes: Data size
• 4 bytes: CRC32 checksum
• 6 bytes: Previous record location (for history)
• N bytes: Actual data

Documentation

Additional documentation is available in the repository:

• API Reference: Detailed API documentation
• Migration Guide: Guide for migrating from the V implementation
• Architecture: Design and implementation details

Examples

The repository includes several examples to demonstrate OurDB usage:

• basic_usage.rs: Simple operations with OurDB
• advanced_usage.rs: More complex features including both operation modes
• benchmark.rs: Performance benchmarking tool

Run an example with:

cargo run --example basic_usage
cargo run --example advanced_usage
cargo run --example benchmark

Performance

OurDB is designed for efficiency and minimal overhead. The benchmark example can be used to evaluate performance on your specific hardware and workload.

Typical performance metrics on modern hardware:

• Write: 10,000+ operations per second
• Read: 50,000+ operations per second

License

This project is licensed under the MIT License.