db/herodb_ourdb_migration_plan.md

Migration Plan: Restructuring herodb to Use ourdb as Backend

This document outlines the plan to restructure herodb to use ourdb as the backend, completely removing all sled references and better aligning with ourdb's design patterns.

Overview

graph TD
    A[Current herodb with sled] --> B[Define new core traits]
    B --> C[Implement ourdb backend]
    C --> D[Create new DB manager]
    D --> E[Implement transaction system]
    E --> F[Update model implementations]
    F --> G[Final restructured herodb with ourdb]

New Architecture

classDiagram
    class Model {
        +get_id() u32
        +db_prefix() &'static str
    }
    class Storable {
        +serialize() Result<Vec<u8>>
        +deserialize() Result<Self>
    }
    class DB {
        -path: PathBuf
        -type_map: HashMap<TypeId, Arc<dyn DbOperations>>
        -transaction: Arc<RwLock<Option<TransactionState>>>
        +new(config: DbConfig) Result<Self>
        +begin_transaction() Result<()>
        +commit_transaction() Result<()>
        +rollback_transaction() Result<()>
        +set<T: Model>(model: &T) Result<()>
        +get<T: Model>(id: u32) Result<T>
        +delete<T: Model>(id: u32) Result<()>
        +list<T: Model>() Result<Vec<T>>
        +register<T: Model>() Result<()>
        +get_history<T: Model>(id: u32, depth: u8) Result<Vec<T>>
    }
    class DbOperations {
        <<interface>>
        +delete(id: u32) Result<()>
        +get(id: u32) Result<Box<dyn Any>>
        +list() Result<Box<dyn Any>>
        +insert(model: &dyn Any) Result<()>
        +get_history(id: u32, depth: u8) Result<Vec<Box<dyn Any>>>
    }
    class OurDbStore~T~ {
        -db: OurDB
        -model_type: PhantomData<T>
        +new(config: OurDBConfig) Result<Self>
        +insert(model: &T) Result<()>
        +get(id: u32) Result<T>
        +delete(id: u32) Result<()>
        +list() Result<Vec<T>>
        +get_history(id: u32, depth: u8) Result<Vec<T>>
    }
    
    Model --|> Storable
    OurDbStore ..|> DbOperations
    DB o-- DbOperations

Detailed Restructuring Steps

1. Define New Core Traits and Types

1. Create a new Model trait to replace SledModel
2. Create a new Storable trait for serialization/deserialization
3. Define a new error type hierarchy based on ourdb's error types
4. Create a DbOperations trait for database operations

2. Implement ourdb Backend

1. Create an OurDbStore<T> type that wraps ourdb
2. Implement the DbOperations trait for OurDbStore<T>
3. Add support for history tracking

3. Create New DB Manager

1. Create a new DB struct that manages multiple model types
2. Implement a builder pattern for configuration
3. Add methods for CRUD operations

4. Implement Transaction System

1. Create a transaction system that works with ourdb
2. Implement transaction operations (begin, commit, rollback)
3. Handle transaction state tracking

5. Update Model Implementations

1. Update all models to use u32 IDs
2. Implement the new Model trait for all models
3. Update model constructors and builders

Implementation Details

1. Core Traits and Types

// Error types
pub enum DbError {
    IoError(std::io::Error),
    SerializationError(bincode::Error),
    NotFound(u32),
    TransactionError(String),
    // Map to ourdb error types
    OurDbError(ourdb::Error),
    // Other error types as needed
}

// Result type alias
pub type DbResult<T> = Result<T, DbError>;

// Storable trait
pub trait Storable: Serialize + for<'de> Deserialize<'de> + Sized {
    fn serialize(&self) -> DbResult<Vec<u8>> {
        // Default implementation using bincode
        Ok(bincode::serialize(self)?)
    }
    
    fn deserialize(data: &[u8]) -> DbResult<Self> {
        // Default implementation using bincode
        Ok(bincode::deserialize(data)?)
    }
}

// Model trait
pub trait Model: Storable + Debug + Clone + Send + Sync + 'static {
    fn get_id(&self) -> u32;
    fn db_prefix() -> &'static str;
}

2. ourdb Backend Implementation

pub struct OurDbStore<T: Model> {
    db: OurDB,
    _phantom: PhantomData<T>,
}

impl<T: Model> OurDbStore<T> {
    pub fn new(config: OurDBConfig) -> DbResult<Self> {
        let db = OurDB::new(config)?;
        Ok(Self {
            db,
            _phantom: PhantomData,
        })
    }
    
    // Implementation of CRUD operations
}

impl<T: Model> DbOperations for OurDbStore<T> {
    // Implementation of DbOperations trait
}

3. DB Manager Implementation

pub struct DB {
    path: PathBuf,
    type_map: HashMap<TypeId, Arc<dyn DbOperations>>,
    transaction: Arc<RwLock<Option<TransactionState>>>,
}

impl DB {
    pub fn new(config: DbConfig) -> DbResult<Self> {
        // Implementation
    }
    
    // CRUD operations and other methods
}

4. Transaction System

pub struct TransactionState {
    operations: Vec<DbOperation>,
    active: bool,
}

enum DbOperation {
    Set {
        model_type: TypeId,
        serialized: Vec<u8>,
    },
    Delete {
        model_type: TypeId,
        id: u32,
    },
}

impl DB {
    pub fn begin_transaction(&self) -> DbResult<()> {
        // Implementation
    }
    
    pub fn commit_transaction(&self) -> DbResult<()> {
        // Implementation
    }
    
    pub fn rollback_transaction(&self) -> DbResult<()> {
        // Implementation
    }
}

5. Model Implementation Updates

// Example for Product model
impl Model for Product {
    fn get_id(&self) -> u32 {
        self.id
    }
    
    fn db_prefix() -> &'static str {
        "product"
    }
}

impl Storable for Product {}

Key Technical Considerations

1. Clean Architecture: The new design provides a cleaner separation of concerns.

2. Incremental IDs: All models will use u32 IDs, and ourdb will be configured in incremental mode.

3. History Tracking: The new API will expose ourdb's history tracking capabilities.

4. Transaction Support: We'll implement a custom transaction system on top of ourdb.

5. Error Handling: New error types will map directly to ourdb's error types.

6. Serialization: We'll use bincode for serialization/deserialization by default.

Migration Risks and Mitigations

Risk	Mitigation
Breaking API changes	Create a compatibility layer if needed
Data migration complexity	Develop a data migration utility
Performance impact	Benchmark before and after
Implementation complexity	Implement in phases with thorough testing
Integration issues	Create comprehensive integration tests

Implementation Phases

1. Phase 1: Define core traits and types
2. Phase 2: Implement ourdb backend
3. Phase 3: Create DB manager
4. Phase 4: Implement transaction system
5. Phase 5: Update model implementations
6. Phase 6: Create tests and benchmarks
7. Phase 7: Develop data migration utility