merge branches and cleanup db

heromodels/docs/herodb_ourdb_migration_plan.md

@@ -0,0 +1,266 @@
+# Migration Plan: Restructuring herodb to Use ourdb as Backend
+
+This document outlines the plan to restructure herodb to use ourdb as the backend, completely removing all sled references and better aligning with ourdb's design patterns.
+
+## Overview
+
+```mermaid
+graph TD
+    A[Current herodb with sled] --> B[Define new core traits]
+    B --> C[Implement ourdb backend]
+    C --> D[Create new DB manager]
+    D --> E[Implement transaction system]
+    E --> F[Update model implementations]
+    F --> G[Final restructured herodb with ourdb]
+```
+
+## New Architecture
+
+```mermaid
+classDiagram
+    class Model {
+        +get_id() u32
+        +db_prefix() &'static str
+    }
+    class Storable {
+        +serialize() Result<Vec<u8>>
+        +deserialize() Result<Self>
+    }
+    class DB {
+        -path: PathBuf
+        -type_map: HashMap<TypeId, Arc<dyn DbOperations>>
+        -transaction: Arc<RwLock<Option<TransactionState>>>
+        +new(config: DbConfig) Result<Self>
+        +begin_transaction() Result<()>
+        +commit_transaction() Result<()>
+        +rollback_transaction() Result<()>
+        +set<T: Model>(model: &T) Result<()>
+        +get<T: Model>(id: u32) Result<T>
+        +delete<T: Model>(id: u32) Result<()>
+        +list<T: Model>() Result<Vec<T>>
+        +register<T: Model>() Result<()>
+        +get_history<T: Model>(id: u32, depth: u8) Result<Vec<T>>
+    }
+    class DbOperations {
+        <<interface>>
+        +delete(id: u32) Result<()>
+        +get(id: u32) Result<Box<dyn Any>>
+        +list() Result<Box<dyn Any>>
+        +insert(model: &dyn Any) Result<()>
+        +get_history(id: u32, depth: u8) Result<Vec<Box<dyn Any>>>
+    }
+    class OurDbStore~T~ {
+        -db: OurDB
+        -model_type: PhantomData<T>
+        +new(config: OurDBConfig) Result<Self>
+        +insert(model: &T) Result<()>
+        +get(id: u32) Result<T>
+        +delete(id: u32) Result<()>
+        +list() Result<Vec<T>>
+        +get_history(id: u32, depth: u8) Result<Vec<T>>
+    }
+    
+    Model --|> Storable
+    OurDbStore ..|> DbOperations
+    DB o-- DbOperations
+```
+
+## Detailed Restructuring Steps
+
+### 1. Define New Core Traits and Types
+
+1. Create a new `Model` trait to replace `SledModel`
+2. Create a new `Storable` trait for serialization/deserialization
+3. Define a new error type hierarchy based on ourdb's error types
+4. Create a `DbOperations` trait for database operations
+
+### 2. Implement ourdb Backend
+
+1. Create an `OurDbStore<T>` type that wraps ourdb
+2. Implement the `DbOperations` trait for `OurDbStore<T>`
+3. Add support for history tracking
+
+### 3. Create New DB Manager
+
+1. Create a new `DB` struct that manages multiple model types
+2. Implement a builder pattern for configuration
+3. Add methods for CRUD operations
+
+### 4. Implement Transaction System
+
+1. Create a transaction system that works with ourdb
+2. Implement transaction operations (begin, commit, rollback)
+3. Handle transaction state tracking
+
+### 5. Update Model Implementations
+
+1. Update all models to use `u32` IDs
+2. Implement the new `Model` trait for all models
+3. Update model constructors and builders
+
+## Implementation Details
+
+### 1. Core Traits and Types
+
+```rust
+// Error types
+pub enum DbError {
+    IoError(std::io::Error),
+    SerializationError(bincode::Error),
+    NotFound(u32),
+    TransactionError(String),
+    // Map to ourdb error types
+    OurDbError(ourdb::Error),
+    // Other error types as needed
+}
+
+// Result type alias
+pub type DbResult<T> = Result<T, DbError>;
+
+// Storable trait
+pub trait Storable: Serialize + for<'de> Deserialize<'de> + Sized {
+    fn serialize(&self) -> DbResult<Vec<u8>> {
+        // Default implementation using bincode
+        Ok(bincode::serialize(self)?)
+    }
+    
+    fn deserialize(data: &[u8]) -> DbResult<Self> {
+        // Default implementation using bincode
+        Ok(bincode::deserialize(data)?)
+    }
+}
+
+// Model trait
+pub trait Model: Storable + Debug + Clone + Send + Sync + 'static {
+    fn get_id(&self) -> u32;
+    fn db_prefix() -> &'static str;
+}
+```
+
+### 2. ourdb Backend Implementation
+
+```rust
+pub struct OurDbStore<T: Model> {
+    db: OurDB,
+    _phantom: PhantomData<T>,
+}
+
+impl<T: Model> OurDbStore<T> {
+    pub fn new(config: OurDBConfig) -> DbResult<Self> {
+        let db = OurDB::new(config)?;
+        Ok(Self {
+            db,
+            _phantom: PhantomData,
+        })
+    }
+    
+    // Implementation of CRUD operations
+}
+
+impl<T: Model> DbOperations for OurDbStore<T> {
+    // Implementation of DbOperations trait
+}
+```
+
+### 3. DB Manager Implementation
+
+```rust
+pub struct DB {
+    path: PathBuf,
+    type_map: HashMap<TypeId, Arc<dyn DbOperations>>,
+    transaction: Arc<RwLock<Option<TransactionState>>>,
+}
+
+impl DB {
+    pub fn new(config: DbConfig) -> DbResult<Self> {
+        // Implementation
+    }
+    
+    // CRUD operations and other methods
+}
+```
+
+### 4. Transaction System
+
+```rust
+pub struct TransactionState {
+    operations: Vec<DbOperation>,
+    active: bool,
+}
+
+enum DbOperation {
+    Set {
+        model_type: TypeId,
+        serialized: Vec<u8>,
+    },
+    Delete {
+        model_type: TypeId,
+        id: u32,
+    },
+}
+
+impl DB {
+    pub fn begin_transaction(&self) -> DbResult<()> {
+        // Implementation
+    }
+    
+    pub fn commit_transaction(&self) -> DbResult<()> {
+        // Implementation
+    }
+    
+    pub fn rollback_transaction(&self) -> DbResult<()> {
+        // Implementation
+    }
+}
+```
+
+### 5. Model Implementation Updates
+
+```rust
+// Example for Product model
+impl Model for Product {
+    fn get_id(&self) -> u32 {
+        self.id
+    }
+    
+    fn db_prefix() -> &'static str {
+        "product"
+    }
+}
+
+impl Storable for Product {}
+```
+
+## Key Technical Considerations
+
+1. **Clean Architecture**: The new design provides a cleaner separation of concerns.
+
+2. **Incremental IDs**: All models will use `u32` IDs, and ourdb will be configured in incremental mode.
+
+3. **History Tracking**: The new API will expose ourdb's history tracking capabilities.
+
+4. **Transaction Support**: We'll implement a custom transaction system on top of ourdb.
+
+5. **Error Handling**: New error types will map directly to ourdb's error types.
+
+6. **Serialization**: We'll use bincode for serialization/deserialization by default.
+
+## Migration Risks and Mitigations
+
+| Risk | Mitigation |
+|------|------------|
+| Breaking API changes | Create a compatibility layer if needed |
+| Data migration complexity | Develop a data migration utility |
+| Performance impact | Benchmark before and after |
+| Implementation complexity | Implement in phases with thorough testing |
+| Integration issues | Create comprehensive integration tests |
+
+## Implementation Phases
+
+1. **Phase 1**: Define core traits and types
+2. **Phase 2**: Implement ourdb backend
+3. **Phase 3**: Create DB manager
+4. **Phase 4**: Implement transaction system
+5. **Phase 5**: Update model implementations
+6. **Phase 6**: Create tests and benchmarks
+7. **Phase 7**: Develop data migration utility