herocode/db
13
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

merge branches and cleanup db

Browse Source
This commit is contained in:
timurgordon
2025-06-27 12:11:04 +03:00
parent 5563d7e27e
commit 1f9ec01934
177 changed files with 1202 additions and 174 deletions
Download Patch File Download Diff File Expand all files Collapse all files

266
heromodels/docs/herodb_ourdb_migration_plan.md Normal file
View File

@@ -0,0 +1,266 @@
# Migration Plan: Restructuring herodb to Use ourdb as Backend
This document outlines the plan to restructure herodb to use ourdb as the backend, completely removing all sled references and better aligning with ourdb's design patterns.
## Overview
```mermaid
graph TD
A[Current herodb with sled] --> B[Define new core traits]
B --> C[Implement ourdb backend]
C --> D[Create new DB manager]
D --> E[Implement transaction system]
E --> F[Update model implementations]
F --> G[Final restructured herodb with ourdb]
```
## New Architecture
```mermaid
classDiagram
class Model {
+get_id() u32
+db_prefix() &'static str
}
class Storable {
+serialize() Result<Vec<u8>>
+deserialize() Result<Self>
}
class DB {
-path: PathBuf
-type_map: HashMap<TypeId, Arc<dyn DbOperations>>
-transaction: Arc<RwLock<Option<TransactionState>>>
+new(config: DbConfig) Result<Self>
+begin_transaction() Result<()>
+commit_transaction() Result<()>
+rollback_transaction() Result<()>
+set<T: Model>(model: &T) Result<()>
+get<T: Model>(id: u32) Result<T>
+delete<T: Model>(id: u32) Result<()>
+list<T: Model>() Result<Vec<T>>
+register<T: Model>() Result<()>
+get_history<T: Model>(id: u32, depth: u8) Result<Vec<T>>
}
class DbOperations {
<<interface>>
+delete(id: u32) Result<()>
+get(id: u32) Result<Box<dyn Any>>
+list() Result<Box<dyn Any>>
+insert(model: &dyn Any) Result<()>
+get_history(id: u32, depth: u8) Result<Vec<Box<dyn Any>>>
}
class OurDbStore~T~ {
-db: OurDB
-model_type: PhantomData<T>
+new(config: OurDBConfig) Result<Self>
+insert(model: &T) Result<()>
+get(id: u32) Result<T>
+delete(id: u32) Result<()>
+list() Result<Vec<T>>
+get_history(id: u32, depth: u8) Result<Vec<T>>
}
Model --|> Storable
OurDbStore ..|> DbOperations
DB o-- DbOperations
```
## Detailed Restructuring Steps
### 1. Define New Core Traits and Types
1. Create a new `Model` trait to replace `SledModel`
2. Create a new `Storable` trait for serialization/deserialization
3. Define a new error type hierarchy based on ourdb's error types
4. Create a `DbOperations` trait for database operations
### 2. Implement ourdb Backend
1. Create an `OurDbStore<T>` type that wraps ourdb
2. Implement the `DbOperations` trait for `OurDbStore<T>`
3. Add support for history tracking
### 3. Create New DB Manager
1. Create a new `DB` struct that manages multiple model types
2. Implement a builder pattern for configuration
3. Add methods for CRUD operations
### 4. Implement Transaction System
1. Create a transaction system that works with ourdb
2. Implement transaction operations (begin, commit, rollback)
3. Handle transaction state tracking
### 5. Update Model Implementations
1. Update all models to use `u32` IDs
2. Implement the new `Model` trait for all models
3. Update model constructors and builders
## Implementation Details
### 1. Core Traits and Types
```rust
// Error types
pub enum DbError {
IoError(std::io::Error),
SerializationError(bincode::Error),
NotFound(u32),
TransactionError(String),
// Map to ourdb error types
OurDbError(ourdb::Error),
// Other error types as needed
}
// Result type alias
pub type DbResult<T> = Result<T, DbError>;
// Storable trait
pub trait Storable: Serialize + for<'de> Deserialize<'de> + Sized {
fn serialize(&self) -> DbResult<Vec<u8>> {
// Default implementation using bincode
Ok(bincode::serialize(self)?)
}
fn deserialize(data: &[u8]) -> DbResult<Self> {
// Default implementation using bincode
Ok(bincode::deserialize(data)?)
}
}
// Model trait
pub trait Model: Storable + Debug + Clone + Send + Sync + 'static {
fn get_id(&self) -> u32;
fn db_prefix() -> &'static str;
}
```
### 2. ourdb Backend Implementation
```rust
pub struct OurDbStore<T: Model> {
db: OurDB,
_phantom: PhantomData<T>,
}
impl<T: Model> OurDbStore<T> {
pub fn new(config: OurDBConfig) -> DbResult<Self> {
let db = OurDB::new(config)?;
Ok(Self {
db,
_phantom: PhantomData,
})
}
// Implementation of CRUD operations
}
impl<T: Model> DbOperations for OurDbStore<T> {
// Implementation of DbOperations trait
}
```
### 3. DB Manager Implementation
```rust
pub struct DB {
path: PathBuf,
type_map: HashMap<TypeId, Arc<dyn DbOperations>>,
transaction: Arc<RwLock<Option<TransactionState>>>,
}
impl DB {
pub fn new(config: DbConfig) -> DbResult<Self> {
// Implementation
}
// CRUD operations and other methods
}
```
### 4. Transaction System
```rust
pub struct TransactionState {
operations: Vec<DbOperation>,
active: bool,
}
enum DbOperation {
Set {
model_type: TypeId,
serialized: Vec<u8>,
},
Delete {
model_type: TypeId,
id: u32,
},
}
impl DB {
pub fn begin_transaction(&self) -> DbResult<()> {
// Implementation
}
pub fn commit_transaction(&self) -> DbResult<()> {
// Implementation
}
pub fn rollback_transaction(&self) -> DbResult<()> {
// Implementation
}
}
```
### 5. Model Implementation Updates
```rust
// Example for Product model
impl Model for Product {
fn get_id(&self) -> u32 {
self.id
}
fn db_prefix() -> &'static str {
"product"
}
}
impl Storable for Product {}
```
## Key Technical Considerations
1. **Clean Architecture**: The new design provides a cleaner separation of concerns.
2. **Incremental IDs**: All models will use `u32` IDs, and ourdb will be configured in incremental mode.
3. **History Tracking**: The new API will expose ourdb's history tracking capabilities.
4. **Transaction Support**: We'll implement a custom transaction system on top of ourdb.
5. **Error Handling**: New error types will map directly to ourdb's error types.
6. **Serialization**: We'll use bincode for serialization/deserialization by default.
## Migration Risks and Mitigations
| Risk | Mitigation |
|------|------------|
| Breaking API changes | Create a compatibility layer if needed |
| Data migration complexity | Develop a data migration utility |
| Performance impact | Benchmark before and after |
| Implementation complexity | Implement in phases with thorough testing |
| Integration issues | Create comprehensive integration tests |
## Implementation Phases
1. **Phase 1**: Define core traits and types
2. **Phase 2**: Implement ourdb backend
3. **Phase 3**: Create DB manager
4. **Phase 4**: Implement transaction system
5. **Phase 5**: Update model implementations
6. **Phase 6**: Create tests and benchmarks
7. **Phase 7**: Develop data migration utility

161
heromodels/docs/mcc_models_standalone_plan.md Normal file
View File

@@ -0,0 +1,161 @@
# MCC Models Standalone Implementation Plan
## Overview
This document outlines the plan to make the MCC models in `herodb/src/models/mcc` completely standalone without dependencies on the database implementation, while ensuring that examples like `herodb/src/cmd/dbexample_mcc` can still work.
## Current Architecture Analysis
```mermaid
graph TD
subgraph "Before"
A1[MCC Models] -->|Depends on| B1[DB Implementation]
A1 -->|Implements| C1[Model Trait]
A1 -->|Implements| D1[Storable Trait]
E1[dbexample_mcc] -->|Uses| A1
E1 -->|Uses| B1
end
subgraph "After"
A2[Standalone MCC Models] -.->|No dependency on| B2[DB Implementation]
B2 -->|Works with| F2[Any Serializable Struct]
B2 -->|Optional trait| G2[ModelAdapter Trait]
E2[dbexample_mcc] -->|Uses| A2
E2 -->|Uses| B2
end
```
## Implementation Plan
### Phase 1: Make MCC Models Standalone
1. For each MCC model file (calendar.rs, event.rs, mail.rs, contacts.rs, message.rs):
- Remove `use crate::db::{Model, Storable, DB, DbError, DbResult}`
- Remove `impl Model for X` and `impl Storable for X` blocks
- Replace methods that use DB (like `get_events(&self, db: &DB)`) with standalone methods
2. For mod.rs and lib.rs:
- Remove `pub use crate::db::{DB, DBBuilder, Model, Storable, DbError, DbResult}`
### Phase 2: Modify DB Implementation
1. Create a new ModelAdapter trait in db/model.rs:
```rust
pub trait ModelAdapter {
fn get_id(&self) -> u32;
fn db_prefix() -> &'static str;
fn db_keys(&self) -> Vec<IndexKey> { Vec::new() }
}
```
2. Modify DB methods in db.rs to work with any serializable struct:
```rust
impl DB {
// Generic set method for any serializable type
pub fn set<T: Serialize + DeserializeOwned + 'static>(&self, item: &T, id: u32, prefix: &str) -> DbResult<()> {
// Implementation
}
// Enhanced version for types implementing ModelAdapter
pub fn set_model<T: Serialize + DeserializeOwned + ModelAdapter + 'static>(&self, item: &T) -> DbResult<()> {
self.set(item, item.get_id(), T::db_prefix())
}
// Similar changes for get, delete, list methods
}
```
3. Update DBBuilder to register any serializable type:
```rust
impl DBBuilder {
pub fn register_type<T: Serialize + DeserializeOwned + 'static>(&mut self, prefix: &'static str) -> &mut Self {
// Implementation
}
pub fn register_model<T: Serialize + DeserializeOwned + ModelAdapter + 'static>(&mut self) -> &mut Self {
self.register_type::<T>(T::db_prefix())
}
}
```
### Phase 3: Update Examples and Tests
1. Update dbexample_mcc/main.rs:
```rust
let db = DBBuilder::new(&db_path)
.register_type::<Calendar>("calendar")
.register_type::<Event>("event")
// etc.
.build()?;
```
2. Create a new standalone example for MCC models similar to circle_standalone.rs
## Detailed Changes Required
### MCC Models Changes
#### calendar.rs
- Remove database-related imports
- Remove Model and Storable trait implementations
- Replace `get_events(&self, db: &DB)` with a standalone method like:
```rust
pub fn filter_events(&self, events: &[Event]) -> Vec<&Event> {
events.iter()
.filter(|event| event.calendar_id == self.id)
.collect()
}
```
#### event.rs
- Remove database-related imports
- Remove Model and Storable trait implementations
- Add standalone methods for event operations
#### mail.rs
- Remove database-related imports
- Remove Model and Storable trait implementations
- Add standalone methods for mail operations
#### contacts.rs
- Remove database-related imports
- Remove Model and Storable trait implementations
- Add standalone methods for contact operations
#### message.rs
- Remove database-related imports
- Remove Model and Storable trait implementations
- Add standalone methods for message operations
#### mod.rs and lib.rs
- Remove re-exports of database components
### DB Implementation Changes
#### model.rs
- Create a new ModelAdapter trait
- Keep existing Model and Storable traits for backward compatibility
- Add helper methods for working with serializable structs
#### db.rs
- Modify DB methods to work with any serializable struct
- Add overloaded methods for ModelAdapter types
- Ensure backward compatibility with existing code
#### DBBuilder
- Update to register any serializable type
- Keep existing methods for backward compatibility
## Testing Strategy
1. Ensure all existing tests pass with the modified DB implementation
2. Create new tests for standalone MCC models
3. Verify dbexample_mcc works with the new implementation
4. Create a new standalone example for MCC models
## Benefits
1. MCC models become more reusable and can be used without database dependencies
2. DB implementation becomes more flexible and can work with any serializable struct
3. Cleaner separation of concerns between models and database operations
4. Easier to test models in isolation

277
heromodels/docs/model_trait_unification_plan.md Normal file
View File

@@ -0,0 +1,277 @@
# Model Trait Unification Plan
## Introduction
This document outlines the plan to unify the `BaseModel` and `ModelBuilder` traits in the heromodels crate into a single `Model` trait. The goal is to simplify the codebase, reduce boilerplate, and make the API more intuitive while maintaining all existing functionality.
## Current Structure
Currently, the codebase has two separate traits:
1. **BaseModel** - Provides database-related functionality:
- `db_prefix()` - Returns a string prefix for database operations
- `db_keys()` - Returns index keys for the model
- `get_id()` - Returns the model's unique ID
2. **ModelBuilder** - Provides builder pattern functionality:
- `base_data_mut()` - Gets a mutable reference to the base data
- `id()` - Sets the ID for the model
- `build()` - Finalizes the model by updating timestamps
Each model (like User and Comment) implements both traits separately, and there are two separate macros for implementing these traits:
```rust
// For BaseModel
impl_base_model!(User, "user");
// For ModelBuilder
impl_model_builder!(User);
```
This leads to duplication and cognitive overhead when working with models.
## Proposed Structure
We will create a unified `Model` trait that combines all the functionality from both existing traits:
```mermaid
classDiagram
class Model {
<<trait>>
+db_prefix() static str
+db_keys() Vec~IndexKey~
+get_id() u32
+base_data_mut() &mut BaseModelData
+id(u32) Self
+build() Self
}
class User {
+base_data BaseModelData
+username String
+email String
+full_name String
+is_active bool
}
class Comment {
+base_data BaseModelData
+user_id u32
+content String
}
Model <|-- User
Model <|-- Comment
```
## Implementation Steps
### 1. Update model.rs
Create the new `Model` trait that combines all methods from both existing traits:
```rust
/// Unified trait for all models
pub trait Model: Debug + Clone + Serialize + for<'de> Deserialize<'de> + Send + Sync + 'static {
/// Get the database prefix for this model type
fn db_prefix() -> &'static str where Self: Sized;
/// Returns a list of index keys for this model instance
/// These keys will be used to create additional indexes in the TST
fn db_keys(&self) -> Vec<IndexKey> {
Vec::new()
}
/// Get the unique ID for this model
fn get_id(&self) -> u32;
/// Get a mutable reference to the base_data field
fn base_data_mut(&mut self) -> &mut BaseModelData;
/// Set the ID for this model
fn id(mut self, id: u32) -> Self where Self: Sized {
self.base_data_mut().id = id;
self
}
/// Build the model, updating the modified timestamp
fn build(mut self) -> Self where Self: Sized {
self.base_data_mut().update_modified();
self
}
}
```
Create a new implementation macro that implements all required methods:
```rust
/// Macro to implement Model for a struct that contains a base_data field of type BaseModelData
#[macro_export]
macro_rules! impl_model {
($type:ty, $prefix:expr) => {
impl $crate::core::model::Model for $type {
fn db_prefix() -> &'static str {
$prefix
}
fn get_id(&self) -> u32 {
self.base_data.id
}
fn base_data_mut(&mut self) -> &mut $crate::core::model::BaseModelData {
&mut self.base_data
}
// Other methods have default implementations
}
};
}
```
Mark the old traits and macros as deprecated (or remove them entirely):
```rust
#[deprecated(
since = "0.2.0",
note = "Use the unified Model trait instead"
)]
pub trait BaseModel { /* ... */ }
#[deprecated(
since = "0.2.0",
note = "Use the unified Model trait instead"
)]
pub trait ModelBuilder { /* ... */ }
#[deprecated(
since = "0.2.0",
note = "Use impl_model instead"
)]
#[macro_export]
macro_rules! impl_base_model { /* ... */ }
#[deprecated(
since = "0.2.0",
note = "Use impl_model instead"
)]
#[macro_export]
macro_rules! impl_model_builder { /* ... */ }
```
### 2. Update User and Comment Implementations
For User:
```rust
// Implement Model for User
impl_model!(User, "user");
// Custom implementation of db_keys
impl Model for User {
fn db_keys(&self) -> Vec<IndexKey> {
vec![
IndexKey {
name: "username",
value: self.username.clone(),
},
IndexKey {
name: "email",
value: self.email.clone(),
},
IndexKey {
name: "is_active",
value: self.is_active.to_string(),
},
]
}
}
```
For Comment:
```rust
// Implement Model for Comment
impl_model!(Comment, "comment");
// Custom implementation of db_keys
impl Model for Comment {
fn db_keys(&self) -> Vec<IndexKey> {
vec![
IndexKey {
name: "user_id",
value: self.user_id.to_string(),
},
]
}
}
```
### 3. Update Imports and References
Update the lib.rs file to export the new trait and macro:
```rust
// Export core module
pub mod core;
// Export userexample module
pub mod userexample;
// Re-export key types for convenience
pub use core::model::{Model, BaseModelData, IndexKey};
pub use core::Comment;
pub use userexample::User;
// Re-export macros
pub use crate::impl_model;
```
Update the example code to use the new trait:
```rust
use heromodels::{Model, Comment, User};
fn main() {
println!("Hero Models - Basic Usage Example");
println!("================================");
// Create a new user using the fluent interface
let user = User::new(1)
.username("johndoe")
.email("john.doe@example.com")
.full_name("John Doe")
.build();
println!("Created user: {:?}", user);
println!("User ID: {}", user.get_id());
println!("User DB Prefix: {}", User::db_prefix());
// Create a comment for the user
let comment = Comment::new(1)
.user_id(2) // commenter's user ID
.content("This is a comment on the user")
.build();
println!("\nCreated comment: {:?}", comment);
println!("Comment ID: {}", comment.get_id());
println!("Comment DB Prefix: {}", Comment::db_prefix());
}
```
### 4. Testing
Run the example to ensure it works as expected:
```bash
cargo run --example basic_user_example
```
Verify that all functionality is preserved and that the output matches the expected output.
## Benefits of This Approach
1. **Simplified Code Structure**: One trait instead of two means less cognitive overhead
2. **Reduced Boilerplate**: A single macro implementation reduces repetitive code
3. **Improved Usability**: No need to import multiple traits or worry about which trait provides which method
4. **Maintained Functionality**: All existing functionality is preserved
5. **Better Encapsulation**: The model concept is more cohesive as a single trait

318
heromodels/docs/payment_usage.md Normal file
View File

@@ -0,0 +1,318 @@
# Payment Model Usage Guide
This document provides comprehensive instructions for AI assistants on how to use the Payment model in the heromodels repository.
## Overview
The Payment model represents a payment transaction in the system, typically associated with company registration or subscription payments. It integrates with Stripe for payment processing and maintains comprehensive status tracking.
## Model Structure
```rust
pub struct Payment {
pub base_data: BaseModelData, // Auto-managed ID, timestamps, comments
pub payment_intent_id: String, // Stripe payment intent ID
pub company_id: u32, // Foreign key to Company
pub payment_plan: String, // "monthly", "yearly", "two_year"
pub setup_fee: f64, // One-time setup fee
pub monthly_fee: f64, // Recurring monthly fee
pub total_amount: f64, // Total amount paid
pub currency: String, // Currency code (defaults to "usd")
pub status: PaymentStatus, // Current payment status
pub stripe_customer_id: Option<String>, // Stripe customer ID (set on completion)
pub created_at: i64, // Payment creation timestamp
pub completed_at: Option<i64>, // Payment completion timestamp
}
pub enum PaymentStatus {
Pending, // Initial state - payment created but not processed
Processing, // Payment is being processed by Stripe
Completed, // Payment successfully completed
Failed, // Payment processing failed
Refunded, // Payment was refunded
}
```
## Basic Usage
### 1. Creating a New Payment
```rust
use heromodels::models::biz::{Payment, PaymentStatus};
// Create a new payment with required fields
let payment = Payment::new(
"pi_1234567890".to_string(), // Stripe payment intent ID
company_id, // Company ID from database
"monthly".to_string(), // Payment plan
100.0, // Setup fee
49.99, // Monthly fee
149.99, // Total amount
);
// Payment defaults:
// - status: PaymentStatus::Pending
// - currency: "usd"
// - stripe_customer_id: None
// - created_at: current timestamp
// - completed_at: None
```
### 2. Using Builder Pattern
```rust
let payment = Payment::new(
"pi_1234567890".to_string(),
company_id,
"yearly".to_string(),
500.0,
99.99,
1699.88,
)
.currency("eur".to_string())
.stripe_customer_id(Some("cus_existing_customer".to_string()));
```
### 3. Database Operations
```rust
use heromodels::db::Collection;
// Save payment to database
let db = get_db()?;
let (payment_id, saved_payment) = db.set(&payment)?;
// Retrieve payment by ID
let retrieved_payment: Payment = db.get_by_id(payment_id)?.unwrap();
// Update payment
let updated_payment = saved_payment.complete_payment(Some("cus_new_customer".to_string()));
let (_, final_payment) = db.set(&updated_payment)?;
```
## Payment Status Management
### Status Transitions
```rust
// 1. Start with Pending status (default)
let payment = Payment::new(/* ... */);
assert!(payment.is_pending());
// 2. Mark as processing when Stripe starts processing
let processing_payment = payment.process_payment();
assert!(processing_payment.is_processing());
// 3. Complete payment when Stripe confirms success
let completed_payment = processing_payment.complete_payment(Some("cus_123".to_string()));
assert!(completed_payment.is_completed());
assert!(completed_payment.completed_at.is_some());
// 4. Handle failure if payment fails
let failed_payment = processing_payment.fail_payment();
assert!(failed_payment.has_failed());
// 5. Refund if needed
let refunded_payment = completed_payment.refund_payment();
assert!(refunded_payment.is_refunded());
```
### Status Check Methods
```rust
// Check current status
if payment.is_pending() {
// Show "Payment Pending" UI
} else if payment.is_processing() {
// Show "Processing Payment" UI
} else if payment.is_completed() {
// Show "Payment Successful" UI
// Enable company features
} else if payment.has_failed() {
// Show "Payment Failed" UI
// Offer retry option
} else if payment.is_refunded() {
// Show "Payment Refunded" UI
}
```
## Integration with Company Model
### Complete Payment Flow
```rust
use heromodels::models::biz::{Company, CompanyStatus, Payment, PaymentStatus};
// 1. Create company with pending payment status
let company = Company::new(
"TechStart Inc.".to_string(),
"REG-TS-2024-001".to_string(),
chrono::Utc::now().timestamp(),
)
.email("contact@techstart.com".to_string())
.status(CompanyStatus::PendingPayment);
let (company_id, company) = db.set(&company)?;
// 2. Create payment for the company
let payment = Payment::new(
stripe_payment_intent_id,
company_id,
"yearly".to_string(),
500.0, // Setup fee
99.0, // Monthly fee
1688.0, // Total (setup + 12 months)
);
let (payment_id, payment) = db.set(&payment)?;
// 3. Process payment through Stripe
let processing_payment = payment.process_payment();
let (_, processing_payment) = db.set(&processing_payment)?;
// 4. On successful Stripe webhook
let completed_payment = processing_payment.complete_payment(Some(stripe_customer_id));
let (_, completed_payment) = db.set(&completed_payment)?;
// 5. Activate company
let active_company = company.status(CompanyStatus::Active);
let (_, active_company) = db.set(&active_company)?;
```
## Database Indexing
The Payment model provides custom indexes for efficient querying:
```rust
// Indexed fields for fast lookups:
// - payment_intent_id: Find payment by Stripe intent ID
// - company_id: Find all payments for a company
// - status: Find payments by status
// Example queries (conceptual - actual implementation depends on your query layer)
// let pending_payments = db.find_by_index("status", "Pending")?;
// let company_payments = db.find_by_index("company_id", company_id.to_string())?;
// let stripe_payment = db.find_by_index("payment_intent_id", "pi_1234567890")?;
```
## Error Handling Best Practices
```rust
use heromodels::db::DbError;
fn process_payment_flow(payment_intent_id: String, company_id: u32) -> Result<Payment, DbError> {
let db = get_db()?;
// Create payment
let payment = Payment::new(
payment_intent_id,
company_id,
"monthly".to_string(),
100.0,
49.99,
149.99,
);
// Save to database
let (payment_id, payment) = db.set(&payment)?;
// Process through Stripe (external API call)
match process_stripe_payment(&payment.payment_intent_id) {
Ok(stripe_customer_id) => {
// Success: complete payment
let completed_payment = payment.complete_payment(Some(stripe_customer_id));
let (_, final_payment) = db.set(&completed_payment)?;
Ok(final_payment)
}
Err(_) => {
// Failure: mark as failed
let failed_payment = payment.fail_payment();
let (_, final_payment) = db.set(&failed_payment)?;
Ok(final_payment)
}
}
}
```
## Testing
The Payment model includes comprehensive tests in `tests/payment.rs`. When working with payments:
1. **Always test status transitions**
2. **Verify timestamp handling**
3. **Test database persistence**
4. **Test integration with Company model**
5. **Test builder pattern methods**
```bash
# Run payment tests
cargo test payment
# Run specific test
cargo test test_payment_completion
```
## Common Patterns
### 1. Payment Retry Logic
```rust
fn retry_failed_payment(payment: Payment) -> Payment {
if payment.has_failed() {
// Reset to pending for retry
Payment::new(
payment.payment_intent_id,
payment.company_id,
payment.payment_plan,
payment.setup_fee,
payment.monthly_fee,
payment.total_amount,
)
.currency(payment.currency)
} else {
payment
}
}
```
### 2. Payment Summary
```rust
fn get_payment_summary(payment: &Payment) -> String {
format!(
"Payment {} for company {}: {} {} ({})",
payment.payment_intent_id,
payment.company_id,
payment.total_amount,
payment.currency.to_uppercase(),
payment.status
)
}
```
### 3. Payment Validation
```rust
fn validate_payment(payment: &Payment) -> Result<(), String> {
if payment.total_amount <= 0.0 {
return Err("Total amount must be positive".to_string());
}
if payment.payment_intent_id.is_empty() {
return Err("Payment intent ID is required".to_string());
}
if payment.company_id == 0 {
return Err("Valid company ID is required".to_string());
}
Ok(())
}
```
## Key Points for AI Assistants
1. **Always use auto-generated IDs** - Don't manually set IDs, let OurDB handle them
2. **Follow status flow** - Pending → Processing → Completed/Failed → (optionally) Refunded
3. **Update timestamps** - `completed_at` is automatically set when calling `complete_payment()`
4. **Use builder pattern** - For optional fields and cleaner code
5. **Test thoroughly** - Payment logic is critical, always verify with tests
6. **Handle errors gracefully** - Payment failures should be tracked, not ignored
7. **Integrate with Company** - Payments typically affect company status
8. **Use proper indexing** - Leverage indexed fields for efficient queries
This model follows the heromodels patterns and integrates seamlessly with the existing codebase architecture.

151
heromodels/docs/prompts/new_rhai_rs_gen.md Normal file
View File

@@ -0,0 +1,151 @@
# AI Prompt: Generate Rhai Module Integration for Rust Models
## Context
You are tasked with creating a Rhai scripting integration for Rust models in the heromodels crate. The integration should follow the builder pattern and provide a comprehensive set of functions for interacting with the models through Rhai scripts.
## Key Concepts to Understand
1. **Builder Pattern vs. Rhai Mutation**: Rust models use builder pattern (methods consume self and return Self), but Rhai requires functions that mutate in place (&mut self).
2. **Database Access**: Database functions are exposed as global functions in a 'db' module namespace (e.g., `db::save_flow`, not `db.save_flow`).
3. **ID Handling**: New objects should use ID 0 to allow database assignment of real IDs.
4. **Function vs. Method Calls**: Rhai scripts must use function calls (e.g., `status(flow, STATUS_DRAFT)`) not method chaining (e.g., `flow.status(STATUS_DRAFT)`).
## Requirements
1. **Module Structure and Imports**:
- Create a `rhai.rs` file that exports a function named `register_[model]_rhai_module` which takes an Engine and an Arc<OurDB>
- **IMPORTANT**: Import `heromodels_core::Model` trait for access to methods like `get_id`
- Use the `#[export_module]` macro to define the Rhai module
- Register the module globally with the engine using the correct module registration syntax
- Create separate database functions that capture the database reference
2. **Builder Pattern Integration**:
- For each model struct marked with `#[model]`, create constructor functions (e.g., `new_[model]`) that accept ID 0 for new objects
- For each builder method in the model, create a corresponding Rhai function that:
- Takes a mutable reference to the model as first parameter
- Uses `std::mem::take` to get ownership (if the model implements Default) or `std::mem::replace` (if it doesn't)
- Calls the builder method on the owned object
- Assigns the result back to the mutable reference
- Returns a clone of the modified object
3. **Field Access**:
- **IMPORTANT**: Do NOT create getter methods for fields that can be accessed directly
- Use direct field access in example code (e.g., `flow.name`, not `flow.get_name()`)
- Only create getter functions for computed properties or when special conversion is needed
- Handle special types appropriately (e.g., convert timestamps, IDs, etc.)
4. **Database Functions**:
- For each model, implement the following database functions in the `db` module namespace:
- `db::save_[model]`: Save the model to the database and return the saved model with assigned ID
- `db::get_[model]_by_id`: Retrieve a model by ID
- `db::delete_[model]`: Delete a model from the database
- `db::list_[model]s`: List all models of this type
- **IMPORTANT**: Use `db::function_name` syntax in scripts, not `db.function_name`
- Handle errors properly with detailed error messages using debug format `{:?}` for error objects
- Convert between Rhai's i64 and Rust's u32/u64 types for IDs and timestamps
- Return saved objects from save functions to ensure proper ID handling
5. **Error Handling**:
- Use `Box<EvalAltResult>` for error returns
- Provide detailed error messages with proper position information
- **IMPORTANT**: Use `context.call_position()` not the deprecated `context.position()`
- Handle type conversions safely
- Format error messages with debug format for error objects: `format!("Error: {:?}", err)`
6. **Type Conversions**:
- Create helper functions for converting between Rhai and Rust types
- Handle special cases like enums with string representations
## Implementation Pattern
```rust
// Helper functions for type conversion
fn i64_to_u32(val: i64, position: Position, param_name: &str, function_name: &str) -> Result<u32, Box<EvalAltResult>> {
u32::try_from(val).map_err(|_| {
Box::new(EvalAltResult::ErrorArithmetic(
format!("{} must be a positive integer less than 2^32 in {}", param_name, function_name).into(),
position
))
})
}
// Model constructor
#[rhai_fn(name = "new_[model]")]
pub fn new_model() -> Model {
Model::new()
}
// Builder method integration
#[rhai_fn(name = "[method_name]", return_raw, global, pure)]
pub fn model_method(model: &mut Model, param: Type) -> Result<Model, Box<EvalAltResult>> {
let owned_model = std::mem::take(model); // or replace if Default not implemented
*model = owned_model.method_name(param);
Ok(model.clone())
}
// Getter method
#[rhai_fn(name = "[field]", global)]
pub fn get_model_field(model: &mut Model) -> Type {
model.field.clone()
}
// Database function
db_module.set_native_fn("save_[model]", move |model: Model| -> Result<Model, Box<EvalAltResult>> {
db_clone.set(&model)
.map_err(|e| Box::new(EvalAltResult::ErrorRuntime(format!("DB Error: {}", e).into(), Position::NONE)))
.map(|(_, model)| model)
});
```
## Important Notes
1. All models must follow the builder pattern where methods take ownership of `self` and return `Self`
2. If a model doesn't implement `Default`, use `std::mem::replace` instead of `std::mem::take`
3. Ensure proper error handling for all operations
4. Register all functions with appropriate Rhai names
5. Follow the naming conventions established in existing modules
6. Ensure all database operations are properly wrapped with error handling
7. **CRITICAL**: When creating example scripts, use function calls not method chaining
8. **CRITICAL**: Always use ID 0 for new objects in scripts to allow database assignment
9. **CRITICAL**: After saving objects, use the returned objects with assigned IDs for further operations
10. **CRITICAL**: In Rust examples, include the Model trait import and use direct field access
## Example Rhai Script
```rhai
// Create a new flow with ID 0 (database will assign real ID)
let flow = new_flow(0, "12345678-1234-1234-1234-123456789012");
name(flow, "Document Approval Flow");
status(flow, STATUS_DRAFT);
// Save flow to database and get saved object with assigned ID
let saved_flow = db::save_flow(flow);
print(`Flow saved to database with ID: ${get_flow_id(saved_flow)}`);
// Retrieve flow from database using assigned ID
let retrieved_flow = db::get_flow_by_id(get_flow_id(saved_flow));
```
## Example Rust Code
```rust
use heromodels_core::Model; // Import Model trait for get_id()
// Create flow using Rhai
let flow: Flow = engine.eval("new_flow(0, \"12345678-1234-1234-1234-123456789012\")").unwrap();
// Access fields directly
println!("Flow name: {}", flow.name);
// Save to database using Rhai function
let save_script = "fn save_it(f) { return db::save_flow(f); }";
let save_ast = engine.compile(save_script).unwrap();
let saved_flow: Flow = engine.call_fn(&mut scope, &save_ast, "save_it", (flow,)).unwrap();
println!("Saved flow ID: {}", saved_flow.get_id());
```
## Reference Implementations
See the existing implementations for examples:
1. `heromodels/src/models/calendar/rhai.rs` - Calendar module Rhai bindings
2. `heromodels/src/models/flow/rhai.rs` - Flow module Rhai bindings
3. `heromodels_rhai/examples/flow/flow_script.rhai` - Example Rhai script
4. `heromodels_rhai/examples/flow/example.rs` - Example Rust code

192
heromodels/docs/prompts/rhai_rs_generation_prompt.md Normal file
View File

@@ -0,0 +1,192 @@
## AI Prompt: Generate `rhai.rs` for a new Rust Model
**Objective:**
Create a `rhai.rs` file to expose the Rust model `[YourModelName]` (and any related owned sub-models like `[YourSubModelName]`) to the Rhai scripting engine. This file should allow Rhai scripts to create, manipulate, and retrieve instances of `[YourModelName]`.
**Input Requirements (Provide this information for your specific model):**
1. **Rust Struct Definition(s):**
```rust
// Example for [YourModelName]
#[derive(Clone, Debug, Serialize, Deserialize)] // Ensure necessary derives
pub struct [YourModelName] {
pub base_data: BaseModelData, // Common field for ID
pub unique_id_field: String, // Example: like flow_uuid
pub name: String,
pub status: String,
// If it owns a collection of sub-models:
pub sub_items: Vec<[YourSubModelName]>,
// Other fields...
}
impl [YourModelName] {
// Constructor
pub fn new(id: u32, unique_id_field: String /*, other essential params */) -> Self {
Self {
base_data: BaseModelData::new(id),
unique_id_field,
name: "".to_string(), // Default or passed in
status: "".to_string(), // Default or passed in
sub_items: Vec::new(),
// ...
}
}
// Builder methods
pub fn name(mut self, name: String) -> Self {
self.name = name;
self
}
pub fn status(mut self, status: String) -> Self {
self.status = status;
self
}
// Method to add sub-items
pub fn add_sub_item(mut self, item: [YourSubModelName]) -> Self {
self.sub_items.push(item);
self
}
// Other methods to expose...
}
// Example for [YourSubModelName] (if applicable)
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct [YourSubModelName] {
pub id: u32,
pub description: String,
// ...
}
impl [YourSubModelName] {
pub fn new(id: u32, description: String) -> Self {
Self { id, description }
}
// Builder methods for sub-model...
}
```
2. **Key ID fields that need `i64` (Rhai) to `u32` (Rust) conversion:** (e.g., `base_data.id`, `[YourSubModelName].id`, any foreign key IDs).
**Implementation Guidelines for `rhai.rs`:**
1. **File Structure:**
* Start with necessary imports: `rhai::{Dynamic, Engine, EvalAltResult, NativeCallContext, Position}`, `std::sync::Arc`, your model structs, `BaseModelData`, `OurDB`.
* Include the `i64_to_u32` helper function.
* Define `pub fn register_[your_model_name]_rhai_module(engine: &mut Engine, db: Arc<OurDB>) { ... }`.
2. **Helper Function for ID Conversion:**
```rust
fn i64_to_u32(val: i64, context_pos: Position, field_name: &str, object_name: &str) -> Result<u32, Box<EvalAltResult>> {
val.try_into().map_err(|_e| {
Box::new(EvalAltResult::ErrorArithmetic(
format!("Conversion error for {} in {} from i64 to u32", field_name, object_name),
context_pos,
))
})
}
```
3. **Constructors (e.g., `new_[your_model_name]`):**
* Use `NativeCallContext` for manual argument parsing and `i64_to_u32` conversion for ID fields.
* Example:
```rust
engine.register_fn("new_[your_model_name]",
move |context: NativeCallContext, id_i64: i64, unique_id_str: String /*, other_args... */|
-> Result<[YourModelName], Box<EvalAltResult>> {
let id_u32 = i64_to_u32(id_i64, context.position(), "id", "new_[your_model_name]")?;
Ok([YourModelName]::new(id_u32, unique_id_str /*, ... */))
});
```
* Do the same for `new_[your_sub_model_name]` if applicable.
* **Note on `adapter_macros`**: `adapt_rhai_i64_input_fn!` is generally NOT suitable for constructors with multiple arguments or mixed types (e.g., `u32` and `String`). Prefer `NativeCallContext`.
4. **Builder Methods:**
* Register functions that take ownership of the model, modify it, and return it.
* Example:
```rust
engine.register_fn("name", |model: [YourModelName], name_val: String| -> [YourModelName] { model.name(name_val) });
engine.register_fn("status", |model: [YourModelName], status_val: String| -> [YourModelName] { model.status(status_val) });
// For adding sub-items (if applicable)
engine.register_fn("add_sub_item", |model: [YourModelName], item: [YourSubModelName]| -> [YourModelName] { model.add_sub_item(item) });
```
5. **Getters:**
* Use `engine.register_get("field_name", |model: &mut [YourModelName]| -> Result<FieldType, Box<EvalAltResult>> { Ok(model.field.clone()) });`
* For `base_data.id` (u32), cast to `i64` for Rhai: `Ok(model.base_data.id as i64)`
* **For `Vec<[YourSubModelName]>` fields (e.g., `sub_items`):** Convert to `rhai::Array`.
```rust
engine.register_get("sub_items", |model: &mut [YourModelName]| -> Result<rhai::Array, Box<EvalAltResult>> {
let rhai_array = model.sub_items.iter().cloned().map(Dynamic::from).collect::<rhai::Array>();
Ok(rhai_array)
});
```
(Ensure `[YourSubModelName]` is `Clone` and works with `Dynamic::from`).
6. **Setters (Less common if using a full builder pattern, but can be useful):**
* Use `engine.register_set("field_name", |model: &mut [YourModelName], value: FieldType| { model.field = value; Ok(()) });`
* If a setter method in Rust takes an ID (e.g., `set_related_id(id: u32)`), and you want to call it from Rhai with an `i64`:
* You *could* use `adapter_macros::adapt_rhai_i64_input_method!(YourModelType::set_related_id, u32)` if `YourModelType::set_related_id` takes `&mut self, u32`.
* Or, handle manually with `NativeCallContext` if the method signature is more complex or doesn't fit the macro.
7. **Other Custom Methods:**
* Register any other public methods from your Rust struct that should be callable.
* Example: `engine.register_fn("custom_method_name", |model: &mut [YourModelName]| model.custom_method_in_rust());`
8. **Database Interaction (Using actual `OurDB` methods like `set` and `get_by_id`):**
* The `Arc<OurDB>` instance passed to `register_[your_model_name]_rhai_module` should be cloned and *captured* by the closures for DB functions.
* The Rhai script will call these functions without explicitly passing the DB instance (e.g., `set_my_model(my_model_instance);`, `let m = get_my_model_by_id(123);`).
* Ensure proper error handling, converting DB errors and `Option::None` (for getters) to `Box<EvalAltResult::ErrorRuntime(...)`.
* **Example for `set_[your_model_name]`:**
```rust
// In register_[your_model_name]_rhai_module(engine: &mut Engine, db: Arc<OurDB>)
let captured_db_for_set = Arc::clone(&db);
engine.register_fn("set_[your_model_name]",
move |model: [YourModelName]| -> Result<(), Box<EvalAltResult>> {
captured_db_for_set.set(&model).map_err(|e| {
Box::new(EvalAltResult::ErrorRuntime(
format!("Failed to set [YourModelName] (ID: {}): {}", model.base_data.id, e).into(),
Position::NONE,
))
})
});
```
* **Example for `get_[your_model_name]_by_id`:**
```rust
// In register_[your_model_name]_rhai_module(engine: &mut Engine, db: Arc<OurDB>)
let captured_db_for_get = Arc::clone(&db);
engine.register_fn("get_[your_model_name]_by_id",
move |context: NativeCallContext, id_i64: i64| -> Result<[YourModelName], Box<EvalAltResult>> {
let id_u32 = i64_to_u32(id_i64, context.position(), "id", "get_[your_model_name]_by_id")?;
captured_db_for_get.get_by_id(id_u32) // Assumes OurDB directly implements Collection<_, [YourModelName]>
.map_err(|e| Box::new(EvalAltResult::ErrorRuntime(
format!("Error getting [YourModelName] (ID: {}): {}", id_u32, e).into(),
Position::NONE,
)))?
.ok_or_else(|| Box::new(EvalAltResult::ErrorRuntime(
format!("[YourModelName] with ID {} not found", id_u32).into(),
Position::NONE,
)))
});
```
**Example Output Snippet (Illustrating a few parts):**
```rust
// #[...]
// pub struct MyItem { /* ... */ }
// impl MyItem { /* ... */ }
// pub fn register_my_item_rhai_module(engine: &mut Engine, db: Arc<OurDB>) {
// fn i64_to_u32(...) { /* ... */ }
//
// engine.register_fn("new_my_item", |ctx: NativeCallContext, id: i64, name: String| { /* ... */ });
// engine.register_fn("name", |item: MyItem, name: String| -> MyItem { /* ... */ });
// engine.register_get("id", |item: &mut MyItem| Ok(item.base_data.id as i64));
// engine.register_get("sub_elements", |item: &mut MyItem| {
// Ok(item.sub_elements.iter().cloned().map(Dynamic::from).collect::<rhai::Array>())
// });
// // ... etc.
// }
```

401
heromodels/docs/sigsocket_architecture.md Normal file
View File

@@ -0,0 +1,401 @@
# SigSocket: WebSocket Signing Server Architecture
Based on my analysis of the existing Actix application structure, I've designed a comprehensive architecture for implementing a WebSocket server that handles signing operations. This server will integrate seamlessly with the existing hostbasket application.
## 1. Overview
SigSocket will:
- Accept WebSocket connections from clients
- Allow clients to identify themselves with a public key
- Provide a `send_to_sign()` function that takes a public key and a message
- Forward the message to the appropriate client for signing
- Wait for a signed response (with a 1-minute timeout)
- Verify the signature using the client's public key
- Return the response message and signature
## 2. Component Architecture
```mermaid
graph TD
A[Actix Web Server] --> B[SigSocket Manager]
B --> C[Connection Registry]
B --> D[Message Handler]
D --> E[Signature Verifier]
F[Client] <--> B
G[Application Code] --> H[SigSocketService]
H --> B
```
### Key Components:
1. **SigSocket Manager**
- Handles WebSocket connections
- Manages connection lifecycle
- Routes messages to appropriate handlers
2. **Connection Registry**
- Maps public keys to active WebSocket connections
- Handles connection tracking and cleanup
- Provides lookup functionality
3. **Message Handler**
- Processes incoming messages
- Implements the message protocol
- Manages timeouts for responses
4. **Signature Verifier**
- Verifies signatures using public keys
- Implements cryptographic operations
- Ensures security of the signing process
5. **SigSocket Service**
- Provides a clean API for the application to use
- Abstracts WebSocket complexity from business logic
- Handles error cases and timeouts
## 3. Directory Structure
```
src/
├── lib.rs # Main library exports
├── manager.rs # WebSocket connection manager
├── registry.rs # Connection registry
├── handler.rs # Message handling logic
├── protocol.rs # Message protocol definitions
├── crypto.rs # Cryptographic operations
└── service.rs # Service API
```
## 4. Data Flow
```mermaid
sequenceDiagram
participant Client
participant SigSocketManager
participant Registry
participant Application
participant SigSocketService
Client->>SigSocketManager: Connect
Client->>SigSocketManager: Introduce(public_key)
SigSocketManager->>Registry: Register(connection, public_key)
Application->>SigSocketService: send_to_sign(public_key, message)
SigSocketService->>Registry: Lookup(public_key)
Registry-->>SigSocketService: connection
SigSocketService->>SigSocketManager: Send message to connection
SigSocketManager->>Client: Message to sign
Client->>SigSocketManager: Signed response
SigSocketManager->>SigSocketService: Forward response
SigSocketService->>SigSocketService: Verify signature
SigSocketService-->>Application: Return verified response
```
## 5. Message Protocol
We'll define a minimalist protocol for communication:
```
// Client introduction (first message upon connection)
<base64_encoded_public_key>
// Sign request (sent from server to client)
<base64_encoded_message>
// Sign response (sent from client to server)
<base64_encoded_message>.<base64_encoded_signature>
```
This simplified protocol reduces overhead and complexity:
- The introduction is always the first message, containing only the public key
- Sign requests contain only the message to be signed
- Responses use a simple "message.signature" format
## 6. Required Dependencies
We'll need to add the following dependencies to the project:
```toml
# WebSocket support
actix-web-actors = "4.2.0"
# Cryptography
secp256k1 = "0.28.0" # For secp256k1 signatures (used in Bitcoin/Ethereum)
sha2 = "0.10.8" # For hashing before signing
hex = "0.4.3" # For hex encoding/decoding
base64 = "0.21.0" # For base64 encoding/decoding
rand = "0.8.5" # For generating random data
```
## 7. Implementation Details
### 7.1 SigSocket Manager
The SigSocket Manager will handle the lifecycle of WebSocket connections:
```rust
pub struct SigSocketManager {
registry: Arc<RwLock<ConnectionRegistry>>,
}
impl Actor for SigSocketManager {
type Context = ws::WebsocketContext<Self>;
fn started(&mut self, ctx: &mut Self::Context) {
// Handle connection start
}
fn stopped(&mut self, ctx: &mut Self::Context) {
// Handle connection close and cleanup
}
}
impl StreamHandler<Result<ws::Message, ws::ProtocolError>> for SigSocketManager {
fn handle(&mut self, msg: Result<ws::Message, ws::ProtocolError>, ctx: &mut Self::Context) {
// Handle different types of WebSocket messages
}
}
```
### 7.2 Connection Registry
The Connection Registry will maintain a mapping of public keys to active connections:
```rust
pub struct ConnectionRegistry {
connections: HashMap<String, Addr<SigSocketManager>>,
}
impl ConnectionRegistry {
pub fn register(&mut self, public_key: String, addr: Addr<SigSocketManager>) {
self.connections.insert(public_key, addr);
}
pub fn unregister(&mut self, public_key: &str) {
self.connections.remove(public_key);
}
pub fn get(&self, public_key: &str) -> Option<&Addr<SigSocketManager>> {
self.connections.get(public_key)
}
}
```
### 7.3 SigSocket Service
The SigSocket Service will provide a clean API for controllers:
```rust
pub struct SigSocketService {
registry: Arc<RwLock<ConnectionRegistry>>,
}
impl SigSocketService {
pub async fn send_to_sign(&self, public_key: &str, message: &[u8])
-> Result<(Vec<u8>, Vec<u8>), SigSocketError> {
// 1. Find the connection for the public key
let connection = self.registry.read().await.get(public_key).cloned();
if let Some(conn) = connection {
// 2. Create a response channel
let (tx, rx) = oneshot::channel();
// 3. Register the response channel with the connection
self.pending_requests.write().await.insert(conn.clone(), tx);
// 4. Send the message to the client (just the raw message)
conn.do_send(base64::encode(message));
// 5. Wait for the response with a timeout
match tokio::time::timeout(Duration::from_secs(60), rx).await {
Ok(Ok(response)) => {
// 6. Parse the response in format "message.signature"
let parts: Vec<&str> = response.split('.').collect();
if parts.len() != 2 {
return Err(SigSocketError::InvalidResponseFormat);
}
let message_b64 = parts[0];
let signature_b64 = parts[1];
// 7. Decode the message and signature
let message_bytes = match base64::decode(message_b64) {
Ok(m) => m,
Err(_) => return Err(SigSocketError::DecodingError),
};
let signature_bytes = match base64::decode(signature_b64) {
Ok(s) => s,
Err(_) => return Err(SigSocketError::DecodingError),
};
// 8. Verify the signature
if self.verify_signature(&signature_bytes, &message_bytes, public_key) {
Ok((message_bytes, signature_bytes))
} else {
Err(SigSocketError::InvalidSignature)
}
},
Ok(Err(_)) => Err(SigSocketError::ChannelClosed),
Err(_) => Err(SigSocketError::Timeout),
}
} else {
Err(SigSocketError::ConnectionNotFound)
}
}
}
```
## 8. Error Handling
We'll define a comprehensive error type for the SigSocket service:
```rust
#[derive(Debug, thiserror::Error)]
pub enum SigSocketError {
#[error("Connection not found for the provided public key")]
ConnectionNotFound,
#[error("Timeout waiting for signature")]
Timeout,
#[error("Invalid signature")]
InvalidSignature,
#[error("Channel closed unexpectedly")]
ChannelClosed,
#[error("Invalid response format, expected 'message.signature'")]
InvalidResponseFormat,
#[error("Error decoding base64 message or signature")]
DecodingError,
#[error("Invalid public key format")]
InvalidPublicKey,
#[error("Invalid signature format")]
InvalidSignature,
#[error("Internal cryptographic error")]
InternalError,
#[error("WebSocket error: {0}")]
WebSocketError(#[from] ws::ProtocolError),
#[error("Base64 decoding error: {0}")]
Base64Error(#[from] base64::DecodeError),
#[error("Hex decoding error: {0}")]
HexError(#[from] hex::FromHexError),
}
```
## 9. Cryptographic Operations
The SigSocket service uses the secp256k1 elliptic curve for cryptographic operations, which is the same curve used in Bitcoin and Ethereum. This makes it compatible with many blockchain applications and wallets.
```rust
use secp256k1::{Secp256k1, Message, PublicKey, Signature};
use sha2::{Sha256, Digest};
pub fn verify_signature(public_key_hex: &str, message: &[u8], signature_hex: &str) -> Result<bool, SigSocketError> {
// 1. Parse the public key
let public_key_bytes = hex::decode(public_key_hex)
.map_err(|_| SigSocketError::InvalidPublicKey)?;
let public_key = PublicKey::from_slice(&public_key_bytes)
.map_err(|_| SigSocketError::InvalidPublicKey)?;
// 2. Parse the signature
let signature_bytes = hex::decode(signature_hex)
.map_err(|_| SigSocketError::InvalidSignature)?;
let signature = Signature::from_compact(&signature_bytes)
.map_err(|_| SigSocketError::InvalidSignature)?;
// 3. Hash the message (secp256k1 requires a 32-byte hash)
let mut hasher = Sha256::new();
hasher.update(message);
let message_hash = hasher.finalize();
// 4. Create a secp256k1 message from the hash
let secp_message = Message::from_slice(&message_hash)
.map_err(|_| SigSocketError::InternalError)?;
// 5. Verify the signature
let secp = Secp256k1::verification_only();
match secp.verify(&secp_message, &signature, &public_key) {
Ok(_) => Ok(true),
Err(_) => Ok(false),
}
}
```
This implementation:
1. Decodes the hex-encoded public key and signature
2. Hashes the message using SHA-256 (required for secp256k1)
3. Verifies the signature using the secp256k1 library
4. Returns a boolean indicating whether the signature is valid
## 10. Security Considerations
1. **Public Key Validation**: Validate public keys upon connection to ensure they are properly formatted secp256k1 keys
2. **Message Authentication**: Consider adding a nonce or timestamp to prevent replay attacks
3. **Rate Limiting**: Implement rate limiting to prevent DoS attacks
4. **Connection Timeouts**: Automatically close inactive connections
5. **Error Logging**: Log errors but avoid exposing sensitive information
6. **Input Validation**: Validate all inputs to prevent injection attacks
7. **Secure Hashing**: Always hash messages before signing to prevent length extension attacks
## 11. Testing Strategy
1. **Unit Tests**: Test individual components in isolation
2. **Integration Tests**: Test the interaction between components
3. **End-to-End Tests**: Test the complete flow from client connection to signature verification
4. **Load Tests**: Test the system under high load to ensure stability
5. **Security Tests**: Test for common security vulnerabilities
## 12. Integration with Existing Code
The SigSocketService can be used by any part of the application that needs signing functionality:
```rust
// Create and initialize the service
let registry = Arc::new(RwLock::new(ConnectionRegistry::new()));
let sigsocket_service = Arc::new(SigSocketService::new(registry.clone()));
// Use the service to send a message for signing
async fn sign_message(service: Arc<SigSocketService>, public_key: String, message: Vec<u8>) -> Result<(Vec<u8>, Vec<u8>), SigSocketError> {
service.send_to_sign(&public_key, &message).await
}
// Example usage in an HTTP handler
async fn handle_sign_request(
service: web::Data<Arc<SigSocketService>>,
req: web::Json<SignRequest>,
) -> HttpResponse {
match service.send_to_sign(&req.public_key, &req.message).await {
Ok((response, signature)) => {
HttpResponse::Ok().json(json!({
"response": base64::encode(response),
"signature": base64::encode(signature),
}))
},
Err(e) => {
HttpResponse::InternalServerError().json(json!({
"error": e.to_string(),
}))
}
}
}
```
## 13. Deployment Considerations
1. **Scalability**: The SigSocket server should be designed to scale horizontally
2. **Monitoring**: Implement metrics for connection count, message throughput, and error rates
3. **Logging**: Log important events for debugging and auditing
4. **Documentation**: Document the API and protocol for client implementers

365
heromodels/docs/tst_implementation_plan.md Normal file
View File

@@ -0,0 +1,365 @@
# Ternary Search Tree (TST) Implementation Plan
## 1. Overview
A Ternary Search Tree (TST) is a type of trie where each node has three children: left, middle, and right. Unlike a RadixTree which compresses common prefixes, a TST stores one character per node and uses a binary search tree-like structure for efficient traversal.
```mermaid
graph TD
A[Root Node 'r'] --> B[Left Child 'a']
A --> C[Middle Child 'o']
A --> D[Right Child 't']
C --> E[Middle Child 'o']
E --> F[Middle Child 'm' - End of Key]
E --> G[Middle Child 't' - End of Key]
```
The TST implementation will use OurDB as the backend for persistent storage, similar to the existing RadixTree implementation. The goal is to provide a more balanced tree structure that offers consistent performance across all operations (set, get, delete, list).
## 2. Core Data Structures
### 2.1 TST Node Structure
```rust
pub struct TSTNode {
// The character stored at this node
pub character: char,
// Value stored at this node (empty if not end of key)
pub value: Vec<u8>,
// Whether this node represents the end of a key
pub is_end_of_key: bool,
// References to child nodes
pub left_id: Option<u32>, // For characters < current character
pub middle_id: Option<u32>, // For characters == current character (next character in key)
pub right_id: Option<u32>, // For characters > current character
}
```
### 2.2 TST Structure
```rust
pub struct TST {
// Database for persistent storage
db: OurDB,
// Database ID of the root node
root_id: Option<u32>,
}
```
## 3. API Design
The TST will maintain similar core functionality to RadixTree but with an API that better suits its structure:
```rust
impl TST {
// Creates a new TST with the specified database path
pub fn new(path: &str, reset: bool) -> Result<Self, Error>;
// Sets a key-value pair in the tree
pub fn set(&mut self, key: &str, value: Vec<u8>) -> Result<(), Error>;
// Gets a value by key from the tree
pub fn get(&mut self, key: &str) -> Result<Vec<u8>, Error>;
// Deletes a key from the tree
pub fn delete(&mut self, key: &str) -> Result<(), Error>;
// Lists all keys with a given prefix
pub fn list(&mut self, prefix: &str) -> Result<Vec<String>, Error>;
// Gets all values for keys with a given prefix
pub fn getall(&mut self, prefix: &str) -> Result<Vec<Vec<u8>>, Error>;
}
```
## 4. Implementation Strategy
### 4.1 Phase 1: Core Data Structures and Serialization
```mermaid
graph TD
A[Define TSTNode and TST structs] --> B[Implement serialization/deserialization]
B --> C[Implement Error handling]
C --> D[Implement OurDB integration]
```
1. Define the `TSTNode` and `TST` structs
2. Implement serialization and deserialization for `TSTNode`
3. Define error types for TST-specific errors
4. Implement OurDB integration for node storage and retrieval
### 4.2 Phase 2: Basic Tree Operations
```mermaid
graph TD
A[Implement new] --> B[Implement set]
B --> C[Implement get]
C --> D[Implement helper functions]
```
1. Implement the `new()` function for creating a new TST
2. Implement the `set()` function for inserting key-value pairs
3. Implement the `get()` function for retrieving values
4. Implement helper functions for node traversal and manipulation
### 4.3 Phase 3: Advanced Tree Operations
```mermaid
graph TD
A[Implement delete] --> B[Implement list]
B --> C[Implement getall]
C --> D[Optimize operations]
```
1. Implement the `delete()` function for removing keys
2. Implement the `list()` function for prefix-based key listing
3. Implement the `getall()` function for retrieving all values with a prefix
4. Optimize operations for balanced performance
### 4.4 Phase 4: Testing and Performance Evaluation
```mermaid
graph TD
A[Create unit tests] --> B[Create integration tests]
B --> C[Create performance tests]
C --> D[Compare with RadixTree]
D --> E[Optimize based on results]
```
1. Create unit tests for each component
2. Create integration tests for the complete system
3. Create performance tests similar to RadixTree's
4. Compare performance with RadixTree
5. Optimize based on performance results
## 5. Implementation Details
### 5.1 Node Structure and Storage
Each TST node will store a single character and have three child pointers (left, middle, right). The nodes will be serialized and stored in OurDB, with node references using OurDB record IDs.
### 5.2 Key Operations
#### 5.2.1 Insertion (set)
```mermaid
graph TD
A[Start at root] --> B{Root exists?}
B -- No --> C[Create root node]
B -- Yes --> D[Compare current char with node char]
D -- Less than --> E[Go to left child]
D -- Equal to --> F[Go to middle child]
D -- Greater than --> G[Go to right child]
E --> H{Child exists?}
F --> H
G --> H
H -- No --> I[Create new node]
H -- Yes --> J[Continue with next char]
I --> J
J --> K{End of key?}
K -- Yes --> L[Set value and mark as end of key]
K -- No --> D
```
1. Start at the root node
2. For each character in the key:
- If the character is less than the current node's character, go to the left child
- If the character is equal to the current node's character, go to the middle child
- If the character is greater than the current node's character, go to the right child
- If the child doesn't exist, create a new node
3. When the end of the key is reached, set the value and mark the node as end of key
#### 5.2.2 Lookup (get)
1. Start at the root node
2. For each character in the key:
- If the character is less than the current node's character, go to the left child
- If the character is equal to the current node's character, go to the middle child
- If the character is greater than the current node's character, go to the right child
- If the child doesn't exist, the key is not found
3. When the end of the key is reached, check if the node is marked as end of key
- If yes, return the value
- If no, the key is not found
#### 5.2.3 Deletion (delete)
1. Find the node corresponding to the end of the key
2. If the node has no children, remove it and update its parent
3. If the node has children, mark it as not end of key and clear its value
4. Recursively clean up any nodes that are no longer needed
#### 5.2.4 Prefix Operations (list, getall)
1. Find the node corresponding to the end of the prefix
2. Perform a traversal of the subtree rooted at that node
3. Collect all keys (for list) or values (for getall) from nodes marked as end of key
### 5.3 Serialization and OurDB Integration
#### 5.3.1 Node Structure for Serialization
Each TSTNode will be serialized with the following logical structure:
1. Version marker (for future format evolution)
2. Character data
3. Is-end-of-key flag
4. Value (if is-end-of-key is true)
5. Child node references (left, middle, right)
#### 5.3.2 OurDB Integration
The TST will use OurDB for node storage and retrieval:
1. **Node Storage**: Each node will be serialized and stored as a record in OurDB.
```rust
fn save_node(&mut self, node_id: Option<u32>, node: &TSTNode) -> Result<u32, Error> {
let data = node.serialize();
let args = OurDBSetArgs {
id: node_id,
data: &data,
};
Ok(self.db.set(args)?)
}
```
2. **Node Retrieval**: Nodes will be retrieved from OurDB and deserialized.
```rust
fn get_node(&mut self, node_id: u32) -> Result<TSTNode, Error> {
let data = self.db.get(node_id)?;
TSTNode::deserialize(&data)
}
```
3. **Root Node Management**: The TST will maintain a root node ID for traversal.
#### 5.3.3 Handling Large Datasets
For large datasets, we'll implement a batching approach similar to the RadixTree's large-scale tests:
1. **Batch Processing**: Process large datasets in manageable batches to avoid OurDB size limitations.
2. **Database Partitioning**: Create separate database instances for very large datasets.
3. **Memory Management**: Implement efficient memory usage patterns to avoid excessive memory consumption.
## 6. Project Structure
```
tst/
├── Cargo.toml
├── src/
│ ├── lib.rs # Public API and re-exports
│ ├── node.rs # TSTNode implementation
│ ├── serialize.rs # Serialization and deserialization
│ ├── error.rs # Error types
│ └── operations.rs # Tree operations implementation
├── tests/
│ ├── basic_test.rs # Basic operations tests
│ ├── prefix_test.rs # Prefix operations tests
│ └── edge_cases.rs # Edge case tests
└── examples/
├── basic_usage.rs # Basic usage example
├── prefix_ops.rs # Prefix operations example
└── performance.rs # Performance benchmark
```
## 7. Performance Considerations
### 7.1 Advantages of TST over RadixTree
1. **Balanced Structure**: TST naturally maintains a more balanced structure, which can lead to more consistent performance across operations.
2. **Character-by-Character Comparison**: TST performs character-by-character comparisons, which can be more efficient for certain workloads.
3. **Efficient Prefix Operations**: TST can efficiently handle prefix operations by traversing the middle child path.
### 7.2 Potential Optimizations
1. **Node Caching**: Cache frequently accessed nodes to reduce database operations.
2. **Balancing Techniques**: Implement balancing techniques to ensure the tree remains balanced.
3. **Batch Operations**: Support batch operations for improved performance.
4. **Memory Management**: Implement efficient memory usage patterns to avoid excessive memory consumption.
## 8. Testing Strategy
### 8.1 Unit Tests
1. Test `TSTNode` serialization/deserialization
2. Test character comparison operations
3. Test error handling
### 8.2 Integration Tests
1. Test basic CRUD operations
2. Test prefix operations
3. Test edge cases (empty keys, very long keys, etc.)
4. Test with large datasets
### 8.3 Performance Tests
1. Measure throughput for set/get operations
2. Measure latency for different operations
3. Test with different tree sizes and key distributions
4. Compare performance with RadixTree
#### 8.3.1 Performance Benchmarking
We'll create comprehensive benchmarks to compare the TST implementation with RadixTree:
```rust
// Example benchmark structure
fn benchmark_set_operations(tree_type: &str, num_records: usize) -> Duration {
let start_time = Instant::now();
// Create tree (TST or RadixTree)
let mut tree = match tree_type {
"tst" => create_tst(),
"radix" => create_radix_tree(),
_ => panic!("Unknown tree type"),
};
// Insert records
for i in 0..num_records {
let key = format!("key:{:08}", i);
let value = format!("val{}", i).into_bytes();
tree.set(&key, value).unwrap();
}
start_time.elapsed()
}
```
We'll benchmark the following operations:
- Set (insertion)
- Get (lookup)
- Delete
- List (prefix search)
- GetAll (prefix values)
For each operation, we'll measure:
- Throughput (operations per second)
- Latency (time per operation)
- Memory usage
- Database size
We'll test with various dataset characteristics:
- Small datasets (100-1,000 keys)
- Medium datasets (10,000-100,000 keys)
- Large datasets (1,000,000+ keys)
- Keys with common prefixes
- Keys with random distribution
- Long keys vs. short keys
## 9. Timeline and Milestones
1. **Week 1**: Core data structures and serialization
2. **Week 2**: Basic tree operations
3. **Week 3**: Advanced tree operations
4. **Week 4**: Testing and performance evaluation
5. **Week 5**: Optimization and documentation
## 10. Conclusion
This implementation plan provides a roadmap for creating a Ternary Search Tree (TST) as an alternative to the RadixTree implementation. The TST will maintain the same core functionality while providing a more balanced tree structure and aiming for balanced performance across all operations.
The implementation will leverage OurDB for persistent storage, similar to RadixTree, but with a different node structure and traversal algorithm that better suits the TST approach.

451
heromodels/docs/tst_integration_plan.md Normal file
View File

@@ -0,0 +1,451 @@
# TST Integration Plan for HeroDB
## Overview
This document outlines the plan for adding generic functionality to the `herodb/src/db` module to use the Ternary Search Tree (TST) for storing objects with prefixed IDs and implementing a generic list function to retrieve all objects with a specific prefix.
## Current Architecture
Currently:
- Each model has a `db_prefix()` method that returns a string prefix (e.g., "vote" for Vote objects)
- Objects are stored in OurDB with numeric IDs
- The `list()` method in `OurDbStore` is not implemented
## Implementation Plan
### 1. Create a TST-based Index Manager (herodb/src/db/tst_index.rs)
Create a new module that manages TST instances for different model prefixes:
```rust
use crate::db::error::{DbError, DbResult};
use std::path::{Path, PathBuf};
use tst::TST;
/// Manages TST-based indexes for model objects
pub struct TSTIndexManager {
/// Base path for TST databases
base_path: PathBuf,
/// Map of model prefixes to their TST instances
tst_instances: std::collections::HashMap<String, TST>,
}
impl TSTIndexManager {
/// Creates a new TST index manager
pub fn new<P: AsRef<Path>>(base_path: P) -> DbResult<Self> {
let base_path = base_path.as_ref().to_path_buf();
// Create directory if it doesn't exist
std::fs::create_dir_all(&base_path).map_err(DbError::IoError)?;
Ok(Self {
base_path,
tst_instances: std::collections::HashMap::new(),
})
}
/// Gets or creates a TST instance for a model prefix
pub fn get_tst(&mut self, prefix: &str) -> DbResult<&mut TST> {
if !self.tst_instances.contains_key(prefix) {
// Create a new TST instance for this prefix
let tst_path = self.base_path.join(format!("{}_tst", prefix));
let tst_path_str = tst_path.to_string_lossy().to_string();
// Create the TST
let tst = TST::new(&tst_path_str, false)
.map_err(|e| DbError::GeneralError(format!("TST error: {:?}", e)))?;
// Insert it into the map
self.tst_instances.insert(prefix.to_string(), tst);
}
// Return a mutable reference to the TST
Ok(self.tst_instances.get_mut(prefix).unwrap())
}
/// Adds or updates an object in the TST index
pub fn set(&mut self, prefix: &str, id: u32, data: Vec<u8>) -> DbResult<()> {
// Get the TST for this prefix
let tst = self.get_tst(prefix)?;
// Create the key in the format prefix_id
let key = format!("{}_{}", prefix, id);
// Set the key-value pair in the TST
tst.set(&key, data)
.map_err(|e| DbError::GeneralError(format!("TST error: {:?}", e)))?;
Ok(())
}
/// Removes an object from the TST index
pub fn delete(&mut self, prefix: &str, id: u32) -> DbResult<()> {
// Get the TST for this prefix
let tst = self.get_tst(prefix)?;
// Create the key in the format prefix_id
let key = format!("{}_{}", prefix, id);
// Delete the key from the TST
tst.delete(&key)
.map_err(|e| DbError::GeneralError(format!("TST error: {:?}", e)))?;
Ok(())
}
/// Lists all objects with a given prefix
pub fn list(&mut self, prefix: &str) -> DbResult<Vec<(u32, Vec<u8>)>> {
// Get the TST for this prefix
let tst = self.get_tst(prefix)?;
// Get all keys with this prefix
let keys = tst.list(prefix)
.map_err(|e| DbError::GeneralError(format!("TST error: {:?}", e)))?;
// Get all values for these keys
let mut result = Vec::with_capacity(keys.len());
for key in keys {
// Extract the ID from the key (format: prefix_id)
let id_str = key.split('_').nth(1).ok_or_else(|| {
DbError::GeneralError(format!("Invalid key format: {}", key))
})?;
let id = id_str.parse::<u32>().map_err(|_| {
DbError::GeneralError(format!("Invalid ID in key: {}", key))
})?;
// Get the value from the TST
let data = tst.get(&key)
.map_err(|e| DbError::GeneralError(format!("TST error: {:?}", e)))?;
result.push((id, data));
}
Ok(result)
}
}
```
### 2. Update DB Module (herodb/src/db/mod.rs)
Add the new module to the db module:
```rust
pub mod db;
pub mod error;
pub mod macros;
pub mod model;
pub mod model_methods;
pub mod store;
pub mod tst_index; // Add the new module
pub use db::DB;
pub use db::DBBuilder;
pub use error::{DbError, DbResult};
pub use model::Model;
pub use model::Storable;
```
### 3. Modify DB Struct (herodb/src/db/db.rs)
Update the DB struct to include the TST index manager:
```rust
/// Main DB manager that automatically handles all models
#[derive(Clone, CustomType)]
pub struct DB {
db_path: PathBuf,
// Type map for generic operations
type_map: HashMap<TypeId, Arc<RwLock<dyn DbOperations>>>,
// TST index manager
tst_index: Arc<RwLock<TSTIndexManager>>,
// Transaction state
transaction: Arc<RwLock<Option<TransactionState>>>,
}
```
### 4. Extend Transaction Handling
Extend the `DbOperation` enum to include model prefix and ID information:
```rust
#[derive(Debug, Clone)]
enum DbOperation {
Set {
model_type: TypeId,
serialized: Vec<u8>,
model_prefix: String, // Add model prefix
model_id: u32, // Add model ID
},
Delete {
model_type: TypeId,
id: u32,
model_prefix: String, // Add model prefix
},
}
```
### 5. Update Transaction Recording
Modify the `set` and `delete` methods to record model prefix and ID in the transaction:
```rust
pub fn set<T: Model>(&self, model: &T) -> DbResult<()> {
// Try to acquire a write lock on the transaction
let mut tx_guard = self.transaction.write().unwrap();
// Check if there's an active transaction
if let Some(tx_state) = tx_guard.as_mut() {
if tx_state.active {
// Serialize the model for later use
let serialized = model.to_bytes()?;
// Record a Set operation in the transaction with prefix and ID
tx_state.operations.push(DbOperation::Set {
model_type: TypeId::of::<T>(),
serialized: serialized.clone(),
model_prefix: T::db_prefix().to_string(),
model_id: model.get_id(),
});
return Ok(());
}
}
// ... rest of the method ...
}
pub fn delete<T: Model>(&self, id: u32) -> DbResult<()> {
// Try to acquire a write lock on the transaction
let mut tx_guard = self.transaction.write().unwrap();
// Check if there's an active transaction
if let Some(tx_state) = tx_guard.as_mut() {
if tx_state.active {
// Record a Delete operation in the transaction with prefix
tx_state.operations.push(DbOperation::Delete {
model_type: TypeId::of::<T>(),
id,
model_prefix: T::db_prefix().to_string(),
});
return Ok(());
}
}
// ... rest of the method ...
}
```
### 6. Update Transaction Commit
Modify the `commit_transaction` method to update both OurDB and the TST index:
```rust
pub fn commit_transaction(&self) -> DbResult<()> {
let mut tx_guard = self.transaction.write().unwrap();
if let Some(tx_state) = tx_guard.take() {
if !tx_state.active {
return Err(DbError::TransactionError("Transaction not active".into()));
}
// Create a backup of the transaction state in case we need to rollback
let backup = tx_state.clone();
// Try to execute all operations
let result = (|| {
for op in &tx_state.operations {
match op {
DbOperation::Set {
model_type,
serialized,
model_prefix,
model_id,
} => {
// Apply to OurDB
self.apply_set_operation(*model_type, serialized)?;
// Apply to TST index
let mut tst_index = self.tst_index.write().unwrap();
tst_index.set(model_prefix, *model_id, serialized.clone())?;
}
DbOperation::Delete {
model_type,
id,
model_prefix,
} => {
// Apply to OurDB
let db_ops = self
.type_map
.get(model_type)
.ok_or_else(|| DbError::TypeError)?;
let mut db_ops_guard = db_ops.write().unwrap();
db_ops_guard.delete(*id)?;
// Apply to TST index
let mut tst_index = self.tst_index.write().unwrap();
tst_index.delete(model_prefix, *id)?;
}
}
}
Ok(())
})();
// If any operation failed, restore the transaction state
if result.is_err() {
*tx_guard = Some(backup);
return result;
}
Ok(())
} else {
Err(DbError::TransactionError("No active transaction".into()))
}
}
```
### 7. Implement List Method
Implement the `list` method to use the TST's prefix search:
```rust
pub fn list<T: Model>(&self) -> DbResult<Vec<T>> {
// Get the prefix for this model type
let prefix = T::db_prefix();
// Use the TST index to get all objects with this prefix
let mut tst_index = self.tst_index.write().unwrap();
let items = tst_index.list(prefix)?;
// Deserialize the objects
let mut result = Vec::with_capacity(items.len());
for (_, data) in items {
let model = T::from_bytes(&data)?;
result.push(model);
}
Ok(result)
}
```
### 8. Add Recovery Mechanism
Add a method to synchronize the TST index with OurDB in case they get out of sync:
```rust
pub fn synchronize_tst_index<T: Model>(&self) -> DbResult<()> {
// Get all models from OurDB
let models = self.list_from_ourdb::<T>()?;
// Clear the TST index for this model type
let mut tst_index = self.tst_index.write().unwrap();
let prefix = T::db_prefix();
// Rebuild the TST index
for model in models {
let id = model.get_id();
let data = model.to_bytes()?;
tst_index.set(prefix, id, data)?;
}
Ok(())
}
// Helper method to list models directly from OurDB (not using TST)
fn list_from_ourdb<T: Model>(&self) -> DbResult<Vec<T>> {
match self.type_map.get(&TypeId::of::<T>()) {
Some(db_ops) => {
let db_ops_guard = db_ops.read().unwrap();
let result_any = db_ops_guard.list()?;
match result_any.downcast::<Vec<T>>() {
Ok(vec_t) => Ok(*vec_t),
Err(_) => Err(DbError::TypeError),
}
}
None => Err(DbError::TypeError),
}
}
```
## Implementation Flow
```mermaid
sequenceDiagram
participant Client
participant DB
participant TransactionState
participant OurDbStore
participant TSTIndexManager
participant TST
Client->>DB: begin_transaction()
DB->>TransactionState: create new transaction
Client->>DB: set(model)
DB->>TransactionState: record Set operation with prefix and ID
Client->>DB: delete(model)
DB->>TransactionState: record Delete operation with prefix and ID
Client->>DB: commit_transaction()
DB->>TransactionState: get all operations
loop For each operation
alt Set operation
DB->>OurDbStore: apply_set_operation()
DB->>TSTIndexManager: set(prefix, id, data)
TSTIndexManager->>TST: set(key, data)
else Delete operation
DB->>OurDbStore: delete(id)
DB->>TSTIndexManager: delete(prefix, id)
TSTIndexManager->>TST: delete(key)
end
end
alt Success
DB-->>Client: Ok(())
else Error
DB->>TransactionState: restore transaction state
DB-->>Client: Err(error)
end
Client->>DB: list<T>()
DB->>TSTIndexManager: list(prefix)
TSTIndexManager->>TST: list(prefix)
TST-->>TSTIndexManager: keys
TSTIndexManager->>TST: get(key) for each key
TST-->>TSTIndexManager: data
TSTIndexManager-->>DB: (id, data) pairs
DB->>DB: deserialize data to models
DB-->>Client: Vec<T>
```
## Testing Strategy
1. Create unit tests for the TST index manager
2. Test the list functionality with different model types
3. Test transaction handling (commit and rollback)
4. Test error recovery mechanisms
5. Test edge cases (empty database, large datasets)
## Implementation Steps
1. Add TST dependency to herodb/Cargo.toml
2. Create the tst_index.rs module
3. Update the DB module to include the TST index manager
4. Extend the transaction handling
5. Implement the list method
6. Add tests for the new functionality
7. Update documentation
## Considerations
1. **Performance**: The TST operations add overhead to insert/delete operations, but provide efficient list functionality.
2. **Consistency**: The enhanced transaction handling ensures consistency between OurDB and the TST index.
3. **Error Handling**: Proper error handling and recovery mechanisms are essential for maintaining data integrity.
4. **Backward Compatibility**: The implementation should maintain backward compatibility with existing code.