herocode/herolib_rust
12
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

add data packages and remove empty submodule

Browse Source
This commit is contained in:
Timur Gordon
2025-08-07 12:13:37 +02:00
parent ca736d62f3
commit d7562ce466
47 changed files with 8639 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

787
packages/data/radixtree/ARCHITECTURE.md Normal file
View File

@@ -0,0 +1,787 @@
# RadixTree: Architecture for V to Rust Port
## 1. Overview
RadixTree is a space-optimized tree data structure that enables efficient string key operations with persistent storage. This document outlines the architecture for porting the RadixTree module from its original V implementation to Rust, maintaining all existing functionality while leveraging Rust's memory safety, performance, and ecosystem.
The Rust implementation will integrate with the existing OurDB Rust implementation for persistent storage.
```mermaid
graph TD
A[Client Code] --> B[RadixTree API]
B --> C[Node Management]
B --> D[Serialization]
B --> E[Tree Operations]
C --> F[OurDB]
D --> F
E --> C
```
## 2. Current Architecture (V Implementation)
The current V implementation of RadixTree consists of the following components:
### 2.1 Core Data Structures
#### Node
```v
struct Node {
mut:
key_segment string // The segment of the key stored at this node
value []u8 // Value stored at this node (empty if not a leaf)
children []NodeRef // References to child nodes
is_leaf bool // Whether this node is a leaf node
}
```
#### NodeRef
```v
struct NodeRef {
mut:
key_part string // The key segment for this child
node_id u32 // Database ID of the node
}
```
#### RadixTree
```v
@[heap]
pub struct RadixTree {
mut:
db &ourdb.OurDB // Database for persistent storage
root_id u32 // Database ID of the root node
}
```
### 2.2 Key Operations
1. **new()**: Creates a new radix tree with a specified database path
2. **set(key, value)**: Sets a key-value pair in the tree
3. **get(key)**: Retrieves a value by key
4. **update(prefix, new_value)**: Updates the value at a given key prefix
5. **delete(key)**: Removes a key from the tree
6. **list(prefix)**: Lists all keys with a given prefix
7. **getall(prefix)**: Gets all values for keys with a given prefix
### 2.3 Serialization
The V implementation uses a custom binary serialization format for nodes:
- Version byte (1 byte)
- Key segment (string)
- Value length (2 bytes) followed by value bytes
- Children count (2 bytes) followed by children
- Is leaf flag (1 byte)
Each child is serialized as:
- Key part (string)
- Node ID (4 bytes)
### 2.4 Integration with OurDB
The RadixTree uses OurDB for persistent storage:
- Each node is serialized and stored as a record in OurDB
- Node references use OurDB record IDs
- The tree maintains a root node ID for traversal
## 3. Proposed Rust Architecture
The Rust implementation will maintain the same overall architecture while leveraging Rust's type system, ownership model, and error handling.
### 3.1 Core Data Structures
#### Node
```rust
pub struct Node {
key_segment: String,
value: Vec<u8>,
children: Vec<NodeRef>,
is_leaf: bool,
}
```
#### NodeRef
```rust
pub struct NodeRef {
key_part: String,
node_id: u32,
}
```
#### RadixTree
```rust
pub struct RadixTree {
db: ourdb::OurDB,
root_id: u32,
}
```
### 3.2 Public API
```rust
impl RadixTree {
/// Creates a new radix tree with the specified database path
pub fn new(path: &str, reset: bool) -> Result<Self, Error> {
// Implementation
}
/// Sets a key-value pair in the tree
pub fn set(&mut self, key: &str, value: Vec<u8>) -> Result<(), Error> {
// Implementation
}
/// Gets a value by key from the tree
pub fn get(&mut self, key: &str) -> Result<Vec<u8>, Error> {
// Implementation
}
/// Updates the value at a given key prefix
pub fn update(&mut self, prefix: &str, new_value: Vec<u8>) -> Result<(), Error> {
// Implementation
}
/// Deletes a key from the tree
pub fn delete(&mut self, key: &str) -> Result<(), Error> {
// Implementation
}
/// Lists all keys with a given prefix
pub fn list(&mut self, prefix: &str) -> Result<Vec<String>, Error> {
// Implementation
}
/// Gets all values for keys with a given prefix
pub fn getall(&mut self, prefix: &str) -> Result<Vec<Vec<u8>>, Error> {
// Implementation
}
}
```
### 3.3 Error Handling
```rust
#[derive(Debug, thiserror::Error)]
pub enum Error {
#[error("OurDB error: {0}")]
OurDB(#[from] ourdb::Error),
#[error("Key not found: {0}")]
KeyNotFound(String),
#[error("Prefix not found: {0}")]
PrefixNotFound(String),
#[error("Serialization error: {0}")]
Serialization(String),
#[error("Deserialization error: {0}")]
Deserialization(String),
#[error("Invalid operation: {0}")]
InvalidOperation(String),
}
```
### 3.4 Serialization
The Rust implementation will maintain the same binary serialization format for compatibility:
```rust
const VERSION: u8 = 1;
impl Node {
/// Serializes a node to bytes for storage
fn serialize(&self) -> Vec<u8> {
// Implementation
}
/// Deserializes bytes to a node
fn deserialize(data: &[u8]) -> Result<Self, Error> {
// Implementation
}
}
```
### 3.5 Integration with OurDB
The Rust implementation will use the existing OurDB Rust implementation:
```rust
impl RadixTree {
fn get_node(&mut self, node_id: u32) -> Result<Node, Error> {
let data = self.db.get(node_id)?;
Node::deserialize(&data)
}
fn save_node(&mut self, node_id: Option<u32>, node: &Node) -> Result<u32, Error> {
let data = node.serialize();
let args = ourdb::OurDBSetArgs {
id: node_id,
data: &data,
};
Ok(self.db.set(args)?)
}
}
```
## 4. Implementation Strategy
### 4.1 Phase 1: Core Data Structures and Serialization
1. Implement the `Node` and `NodeRef` structs
2. Implement serialization and deserialization functions
3. Implement the `Error` enum for error handling
### 4.2 Phase 2: Basic Tree Operations
1. Implement the `RadixTree` struct with OurDB integration
2. Implement the `new()` function for creating a new tree
3. Implement the `get()` and `set()` functions for basic operations
### 4.3 Phase 3: Advanced Tree Operations
1. Implement the `delete()` function for removing keys
2. Implement the `update()` function for updating values
3. Implement the `list()` and `getall()` functions for prefix operations
### 4.4 Phase 4: Testing and Optimization
1. Port existing tests from V to Rust
2. Add new tests for Rust-specific functionality
3. Benchmark and optimize performance
4. Ensure compatibility with existing RadixTree data
## 5. Implementation Considerations
### 5.1 Memory Management
Leverage Rust's ownership model for safe and efficient memory management:
- Use `String` and `Vec<u8>` for data buffers instead of raw pointers
- Use references and borrows to avoid unnecessary copying
- Implement proper RAII for resource management
### 5.2 Error Handling
Use Rust's `Result` type for comprehensive error handling:
- Define custom error types for RadixTree-specific errors
- Propagate errors using the `?` operator
- Provide detailed error messages
- Implement proper error conversion using the `From` trait
### 5.3 Performance Optimizations
Identify opportunities for performance improvements:
- Use efficient string operations for prefix matching
- Minimize database operations by caching nodes when appropriate
- Use iterators for efficient traversal
- Consider using `Cow<str>` for string operations to avoid unnecessary cloning
### 5.4 Compatibility
Ensure compatibility with the V implementation:
- Maintain the same serialization format
- Ensure identical behavior for all operations
- Support reading existing RadixTree data
## 6. Testing Strategy
### 6.1 Unit Tests
Write comprehensive unit tests for each component:
- Test `Node` serialization/deserialization
- Test string operations (common prefix, etc.)
- Test error handling
### 6.2 Integration Tests
Write integration tests for the complete system:
- Test basic CRUD operations
- Test prefix operations
- Test edge cases (empty keys, very long keys, etc.)
- Test with large datasets
### 6.3 Compatibility Tests
Ensure compatibility with existing RadixTree data:
- Test reading existing V-created RadixTree data
- Test writing data that can be read by the V implementation
### 6.4 Performance Tests
Benchmark performance against the V implementation:
- Measure throughput for set/get operations
- Measure latency for different operations
- Test with different tree sizes and key distributions
## 7. Project Structure
```
radixtree/
├── Cargo.toml
├── src/
│ ├── lib.rs # Public API and re-exports
│ ├── node.rs # Node and NodeRef implementations
│ ├── serialize.rs # Serialization and deserialization
│ ├── error.rs # Error types
│ └── operations.rs # Tree operations implementation
├── tests/
│ ├── basic_test.rs # Basic operations tests
│ ├── prefix_test.rs # Prefix operations tests
│ └── edge_cases.rs # Edge case tests
└── examples/
├── basic.rs # Basic usage example
├── prefix.rs # Prefix operations example
└── performance.rs # Performance benchmark
```
## 8. Dependencies
The Rust implementation will use the following dependencies:
- `ourdb` for persistent storage
- `thiserror` for error handling
- `log` for logging
- `criterion` for benchmarking (dev dependency)
## 9. Compatibility Considerations
To ensure compatibility with the V implementation:
1. Maintain the same serialization format for nodes
2. Ensure identical behavior for all operations
3. Support reading existing RadixTree data
4. Maintain the same performance characteristics
## 10. Future Extensions
Potential future extensions to consider:
1. Async API for non-blocking operations
2. Iterator interface for efficient traversal
3. Batch operations for improved performance
4. Custom serialization formats for specific use cases
5. Compression support for values
6. Concurrency support for parallel operations
## 11. Conclusion
This architecture provides a roadmap for porting RadixTree from V to Rust while maintaining compatibility and leveraging Rust's strengths. The implementation will follow a phased approach, starting with core data structures and gradually building up to the complete system.
The Rust implementation aims to be:
- **Safe**: Leveraging Rust's ownership model for memory safety
- **Fast**: Maintaining or improving performance compared to V
- **Compatible**: Working with existing RadixTree data
- **Extensible**: Providing a foundation for future enhancements
- **Well-tested**: Including comprehensive test coverage
## 12. Implementation Files
### 12.1 Cargo.toml
```toml
[package]
name = "radixtree"
version = "0.1.0"
edition = "2021"
description = "A persistent radix tree implementation using OurDB for storage"
authors = ["OurWorld Team"]
[dependencies]
ourdb = { path = "../ourdb" }
thiserror = "1.0.40"
log = "0.4.17"
[dev-dependencies]
criterion = "0.5.1"
[[bench]]
name = "radixtree_benchmarks"
harness = false
[[example]]
name = "basic_usage"
path = "examples/basic_usage.rs"
[[example]]
name = "prefix_operations"
path = "examples/prefix_operations.rs"
```
### 12.2 src/lib.rs
```rust
//! RadixTree is a space-optimized tree data structure that enables efficient string key operations
//! with persistent storage using OurDB as a backend.
//!
//! This implementation provides a persistent radix tree that can be used for efficient
//! prefix-based key operations, such as auto-complete, routing tables, and more.
mod error;
mod node;
mod operations;
mod serialize;
pub use error::Error;
pub use node::{Node, NodeRef};
use ourdb::{OurDB, OurDBConfig, OurDBSetArgs};
use std::path::PathBuf;
/// RadixTree represents a radix tree data structure with persistent storage.
pub struct RadixTree {
db: OurDB,
root_id: u32,
}
impl RadixTree {
/// Creates a new radix tree with the specified database path.
///
/// # Arguments
///
/// * `path` - The path to the database directory
/// * `reset` - Whether to reset the database if it exists
///
/// # Returns
///
/// A new `RadixTree` instance
///
/// # Errors
///
/// Returns an error if the database cannot be created or opened
pub fn new(path: &str, reset: bool) -> Result<Self, Error> {
// Implementation will go here
unimplemented!()
}
/// Sets a key-value pair in the tree.
///
/// # Arguments
///
/// * `key` - The key to set
/// * `value` - The value to set
///
/// # Errors
///
/// Returns an error if the operation fails
pub fn set(&mut self, key: &str, value: Vec<u8>) -> Result<(), Error> {
// Implementation will go here
unimplemented!()
}
/// Gets a value by key from the tree.
///
/// # Arguments
///
/// * `key` - The key to get
///
/// # Returns
///
/// The value associated with the key
///
/// # Errors
///
/// Returns an error if the key is not found or the operation fails
pub fn get(&mut self, key: &str) -> Result<Vec<u8>, Error> {
// Implementation will go here
unimplemented!()
}
/// Updates the value at a given key prefix.
///
/// # Arguments
///
/// * `prefix` - The key prefix to update
/// * `new_value` - The new value to set
///
/// # Errors
///
/// Returns an error if the prefix is not found or the operation fails
pub fn update(&mut self, prefix: &str, new_value: Vec<u8>) -> Result<(), Error> {
// Implementation will go here
unimplemented!()
}
/// Deletes a key from the tree.
///
/// # Arguments
///
/// * `key` - The key to delete
///
/// # Errors
///
/// Returns an error if the key is not found or the operation fails
pub fn delete(&mut self, key: &str) -> Result<(), Error> {
// Implementation will go here
unimplemented!()
}
/// Lists all keys with a given prefix.
///
/// # Arguments
///
/// * `prefix` - The prefix to search for
///
/// # Returns
///
/// A list of keys that start with the given prefix
///
/// # Errors
///
/// Returns an error if the operation fails
pub fn list(&mut self, prefix: &str) -> Result<Vec<String>, Error> {
// Implementation will go here
unimplemented!()
}
/// Gets all values for keys with a given prefix.
///
/// # Arguments
///
/// * `prefix` - The prefix to search for
///
/// # Returns
///
/// A list of values for keys that start with the given prefix
///
/// # Errors
///
/// Returns an error if the operation fails
pub fn getall(&mut self, prefix: &str) -> Result<Vec<Vec<u8>>, Error> {
// Implementation will go here
unimplemented!()
}
}
```
### 12.3 src/error.rs
```rust
//! Error types for the RadixTree module.
use thiserror::Error;
/// Error type for RadixTree operations.
#[derive(Debug, Error)]
pub enum Error {
/// Error from OurDB operations.
#[error("OurDB error: {0}")]
OurDB(#[from] ourdb::Error),
/// Error when a key is not found.
#[error("Key not found: {0}")]
KeyNotFound(String),
/// Error when a prefix is not found.
#[error("Prefix not found: {0}")]
PrefixNotFound(String),
/// Error during serialization.
#[error("Serialization error: {0}")]
Serialization(String),
/// Error during deserialization.
#[error("Deserialization error: {0}")]
Deserialization(String),
/// Error for invalid operations.
#[error("Invalid operation: {0}")]
InvalidOperation(String),
}
```
### 12.4 src/node.rs
```rust
//! Node types for the RadixTree module.
/// Represents a node in the radix tree.
pub struct Node {
/// The segment of the key stored at this node.
pub key_segment: String,
/// Value stored at this node (empty if not a leaf).
pub value: Vec<u8>,
/// References to child nodes.
pub children: Vec<NodeRef>,
/// Whether this node is a leaf node.
pub is_leaf: bool,
}
/// Reference to a node in the database.
pub struct NodeRef {
/// The key segment for this child.
pub key_part: String,
/// Database ID of the node.
pub node_id: u32,
}
impl Node {
/// Creates a new node.
pub fn new(key_segment: String, value: Vec<u8>, is_leaf: bool) -> Self {
Self {
key_segment,
value,
children: Vec::new(),
is_leaf,
}
}
/// Creates a new root node.
pub fn new_root() -> Self {
Self {
key_segment: String::new(),
value: Vec::new(),
children: Vec::new(),
is_leaf: false,
}
}
}
impl NodeRef {
/// Creates a new node reference.
pub fn new(key_part: String, node_id: u32) -> Self {
Self {
key_part,
node_id,
}
}
}
```
### 12.5 src/serialize.rs
```rust
//! Serialization and deserialization for RadixTree nodes.
use crate::error::Error;
use crate::node::{Node, NodeRef};
/// Current binary format version.
const VERSION: u8 = 1;
impl Node {
/// Serializes a node to bytes for storage.
pub fn serialize(&self) -> Vec<u8> {
// Implementation will go here
unimplemented!()
}
/// Deserializes bytes to a node.
pub fn deserialize(data: &[u8]) -> Result<Self, Error> {
// Implementation will go here
unimplemented!()
}
}
```
### 12.6 src/operations.rs
```rust
//! Implementation of RadixTree operations.
use crate::error::Error;
use crate::node::{Node, NodeRef};
use crate::RadixTree;
impl RadixTree {
/// Helper function to get a node from the database.
pub(crate) fn get_node(&mut self, node_id: u32) -> Result<Node, Error> {
// Implementation will go here
unimplemented!()
}
/// Helper function to save a node to the database.
pub(crate) fn save_node(&mut self, node_id: Option<u32>, node: &Node) -> Result<u32, Error> {
// Implementation will go here
unimplemented!()
}
/// Helper function to find all keys with a given prefix.
fn find_keys_with_prefix(
&mut self,
node_id: u32,
current_path: &str,
prefix: &str,
result: &mut Vec<String>,
) -> Result<(), Error> {
// Implementation will go here
unimplemented!()
}
/// Helper function to recursively collect all keys under a node.
fn collect_all_keys(
&mut self,
node_id: u32,
current_path: &str,
result: &mut Vec<String>,
) -> Result<(), Error> {
// Implementation will go here
unimplemented!()
}
/// Helper function to get the common prefix of two strings.
fn get_common_prefix(a: &str, b: &str) -> String {
// Implementation will go here
unimplemented!()
}
}
```
### 12.7 examples/basic_usage.rs
```rust
//! Basic usage example for RadixTree.
use radixtree::RadixTree;
fn main() -> Result<(), radixtree::Error> {
// Create a temporary directory for the database
let db_path = std::env::temp_dir().join("radixtree_example");
std::fs::create_dir_all(&db_path)?;
println!("Creating radix tree at: {}", db_path.display());
// Create a new radix tree
let mut tree = RadixTree::new(db_path.to_str().unwrap(), true)?;
// Store some data
tree.set("hello", b"world".to_vec())?;
tree.set("help", b"me".to_vec())?;
tree.set("helicopter", b"flying".to_vec())?;
// Retrieve and print the data
let value = tree.get("hello")?;
println!("hello: {}", String::from_utf8_lossy(&value));
// List keys with prefix
let keys = tree.list("hel")?;
println!("Keys with prefix 'hel': {:?}", keys);
// Get all values with prefix
let values = tree.getall("hel")?;
println!("Values with prefix 'hel':");
for (i, value) in values.iter().enumerate() {
println!(" {}: {}", i, String::from_utf8_lossy(value));
}
// Delete a key
tree.delete("help")?;
println!("Deleted 'help'");
// Verify deletion
let keys_after = tree.list("hel")?;
println!("Keys with prefix 'hel' after deletion: {:?}", keys_after);
// Clean up (optional)
if std::env::var("KEEP_DB").is_err() {
std::fs::remove_dir_all(&db_path)?;
println!("Cleaned up database directory");
} else {
println!("Database kept at: {}", db_path.display());
}
Ok(())
}
```

27
packages/data/radixtree/Cargo.toml Normal file
View File

@@ -0,0 +1,27 @@
[package]
name = "radixtree"
version = "0.1.0"
edition = "2021"
description = "A persistent radix tree implementation using OurDB for storage"
authors = ["OurWorld Team"]
[dependencies]
ourdb = { path = "../ourdb" }
thiserror = "1.0.40"
log = "0.4.17"
[dev-dependencies]
criterion = "0.5.1"
tempfile = "3.8.0"
[[bench]]
name = "radixtree_benchmarks"
harness = false
[[example]]
name = "basic_usage"
path = "examples/basic_usage.rs"
[[example]]
name = "prefix_operations"
path = "examples/prefix_operations.rs"

265
packages/data/radixtree/MIGRATION.md Normal file
View File

@@ -0,0 +1,265 @@
# Migration Guide: V to Rust RadixTree
This document provides guidance for migrating from the V implementation of RadixTree to the Rust implementation.
## API Changes
The Rust implementation maintains API compatibility with the V implementation, but with some idiomatic Rust changes:
### V API
```v
// Create a new radix tree
mut rt := radixtree.new(path: '/tmp/radixtree_test', reset: true)!
// Set a key-value pair
rt.set('test', 'value1'.bytes())!
// Get a value by key
value := rt.get('test')!
// Update a value at a prefix
rt.update('prefix', 'new_value'.bytes())!
// Delete a key
rt.delete('test')!
// List keys with a prefix
keys := rt.list('prefix')!
// Get all values with a prefix
values := rt.getall('prefix')!
```
### Rust API
```rust
// Create a new radix tree
let mut tree = RadixTree::new("/tmp/radixtree_test", true)?;
// Set a key-value pair
tree.set("test", b"value1".to_vec())?;
// Get a value by key
let value = tree.get("test")?;
// Update a value at a prefix
tree.update("prefix", b"new_value".to_vec())?;
// Delete a key
tree.delete("test")?;
// List keys with a prefix
let keys = tree.list("prefix")?;
// Get all values with a prefix
let values = tree.getall("prefix")?;
```
## Key Differences
1. **Error Handling**: The Rust implementation uses Rust's `Result` type for error handling, while the V implementation uses V's `!` operator.
2. **String Handling**: The Rust implementation uses Rust's `&str` for string parameters and `String` for string return values, while the V implementation uses V's `string` type.
3. **Binary Data**: The Rust implementation uses Rust's `Vec<u8>` for binary data, while the V implementation uses V's `[]u8` type.
4. **Constructor**: The Rust implementation uses a constructor function with separate parameters, while the V implementation uses a struct with named parameters.
5. **Ownership**: The Rust implementation follows Rust's ownership model, requiring mutable references for methods that modify the tree.
## Data Compatibility
The Rust implementation maintains data compatibility with the V implementation:
- The same serialization format is used for nodes
- The same OurDB storage format is used
- Existing RadixTree data created with the V implementation can be read by the Rust implementation
## Migration Steps
1. **Update Dependencies**: Replace the V RadixTree dependency with the Rust RadixTree dependency in your project.
2. **Update Import Statements**: Replace V import statements with Rust use statements.
```v
// V
import freeflowuniverse.herolib.data.radixtree
```
```rust
// Rust
use radixtree::RadixTree;
```
3. **Update Constructor Calls**: Replace V constructor calls with Rust constructor calls.
```v
// V
mut rt := radixtree.new(path: '/path/to/db', reset: false)!
```
```rust
// Rust
let mut tree = RadixTree::new("/path/to/db", false)?;
```
4. **Update Method Calls**: Replace V method calls with Rust method calls.
```v
// V
rt.set('key', 'value'.bytes())!
```
```rust
// Rust
tree.set("key", b"value".to_vec())?;
```
5. **Update Error Handling**: Replace V error handling with Rust error handling.
```v
// V
if value := rt.get('key') {
println('Found: ${value.bytestr()}')
} else {
println('Error: ${err}')
}
```
```rust
// Rust
match tree.get("key") {
Ok(value) => println!("Found: {}", String::from_utf8_lossy(&value)),
Err(e) => println!("Error: {}", e),
}
```
6. **Update String Conversions**: Replace V string conversions with Rust string conversions.
```v
// V
value.bytestr() // Convert []u8 to string
```
```rust
// Rust
String::from_utf8_lossy(&value) // Convert Vec<u8> to string
```
## Example Migration
### V Code
```v
module main
import freeflowuniverse.herolib.data.radixtree
fn main() {
mut rt := radixtree.new(path: '/tmp/radixtree_test', reset: true) or {
println('Error creating RadixTree: ${err}')
return
}
rt.set('hello', 'world'.bytes()) or {
println('Error setting key: ${err}')
return
}
rt.set('help', 'me'.bytes()) or {
println('Error setting key: ${err}')
return
}
if value := rt.get('hello') {
println('hello: ${value.bytestr()}')
} else {
println('Error getting key: ${err}')
return
}
keys := rt.list('hel') or {
println('Error listing keys: ${err}')
return
}
println('Keys with prefix "hel": ${keys}')
values := rt.getall('hel') or {
println('Error getting all values: ${err}')
return
}
println('Values with prefix "hel":')
for i, value in values {
println(' ${i}: ${value.bytestr()}')
}
rt.delete('help') or {
println('Error deleting key: ${err}')
return
}
println('Deleted "help"')
}
```
### Rust Code
```rust
use radixtree::RadixTree;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let mut tree = RadixTree::new("/tmp/radixtree_test", true)
.map_err(|e| format!("Error creating RadixTree: {}", e))?;
tree.set("hello", b"world".to_vec())
.map_err(|e| format!("Error setting key: {}", e))?;
tree.set("help", b"me".to_vec())
.map_err(|e| format!("Error setting key: {}", e))?;
let value = tree.get("hello")
.map_err(|e| format!("Error getting key: {}", e))?;
println!("hello: {}", String::from_utf8_lossy(&value));
let keys = tree.list("hel")
.map_err(|e| format!("Error listing keys: {}", e))?;
println!("Keys with prefix \"hel\": {:?}", keys);
let values = tree.getall("hel")
.map_err(|e| format!("Error getting all values: {}", e))?;
println!("Values with prefix \"hel\":");
for (i, value) in values.iter().enumerate() {
println!(" {}: {}", i, String::from_utf8_lossy(value));
}
tree.delete("help")
.map_err(|e| format!("Error deleting key: {}", e))?;
println!("Deleted \"help\"");
Ok(())
}
```
## Performance Considerations
The Rust implementation should provide similar or better performance compared to the V implementation. However, there are some considerations:
1. **Memory Usage**: The Rust implementation may have different memory usage patterns due to Rust's ownership model.
2. **Error Handling**: The Rust implementation uses Rust's `Result` type, which may have different performance characteristics compared to V's error handling.
3. **String Handling**: The Rust implementation uses Rust's string types, which may have different performance characteristics compared to V's string types.
## Troubleshooting
If you encounter issues during migration, check the following:
1. **Data Compatibility**: Ensure that the data format is compatible between the V and Rust implementations.
2. **API Usage**: Ensure that you're using the correct API for the Rust implementation.
3. **Error Handling**: Ensure that you're handling errors correctly in the Rust implementation.
4. **String Encoding**: Ensure that string encoding is consistent between the V and Rust implementations.
If you encounter any issues that are not covered in this guide, please report them to the project maintainers.

189
packages/data/radixtree/README.md Normal file
View File

@@ -0,0 +1,189 @@
# RadixTree
A persistent radix tree implementation in Rust using OurDB for storage.
## Overview
RadixTree is a space-optimized tree data structure that enables efficient string key operations with persistent storage. This implementation provides a persistent radix tree that can be used for efficient prefix-based key operations, such as auto-complete, routing tables, and more.
A radix tree (also known as a patricia trie or radix trie) is a space-optimized tree data structure that enables efficient string key operations. Unlike a standard trie where each node represents a single character, a radix tree compresses paths by allowing nodes to represent multiple characters (key segments).
Key characteristics:
- Each node stores a segment of a key (not just a single character)
- Nodes can have multiple children, each representing a different branch
- Leaf nodes contain the actual values
- Optimizes storage by compressing common prefixes
## Features
- Efficient prefix-based key operations
- Persistent storage using OurDB backend
- Memory-efficient storage of strings with common prefixes
- Support for binary values
- Thread-safe operations through OurDB
## Usage
Add the dependency to your `Cargo.toml`:
```toml
[dependencies]
radixtree = { path = "../radixtree" }
```
### Basic Example
```rust
use radixtree::RadixTree;
fn main() -> Result<(), radixtree::Error> {
// Create a new radix tree
let mut tree = RadixTree::new("/tmp/radix", false)?;
// Set key-value pairs
tree.set("hello", b"world".to_vec())?;
tree.set("help", b"me".to_vec())?;
// Get values by key
let value = tree.get("hello")?;
println!("hello: {}", String::from_utf8_lossy(&value)); // Prints: world
// List keys by prefix
let keys = tree.list("hel")?; // Returns ["hello", "help"]
println!("Keys with prefix 'hel': {:?}", keys);
// Get all values by prefix
let values = tree.getall("hel")?; // Returns [b"world", b"me"]
// Delete keys
tree.delete("help")?;
Ok(())
}
```
## API
### Creating a RadixTree
```rust
// Create a new radix tree
let mut tree = RadixTree::new("/tmp/radix", false)?;
// Create a new radix tree and reset if it exists
let mut tree = RadixTree::new("/tmp/radix", true)?;
```
### Setting Values
```rust
// Set a key-value pair
tree.set("key", b"value".to_vec())?;
```
### Getting Values
```rust
// Get a value by key
let value = tree.get("key")?;
```
### Updating Values
```rust
// Update a value at a given prefix
tree.update("prefix", b"new_value".to_vec())?;
```
### Deleting Keys
```rust
// Delete a key
tree.delete("key")?;
```
### Listing Keys by Prefix
```rust
// List all keys with a given prefix
let keys = tree.list("prefix")?;
```
### Getting All Values by Prefix
```rust
// Get all values for keys with a given prefix
let values = tree.getall("prefix")?;
```
## Performance Characteristics
- Search: O(k) where k is the key length
- Insert: O(k) for new keys, may require node splitting
- Delete: O(k) plus potential node cleanup
- Space: O(n) where n is the total length of all keys
## Use Cases
RadixTree is particularly useful for:
- Prefix-based searching
- IP routing tables
- Dictionary implementations
- Auto-complete systems
- File system paths
- Any application requiring efficient string key operations with persistence
## Implementation Details
The RadixTree implementation uses OurDB for persistent storage:
- Each node is serialized and stored as a record in OurDB
- Node references use OurDB record IDs
- The tree maintains a root node ID for traversal
- Node serialization includes version tracking for format evolution
For more detailed information about the implementation, see the [ARCHITECTURE.md](./ARCHITECTURE.md) file.
## Running Tests
The project includes a comprehensive test suite that verifies all functionality:
```bash
# Run all tests
cargo test
# Run specific test file
cargo test --test basic_test
cargo test --test prefix_test
cargo test --test getall_test
cargo test --test serialize_test
```
## Running Examples
The project includes example applications that demonstrate how to use the RadixTree:
```bash
# Run the basic usage example
cargo run --example basic_usage
# Run the prefix operations example
cargo run --example prefix_operations
```
## Benchmarking
The project includes benchmarks to measure performance:
```bash
# Run all benchmarks
cargo bench
# Run specific benchmark
cargo bench -- set
cargo bench -- get
cargo bench -- prefix_operations
```
## License
This project is licensed under the same license as the HeroCode project.

141
packages/data/radixtree/benches/radixtree_benchmarks.rs Normal file
View File

@@ -0,0 +1,141 @@
use criterion::{black_box, criterion_group, criterion_main, Criterion};
use radixtree::RadixTree;
use std::path::PathBuf;
use tempfile::tempdir;
fn criterion_benchmark(c: &mut Criterion) {
// Create a temporary directory for benchmarks
let temp_dir = tempdir().expect("Failed to create temp directory");
let db_path = temp_dir.path().to_str().unwrap();
// Benchmark set operation
c.bench_function("set", |b| {
let mut tree = RadixTree::new(db_path, true).unwrap();
let mut i = 0;
b.iter(|| {
let key = format!("benchmark_key_{}", i);
let value = format!("benchmark_value_{}", i).into_bytes();
tree.set(&key, value).unwrap();
i += 1;
});
});
// Setup tree with data for get/list/delete benchmarks
let mut setup_tree = RadixTree::new(db_path, true).unwrap();
for i in 0..1000 {
let key = format!("benchmark_key_{}", i);
let value = format!("benchmark_value_{}", i).into_bytes();
setup_tree.set(&key, value).unwrap();
}
// Benchmark get operation
c.bench_function("get", |b| {
let mut tree = RadixTree::new(db_path, false).unwrap();
let mut i = 0;
b.iter(|| {
let key = format!("benchmark_key_{}", i % 1000);
let _value = tree.get(&key).unwrap();
i += 1;
});
});
// Benchmark list operation
c.bench_function("list", |b| {
let mut tree = RadixTree::new(db_path, false).unwrap();
b.iter(|| {
let _keys = tree.list("benchmark_key_1").unwrap();
});
});
// Benchmark getall operation
c.bench_function("getall", |b| {
let mut tree = RadixTree::new(db_path, false).unwrap();
b.iter(|| {
let _values = tree.getall("benchmark_key_1").unwrap();
});
});
// Benchmark update operation
c.bench_function("update", |b| {
let mut tree = RadixTree::new(db_path, false).unwrap();
let mut i = 0;
b.iter(|| {
let key = format!("benchmark_key_{}", i % 1000);
let new_value = format!("updated_value_{}", i).into_bytes();
tree.update(&key, new_value).unwrap();
i += 1;
});
});
// Benchmark delete operation
c.bench_function("delete", |b| {
// Create a fresh tree for deletion benchmarks
let delete_dir = tempdir().expect("Failed to create temp directory");
let delete_path = delete_dir.path().to_str().unwrap();
let mut tree = RadixTree::new(delete_path, true).unwrap();
// Setup keys to delete
for i in 0..1000 {
let key = format!("delete_key_{}", i);
let value = format!("delete_value_{}", i).into_bytes();
tree.set(&key, value).unwrap();
}
let mut i = 0;
b.iter(|| {
let key = format!("delete_key_{}", i % 1000);
// Only try to delete if it exists
if tree.get(&key).is_ok() {
tree.delete(&key).unwrap();
}
i += 1;
});
});
// Benchmark prefix operations with varying tree sizes
let mut group = c.benchmark_group("prefix_operations");
for &size in &[100, 1000, 10000] {
// Create a fresh tree for each size
let size_dir = tempdir().expect("Failed to create temp directory");
let size_path = size_dir.path().to_str().unwrap();
let mut tree = RadixTree::new(size_path, true).unwrap();
// Insert data with common prefixes
for i in 0..size {
let prefix = match i % 5 {
0 => "user",
1 => "post",
2 => "comment",
3 => "product",
_ => "category",
};
let key = format!("{}_{}", prefix, i);
let value = format!("value_{}", i).into_bytes();
tree.set(&key, value).unwrap();
}
// Benchmark list operation for this size
group.bench_function(format!("list_size_{}", size), |b| {
b.iter(|| {
for prefix in &["user", "post", "comment", "product", "category"] {
let _keys = tree.list(prefix).unwrap();
}
});
});
// Benchmark getall operation for this size
group.bench_function(format!("getall_size_{}", size), |b| {
b.iter(|| {
for prefix in &["user", "post", "comment", "product", "category"] {
let _values = tree.getall(prefix).unwrap();
}
});
});
}
group.finish();
}
criterion_group!(benches, criterion_benchmark);
criterion_main!(benches);

51
packages/data/radixtree/examples/basic_usage.rs Normal file
View File

@@ -0,0 +1,51 @@
use radixtree::RadixTree;
use std::path::PathBuf;
fn main() -> Result<(), radixtree::Error> {
// Create a temporary directory for the database
let db_path = std::env::temp_dir().join("radixtree_example");
std::fs::create_dir_all(&db_path)?;
println!("Creating radix tree at: {}", db_path.display());
// Create a new radix tree
let mut tree = RadixTree::new(db_path.to_str().unwrap(), true)?;
// Store some data
println!("Storing data...");
tree.set("hello", b"world".to_vec())?;
tree.set("help", b"me".to_vec())?;
tree.set("helicopter", b"flying".to_vec())?;
// Retrieve and print the data
let value = tree.get("hello")?;
println!("hello: {}", String::from_utf8_lossy(&value));
// Update a value
println!("Updating value...");
tree.update("hello", b"updated world".to_vec())?;
// Retrieve the updated value
let updated_value = tree.get("hello")?;
println!("hello (updated): {}", String::from_utf8_lossy(&updated_value));
// Delete a key
println!("Deleting 'help'...");
tree.delete("help")?;
// Try to retrieve the deleted key (should fail)
match tree.get("help") {
Ok(value) => println!("Unexpected: help still exists with value: {}", String::from_utf8_lossy(&value)),
Err(e) => println!("As expected, help was deleted: {}", e),
}
// Clean up (optional)
if std::env::var("KEEP_DB").is_err() {
std::fs::remove_dir_all(&db_path)?;
println!("Cleaned up database directory");
} else {
println!("Database kept at: {}", db_path.display());
}
Ok(())
}

121
packages/data/radixtree/examples/large_scale_test.rs Normal file
View File

@@ -0,0 +1,121 @@
use radixtree::RadixTree;
use std::time::{Duration, Instant};
use std::io::{self, Write};
// Use much smaller batches to avoid hitting OurDB's size limit
const BATCH_SIZE: usize = 1_000;
const NUM_BATCHES: usize = 1_000; // Total records: 1,000,000
const PROGRESS_INTERVAL: usize = 100;
fn main() -> Result<(), radixtree::Error> {
// Overall metrics
let total_start_time = Instant::now();
let mut total_records_inserted = 0;
let mut batch_times = Vec::with_capacity(NUM_BATCHES);
println!("Will insert up to {} records in batches of {}",
BATCH_SIZE * NUM_BATCHES, BATCH_SIZE);
// Process in batches to avoid OurDB size limits
for batch in 0..NUM_BATCHES {
// Create a new database for each batch
let batch_path = std::env::temp_dir().join(format!("radixtree_batch_{}", batch));
// Clean up any existing database
if batch_path.exists() {
std::fs::remove_dir_all(&batch_path)?;
}
std::fs::create_dir_all(&batch_path)?;
println!("\nBatch {}/{}: Creating new radix tree...", batch + 1, NUM_BATCHES);
let mut tree = RadixTree::new(batch_path.to_str().unwrap(), true)?;
let batch_start_time = Instant::now();
let mut last_progress_time = Instant::now();
let mut last_progress_count = 0;
// Insert records for this batch
for i in 0..BATCH_SIZE {
let global_index = batch * BATCH_SIZE + i;
let key = format!("key:{:08}", global_index);
let value = format!("val{}", global_index).into_bytes();
tree.set(&key, value)?;
// Show progress at intervals
if (i + 1) % PROGRESS_INTERVAL == 0 || i == BATCH_SIZE - 1 {
let records_since_last = i + 1 - last_progress_count;
let time_since_last = last_progress_time.elapsed();
let records_per_second = records_since_last as f64 / time_since_last.as_secs_f64();
print!("\rProgress: {}/{} records ({:.2}%) - {:.2} records/sec",
i + 1, BATCH_SIZE,
(i + 1) as f64 / BATCH_SIZE as f64 * 100.0,
records_per_second);
io::stdout().flush().unwrap();
last_progress_time = Instant::now();
last_progress_count = i + 1;
}
}
let batch_duration = batch_start_time.elapsed();
batch_times.push(batch_duration);
total_records_inserted += BATCH_SIZE;
println!("\nBatch {}/{} completed in {:?} ({:.2} records/sec)",
batch + 1, NUM_BATCHES,
batch_duration,
BATCH_SIZE as f64 / batch_duration.as_secs_f64());
// Test random access performance for this batch
println!("Testing access performance for batch {}...", batch + 1);
let mut total_get_time = Duration::new(0, 0);
let num_samples = 100;
// Use a simple distribution pattern
for i in 0..num_samples {
// Distribute samples across the batch
let sample_id = batch * BATCH_SIZE + (i * (BATCH_SIZE / num_samples));
let key = format!("key:{:08}", sample_id);
let get_start = Instant::now();
let _ = tree.get(&key)?;
total_get_time += get_start.elapsed();
}
println!("Average time to retrieve a record: {:?}",
total_get_time / num_samples as u32);
// Test prefix search performance
println!("Testing prefix search performance...");
let prefix = format!("key:{:02}", batch % 100);
let list_start = Instant::now();
let keys = tree.list(&prefix)?;
let list_duration = list_start.elapsed();
println!("Found {} keys with prefix '{}' in {:?}",
keys.len(), prefix, list_duration);
}
// Overall performance summary
let total_duration = total_start_time.elapsed();
println!("\n\nPerformance Summary:");
println!("Total time to insert {} records: {:?}", total_records_inserted, total_duration);
println!("Average insertion rate: {:.2} records/second",
total_records_inserted as f64 / total_duration.as_secs_f64());
// Show performance trend
println!("\nPerformance Trend (batch number vs. time):");
for (i, duration) in batch_times.iter().enumerate() {
if i % 10 == 0 || i == batch_times.len() - 1 { // Only show every 10th point
println!(" Batch {}: {:?} ({:.2} records/sec)",
i + 1,
duration,
BATCH_SIZE as f64 / duration.as_secs_f64());
}
}
Ok(())
}

134
packages/data/radixtree/examples/performance_test.rs Normal file
View File

@@ -0,0 +1,134 @@
use radixtree::RadixTree;
use std::time::{Duration, Instant};
use std::io::{self, Write};
// Number of records to insert
const TOTAL_RECORDS: usize = 1_000_000;
// How often to report progress (every X records)
const PROGRESS_INTERVAL: usize = 10_000;
// How many records to use for performance sampling
const PERFORMANCE_SAMPLE_SIZE: usize = 1000;
fn main() -> Result<(), radixtree::Error> {
// Create a temporary directory for the database
let db_path = std::env::temp_dir().join("radixtree_performance_test");
// Completely remove and recreate the directory to ensure a clean start
if db_path.exists() {
std::fs::remove_dir_all(&db_path)?;
}
std::fs::create_dir_all(&db_path)?;
println!("Creating radix tree at: {}", db_path.display());
println!("Will insert {} records and show progress...", TOTAL_RECORDS);
// Create a new radix tree
let mut tree = RadixTree::new(db_path.to_str().unwrap(), true)?;
// Track overall time
let start_time = Instant::now();
// Track performance metrics
let mut insertion_times = Vec::with_capacity(TOTAL_RECORDS / PROGRESS_INTERVAL);
let mut last_batch_time = Instant::now();
let mut last_batch_records = 0;
// Insert records and track progress
for i in 0..TOTAL_RECORDS {
let key = format!("key:{:08}", i);
// Use smaller values to avoid exceeding OurDB's size limit
let value = format!("val{}", i).into_bytes();
// Time the insertion of every Nth record for performance sampling
if i % PERFORMANCE_SAMPLE_SIZE == 0 {
let insert_start = Instant::now();
tree.set(&key, value)?;
let insert_duration = insert_start.elapsed();
// Only print detailed timing for specific samples to avoid flooding output
if i % (PERFORMANCE_SAMPLE_SIZE * 10) == 0 {
println!("Record {}: Insertion took {:?}", i, insert_duration);
}
} else {
tree.set(&key, value)?;
}
// Show progress at intervals
if (i + 1) % PROGRESS_INTERVAL == 0 || i == TOTAL_RECORDS - 1 {
let records_in_batch = i + 1 - last_batch_records;
let batch_duration = last_batch_time.elapsed();
let records_per_second = records_in_batch as f64 / batch_duration.as_secs_f64();
insertion_times.push((i + 1, batch_duration));
print!("\rProgress: {}/{} records ({:.2}%) - {:.2} records/sec",
i + 1, TOTAL_RECORDS,
(i + 1) as f64 / TOTAL_RECORDS as f64 * 100.0,
records_per_second);
io::stdout().flush().unwrap();
last_batch_time = Instant::now();
last_batch_records = i + 1;
}
}
let total_duration = start_time.elapsed();
println!("\n\nPerformance Summary:");
println!("Total time to insert {} records: {:?}", TOTAL_RECORDS, total_duration);
println!("Average insertion rate: {:.2} records/second",
TOTAL_RECORDS as f64 / total_duration.as_secs_f64());
// Show performance trend
println!("\nPerformance Trend (records inserted vs. time per batch):");
for (i, (record_count, duration)) in insertion_times.iter().enumerate() {
if i % 10 == 0 || i == insertion_times.len() - 1 { // Only show every 10th point to avoid too much output
println!(" After {} records: {:?} for {} records ({:.2} records/sec)",
record_count,
duration,
PROGRESS_INTERVAL,
PROGRESS_INTERVAL as f64 / duration.as_secs_f64());
}
}
// Test access performance with distributed samples
println!("\nTesting access performance with distributed samples...");
let mut total_get_time = Duration::new(0, 0);
let num_samples = 1000;
// Use a simple distribution pattern instead of random
for i in 0..num_samples {
// Distribute samples across the entire range
let sample_id = (i * (TOTAL_RECORDS / num_samples)) % TOTAL_RECORDS;
let key = format!("key:{:08}", sample_id);
let get_start = Instant::now();
let _ = tree.get(&key)?;
total_get_time += get_start.elapsed();
}
println!("Average time to retrieve a record: {:?}",
total_get_time / num_samples as u32);
// Test prefix search performance
println!("\nTesting prefix search performance...");
let prefixes = ["key:0", "key:1", "key:5", "key:9"];
for prefix in &prefixes {
let list_start = Instant::now();
let keys = tree.list(prefix)?;
let list_duration = list_start.elapsed();
println!("Found {} keys with prefix '{}' in {:?}",
keys.len(), prefix, list_duration);
}
// Clean up (optional)
if std::env::var("KEEP_DB").is_err() {
std::fs::remove_dir_all(&db_path)?;
println!("\nCleaned up database directory");
} else {
println!("\nDatabase kept at: {}", db_path.display());
}
Ok(())
}

97
packages/data/radixtree/examples/prefix_operations.rs Normal file
View File

@@ -0,0 +1,97 @@
use radixtree::RadixTree;
use std::path::PathBuf;
fn main() -> Result<(), radixtree::Error> {
// Create a temporary directory for the database
let db_path = std::env::temp_dir().join("radixtree_prefix_example");
std::fs::create_dir_all(&db_path)?;
println!("Creating radix tree at: {}", db_path.display());
// Create a new radix tree
let mut tree = RadixTree::new(db_path.to_str().unwrap(), true)?;
// Store data with common prefixes
println!("Storing data with common prefixes...");
// User data
tree.set("user:1:name", b"Alice".to_vec())?;
tree.set("user:1:email", b"alice@example.com".to_vec())?;
tree.set("user:2:name", b"Bob".to_vec())?;
tree.set("user:2:email", b"bob@example.com".to_vec())?;
// Post data
tree.set("post:1:title", b"First Post".to_vec())?;
tree.set("post:1:content", b"Hello World!".to_vec())?;
tree.set("post:2:title", b"Second Post".to_vec())?;
tree.set("post:2:content", b"Another post content".to_vec())?;
// Demonstrate listing keys with a prefix
println!("\nListing keys with prefix 'user:1:'");
let user1_keys = tree.list("user:1:")?;
for key in &user1_keys {
println!(" Key: {}", key);
}
println!("\nListing keys with prefix 'post:'");
let post_keys = tree.list("post:")?;
for key in &post_keys {
println!(" Key: {}", key);
}
// Demonstrate getting all values with a prefix
println!("\nGetting all values with prefix 'user:1:'");
let user1_values = tree.getall("user:1:")?;
for (i, value) in user1_values.iter().enumerate() {
println!(" Value {}: {}", i + 1, String::from_utf8_lossy(value));
}
// Demonstrate finding all user names
println!("\nFinding all user names (prefix 'user:*:name')");
let mut user_names = Vec::new();
let all_keys = tree.list("user:")?;
for key in all_keys {
if key.ends_with(":name") {
if let Ok(value) = tree.get(&key) {
user_names.push((key, String::from_utf8_lossy(&value).to_string()));
}
}
}
for (key, name) in user_names {
println!(" {}: {}", key, name);
}
// Demonstrate updating values with a specific prefix
println!("\nUpdating all post titles...");
let post_title_keys = tree.list("post:")?.into_iter().filter(|k| k.ends_with(":title")).collect::<Vec<_>>();
for key in post_title_keys {
let old_value = tree.get(&key)?;
let old_title = String::from_utf8_lossy(&old_value);
let new_title = format!("UPDATED: {}", old_title);
println!(" Updating '{}' to '{}'", old_title, new_title);
tree.update(&key, new_title.as_bytes().to_vec())?;
}
// Verify updates
println!("\nVerifying updates:");
let post_keys = tree.list("post:")?;
for key in post_keys {
if key.ends_with(":title") {
let value = tree.get(&key)?;
println!(" {}: {}", key, String::from_utf8_lossy(&value));
}
}
// Clean up (optional)
if std::env::var("KEEP_DB").is_err() {
std::fs::remove_dir_all(&db_path)?;
println!("\nCleaned up database directory");
} else {
println!("\nDatabase kept at: {}", db_path.display());
}
Ok(())
}

35
packages/data/radixtree/src/error.rs Normal file
View File

@@ -0,0 +1,35 @@
//! Error types for the RadixTree module.
use thiserror::Error;
/// Error type for RadixTree operations.
#[derive(Debug, Error)]
pub enum Error {
/// Error from OurDB operations.
#[error("OurDB error: {0}")]
OurDB(#[from] ourdb::Error),
/// Error when a key is not found.
#[error("Key not found: {0}")]
KeyNotFound(String),
/// Error when a prefix is not found.
#[error("Prefix not found: {0}")]
PrefixNotFound(String),
/// Error during serialization.
#[error("Serialization error: {0}")]
Serialization(String),
/// Error during deserialization.
#[error("Deserialization error: {0}")]
Deserialization(String),
/// Error for invalid operations.
#[error("Invalid operation: {0}")]
InvalidOperation(String),
/// Error for I/O operations.
#[error("I/O error: {0}")]
IO(#[from] std::io::Error),
}

133
packages/data/radixtree/src/lib.rs Normal file
View File

@@ -0,0 +1,133 @@
//! RadixTree is a space-optimized tree data structure that enables efficient string key operations
//! with persistent storage using OurDB as a backend.
//!
//! This implementation provides a persistent radix tree that can be used for efficient
//! prefix-based key operations, such as auto-complete, routing tables, and more.
mod error;
mod node;
mod operations;
mod serialize;
pub use error::Error;
pub use node::{Node, NodeRef};
use ourdb::OurDB;
/// RadixTree represents a radix tree data structure with persistent storage.
pub struct RadixTree {
db: OurDB,
root_id: u32,
}
impl RadixTree {
/// Creates a new radix tree with the specified database path.
///
/// # Arguments
///
/// * `path` - The path to the database directory
/// * `reset` - Whether to reset the database if it exists
///
/// # Returns
///
/// A new `RadixTree` instance
///
/// # Errors
///
/// Returns an error if the database cannot be created or opened
pub fn new(path: &str, reset: bool) -> Result<Self, Error> {
operations::new_radix_tree(path, reset)
}
/// Sets a key-value pair in the tree.
///
/// # Arguments
///
/// * `key` - The key to set
/// * `value` - The value to set
///
/// # Errors
///
/// Returns an error if the operation fails
pub fn set(&mut self, key: &str, value: Vec<u8>) -> Result<(), Error> {
operations::set(self, key, value)
}
/// Gets a value by key from the tree.
///
/// # Arguments
///
/// * `key` - The key to get
///
/// # Returns
///
/// The value associated with the key
///
/// # Errors
///
/// Returns an error if the key is not found or the operation fails
pub fn get(&mut self, key: &str) -> Result<Vec<u8>, Error> {
operations::get(self, key)
}
/// Updates the value at a given key prefix.
///
/// # Arguments
///
/// * `prefix` - The key prefix to update
/// * `new_value` - The new value to set
///
/// # Errors
///
/// Returns an error if the prefix is not found or the operation fails
pub fn update(&mut self, prefix: &str, new_value: Vec<u8>) -> Result<(), Error> {
operations::update(self, prefix, new_value)
}
/// Deletes a key from the tree.
///
/// # Arguments
///
/// * `key` - The key to delete
///
/// # Errors
///
/// Returns an error if the key is not found or the operation fails
pub fn delete(&mut self, key: &str) -> Result<(), Error> {
operations::delete(self, key)
}
/// Lists all keys with a given prefix.
///
/// # Arguments
///
/// * `prefix` - The prefix to search for
///
/// # Returns
///
/// A list of keys that start with the given prefix
///
/// # Errors
///
/// Returns an error if the operation fails
pub fn list(&mut self, prefix: &str) -> Result<Vec<String>, Error> {
operations::list(self, prefix)
}
/// Gets all values for keys with a given prefix.
///
/// # Arguments
///
/// * `prefix` - The prefix to search for
///
/// # Returns
///
/// A list of values for keys that start with the given prefix
///
/// # Errors
///
/// Returns an error if the operation fails
pub fn getall(&mut self, prefix: &str) -> Result<Vec<Vec<u8>>, Error> {
operations::getall(self, prefix)
}
}

59
packages/data/radixtree/src/node.rs Normal file
View File

@@ -0,0 +1,59 @@
//! Node types for the RadixTree module.
/// Represents a node in the radix tree.
#[derive(Debug, Clone, PartialEq)]
pub struct Node {
/// The segment of the key stored at this node.
pub key_segment: String,
/// Value stored at this node (empty if not a leaf).
pub value: Vec<u8>,
/// References to child nodes.
pub children: Vec<NodeRef>,
/// Whether this node is a leaf node.
pub is_leaf: bool,
}
/// Reference to a node in the database.
#[derive(Debug, Clone, PartialEq)]
pub struct NodeRef {
/// The key segment for this child.
pub key_part: String,
/// Database ID of the node.
pub node_id: u32,
}
impl Node {
/// Creates a new node.
pub fn new(key_segment: String, value: Vec<u8>, is_leaf: bool) -> Self {
Self {
key_segment,
value,
children: Vec::new(),
is_leaf,
}
}
/// Creates a new root node.
pub fn new_root() -> Self {
Self {
key_segment: String::new(),
value: Vec::new(),
children: Vec::new(),
is_leaf: false,
}
}
}
impl NodeRef {
/// Creates a new node reference.
pub fn new(key_part: String, node_id: u32) -> Self {
Self {
key_part,
node_id,
}
}
}

508
packages/data/radixtree/src/operations.rs Normal file
View File

@@ -0,0 +1,508 @@
//! Implementation of RadixTree operations.
use crate::error::Error;
use crate::node::{Node, NodeRef};
use crate::RadixTree;
use crate::serialize::get_common_prefix;
use ourdb::{OurDB, OurDBConfig, OurDBSetArgs};
use std::path::PathBuf;
/// Creates a new radix tree with the specified database path.
pub fn new_radix_tree(path: &str, reset: bool) -> Result<RadixTree, Error> {
let config = OurDBConfig {
path: PathBuf::from(path),
incremental_mode: true,
file_size: Some(1024 * 1024 * 10), // 10MB file size for better performance with large datasets
keysize: Some(6), // Use keysize=6 to support multiple files (file_nr + position)
reset: None, // Don't reset existing database
};
let mut db = OurDB::new(config)?;
// If reset is true, we would clear the database
// Since OurDB doesn't have a reset method, we'll handle it by
// creating a fresh database when reset is true
// We'll implement this by checking if it's a new database (next_id == 1)
let root_id = if db.get_next_id()? == 1 {
// Create a new root node
let root = Node::new_root();
let root_id = db.set(OurDBSetArgs {
id: None,
data: &root.serialize(),
})?;
// First ID should be 1
assert_eq!(root_id, 1);
root_id
} else {
// Use existing root node
1 // Root node always has ID 1
};
Ok(RadixTree {
db,
root_id,
})
}
/// Sets a key-value pair in the tree.
pub fn set(tree: &mut RadixTree, key: &str, value: Vec<u8>) -> Result<(), Error> {
let mut current_id = tree.root_id;
let mut offset = 0;
// Handle empty key case
if key.is_empty() {
let mut root_node = tree.get_node(current_id)?;
root_node.is_leaf = true;
root_node.value = value;
tree.save_node(Some(current_id), &root_node)?;
return Ok(());
}
while offset < key.len() {
let mut node = tree.get_node(current_id)?;
// Find matching child
let mut matched_child = None;
for (i, child) in node.children.iter().enumerate() {
if key[offset..].starts_with(&child.key_part) {
matched_child = Some((i, child.clone()));
break;
}
}
if matched_child.is_none() {
// No matching child found, create new leaf node
let key_part = key[offset..].to_string();
let new_node = Node {
key_segment: key_part.clone(),
value: value.clone(),
children: Vec::new(),
is_leaf: true,
};
let new_id = tree.save_node(None, &new_node)?;
// Create new child reference and update parent node
node.children.push(NodeRef {
key_part,
node_id: new_id,
});
tree.save_node(Some(current_id), &node)?;
return Ok(());
}
let (child_index, mut child) = matched_child.unwrap();
let common_prefix = get_common_prefix(&key[offset..], &child.key_part);
if common_prefix.len() < child.key_part.len() {
// Split existing node
let child_node = tree.get_node(child.node_id)?;
// Create new intermediate node
let new_node = Node {
key_segment: child.key_part[common_prefix.len()..].to_string(),
value: child_node.value.clone(),
children: child_node.children.clone(),
is_leaf: child_node.is_leaf,
};
let new_id = tree.save_node(None, &new_node)?;
// Update current node
node.children[child_index] = NodeRef {
key_part: common_prefix.to_string(),
node_id: new_id,
};
tree.save_node(Some(current_id), &node)?;
// Update child node reference
child.node_id = new_id;
}
if offset + common_prefix.len() == key.len() {
// Update value at existing node
let mut child_node = tree.get_node(child.node_id)?;
child_node.value = value;
child_node.is_leaf = true;
tree.save_node(Some(child.node_id), &child_node)?;
return Ok(());
}
offset += common_prefix.len();
current_id = child.node_id;
}
Ok(())
}
/// Gets a value by key from the tree.
pub fn get(tree: &mut RadixTree, key: &str) -> Result<Vec<u8>, Error> {
let mut current_id = tree.root_id;
let mut offset = 0;
// Handle empty key case
if key.is_empty() {
let root_node = tree.get_node(current_id)?;
if root_node.is_leaf {
return Ok(root_node.value.clone());
}
return Err(Error::KeyNotFound(key.to_string()));
}
while offset < key.len() {
let node = tree.get_node(current_id)?;
let mut found = false;
for child in &node.children {
if key[offset..].starts_with(&child.key_part) {
if offset + child.key_part.len() == key.len() {
let child_node = tree.get_node(child.node_id)?;
if child_node.is_leaf {
return Ok(child_node.value);
}
}
current_id = child.node_id;
offset += child.key_part.len();
found = true;
break;
}
}
if !found {
return Err(Error::KeyNotFound(key.to_string()));
}
}
Err(Error::KeyNotFound(key.to_string()))
}
/// Updates the value at a given key prefix.
pub fn update(tree: &mut RadixTree, prefix: &str, new_value: Vec<u8>) -> Result<(), Error> {
let mut current_id = tree.root_id;
let mut offset = 0;
// Handle empty prefix case
if prefix.is_empty() {
return Err(Error::InvalidOperation("Empty prefix not allowed".to_string()));
}
while offset < prefix.len() {
let node = tree.get_node(current_id)?;
let mut found = false;
for child in &node.children {
if prefix[offset..].starts_with(&child.key_part) {
if offset + child.key_part.len() == prefix.len() {
// Found exact prefix match
let mut child_node = tree.get_node(child.node_id)?;
if child_node.is_leaf {
// Update the value
child_node.value = new_value;
tree.save_node(Some(child.node_id), &child_node)?;
return Ok(());
}
}
current_id = child.node_id;
offset += child.key_part.len();
found = true;
break;
}
}
if !found {
return Err(Error::PrefixNotFound(prefix.to_string()));
}
}
Err(Error::PrefixNotFound(prefix.to_string()))
}
/// Deletes a key from the tree.
pub fn delete(tree: &mut RadixTree, key: &str) -> Result<(), Error> {
let mut current_id = tree.root_id;
let mut offset = 0;
let mut path = Vec::new();
// Handle empty key case
if key.is_empty() {
let mut root_node = tree.get_node(current_id)?;
if !root_node.is_leaf {
return Err(Error::KeyNotFound(key.to_string()));
}
// For the root node, we just mark it as non-leaf
root_node.is_leaf = false;
root_node.value = Vec::new();
tree.save_node(Some(current_id), &root_node)?;
return Ok(());
}
// Find the node to delete
while offset < key.len() {
let node = tree.get_node(current_id)?;
let mut found = false;
for child in &node.children {
if key[offset..].starts_with(&child.key_part) {
path.push(child.clone());
current_id = child.node_id;
offset += child.key_part.len();
found = true;
// Check if we've matched the full key
if offset == key.len() {
let child_node = tree.get_node(child.node_id)?;
if child_node.is_leaf {
found = true;
break;
}
}
break;
}
}
if !found {
return Err(Error::KeyNotFound(key.to_string()));
}
}
if path.is_empty() {
return Err(Error::KeyNotFound(key.to_string()));
}
// Get the node to delete
let mut last_node = tree.get_node(path.last().unwrap().node_id)?;
// If the node has children, just mark it as non-leaf
if !last_node.children.is_empty() {
last_node.is_leaf = false;
last_node.value = Vec::new();
tree.save_node(Some(path.last().unwrap().node_id), &last_node)?;
return Ok(());
}
// If node has no children, remove it from parent
if path.len() > 1 {
let parent_id = path[path.len() - 2].node_id;
let mut parent_node = tree.get_node(parent_id)?;
// Find and remove the child from parent
for i in 0..parent_node.children.len() {
if parent_node.children[i].node_id == path.last().unwrap().node_id {
parent_node.children.remove(i);
break;
}
}
tree.save_node(Some(parent_id), &parent_node)?;
// Delete the node from the database
tree.db.delete(path.last().unwrap().node_id)?;
} else {
// If this is a direct child of the root, just mark it as non-leaf
last_node.is_leaf = false;
last_node.value = Vec::new();
tree.save_node(Some(path.last().unwrap().node_id), &last_node)?;
}
Ok(())
}
/// Lists all keys with a given prefix.
pub fn list(tree: &mut RadixTree, prefix: &str) -> Result<Vec<String>, Error> {
let mut result = Vec::new();
// Handle empty prefix case - will return all keys
if prefix.is_empty() {
collect_all_keys(tree, tree.root_id, "", &mut result)?;
return Ok(result);
}
// Start from the root and find all matching keys
find_keys_with_prefix(tree, tree.root_id, "", prefix, &mut result)?;
Ok(result)
}
/// Helper function to find all keys with a given prefix.
fn find_keys_with_prefix(
tree: &mut RadixTree,
node_id: u32,
current_path: &str,
prefix: &str,
result: &mut Vec<String>,
) -> Result<(), Error> {
let node = tree.get_node(node_id)?;
// If the current path already matches or exceeds the prefix length
if current_path.len() >= prefix.len() {
// Check if the current path starts with the prefix
if current_path.starts_with(prefix) {
// If this is a leaf node, add it to the results
if node.is_leaf {
result.push(current_path.to_string());
}
// Collect all keys from this subtree
for child in &node.children {
let child_path = format!("{}{}", current_path, child.key_part);
find_keys_with_prefix(tree, child.node_id, &child_path, prefix, result)?;
}
}
return Ok(());
}
// Current path is shorter than the prefix, continue searching
for child in &node.children {
let child_path = format!("{}{}", current_path, child.key_part);
// Check if this child's path could potentially match the prefix
if prefix.starts_with(current_path) {
// The prefix starts with the current path, so we need to check if
// the child's key_part matches the next part of the prefix
let prefix_remainder = &prefix[current_path.len()..];
// If the prefix remainder starts with the child's key_part or vice versa
if prefix_remainder.starts_with(&child.key_part)
|| (child.key_part.starts_with(prefix_remainder)
&& child.key_part.len() >= prefix_remainder.len()) {
find_keys_with_prefix(tree, child.node_id, &child_path, prefix, result)?;
}
}
}
Ok(())
}
/// Helper function to recursively collect all keys under a node.
fn collect_all_keys(
tree: &mut RadixTree,
node_id: u32,
current_path: &str,
result: &mut Vec<String>,
) -> Result<(), Error> {
let node = tree.get_node(node_id)?;
// If this node is a leaf, add its path to the result
if node.is_leaf {
result.push(current_path.to_string());
}
// Recursively collect keys from all children
for child in &node.children {
let child_path = format!("{}{}", current_path, child.key_part);
collect_all_keys(tree, child.node_id, &child_path, result)?;
}
Ok(())
}
/// Gets all values for keys with a given prefix.
pub fn getall(tree: &mut RadixTree, prefix: &str) -> Result<Vec<Vec<u8>>, Error> {
// Get all matching keys
let keys = list(tree, prefix)?;
// Get values for each key
let mut values = Vec::new();
for key in keys {
if let Ok(value) = get(tree, &key) {
values.push(value);
}
}
Ok(values)
}
impl RadixTree {
/// Helper function to get a node from the database.
pub(crate) fn get_node(&mut self, node_id: u32) -> Result<Node, Error> {
let data = self.db.get(node_id)?;
Node::deserialize(&data)
}
/// Helper function to save a node to the database.
pub(crate) fn save_node(&mut self, node_id: Option<u32>, node: &Node) -> Result<u32, Error> {
let data = node.serialize();
let args = OurDBSetArgs {
id: node_id,
data: &data,
};
Ok(self.db.set(args)?)
}
/// Helper function to find all keys with a given prefix.
fn find_keys_with_prefix(
&mut self,
node_id: u32,
current_path: &str,
prefix: &str,
result: &mut Vec<String>,
) -> Result<(), Error> {
let node = self.get_node(node_id)?;
// If the current path already matches or exceeds the prefix length
if current_path.len() >= prefix.len() {
// Check if the current path starts with the prefix
if current_path.starts_with(prefix) {
// If this is a leaf node, add it to the results
if node.is_leaf {
result.push(current_path.to_string());
}
// Collect all keys from this subtree
for child in &node.children {
let child_path = format!("{}{}", current_path, child.key_part);
self.find_keys_with_prefix(child.node_id, &child_path, prefix, result)?;
}
}
return Ok(());
}
// Current path is shorter than the prefix, continue searching
for child in &node.children {
let child_path = format!("{}{}", current_path, child.key_part);
// Check if this child's path could potentially match the prefix
if prefix.starts_with(current_path) {
// The prefix starts with the current path, so we need to check if
// the child's key_part matches the next part of the prefix
let prefix_remainder = &prefix[current_path.len()..];
// If the prefix remainder starts with the child's key_part or vice versa
if prefix_remainder.starts_with(&child.key_part)
|| (child.key_part.starts_with(prefix_remainder)
&& child.key_part.len() >= prefix_remainder.len()) {
self.find_keys_with_prefix(child.node_id, &child_path, prefix, result)?;
}
}
}
Ok(())
}
/// Helper function to recursively collect all keys under a node.
fn collect_all_keys(
&mut self,
node_id: u32,
current_path: &str,
result: &mut Vec<String>,
) -> Result<(), Error> {
let node = self.get_node(node_id)?;
// If this node is a leaf, add its path to the result
if node.is_leaf {
result.push(current_path.to_string());
}
// Recursively collect keys from all children
for child in &node.children {
let child_path = format!("{}{}", current_path, child.key_part);
self.collect_all_keys(child.node_id, &child_path, result)?;
}
Ok(())
}
}

156
packages/data/radixtree/src/serialize.rs Normal file
View File

@@ -0,0 +1,156 @@
//! Serialization and deserialization for RadixTree nodes.
use crate::error::Error;
use crate::node::{Node, NodeRef};
use std::io::{Cursor, Read};
use std::mem::size_of;
/// Current binary format version.
const VERSION: u8 = 1;
impl Node {
/// Serializes a node to bytes for storage.
pub fn serialize(&self) -> Vec<u8> {
let mut buffer = Vec::new();
// Add version byte
buffer.push(VERSION);
// Add key segment
write_string(&mut buffer, &self.key_segment);
// Add value as []u8
write_u16(&mut buffer, self.value.len() as u16);
buffer.extend_from_slice(&self.value);
// Add children
write_u16(&mut buffer, self.children.len() as u16);
for child in &self.children {
write_string(&mut buffer, &child.key_part);
write_u32(&mut buffer, child.node_id);
}
// Add leaf flag
buffer.push(if self.is_leaf { 1 } else { 0 });
buffer
}
/// Deserializes bytes to a node.
pub fn deserialize(data: &[u8]) -> Result<Self, Error> {
if data.is_empty() {
return Err(Error::Deserialization("Empty data".to_string()));
}
let mut cursor = Cursor::new(data);
// Read and verify version
let mut version_byte = [0u8; 1];
cursor.read_exact(&mut version_byte)
.map_err(|e| Error::Deserialization(format!("Failed to read version byte: {}", e)))?;
if version_byte[0] != VERSION {
return Err(Error::Deserialization(
format!("Invalid version byte: expected {}, got {}", VERSION, version_byte[0])
));
}
// Read key segment
let key_segment = read_string(&mut cursor)
.map_err(|e| Error::Deserialization(format!("Failed to read key segment: {}", e)))?;
// Read value as []u8
let value_len = read_u16(&mut cursor)
.map_err(|e| Error::Deserialization(format!("Failed to read value length: {}", e)))?;
let mut value = vec![0u8; value_len as usize];
cursor.read_exact(&mut value)
.map_err(|e| Error::Deserialization(format!("Failed to read value: {}", e)))?;
// Read children
let children_len = read_u16(&mut cursor)
.map_err(|e| Error::Deserialization(format!("Failed to read children length: {}", e)))?;
let mut children = Vec::with_capacity(children_len as usize);
for _ in 0..children_len {
let key_part = read_string(&mut cursor)
.map_err(|e| Error::Deserialization(format!("Failed to read child key part: {}", e)))?;
let node_id = read_u32(&mut cursor)
.map_err(|e| Error::Deserialization(format!("Failed to read child node ID: {}", e)))?;
children.push(NodeRef {
key_part,
node_id,
});
}
// Read leaf flag
let mut is_leaf_byte = [0u8; 1];
cursor.read_exact(&mut is_leaf_byte)
.map_err(|e| Error::Deserialization(format!("Failed to read leaf flag: {}", e)))?;
let is_leaf = is_leaf_byte[0] == 1;
Ok(Node {
key_segment,
value,
children,
is_leaf,
})
}
}
// Helper functions for serialization
fn write_string(buffer: &mut Vec<u8>, s: &str) {
let bytes = s.as_bytes();
write_u16(buffer, bytes.len() as u16);
buffer.extend_from_slice(bytes);
}
fn write_u16(buffer: &mut Vec<u8>, value: u16) {
buffer.extend_from_slice(&value.to_le_bytes());
}
fn write_u32(buffer: &mut Vec<u8>, value: u32) {
buffer.extend_from_slice(&value.to_le_bytes());
}
// Helper functions for deserialization
fn read_string(cursor: &mut Cursor<&[u8]>) -> std::io::Result<String> {
let len = read_u16(cursor)? as usize;
let mut bytes = vec![0u8; len];
cursor.read_exact(&mut bytes)?;
String::from_utf8(bytes)
.map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e))
}
fn read_u16(cursor: &mut Cursor<&[u8]>) -> std::io::Result<u16> {
let mut bytes = [0u8; size_of::<u16>()];
cursor.read_exact(&mut bytes)?;
Ok(u16::from_le_bytes(bytes))
}
fn read_u32(cursor: &mut Cursor<&[u8]>) -> std::io::Result<u32> {
let mut bytes = [0u8; size_of::<u32>()];
cursor.read_exact(&mut bytes)?;
Ok(u32::from_le_bytes(bytes))
}
/// Helper function to get the common prefix of two strings.
pub fn get_common_prefix(a: &str, b: &str) -> String {
let mut i = 0;
let a_bytes = a.as_bytes();
let b_bytes = b.as_bytes();
while i < a.len() && i < b.len() && a_bytes[i] == b_bytes[i] {
i += 1;
}
a[..i].to_string()
}

144
packages/data/radixtree/tests/basic_test.rs Normal file
View File

@@ -0,0 +1,144 @@
use radixtree::RadixTree;
use std::path::PathBuf;
use tempfile::tempdir;
#[test]
fn test_basic_operations() -> Result<(), radixtree::Error> {
// Create a temporary directory for the test
let temp_dir = tempdir().expect("Failed to create temp directory");
let db_path = temp_dir.path().to_str().unwrap();
// Create a new radix tree
let mut tree = RadixTree::new(db_path, true)?;
// Test setting and getting values
let key = "test_key";
let value = b"test_value".to_vec();
tree.set(key, value.clone())?;
let retrieved_value = tree.get(key)?;
assert_eq!(retrieved_value, value);
// Test updating a value
let new_value = b"updated_value".to_vec();
tree.update(key, new_value.clone())?;
let updated_value = tree.get(key)?;
assert_eq!(updated_value, new_value);
// Test deleting a value
tree.delete(key)?;
// Trying to get a deleted key should return an error
let result = tree.get(key);
assert!(result.is_err());
Ok(())
}
#[test]
fn test_empty_key() -> Result<(), radixtree::Error> {
// Create a temporary directory for the test
let temp_dir = tempdir().expect("Failed to create temp directory");
let db_path = temp_dir.path().to_str().unwrap();
// Create a new radix tree
let mut tree = RadixTree::new(db_path, true)?;
// Test setting and getting empty key
let key = "";
let value = b"value_for_empty_key".to_vec();
tree.set(key, value.clone())?;
let retrieved_value = tree.get(key)?;
assert_eq!(retrieved_value, value);
// Test deleting empty key
tree.delete(key)?;
// Trying to get a deleted key should return an error
let result = tree.get(key);
assert!(result.is_err());
Ok(())
}
#[test]
fn test_multiple_keys() -> Result<(), radixtree::Error> {
// Create a temporary directory for the test
let temp_dir = tempdir().expect("Failed to create temp directory");
let db_path = temp_dir.path().to_str().unwrap();
// Create a new radix tree
let mut tree = RadixTree::new(db_path, true)?;
// Insert multiple keys
let test_data = [
("key1", b"value1".to_vec()),
("key2", b"value2".to_vec()),
("key3", b"value3".to_vec()),
];
for (key, value) in &test_data {
tree.set(key, value.clone())?;
}
// Verify all keys can be retrieved
for (key, expected_value) in &test_data {
let retrieved_value = tree.get(key)?;
assert_eq!(&retrieved_value, expected_value);
}
Ok(())
}
#[test]
fn test_shared_prefixes() -> Result<(), radixtree::Error> {
// Create a temporary directory for the test
let temp_dir = tempdir().expect("Failed to create temp directory");
let db_path = temp_dir.path().to_str().unwrap();
// Create a new radix tree
let mut tree = RadixTree::new(db_path, true)?;
// Insert keys with shared prefixes
let test_data = [
("test", b"value_test".to_vec()),
("testing", b"value_testing".to_vec()),
("tested", b"value_tested".to_vec()),
];
for (key, value) in &test_data {
tree.set(key, value.clone())?;
}
// Verify all keys can be retrieved
for (key, expected_value) in &test_data {
let retrieved_value = tree.get(key)?;
assert_eq!(&retrieved_value, expected_value);
}
Ok(())
}
#[test]
fn test_persistence() -> Result<(), radixtree::Error> {
// Create a temporary directory for the test
let temp_dir = tempdir().expect("Failed to create temp directory");
let db_path = temp_dir.path().to_str().unwrap();
// Create a new radix tree and add some data
{
let mut tree = RadixTree::new(db_path, true)?;
tree.set("persistent_key", b"persistent_value".to_vec())?;
} // Tree is dropped here
// Create a new tree instance with the same path
{
let mut tree = RadixTree::new(db_path, false)?;
let value = tree.get("persistent_key")?;
assert_eq!(value, b"persistent_value".to_vec());
}
Ok(())
}

153
packages/data/radixtree/tests/getall_test.rs Normal file
View File

@@ -0,0 +1,153 @@
use radixtree::RadixTree;
use std::collections::HashMap;
use tempfile::tempdir;
#[test]
fn test_getall() -> Result<(), radixtree::Error> {
// Create a temporary directory for the test
let temp_dir = tempdir().expect("Failed to create temp directory");
let db_path = temp_dir.path().to_str().unwrap();
// Create a new radix tree
let mut tree = RadixTree::new(db_path, true)?;
// Set up test data with common prefixes
let test_data: HashMap<&str, &str> = [
("user_1", "data1"),
("user_2", "data2"),
("user_3", "data3"),
("admin_1", "admin_data1"),
("admin_2", "admin_data2"),
("guest", "guest_data"),
].iter().cloned().collect();
// Set all test data
for (key, value) in &test_data {
tree.set(key, value.as_bytes().to_vec())?;
}
// Test getall with 'user_' prefix
let user_values = tree.getall("user_")?;
// Should return 3 values
assert_eq!(user_values.len(), 3);
// Convert byte arrays to strings for easier comparison
let user_value_strings: Vec<String> = user_values
.iter()
.map(|v| String::from_utf8_lossy(v).to_string())
.collect();
// Check all expected values are present
assert!(user_value_strings.contains(&"data1".to_string()));
assert!(user_value_strings.contains(&"data2".to_string()));
assert!(user_value_strings.contains(&"data3".to_string()));
// Test getall with 'admin_' prefix
let admin_values = tree.getall("admin_")?;
// Should return 2 values
assert_eq!(admin_values.len(), 2);
// Convert byte arrays to strings for easier comparison
let admin_value_strings: Vec<String> = admin_values
.iter()
.map(|v| String::from_utf8_lossy(v).to_string())
.collect();
// Check all expected values are present
assert!(admin_value_strings.contains(&"admin_data1".to_string()));
assert!(admin_value_strings.contains(&"admin_data2".to_string()));
// Test getall with empty prefix (should return all values)
let all_values = tree.getall("")?;
// Should return all 6 values
assert_eq!(all_values.len(), test_data.len());
// Test getall with non-existent prefix
let non_existent_values = tree.getall("xyz")?;
// Should return empty array
assert_eq!(non_existent_values.len(), 0);
Ok(())
}
#[test]
fn test_getall_with_updates() -> Result<(), radixtree::Error> {
// Create a temporary directory for the test
let temp_dir = tempdir().expect("Failed to create temp directory");
let db_path = temp_dir.path().to_str().unwrap();
// Create a new radix tree
let mut tree = RadixTree::new(db_path, true)?;
// Set initial values
tree.set("key1", b"value1".to_vec())?;
tree.set("key2", b"value2".to_vec())?;
tree.set("key3", b"value3".to_vec())?;
// Get initial values
let initial_values = tree.getall("key")?;
assert_eq!(initial_values.len(), 3);
// Update a value
tree.update("key2", b"updated_value2".to_vec())?;
// Get values after update
let updated_values = tree.getall("key")?;
assert_eq!(updated_values.len(), 3);
// Convert to strings for easier comparison
let updated_value_strings: Vec<String> = updated_values
.iter()
.map(|v| String::from_utf8_lossy(v).to_string())
.collect();
// Check the updated value is present
assert!(updated_value_strings.contains(&"value1".to_string()));
assert!(updated_value_strings.contains(&"updated_value2".to_string()));
assert!(updated_value_strings.contains(&"value3".to_string()));
Ok(())
}
#[test]
fn test_getall_with_deletions() -> Result<(), radixtree::Error> {
// Create a temporary directory for the test
let temp_dir = tempdir().expect("Failed to create temp directory");
let db_path = temp_dir.path().to_str().unwrap();
// Create a new radix tree
let mut tree = RadixTree::new(db_path, true)?;
// Set initial values
tree.set("prefix_1", b"value1".to_vec())?;
tree.set("prefix_2", b"value2".to_vec())?;
tree.set("prefix_3", b"value3".to_vec())?;
tree.set("other", b"other_value".to_vec())?;
// Get initial values
let initial_values = tree.getall("prefix_")?;
assert_eq!(initial_values.len(), 3);
// Delete a key
tree.delete("prefix_2")?;
// Get values after deletion
let after_delete_values = tree.getall("prefix_")?;
assert_eq!(after_delete_values.len(), 2);
// Convert to strings for easier comparison
let after_delete_strings: Vec<String> = after_delete_values
.iter()
.map(|v| String::from_utf8_lossy(v).to_string())
.collect();
// Check the remaining values
assert!(after_delete_strings.contains(&"value1".to_string()));
assert!(after_delete_strings.contains(&"value3".to_string()));
Ok(())
}

185
packages/data/radixtree/tests/prefix_test.rs Normal file
View File

@@ -0,0 +1,185 @@
use radixtree::RadixTree;
use std::collections::HashMap;
use tempfile::tempdir;
#[test]
fn test_list() -> Result<(), radixtree::Error> {
// Create a temporary directory for the test
let temp_dir = tempdir().expect("Failed to create temp directory");
let db_path = temp_dir.path().to_str().unwrap();
// Create a new radix tree
let mut tree = RadixTree::new(db_path, true)?;
// Insert keys with various prefixes
let test_data: HashMap<&str, &str> = [
("apple", "fruit1"),
("application", "software1"),
("apply", "verb1"),
("banana", "fruit2"),
("ball", "toy1"),
("cat", "animal1"),
("car", "vehicle1"),
("cargo", "shipping1"),
].iter().cloned().collect();
// Set all test data
for (key, value) in &test_data {
tree.set(key, value.as_bytes().to_vec())?;
}
// Test prefix 'app' - should return apple, application, apply
let app_keys = tree.list("app")?;
assert_eq!(app_keys.len(), 3);
assert!(app_keys.contains(&"apple".to_string()));
assert!(app_keys.contains(&"application".to_string()));
assert!(app_keys.contains(&"apply".to_string()));
// Test prefix 'ba' - should return banana, ball
let ba_keys = tree.list("ba")?;
assert_eq!(ba_keys.len(), 2);
assert!(ba_keys.contains(&"banana".to_string()));
assert!(ba_keys.contains(&"ball".to_string()));
// Test prefix 'car' - should return car, cargo
let car_keys = tree.list("car")?;
assert_eq!(car_keys.len(), 2);
assert!(car_keys.contains(&"car".to_string()));
assert!(car_keys.contains(&"cargo".to_string()));
// Test prefix 'z' - should return empty list
let z_keys = tree.list("z")?;
assert_eq!(z_keys.len(), 0);
// Test empty prefix - should return all keys
let all_keys = tree.list("")?;
assert_eq!(all_keys.len(), test_data.len());
for key in test_data.keys() {
assert!(all_keys.contains(&key.to_string()));
}
// Test exact key as prefix - should return just that key
let exact_key = tree.list("apple")?;
assert_eq!(exact_key.len(), 1);
assert_eq!(exact_key[0], "apple");
Ok(())
}
#[test]
fn test_list_with_deletion() -> Result<(), radixtree::Error> {
// Create a temporary directory for the test
let temp_dir = tempdir().expect("Failed to create temp directory");
let db_path = temp_dir.path().to_str().unwrap();
// Create a new radix tree
let mut tree = RadixTree::new(db_path, true)?;
// Set keys with common prefixes
tree.set("test1", b"value1".to_vec())?;
tree.set("test2", b"value2".to_vec())?;
tree.set("test3", b"value3".to_vec())?;
tree.set("other", b"value4".to_vec())?;
// Initial check
let test_keys = tree.list("test")?;
assert_eq!(test_keys.len(), 3);
assert!(test_keys.contains(&"test1".to_string()));
assert!(test_keys.contains(&"test2".to_string()));
assert!(test_keys.contains(&"test3".to_string()));
// Delete one key
tree.delete("test2")?;
// Check after deletion
let test_keys_after = tree.list("test")?;
assert_eq!(test_keys_after.len(), 2);
assert!(test_keys_after.contains(&"test1".to_string()));
assert!(!test_keys_after.contains(&"test2".to_string()));
assert!(test_keys_after.contains(&"test3".to_string()));
// Check all keys
let all_keys = tree.list("")?;
assert_eq!(all_keys.len(), 3);
assert!(all_keys.contains(&"other".to_string()));
Ok(())
}
#[test]
fn test_list_edge_cases() -> Result<(), radixtree::Error> {
// Create a temporary directory for the test
let temp_dir = tempdir().expect("Failed to create temp directory");
let db_path = temp_dir.path().to_str().unwrap();
// Create a new radix tree
let mut tree = RadixTree::new(db_path, true)?;
// Test with empty tree
let empty_result = tree.list("any")?;
assert_eq!(empty_result.len(), 0);
// Set a single key
tree.set("single", b"value".to_vec())?;
// Test with prefix that's longer than any key
let long_prefix = tree.list("singlelonger")?;
assert_eq!(long_prefix.len(), 0);
// Test with partial prefix match
let partial = tree.list("sing")?;
assert_eq!(partial.len(), 1);
assert_eq!(partial[0], "single");
// Test with very long keys
let long_key1 = "a".repeat(100) + "key1";
let long_key2 = "a".repeat(100) + "key2";
tree.set(&long_key1, b"value1".to_vec())?;
tree.set(&long_key2, b"value2".to_vec())?;
let long_prefix_result = tree.list(&"a".repeat(100))?;
assert_eq!(long_prefix_result.len(), 2);
assert!(long_prefix_result.contains(&long_key1));
assert!(long_prefix_result.contains(&long_key2));
Ok(())
}
#[test]
fn test_list_performance() -> Result<(), radixtree::Error> {
// Create a temporary directory for the test
let temp_dir = tempdir().expect("Failed to create temp directory");
let db_path = temp_dir.path().to_str().unwrap();
// Create a new radix tree
let mut tree = RadixTree::new(db_path, true)?;
// Insert a large number of keys with different prefixes
let prefixes = ["user", "post", "comment", "like", "share"];
// Set 100 keys for each prefix (500 total)
for prefix in &prefixes {
for i in 0..100 {
let key = format!("{}_{}", prefix, i);
tree.set(&key, format!("value_{}", key).as_bytes().to_vec())?;
}
}
// Test retrieving by each prefix
for prefix in &prefixes {
let keys = tree.list(prefix)?;
assert_eq!(keys.len(), 100);
// Verify all keys have the correct prefix
for key in &keys {
assert!(key.starts_with(prefix));
}
}
// Test retrieving all keys
let all_keys = tree.list("")?;
assert_eq!(all_keys.len(), 500);
Ok(())
}

180
packages/data/radixtree/tests/serialize_test.rs Normal file
View File

@@ -0,0 +1,180 @@
use radixtree::{Node, NodeRef};
#[test]
fn test_node_serialization() {
// Create a node with some data
let node = Node {
key_segment: "test".to_string(),
value: b"test_value".to_vec(),
children: vec![
NodeRef {
key_part: "child1".to_string(),
node_id: 1,
},
NodeRef {
key_part: "child2".to_string(),
node_id: 2,
},
],
is_leaf: true,
};
// Serialize the node
let serialized = node.serialize();
// Deserialize the node
let deserialized = Node::deserialize(&serialized).expect("Failed to deserialize node");
// Verify the deserialized node matches the original
assert_eq!(deserialized.key_segment, node.key_segment);
assert_eq!(deserialized.value, node.value);
assert_eq!(deserialized.is_leaf, node.is_leaf);
assert_eq!(deserialized.children.len(), node.children.len());
for (i, child) in node.children.iter().enumerate() {
assert_eq!(deserialized.children[i].key_part, child.key_part);
assert_eq!(deserialized.children[i].node_id, child.node_id);
}
}
#[test]
fn test_empty_node_serialization() {
// Create an empty node
let node = Node {
key_segment: "".to_string(),
value: vec![],
children: vec![],
is_leaf: false,
};
// Serialize the node
let serialized = node.serialize();
// Deserialize the node
let deserialized = Node::deserialize(&serialized).expect("Failed to deserialize node");
// Verify the deserialized node matches the original
assert_eq!(deserialized.key_segment, node.key_segment);
assert_eq!(deserialized.value, node.value);
assert_eq!(deserialized.is_leaf, node.is_leaf);
assert_eq!(deserialized.children.len(), node.children.len());
}
#[test]
fn test_node_with_many_children() {
// Create a node with many children
let mut children = Vec::new();
for i in 0..100 {
children.push(NodeRef {
key_part: format!("child{}", i),
node_id: i as u32,
});
}
let node = Node {
key_segment: "parent".to_string(),
value: b"parent_value".to_vec(),
children,
is_leaf: true,
};
// Serialize the node
let serialized = node.serialize();
// Deserialize the node
let deserialized = Node::deserialize(&serialized).expect("Failed to deserialize node");
// Verify the deserialized node matches the original
assert_eq!(deserialized.key_segment, node.key_segment);
assert_eq!(deserialized.value, node.value);
assert_eq!(deserialized.is_leaf, node.is_leaf);
assert_eq!(deserialized.children.len(), node.children.len());
for (i, child) in node.children.iter().enumerate() {
assert_eq!(deserialized.children[i].key_part, child.key_part);
assert_eq!(deserialized.children[i].node_id, child.node_id);
}
}
#[test]
fn test_node_with_large_value() {
// Create a node with a large value
let large_value = vec![0u8; 4096]; // 4KB value
let node = Node {
key_segment: "large_value".to_string(),
value: large_value.clone(),
children: vec![],
is_leaf: true,
};
// Serialize the node
let serialized = node.serialize();
// Deserialize the node
let deserialized = Node::deserialize(&serialized).expect("Failed to deserialize node");
// Verify the deserialized node matches the original
assert_eq!(deserialized.key_segment, node.key_segment);
assert_eq!(deserialized.value, node.value);
assert_eq!(deserialized.is_leaf, node.is_leaf);
assert_eq!(deserialized.children.len(), node.children.len());
}
#[test]
fn test_version_compatibility() {
// This test ensures that the serialization format is compatible with version 1
// Create a node
let node = Node {
key_segment: "test".to_string(),
value: b"test_value".to_vec(),
children: vec![
NodeRef {
key_part: "child".to_string(),
node_id: 1,
},
],
is_leaf: true,
};
// Serialize the node
let serialized = node.serialize();
// Verify the first byte is the version byte (1)
assert_eq!(serialized[0], 1);
// Deserialize the node
let deserialized = Node::deserialize(&serialized).expect("Failed to deserialize node");
// Verify the deserialized node matches the original
assert_eq!(deserialized.key_segment, node.key_segment);
assert_eq!(deserialized.value, node.value);
assert_eq!(deserialized.is_leaf, node.is_leaf);
assert_eq!(deserialized.children.len(), node.children.len());
}
#[test]
fn test_invalid_serialization() {
// Test with empty data
let result = Node::deserialize(&[]);
assert!(result.is_err());
// Test with invalid version
let result = Node::deserialize(&[2, 0, 0, 0, 0]);
assert!(result.is_err());
// Test with truncated data
let node = Node {
key_segment: "test".to_string(),
value: b"test_value".to_vec(),
children: vec![],
is_leaf: true,
};
let serialized = node.serialize();
let truncated = &serialized[0..serialized.len() / 2];
let result = Node::deserialize(truncated);
assert!(result.is_err());
}