Start CQRS refactoring: Create Osiris client crate
- Added workspace structure to Osiris Cargo.toml - Created osiris-client crate for query operations (GET requests) - Implemented generic get(), list(), query() methods - Added KYC, payment, and communication query modules - Created comprehensive refactoring plan document CQRS Pattern: - Commands (writes) → Supervisor client → Rhai scripts - Queries (reads) → Osiris client → REST API Next steps: - Implement Osiris server with Axum - Restructure SDK client by category (kyc/, payment/, etc.) - Update FreezoneClient to use both supervisor and osiris clients
This commit is contained in:
Timur Gordon
303
README.md
303
README.md
@@ -1,124 +1,251 @@
|
||||
# OSIRIS
|
||||
|
||||
**Object Storage, Indexing & Retrieval Intelligent System**
|
||||
**Object Storage with Rhai Scripting Integration**
|
||||
|
||||
OSIRIS is a Rust-native object storage and retrieval layer built on top of HeroDB, providing structured storage with metadata, field indexing, and search capabilities.
|
||||
OSIRIS is a Rust-native object storage layer built on HeroDB, providing structured storage with automatic indexing, Rhai scripting support, and signatory-based access control.
|
||||
|
||||
## Features
|
||||
## Overview
|
||||
|
||||
- **Object Storage**: Store structured objects with metadata (title, tags, MIME type, timestamps)
|
||||
- **Namespace Management**: Organize objects into isolated namespaces
|
||||
- **Field Indexing**: Fast filtering by tags and metadata fields
|
||||
- **Text Search**: Simple keyword-based search across object content
|
||||
- **CLI Interface**: Command-line tools for object management and search
|
||||
- **9P Filesystem**: Mount OSIRIS as a filesystem (future)
|
||||
OSIRIS provides a trait-based architecture for storing and retrieving typed objects with:
|
||||
- **Automatic Indexing**: Fields marked with `#[index]` are automatically indexed
|
||||
- **Rhai Integration**: Full scripting support with builder patterns
|
||||
- **Context-Based Storage**: Multi-tenant contexts with signatory-based access control
|
||||
- **Type Safety**: Compile-time guarantees through the `Object` trait
|
||||
- **HeroDB Backend**: Built on top of HeroDB (Redis-compatible)
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Prerequisites
|
||||
### Build OSIRIS
|
||||
|
||||
Start HeroDB:
|
||||
```bash
|
||||
cd /path/to/herodb
|
||||
cargo run --release -- --dir ./data --admin-secret mysecret --port 6379
|
||||
cargo build --release --features rhai-support
|
||||
```
|
||||
|
||||
### Installation
|
||||
### Run a Rhai Script
|
||||
|
||||
```bash
|
||||
cd /path/to/osiris
|
||||
cargo build --release
|
||||
cargo run --bin runner --features rhai-support -- runner1 \
|
||||
--redis-url redis://localhost:6379 \
|
||||
--db-id 1 \
|
||||
--script-file examples/engine/01_note.rhai
|
||||
```
|
||||
|
||||
### Initialize
|
||||
### Example Script
|
||||
|
||||
```bash
|
||||
# Create configuration
|
||||
mkdir -p ~/.config/osiris
|
||||
cat > ~/.config/osiris/config.toml <<EOF
|
||||
[herodb]
|
||||
url = "redis://localhost:6379"
|
||||
```rhai
|
||||
// Get a context (requires signatories)
|
||||
let ctx = get_context(["alice_pk", "bob_pk"]);
|
||||
|
||||
[namespaces.notes]
|
||||
db_id = 1
|
||||
EOF
|
||||
// Create a note
|
||||
let note = note("notes")
|
||||
.title("My First Note")
|
||||
.content("This is the content")
|
||||
.tag("topic", "rust")
|
||||
.tag("priority", "high");
|
||||
|
||||
# Initialize OSIRIS
|
||||
./target/release/osiris init --herodb redis://localhost:6379
|
||||
// Save to context
|
||||
ctx.save(note);
|
||||
|
||||
# Create a namespace
|
||||
./target/release/osiris ns create notes
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
# Add objects
|
||||
./target/release/osiris put notes/my-note.md ./my-note.md --tags topic=rust,project=osiris
|
||||
|
||||
# Get objects
|
||||
./target/release/osiris get notes/my-note.md
|
||||
|
||||
# Search
|
||||
./target/release/osiris find --ns notes --filter topic=rust
|
||||
./target/release/osiris find "retrieval" --ns notes
|
||||
|
||||
# Delete objects
|
||||
./target/release/osiris del notes/my-note.md
|
||||
|
||||
# Show statistics
|
||||
./target/release/osiris stats --ns notes
|
||||
// Query by index
|
||||
let ids = ctx.query("notes", "tags:tag", "topic=rust");
|
||||
print(`Found ${ids.len()} notes`);
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
HeroDB (unmodified)
|
||||
│
|
||||
├── KV store + encryption
|
||||
└── RESP protocol
|
||||
↑
|
||||
│
|
||||
└── OSIRIS
|
||||
├── store/ – object schema + persistence
|
||||
├── index/ – field index & keyword scanning
|
||||
├── retrieve/ – query planner + filtering
|
||||
├── interfaces/ – CLI, 9P
|
||||
└── config/ – namespaces + settings
|
||||
```
|
||||
|
||||
## Data Model
|
||||
|
||||
Objects are stored with metadata:
|
||||
- **ID**: Unique identifier (UUID or user-assigned)
|
||||
- **Namespace**: Logical grouping (e.g., "notes", "calendar")
|
||||
- **Title**: Optional human-readable title
|
||||
- **MIME Type**: Content type
|
||||
- **Tags**: Key-value pairs for categorization
|
||||
- **Timestamps**: Created and updated times
|
||||
- **Text Content**: Optional plain text content
|
||||
|
||||
## Keyspace Design
|
||||
### Core Components
|
||||
|
||||
```
|
||||
meta:<id> → serialized OsirisObject
|
||||
field:<field>:<val> → Set of IDs (for equality filtering)
|
||||
scan:index → list of IDs for text scan
|
||||
OSIRIS
|
||||
├── objects/ – Domain objects (Note, Event, User, etc.)
|
||||
│ ├── note/
|
||||
│ │ ├── mod.rs – Note struct and impl
|
||||
│ │ └── rhai.rs – Rhai bindings
|
||||
│ └── event/
|
||||
│ ├── mod.rs – Event struct and impl
|
||||
│ └── rhai.rs – Rhai bindings
|
||||
├── store/ – Storage layer
|
||||
│ ├── generic_store.rs – Type-safe storage
|
||||
│ └── type_registry.rs – Custom type registration
|
||||
├── rhai/ – Rhai integration
|
||||
│ ├── instance.rs – OsirisContext (multi-tenant)
|
||||
│ └── engine.rs – Engine configuration
|
||||
└── bin/
|
||||
└── runner.rs – Standalone script runner
|
||||
```
|
||||
|
||||
### Object Trait
|
||||
|
||||
All OSIRIS objects implement the `Object` trait:
|
||||
|
||||
```rust
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, DeriveObject)]
|
||||
pub struct Note {
|
||||
pub base_data: BaseData,
|
||||
|
||||
#[index]
|
||||
pub title: Option<String>,
|
||||
|
||||
pub content: Option<String>,
|
||||
|
||||
#[index]
|
||||
pub tags: BTreeMap<String, String>,
|
||||
}
|
||||
```
|
||||
|
||||
The `#[derive(DeriveObject)]` macro automatically:
|
||||
- Implements the `Object` trait
|
||||
- Generates index keys from `#[index]` fields
|
||||
- Provides serialization/deserialization
|
||||
|
||||
## Key Features
|
||||
|
||||
### 1. Signatory-Based Access Control
|
||||
|
||||
Contexts use signatory-based access instead of owner-based permissions:
|
||||
|
||||
```rhai
|
||||
// Single participant
|
||||
let ctx = get_context(["alice_pk"]);
|
||||
|
||||
// Shared context (all must be signatories)
|
||||
let ctx = get_context(["alice_pk", "bob_pk", "charlie_pk"]);
|
||||
```
|
||||
|
||||
Access is granted only if all participants are signatories of the script.
|
||||
|
||||
### 2. Automatic Indexing
|
||||
|
||||
Fields marked with `#[index]` are automatically indexed in HeroDB:
|
||||
|
||||
```rust
|
||||
#[index]
|
||||
pub title: Option<String>, // Indexed for fast queries
|
||||
|
||||
pub content: Option<String>, // Not indexed
|
||||
```
|
||||
|
||||
Query by indexed fields:
|
||||
```rhai
|
||||
let ids = ctx.query("notes", "title", "My Note");
|
||||
let ids = ctx.query("notes", "tags:tag", "topic=rust");
|
||||
```
|
||||
|
||||
### 3. Multi-Instance Support
|
||||
|
||||
Create multiple OSIRIS instances pointing to different databases:
|
||||
|
||||
```bash
|
||||
cargo run --bin runner --features rhai-support -- runner1 \
|
||||
--instance freezone:redis://localhost:6379:1 \
|
||||
--instance my:redis://localhost:6379:2 \
|
||||
--script-file script.rhai
|
||||
```
|
||||
|
||||
```rhai
|
||||
// Instances are automatically available
|
||||
freezone.save(note);
|
||||
my.save(note);
|
||||
```
|
||||
|
||||
### 4. Builder Pattern
|
||||
|
||||
Fluent API for creating objects:
|
||||
|
||||
```rhai
|
||||
let event = event("calendar", "Team Meeting")
|
||||
.description("Weekly sync")
|
||||
.location("Conference Room A")
|
||||
.category("meetings")
|
||||
.all_day(false);
|
||||
|
||||
ctx.save(event);
|
||||
```
|
||||
|
||||
## Creating Custom Objects
|
||||
|
||||
See [docs/CREATING_NEW_OBJECTS.md](docs/CREATING_NEW_OBJECTS.md) for a complete guide on creating new object types.
|
||||
|
||||
Quick example:
|
||||
|
||||
```rust
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, DeriveObject)]
|
||||
pub struct Task {
|
||||
pub base_data: BaseData,
|
||||
|
||||
#[index]
|
||||
pub title: String,
|
||||
|
||||
pub completed: bool,
|
||||
}
|
||||
```
|
||||
|
||||
## Storage Model
|
||||
|
||||
### HeroDB Keyspace
|
||||
|
||||
```
|
||||
obj:<ns>:<id> → JSON serialized object
|
||||
idx:<ns>:<field>:<value> → Set of object IDs
|
||||
scan:<ns> → Set of all object IDs in namespace
|
||||
```
|
||||
|
||||
Example:
|
||||
```
|
||||
field:tag:project=osiris → {note_1, note_2}
|
||||
field:mime:text/markdown → {note_1, note_3}
|
||||
obj:notes:abc123 → {"base_data":{...},"title":"My Note",...}
|
||||
idx:notes:title:My Note → {abc123, def456}
|
||||
idx:notes:tag:topic:rust → {abc123, xyz789}
|
||||
scan:notes → {abc123, def456, xyz789}
|
||||
```
|
||||
|
||||
## Future Enhancements
|
||||
### Context Storage
|
||||
|
||||
- Content-addressable deduplication
|
||||
- Vector embeddings for semantic search
|
||||
- Relation graphs
|
||||
- Full-text search with Tantivy
|
||||
- 9P filesystem interface
|
||||
Contexts store member privileges and metadata:
|
||||
```
|
||||
ctx:<context_id>:members → Map of user_id → privileges
|
||||
ctx:<context_id>:meta → Context metadata
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
The `examples/` directory contains comprehensive examples:
|
||||
|
||||
- **`examples/engine/`** - Object creation and storage examples
|
||||
- `01_note.rhai` - Note creation and querying
|
||||
- `02_event.rhai` - Event management
|
||||
- `03_user.rhai` - User objects
|
||||
- And more...
|
||||
|
||||
- **`examples/freezone/`** - Complete freezone registration flow
|
||||
|
||||
Run examples:
|
||||
```bash
|
||||
cargo run --example engine examples/engine/01_note.rhai
|
||||
```
|
||||
|
||||
## Documentation
|
||||
|
||||
- **[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md)** - Detailed architecture and design patterns
|
||||
- **[docs/CREATING_NEW_OBJECTS.md](docs/CREATING_NEW_OBJECTS.md)** - Guide for creating custom objects
|
||||
|
||||
## Building and Testing
|
||||
|
||||
```bash
|
||||
# Build with Rhai support
|
||||
cargo build --features rhai-support
|
||||
|
||||
# Run tests
|
||||
cargo test --lib --features rhai-support
|
||||
|
||||
# Build release binary
|
||||
cargo build --release --features rhai-support --bin runner
|
||||
```
|
||||
|
||||
## Integration
|
||||
|
||||
OSIRIS is used by:
|
||||
- **ZDFZ Backend** - Freezone company and resident management
|
||||
- **Hero Actor System** - Distributed job execution with object storage
|
||||
|
||||
## License
|
||||
|
||||
|
||||
Reference in New Issue
Block a user