move rhailib to herolib
This commit is contained in:
Timur Gordon
294
rhailib/docs/ARCHITECTURE.md
Normal file
294
rhailib/docs/ARCHITECTURE.md
Normal file
@@ -0,0 +1,294 @@
|
||||
# Rhailib Architecture Overview
|
||||
|
||||
Rhailib is a comprehensive Rust-based ecosystem for executing Rhai scripts in distributed environments with full business domain support, authorization, and scalability features.
|
||||
|
||||
## System Architecture
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "Client Layer"
|
||||
A[rhai_dispatcher] --> B[Redis Task Queues]
|
||||
UI[rhai_engine_ui] --> B
|
||||
REPL[ui_repl] --> B
|
||||
end
|
||||
|
||||
subgraph "Processing Layer"
|
||||
B --> C[rhailib_worker]
|
||||
C --> D[rhailib_engine]
|
||||
D --> E[rhailib_dsl]
|
||||
end
|
||||
|
||||
subgraph "Core Infrastructure"
|
||||
E --> F[derive - Procedural Macros]
|
||||
E --> G[macros - Authorization]
|
||||
D --> H[mock_db - Testing]
|
||||
end
|
||||
|
||||
subgraph "Operations Layer"
|
||||
I[monitor] --> B
|
||||
I --> C
|
||||
end
|
||||
|
||||
subgraph "Data Layer"
|
||||
J[Redis] --> B
|
||||
K[Database] --> E
|
||||
end
|
||||
```
|
||||
|
||||
## Crate Overview
|
||||
|
||||
### Core Engine Components
|
||||
|
||||
#### [`rhailib_engine`](../src/engine/docs/ARCHITECTURE.md)
|
||||
The central Rhai scripting engine that orchestrates all business domain modules.
|
||||
- **Purpose**: Unified engine creation and script execution
|
||||
- **Features**: Mock database, feature-based architecture, performance optimization
|
||||
- **Key Functions**: `create_heromodels_engine()`, script compilation and execution
|
||||
|
||||
#### [`rhailib_dsl`](../src/dsl/docs/ARCHITECTURE.md)
|
||||
Comprehensive Domain-Specific Language implementation exposing business models to Rhai.
|
||||
- **Purpose**: Business domain integration with Rhai scripting
|
||||
- **Domains**: Business operations, finance, content management, workflows, access control
|
||||
- **Features**: Fluent APIs, type safety, authorization integration
|
||||
|
||||
### Code Generation and Utilities
|
||||
|
||||
#### [`derive`](../src/derive/docs/ARCHITECTURE.md)
|
||||
Procedural macros for automatic Rhai integration code generation.
|
||||
- **Purpose**: Simplify Rhai integration for custom types
|
||||
- **Macros**: `RhaiApi` for DSL generation, `FromVec` for type conversion
|
||||
- **Features**: Builder pattern generation, error handling
|
||||
|
||||
#### [`macros`](../src/macros/docs/ARCHITECTURE.md)
|
||||
Authorization macros and utilities for secure database operations.
|
||||
- **Purpose**: Declarative security for Rhai functions
|
||||
- **Features**: CRUD operation macros, access control, context management
|
||||
- **Security**: Multi-level authorization, audit trails
|
||||
|
||||
### Client and Communication
|
||||
|
||||
#### [`rhai_dispatcher`](../src/client/docs/ARCHITECTURE.md)
|
||||
Redis-based client library for distributed script execution.
|
||||
- **Purpose**: Submit and manage Rhai script execution requests
|
||||
- **Features**: Builder pattern API, timeout handling, request-reply pattern
|
||||
- **Architecture**: Async operations, connection pooling, error handling
|
||||
|
||||
#### [`rhailib_worker`](../src/worker/docs/ARCHITECTURE.md)
|
||||
Distributed task execution system for processing Rhai scripts.
|
||||
- **Purpose**: Scalable script processing with queue-based architecture
|
||||
- **Features**: Multi-context support, horizontal scaling, fault tolerance, context injection
|
||||
- **Architecture**: Workers decoupled from contexts, allowing single worker to serve multiple circles
|
||||
- **Integration**: Full engine and DSL access, secure execution
|
||||
|
||||
### User Interfaces
|
||||
|
||||
#### [`ui_repl`](../src/repl/docs/ARCHITECTURE.md)
|
||||
Interactive development environment for Rhai script development.
|
||||
- **Purpose**: Real-time script development and testing
|
||||
- **Features**: Enhanced CLI, dual execution modes, worker management
|
||||
- **Development**: Syntax highlighting, script editing, immediate feedback
|
||||
|
||||
#### [`rhai_engine_ui`](../src/rhai_engine_ui/docs/ARCHITECTURE.md)
|
||||
Web-based interface for Rhai script management and execution.
|
||||
- **Purpose**: Browser-based script execution and management
|
||||
- **Architecture**: WebAssembly frontend with optional server backend
|
||||
- **Features**: Real-time updates, task management, visual interface
|
||||
|
||||
### Operations and Monitoring
|
||||
|
||||
#### [`monitor`](../src/monitor/docs/ARCHITECTURE.md)
|
||||
Command-line monitoring and management tool for the rhailib ecosystem.
|
||||
- **Purpose**: System observability and task management
|
||||
- **Features**: Real-time monitoring, performance metrics, queue management
|
||||
- **Operations**: Multi-worker support, interactive CLI, visualization
|
||||
|
||||
## Data Flow Architecture
|
||||
|
||||
### Script Execution Flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client as rhai_dispatcher
|
||||
participant Redis as Redis Queue
|
||||
participant Worker as rhailib_worker
|
||||
participant Engine as rhailib_engine
|
||||
participant DSL as rhailib_dsl
|
||||
participant DB as Database
|
||||
|
||||
Client->>Redis: Submit script task (worker_id + context_id)
|
||||
Worker->>Redis: Poll worker queue (worker_id)
|
||||
Redis->>Worker: Return task with context_id
|
||||
Worker->>Engine: Create configured engine
|
||||
Engine->>DSL: Register domain modules
|
||||
Worker->>Engine: Execute script with context_id
|
||||
Engine->>DSL: Call business functions (context_id)
|
||||
DSL->>DB: Perform authorized operations (context_id)
|
||||
DB->>DSL: Return results
|
||||
DSL->>Engine: Return processed data
|
||||
Engine->>Worker: Return execution result
|
||||
Worker->>Redis: Publish result to reply queue
|
||||
Redis->>Client: Deliver result
|
||||
```
|
||||
|
||||
### Authorization Flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Script as Rhai Script
|
||||
participant Macro as Authorization Macro
|
||||
participant Context as Execution Context
|
||||
participant Access as Access Control
|
||||
participant DB as Database
|
||||
|
||||
Script->>Macro: Call authorized function
|
||||
Macro->>Context: Extract caller credentials
|
||||
Context->>Access: Validate permissions
|
||||
Access->>DB: Check resource access
|
||||
DB->>Access: Return authorization result
|
||||
Access->>Macro: Grant/deny access
|
||||
Macro->>DB: Execute authorized operation
|
||||
DB->>Script: Return results
|
||||
```
|
||||
|
||||
## Worker-Context Decoupling Architecture
|
||||
|
||||
A key architectural feature of rhailib is the decoupling of worker assignment from context management:
|
||||
|
||||
### Traditional Model (Previous)
|
||||
- **One Worker Per Circle**: Each worker was dedicated to a specific circle/context
|
||||
- **Queue Per Circle**: Workers listened to circle-specific queues
|
||||
- **Tight Coupling**: Worker identity was directly tied to context identity
|
||||
|
||||
### New Decoupled Model (Current)
|
||||
- **Worker ID**: Determines which queue the worker listens to (`rhailib:<worker_id>`)
|
||||
- **Context ID**: Provided in task details, determines execution context and database access
|
||||
- **Flexible Assignment**: Single worker can process tasks for multiple contexts
|
||||
|
||||
### Benefits of Decoupling
|
||||
|
||||
1. **Resource Efficiency**: Better worker utilization across multiple contexts
|
||||
2. **Deployment Flexibility**: Easier scaling and resource allocation
|
||||
3. **Cost Optimization**: Fewer worker instances needed for multi-context scenarios
|
||||
4. **Operational Simplicity**: Centralized worker management with distributed contexts
|
||||
|
||||
### Implementation Details
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
subgraph "Client Layer"
|
||||
C[Client] --> |worker_id + context_id| Q[Redis Queue]
|
||||
end
|
||||
|
||||
subgraph "Worker Layer"
|
||||
W1[Worker 1] --> |listens to| Q1[Queue: worker-1]
|
||||
W2[Worker 2] --> |listens to| Q2[Queue: worker-2]
|
||||
end
|
||||
|
||||
subgraph "Context Layer"
|
||||
W1 --> |processes| CTX1[Context A]
|
||||
W1 --> |processes| CTX2[Context B]
|
||||
W2 --> |processes| CTX1
|
||||
W2 --> |processes| CTX3[Context C]
|
||||
end
|
||||
```
|
||||
|
||||
## Key Design Principles
|
||||
|
||||
### 1. Security First
|
||||
- **Multi-layer Authorization**: Context-based, resource-specific, and operation-level security
|
||||
- **Secure Execution**: Isolated script execution with proper context injection
|
||||
- **Audit Trails**: Comprehensive logging and monitoring of all operations
|
||||
|
||||
### 2. Scalability
|
||||
- **Horizontal Scaling**: Multiple worker instances for load distribution
|
||||
- **Queue-based Architecture**: Reliable task distribution and processing
|
||||
- **Async Operations**: Non-blocking I/O throughout the system
|
||||
|
||||
### 3. Developer Experience
|
||||
- **Type Safety**: Comprehensive type checking and conversion utilities
|
||||
- **Error Handling**: Detailed error messages and proper error propagation
|
||||
- **Interactive Development**: REPL and web interfaces for immediate feedback
|
||||
|
||||
### 4. Modularity
|
||||
- **Feature Flags**: Configurable compilation based on requirements
|
||||
- **Crate Separation**: Clear boundaries and responsibilities
|
||||
- **Plugin Architecture**: Easy extension and customization
|
||||
|
||||
## Deployment Patterns
|
||||
|
||||
### Development Environment
|
||||
```
|
||||
REPL + Local Engine + Mock Database
|
||||
```
|
||||
- Interactive development with immediate feedback
|
||||
- Full DSL access without external dependencies
|
||||
- Integrated testing and debugging
|
||||
|
||||
### Testing Environment
|
||||
```
|
||||
Client + Worker + Redis + Mock Database
|
||||
```
|
||||
- Distributed execution testing
|
||||
- Queue-based communication validation
|
||||
- Performance and scalability testing
|
||||
|
||||
### Production Environment
|
||||
```
|
||||
Multiple Clients + Redis Cluster + Worker Pool + Production Database
|
||||
```
|
||||
- High availability and fault tolerance
|
||||
- Horizontal scaling and load distribution
|
||||
- Comprehensive monitoring and observability
|
||||
|
||||
## Integration Points
|
||||
|
||||
### External Systems
|
||||
- **Redis**: Task queues, result delivery, system coordination
|
||||
- **Databases**: Business data persistence and retrieval
|
||||
- **Web Browsers**: WebAssembly-based user interfaces
|
||||
- **Command Line**: Development and operations tooling
|
||||
|
||||
### Internal Integration
|
||||
- **Macro System**: Code generation and authorization
|
||||
- **Type System**: Safe conversions and error handling
|
||||
- **Module System**: Domain-specific functionality organization
|
||||
- **Context System**: Security and execution environment management
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
### Throughput
|
||||
- **Concurrent Execution**: Multiple workers processing tasks simultaneously
|
||||
- **Connection Pooling**: Efficient database and Redis connection management
|
||||
- **Compiled Scripts**: AST caching for repeated execution optimization
|
||||
|
||||
### Latency
|
||||
- **Local Execution**: Direct engine access for development scenarios
|
||||
- **Queue Optimization**: Efficient task distribution and result delivery
|
||||
- **Context Caching**: Reduced overhead for authorization and setup
|
||||
|
||||
### Resource Usage
|
||||
- **Memory Management**: Efficient ownership and borrowing patterns
|
||||
- **CPU Utilization**: Async operations and non-blocking I/O
|
||||
- **Network Efficiency**: Optimized serialization and communication protocols
|
||||
|
||||
## Future Extensibility
|
||||
|
||||
### Adding New Domains
|
||||
1. Create domain module in `rhailib_dsl`
|
||||
2. Implement authorization macros in `macros`
|
||||
3. Add feature flags and conditional compilation
|
||||
4. Update engine registration and documentation
|
||||
|
||||
### Custom Authorization
|
||||
1. Extend authorization macros with custom logic
|
||||
2. Implement domain-specific access control functions
|
||||
3. Add audit and logging capabilities
|
||||
4. Update security documentation
|
||||
|
||||
### New Interfaces
|
||||
1. Implement client interface following existing patterns
|
||||
2. Integrate with Redis communication layer
|
||||
3. Add monitoring and observability features
|
||||
4. Provide comprehensive documentation
|
||||
|
||||
This architecture provides a robust, secure, and scalable foundation for distributed Rhai script execution while maintaining excellent developer experience and operational visibility.
|
||||
Reference in New Issue
Block a user