herocode/rhailib
Archived
12
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

change context syntax

Browse Source
This commit is contained in:
timurgordon
2025-06-27 12:09:50 +03:00
parent 7619e3b944
commit 36cf5b7e38
68 changed files with 4236 additions and 833 deletions
Download Patch File Download Diff File Expand all files Collapse all files

370
docs/ARCHITECTURE.md
View File

@@ -1,126 +1,294 @@
# Rhailib Architecture: Distributed Rhai Scripting
# Rhailib Architecture Overview
## 1. Overview
Rhailib is a comprehensive Rust-based ecosystem for executing Rhai scripts in distributed environments with full business domain support, authorization, and scalability features.
`rhailib` provides a robust infrastructure for executing Rhai scripts in a distributed manner, primarily designed to integrate with and extend the HeroModels ecosystem. It allows for dynamic scripting capabilities, offloading computation, and enabling flexible automation. This document describes the target architecture utilizing dedicated reply queues for efficient result notification.
## System Architecture
## 2. Core Components
```mermaid
graph TB
subgraph "Client Layer"
A[rhai_client] --> B[Redis Task Queues]
UI[rhai_engine_ui] --> B
REPL[ui_repl] --> B
end
subgraph "Processing Layer"
B --> C[rhailib_worker]
C --> D[rhailib_engine]
D --> E[rhailib_dsl]
end
subgraph "Core Infrastructure"
E --> F[derive - Procedural Macros]
E --> G[macros - Authorization]
D --> H[mock_db - Testing]
end
subgraph "Operations Layer"
I[monitor] --> B
I --> C
end
subgraph "Data Layer"
J[Redis] --> B
K[Database] --> E
end
```
The `rhailib` system is composed of the following main components, leveraging Redis for task queuing, state management, and result notification:
## Crate Overview
1. **Rhai Engine (`src/engine`):**
* The core scripting capability. Provides a Rhai engine pre-configured with various modules.
* Utilized by the `rhai_worker` to process tasks.
### Core Engine Components
2. **Rhai Client (`src/client`):**
* Offers an interface for applications to submit Rhai scripts as tasks.
* Submits tasks to named Redis queues ("circles").
* Waits for results on a dedicated reply queue, avoiding polling.
#### [`rhailib_engine`](../src/engine/docs/ARCHITECTURE.md)
The central Rhai scripting engine that orchestrates all business domain modules.
- **Purpose**: Unified engine creation and script execution
- **Features**: Mock database, feature-based architecture, performance optimization
- **Key Functions**: `create_heromodels_engine()`, script compilation and execution
3. **Rhai Worker (`src/worker`):**
* Listens to Redis task queues ("circles") for incoming task IDs.
* Fetches task details, executes the script using the `rhai_engine`.
* Updates task status and results in Redis.
* Injects the caller's public key into the script's scope as `CALLER_PUBLIC_KEY` if available.
* Sends a notification/result to the client's dedicated reply queue.
#### [`rhailib_dsl`](../src/dsl/docs/ARCHITECTURE.md)
Comprehensive Domain-Specific Language implementation exposing business models to Rhai.
- **Purpose**: Business domain integration with Rhai scripting
- **Domains**: Business operations, finance, content management, workflows, access control
- **Features**: Fluent APIs, type safety, authorization integration
4. **Redis:**
* Acts as the message broker and data store for task queues, detailed task information (including scripts, status, and results), and reply queues for notifications.
### Code Generation and Utilities
## 3. Architecture & Workflow (Dedicated Reply Queues)
#### [`derive`](../src/derive/docs/ARCHITECTURE.md)
Procedural macros for automatic Rhai integration code generation.
- **Purpose**: Simplify Rhai integration for custom types
- **Macros**: `RhaiApi` for DSL generation, `FromVec` for type conversion
- **Features**: Builder pattern generation, error handling
The system employs a "Dedicated Reply Queue" pattern to notify clients of task completion, enhancing efficiency by eliminating client-side polling for results.
#### [`macros`](../src/macros/docs/ARCHITECTURE.md)
Authorization macros and utilities for secure database operations.
- **Purpose**: Declarative security for Rhai functions
- **Features**: CRUD operation macros, access control, context management
- **Security**: Multi-level authorization, audit trails
**Workflow:**
### Client and Communication
1. **Task Submission (Client):**
a. The client generates a unique `task_id` and a unique `reply_queue_name` (e.g., `rhai_reply:<uuid>`).
b. Task details, including the script, initial status ("pending"), and the `reply_queue_name`, are stored in a Redis Hash: `rhai_task_details:<task_id>`.
c. The `task_id` is pushed onto a Redis List acting as a task queue for a specific "circle": `rhai_tasks:<circle_name>`.
d. The client then performs a blocking pop (`BLPOP`) on its `reply_queue_name`, waiting for the result message.
#### [`rhai_client`](../src/client/docs/ARCHITECTURE.md)
Redis-based client library for distributed script execution.
- **Purpose**: Submit and manage Rhai script execution requests
- **Features**: Builder pattern API, timeout handling, request-reply pattern
- **Architecture**: Async operations, connection pooling, error handling
2. **Task Consumption & Processing (Worker):**
a. A `rhai_worker` instance, listening to one or more `rhai_tasks:<circle_name>` queues, picks up a `task_id` using `BLPOP`.
b. The worker retrieves the full task details (including the script and `reply_queue_name`) from the `rhai_task_details:<task_id>` hash.
c. The worker updates the task's status in the hash to "processing".
d. The Rhai script is executed using an instance of the `rhai_engine`.
#### [`rhailib_worker`](../src/worker/docs/ARCHITECTURE.md)
Distributed task execution system for processing Rhai scripts.
- **Purpose**: Scalable script processing with queue-based architecture
- **Features**: Multi-context support, horizontal scaling, fault tolerance, context injection
- **Architecture**: Workers decoupled from contexts, allowing single worker to serve multiple circles
- **Integration**: Full engine and DSL access, secure execution
3. **Result Storage & Notification (Worker):**
a. Upon completion (or error), the worker updates the `rhai_task_details:<task_id>` hash with the final status ("completed" or "error") and the script's output or error message.
b. If a `reply_queue_name` was provided in the task details, the worker constructs a result message (e.g., JSON containing `task_id`, final `status`, `output`/`error`).
c. The worker pushes this result message onto the specified `reply_queue_name` using `LPUSH`.
### User Interfaces
4. **Result Reception (Client):**
a. The client's `BLPOP` on its `reply_queue_name` receives the result message.
b. The client processes the result or error.
c. Optionally, the client may `DEL`ete its temporary reply queue.
#### [`ui_repl`](../src/repl/docs/ARCHITECTURE.md)
Interactive development environment for Rhai script development.
- **Purpose**: Real-time script development and testing
- **Features**: Enhanced CLI, dual execution modes, worker management
- **Development**: Syntax highlighting, script editing, immediate feedback
**Diagram:**
#### [`rhai_engine_ui`](../src/rhai_engine_ui/docs/ARCHITECTURE.md)
Web-based interface for Rhai script management and execution.
- **Purpose**: Browser-based script execution and management
- **Architecture**: WebAssembly frontend with optional server backend
- **Features**: Real-time updates, task management, visual interface
### Operations and Monitoring
#### [`monitor`](../src/monitor/docs/ARCHITECTURE.md)
Command-line monitoring and management tool for the rhailib ecosystem.
- **Purpose**: System observability and task management
- **Features**: Real-time monitoring, performance metrics, queue management
- **Operations**: Multi-worker support, interactive CLI, visualization
## Data Flow Architecture
### Script Execution Flow
```mermaid
sequenceDiagram
participant Client
participant Redis
participant Worker
Client->>Client: 1. Generate task_id & reply_queue_name
Client->>Redis: 2. LPUSH task_id to rhai_tasks:<circle>
Client->>Redis: 3. HSET rhai_task_details:<task_id> (script, status:pending, reply_to:reply_queue_name)
Client->>Redis: 4. BLPOP from reply_queue_name (waits with timeout)
Worker->>Redis: 5. BLPOP from rhai_tasks:<circle>
Redis-->>Worker: task_id
Worker->>Redis: 6. HGETALL rhai_task_details:<task_id>
Redis-->>Worker: task_details (script, reply_to)
Worker->>Redis: 7. HSET rhai_task_details:<task_id> (status:processing)
Note over Worker: Executes Rhai script
alt Script Success
Worker->>Redis: 8a. HSET rhai_task_details:<task_id> (status:completed, output)
Worker->>Redis: 9a. LPUSH to reply_queue_name (message: {task_id, status:completed, output})
else Script Error
Worker->>Redis: 8b. HSET rhai_task_details:<task_id> (status:error, error_msg)
Worker->>Redis: 9b. LPUSH to reply_queue_name (message: {task_id, status:error, error_msg})
end
Redis-->>Client: Result message from reply_queue_name
Client->>Client: Process result/error
Client->>Redis: 10. DEL reply_queue_name (optional cleanup)
participant Client as rhai_client
participant Redis as Redis Queue
participant Worker as rhailib_worker
participant Engine as rhailib_engine
participant DSL as rhailib_dsl
participant DB as Database
Client->>Redis: Submit script task (worker_id + context_id)
Worker->>Redis: Poll worker queue (worker_id)
Redis->>Worker: Return task with context_id
Worker->>Engine: Create configured engine
Engine->>DSL: Register domain modules
Worker->>Engine: Execute script with context_id
Engine->>DSL: Call business functions (context_id)
DSL->>DB: Perform authorized operations (context_id)
DB->>DSL: Return results
DSL->>Engine: Return processed data
Engine->>Worker: Return execution result
Worker->>Redis: Publish result to reply queue
Redis->>Client: Deliver result
```
This architecture allows for:
* Asynchronous and non-blocking script execution for the client.
* Scalable processing of Rhai scripts by running multiple workers.
* Efficient, event-driven result notification to clients.
* Robust task state tracking and observability.
### Authorization Flow
## 4. Redis Data Structures
```mermaid
sequenceDiagram
participant Script as Rhai Script
participant Macro as Authorization Macro
participant Context as Execution Context
participant Access as Access Control
participant DB as Database
Script->>Macro: Call authorized function
Macro->>Context: Extract caller credentials
Context->>Access: Validate permissions
Access->>DB: Check resource access
DB->>Access: Return authorization result
Access->>Macro: Grant/deny access
Macro->>DB: Execute authorized operation
DB->>Script: Return results
```
* **Task Queues:**
* Key Pattern: `rhai_tasks:<circle_name>`
* Type: List
* Purpose: FIFO queue for `task_id`s waiting to be processed by workers assigned to a specific circle. Workers use `BLPOP`. Clients use `LPUSH`.
* **Task Details:**
* Key Pattern: `rhai_task_details:<task_id>`
* Type: Hash
* Purpose: Stores all information about a specific task.
* Key Fields:
* `script`: The Rhai script content.
* `status`: Current state of the task (e.g., "pending", "processing", "completed", "error").
* `client_rpc_id`: Optional client-provided identifier.
* `output`: The result of a successful script execution.
* `error`: Error message if script execution failed.
* `created_at`: Timestamp of task creation.
* `updated_at`: Timestamp of the last update to the task details.
* `reply_to_queue`: (New) The name of the dedicated Redis List the client is listening on for the result.
* `publicKey`: (Optional) The public key of the user who submitted the task.
* **Reply Queues:**
* Key Pattern: `rhai_reply:<unique_identifier>` (e.g., `rhai_reply:<uuid_generated_by_client>`)
* Type: List
* Purpose: A temporary, client-specific queue where the worker pushes the final result/notification for a particular task. The client uses `BLPOP` to wait for a message on this queue.
## Worker-Context Decoupling Architecture
## 5. Key Design Choices
A key architectural feature of rhailib is the decoupling of worker assignment from context management:
* **Hashes for Task Details:** Storing comprehensive task details in a Redis Hash provides a persistent, inspectable, and easily updatable record of each task's lifecycle. This aids in monitoring, debugging, and potential recovery scenarios.
* **Dedicated Reply Queues:** Using a unique Redis List per client request for result notification offers reliable message delivery (compared to Pub/Sub's fire-and-forget) and allows the client to efficiently block until its specific result is ready, eliminating polling.
* **Status Tracking in Hash:** Maintaining the `status` field within the task details hash ("pending", "processing", "completed", "error") offers crucial observability into the system's state and the progress of individual tasks, independent of the client-worker notification flow.
### Traditional Model (Previous)
- **One Worker Per Circle**: Each worker was dedicated to a specific circle/context
- **Queue Per Circle**: Workers listened to circle-specific queues
- **Tight Coupling**: Worker identity was directly tied to context identity
### New Decoupled Model (Current)
- **Worker ID**: Determines which queue the worker listens to (`rhailib:<worker_id>`)
- **Context ID**: Provided in task details, determines execution context and database access
- **Flexible Assignment**: Single worker can process tasks for multiple contexts
### Benefits of Decoupling
1. **Resource Efficiency**: Better worker utilization across multiple contexts
2. **Deployment Flexibility**: Easier scaling and resource allocation
3. **Cost Optimization**: Fewer worker instances needed for multi-context scenarios
4. **Operational Simplicity**: Centralized worker management with distributed contexts
### Implementation Details
```mermaid
graph LR
subgraph "Client Layer"
C[Client] --> |worker_id + context_id| Q[Redis Queue]
end
subgraph "Worker Layer"
W1[Worker 1] --> |listens to| Q1[Queue: worker-1]
W2[Worker 2] --> |listens to| Q2[Queue: worker-2]
end
subgraph "Context Layer"
W1 --> |processes| CTX1[Context A]
W1 --> |processes| CTX2[Context B]
W2 --> |processes| CTX1
W2 --> |processes| CTX3[Context C]
end
```
## Key Design Principles
### 1. Security First
- **Multi-layer Authorization**: Context-based, resource-specific, and operation-level security
- **Secure Execution**: Isolated script execution with proper context injection
- **Audit Trails**: Comprehensive logging and monitoring of all operations
### 2. Scalability
- **Horizontal Scaling**: Multiple worker instances for load distribution
- **Queue-based Architecture**: Reliable task distribution and processing
- **Async Operations**: Non-blocking I/O throughout the system
### 3. Developer Experience
- **Type Safety**: Comprehensive type checking and conversion utilities
- **Error Handling**: Detailed error messages and proper error propagation
- **Interactive Development**: REPL and web interfaces for immediate feedback
### 4. Modularity
- **Feature Flags**: Configurable compilation based on requirements
- **Crate Separation**: Clear boundaries and responsibilities
- **Plugin Architecture**: Easy extension and customization
## Deployment Patterns
### Development Environment
```
REPL + Local Engine + Mock Database
```
- Interactive development with immediate feedback
- Full DSL access without external dependencies
- Integrated testing and debugging
### Testing Environment
```
Client + Worker + Redis + Mock Database
```
- Distributed execution testing
- Queue-based communication validation
- Performance and scalability testing
### Production Environment
```
Multiple Clients + Redis Cluster + Worker Pool + Production Database
```
- High availability and fault tolerance
- Horizontal scaling and load distribution
- Comprehensive monitoring and observability
## Integration Points
### External Systems
- **Redis**: Task queues, result delivery, system coordination
- **Databases**: Business data persistence and retrieval
- **Web Browsers**: WebAssembly-based user interfaces
- **Command Line**: Development and operations tooling
### Internal Integration
- **Macro System**: Code generation and authorization
- **Type System**: Safe conversions and error handling
- **Module System**: Domain-specific functionality organization
- **Context System**: Security and execution environment management
## Performance Characteristics
### Throughput
- **Concurrent Execution**: Multiple workers processing tasks simultaneously
- **Connection Pooling**: Efficient database and Redis connection management
- **Compiled Scripts**: AST caching for repeated execution optimization
### Latency
- **Local Execution**: Direct engine access for development scenarios
- **Queue Optimization**: Efficient task distribution and result delivery
- **Context Caching**: Reduced overhead for authorization and setup
### Resource Usage
- **Memory Management**: Efficient ownership and borrowing patterns
- **CPU Utilization**: Async operations and non-blocking I/O
- **Network Efficiency**: Optimized serialization and communication protocols
## Future Extensibility
### Adding New Domains
1. Create domain module in `rhailib_dsl`
2. Implement authorization macros in `macros`
3. Add feature flags and conditional compilation
4. Update engine registration and documentation
### Custom Authorization
1. Extend authorization macros with custom logic
2. Implement domain-specific access control functions
3. Add audit and logging capabilities
4. Update security documentation
### New Interfaces
1. Implement client interface following existing patterns
2. Integrate with Redis communication layer
3. Add monitoring and observability features
4. Provide comprehensive documentation
This architecture provides a robust, secure, and scalable foundation for distributed Rhai script execution while maintaining excellent developer experience and operational visibility.