rhailib/docs/ARCHITECTURE.md

Rhailib Architecture Overview

Rhailib is a comprehensive Rust-based ecosystem for executing Rhai scripts in distributed environments with full business domain support, authorization, and scalability features.

System Architecture

graph TB
    subgraph "Client Layer"
        A[rhai_client] --> B[Redis Task Queues]
        UI[rhai_engine_ui] --> B
        REPL[ui_repl] --> B
    end
    
    subgraph "Processing Layer"
        B --> C[rhailib_worker]
        C --> D[rhailib_engine]
        D --> E[rhailib_dsl]
    end
    
    subgraph "Core Infrastructure"
        E --> F[derive - Procedural Macros]
        E --> G[macros - Authorization]
        D --> H[mock_db - Testing]
    end
    
    subgraph "Operations Layer"
        I[monitor] --> B
        I --> C
    end
    
    subgraph "Data Layer"
        J[Redis] --> B
        K[Database] --> E
    end

Crate Overview

Core Engine Components

rhailib_engine

The central Rhai scripting engine that orchestrates all business domain modules.

• Purpose: Unified engine creation and script execution
• Features: Mock database, feature-based architecture, performance optimization
• Key Functions: create_heromodels_engine(), script compilation and execution

rhailib_dsl

Comprehensive Domain-Specific Language implementation exposing business models to Rhai.

• Purpose: Business domain integration with Rhai scripting
• Domains: Business operations, finance, content management, workflows, access control
• Features: Fluent APIs, type safety, authorization integration

Code Generation and Utilities

derive

Procedural macros for automatic Rhai integration code generation.

• Purpose: Simplify Rhai integration for custom types
• Macros: RhaiApi for DSL generation, FromVec for type conversion
• Features: Builder pattern generation, error handling

macros

Authorization macros and utilities for secure database operations.

• Purpose: Declarative security for Rhai functions
• Features: CRUD operation macros, access control, context management
• Security: Multi-level authorization, audit trails

Client and Communication

rhai_client

Redis-based client library for distributed script execution.

• Purpose: Submit and manage Rhai script execution requests
• Features: Builder pattern API, timeout handling, request-reply pattern
• Architecture: Async operations, connection pooling, error handling

rhailib_worker

Distributed task execution system for processing Rhai scripts.

• Purpose: Scalable script processing with queue-based architecture
• Features: Multi-context support, horizontal scaling, fault tolerance, context injection
• Architecture: Workers decoupled from contexts, allowing single worker to serve multiple circles
• Integration: Full engine and DSL access, secure execution

User Interfaces

ui_repl

Interactive development environment for Rhai script development.

• Purpose: Real-time script development and testing
• Features: Enhanced CLI, dual execution modes, worker management
• Development: Syntax highlighting, script editing, immediate feedback

rhai_engine_ui

Web-based interface for Rhai script management and execution.

• Purpose: Browser-based script execution and management
• Architecture: WebAssembly frontend with optional server backend
• Features: Real-time updates, task management, visual interface

Operations and Monitoring

monitor

Command-line monitoring and management tool for the rhailib ecosystem.

• Purpose: System observability and task management
• Features: Real-time monitoring, performance metrics, queue management
• Operations: Multi-worker support, interactive CLI, visualization

Data Flow Architecture

Script Execution Flow

sequenceDiagram
    participant Client as rhai_client
    participant Redis as Redis Queue
    participant Worker as rhailib_worker
    participant Engine as rhailib_engine
    participant DSL as rhailib_dsl
    participant DB as Database
    
    Client->>Redis: Submit script task (worker_id + context_id)
    Worker->>Redis: Poll worker queue (worker_id)
    Redis->>Worker: Return task with context_id
    Worker->>Engine: Create configured engine
    Engine->>DSL: Register domain modules
    Worker->>Engine: Execute script with context_id
    Engine->>DSL: Call business functions (context_id)
    DSL->>DB: Perform authorized operations (context_id)
    DB->>DSL: Return results
    DSL->>Engine: Return processed data
    Engine->>Worker: Return execution result
    Worker->>Redis: Publish result to reply queue
    Redis->>Client: Deliver result

Authorization Flow

sequenceDiagram
    participant Script as Rhai Script
    participant Macro as Authorization Macro
    participant Context as Execution Context
    participant Access as Access Control
    participant DB as Database
    
    Script->>Macro: Call authorized function
    Macro->>Context: Extract caller credentials
    Context->>Access: Validate permissions
    Access->>DB: Check resource access
    DB->>Access: Return authorization result
    Access->>Macro: Grant/deny access
    Macro->>DB: Execute authorized operation
    DB->>Script: Return results

Worker-Context Decoupling Architecture

A key architectural feature of rhailib is the decoupling of worker assignment from context management:

Traditional Model (Previous)

• One Worker Per Circle: Each worker was dedicated to a specific circle/context
• Queue Per Circle: Workers listened to circle-specific queues
• Tight Coupling: Worker identity was directly tied to context identity

New Decoupled Model (Current)

• Worker ID: Determines which queue the worker listens to (rhailib:<worker_id>)
• Context ID: Provided in task details, determines execution context and database access
• Flexible Assignment: Single worker can process tasks for multiple contexts

Benefits of Decoupling

1. Resource Efficiency: Better worker utilization across multiple contexts
2. Deployment Flexibility: Easier scaling and resource allocation
3. Cost Optimization: Fewer worker instances needed for multi-context scenarios
4. Operational Simplicity: Centralized worker management with distributed contexts

Implementation Details

graph LR
    subgraph "Client Layer"
        C[Client] --> |worker_id + context_id| Q[Redis Queue]
    end
    
    subgraph "Worker Layer"
        W1[Worker 1] --> |listens to| Q1[Queue: worker-1]
        W2[Worker 2] --> |listens to| Q2[Queue: worker-2]
    end
    
    subgraph "Context Layer"
        W1 --> |processes| CTX1[Context A]
        W1 --> |processes| CTX2[Context B]
        W2 --> |processes| CTX1
        W2 --> |processes| CTX3[Context C]
    end

Key Design Principles

1. Security First

• Multi-layer Authorization: Context-based, resource-specific, and operation-level security
• Secure Execution: Isolated script execution with proper context injection
• Audit Trails: Comprehensive logging and monitoring of all operations

2. Scalability

• Horizontal Scaling: Multiple worker instances for load distribution
• Queue-based Architecture: Reliable task distribution and processing
• Async Operations: Non-blocking I/O throughout the system

3. Developer Experience

• Type Safety: Comprehensive type checking and conversion utilities
• Error Handling: Detailed error messages and proper error propagation
• Interactive Development: REPL and web interfaces for immediate feedback

4. Modularity

• Feature Flags: Configurable compilation based on requirements
• Crate Separation: Clear boundaries and responsibilities
• Plugin Architecture: Easy extension and customization

Deployment Patterns

Development Environment

REPL + Local Engine + Mock Database

• Interactive development with immediate feedback
• Full DSL access without external dependencies
• Integrated testing and debugging

Testing Environment

Client + Worker + Redis + Mock Database

• Distributed execution testing
• Queue-based communication validation
• Performance and scalability testing

Production Environment

Multiple Clients + Redis Cluster + Worker Pool + Production Database

• High availability and fault tolerance
• Horizontal scaling and load distribution
• Comprehensive monitoring and observability

Integration Points

External Systems

• Redis: Task queues, result delivery, system coordination
• Databases: Business data persistence and retrieval
• Web Browsers: WebAssembly-based user interfaces
• Command Line: Development and operations tooling

Internal Integration

• Macro System: Code generation and authorization
• Type System: Safe conversions and error handling
• Module System: Domain-specific functionality organization
• Context System: Security and execution environment management

Performance Characteristics

Throughput

• Concurrent Execution: Multiple workers processing tasks simultaneously
• Connection Pooling: Efficient database and Redis connection management
• Compiled Scripts: AST caching for repeated execution optimization

Latency

• Local Execution: Direct engine access for development scenarios
• Queue Optimization: Efficient task distribution and result delivery
• Context Caching: Reduced overhead for authorization and setup

Resource Usage

• Memory Management: Efficient ownership and borrowing patterns
• CPU Utilization: Async operations and non-blocking I/O
• Network Efficiency: Optimized serialization and communication protocols

Future Extensibility

Adding New Domains

1. Create domain module in rhailib_dsl
2. Implement authorization macros in macros
3. Add feature flags and conditional compilation
4. Update engine registration and documentation

Custom Authorization

1. Extend authorization macros with custom logic
2. Implement domain-specific access control functions
3. Add audit and logging capabilities
4. Update security documentation

New Interfaces

1. Implement client interface following existing patterns
2. Integrate with Redis communication layer
3. Add monitoring and observability features
4. Provide comprehensive documentation

This architecture provides a robust, secure, and scalable foundation for distributed Rhai script execution while maintaining excellent developer experience and operational visibility.