Rhailib Architecture: Distributed Rhai Scripting

1. Overview

rhailib provides a robust infrastructure for executing Rhai scripts in a distributed manner, primarily designed to integrate with and extend the HeroModels ecosystem. It allows for dynamic scripting capabilities, offloading computation, and enabling flexible automation. This document describes the target architecture utilizing dedicated reply queues for efficient result notification.

2. Core Components

The rhailib system is composed of the following main components, leveraging Redis for task queuing, state management, and result notification:

Rhai Engine (src/engine):
- The core scripting capability. Provides a Rhai engine pre-configured with various modules.
- Utilized by the rhai_worker to process tasks.
Rhai Client (src/client):
- Offers an interface for applications to submit Rhai scripts as tasks.
- Submits tasks to named Redis queues ("circles").
- Waits for results on a dedicated reply queue, avoiding polling.
Rhai Worker (src/worker):
- Listens to Redis task queues ("circles") for incoming task IDs.
- Fetches task details, executes the script using the rhai_engine.
- Updates task status and results in Redis.
- Sends a notification/result to the client's dedicated reply queue.
Redis:
- Acts as the message broker and data store for task queues, detailed task information (including scripts, status, and results), and reply queues for notifications.

3. Architecture & Workflow (Dedicated Reply Queues)

The system employs a "Dedicated Reply Queue" pattern to notify clients of task completion, enhancing efficiency by eliminating client-side polling for results.

Workflow:

Task Submission (Client): a. The client generates a unique task_id and a unique reply_queue_name (e.g., rhai_reply:<uuid>). b. Task details, including the script, initial status ("pending"), and the reply_queue_name, are stored in a Redis Hash: rhai_task_details:<task_id>. c. The task_id is pushed onto a Redis List acting as a task queue for a specific "circle": rhai_tasks:<circle_name>. d. The client then performs a blocking pop (BLPOP) on its reply_queue_name, waiting for the result message.
Task Consumption & Processing (Worker): a. A rhai_worker instance, listening to one or more rhai_tasks:<circle_name> queues, picks up a task_id using BLPOP. b. The worker retrieves the full task details (including the script and reply_queue_name) from the rhai_task_details:<task_id> hash. c. The worker updates the task's status in the hash to "processing". d. The Rhai script is executed using an instance of the rhai_engine.
Result Storage & Notification (Worker): a. Upon completion (or error), the worker updates the rhai_task_details:<task_id> hash with the final status ("completed" or "error") and the script's output or error message. b. If a reply_queue_name was provided in the task details, the worker constructs a result message (e.g., JSON containing task_id, final status, output/error). c. The worker pushes this result message onto the specified reply_queue_name using LPUSH.
Result Reception (Client): a. The client's BLPOP on its reply_queue_name receives the result message. b. The client processes the result or error. c. Optionally, the client may DELete its temporary reply queue.

Diagram:

sequenceDiagram
    participant Client
    participant Redis
    participant Worker

    Client->>Client: 1. Generate task_id & reply_queue_name
    Client->>Redis: 2. LPUSH task_id to rhai_tasks:<circle>
    Client->>Redis: 3. HSET rhai_task_details:<task_id> (script, status:pending, reply_to:reply_queue_name)
    Client->>Redis: 4. BLPOP from reply_queue_name (waits with timeout)

    Worker->>Redis: 5. BLPOP from rhai_tasks:<circle>
    Redis-->>Worker: task_id
    Worker->>Redis: 6. HGETALL rhai_task_details:<task_id>
    Redis-->>Worker: task_details (script, reply_to)
    Worker->>Redis: 7. HSET rhai_task_details:<task_id> (status:processing)
    Note over Worker: Executes Rhai script

    alt Script Success
        Worker->>Redis: 8a. HSET rhai_task_details:<task_id> (status:completed, output)
        Worker->>Redis: 9a. LPUSH to reply_queue_name (message: {task_id, status:completed, output})
    else Script Error
        Worker->>Redis: 8b. HSET rhai_task_details:<task_id> (status:error, error_msg)
        Worker->>Redis: 9b. LPUSH to reply_queue_name (message: {task_id, status:error, error_msg})
    end

    Redis-->>Client: Result message from reply_queue_name
    Client->>Client: Process result/error
    Client->>Redis: 10. DEL reply_queue_name (optional cleanup)

This architecture allows for:

Asynchronous and non-blocking script execution for the client.
Scalable processing of Rhai scripts by running multiple workers.
Efficient, event-driven result notification to clients.
Robust task state tracking and observability.

4. Redis Data Structures

Task Queues:
- Key Pattern: rhai_tasks:<circle_name>
- Type: List
- Purpose: FIFO queue for task_ids waiting to be processed by workers assigned to a specific circle. Workers use BLPOP. Clients use LPUSH.
Task Details:
- Key Pattern: rhai_task_details:<task_id>
- Type: Hash
- Purpose: Stores all information about a specific task.
- Key Fields:
  - script: The Rhai script content.
  - status: Current state of the task (e.g., "pending", "processing", "completed", "error").
  - client_rpc_id: Optional client-provided identifier.
  - output: The result of a successful script execution.
  - error: Error message if script execution failed.
  - created_at: Timestamp of task creation.
  - updated_at: Timestamp of the last update to the task details.
  - reply_to_queue: (New) The name of the dedicated Redis List the client is listening on for the result.
Reply Queues:
- Key Pattern: rhai_reply:<unique_identifier> (e.g., rhai_reply:<uuid_generated_by_client>)
- Type: List
- Purpose: A temporary, client-specific queue where the worker pushes the final result/notification for a particular task. The client uses BLPOP to wait for a message on this queue.

5. Key Design Choices

Hashes for Task Details: Storing comprehensive task details in a Redis Hash provides a persistent, inspectable, and easily updatable record of each task's lifecycle. This aids in monitoring, debugging, and potential recovery scenarios.
Dedicated Reply Queues: Using a unique Redis List per client request for result notification offers reliable message delivery (compared to Pub/Sub's fire-and-forget) and allows the client to efficiently block until its specific result is ready, eliminating polling.
Status Tracking in Hash: Maintaining the status field within the task details hash ("pending", "processing", "completed", "error") offers crucial observability into the system's state and the progress of individual tasks, independent of the client-worker notification flow.

7.2 KiB Raw Blame History