herocode/baobab
Archived
13
0
Fork 0
Code Issues 8 Pull Requests Actions Packages Projects Releases Wiki Activity

initial commit

Browse Source
This commit is contained in:
Timur Gordon
2025-07-29 01:15:23 +02:00
commit 7d7ff0f0ab
108 changed files with 24713 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
core/dispatcher/.gitignore vendored Normal file
View File

@@ -0,0 +1 @@
/target

25
core/dispatcher/Cargo.toml Normal file
View File

@@ -0,0 +1,25 @@
[package]
name = "hero_dispatcher"
version = "0.1.0"
edition = "2021"
[[bin]]
name = "dispatcher"
path = "cmd/dispatcher.rs"
[dependencies]
clap = { version = "4.4", features = ["derive"] }
env_logger = "0.10"
redis = { version = "0.25.0", features = ["tokio-comp"] }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
uuid = { version = "1.6", features = ["v4", "serde"] }
chrono = { version = "0.4", features = ["serde"] }
log = "0.4"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] } # For async main in examples, and general async
colored = "2.0"
hero_job = { path = "../job" }
[dev-dependencies] # For examples later
env_logger = "0.10"
rhai = "1.18.0" # For examples that might need to show engine setup

128
core/dispatcher/README.md Normal file
View File

@@ -0,0 +1,128 @@
# Hero Dispatcher
A Redis-based job dispatcher for managing Rhai/HeroScript execution across distributed workers.
## Overview
The Hero Dispatcher provides a robust job queue system where:
- **Jobs** represent script execution requests (Rhai or HeroScript)
- **Creating a job** stores job parameters in Redis as an hset entry
- **Submitting a job** pushes the job ID to a worker's queue
- **Running a job** creates, submits, and awaits results on a dedicated reply queue
## Key Features
- **Asynchronous Operations**: Built with `tokio` for non-blocking I/O
- **Request-Reply Pattern**: Submit jobs and await results without polling
- **Configurable Jobs**: Set timeouts, retries, concurrency, and logging options
- **Worker Targeting**: Direct job routing to specific worker queues
- **Job Lifecycle**: Create, submit, monitor status, and retrieve results
## Core Components
### `DispatcherBuilder`
Builder for creating `Dispatcher` instances with caller ID, worker ID, context ID, and Redis URL.
### `Dispatcher`
Main interface for job management:
- `new_job()` - Create a new `JobBuilder`
- `create_job()` - Store job in Redis
- `run_job_and_await_result()` - Execute job and wait for completion
- `get_job_status()` - Check job execution status
- `get_job_output()` - Retrieve job results
### `JobBuilder`
Fluent builder for configuring jobs:
- `script()` - Set the script content
- `worker_id()` - Target specific worker
- `timeout()` - Set execution timeout
- `build()` - Create the job
- `submit()` - Fire-and-forget submission
- `await_response()` - Submit and wait for result
### `Job`
Represents a script execution request with:
- Unique ID and timestamps
- Script content and target worker
- Execution settings (timeout, retries, concurrency)
- Logging configuration
## Redis Schema
Jobs are stored using the `hero:` namespace:
- `hero:job:{job_id}` - Job parameters as Redis hash
- `hero:work_queue:{worker_id}` - Worker-specific job queues
- `hero:reply:{job_id}` - Dedicated reply queues for results
## Prerequisites
- Redis server accessible by dispatcher and workers
## Usage Example
### Basic Job Creation and Submission
```rust
use hero_dispatcher::{DispatcherBuilder, DispatcherError};
use std::time::Duration;
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
// Create dispatcher
let dispatcher = DispatcherBuilder::new()
.caller_id("my-app")
.worker_id("worker-1")
.context_id("my-context")
.redis_url("redis://127.0.0.1:6379")
.build()?;
// Create a job
let job = dispatcher
.new_job()
.script(r#"print("Hello from worker!"); "success""#)
.timeout(Duration::from_secs(30))
.build()?;
// Store job in Redis
dispatcher.create_job(&job)?;
println!("Job {} created and stored in Redis", job.id);
// Run job and await result (requires worker)
match dispatcher.run_job_and_await_result(&job, "worker-1".to_string()) {
Ok(result) => println!("Job completed: {}", result),
Err(DispatcherError::Timeout(_)) => println!("Job timed out"),
Err(e) => println!("Job failed: {}", e),
}
Ok(())
}
```
### Job Status Monitoring
```rust
// Check job status
match dispatcher.get_job_status(&job.id) {
Ok(status) => println!("Job status: {:?}", status),
Err(e) => println!("Error getting status: {}", e),
}
// Get job output
match dispatcher.get_job_output(&job.id) {
Ok(output) => println!("Job output: {:?}", output),
Err(e) => println!("Error getting output: {}", e),
}
```
## Examples
Run the comprehensive demo to see dispatcher functionality and Redis entries:
```bash
cargo run --example dispatcher_demo
```
Other examples:
- `timeout_example.rs` - Demonstrates timeout handling
Ensure Redis is running at `redis://127.0.0.1:6379`.

157
core/dispatcher/cmd/README.md Normal file
View File

@@ -0,0 +1,157 @@
# Rhai Client Binary
A command-line client for executing Rhai scripts on remote workers via Redis.
## Binary: `client`
### Installation
Build the binary:
```bash
cargo build --bin client --release
```
### Usage
```bash
# Basic usage - requires caller and circle keys
client --caller-key <CALLER_KEY> --circle-key <CIRCLE_KEY>
# Execute inline script
client -c <CALLER_KEY> -k <CIRCLE_KEY> --script "print('Hello World!')"
# Execute script from file
client -c <CALLER_KEY> -k <CIRCLE_KEY> --file script.rhai
# Use specific worker (defaults to circle key)
client -c <CALLER_KEY> -k <CIRCLE_KEY> -w <WORKER_KEY> --script "2 + 2"
# Custom Redis and timeout
client -c <CALLER_KEY> -k <CIRCLE_KEY> --redis-url redis://localhost:6379/1 --timeout 60
# Remove timestamps from logs
client -c <CALLER_KEY> -k <CIRCLE_KEY> --no-timestamp
# Increase verbosity
client -c <CALLER_KEY> -k <CIRCLE_KEY> -v --script "debug_info()"
```
### Command-Line Options
| Option | Short | Default | Description |
|--------|-------|---------|-------------|
| `--caller-key` | `-c` | **Required** | Caller public key (your identity) |
| `--circle-key` | `-k` | **Required** | Circle public key (execution context) |
| `--worker-key` | `-w` | `circle-key` | Worker public key (target worker) |
| `--redis-url` | `-r` | `redis://localhost:6379` | Redis connection URL |
| `--script` | `-s` | | Rhai script to execute |
| `--file` | `-f` | | Path to Rhai script file |
| `--timeout` | `-t` | `30` | Timeout for script execution (seconds) |
| `--no-timestamp` | | `false` | Remove timestamps from log output |
| `--verbose` | `-v` | | Increase verbosity (stackable) |
### Execution Modes
#### Inline Script Execution
```bash
# Execute a simple calculation
client -c caller_123 -k circle_456 -s "let result = 2 + 2; print(result);"
# Execute with specific worker
client -c caller_123 -k circle_456 -w worker_789 -s "get_user_data()"
```
#### Script File Execution
```bash
# Execute script from file
client -c caller_123 -k circle_456 -f examples/data_processing.rhai
# Execute with custom timeout
client -c caller_123 -k circle_456 -f long_running_script.rhai -t 120
```
#### Interactive Mode
```bash
# Enter interactive REPL mode (when no script or file provided)
client -c caller_123 -k circle_456
# Interactive mode with verbose logging
client -c caller_123 -k circle_456 -v --no-timestamp
```
### Interactive Mode
When no script (`-s`) or file (`-f`) is provided, the client enters interactive mode:
```
🔗 Starting Rhai Client
📋 Configuration:
Caller Key: caller_123
Circle Key: circle_456
Worker Key: circle_456
Redis URL: redis://localhost:6379
Timeout: 30s
✅ Connected to Redis at redis://localhost:6379
🎮 Entering interactive mode
Type Rhai scripts and press Enter to execute. Type 'exit' or 'quit' to close.
rhai> let x = 42; print(x);
Status: completed
Output: 42
rhai> exit
👋 Goodbye!
```
### Configuration Examples
#### Development Usage
```bash
# Simple development client
client -c dev_user -k dev_circle
# Development with clean logs
client -c dev_user -k dev_circle --no-timestamp -v
```
#### Production Usage
```bash
# Production client with specific worker
client \
--caller-key prod_user_123 \
--circle-key prod_circle_456 \
--worker-key prod_worker_789 \
--redis-url redis://redis-cluster:6379/0 \
--timeout 300 \
--file production_script.rhai
```
#### Batch Processing
```bash
# Process multiple scripts
for script in scripts/*.rhai; do
client -c batch_user -k batch_circle -f "$script" --no-timestamp
done
```
### Key Concepts
- **Caller Key**: Your identity - used for authentication and tracking
- **Circle Key**: Execution context - defines the environment/permissions
- **Worker Key**: Target worker - which worker should execute the script (defaults to circle key)
### Error Handling
The client provides clear error messages for:
- Missing required keys
- Redis connection failures
- Script execution timeouts
- Worker unavailability
- Script syntax errors
### Dependencies
- `rhai_dispatcher`: Core client library for Redis-based script execution
- `redis`: Redis client for task queue communication
- `clap`: Command-line argument parsing
- `env_logger`: Logging infrastructure
- `tokio`: Async runtime

271
core/dispatcher/cmd/dispatcher.rs Normal file
View File

@@ -0,0 +1,271 @@
use clap::Parser;
use hero_dispatcher::{Dispatcher, DispatcherBuilder, ScriptType};
use log::{error, info};
use colored::Colorize;
use std::io::{self, Write};
use std::time::Duration;
#[derive(Parser, Debug)]
#[command(author, version, about = "Rhai Client - Script execution client", long_about = None)]
struct Args {
/// Caller ID (your identity)
#[arg(short = 'c', long = "caller-id", help = "Caller ID (your identity)")]
caller_id: String,
/// Context ID (execution context)
#[arg(short = 'k', long = "context-id", help = "Context ID (execution context)")]
context_id: String,
/// Script type to execute (heroscript, rhai-sal, rhai-dsl)
#[arg(short = 'T', long = "script-type", default_value = "heroscript", help = "Script type: heroscript, rhai-sal, or rhai-dsl")]
script_type: String,
/// HeroScript workers (comma-separated)
#[arg(long = "hero-workers", default_value = "hero-worker-1", help = "HeroScript worker IDs (comma-separated)")]
hero_workers: String,
/// Rhai SAL workers (comma-separated)
#[arg(long = "rhai-sal-workers", default_value = "rhai-sal-worker-1", help = "Rhai SAL worker IDs (comma-separated)")]
rhai_sal_workers: String,
/// Rhai DSL workers (comma-separated)
#[arg(long = "rhai-dsl-workers", default_value = "rhai-dsl-worker-1", help = "Rhai DSL worker IDs (comma-separated)")]
rhai_dsl_workers: String,
/// Redis URL
#[arg(short, long, default_value = "redis://localhost:6379", help = "Redis connection URL")]
redis_url: String,
/// Rhai script to execute
#[arg(short, long, help = "Rhai script to execute")]
script: Option<String>,
/// Path to Rhai script file
#[arg(short, long, help = "Path to Rhai script file")]
file: Option<String>,
/// Timeout for script execution (in seconds)
#[arg(short, long, default_value = "30", help = "Timeout for script execution in seconds")]
timeout: u64,
/// Increase verbosity (can be used multiple times)
#[arg(short, long, action = clap::ArgAction::Count, help = "Increase verbosity (-v for debug, -vv for trace)")]
verbose: u8,
/// Disable timestamps in log output
#[arg(long, help = "Remove timestamps from log output")]
no_timestamp: bool,
}
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let args = Args::parse();
// Configure logging based on verbosity level
let log_config = match args.verbose {
0 => "warn,hero_dispatcher=warn",
1 => "info,hero_dispatcher=info",
2 => "debug,hero_dispatcher=debug",
_ => "trace,hero_dispatcher=trace",
};
std::env::set_var("RUST_LOG", log_config);
// Configure env_logger with or without timestamps
if args.no_timestamp {
env_logger::Builder::from_default_env()
.format_timestamp(None)
.init();
} else {
env_logger::init();
}
// Parse worker lists
let hero_workers: Vec<String> = args.hero_workers.split(',').map(|s| s.trim().to_string()).filter(|s| !s.is_empty()).collect();
let rhai_sal_workers: Vec<String> = args.rhai_sal_workers.split(',').map(|s| s.trim().to_string()).filter(|s| !s.is_empty()).collect();
let rhai_dsl_workers: Vec<String> = args.rhai_dsl_workers.split(',').map(|s| s.trim().to_string()).filter(|s| !s.is_empty()).collect();
// Validate that at least one worker is provided for the selected script type
match args.script_type.to_lowercase().as_str() {
"heroscript" => {
if hero_workers.is_empty() {
error!("❌ No HeroScript workers provided. Use --hero-workers to specify at least one worker.");
return Err("At least one HeroScript worker must be provided".into());
}
}
"rhai-sal" => {
if rhai_sal_workers.is_empty() {
error!("❌ No Rhai SAL workers provided. Use --rhai-sal-workers to specify at least one worker.");
return Err("At least one Rhai SAL worker must be provided".into());
}
}
"rhai-dsl" => {
if rhai_dsl_workers.is_empty() {
error!("❌ No Rhai DSL workers provided. Use --rhai-dsl-workers to specify at least one worker.");
return Err("At least one Rhai DSL worker must be provided".into());
}
}
_ => {
error!("❌ Invalid script type: {}. Valid types: heroscript, rhai-sal, rhai-dsl", args.script_type);
return Err(format!("Invalid script type: {}", args.script_type).into());
}
}
if args.verbose > 0 {
info!("🔗 Starting Hero Dispatcher");
info!("📋 Configuration:");
info!(" Caller ID: {}", args.caller_id);
info!(" Context ID: {}", args.context_id);
info!(" Script Type: {}", args.script_type);
info!(" HeroScript Workers: {:?}", hero_workers);
info!(" Rhai SAL Workers: {:?}", rhai_sal_workers);
info!(" Rhai DSL Workers: {:?}", rhai_dsl_workers);
info!(" Redis URL: {}", args.redis_url);
info!(" Timeout: {}s", args.timeout);
info!("");
}
// Create the dispatcher client
let client = DispatcherBuilder::new()
.caller_id(&args.caller_id)
.context_id(&args.context_id)
.heroscript_workers(hero_workers)
.rhai_sal_workers(rhai_sal_workers)
.rhai_dsl_workers(rhai_dsl_workers)
.redis_url(&args.redis_url)
.build()?;
if args.verbose > 0 {
info!("✅ Connected to Redis at {}", args.redis_url);
}
// Determine execution mode
if let Some(script_content) = args.script {
// Execute inline script
if args.verbose > 0 {
info!("📜 Executing inline script");
}
execute_script(&client, script_content, &args.script_type, args.timeout).await?;
} else if let Some(file_path) = args.file {
// Execute script from file
if args.verbose > 0 {
info!("📁 Loading script from file: {}", file_path);
}
let script_content = std::fs::read_to_string(&file_path)
.map_err(|e| format!("Failed to read script file '{}': {}", file_path, e))?;
execute_script(&client, script_content, &args.script_type, args.timeout).await?;
} else {
// Interactive mode
info!("🎮 Entering interactive mode");
info!("Type Rhai scripts and press Enter to execute. Type 'exit' or 'quit' to close.");
run_interactive_mode(&client, &args.script_type, args.timeout, args.verbose).await?;
}
Ok(())
}
async fn execute_script(
client: &Dispatcher,
script: String,
script_type_str: &str,
timeout_secs: u64,
) -> Result<(), Box<dyn std::error::Error>> {
info!("⚡ Executing script: {:.50}...", script);
// Parse script type
let script_type = match script_type_str.to_lowercase().as_str() {
"heroscript" => ScriptType::HeroScript,
"rhai-sal" => ScriptType::RhaiSAL,
"rhai-dsl" => ScriptType::RhaiDSL,
_ => {
error!("❌ Invalid script type: {}. Valid types: heroscript, rhai-sal, rhai-dsl", script_type_str);
return Err(format!("Invalid script type: {}", script_type_str).into());
}
};
let timeout = Duration::from_secs(timeout_secs);
match client
.new_job()
.script_type(script_type)
.script(&script)
.timeout(timeout)
.await_response()
.await
{
Ok(result) => {
info!("✅ Script execution completed");
println!("{}", "Result:".green().bold());
println!("{}", result);
}
Err(e) => {
error!("❌ Script execution failed: {}", e);
return Err(Box::new(e));
}
}
Ok(())
}
async fn run_interactive_mode(
client: &Dispatcher,
script_type_str: &str,
timeout_secs: u64,
verbose: u8,
) -> Result<(), Box<dyn std::error::Error>> {
// Parse script type
let script_type = match script_type_str.to_lowercase().as_str() {
"heroscript" => ScriptType::HeroScript,
"rhai-sal" => ScriptType::RhaiSAL,
"rhai-dsl" => ScriptType::RhaiDSL,
_ => {
error!("❌ Invalid script type: {}. Valid types: heroscript, rhai-sal, rhai-dsl", script_type_str);
return Err(format!("Invalid script type: {}", script_type_str).into());
}
};
let timeout = Duration::from_secs(timeout_secs);
loop {
print!("rhai> ");
io::stdout().flush()?;
let mut input = String::new();
io::stdin().read_line(&mut input)?;
let input = input.trim();
if input.is_empty() {
continue;
}
if input == "exit" || input == "quit" {
info!("👋 Goodbye!");
break;
}
if verbose > 0 {
info!("⚡ Executing: {}", input);
}
match client
.new_job()
.script_type(script_type.clone())
.script(input)
.timeout(timeout)
.await_response()
.await
{
Ok(result) => {
println!("{}", result.green());
}
Err(e) => {
println!("{}", format!("error: {}", e).red());
}
}
println!(); // Add blank line for readability
}
Ok(())
}

190
core/dispatcher/docs/ARCHITECTURE.md Normal file
View File

@@ -0,0 +1,190 @@
# Architecture of the `rhai_dispatcher` Crate
The `rhai_dispatcher` crate provides a Redis-based client library for submitting Rhai scripts to distributed worker services and awaiting their execution results. It implements a request-reply pattern using Redis as the message broker.
## Core Architecture
The client follows a builder pattern design with clear separation of concerns:
```mermaid
graph TD
A[RhaiDispatcherBuilder] --> B[RhaiDispatcher]
B --> C[PlayRequestBuilder]
C --> D[PlayRequest]
D --> E[Redis Task Queue]
E --> F[Worker Service]
F --> G[Redis Reply Queue]
G --> H[Client Response]
subgraph "Client Components"
A
B
C
D
end
subgraph "Redis Infrastructure"
E
G
end
subgraph "External Services"
F
end
```
## Key Components
### 1. RhaiDispatcherBuilder
A builder pattern implementation for constructing `RhaiDispatcher` instances with proper configuration validation.
**Responsibilities:**
- Configure Redis connection URL
- Set caller ID for task attribution
- Validate configuration before building client
**Key Methods:**
- `caller_id(id: &str)` - Sets the caller identifier
- `redis_url(url: &str)` - Configures Redis connection
- `build()` - Creates the final `RhaiDispatcher` instance
### 2. RhaiDispatcher
The main client interface that manages Redis connections and provides factory methods for creating play requests.
**Responsibilities:**
- Maintain Redis connection pool
- Provide factory methods for request builders
- Handle low-level Redis operations
- Manage task status queries
**Key Methods:**
- `new_play_request()` - Creates a new `PlayRequestBuilder`
- `get_task_status(task_id)` - Queries task status from Redis
- Internal methods for Redis operations
### 3. PlayRequestBuilder
A fluent builder for constructing and submitting script execution requests.
**Responsibilities:**
- Configure script execution parameters
- Handle script loading from files or strings
- Manage request timeouts
- Provide submission methods (fire-and-forget vs await-response)
**Key Methods:**
- `worker_id(id: &str)` - Target worker queue (determines which worker processes the task)
- `context_id(id: &str)` - Target context ID (determines execution context/circle)
- `script(content: &str)` - Set script content directly
- `script_path(path: &str)` - Load script from file
- `timeout(duration: Duration)` - Set execution timeout
- `submit()` - Fire-and-forget submission
- `await_response()` - Submit and wait for result
**Architecture Note:** The decoupling of `worker_id` and `context_id` allows a single worker to process tasks for multiple contexts (circles), providing greater deployment flexibility.
### 4. Data Structures
#### RhaiTaskDetails
Represents the complete state of a task throughout its lifecycle.
```rust
pub struct RhaiTaskDetails {
pub task_id: String,
pub script: String,
pub status: String, // "pending", "processing", "completed", "error"
pub output: Option<String>,
pub error: Option<String>,
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
pub caller_id: String,
}
```
#### RhaiDispatcherError
Comprehensive error handling for various failure scenarios:
- `RedisError` - Redis connection/operation failures
- `SerializationError` - JSON serialization/deserialization issues
- `Timeout` - Task execution timeouts
- `TaskNotFound` - Missing tasks after submission
## Communication Protocol
### Task Submission Flow
1. **Task Creation**: Client generates unique UUID for task identification
2. **Task Storage**: Task details stored in Redis hash: `rhailib:<task_id>`
3. **Queue Submission**: Task ID pushed to worker queue: `rhailib:<worker_id>`
4. **Reply Queue Setup**: Client listens on: `rhailib:reply:<task_id>`
### Redis Key Patterns
- **Task Storage**: `rhailib:<task_id>` (Redis Hash)
- **Worker Queues**: `rhailib:<worker_id>` (Redis List)
- **Reply Queues**: `rhailib:reply:<task_id>` (Redis List)
### Message Flow Diagram
```mermaid
sequenceDiagram
participant C as Client
participant R as Redis
participant W as Worker
C->>R: HSET rhailib:task_id (task details)
C->>R: LPUSH rhailib:worker_id task_id
C->>R: BLPOP rhailib:reply:task_id (blocking)
W->>R: BRPOP rhailib:worker_id (blocking)
W->>W: Execute Rhai Script
W->>R: LPUSH rhailib:reply:task_id (result)
R->>C: Return result from BLPOP
C->>R: DEL rhailib:reply:task_id (cleanup)
```
## Concurrency and Async Design
The client is built on `tokio` for asynchronous operations:
- **Connection Pooling**: Uses Redis multiplexed connections for efficiency
- **Non-blocking Operations**: All Redis operations are async
- **Timeout Handling**: Configurable timeouts with proper cleanup
- **Error Propagation**: Comprehensive error handling with context
## Configuration and Deployment
### Prerequisites
- Redis server accessible to both client and workers
- Proper network connectivity between components
- Sufficient Redis memory for task storage
### Configuration Options
- **Redis URL**: Connection string for Redis instance
- **Caller ID**: Unique identifier for client instance
- **Timeouts**: Per-request timeout configuration
- **Worker Targeting**: Direct worker queue addressing
## Security Considerations
- **Task Isolation**: Each task uses unique identifiers
- **Queue Separation**: Worker-specific queues prevent cross-contamination
- **Cleanup**: Automatic cleanup of reply queues after completion
- **Error Handling**: Secure error propagation without sensitive data leakage
## Performance Characteristics
- **Scalability**: Horizontal scaling through multiple worker instances
- **Throughput**: Limited by Redis performance and network latency
- **Memory Usage**: Efficient with connection pooling and cleanup
- **Latency**: Low latency for local Redis deployments
## Integration Points
The client integrates with:
- **Worker Services**: Via Redis queue protocol
- **Monitoring Systems**: Through structured logging
- **Application Code**: Via builder pattern API
- **Configuration Systems**: Through environment variables and builders

272
core/dispatcher/docs/protocol.md Normal file
View File

@@ -0,0 +1,272 @@
# Hero Dispatcher Protocol
This document describes the Redis-based protocol used by the Hero Dispatcher for job management and worker communication.
## Overview
The Hero Dispatcher uses Redis as a message broker and data store for managing distributed job execution. Jobs are stored as Redis hashes, and communication with workers happens through Redis lists (queues).
## Redis Namespace
All dispatcher-related keys use the `hero:` namespace prefix to avoid conflicts with other Redis usage.
## Data Structures
### Job Storage
Jobs are stored as Redis hashes with the following key pattern:
```
hero:job:{job_id}
```
**Job Hash Fields:**
- `id`: Unique job identifier (UUID v4)
- `caller_id`: Identifier of the client that created the job
- `worker_id`: Target worker identifier
- `context_id`: Execution context identifier
- `script`: Script content to execute (Rhai or HeroScript)
- `timeout`: Execution timeout in seconds
- `retries`: Number of retry attempts
- `concurrent`: Whether to execute in separate thread (true/false)
- `log_path`: Optional path to log file for job output
- `created_at`: Job creation timestamp (ISO 8601)
- `updated_at`: Job last update timestamp (ISO 8601)
- `status`: Current job status (dispatched/started/error/finished)
- `env_vars`: Environment variables as JSON object (optional)
- `prerequisites`: JSON array of job IDs that must complete before this job (optional)
- `dependents`: JSON array of job IDs that depend on this job completing (optional)
- `output`: Job execution result (set by worker)
- `error`: Error message if job failed (set by worker)
- `dependencies`: List of job IDs that this job depends on
### Job Dependencies
Jobs can have dependencies on other jobs, which are stored in the `dependencies` field. A job will not be dispatched until all its dependencies have completed successfully.
### Work Queues
Jobs are queued for execution using Redis lists:
```
hero:work_queue:{worker_id}
```
Workers listen on their specific queue using `BLPOP` for job IDs to process.
### Stop Queues
Job stop requests are sent through dedicated stop queues:
```
hero:stop_queue:{worker_id}
```
Workers monitor these queues to receive stop requests for running jobs.
### Reply Queues
For synchronous job execution, dedicated reply queues are used:
```
hero:reply:{job_id}
```
Workers send results to these queues when jobs complete.
## Job Lifecycle
### 1. Job Creation
```
Client -> Redis: HSET hero:job:{job_id} {job_fields}
```
### 2. Job Submission
```
Client -> Redis: LPUSH hero:work_queue:{worker_id} {job_id}
```
### 3. Job Processing
```
Worker -> Redis: BLPOP hero:work_queue:{worker_id}
Worker -> Redis: HSET hero:job:{job_id} status "started"
Worker: Execute script
Worker -> Redis: HSET hero:job:{job_id} status "finished" output "{result}"
```
### 4. Job Completion (Async)
```
Worker -> Redis: LPUSH hero:reply:{job_id} {result}
```
## API Operations
### List Jobs
```rust
dispatcher.list_jobs() -> Vec<String>
```
**Redis Operations:**
- `KEYS hero:job:*` - Get all job keys
- Extract job IDs from key names
### Stop Job
```rust
dispatcher.stop_job(job_id) -> Result<(), DispatcherError>
```
**Redis Operations:**
- `LPUSH hero:stop_queue:{worker_id} {job_id}` - Send stop request
### Get Job Status
```rust
dispatcher.get_job_status(job_id) -> Result<JobStatus, DispatcherError>
```
**Redis Operations:**
- `HGETALL hero:job:{job_id}` - Get job data
- Parse `status` field
### Get Job Logs
```rust
dispatcher.get_job_logs(job_id) -> Result<Option<String>, DispatcherError>
```
**Redis Operations:**
- `HGETALL hero:job:{job_id}` - Get job data
- Read `log_path` field
- Read log file from filesystem
### Run Job and Await Result
```rust
dispatcher.run_job_and_await_result(job, worker_id) -> Result<String, DispatcherError>
```
**Redis Operations:**
1. `HSET hero:job:{job_id} {job_fields}` - Store job
2. `LPUSH hero:work_queue:{worker_id} {job_id}` - Submit job
3. `BLPOP hero:reply:{job_id} {timeout}` - Wait for result
## Worker Protocol
### Job Processing Loop
```rust
loop {
// 1. Wait for job
job_id = BLPOP hero:work_queue:{worker_id}
// 2. Get job details
job_data = HGETALL hero:job:{job_id}
// 3. Update status
HSET hero:job:{job_id} status "started"
// 4. Check for stop requests
if LLEN hero:stop_queue:{worker_id} > 0 {
stop_job_id = LPOP hero:stop_queue:{worker_id}
if stop_job_id == job_id {
HSET hero:job:{job_id} status "error" error "stopped"
continue
}
}
// 5. Execute script
result = execute_script(job_data.script)
// 6. Update job with result
HSET hero:job:{job_id} status "finished" output result
// 7. Send reply if needed
if reply_queue_exists(hero:reply:{job_id}) {
LPUSH hero:reply:{job_id} result
}
}
```
### Stop Request Handling
Workers should periodically check the stop queue during long-running jobs:
```rust
if LLEN hero:stop_queue:{worker_id} > 0 {
stop_requests = LRANGE hero:stop_queue:{worker_id} 0 -1
if stop_requests.contains(current_job_id) {
// Stop current job execution
HSET hero:job:{current_job_id} status "error" error "stopped_by_request"
// Remove stop request
LREM hero:stop_queue:{worker_id} 1 current_job_id
return
}
}
```
## Error Handling
### Job Timeouts
- Client sets timeout when creating job
- Worker should respect timeout and stop execution
- If timeout exceeded: `HSET hero:job:{job_id} status "error" error "timeout"`
### Worker Failures
- If worker crashes, job remains in "started" status
- Monitoring systems can detect stale jobs and retry
- Jobs can be requeued: `LPUSH hero:work_queue:{worker_id} {job_id}`
### Redis Connection Issues
- Clients should implement retry logic with exponential backoff
- Workers should reconnect and resume processing
- Use Redis persistence to survive Redis restarts
## Monitoring and Observability
### Queue Monitoring
```bash
# Check work queue length
LLEN hero:work_queue:{worker_id}
# Check stop queue length
LLEN hero:stop_queue:{worker_id}
# List all jobs
KEYS hero:job:*
# Get job details
HGETALL hero:job:{job_id}
```
### Metrics to Track
- Jobs created per second
- Jobs completed per second
- Average job execution time
- Queue depths
- Worker availability
- Error rates by job type
## Security Considerations
### Redis Security
- Use Redis AUTH for authentication
- Enable TLS for Redis connections
- Restrict Redis network access
- Use Redis ACLs to limit worker permissions
### Job Security
- Validate script content before execution
- Sandbox script execution environment
- Limit resource usage (CPU, memory, disk)
- Log all job executions for audit
### Log File Security
- Ensure log paths are within allowed directories
- Validate log file permissions
- Rotate and archive logs regularly
- Sanitize sensitive data in logs
## Performance Considerations
### Redis Optimization
- Use Redis pipelining for batch operations
- Configure appropriate Redis memory limits
- Use Redis clustering for high availability
- Monitor Redis memory usage and eviction
### Job Optimization
- Keep job payloads small
- Use efficient serialization formats
- Batch similar jobs when possible
- Implement job prioritization if needed
### Worker Optimization
- Pool worker connections to Redis
- Use async I/O for Redis operations
- Implement graceful shutdown handling
- Monitor worker resource usage

559
core/dispatcher/examples/dispatcher_demo.rs Normal file
View File

@@ -0,0 +1,559 @@
use hero_dispatcher::{Dispatcher, DispatcherBuilder, ScriptType};
use log::info;
use redis::AsyncCommands;
use std::collections::HashMap;
use std::time::Duration;
use tokio::time::sleep;
/// Comprehensive example demonstrating the Hero Dispatcher functionality.
///
/// This example shows:
/// 1. Creating a dispatcher instance
/// 2. Creating jobs with different configurations
/// 3. Submitting jobs to the queue
/// 4. Inspecting Redis entries created by the dispatcher
/// 5. Running jobs and awaiting results
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
env_logger::init();
println!("🚀 Hero Dispatcher Demo");
println!("======================\n");
// Create dispatcher client with worker vectors per script type
let dispatcher = DispatcherBuilder::new()
.caller_id("demo-caller")
.context_id("demo-context")
.heroscript_workers(vec!["hero-worker-1".to_string(), "hero-worker-2".to_string()])
.rhai_sal_workers(vec!["rhai-sal-worker-1".to_string()])
.rhai_dsl_workers(vec!["rhai-dsl-worker-1".to_string()])
.redis_url("redis://127.0.0.1/")
.build()?;
println!("✅ Dispatcher created with:");
println!(" - Caller ID: demo-caller");
println!(" - Worker ID: demo-worker");
println!(" - Context ID: demo-context\n");
// Create Redis connection for inspection
let redis_client = redis::Client::open("redis://127.0.0.1:6379")?;
let mut redis_conn = redis_client.get_multiplexed_async_connection().await?;
// Demo 1: Create a simple job
println!("📝 Demo 1: Creating a simple job");
println!("--------------------------------");
let job1 = dispatcher
.new_job()
.script_type(ScriptType::HeroScript)
.script(r#"print("Hello from job 1!");"#)
.timeout(Duration::from_secs(10))
.build()?;
println!("Job 1 created with ID: {}", job1.id);
// Create the job (stores in Redis)
dispatcher.create_job(&job1).await?;
println!("✅ Job 1 stored in Redis");
// Inspect Redis entries for this job
print_job_redis_entries(&mut redis_conn, &job1.id).await?;
println!();
// Demo 2: Create a job with custom settings
println!("📝 Demo 2: Creating a job with custom settings");
println!("----------------------------------------------");
let job2 = dispatcher
.new_job()
.script_type(ScriptType::RhaiSAL)
.script(r#"
let result = 42 * 2;
print("Calculation result: " + result);
result
"#)
.timeout(Duration::from_secs(30))
.build()?;
println!("Job 2 created with ID: {}", job2.id);
// Create the job
dispatcher.create_job(&job2).await?;
println!("✅ Job 2 stored in Redis");
// Inspect Redis entries
print_job_redis_entries(&mut redis_conn, &job2.id).await?;
println!();
// Demo 3: Environment Variables
println!("📝 Demo 3: Jobs with Environment Variables");
println!("------------------------------------------");
// Create environment variables map
let mut env_vars = HashMap::new();
env_vars.insert("API_KEY".to_string(), "secret-api-key-123".to_string());
env_vars.insert("DEBUG_MODE".to_string(), "true".to_string());
env_vars.insert("MAX_RETRIES".to_string(), "5".to_string());
env_vars.insert("SERVICE_URL".to_string(), "https://api.example.com".to_string());
let job_with_env = dispatcher
.new_job()
.script_type(ScriptType::HeroScript)
.script(r#"
print("Environment variables available:");
print("API_KEY: " + env.API_KEY);
print("DEBUG_MODE: " + env.DEBUG_MODE);
print("MAX_RETRIES: " + env.MAX_RETRIES);
print("SERVICE_URL: " + env.SERVICE_URL);
"Environment variables processed successfully"
"#)
.env_vars(env_vars.clone())
.timeout(Duration::from_secs(15))
.build()?;
println!("Job with environment variables created: {}", job_with_env.id);
// Store job in Redis
dispatcher.create_job(&job_with_env).await?;
println!("✅ Job with env vars stored in Redis");
// Show Redis entries including environment variables
print_job_redis_entries(&mut redis_conn, &job_with_env.id).await?;
// Demonstrate individual env var setting
let job_individual_env = dispatcher
.new_job()
.script_type(ScriptType::RhaiSAL)
.script("print('Single env var: ' + env.SINGLE_VAR); 'done'")
.env_var("SINGLE_VAR", "individual-value")
.env_var("ANOTHER_VAR", "another-value")
.build()?;
println!("Job with individual env vars created: {}", job_individual_env.id);
dispatcher.create_job(&job_individual_env).await?;
println!("✅ Job with individual env vars stored in Redis");
print_job_redis_entries(&mut redis_conn, &job_individual_env.id).await?;
println!();
// Demo 4: Create multiple jobs and show queue state
println!("📝 Demo 4: Creating multiple jobs and inspecting queue");
println!("----------------------------------------------------");
let mut job_ids = Vec::new();
for i in 3..=5 {
let script_type = match i {
3 => ScriptType::HeroScript,
4 => ScriptType::RhaiSAL,
5 => ScriptType::RhaiDSL,
_ => ScriptType::HeroScript,
};
let job = dispatcher
.new_job()
.script_type(script_type)
.script(&format!(r#"print("Job {} is running");"#, i))
.timeout(Duration::from_secs(15))
.build()?;
job_ids.push(job.id.clone());
dispatcher.create_job(&job).await?;
println!("✅ Job {} created with ID: {}", i, job.id);
}
// Show all Redis keys related to our jobs
print_all_dispatcher_redis_keys(&mut redis_conn).await?;
println!();
// Demo 4: Show job status checking
println!("📝 Demo 4: Checking job statuses");
println!("--------------------------------");
for job_id in &job_ids {
match dispatcher.get_job_status(job_id).await {
Ok(status) => println!("Job {}: {:?}", job_id, status),
Err(e) => println!("Error getting status for job {}: {}", job_id, e),
}
}
println!();
// Demo 5: Simulate running a job and getting result (if worker is available)
println!("📝 Demo 5: Attempting to run job and await result");
println!("------------------------------------------------");
let simple_job = dispatcher
.new_job()
.script_type(ScriptType::HeroScript)
.script(r#"print("This job will complete quickly"); "success""#)
.timeout(Duration::from_secs(5))
.build()?;
println!("Created job for execution: {}", simple_job.id);
// Try to run the job (this will timeout if no worker is available)
match dispatcher.run_job_and_await_result(&simple_job).await {
Ok(result) => {
println!("✅ Job completed successfully!");
println!("Result: {}", result);
}
Err(e) => {
println!("⚠️ Job execution failed (likely no worker available): {}", e);
println!(" This is expected if no Hero worker is running");
}
}
// Demo 6: List all jobs
println!("📝 Demo 6: Listing all jobs");
println!("-------------------------");
let all_job_ids = match dispatcher.list_jobs().await {
Ok(job_ids) => {
println!("Found {} jobs:", job_ids.len());
for job_id in &job_ids {
println!(" - {}", job_id);
}
job_ids
}
Err(e) => {
println!("Error listing jobs: {}", e);
Vec::new()
}
};
println!();
// Demo 7: Create a job with log path and demonstrate logs functionality
println!("📝 Demo 7: Job with log path and logs retrieval");
println!("-----------------------------------------------");
let log_job = dispatcher
.new_job()
.script(r#"print("This job writes to logs"); "log_test""#)
.log_path("/tmp/hero_job_demo.log")
.timeout(Duration::from_secs(10))
.build()?;
println!("Created job with log path: {}", log_job.id);
dispatcher.create_job(&log_job).await?;
// Try to get logs (will be empty since job hasn't run)
match dispatcher.get_job_logs(&log_job.id).await {
Ok(Some(logs)) => println!("Job logs: {}", logs),
Ok(None) => println!("No logs available for job (expected - job hasn't run or no log file)"),
Err(e) => println!("Error getting logs: {}", e),
}
println!();
// Demo 8: Stop job functionality
println!("📝 Demo 8: Stopping a job");
println!("-------------------------");
if let Some(job_id) = all_job_ids.first() {
println!("Attempting to stop job: {}", job_id);
match dispatcher.stop_job(job_id).await {
Ok(()) => println!("✅ Stop request sent for job {}", job_id),
Err(e) => println!("Error stopping job: {}", e),
}
// Show stop queue
let stop_queue_key = "hero:stop_queue:demo-worker";
let stop_queue_length: i64 = redis_conn.llen(stop_queue_key).await?;
println!("📤 Stop queue length ({}): {}", stop_queue_key, stop_queue_length);
if stop_queue_length > 0 {
let stop_items: Vec<String> = redis_conn.lrange(stop_queue_key, 0, -1).await?;
println!("📋 Stop queue items:");
for (i, item) in stop_items.iter().enumerate() {
println!(" {}: {}", i, item);
}
}
} else {
println!("No jobs available to stop");
}
println!();
// Demo 9: Final Redis state inspection
println!("📝 Demo 9: Final Redis state");
println!("----------------------------");
print_all_dispatcher_redis_keys(&mut redis_conn).await?;
for job_id in &job_ids {
match dispatcher.get_job_status(job_id).await {
Ok(status) => println!("Job {}: {:?}", job_id, status),
Err(e) => println!("Error getting status for job {}: {}", job_id, e),
}
}
println!();
// Demo 5: Simulate running a job and getting result (if worker is available)
println!("📝 Demo 5: Attempting to run job and await result");
println!("------------------------------------------------");
let simple_job = dispatcher
.new_job()
.script_type(ScriptType::HeroScript)
.script(r#"print("This job will complete quickly"); "success""#)
.timeout(Duration::from_secs(5))
.build()?;
println!("Created job for execution: {}", simple_job.id);
// Try to run the job (this will timeout if no worker is available)
match dispatcher.run_job_and_await_result(&simple_job).await {
Ok(result) => {
println!("✅ Job completed successfully!");
println!("Result: {}", result);
}
Err(e) => {
println!("⚠️ Job execution failed (likely no worker available): {}", e);
println!(" This is expected if no Hero worker is running");
}
}
// Demo 6: List all jobs
println!("📝 Demo 6: Listing all jobs");
println!("-------------------------");
let all_job_ids = match dispatcher.list_jobs().await {
Ok(job_ids) => {
println!("Found {} jobs:", job_ids.len());
for job_id in &job_ids {
println!(" - {}", job_id);
}
job_ids
}
Err(e) => {
println!("Error listing jobs: {}", e);
Vec::new()
}
};
println!();
// Demo 7: Create a job with log path and demonstrate logs functionality
println!("📝 Demo 7: Job with log path and logs retrieval");
println!("-----------------------------------------------");
let log_job = dispatcher
.new_job()
.script(r#"print("This job writes to logs"); "log_test""#)
.log_path("/tmp/hero_job_demo.log")
.timeout(Duration::from_secs(10))
.build()?;
println!("Created job with log path: {}", log_job.id);
dispatcher.create_job(&log_job).await?;
// Try to get logs (will be empty since job hasn't run)
match dispatcher.get_job_logs(&log_job.id).await {
Ok(Some(logs)) => println!("Job logs: {}", logs),
Ok(None) => println!("No logs available for job (expected - job hasn't run or no log file)"),
Err(e) => println!("Error getting logs: {}", e),
}
println!();
// Demo 8: Stop job functionality
println!("📝 Demo 8: Stopping a job");
println!("-------------------------");
if let Some(job_id) = all_job_ids.first() {
println!("Attempting to stop job: {}", job_id);
match dispatcher.stop_job(job_id).await {
Ok(()) => println!("✅ Stop request sent for job {}", job_id),
Err(e) => println!("Error stopping job: {}", e),
}
// Show stop queue
let stop_queue_key = "hero:stop_queue:demo-worker";
let stop_queue_length: i64 = redis_conn.llen(stop_queue_key).await?;
println!("📤 Stop queue length ({}): {}", stop_queue_key, stop_queue_length);
if stop_queue_length > 0 {
let stop_items: Vec<String> = redis_conn.lrange(stop_queue_key, 0, -1).await?;
println!("📋 Stop queue items:");
for (i, item) in stop_items.iter().enumerate() {
println!(" {}: {}", i, item);
}
}
} else {
println!("No jobs available to stop");
}
println!();
// Demo 9: Final Redis state inspection
println!("📝 Demo 9: Final Redis state");
println!("----------------------------");
print_all_dispatcher_redis_keys(&mut redis_conn).await?;
println!("\n🎉 Dispatcher demo completed!");
println!("💡 New features demonstrated:");
println!(" - list_jobs(): List all job IDs");
println!(" - stop_job(): Send stop request to worker");
println!(" - get_job_logs(): Retrieve job logs from file");
println!(" - log_path(): Configure log file for jobs");
println!("💡 To see job execution in action, start a Hero worker that processes the 'demo-worker' queue");
// Demo 6: Demonstrate new job management features
println!("📝 Demo 6: Job Management - Delete and Clear Operations");
println!("--------------------------------------------------------");
// List all current jobs
match dispatcher.list_jobs().await {
Ok(jobs) => {
println!("Current jobs in system: {:?}", jobs);
if !jobs.is_empty() {
// Delete the first job as an example
let job_to_delete = &jobs[0];
println!("Deleting job: {}", job_to_delete);
match dispatcher.delete_job(job_to_delete).await {
Ok(()) => println!("✅ Job {} deleted successfully", job_to_delete),
Err(e) => println!("❌ Error deleting job {}: {}", job_to_delete, e),
}
// Show updated job list
match dispatcher.list_jobs().await {
Ok(remaining_jobs) => println!("Remaining jobs: {:?}", remaining_jobs),
Err(e) => println!("Error listing jobs: {}", e),
}
}
}
Err(e) => println!("Error listing jobs: {}", e),
}
println!();
// Demonstrate clear all jobs
println!("Clearing all remaining jobs...");
match dispatcher.clear_all_jobs().await {
Ok(count) => println!("✅ Cleared {} jobs from Redis", count),
Err(e) => println!("❌ Error clearing jobs: {}", e),
}
// Verify all jobs are cleared
match dispatcher.list_jobs().await {
Ok(jobs) => {
if jobs.is_empty() {
println!("✅ All jobs successfully cleared from Redis");
} else {
println!("⚠️ Some jobs remain: {:?}", jobs);
}
}
Err(e) => println!("Error verifying job clearance: {}", e),
}
println!();
println!("🎉 Demo completed! The dispatcher now supports:");
println!(" • Script type routing (HeroScript, RhaiSAL, RhaiDSL)");
println!(" • Multiple workers per script type for load balancing");
println!(" • Automatic worker selection based on job script type");
println!(" • Job management: list, delete, and clear operations");
println!(" • Enhanced job logging and monitoring");
Ok(())
}
/// Print Redis entries for a specific job
async fn print_job_redis_entries(
conn: &mut redis::aio::MultiplexedConnection,
job_id: &str,
) -> Result<(), redis::RedisError> {
let job_key = format!("hero:job:{}", job_id);
println!("🔍 Redis entries for job {}:", job_id);
// Check if job hash exists
let exists: bool = conn.exists(&job_key).await?;
if exists {
// Check if the key is actually a hash before trying to get all fields
let key_type: String = redis::cmd("TYPE").arg(&job_key).query_async(conn).await?;
if key_type == "hash" {
let job_data: std::collections::HashMap<String, String> = conn.hgetall(&job_key).await?;
println!(" 📋 Job data ({}): ", job_key);
for (field, value) in job_data {
println!(" {}: {}", field, value);
}
} else {
println!(" ⚠️ Key {} exists but is not a hash (type: {})", job_key, key_type);
}
} else {
println!(" ❌ No job data found at key: {}", job_key);
}
// Check work queue
let queue_key = "hero:work_queue:demo-worker";
let queue_length: i64 = conn.llen(queue_key).await?;
println!(" 📤 Work queue length ({}): {}", queue_key, queue_length);
if queue_length > 0 {
let queue_items: Vec<String> = conn.lrange(queue_key, 0, -1).await?;
println!(" 📋 Queue items:");
for (i, item) in queue_items.iter().enumerate() {
println!(" {}: {}", i, item);
}
}
Ok(())
}
/// Print all dispatcher-related Redis keys
async fn print_all_dispatcher_redis_keys(
conn: &mut redis::aio::MultiplexedConnection,
) -> Result<(), redis::RedisError> {
println!("🔍 All Hero Dispatcher Redis keys:");
// Get all keys with hero: prefix
let keys: Vec<String> = conn.keys("hero:*").await?;
if keys.is_empty() {
println!(" ❌ No Hero keys found in Redis");
return Ok(());
}
// Group keys by type
let mut job_keys = Vec::new();
let mut queue_keys = Vec::new();
let mut other_keys = Vec::new();
for key in keys {
if key.starts_with("hero:job:") {
job_keys.push(key);
} else if key.contains("queue") {
queue_keys.push(key);
} else {
other_keys.push(key);
}
}
// Print job keys
if !job_keys.is_empty() {
println!(" 📋 Job entries:");
for key in job_keys {
// Check if the key is actually a hash before trying to get all fields
let key_type: String = redis::cmd("TYPE").arg(&key).query_async(conn).await?;
if key_type == "hash" {
let job_data: std::collections::HashMap<String, String> = conn.hgetall(&key).await?;
println!(" {}: {} fields", key, job_data.len());
} else {
println!(" {}: {} (not a hash, skipping)", key, key_type);
}
}
}
// Print queue keys
if !queue_keys.is_empty() {
println!(" 📤 Queue entries:");
for key in queue_keys {
let length: i64 = conn.llen(&key).await?;
println!(" {}: {} items", key, length);
}
}
// Print other keys
if !other_keys.is_empty() {
println!(" 🔧 Other entries:");
for key in other_keys {
println!(" {}", key);
}
}
Ok(())
}

90
core/dispatcher/examples/timeout_example.rs Normal file
View File

@@ -0,0 +1,90 @@
use log::info;
use hero_dispatcher::{DispatcherBuilder, DispatcherError, ScriptType};
use std::time::{Duration, Instant};
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
env_logger::builder()
.filter_level(log::LevelFilter::Info)
.init();
// Build the client using the new builder pattern
let client = DispatcherBuilder::new()
.caller_id("timeout-example-runner")
.redis_url("redis://127.0.0.1/")
.build()?;
info!("Dispatcher created.");
let script_content = r#"
// This script will never be executed by a worker because the recipient does not exist.
let x = 10;
let y = x + 32;
y
"#;
// The worker_id points to a worker queue that doesn't have a worker.
let non_existent_recipient = "non_existent_worker_for_timeout_test";
let very_short_timeout = Duration::from_secs(2);
info!(
"Submitting script to non-existent recipient '{}' with a timeout of {:?}...",
non_existent_recipient, very_short_timeout
);
let start_time = Instant::now();
// Use the new JobBuilder
let result = client
.new_job()
.script_type(ScriptType::HeroScript)
.script(script_content)
.timeout(very_short_timeout)
.await_response()
.await;
match result {
Ok(details) => {
log::error!(
"Timeout Example FAILED: Expected a timeout, but got Ok: {:?}",
details
);
Err("Expected timeout, but task completed successfully.".into())
}
Err(e) => {
let elapsed = start_time.elapsed();
info!("Timeout Example: Received error as expected: {}", e);
info!("Elapsed time: {:?}", elapsed);
match e {
DispatcherError::Timeout(task_id) => {
info!("Timeout Example PASSED: Correctly received DispatcherError::Timeout for task_id: {}", task_id);
// Ensure the elapsed time is close to the timeout duration
// Allow for some buffer for processing
assert!(
elapsed >= very_short_timeout
&& elapsed < very_short_timeout + Duration::from_secs(1),
"Elapsed time {:?} should be close to timeout {:?}",
elapsed,
very_short_timeout
);
info!(
"Elapsed time {:?} is consistent with timeout duration {:?}.",
elapsed, very_short_timeout
);
Ok(())
}
other_error => {
log::error!(
"Timeout Example FAILED: Expected DispatcherError::Timeout, but got other error: {:?}",
other_error
);
Err(format!(
"Expected DispatcherError::Timeout, got other error: {:?}",
other_error
)
.into())
}
}
}
}
}

57
core/dispatcher/src/error.rs Normal file
View File

@@ -0,0 +1,57 @@
// Added error
// Duration is still used, Instant and sleep were removed
/// Comprehensive error type for all possible failures in the Rhai client.
///
/// This enum covers all error scenarios that can occur during client operations,
/// from Redis connectivity issues to task execution timeouts.
#[derive(Debug)]
pub enum DispatcherError {
/// Redis connection or operation error
RedisError(redis::RedisError),
/// JSON serialization/deserialization error
SerializationError(serde_json::Error),
/// Task execution timeout - contains the task_id that timed out
Timeout(String),
/// Task not found after submission - contains the task_id (rare occurrence)
TaskNotFound(String),
/// Context ID is missing
ContextIdMissing,
/// Invalid input provided
InvalidInput(String),
}
impl From<redis::RedisError> for DispatcherError {
fn from(err: redis::RedisError) -> Self {
DispatcherError::RedisError(err)
}
}
impl From<serde_json::Error> for DispatcherError {
fn from(err: serde_json::Error) -> Self {
DispatcherError::SerializationError(err)
}
}
impl std::fmt::Display for DispatcherError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
DispatcherError::RedisError(e) => write!(f, "Redis error: {}", e),
DispatcherError::SerializationError(e) => write!(f, "Serialization error: {}", e),
DispatcherError::Timeout(task_id) => {
write!(f, "Timeout waiting for task {} to complete", task_id)
}
DispatcherError::TaskNotFound(task_id) => {
write!(f, "Task {} not found after submission", task_id)
}
DispatcherError::ContextIdMissing => {
write!(f, "Context ID is missing")
}
DispatcherError::InvalidInput(msg) => {
write!(f, "Invalid input: {}", msg)
}
}
}
}
impl std::error::Error for DispatcherError {}

261
core/dispatcher/src/job.rs Normal file
View File

@@ -0,0 +1,261 @@
use chrono::Utc;
use std::collections::HashMap;
use std::time::Duration;
use uuid::Uuid;
use crate::{Dispatcher, DispatcherError};
use hero_job::{Job, ScriptType};
/// Builder for constructing and submitting script execution requests.
///
/// This builder provides a fluent interface for configuring script execution
/// parameters and offers two submission modes: fire-and-forget (`submit()`)
/// and request-reply (`await_response()`).
///
/// # Example
///
/// ```rust,no_run
/// use std::time::Duration;
/// use hero_dispatcher::ScriptType;
///
/// # async fn example(client: &hero_dispatcher::Dispatcher) -> Result<String, hero_dispatcher::DispatcherError> {
/// let result = client
/// .new_job()
/// .script_type(ScriptType::HeroScript)
/// .script(r#"print("Hello, World!");"#)
/// .timeout(Duration::from_secs(30))
/// .await_response()
/// .await?;
/// # Ok(result)
/// # }
/// ```
pub struct JobBuilder<'a> {
client: &'a Dispatcher,
request_id: String,
context_id: String,
caller_id: String,
script: String,
script_type: ScriptType,
timeout: Duration,
retries: u32,
concurrent: bool,
log_path: Option<String>,
env_vars: HashMap<String, String>,
prerequisites: Vec<String>,
dependents: Vec<String>
}
impl<'a> JobBuilder<'a> {
pub fn new(client: &'a Dispatcher) -> Self {
Self {
client,
request_id: "".to_string(),
context_id: client.context_id.clone(),
caller_id: client.caller_id.clone(),
script: "".to_string(),
script_type: ScriptType::HeroScript, // Default to HeroScript
timeout: Duration::from_secs(5),
retries: 0,
concurrent: false,
log_path: None,
env_vars: HashMap::new(),
prerequisites: Vec::new(),
dependents: Vec::new(),
}
}
pub fn request_id(mut self, request_id: &str) -> Self {
self.request_id = request_id.to_string();
self
}
pub fn script_type(mut self, script_type: ScriptType) -> Self {
self.script_type = script_type;
self
}
pub fn context_id(mut self, context_id: &str) -> Self {
self.context_id = context_id.to_string();
self
}
pub fn script(mut self, script: &str) -> Self {
self.script = script.to_string();
self
}
pub fn script_path(mut self, script_path: &str) -> Self {
self.script = std::fs::read_to_string(script_path).unwrap();
self
}
pub fn timeout(mut self, timeout: Duration) -> Self {
self.timeout = timeout;
self
}
pub fn log_path(mut self, log_path: &str) -> Self {
self.log_path = Some(log_path.to_string());
self
}
/// Set a single environment variable
pub fn env_var(mut self, key: &str, value: &str) -> Self {
self.env_vars.insert(key.to_string(), value.to_string());
self
}
/// Set multiple environment variables from a HashMap
pub fn env_vars(mut self, env_vars: HashMap<String, String>) -> Self {
self.env_vars.extend(env_vars);
self
}
/// Clear all environment variables
pub fn clear_env_vars(mut self) -> Self {
self.env_vars.clear();
self
}
/// Add a prerequisite job ID that must complete before this job can run
pub fn prerequisite(mut self, job_id: &str) -> Self {
self.prerequisites.push(job_id.to_string());
self
}
/// Set multiple prerequisite job IDs
pub fn prerequisites(mut self, job_ids: Vec<String>) -> Self {
self.prerequisites.extend(job_ids);
self
}
/// Add a dependent job ID that depends on this job completing
pub fn dependent(mut self, job_id: &str) -> Self {
self.dependents.push(job_id.to_string());
self
}
/// Set multiple dependent job IDs
pub fn dependents(mut self, job_ids: Vec<String>) -> Self {
self.dependents.extend(job_ids);
self
}
/// Clear all prerequisites
pub fn clear_prerequisites(mut self) -> Self {
self.prerequisites.clear();
self
}
/// Clear all dependents
pub fn clear_dependents(mut self) -> Self {
self.dependents.clear();
self
}
pub fn build(self) -> Result<Job, DispatcherError> {
let request_id = if self.request_id.is_empty() {
// Generate a UUID for the request_id
Uuid::new_v4().to_string()
} else {
self.request_id.clone()
};
if self.context_id.is_empty() {
return Err(DispatcherError::ContextIdMissing);
}
if self.caller_id.is_empty() {
return Err(DispatcherError::ContextIdMissing);
}
let now = Utc::now();
Ok(Job {
id: request_id,
caller_id: self.caller_id,
context_id: self.context_id,
script: self.script,
script_type: self.script_type,
timeout: self.timeout,
retries: self.retries as u8,
concurrent: self.concurrent,
log_path: self.log_path.clone(),
env_vars: self.env_vars.clone(),
prerequisites: self.prerequisites.clone(),
dependents: self.dependents.clone(),
created_at: now,
updated_at: now,
})
}
pub async fn submit(self) -> Result<(), DispatcherError> {
// Create job first, then use client reference
let request_id = if self.request_id.is_empty() {
Uuid::new_v4().to_string()
} else {
self.request_id
};
if self.context_id.is_empty() {
return Err(DispatcherError::ContextIdMissing);
}
let now = Utc::now();
let job = Job {
id: request_id,
caller_id: self.caller_id,
context_id: self.context_id,
script: self.script,
script_type: self.script_type.clone(),
timeout: self.timeout,
retries: self.retries as u8,
concurrent: self.concurrent,
log_path: self.log_path.clone(),
env_vars: self.env_vars.clone(),
prerequisites: self.prerequisites.clone(),
dependents: self.dependents.clone(),
created_at: now,
updated_at: now,
};
self.client.create_job(&job).await?;
Ok(())
}
pub async fn await_response(self) -> Result<String, DispatcherError> {
// Create job first, then use client reference
let request_id = if self.request_id.is_empty() {
Uuid::new_v4().to_string()
} else {
self.request_id
};
if self.context_id.is_empty() {
return Err(DispatcherError::ContextIdMissing);
}
let now = Utc::now();
let job = Job {
id: request_id,
caller_id: self.caller_id.clone(),
context_id: self.context_id,
script: self.script,
script_type: self.script_type.clone(),
timeout: self.timeout,
retries: self.retries as u8,
concurrent: self.concurrent,
log_path: self.log_path.clone(),
env_vars: self.env_vars.clone(),
prerequisites: self.prerequisites.clone(),
dependents: self.dependents.clone(),
created_at: now,
updated_at: now,
};
let result = self.client.run_job_and_await_result(&job).await?;
Ok(result)
}
}

498
core/dispatcher/src/lib.rs Normal file
View File

@@ -0,0 +1,498 @@
use log::{debug, error, info, warn};
use redis::AsyncCommands;
use std::time::Duration;
use hero_job::NAMESPACE_PREFIX;
mod job;
mod error;
pub use crate::error::DispatcherError;
pub use crate::job::JobBuilder;
// Re-export types from hero_job for public API
pub use hero_job::{Job, JobStatus, ScriptType};
pub struct Dispatcher {
redis_client: redis::Client,
caller_id: String,
context_id: String,
heroscript_workers: Vec<String>,
rhai_sal_workers: Vec<String>,
rhai_dsl_workers: Vec<String>,
}
pub struct DispatcherBuilder {
redis_url: Option<String>,
caller_id: Option<String>,
context_id: Option<String>,
heroscript_workers: Vec<String>,
rhai_sal_workers: Vec<String>,
rhai_dsl_workers: Vec<String>,
}
impl DispatcherBuilder {
pub fn new() -> Self {
Self {
redis_url: None,
caller_id: Some("default_caller".to_string()),
context_id: Some("default_context".to_string()),
heroscript_workers: Vec::new(),
rhai_sal_workers: Vec::new(),
rhai_dsl_workers: Vec::new(),
}
}
pub fn caller_id(mut self, caller_id: &str) -> Self {
self.caller_id = Some(caller_id.to_string());
self
}
pub fn context_id(mut self, context_id: &str) -> Self {
self.context_id = Some(context_id.to_string());
self
}
pub fn heroscript_workers(mut self, workers: Vec<String>) -> Self {
self.heroscript_workers = workers;
self
}
pub fn rhai_sal_workers(mut self, workers: Vec<String>) -> Self {
self.rhai_sal_workers = workers;
self
}
pub fn rhai_dsl_workers(mut self, workers: Vec<String>) -> Self {
self.rhai_dsl_workers = workers;
self
}
pub fn redis_url(mut self, url: &str) -> Self {
self.redis_url = Some(url.to_string());
self
}
/// Builds the final `Dispatcher` instance.
///
/// This method validates the configuration and creates the Redis client.
/// It will return an error if the caller ID is empty or if the Redis
/// connection cannot be established.
///
/// # Returns
///
/// * `Ok(Dispatcher)` - Successfully configured client
/// * `Err(DispatcherError)` - Configuration or connection error
pub fn build(self) -> Result<Dispatcher, DispatcherError> {
let url = self
.redis_url
.unwrap_or_else(|| "redis://127.0.0.1/".to_string());
let client = redis::Client::open(url)?;
Ok(Dispatcher {
redis_client: client,
caller_id: self.caller_id.unwrap_or_else(|| "default_caller".to_string()),
context_id: self.context_id.unwrap_or_else(|| "default_context".to_string()),
heroscript_workers: self.heroscript_workers,
rhai_sal_workers: self.rhai_sal_workers,
rhai_dsl_workers: self.rhai_dsl_workers,
})
}
}
impl Dispatcher {
/// Select a worker ID based on the script type using round-robin or first available
fn select_worker_for_script_type(&self, script_type: &ScriptType) -> Result<String, DispatcherError> {
let workers = match script_type {
ScriptType::HeroScript => &self.heroscript_workers,
ScriptType::RhaiSAL => &self.rhai_sal_workers,
ScriptType::RhaiDSL => &self.rhai_dsl_workers,
};
if workers.is_empty() {
return Err(DispatcherError::InvalidInput(format!(
"No workers configured for script type: {:?}", script_type
)));
}
// For now, use simple round-robin by selecting first available worker
// TODO: Implement proper load balancing
Ok(workers[0].clone())
}
pub fn new_job(&self) -> JobBuilder {
JobBuilder::new(self)
}
// Internal helper to submit script details and push to work queue
async fn create_job_using_connection(
&self,
conn: &mut redis::aio::MultiplexedConnection,
job: &Job,
) -> Result<(), DispatcherError> {
debug!(
"Submitting play request: {} for script type: {:?} with namespace prefix: {}",
job.id, job.script_type, NAMESPACE_PREFIX
);
// Use the shared Job struct's Redis storage method
job.store_in_redis(conn).await
.map_err(|e| DispatcherError::InvalidInput(format!("Failed to store job in Redis: {}", e)))?;
Ok(())
}
// Internal helper to submit script details and push to work queue
async fn start_job_using_connection(
&self,
conn: &mut redis::aio::MultiplexedConnection,
job_id: String,
worker_id: String
) -> Result<(), DispatcherError> {
let worker_queue_key = format!(
"{}{}",
NAMESPACE_PREFIX,
worker_id.replace(" ", "_").to_lowercase()
);
// lpush also infers its types, RV is typically i64 (length of list) or () depending on exact command variant
// For `redis::AsyncCommands::lpush`, it's `RedisResult<R>` where R: FromRedisValue
// Often this is the length of the list. Let's allow inference or specify if needed.
let _: redis::RedisResult<i64> =
conn.lpush(&worker_queue_key, job_id.clone()).await;
Ok(())
}
// Internal helper to await response from worker
async fn await_response_from_connection(
&self,
conn: &mut redis::aio::MultiplexedConnection,
job_key: &String,
reply_queue_key: &String,
timeout: Duration,
) -> Result<String, DispatcherError> {
// BLPOP on the reply queue
// The timeout for BLPOP is in seconds (integer)
let blpop_timeout_secs = timeout.as_secs().max(1); // Ensure at least 1 second for BLPOP timeout
match conn
.blpop::<&String, Option<(String, String)>>(reply_queue_key, blpop_timeout_secs as f64)
.await
{
Ok(Some((_queue, result_message_str))) => {
Ok(result_message_str)
}
Ok(None) => {
// BLPOP timed out
warn!(
"Timeout waiting for result on reply queue {} for job {}",
reply_queue_key, job_key
);
// Optionally, delete the reply queue
let _: redis::RedisResult<i32> = conn.del(&reply_queue_key).await;
Err(DispatcherError::Timeout(job_key.clone()))
}
Err(e) => {
// Redis error
error!(
"Redis error on BLPOP for reply queue {}: {}",
reply_queue_key, e
);
// Optionally, delete the reply queue
let _: redis::RedisResult<i32> = conn.del(&reply_queue_key).await;
Err(DispatcherError::RedisError(e))
}
}
}
// New method using dedicated reply queue
pub async fn create_job(
&self,
job: &Job,
) -> Result<(), DispatcherError> {
let mut conn = self.redis_client.get_multiplexed_async_connection().await?;
self.create_job_using_connection(
&mut conn,
&job, // Pass the job_id parameter
)
.await?;
Ok(())
}
// New method using dedicated reply queue with automatic worker selection
pub async fn run_job_and_await_result(
&self,
job: &Job
) -> Result<String, DispatcherError> {
// Select worker based on script type
let worker_id = self.select_worker_for_script_type(&job.script_type)?;
let mut conn = self.redis_client.get_multiplexed_async_connection().await?;
let reply_queue_key = format!("{}:reply:{}", NAMESPACE_PREFIX, job.id); // Derived from the passed job_id
self.create_job_using_connection(
&mut conn,
&job, // Pass the job_id parameter
)
.await?;
self.start_job_using_connection(&mut conn, job.id.clone(), worker_id).await?;
info!(
"Task {} submitted. Waiting for result on queue {} with timeout {:?}...",
job.id, // This is the UUID
reply_queue_key,
job.timeout
);
self.await_response_from_connection(
&mut conn,
&job.id,
&reply_queue_key,
job.timeout,
)
.await
}
// Method to get job status
pub async fn get_job_status(
&self,
job_id: &str,
) -> Result<JobStatus, DispatcherError> {
let mut conn = self.redis_client.get_multiplexed_async_connection().await?;
let job_key = format!("{}{}", NAMESPACE_PREFIX, job_id);
let result_map: Option<std::collections::HashMap<String, String>> =
conn.hgetall(&job_key).await?;
match result_map {
Some(map) => {
let status_str = map.get("status").cloned().unwrap_or_else(|| {
warn!("Task {}: 'status' field missing from Redis hash, defaulting to empty.", job_id);
String::new()
});
let status = match status_str.as_str() {
"dispatched" => JobStatus::Dispatched,
"started" => JobStatus::Started,
"error" => JobStatus::Error,
"finished" => JobStatus::Finished,
_ => JobStatus::Dispatched, // default
};
Ok(status)
}
None => {
warn!("Job {} not found in Redis", job_id);
Ok(JobStatus::Dispatched) // default for missing jobs
}
}
}
// Method to get job output
pub async fn get_job_output(
&self,
job_id: &str,
) -> Result<Option<String>, DispatcherError> {
let mut conn = self.redis_client.get_multiplexed_async_connection().await?;
let job_key = format!("{}{}", NAMESPACE_PREFIX, job_id);
let result_map: Option<std::collections::HashMap<String, String>> =
conn.hgetall(&job_key).await?;
match result_map {
Some(map) => {
Ok(map.get("output").cloned())
}
None => {
warn!("Job {} not found in Redis", job_id);
Ok(None)
}
}
}
/// List all jobs in Redis
pub async fn list_jobs(&self) -> Result<Vec<String>, DispatcherError> {
let mut conn = self.redis_client.get_multiplexed_async_connection().await?;
// Use the shared Job struct's list method
Job::list_all_job_ids(&mut conn).await
.map_err(|e| DispatcherError::InvalidInput(format!("Failed to list jobs: {}", e)))
}
/// Stop a job by pushing its ID to the stop queue
pub async fn stop_job(&self, job_id: &str) -> Result<(), DispatcherError> {
let mut conn = self.redis_client.get_multiplexed_async_connection().await?;
// Get job details to determine script type and appropriate worker
let job_key = format!("{}job:{}", NAMESPACE_PREFIX, job_id);
let job_data: std::collections::HashMap<String, String> = conn.hgetall(&job_key).await?;
if job_data.is_empty() {
return Err(DispatcherError::InvalidInput(format!("Job {} not found", job_id)));
}
// Parse script type from job data
let script_type_str = job_data.get("script_type")
.ok_or_else(|| DispatcherError::InvalidInput("Job missing script_type field".to_string()))?;
let script_type: ScriptType = serde_json::from_str(&format!("\"{}\"", script_type_str))
.map_err(|e| DispatcherError::InvalidInput(format!("Invalid script type: {}", e)))?;
// Select appropriate worker for this script type
let worker_id = self.select_worker_for_script_type(&script_type)?;
let stop_queue_key = format!("{}stop_queue:{}", NAMESPACE_PREFIX, worker_id);
// Push job ID to the stop queue
conn.lpush::<_, _, ()>(&stop_queue_key, job_id).await?;
info!("Job {} added to stop queue {} for script type {:?}", job_id, stop_queue_key, script_type);
Ok(())
}
/// Get logs for a job by reading from its log file
pub async fn get_job_logs(&self, job_id: &str) -> Result<Option<String>, DispatcherError> {
let mut conn = self.redis_client.get_multiplexed_async_connection().await?;
let job_key = format!("{}job:{}", NAMESPACE_PREFIX, job_id);
// Get the job data to find the log path
let result_map: Option<std::collections::HashMap<String, String>> =
conn.hgetall(&job_key).await?;
match result_map {
Some(map) => {
if let Some(log_path) = map.get("log_path") {
// Try to read the log file
match std::fs::read_to_string(log_path) {
Ok(contents) => Ok(Some(contents)),
Err(e) => {
warn!("Failed to read log file {}: {}", log_path, e);
Ok(None)
}
}
} else {
// No log path configured for this job
Ok(None)
}
}
None => {
warn!("Job {} not found in Redis", job_id);
Ok(None)
}
}
}
/// Delete a specific job by ID
pub async fn delete_job(&self, job_id: &str) -> Result<(), DispatcherError> {
let mut conn = self.redis_client.get_multiplexed_async_connection().await?;
// Use the shared Job struct's delete method
Job::delete_from_redis(&mut conn, job_id).await
.map_err(|e| DispatcherError::InvalidInput(format!("Failed to delete job: {}", e)))?;
info!("Job {} deleted successfully", job_id);
Ok(())
}
/// Clear all jobs from Redis
pub async fn clear_all_jobs(&self) -> Result<usize, DispatcherError> {
let mut conn = self.redis_client.get_multiplexed_async_connection().await?;
// Get all job IDs first
let job_ids = Job::list_all_job_ids(&mut conn).await
.map_err(|e| DispatcherError::InvalidInput(format!("Failed to list jobs: {}", e)))?;
let count = job_ids.len();
// Delete each job using the shared method
for job_id in job_ids {
Job::delete_from_redis(&mut conn, &job_id).await
.map_err(|e| DispatcherError::InvalidInput(format!("Failed to delete job {}: {}", job_id, e)))?;
}
Ok(count)
}
/// Check if all prerequisites for a job are completed
pub async fn check_prerequisites_completed(&self, job_id: &str) -> Result<bool, DispatcherError> {
let mut conn = self.redis_client.get_multiplexed_async_connection().await?;
// Load the job using the shared Job struct
let job = Job::load_from_redis(&mut conn, job_id).await
.map_err(|e| DispatcherError::InvalidInput(format!("Failed to load job: {}", e)))?;
// Check each prerequisite job status
for prereq_id in &job.prerequisites {
let status = Job::get_status(&mut conn, prereq_id).await
.map_err(|e| DispatcherError::InvalidInput(format!("Failed to get prerequisite status: {}", e)))?;
if status != JobStatus::Finished {
return Ok(false); // Prerequisite not completed
}
}
Ok(true) // All prerequisites completed (or no prerequisites)
}
/// Update job status and check dependent jobs for readiness
pub async fn update_job_status_and_check_dependents(&self, job_id: &str, new_status: JobStatus) -> Result<Vec<String>, DispatcherError> {
let mut conn = self.redis_client.get_multiplexed_async_connection().await?;
// Update job status using shared Job method
Job::update_status(&mut conn, job_id, new_status.clone()).await
.map_err(|e| DispatcherError::InvalidInput(format!("Failed to update job status: {}", e)))?;
let mut ready_jobs = Vec::new();
// If job finished, check dependent jobs
if new_status == JobStatus::Finished {
// Load the job to get its dependents
let job = Job::load_from_redis(&mut conn, job_id).await
.map_err(|e| DispatcherError::InvalidInput(format!("Failed to load job: {}", e)))?;
// Check each dependent job
for dependent_id in &job.dependents {
let dependent_status = Job::get_status(&mut conn, dependent_id).await
.map_err(|e| DispatcherError::InvalidInput(format!("Failed to get dependent status: {}", e)))?;
// Only check jobs that are waiting for prerequisites
if dependent_status == JobStatus::WaitingForPrerequisites {
// Check if all prerequisites are now completed
if self.check_prerequisites_completed(dependent_id).await? {
// Update status to dispatched and add to ready jobs
Job::update_status(&mut conn, dependent_id, JobStatus::Dispatched).await
.map_err(|e| DispatcherError::InvalidInput(format!("Failed to update dependent status: {}", e)))?;
ready_jobs.push(dependent_id.clone());
}
}
}
}
Ok(ready_jobs)
}
/// Dispatch jobs that are ready (have all prerequisites completed)
pub async fn dispatch_ready_jobs(&self, ready_job_ids: Vec<String>) -> Result<(), DispatcherError> {
for job_id in ready_job_ids {
// Get job data to determine script type and select worker
let mut conn = self.redis_client.get_multiplexed_async_connection().await?;
let job_key = format!("{}job:{}", NAMESPACE_PREFIX, job_id);
let job_data: std::collections::HashMap<String, String> = conn.hgetall(&job_key).await?;
if let Some(script_type_str) = job_data.get("script_type") {
// Parse script type (stored as Debug format, e.g., "HeroScript")
let script_type = match script_type_str.as_str() {
"HeroScript" => ScriptType::HeroScript,
"RhaiSAL" => ScriptType::RhaiSAL,
"RhaiDSL" => ScriptType::RhaiDSL,
_ => return Err(DispatcherError::InvalidInput(format!("Unknown script type: {}", script_type_str))),
};
// Select worker and dispatch job
let worker_id = self.select_worker_for_script_type(&script_type)?;
self.start_job_using_connection(&mut conn, job_id, worker_id).await?;
}
}
Ok(())
}
}