herocode/supervisor
Archived
14
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Clean up documentation: consolidate docs in docs/ directory, simplify README

Browse Source
This commit is contained in:
Timur Gordon
2025-11-06 23:36:10 +01:00
parent 6d518599b8
commit a65c721c64
11 changed files with 21 additions and 1146 deletions
Download Patch File Download Diff File Expand all files Collapse all files

268
docs/MYCELIUM.md Normal file
View File

@@ -0,0 +1,268 @@
# Mycelium Integration - Now Optional!
The Mycelium integration is now an optional feature. The supervisor can run with just the OpenRPC HTTP server, making it simpler to use and deploy.
## What Changed
### Before
- Mycelium integration was always enabled
- Supervisor would continuously try to connect to Mycelium on port 8990
- Error logs if Mycelium wasn't available
- Required additional dependencies
### After
- ✅ Mycelium is now an optional feature
- ✅ Supervisor runs with clean OpenRPC HTTP server by default
- ✅ No connection errors when Mycelium isn't needed
- ✅ Smaller binary size without Mycelium dependencies
## Running the Supervisor
### Option 1: Simple OpenRPC Server (Recommended)
**No Mycelium, just OpenRPC:**
```bash
# Using the helper script
./run_supervisor_simple.sh
# Or manually
MYCELIUM_URL="" cargo run --bin supervisor -- \
--redis-url redis://localhost:6379 \
--port 3030
```
This starts:
- ✅ OpenRPC HTTP server on port 3030
- ✅ Redis connection for job queuing
- ❌ No Mycelium integration
### Option 2: With Mycelium Integration
**Enable Mycelium feature:**
```bash
# Build with Mycelium support
cargo build --bin supervisor --features mycelium
# Run with Mycelium URL
MYCELIUM_URL="http://localhost:8990" cargo run --bin supervisor --features mycelium -- \
--redis-url redis://localhost:6379 \
--port 3030
```
This starts:
- ✅ OpenRPC HTTP server on port 3030
- ✅ Redis connection for job queuing
- ✅ Mycelium integration (connects to daemon)
## Feature Flags
### Available Features
| Feature | Description | Default |
|---------|-------------|---------|
| `cli` | Command-line interface | ✅ Yes |
| `mycelium` | Mycelium integration | ❌ No |
### Building with Features
```bash
# Default build (CLI only, no Mycelium)
cargo build --bin supervisor
# With Mycelium
cargo build --bin supervisor --features mycelium
# Minimal (no CLI, no Mycelium)
cargo build --bin supervisor --no-default-features
```
## Architecture
### Without Mycelium (Default)
```
┌─────────────────┐
│ Client │
└────────┬────────┘
│ HTTP/JSON-RPC
▼
┌─────────────────┐
│ Supervisor │
│ OpenRPC Server │
│ (Port 3030) │
└────────┬────────┘
│ Redis
▼
┌─────────────────┐
│ Runners │
└─────────────────┘
```
### With Mycelium (Optional)
```
┌─────────────────┐
│ Client │
└────────┬────────┘
│ HTTP/JSON-RPC
▼
┌─────────────────┐ ┌──────────────┐
│ Supervisor │◄────►│ Mycelium │
│ OpenRPC Server │ │ Daemon │
│ (Port 3030) │ │ (Port 8990) │
└────────┬────────┘ └──────────────┘
│ Redis
▼
┌─────────────────┐
│ Runners │
└─────────────────┘
```
## Environment Variables
| Variable | Description | Default | Required |
|----------|-------------|---------|----------|
| `MYCELIUM_URL` | Mycelium daemon URL | `http://127.0.0.1:8990` | No |
| `RUST_LOG` | Log level | `info` | No |
**To disable Mycelium:**
```bash
export MYCELIUM_URL=""
```
## Dependencies
### Core Dependencies (Always)
- `tokio` - Async runtime
- `redis` - Job queuing
- `jsonrpsee` - OpenRPC server
- `runner_rust` - Job model
### Mycelium Dependencies (Optional)
- `reqwest` - HTTP client
- `base64` - Encoding
- `rand` - Random IDs
## Examples
All examples work without Mycelium:
```bash
# Simple end-to-end example
RUST_LOG=info cargo run --example simple_e2e
# Full automated demo
RUST_LOG=info cargo run --example end_to_end_demo
```
## Migration Guide
### If you were using Mycelium
**Before:**
```bash
cargo run --bin supervisor
# Would try to connect to Mycelium automatically
```
**After:**
```bash
# Option A: Disable Mycelium (recommended for most use cases)
MYCELIUM_URL="" cargo run --bin supervisor
# Option B: Enable Mycelium feature
cargo run --bin supervisor --features mycelium
```
### If you weren't using Mycelium
**Before:**
```bash
cargo run --bin supervisor
# Would see connection errors to port 8990
```
**After:**
```bash
cargo run --bin supervisor
# Clean startup, no connection errors! 🎉
```
## Benefits
### For Development
- ✅ Faster builds (fewer dependencies)
- ✅ Simpler setup (no Mycelium daemon needed)
- ✅ Cleaner logs (no connection errors)
- ✅ Easier debugging
### For Production
- ✅ Smaller binary size
- ✅ Fewer runtime dependencies
- ✅ More flexible deployment
- ✅ Optional advanced features
## Testing
### Test without Mycelium
```bash
# Build
cargo build --bin supervisor
# Run tests
cargo test
# Run examples
cargo run --example simple_e2e
```
### Test with Mycelium
```bash
# Build with feature
cargo build --bin supervisor --features mycelium
# Start Mycelium daemon (if you have one)
# mycelium-daemon --port 8990
# Run supervisor
MYCELIUM_URL="http://localhost:8990" cargo run --bin supervisor --features mycelium
```
## Troubleshooting
### "Mycelium integration not enabled"
This is informational, not an error. If you need Mycelium:
```bash
cargo build --features mycelium
```
### "HTTP request failed: error sending request"
If you see this with Mycelium enabled, check:
1. Is Mycelium daemon running?
2. Is the URL correct? (`MYCELIUM_URL`)
3. Is the port accessible?
Or simply disable Mycelium:
```bash
export MYCELIUM_URL=""
```
## Summary
🎉 **The supervisor now runs cleanly with just OpenRPC!**
- Default: OpenRPC HTTP server only
- Optional: Enable Mycelium with `--features mycelium`
- No more connection errors when Mycelium isn't needed
- Simpler, faster, cleaner!
---
**Status:** ✅ Complete
**Version:** 0.1.0
**Last Updated:** 2025-10-24

214
docs/QUICK_START.md Normal file
View File

@@ -0,0 +1,214 @@
# Quick Start Guide
Complete guide to running the Hero Supervisor with OSIS runners and examples.
## Prerequisites
1. **Redis** - Must be running
2. **Rust** - Version 1.88+ (run `rustup update`)
## 1. Start Redis
```bash
redis-server
```
## 2. Start Supervisor
```bash
cd /Users/timurgordon/code/git.ourworld.tf/herocode/supervisor
cargo run --bin supervisor
```
You should see:
```
╔════════════════════════════════════════════════════════════╗
║ Hero Supervisor Started ║
╚════════════════════════════════════════════════════════════╝
📡 OpenRPC Server: http://127.0.0.1:3030
🔗 Redis: redis://localhost:6379
🌐 Mycelium: Not compiled (use --features mycelium)
╚════════════════════════════════════════════════════════════╝
```
## 3. Start OSIS Runner
```bash
cd /Users/timurgordon/code/git.ourworld.tf/herocode/runner_rust
cargo run --bin runner_osis -- test_runner \
--redis-url redis://localhost:6379 \
--db-path /tmp/test_runner.db
```
You should see:
```
Starting OSIS Sync Runner with ID: test_runner
Database path: /tmp/test_runner.db
Redis URL: redis://localhost:6379
OSIS Sync Runner 'test_runner' started successfully
```
## 4. Run Example
```bash
cd /Users/timurgordon/code/git.ourworld.tf/herocode/supervisor
RUST_LOG=info cargo run --example simple_e2e
```
## Terminal Layout
```
┌─────────────────────┬─────────────────────┐
│ Terminal 1 │ Terminal 2 │
│ Redis │ Supervisor │
│ redis-server │ cargo run --bin │
│ │ supervisor │
├─────────────────────┼─────────────────────┤
│ Terminal 3 │ Terminal 4 │
│ OSIS Runner │ Example │
│ cargo run --bin │ cargo run │
│ runner_osis │ --example │
│ │ simple_e2e │
└─────────────────────┴─────────────────────┘
```
## What Each Component Does
### Redis
- Job queue storage
- Job result storage
- Runner coordination
### Supervisor
- OpenRPC HTTP server (port 3030)
- Job dispatch to runners
- Runner registration
- Job execution coordination
### OSIS Runner
- Listens for jobs on Redis queue
- Executes Rhai scripts
- Stores results back to Redis
- Uses HeroDB for data persistence
### Example
- Creates jobs with Rhai scripts
- Sends jobs to supervisor via OpenRPC
- Receives results
- Demonstrates both blocking and non-blocking modes
## Architecture
```
┌─────────────┐
│ Example │ (simple_e2e.rs)
└──────┬──────┘
│ HTTP/JSON-RPC
▼
┌─────────────┐
│ Supervisor │ (port 3030)
└──────┬──────┘
│ Redis Queue
▼
┌─────────────┐
│ OSIS Runner │ (test_runner)
└──────┬──────┘
│
▼
┌─────────────┐
│ HeroDB │ (Redis + local DB)
└─────────────┘
```
## Troubleshooting
### "Connection refused" on port 3030
- Make sure supervisor is running
- Check if another process is using port 3030
### "Connection refused" on port 6379
- Make sure Redis is running
- Check: `redis-cli ping` (should return "PONG")
### Runner not receiving jobs
- Check runner is registered: Look for "Runner registered successfully" in example output
- Check Redis connection: Both supervisor and runner must use same Redis URL
- Check queue name matches: Should be `hero:q:work:type:osis:group:default:inst:test_runner`
### "Job execution timeout"
- Increase timeout in job builder: `.timeout(120)`
- Check if runner is actually processing jobs (look for logs)
## Example Output
### Successful Run
```
╔════════════════════════════════════════╗
║ Simple End-to-End Demo ║
╚════════════════════════════════════════╝
📋 Step 1: Registering Runner
─────────────────────────────────────────
✅ Runner registered successfully
📋 Step 2: Running a Simple Job (Blocking)
─────────────────────────────────────────
✅ Job completed!
Result: {"message":"Hello from the runner!","number":42}
📋 Step 3: Running a Calculation Job
─────────────────────────────────────────
✅ Calculation completed!
Result: {"sum":55,"product":3628800,"count":10}
📋 Step 4: Starting a Non-Blocking Job
─────────────────────────────────────────
✅ Job started!
Job ID: abc-123 (running in background)
🎉 Demo completed successfully!
```
## Next Steps
1. **Try different Rhai scripts** - Modify the payload in examples
2. **Add more runners** - Start multiple runners with different IDs
3. **Explore the API** - Use the OpenRPC client library
4. **Build your own client** - See `client/` for examples
## Useful Commands
```bash
# Check Redis
redis-cli ping
# List Redis keys
redis-cli keys "hero:*"
# Monitor Redis commands
redis-cli monitor
# Check supervisor is running
curl http://localhost:3030
# View runner logs
# (check terminal where runner is running)
```
## Clean Up
```bash
# Stop all processes (Ctrl+C in each terminal)
# Clean up test database
rm /tmp/test_runner.db
# (Optional) Flush Redis
redis-cli FLUSHALL
```
---
**Status:** ✅ Ready to Use
**Last Updated:** 2025-10-24

280
docs/README.md
View File

@@ -1,280 +0,0 @@
# Hero Supervisor Documentation
## Overview
Hero Supervisor is a distributed job execution system that manages runners and coordinates job processing across multiple worker nodes. It provides a robust OpenRPC API for job management and runner administration.
## Architecture
The supervisor consists of several key components:
- **Supervisor Core**: Central coordinator that manages runners and job dispatch
- **OpenRPC Server**: JSON-RPC API server for remote management
- **Redis Backend**: Job queue and state management
- **Process Manager**: Runner lifecycle management (Simple or Tmux)
- **Client Libraries**: Native Rust and WASM clients for integration
## Quick Start
### Starting the Supervisor
```bash
# With default configuration
./supervisor
# With custom configuration file
./supervisor --config /path/to/config.toml
```
### Example Configuration
```toml
# config.toml
redis_url = "redis://localhost:6379"
namespace = "hero"
bind_address = "127.0.0.1"
port = 3030
# Admin secrets for full access
admin_secrets = ["admin-secret-123"]
# User secrets for job operations
user_secrets = ["user-secret-456"]
# Register secrets for runner registration
register_secrets = ["register-secret-789"]
[[actors]]
id = "sal_runner_1"
name = "sal_runner_1"
binary_path = "/path/to/sal_runner"
db_path = "/tmp/sal_db"
redis_url = "redis://localhost:6379"
process_manager = "simple"
[[actors]]
id = "osis_runner_1"
name = "osis_runner_1"
binary_path = "/path/to/osis_runner"
db_path = "/tmp/osis_db"
redis_url = "redis://localhost:6379"
process_manager = "tmux:osis_session"
```
## API Documentation
### Job API Convention
The Hero Supervisor follows a consistent naming convention for job operations:
- **`jobs.`** - General job operations (create, list)
- **`job.`** - Specific job operations (run, start, status, result)
See [Job API Convention](job-api-convention.md) for detailed documentation.
### Core Methods
#### Runner Management
- `register_runner` - Register a new runner
- `list_runners` - List all registered runners
- `start_runner` / `stop_runner` - Control runner lifecycle
- `get_runner_status` - Get runner status
- `get_runner_logs` - Retrieve runner logs
#### Job Management
- `jobs.create` - Create a job without queuing
- `jobs.list` - List all jobs with full details
- `job.run` - Run a job and return result
- `job.start` - Start a created job
- `job.stop` - Stop a running job
- `job.delete` - Delete a job from the system
- `job.status` - Get job status (non-blocking)
- `job.result` - Get job result (blocking)
#### Administration
- `add_secret` / `remove_secret` - Manage authentication secrets
- `get_supervisor_info` - Get system information
- `rpc.discover` - OpenRPC specification discovery
## Client Usage
### Rust Client
```rust
use hero_supervisor_openrpc_client::{SupervisorClient, JobBuilder};
// Create client
let client = SupervisorClient::new("http://localhost:3030")?;
// Create a job
let job = JobBuilder::new()
.caller_id("my_client")
.context_id("my_context")
.payload("print('Hello World')")
.executor("osis")
.runner("osis_runner_1")
.timeout(60)
.build()?;
// Option 1: Fire-and-forget execution
let result = client.job_run("user-secret", job.clone()).await?;
match result {
JobResult::Success { success } => println!("Output: {}", success),
JobResult::Error { error } => println!("Error: {}", error),
}
// Option 2: Asynchronous execution
let job_id = client.jobs_create("user-secret", job).await?;
client.job_start("user-secret", &job_id).await?;
// Poll for completion
loop {
let status = client.job_status(&job_id).await?;
if status.status == "completed" || status.status == "failed" {
break;
}
tokio::time::sleep(Duration::from_secs(1)).await;
}
let result = client.job_result(&job_id).await?;
// Option 3: Job management
// Stop a running job
client.job_stop("user-secret", &job_id).await?;
// Delete a job
client.job_delete("user-secret", &job_id).await?;
// List all jobs (returns full Job objects)
let jobs = client.jobs_list("user-secret").await?;
for job in jobs {
println!("Job {}: {} ({})", job.id, job.executor, job.payload);
}
```
### WASM Client
```javascript
import { WasmSupervisorClient, WasmJob } from 'hero-supervisor-openrpc-client';
// Create client
const client = new WasmSupervisorClient('http://localhost:3030');
// Create and run job
const job = new WasmJob('job-id', 'print("Hello")', 'osis', 'osis_runner_1');
const result = await client.create_job('user-secret', job);
```
## Security
### Authentication Levels
1. **Admin Secrets**: Full system access
- All runner management operations
- All job operations
- Secret management
- System information access
2. **User Secrets**: Job operations only
- Create, run, start jobs
- Get job status and results
- No runner or secret management
3. **Register Secrets**: Runner registration only
- Register new runners
- No other operations
### Best Practices
- Use different secret types for different access levels
- Rotate secrets regularly
- Store secrets securely (environment variables, secret management systems)
- Use HTTPS in production environments
- Implement proper logging and monitoring
## Development
### Building
```bash
# Build supervisor binary
cargo build --release
# Build with OpenRPC feature
cargo build --release --features openrpc
# Build client library
cd client
cargo build --release
```
### Testing
```bash
# Run tests
cargo test
# Run with Redis (requires Redis server)
docker run -d -p 6379:6379 redis:alpine
cargo test -- --ignored
```
### Examples
See the `examples/` directory for:
- Basic supervisor setup
- Mock runner implementation
- Comprehensive OpenRPC client usage
- Configuration examples
## Troubleshooting
### Common Issues
1. **Redis Connection Failed**
- Ensure Redis server is running
- Check Redis URL in configuration
- Verify network connectivity
2. **Runner Registration Failed**
- Check register secret validity
- Verify runner binary path exists
- Ensure runner has proper permissions
3. **Job Execution Timeout**
- Increase job timeout value
- Check runner resource availability
- Monitor runner logs for issues
4. **OpenRPC Method Not Found**
- Verify method name spelling
- Check OpenRPC specification
- Ensure server supports the method
### Logging
Enable debug logging:
```bash
RUST_LOG=debug ./supervisor --config config.toml
```
### Monitoring
Monitor key metrics:
- Runner status and health
- Job queue lengths
- Job success/failure rates
- Response times
- Redis connection status
## Contributing
1. Fork the repository
2. Create a feature branch
3. Make changes with tests
4. Update documentation
5. Submit a pull request
## License
[License information here]

58
docs/RESTRUCTURE.md Normal file
View File

@@ -0,0 +1,58 @@
# Repository Restructure
## Changes Made
The supervisor repository has been restructured to follow a cleaner organization:
### Before:
```
supervisor/
├── clients/
│ ├── openrpc/ # OpenRPC client library
│ └── admin-ui/ # Admin UI (Yew WASM app)
├── src/ # Main supervisor library
└── cmd/ # Supervisor binary
```
### After:
```
supervisor/
├── client/ # OpenRPC client library (renamed from clients/openrpc)
├── ui/ # Admin UI (renamed from clients/admin-ui)
├── src/ # Main supervisor library
└── cmd/ # Supervisor binary
```
## Package Names
The package names remain unchanged:
- **Client**: `hero-supervisor-openrpc-client`
- **UI**: `supervisor-admin-ui`
- **Main**: `hero-supervisor`
## Git Dependencies
External projects using Git URLs will automatically pick up the new structure:
```toml
# This continues to work
hero-supervisor-openrpc-client = { git = "https://git.ourworld.tf/herocode/supervisor.git" }
```
Cargo will find the package by name regardless of its location in the repository.
## Local Path Dependencies
If you have local path dependencies, update them:
```toml
# Old
hero-supervisor-openrpc-client = { path = "../supervisor/clients/openrpc" }
# New
hero-supervisor-openrpc-client = { path = "../supervisor/client" }
```
## Scripts and Documentation
All references in scripts, documentation, and examples have been updated to reflect the new structure.

80
docs/test_keypairs.md Normal file
View File

@@ -0,0 +1,80 @@
# Test Keypairs for Supervisor Auth
These are secp256k1 keypairs for testing the supervisor authentication system.
## Keypair 1 (Alice - Admin)
```
Private Key: 0x1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef
Public Key: 0x04a34b99f22c790c4e36b2b3c2c35a36db06226e41c692fc82b8b56ac1c540c5bd5b8dec5235a0fa8722476c7709c02559e3aa73aa03918ba2d492eea75abea235
Address: 0x1234567890abcdef1234567890abcdef12345678
```
## Keypair 2 (Bob - User)
```
Private Key: 0xfedcba0987654321fedcba0987654321fedcba0987654321fedcba0987654321
Public Key: 0x04d0de0aaeaefad02b8bdf8a56451a9852d7f851fee0cc8b4d42f3a0a4c3c2f66c1e5e3e8e3c3e3e3e3e3e3e3e3e3e3e3e3e3e3e3e3e3e3e3e3e3e3e3e3e3e3e3e
Address: 0xfedcba0987654321fedcba0987654321fedcba09
```
## Keypair 3 (Charlie - Register)
```
Private Key: 0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
Public Key: 0x04e68acfc0253a10620dff706b0a1b1f1f5833ea3beb3bde6250d4e5e1e283bb4e9504be11a68d7a263f8e2000d1f8b8c5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e
Address: 0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
```
## Keypair 4 (Dave - Test)
```
Private Key: 0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
Public Key: 0x04f71e8f6c7e8f6c7e8f6c7e8f6c7e8f6c7e8f6c7e8f6c7e8f6c7e8f6c7e8f6c7e8f6c7e8f6c7e8f6c7e8f6c7e8f6c7e8f6c7e8f6c7e8f6c7e8f6c7e8f6c7e
Address: 0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
```
## Keypair 5 (Eve - Test)
```
Private Key: 0xcccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
Public Key: 0x04a0b1c2d3e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0
Address: 0xcccccccccccccccccccccccccccccccccccccccc
```
## Usage Examples
### Using with OpenRPC Client
```rust
use secp256k1::{Secp256k1, SecretKey};
use hex;
// Alice's private key
let alice_privkey = SecretKey::from_slice(
&hex::decode("1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef").unwrap()
).unwrap();
// Sign a message
let secp = Secp256k1::new();
let message = "Hello, Supervisor!";
// ... sign with alice_privkey
```
### Using with Admin UI
You can use the public keys as identifiers when creating API keys:
- Alice: `0x04a34b99f22c790c4e36b2b3c2c35a36db06226e41c692fc82b8b56ac1c540c5bd...`
- Bob: `0x04d0de0aaeaefad02b8bdf8a56451a9852d7f851fee0cc8b4d42f3a0a4c3c2f66c...`
### Testing Different Scopes
1. **Admin Scope** - Use Alice's keypair for full admin access
2. **User Scope** - Use Bob's keypair for limited user access
3. **Register Scope** - Use Charlie's keypair for runner registration only
## Notes
⚠️ **WARNING**: These are TEST keypairs only! Never use these in production!
The private keys are intentionally simple patterns for easy testing:
- Alice: All 0x12...ef pattern
- Bob: Reverse pattern 0xfe...21
- Charlie: All 0xaa
- Dave: All 0xbb
- Eve: All 0xcc