wip

core/supervisor/LIFECYCLE.md

@@ -0,0 +1,315 @@
+# Worker Lifecycle Management
+
+The Hero Supervisor includes comprehensive worker lifecycle management functionality using [Zinit](https://github.com/threefoldtech/zinit) as the process manager. This enables the supervisor to manage worker processes, perform health monitoring, and implement load balancing.
+
+## Overview
+
+The lifecycle management system provides:
+
+- **Worker Process Management**: Start, stop, restart, and monitor worker binaries
+- **Health Monitoring**: Automatic ping jobs every 10 minutes for idle workers
+- **Load Balancing**: Dynamic scaling of workers based on demand
+- **Service Dependencies**: Proper startup ordering with dependency management
+- **Graceful Shutdown**: Clean termination of worker processes
+
+## Architecture
+
+```
+┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
+│   Supervisor    │    │ WorkerLifecycle  │    │     Zinit       │
+│                 │◄──►│    Manager       │◄──►│   (Process      │
+│  (Job Dispatch) │    │                  │    │    Manager)     │
+└─────────────────┘    └──────────────────┘    └─────────────────┘
+         │                       │                       │
+         │                       │                       │
+         ▼                       ▼                       ▼
+┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
+│     Redis       │    │ Health Monitor   │    │ Worker Binaries │
+│   (Job Queue)   │    │  (Ping Jobs)     │    │  (OSIS/SAL/V)   │
+└─────────────────┘    └──────────────────┘    └─────────────────┘
+```
+
+## Components
+
+### WorkerConfig
+
+Defines configuration for a worker binary:
+
+```rust
+use hero_supervisor::{WorkerConfig, ScriptType};
+use std::path::PathBuf;
+use std::collections::HashMap;
+
+let config = WorkerConfig::new(
+    "osis_worker_0".to_string(),
+    PathBuf::from("/usr/local/bin/osis_worker"),
+    ScriptType::OSIS,
+)
+.with_args(vec![
+    "--redis-url".to_string(),
+    "redis://localhost:6379".to_string(),
+    "--worker-id".to_string(),
+    "osis_worker_0".to_string(),
+])
+.with_env({
+    let mut env = HashMap::new();
+    env.insert("RUST_LOG".to_string(), "info".to_string());
+    env.insert("WORKER_TYPE".to_string(), "osis".to_string());
+    env
+})
+.with_health_check("/usr/local/bin/osis_worker --health-check".to_string())
+.with_dependencies(vec!["redis".to_string()]);
+```
+
+### WorkerLifecycleManager
+
+Main component for managing worker lifecycles:
+
+```rust
+use hero_supervisor::{WorkerLifecycleManagerBuilder, Supervisor};
+
+let supervisor = SupervisorBuilder::new()
+    .redis_url("redis://localhost:6379")
+    .caller_id("my_supervisor")
+    .context_id("production")
+    .build()?;
+
+let mut lifecycle_manager = WorkerLifecycleManagerBuilder::new("/var/run/zinit.sock".to_string())
+    .with_supervisor(supervisor.clone())
+    .add_worker(osis_worker_config)
+    .add_worker(sal_worker_config)
+    .add_worker(v_worker_config)
+    .build();
+```
+
+## Supported Script Types
+
+The lifecycle manager supports all Hero script types:
+
+- **OSIS**: Rhai/HeroScript execution workers
+- **SAL**: System Abstraction Layer workers  
+- **V**: HeroScript execution in V language
+- **Python**: HeroScript execution in Python
+
+## Key Features
+
+### 1. Worker Management
+
+```rust
+// Start all configured workers
+lifecycle_manager.start_all_workers().await?;
+
+// Stop all workers
+lifecycle_manager.stop_all_workers().await?;
+
+// Restart specific worker
+lifecycle_manager.restart_worker("osis_worker_0").await?;
+
+// Get worker status
+let status = lifecycle_manager.get_worker_status("osis_worker_0").await?;
+println!("Worker state: {:?}, PID: {}", status.state, status.pid);
+```
+
+### 2. Health Monitoring
+
+The system automatically monitors worker health:
+
+- Tracks last job execution time for each worker
+- Sends ping jobs to workers idle for 10+ minutes
+- Restarts workers that fail ping checks 3 times
+- Updates job times when workers receive tasks
+
+```rust
+// Manual health check
+lifecycle_manager.monitor_worker_health().await?;
+
+// Update job time (called automatically by supervisor)
+lifecycle_manager.update_worker_job_time("osis_worker_0");
+
+// Start continuous health monitoring
+lifecycle_manager.start_health_monitoring().await; // Runs forever
+```
+
+### 3. Dynamic Scaling
+
+Scale workers up or down based on demand:
+
+```rust
+// Scale OSIS workers to 5 instances
+lifecycle_manager.scale_workers(&ScriptType::OSIS, 5).await?;
+
+// Scale down SAL workers to 1 instance  
+lifecycle_manager.scale_workers(&ScriptType::SAL, 1).await?;
+
+// Check current running count
+let count = lifecycle_manager.get_running_worker_count(&ScriptType::V).await;
+println!("Running V workers: {}", count);
+```
+
+### 4. Service Dependencies
+
+Workers can depend on other services:
+
+```rust
+let config = WorkerConfig::new(name, binary, script_type)
+    .with_dependencies(vec![
+        "redis".to_string(),
+        "database".to_string(),
+        "auth_service".to_string(),
+    ]);
+```
+
+Zinit ensures dependencies start before the worker.
+
+## Integration with Supervisor
+
+The lifecycle manager integrates seamlessly with the supervisor:
+
+```rust
+use hero_supervisor::{Supervisor, WorkerLifecycleManager};
+
+// Create supervisor and lifecycle manager
+let supervisor = SupervisorBuilder::new().build()?;
+let mut lifecycle_manager = WorkerLifecycleManagerBuilder::new(zinit_socket)
+    .with_supervisor(supervisor.clone())
+    .build();
+
+// Start workers
+lifecycle_manager.start_all_workers().await?;
+
+// Create and execute jobs (supervisor automatically routes to workers)
+let job = supervisor
+    .new_job()
+    .script_type(ScriptType::OSIS)
+    .script_content("println!(\"Hello World!\");".to_string())
+    .build()?;
+
+let result = supervisor.run_job_and_await_result(&job).await?;
+println!("Job result: {}", result);
+```
+
+## Zinit Service Configuration
+
+The lifecycle manager automatically creates Zinit service configurations:
+
+```yaml
+# Generated service config for osis_worker_0
+exec: "/usr/local/bin/osis_worker --redis-url redis://localhost:6379 --worker-id osis_worker_0"
+test: "/usr/local/bin/osis_worker --health-check"
+oneshot: false  # Restart on exit
+after:
+  - redis
+env:
+  RUST_LOG: "info"
+  WORKER_TYPE: "osis"
+```
+
+## Error Handling
+
+The system provides comprehensive error handling:
+
+```rust
+use hero_supervisor::SupervisorError;
+
+match lifecycle_manager.start_worker(&config).await {
+    Ok(_) => println!("Worker started successfully"),
+    Err(SupervisorError::WorkerStartFailed(worker, reason)) => {
+        eprintln!("Failed to start {}: {}", worker, reason);
+    }
+    Err(e) => eprintln!("Other error: {}", e),
+}
+```
+
+## Example Usage
+
+See `examples/lifecycle_demo.rs` for a comprehensive demonstration:
+
+```bash
+# Run the lifecycle demo
+cargo run --example lifecycle_demo
+
+# Run with custom Redis URL
+REDIS_URL=redis://localhost:6379 cargo run --example lifecycle_demo
+```
+
+## Prerequisites
+
+1. **Zinit**: Install and run Zinit process manager
+   ```bash
+   curl https://raw.githubusercontent.com/threefoldtech/zinit/refs/heads/master/install.sh | bash
+   zinit init --config /etc/zinit/ --socket /var/run/zinit.sock
+   ```
+
+2. **Redis**: Running Redis instance for job queues
+   ```bash
+   redis-server
+   ```
+
+3. **Worker Binaries**: Compiled worker binaries for each script type
+   - `/usr/local/bin/osis_worker`
+   - `/usr/local/bin/sal_worker` 
+   - `/usr/local/bin/v_worker`
+   - `/usr/local/bin/python_worker`
+
+## Configuration Best Practices
+
+1. **Resource Limits**: Configure appropriate resource limits in Zinit
+2. **Health Checks**: Implement meaningful health check commands
+3. **Dependencies**: Define proper service dependencies
+4. **Environment**: Set appropriate environment variables
+5. **Logging**: Configure structured logging for debugging
+6. **Monitoring**: Use health monitoring for production deployments
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Zinit Connection Failed**
+   - Ensure Zinit is running: `ps aux | grep zinit`
+   - Check socket permissions: `ls -la /var/run/zinit.sock`
+   - Verify socket path in configuration
+
+2. **Worker Start Failed**
+   - Check binary exists and is executable
+   - Verify dependencies are running
+   - Review Zinit logs: `zinit logs <service-name>`
+
+3. **Health Check Failures**
+   - Implement proper health check endpoint in workers
+   - Verify health check command syntax
+   - Check worker responsiveness
+
+4. **Redis Connection Issues**
+   - Ensure Redis is running and accessible
+   - Verify Redis URL configuration
+   - Check network connectivity
+
+### Debug Commands
+
+```bash
+# Check Zinit status
+zinit list
+
+# View service logs
+zinit logs osis_worker_0
+
+# Check service status
+zinit status osis_worker_0
+
+# Monitor Redis queues
+redis-cli keys "hero:job:*"
+```
+
+## Performance Considerations
+
+- **Scaling**: Start with minimal workers and scale based on queue depth
+- **Health Monitoring**: Adjust ping intervals based on workload patterns
+- **Resource Usage**: Monitor CPU/memory usage of worker processes
+- **Queue Depth**: Monitor Redis queue lengths for scaling decisions
+
+## Security
+
+- **Process Isolation**: Zinit provides process isolation
+- **User Permissions**: Run workers with appropriate user permissions
+- **Network Security**: Secure Redis and Zinit socket access
+- **Binary Validation**: Verify worker binary integrity before deployment