316 lines
9.6 KiB
Markdown
316 lines
9.6 KiB
Markdown
# Worker Lifecycle Management
|
|
|
|
The Hero Supervisor includes comprehensive worker lifecycle management functionality using [Zinit](https://github.com/threefoldtech/zinit) as the process manager. This enables the supervisor to manage worker processes, perform health monitoring, and implement load balancing.
|
|
|
|
## Overview
|
|
|
|
The lifecycle management system provides:
|
|
|
|
- **Worker Process Management**: Start, stop, restart, and monitor worker binaries
|
|
- **Health Monitoring**: Automatic ping jobs every 10 minutes for idle workers
|
|
- **Load Balancing**: Dynamic scaling of workers based on demand
|
|
- **Service Dependencies**: Proper startup ordering with dependency management
|
|
- **Graceful Shutdown**: Clean termination of worker processes
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
|
|
│ Supervisor │ │ WorkerLifecycle │ │ Zinit │
|
|
│ │◄──►│ Manager │◄──►│ (Process │
|
|
│ (Job Dispatch) │ │ │ │ Manager) │
|
|
└─────────────────┘ └──────────────────┘ └─────────────────┘
|
|
│ │ │
|
|
│ │ │
|
|
▼ ▼ ▼
|
|
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
|
|
│ Redis │ │ Health Monitor │ │ Worker Binaries │
|
|
│ (Job Queue) │ │ (Ping Jobs) │ │ (OSIS/SAL/V) │
|
|
└─────────────────┘ └──────────────────┘ └─────────────────┘
|
|
```
|
|
|
|
## Components
|
|
|
|
### WorkerConfig
|
|
|
|
Defines configuration for a worker binary:
|
|
|
|
```rust
|
|
use hero_supervisor::{WorkerConfig, ScriptType};
|
|
use std::path::PathBuf;
|
|
use std::collections::HashMap;
|
|
|
|
let config = WorkerConfig::new(
|
|
"osis_worker_0".to_string(),
|
|
PathBuf::from("/usr/local/bin/osis_worker"),
|
|
ScriptType::OSIS,
|
|
)
|
|
.with_args(vec![
|
|
"--redis-url".to_string(),
|
|
"redis://localhost:6379".to_string(),
|
|
"--worker-id".to_string(),
|
|
"osis_worker_0".to_string(),
|
|
])
|
|
.with_env({
|
|
let mut env = HashMap::new();
|
|
env.insert("RUST_LOG".to_string(), "info".to_string());
|
|
env.insert("WORKER_TYPE".to_string(), "osis".to_string());
|
|
env
|
|
})
|
|
.with_health_check("/usr/local/bin/osis_worker --health-check".to_string())
|
|
.with_dependencies(vec!["redis".to_string()]);
|
|
```
|
|
|
|
### WorkerLifecycleManager
|
|
|
|
Main component for managing worker lifecycles:
|
|
|
|
```rust
|
|
use hero_supervisor::{WorkerLifecycleManagerBuilder, Supervisor};
|
|
|
|
let supervisor = SupervisorBuilder::new()
|
|
.redis_url("redis://localhost:6379")
|
|
.caller_id("my_supervisor")
|
|
.context_id("production")
|
|
.build()?;
|
|
|
|
let mut lifecycle_manager = WorkerLifecycleManagerBuilder::new("/var/run/zinit.sock".to_string())
|
|
.with_supervisor(supervisor.clone())
|
|
.add_worker(osis_worker_config)
|
|
.add_worker(sal_worker_config)
|
|
.add_worker(v_worker_config)
|
|
.build();
|
|
```
|
|
|
|
## Supported Script Types
|
|
|
|
The lifecycle manager supports all Hero script types:
|
|
|
|
- **OSIS**: Rhai/HeroScript execution workers
|
|
- **SAL**: System Abstraction Layer workers
|
|
- **V**: HeroScript execution in V language
|
|
- **Python**: HeroScript execution in Python
|
|
|
|
## Key Features
|
|
|
|
### 1. Worker Management
|
|
|
|
```rust
|
|
// Start all configured workers
|
|
lifecycle_manager.start_all_workers().await?;
|
|
|
|
// Stop all workers
|
|
lifecycle_manager.stop_all_workers().await?;
|
|
|
|
// Restart specific worker
|
|
lifecycle_manager.restart_worker("osis_worker_0").await?;
|
|
|
|
// Get worker status
|
|
let status = lifecycle_manager.get_worker_status("osis_worker_0").await?;
|
|
println!("Worker state: {:?}, PID: {}", status.state, status.pid);
|
|
```
|
|
|
|
### 2. Health Monitoring
|
|
|
|
The system automatically monitors worker health:
|
|
|
|
- Tracks last job execution time for each worker
|
|
- Sends ping jobs to workers idle for 10+ minutes
|
|
- Restarts workers that fail ping checks 3 times
|
|
- Updates job times when workers receive tasks
|
|
|
|
```rust
|
|
// Manual health check
|
|
lifecycle_manager.monitor_worker_health().await?;
|
|
|
|
// Update job time (called automatically by supervisor)
|
|
lifecycle_manager.update_worker_job_time("osis_worker_0");
|
|
|
|
// Start continuous health monitoring
|
|
lifecycle_manager.start_health_monitoring().await; // Runs forever
|
|
```
|
|
|
|
### 3. Dynamic Scaling
|
|
|
|
Scale workers up or down based on demand:
|
|
|
|
```rust
|
|
// Scale OSIS workers to 5 instances
|
|
lifecycle_manager.scale_workers(&ScriptType::OSIS, 5).await?;
|
|
|
|
// Scale down SAL workers to 1 instance
|
|
lifecycle_manager.scale_workers(&ScriptType::SAL, 1).await?;
|
|
|
|
// Check current running count
|
|
let count = lifecycle_manager.get_running_worker_count(&ScriptType::V).await;
|
|
println!("Running V workers: {}", count);
|
|
```
|
|
|
|
### 4. Service Dependencies
|
|
|
|
Workers can depend on other services:
|
|
|
|
```rust
|
|
let config = WorkerConfig::new(name, binary, script_type)
|
|
.with_dependencies(vec![
|
|
"redis".to_string(),
|
|
"database".to_string(),
|
|
"auth_service".to_string(),
|
|
]);
|
|
```
|
|
|
|
Zinit ensures dependencies start before the worker.
|
|
|
|
## Integration with Supervisor
|
|
|
|
The lifecycle manager integrates seamlessly with the supervisor:
|
|
|
|
```rust
|
|
use hero_supervisor::{Supervisor, WorkerLifecycleManager};
|
|
|
|
// Create supervisor and lifecycle manager
|
|
let supervisor = SupervisorBuilder::new().build()?;
|
|
let mut lifecycle_manager = WorkerLifecycleManagerBuilder::new(zinit_socket)
|
|
.with_supervisor(supervisor.clone())
|
|
.build();
|
|
|
|
// Start workers
|
|
lifecycle_manager.start_all_workers().await?;
|
|
|
|
// Create and execute jobs (supervisor automatically routes to workers)
|
|
let job = supervisor
|
|
.new_job()
|
|
.script_type(ScriptType::OSIS)
|
|
.script_content("println!(\"Hello World!\");".to_string())
|
|
.build()?;
|
|
|
|
let result = supervisor.run_job_and_await_result(&job).await?;
|
|
println!("Job result: {}", result);
|
|
```
|
|
|
|
## Zinit Service Configuration
|
|
|
|
The lifecycle manager automatically creates Zinit service configurations:
|
|
|
|
```yaml
|
|
# Generated service config for osis_worker_0
|
|
exec: "/usr/local/bin/osis_worker --redis-url redis://localhost:6379 --worker-id osis_worker_0"
|
|
test: "/usr/local/bin/osis_worker --health-check"
|
|
oneshot: false # Restart on exit
|
|
after:
|
|
- redis
|
|
env:
|
|
RUST_LOG: "info"
|
|
WORKER_TYPE: "osis"
|
|
```
|
|
|
|
## Error Handling
|
|
|
|
The system provides comprehensive error handling:
|
|
|
|
```rust
|
|
use hero_supervisor::SupervisorError;
|
|
|
|
match lifecycle_manager.start_worker(&config).await {
|
|
Ok(_) => println!("Worker started successfully"),
|
|
Err(SupervisorError::WorkerStartFailed(worker, reason)) => {
|
|
eprintln!("Failed to start {}: {}", worker, reason);
|
|
}
|
|
Err(e) => eprintln!("Other error: {}", e),
|
|
}
|
|
```
|
|
|
|
## Example Usage
|
|
|
|
See `examples/lifecycle_demo.rs` for a comprehensive demonstration:
|
|
|
|
```bash
|
|
# Run the lifecycle demo
|
|
cargo run --example lifecycle_demo
|
|
|
|
# Run with custom Redis URL
|
|
REDIS_URL=redis://localhost:6379 cargo run --example lifecycle_demo
|
|
```
|
|
|
|
## Prerequisites
|
|
|
|
1. **Zinit**: Install and run Zinit process manager
|
|
```bash
|
|
curl https://raw.githubusercontent.com/threefoldtech/zinit/refs/heads/master/install.sh | bash
|
|
zinit init --config /etc/zinit/ --socket /var/run/zinit.sock
|
|
```
|
|
|
|
2. **Redis**: Running Redis instance for job queues
|
|
```bash
|
|
redis-server
|
|
```
|
|
|
|
3. **Worker Binaries**: Compiled worker binaries for each script type
|
|
- `/usr/local/bin/osis_worker`
|
|
- `/usr/local/bin/sal_worker`
|
|
- `/usr/local/bin/v_worker`
|
|
- `/usr/local/bin/python_worker`
|
|
|
|
## Configuration Best Practices
|
|
|
|
1. **Resource Limits**: Configure appropriate resource limits in Zinit
|
|
2. **Health Checks**: Implement meaningful health check commands
|
|
3. **Dependencies**: Define proper service dependencies
|
|
4. **Environment**: Set appropriate environment variables
|
|
5. **Logging**: Configure structured logging for debugging
|
|
6. **Monitoring**: Use health monitoring for production deployments
|
|
|
|
## Troubleshooting
|
|
|
|
### Common Issues
|
|
|
|
1. **Zinit Connection Failed**
|
|
- Ensure Zinit is running: `ps aux | grep zinit`
|
|
- Check socket permissions: `ls -la /var/run/zinit.sock`
|
|
- Verify socket path in configuration
|
|
|
|
2. **Worker Start Failed**
|
|
- Check binary exists and is executable
|
|
- Verify dependencies are running
|
|
- Review Zinit logs: `zinit logs <service-name>`
|
|
|
|
3. **Health Check Failures**
|
|
- Implement proper health check endpoint in workers
|
|
- Verify health check command syntax
|
|
- Check worker responsiveness
|
|
|
|
4. **Redis Connection Issues**
|
|
- Ensure Redis is running and accessible
|
|
- Verify Redis URL configuration
|
|
- Check network connectivity
|
|
|
|
### Debug Commands
|
|
|
|
```bash
|
|
# Check Zinit status
|
|
zinit list
|
|
|
|
# View service logs
|
|
zinit logs osis_worker_0
|
|
|
|
# Check service status
|
|
zinit status osis_worker_0
|
|
|
|
# Monitor Redis queues
|
|
redis-cli keys "hero:job:*"
|
|
```
|
|
|
|
## Performance Considerations
|
|
|
|
- **Scaling**: Start with minimal workers and scale based on queue depth
|
|
- **Health Monitoring**: Adjust ping intervals based on workload patterns
|
|
- **Resource Usage**: Monitor CPU/memory usage of worker processes
|
|
- **Queue Depth**: Monitor Redis queue lengths for scaling decisions
|
|
|
|
## Security
|
|
|
|
- **Process Isolation**: Zinit provides process isolation
|
|
- **User Permissions**: Run workers with appropriate user permissions
|
|
- **Network Security**: Secure Redis and Zinit socket access
|
|
- **Binary Validation**: Verify worker binary integrity before deployment
|