circles/research/launcher/README.md

# Circle Launcher

Crate for launching and managing [circle workers](../worker) and the [circles ws server](../server). 

## Features

- **Single-server multi-circle architecture**: One WebSocket server handles all circles via path-based routing
- **Dual operation modes**: Direct spawning or service manager integration
- **Initialization scripts**: Send Rhai scripts to workers on startup
- **Service management**: Automatic restart and background operation support
- **Cross-platform**: macOS (launchctl) and Linux (systemd) support
- **Service cleanup**: Automatic cleanup of background services on exit

## Installation

Build the launcher:

```bash
cargo build --release --bin launcher
```

## Usage

### Basic Syntax

```bash
launcher [OPTIONS] -c <CIRCLE>...
```

### Circle Configuration Format

Circles are specified using the `-c/--circle` option with the format:

```
public_key[:init_script.rhai]
```

- `public_key`: secp256k1 public key in hex format (required)
- `init_script.rhai`: Optional initialization script to send to the worker

### Examples

#### Development Mode (Direct Spawning)

```bash
# Single circle without initialization script
launcher -c 02a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d4

# Multiple circles with initialization scripts
launcher -c 02a1b2c3d4e5f6a7...d4:setup.rhai -c 03b2c3d4e5f6a7...e5:config.rhai

# Mixed configuration (some with scripts, some without)
launcher -c 02a1b2c3d4e5f6a7...d4:init.rhai -c 03b2c3d4e5f6a7...e5
```

#### Production Mode (Service Manager)

```bash
# Using service manager with worker binary
launcher --use-service-manager --worker-binary ./target/release/worker \
  -c 02a1b2c3d4e5f6a7...d4:prod_init.rhai \
  -c 03b2c3d4e5f6a7...e5

# Service manager without initialization scripts
launcher --use-service-manager --worker-binary /usr/local/bin/worker \
  -c 02a1b2c3d4e5f6a7...d4 \
  -c 03b2c3d4e5f6a7...e5
```

## Command Line Options

| Option | Short | Description | Default |
|--------|-------|-------------|---------|
| `--circle` | `-c` | Circle configuration: `public_key[:init_script.rhai]` | Required |
| `--port` | `-p` | WebSocket server port | 8080 |
| `--redis-url` | | Redis connection URL | `redis://127.0.0.1:6379` |
| `--enable-auth` | | Enable WebSocket authentication | false |
| `--use-service-manager` | | Use service manager instead of direct spawning | false |
| `--worker-binary` | | Path to worker binary (required with service manager) | None |
| `--debug` | `-d` | Enable debug logging | false |
| `--verbose` | `-v` | Increase verbosity (can be used multiple times) | 0 |

## Operation Modes

### Direct Spawn Mode (Default)

In direct spawn mode, workers run as Tokio tasks within the launcher process:

- **Pros**: Simple setup, immediate shutdown, ideal for development
- **Cons**: Workers stop when launcher exits, no automatic restart

```bash
launcher -c 02a1b2c3d4e5f6a7...d4:init.rhai
```

### Service Manager Mode

In service manager mode, workers are managed by the system service manager:

- **Pros**: Background operation, automatic restart, production-ready
- **Cons**: Requires worker binary, platform-specific setup

```bash
launcher --use-service-manager --worker-binary ./target/release/worker \
  -c 02a1b2c3d4e5f6a7...d4:init.rhai
```

#### Service Manager Support

- **macOS**: Uses `launchctl` with launch agents
- **Linux**: Uses `systemd` (implementation in progress)

Services are named: `tf.ourworld.circles.circle-worker-{public_key}`

## Architecture

### Single-Server Multi-Circle

The launcher creates one WebSocket server that handles multiple circles through path-based routing:

- **Server URL**: `ws://127.0.0.1:8080`
- **Circle URLs**: `ws://127.0.0.1:8080/{circle_public_key}`
- **Worker Queues**: `rhai_tasks:{circle_public_key}`

### Initialization Scripts

When a circle configuration includes an initialization script:

1. Worker starts and connects to Redis
2. Launcher waits 2 seconds for worker startup
3. Launcher sends script content via RhaiDispatcher to worker's queue
4. Worker executes the initialization script

## Configuration

### Environment Variables

- `RUST_LOG`: Controls logging level (auto-configured based on verbosity)
- `PRESERVE_TASKS`: Preserve Redis tasks on worker shutdown

### Data Directory

The launcher creates a `./launch_data` directory for:
- Worker databases: `circle_db_{public_key}.db`
- Service configuration files (service manager mode)

## Error Handling

Common error scenarios and solutions:

| Error | Cause | Solution |
|-------|-------|----------|
| "Invalid public key" | Malformed secp256k1 key | Verify key format (64 hex chars) |
| "Worker binary path required" | Missing `--worker-binary` in service mode | Provide path to worker executable |
| "Failed to read init script" | Script file not found | Check script file path and permissions |
| "Service already exists" | Service name conflict | Stop existing service or use different key |

## Development

### Building

```bash
# Debug build
cargo build --bin launcher

# Release build
cargo build --release --bin launcher
```

### Testing

```bash
# Run with debug logging
RUST_LOG=debug cargo run --bin launcher -- -c 02a1b2c3d4e5f6a7...d4

# Test service manager mode
cargo run --bin launcher -- --use-service-manager \
  --worker-binary ./target/debug/worker \
  -c 02a1b2c3d4e5f6a7...d4:test.rhai
```

## Troubleshooting

### Worker Connection Issues

1. Verify Redis is running on the specified URL
2. Check worker binary exists and is executable
3. Ensure public keys are valid secp256k1 format

### Service Manager Issues

1. Check service manager logs: `launchctl log show --predicate 'subsystem == "tf.ourworld.circles"'`
2. Verify worker binary permissions and dependencies
3. Ensure working directory is accessible

### Script Execution Issues

1. Verify script file exists and is readable
2. Check Redis connectivity for script transmission
3. Monitor worker logs for script execution errors

## Security Considerations

- **Public Key Validation**: All keys are validated as proper secp256k1 public keys
- **Script Execution**: Initialization scripts run with worker privileges
- **Service Isolation**: Each worker runs as a separate service in service manager mode
- **Redis Security**: Ensure Redis instance is properly secured in production

## Performance

- **Concurrent Workers**: No hard limit on number of circles
- **Resource Usage**: Each worker consumes memory for database and Rhai engine
- **Network**: Single WebSocket server reduces port usage
- **Startup Time**: ~2 second delay for initialization script transmission

## Service Cleanup

The launcher provides automatic cleanup functionality to stop and remove all circle-related services:

### Automatic Cleanup

Examples automatically clean up services on exit:

```bash
# Run example - services are cleaned up automatically on exit or Ctrl+C
cargo run --example circle_launcher_example
```

### Manual Cleanup

Clean up all circle services manually:

```bash
# Using the cleanup example
cargo run --example cleanup_example

# Or using the library function
use circles_launcher::cleanup_launcher;
cleanup_launcher().await?;
```

### What Gets Cleaned Up

The cleanup function removes:
- All worker services (`circle-worker-{public_key}`)
- WebSocket server service (`circle-ws-server`)
- Associated service configuration files (plist files on macOS)

### Signal Handling

Examples include signal handling for graceful cleanup:
- **Ctrl+C**: Triggers cleanup before exit
- **Normal exit**: Always runs cleanup before termination
- **Error exit**: Cleanup still runs to prevent orphaned services