heroagent/pkg/servers/heroagent/README.md

HeroAgent Server Factory

The HeroAgent Server Factory is a comprehensive server management system that integrates multiple services:

• Redis Server
• WebDAV Server
• UI Server
• Job Management System

Overview

The server factory provides a unified interface for starting, managing, and stopping these services. Each service can be enabled or disabled independently through configuration.

Job Management System

The job management system provides a robust solution for handling asynchronous tasks with persistence and reliability. It combines the strengths of OurDB for persistent storage and Redis for active job queuing.

Architecture

The job system follows a specific flow:

1. Job Creation:

   • When a job is created, it's stored in both OurDB and Redis
   • OurDB provides persistent storage with history tracking
   • Redis stores the job data and adds the job ID to a queue for processing
   • Each job is stored in Redis using a key pattern: herojobs:<topic>:<jobID>
   • Each job ID is added to a queue using a key pattern: heroqueue:<topic>

2. Job Processing:

   • Workers continuously poll Redis queues for new jobs
   • When a job is found, it's fetched from Redis and updated to "active" status
   • The updated job is stored in both OurDB and Redis
   • The job is processed based on its parameters

3. Job Completion:

   • When a job completes (success or error), it's updated in OurDB
   • The job is removed from Redis to keep only active jobs in memory
   • This approach ensures efficient memory usage while maintaining a complete history

Data Flow Diagram

Job Creation:
  ┌─────────┐     ┌─────────┐     ┌─────────┐
  │  Client │────▶│  OurDB  │     │  Redis  │
  └─────────┘     └────┬────┘     └────┬────┘
                       │               │
                       │ Store Job     │ Store Job
                       │               │
                       ▼               ▼
                  ┌─────────┐     ┌─────────┐
                  │ Job Data│     │ Job Data│
                  └─────────┘     └─────────┘
                                      │
                                      │ Add to Queue
                                      │
                                      ▼
                                 ┌─────────┐
                                 │  Queue  │
                                 └─────────┘

Job Processing:
  ┌─────────┐     ┌─────────┐     ┌─────────┐
  │  Worker │────▶│  Redis  │────▶│  OurDB  │
  └─────────┘     └────┬────┘     └────┬────┘
                       │               │
                       │ Fetch Job     │ Update Job
                       │ from Queue    │
                       ▼               ▼
                  ┌─────────┐     ┌─────────┐
                  │ Job Data│     │ Job Data│
                  └─────────┘     └─────────┘
                       │
                       │ Process Job
                       │
                       ▼
                  ┌─────────┐
                  │ Result  │
                  └─────────┘

Job Completion:
  ┌─────────┐     ┌─────────┐     ┌─────────┐
  │  Worker │────▶│  OurDB  │     │  Redis  │
  └─────────┘     └────┬────┘     └────┬────┘
                       │               │
                       │ Update Job    │ Remove Job
                       │               │
                       ▼               ▼
                  ┌─────────┐     ┌─────────┐
                  │ Job Data│     │   ✓     │
                  └─────────┘     └─────────┘

Components

• JobManager: Coordinates job operations between OurDB and Redis
• RedisJobManager: Handles Redis-specific operations for jobs
• JobWorker: Processes jobs from Redis queues
• OurDB: Provides persistent storage for all jobs

Job States

Jobs can be in one of four states:

• New: Job has been created but not yet processed
• Active: Job is currently being processed
• Done: Job has completed successfully
• Error: Job encountered an error during processing

Usage

Configuration

config := heroagent.DefaultConfig()

// Configure Redis
config.Redis.TCPPort = 6379
config.Redis.UnixSocketPath = "/tmp/redis.sock"

// Configure job system
config.Jobs.OurDBPath = "./data/jobsdb"
config.Jobs.WorkerCount = 5
config.Jobs.QueuePollInterval = 100 * time.Millisecond

// Enable/disable services
config.EnableRedis = true
config.EnableWebDAV = true
config.EnableUI = true
config.EnableJobs = true

Starting the Server Factory

// Create server factory
factory := heroagent.New(config)

// Start servers
if err := factory.Start(); err != nil {
    log.Fatalf("Failed to start servers: %v", err)
}

Creating and Managing Jobs

// Get job manager
jobManager := factory.GetJobManager()

// Create a job
job, err := jobManager.CreateJob("email", `{"to": "user@example.com", "subject": "Hello"}`)
if err != nil {
    log.Fatalf("Failed to create job: %v", err)
}

// Get job status
job, err = jobManager.GetJob(job.JobID)
if err != nil {
    log.Fatalf("Failed to get job: %v", err)
}

// Update job status
err = jobManager.UpdateJobStatus(job.JobID, heroagent.JobStatusActive)
if err != nil {
    log.Fatalf("Failed to update job status: %v", err)
}

// Complete a job
err = jobManager.CompleteJob(job.JobID, "Job completed successfully")
if err != nil {
    log.Fatalf("Failed to complete job: %v", err)
}

// Mark a job as failed
err = jobManager.FailJob(job.JobID, "Job failed due to network error")
if err != nil {
    log.Fatalf("Failed to mark job as failed: %v", err)
}

Stopping the Server Factory

// Stop servers
if err := factory.Stop(); err != nil {
    log.Fatalf("Failed to stop servers: %v", err)
}

Implementation Details

OurDB Integration

OurDB provides persistent storage for all jobs, including their complete history. It uses an auto-incrementing ID system to assign unique IDs to jobs.

Redis Integration

Redis is used for active job queuing and temporary storage. Jobs are stored in Redis using the following key patterns:

• Queue keys: heroqueue:<topic>
• Job storage keys: herojobs:<topic>:<jobID>

When a job reaches a terminal state (done or error), it's removed from Redis but remains in OurDB for historical reference.

Worker Pool

The job system uses a configurable worker pool to process jobs concurrently. Each worker polls Redis queues for new jobs and processes them independently.