herocode/home

Fork 0

Files

Timur Gordon ac6020d883 Add comprehensive architecture documentation with Freezone reference

2025-10-31 02:29:10 +01:00

20 KiB

Raw Permalink Blame History

Hero Architecture: Scalable Backend System

Proven with Zanzibar Freezone - Digital Residency & Company Registration

The Stack

┌─────────────────────────────────────────────────────────┐
│ CLIENT (HTTP/SDK)                                        │
│ • Signs jobs with secp256k1                             │
│ • Submits to Supervisor                                 │
└──────────────────────┬──────────────────────────────────┘
                       │
┌──────────────────────▼──────────────────────────────────┐
│ SUPERVISOR (https://git.ourworld.tf/herocode/supervisor)│
│ • Verifies signatures                                   │
│ • Queues to Redis                                       │
│ • Routes to runners                                     │
└──────────────────────┬──────────────────────────────────┘
                       │
┌──────────────────────▼──────────────────────────────────┐
│ RUNNER (https://git.ourworld.tf/herocode/runner_rust)   │
│ • Executes Rhai scripts                                 │
│ • Access control via signatures                         │
│ • Registers domain models                               │
└──────────────────────┬──────────────────────────────────┘
                       │
┌──────────────────────▼──────────────────────────────────┐
│ OSIRIS (https://git.ourworld.tf/herocode/osiris)        │
│ • Generic object storage                                │
│ • Automatic indexing                                    │
│ • Context isolation                                     │
└──────────────────────┬──────────────────────────────────┘
                       │
┌──────────────────────▼──────────────────────────────────┐
│ HERODB (https://git.ourworld.tf/herocode/herodb)        │
│ • Redis-compatible                                      │
│ • Age encryption                                        │
│ • Per-database keys                                     │
└─────────────────────────────────────────────────────────┘

Freezone: Production Implementation

Repository: https://git.ourworld.tf/zdfz/backend

What It Does

Digital residency registration with:

Email verification (SMTP)
Payment processing (Pesapal)
KYC verification (Idenfy)
Company registration
Invoice management

Architecture

// HTTP API receives request
POST /api/v1/digital-residents

// API creates Rhai script
let script = format!(r#"
    let ctx = get_context(["freezone_pubkey"]);
    
    let user = digital_resident()
        .username("{}")
        .email("{}")
        .pubkey("{}");
    
    ctx.save(user);
    
    send_verification_email(user.email);
"#, username, email, pubkey);

// Submit to Supervisor
supervisor_client.queue_job(script).await?;

// Runner executes with models registered
// Data stored in HeroDB with automatic indexing

Key files:

src/bin/server.rs - HTTP API
src/bin/runner_zdfz/ - Osiris runner
sdk/models/ - Domain models

Core Components

1. Models (Define Your Domain)

Location: Your repo (e.g., zdfz/sdk/models)

// models/src/digital_resident/model.rs

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct DigitalResident {
    pub base_data: BaseData,
    #[index]
    pub email: String,
    #[index]
    pub pubkey: String,
    pub username: String,
    pub verification_status: VerificationStatus,
}

impl Object for DigitalResident {
    fn object_type() -> &'static str { "digital_resident" }
    fn base_data(&self) -> &BaseData { &self.base_data }
    
    fn index_keys(&self) -> Vec<IndexKey> {
        vec![
            IndexKey::new("email", &self.email),
            IndexKey::new("pubkey", &self.pubkey),
        ]
    }
    
    // ... serialization methods
}

Result:

HeroDB stores: obj:digital_residents:<id> → JSON
HeroDB indexes: idx:digital_residents:email:<email> → ID
Queries: O(1) lookup by email or pubkey

2. Rhai Builders (Script API)

Location: Your repo (e.g., zdfz/sdk/models/digital_resident/rhai.rs)

// models/src/digital_resident/rhai.rs

pub fn register_digital_resident_builders(engine: &mut Engine) {
    engine.register_fn("digital_resident", || DigitalResident {
        base_data: BaseData::new("digital_residents"),
        email: String::new(),
        pubkey: String::new(),
        username: String::new(),
        verification_status: VerificationStatus::Pending,
    });
    
    engine.register_fn("email", |mut dr, email: String| {
        dr.email = email;
        dr
    });
    
    engine.register_fn("username", |mut dr, username: String| {
        dr.username = username;
        dr
    });
}

Result: Fluent API in Rhai scripts:

let user = digital_resident()
    .email("alice@example.com")
    .username("alice")
    .pubkey("0x123...");

3. Runner (Execute Scripts)

Location: Your repo (e.g., zdfz/backend/src/bin/runner_zdfz)

// src/bin/runner_zdfz/engine.rs

pub fn create_zdfz_engine() -> Engine {
    let mut engine = Engine::new();
    
    // Load OSIRIS core
    let osiris_package = OsirisPackage::new();
    osiris_package.register_into_engine(&mut engine);
    
    // Register your models
    register_digital_resident_builders(&mut engine);
    register_freezone_company_builders(&mut engine);
    register_invoice_builders(&mut engine);
    
    // Register external services
    register_email_client(&mut engine);
    register_payment_client(&mut engine);
    register_kyc_client(&mut engine);
    
    engine
}

// src/bin/runner_zdfz/main.rs

#[tokio::main]
async fn main() {
    let redis_url = env::var("REDIS_URL").unwrap();
    let queue = env::var("QUEUE_NAME").unwrap();
    
    let client = redis::Client::open(redis_url).unwrap();
    let mut conn = client.get_connection().unwrap();
    
    loop {
        // Block until job available
        let result: Vec<String> = redis::cmd("BLPOP")
            .arg(&queue)
            .arg(0)
            .query(&mut conn)
            .unwrap();
        
        let job: Job = serde_json::from_str(&result[1]).unwrap();
        
        // Create engine with signatories
        let mut engine = create_zdfz_engine();
        set_signatories(&mut engine, &job);
        
        // Execute
        match engine.run(&job.payload) {
            Ok(_) => println!("Job {} completed", job.id),
            Err(e) => eprintln!("Job {} failed: {}", job.id, e),
        }
    }
}

Result: Runner polls Redis, executes scripts with your models

4. Supervisor (Job Queue)

Repository: https://git.ourworld.tf/herocode/supervisor

What it does:

Verifies job signatures (secp256k1)
Queues to Redis
Routes to runners
Returns results

API:

# Submit job
curl -X POST http://supervisor:3030 \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc": "2.0",
    "method": "queue_job_to_runner",
    "params": {
      "runner_name": "zdfz_runner",
      "job": {
        "id": "job-123",
        "payload": "let ctx = get_context([\"user_pubkey\"]); ...",
        "signatures": [{"public_key": "0x...", "signature": "0x..."}]
      }
    },
    "id": 1
  }'

Key files:

src/supervisor.rs - Core logic
src/openrpc.rs - JSON-RPC API
src/app.rs - Signature verification

5. OSIRIS (Object Storage)

Repository: https://git.ourworld.tf/herocode/osiris

What it provides:

Object trait for models
Automatic indexing in HeroDB
Context-based access control
Rhai integration

Key files:

src/context.rs - Context API
src/engine.rs - Rhai engine setup
src/store.rs - Generic storage

Usage in Rhai:

// Get context (verifies signatories)
let ctx = get_context(["user_pubkey"]);

// Save object (automatic indexing)
ctx.save(user);

// Query by index (O(1) lookup)
let users = ctx.query("digital_residents", "email", "alice@example.com");

// Get by ID
let user = ctx.get("digital_residents", "user-123");

6. HeroDB (Storage)

Repository: https://git.ourworld.tf/herocode/herodb

What it provides:

Redis protocol compatibility
Age encryption at rest
Per-database keys
Admin database (DB 0)

Run:

cd herocode/herodb
cargo run -- --admin-secret secret --port 6379

Result: Drop-in Redis replacement with encryption

Signature-Based Access Control

Core concept: Signatures determine access. No central auth server.

Single Party

// Client signs job
let mut job = Job::new(script);
job.sign(&alice_secret_key)?;
supervisor.queue_job(job).await?;

// Script: only Alice can access
let ctx = get_context(["alice_pubkey"]);  // ✓ Works
let ctx = get_context(["bob_pubkey"]);    // ✗ Access denied

Multi-Party

// Alice creates and signs
let mut job = Job::new(script);
job.sign(&alice_secret_key)?;

// Bob adds signature
job.sign(&bob_secret_key)?;

// Submit with both signatures
supervisor.queue_job(job).await?;

// Both can access shared context
let ctx = get_context(["alice_pubkey", "bob_pubkey"]);

let shared_data = company()
    .name("Acme Corp")
    .add_shareholder("alice_pubkey")
    .add_shareholder("bob_pubkey");

ctx.save(shared_data);

Implementation: osiris/src/engine.rs - get_context() function

Scalability

Horizontal Scaling

Runners are stateless:

# Start 10 runners for same queue
for i in {1..10}; do
    REDIS_URL=redis://localhost:6379 \
    QUEUE_NAME=zdfz_runner \
    ./runner_zdfz &
done

Jobs automatically distributed via Redis BLPOP.

Freezone production: 3 runners handling 1000+ registrations/day

Queue Partitioning

let queue = match priority {
    Priority::Urgent => "zdfz_urgent",
    Priority::Normal => "zdfz_normal",
};

supervisor.queue_job_to_runner(queue, job).await?;

Freezone production: Separate queues for registration, payment, KYC

Database Sharding

let shard = hash(context_id) % num_shards;
let herodb_url = format!("redis://herodb-{}.internal:6379", shard);

OsirisContext::builder()
    .herodb_url(&herodb_url)
    .build()?;

Freezone ready: Can shard by country/region

Multi-Region

Region A (EU)          Region B (Asia)        Region C (US)
├─ Supervisor          ├─ Supervisor          ├─ Supervisor
├─ Runners (3)         ├─ Runners (3)         ├─ Runners (3)
└─ HeroDB              └─ HeroDB              └─ HeroDB
        │                      │                      │
        └──────────────────────┴──────────────────────┘
                               │
                        Redis Cluster

Freezone ready: Can deploy per jurisdiction

External Service Integration

Email (SMTP)

Location: osiris/src/objects/communication/

// Register in runner
register_email_client(&mut engine);

// Use in script
send_email(
    "user@example.com",
    "Verify your email",
    "Click here: https://..."
);

Freezone: Brevo SMTP for verification emails

Payment (Pesapal)

Location: osiris/src/objects/money/

// Register in runner
register_payment_client(&mut engine);

// Use in script
let payment = create_payment_link(
    100.0,
    "USD",
    "Registration fee"
);

print("Pay here: " + payment.url);

Freezone: Pesapal for registration fees

KYC (Idenfy)

Location: osiris/src/objects/kyc/

// Register in runner
register_kyc_client(&mut engine);

// Use in script
let kyc_session = create_kyc_verification(
    "user-123",
    "Alice",
    "Smith",
    "alice@example.com"
);

print("Verify here: " + kyc_session.url);

Freezone: Idenfy for identity verification

Coordinator (Optional)

Repository: https://git.ourworld.tf/herocode/herocoordinator

Purpose: Multi-step workflows (DAGs)

When to use:

Complex workflows with dependencies
Conditional execution
Long-running processes

Example: Freezone registration flow

1. Create user → 2. Send email → 3. Wait verification
                                        ↓
                                 4. Create payment
                                        ↓
                                 5. Wait payment
                                        ↓
                                 6. Create KYC
                                        ↓
                                 7. Wait KYC
                                        ↓
                                 8. Activate account

UI: herocoordinator/clients/coordinator-ui/ - Visual DAG editor

Deployment

Development (Single Node)

# 1. HeroDB
cd herocode/herodb
cargo run -- --admin-secret secret

# 2. Supervisor
cd herocode/supervisor
cargo run -- --redis-url redis://localhost:6379

# 3. Your Runner
cd your_backend
REDIS_URL=redis://localhost:6379 \
QUEUE_NAME=your_runner \
cargo run --bin runner

Production (Multi-Node)

# Node 1: Supervisor
cd herocode/supervisor
cargo run --release -- \
    --redis-url redis://cluster:6379 \
    --admin-secret $ADMIN_SECRET

# Node 2-N: Workers
cd your_backend
REDIS_URL=redis://cluster:6379 \
QUEUE_NAME=your_runner \
./runner &

cd herocode/herodb
./herodb --admin-secret $ADMIN_SECRET --port 6380

Mycelium (P2P)

Documentation: home/MYCELIUM_INTEGRATION_SUMMARY.md

# Start Mycelium daemon
mycelium --peers tcp://188.40.132.242:9651 \
         --no-tun \
         --jsonrpc-addr 127.0.0.1:8990

# Start Supervisor with Mycelium
cd herocode/supervisor
cargo run -- \
    --mycelium-url http://127.0.0.1:8990 \
    --topic supervisor.rpc

Benefits:

P2P communication
Encrypted overlay network
NAT traversal
No central server

Why This Architecture Scales

1. Stateless Runners

No session state
All data in HeroDB
Scale by adding processes

Freezone: 3 runners → 10 runners = 3x throughput

2. Signature-Based Auth

No central auth server
No session management
Cryptographic proof

Freezone: No auth server to scale or fail

3. Context Isolation

Multi-tenant by design
Per-context access control
Natural sharding boundary

Freezone: Each user has isolated context

4. Redis Queue

Proven at scale
BLPOP for fair distribution
Can cluster for HA

Freezone: Redis handles 10k+ jobs/day

5. Automatic Indexing

Define index_keys() → automatic indexes
O(1) lookups
No manual index management

Freezone: Query by email, pubkey, status - all O(1)

Building Your Backend

1. Define Models

Implement Object trait:

use osiris::{BaseData, Object, IndexKey};

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct YourModel {
    pub base_data: BaseData,
    #[index]
    pub indexed_field: String,
    pub data_field: String,
}

impl Object for YourModel {
    fn object_type() -> &'static str { "your_model" }
    fn base_data(&self) -> &BaseData { &self.base_data }
    
    fn index_keys(&self) -> Vec<IndexKey> {
        vec![IndexKey::new("indexed_field", &self.indexed_field)]
    }
    
    // ... serialization
}

2. Register Rhai Builders

pub fn register_your_model_builders(engine: &mut Engine) {
    engine.register_fn("your_model", || YourModel::default());
    
    engine.register_fn("indexed_field", |mut m, val: String| {
        m.indexed_field = val;
        m
    });
}

3. Create Runner

fn create_engine() -> Engine {
    let mut engine = Engine::new();
    
    // OSIRIS core
    let osiris_package = OsirisPackage::new();
    osiris_package.register_into_engine(&mut engine);
    
    // Your models
    register_your_model_builders(&mut engine);
    
    engine
}

#[tokio::main]
async fn main() {
    // Poll Redis queue
    // Execute scripts
    // Return results
}

4. Write Scripts

let ctx = get_context(["user_pubkey"]);

let obj = your_model()
    .indexed_field("value");

ctx.save(obj);

5. Build Client

// Sign job
let mut job = Job::new(script);
job.sign(&secret_key)?;

// Submit
supervisor.queue_job(job).await?;

Repository Links

Summary

Freezone demonstrates:

✓ Production-ready (digital residency live)
✓ External integrations (Email, Payment, KYC)
✓ Multi-tenant (context isolation)
✓ Scalable (stateless runners)
✓ Secure (signature-based auth)
✓ Fast (automatic indexing)

Architecture enables:

Any domain models (implement Object trait)
Any external services (register in runner)
Any scale (horizontal scaling)
Any deployment (single-node → multi-region)

What you reuse:

Supervisor, HeroDB, OSIRIS, Job model

What you customize:

Models, Rhai builders, scripts, integrations

Result: Build backends fast, scale easily, no central auth server.

20 KiB Raw Permalink Blame History

Hero Architecture: Scalable Backend System

The Stack

Freezone: Production Implementation

What It Does

Architecture

Core Components

1. Models (Define Your Domain)

2. Rhai Builders (Script API)

3. Runner (Execute Scripts)

4. Supervisor (Job Queue)

5. OSIRIS (Object Storage)

6. HeroDB (Storage)

Signature-Based Access Control

Single Party

Multi-Party

Scalability

Horizontal Scaling

Queue Partitioning

Database Sharding

Multi-Region

External Service Integration

Email (SMTP)

Payment (Pesapal)

KYC (Idenfy)

Coordinator (Optional)

Deployment

Development (Single Node)

Production (Multi-Node)

Mycelium (P2P)

Why This Architecture Scales

1. Stateless Runners

2. Signature-Based Auth

3. Context Isolation

4. Redis Queue

5. Automatic Indexing

Building Your Backend

1. Define Models

2. Register Rhai Builders

3. Create Runner

4. Write Scripts

5. Build Client

Repository Links

Core Infrastructure

Core Framework

Reference Implementation

Documentation

Summary

20 KiB

Raw Permalink Blame History