lhumina_code/hero_biz
0
0
Fork 0
Code Issues 2 Pull requests Projects Releases 5 Packages Wiki Activity Actions

Route AiClient through local AI Broker instead of direct provider calls #15

New issue
Closed
opened 2026-05-04 09:30:14 +00:00 by casper-stevens · 5 comments
casper-stevens commented 2026-05-04 09:30:14 +00:00
Member
Copy link

Problem

The current AiClient in crates/hero_biz_ui/src/ai/client.rs is an OpenAI-compatible HTTP client that calls external AI providers directly via base_url + api_key on AiProvider. This breaks the intended Hero stack architecture.

Required Architecture

All AI calls must flow through the local AI Broker:

Tool / Service
   ↓
AI Client
   ↓
hero_aibroker (local)
   ↓
hero_aibroker (mother) → External providers

The broker already exists (hero_aibroker) and exposes an OpenAI-compatible REST surface at ~/hero/var/sockets/hero_aibroker/rest.sock.

SDK Gap — Blocker

hero_biz uses non-streaming chat exclusively (analyze_intent, generate_suggestions, all assistant calls go through AiClient.chat()). The current hero_aibroker_sdk only provides:

  • AIBrokerAdminAPIClient — admin RPC only (health, models, info), no chat
  • StreamingClient — forces stream: true, streaming only
  • AIBrokerRawClient — raw JSON-RPC; chat is not exposed over the RPC socket

There is no non-streaming chat client in the SDK. This must be resolved first.

Path Forward

Step 1 (prerequisite — work in hero_aibroker): Add a non-streaming REST client to hero_aibroker_sdk that POSTs to rest.sock at /v1/chat/completions without forcing streaming. Something like a RestClient alongside the existing StreamingClient.

Step 2 (this repo): Replace hero_biz's AiClient with the new SDK client:

  • Remove AiProvider, AiConfig, and all provider/API key config from hero_biz — that becomes the broker's concern
  • Remove the try_chat fallback loop — failover is handled inside the broker
  • Wire chat(), transcribe(), and tts() calls to the broker REST socket via the SDK

References

  • crates/hero_biz_ui/src/ai/client.rs — AiClient, try_chat, try_transcribe, try_tts
  • crates/hero_biz_ui/src/ai/config.rs — AiProvider, AiConfig, AiModel
  • hero_aibroker/crates/hero_aibroker_sdk/src/streaming.rs — existing StreamingClient (streaming only)
  • hero_aibroker/crates/hero_aibroker_sdk/src/lib.rs — AIBrokerAdminAPIClient, AIBrokerRawClient
  • Meeting: Engineering flow / Ubuntu 4 May — Section 8 (AI Client Cleanup)
## Problem The current `AiClient` in `crates/hero_biz_ui/src/ai/client.rs` is an OpenAI-compatible HTTP client that calls external AI providers directly via `base_url` + `api_key` on `AiProvider`. This breaks the intended Hero stack architecture. ## Required Architecture All AI calls must flow through the local AI Broker: ``` Tool / Service ↓ AI Client ↓ hero_aibroker (local) ↓ hero_aibroker (mother) → External providers ``` The broker already exists (`hero_aibroker`) and exposes an OpenAI-compatible REST surface at `~/hero/var/sockets/hero_aibroker/rest.sock`. ## SDK Gap — Blocker hero_biz uses **non-streaming chat exclusively** (`analyze_intent`, `generate_suggestions`, all assistant calls go through `AiClient.chat()`). The current `hero_aibroker_sdk` only provides: - `AIBrokerAdminAPIClient` — admin RPC only (health, models, info), no chat - `StreamingClient` — forces `stream: true`, streaming only - `AIBrokerRawClient` — raw JSON-RPC; chat is not exposed over the RPC socket There is no non-streaming chat client in the SDK. This must be resolved first. ## Path Forward **Step 1 (prerequisite — work in `hero_aibroker`):** Add a non-streaming REST client to `hero_aibroker_sdk` that POSTs to `rest.sock` at `/v1/chat/completions` without forcing streaming. Something like a `RestClient` alongside the existing `StreamingClient`. **Step 2 (this repo):** Replace hero_biz's `AiClient` with the new SDK client: - Remove `AiProvider`, `AiConfig`, and all provider/API key config from hero_biz — that becomes the broker's concern - Remove the `try_chat` fallback loop — failover is handled inside the broker - Wire `chat()`, `transcribe()`, and `tts()` calls to the broker REST socket via the SDK ## References - `crates/hero_biz_ui/src/ai/client.rs` — `AiClient`, `try_chat`, `try_transcribe`, `try_tts` - `crates/hero_biz_ui/src/ai/config.rs` — `AiProvider`, `AiConfig`, `AiModel` - `hero_aibroker/crates/hero_aibroker_sdk/src/streaming.rs` — existing `StreamingClient` (streaming only) - `hero_aibroker/crates/hero_aibroker_sdk/src/lib.rs` — `AIBrokerAdminAPIClient`, `AIBrokerRawClient` - Meeting: Engineering flow / Ubuntu 4 May — Section 8 (AI Client Cleanup)
mik-tf added this to the ACTIVE project 2026-05-06 17:32:00 +00:00
casper-stevens commented 2026-05-08 11:48:41 +00:00
Author
Member
Copy link

Blocked by lhumina_code/hero_aibroker#63

This issue cannot start until hero_aibroker_sdk exposes a non-streaming REST chat client. The SDK currently only has StreamingClient and AIBrokerAdminAPIClient — neither supports a plain POST /v1/chat/completions without forcing streaming.

The prerequisite work is tracked in hero_aibroker#63 — Consolidate herolib_ai into hero_aibroker_sdk. Specifically, step 1 of that issue (add from_default_socket() + non-streaming chat to HeroAibrokerClient) must land before this can proceed.

**Blocked by** lhumina_code/hero_aibroker#63 This issue cannot start until `hero_aibroker_sdk` exposes a non-streaming REST chat client. The SDK currently only has `StreamingClient` and `AIBrokerAdminAPIClient` — neither supports a plain `POST /v1/chat/completions` without forcing streaming. The prerequisite work is tracked in [hero_aibroker#63 — Consolidate herolib_ai into hero_aibroker_sdk](https://forge.ourworld.tf/lhumina_code/hero_aibroker/issues/63). Specifically, step 1 of that issue (add `from_default_socket()` + non-streaming chat to `HeroAibrokerClient`) must land before this can proceed.
casper-stevens commented 2026-05-11 09:04:34 +00:00
Author
Member
Copy link

Implementation Spec for Issue #15

Objective

Replace hero_biz_admin's direct-provider AiClient with calls routed through the local hero_aibroker REST socket, eliminating all provider/API-key configuration from hero_biz. The public surface of AiClient (the methods used by assistant.rs and server.rs) is preserved so callers need only minimal changes.

Key Findings

Correction vs. issue text: The files live in crates/hero_biz_admin/, not crates/hero_biz_ui/. The issue refers to an older layout.

SDK state: The hero_aibroker_sdk StreamingClient talks to rest.sock over raw Unix HTTP. The same pattern must be used for non-streaming calls — there is no pre-built non-streaming client in the SDK, so a thin helper is written directly in client.rs.

Broker REST socket path: $HERO_SOCKET_DIR/hero_aibroker/rest.sock

Callers of AiClient:

  • crates/hero_biz_admin/src/ai/assistant.rs — calls .chat(ModelPurpose::*, …)
  • crates/hero_biz_admin/src/web/server.rs — constructs AiClient from AiConfig; holds Option<AiClient> in AppState
  • crates/hero_biz_admin/src/lib.rs — passes Option<AiConfig> into create_web_router

Requirements

  • All AI calls must flow through the broker's REST socket (rest.sock), never directly to external providers
  • Remove AiProvider, AiConfig, AiModel from hero_biz — no provider URLs, no API keys, no model selection
  • Remove try_chat, try_transcribe, try_tts fallback loops — the broker handles routing/fallback
  • New AiClient constructable from HERO_SOCKET_DIR env var only
  • Preserve chat(), transcribe(), text_to_speech(), text_chat(), morph() signatures for existing callers
  • Keep ChatMessage, MessageRole, TranscriptionRequest public (used by handlers)
  • Remove reqwest from hero_biz_admin deps (only used in ai/client.rs)
  • create_web_router no longer accepts ai_config parameter

Files to Modify

File Action
Cargo.toml (workspace root) Add hero_aibroker_sdk to [workspace.dependencies]
crates/hero_biz_admin/Cargo.toml Add hero_aibroker_sdk; remove reqwest
crates/hero_biz_admin/src/ai/client.rs Rewrite — broker socket HTTP client, keep public types
crates/hero_biz_admin/src/ai/config.rs Remove AiProvider, AiConfig, AiModel; keep ModelPurpose
crates/hero_biz_admin/src/ai/mod.rs Update re-exports
crates/hero_biz_admin/src/web/server.rs Replace AiConfig-based construction with AiClient::from_env()
crates/hero_biz_admin/src/lib.rs Remove ai_config parameter from create_web_router

Implementation Plan

Step 1 — Update Cargo.toml files

Files: Cargo.toml, crates/hero_biz_admin/Cargo.toml

  • Add hero_aibroker_sdk git dependency to workspace
  • Add it to hero_biz_admin deps
  • Remove reqwest from hero_biz_admin (verify no other usage first)
    Dependencies: none

Step 2 — Rewrite crates/hero_biz_admin/src/ai/client.rs

Files: crates/hero_biz_admin/src/ai/client.rs

  • Add private unix_post_json(path, body) -> Result<serde_json::Value> helper using tokio::net::UnixStream raw HTTP/1.1
  • Add private unix_post_multipart helper for STT
  • New AiClient { socket_path: PathBuf } with from_env() constructor
  • Implement chat() (sends model: "auto", ignores ModelPurpose), transcribe(), text_to_speech(), text_chat(), morph()
  • Keep ChatMessage, MessageRole, TranscriptionRequest public; remove all private OpenAI-client structs
    Dependencies: none (Step 1 for compile)

Step 3 — Update config.rs and mod.rs

Files: crates/hero_biz_admin/src/ai/config.rs, crates/hero_biz_admin/src/ai/mod.rs

  • Remove AiProvider, AiConfig, AiModel from config.rs
  • Keep ModelPurpose enum and its trait impls
  • Update mod.rs to not re-export removed types
    Dependencies: Step 2

Step 4 — Update server.rs and lib.rs

Files: crates/hero_biz_admin/src/web/server.rs, crates/hero_biz_admin/src/lib.rs

  • Replace AiConfig-based construction with AiClient::from_env()
  • Remove ai_config param from create_web_router and create_router
  • Keep Option<AiClient> in AppState but always Some(AiClient::from_env())
  • Update handler code that unwraps ai_client (grep for .ai_client)
    Dependencies: Step 3

Step 5 — Verify build and fix compile errors

  • Run cargo build -p hero_biz_admin
  • Fix remaining errors
  • Run cargo test -p hero_biz_admin
    Dependencies: Step 4

Acceptance Criteria

  • cargo build -p hero_biz_admin succeeds
  • AiProvider, AiConfig, AiModel no longer exist in hero_biz
  • reqwest removed from hero_biz_admin/Cargo.toml
  • All AI calls route to unix://.../hero_aibroker/rest.sock
  • No hardcoded base_url or api_key remains
  • create_web_router no longer accepts ai_config
  • cargo test -p hero_biz_admin passes

Notes

  • ModelPurpose is kept in config.rs for API compatibility with assistant.rs; the implementation ignores it and sends model: "auto" to the broker
  • STT multipart is written manually over UnixStream (same raw-socket approach as chat)
  • TTS reads raw bytes response rather than JSON
  • The broker handles all routing, fallback, and model selection internally
## Implementation Spec for Issue #15 ### Objective Replace `hero_biz_admin`'s direct-provider `AiClient` with calls routed through the local `hero_aibroker` REST socket, eliminating all provider/API-key configuration from hero_biz. The public surface of `AiClient` (the methods used by `assistant.rs` and `server.rs`) is preserved so callers need only minimal changes. ### Key Findings **Correction vs. issue text:** The files live in `crates/hero_biz_admin/`, not `crates/hero_biz_ui/`. The issue refers to an older layout. **SDK state:** The `hero_aibroker_sdk` `StreamingClient` talks to `rest.sock` over raw Unix HTTP. The same pattern must be used for non-streaming calls — there is no pre-built non-streaming client in the SDK, so a thin helper is written directly in `client.rs`. **Broker REST socket path:** `$HERO_SOCKET_DIR/hero_aibroker/rest.sock` **Callers of `AiClient`:** - `crates/hero_biz_admin/src/ai/assistant.rs` — calls `.chat(ModelPurpose::*, …)` - `crates/hero_biz_admin/src/web/server.rs` — constructs `AiClient` from `AiConfig`; holds `Option<AiClient>` in `AppState` - `crates/hero_biz_admin/src/lib.rs` — passes `Option<AiConfig>` into `create_web_router` ### Requirements - All AI calls must flow through the broker's REST socket (`rest.sock`), never directly to external providers - Remove `AiProvider`, `AiConfig`, `AiModel` from hero_biz — no provider URLs, no API keys, no model selection - Remove `try_chat`, `try_transcribe`, `try_tts` fallback loops — the broker handles routing/fallback - New `AiClient` constructable from `HERO_SOCKET_DIR` env var only - Preserve `chat()`, `transcribe()`, `text_to_speech()`, `text_chat()`, `morph()` signatures for existing callers - Keep `ChatMessage`, `MessageRole`, `TranscriptionRequest` public (used by handlers) - Remove `reqwest` from `hero_biz_admin` deps (only used in `ai/client.rs`) - `create_web_router` no longer accepts `ai_config` parameter ### Files to Modify | File | Action | |---|---| | `Cargo.toml` (workspace root) | Add `hero_aibroker_sdk` to `[workspace.dependencies]` | | `crates/hero_biz_admin/Cargo.toml` | Add `hero_aibroker_sdk`; remove `reqwest` | | `crates/hero_biz_admin/src/ai/client.rs` | Rewrite — broker socket HTTP client, keep public types | | `crates/hero_biz_admin/src/ai/config.rs` | Remove `AiProvider`, `AiConfig`, `AiModel`; keep `ModelPurpose` | | `crates/hero_biz_admin/src/ai/mod.rs` | Update re-exports | | `crates/hero_biz_admin/src/web/server.rs` | Replace `AiConfig`-based construction with `AiClient::from_env()` | | `crates/hero_biz_admin/src/lib.rs` | Remove `ai_config` parameter from `create_web_router` | ### Implementation Plan #### Step 1 — Update Cargo.toml files Files: `Cargo.toml`, `crates/hero_biz_admin/Cargo.toml` - Add `hero_aibroker_sdk` git dependency to workspace - Add it to `hero_biz_admin` deps - Remove `reqwest` from `hero_biz_admin` (verify no other usage first) Dependencies: none #### Step 2 — Rewrite `crates/hero_biz_admin/src/ai/client.rs` Files: `crates/hero_biz_admin/src/ai/client.rs` - Add private `unix_post_json(path, body) -> Result<serde_json::Value>` helper using `tokio::net::UnixStream` raw HTTP/1.1 - Add private `unix_post_multipart` helper for STT - New `AiClient { socket_path: PathBuf }` with `from_env()` constructor - Implement `chat()` (sends `model: "auto"`, ignores `ModelPurpose`), `transcribe()`, `text_to_speech()`, `text_chat()`, `morph()` - Keep `ChatMessage`, `MessageRole`, `TranscriptionRequest` public; remove all private OpenAI-client structs Dependencies: none (Step 1 for compile) #### Step 3 — Update `config.rs` and `mod.rs` Files: `crates/hero_biz_admin/src/ai/config.rs`, `crates/hero_biz_admin/src/ai/mod.rs` - Remove `AiProvider`, `AiConfig`, `AiModel` from `config.rs` - Keep `ModelPurpose` enum and its trait impls - Update `mod.rs` to not re-export removed types Dependencies: Step 2 #### Step 4 — Update `server.rs` and `lib.rs` Files: `crates/hero_biz_admin/src/web/server.rs`, `crates/hero_biz_admin/src/lib.rs` - Replace `AiConfig`-based construction with `AiClient::from_env()` - Remove `ai_config` param from `create_web_router` and `create_router` - Keep `Option<AiClient>` in `AppState` but always `Some(AiClient::from_env())` - Update handler code that unwraps `ai_client` (grep for `.ai_client`) Dependencies: Step 3 #### Step 5 — Verify build and fix compile errors - Run `cargo build -p hero_biz_admin` - Fix remaining errors - Run `cargo test -p hero_biz_admin` Dependencies: Step 4 ### Acceptance Criteria - [ ] `cargo build -p hero_biz_admin` succeeds - [ ] `AiProvider`, `AiConfig`, `AiModel` no longer exist in hero_biz - [ ] `reqwest` removed from `hero_biz_admin/Cargo.toml` - [ ] All AI calls route to `unix://.../hero_aibroker/rest.sock` - [ ] No hardcoded `base_url` or `api_key` remains - [ ] `create_web_router` no longer accepts `ai_config` - [ ] `cargo test -p hero_biz_admin` passes ### Notes - `ModelPurpose` is kept in `config.rs` for API compatibility with `assistant.rs`; the implementation ignores it and sends `model: "auto"` to the broker - STT multipart is written manually over `UnixStream` (same raw-socket approach as chat) - TTS reads raw bytes response rather than JSON - The broker handles all routing, fallback, and model selection internally
casper-stevens commented 2026-05-11 09:11:35 +00:00
Author
Member
Copy link

Test Results

  • Total: 7
  • Passed: 7
  • Failed: 0
cargo test -p hero_biz_admin

running 7 tests
test services::tests::empty_string_is_rejected ... ok
test services::tests::null_byte_is_rejected ... ok
test services::tests::dot_dot_is_rejected ... ok
test services::tests::path_traversal_sequences_rejected ... ok
test parser::tests::test_name_fix ... ok
test services::tests::slash_is_rejected ... ok
test services::tests::valid_components_pass ... ok

test result: ok. 7 passed; 0 failed; 0 ignored
## Test Results - Total: 7 - Passed: 7 - Failed: 0 ``` cargo test -p hero_biz_admin running 7 tests test services::tests::empty_string_is_rejected ... ok test services::tests::null_byte_is_rejected ... ok test services::tests::dot_dot_is_rejected ... ok test services::tests::path_traversal_sequences_rejected ... ok test parser::tests::test_name_fix ... ok test services::tests::slash_is_rejected ... ok test services::tests::valid_components_pass ... ok test result: ok. 7 passed; 0 failed; 0 ignored ```
casper-stevens commented 2026-05-11 09:11:52 +00:00
Author
Member
Copy link

Implementation Complete

Changes made

crates/hero_biz_admin/src/ai/client.rs — Rewritten from scratch. Removed all reqwest-based direct provider calls (try_chat, try_transcribe, try_tts, fallback loops, API key handling). New AiClient holds only a socket_path: PathBuf resolved from HERO_SOCKET_DIR. Communicates with the broker via raw tokio::net::UnixStream HTTP/1.1 (same pattern as hero_aibroker_sdk's StreamingClient). Kept public types ChatMessage, MessageRole, TranscriptionRequest unchanged — callers need no updates.

crates/hero_biz_admin/src/ai/config.rs — Removed AiProvider, AiConfig, AiModel. Kept only ModelPurpose enum (still passed to AiClient::chat() for API compatibility; ignored by the broker since it auto-routes).

crates/hero_biz_admin/src/ai/mod.rs — Narrowed config re-export to ModelPurpose only (was pub use config::*).

crates/hero_biz_admin/src/web/server.rs — create_router no longer accepts ai_config: Option<AiConfig>. Constructs AiClient::from_env() unconditionally at startup. ai_configured is always true.

crates/hero_biz_admin/src/lib.rs — create_web_router signature simplified: removed ai_config: Option<AiConfig> parameter.

crates/hero_biz_admin/src/main.rs — Updated call site to match new signature.

crates/hero_biz_admin/Cargo.toml — Removed reqwest from both [dependencies] and [dev-dependencies].

Broker routing

All AI calls now route through $HERO_SOCKET_DIR/hero_aibroker/rest.sock:

  • Chat: POST /v1/chat/completions with model: "auto"
  • STT: POST /v1/audio/transcriptions (multipart/form-data over Unix socket)
  • TTS: POST /v1/audio/speech

Test results

cargo test -p hero_biz_admin: 7 passed, 0 failed.

## Implementation Complete ### Changes made **`crates/hero_biz_admin/src/ai/client.rs`** — Rewritten from scratch. Removed all `reqwest`-based direct provider calls (`try_chat`, `try_transcribe`, `try_tts`, fallback loops, API key handling). New `AiClient` holds only a `socket_path: PathBuf` resolved from `HERO_SOCKET_DIR`. Communicates with the broker via raw `tokio::net::UnixStream` HTTP/1.1 (same pattern as `hero_aibroker_sdk`'s `StreamingClient`). Kept public types `ChatMessage`, `MessageRole`, `TranscriptionRequest` unchanged — callers need no updates. **`crates/hero_biz_admin/src/ai/config.rs`** — Removed `AiProvider`, `AiConfig`, `AiModel`. Kept only `ModelPurpose` enum (still passed to `AiClient::chat()` for API compatibility; ignored by the broker since it auto-routes). **`crates/hero_biz_admin/src/ai/mod.rs`** — Narrowed `config` re-export to `ModelPurpose` only (was `pub use config::*`). **`crates/hero_biz_admin/src/web/server.rs`** — `create_router` no longer accepts `ai_config: Option<AiConfig>`. Constructs `AiClient::from_env()` unconditionally at startup. `ai_configured` is always `true`. **`crates/hero_biz_admin/src/lib.rs`** — `create_web_router` signature simplified: removed `ai_config: Option<AiConfig>` parameter. **`crates/hero_biz_admin/src/main.rs`** — Updated call site to match new signature. **`crates/hero_biz_admin/Cargo.toml`** — Removed `reqwest` from both `[dependencies]` and `[dev-dependencies]`. ### Broker routing All AI calls now route through `$HERO_SOCKET_DIR/hero_aibroker/rest.sock`: - Chat: `POST /v1/chat/completions` with `model: "auto"` - STT: `POST /v1/audio/transcriptions` (multipart/form-data over Unix socket) - TTS: `POST /v1/audio/speech` ### Test results `cargo test -p hero_biz_admin`: 7 passed, 0 failed.
casper-stevens commented 2026-05-11 10:11:19 +00:00
Author
Member
Copy link

Implemented in commit 387341c. AiClient now routes all calls (chat, STT, TTS, morph) through the local hero_aibroker REST socket at $HERO_SOCKET_DIR/hero_aibroker/rest.sock. Direct provider calls removed.

Implemented in commit `387341c`. `AiClient` now routes all calls (chat, STT, TTS, morph) through the local `hero_aibroker` REST socket at `$HERO_SOCKET_DIR/hero_aibroker/rest.sock`. Direct provider calls removed.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
development
development_casper
main
v0.5.0-rc1
v0.1.3-rc4
v0.1.3-rc3
v0.1.3-rc2
development_mik_biz_from_ci
v0.1.3-rc1
v0.1.2
v0.1.1
v0.1.0
v0.0.1-rc1
Labels
Clear labels   
prio_critical

   
prio_low

   
type_bug

   
type_contact

   
type_issue

   
type_lead

   
type_question

   
type_story

   
type_task
No labels
prio_critical
prio_low
type_bug
type_contact
type_issue
type_lead
type_question
type_story
type_task
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
ACTIVE
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
lhumina_code/hero_biz#15
No description provided.