[vision] Hero OS as an ambient AI desktop — sovereign, voice-native, multi-context #52
mik-tf
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
§0 Current state
Last session: 121 (2026-05-19) — T2
hero_lib_rhaiD-10 closure: 5-binary wholesale sweep (gpu_agent + hero_do + hero_runner_rhai CLI + _server + _admin) + lesson #19 wired in CLI orchestrator + lesson #21 shape 1 trivial collapse onproc_log.rs(HeroLogger fully gone at hero_proc tip — anticipated shape 3 did NOT materialize) + adjacentproc_rhaicascade unblock + 9 zero-match dep strips. 30/35 effective clean (~86%).s121
hero_lib_rhai(squashb4d138adirect to origin/development; no PR): 20 files +427/-385 — 5-binary wholesale across the 5 service-shaped crates (the 13 lib-only*_rhaicrates have no main.rs and infocheck ignores them). 5 service.toml at crate roots all ship[[env]] PATH_ROOT default="~/hero".service_base!()triad on all 5 main.rs + deleted ~280 LOC hand-rolledprint_info_json/print_startup_info/print_help. Lesson #19 wired in hero_runner_rhai/src/main.rs —forward_env_if_setthreads PATH_ROOT/PATH_VAR/PATH_BUILD/PATH_CODE/HERO_SOCKET_DIR into both spawned ActionSpecs; verified via /proc/$pid/environ on 8 CLI-spawned PIDs. Lesson #21 anticipation INVERTED: §3 predicted shape 3 (function-param threading) at proc_log.rs:48,55,99; probe-at-start found HeroLogger fully removed at hero_proc tip 32af77cd. Resolution: shape 1 trivial collapse — spawn_forwarder signature preserved (SessionManager / OpenRPC / SDK contracts intact), body collapses to warn-once + drop broadcast subscription. forward_logs flag inert pending logs.insert RPC re-migration. Adjacent proc_rhai cascade-absorb (pre-existing latent unblocked the workspace test build):JobLogsInput +attempt: Option<i64>,LogLine.lineplain String,ServiceSpec +probe +require_ready +sockets +tags. Service.toml authoring quirks discovered:categoryrestricted enum (core/storage/ai/network/ui/tool —"rhai"/"gpu"panic at validate);protocolflips smoke coverage (http=2/openrpc=4 tests per socket);[[binaries.tcp]]shape isaddress + port + purpose. D-10 5/5: infocheck 5 crates clean / 0 findings, smoke 6/6 (server rpc.sock × {health, openrpc.json, well-known/heroservice.json, /rpc system.ping} = 4/4 + admin admin.sock × {health, well-known/heroservice.json} = 2/2),cargo test --workspace --release102 passed / 0 failed / 5 ignored.s120
hero_agent(squasha141d48direct to origin/development; no PR): wholesale 3-binary shape (cli + server + admin) — same template as s109/s111/s115/s119. 3 service.toml + service_base!() triad on 3 main.rs +.well-known/heroservice.jsonadded to admin. Lesson #19 verified in /proc/$pid/environ viahero_agent --startCLI path. Last AI-rename ripple absorbed — single-consumer transcribe-surface recipe atcrates/hero_agent_server/src/routes.rs voice_transcribe. 0 indirect callers (Agent struct holds LlmClient not AiClient — confirmed by Explore probe). D-10 5/5: infocheck 3 clean / 0 findings, smoke 4/4, cargo test workspace 0 FAILED. Runtime probe: POST /api/voice/transcribe → HTTP 401 expired_api_key (validates wire chain). Filed hero_skills#263.s119
hero_voice(squashf8503b3): wholesale 4-binary sweep + first end-to-end transcribe/TTS validation of herolib_ai v0.6.0 + src/bin → src/main restructure (drops lesson #20 inline workaround).s118
hero_researcher(squashee3431bdirect to origin/development; no PR): wholesale shape; first single-process two-socket binaries layout. First non-dead-code Lesson #21 4th-shape application — migratedchat_sync(5 hot-path callers) via sync-wrapper-preservation pattern. Pairedweb.sock → admin.sockrename.s117
hero_webbuilder(squash5f726afdirect to origin/development; no PR): 4-binary wholesale. First Lesson #21 4th-shape application — full herolib_ai v0.6.0 API rewrite on JobManager (0 workspace callers, runtime-verified). Wrotememory/investigation_herolib_ai_migration.mdplaybook.Next: s122 =
hero_os(next wholesale; OS-WASM surface; full session minimum; high effort tier — ~25-min cold WASM compile viamake buildpipeline). Alternate:hero_osis(per-domain socket complexity, 17 sockets).What Hero OS becomes
Hero OS is a sovereign personal operating system that runs in your browser, owns its own data, and is operated by an always-on AI assistant that understands your workspace as well as you do. The interface isn't an app launcher — it's a conversation.
This issue is the long-form vision. Every other issue in the program tree (hero_demo, hero_agent, hero_books, hero_proc, hero_osis, …) implements one piece of this picture. The vision is what we're optimising for; the issues are how we get there.
The product story (what a user lives in)
You wake up, open the browser, and Hero OS is your second brain. You don't navigate menus. You say:
A small widget in the top right wakes up, listens, transcribes, queries
hero_osis_calendarfor the current context, summarises:You say "yes." It pulls the doc from
hero_foundry, embeds it viahero_embedder, summarises throughhero_aibroker. You hear the answer, no clicks.You say:
It switches context, calls
business.contact.createvia curated MCP tools, confirms:You say:
It searches
hero_books+hero_foundryfor documents tagged Alex, ranks viahero_indexer, opens the most likely match in the Slides island.That's the vision. The whole OS is reachable by voice. The AI is the operating layer. Every island is also a tool the AI can invoke.
Why this is achievable now (the stack we already have)
The pieces are mostly built — what's needed is wiring + content + polish, not new features.
What's verified live as of 2026-04-30:
So we're closer to the vision than this issue makes it sound. The gap is polish + ambient access + wiring.
The four enabling moves (everything else flows from these)
1. AI Assistant becomes ambient (OS-wide, always-on)
Today: AI Assistant lives in its own island. You have to open it.
Vision:
hero_routepostMessage protocol the dock already uses.The wake-word machinery is built; it's the always-on widget + conversation-mode UI + OS-routing intent layer that's missing. Tracked in hero_agent issue (filed alongside this one).
2. Curated MCP tool surface (instead of socket-path brute force)
Today (verified live in screenshots): when asked "who are my contacts in hero_biz?", the AI Assistant emitted ~30 lines of "Let me try this... let me check the socket path... let me search for the RPC method..." before eventually returning the answer. It got there, but it brute-forced through service paths.
Vision:
X-Hero-Context: geomindwithout the AI having to think about it.Tracked in hero_agent issue (filed alongside this one).
3. docs_hero as the agent's brain
Today: docs_hero exists, the AI grounds on it, the answers are good when the docs cover the question. When they don't, the AI falls back to brute force.
Vision:
This is the multiplier. Investing in docs_hero makes the AI feel measurably smarter without changing a line of agent code. Tracked in docs_hero issue (filed alongside this one).
4. Reliability so the demo doesn't fall over
Today: supervisor reports green when reality is broken; service restarts can knock children into half-broken states; logs can OOM the supervisor. Filed in hero_proc#86 META + sister issues.
Vision:
Tracked in lhumina_code/hero_proc#86 (META) and sister issues (#83/#84/#85, home#201/#202/#204).
What v1 of Hero OS looks like
A user can:
v1 acceptance: that user-flow works end-to-end on a fresh deploy with no manual interventions, no env-var workarounds, no half-broken state mid-flow.
Three-track roadmap
Cross-references — every implementing issue
Demo capabilities:
Reliability:
Architecture:
Known blockers:
Signed-off-by: mik-tf
Capabilities the vision under-pitches (session 52 source-grounded read)
After reading the 13 CORE repos for docs_hero Phase 1, five capabilities the demo ALREADY ships are not headlined in the vision body:
Plug Claude Code into any Hero service via
claude mcp add --transport http <slug> <endpoint>. Every healthy Hero service auto-becomes an MCP server — hero_router exposesPOST /mcp/{service_name}and auto-derives tools from each service's OpenRPC spec (hero_router/.../mcp.rs:30, :591). The sovereignty + composability story this enables — "your AI tools, your hardware, plug in any MCP-speaking client" — isn't in the vision today.Every service has a self-driving Python agent built-in.
agent_runis auto-appended as a free MCP tool per service (mcp.rs:132-158) — natural-language description → generated Python → executed against the live service with retry. This is a top-tier "ambient AI" feature.Auto-generated Python clients per service via
router.python_client/router.python_interface. DX story: spin up a typed client for any Hero service from its OpenRPC.Embedded PTY terminal sessions deep-linked from the browser via
terminal.create(supports nu/bash/tmux). Power-user / admin-grade capability that many "AI desktop" pitches don't have.Inline AI agent pipeline —
router.agent.run/.start/.play/.modelsrunning hero_logic'sservice_agent_v3workflow directly from the router's own surface.Suggested: update the vision body to headline (1) + (2) explicitly. (3)–(5) are demo-amplifiers worth mentioning in a "developer experience" section.
The auto-MCP gateway also has implications for hero_agent#15 — see comment there for the rescope.
Reconciliation memo:
memory/investigation_roadmap_reconciliation.md(private workspace, session 52).Session 52 — 2026-05-01 status update
Tier A landed on
docs_hero/developmentat commit6ed62f69— 10 source-grounded service pages (1891 LOC). Session then pivoted into a multi-hour herodemo VM redeploy stabilization. Demo verified working end-to-end at session close.Demo state (verified live, 2026-05-01 23:35 UTC):
Open work surfaced:
load_init_shdoesn't followsourcedirectives (env.sh exports silently lost)agent.rs:853on multi-byte chars in tool resultsFull session manifest:
sessions/52.ymlin the workspace.Signed-off-by: mik-tf
Session 62 closed (2026-05-06) — verification + drift cleanup.
Outcome: confirmed CEO restructure (
tools/→nutools/,--from-ci→--download) preserved sessions 53–61's work verbatim. Re-validated hero_voice end-to-end on heroci againsthero_skills @ 1b721d8(now908c775after the dispatcher fix below) —service_voice install --download --resetfetched all 4 v0.1.0-rc2 assets cleanly,libonnxruntime.so.1.25.1exports the expectedOrtGetApiBase@@VERS_1.25.1symbol, binary boots underORT_DYLIB_PATH. Coverage stable at 16/15 (15 original Phase 2 set + editor + voice). hero_embedder remains the last D-05 service.Shipped:
hero_voice.tomlcomment rename--from-ci→--download.fix(dispatcher): forward --download/--version to 9 service install aliases. Closes the alias-form gap surfaced this session:service voice install --download(alias) was silently degrading to a cargo build because dispatcher dropped the flag. 9 install entries fixed (biz, books, browser, editor, foundry, matrixchat, slides, voice, whiteboard) plus voice start. embedder intentionally left for session 63.--downloadreality.Not shipped (intentional):
svx_install_onnx_runtime_downloadintolib.nu).See
sessions/62.ymlfor the full manifest.Session 71 close (2026-05-07): Bucket D item 1 —
hero_computev0.1.8 sentinel rename + nutools--downloadwiring. Coverage reframed: 25/27 in-scope (93%) + 4 Tier D deferred (mail/sync/tfspores/ledger). Strategic pivot to ecosystem-on-heroci trio.Producer: lhumina_code/hero_compute#105
d710d48(matrixartifact:flip).Consumer: lhumina_code/hero_skills#227
10493a9(service_compute.nu+ dispatcher gap).8 Forgejo PATCH renames on v0.1.8 in-place (no rebuild). Heroci smoke: 4 musl static-pie binaries land green;
hero_compute --version → 0.1.8.Hygiene: cleaned
hero_logicv0.1.0-rc1 dup assets (6→3, carried over from session 70).End-of-session archipelagos recon (highest-leverage): library-only hypothesis disproved. archipelagos ships 1 native binary (
archipelagos_server, port 8886) + 24 WASM islands packaged ashero_archipelagos-wasm.tar.gzand uploaded to Forgejo'''s Generic Package registry (which bypasses Releases — that'''s why "0 releases" on the API). Session 72 path: ADD native-binary publish alongside existing WASM flow.Pinned next: session 72 =
hero_archipelagosfirst release (native side; WASM stays as today). Then session 73+74 =hero_os(mixed-artifact, planned 1.5 sessions). After: ecosystem-on-heroci unblocked.Session 77 complete — workspace-wide
_ui → _adminconvention rollout closed in one sweepScope expansion mid-session per user direction: from "hero_proc + finalize hero_db" (s76 §3 plan) to "all 26 service producers so we can build it all".
Outcome
_web), hero_livekit_web— correct per skill), hero_matrixchat, hero_osisbuild-linux.yamlCI.hero_skills/service_*.nu) already aligned per pre-sweep audit; no consumer PRs needed.hero_foundry_uirepo-rename decision (deferred).Sweep correctness gap (s78 follow-up)
The sweep defaulted single-
_uirepos to_admin. Perhero_service_testskill §2,_adminis admin Askama UI and_webis end-user-facing. hero_collab was caught by upstream + corrected to_web. Sister audit candidates: hero_office, hero_biz, hero_browser, hero_editor — flagged for per-repo classification in s78.Sweep-mechanics bugs caught + filed for posterity
grep -Zdoes not mean--nullon this GNU grep build (silently emits newline-separated). Usegrep -rl --nullor-l0.set -e + pipefailtrips when grep finds zero matches (rc=1). Wrap with{ grep ... || true; }so xargs -r handles empty input cleanly.Standing instruction
Never merge lhumina_code/hero_router#92 (saved as
feedback_never_merge_hero_router_pr92.md).Next session (s78)
hero.threefold.store— target 17/17 services starting from CI artifacts with zero hand-patches (s75 had 14/17; the s77 sweep should close the cascade)._adminvs_webcorrectness audit on s77 sweep set.hero_foundry_uifate.packages.nustarts_extra(one-line).META home#228 is now substantively closed pending CI green + DO re-validation.
Session manifest:
sessions/77.yml.Signed-off-by: mik-tf
Session 92 (2026-05-11) — closed hero_slides#60.
hero_slides PR #61
f34f18a(3 files, +112/-3):slide.getIncomingLinks— new RPC handler walks the collection root scanningmetadata.toml.slides[*].source_linkwith canonical-path equality. Closes the ~11x-per-dashboard-loadMethod not foundserver log spam since PR #55 added the "used as link source" badge.bg.listFolders— returns{folders: [], root_files: []}for absent backgrounds dirs instead of-32000. "No backgrounds" is the steady state for new decks.Verified locally:
python3 scripts/smoke_openrpc.py— 91 methods, METHOD_MISSING 0, no new dispatch regressions.hero_slides_intro/07_s92_link_probe→sample_deck/01_introreturned the linker; probe cleaned up.{incoming_links: []}.Latent finding (documented, not fixed):
legacy_param_shimsynthesizespath = <deck_path>/backgroundsfor allbg.*methods (rpc.rs:399), but every lib helper expects to receive a deck root and internally appendscontent/background/. So allbg.*mutation paths (createFolder, uploadFile, deleteFile, moveFile, ...) silently write into a non-existent<deck_path>/backgrounds/content/background/in steady state. Not user-visible if the dashboard's background-folder feature isn't actively used; user-visible the moment someone tries to upload a logo via that path. Candidate for next session if not Rule 6 redeploy.Pinned next: Rule 6 UX gate (herodemo.gent01 redeploy now 5+ sessions deferred) | fix bg.* shim path | re-engage #58 if multimodal still broken on boss's end.
Signed-off-by: mik-tf
Session 94 — setup complete
Setup-only session for the hero_proc#102 workspace
service.toml+lab servicecleanup sweep that boss filed today.Zero code commits. Pipeline state shipped:
decisions/D-10-service-toml-sweep-scope-order-acceptance.mdso the multi-session loop resumes consistently across/startboundaries.lhumina_code/hero_skillsfast-forwarded73762cb → be20458(12 commits today). All commits map to sections of #102's body — the issue is the writeup of code boss pushed 03:57–15:46 UTC.prompt.md §3rewired for the multi-session loop with a per-repo effort table: bulk =medium, foundational/adversarial =high, noxhighunless boss-WIP collision forces it. Rationale: mechanicallab infocheckfix loops don't benefit from xhigh-grade reasoning; cost arithmetic favors per-tier sizing.STT skill-name reconciliation (Issue body uses friendly notation from voice recording → real artifacts):
service.infohero_service_infoSKILLservice.checklab infocheckservice.fixhero_tests_fix_errorsSKILL +lab infocheckfix-loopResolved side-fire: hero_skills#245 — dispatcher.nu booby-trap resolved by completion at
be20458. 47service_*.nu+dispatcher.nuarchived under_archive/nutools/modules/services/.lab servicereplaces nu dispatch. Runtime clone pull is now safe (queued for s95 prep).One open question for boss when awake: "Singleton services in hero_proc" mentioned in #102 §"Important Proc Updates" has no visible commit in today's hero_proc 3-commit window (
4b2afbctags /4495a09run.submit concurrency /e7f1d2aRunBuilder SDK). Either pre-existing, or local-WIP. Not blocking — sweep proceeds; will note if a repo's smoke checks depend on singleton behavior.Estimate: 10–15 sessions to close 35/35. Most likely 12. Wall-clock 3–5 days continuous / 1–2 weeks intermittent given per-PR squash-merge human gate.
Next session (s95): prep (re-mirror
~/.claude/skills/, FF runtime clone, rebuildlab, re-read updatedhero_service_info) + Tier 1 / 1 of 5:hero_procsweep. Effort: high.— Signed-off-by: mik-tf