[vision] Hero OS as an ambient AI desktop — sovereign, voice-native, multi-context #52
mik-tf
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
§0 Current state
Last session: 140 (2026-05-21) — Track D s140 methodology-only session: D2 Forge user lifecycle decision-locking; D-22 BYO landing locked end-to-end; zero code commits — Triggered by Phase B.5 catching two load-bearing gaps in the initial "D2 code start" plan: (a)
herolib_tools::forge::ForgeClient::connect()hard-readscore/FORGE_TOKEN(D-16 conflation footgun), (b) D2 Q2 "can admin mint on behalf?" unresolved. Live API probe answered Q2 definitively against forge.ourworld.tf:POST /api/v1/users/<u>/tokensrefuses bearer-token auth on both admin + Sudo paths (HTTP 401 "auth method not allowed"); basic-auth-with-deployer-known-password bridge works (HTTP 201) but rejected on security grounds (deployer would become long-lived impersonation-credential vault; repudiation breaks; token outlives password change). Throwaway test user purged. Spec re-read showed 4 canonical sources converging on BYO: #1 §3.4 "where needed" + #1 §4 "required" + #1 §7 cockpit "Set or update: Forge token" UI feature + #2 step 1 "admin captures the password to share OOB". D-22 locked: deployer creates Forge account viaPOST /api/v1/admin/userswithmust_change_password=true, hands password OOB; user mints own scopedwrite:issuetoken via Forge UI in same forge.ourworld.tf session they're in from OAuth-gated VM front door (~30s detour); paste into cockpit Settings lands atcockpit/USER_FORGE_TOKENper D-16; deployer admin token namespaced atdeployer/FORGE_TOKENextending D-16's three-actor isolation matrix. Posted hero_os_tfgrid_deployer#4#35866 closing the spec hedge + droppinggenerate_token_forfrom D2 scope. Filed hero_cockpit#2 — Option 3 OAuth-session-reuse polish (zero-setup if hero_proxy passes OAuth access token to backend; forward-compatible with D-22). Wrotememory/investigation_forge_admin_token_mint_2026_05_21.mddocumenting probe + basic-auth bridge as known-but-rejected escape hatch. Perfeedback_phase_b_5_finding_is_slow_down_signal.md, code work cleanly deferred to s141. s141 next: (1) upstreamForgeClient::connect_with_secret(context, key)inherolib_tools(standalone squash onhero_lib); (2) deployer-sideforge.rs+ 3 OpenRPC methods (deployer.create_user/get_user/list_users) + sqlite users table. No new D-NN expected. Effort tier: high. Full narrative: sessions/140.yml.deployer-forge-token-namespacing-and-byo-landing; D-NN next-free bumped D-22 → D-23.Previous: s139 (2026-05-21) — hero_cockpit A7 closes Track A; herolab destroyed; deployer becomes canonical TFGrid deploy path; Track D critical-path: hero_cockpit
f880247—cockpit.expose/unexpose/list_exposures+get/set_base_domain(13→18 OpenRPC methods); newexposures.rs. Track A closed (A1-A7). Post-session pivot:herolab.gent02.grid.tfdestroyed viamake destroy ENV=herolab. D-20 SendGrid locked.Demo-deployer arc (s133-onward): Track A (cockpit) CLOSED at s139 (A1-A7). Track D (deployer) CRITICAL-PATH s140-s144: s140 = decision-lock (this session), s141 = code start, s142-s144 = D3-D5. Tracks B/C local-code parallel; Track E feeds D4; Track F gated on all. Full 6-track A-F roadmap in home#235.
What Hero OS becomes
Hero OS is a sovereign personal operating system that runs in your browser, owns its own data, and is operated by an always-on AI assistant that understands your workspace as well as you do. The interface isn't an app launcher — it's a conversation.
This issue is the long-form vision. Every other issue in the program tree (hero_demo, hero_agent, hero_books, hero_proc, hero_osis, …) implements one piece of this picture. The vision is what we're optimising for; the issues are how we get there.
The product story (what a user lives in)
You wake up, open the browser, and Hero OS is your second brain. You don't navigate menus. You say:
A small widget in the top right wakes up, listens, transcribes, queries
hero_osis_calendarfor the current context, summarises:You say "yes." It pulls the doc from
hero_foundry, embeds it viahero_embedder, summarises throughhero_aibroker. You hear the answer, no clicks.You say:
It switches context, calls
business.contact.createvia curated MCP tools, confirms:You say:
It searches
hero_books+hero_foundryfor documents tagged Alex, ranks viahero_indexer, opens the most likely match in the Slides island.That's the vision. The whole OS is reachable by voice. The AI is the operating layer. Every island is also a tool the AI can invoke.
Why this is achievable now (the stack we already have)
The pieces are mostly built — what's needed is wiring + content + polish, not new features.
What's verified live as of 2026-04-30:
So we're closer to the vision than this issue makes it sound. The gap is polish + ambient access + wiring.
The four enabling moves (everything else flows from these)
1. AI Assistant becomes ambient (OS-wide, always-on)
Today: AI Assistant lives in its own island. You have to open it.
Vision:
hero_routepostMessage protocol the dock already uses.The wake-word machinery is built; it's the always-on widget + conversation-mode UI + OS-routing intent layer that's missing. Tracked in hero_agent issue (filed alongside this one).
2. Curated MCP tool surface (instead of socket-path brute force)
Today (verified live in screenshots): when asked "who are my contacts in hero_biz?", the AI Assistant emitted ~30 lines of "Let me try this... let me check the socket path... let me search for the RPC method..." before eventually returning the answer. It got there, but it brute-forced through service paths.
Vision:
X-Hero-Context: geomindwithout the AI having to think about it.Tracked in hero_agent issue (filed alongside this one).
3. docs_hero as the agent's brain
Today: docs_hero exists, the AI grounds on it, the answers are good when the docs cover the question. When they don't, the AI falls back to brute force.
Vision:
This is the multiplier. Investing in docs_hero makes the AI feel measurably smarter without changing a line of agent code. Tracked in docs_hero issue (filed alongside this one).
4. Reliability so the demo doesn't fall over
Today: supervisor reports green when reality is broken; service restarts can knock children into half-broken states; logs can OOM the supervisor. Filed in hero_proc#86 META + sister issues.
Vision:
Tracked in lhumina_code/hero_proc#86 (META) and sister issues (#83/#84/#85, home#201/#202/#204).
What v1 of Hero OS looks like
A user can:
v1 acceptance: that user-flow works end-to-end on a fresh deploy with no manual interventions, no env-var workarounds, no half-broken state mid-flow.
Three-track roadmap
Cross-references — every implementing issue
Demo capabilities:
Reliability:
Architecture:
Known blockers:
Signed-off-by: mik-tf
Capabilities the vision under-pitches (session 52 source-grounded read)
After reading the 13 CORE repos for docs_hero Phase 1, five capabilities the demo ALREADY ships are not headlined in the vision body:
Plug Claude Code into any Hero service via
claude mcp add --transport http <slug> <endpoint>. Every healthy Hero service auto-becomes an MCP server — hero_router exposesPOST /mcp/{service_name}and auto-derives tools from each service's OpenRPC spec (hero_router/.../mcp.rs:30, :591). The sovereignty + composability story this enables — "your AI tools, your hardware, plug in any MCP-speaking client" — isn't in the vision today.Every service has a self-driving Python agent built-in.
agent_runis auto-appended as a free MCP tool per service (mcp.rs:132-158) — natural-language description → generated Python → executed against the live service with retry. This is a top-tier "ambient AI" feature.Auto-generated Python clients per service via
router.python_client/router.python_interface. DX story: spin up a typed client for any Hero service from its OpenRPC.Embedded PTY terminal sessions deep-linked from the browser via
terminal.create(supports nu/bash/tmux). Power-user / admin-grade capability that many "AI desktop" pitches don't have.Inline AI agent pipeline —
router.agent.run/.start/.play/.modelsrunning hero_logic'sservice_agent_v3workflow directly from the router's own surface.Suggested: update the vision body to headline (1) + (2) explicitly. (3)–(5) are demo-amplifiers worth mentioning in a "developer experience" section.
The auto-MCP gateway also has implications for hero_agent#15 — see comment there for the rescope.
Reconciliation memo:
memory/investigation_roadmap_reconciliation.md(private workspace, session 52).Session 52 — 2026-05-01 status update
Tier A landed on
docs_hero/developmentat commit6ed62f69— 10 source-grounded service pages (1891 LOC). Session then pivoted into a multi-hour herodemo VM redeploy stabilization. Demo verified working end-to-end at session close.Demo state (verified live, 2026-05-01 23:35 UTC):
Open work surfaced:
load_init_shdoesn't followsourcedirectives (env.sh exports silently lost)agent.rs:853on multi-byte chars in tool resultsFull session manifest:
sessions/52.ymlin the workspace.Signed-off-by: mik-tf
Session 62 closed (2026-05-06) — verification + drift cleanup.
Outcome: confirmed CEO restructure (
tools/→nutools/,--from-ci→--download) preserved sessions 53–61's work verbatim. Re-validated hero_voice end-to-end on heroci againsthero_skills @ 1b721d8(now908c775after the dispatcher fix below) —service_voice install --download --resetfetched all 4 v0.1.0-rc2 assets cleanly,libonnxruntime.so.1.25.1exports the expectedOrtGetApiBase@@VERS_1.25.1symbol, binary boots underORT_DYLIB_PATH. Coverage stable at 16/15 (15 original Phase 2 set + editor + voice). hero_embedder remains the last D-05 service.Shipped:
hero_voice.tomlcomment rename--from-ci→--download.fix(dispatcher): forward --download/--version to 9 service install aliases. Closes the alias-form gap surfaced this session:service voice install --download(alias) was silently degrading to a cargo build because dispatcher dropped the flag. 9 install entries fixed (biz, books, browser, editor, foundry, matrixchat, slides, voice, whiteboard) plus voice start. embedder intentionally left for session 63.--downloadreality.Not shipped (intentional):
svx_install_onnx_runtime_downloadintolib.nu).See
sessions/62.ymlfor the full manifest.Session 71 close (2026-05-07): Bucket D item 1 —
hero_computev0.1.8 sentinel rename + nutools--downloadwiring. Coverage reframed: 25/27 in-scope (93%) + 4 Tier D deferred (mail/sync/tfspores/ledger). Strategic pivot to ecosystem-on-heroci trio.Producer: lhumina_code/hero_compute#105
d710d48(matrixartifact:flip).Consumer: lhumina_code/hero_skills#227
10493a9(service_compute.nu+ dispatcher gap).8 Forgejo PATCH renames on v0.1.8 in-place (no rebuild). Heroci smoke: 4 musl static-pie binaries land green;
hero_compute --version → 0.1.8.Hygiene: cleaned
hero_logicv0.1.0-rc1 dup assets (6→3, carried over from session 70).End-of-session archipelagos recon (highest-leverage): library-only hypothesis disproved. archipelagos ships 1 native binary (
archipelagos_server, port 8886) + 24 WASM islands packaged ashero_archipelagos-wasm.tar.gzand uploaded to Forgejo'''s Generic Package registry (which bypasses Releases — that'''s why "0 releases" on the API). Session 72 path: ADD native-binary publish alongside existing WASM flow.Pinned next: session 72 =
hero_archipelagosfirst release (native side; WASM stays as today). Then session 73+74 =hero_os(mixed-artifact, planned 1.5 sessions). After: ecosystem-on-heroci unblocked.Session 77 complete — workspace-wide
_ui → _adminconvention rollout closed in one sweepScope expansion mid-session per user direction: from "hero_proc + finalize hero_db" (s76 §3 plan) to "all 26 service producers so we can build it all".
Outcome
_web), hero_livekit_web— correct per skill), hero_matrixchat, hero_osisbuild-linux.yamlCI.hero_skills/service_*.nu) already aligned per pre-sweep audit; no consumer PRs needed.hero_foundry_uirepo-rename decision (deferred).Sweep correctness gap (s78 follow-up)
The sweep defaulted single-
_uirepos to_admin. Perhero_service_testskill §2,_adminis admin Askama UI and_webis end-user-facing. hero_collab was caught by upstream + corrected to_web. Sister audit candidates: hero_office, hero_biz, hero_browser, hero_editor — flagged for per-repo classification in s78.Sweep-mechanics bugs caught + filed for posterity
grep -Zdoes not mean--nullon this GNU grep build (silently emits newline-separated). Usegrep -rl --nullor-l0.set -e + pipefailtrips when grep finds zero matches (rc=1). Wrap with{ grep ... || true; }so xargs -r handles empty input cleanly.Standing instruction
Never merge lhumina_code/hero_router#92 (saved as
feedback_never_merge_hero_router_pr92.md).Next session (s78)
hero.threefold.store— target 17/17 services starting from CI artifacts with zero hand-patches (s75 had 14/17; the s77 sweep should close the cascade)._adminvs_webcorrectness audit on s77 sweep set.hero_foundry_uifate.packages.nustarts_extra(one-line).META home#228 is now substantively closed pending CI green + DO re-validation.
Session manifest:
sessions/77.yml.Signed-off-by: mik-tf
Session 92 (2026-05-11) — closed hero_slides#60.
hero_slides PR #61
f34f18a(3 files, +112/-3):slide.getIncomingLinks— new RPC handler walks the collection root scanningmetadata.toml.slides[*].source_linkwith canonical-path equality. Closes the ~11x-per-dashboard-loadMethod not foundserver log spam since PR #55 added the "used as link source" badge.bg.listFolders— returns{folders: [], root_files: []}for absent backgrounds dirs instead of-32000. "No backgrounds" is the steady state for new decks.Verified locally:
python3 scripts/smoke_openrpc.py— 91 methods, METHOD_MISSING 0, no new dispatch regressions.hero_slides_intro/07_s92_link_probe→sample_deck/01_introreturned the linker; probe cleaned up.{incoming_links: []}.Latent finding (documented, not fixed):
legacy_param_shimsynthesizespath = <deck_path>/backgroundsfor allbg.*methods (rpc.rs:399), but every lib helper expects to receive a deck root and internally appendscontent/background/. So allbg.*mutation paths (createFolder, uploadFile, deleteFile, moveFile, ...) silently write into a non-existent<deck_path>/backgrounds/content/background/in steady state. Not user-visible if the dashboard's background-folder feature isn't actively used; user-visible the moment someone tries to upload a logo via that path. Candidate for next session if not Rule 6 redeploy.Pinned next: Rule 6 UX gate (herodemo.gent01 redeploy now 5+ sessions deferred) | fix bg.* shim path | re-engage #58 if multimodal still broken on boss's end.
Signed-off-by: mik-tf
Session 94 — setup complete
Setup-only session for the hero_proc#102 workspace
service.toml+lab servicecleanup sweep that boss filed today.Zero code commits. Pipeline state shipped:
decisions/D-10-service-toml-sweep-scope-order-acceptance.mdso the multi-session loop resumes consistently across/startboundaries.lhumina_code/hero_skillsfast-forwarded73762cb → be20458(12 commits today). All commits map to sections of #102's body — the issue is the writeup of code boss pushed 03:57–15:46 UTC.prompt.md §3rewired for the multi-session loop with a per-repo effort table: bulk =medium, foundational/adversarial =high, noxhighunless boss-WIP collision forces it. Rationale: mechanicallab infocheckfix loops don't benefit from xhigh-grade reasoning; cost arithmetic favors per-tier sizing.STT skill-name reconciliation (Issue body uses friendly notation from voice recording → real artifacts):
service.infohero_service_infoSKILLservice.checklab infocheckservice.fixhero_tests_fix_errorsSKILL +lab infocheckfix-loopResolved side-fire: hero_skills#245 — dispatcher.nu booby-trap resolved by completion at
be20458. 47service_*.nu+dispatcher.nuarchived under_archive/nutools/modules/services/.lab servicereplaces nu dispatch. Runtime clone pull is now safe (queued for s95 prep).One open question for boss when awake: "Singleton services in hero_proc" mentioned in #102 §"Important Proc Updates" has no visible commit in today's hero_proc 3-commit window (
4b2afbctags /4495a09run.submit concurrency /e7f1d2aRunBuilder SDK). Either pre-existing, or local-WIP. Not blocking — sweep proceeds; will note if a repo's smoke checks depend on singleton behavior.Estimate: 10–15 sessions to close 35/35. Most likely 12. Wall-clock 3–5 days continuous / 1–2 weeks intermittent given per-PR squash-merge human gate.
Next session (s95): prep (re-mirror
~/.claude/skills/, FF runtime clone, rebuildlab, re-read updatedhero_service_info) + Tier 1 / 1 of 5:hero_procsweep. Effort: high.— Signed-off-by: mik-tf
s127 — CI auto-publish wired (closes 5/6 of hero_skills#268 DoD)
Canonical
lab-publish.yamlworkflow now lives in 31 of 35 D-07 demo-set repos. 17 of those 31 auto-publish fresh musl-x86_64 binaries toreleases/tag/lateston every push todevelopment— the ecosystem-bootstrap path (VM doeslab build --downloadand gets the whole stack in ~5 min) works out-of-the-box for them.Shipped this session
crates/lab/install.sh(63e0ef7) — fixes the documented-but-404 install URL the lab skill has been advertising.forgejo/workflows/lab-publish.yaml(ad537ae+ 3 iterations →4767f67) — canonical workflow: Rust 1.95 install / musl-tools / lab install via install.sh /lab build --release --platforms linux-musl-x86_64 --upload --workspacescripts/wire-lab-publish.sh(7ca5874+ fixb9f7ae0) — idempotent templating script, rolled the workflow to 31 repos in 6 batches of 5crates/lab/src/builder/orchestrator.rs(777a7ec) —MIN_BINARY_BYTESmade env-overridable viaHERO_MIN_BIN_BYTES; CI sets it to 0 so legitimately-small Rust tools (mcp_*, *_admin shims) don't false-trip the < 512 KB "likely corrupt" guard. Unblocked hero_aibroker + hero_compute + hero_lib without touching their source.skills/lab/lab.md(d660a96) — drift fixlab-latest → latest(lab source has always usedlatest; skill docs were wrong from day one)lab build --uploadis the bypass caseOutcome
17 green / 14 red / 4 deferred = 17/35 actually shipping today. The 14 red are pre-existing per-repo bugs (path-deps, cargo feature errors, musl portability, etc.) — none are CI-workflow issues. The workflow file is in place, so each goes green automatically on the next push after its per-repo fix lands. The 4 deferred (hero_voice / hero_os / hero_office / mycelium_network) need workflow variants (gnu matrix-swap, WASM target, private auth, branch override).
Next session (s128) target: drive to 35/35 shipping. Two parallel work surfaces detailed in prompt.md §3 — Surface A clears the 14 per-repo bugs (recommend path-dep cluster of 6 repos as first head, mechanical one-pattern fix), Surface B wires the 4 special-case repos. Each fix unlocks one or more repos toward the ecosystem-bootstrap goal.
Sessions/127.yml has the full narrative + commit ledger across all 31 repos.
s128 complete — #269 Surface A swept
9 per-repo lab-publish unblock pushes squashed direct-to-development:
3bc4d07febfa4289a983a231f268e83336951c6551rustls-tls2f601c7with-axuma55decatarget_env-gatedRlimitResourcea4a9e5ddep_overridesrusqlite=0.31Followups filed:
6e39d677(138 markers / 46 regions across 10 files); needs despiegk + Casper to reconcile.executor.rs:54CWD bug on nested workspaces; blocks hero_rpc CI.Diagnoses corrected: hero_code already shipping pre-session; hero_books APIs exist + builds clean (infra CI failure); hero_rpc failure is lab orchestrator bug not per-repo bug.
Coverage projected post-CI: 25/31 wired (81%) / 25/35 D-07 (71%).
Tracker close-out: hero_skills#269#34413. Full narrative in
sessions/128.yml(local; not committed).s129 recommended head: land #270 lab patch (
executor.rs+ plumbing) to unblock hero_rpc — small, high-leverage, single-site bug with two documented fix shapes.