[ops] Formalize the nu-shell OSIS seed step — canonical hero_zero_seed binary + schema-aligned TOMLs #30

New issue

Open

opened 2026-04-28 12:21:05 +00:00 by mik-tf · 0 comments

mik-tf commented

2026-04-28 12:21:05 +00:00

Owner

Why

Today hero_zero's legacy seed data (hero_zero/dist/var/seed/<context>/<domain>/*.toml, 572 files across 5 contexts) is the only thing that turns a freshly-installed OSIS into a demo-ready state with Business/Calendar/Projects/Media populated. In the nu-shell deploy path, nothing automates this — it took a manual scp + 2 rounds of running the binary + a custom Python uploader on heronu.

Issues encountered running the legacy binary against the current OSIS:

Cosmetic phantom errors — every .set write reports Failed to parse response: error decoding response body, but writes actually succeed. The binary expects an older OSIS response format.
Schema drift — projects require status ∈ {todo, in_progress, done} but seed TOMLs use status = "active" / "not_started". Had to patch via a Python transformer.
OTOML format requirement — current <domain>.set rejects otoml param name, needs obj param with O:\n prefix. Binary doesn't know this.
Some objects fail silently — 24/54 project+milestone TOMLs hit TOML parse errors at various columns because of enum/field renames.
OSIS URL format assumption — binary defaults to http://localhost:3377/api (hero_proxy era); nu-shell deploy uses hero_router http://10.1.2.2:9988/hero_osis/ui.

Net result: the legacy binary works for about 70% of the seed corpus; the rest requires hand-written shims per domain.

Proposed architecture

Rewrite as hero_zero_seed (or hero_osis_seed — either way, maintained in-tree):

/* crates/hero_zero_seed/
 *   src/main.rs        — clap CLI
 *   src/load.rs        — read TOML dir → typed objects
 *   src/migrate/       — per-schema-version field renames + enum maps
 *   src/client.rs      — current OSIS RPC client (aware of per-domain sockets)
 *   src/verify.rs      — post-seed smoke checks
 * tests/
 *   fixtures/          — the canonical seed TOMLs (moved from hero_zero/dist/var/seed)
 */

Design points:

Seed TOMLs live in-tree, not in hero_zero. Currently the seed files are buried under hero_zero/dist/var/seed/ — a build artifact path. They should be versioned source, e.g. lhumina_code/demo_seed/<context>/<domain>/*.toml.
Schema alignment is enforced at commit time. CI runs cargo test -p hero_zero_seed which attempts to load every seed TOML against the current OSIS types. Any drift fails CI immediately. No more surprise runtime failures.
Per-domain + per-context surgical targeting. hero_zero_seed --context geomind --domain projects should idempotently (re-)seed ONLY those records, leaving everything else untouched.
OTOML encoding is internal. The binary handles O:\n prefix, .set vs .create, obj vs data param selection based on OpenRPC spec.
Verification mode. hero_zero_seed --verify queries OSIS, compares to expected TOML set, prints diff. Used by CI and by the disaster-recovery restore flow (see #XYZ) to confirm the replay worked.
Works via hero_router OR direct sockets. Auto-detect from env: if HERO_SOCKET_DIR exists, use Unix sockets per-domain; else use the configured HTTP URL.

Concrete migration tasks

Move hero_zero/dist/var/seed/* → new demo_seed/ repo (or subdir of hero_zero).
Rewrite hero_osis_seed binary against current hero_osis_sdk types.
Port status-enum / field-name migrations for the 5 domains where current state diverges: projects, identity, business, ai, files.
Add CI job: cargo test -p hero_zero_seed -- --fixtures=demo_seed/ — fails on schema drift.
Integrate as post-install step in hero_skills install-all: after hero_proc start completes, run hero_zero_seed --context default --all-domains for a first-boot baseline.
Add smoke test: after install, verify (a) each context has ≥1 business record, (b) calendar has events, (c) projects has at least one entry.
Extend to also seed hero_office's internal document store (see #160 — PDF island reads from there, not from hero_foundry webdav).

What's in scope vs not

In scope: OSIS per-domain data (business, calendar, projects, identity, communication, media, base, ai, files).

Out of scope for this issue: hero_books library content (that's driven by libraries.txt already), hero_foundry webdav media (already seeded from /home/driver/hero/var/seed_media), hero_office's own document store (separate issue).

#160 — consolidated demo state (§10-11 Projects milestone seed partial failure)
#129 — service_osis.nu doesn't pass HERO_CONTEXTS/HERO_SEED_DIR
#143 — OSIS SDK delete bool/String mismatch (related schema-drift pattern)

Signed-off-by: mik-tf

Previous comments from home#162

mik-tf — 2026-04-25T16:21:59Z

Resolved by lhumina_code/hero_skills@7c823d1 (PR lhumina_code/hero_skills#126).

Part of Phase 2 tracker lhumina_code/home#185.

mik-tf — 2026-04-25T16:31:55Z

Reopening — closed in error earlier today. The hero_demo runbook §13 had this issue listed as the tracker for an unrelated deploy step (ONNX install for #162 / HERO_ROOTDIR override for #164), and I trusted the reference without checking the actual issue body. Apologies for the noise. The actual scope of this issue is unchanged from when it was filed.

The correct trackers for the work that just landed: ONNX install + HERO_ROOTDIR are covered directly by lhumina_code/hero_skills@7c823d1 and tracked under lhumina_code/home#185 (no separate sub-issues filed).

Originally filed as home#162 on 2026-04-24 by mik-tf — moved to hero_demo as part of consolidating issue tracking.

## Why Today hero_zero's legacy seed data (`hero_zero/dist/var/seed/<context>/<domain>/*.toml`, 572 files across 5 contexts) is the only thing that turns a freshly-installed OSIS into a demo-ready state with Business/Calendar/Projects/Media populated. In the nu-shell deploy path, **nothing automates this** — it took a manual scp + 2 rounds of running the binary + a custom Python uploader on heronu. Issues encountered running the legacy binary against the current OSIS: 1. **Cosmetic phantom errors** — every `.set` write reports `Failed to parse response: error decoding response body`, but writes actually succeed. The binary expects an older OSIS response format. 2. **Schema drift** — projects require `status ∈ {todo, in_progress, done}` but seed TOMLs use `status = "active"` / `"not_started"`. Had to patch via a Python transformer. 3. **OTOML format requirement** — current `<domain>.set` rejects `otoml` param name, needs `obj` param with `O:\n` prefix. Binary doesn't know this. 4. **Some objects fail silently** — 24/54 project+milestone TOMLs hit TOML parse errors at various columns because of enum/field renames. 5. **OSIS URL format assumption** — binary defaults to `http://localhost:3377/api` (hero_proxy era); nu-shell deploy uses hero_router `http://10.1.2.2:9988/hero_osis/ui`. Net result: the legacy binary works for about 70% of the seed corpus; the rest requires hand-written shims per domain. ## Proposed architecture Rewrite as `hero_zero_seed` (or `hero_osis_seed` — either way, maintained in-tree): ``` /* crates/hero_zero_seed/ * src/main.rs — clap CLI * src/load.rs — read TOML dir → typed objects * src/migrate/ — per-schema-version field renames + enum maps * src/client.rs — current OSIS RPC client (aware of per-domain sockets) * src/verify.rs — post-seed smoke checks * tests/ * fixtures/ — the canonical seed TOMLs (moved from hero_zero/dist/var/seed) */ ``` Design points: 1. **Seed TOMLs live in-tree, not in hero_zero.** Currently the seed files are buried under `hero_zero/dist/var/seed/` — a build artifact path. They should be versioned source, e.g. `lhumina_code/demo_seed/<context>/<domain>/*.toml`. 2. **Schema alignment is enforced at commit time.** CI runs `cargo test -p hero_zero_seed` which attempts to load every seed TOML against the current OSIS types. Any drift fails CI immediately. No more surprise runtime failures. 3. **Per-domain + per-context surgical targeting.** `hero_zero_seed --context geomind --domain projects` should idempotently (re-)seed ONLY those records, leaving everything else untouched. 4. **OTOML encoding is internal.** The binary handles `O:\n` prefix, `.set` vs `.create`, `obj` vs `data` param selection based on OpenRPC spec. 5. **Verification mode.** `hero_zero_seed --verify` queries OSIS, compares to expected TOML set, prints diff. Used by CI and by the disaster-recovery restore flow (see [#XYZ](#)) to confirm the replay worked. 6. **Works via hero_router OR direct sockets.** Auto-detect from env: if `HERO_SOCKET_DIR` exists, use Unix sockets per-domain; else use the configured HTTP URL. ## Concrete migration tasks - [ ] Move `hero_zero/dist/var/seed/*` → new `demo_seed/` repo (or subdir of `hero_zero`). - [ ] Rewrite `hero_osis_seed` binary against current hero_osis_sdk types. - [ ] Port status-enum / field-name migrations for the 5 domains where current state diverges: projects, identity, business, ai, files. - [ ] Add CI job: `cargo test -p hero_zero_seed -- --fixtures=demo_seed/` — fails on schema drift. - [ ] Integrate as post-install step in `hero_skills install-all`: after `hero_proc start` completes, run `hero_zero_seed --context default --all-domains` for a first-boot baseline. - [ ] Add smoke test: after install, verify (a) each context has ≥1 business record, (b) calendar has events, (c) projects has at least one entry. - [ ] Extend to also seed hero_office's internal document store (see [#160](https://forge.ourworld.tf/lhumina_code/home/issues/160) — PDF island reads from there, not from hero_foundry webdav). ## What's in scope vs not **In scope:** OSIS per-domain data (business, calendar, projects, identity, communication, media, base, ai, files). **Out of scope for this issue:** hero_books library content (that's driven by `libraries.txt` already), hero_foundry webdav media (already seeded from `/home/driver/hero/var/seed_media`), hero_office's own document store (separate issue). ## Related - [#160](https://forge.ourworld.tf/lhumina_code/home/issues/160) — consolidated demo state (§10-11 Projects milestone seed partial failure) - [#129](https://forge.ourworld.tf/lhumina_code/home/issues/129) — service_osis.nu doesn't pass HERO_CONTEXTS/HERO_SEED_DIR - [#143](https://forge.ourworld.tf/lhumina_code/home/issues/143) — OSIS SDK delete bool/String mismatch (related schema-drift pattern) Signed-off-by: mik-tf --- ### Previous comments from home#162 #### mik-tf — 2026-04-25T16:21:59Z Resolved by https://forge.ourworld.tf/lhumina_code/hero_skills/commit/7c823d1 (PR https://forge.ourworld.tf/lhumina_code/hero_skills/pulls/126). Part of Phase 2 tracker https://forge.ourworld.tf/lhumina_code/home/issues/185. #### mik-tf — 2026-04-25T16:31:55Z Reopening — closed in error earlier today. The hero_demo runbook §13 had this issue listed as the tracker for an unrelated deploy step (ONNX install for #162 / HERO_ROOTDIR override for #164), and I trusted the reference without checking the actual issue body. Apologies for the noise. The actual scope of this issue is unchanged from when it was filed. The correct trackers for the work that just landed: ONNX install + HERO_ROOTDIR are covered directly by https://forge.ourworld.tf/lhumina_code/hero_skills/commit/7c823d1 and tracked under https://forge.ourworld.tf/lhumina_code/home/issues/185 (no separate sub-issues filed). --- *Originally filed as [home#162](https://forge.ourworld.tf/lhumina_code/home/issues/162) on 2026-04-24 by mik-tf — moved to hero_demo as part of consolidating issue tracking.*