zosbuilder/docs/NOTES.md

Zero-OS Builder – Working Notes and Repository Map

Purpose
- This document captures operational knowledge of this repository: build flow, key files, flags, and recent behavior decisions (e.g., passwordless root).
- Links below point to exact functions and files for fast triage, using code navigation-friendly anchors.

Repository Overview
- Build entrypoint: [scripts/build.sh](scripts/build.sh)
  - Orchestrates incremental stages using stage markers.
  - Runs inside a container defined by [Dockerfile](Dockerfile) for reproducibility.
- Common utilities and config loading: [bash.common.sh](scripts/lib/common.sh:1)
  - Loads [config/build.conf](config/build.conf), normalizes directory paths, provides logging and safe execution wrappers.
- Initramfs assembly and finalization: [bash.initramfs_* functions](scripts/lib/initramfs.sh:1)
  - Copies components, sets up zinit configs, finalizes branding, creates CPIO archive, validates contents.
- Kernel integration (optional embedded initramfs): [bash.kernel_* functions](scripts/lib/kernel.sh:1)
  - Downloads/configures/builds kernel and modules, embeds initramfs, runs depmod.
- zinit configuration: [config/zinit/](config/zinit/)
  - YAML service definitions and init scripts used by zinit inside the initramfs rootfs.
- RFS tooling (modules/firmware flists): [scripts/rfs/](scripts/rfs/)
  - Packs module/firmware flists and embeds them into initramfs at /etc/rfs.

Container Tooling (dev-container)
- Base image: Alpine 3.22 in [Dockerfile](Dockerfile:1)
- Tools:
  - shadow (passwd/chpasswd): required for root password management in initramfs.
  - openssl, openssl-dev: kept for other build steps and potential hashing utilities.
  - build-base, rustup, kmod, upx, etc.: required by various build stages.
- Removed: perl, not required for password handling after switching to passwd/chpasswd workflow.

Configuration – build.conf
- File: [config/build.conf](config/build.conf)
- Key variables:
  - Versions: ALPINE_VERSION, KERNEL_VERSION
  - Directories (relative in config, normalized to absolute during runtime):
    - INSTALL_DIR="initramfs"
    - COMPONENTS_DIR="components"
    - KERNEL_DIR="kernel"
    - DIST_DIR="dist"
  - Flags:
    - ZEROOS_BRANDING="true"
    - ZEROOS_REBRANDING="true"
  - Branding behavior:
    - ZEROOS_PASSWORDLESS_ROOT="true" (default for branded builds in current policy)
    - ZEROOS_ROOT_PASSWORD_HASH / ROOT_PASSWORD_HASH (not used in current policy)
    - ZEROOS_ROOT_PASSWORD / ROOT_PASSWORD (not used in current policy)
  - FIRMWARE_TAG optional for reproducible firmware flist naming.

Absolute Path Normalization
- Location: [bash.common.sh](scripts/lib/common.sh:236)
- After sourcing build.conf, the following variables are normalized to absolute paths anchored at PROJECT_ROOT:
  - INSTALL_DIR, COMPONENTS_DIR, KERNEL_DIR, DIST_DIR
- Rationale: Prevents path resolution errors when CWD changes (e.g., when kernel build operates in /workspace/kernel/current, validation now resolves to /workspace/initramfs instead of /workspace/kernel/current/initramfs).

Build Pipeline – High Level
- Orchestrator: [bash.main_build_process()](scripts/build.sh:214)
  - Stage list:
    - alpine_extract
    - alpine_configure
    - alpine_packages
    - alpine_firmware
    - components_build
    - components_verify
    - kernel_modules
    - init_script
    - components_copy
    - zinit_setup
    - modules_setup
    - modules_copy
    - cleanup
    - rfs_flists
    - validation
    - initramfs_create
    - initramfs_test
    - kernel_build
    - boot_tests
  - Each stage wrapped with [bash.stage_run()](scripts/lib/stages.sh:99) and tracked under .build-stages/
- Container use:
  - Always run in container for stable toolchain (podman/docker auto-detected).
  - Inside container, CWD normalized to PROJECT_ROOT.

Initramfs Assembly – Key Functions
- zinit setup: [bash.initramfs_setup_zinit()](scripts/lib/initramfs.sh:12)
  - Copies [config/zinit](config/zinit/) into /etc/zinit, fixes perms, removes OpenRC remnants.
- Install init script: [bash.initramfs_install_init_script()](scripts/lib/initramfs.sh:74)
  - Installs [config/init](config/init) as /init in initramfs root.
- Components copy: [bash.initramfs_copy_components()](scripts/lib/initramfs.sh:101)
-   Installs built components (zinit/rfs/mycelium/corex/zosstorage) into proper locations, strips/UPX where applicable.
- Modules setup: [bash.initramfs_setup_modules()](scripts/lib/initramfs.sh:229)
  - Reads [config/modules.conf](config/modules.conf), resolves deps via [bash.initramfs_resolve_module_dependencies()](scripts/lib/initramfs.sh:318), generates stage1 list (firmware hints in modules.conf are ignored; firmware.conf is authoritative).
- Create module scripts: [bash.initramfs_create_module_scripts()](scripts/lib/initramfs.sh:427)
  - Writes /etc/zinit/init/stage1-modules.sh and stage2-modules.sh for zinit to load modules.
- Binary size optimization: [bash.initramfs_strip_and_upx()](scripts/lib/initramfs.sh:491)
- Final customization: [bash.initramfs_finalize_customization()](scripts/lib/initramfs.sh:575)
  - See “Branding behavior” below.
- Create archive: [bash.initramfs_create_cpio()](scripts/lib/initramfs.sh:688)
  - Calls finalize, runs sanity checks, and creates initramfs.cpio.xz.
- Validate: [bash.initramfs_validate()](scripts/lib/initramfs.sh:799)
  - Ensures essential items exist, logs debug info:
    - Prints “Validation debug:” lines showing input, PWD, PROJECT_ROOT, INSTALL_DIR, and resolved path.

Kernel Integration
- Get full kernel version: [bash.kernel_get_full_version()](scripts/lib/kernel.sh:13)
- Apply config (embed initramfs): [bash.kernel_apply_config()](scripts/lib/kernel.sh:81)
  - Updates CONFIG_INITRAMFS_SOURCE to the archive’s absolute path via [bash.kernel_modify_config_for_initramfs()](scripts/lib/kernel.sh:130).
- Build kernel: [bash.kernel_build_with_initramfs()](scripts/lib/kernel.sh:173)
- Build and install modules in container: [bash.kernel_build_modules()](scripts/lib/kernel.sh:228)
  - Installs modules to /lib/modules/$FULL_VERSION in container, runs depmod -a.

RFS Flists (modules/firmware)
- Packing scripts:
  - Modules: [bash.pack-modules.sh](scripts/rfs/pack-modules.sh:1)
  - Firmware: [bash.pack-firmware.sh](scripts/rfs/pack-firmware.sh:1)
- Firmware policy:
  - For initramfs: [config/firmware.conf](config/firmware.conf) is the single source of truth for preinstalled firmware; modules.conf hints are ignored.
  - For RFS: install all Alpine linux-firmware* packages into the build container and pack from /lib/firmware (full set for runtime).
- Integrated in stage_rfs_flists:
  - Embeds /etc/rfs/modules-<FULL_KERNEL_VERSION>.fl
  - Embeds /etc/rfs/firmware-latest.fl (or tagged by FIRMWARE_TAG)
  - See [bash.main_build_process() — stage_rfs_flists](scripts/build.sh:298)
- Runtime mount/readiness:
  - Firmware flist mounts over /lib/firmware (overmount hides any initramfs firmware).
  - Modules flist mounts at /lib/modules/$(uname -r).
  - Init scripts probe BASE_URL reachability (accepts FLISTS_BASE_URL or FLIST_BASE_URL) and wait for HTTP(S) before fetching:
    - Firmware: [sh.firmware.sh](config/zinit/init/firmware.sh:1)
    - Modules: [sh.modules.sh](config/zinit/init/modules.sh:1)

Branding Behavior (Passwordless Root, motd/issue)
- Finalization hook: [bash.initramfs_finalize_customization()](scripts/lib/initramfs.sh:575)
- Behavior (current policy):
  - Passwordless root enforced using passwd for shadow-aware deletion:
    - [bash.initramfs_finalize_customization()](scripts/lib/initramfs.sh:616):
      - passwd -d -R "${initramfs_dir}" root
      - Ensures /etc/shadow has root:: (empty password) inside initramfs root, not host.
  - Branding toggles: ZEROOS_BRANDING and ZEROOS_REBRANDING (branding guard printed in logs).
  - Branding also updates /etc/motd and /etc/issue to Zero-OS.

Console and getty
- Early keyboard and debug:
  - [config/init](config/init) preloads input/HID and USB HCD modules (i8042, atkbd, usbhid, hid, hid_generic, evdev, xhci/ehci/ohci/uhci) so console input works before zinit/rfs.
  - Kernel cmdline initdebug=true opens an early interactive shell; if /init-debug exists and is executable, it runs preferentially.
- Serial and console getty configs (zinit service YAML):
  - [config/zinit/getty-tty1.yaml](config/zinit/getty-tty1.yaml)
  - [config/zinit/getty-console.yaml](config/zinit/getty-console.yaml)
- Optional ash login loop (not enabled unless referenced):
  - [bash.ashloging.sh](config/zinit/init/ashloging.sh:1)

Validation Diagnostics and Triage
- Common error previously observed:
  - “Initramfs directory not found: initramfs (resolved: /workspace/kernel/current/initramfs)”
- Root cause:
  - INSTALL_DIR re-sourced in a different CWD and interpreted as relative.
- Fix:
  - Absolute path normalization of INSTALL_DIR/COMPONENTS_DIR/KERNEL_DIR/DIST_DIR after sourcing build.conf in [bash.common.sh](scripts/lib/common.sh:236).
  - Additional “Validation debug” prints added in [bash.initramfs_validate()](scripts/lib/initramfs.sh:799).
- Expected logs now:
  - “Validation debug: input='initramfs' PWD=/workspace PROJECT_ROOT=/workspace INSTALL_DIR=/workspace/initramfs”
  - Resolves correctly even if called from a different stage CWD.

How to Verify Passwordless Root
- After build, check archive:
  - mkdir -p dist/_inspect && cd dist/_inspect
  - xz -dc ../initramfs.cpio.xz | cpio -idmv
  - grep '^root:' ./etc/shadow
  - Expect root:: (empty field) indicating passwordless root.
- At runtime on console:
  - When prompted for root’s password, press Enter.

Stage System and Incremental Rebuilds
- Stage markers stored in .build-stages/ (one file per stage).
- Minimal rebuild helper (host or container):
  - [scripts/rebuild-after-zinit.sh](scripts/rebuild-after-zinit.sh) clears only: modules_setup, modules_copy, init_script, zinit_setup, validation, initramfs_create, initramfs_test (kernel_build only with --with-kernel; kernel_modules only with --refresh-container-mods).
  - Flags:
    - --with-kernel (also rebuild kernel; ensures cpio is recreated right before embedding)
    - --refresh-container-mods (rebuild container /lib/modules for fresh containers)
    - --verify-only (report changed files and stage status; no rebuild)
  - Shows stage status before/after marker removal; no --rebuild-from is passed by default (relies on markers only).
- Manual minimal rebuild:
  - Remove relevant .done files, e.g.: initramfs_create.done initramfs_test.done validation.done
  - Rerun: DEBUG=1 ./scripts/build.sh
- Show status:
  - ./scripts/build.sh --show-stages
- Test built kernel:
  - ./runit.sh --hypervisor qemu
  - ./runit.sh --hypervisor ch --disks 5 --reset

Key Decisions (current)
- Firmware selection for initramfs comes exclusively from [config/firmware.conf](config/firmware.conf); firmware hints in modules.conf are ignored to avoid duplication/mismatch.
- Runtime firmware flist overmounts /lib/firmware after network readiness; init scripts wait for FLISTS_BASE_URL/FLIST_BASE_URL HTTP reachability before fetching.
- Early keyboard and debug shell added to [config/init](config/init) as described above.
- Branding enforces passwordless root via passwd -d -R inside initramfs finalization, avoiding direct edits of passwd/shadow files.
- Directory paths normalized to absolute after loading config to avoid CWD-sensitive behavior.
- Container image contains shadow suite to ensure passwd/chpasswd availability; perl removed.

File Pointers (quick jump)
- Orchestrator: [scripts/build.sh](scripts/build.sh)
- Common and config loading: [bash.common.sh](scripts/lib/common.sh:1)
- Finalization hook: [bash.initramfs_finalize_customization()](scripts/lib/initramfs.sh:575)
- Validation entry: [bash.initramfs_validate()](scripts/lib/initramfs.sh:799)
- CPIO creation: [bash.initramfs_create_cpio()](scripts/lib/initramfs.sh:688)
- Kernel embed config: [bash.kernel_modify_config_for_initramfs()](scripts/lib/kernel.sh:130)
- RFS packers: [bash.pack-modules.sh](scripts/rfs/pack-modules.sh:1), [bash.pack-firmware.sh](scripts/rfs/pack-firmware.sh:1)
- USB creator: [scripts/make-grub-usb.sh](scripts/make-grub-usb.sh)

Change Log
- 2025-09-09:
  - Enforce passwordless root using passwd -d -R in finalization.
  - Normalize INSTALL_DIR/COMPONENTS_DIR/KERNEL_DIR/DIST_DIR to absolute paths post-config load.
  - Add validation diagnostics prints (input/PWD/PROJECT_ROOT/INSTALL_DIR/resolved).
  - Ensure shadow package in container for passwd/chpasswd; keep openssl and openssl-dev; remove perl earlier.
- 2025-10-10:
  - Record zosstorage as a standard component installed during [bash.initramfs_copy_components()](scripts/lib/initramfs.sh:101) and sync component documentation references.
Updates 2025-10-01

- Function index regenerated: see [scripts/functionlist.md](scripts/functionlist.md) for an authoritative map of all functions with current line numbers. Use it alongside the quick links below to jump into code fast.
- Key jump-points (current lines):
  - Finalization: [bash.initramfs_finalize_customization()](scripts/lib/initramfs.sh:568)
  - CPIO creation: [bash.initramfs_create_cpio()](scripts/lib/initramfs.sh:691)
  - Validation: [bash.initramfs_validate()](scripts/lib/initramfs.sh:820)
  - Kernel embed config: [bash.kernel_modify_config_for_initramfs()](scripts/lib/kernel.sh:130)
  - Stage orchestrator entry: [bash.main_build_process()](scripts/build.sh:214)
  - Repo-wide index: [scripts/functionlist.md](scripts/functionlist.md)

Roadmap / TODO (tracked in tool todo list)

- Zosception (zinit service graph and ordering)
  - Define additional services and ordering for nested/recursive orchestration.
  - Likely integration points:
    - Networking readiness before RFS: [config/zinit/network.yaml](config/zinit/network.yaml)
    - Early udev coldplug: [config/zinit/udev-trigger.yaml](config/zinit/udev-trigger.yaml)
    - Post-RFS coldplug: [config/zinit/udev-rfs.yaml](config/zinit/udev-rfs.yaml)
  - Ensure dependency edges are correct in the service DAG image (see docs/img_*.png).

- Add zosstorage to initramfs
  - Source:
    - If packaged: add to [config/packages.list](config/packages.list).
    - If built from source: extend [bash.components_parse_sources_conf()](scripts/lib/components.sh:13) and add a build_* function; install via [bash.initramfs_copy_components()](scripts/lib/initramfs.sh:102).
  - Zinit unit:
    - Add YAML under [config/zinit/](config/zinit/) and hook into the network-ready path.
  - Ordering:
    - Start after "network" and before/with RFS mounts if it provides storage functionality used by rfs.

- RFS blob store backends (design + docs; http and s3 exist)
  - Current S3 store URI construction: [bash.rfs_common_build_s3_store_uri()](scripts/rfs/common.sh:137)
  - Flist manifest store patching: [bash.rfs_common_patch_flist_stores()](scripts/rfs/common.sh:385)
  - Route URL patching: [bash.rfs_common_patch_flist_route_url()](scripts/rfs/common.sh:494)
  - Packers entrypoints:
    - [scripts/rfs/pack-modules.sh](scripts/rfs/pack-modules.sh:1)
    - [scripts/rfs/pack-firmware.sh](scripts/rfs/pack-firmware.sh:1)
  - Proposed additional backend: RESP/DB-style store
    - Goal: Allow rfs to push/fetch content-addressed blobs via a RESP-compatible endpoint (e.g., Redis/KeyDB/Dragonfly-like), or a thin HTTP/RESP adapter.
    - Draft URI scheme examples:
      - resp://host:port/db?tls=0&amp;prefix=blobs
      - resp+tls://host:port/db?prefix=blobs&amp;ca=/etc/ssl/certs/ca.pem
      - resp+sentinel://sentinelHost:26379/mymaster?prefix=blobs
    - Minimum operations:
      - PUT blob: SETEX prefix/ab/cd/hash ttl file-bytes or HSET prefix/hash data file-bytes
      - GET blob: GET or HGET
      - HEAD/exists: EXISTS
      - Optional: pipelined/mget for batch prefetch
    - Client integration layers:
      - Pack-time: extend rfs CLI store resolver (design doc first; scripts/rfs/common.sh can map scheme→uploader if CLI not ready).
      - Manifest post-process: still supported; stores table may include multiple URIs (s3 + resp) for redundancy.
    - Caching and retries:
      - Local on-disk cache under dist/.rfs-cache keyed by hash with LRU GC.
      - Exponential backoff on GET failures; fall back across stores in order.
    - Auth:
      - RESP: optional username/password in URI; TLS with cert pinning parameters.
      - Keep secrets in config/rfs.conf or env; do not embed write creds in manifests (read-credential routes only).
    - Deliverables:
      - Design section in docs/rfs-flists.md (to be added)
      - Config keys in config/rfs.conf.example for RESP endpoints
      - Optional shim uploader script if CLI support lags.

- Documentation refresh tasks
  - Cross-check this file’s clickable references against [scripts/functionlist.md](scripts/functionlist.md) after changes in lib files.
  - Keep “Branding behavior” and “Absolute Path Normalization” pointers aligned with:
    - [bash.common.sh normalization](scripts/lib/common.sh:244)
    - [bash.initramfs_finalize_customization()](scripts/lib/initramfs.sh:568)

Diagnostics-first reminder

- Use DEBUG=1 and stage markers for minimal rebuilds.
- Quick commands:
  - Show stages: ./scripts/build.sh --show-stages
  - Minimal rebuild after zinit/init edits: [scripts/rebuild-after-zinit.sh](scripts/rebuild-after-zinit.sh)
  - Validate archive: [bash.initramfs_create_cpio()](scripts/lib/initramfs.sh:691), then [bash.initramfs_test_archive()](scripts/lib/initramfs.sh:953)