14 Commits

Author SHA1 Message Date
721e26a855 build: remove testing.sh in favor of runit.sh; add claude.md reference
Replace inline boot testing with standalone runit.sh runner for clarity:
- Remove scripts/lib/testing.sh source and boot_tests stage from build.sh
- Remove --skip-tests option from build.sh and rebuild-after-zinit.sh
- Update all docs to reference runit.sh for QEMU/cloud-hypervisor testing
- Add comprehensive claude.md as AI assistant entry point with guidelines

Testing is now fully decoupled from build pipeline; use ./runit.sh for
QEMU/cloud-hypervisor validation after builds complete.
2025-11-04 13:47:24 +01:00
334821dacf Integrate zosstorage build path and runtime orchestration
Some checks failed
Build Zero OS Initramfs / build (push) Has been cancelled
Build Zero OS Initramfs / test-matrix (qemu, basic) (push) Has been cancelled
Build Zero OS Initramfs / test-matrix (qemu, serial) (push) Has been cancelled
Summary:

* add openssh-client to the builder image and mount host SSH keys into the dev container when available

* switch RFS to git builds, register the zosstorage source, and document the extra Rust component

* wire zosstorage into the build: add build_zosstorage(), ship the binary in the initramfs, and extend component validation

* refresh kernel configuration to 6.12.49 while dropping Xen guest selections and enabling counted-by support

* tighten runtime configs: use cached mycelium key path, add zosstorage zinit unit, bootstrap ovsdb-server, and enable openvswitch module

* adjust the network health check ping invocation and fix the RFS pack-tree --debug flag order

* update NOTES changelog, README component list, and introduce a runit helper for qemu/cloud-hypervisor testing

* add ovsdb init script wiring under config/zinit/init and ensure zosstorage is available before mycelium
2025-10-14 17:47:13 +02:00
818f5037f4 docs(TODO): use relative links from docs/ to ../scripts and ../config so links work in GitHub and VS Code 2025-10-02 11:44:41 +02:00
d5e9bf2d9a docs: add persistent TODO.md checklist with clickable references to code and configs 2025-10-01 18:09:28 +02:00
10ba31acb4 docs: regenerate scripts/functionlist.md; refresh NOTES with jump-points and roadmap; extend rfs-flists with RESP backend design. config: add RESP placeholders to rfs.conf.example. components: keep previous non-destructive git clone logic. 2025-10-01 18:06:13 +02:00
ad0a06e267 initramfs+modules: robust copy aliasing, curated stage1 + PHYs, firmware policy via firmware.conf, runtime readiness, build ID; docs sync
Some checks failed
Build Zero OS Initramfs / build (push) Has been cancelled
Build Zero OS Initramfs / test-matrix (qemu, basic) (push) Has been cancelled
Build Zero OS Initramfs / test-matrix (qemu, serial) (push) Has been cancelled
Summary of changes (with references):\n\nModules + PHY coverage\n- Curated and normalized stage1 list in [config.modules.conf](config/modules.conf:1):\n  - Boot-critical storage, core virtio, common NICs (Intel/Realtek/Broadcom), overlay/fuse, USB HCD/HID.\n  - Added PHY drivers required by NIC MACs:\n    * realtek (for r8169, etc.)\n    * broadcom families: broadcom, bcm7xxx, bcm87xx, bcm_phy_lib, bcm_phy_ptp\n- Robust underscore↔hyphen aliasing during copy so e.g. xhci_pci → xhci-pci.ko, hid_generic → hid-generic.ko:\n  - [bash.initramfs_copy_resolved_modules()](scripts/lib/initramfs.sh:990)\n\nFirmware policy and coverage\n- Firmware selection now authoritative via [config/firmware.conf](config/firmware.conf:1); ignore modules.conf firmware hints:\n  - [bash.initramfs_setup_modules()](scripts/lib/initramfs.sh:229)\n  - Count from firmware.conf for reporting; remove stale required-firmware.list.\n- Expanded NIC firmware set (bnx2, bnx2x, tigon, intel, realtek, rtl_nic, qlogic, e100) in [config.firmware.conf](config/firmware.conf:1).\n- Installer enforces firmware.conf source-of-truth in [bash.alpine_install_firmware()](scripts/lib/alpine.sh:392).\n\nEarly input & build freshness\n- Write a runtime build stamp to /etc/zero-os-build-id for embedded initramfs verification:\n  - [bash.initramfs_finalize_customization()](scripts/lib/initramfs.sh:568)\n- Minor init refinements in [config.init](config/init:1) (ensures /home, consistent depmod path).\n\nRebuild helper improvements\n- [scripts/rebuild-after-zinit.sh](scripts/rebuild-after-zinit.sh:1):\n  - Added --verify-only; container-aware execution; selective marker clears only.\n  - Prints stage status before/after; avoids --rebuild-from; resolves full kernel version for diagnostics.\n\nRemote flist readiness + zinit\n- Init scripts now probe BASE_URL readiness and accept FLISTS_BASE_URL/FLIST_BASE_URL; firmware target is /lib/firmware:\n  - [sh.firmware.sh](config/zinit/init/firmware.sh:1)\n  - [sh.modules.sh](config/zinit/init/modules.sh:1)\n\nContainer, docs, and utilities\n- Stream container build logs by calling runtime build directly in [bash.docker_build_container()](scripts/lib/docker.sh:56).\n- Docs updated to reflect firmware policy, runtime readiness, rebuild helper, early input, and GRUB USB:\n  - [docs.NOTES.md](docs/NOTES.md)\n  - [docs.PROMPT.md](docs/PROMPT.md)\n  - [docs.review-rfs-integration.md](docs/review-rfs-integration.md)\n- Added GRUB USB creator (referenced in docs): [scripts/make-grub-usb.sh](scripts/make-grub-usb.sh)\n\nCleanup\n- Removed legacy/duplicated config trees under configs/ and config/zinit.old/.\n- Minor newline and ignore fixes: [.gitignore](.gitignore:1)\n\nNet effect\n- Runtime now has correct USB HCDs/HID-generic and NIC+PHY coverage (Realtek/Broadcom), with matching firmware installed in initramfs.\n- Rebuild workflow is minimal and host/container-aware; docs are aligned with implemented behavior.\n
2025-09-23 14:03:01 +02:00
310e11d2bf rfs(firmware): pack full Alpine linux-firmware set from container and overmount /lib/firmware
Some checks failed
Build Zero OS Initramfs / build (push) Has been cancelled
Build Zero OS Initramfs / test-matrix (qemu, basic) (push) Has been cancelled
Build Zero OS Initramfs / test-matrix (qemu, serial) (push) Has been cancelled
Change pack source: install all linux-firmware* packages in container and pack from /lib/firmware via [bash.rfs_common_install_all_alpine_firmware_packages()](scripts/rfs/common.sh:290) used by [bash.pack-firmware.sh](scripts/rfs/pack-firmware.sh:21). At runtime, overmount firmware flist on /lib/firmware by updating [sh.firmware.sh](config/zinit/init/firmware.sh:10). Update docs to reflect /lib/firmware mount and new pack strategy.
2025-09-19 08:27:10 +02:00
79ed723303 Notes.md, absolute path normalizing
Some checks failed
Build Zero OS Initramfs / build (push) Has been cancelled
Build Zero OS Initramfs / test-matrix (qemu, basic) (push) Has been cancelled
Build Zero OS Initramfs / test-matrix (qemu, serial) (push) Has been cancelled
- edit NOTES.md for updating line numbers
- add check for using normalized path in initramfs normalization
2025-09-18 21:45:21 +02:00
4f67ea488f docs: add PROMPT.md – on-repo prompt for debugging, build, and ops with function/file jump-points
Some checks failed
Build Zero OS Initramfs / build (push) Has been cancelled
Build Zero OS Initramfs / test-matrix (qemu, basic) (push) Has been cancelled
Build Zero OS Initramfs / test-matrix (qemu, serial) (push) Has been cancelled
2025-09-18 16:18:06 +02:00
fe8c48a862 sync: apply remote flist fallback, passwordless root finalize, path normalization, INITRAMFS_ARCHIVE guard, /home ensure, and notes
Some checks failed
Build Zero OS Initramfs / build (push) Has been cancelled
Build Zero OS Initramfs / test-matrix (qemu, basic) (push) Has been cancelled
Build Zero OS Initramfs / test-matrix (qemu, serial) (push) Has been cancelled
2025-09-09 21:24:28 +02:00
0db55fdc6e docs: add comprehensive repository map and operational notes (build flow, branding passwordless policy, path normalization, container tools)
Some checks failed
Build Zero OS Initramfs / build (push) Has been cancelled
Build Zero OS Initramfs / test-matrix (qemu, basic) (push) Has been cancelled
Build Zero OS Initramfs / test-matrix (qemu, serial) (push) Has been cancelled
2025-09-09 16:17:10 +02:00
c10580d171 branding: enforce passwordless root via passwd -d -R; remove direct passwd/shadow edits
Some checks failed
Build Zero OS Initramfs / build (push) Has been cancelled
Build Zero OS Initramfs / test-matrix (qemu, basic) (push) Has been cancelled
Build Zero OS Initramfs / test-matrix (qemu, serial) (push) Has been cancelled
initramfs: switch to passwd -d -R in scripts/lib/initramfs.sh:initramfs_finalize_customization() for shadow-aware passwordless root (aligned with 9423b708 intent), drop sed and chpasswd paths, and add validation diagnostics. common: normalize INSTALL_DIR/COMPONENTS_DIR/KERNEL_DIR/DIST_DIR to absolute paths after sourcing config to prevent validation resolving under kernel/current. Dockerfile: include shadow (for passwd/chpasswd), ensure openssl and openssl-dev present; remove perl. config: introduce ZEROOS_PASSWORDLESS_ROOT default true and comment password vars. docs: NOTES.md updated with diagnostics and flow.
2025-09-09 13:59:44 +02:00
652d38abb1 build/rfs: integrate RFS flists + runtime orchestration
• Add standalone RFS tooling: scripts/rfs/common.sh, pack-modules.sh, pack-firmware.sh, verify-flist.sh

• Patch flist route.url with read-only Garage S3 credentials; optional HTTPS store row; optional manifest upload via mcli

• Build integration: stage_rfs_flists in scripts/build.sh to pack and embed manifests under initramfs/etc/rfs

• Runtime: add zinit units rfs-modules (after: network), rfs-firmware (after: network) as daemons; add udev-rfs oneshot post-mount

• Keep early udev-trigger oneshot to coldplug NICs before RFS mounts

• Firmware flist reproducible naming: respect FIRMWARE_TAG from env or config/build.conf, default to latest

• Docs: update docs/rfs-flists.md with runtime ordering, reproducible tagging, verification steps
2025-09-08 23:39:20 +02:00
afd4f4c6f9 feat(rfs): flist pack to S3 + read-only route embedding + zinit mount scripts; docs; dev-container tooling
Summary
- Implemented plain S3-only flist workflow (no web endpoint). rfs pack uploads blobs using write creds; flist route.url is patched to embed read-only S3 credentials so rfs mount reads directly from S3.

Changes
1) New RFS tooling (scripts/rfs/)
   - common.sh:
     - Compute FULL_KERNEL_VERSION from configs (no uname).
     - Load S3 config and construct pack store URI.
     - Build read-only S3 route URL and patch flist (sqlite).
     - Helpers to locate modules/firmware trees and rfs binary.
   - pack-modules.sh:
     - Pack /lib/modules/<FULL_KERNEL_VERSION> to dist/flists/modules-<FULL_KERNEL_VERSION>.fl
     - Patch flist route to s3://READ:READ@host:port/ROUTE_PATH?region=ROUTE_REGION (default /blobs, garage).
     - Optional upload of .fl using MinIO client (mcli/mc).
   - pack-firmware.sh:
     - Source firmware from $PROJECT_ROOT/firmware (fallback to initramfs/lib/firmware).
     - Pack to dist/flists/firmware-<TAG_OR_DATE>.fl (FIRMWARE_TAG or YYYYMMDD).
     - Patch flist route to read-only S3; optional .fl upload via mcli/mc.
   - verify-flist.sh:
     - rfs flist inspect/tree; optional mount test (best effort).
   - patch-stores.sh:
     - Helper to patch stores (kept though not used by default).

2) Dev-container (Dockerfile)
   - Added sqlite and MinIO client package for manifest patching/upload (expect mcli binary at runtime; scripts support both mcli/mc).
   - Retains rustup and musl target for building rfs/zinit/mycelium.

3) Config and examples
   - config/rfs.conf.example:
     - S3_ENDPOINT/S3_REGION/S3_BUCKET/S3_PREFIX
     - S3_ACCESS_KEY/S3_SECRET_KEY (write)
     - READ_ACCESS_KEY/READ_SECRET_KEY (read-only)
     - ROUTE_ENDPOINT (defaults to S3_ENDPOINT), ROUTE_PATH=/blobs, ROUTE_REGION=garage
     - MANIFESTS_SUBPATH, UPLOAD_MANIFESTS (mcli upload optional)
   - config/rfs.conf updated by user with real values (not committed here; example included).
   - config/modules.conf minor tweak (staged).

4) Zinit mount scripts (config/zinit/init/)
   - firmware.sh:
     - Mounts firmware-latest.fl over /usr/lib/firmware using rfs mount (env override FIRMWARE_FLIST supported).
   - modules.sh:
     - Mounts modules-$(uname -r).fl over /lib/modules/$(uname -r) (env override MODULES_FLIST supported).
   - Both skip if target already mounted and respect RFS_BIN env.

5) Documentation
   - docs/rfs-flists.md:
     - End-to-end flow, S3-only route URL patching, mcli upload notes.
   - docs/review-rfs-integration.md:
     - Integration points, build flow, and post-build standalone usage.
   - docs/depmod-behavior.md:
     - depmod reads .modinfo; recommend prebuilt modules.*(.bin); use depmod -A only on mismatch.

6) Utility
   - scripts/functionlist.md synced with current functions.

Behavioral details
- Pack (write):
  s3://S3_ACCESS_KEY:S3_SECRET_KEY@HOST:PORT/S3_BUCKET/S3_PREFIX?region=REGION
- Flist route (read, post-patch):
  s3://READ_ACCESS_KEY:READ_SECRET_KEY@HOST:PORT/ROUTE_PATH?region=ROUTE_REGION
  Defaults: ROUTE_PATH=/blobs, ROUTE_REGION=garage; ROUTE_ENDPOINT derived from S3_ENDPOINT if not set.

Runtime mount examples
- Modules:
  rfs mount -m dist/flists/modules-6.12.44-Zero-OS.fl /lib/modules/6.12.44-Zero-OS
- Firmware:
  rfs mount -m dist/flists/firmware-YYYYMMDD.fl /usr/lib/firmware

Notes
- FUSE policy: If "allow_other" error occurs, enable user_allow_other in /etc/fuse.conf or run mounts as root.
- WEB_ENDPOINT rewrite is disabled by default (set WEB_ENDPOINT=""). Plain S3 route is embedded in flists.
- MinIO client binary in dev-container is mcli; scripts support mcli (preferred) and mc (fallback).

Files added/modified
- Added: scripts/rfs/{common.sh,pack-modules.sh,pack-firmware.sh,verify-flist.sh,patch-stores.sh}
- Added: config/zinit/init/{firmware.sh,modules.sh}
- Added: docs/{rfs-flists.md,review-rfs-integration.md,depmod-behavior.md}
- Added: config/rfs.conf.example
- Modified: Dockerfile, scripts/functionlist.md, config/modules.conf, config/zinit/sshd-setup.yaml, .gitignore
2025-09-08 22:51:53 +02:00