zosbuilder/docs/rfs-flists.md

# RFS flist creation and runtime overmounts (design)

Goal
- Produce two flists without modifying existing build scripts:
  - firmware-VERSION.fl
  - modules-KERNEL_FULL_VERSION.fl
- Store blobs in S3 via rfs store; upload .fl manifest (sqlite) separately to S3.
- Overmount these at runtime later to enable extended hardware, then depmod + udev trigger.

Scope of this change
- Add standalone scripts under [scripts/rfs](scripts/rfs) (no changes in existing libs or stages).
- Add a config file [config/rfs.conf](config/rfs.conf) for S3 credentials and addressing.
- Document the flow and usage here; scripting comes next.

Inputs
- Built kernel modules present in the dev-container (from kernel build stages):
  - Preferred: /lib/modules/KERNEL_FULL_VERSION
- Firmware source for RFS pack:
-   Install all Alpine linux-firmware* packages into the build container and use /lib/firmware as the source (full set).
- Initramfs fallback (build-time):
-   Selective firmware packages installed by [bash.alpine_install_firmware()](scripts/lib/alpine.sh:392) into initramfs/lib/firmware (kept inside the initramfs).
- Kernel version derivation (never use uname -r in container):
  - Combine KERNEL_VERSION from [config/build.conf](config/build.conf) and LOCALVERSION from [config/kernel.config](config/kernel.config).
  - This matches [kernel_get_full_version()](scripts/lib/kernel.sh:14).

Outputs and locations
- Flists:
  - [dist/flists/firmware-VERSION.fl](dist/flists/firmware-VERSION.fl)
  - [dist/flists/modules-KERNEL_FULL_VERSION.fl](dist/flists/modules-KERNEL_FULL_VERSION.fl)
- Blobs are uploaded by rfs to the configured S3 store.
- Manifests (.fl sqlite) are uploaded by script as S3 objects (separate from blob store).

Configuration: [config/rfs.conf](config/rfs.conf)
Required values:
- S3_ENDPOINT=https://s3.example.com:9000
- S3_REGION=us-east-1
- S3_BUCKET=zos
- S3_PREFIX=flists/zosbuilder
- S3_ACCESS_KEY=AKIA...
- S3_SECRET_KEY=...

Notes:
- We construct an rfs S3 store URI for pack operations (for blob uploads during pack):
  - s3://S3_ACCESS_KEY:S3_SECRET_KEY@HOST:PORT/S3_BUCKET/S3_PREFIX?region=S3_REGION
- After pack, we correct the flist route URL to include READ-ONLY credentials so mounts can read directly from Garage:
  - UPDATE route SET url='s3://READ_ACCESS_KEY:READ_SECRET_KEY@HOST:PORT/ROUTE_PATH?region=ROUTE_REGION'
  - Defaults: ROUTE_PATH=/blobs, ROUTE_REGION=garage, ROUTE_ENDPOINT=S3_ENDPOINT (overridable)

Scripts to add (standalone)
- [scripts/rfs/common.sh](scripts/rfs/common.sh)
  - Read [config/build.conf](config/build.conf) and [config/kernel.config](config/kernel.config).
  - Compute FULL_KERNEL_VERSION exactly as [kernel_get_full_version()](scripts/lib/kernel.sh:14).
  - Read and validate [config/rfs.conf](config/rfs.conf).
  - Build S3 store URI for rfs.
  - Locate module and firmware source trees (with priority rules).
  - Locate rfs binary (PATH first, fallback to [components/rfs/target/x86_64-unknown-linux-musl/release/rfs](components/rfs/target/x86_64-unknown-linux-musl/release/rfs)).

- [scripts/rfs/pack-modules.sh](scripts/rfs/pack-modules.sh)
  - Name: modules-KERNEL_FULL_VERSION.fl (e.g., modules-6.12.44-Zero-OS.fl).
  - rfs pack -m dist/flists/modules-...fl -s s3://... /lib/modules/KERNEL_FULL_VERSION
  - Optional: upload dist/flists/modules-...fl to s3://S3_BUCKET/S3_PREFIX/manifests/ using MinIO Client (mc) if present.

- [scripts/rfs/pack-firmware.sh](scripts/rfs/pack-firmware.sh)
  - Source: $PROJECT_ROOT/firmware if exists, else initramfs/lib/firmware.
  - Name: firmware-YYYYMMDD.fl by default; override with FIRMWARE_TAG env to firmware-FIRMWARE_TAG.fl.
  - rfs pack as above; optional upload of the .fl manifest using MinIO Client (mc) if present.

- [scripts/rfs/verify-flist.sh](scripts/rfs/verify-flist.sh)
  - rfs flist inspect dist/flists/NAME.fl
  - rfs flist tree dist/flists/NAME.fl | head
  - Optional: test mount if run with --mount (mountpoint under /tmp).

Runtime (deferred to a follow-up)
- New zinit units to mount and coldplug:
  -   Mount firmware flist read-only at /lib/firmware
  - Mount modules flist at /lib/modules/KERNEL_FULL_VERSION
  - Run depmod -a KERNEL_FULL_VERSION
  - udevadm control --reload; udevadm trigger --action=add; udevadm settle
- Placement examples (to be created later):
  - [config/zinit/rfs-modules.yaml](config/zinit/rfs-modules.yaml)
  - [config/zinit/rfs-firmware.yaml](config/zinit/rfs-firmware.yaml)
  - Keep in correct dependency order before [config/zinit/udev-trigger.yaml](config/zinit/udev-trigger.yaml).

Naming policy
- modules flist:
  - modules-KERNEL_FULL_VERSION.fl
- firmware flist:
  - firmware-YYYYMMDD.fl by default
  - firmware-FIRMWARE_TAG.fl if env FIRMWARE_TAG is set

Usage flow (after your normal build inside dev-container)
1) Create config for S3: [config/rfs.conf](config/rfs.conf)
2) Generate modules flist: [scripts/rfs/pack-modules.sh](scripts/rfs/pack-modules.sh)
3) Generate firmware flist: [scripts/rfs/pack-firmware.sh](scripts/rfs/pack-firmware.sh)
4) Verify manifests: [scripts/rfs/verify-flist.sh](scripts/rfs/verify-flist.sh) dist/flists/modules-...fl

Assumptions
- rfs supports s3 store URIs as described (per [components/rfs/README.md](components/rfs/README.md)).
- The dev-container has the built kernel modules in /lib/modules/KERNEL_FULL_VERSION (as produced via [kernel_build_modules()](scripts/lib/kernel.sh:228)).
- No changes are made to existing build scripts. The new scripts are run on-demand.

Open question for confirm
- Confirm S3 endpoint form (with or without explicit port) and whether we should prefer AWS_REGION env over query param; scripts will support both patterns.

Note on route URL vs HTTP endpoint
- rfs mount reads blobs via s3:// URLs, not via an arbitrary HTTP(S) endpoint. A reverse proxy is not required if you embed read-only S3 credentials in the flist.
- This project now patches the flist after pack to set route.url to a read-only Garage S3 URL:
  - Example SQL equivalent:
    - UPDATE route SET url='s3://READ_ACCESS_KEY:READ_SECRET_KEY@[HOST]:3900/blobs?region=garage';
- Configure these in config/rfs.conf:
  - READ_ACCESS_KEY / READ_SECRET_KEY: read-only credentials
  - ROUTE_ENDPOINT (defaults to S3_ENDPOINT), ROUTE_PATH=/blobs, ROUTE_REGION=garage
- Do not set ROUTE_PATH to S3_PREFIX. ROUTE_PATH is the gateway’s blob route (usually /blobs). S3_PREFIX is only for the pack-time store path.

## Runtime units and ordering (zinit)

This repo now includes runtime zinit units and init scripts to mount the RFS flists and perform dual udev coldplug sequences.

- Early coldplug (before RFS mounts):
  - [config/zinit/udev-trigger.yaml](config/zinit/udev-trigger.yaml) calls [config/zinit/init/udev.sh](config/zinit/init/udev.sh).
  - Runs after depmod/udev daemons to initialize NICs and other devices using what is already in the initramfs.
  - Purpose: bring up networking so RFS can reach Garage S3.

- RFS mounts (daemons, after network):
  - [config/zinit/rfs-modules.yaml](config/zinit/rfs-modules.yaml) runs [config/zinit/init/modules.sh](config/zinit/init/modules.sh) to mount modules-$(uname -r).fl onto /lib/modules/$(uname -r).
  - [config/zinit/rfs-firmware.yaml](config/zinit/rfs-firmware.yaml) runs [config/zinit/init/firmware.sh](config/zinit/init/firmware.sh) to mount firmware-latest.fl onto /usr/lib/firmware.
  - Both are defined as restart: always and include after: network to ensure the Garage S3 route is reachable.

- Post-mount coldplug (after RFS mounts):
  - [config/zinit/udev-rfs.yaml](config/zinit/udev-rfs.yaml) performs:
    - udevadm control --reload
    - udevadm trigger --action=add --type=subsystems
    - udevadm trigger --action=add --type=devices
    - udevadm settle
  - This re-probes hardware so new modules/firmware from the overmounted flists are considered.

- Embedded manifests in initramfs:
  - The build embeds the flists under /etc/rfs:
    - modules-KERNEL_FULL_VERSION.fl
    - firmware-latest.fl
  - Creation happens in [scripts/rfs/pack-modules.sh](scripts/rfs/pack-modules.sh) and [scripts/rfs/pack-firmware.sh](scripts/rfs/pack-firmware.sh), and embedding is orchestrated by [scripts/build.sh](scripts/build.sh).

## Reproducible firmware tagging

- The firmware flist name can be pinned via FIRMWARE_TAG in [config/build.conf](config/build.conf).
  - If set: firmware-FIRMWARE_TAG.fl
  - If unset: the build uses firmware-latest.fl for embedding (standalone pack may default to date-based).
- The build logic picks the tag with this precedence:
  1) Environment FIRMWARE_TAG
  2) FIRMWARE_TAG from [config/build.conf](config/build.conf)
  3) "latest"
- Build integration implemented in [scripts/build.sh](scripts/build.sh).

Example:
- Set FIRMWARE_TAG in config: add FIRMWARE_TAG="20250908" in [config/build.conf](config/build.conf)
- Or export at build time: export FIRMWARE_TAG="v1"

## Verifying flists

Use the helper to inspect a manifest, optionally listing entries and testing a local mount (root + proper FUSE policy required):

- Inspect only:
  - scripts/rfs/verify-flist.sh -m dist/flists/modules-6.12.44-Zero-OS.fl
- Inspect + tree:
  - scripts/rfs/verify-flist.sh -m dist/flists/firmware-latest.fl --tree
- Inspect + mount test to a temp dir:
  - sudo scripts/rfs/verify-flist.sh -m dist/flists/modules-6.12.44-Zero-OS.fl --mount

## Additional blob store backends (design)

This extends the existing S3/HTTP approach with a RESP/DB-style backend option for rfs blob storage. It is a design-only addition; CLI and scripts will be extended in a follow-up.

Scope
- Keep S3 flow intact via [scripts/rfs/common.sh](scripts/rfs/common.sh:137), [scripts/rfs/common.sh](scripts/rfs/common.sh:385), and [scripts/rfs/common.sh](scripts/rfs/common.sh:494).
- Introduce RESP URIs that can be encoded in config and, later, resolved by rfs or a thin uploader shim invoked by:
  - [scripts/rfs/pack-modules.sh](scripts/rfs/pack-modules.sh:1)
  - [scripts/rfs/pack-firmware.sh](scripts/rfs/pack-firmware.sh:1)

URI schemes (draft)
- resp://host:port/db?prefix=blobs
- resp+tls://host:port/db?prefix=blobs&ca=/etc/ssl/certs/ca.pem
- resp+sentinel://sentinelHost:26379/mymaster?prefix=blobs
- Credentials may be provided via URI userinfo or config (recommended: config only).

Operations (minimal set)
- PUT blob: write content-addressed key (e.g., prefix/ab/cd/hash)
- GET blob: fetch by exact key
- Exists/HEAD: presence test by key
- Optional batching: pipelined MGET for prefetch

Config keys (see example additions in config/rfs.conf.example)
- RESP_ENDPOINT (host:port), RESP_DB (integer), RESP_PREFIX (path namespace)
- RESP_USERNAME/RESP_PASSWORD (optional), RESP_TLS=0/1 (+ RESP_CA if needed)
- RESP_SENTINEL and RESP_MASTER for sentinel deployments

Manifests and routes
- Keep S3 store in flist stores table (fallback) while enabling route.url patching to HTTP/S3 for read-only access:
  - Patch stores table as today via [scripts/rfs/common.sh](scripts/rfs/common.sh:385)
  - Patch route.url as today via [scripts/rfs/common.sh](scripts/rfs/common.sh:494)
- RESP may be used primarily for pack-time blob uploads or as an additional store the CLI can consume later.

Security
- Do not embed write credentials in manifests.
- Read-only credentials may be embedded in route.url if required, mirroring S3 pattern.

Next steps
- Implement RESP uploader shim called from pack scripts; keep the CLI S3 flow unchanged.
- Extend config loader in [scripts/rfs/common.sh](scripts/rfs/common.sh:82) to parse RESP_* variables.
- Add verification routines to sanity-check connectivity before pack.