feat(rfs): flist pack to S3 + read-only route embedding + zinit mount scripts; docs; dev-container tooling

Summary
- Implemented plain S3-only flist workflow (no web endpoint). rfs pack uploads blobs using write creds; flist route.url is patched to embed read-only S3 credentials so rfs mount reads directly from S3.

Changes
1) New RFS tooling (scripts/rfs/)
   - common.sh:
     - Compute FULL_KERNEL_VERSION from configs (no uname).
     - Load S3 config and construct pack store URI.
     - Build read-only S3 route URL and patch flist (sqlite).
     - Helpers to locate modules/firmware trees and rfs binary.
   - pack-modules.sh:
     - Pack /lib/modules/<FULL_KERNEL_VERSION> to dist/flists/modules-<FULL_KERNEL_VERSION>.fl
     - Patch flist route to s3://READ:READ@host:port/ROUTE_PATH?region=ROUTE_REGION (default /blobs, garage).
     - Optional upload of .fl using MinIO client (mcli/mc).
   - pack-firmware.sh:
     - Source firmware from $PROJECT_ROOT/firmware (fallback to initramfs/lib/firmware).
     - Pack to dist/flists/firmware-<TAG_OR_DATE>.fl (FIRMWARE_TAG or YYYYMMDD).
     - Patch flist route to read-only S3; optional .fl upload via mcli/mc.
   - verify-flist.sh:
     - rfs flist inspect/tree; optional mount test (best effort).
   - patch-stores.sh:
     - Helper to patch stores (kept though not used by default).

2) Dev-container (Dockerfile)
   - Added sqlite and MinIO client package for manifest patching/upload (expect mcli binary at runtime; scripts support both mcli/mc).
   - Retains rustup and musl target for building rfs/zinit/mycelium.

3) Config and examples
   - config/rfs.conf.example:
     - S3_ENDPOINT/S3_REGION/S3_BUCKET/S3_PREFIX
     - S3_ACCESS_KEY/S3_SECRET_KEY (write)
     - READ_ACCESS_KEY/READ_SECRET_KEY (read-only)
     - ROUTE_ENDPOINT (defaults to S3_ENDPOINT), ROUTE_PATH=/blobs, ROUTE_REGION=garage
     - MANIFESTS_SUBPATH, UPLOAD_MANIFESTS (mcli upload optional)
   - config/rfs.conf updated by user with real values (not committed here; example included).
   - config/modules.conf minor tweak (staged).

4) Zinit mount scripts (config/zinit/init/)
   - firmware.sh:
     - Mounts firmware-latest.fl over /usr/lib/firmware using rfs mount (env override FIRMWARE_FLIST supported).
   - modules.sh:
     - Mounts modules-$(uname -r).fl over /lib/modules/$(uname -r) (env override MODULES_FLIST supported).
   - Both skip if target already mounted and respect RFS_BIN env.

5) Documentation
   - docs/rfs-flists.md:
     - End-to-end flow, S3-only route URL patching, mcli upload notes.
   - docs/review-rfs-integration.md:
     - Integration points, build flow, and post-build standalone usage.
   - docs/depmod-behavior.md:
     - depmod reads .modinfo; recommend prebuilt modules.*(.bin); use depmod -A only on mismatch.

6) Utility
   - scripts/functionlist.md synced with current functions.

Behavioral details
- Pack (write):
  s3://S3_ACCESS_KEY:S3_SECRET_KEY@HOST:PORT/S3_BUCKET/S3_PREFIX?region=REGION
- Flist route (read, post-patch):
  s3://READ_ACCESS_KEY:READ_SECRET_KEY@HOST:PORT/ROUTE_PATH?region=ROUTE_REGION
  Defaults: ROUTE_PATH=/blobs, ROUTE_REGION=garage; ROUTE_ENDPOINT derived from S3_ENDPOINT if not set.

Runtime mount examples
- Modules:
  rfs mount -m dist/flists/modules-6.12.44-Zero-OS.fl /lib/modules/6.12.44-Zero-OS
- Firmware:
  rfs mount -m dist/flists/firmware-YYYYMMDD.fl /usr/lib/firmware

Notes
- FUSE policy: If "allow_other" error occurs, enable user_allow_other in /etc/fuse.conf or run mounts as root.
- WEB_ENDPOINT rewrite is disabled by default (set WEB_ENDPOINT=""). Plain S3 route is embedded in flists.
- MinIO client binary in dev-container is mcli; scripts support mcli (preferred) and mc (fallback).

Files added/modified
- Added: scripts/rfs/{common.sh,pack-modules.sh,pack-firmware.sh,verify-flist.sh,patch-stores.sh}
- Added: config/zinit/init/{firmware.sh,modules.sh}
- Added: docs/{rfs-flists.md,review-rfs-integration.md,depmod-behavior.md}
- Added: config/rfs.conf.example
- Modified: Dockerfile, scripts/functionlist.md, config/modules.conf, config/zinit/sshd-setup.yaml, .gitignore
This commit is contained in:
2025-09-08 22:51:53 +02:00
parent acc2fce84a
commit afd4f4c6f9
16 changed files with 1344 additions and 167 deletions

65
scripts/rfs/verify-flist.sh Executable file
View File

@@ -0,0 +1,65 @@
#!/bin/bash
# Verify an RFS flist (.fl) by inspecting, listing tree, and optional mount test.
# Usage:
# ./scripts/rfs/verify-flist.sh /path/to/foo.fl
# ./scripts/rfs/verify-flist.sh /path/to/foo.fl --mount
#
# Notes:
# - Requires the rfs binary (PATH or components fallback).
# - --mount performs a temporary mount (needs FUSE and privileges).
set -euo pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=/dev/null
source "${HERE}/common.sh"
section() { echo -e "\n==== $* ====\n"; }
if [[ $# -lt 1 ]]; then
echo "Usage: $0 /path/to/foo.fl [--mount]"
exit 1
fi
FL_PATH="$1"
DO_MOUNT="${2:-}"
if [[ ! -f "$FL_PATH" ]]; then
echo "[ERROR] flist not found: ${FL_PATH}" >&2
exit 1
fi
section "Locating rfs binary"
rfs_common_locate_rfs
section "Inspect flist"
safe_execute "${RFS_BIN}" flist inspect "${FL_PATH}" || true
section "Tree (first 100 entries)"
safe_execute "${RFS_BIN}" flist tree "${FL_PATH}" | head -n 100 || true
if [[ "$DO_MOUNT" == "--mount" ]]; then
section "Attempting temporary mount"
MNT="$(mktemp -d /tmp/rfs-mnt-XXXXXX)"
cleanup() {
set +e
if mountpoint -q "${MNT}" 2>/dev/null; then
echo "[INFO] Unmounting ${MNT}"
fusermount -u "${MNT}" 2>/dev/null || umount "${MNT}" 2>/dev/null || true
fi
rmdir "${MNT}" 2>/dev/null || true
}
trap cleanup EXIT INT TERM
echo "[INFO] Mountpoint: ${MNT}"
# Try mount; some environments require sudo or have limited FUSE. Best-effort.
if "${RFS_BIN}" mount -m "${FL_PATH}" "${MNT}"; then
echo "[INFO] Mounted. Listing top-level entries:"
ls -la "${MNT}" | head -n 50 || true
else
echo "[WARN] rfs mount failed (FUSE/permissions?). Skipping mount verification." >&2
fi
fi
section "Done"
echo "[INFO] Verified flist: ${FL_PATH}"