rfs(firmware): pack full Alpine linux-firmware set from container and overmount /lib/firmware
Some checks failed
Build Zero OS Initramfs / build (push) Has been cancelled
Build Zero OS Initramfs / test-matrix (qemu, basic) (push) Has been cancelled
Build Zero OS Initramfs / test-matrix (qemu, serial) (push) Has been cancelled

Change pack source: install all linux-firmware* packages in container and pack from /lib/firmware via [bash.rfs_common_install_all_alpine_firmware_packages()](scripts/rfs/common.sh:290) used by [bash.pack-firmware.sh](scripts/rfs/pack-firmware.sh:21). At runtime, overmount firmware flist on /lib/firmware by updating [sh.firmware.sh](config/zinit/init/firmware.sh:10). Update docs to reflect /lib/firmware mount and new pack strategy.
This commit is contained in:
2025-09-19 08:27:10 +02:00
parent 79ed723303
commit 310e11d2bf
5 changed files with 100 additions and 13 deletions

View File

@@ -1,5 +1,5 @@
#!/bin/sh #!/bin/sh
# rfs mount firmware flist over /usr/lib/firmware (plain S3 route inside the .fl) # rfs mount firmware flist over /lib/firmware (plain S3 route inside the .fl)
# Looks for firmware-latest.fl in known locations; can be overridden via FIRMWARE_FLIST env # Looks for firmware-latest.fl in known locations; can be overridden via FIRMWARE_FLIST env
set -eu set -eu
@@ -7,7 +7,7 @@ set -eu
log() { echo "[rfs-firmware] $*"; } log() { echo "[rfs-firmware] $*"; }
RFS_BIN="${RFS_BIN:-rfs}" RFS_BIN="${RFS_BIN:-rfs}"
TARGET="/usr/lib/firmware" TARGET="/lib/firmware"
BASE_URL="${FLISTS_BASE_URL:-https://zos.grid.tf/store/flists}" BASE_URL="${FLISTS_BASE_URL:-https://zos.grid.tf/store/flists}"
# Allow override via env # Allow override via env

View File

@@ -31,7 +31,7 @@ Key property: depmods default operation opens many .ko files to read .modinfo
- Option C: Run depmod -a only if you changed the module set or path layout. Expect many small reads on first run. - Option C: Run depmod -a only if you changed the module set or path layout. Expect many small reads on first run.
3) Firmware implications 3) Firmware implications
- No depmod impact, but udev coldplug will probe devices. Keep firmware files accessible via the firmware flist mount (e.g., /usr/lib/firmware). - No depmod impact, but udev coldplug will probe devices. Keep firmware files accessible via the firmware flist mount (e.g., /lib/firmware).
- Since firmware loads on-demand by the kernel/driver, the lazy store will fetch only needed blobs. - Since firmware loads on-demand by the kernel/driver, the lazy store will fetch only needed blobs.
4) Post-processing .fl to use a web endpoint (garage S3 private) 4) Post-processing .fl to use a web endpoint (garage S3 private)

View File

@@ -15,9 +15,10 @@ Scope of this change
Inputs Inputs
- Built kernel modules present in the dev-container (from kernel build stages): - Built kernel modules present in the dev-container (from kernel build stages):
- Preferred: /lib/modules/KERNEL_FULL_VERSION - Preferred: /lib/modules/KERNEL_FULL_VERSION
- Firmware tree: - Firmware source for RFS pack:
- Preferred: $PROJECT_ROOT/firmware (prepopulated tree from dev-container: “$root/firmware”) - Install all Alpine linux-firmware* packages into the build container and use /lib/firmware as the source (full set).
- Fallback: initramfs/lib/firmware created by apk install of firmware packages - Initramfs fallback (build-time):
- Selective firmware packages installed by [bash.alpine_install_firmware()](scripts/lib/alpine.sh:392) into initramfs/lib/firmware (kept inside the initramfs).
- Kernel version derivation (never use uname -r in container): - Kernel version derivation (never use uname -r in container):
- Combine KERNEL_VERSION from [config/build.conf](config/build.conf) and LOCALVERSION from [config/kernel.config](config/kernel.config). - Combine KERNEL_VERSION from [config/build.conf](config/build.conf) and LOCALVERSION from [config/kernel.config](config/kernel.config).
- This matches [kernel_get_full_version()](scripts/lib/kernel.sh:14). - This matches [kernel_get_full_version()](scripts/lib/kernel.sh:14).
@@ -71,7 +72,7 @@ Scripts to add (standalone)
Runtime (deferred to a follow-up) Runtime (deferred to a follow-up)
- New zinit units to mount and coldplug: - New zinit units to mount and coldplug:
- Mount firmware flist read-only at /usr/lib/firmware - Mount firmware flist read-only at /lib/firmware
- Mount modules flist at /lib/modules/KERNEL_FULL_VERSION - Mount modules flist at /lib/modules/KERNEL_FULL_VERSION
- Run depmod -a KERNEL_FULL_VERSION - Run depmod -a KERNEL_FULL_VERSION
- udevadm control --reload; udevadm trigger --action=add; udevadm settle - udevadm control --reload; udevadm trigger --action=add; udevadm settle

View File

@@ -287,6 +287,92 @@ rfs_common_validate_modules_metadata() {
fi fi
} }
# -----------------------------------------------------------------------------
# Alpine firmware installation in container (for flist packing)
# -----------------------------------------------------------------------------
# Install all available linux-firmware* packages into the build container root.
# This prepares a full /lib/firmware tree to be packed into an RFS flist.
# Notes:
# - Requires root inside the container (the builder image runs scripts as root).
# - Uses apk search to enumerate all linux-firmware packages; installs them all.
rfs_common_install_all_alpine_firmware_packages() {
log_info "Installing all Alpine linux-firmware* packages into container (/lib/firmware)"
# Ensure apk index is fresh
safe_execute apk update
# Enumerate firmware packages. Output of 'apk search' looks like:
# linux-firmware-rtl_nic-20231030-r0
# Strip version suffix to get the package names acceptable to apk add.
local pkgs_raw
pkgs_raw="$(apk search 'linux-firmware*' 2>/dev/null || true)"
if [[ -z "${pkgs_raw// }" ]]; then
log_warn "No linux-firmware* packages found via apk search"
fi
# Build unique package list without versions
# 1) take first column (package-with-version)
# 2) strip trailing '-<digits...>' (version/revision)
# 3) dedupe
local pkgs=()
while IFS= read -r line; do
[[ -z "${line// }" ]] && continue
local name="${line%% *}"
name="$(echo "$name" | sed -E 's/-[0-9].*$//')"
if [[ -n "$name" ]]; then
pkgs+=("$name")
fi
done <<< "$pkgs_raw"
if [[ ${#pkgs[@]} -eq 0 ]]; then
log_warn "Firmware package list is empty after parsing; attempting base meta-package 'linux-firmware'"
pkgs=("linux-firmware")
fi
# Deduplicate while preserving order
local seen="" final_pkgs=()
for p in "${pkgs[@]}"; do
if ! grep -qx "$p" <<< "$seen"; then
final_pkgs+=("$p")
seen+=$'\n'"$p"
fi
done
log_info "Installing ${#final_pkgs[@]} firmware packages:"
for p in "${final_pkgs[@]}"; do
log_info " - $p"
done
# Install all firmware packages; allow some failures (not all subpackages exist on all arches)
local failed=()
for p in "${final_pkgs[@]}"; do
if apk add --no-cache "$p" >/dev/null 2>&1; then
log_debug "Installed: $p"
else
log_warn "Failed to install: $p"
failed+=("$p")
fi
done
# Quick check that /lib/firmware exists and is populated
if [[ -d "/lib/firmware" ]]; then
local cnt
cnt=$(find /lib/firmware -type f | wc -l || echo 0)
log_info "/lib/firmware population: ${cnt} files"
if [[ "$cnt" -eq 0 ]]; then
log_warn "/lib/firmware exists but is empty after installation"
fi
else
log_error "/lib/firmware directory not found after firmware installation"
return 1
fi
if [[ ${#failed[@]} -gt 0 ]]; then
log_warn "Some firmware packages failed to install (${#failed[@]} failures); proceeding with available set"
fi
return 0
}
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
# Manifest patching (sqlite .fl) # Manifest patching (sqlite .fl)
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------

View File

@@ -1,10 +1,8 @@
#!/bin/bash #!/bin/bash
# Pack firmware tree into an RFS flist and patch manifest stores for Garage web endpoint. # Pack firmware tree into an RFS flist and patch manifest stores for Garage web endpoint.
# - Computes FULL_KERNEL_VERSION from configs (not strictly needed for firmware, but kept uniform) # - Computes FULL_KERNEL_VERSION from configs (not strictly needed for firmware, but kept uniform)
# - Selects firmware directory with priority: # - Installs all Alpine linux-firmware* packages into the build container (/lib/firmware)
# 1) $PROJECT_ROOT/firmware # - Packs from container's /lib/firmware into an RFS flist (reproducible, full set)
# 2) $PROJECT_ROOT/initramfs/lib/firmware
# 3) /lib/firmware
# - Manifest name: firmware-<FIRMWARE_TAG or YYYYMMDD>.fl # - Manifest name: firmware-<FIRMWARE_TAG or YYYYMMDD>.fl
# - Uploads blobs to S3 (Garage) via rfs store URI # - Uploads blobs to S3 (Garage) via rfs store URI
# - Patches .fl sqlite stores table to use WEB_ENDPOINT for read-only fetches # - Patches .fl sqlite stores table to use WEB_ENDPOINT for read-only fetches
@@ -25,8 +23,10 @@ rfs_common_load_rfs_s3_config
rfs_common_build_s3_store_uri rfs_common_build_s3_store_uri
rfs_common_locate_rfs rfs_common_locate_rfs
section "Locating firmware directory" section "Installing full Alpine firmware set in container (source: /lib/firmware)"
rfs_common_locate_firmware_dir rfs_common_install_all_alpine_firmware_packages
export FIRMWARE_DIR="/lib/firmware"
log_info "Using firmware source dir: ${FIRMWARE_DIR}"
TAG="$(rfs_common_firmware_tag)" TAG="$(rfs_common_firmware_tag)"
MANIFEST_NAME="firmware-${TAG}.fl" MANIFEST_NAME="firmware-${TAG}.fl"