forked from tfgrid/zosbuilder
rfs(firmware): pack full Alpine linux-firmware set from container and overmount /lib/firmware
Change pack source: install all linux-firmware* packages in container and pack from /lib/firmware via [bash.rfs_common_install_all_alpine_firmware_packages()](scripts/rfs/common.sh:290) used by [bash.pack-firmware.sh](scripts/rfs/pack-firmware.sh:21). At runtime, overmount firmware flist on /lib/firmware by updating [sh.firmware.sh](config/zinit/init/firmware.sh:10). Update docs to reflect /lib/firmware mount and new pack strategy.
This commit is contained in:
@@ -1,5 +1,5 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# rfs mount firmware flist over /usr/lib/firmware (plain S3 route inside the .fl)
|
# rfs mount firmware flist over /lib/firmware (plain S3 route inside the .fl)
|
||||||
# Looks for firmware-latest.fl in known locations; can be overridden via FIRMWARE_FLIST env
|
# Looks for firmware-latest.fl in known locations; can be overridden via FIRMWARE_FLIST env
|
||||||
|
|
||||||
set -eu
|
set -eu
|
||||||
@@ -7,7 +7,7 @@ set -eu
|
|||||||
log() { echo "[rfs-firmware] $*"; }
|
log() { echo "[rfs-firmware] $*"; }
|
||||||
|
|
||||||
RFS_BIN="${RFS_BIN:-rfs}"
|
RFS_BIN="${RFS_BIN:-rfs}"
|
||||||
TARGET="/usr/lib/firmware"
|
TARGET="/lib/firmware"
|
||||||
BASE_URL="${FLISTS_BASE_URL:-https://zos.grid.tf/store/flists}"
|
BASE_URL="${FLISTS_BASE_URL:-https://zos.grid.tf/store/flists}"
|
||||||
|
|
||||||
# Allow override via env
|
# Allow override via env
|
||||||
|
|||||||
@@ -31,7 +31,7 @@ Key property: depmod’s default operation opens many .ko files to read .modinfo
|
|||||||
- Option C: Run depmod -a only if you changed the module set or path layout. Expect many small reads on first run.
|
- Option C: Run depmod -a only if you changed the module set or path layout. Expect many small reads on first run.
|
||||||
|
|
||||||
3) Firmware implications
|
3) Firmware implications
|
||||||
- No depmod impact, but udev coldplug will probe devices. Keep firmware files accessible via the firmware flist mount (e.g., /usr/lib/firmware).
|
- No depmod impact, but udev coldplug will probe devices. Keep firmware files accessible via the firmware flist mount (e.g., /lib/firmware).
|
||||||
- Since firmware loads on-demand by the kernel/driver, the lazy store will fetch only needed blobs.
|
- Since firmware loads on-demand by the kernel/driver, the lazy store will fetch only needed blobs.
|
||||||
|
|
||||||
4) Post-processing .fl to use a web endpoint (garage S3 private)
|
4) Post-processing .fl to use a web endpoint (garage S3 private)
|
||||||
|
|||||||
@@ -15,9 +15,10 @@ Scope of this change
|
|||||||
Inputs
|
Inputs
|
||||||
- Built kernel modules present in the dev-container (from kernel build stages):
|
- Built kernel modules present in the dev-container (from kernel build stages):
|
||||||
- Preferred: /lib/modules/KERNEL_FULL_VERSION
|
- Preferred: /lib/modules/KERNEL_FULL_VERSION
|
||||||
- Firmware tree:
|
- Firmware source for RFS pack:
|
||||||
- Preferred: $PROJECT_ROOT/firmware (prepopulated tree from dev-container: “$root/firmware”)
|
- Install all Alpine linux-firmware* packages into the build container and use /lib/firmware as the source (full set).
|
||||||
- Fallback: initramfs/lib/firmware created by apk install of firmware packages
|
- Initramfs fallback (build-time):
|
||||||
|
- Selective firmware packages installed by [bash.alpine_install_firmware()](scripts/lib/alpine.sh:392) into initramfs/lib/firmware (kept inside the initramfs).
|
||||||
- Kernel version derivation (never use uname -r in container):
|
- Kernel version derivation (never use uname -r in container):
|
||||||
- Combine KERNEL_VERSION from [config/build.conf](config/build.conf) and LOCALVERSION from [config/kernel.config](config/kernel.config).
|
- Combine KERNEL_VERSION from [config/build.conf](config/build.conf) and LOCALVERSION from [config/kernel.config](config/kernel.config).
|
||||||
- This matches [kernel_get_full_version()](scripts/lib/kernel.sh:14).
|
- This matches [kernel_get_full_version()](scripts/lib/kernel.sh:14).
|
||||||
@@ -71,7 +72,7 @@ Scripts to add (standalone)
|
|||||||
|
|
||||||
Runtime (deferred to a follow-up)
|
Runtime (deferred to a follow-up)
|
||||||
- New zinit units to mount and coldplug:
|
- New zinit units to mount and coldplug:
|
||||||
- Mount firmware flist read-only at /usr/lib/firmware
|
- Mount firmware flist read-only at /lib/firmware
|
||||||
- Mount modules flist at /lib/modules/KERNEL_FULL_VERSION
|
- Mount modules flist at /lib/modules/KERNEL_FULL_VERSION
|
||||||
- Run depmod -a KERNEL_FULL_VERSION
|
- Run depmod -a KERNEL_FULL_VERSION
|
||||||
- udevadm control --reload; udevadm trigger --action=add; udevadm settle
|
- udevadm control --reload; udevadm trigger --action=add; udevadm settle
|
||||||
|
|||||||
@@ -287,6 +287,92 @@ rfs_common_validate_modules_metadata() {
|
|||||||
fi
|
fi
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------
|
||||||
|
# Alpine firmware installation in container (for flist packing)
|
||||||
|
# -----------------------------------------------------------------------------
|
||||||
|
# Install all available linux-firmware* packages into the build container root.
|
||||||
|
# This prepares a full /lib/firmware tree to be packed into an RFS flist.
|
||||||
|
# Notes:
|
||||||
|
# - Requires root inside the container (the builder image runs scripts as root).
|
||||||
|
# - Uses apk search to enumerate all linux-firmware packages; installs them all.
|
||||||
|
rfs_common_install_all_alpine_firmware_packages() {
|
||||||
|
log_info "Installing all Alpine linux-firmware* packages into container (/lib/firmware)"
|
||||||
|
# Ensure apk index is fresh
|
||||||
|
safe_execute apk update
|
||||||
|
|
||||||
|
# Enumerate firmware packages. Output of 'apk search' looks like:
|
||||||
|
# linux-firmware-rtl_nic-20231030-r0
|
||||||
|
# Strip version suffix to get the package names acceptable to apk add.
|
||||||
|
local pkgs_raw
|
||||||
|
pkgs_raw="$(apk search 'linux-firmware*' 2>/dev/null || true)"
|
||||||
|
|
||||||
|
if [[ -z "${pkgs_raw// }" ]]; then
|
||||||
|
log_warn "No linux-firmware* packages found via apk search"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Build unique package list without versions
|
||||||
|
# 1) take first column (package-with-version)
|
||||||
|
# 2) strip trailing '-<digits...>' (version/revision)
|
||||||
|
# 3) dedupe
|
||||||
|
local pkgs=()
|
||||||
|
while IFS= read -r line; do
|
||||||
|
[[ -z "${line// }" ]] && continue
|
||||||
|
local name="${line%% *}"
|
||||||
|
name="$(echo "$name" | sed -E 's/-[0-9].*$//')"
|
||||||
|
if [[ -n "$name" ]]; then
|
||||||
|
pkgs+=("$name")
|
||||||
|
fi
|
||||||
|
done <<< "$pkgs_raw"
|
||||||
|
|
||||||
|
if [[ ${#pkgs[@]} -eq 0 ]]; then
|
||||||
|
log_warn "Firmware package list is empty after parsing; attempting base meta-package 'linux-firmware'"
|
||||||
|
pkgs=("linux-firmware")
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Deduplicate while preserving order
|
||||||
|
local seen="" final_pkgs=()
|
||||||
|
for p in "${pkgs[@]}"; do
|
||||||
|
if ! grep -qx "$p" <<< "$seen"; then
|
||||||
|
final_pkgs+=("$p")
|
||||||
|
seen+=$'\n'"$p"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
log_info "Installing ${#final_pkgs[@]} firmware packages:"
|
||||||
|
for p in "${final_pkgs[@]}"; do
|
||||||
|
log_info " - $p"
|
||||||
|
done
|
||||||
|
|
||||||
|
# Install all firmware packages; allow some failures (not all subpackages exist on all arches)
|
||||||
|
local failed=()
|
||||||
|
for p in "${final_pkgs[@]}"; do
|
||||||
|
if apk add --no-cache "$p" >/dev/null 2>&1; then
|
||||||
|
log_debug "Installed: $p"
|
||||||
|
else
|
||||||
|
log_warn "Failed to install: $p"
|
||||||
|
failed+=("$p")
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
# Quick check that /lib/firmware exists and is populated
|
||||||
|
if [[ -d "/lib/firmware" ]]; then
|
||||||
|
local cnt
|
||||||
|
cnt=$(find /lib/firmware -type f | wc -l || echo 0)
|
||||||
|
log_info "/lib/firmware population: ${cnt} files"
|
||||||
|
if [[ "$cnt" -eq 0 ]]; then
|
||||||
|
log_warn "/lib/firmware exists but is empty after installation"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
log_error "/lib/firmware directory not found after firmware installation"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ ${#failed[@]} -gt 0 ]]; then
|
||||||
|
log_warn "Some firmware packages failed to install (${#failed[@]} failures); proceeding with available set"
|
||||||
|
fi
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
# -----------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------
|
||||||
# Manifest patching (sqlite .fl)
|
# Manifest patching (sqlite .fl)
|
||||||
# -----------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------
|
||||||
|
|||||||
@@ -1,10 +1,8 @@
|
|||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
# Pack firmware tree into an RFS flist and patch manifest stores for Garage web endpoint.
|
# Pack firmware tree into an RFS flist and patch manifest stores for Garage web endpoint.
|
||||||
# - Computes FULL_KERNEL_VERSION from configs (not strictly needed for firmware, but kept uniform)
|
# - Computes FULL_KERNEL_VERSION from configs (not strictly needed for firmware, but kept uniform)
|
||||||
# - Selects firmware directory with priority:
|
# - Installs all Alpine linux-firmware* packages into the build container (/lib/firmware)
|
||||||
# 1) $PROJECT_ROOT/firmware
|
# - Packs from container's /lib/firmware into an RFS flist (reproducible, full set)
|
||||||
# 2) $PROJECT_ROOT/initramfs/lib/firmware
|
|
||||||
# 3) /lib/firmware
|
|
||||||
# - Manifest name: firmware-<FIRMWARE_TAG or YYYYMMDD>.fl
|
# - Manifest name: firmware-<FIRMWARE_TAG or YYYYMMDD>.fl
|
||||||
# - Uploads blobs to S3 (Garage) via rfs store URI
|
# - Uploads blobs to S3 (Garage) via rfs store URI
|
||||||
# - Patches .fl sqlite stores table to use WEB_ENDPOINT for read-only fetches
|
# - Patches .fl sqlite stores table to use WEB_ENDPOINT for read-only fetches
|
||||||
@@ -25,8 +23,10 @@ rfs_common_load_rfs_s3_config
|
|||||||
rfs_common_build_s3_store_uri
|
rfs_common_build_s3_store_uri
|
||||||
rfs_common_locate_rfs
|
rfs_common_locate_rfs
|
||||||
|
|
||||||
section "Locating firmware directory"
|
section "Installing full Alpine firmware set in container (source: /lib/firmware)"
|
||||||
rfs_common_locate_firmware_dir
|
rfs_common_install_all_alpine_firmware_packages
|
||||||
|
export FIRMWARE_DIR="/lib/firmware"
|
||||||
|
log_info "Using firmware source dir: ${FIRMWARE_DIR}"
|
||||||
|
|
||||||
TAG="$(rfs_common_firmware_tag)"
|
TAG="$(rfs_common_firmware_tag)"
|
||||||
MANIFEST_NAME="firmware-${TAG}.fl"
|
MANIFEST_NAME="firmware-${TAG}.fl"
|
||||||
|
|||||||
Reference in New Issue
Block a user