Jan De Landtsheer
afd4f4c6f9
Summary
- Implemented plain S3-only flist workflow (no web endpoint). rfs pack uploads blobs using write creds; flist route.url is patched to embed read-only S3 credentials so rfs mount reads directly from S3.
Changes
1) New RFS tooling (scripts/rfs/)
- common.sh:
- Compute FULL_KERNEL_VERSION from configs (no uname).
- Load S3 config and construct pack store URI.
- Build read-only S3 route URL and patch flist (sqlite).
- Helpers to locate modules/firmware trees and rfs binary.
- pack-modules.sh:
- Pack /lib/modules/<FULL_KERNEL_VERSION> to dist/flists/modules-<FULL_KERNEL_VERSION>.fl
- Patch flist route to s3://READ:READ@host:port/ROUTE_PATH?region=ROUTE_REGION (default /blobs, garage).
- Optional upload of .fl using MinIO client (mcli/mc).
- pack-firmware.sh:
- Source firmware from $PROJECT_ROOT/firmware (fallback to initramfs/lib/firmware).
- Pack to dist/flists/firmware-<TAG_OR_DATE>.fl (FIRMWARE_TAG or YYYYMMDD).
- Patch flist route to read-only S3; optional .fl upload via mcli/mc.
- verify-flist.sh:
- rfs flist inspect/tree; optional mount test (best effort).
- patch-stores.sh:
- Helper to patch stores (kept though not used by default).
2) Dev-container (Dockerfile)
- Added sqlite and MinIO client package for manifest patching/upload (expect mcli binary at runtime; scripts support both mcli/mc).
- Retains rustup and musl target for building rfs/zinit/mycelium.
3) Config and examples
- config/rfs.conf.example:
- S3_ENDPOINT/S3_REGION/S3_BUCKET/S3_PREFIX
- S3_ACCESS_KEY/S3_SECRET_KEY (write)
- READ_ACCESS_KEY/READ_SECRET_KEY (read-only)
- ROUTE_ENDPOINT (defaults to S3_ENDPOINT), ROUTE_PATH=/blobs, ROUTE_REGION=garage
- MANIFESTS_SUBPATH, UPLOAD_MANIFESTS (mcli upload optional)
- config/rfs.conf updated by user with real values (not committed here; example included).
- config/modules.conf minor tweak (staged).
4) Zinit mount scripts (config/zinit/init/)
- firmware.sh:
- Mounts firmware-latest.fl over /usr/lib/firmware using rfs mount (env override FIRMWARE_FLIST supported).
- modules.sh:
- Mounts modules-$(uname -r).fl over /lib/modules/$(uname -r) (env override MODULES_FLIST supported).
- Both skip if target already mounted and respect RFS_BIN env.
5) Documentation
- docs/rfs-flists.md:
- End-to-end flow, S3-only route URL patching, mcli upload notes.
- docs/review-rfs-integration.md:
- Integration points, build flow, and post-build standalone usage.
- docs/depmod-behavior.md:
- depmod reads .modinfo; recommend prebuilt modules.*(.bin); use depmod -A only on mismatch.
6) Utility
- scripts/functionlist.md synced with current functions.
Behavioral details
- Pack (write):
s3://S3_ACCESS_KEY:S3_SECRET_KEY@HOST:PORT/S3_BUCKET/S3_PREFIX?region=REGION
- Flist route (read, post-patch):
s3://READ_ACCESS_KEY:READ_SECRET_KEY@HOST:PORT/ROUTE_PATH?region=ROUTE_REGION
Defaults: ROUTE_PATH=/blobs, ROUTE_REGION=garage; ROUTE_ENDPOINT derived from S3_ENDPOINT if not set.
Runtime mount examples
- Modules:
rfs mount -m dist/flists/modules-6.12.44-Zero-OS.fl /lib/modules/6.12.44-Zero-OS
- Firmware:
rfs mount -m dist/flists/firmware-YYYYMMDD.fl /usr/lib/firmware
Notes
- FUSE policy: If "allow_other" error occurs, enable user_allow_other in /etc/fuse.conf or run mounts as root.
- WEB_ENDPOINT rewrite is disabled by default (set WEB_ENDPOINT=""). Plain S3 route is embedded in flists.
- MinIO client binary in dev-container is mcli; scripts support mcli (preferred) and mc (fallback).
Files added/modified
- Added: scripts/rfs/{common.sh,pack-modules.sh,pack-firmware.sh,verify-flist.sh,patch-stores.sh}
- Added: config/zinit/init/{firmware.sh,modules.sh}
- Added: docs/{rfs-flists.md,review-rfs-integration.md,depmod-behavior.md}
- Added: config/rfs.conf.example
- Modified: Dockerfile, scripts/functionlist.md, config/modules.conf, config/zinit/sshd-setup.yaml, .gitignore
72 lines
5.2 KiB
Markdown
72 lines
5.2 KiB
Markdown
# depmod behavior, impact on lazy-mounted module stores, and flist store rewriting
|
||
|
||
Summary (short answer)
|
||
- depmod builds the modules dependency/alias databases by scanning the modules tree under /lib/modules/<kernel>. It reads metadata from each .ko file (.modinfo section) to generate:
|
||
- modules.dep(.bin), modules.alias(.bin), modules.symbols(.bin), modules.devname, modules.order, etc.
|
||
- It does not load modules; it opens many files for small reads. On a lazy store, the first depmod run can trigger many object fetches.
|
||
- If modules metadata files are already present and consistent (as produced during build), modprobe can work without re-running depmod. Use depmod -A (update only) or skip depmod entirely if timestamps and paths are unchanged.
|
||
- For private S3 (garage) without anonymous read, post-process the .fl manifest to replace the store URI with your HTTPS web endpoint for that bucket, so runtime mounts fetch over the web endpoint instead of signed S3.
|
||
|
||
Details
|
||
|
||
1) What depmod actually reads/builds
|
||
- Inputs scanned under /lib/modules/<kernel>:
|
||
- .ko files: depmod reads ELF .modinfo to collect depends=, alias=, vermagic, etc. It does not execute or load modules.
|
||
- modules.builtin and modules.builtin.modinfo: indicate built-in drivers so they are excluded from dep graph.
|
||
- Optional flags:
|
||
- depmod -F <System.map> and -E <Module.symvers> allow symbol/CRC checks; these are typically not required on target systems for generating dependency/alias maps.
|
||
- Outputs (consumed by modprobe/kmod):
|
||
- modules.dep and modules.dep.bin: dependency lists and fast index
|
||
- modules.alias and modules.alias.bin: modalias to module name mapping
|
||
- modules.symbols(.bin), modules.devname, modules.order, etc.
|
||
|
||
Key property: depmod’s default operation opens many .ko files to read .modinfo, which on a lazy FUSE-backed store causes many small reads.
|
||
|
||
2) Recommended strategy with lazy flists
|
||
- Precompute metadata during build:
|
||
- In the dev container, your pipeline already runs depmod (see [kernel_build_modules()](scripts/lib/kernel.sh:228)). Ensure the resulting metadata files in /lib/modules/<kernel> are included in the modules flist.
|
||
- At runtime after overmounting the modules flist:
|
||
- Option A: Do nothing. If your path is the same (/lib/modules/<kernel>), modprobe will use the precomputed .bin maps and will not need to rescan .ko files. This minimizes object fetches (only when a module is actually loaded).
|
||
- Option B: Run depmod -A <kernel> (update only if any .ko newer than modules.dep). This typically performs stats on files and only rebuilds if needed, avoiding a full read of all .ko files.
|
||
- Option C: Run depmod -a only if you changed the module set or path layout. Expect many small reads on first run.
|
||
|
||
3) Firmware implications
|
||
- No depmod impact, but udev coldplug will probe devices. Keep firmware files accessible via the firmware flist mount (e.g., /usr/lib/firmware).
|
||
- Since firmware loads on-demand by the kernel/driver, the lazy store will fetch only needed blobs.
|
||
|
||
4) Post-processing .fl to use a web endpoint (garage S3 private)
|
||
- Goal: Pack/upload blobs to private S3 using credentials, but ship a manifest (.fl) that references a public HTTPS endpoint (or authenticated gateway) that your rfs mount can fetch from without S3 signing.
|
||
- Approach A: Use rfs CLI (if supported) to edit store URIs within the manifest.
|
||
- Example (conceptual): rfs flist edit-store -m dist/flists/modules-...fl --set https://web.example.com/bucket/prefix
|
||
- Approach B: Use sqlite3 to patch the manifest directly (the .fl is sqlite):
|
||
- Inspect stores:
|
||
- sqlite3 dist/flists/modules-...fl "SELECT id, uri FROM stores;"
|
||
- Replace s3 store with web endpoint:
|
||
- sqlite3 dist/flists/modules-...fl "UPDATE stores SET uri='https://web.example.com/bucket/prefix' WHERE uri LIKE 's3://%';"
|
||
- Validate:
|
||
- rfs flist inspect dist/flists/modules-...fl
|
||
- Notes:
|
||
- The web endpoint you provide must serve the same content-addressed paths that rfs expects. Confirm the object path layout (e.g., /bucket/prefix/ab/cd/abcdef...).
|
||
- You can maintain multiple store rows to provide fallbacks (if rfs supports trying multiple stores).
|
||
|
||
5) Suggested runtime sequence after overmount (with precomputed metadata)
|
||
- Mount modules flist read-only at /lib/modules/<kernel>.
|
||
- Optionally depmod -A <kernel> (cheap; no full scan).
|
||
- udevadm control --reload; udevadm trigger --action=add; udevadm settle
|
||
- Load required baseline modules (stage1) if needed; the lazy store ensures only requested .ko files are fetched.
|
||
|
||
6) Practical checklist for our scripts
|
||
- Ensure pack-modules includes:
|
||
- /lib/modules/<kernel>/*.ko*
|
||
- All modules.* metadata files (dep, alias, symbols, order, builtin, *.bin)
|
||
- After pack completes and blobs are uploaded to S3, patch the .fl manifest’s stores table to the public HTTPS endpoint of your garage bucket/web gateway.
|
||
- Provide verify utilities:
|
||
- rfs flist inspect/tree
|
||
- Optional local mount test against the web endpoint referenced in the manifest.
|
||
|
||
Appendix: Commands and flags
|
||
- Generate/update metadata (build-time): depmod -a <kernel>
|
||
- Fast update at boot: depmod -A <kernel> # only if newer/changed
|
||
- Chroot/base path (useful for initramfs image pathing): depmod -b <base> -a <kernel>
|
||
- Modprobe uses *.bin maps when present, which avoids parsing large text maps on every lookup.
|