tfgrid/zosbuilder
11
0
Fork 1
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
652d38abb1ce41386dd173457e58d7d8aab3ce4a
zosbuilder/scripts/functionlist.md
Jan De Landtsheer afd4f4c6f9 feat(rfs): flist pack to S3 + read-only route embedding + zinit mount scripts; docs; dev-container tooling 
Summary
- Implemented plain S3-only flist workflow (no web endpoint). rfs pack uploads blobs using write creds; flist route.url is patched to embed read-only S3 credentials so rfs mount reads directly from S3.

Changes
1) New RFS tooling (scripts/rfs/)
   - common.sh:
     - Compute FULL_KERNEL_VERSION from configs (no uname).
     - Load S3 config and construct pack store URI.
     - Build read-only S3 route URL and patch flist (sqlite).
     - Helpers to locate modules/firmware trees and rfs binary.
   - pack-modules.sh:
     - Pack /lib/modules/<FULL_KERNEL_VERSION> to dist/flists/modules-<FULL_KERNEL_VERSION>.fl
     - Patch flist route to s3://READ:READ@host:port/ROUTE_PATH?region=ROUTE_REGION (default /blobs, garage).
     - Optional upload of .fl using MinIO client (mcli/mc).
   - pack-firmware.sh:
     - Source firmware from $PROJECT_ROOT/firmware (fallback to initramfs/lib/firmware).
     - Pack to dist/flists/firmware-<TAG_OR_DATE>.fl (FIRMWARE_TAG or YYYYMMDD).
     - Patch flist route to read-only S3; optional .fl upload via mcli/mc.
   - verify-flist.sh:
     - rfs flist inspect/tree; optional mount test (best effort).
   - patch-stores.sh:
     - Helper to patch stores (kept though not used by default).

2) Dev-container (Dockerfile)
   - Added sqlite and MinIO client package for manifest patching/upload (expect mcli binary at runtime; scripts support both mcli/mc).
   - Retains rustup and musl target for building rfs/zinit/mycelium.

3) Config and examples
   - config/rfs.conf.example:
     - S3_ENDPOINT/S3_REGION/S3_BUCKET/S3_PREFIX
     - S3_ACCESS_KEY/S3_SECRET_KEY (write)
     - READ_ACCESS_KEY/READ_SECRET_KEY (read-only)
     - ROUTE_ENDPOINT (defaults to S3_ENDPOINT), ROUTE_PATH=/blobs, ROUTE_REGION=garage
     - MANIFESTS_SUBPATH, UPLOAD_MANIFESTS (mcli upload optional)
   - config/rfs.conf updated by user with real values (not committed here; example included).
   - config/modules.conf minor tweak (staged).

4) Zinit mount scripts (config/zinit/init/)
   - firmware.sh:
     - Mounts firmware-latest.fl over /usr/lib/firmware using rfs mount (env override FIRMWARE_FLIST supported).
   - modules.sh:
     - Mounts modules-$(uname -r).fl over /lib/modules/$(uname -r) (env override MODULES_FLIST supported).
   - Both skip if target already mounted and respect RFS_BIN env.

5) Documentation
   - docs/rfs-flists.md:
     - End-to-end flow, S3-only route URL patching, mcli upload notes.
   - docs/review-rfs-integration.md:
     - Integration points, build flow, and post-build standalone usage.
   - docs/depmod-behavior.md:
     - depmod reads .modinfo; recommend prebuilt modules.*(.bin); use depmod -A only on mismatch.

6) Utility
   - scripts/functionlist.md synced with current functions.

Behavioral details
- Pack (write):
  s3://S3_ACCESS_KEY:S3_SECRET_KEY@HOST:PORT/S3_BUCKET/S3_PREFIX?region=REGION
- Flist route (read, post-patch):
  s3://READ_ACCESS_KEY:READ_SECRET_KEY@HOST:PORT/ROUTE_PATH?region=ROUTE_REGION
  Defaults: ROUTE_PATH=/blobs, ROUTE_REGION=garage; ROUTE_ENDPOINT derived from S3_ENDPOINT if not set.

Runtime mount examples
- Modules:
  rfs mount -m dist/flists/modules-6.12.44-Zero-OS.fl /lib/modules/6.12.44-Zero-OS
- Firmware:
  rfs mount -m dist/flists/firmware-YYYYMMDD.fl /usr/lib/firmware

Notes
- FUSE policy: If "allow_other" error occurs, enable user_allow_other in /etc/fuse.conf or run mounts as root.
- WEB_ENDPOINT rewrite is disabled by default (set WEB_ENDPOINT=""). Plain S3 route is embedded in flists.
- MinIO client binary in dev-container is mcli; scripts support mcli (preferred) and mc (fallback).

Files added/modified
- Added: scripts/rfs/{common.sh,pack-modules.sh,pack-firmware.sh,verify-flist.sh,patch-stores.sh}
- Added: config/zinit/init/{firmware.sh,modules.sh}
- Added: docs/{rfs-flists.md,review-rfs-integration.md,depmod-behavior.md}
- Added: config/rfs.conf.example
- Modified: Dockerfile, scripts/functionlist.md, config/modules.conf, config/zinit/sshd-setup.yaml, .gitignore
2025-09-08 22:51:53 +02:00

110 lines
5.6 KiB
Markdown
Raw Blame History

# Function List - scripts/lib Library
This document lists all functions currently defined under [scripts/lib](scripts/lib) with their source locations.
## alpine.sh - Alpine Linux operations
File: [scripts/lib/alpine.sh](scripts/lib/alpine.sh)
- [alpine_extract_miniroot()](scripts/lib/alpine.sh:14) - Download and extract Alpine miniroot
- [alpine_setup_chroot()](scripts/lib/alpine.sh:70) - Setup chroot mounts and resolv.conf
- [alpine_cleanup_chroot()](scripts/lib/alpine.sh:115) - Unmount chroot mounts
- [alpine_install_packages()](scripts/lib/alpine.sh:142) - Install packages from packages.list
- [alpine_aggressive_cleanup()](scripts/lib/alpine.sh:211) - Reduce image size by removing docs/locales/etc
- [alpine_configure_repos()](scripts/lib/alpine.sh:321) - Configure APK repositories
- [alpine_configure_system()](scripts/lib/alpine.sh:339) - Configure hostname, hosts, timezone, profile
- [alpine_install_firmware()](scripts/lib/alpine.sh:392) - Install required firmware packages
## common.sh - Core utilities
File: [scripts/lib/common.sh](scripts/lib/common.sh)
- [log_info()](scripts/lib/common.sh:31)
- [log_warn()](scripts/lib/common.sh:36)
- [log_error()](scripts/lib/common.sh:41)
- [log_debug()](scripts/lib/common.sh:46)
- [safe_execute()](scripts/lib/common.sh:54)
- [section_header()](scripts/lib/common.sh:79)
- [command_exists()](scripts/lib/common.sh:89)
- [in_container()](scripts/lib/common.sh:94)
- [check_dependencies()](scripts/lib/common.sh:99)
- [safe_mkdir()](scripts/lib/common.sh:142)
- [safe_rmdir()](scripts/lib/common.sh:149)
- [safe_copy()](scripts/lib/common.sh:158)
- [is_absolute_path()](scripts/lib/common.sh:166)
- [resolve_path()](scripts/lib/common.sh:171)
- [get_file_size()](scripts/lib/common.sh:181)
- [wait_for_file()](scripts/lib/common.sh:191)
- [cleanup_on_exit()](scripts/lib/common.sh:205)
## components.sh - Component management
File: [scripts/lib/components.sh](scripts/lib/components.sh)
- [components_parse_sources_conf()](scripts/lib/components.sh:13)
- [components_download_git()](scripts/lib/components.sh:72)
- [components_download_release()](scripts/lib/components.sh:104)
- [components_process_extra_options()](scripts/lib/components.sh:144)
- [components_build_component()](scripts/lib/components.sh:183)
- [components_setup_rust_env()](scripts/lib/components.sh:217)
- [build_zinit()](scripts/lib/components.sh:252)
- [build_rfs()](scripts/lib/components.sh:299)
- [build_mycelium()](scripts/lib/components.sh:346)
- [install_rfs()](scripts/lib/components.sh:386)
- [install_corex()](scripts/lib/components.sh:409)
- [components_verify_installation()](scripts/lib/components.sh:436)
- [components_cleanup()](scripts/lib/components.sh:472)
## docker.sh - Container runtime management
File: [scripts/lib/docker.sh](scripts/lib/docker.sh)
- [docker_detect_runtime()](scripts/lib/docker.sh:14)
- [docker_verify_rootless()](scripts/lib/docker.sh:33)
- [docker_build_container()](scripts/lib/docker.sh:47)
- [docker_create_dockerfile()](scripts/lib/docker.sh:65)
- [docker_start_rootless()](scripts/lib/docker.sh:116)
- [docker_run_build()](scripts/lib/docker.sh:154)
- [docker_commit_builder()](scripts/lib/docker.sh:196)
- [docker_cleanup()](scripts/lib/docker.sh:208)
- [docker_check_capabilities()](scripts/lib/docker.sh:248)
- [docker_setup_rootless()](scripts/lib/docker.sh:279)
## initramfs.sh - Initramfs assembly
File: [scripts/lib/initramfs.sh](scripts/lib/initramfs.sh)
- [initramfs_setup_zinit()](scripts/lib/initramfs.sh:13)
- [initramfs_install_init_script()](scripts/lib/initramfs.sh:70)
- [initramfs_copy_components()](scripts/lib/initramfs.sh:97)
- [initramfs_setup_modules()](scripts/lib/initramfs.sh:225)
- [initramfs_resolve_module_dependencies()](scripts/lib/initramfs.sh:313)
- [initramfs_create_module_scripts()](scripts/lib/initramfs.sh:422)
- [initramfs_strip_and_upx()](scripts/lib/initramfs.sh:486)
- [initramfs_finalize_customization()](scripts/lib/initramfs.sh:569)
- [initramfs_create_cpio()](scripts/lib/initramfs.sh:642)
- [initramfs_validate()](scripts/lib/initramfs.sh:710)
- [initramfs_test_archive()](scripts/lib/initramfs.sh:809)
- [initramfs_copy_resolved_modules()](scripts/lib/initramfs.sh:846)
## kernel.sh - Kernel building
File: [scripts/lib/kernel.sh](scripts/lib/kernel.sh)
- [kernel_get_full_version()](scripts/lib/kernel.sh:14)
- [kernel_download_source()](scripts/lib/kernel.sh:28)
- [kernel_apply_config()](scripts/lib/kernel.sh:82)
- [kernel_modify_config_for_initramfs()](scripts/lib/kernel.sh:129)
- [kernel_build_with_initramfs()](scripts/lib/kernel.sh:174)
- [kernel_build_modules()](scripts/lib/kernel.sh:228)
- [kernel_cleanup()](scripts/lib/kernel.sh:284)
## stages.sh - Build stage tracking
File: [scripts/lib/stages.sh](scripts/lib/stages.sh)
- [stages_init()](scripts/lib/stages.sh:12)
- [stage_is_completed()](scripts/lib/stages.sh:33)
- [stage_mark_completed()](scripts/lib/stages.sh:48)
- [stage_force_rebuild()](scripts/lib/stages.sh:69)
- [stages_clear_all()](scripts/lib/stages.sh:82)
- [stage_run()](scripts/lib/stages.sh:99)
- [stages_status()](scripts/lib/stages.sh:134)
## testing.sh - Boot testing
File: [scripts/lib/testing.sh](scripts/lib/testing.sh)
- [testing_qemu_boot()](scripts/lib/testing.sh:14)
- [testing_qemu_basic_boot()](scripts/lib/testing.sh:55)
- [testing_qemu_serial_boot()](scripts/lib/testing.sh:90)
- [testing_qemu_interactive_boot()](scripts/lib/testing.sh:113)
- [testing_cloud_hypervisor_boot()](scripts/lib/testing.sh:135)
- [testing_cloud_hypervisor_basic()](scripts/lib/testing.sh:171)
- [testing_cloud_hypervisor_serial()](scripts/lib/testing.sh:206)
- [testing_analyze_boot_log()](scripts/lib/testing.sh:227)
- [testing_run_all()](scripts/lib/testing.sh:299)
Reference in New Issue View Git Blame Copy Permalink