afd4f4c6f9790ee7d8a97a49c5619309452594f0
Summary
- Implemented plain S3-only flist workflow (no web endpoint). rfs pack uploads blobs using write creds; flist route.url is patched to embed read-only S3 credentials so rfs mount reads directly from S3.
Changes
1) New RFS tooling (scripts/rfs/)
- common.sh:
- Compute FULL_KERNEL_VERSION from configs (no uname).
- Load S3 config and construct pack store URI.
- Build read-only S3 route URL and patch flist (sqlite).
- Helpers to locate modules/firmware trees and rfs binary.
- pack-modules.sh:
- Pack /lib/modules/<FULL_KERNEL_VERSION> to dist/flists/modules-<FULL_KERNEL_VERSION>.fl
- Patch flist route to s3://READ:READ@host:port/ROUTE_PATH?region=ROUTE_REGION (default /blobs, garage).
- Optional upload of .fl using MinIO client (mcli/mc).
- pack-firmware.sh:
- Source firmware from $PROJECT_ROOT/firmware (fallback to initramfs/lib/firmware).
- Pack to dist/flists/firmware-<TAG_OR_DATE>.fl (FIRMWARE_TAG or YYYYMMDD).
- Patch flist route to read-only S3; optional .fl upload via mcli/mc.
- verify-flist.sh:
- rfs flist inspect/tree; optional mount test (best effort).
- patch-stores.sh:
- Helper to patch stores (kept though not used by default).
2) Dev-container (Dockerfile)
- Added sqlite and MinIO client package for manifest patching/upload (expect mcli binary at runtime; scripts support both mcli/mc).
- Retains rustup and musl target for building rfs/zinit/mycelium.
3) Config and examples
- config/rfs.conf.example:
- S3_ENDPOINT/S3_REGION/S3_BUCKET/S3_PREFIX
- S3_ACCESS_KEY/S3_SECRET_KEY (write)
- READ_ACCESS_KEY/READ_SECRET_KEY (read-only)
- ROUTE_ENDPOINT (defaults to S3_ENDPOINT), ROUTE_PATH=/blobs, ROUTE_REGION=garage
- MANIFESTS_SUBPATH, UPLOAD_MANIFESTS (mcli upload optional)
- config/rfs.conf updated by user with real values (not committed here; example included).
- config/modules.conf minor tweak (staged).
4) Zinit mount scripts (config/zinit/init/)
- firmware.sh:
- Mounts firmware-latest.fl over /usr/lib/firmware using rfs mount (env override FIRMWARE_FLIST supported).
- modules.sh:
- Mounts modules-$(uname -r).fl over /lib/modules/$(uname -r) (env override MODULES_FLIST supported).
- Both skip if target already mounted and respect RFS_BIN env.
5) Documentation
- docs/rfs-flists.md:
- End-to-end flow, S3-only route URL patching, mcli upload notes.
- docs/review-rfs-integration.md:
- Integration points, build flow, and post-build standalone usage.
- docs/depmod-behavior.md:
- depmod reads .modinfo; recommend prebuilt modules.*(.bin); use depmod -A only on mismatch.
6) Utility
- scripts/functionlist.md synced with current functions.
Behavioral details
- Pack (write):
s3://S3_ACCESS_KEY:S3_SECRET_KEY@HOST:PORT/S3_BUCKET/S3_PREFIX?region=REGION
- Flist route (read, post-patch):
s3://READ_ACCESS_KEY:READ_SECRET_KEY@HOST:PORT/ROUTE_PATH?region=ROUTE_REGION
Defaults: ROUTE_PATH=/blobs, ROUTE_REGION=garage; ROUTE_ENDPOINT derived from S3_ENDPOINT if not set.
Runtime mount examples
- Modules:
rfs mount -m dist/flists/modules-6.12.44-Zero-OS.fl /lib/modules/6.12.44-Zero-OS
- Firmware:
rfs mount -m dist/flists/firmware-YYYYMMDD.fl /usr/lib/firmware
Notes
- FUSE policy: If "allow_other" error occurs, enable user_allow_other in /etc/fuse.conf or run mounts as root.
- WEB_ENDPOINT rewrite is disabled by default (set WEB_ENDPOINT=""). Plain S3 route is embedded in flists.
- MinIO client binary in dev-container is mcli; scripts support mcli (preferred) and mc (fallback).
Files added/modified
- Added: scripts/rfs/{common.sh,pack-modules.sh,pack-firmware.sh,verify-flist.sh,patch-stores.sh}
- Added: config/zinit/init/{firmware.sh,modules.sh}
- Added: docs/{rfs-flists.md,review-rfs-integration.md,depmod-behavior.md}
- Added: config/rfs.conf.example
- Modified: Dockerfile, scripts/functionlist.md, config/modules.conf, config/zinit/sshd-setup.yaml, .gitignore
Zero OS Alpine Initramfs Builder
A comprehensive build system for creating custom Alpine Linux 3.22 x86_64 initramfs with zinit process management, designed for Zero OS deployment.
Features
- Alpine Linux 3.22 miniroot as base system
- zinit process manager (complete OpenRC replacement)
- Rootless containers (Docker/Podman compatible)
- Rust components with musl targeting (zinit, rfs, mycelium)
- Aggressive optimization (strip + UPX compression)
- 2-stage module loading for hardware support
- GitHub Actions compatible build pipeline
- Final output:
vmlinuz.efiwith embeddedinitramfs.cpio.xz
Quick Start
Prerequisites
Ubuntu/Debian
sudo apt-get update
sudo apt-get install -y \
build-essential \
rustc \
cargo \
upx-ucl \
binutils \
git \
wget \
qemu-system-x86 \
podman
# Add Rust musl target
rustup target add x86_64-unknown-linux-musl
sudo apt-get install -y musl-tools
Alpine Linux
apk add --no-cache \
build-base \
rust \
cargo \
upx \
git \
wget \
qemu-system-x86 \
podman
# Add Rust musl target
rustup target add x86_64-unknown-linux-musl
Rootless Container Setup
For rootless Docker/Podman support:
# Configure subuid/subgid (if not already configured)
echo "$(whoami):100000:65536" | sudo tee -a /etc/subuid
echo "$(whoami):100000:65536" | sudo tee -a /etc/subgid
# Verify setup
podman system info
Build
# Clone the repository
git clone <repository-url>
cd zosbuilder
# Make scripts executable
chmod +x scripts/build.sh scripts/clean.sh
# Build initramfs
./scripts/build.sh
# Output will be in dist/
ls -la dist/
# vmlinuz.efi - Kernel with embedded initramfs
# initramfs.cpio.xz - Standalone initramfs archive
Project Structure
zosbuilder/
├── config/
│ ├── zinit/ # zinit service definitions
│ │ ├── services/ # individual service files
│ │ └── zinit.conf # main zinit configuration
│ ├── packages.list # Alpine packages to install
│ ├── sources.conf # components to build (ThreeFold)
│ ├── kernel.config # Linux kernel configuration
│ └── modules.conf # 2-stage module loading
├── configs/ # existing configurations (migrated)
├── scripts/
│ ├── lib/
│ │ ├── docker.sh # container management
│ │ ├── alpine.sh # Alpine operations
│ │ ├── components.sh # source building
│ │ ├── initramfs.sh # assembly & optimization
│ │ ├── kernel.sh # kernel building
│ │ └── testing.sh # QEMU/cloud-hypervisor
│ ├── build.sh # main orchestrator
│ └── clean.sh # cleanup script
├── initramfs/ # build output (generated)
├── components/ # component sources (generated)
├── kernel/ # kernel source (generated)
└── dist/ # final artifacts (generated)
Configuration
Component Sources (config/sources.conf)
Define components to download and build:
# Format: TYPE:NAME:URL:VERSION:BUILD_FUNCTION[:EXTRA_OPTIONS]
# Git repositories (Rust components with musl)
git:zinit:https://github.com/threefoldtech/zinit:master:build_zinit
git:mycelium:https://github.com/threefoldtech/mycelium:0.6.1:build_mycelium
git:rfs:https://github.com/threefoldtech/rfs:development:build_rfs
# Pre-built releases
release:corex:https://github.com/threefoldtech/corex/releases/download/2.1.4/corex-2.1.4-amd64-linux-static:2.1.4:install_corex:rename=corex
Package List (config/packages.list)
Alpine packages to install (NO OpenRC):
# Core system
alpine-baselayout
busybox
musl
# Hardware detection & modules
eudev
eudev-hwids
kmod
# Networking
iproute2
ethtool
dhcpcd
# Filesystems
btrfs-progs
dosfstools
# Security & SSH
haveged
openssh-server
# Tools
zellij
tcpdump
bmon
Module Loading (config/modules.conf)
2-stage hardware module loading:
# Stage 1: Critical boot modules
stage1:virtio_net
stage1:virtio_scsi
stage1:virtio_blk
stage1:e1000
stage1:e1000e
# Stage 2: Extended hardware support
stage2:igb
stage2:ixgbe
stage2:i40e
stage2:r8169
stage2:bnx2
stage2:bnx2x
zinit Configuration (config/zinit/)
Main config (config/zinit/zinit.conf)
log_level: debug
init:
- stage1-modules
- stage2-modules
- networking
- services
Service definitions (config/zinit/services/)
Services are migrated from existing configs/zinit/ directory with proper initialization order.
Build Process
Phase 1: Environment Setup
- Create build directories
- Install build dependencies
- Setup Rust musl target
Phase 2: Alpine Base
- Download Alpine 3.22 miniroot
- Extract to initramfs directory
- Install packages from
config/packages.list - NO OpenRC installation
Phase 3: Component Building
- Parse
config/sources.conf - Download/clone sources to
components/ - Build Rust components with musl:
- zinit: Standard cargo build
- rfs: Standard cargo build
- mycelium: Build in
myceliumd/subdirectory
- Install binaries to initramfs
Phase 4: System Configuration
- Replace
/sbin/initwith zinit - Copy zinit configuration
- Setup 2-stage module loading
- Configure system services
Phase 5: Optimization
- Aggressive cleanup:
- Remove docs, man pages, locales
- Remove headers, development files
- Remove APK cache
- Binary optimization:
- Strip all executables and libraries
- UPX compress all binaries
- Size verification
Phase 6: Packaging
- Create
initramfs.cpio.xzwith XZ compression - Build kernel with embedded initramfs
- Generate
vmlinuz.efi
Testing
QEMU Testing
# Boot test with QEMU
./scripts/test.sh --qemu
# With serial console
./scripts/test.sh --qemu --serial
cloud-hypervisor Testing
# Boot test with cloud-hypervisor
./scripts/test.sh --cloud-hypervisor
Custom Testing
# Manual QEMU command
qemu-system-x86_64 \
-kernel dist/vmlinuz.efi \
-m 512M \
-nographic \
-serial mon:stdio \
-append "console=ttyS0,115200 console=tty1 loglevel=7"
Size Optimization
The build system achieves minimal size through:
Package Selection
- Minimal Alpine packages (~50MB target)
- No OpenRC or systemd
- Essential tools only
Binary Optimization
- strip: Remove debug symbols
- UPX: Maximum compression
- musl static linking: No runtime dependencies
Filesystem Cleanup
- Remove documentation
- Remove locales (except C)
- Remove development headers
- Remove package manager cache
Expected Sizes
- Base Alpine: ~5MB
- With packages: ~25MB
- With components: ~40MB
- After optimization: ~15-20MB
- Final initramfs.cpio.xz: ~8-12MB
GitHub Actions Integration
See GITHUB_ACTIONS.md for complete CI/CD setup.
Basic Workflow
name: Build Zero OS
on: [push, pull_request]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup rootless containers
run: |
echo "runner:100000:65536" | sudo tee -a /etc/subuid
echo "runner:100000:65536" | sudo tee -a /etc/subgid
- name: Build
run: ./scripts/build.sh
- name: Test
run: ./scripts/test.sh --qemu
Advanced Usage
Custom Components
Add custom components to config/sources.conf:
# Custom Git component
git:myapp:https://github.com/user/myapp:v1.0:build_myapp
# Custom release
release:mytool:https://releases.example.com/mytool-x86_64:v2.0:install_mytool
Implement build function in scripts/lib/components.sh:
function build_myapp() {
local name="$1"
local component_dir="$2"
# Custom build logic
export RUSTFLAGS="-C target-feature=+crt-static"
cargo build --release --target x86_64-unknown-linux-musl
# Install binary
cp target/x86_64-unknown-linux-musl/release/myapp "${INSTALL_DIR}/usr/bin/"
}
Container Builds
Build in isolated container:
# Build container image
podman build -t zero-os-builder .
# Run build in container
podman run --rm \
-v $(pwd):/workspace \
-w /workspace \
zero-os-builder \
./scripts/build.sh
Cross-Platform Support
The build system supports multiple architectures:
# Build for different targets
export RUST_TARGET="aarch64-unknown-linux-musl"
export ALPINE_ARCH="aarch64"
./scripts/build.sh
Troubleshooting
Common Issues
Build Failures
# Clean and retry
./scripts/clean.sh
./scripts/build.sh
# Check dependencies
./scripts/build.sh --check-deps
Container Issues
# Verify rootless setup
podman system info
# Reset user namespace
podman system reset
Rust Build Issues
# Verify musl target
rustup target list --installed | grep musl
# Add if missing
rustup target add x86_64-unknown-linux-musl
Debug Mode
# Enable verbose output
export DEBUG=1
./scripts/build.sh
Size Analysis
# Analyze initramfs contents
./scripts/analyze-size.sh
# Show largest files
find initramfs/ -type f -exec du -h {} \; | sort -rh | head -20
Contributing
- Fork the repository
- Create feature branch
- Test thoroughly with both QEMU and cloud-hypervisor
- Ensure size optimization targets are met
- Submit pull request with detailed description
Development Workflow
# Setup development environment
./scripts/setup-dev.sh
# Run tests
./scripts/test.sh --all
# Check size impact
./scripts/analyze-size.sh --compare
License
[License information]
Support
- Issues: GitHub Issues
- Documentation: See
docs/directory - Examples: See
examples/directory
Related Projects
Description
dockerized/podman-ized build system for zos, based on Alpine pakaging system, zinit and zos-specific binaries
Languages
Shell
99.5%
Dockerfile
0.5%