geomind_code/my_fs
23
0
Fork 0
Code Issues 1 Pull requests Projects Releases 2 Packages Wiki Activity Actions

feature/workspace-restructuring-worker-pool-fixes #27

Closed
delandtj wants to merge 21 commits from feature/workspace-restructuring-worker-pool-fixes into development
pull from: feature/workspace-restructuring-worker-pool-fixes
merge into: geomind_code:development
Conversation 1 Commits 21 Files changed 99 +8012 -5700
delandtj commented 2026-02-26 20:58:54 +00:00
Owner
Copy link

Summary

  • Restructure the codebase from a flat src/ layout into a Cargo workspace with 4 crates: myfs-core, myfs-server, myfs (FUSE mount), and myfstool (CLI utilities)
  • Remove async-trait crate entirely, using native Rust async fn in traits (requires Rust 1.75+)
  • Upgrade bb8-redis 0.13 → 0.26 (bb8 0.8 → 0.9, redis 0.23 → 1.0) to eliminate last async-trait dependency
  • Update workers (tokio-worker-pool) to hanging_myfs_fixes branch with new Work trait API
  • Port all commits from the development branch into the new workspace layout
  • Add comprehensive test suite (174 tests across all crates)
  • Add build system (Makefile, build_lib.sh, Forgejo workflows)

Ported from development

All meaningful commits from development have been cherry-picked and adapted to the workspace layout:

  • Local image check before container conversion
  • String entrypoint support
  • Recursive tree traversal → iterative (stack overflow fix)
  • Path traversal prevention in website_handlers
  • Configurable server concurrency/timeouts
  • Atomic metadata writes (temp file + rename pattern)
  • Complete Error-to-Errno mapping for FUSE
  • AES-GCM random nonce per block (security fix)
  • Constant-time password comparison (timing attack fix)
  • std::sync::Mutex → tokio::sync::Mutex for async safety

Not integrated

The development_consistent_bugs branch commit a08590f ("follow consistent error structure across all crate modules") is not included. This is a sweeping refactor that replaces
anyhow::Result with crate::error::Result across 19 files. It conflicts with the workspace restructuring and the error/errno changes already ported here. It should be re-applied on top of
the workspace layout after this PR lands.

Post-merge

After merging, the development branch can be reset to main — all its changes are already here. Development continues from the workspace layout going forward.

Test plan

  • cargo test — 174 tests pass (86 core + 53 server + 35 tool)
  • cargo check / cargo clippy clean
  • Manual smoke test of FUSE mount + server
Summary - Restructure the codebase from a flat src/ layout into a Cargo workspace with 4 crates: myfs-core, myfs-server, myfs (FUSE mount), and myfstool (CLI utilities) - Remove async-trait crate entirely, using native Rust async fn in traits (requires Rust 1.75+) - Upgrade bb8-redis 0.13 → 0.26 (bb8 0.8 → 0.9, redis 0.23 → 1.0) to eliminate last async-trait dependency - Update workers (tokio-worker-pool) to hanging_myfs_fixes branch with new Work trait API - Port all commits from the development branch into the new workspace layout - Add comprehensive test suite (174 tests across all crates) - Add build system (Makefile, build_lib.sh, Forgejo workflows) Ported from development All meaningful commits from development have been cherry-picked and adapted to the workspace layout: - Local image check before container conversion - String entrypoint support - Recursive tree traversal → iterative (stack overflow fix) - Path traversal prevention in website_handlers - Configurable server concurrency/timeouts - Atomic metadata writes (temp file + rename pattern) - Complete Error-to-Errno mapping for FUSE - AES-GCM random nonce per block (security fix) - Constant-time password comparison (timing attack fix) - std::sync::Mutex → tokio::sync::Mutex for async safety Not integrated The development_consistent_bugs branch commit a08590f ("follow consistent error structure across all crate modules") is not included. This is a sweeping refactor that replaces anyhow::Result with crate::error::Result across 19 files. It conflicts with the workspace restructuring and the error/errno changes already ported here. It should be re-applied on top of the workspace layout after this PR lands. Post-merge After merging, the development branch can be reset to main — all its changes are already here. Development continues from the workspace layout going forward. Test plan - cargo test — 174 tests pass (86 core + 53 server + 35 tool) - cargo check / cargo clippy clean - Manual smoke test of FUSE mount + server
Split the monolithic myfs crate into a workspace as per ADR-002:

- myfs-core: Shared library with store backends, fungi metadata,
  cache, error handling, pack/unpack, upload/download, and utilities
- myfs-server: Standalone HTTP server binary with REST API and JWT auth
- myfstool: Advanced tools binary (container, clone, merge, sync)
- myfs: FUSE mount binary with core flist operations

This separation provides clearer ownership, smaller binaries,
faster builds, and independent versioning for each component.

All 21 tests pass and clippy is clean.
Implement testing strategy with 176 tests covering:
- myfs-core: server_api, upload, download, cache, store/dir (88 tests)
- myfs-server: auth, sqlite db, block_handlers (53 tests)
- myfstool: clone, merge, sync operations (35 tests)

Add shared test utilities in myfs-core/src/testutils.rs with InMemoryStore
and helper functions. Add integration tests for server API and pack/unpack
round-trips. Add wiremock and tokio-test dev-dependencies for HTTP mocking.
Update README to reflect current workspace structure
Some checks failed
Unit and Integration Test / test (push) Failing after 20s
Details
2721004c92
 
- Document 4-crate workspace architecture (myfs-core, myfs-server, myfs, myfstool)
- Update command reference for myfs and myfstool CLIs
- Add accurate server configuration example with all required fields
- Document storage backends with URL formats
- Add project structure overview
- Remove outdated README MyFS.md (contained incorrect command references)
chore: migrate polyfuse source and comprehensively realign docs to current CLI/server
Some checks failed
Unit and Integration Test / test (push) Failing after 6s
Details
3e1c2fec34
 
This commit consolidates a full documentation and build-tooling cleanup while
also switching the myfs FUSE dependency to the requested upstream branch.

Dependency/tooling updates:
- Switch myfs polyfuse dependency from
  https://github.com/muhamadazmy/polyfuse (master)
  to ssh://git@github.com/ubnt-intrepid/polyfuse.git (0.5.x)
- Refresh Cargo.lock to the new polyfuse/polyfuse-kernel revisions and
  related transitive dependency graph changes
- Add a root Makefile with practical release workflows:
  - build / test
  - build-musl
  - strip-binaries
  - upx-binaries (upx --best)
  - release-musl pipeline and dist output handling

Documentation realignment (broad sweep):
- Correct server startup docs from legacy myfs server/config-path usage to
  current myfs-server --config flow
- Align command naming to implemented CLI surface:
  - myfs flist-create (not myfs flist create)
  - advanced operations under myfstool (container/clone/merge/sync)
  - remove stale references to removed myfs subcommands (docker/clone/server/
    website-publish/token/track* where no longer available)
- Update server API references to the current routes:
  - auth via /api/v1/signin
  - current /api/v1/* endpoint family
  - website serving via /website/{flist_hash}/...
- Rewrite/refresh high-drift docs so examples are now executable with current
  binaries and flags

Files updated include:
- root README and docs/README command summaries
- user guides (myfs-cli, fl-server, frontend, troubleshooting,
  performance-tuning)
- tutorials (server-setup, website-publishing, docker-conversion,
  end-to-end-docker-workflow)
- architecture and concepts pages
- zdb and legacy guidance pages
- ADR wording to explicitly label migration tables as historical mappings

Outcome:
- Docs now match actual binaries and command behavior
- release build/compression workflow is standardized in-repo
- dependency source change requested for polyfuse is fully applied
fix: update workers crate to hanging_myfs_fixes branch and adapt to new Work trait API
Some checks failed
Unit and Integration Test / test (push) Failing after 8s
Details
49e8f39e35
 
Switch tokio-worker-pool dependency to the hanging_myfs_fixes branch which
removes async_trait from the Work trait in favor of impl Future return types.
Adapt Uploader (pack.rs) and BlobCloner (clone.rs) implementations accordingly,
and handle the new Option return from WorkerPool::get().
refactor: remove async-trait crate, upgrade bb8-redis to 0.26
Some checks failed
Unit and Integration Test / test (push) Failing after 25s
Details
590f250557
 
Remove all async_trait usage across the codebase by leveraging native
async fn in traits (stable since Rust 1.75). The Store and WalkVisitor
traits now use desugared return types with explicit Send bounds.

Upgrade bb8-redis 0.13 → 0.26 (bb8 0.8 → 0.9, redis 0.23 → 1.0) to
eliminate bb8's async_trait dependency. Rework zdb.rs for redis 1.0 API
changes (private ConnectionInfo fields, MultiplexedConnection type).
(cherry-picked from development 8383c6f)
(cherry-picked from development ed9e617)
Replace manual workflow scripts with build_lib pattern: buildenv.sh for
config, scripts/build_lib.sh for shared logic, Makefile for dev targets.
Add Linux ARM64 cross-compilation and macOS build workflows.
Fix Makefile cargo and shell compatibility.

(cherry-picked from development a0f9fdd + 289cf64)
(cherry-picked from development 0d47b7d)
(cherry-picked from development abbc34b)
(cherry-picked from development 453c6d4)
(cherry-picked from development 7f81d77)
(cherry-picked from development a9f84b5)
Generate unique random nonces for each AES-GCM encryption instead of
deriving from the key. Add nonce column to block schema with backward
compatibility for old flists.

(cherry-picked from development 2e699fb)
(cherry-picked from development cdd4319)
Switch from std::sync::Mutex to tokio::sync::Mutex for async-safe
locking of jobs_state and flists_progress.

(cherry-picked from development 1450ed8)
fix: update tests for nonce and atomic write changes
Some checks failed
Build and Test / build (push) Failing after 7s
Details
be8ced9c3c
 
Add nonce parameters to meta.block() test calls and Block struct
initializer. Add Writer::finalize() calls before Reader creation
in all meta.rs tests to match the atomic write pattern.
fix: remove build-binary feature flag from build system
Some checks failed
Build and Test / build (push) Failing after 10s
Details
c0846b40d1
 
The workspace restructuring eliminated the need for the build-binary
feature gate — binary targets are now separate crates. Update Makefile,
buildenv.sh, and build_lib.sh accordingly.
fix: use HTTPS URL for polyfuse dependency
All checks were successful
Build and Test / build (push) Successful in 2m52s
Details
37636e1618
 
SSH git URL fails in CI runners without SSH keys configured.
Switch to HTTPS which works without authentication.
rawdaGastan commented 2026-03-04 11:48:03 +00:00
Owner
Copy link

Can you please solve conflicts?

Can you please solve conflicts?
rawdaGastan changed target branch from main to development 2026-03-10 09:54:40 +00:00
delandtj closed this pull request 2026-04-02 11:54:39 +00:00
All checks were successful
Build and Test / build (push) Successful in 3m35s
Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
geomind_code/my_fs!27
No description provided.