projectmycelium/docs/dev/design/threefold-marketplace-design-architecture.md

Project Mycelium - Design & Architecture

Last Updated: 2025-08-23 10:21:27 (EDT)

Purpose: Comprehensive architectural vision and complete UX specification (patterns, guidelines, ADRs) for the Project Mycelium. Roadmap and status live in the separate roadmap document.

See also: Roadmap

1. Architecture Vision (Target State)

Core Design Principles

User Experience as Single Source of Truth

• Complete UX specification drives all development priorities
• All features must support the defined user workflows
• UX testing validates every implementation milestone
• API development derives from UX requirements

Builder Pattern as Single Source of Truth

• SessionDataBuilder, ConfigurationBuilder, ResponseBuilder, ServiceFactory centralize construction and lifecycles
• All HTTP endpoints return via ResponseBuilder with consistent JSON envelopes

ResponseBuilder Envelope & Frontend Contract

{
  "success": true|false,
  "data": { ... },
  "error"?: { ... }
}

• Frontend must always unwrap: const data = result.data || result; before accessing fields

CSP-Compliant Frontend: External JS + JSON Hydration

• Zero inline scripts/handlers - code lives in src/static/js/*.js and is served under /static/js/*
• Data hydration via <script type="application/json" id="..."> blocks
• Individual field encoding: Each field is JSON-encoded individually (e.g., via server-side serializer)
• Template structure: Templates expose {% block scripts %} and {% block head %} at top level only
• Static mapping: Actix serves static assets at /static/* from ./src/static (see src/main.rs)
• Hydration safety: Mark embedded JSON as safe in templates (e.g., Tera | safe) to avoid HTML entity escaping that breaks JSON.parse

Persistent-Data-Only Runtime

• No mock data in production code
• Canonical user data model persisted per user under ./user_data/{email_encoded}.json via UserPersistence
• All operations (orders, services, wallet, products) read/written through persistence services

Product Model: User-Owned SOT + Derived Catalog

• Products owned by provider users in their persistent files
• ProductService aggregates derived marketplace catalog
• Category ID normalization with optional dev TTL cache (disabled by default in prod)

Currency System (Updated)

• TFC as base currency with display currencies: TFC/USD/EUR/CAD (extensible)
• TFC credits settle at 1 TFC = 1 USD (fixed exchange rate)
• Server-formatted amounts: Server returns formatted display amounts and currency code
• Frontend renders without recomputing
• User preference: Users can choose display currency in settings

Unified Insufficient-Balance Contract

• Target: HTTP 402 Payment Required for all insufficient funds cases
• Canonical error payload with error.details containing currency-aware amounts and deficit

---

2. Complete User Experience Specification

Public Access (No Authentication Required)

Information Pages

• /docs - Complete marketplace documentation
• /privacy - Privacy policy and data handling
• /terms - Terms and conditions
• /about - Marketplace information and mission
• /contact - Contact ThreeFold support

Marketplace Browsing

• /marketplace - Full marketplace access for browsing
• Anonymous cart functionality - Add items without login
• Search and filtering across all categories
• Product details and pricing information

Authentication & Registration

User Registration

• /register - New user registration flow
• GitEa OAuth integration with seamless account creation
• Profile setup during registration

User Login

• /login - User authentication
• Session management with secure token handling
• Cart migration from anonymous to authenticated session

Purchase Workflows

Shopping Cart Management

• Anonymous users: /cart
  • Add/remove items
  • View cart contents
  • Login prompt for checkout
  • Cart preservation during login transition
• Authenticated users: /dashboard/cart
  • Full cart editing (quantity adjustment)
  • Item removal and cart clearing
  • Enhanced cart management with user preferences

Purchase Options

• Buy Now - Direct purchase flow
• Add to Cart - Traditional shopping cart workflow
• Checkout process with credit validation
• Order confirmation and invoice generation

Marketplace Categories

Compute Resources

• /marketplace/compute - Virtual machine slices
• VM configuration options (CPU, RAM, storage)
• Kubernetes cluster deployment options
• Self-managed resource mode
  • Users set SSH public key to access VM

Infrastructure

• /marketplace/3nodes - Complete server reservations
• /marketplace/gateways - Mycelium gateway services

Applications & Services

• /marketplace/applications - Published applications
• /marketplace/services - Professional services

User Dashboard & Profile Management

Dashboard Overview

• /dashboard - Activity overview and notifications
• Recent transactions and order status
• Quick actions and shortcuts

User Profile

• /dashboard/user - Profile and activity management
• My Applications - Purchased and deployed apps
• My Service Bookings - Active service engagements
• Deployment management for purchased resources

Wallet & Credits

• /dashboard/wallet - Complete wallet management
• Buy Credits - Credit purchase functionality
• Transfer Credits - User-to-user transfers
• Auto Top-Up Settings - Automated balance management
• Recent Transactions - Complete transaction history

Orders & History

• /dashboard/orders - Complete order history
• Order details and status tracking
• Invoice access and download

Settings & Preferences

Profile Settings

• /dashboard/settings - Comprehensive settings management
• Profile Information:
  • Name (editable)
  • Email (read-only)
  • Country
  • Time Zone
• Password Management - Secure password updates
• Account Deletion - Data retention policy compliance

SSH Key Management

• /dashboard/settings - SSH Keys section
• Public key upload for self-managed resources
• Key management (add, remove, edit)
• Multiple SSH keys per user with duplicate prevention
• Real-time validation with security level indicators
• Integration with VM and cluster deployments

SSH Keys UX Contract — Canonical Pattern

• Endpoints (ResponseBuilder envelope; always unwrap const data = result.data || result;):
  • GET /api/dashboard/ssh-keys
  • POST /api/dashboard/ssh-keys
  • PUT /api/dashboard/ssh-keys/{id}
  • DELETE /api/dashboard/ssh-keys/{id}
  • POST /api/dashboard/ssh-keys/{id}/set-default
  • GET /api/dashboard/ssh-keys/{id}
• Frontend implementation:
  • CSP-compliant JS in src/static/js/dashboard-ssh-keys.js
  • Views in src/views/dashboard/settings.html (no inline JS)
  • Read/write via fetch; unwrap data before accessing fields
• Verification workflow:
  • Use browser console to call each endpoint and confirm { success, data } envelope
  • In UI: add/edit/delete keys, set default; verify list updates and validation messages
• Regression tests:
  • Reference: tests/frontend_ux/settings_management_ux_test.rs (per-step runner pattern)
  • Ensure tests assert ResponseBuilder unwrapping and persistence via user_data

Notification Preferences

• Security Alerts - Account security notifications
• Billing Notifications - Payment and credit alerts
• System Alerts - Downtime and maintenance
• Newsletter - Product updates and news
• Dashboard Notifications - In-app alert preferences
• Message Notifications - Real-time message alerts and desktop notifications

Currency Preferences

• Display Currency Selection:
  • TFC (base currency)
  • USD
  • EUR
  • CAD
• Real-time conversion display

Settings UX Contract — Canonical Pattern

• Endpoints (ResponseBuilder envelope; always unwrap const data = result.data || result;):
  • Profile: POST /api/dashboard/settings/profile
  • Password: POST /api/dashboard/settings/password
  • Verify Password: POST /api/dashboard/settings/verify-password
  • Notifications: POST /api/dashboard/settings/notifications
  • Delete Account: POST /api/dashboard/settings/delete-account
  • Billing History: GET /api/dashboard/settings/billing-history
  • Currency Preference: GET /api/user/currency, POST /api/user/currency
• Frontend contract:
  • Forms or fetch-based actions must read from data and render messages/status accordingly
  • Avoid recomputation; use server-formatted values when provided
  • Keep CSP compliance (no inline JS); hydrate via JSON or fetch
• Verification workflow:
  • Console: test endpoints and confirm { success, data } envelope
  • UI: submit profile/password/notifications; verify success toasts and field updates
  • Currency: change preference and verify display reflects new currency
• Regression tests:
  • Covered by tests/frontend_ux/settings_management_ux_test.rs (validated passing)
  • Use per-step runner with persistence checks in user_data/

Communication & Messaging

Message Notification System

• Industry-standard notifications across all platform touchpoints
• Navbar integration - Messages item in user dropdown with unread counter
• Sidebar integration - Enhanced message counter in dashboard navigation
• Real-time updates - 15-second polling with visual feedback
• Desktop notifications - Browser notifications for new messages (with permission)
• Progressive disclosure - Badge → dropdown preview → full messaging interface
• Cross-platform consistency - Unified notification badges and counters

Messaging UX Contract — Canonical Pattern

• Core API endpoints (ResponseBuilder envelope; always unwrap const data = result.data || result;):
  • GET /api/messages/threads - Thread list with unread counts and last messages
  • POST /api/messages/threads - Create new conversation thread (requires Content-Type: application/json)
  • GET /api/messages/threads/{id}/messages - Get messages for specific thread
  • POST /api/messages/threads/{id}/messages - Send message to thread
  • PUT /api/messages/threads/{id}/read - Mark thread as read (resets unread count)
• Frontend implementation:
  • src/static/js/messaging-system.js - Core messaging functionality and modal interface
  • src/static/js/notification-system.js - Cross-platform notification management (15s polling)
  • src/static/js/dashboard-messages.js - Full-page messaging interface at /dashboard/messages
  • Real-time badge updates via polling and custom event system
• Thread creation workflow:
  • Contact Provider buttons redirect to /dashboard/messages?recipient=email&context=type&booking_id=id
  • System checks for existing thread by recipient_email, context_type, and context_id
  • If no thread exists, creates new thread via POST with proper headers
  • Auto-selects newly created thread by matching both recipient_email AND context_id
• Notification locations:
  • Navbar user dropdown (between Wallet and Settings) with unread counter badge
  • Dashboard sidebar (Communication section) with red danger badge and pulse animation
  • Document title updates showing unread count
  • Desktop notifications (with user permission) for new messages
• Visual standards:
  • Red danger badges for unread counts with 99+ cap for high counts
  • Pulse animation for new notifications
  • Consistent styling with cart counter patterns
  • Toast notifications (top-right, error-only for cleaner UX)

Full-Page Messaging Interface

• Route: /dashboard/messages - Dedicated messaging dashboard page
• Controller: DashboardController::messages_page() with proper context setup
• Template: src/views/dashboard/messages.html - Industry-standard messaging layout
• Architecture:
  • Left panel: Conversation list with unread indicators and context badges
  • Right panel: Message view with fixed header, scrollable messages, anchored input
  • Flexbox layout: calc(100vh - 200px) height with proper viewport utilization
  • Message input: Fixed at bottom using card-footer with flex-shrink-0
• UX Standards:
  • Toast notifications: Top-right positioning, error-only (success removed for cleaner UX)
  • Real-time updates: 15-second polling with immediate message display
  • Character validation: 1000 character limit with live counter
  • Responsive design: Mobile and desktop optimized
  • CSP compliance: External JS with JSON hydration pattern

Message Thread Management

• Thread Structure: Each thread connects two users (user_a_email, user_b_email) with context
• Context Types: service_booking, slice_rental, general - determines thread purpose
• Context ID: Optional identifier (e.g., booking ID) for specific business objects
• Unread Tracking: Per-user unread counters (user_a_unread_count, user_b_unread_count)
• Persistence: Stored in user data files under message_threads and messages arrays
• Thread Creation: Automatic via Contact Provider buttons or manual via messaging interface
• Thread Selection: Matches by recipient_email + context_id for precise targeting

Integration Points

• Service Bookings: Contact Provider buttons in /dashboard/user My Service Bookings
• Slice Rentals: Contact Provider functionality for compute resource support
• General Messaging: Direct user-to-user communication without specific context
• Provider Dashboards: Incoming message notifications for service/app/farmer providers
• Notification System: Cross-platform badges, desktop notifications, document title updates

Provider Dashboards

Farmer Dashboard

• /dashboard/farmer - Resource provider management
• Add Nodes - Grid node registration
• Node Management - Active node monitoring
• Slice Distribution - Resource allocation management
• Revenue Tracking - Earnings and analytics

App Provider Dashboard

• /dashboard/app-provider - Application provider tools
• Register Applications - New app publishing
• My Published Applications - App management table
• Active Deployments - Customer deployment tracking
• Revenue Analytics - App earnings dashboard

Service Provider Dashboard

• /dashboard/service-provider - Professional service management
• Register Services - New service offerings
• My Service Offerings - Service catalog management
• Service Requests - Customer request pipeline:
  • Open
  • In Progress
  • Completed
• Client Management - Customer relationship tools

---

5. Technical Implementation Guidelines

Currency System Implementation

// Updated currency constants
const BASE_CURRENCY: &str = "TFC";
const TFC_TO_USD_RATE: f64 = 1.0;

// Currency display preferences
struct CurrencyPreference {
    user_id: String,
    display_currency: String, // TFC, USD, EUR, CAD, where USD is default
    created_at: DateTime<Utc>,
}

SSH Key Management

// SSH key storage structure
struct SSHKey {
    id: String,
    user_id: String,
    name: String,
    public_key: String,
    fingerprint: String,
    created_at: DateTime<Utc>,
}

Cart Management Pattern

// Cart persistence and migration
struct CartItem {
    product_id: String,
    quantity: u32,
    configuration: Option<serde_json::Value>,
}

struct Cart {
    session_id: Option<String>, // For anonymous carts
    user_id: Option<String>,    // For authenticated carts
    items: Vec<CartItem>,
    created_at: DateTime<Utc>,
    updated_at: DateTime<Utc>,
}

---

8. Architecture Decision Records (Updated)

UX-First Development

• Decision: Complete UX implementation before backend infrastructure
• Rationale: User experience validates product-market fit and drives API requirements
• Implementation: Phased approach with UX testing as validation gate

TFC Base Currency

• Decision: TFC as base currency instead of USD
• Rationale: Align with ThreeFold ecosystem and user expectations
• Implementation: 1 TFC = 1 USD fixed rate with display preferences

9. Concrete UX List

• Publicly available information
  • A user can consult the docs at /docs
  • A user can consult the privacy policy at /privacy
  • A user can consult the Terms and Conditions at /terms
  • A user can read about the marketplace at /about
  • A user can contact ThreeFold by consulting the page /contact
• Registration and Login
  • A user can register at /register
  • Once registered they can log out and then log in at /login
• Purchases
  • A non-logged in user can check the marketplace at /marketplace and add items to cart
  • To buy an item, the user must be logged in and have enough credits
• Purchase UX
  • A user can always buy a product or service in two ways
    • They can Buy Now, or Add to Cart (then proceed to checkout and complete the purchase)
  • A non-logged in user will have access to the cart at /cart
    • If they then log in, the cart items will be added to their account's cart
    • In the /cart page, they can clear cart or delete individual items in the cart
    • If they try to edit the cart, a message appears telling them to log in to edit the cart
  • A logged in user will have access to the cart at /dashboard/cart
    • Then they add to cart, they can see their cart at /dashboard/cart
    • In the /cart page, they can clear cart or delete individual items in the cart
    • As a logged in user, they can edit the cart, e.g. click + to add more of that same item, or click - to remove that item
  • A user can see all their purchases at /dashboard/orders
  • A user can see their recent transactions at /dashboard/wallet Recent Transactions
• Credits
  • A user can buy credits at /dashboard/wallet, Buy Credits
  • A user can transfer credits to another user at /dashboard/wallet, Transfer Credits
  • A user can set up Auto Top-Up Settings at /dashboard/wallet Auto Top-up Settings
    • Thus if a user buys an app that costs e.g. 10 USD per month, they can set an automatic wallet top-up to never go below that number, ensuring their app will never run out of credits
• Marketplace
  • In the marketplace, a user can search and filter items in different categories
    • They can buy compute resources (slices) at /marketplace/compute
    • They can reserve a complete node (server) at /marketplace/3nodes
    • They can buy Mycelium gateways at /marketplace/gateways
    • They can buy applications (solutions) at /marketplace/applications
    • They can buy services at /marketplace/services
• Settings
  • A user can change settings at /dashboard/settings
    • They can
      • Update the profile information (email can't be changed)
        • Name
        • Country
        • Time Zone
      • Change the password
      • Set Up SSH Public Keys
      • Update notification settings
        • Security alerts
        • Billing notifications
        • System alerts (downtime, maintenance)
        • Newsletter and product updates
        • Dashboard Notifications
          • Show alerts in dashboard
          • Show update notifications
      • Update their currency preferences
        • They can decide to have the prices displayed in
          • USD
          • TFC
          • EUR
          • CAD
      • Delete account
        • When a user deletes their account, their data is still available on the marketplace backend for security, audit and legal purposes
• Dashboard UX and Provider+Consumer Interactions
  • A user can see their dashboard overview activities at /dashboard
  • A user can see their user profile and activities at /dashboard/user
  • A user provoding resources to the grid (aka a farmer) at /dashboard/farmer
    • There they can add resources, e.g. add a node to the marketplace which will be available for purchase by other users as slices (a node is distributed as slices)
  • A user can become an app provider at /dashboard/app-provider by registering new applications
    • When a user register an application
      • The application is displayed publicly at /marketplace/applications
      • The application is shown at the app provider dashboard /dashboard/app-provider at the table My Published Applications
      • When another user buys that app
        • The app provider will see that info at the table /dashboard/app-provider Active Deployments
        • The app purchaser will see the app info at /dashboard/user My Applications
  • A user can become a service provider at /dashboard/service-provider by registering new services
    • When a user register a service
      • The service is displayed publicly at /marketplace/services
      • The service is shown at the service provider dashboard /dashboard/service-provider at the table My Service Offerings
      • When another user buys that service
        • The service provider will see that info at the table /dashboard/service-provider Service Requests
          • There are 3 stages to this service request from the service provider POV
            • Open
            • In Progress
            • Completed
        • The service purchaser will see the service info at /dashboard/user My Service Bookings
  • A user can become a resource provider by adding nodes and thus resources to the marketplace at /dashboard/farmer
    • They can Add Nodes
      • Then it fetches the nodes on the ThreeFold Grid and distribute the node into slices
      • Then the slices are can be seen publicly at /marketplace/compute
    • Any user can then purchase slices from that farmer and access the slices
• Products
  • Farmers at the core offer compute resources (slices) to the marketplace.
  • On their end, farmers host nodes on the threefold grid. nodes are split into slices.
  • Users can use slices for individual VMs, or for Kubernetes cluster
    • The UI is intuitive
      • .html_template_tests
• Apps can be self-managed and managed
  • An app is at its core a slice (VM or Kubernetes cluster) with an app on top
  • for self-managed node, users can set their SSH public key, they set it in /dashboard/settings SSH Keys page
  • for managed node, users will have access to the credentials on their marketplace dashboard in /dashboard/user page at the section of the app/product they rent/bought

Testing the UX

• The marketplace should have a complete test suite to confirm the frontend UX is as expected for usersT
• Thus, there should be a series of test for the UX above
• We should have the tests mentioned above in an isolated section, e.g. not shared with other tests
  • This ensures that the tests are not affected by other tests

Marketplace-Wide UX Contract Audit Plan

Goal: Ensure every UX flow follows the canonical API envelope and CSP-compliant frontend contract, with deterministic tests and persistence-backed verification. The complete UX should be complete and functioning.

• Scope (frontend code):
  • src/static/js/**/*.js and static/js/**/*.js (external JS only; no inline handlers)
  • Views in src/views/** (JSON hydration blocks only; keep CSP clean)
• Scope (backend routes):
  • src/routes/mod.rs is the single source of truth for endpoint paths
• Contract checks (apply to every fetch):
  • Unwrap ResponseBuilder: const result = await response.json(); const data = result.data || result;
  • Render optional numbers safely: show "No limit" for null/undefined values
  • Prefer server-formatted currency/amounts; avoid client recomputation
  • Keep all logic in external JS; hydrate via <script type="application/json">
• Priority UX areas to audit:
  • Wallet & credits: src/static/js/dashboard_wallet.js, /api/wallet/*
  • Auto top-up status: GET /api/wallet/auto-topup/status
  • SSH keys: src/static/js/dashboard-ssh-keys.js, /api/dashboard/ssh-keys*
  • Settings: /api/dashboard/settings/* forms and responses
  • Cart, orders, products: /api/cart*, /api/orders*, /api/products*
• Verification workflow:
  • Console: fetch('<endpoint>').then(r=>r.json()).then(console.log) to confirm { success, data }
  • UI: ensure badges/labels and form prefills reflect data
  • Cross-check each referenced path with src/routes/mod.rs
• Test adoption:
  • Use per-step runner pattern (see tests/frontend_ux/settings_management_ux_test.rs)
  • Persist state under user_data/ and assert post-conditions
  • Run: cargo test --features ux_testing -- --nocapture
• Reporting:
  • Track per-feature audit items in this roadmap; document deviations and fixes under each UX contract subsection