projectmycelium/docs/dev/design/archive/threefold-marketplace-design-architecture.md

# Project Mycelium - Design & Architecture

**Last Updated:** 2025-08-23 10:21:27 (EDT)

**Purpose:** Comprehensive architectural vision and complete UX specification (patterns, guidelines, ADRs) for the Project Mycelium. Roadmap and status live in the separate roadmap document.

See also: [Roadmap](./projectmycelium-roadmap.md)

## 1. Architecture Vision (Target State)

### Core Design Principles

#### **User Experience as Single Source of Truth**
- Complete UX specification drives all development priorities
- All features must support the defined user workflows
- UX testing validates every implementation milestone
- API development derives from UX requirements

#### **Builder Pattern as Single Source of Truth**
- [`SessionDataBuilder`](src/services/session_data.rs), [`ConfigurationBuilder`](src/config/builder.rs), [`ResponseBuilder`](src/utils/response_builder.rs), [`ServiceFactory`](src/services/factory.rs) centralize construction and lifecycles
- All HTTP endpoints return via [`ResponseBuilder`](src/utils/response_builder.rs) with consistent JSON envelopes

#### **ResponseBuilder Envelope & Frontend Contract**
```json
{
  "success": true|false,
  "data": { ... },
  "error"?: { ... }
}
```
- Frontend must always unwrap: `const data = result.data || result;` before accessing fields

#### **CSP-Compliant Frontend: External JS + JSON Hydration**
- **Zero inline scripts/handlers** - code lives in [`src/static/js/*.js`](src/static/js/) and is served under `/static/js/*`
- **Data hydration** via `<script type="application/json" id="...">` blocks
- **Individual field encoding**: Each field is JSON-encoded individually (e.g., via server-side serializer)
- **Template structure**: Templates expose `{% block scripts %}` and `{% block head %}` at top level only
- **Static mapping**: Actix serves static assets at `/static/*` from `./src/static` (see `src/main.rs`)
- **Hydration safety**: Mark embedded JSON as safe in templates (e.g., Tera `| safe`) to avoid HTML entity escaping that breaks `JSON.parse`

#### **Persistent-Data-Only Runtime**
- **No mock data** in production code
- **Canonical user data model** persisted per user under [`./user_data/{email_encoded}.json`](user_data/) via [`UserPersistence`](src/services/user_persistence.rs)
- **All operations** (orders, services, wallet, products) read/written through persistence services

#### **Product Model: User-Owned SOT + Derived Catalog**
- **Products owned by provider users** in their persistent files
- **[`ProductService`](src/services/product.rs)** aggregates derived marketplace catalog
- **Category ID normalization** with optional dev TTL cache (disabled by default in prod)

#### **Currency System (Updated)**
- **TFC as base currency** with display currencies: TFC/USD/EUR/CAD (extensible)
- **TFC credits settle** at 1 TFC = 1 USD (fixed exchange rate)
- **Server-formatted amounts**: Server returns formatted display amounts and currency code
- **Frontend renders** without recomputing
- **User preference**: Users can choose display currency in settings

#### **Unified Insufficient-Balance Contract**
- **Target**: HTTP 402 Payment Required for all insufficient funds cases
- **Canonical error payload** with `error.details` containing currency-aware amounts and deficit

---

## 2. Complete User Experience Specification

### **Public Access (No Authentication Required)**

#### **Information Pages**
- `/docs` - Complete marketplace documentation
- `/privacy` - Privacy policy and data handling
- `/terms` - Terms and conditions
- `/about` - Marketplace information and mission
- `/contact` - Contact ThreeFold support

#### **Marketplace Browsing**
- `/marketplace` - Full marketplace access for browsing
- **Anonymous cart functionality** - Add items without login
- **Search and filtering** across all categories
- **Product details** and pricing information

### **Authentication & Registration**

#### **User Registration**
- `/register` - New user registration flow
- **GitEa OAuth integration** with seamless account creation
- **Profile setup** during registration

#### **User Login**
- `/login` - User authentication
- **Session management** with secure token handling
- **Cart migration** from anonymous to authenticated session

### **Purchase Workflows**

#### **Shopping Cart Management**
- **Anonymous users**: `/cart`
  - Add/remove items
  - View cart contents
  - Login prompt for checkout
  - **Cart preservation** during login transition
- **Authenticated users**: `/dashboard/cart`
  - Full cart editing (quantity adjustment)
  - Item removal and cart clearing
  - **Enhanced cart management** with user preferences

#### **Purchase Options**
- **Buy Now** - Direct purchase flow
- **Add to Cart** - Traditional shopping cart workflow
- **Checkout process** with credit validation
- **Order confirmation** and invoice generation

### **Marketplace Categories**

#### **Compute Resources**
- `/marketplace/compute` - Virtual machine slices
- **VM configuration** options (CPU, RAM, storage)
- **Kubernetes cluster** deployment options
- **Self-managed** resource mode
  - Users set SSH public key to access VM

#### **Infrastructure**
- `/marketplace/3nodes` - Complete server reservations
- `/marketplace/gateways` - Mycelium gateway services

#### **Applications & Services**
- `/marketplace/applications` - Published applications
- `/marketplace/services` - Professional services

### **User Dashboard & Profile Management**

#### **Dashboard Overview**
- `/dashboard` - Activity overview and notifications
- **Recent transactions** and order status
- **Quick actions** and shortcuts

#### **User Profile**
- `/dashboard/user` - Profile and activity management
- **My Applications** - Purchased and deployed apps
- **My Service Bookings** - Active service engagements
- **Deployment management** for purchased resources

#### **Wallet & Credits**
- `/dashboard/wallet` - Complete wallet management
- **Buy Credits** - Credit purchase functionality
- **Transfer Credits** - User-to-user transfers
- **Auto Top-Up Settings** - Automated balance management
- **Recent Transactions** - Complete transaction history

#### **Orders & History**
- `/dashboard/orders` - Complete order history
- **Order details** and status tracking
- **Invoice access** and download

### **Settings & Preferences**

#### **Profile Settings**
- `/dashboard/settings` - Comprehensive settings management
- **Profile Information**:
  - Name (editable)
  - Email (read-only)
  - Country
  - Time Zone
- **Password Management** - Secure password updates
- **Account Deletion** - Data retention policy compliance

#### **SSH Key Management**
- `/dashboard/settings` - SSH Keys section
- **Public key upload** for self-managed resources
- **Key management** (add, remove, edit)
- **Multiple SSH keys per user** with duplicate prevention
- **Real-time validation** with security level indicators
- **Integration** with VM and cluster deployments

##### SSH Keys UX Contract — Canonical Pattern

- Endpoints (ResponseBuilder envelope; always unwrap `const data = result.data || result;`):
  - `GET /api/dashboard/ssh-keys`
  - `POST /api/dashboard/ssh-keys`
  - `PUT /api/dashboard/ssh-keys/{id}`
  - `DELETE /api/dashboard/ssh-keys/{id}`
  - `POST /api/dashboard/ssh-keys/{id}/set-default`
  - `GET /api/dashboard/ssh-keys/{id}`
- Frontend implementation:
  - CSP-compliant JS in `src/static/js/dashboard-ssh-keys.js`
  - Views in `src/views/dashboard/settings.html` (no inline JS)
  - Read/write via fetch; unwrap `data` before accessing fields
- Verification workflow:
  - Use browser console to call each endpoint and confirm `{ success, data }` envelope
  - In UI: add/edit/delete keys, set default; verify list updates and validation messages
- Regression tests:
  - Reference: `tests/frontend_ux/settings_management_ux_test.rs` (per-step runner pattern)
  - Ensure tests assert ResponseBuilder unwrapping and persistence via user_data

#### **Notification Preferences**
- **Security Alerts** - Account security notifications
- **Billing Notifications** - Payment and credit alerts
- **System Alerts** - Downtime and maintenance
- **Newsletter** - Product updates and news
- **Dashboard Notifications** - In-app alert preferences
- **Message Notifications** - Real-time message alerts and desktop notifications

#### **Currency Preferences**
- **Display Currency Selection**:
  - TFC (base currency)
  - USD
  - EUR
  - CAD
- **Real-time conversion** display

##### Settings UX Contract — Canonical Pattern

- Endpoints (ResponseBuilder envelope; always unwrap `const data = result.data || result;`):
  - Profile: `POST /api/dashboard/settings/profile`
  - Password: `POST /api/dashboard/settings/password`
  - Verify Password: `POST /api/dashboard/settings/verify-password`
  - Notifications: `POST /api/dashboard/settings/notifications`
  - Delete Account: `POST /api/dashboard/settings/delete-account`
  - Billing History: `GET /api/dashboard/settings/billing-history`
  - Currency Preference: `GET /api/user/currency`, `POST /api/user/currency`
- Frontend contract:
  - Forms or fetch-based actions must read from `data` and render messages/status accordingly
  - Avoid recomputation; use server-formatted values when provided
  - Keep CSP compliance (no inline JS); hydrate via JSON or fetch
- Verification workflow:
  - Console: test endpoints and confirm `{ success, data }` envelope
  - UI: submit profile/password/notifications; verify success toasts and field updates
  - Currency: change preference and verify display reflects new currency
- Regression tests:
  - Covered by `tests/frontend_ux/settings_management_ux_test.rs` (validated passing)
  - Use per-step runner with persistence checks in `user_data/`

### **Communication & Messaging**

#### **Message Notification System**
- **Industry-standard notifications** across all platform touchpoints
- **Navbar integration** - Messages item in user dropdown with unread counter
- **Sidebar integration** - Enhanced message counter in dashboard navigation
- **Real-time updates** - 15-second polling with visual feedback
- **Desktop notifications** - Browser notifications for new messages (with permission)
- **Progressive disclosure** - Badge → dropdown preview → full messaging interface
- **Cross-platform consistency** - Unified notification badges and counters

##### Messaging UX Contract — Canonical Pattern

- **Core API endpoints** (ResponseBuilder envelope; always unwrap `const data = result.data || result;`):
  - `GET /api/messages/threads` - Thread list with unread counts and last messages
  - `POST /api/messages/threads` - Create new conversation thread (requires `Content-Type: application/json`)
  - `GET /api/messages/threads/{id}/messages` - Get messages for specific thread
  - `POST /api/messages/threads/{id}/messages` - Send message to thread
  - `PUT /api/messages/threads/{id}/read` - Mark thread as read (resets unread count)
- **Frontend implementation**:
  - `src/static/js/messaging-system.js` - Core messaging functionality and modal interface
  - `src/static/js/notification-system.js` - Cross-platform notification management (15s polling)
  - `src/static/js/dashboard-messages.js` - Full-page messaging interface at `/dashboard/messages`
  - Real-time badge updates via polling and custom event system
- **Thread creation workflow**:
  - Contact Provider buttons redirect to `/dashboard/messages?recipient=email&context=type&booking_id=id`
  - System checks for existing thread by `recipient_email`, `context_type`, and `context_id`
  - If no thread exists, creates new thread via POST with proper headers
  - Auto-selects newly created thread by matching both `recipient_email` AND `context_id`
- **Notification locations**:
  - Navbar user dropdown (between Wallet and Settings) with unread counter badge
  - Dashboard sidebar (Communication section) with red danger badge and pulse animation
  - Document title updates showing unread count
  - Desktop notifications (with user permission) for new messages
- **Visual standards**:
  - Red danger badges for unread counts with 99+ cap for high counts
  - Pulse animation for new notifications
  - Consistent styling with cart counter patterns
  - Toast notifications (top-right, error-only for cleaner UX)

#### **Full-Page Messaging Interface**
- **Route**: `/dashboard/messages` - Dedicated messaging dashboard page
- **Controller**: `DashboardController::messages_page()` with proper context setup
- **Template**: `src/views/dashboard/messages.html` - Industry-standard messaging layout
- **Architecture**:
  - **Left panel**: Conversation list with unread indicators and context badges
  - **Right panel**: Message view with fixed header, scrollable messages, anchored input
  - **Flexbox layout**: `calc(100vh - 200px)` height with proper viewport utilization
  - **Message input**: Fixed at bottom using `card-footer` with `flex-shrink-0`
- **UX Standards**:
  - **Toast notifications**: Top-right positioning, error-only (success removed for cleaner UX)
  - **Real-time updates**: 15-second polling with immediate message display
  - **Character validation**: 1000 character limit with live counter
  - **Responsive design**: Mobile and desktop optimized
  - **CSP compliance**: External JS with JSON hydration pattern

#### **Message Thread Management**
- **Thread Structure**: Each thread connects two users (`user_a_email`, `user_b_email`) with context
- **Context Types**: `service_booking`, `slice_rental`, `general` - determines thread purpose
- **Context ID**: Optional identifier (e.g., booking ID) for specific business objects
- **Unread Tracking**: Per-user unread counters (`user_a_unread_count`, `user_b_unread_count`)
- **Persistence**: Stored in user data files under `message_threads` and `messages` arrays
- **Thread Creation**: Automatic via Contact Provider buttons or manual via messaging interface
- **Thread Selection**: Matches by `recipient_email` + `context_id` for precise targeting

#### **Integration Points**
- **Service Bookings**: Contact Provider buttons in `/dashboard/user` My Service Bookings
- **Slice Rentals**: Contact Provider functionality for compute resource support
- **General Messaging**: Direct user-to-user communication without specific context
- **Provider Dashboards**: Incoming message notifications for service/app/farmer providers
- **Notification System**: Cross-platform badges, desktop notifications, document title updates

### **Provider Dashboards**

#### **Farmer Dashboard**
- `/dashboard/farmer` - Resource provider management
- **Add Nodes** - Grid node registration
- **Node Management** - Active node monitoring
- **Slice Distribution** - Resource allocation management
- **Revenue Tracking** - Earnings and analytics

#### **App Provider Dashboard**
- `/dashboard/app-provider` - Application provider tools
- **Register Applications** - New app publishing
- **My Published Applications** - App management table
- **Active Deployments** - Customer deployment tracking
- **Revenue Analytics** - App earnings dashboard

#### **Service Provider Dashboard**
- `/dashboard/service-provider` - Professional service management
- **Register Services** - New service offerings
- **My Service Offerings** - Service catalog management
- **Service Requests** - Customer request pipeline:
  - Open
  - In Progress
  - Completed
- **Client Management** - Customer relationship tools

---

## 5. Technical Implementation Guidelines

### **Currency System Implementation**
```rust
// Updated currency constants
const BASE_CURRENCY: &str = "TFC";
const TFC_TO_USD_RATE: f64 = 1.0;

// Currency display preferences
struct CurrencyPreference {
    user_id: String,
    display_currency: String, // TFC, USD, EUR, CAD, where USD is default
    created_at: DateTime<Utc>,
}
```

### **SSH Key Management**
```rust
// SSH key storage structure
struct SSHKey {
    id: String,
    user_id: String,
    name: String,
    public_key: String,
    fingerprint: String,
    created_at: DateTime<Utc>,
}
```

### **Cart Management Pattern**
```rust
// Cart persistence and migration
struct CartItem {
    product_id: String,
    quantity: u32,
    configuration: Option<serde_json::Value>,
}

struct Cart {
    session_id: Option<String>, // For anonymous carts
    user_id: Option<String>,    // For authenticated carts
    items: Vec<CartItem>,
    created_at: DateTime<Utc>,
    updated_at: DateTime<Utc>,
}
```

---

## 8. Architecture Decision Records (Updated)

### **UX-First Development**
- **Decision**: Complete UX implementation before backend infrastructure
- **Rationale**: User experience validates product-market fit and drives API requirements
- **Implementation**: Phased approach with UX testing as validation gate

### **TFC Base Currency**
- **Decision**: TFC as base currency instead of USD
- **Rationale**: Align with ThreeFold ecosystem and user expectations
- **Implementation**: 1 TFC = 1 USD fixed rate with display preferences

## 9. Concrete UX List

- Publicly available information
  - A user can consult the docs at /docs
  - A user can consult the privacy policy at /privacy
  - A user can consult the Terms and Conditions at /terms
  - A user can read about the marketplace at /about
  - A user can contact ThreeFold by consulting the page /contact
- Registration and Login
  - A user can register at /register
  - Once registered they can log out and then log in at /login
- Purchases
  - A non-logged in user can check the marketplace at /marketplace and add items to cart
  - To buy an item, the user must be logged in and have enough credits
- Purchase UX
  - A user can always buy a product or service in two ways
    - They can Buy Now, or Add to Cart (then proceed to checkout and complete the purchase)
  - A non-logged in user will have access to the cart at /cart
    - If they then log in, the cart items will be added to their account's cart
    - In the /cart page, they can clear cart or delete individual items in the cart
    - If they try to edit the cart, a message appears telling them to log in to edit the cart
  - A logged in user will have access to the cart at /dashboard/cart
    - Then they add to cart, they can see their cart at /dashboard/cart
    - In the /cart page, they can clear cart or delete individual items in the cart
    - As a logged in user, they can edit the cart, e.g. click + to add more of that same item, or click - to remove that item
  - A user can see all their purchases at /dashboard/orders
  - A user can see their recent transactions at /dashboard/wallet Recent Transactions
- Credits
  - A user can buy credits at /dashboard/wallet, Buy Credits
  - A user can transfer credits to another user at /dashboard/wallet, Transfer Credits
  - A user can set up Auto Top-Up Settings at /dashboard/wallet Auto Top-up Settings
    - Thus if a user buys an app that costs e.g. 10 USD per month, they can set an automatic wallet top-up to never go below that number, ensuring their app will never run out of credits
- Marketplace
  - In the marketplace, a user can search and filter items in different categories
    - They can buy compute resources (slices) at /marketplace/compute
    - They can reserve a complete node (server) at /marketplace/3nodes
    - They can buy Mycelium gateways at /marketplace/gateways
    - They can buy applications (solutions) at /marketplace/applications
    - They can buy services at /marketplace/services
- Settings
  - A user can change settings at /dashboard/settings
    - They can
      - Update the profile information (email can't be changed)
        - Name
        - Country
        - Time Zone
      - Change the password
      - Set Up SSH Public Keys
      - Update notification settings
        - Security alerts
        - Billing notifications
        - System alerts (downtime, maintenance)
        - Newsletter and product updates
        - Dashboard Notifications
          - Show alerts in dashboard
          - Show update notifications
      - Update their currency preferences
        - They can decide to have the prices displayed in
          - USD
          - TFC
          - EUR
          - CAD
      - Delete account
        - When a user deletes their account, their data is still available on the marketplace backend for security, audit and legal purposes
- Dashboard UX and Provider+Consumer Interactions
  - A user can see their dashboard overview activities at /dashboard
  - A user can see their user profile and activities at /dashboard/user
  - A user provoding resources to the grid (aka a farmer) at /dashboard/farmer
    - There they can add resources, e.g. add a node to the marketplace which will be available for purchase by other users as slices (a node is distributed as slices)
  - A user can become an app provider at /dashboard/app-provider by registering new applications
    - When a user register an application
      - The application is displayed publicly at /marketplace/applications
      - The application is shown at the app provider dashboard /dashboard/app-provider at the table My Published Applications
      - When another user buys that app
        - The app provider will see that info at the table /dashboard/app-provider Active Deployments
        - The app purchaser will see the app info at /dashboard/user My Applications
  - A user can become a service provider at /dashboard/service-provider by registering new services
    - When a user register a service
      - The service is displayed publicly at /marketplace/services
      - The service is shown at the service provider dashboard /dashboard/service-provider at the table My Service Offerings
      - When another user buys that service
        - The service provider will see that info at the table /dashboard/service-provider Service Requests
          - There are 3 stages to this service request from the service provider POV
            - Open
            - In Progress
            - Completed
        - The service purchaser will see the service info at /dashboard/user My Service Bookings
  - A user can become a resource provider by adding nodes and thus resources to the marketplace at /dashboard/farmer
    - They can Add Nodes
      - Then it fetches the nodes on the ThreeFold Grid and distribute the node into slices
      - Then the slices are can be seen publicly at /marketplace/compute
    - Any user can then purchase slices from that farmer and access the slices
- Products
  - Farmers at the core offer compute resources (slices) to the marketplace.
  - On their end, farmers host nodes on the threefold grid. nodes are split into slices.
  - Users can use slices for individual VMs, or for Kubernetes cluster
    - The UI is intuitive
      - .html_template_tests
- Apps can be self-managed and managed
  - An app is at its core a slice (VM or Kubernetes cluster) with an app on top
  - for self-managed node, users can set their SSH public key, they set it in /dashboard/settings SSH Keys page
  - for managed node, users will have access to the credentials on their marketplace dashboard in /dashboard/user page at the section of the app/product they rent/bought

### Testing the UX

- The marketplace should have a complete test suite to confirm the frontend UX is as expected for usersT
- Thus, there should be a series of test for the UX above
- We should have the tests mentioned above in an isolated section, e.g. not shared with other tests
  - This ensures that the tests are not affected by other tests

### Marketplace-Wide UX Contract Audit Plan

**Goal:** Ensure every UX flow follows the canonical API envelope and CSP-compliant frontend contract, with deterministic tests and persistence-backed verification. The complete UX should be complete and functioning.

- **Scope (frontend code):**
  - `src/static/js/**/*.js` and `static/js/**/*.js` (external JS only; no inline handlers)
  - Views in `src/views/**` (JSON hydration blocks only; keep CSP clean)
- **Scope (backend routes):**
  - `src/routes/mod.rs` is the single source of truth for endpoint paths
- **Contract checks (apply to every fetch):**
  - Unwrap ResponseBuilder: `const result = await response.json(); const data = result.data || result;`
  - Render optional numbers safely: show "No limit" for null/undefined values
  - Prefer server-formatted currency/amounts; avoid client recomputation
  - Keep all logic in external JS; hydrate via `<script type="application/json">`
- **Priority UX areas to audit:**
  - Wallet & credits: `src/static/js/dashboard_wallet.js`, `/api/wallet/*`
  - Auto top-up status: `GET /api/wallet/auto-topup/status`
  - SSH keys: `src/static/js/dashboard-ssh-keys.js`, `/api/dashboard/ssh-keys*`
  - Settings: `/api/dashboard/settings/*` forms and responses
  - Cart, orders, products: `/api/cart*`, `/api/orders*`, `/api/products*`
- **Verification workflow:**
  - Console: `fetch('<endpoint>').then(r=>r.json()).then(console.log)` to confirm `{ success, data }`
  - UI: ensure badges/labels and form prefills reflect `data`
  - Cross-check each referenced path with `src/routes/mod.rs`
- **Test adoption:**
  - Use per-step runner pattern (see `tests/frontend_ux/settings_management_ux_test.rs`)
  - Persist state under `user_data/` and assert post-conditions
  - Run: `cargo test --features ux_testing -- --nocapture`
- **Reporting:**
  - Track per-feature audit items in this roadmap; document deviations and fixes under each UX contract subsection