Files
projectmycelium/docs/dev/design/archive/marketplace-architecture.md
2025-09-01 21:37:01 -04:00

80 KiB
Raw Blame History

Project Mycelium - Master Architecture Guide v2.0

Complete Single Source of Truth for AI-Assisted Development

Document Purpose: Comprehensive architectural reference consolidating ALL design decisions, implementation patterns, roadmaps, and development standards for the Project Mycelium. This serves as the definitive guide for AI coders, developers, and system architects.

Last Updated: 2025-08-15 12:41 Architecture Version: 3.2 Status: Production Implementation Active


🎯 Executive Summary

The Project Mycelium has evolved into a production-ready, industry-standard platform built on comprehensive builder pattern architecture, log-free codebase principles, and persistent data-only operations. The system eliminates all mock data, centralizes all construction patterns, and maintains zero compilation errors through systematic architectural decisions.

Key Achievements (2025)

  • 881+ Log Statements Removed: Complete log-free codebase
  • 245+ ResponseBuilder Patterns Migrated: 10 controllers complete, 1 in progress
  • 1,120+ Lines of Code Reduced: Through builder pattern consolidation
  • 100% Mock Data Elimination: Persistent data-only architecture
  • Zero Compilation Errors: Clean builds maintained throughout
  • Single Source of Truth: All construction patterns centralized
  • TFC Credits System: Industry-standard credit model (1 TFC = 1 USD)
  • Major Controllers Completed: Marketplace, Rental, Pool, Order, Simple Controllers
  • Critical Milestone: Dashboard compilation errors fixed (18 errors → 0 errors)
  • Public Pages Added: Changelog and Roadmap pages with comprehensive project status
  • Public Controller: New PublicController for legal, changelog, and roadmap pages
  • Authentication & Buy-Now Flow Fixed: Critical wallet balance issue resolved
  • Middleware Authentication: Fixed route exclusion and session validation
  • Frontend-Backend Integration: ResponseBuilder pattern compatibility established
  • Order Management System: Complete user-centric order persistence and display
  • Payment Method Persistence: Enhanced UX with last payment method remembering
  • Authentication Flow Enhancement: Improved buy-now messaging and redirect to dashboard
  • Preferred Currency Displayed Throughout: Navbar, Wallet, Orders, and Dashboard use server-formatted values; dashboard overview wallet label made dynamic via navbar dropdown API

🏗️ Core Architectural Principles

1. Log-Free Codebase Policy 🚫📝

Policy: Zero log:: statements in main/development branches

Implementation:

  • Developers may use log:: for troubleshooting in feature branches
  • All logs must be removed before merging to main/development
  • Automated cleanup: find src -name "*.rs" -exec perl -i -pe 'BEGIN{undef $/;} s/\s*log::[^;]*;//g' {} \;

Benefits:

  • Clean, production-ready code
  • Simplified AI-assisted development
  • Reduced file complexity and size
  • Eliminated performance overhead

2. Builder Pattern Architecture 🏗️

Philosophy: Single-source-of-truth construction for all complex objects

Core Builders:

  • SessionDataBuilder: UserPersistentData construction
  • ConfigurationBuilder: Environment variable access
  • ResponseBuilder: HTTP response handling
  • ServiceFactory: Service instantiation

ResponseBuilder Pattern - Critical Frontend Integration ⚠️

IMPORTANT: All API responses are automatically wrapped by ResponseBuilder

Response Structure:

// Backend Controller Code:
ResponseBuilder::ok().json(serde_json::json!({
    "success": true,
    "last_payment_method": "credit_card"
})).build()

// Actual API Response (wrapped by ResponseBuilder):
{
    "data": {
        "success": true,
        "last_payment_method": "credit_card"
    },
    "success": true
}

Frontend Compatibility Pattern:

// ✅ CORRECT - Handle ResponseBuilder wrapping
const response = await fetch('/api/endpoint');
const result = await response.json();
const data = result.data || result;  // Handle both wrapped and unwrapped

if (data.success && data.field) {
    // Use data.field
}

// ❌ INCORRECT - Direct access fails with ResponseBuilder
if (result.success && result.field) {
    // This fails because actual data is in result.data.field
}

Implementation Examples:

  • Orders: data.data.orders (nested due to ResponseBuilder + order response structure)
  • Payment Methods: data.last_payment_method (single ResponseBuilder wrapping)
  • Wallet Balance (Navbar dropdown): data.wallet_balance_formatted and data.display_currency from /api/navbar/dropdown-data
  • Wallet Balance (direct API, if used): data.balance or similar numeric field; always unwrap via const data = result.data || result;

Key Rules for AI Coders:

  1. Always use: const data = result.data || result; for API responses
  2. Test both: Wrapped and unwrapped response structures
  3. Debug with: console.log('API response:', result); to see actual structure
  4. Remember: ResponseBuilder adds automatic {"data": {...}, "success": true} wrapper

CSP-Compliant Frontend: External JS + JSON Hydration (2025-08-11)

Policy: Zero inline scripts and zero inline event handlers. All page behavior lives in external JS with JSON hydration blocks.

Implementation:

  • Externalized base and page scripts to src/static/js/ (e.g., base.js, wallet.js, statistics.js).
  • Added {% block scripts %}{% endblock %} in src/views/base.html for page-specific includes.
  • Pages provide a <script type="application/json" id="..."> hydration block. All injected values are server-rendered and must use json_encode to produce valid JSON under CSP.

Hydration example (wallet):

<script type="application/json" id="wallet-hydration">
  {"currency_symbol": {{ currency_symbol | default(value='$') | json_encode() }}}
</script>

Recent Fix (root cause of server exit):

  • Tera failed to parse src/views/wallet/index.html due to invalid filter usage inside JSON and quoting issues.
  • Fixed by using the named-arg form default(value='$') and emitting JSON via json_encode().
  • Added explicit Tera init logging in src/main.rs so startup errors print to stderr:
    • File: src/main.rs → log Tera initialization error: {e} before exiting.
  • Outcome: cargo run --bin projectmycelium now keeps the Actix server running; template errors are clearly reported.

Benefits:

  • Security: Strict CSP compliance (no inline code) reduces XSS risk.
  • Maintainability: Clear separation of structure (HTML), data (hydration JSON), and behavior (external JS).
  • Performance: External JS is cacheable across pages; smaller HTML payloads.
  • Testability: Hydration data is deterministic JSON; easier unit/integration tests.
  • Consistency: One binding pattern via DOMContentLoaded and data attributes, not scattered inline handlers.

Rules of use:

  • Never use onclick/onchange in templates; bind events in JS on DOMContentLoaded.
  • Always emit hydration via <script type="application/json" id="..."> and parse it in JS.
  • Always json_encode any value rendered into hydration JSON; prefer named-arg filters (e.g., default(value='...')).

Files updated:

  • src/static/js/base.js (navbar/cart globals), src/static/js/wallet.js, src/static/js/statistics.js, src/static/js/checkout.js, src/static/js/services.js.
  • src/views/base.html (external scripts block), src/views/wallet/index.html (hydration block fix), src/views/marketplace/checkout.html, src/views/marketplace/services.html.
  • src/main.rs (Tera initialization error logging for startup diagnostics).

Next steps to complete CSP externalization:

  • Externalize remaining inline handlers across marketplace and dashboard templates (cart/checkout/orders/services, pools, user, farmer, service_provider, invoice/report print buttons) using per-page JS and hydration.
    • Replace inline window.print() with a shared print utility and bind via data-action="print".
    • Audit all hydration blocks to use json_encode and provide {} when no data is needed.

Services Page Externalization — Completed 2025-08-11

  • Removed all inline scripts and onclick/onchange handlers from src/views/marketplace/services.html.
  • Added CSP-safe hydration block: <script type="application/json" id="services-data">{}</script> and stable container id id="services-grid".
  • Implemented src/static/js/services.js to bind .add-to-cart-btn clicks, show auth modal on 401, listen for serviceCreated events, and render service cards without inline handlers.
  • Integrates with base.js globals when present (updateCartCount, emitCartUpdated).

Dashboard Orders Page Externalization — Completed 2025-08-11

  • Removed all inline scripts and event handlers from src/views/dashboard/orders.html.
  • Added CSP-safe hydration block #orders-hydration with per-field JSON encoding:
<script type="application/json" id="orders-hydration">
  {
    "currency_symbol": {{ currency_symbol | default(value='$') | json_encode() }},
    "display_currency": {{ display_currency | default(value='USD') | json_encode() }}
  }
</script>
  • Included external script: src/static/js/dashboard_orders.js.
  • Event delegation via data-action attributes:
    • toggle-details, view-invoice, contact-support, view-invoice-from-modal.
  • ResponseBuilder compatibility handled in JS: const payload = result.data || result;.
  • Tera parse fix: replaced quoted defaults inside JSON with named-arg defaults and json_encode() per field to emit valid JSON; resolved the startup error “expected a value that can be negated or an array of values”.

CSP Externalization COMPLETE — 2025-08-12

ACHIEVEMENT: The entire Project Mycelium is now 100% CSP-compliant with zero inline handlers across all templates.

Dashboard Service Provider Externalization — COMPLETED:

  • Removed inline handlers for availability and service creation in src/views/dashboard/service_provider.html by using data-action attributes and external JS.
  • Implemented delegated handlers in src/static/js/dashboard-service-provider.js:
    • availability.toggletoggleAvailability() for checkbox changes (no API write).
    • availability.updateupdateAvailability() PUT /api/dashboard/availability with validation and notifications.
    • services.createcreateNewService() opens modal, validates, submits to API, and refreshes lists.
  • Fixed and initialized loadAvailabilitySettings() on DOMContentLoaded to hydrate UI safely under CSP.
  • Added delegated handler: data-action="services.saveChanges" → calls existing saveServiceChanges() for the Save Changes button (no inline JS).
  • Hydration: confirmed CSP-safe JSON block #sp-dashboard-hydration present and parsed in initialization.
  • Tera parse fix applied to src/views/dashboard/service_provider.html: moved page-specific styles to {% block head %} and hydration/external scripts to {% block scripts %} at the top level; dashboard_content now contains only markup/modals.

Final CSP Cleanup — COMPLETED 2025-08-12:

  • src/views/cart.html — Converted editCartItem() & removeCartItem() to data-action="cart.edit" & data-action="cart.remove"
  • src/views/dashboard/user.html — Converted viewBookingDetails() & contactProvider() to data-action="booking.view" & data-action="provider.contact"
  • src/views/dashboard/farmer.html — Converted 6 inline handlers to data-action patterns:
    • refreshSliceCalculations()data-action="slice.refresh"
    • syncWithGrid()data-action="grid.sync"
    • viewNodeSlices()data-action="node.view"
    • setMaintenanceMode()data-action="node.maintenance"
    • restartNode()data-action="node.restart"

Create Service Modal Flicker — Root Cause Fix (2025-08-12)

  • Root cause: Background elements (cards, rows, service items) had broad transition: all and hover effects. During .modal.fade, these hover transitions triggered expensive repaints, causing visible flicker on modal buttons.
  • Fix: Preserve Bootstrap fade while eliminating repaint conflicts.
    • Tighten transitions to specific properties (e.g., box-shadow, transform, background-color).
    • Suppress hover-driven transitions while any modal is open via body.modal-open rules (reset transforms/box-shadows/bg-color to static values).
    • Restore .modal.fade on #createServiceModal to keep smooth animation.
    • Add light GPU compositing hints on modal buttons (will-change, transform: translateZ(0), backface-visibility: hidden).
    • Verified single-initialization for delegated handlers and Enter-key submit prevention on modal forms.
  • Files changed: src/static/css/styles.css, src/views/dashboard/service_provider.html.

CSP Verification Results — 2025-08-12:

  • Zero inline handlers found across all templates (confirmed via comprehensive audit)
  • All external JS files exist and properly structured
  • JSON hydration blocks in place for CSP-safe data passing
  • Print utilities confirmed available

COMPLETE: Global CSP Compliance Achieved (100%):

  • Dashboard templates
    • src/views/dashboard/wallet.html (done)
    • src/views/dashboard/pools.html (done)
    • src/views/dashboard/service_provider.html (COMPLETED)
    • src/views/dashboard/user.html (COMPLETED 2025-08-12)
    • src/views/dashboard/farmer.html (COMPLETED 2025-08-12)
    • src/views/dashboard/orders.html (done)
    • src/views/dashboard/cart.html (done)
    • src/views/dashboard/welcome.html (no inline handlers found)
  • Dashboard JS
    • src/static/js/dashboard_pools.js
    • src/static/js/dashboard-service-provider.js (COMPLETED)
    • src/static/js/dashboard_user.js (verified existing)
    • src/static/js/dashboard_farmer.js (verified existing)
    • src/static/js/dashboard_orders.js (done)
    • src/static/js/dashboard_cart.js (done)
    • src/static/js/print-utils.js (verified existing)
  • Marketplace templates
    • src/views/marketplace/services.html (done)
    • src/views/marketplace/checkout.html (done)
    • src/views/cart.html (COMPLETED 2025-08-12)
    • All other marketplace templates (no inline handlers found)
  • Marketplace JS
    • src/static/js/services.js (done)
    • src/static/js/checkout.js (done)
    • src/static/js/cart.js (verified existing)
    • All other required JS files (verified existing)

Security & Compliance Benefits Achieved:

  • Strict CSP compliance — No inline code execution vectors
  • XSS risk reduction — All user interactions via external event delegation
  • Maintainability — Clean separation of HTML structure, JSON data, and JS behavior
  • Performance — External JS cacheable across pages, smaller HTML payloads
  • Testability — Deterministic hydration JSON, easier unit/integration tests

Tera JSON Hydration & Template Safety Guidelines (Must Read)

To avoid Tera parse errors and ensure CSP-safe, valid JSON in hydration blocks, follow these rules:

  • Always encode each field individually

    • Use: {{ value | default(value=...) | json_encode() }} per field.
    • Strings: use named-arg defaults, e.g., default(value='USD').
    • Optionals: use default(value=null) to produce null.
    • Note: Avoid default(value=none) as some template contexts can treat none as an unresolved variable. If in doubt, prefer explicit null or guard with {% if var is defined %}{{ var | json_encode() }}{% else %}null{% endif %}.
  • Do NOT create an object literal inside a single interpolation

    • Dont: {{ { "user_currency": user_currency, "cart_details": cart_details } | json_encode }}
    • This can trigger: “expected a value that can be negated or an array of values”.
  • Recommended pattern for hydration blocks

    • Build the JSON shape literally and encode each field:
    <script type="application/json" id="checkout-hydration">
      {
        "user_currency": {{ user_currency | default(value='USD') | json_encode() }},
        "cart_details": {{ cart_details | default(value=none) | json_encode() }}
      }
    </script>
    
  • When to use safe

    • type="application/json" blocks are not executed; JSON encoding avoids HTML escaping.
    • safe is typically unnecessary if you encode each field. If you emit one fully-encoded JSON string, you may append | safe.
  • General Tera tips

    • Prefer named arguments in filters: default(value='...').
    • Never mix template control structures inside JSON literals.
    • Log startup errors: we print Tera init errors in src/main.rs to surface issues early.
    • Validate JSON quickly by copying the block into a linter when in doubt.

Checkout fix (2025-08-11): src/views/marketplace/checkout.html switched from a single interpolated object to the per-field encoding pattern above, resolving the runtime Tera parse error.

Tera Block Structure Common Pitfall (Fixed 2025-08-12)

Tera blocks cannot be nested inside other blocks. Child templates must keep structural blocks (head, scripts) at the top level and place page markup inside the content block only.

  • Do:

    • Put page styles inside {% block head %}...{% endblock %}.
    • Put hydration JSON and external JS includes inside {% block scripts %}...{% endblock %}.
    • Keep dashboard_content (or content) limited to HTML and modal markup only.
  • Don't:

    • Nest {% block scripts %} or style blocks inside dashboard_content.
  • Recent fix: src/views/dashboard/service_provider.html raised an "unexpected {% endblock %}" parse error due to nested blocks. We refactored by moving:

    • Styles → top-level {% block head %}
    • Hydration JSON (#sp-dashboard-hydration) and external JS includes → top-level {% block scripts %}
    • Result: Tera parses cleanly; CSP structure is preserved.

DevX: Rust Error-Only Compilation Logs — Added 2025-08-12

To streamline debugging and keep output readable, we added a small toolchain to capture only Rust compiler errors (no warnings) from cargo check:

  • Script: scripts/dev/cargo-errors.sh
  • Make targets:
    • make check-errors
    • make fixtures-errors (runs with fixtures env)
  • Output: /tmp/cargo_errors_only.log (override with OUT=/path/to/file.log)
  • Dependencies: Prefers jq for precise filtering; falls back to a sed/awk parser if jq is not installed.

Usage examples:

# Standard
make check-errors

# Fixtures mode
make fixtures-errors

# Custom output path
OUT=/tmp/my_errors.log make check-errors

# Quick count/peek
grep -c '^error' /tmp/cargo_errors_only.log
sed -n '1,80p' /tmp/cargo_errors_only.log

Behavior:

  • Suppresses warnings entirely; logs only error diagnostics.
  • Returns cargo check exit code (nonzero when errors present), so it fits CI or hooks.
  • Color is disabled via CARGO_TERM_COLOR=never to keep the log clean.

3. Persistent Data Only 💾

Policy: No mock data in production code

  • All data operations use user_data/ directory
  • Real user data drives marketplace content
  • MockDataService completely eliminated

User-Centric Data Storage Pattern 🎯

Architectural Decision: Orders stored as part of user's persistent data (UserPersistence)

  • Localized Source of Truth: Each user owns their complete data set
  • Industry Standard: Follows GDPR compliance and data portability requirements
  • Implementation: Orders stored in UserPersistentData.orders vector
  • Data Flow: Purchase → UserPersistence → API → ResponseBuilder → Frontend
  • Benefits: Simplified data management, easier account operations, better user privacy

Order Management Implementation:

  • instant_purchase.rs: Orders pushed to persistent_data.orders vector
  • order.rs: get_user_orders() reads from UserPersistence, sorts by created_at DESC
  • orders.html: Frontend handles nested ResponseBuilder JSON structure (data.data.orders)
  • ResponseBuilder Compatibility: All API responses wrapped in {"data": {...}, "success": true}

Marketplace Order Invoices (2025-08-09)

  • Endpoint: GET /orders/{id}/invoice renders an HTML invoice view for printing.
  • Controller: OrderController::get_order_invoice in src/controllers/order.rs.
    • Ownership check prefers session["user_email"] for user identity.
    • Dual-source order retrieval: first check in-memory OrderStorage; if not found, fallback to persistent UserPersistence (email-based). This fixes buy-now invoice access.
  • Template: src/views/marketplace/order_invoice.html renders order id, date, items, qty, unit price, line totals, subtotal/total, payment method, billing email, and includes a Print button.
  • Frontend UI: src/views/dashboard/orders.html opens the HTML view in a new tab.
    • CTA text changed to "View Invoice" with eye icon.
    • JS handlers renamed: downloadInvoiceviewInvoice, downloadInvoiceFromModalviewInvoiceFromModal.
  • Not implemented (by design): direct download API (/api/orders/{id}/invoice).
    • Rationale: Current UX uses "View Invoice" + browser Print → Save as PDF; avoids extra dependencies/complexity; no immediate need for programmatic downloads. Can be added later if integrations require it.

Service Requests and Bookings Flow — Current Status and Fix Plan (2025-08-14)

• Current state

  • Providers can create services; buyers can purchase them.
  • Buyer sees entries under “My Service Requests” (/dashboard/user).
  • Provider sees entries under “Service Requests” (/dashboard/service-provider).
  • Provider lookup/persistence fixed: requests saved under correct provider file.

• Issues observed

  • Status mismatch: when provider accepts (In Progress), buyer still sees Pending.
  • View CTA error: “Cannot read properties of undefined (reading 'id')” (likely envelope/shape mismatch).
  • Update CTA causes the request to disappear (tab filtering not refreshed or destructive overwrite/mock fallback).
  • Buyer booking details view shows only a success button (insufficient detail rendering).

• Backend actions

  • Synchronize status updates across users:
    • In DashboardController::update_service_request_progress (and update_service_request), update both:
      1. Provider service_requests in user_data/{provider}.json.
      2. Buyer service_bookings in user_data/{buyer}.json.
    • Use stable identifiers: request.id, customer_email, and optionally linked_booking_id.
    • Ensure non-destructive in-place updates via UserPersistence (find by id; mutate fields; keep others).
    • Add structured logs: before/after counts, updated ids, file paths; bubble errors.
  • Normalize details endpoints to a consistent DTO and ResponseBuilder envelope:
    • GET /api/dashboard/service-requests/{id}/details, .../completed-details, .../report → return { success, data: { id, status, product, customer_email, ... } }.

• Frontend actions

  • Always unwrap API with const data = result.data || result; and reference data.id.
  • After updates, re-render tables and migrate rows between tabs (Pending → In Progress → Completed) instead of removing.
  • Improve buyer “My Service Bookings” details to render provider/product/status timeline.
  • Guard null/undefined records and missing properties.

• Testing

  • E2E: provider creates → buyer purchases → provider accepts.
    • Assert provider request status and buyer booking status both advance to In Progress.
  • API contract tests: details endpoints include id and consistent field names.

• Observability

  • Add structured logging to update endpoints and persistence helpers to trace ids, counts, and file paths during updates.

Progress Update — Service Requests & Bookings (2025-08-14 13:35)

  • Removed inline mock client_requests arrays in src/models/user.rs (replaced with Vec::new()), aligning with the persistentdataonly policy and fixing compile errors from new fields.
  • Confirmed ServiceRequest includes hours_worked: Option<f64> and notes: Option<String> with #[serde(default)]; all construction sites initialize them (e.g., src/services/order.rs sets None).
  • Controllers dashboard::update_service_request and dashboard::update_service_request_progress now normalize responses via ResponseBuilder, reload the updated request, and synchronize the buyers service_bookings with the providers service_requests.
  • Persistence (UserPersistence): added/used helpers to update buyer booking fields; persist hours_worked and notes; set completed_date when status is Completed or progress ≥ 100.
  • Build status: cargo check passes; only warnings remain (unrelated to this flow).

Next:

  • Remove remaining inline mocks (e.g., revenue_history) and source exclusively from persistent storage.
  • Audit update paths to ensure APP_ENABLE_MOCKS=0 has zero mock fallbacks.

Progress Update — Service Provider Dashboard Persistent Data (2025-08-15 08:46)

  • Template refactor: src/views/dashboard/service_provider.html now binds exclusively to injected persistent context service_provider_data (e.g., active_services, total_clients, monthly_revenue_usd). Removed all user.mock_data.service_provider_data.* references.
  • Hydration: The #sp-dashboard-hydration JSON block outputs the persistent service_provider_data using json_encode() for CSP-safe parsing.
  • Frontend JS: src/static/js/dashboard-service-provider.js aligned to persistent USD fields (monthly_revenue_usd, total_revenue_usd, price_per_hour_usd). Fallback init payload keys updated to *_usd. ResponseBuilder unwrap pattern enforced (const data = result.data || result;). Removed legacy/mock field usage.
  • Backend controller: DashboardController::service_provider_section constructs and injects persistent ServiceProviderData via UserPersistence (summary metrics, services, client requests). No mock fallbacks remain. Placeholder series like revenue_history are empty-by-design, not mock-driven.
  • Verification: cargo check passes; manual dashboard UI test under APP_ENABLE_MOCKS=0 confirms cards, tables, and charts render from persistent data/hydration.

Next:

  • Apply the same persistent-only mapping to /dashboard/app-provider and /dashboard/user (templates + JS + controller sections). Ensure *_usd naming in UI logic and remove any user.mock_data access.
  • Replace/remove any remaining placeholder arrays (e.g., revenue_history) with persisted metrics or clearly mark as not-yet-implemented (no mock values).

Progress Update — App Provider Dashboard Persistent Data (2025-08-15 10:30)

  • Frontend JS (src/static/js/dashboard-app-provider.js):
    • Removed legacy monthly_revenue fallbacks and any sessionStorage merges; all revenue logic uses persistent *_usd fields only (e.g., monthly_revenue_usd).
    • Treated deployment status "Running" as "Active" for UI consistency.
    • Exposed window.__appProviderDashboard for modal helpers to estimate revenue.
    • Updated deployment details modal to use estimated deployment revenue derived from app-level revenue.
  • Backend (src/controllers/dashboard.rs):
    • Fixed GET /api/dashboard/deployment/{id} to return proper JSON success/failure envelopes via ResponseBuilder.
    • json_to_app helper reads monthly_revenue_usd with a soft fallback to legacy monthly_revenue for backward compatibility.
  • Verification: UI tested under APP_ENABLE_MOCKS=0; charts, tables, and modals source only persistent data. ResponseBuilder unwrap pattern enforced on the frontend.

Progress Update — Dashboard Overview Activities Mapping — COMPLETED (2025-08-15 10:30)

  • Root cause: src/views/dashboard/index.html referenced activity.date, but backend provided UserActivity without a date field, causing a Tera render error.
  • Fix: In src/controllers/dashboard.rs, map UserActivity into a template-friendly recent_activities table with fields:
    • date: formatted from timestamp as %Y-%m-%d %H:%M
    • action: derived from ActivityType (e.g., Login, Purchase, Deployment, ...)
    • status: from metadata.status when present; defaults to Active for deployments, otherwise Completed
    • details: from description
  • Outcome: /dashboard renders successfully with persistent activities. No template changes required; context now matches template expectations.

Progress Update — Mock Gating & Status Normalization (2025-08-15 12:41)

  • Backend: Gated legacy mock initialization in OAuth flow. src/controllers/gitea_auth.rs now attaches MockUserData::new_user() only when APP_ENABLE_MOCKS is enabled via get_app_config().enable_mock_data().
  • Model/Docs: Normalized deployment status nomenclature to “Active”. Updated src/services/slice_calculator.rs comment from "Running" → "Active" for deployment status examples.
  • Build: cargo check passing after changes (warnings only).

Recent related changes:

  • Mock data normalization: In src/models/user.rs mock DeploymentStat for "Enterprise AI Suite", status was changed from "Running" → "Active".
  • Frontend demo: src/static/js/demo-workflow.js emits "Active" in simulated deploymentStatusChange events.

Next:

  • Audit all controllers for unconditional mock usage (e.g., src/controllers/dashboard.rs) and gate behind APP_ENABLE_MOCKS.
  • Normalize remaining "Running" → "Active" across:
    • Rust builders: src/models/builders.rs (DeploymentStatBuilder default status)
    • Frontend: src/static/js/marketplace-integration.js, src/static/js/dashboard-user.js
    • Templates: any status badge strings
  • Run cargo test and perform manual UI verification with APP_ENABLE_MOCKS=0.

User Data Loading & Scan Hardening — COMPLETED 2025-08-14 15:29

• Problem

  • Multiple modules scanned user_data/ broadly and attempted to parse non-user fixture files (e.g., session_data.json, carts, aggregated fixtures), causing serde parse errors and runtime failures.

• Design Decision

  • Enforce a strict, uniform filter for per-user JSON files across the entire codebase.
  • Criteria for a valid user file:
    • Filename ends with .json
    • Filename contains _at_ (email encoding)
    • Filename does NOT contain _cart
    • Filename is not session_data.json

• Implementation (Rust)

  • Updated directory scans to apply the filter in:
    • src/services/node_marketplace.rsget_all_marketplace_nodes(), get_all_slice_combinations()
    • src/services/node_rental.rsfind_node_owner()
    • src/services/order.rsfind_app_provider(), find_service_provider()
    • src/controllers/dashboard.rs → deployment counting logic
    • src/utils/data_cleanup.rsDataCleanup::cleanup_all_users()
    • src/utils/data_validator.rsDataValidator::validate_all_user_files()
  • Confirmed existing aggregators in src/services/user_persistence.rs already applied equivalent filtering for services/apps/products.

• Behavior & Guarantees

  • Only real per-user files are parsed; non-user fixtures are ignored.
  • Eliminates deserialization errors and runtime crashes from non-user files.
  • Provider dashboard data accesses go through UserPersistence; no production mock fallbacks (APP_ENABLE_MOCKS=0).

• Build & Verification

  • cargo check passed after changes.
  • Integration tests that parse user JSON remain valid; further E2E will confirm dashboard stability.

Insufficient Balance Unified Error Contract (2025-08-09)

  • Decision: Standardize the API error response for insufficient funds across checkout, buy-now, and cart-related validations.
  • Status Code: Recommend 402 Payment Required as the single status for insufficient funds. If current implementations differ, document and migrate to 402.
  • Canonical Response (via ResponseBuilder):
{
  "success": false,
  "error": {
    "code": "INSUFFICIENT_FUNDS",
    "message": "Insufficient balance",
    "details": {
      "currency": "USD",
      "wallet_balance_usd": 0,
      "required_usd": 0,
      "deficit_usd": 0
    }
  }
}
  • Apply in:
    • Controllers/Services that perform funds checks: src/controllers/order.rs, src/services/order.rs, src/services/instant_purchase.rs
    • Wallet-dependent controllers: src/controllers/wallet.rs, src/controllers/pool.rs, src/controllers/rental.rs
  • Frontend Guidance:
    • Consume error.details to render a consistent message: "Insufficient balance. Need $ more." (currency-aware)
    • Do not hardcode numbers; read from details. Maintain const data = result.data || result; wrapper tolerance.
  • Testing:
    • API tests: assert status code and JSON shape for checkout and buy-now flows (and cart validations if applicable).
    • UI tests/steps: verify consistent rendering and clear CTAs (e.g., "Add Funds").
  • Compatibility:
    • If legacy shapes exist, document temporary adapter logic and a removal timeline.

Product Data Architecture: User-Owned Products & Derived Catalog (2025-08-10)

Decision: Products are owned by the creating user (single source of truth). The marketplace catalog is a derived read model. Avoid dual writes.

  • Single Source of Truth (now): user-persistent storage via UserPersistence (per-user JSON) holds canonical product records owned by that user (provider_id/user_email).
  • Catalog (read model): ProductService aggregates user-owned products (and fixture seeds in fixtures mode), dedupes by id, and derives categories at runtime. Do not write to the catalog directly.
  • Fixtures Mode: user_data/products.json is seed/demo-only and not a write target. Optionally, a dev-only generated cache user_data/catalog.products.json may be produced by an indexing task and must be marked as generated.
  • Implemented: id-based deduplication in ProductService::get_all_products() prevents overlap between seeds and user-owned items. Later sources override earlier ones to honor the SOT.
  • Consistency: ProductService::get_product_by_id() now queries the aggregated, de-duplicated catalog to ensure the same precedence rules apply to detail views.

Category ID Normalization (Fixtures) — 2025-08-10

  • Rationale: Historical fixture data used plural/alias category IDs that did not match frontend filters/routes.
  • Implementation: During fixture load, ProductService normalizes category IDs to canonical singular forms (e.g., "applications" → "application", "gateways" → "gateway").
  • Data Hygiene: user_data/products.json is updated to store singular category_id values.
  • Outcome: All marketplace category views (/marketplace/applications, /marketplace/gateways, etc.) display the correct products.

Provider → Marketplace → Consumer Flow — Implemented 2025-08-12

  • Creation (Provider)
    • Service/App creation via dashboard APIs:
      • Services: GET/POST /api/dashboard/services, PUT /api/dashboard/services/{id}, DELETE /api/dashboard/services/{id}
      • Apps: GET/POST /api/dashboard/apps, PUT /api/dashboard/apps/{id}, DELETE /api/dashboard/apps/{id}
      • Generic products: GET/POST /api/dashboard/products
    • Persistence: UserPersistence writes to user_data/{email}.json under products (single source of truth).
  • Aggregation (Marketplace)
    • src/services/product.rs::ProductService::get_all_products() aggregates fixtures + user products (+ optional slice products) and de-duplicates by id.
    • Category normalization applied uniformly via canonical_category_id (see below).
    • Optional dev TTL cache controlled by APP_CATALOG_CACHE and APP_CATALOG_CACHE_TTL_SECS.
  • Visibility (Consumer)
    • src/controllers/marketplace.rs::MarketplaceController::services() filters by canonical categories service and application, converts prices to user currency, and injects service_products into src/views/marketplace/services.html.
    • Template contract: Each entry is { product, price, formatted_price }. Page is CSP-compliant with JSON hydration and external JS (src/static/js/services.js).
  • Purchase Flows
    • Add to Cart: POST /api/cart/addOrderController::add_to_cart.
    • Buy Now:
      • Affordability: GET /api/wallet/check-affordability?amount=<Decimal>
      • Instant Purchase: POST /api/wallet/instant-purchase
    • ResponseBuilder envelope: Frontend unwraps with const data = result.data || result;.
  • Generalization
    • App providers follow the same flow; canonical application category is normalized and displayed on /marketplace/applications with equivalent pricing and purchase behaviors.

Category Normalization (User Products) — 2025-08-12

  • Problem: Providers historically saved subcategory IDs (e.g., Consulting, Deployment) causing products to be filtered out of /marketplace/services.
  • Solution: src/services/product.rs::canonical_category_id maps professional service subcategories (Consulting, Deployment, Support, Training, Development, Maintenance) to canonical service.
  • Result: User-created services stored in user_data/{email}.json are visible post-aggregation; controller still tolerates both service and application during transition.

Catalog Dev Cache (Development) — 2025-08-10

  • Purpose: Speed up local development by caching the aggregated catalog derived from fixtures, user-owned products, and optional slice combinations.
  • Implementation: In-memory TTL cache inside ProductService keyed by include_slice_products.
    • Structure: static CATALOG_CACHE: OnceLock<Mutex<CatalogCache>> holding two buckets (with/without slices).
    • Entry: CacheEntry { products: Vec<Product>, fetched_at: Instant }.
    • Behavior: get_all_products() checks TTL and returns cached vector; on miss/expiry, recomputes via aggregate_all_products_uncached() and stores new entry. De-duplication semantics are preserved.
  • Configuration:
    • Flags: APP_CATALOG_CACHE (true/false), APP_CATALOG_CACHE_TTL_SECS (u64 seconds).
    • Defaults: Development/Test enabled with TTL=5s; Production disabled unless explicitly enabled.
    • Accessors: AppConfiguration::is_catalog_cache_enabled() and catalog_cache_ttl_secs().
  • Scope & Consistency:
    • Caches only the aggregated catalog vector; downstream helpers (e.g., categories) continue to derive from that consistent list.
    • Slice toggle maintains distinct caches; category normalization and id-based dedupe remain intact.
    • Invalidation & Non-goals (Phase 0):
      • No write-through invalidation yet; rely on small TTL for freshness during dev.
      • Manual bust options: restart the server or wait for TTL to lapse. Changing env TTL requires restart to take effect.
    • Code pointers: src/services/product.rs::ProductService::get_all_products(), aggregate_all_products_uncached(); configuration in src/config/builder.rs.
    • Generated artifacts: If you generate user_data/catalog.products.json, keep it out of VCS (e.g., add to .gitignore).
    • Acceptance:
      • Manual test: results update after TTL; no duplicate products; category pages reflect normalized IDs.
      • Build: cargo check passes.
  • Phase Roadmap:
    • Phase 0 (now): In-memory TTL cache for dev.
    • Phase 1: Optional dev-only cache-bust endpoint or keyboard shortcut in debug UI.
    • Phase 2: Optional fine-grained caches (by category/provider) or Redis toggle for production scalability.

Marketplace Overview Composition & De-duplication — 2025-08-10

  • Overview Sections: The dashboard composes multiple product sections, notably "Featured Items" (curated) and "Popular Applications" (category-filtered).
  • De-duplication Rule: Items already rendered in "Featured Items" are excluded from "Popular Applications" to prevent duplicate cards.
  • Implementation: src/controllers/marketplace.rs collects featured product IDs into a set and filters the popular list prior to price conversion and render.
  • Scope: De-duplication is local to the overview page; category/detail pages show full results without cross-section filtering.

Migration Blueprint: PostgreSQL + PostgREST

  • Tables:
    • users(id uuid pk, email text unique, ...)
    • products(id uuid pk, owner_user_id uuid fk, name text, description text, category_id text, base_price numeric(18,8), base_currency text, availability text, attributes jsonb, metadata jsonb, status text check in (draft,published,archived), created_at timestamptz, updated_at timestamptz)
    • categories(id text pk, name text, ... )
  • Views/Indexes:
    • public_products view exposing only status = 'published' rows for catalog queries
    • GIN/trigram indexes on searchable text/jsonb fields for efficient search
  • RLS Policies:
    • Owners: INSERT/UPDATE/DELETE where owner_user_id = auth.uid()
    • Public: SELECT from public_products only
    • Auth via JWT (GoTrue) used by PostgREST
  • Lifecycle:
    • Publishing controlled by status (draft→published→archived), separate from availability (e.g., InStock/OutOfStock)
    • Prefer soft-delete with audit columns over hard delete

Env/Mode Notes:

  • Fixtures: APP_DATA_SOURCE=fixtures, APP_FIXTURES_PATH=./user_data, APP_ENABLE_MOCKS=0
  • Mock (dev-only): APP_DATA_SOURCE=mock APP_ENABLE_MOCKS=1 to visualize legacy mock data during development.
  • Personas for demos: user1..user10@example.com with password password (see docs/dev/design/current/ux/current_personas.md)
  • Manual test checklist: docs/dev/design/current/ux/manual_fixture_test_checklist.md

Quick .env for Phase 0 (fixtures/dev):

APP_DATA_SOURCE=fixtures
APP_FIXTURES_PATH=./user_data
APP_ENABLE_MOCKS=0
APP_CATALOG_CACHE=true
APP_CATALOG_CACHE_TTL_SECS=5

Roadmap to Final Marketplace (2025-08-10)

Phases are incremental and shippable; each has crisp acceptance criteria.

Phase 0: Polish fixtures/dev mode (nowshort)

  • Catalog dev cache: Optional generated user_data/catalog.products.json from aggregator; documented as generated-only.
  • Acceptance: cold-start marketplace loads ≥300 ms faster locally; cache invalidation documented.
  • Mock cleanup: Ensure prod path never touches mock data/services; annotate remaining dev-only templates (e.g., mock timeline) and gate behind config.
  • Acceptance: production run with APP_ENABLE_MOCKS=0 has zero mock reads/writes.
  • Tests:
  • Invoices: ownership checks, 403/404, print view loads.
  • Cart badge/events: add/remove/clear flows consistent after reload.
  • Marketplace dashboard loads with mocks disabled (HTTP 200).
  • Rental endpoints return 404 when mocks are disabled and resource is missing.
  • Acceptance: tests green in CI; fixtures-run smoke passes.

Phase 1: Unified insufficient balance contract

  • Backend: Standardize on 402 Payment Required and canonical ResponseBuilder error payload across src/controllers/order.rs, src/services/order.rs, src/services/instant_purchase.rs, and wallet-related controllers.
  • Reference: See prompt doc docs/dev/design/current/prompts/prompt-1.md — section "🔥 Critical: Insufficient Balance Unified Error Contract" for the JSON envelope and acceptance details.
  • Frontend: One renderer consumes error.details (e.g., “Insufficient balance. Need $X more.”).
  • Acceptance: Each major purchase flow has one insufficient-funds behavior; e2e verified.

Phase 2: Orders API enrichment — COMPLETED 2025-08-13

  • Added invoice_available and invoice_url in /api/orders and /api/orders/{id}.
  • UI enables/disables invoice CTAs from payload.
  • Acceptance: No dead CTAs; invoices open consistently from both list and detail.

Phase 3: Provider and catalog readiness

  • Minimal “publishing status” on products (draft/published) respected by aggregator (fixtures treat all as published).
  • Featured curation source-of-truth file (config JSON) for Featured Items, decoupled from categories.
  • Acceptance: Only published items appear; Featured fully driven by config.

Phase 4: Search, filters, and category UX

  • Add keyword search and common filters (category, price range, availability).
  • Acceptance: Search/filters consistent across overview and category pages; no duplicates with Featured on overview.

Phase 5: Database migration (PostgreSQL + PostgREST)

  • Implement schema and public_products view with RLS as per blueprint in this guide.
  • Wire ProductService reads to PostgREST in “db mode”; keep fixtures mode intact for demos.
  • Migration scripts + seed path for demo data.
  • Acceptance: App runs in db mode with same UX; fixtures mode remains supported for demos.

Phase 6: Payments and wallet top-up

  • Stripe integration to purchase TFC credits; lock rates at checkout; record payment method.
  • Commission logic applied to marketplace orders.
  • Acceptance: Successful top-up flows; orders complete with correct balances and auditable records.

Phase 7: Security and reliability

  • CSRF, rate limiting, session hardening; structured logging where appropriate; metrics & health endpoints.
  • Acceptance: Security checklist green; basic SLOs monitored.

Phase 8: Launch readiness

  • CI/CD, load testing, PWA/performance polish, accessibility, documentation/runbooks.
  • Acceptance: Go-live checklist complete; rollback plan defined.

4. TFC Credits System 💰

Decision: Pure ThreeFold Credits (1 TFC = 1 USD configurable)

  • Eliminates complex TFP/USD conversion logic
  • Industry-standard credit model (AWS/Google/Azure pattern)
  • Simplified user experience and developer maintenance

5. Cart Count Consistency (Navbar Badge)

Backend (src/services/order.rs):

  • OrderService::get_cart_with_details() cleans orphaned cart items (missing products), recomputes item_count from valid items, and persists cleaned carts to session and user storage.
  • remove_from_cart(), update_cart_item_quantity(), and clear_cart() save to session and, for logged-in users, to persistent JSON under user_data/*_cart.json.
  • transfer_guest_cart_to_user() merges guest items into the user cart on login and saves.

Frontend:

  • Global updateCartCount() in src/views/base.html fetches /api/cart with cache: 'no-store' and updates the navbar badge.
  • window.updateCartCount is called on DOMContentLoaded and on a global cartUpdated event.
  • Pages that modify the cart call window.updateCartCount() and dispatch cartUpdated after changes.
  • src/views/marketplace/dashboard.html uses a locally named updateCartCountLocal() to avoid overriding the global function.

Result: After restarts or rebuilds, the navbar badge accurately reflects the backend cart state.

6. Cart Clear UX (Post-Reload Success Toast)

Rationale: After clearing the cart, we force a short-delayed full page reload to guarantee a consistent empty-state UI. To preserve positive UX, a success toast is shown after the reload.

Pattern:

  • Before reload (after successful DELETE /api/cart):
    • Call window.emitCartUpdated(0) and window.updateCartCount() to update the navbar badge immediately.
    • Set a session flag: sessionStorage.setItem('cartCleared', '1').
    • setTimeout(() => window.location.reload(), 50);
  • On page load:
    • In DOMContentLoaded, check the flag; if present, remove it and show a success toast.

Implementation:

  • Guest Cart: src/views/cart.html
    • Sets cartCleared flag before reload.
    • On load, shows showToast('Cart cleared', 'success') and removes the flag.
  • Dashboard Cart (logged-in): src/views/dashboard/cart.html
    • Sets cartCleared flag before reload; sets window._suppressCartLoadToast = true to avoid false error toasts during reload.
    • On load, shows showToast('Cart cleared', 'success') and removes the flag.
  • Marketplace Cart View: src/views/marketplace/cart.html
    • Sets cartCleared flag before reload.
    • On load, shows local showSuccess('Cart cleared') and removes the flag.

Notes:

  • All flows include credentials where required and handle ResponseBuilder responses safely (204/non-JSON tolerant where applicable).
  • This pattern ensures both guest and logged-in users receive immediate feedback post-reload without UI flicker or mixed toasts.

7. Cart Fetch & Event Standardization (2025-08-09)

To ensure consistent cart state, session handling, and UI updates across all views, cart-related reads and mutations have been standardized.

  • Global Event Helper:
    • Use window.emitCartUpdated(cartCount?) instead of manually dispatching CustomEvent('cartUpdated', ...).
    • Global badge updater lives in src/views/base.html as window.updateCartCount() and listens to cartUpdated.
  • Reads (GET /api/cart):
    • Always include { cache: 'no-store', credentials: 'same-origin' } to avoid stale data and ensure cookies/sessions.
  • Mutations (POST/PUT/DELETE to /api/cart and /api/cart/item/{id}):
    • Always include { credentials: 'same-origin' }.
  • Robustness:
    • Tolerant JSON parsing where 204/No Content or non-JSON responses can occur.
  • Affected Templates:
    • src/views/dashboard/cart.html
    • src/views/marketplace/cart.html
    • src/views/marketplace/cart_full.html
    • src/views/marketplace/cart_standalone.html
    • src/views/cart.html (guest)
  • Outcome:
    • Navbar badge updates instantly and reliably across guest and logged-in flows.
    • No stale reads; session credentials are sent for all cart operations.

🗂️ System Architecture

Application Layer Structure

src/
├── controllers/           # HTTP request handlers
│   ├── auth.rs           ✅ ResponseBuilder complete (10 patterns)
│   ├── wallet.rs         ✅ ResponseBuilder complete (49 patterns)
│   ├── product.rs        ✅ ResponseBuilder complete (7 patterns)
│   ├── currency.rs       ✅ ResponseBuilder complete (12 patterns)
│   ├── marketplace.rs    ✅ ResponseBuilder complete (44 patterns)
│   ├── rental.rs         ✅ ResponseBuilder complete (24 patterns)
│   ├── pool.rs           ✅ ResponseBuilder complete (13 patterns)
│   ├── order.rs          ✅ ResponseBuilder complete (26 patterns)
│   ├── debug.rs          ✅ ResponseBuilder complete (1 pattern)
│   ├── gitea_auth.rs     ✅ ResponseBuilder complete (2 patterns)
│   ├── public.rs         ✅ Uses render_template utility (no direct patterns)
│   └── dashboard.rs      ✅ ResponseBuilder complete (331 patterns)
├── services/             # Business logic layer
│   ├── farmer.rs         ✅ ServiceFactory migrated
│   ├── currency.rs       ✅ ServiceFactory migrated
│   ├── user_persistence.rs ✅ ServiceFactory migrated
│   ├── session_manager.rs ✅ ServiceFactory migrated
│   ├── node_rental.rs    ✅ ServiceFactory migrated
│   ├── slice_rental.rs   ✅ ServiceFactory migrated
│   └── order.rs          ✅ ServiceFactory migrated
├── models/               # Data structures and builders
│   └── builders.rs       ✅ SessionDataBuilder complete
├── utils/                # Utility functions and builders
│   ├── response_builder.rs ✅ Centralized HTTP responses
│   ├── configuration.rs  ✅ ConfigurationBuilder complete
│   ├── mod.rs            ✅ render_template utility integrated with ResponseBuilder (HTML support)
│   └── data_cleanup.rs   ✅ Log-free utilities
└── user_data/            # Persistent data storage
    ├── *.json           # User persistent data files
    └── *_cart.json      # User cart persistence

📊 Implementation Status

Completed Migrations

1. Log-Free Codebase (100% Complete)

  • Scope: Entire codebase (24 files)
  • Impact: 881+ log statements removed
  • Result: Clean, production-ready code

2. SessionDataBuilder (100% Complete)

  • Scope: All UserPersistentData construction
  • Impact: ~800 lines of code reduced
  • Usage: SessionDataBuilder::load_or_create(email)

3. ConfigurationBuilder (100% Complete)

  • Scope: All environment variable access
  • Impact: ~150 lines of code reduced
  • Usage: ConfigurationBuilder::new().jwt_secret().build()

4. ServiceFactory (100% Complete)

  • Scope: All service instantiation
  • Impact: ~100+ lines of code reduced
  • Usage: ServiceFactory::currency_service()

5. ResponseBuilder (521 patterns complete)

  • Auth Controller: 10/10 patterns
  • Wallet Controller: 49/49 patterns
  • Product Controller: 7/7 patterns
  • Currency Controller: 12/12 patterns
  • Marketplace Controller: 44/44 patterns
  • Rental Controller: 24/24 patterns
  • Pool Controller: 13/13 patterns
  • Order Controller: 26/26 patterns
  • Debug Controller: 1/1 patterns
  • Gitea Auth Controller: 2/2 patterns
  • Public Controller: Uses render_template utility (no direct patterns)
  • Dashboard Controller: 331/331 patterns (COMPLETE - 100% migrated)
  • Utils Module: 1/1 patterns (render_template function complete with HTML support)

Completed

1. Dashboard Controller ResponseBuilder Migration Complete

  • Status: COMPLETE - All 331/331 patterns successfully migrated
  • Achievement: 100% ResponseBuilder pattern coverage across entire codebase
  • Patterns Migrated: 17 HttpResponse patterns including redirects, JSON responses, and error handling
  • Quality: Zero compilation errors maintained throughout entire migration
  • Testing: Full functional verification completed - dashboard application runs successfully

🛠️ Technical Specifications

ResponseBuilder API

// Success responses
ResponseBuilder::ok().json(data).build()

// Error responses
ResponseBuilder::bad_request().json(error_data).build()
ResponseBuilder::unauthorized().json(auth_error).build()
ResponseBuilder::internal_error().json(server_error).build()

// Specialized responses
ResponseBuilder::not_found().json(not_found_error).build()
ResponseBuilder::payment_required().json(payment_error).build()

// Plain text responses
ResponseBuilder::internal_error().body("Template rendering failed").build()

SessionDataBuilder Pattern

// Create new user
let user_data = SessionDataBuilder::new_user(email);

// Load existing or create new
let user_data = SessionDataBuilder::load_or_create(email);

// Automatic field inclusion via Default
UserPersistentData {
    user_email: email.to_string(),
    wallet_balance_usd: dec!(0),
    ..Default::default()  // Includes all new fields automatically
}

🚀 Strategic Roadmap

Database Evolution: Supabase Architecture

Current State

  • Storage: JSON file-based (user_data/ directory)
  • Scalability: Limited to single-node deployment
  • Features: Basic CRUD operations

Target Architecture: Self-Hosted Supabase

  • Database: PostgreSQL with ACID compliance
  • API: PostgREST auto-generated REST endpoints
  • Auth: GoTrue JWT-based authentication
  • Real-time: WebSocket subscriptions
  • Management: Supabase Studio web interface

Three-Phase Deployment Strategy

Phase 1: Development/Testing

  • Local Ubuntu 24.04 with Docker Compose
  • Complete TFP→TFC refactor
  • Migrate from JSON to PostgreSQL
  • Access: http://localhost:8000

Phase 2: Production (Single Node)

  • ThreeFold Grid Ubuntu 24.04 VM
  • Same Docker Compose as development
  • Production deployment with acceptable SPOF
  • Cost-effective, decentralized

Phase 3: High Availability

  • k3s cluster with multiple masters + workers
  • Tools: tfgrid-k3s or k3scluster
  • Zero downtime, automatic failover
  • Horizontal scaling capabilities

Complete Marketplace Ecosystem

Payment Integration Strategy

graph TD
    A[User Browses Marketplace] --> B[Selects App/VM/Service]
    B --> C[Add to Cart]
    C --> D[Review Cart]
    D --> E{Payment Method}
    E -->|Has TFC Credits| F[Pay with TFC]
    E -->|Needs Credits| G[Buy TFC with Stripe]
    G --> H[Stripe Payment]
    H --> I[TFC Credits Added]
    I --> F
    F --> J[Service Deployment Queue]
    J --> K[TFC → TFT Conversion]
    K --> L[ThreeFold Grid Deployment]
    L --> M[Service Active]

Revenue Model & Commission Structure

  • Marketplace Commission: 5-15% on all transactions
  • Payment Processing: Stripe fees (2.9% + $0.30)
  • Grid Deployment: TFT conversion at market rates
  • Provider Revenue: 85-95% after commissions

Deployment Automation Pipeline

// Core deployment service structure
pub struct DeploymentJob {
    pub id: String,
    pub user_id: String,
    pub service_spec: ServiceSpec,
    pub tfc_amount: Decimal,
    pub status: DeploymentStatus,
}

impl DeploymentQueue {
    pub async fn process_deployment(&self, job: DeploymentJob) -> Result<(), DeploymentError> {
        // 1. Calculate commission
        let commission = job.tfc_amount * self.commission_rate;
        let deployment_amount = job.tfc_amount - commission;
        
        // 2. Convert TFC to TFT
        let tft_amount = self.convert_tfc_to_tft(deployment_amount).await?;
        
        // 3. Deploy to ThreeFold Grid using GridDriver
        let deployment = self.grid_driver.deploy_service(&job.service_spec, tft_amount).await?;
        
        // 4. Update user with deployment details
        self.update_user_deployment(job.user_id, deployment).await?;
        
        Ok(())
    }
}

ThreeFold Grid Integration

GridDriver: Official ThreeFold Grid deployment interface

  • Repository: https://github.com/threefoldtech/griddriver
  • Purpose: Standardized API for ThreeFold Grid deployments
  • Features: VM deployment, Kubernetes clusters, storage allocation
  • Integration: Direct Rust bindings for marketplace automation
  • Benefits: Official support, maintained by ThreeFold team, production-ready

📋 Development Standards

Code Quality Requirements

  • Zero Compilation Errors: All migrations maintain clean builds
  • Minimal Warnings: Only unused variable warnings acceptable
  • Builder Pattern Usage: Mandatory for complex object construction
  • Persistent Data Only: No mock data in production code
  • Log-Free Code: No log:: statements in main/development branches

Migration Methodology

  • Manual Systematic Migration: Preferred over automated scripts
  • Bulk All-at-Once Edits: Efficient for simple controllers
  • Section-by-Section: For complex controllers (dashboard, order)
  • Compilation Verification: cargo check after each batch
  • Exact Target Content: Precise pattern matching to avoid errors

🔧 Decision Trail & Historical Context

Decision #001: TFP → TFC Credits System Refactor

Date: 2025-08-04
Status: APPROVED - Implementation Complete

Problem Statement

The marketplace used a hybrid TFP (ThreeFold Points) to USD conversion system that created:

  • Complex conversion logic throughout the codebase
  • Confusing user experience with dual currency displays
  • Maintenance overhead with exchange rate calculations

Solution: TFC (ThreeFold Credits)

  • Model: 1 TFC = 1 USD (configurable)
  • Benefits: Industry standard (AWS/Google/Azure pattern)
  • Implementation: Global rename TFP → TFC, remove conversion logic
  • User Experience: Simple mental model, familiar credit system

Decision #002: Log-Free Codebase Policy

Date: 2025-08-05
Status: IMPLEMENTED - 881+ logs removed

Rationale

  • Production-ready code should not contain debug logging
  • Simplified AI-assisted development and code analysis
  • Reduced file complexity and compilation overhead
  • Industry best practice for clean codebases

Decision #003: Builder Pattern Architecture

Date: 2025-08-03
Status: IMPLEMENTED - Single source of truth

Benefits Achieved

  • 95.8% compilation error reduction (24 → 0 errors)
  • ~1,120+ lines of code reduction
  • Future-proof field additions via ..Default::default()
  • Consistent patterns across entire codebase

Decision #004: Multi-Currency Display with USD Settlement

Date: 2025-08-07
Status: IMPLEMENTED

Summary

  • Design: Users can select a display currency (USD, TFC, CAD, EUR, …). All settlements occur in USD (via Stripe at a later stage).
  • Backend Support: Added currencies in src/services/currency.rs::CurrencyService::get_supported_currencies()
    • TFC: Custom("credits"), exchange_rate_to_base = 1.0, decimal_places = 2, is_active = true
    • CAD: Fiat, exchange_rate_to_base = 1.35 (placeholder), decimal_places = 2, is_active = true
    • EUR: already supported
  • Controller: Preference and listing endpoints already validate/accept these currencies.
    • File: src/controllers/currency.rs
  • UI Fix: Removed duplicate USD and added TFC to the currency selector.
    • File: src/views/dashboard/settings.html

Rationale

  • Standard e-commerce pattern: multi-currency display with single-currency settlement simplifies accounting and Stripe integration.
  • Keeps USD as canonical base for pricing, accounting, and refunds.

Implementation Notes

  • Store canonical base_usd_amount on orders, plus display_currency, display_amount, rate_used, and timestamp for transparency.
  • Server-side conversions via CurrencyService.convert_amount(); do not trust client-side for totals.
  • Navbar dropdown API (/api/navbar/dropdown-data) provides fully formatted wallet balance (wallet_balance_formatted) and display_currency for consistent UI display.
  • Dashboard overview wallet section (src/views/dashboard/index.html) updates #dashboardWalletBalance and #dashboardCurrencyCode on load using:
    • Safe unwrap pattern: const data = result.data || result;
    • Values: data.wallet_balance_formatted and data.display_currency

UX Guidance

  • Show both: e.g., Total: $123.45 USD (≈ 167.89 CAD) and note: “final charge in USD; conversions approximate.”
  • On rate refresh/expiry, prompt user to accept updated totals before checkout.

Future Enhancements (Optional)

  • Populate currency selector dynamically from /api/currency/supported to avoid UI/backend drift.
  • Add rate locking at checkout and audit trails for exchange rates used per order.

Decision #005: Externalized Dashboard Scripts & JSON Hydration

Date: 2025-08-11
Status: IMPLEMENTED

Summary

  • Removed inline JavaScript containing template syntax from src/views/dashboard/index.html to eliminate template/JS lint conflicts and improve maintainability.
  • Introduced a safe data hydration pattern via <script type="application/json" id="dashboard-chart-data"> that embeds server-provided values as pure JSON.
  • Moved dashboard logic (chart initialization and wallet balance updater) to static/js/dashboard.js.

Rationale

  • CSP compliance and security (no inline scripts required for charts logic).
  • Better caching and performance for static JS assets.
  • Cleaner separation of concerns and reduced IDE false-positive lints from mixed template/JS syntax.

Implementation Notes

  • Chart data (resource utilization, usage trends, user activity, deployment distribution) is emitted as JSON; the external JS reads and initializes charts accordingly.
  • Wallet balance and display currency continue to come from /api/navbar/dropdown-data and are applied to #dashboardWalletBalance and #dashboardCurrencyCode at page load.
  • Static asset path convention: /static/js/... as used elsewhere in the app.

🎯 Next Steps & Priorities

Immediate (Current Sprint) COMPLETE

  1. Dashboard Controller Migration Complete

    • MILESTONE ACHIEVED: All 331/331 patterns successfully migrated to ResponseBuilder
    • 17 Pattern Types: Redirect (3), JSON Success (6), JSON Error (5), Unauthorized (1), Internal Error (1), Plain Text (1)
    • Zero Compilation Errors: Clean build maintained throughout entire migration
    • Full Functional Testing: Dashboard application runs successfully with all features working
  2. Public Pages Implementation Complete

    • Added comprehensive changelog page with version history
    • Added detailed roadmap page with current progress tracking
    • Implemented PublicController with 7 new endpoints
    • Added routes for /changelog and /roadmap pages
  3. Authentication & Buy-Now Flow Critical Fix

    • RESOLVED: Wallet balance issue where users with sufficient funds saw "insufficient balance"
    • ROOT CAUSE: Frontend JavaScript expecting {can_afford: true} but receiving {data: {can_afford: true}, success: true}
    • SOLUTION: Updated frontend to handle ResponseBuilder's nested response format
    • IMPACT: Buy-now functionality now works correctly for authenticated users
    • TECHNICAL: Fixed middleware route exclusion and session validation
    • DEBUGGING: Added comprehensive logging to identify ResponseBuilder pattern mismatches
  4. Utils Module ResponseBuilder Migration Complete

    • COMPLETED: Migrated 1 HttpResponse pattern in render_template
    • Applied ResponseBuilder pattern to template rendering utility
    • All controllers that use template rendering now benefit from centralized pattern
    • HTML response support added to ResponseBuilder architecture
  5. Builder Pattern Architecture Complete

    • 100% Coverage: All 521/521 patterns across entire codebase migrated
    • All Controllers: Complete ResponseBuilder implementation
    • Single Source of Truth: Centralized HTTP response construction
    • Architecture Goal Achieved: Complete builder pattern implementation

Short Term (Next 2-4 weeks)

  1. Builder Pattern Architecture Complete

    • All ResponseBuilder migrations completed across entire codebase
    • Zero compilation errors maintained across entire codebase
    • Unused HttpResponse imports can be cleaned up
    • All ResponseBuilder patterns verified and tested
  2. Code Cleanup and Optimization

    • Remove unused HttpResponse imports from all controllers
    • Clean up any remaining unused variables or imports
    • Optimize ResponseBuilder usage patterns
    • Final code quality improvements

Medium Term (1-2 months)

  1. Payment Integration Implementation

    • Stripe integration for TFC purchases
    • TFC → TFT conversion service
    • Commission calculation system
  2. Database Migration Planning

    • Design PostgreSQL schema
    • Plan JSON to SQL migration
    • Supabase deployment preparation

Long Term (3-6 months)

  1. Complete Marketplace Ecosystem

    • Deployment automation pipeline
    • ThreeFold Grid integration
    • Real-time deployment status
  2. High Availability Implementation

    • k3s cluster deployment
    • Multi-node redundancy
    • Horizontal scaling capabilities

Current Architecture Documents (./parts/)

Strategic Roadmap Documents (./parts/roadmap/)


📈 Performance Metrics

Before Architecture Implementation

  • Log Statements: 881+ across 24 files
  • Compilation Errors: 67+ scattered errors
  • Code Duplication: ~1,120+ lines of duplicate code
  • Mock Data: Scattered throughout codebase
  • Response Patterns: Direct HttpResponse usage

After Architecture Implementation

  • Log Statements: 0 (100% reduction)
  • Compilation Errors: 0 (100% reduction)
  • Code Duplication: ~1,120+ lines eliminated
  • Mock Data: 100% eliminated
  • Response Patterns: 521 patterns centralized (ALL controllers complete, 1 new controller added, 1 utils module complete)

Quantitative Benefits

  • File Size Reduction: Significant across all migrated files
  • Compilation Time: Improved due to reduced complexity
  • Maintainability: Single source of truth for all patterns
  • Developer Experience: Consistent patterns across codebase
  • AI Assistance: Simplified code analysis and generation
  • Migration Progress: 521/521 patterns complete (100% overall) 🎉
  • Architecture Complete: All controllers fully migrated to ResponseBuilder
  • Critical Milestone: Complete builder pattern architecture implementation achieved
  • New Features: Comprehensive changelog and roadmap pages provide project transparency
  • Utils Module Complete: render_template function successfully migrated with HTML support
  • HTML Template Support: ResponseBuilder now supports both JSON and HTML responses
  • Dashboard Controller: Largest migration component (331 patterns) successfully completed


🎉 MAJOR MILESTONE ACHIEVED (2025-08-08)

Complete Builder Pattern Architecture Implementation

ACHIEVEMENT: 100% ResponseBuilder pattern coverage across entire Project Mycelium codebase

Dashboard Controller Migration Complete

  • Scope: Largest remaining migration component (331 HttpResponse patterns)
  • Patterns Migrated: 17 distinct HttpResponse usage types
    • Redirect Patterns: 3 (HttpResponse::FoundResponseBuilder::redirect)
    • JSON Success Patterns: 6 (HttpResponse::OkResponseBuilder::ok)
    • JSON Error Patterns: 5 (HttpResponse::BadRequestResponseBuilder::bad_request)
    • Unauthorized Pattern: 1 (HttpResponse::UnauthorizedResponseBuilder::unauthorized)
    • Internal Error Pattern: 1 (HttpResponse::InternalServerErrorResponseBuilder::internal_error)
    • Plain Text Pattern: 1 (HttpResponse::Ok with text/plain → ResponseBuilder::ok with content_type)

Technical Excellence Maintained

  • Zero Compilation Errors: Clean build maintained throughout entire migration process
  • Functional Verification: Complete dashboard application testing performed
  • API Compatibility: All endpoints verified to return proper HTTP responses
  • Authentication Flow: Login, wallet, cart, and dashboard functionality confirmed working

Architecture Impact

  • Total Patterns: 521/521 complete (100% coverage)
  • All Controllers: Complete ResponseBuilder implementation
  • Single Source of Truth: Centralized HTTP response construction achieved
  • Future-Proof: Easy addition of new response types through ResponseBuilder
  • Maintainability: Consistent response patterns across entire application

Project Status

The Project Mycelium now has complete builder pattern architecture with:

  • SessionDataBuilder: 100% complete
  • ConfigurationBuilder: 100% complete
  • ServiceFactory: 100% complete
  • ResponseBuilder: 100% complete (521/521 patterns)
  • Log-Free Codebase: 100% complete
  • Persistent Data Only: 100% complete

This represents the completion of the comprehensive architectural modernization initiative.

---

🔧 Previous Session Accomplishments (2025-08-07)

Critical Authentication & Buy-Now Flow Fix

Problem: Users with sufficient wallet balance (e.g., $5989) were incorrectly shown "You need $X.XX more in your wallet" when attempting purchases.

Root Cause Analysis:

  1. Middleware Issue: Buy-now routes were excluded from authentication checks
  2. Session Validation: Multi-factor session validation needed enhancement
  3. ResponseBuilder Pattern Mismatch: Frontend expected {can_afford: true} but backend returned {data: {can_afford: true}, success: true}

Technical Solutions Implemented:

  1. Middleware Fix (src/middleware/mod.rs:197):

    • Removed buy-now route exclusion from authentication middleware
    • Ensured all purchase routes require proper authentication
  2. Enhanced Session Validation (src/controllers/auth.rs:387):

    • Improved auth_status endpoint with multi-factor session validation
    • Added comprehensive session state checking
  3. Frontend ResponseBuilder Compatibility (src/static/js/buy-now.js:186):

    • Updated JavaScript to handle ResponseBuilder's nested response format
    • Added fallback logic: affordabilityData = affordabilityResult.data || affordabilityResult
    • Maintained backward compatibility with direct response format
  4. Debug Infrastructure (src/services/instant_purchase.rs:110-130):

    • Added comprehensive debugging to wallet balance loading
    • Enhanced error tracking for user data persistence
    • Improved logging for affordability calculations

Impact & Results:

  • Authentication Flow: Now properly validates users before purchase attempts
  • Wallet Balance: Correctly reads $5989 balance for user1@example.com
  • Buy-Now Functionality: Successfully proceeds with purchases when funds are sufficient
  • ResponseBuilder Integration: Frontend now compatible with backend response patterns
  • User Experience: Eliminated false "insufficient funds" messages

Technical Debt Addressed:

  • Fixed inconsistency between ResponseBuilder pattern usage and frontend expectations
  • Resolved middleware authentication gaps in purchase flow
  • Established proper debugging infrastructure for future troubleshooting

🧩 New Findings & Required Actions (2025-08-09)

The following updates align the implementation with existing architecture and codify next steps.

1) Marketplace Order Invoices Endpoints, Views, Wiring

  • Backend endpoints:
    • GET /orders/{id}/invoice → Render HTML invoice (Tera) for print/view. Implemented 2025-08-09.
    • Not planned: GET /api/orders/{id}/invoice direct download. See rationale in "Marketplace Order Invoices (2025-08-09)"; may be added later if integrations require it.
  • Controller: get_order_invoice() implemented in src/controllers/order.rs with ownership checks and invoice data assembly.
  • Template: src/views/marketplace/order_invoice.html exists and renders order id, date, items, quantities, unit price, line totals, subtotal/total, payment method, billing email, and a Print button.
  • Frontend wiring: src/views/dashboard/orders.html opens /orders/{id}/invoice in a new tab; CTA text "View Invoice"; handlers renamed: downloadInvoiceviewInvoice, downloadInvoiceFromModalviewInvoiceFromModal.
  • API contract enrichment — Implemented 2025-08-13: include invoice_available: bool and invoice_url: String (empty when unavailable) in /api/orders and /api/orders/{id} responses to drive UI state.

2) Insufficient Balance Unified Error Contract

  • Standard error envelope when funds are insufficient (JSON):
    {
      "success": false,
      "code": "insufficient_balance",
      "message": "Insufficient balance",
      "insufficient_balance": {
        "required": 0.00,
        "available": 0.00,
        "shortfall": 0.00,
        "currency": "USD"
      }
    }
    
  • Apply consistently via ResponseBuilder in:
    • src/controllers/order.rs (checkout/place order), src/services/order.rs (payment paths)
    • src/services/instant_purchase.rs (buy-now), and any wallet-dependent controllers (wallet.rs, pool.rs, rental.rs) that currently emit varied structures/messages.
  • Frontend handling: branch on payload.insufficient_balance when present and render a unified copy: "Insufficient balance. Need $X more." (shortfall with currency-aware formatting). Keep const data = result.data || result; pattern.

3) Cart Events, Cache, and Credentials Standardization

  • Event emission: use window.emitCartUpdated(cartCount?) everywhere instead of ad-hoc new CustomEvent('cartUpdated', ...).
  • Fresh reads: all fetch('/api/cart') calls must include { cache: 'no-store' }.
  • Mutations: all cart-changing requests must include { credentials: 'same-origin' }.
  • Rationale: avoids stale navbar badge and ensures session coherence across logged-in and guest flows. Aligns with global updater defined in src/views/base.html.

4) Mock/Fixture/Stub Data Production Gating

  • Compliance audit found residual dev-only modules/usages (e.g., MockDataService, selective fields in session_manager.rs) still present in tree. Architecture requires persistent data only for production runtime.
  • Action:
    • Introduce AppConfig.enable_mock_data flag (default: false in production). Gate any legacy mock readers/writers behind this flag; remove from production routes/services.
    • Ensure product and session flows source exclusively from persistent stores in production (user_data/ via UserPersistence).
    • Document dev-only paths and make them opt-in for local demos/tests.

5) Tests & Observability

  • Add integration tests covering:

    • Invoice endpoints (ownership checks, HTML view, file headers, success path)
    • Insufficient balance contract across checkout and buy-now endpoints
    • Cart badge update: mutation → emitCartUpdated → global updateCartCount() effect
  • Extend API docs to include the insufficient balance envelope and invoice routes.

  • Template Engine Parity (Tera):

    • In tests, initialize Tera with the same template glob as production (e.g., src/views/**/*.html).
    • Register all custom Tera functions before configuring routes, so templates render identically in tests and production.
    • Tera syntax rule: avoid parentheses around filter expressions in conditions. Use collection | length > 0 instead of (collection | length) > 0.

Buyer and Provider Dashboard Flows — Offer → Buy → Track (2025-08-13)

  • Scope: buyers see purchased services in /dashboard/user ("My Services"); providers see incoming service requests in /dashboard/service-provider ("Service Requests"). Extend this pattern to all product types.
  • Data model: derive from persisted orders joined with products; ownership via buyer_id and seller_id. Currency via CurrencyService; persistent data only (no mocks in production).
  • Endpoints (ResponseBuilder JSON):
    • GET /api/dashboard/user/services/purchased?product_type=service|app|bundle|any
      • Items purchased by current user (buyer). Default product_type=service; any returns all types.
    • GET /api/dashboard/service-provider/requests?product_type=service|app|bundle|any
      • Orders where current user is seller/provider. Include order status and buyer alias/contact where available.
    • Errors: mutations related to these flows must emit the unified insufficient balance envelope (see “Insufficient Balance Unified Error Contract”).
  • Payload shape (example):
    {
      "success": true,
      "data": {
        "items": [
          {
            "order_id": "ord_123",
            "product_id": "prod_abc",
            "product_type": "service",
            "title": "Consulting Hours",
            "status": "completed",
            "unit_price": { "amount": 99.0, "currency": "USD", "display": "$99.00" },
            "quantity": 1,
            "buyer_id": "user_b",
            "seller_id": "user_a",
            "purchased_at": "2025-08-13T00:00:00Z"
          }
        ],
        "currency": "USD"
      }
    }
    
  • Frontend bindings:
    • src/views/dashboard/user.html + src/static/js/dashboard-user.js render purchased services. Use const data = result.data || result;, tolerate empty items.
    • src/views/dashboard/service_provider.html + src/static/js/dashboard-service-provider.js render “Service Requests”.
    • Use data-action handlers only; reads use { cache: 'no-store' }; mutations use { credentials: 'same-origin' }.
  • Extension to all product types:
    • Drive by product_type filter; reuse ProductService::get_all_products() aggregation and category normalization to surface types in the correct marketplace views.
  • Acceptance:
    • Buyer sees purchased services tracked in dashboard; provider sees incoming requests; consistent currency formatting and ResponseBuilder usage.

Document Maintainer: AI Architecture Team Review Cycle: Updated with each major architectural change Approval: Required for any deviations from established patterns Distribution: All developers, AI systems, and stakeholders