tfgrid/projectmycelium
11
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

init projectmycelium

Browse Source
This commit is contained in:
mik-tf
2025-09-01 21:37:01 -04:00
commit b41efb0e99
319 changed files with 128160 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

532
docs/dev/design/threefold-marketplace-design-architecture.md Normal file
View File

@@ -0,0 +1,532 @@
# Project Mycelium - Design & Architecture
**Last Updated:** 2025-08-23 10:21:27 (EDT)
**Purpose:** Comprehensive architectural vision and complete UX specification (patterns, guidelines, ADRs) for the Project Mycelium. Roadmap and status live in the separate roadmap document.
See also: [Roadmap](./projectmycelium-roadmap.md)
## 1. Architecture Vision (Target State)
### Core Design Principles
#### **User Experience as Single Source of Truth**
- Complete UX specification drives all development priorities
- All features must support the defined user workflows
- UX testing validates every implementation milestone
- API development derives from UX requirements
#### **Builder Pattern as Single Source of Truth**
- [`SessionDataBuilder`](src/services/session_data.rs), [`ConfigurationBuilder`](src/config/builder.rs), [`ResponseBuilder`](src/utils/response_builder.rs), [`ServiceFactory`](src/services/factory.rs) centralize construction and lifecycles
- All HTTP endpoints return via [`ResponseBuilder`](src/utils/response_builder.rs) with consistent JSON envelopes
#### **ResponseBuilder Envelope & Frontend Contract**
```json
{
"success": true|false,
"data": { ... },
"error"?: { ... }
}
```
- Frontend must always unwrap: `const data = result.data || result;` before accessing fields
#### **CSP-Compliant Frontend: External JS + JSON Hydration**
- **Zero inline scripts/handlers** - code lives in [`src/static/js/*.js`](src/static/js/) and is served under `/static/js/*`
- **Data hydration** via `<script type="application/json" id="...">` blocks
- **Individual field encoding**: Each field is JSON-encoded individually (e.g., via server-side serializer)
- **Template structure**: Templates expose `{% block scripts %}` and `{% block head %}` at top level only
- **Static mapping**: Actix serves static assets at `/static/*` from `./src/static` (see `src/main.rs`)
- **Hydration safety**: Mark embedded JSON as safe in templates (e.g., Tera `| safe`) to avoid HTML entity escaping that breaks `JSON.parse`
#### **Persistent-Data-Only Runtime**
- **No mock data** in production code
- **Canonical user data model** persisted per user under [`./user_data/{email_encoded}.json`](user_data/) via [`UserPersistence`](src/services/user_persistence.rs)
- **All operations** (orders, services, wallet, products) read/written through persistence services
#### **Product Model: User-Owned SOT + Derived Catalog**
- **Products owned by provider users** in their persistent files
- **[`ProductService`](src/services/product.rs)** aggregates derived marketplace catalog
- **Category ID normalization** with optional dev TTL cache (disabled by default in prod)
#### **Currency System (Updated)**
- **TFC as base currency** with display currencies: TFC/USD/EUR/CAD (extensible)
- **TFC credits settle** at 1 TFC = 1 USD (fixed exchange rate)
- **Server-formatted amounts**: Server returns formatted display amounts and currency code
- **Frontend renders** without recomputing
- **User preference**: Users can choose display currency in settings
#### **Unified Insufficient-Balance Contract**
- **Target**: HTTP 402 Payment Required for all insufficient funds cases
- **Canonical error payload** with `error.details` containing currency-aware amounts and deficit
---
## 2. Complete User Experience Specification
### **Public Access (No Authentication Required)**
#### **Information Pages**
- `/docs` - Complete marketplace documentation
- `/privacy` - Privacy policy and data handling
- `/terms` - Terms and conditions
- `/about` - Marketplace information and mission
- `/contact` - Contact ThreeFold support
#### **Marketplace Browsing**
- `/marketplace` - Full marketplace access for browsing
- **Anonymous cart functionality** - Add items without login
- **Search and filtering** across all categories
- **Product details** and pricing information
### **Authentication & Registration**
#### **User Registration**
- `/register` - New user registration flow
- **GitEa OAuth integration** with seamless account creation
- **Profile setup** during registration
#### **User Login**
- `/login` - User authentication
- **Session management** with secure token handling
- **Cart migration** from anonymous to authenticated session
### **Purchase Workflows**
#### **Shopping Cart Management**
- **Anonymous users**: `/cart`
- Add/remove items
- View cart contents
- Login prompt for checkout
- **Cart preservation** during login transition
- **Authenticated users**: `/dashboard/cart`
- Full cart editing (quantity adjustment)
- Item removal and cart clearing
- **Enhanced cart management** with user preferences
#### **Purchase Options**
- **Buy Now** - Direct purchase flow
- **Add to Cart** - Traditional shopping cart workflow
- **Checkout process** with credit validation
- **Order confirmation** and invoice generation
### **Marketplace Categories**
#### **Compute Resources**
- `/marketplace/compute` - Virtual machine slices
- **VM configuration** options (CPU, RAM, storage)
- **Kubernetes cluster** deployment options
- **Self-managed** resource mode
- Users set SSH public key to access VM
#### **Infrastructure**
- `/marketplace/3nodes` - Complete server reservations
- `/marketplace/gateways` - Mycelium gateway services
#### **Applications & Services**
- `/marketplace/applications` - Published applications
- `/marketplace/services` - Professional services
### **User Dashboard & Profile Management**
#### **Dashboard Overview**
- `/dashboard` - Activity overview and notifications
- **Recent transactions** and order status
- **Quick actions** and shortcuts
#### **User Profile**
- `/dashboard/user` - Profile and activity management
- **My Applications** - Purchased and deployed apps
- **My Service Bookings** - Active service engagements
- **Deployment management** for purchased resources
#### **Wallet & Credits**
- `/dashboard/wallet` - Complete wallet management
- **Buy Credits** - Credit purchase functionality
- **Transfer Credits** - User-to-user transfers
- **Auto Top-Up Settings** - Automated balance management
- **Recent Transactions** - Complete transaction history
#### **Orders & History**
- `/dashboard/orders` - Complete order history
- **Order details** and status tracking
- **Invoice access** and download
### **Settings & Preferences**
#### **Profile Settings**
- `/dashboard/settings` - Comprehensive settings management
- **Profile Information**:
- Name (editable)
- Email (read-only)
- Country
- Time Zone
- **Password Management** - Secure password updates
- **Account Deletion** - Data retention policy compliance
#### **SSH Key Management**
- `/dashboard/settings` - SSH Keys section
- **Public key upload** for self-managed resources
- **Key management** (add, remove, edit)
- **Multiple SSH keys per user** with duplicate prevention
- **Real-time validation** with security level indicators
- **Integration** with VM and cluster deployments
##### SSH Keys UX Contract — Canonical Pattern
- Endpoints (ResponseBuilder envelope; always unwrap `const data = result.data || result;`):
- `GET /api/dashboard/ssh-keys`
- `POST /api/dashboard/ssh-keys`
- `PUT /api/dashboard/ssh-keys/{id}`
- `DELETE /api/dashboard/ssh-keys/{id}`
- `POST /api/dashboard/ssh-keys/{id}/set-default`
- `GET /api/dashboard/ssh-keys/{id}`
- Frontend implementation:
- CSP-compliant JS in `src/static/js/dashboard-ssh-keys.js`
- Views in `src/views/dashboard/settings.html` (no inline JS)
- Read/write via fetch; unwrap `data` before accessing fields
- Verification workflow:
- Use browser console to call each endpoint and confirm `{ success, data }` envelope
- In UI: add/edit/delete keys, set default; verify list updates and validation messages
- Regression tests:
- Reference: `tests/frontend_ux/settings_management_ux_test.rs` (per-step runner pattern)
- Ensure tests assert ResponseBuilder unwrapping and persistence via user_data
#### **Notification Preferences**
- **Security Alerts** - Account security notifications
- **Billing Notifications** - Payment and credit alerts
- **System Alerts** - Downtime and maintenance
- **Newsletter** - Product updates and news
- **Dashboard Notifications** - In-app alert preferences
- **Message Notifications** - Real-time message alerts and desktop notifications
#### **Currency Preferences**
- **Display Currency Selection**:
- TFC (base currency)
- USD
- EUR
- CAD
- **Real-time conversion** display
##### Settings UX Contract — Canonical Pattern
- Endpoints (ResponseBuilder envelope; always unwrap `const data = result.data || result;`):
- Profile: `POST /api/dashboard/settings/profile`
- Password: `POST /api/dashboard/settings/password`
- Verify Password: `POST /api/dashboard/settings/verify-password`
- Notifications: `POST /api/dashboard/settings/notifications`
- Delete Account: `POST /api/dashboard/settings/delete-account`
- Billing History: `GET /api/dashboard/settings/billing-history`
- Currency Preference: `GET /api/user/currency`, `POST /api/user/currency`
- Frontend contract:
- Forms or fetch-based actions must read from `data` and render messages/status accordingly
- Avoid recomputation; use server-formatted values when provided
- Keep CSP compliance (no inline JS); hydrate via JSON or fetch
- Verification workflow:
- Console: test endpoints and confirm `{ success, data }` envelope
- UI: submit profile/password/notifications; verify success toasts and field updates
- Currency: change preference and verify display reflects new currency
- Regression tests:
- Covered by `tests/frontend_ux/settings_management_ux_test.rs` (validated passing)
- Use per-step runner with persistence checks in `user_data/`
### **Communication & Messaging**
#### **Message Notification System**
- **Industry-standard notifications** across all platform touchpoints
- **Navbar integration** - Messages item in user dropdown with unread counter
- **Sidebar integration** - Enhanced message counter in dashboard navigation
- **Real-time updates** - 15-second polling with visual feedback
- **Desktop notifications** - Browser notifications for new messages (with permission)
- **Progressive disclosure** - Badge → dropdown preview → full messaging interface
- **Cross-platform consistency** - Unified notification badges and counters
##### Messaging UX Contract — Canonical Pattern
- **Core API endpoints** (ResponseBuilder envelope; always unwrap `const data = result.data || result;`):
- `GET /api/messages/threads` - Thread list with unread counts and last messages
- `POST /api/messages/threads` - Create new conversation thread (requires `Content-Type: application/json`)
- `GET /api/messages/threads/{id}/messages` - Get messages for specific thread
- `POST /api/messages/threads/{id}/messages` - Send message to thread
- `PUT /api/messages/threads/{id}/read` - Mark thread as read (resets unread count)
- **Frontend implementation**:
- `src/static/js/messaging-system.js` - Core messaging functionality and modal interface
- `src/static/js/notification-system.js` - Cross-platform notification management (15s polling)
- `src/static/js/dashboard-messages.js` - Full-page messaging interface at `/dashboard/messages`
- Real-time badge updates via polling and custom event system
- **Thread creation workflow**:
- Contact Provider buttons redirect to `/dashboard/messages?recipient=email&context=type&booking_id=id`
- System checks for existing thread by `recipient_email`, `context_type`, and `context_id`
- If no thread exists, creates new thread via POST with proper headers
- Auto-selects newly created thread by matching both `recipient_email` AND `context_id`
- **Notification locations**:
- Navbar user dropdown (between Wallet and Settings) with unread counter badge
- Dashboard sidebar (Communication section) with red danger badge and pulse animation
- Document title updates showing unread count
- Desktop notifications (with user permission) for new messages
- **Visual standards**:
- Red danger badges for unread counts with 99+ cap for high counts
- Pulse animation for new notifications
- Consistent styling with cart counter patterns
- Toast notifications (top-right, error-only for cleaner UX)
#### **Full-Page Messaging Interface**
- **Route**: `/dashboard/messages` - Dedicated messaging dashboard page
- **Controller**: `DashboardController::messages_page()` with proper context setup
- **Template**: `src/views/dashboard/messages.html` - Industry-standard messaging layout
- **Architecture**:
- **Left panel**: Conversation list with unread indicators and context badges
- **Right panel**: Message view with fixed header, scrollable messages, anchored input
- **Flexbox layout**: `calc(100vh - 200px)` height with proper viewport utilization
- **Message input**: Fixed at bottom using `card-footer` with `flex-shrink-0`
- **UX Standards**:
- **Toast notifications**: Top-right positioning, error-only (success removed for cleaner UX)
- **Real-time updates**: 15-second polling with immediate message display
- **Character validation**: 1000 character limit with live counter
- **Responsive design**: Mobile and desktop optimized
- **CSP compliance**: External JS with JSON hydration pattern
#### **Message Thread Management**
- **Thread Structure**: Each thread connects two users (`user_a_email`, `user_b_email`) with context
- **Context Types**: `service_booking`, `slice_rental`, `general` - determines thread purpose
- **Context ID**: Optional identifier (e.g., booking ID) for specific business objects
- **Unread Tracking**: Per-user unread counters (`user_a_unread_count`, `user_b_unread_count`)
- **Persistence**: Stored in user data files under `message_threads` and `messages` arrays
- **Thread Creation**: Automatic via Contact Provider buttons or manual via messaging interface
- **Thread Selection**: Matches by `recipient_email` + `context_id` for precise targeting
#### **Integration Points**
- **Service Bookings**: Contact Provider buttons in `/dashboard/user` My Service Bookings
- **Slice Rentals**: Contact Provider functionality for compute resource support
- **General Messaging**: Direct user-to-user communication without specific context
- **Provider Dashboards**: Incoming message notifications for service/app/farmer providers
- **Notification System**: Cross-platform badges, desktop notifications, document title updates
### **Provider Dashboards**
#### **Farmer Dashboard**
- `/dashboard/farmer` - Resource provider management
- **Add Nodes** - Grid node registration
- **Node Management** - Active node monitoring
- **Slice Distribution** - Resource allocation management
- **Revenue Tracking** - Earnings and analytics
#### **App Provider Dashboard**
- `/dashboard/app-provider` - Application provider tools
- **Register Applications** - New app publishing
- **My Published Applications** - App management table
- **Active Deployments** - Customer deployment tracking
- **Revenue Analytics** - App earnings dashboard
#### **Service Provider Dashboard**
- `/dashboard/service-provider` - Professional service management
- **Register Services** - New service offerings
- **My Service Offerings** - Service catalog management
- **Service Requests** - Customer request pipeline:
- Open
- In Progress
- Completed
- **Client Management** - Customer relationship tools
---
## 5. Technical Implementation Guidelines
### **Currency System Implementation**
```rust
// Updated currency constants
const BASE_CURRENCY: &str = "TFC";
const TFC_TO_USD_RATE: f64 = 1.0;
// Currency display preferences
struct CurrencyPreference {
user_id: String,
display_currency: String, // TFC, USD, EUR, CAD, where USD is default
created_at: DateTime<Utc>,
}
```
### **SSH Key Management**
```rust
// SSH key storage structure
struct SSHKey {
id: String,
user_id: String,
name: String,
public_key: String,
fingerprint: String,
created_at: DateTime<Utc>,
}
```
### **Cart Management Pattern**
```rust
// Cart persistence and migration
struct CartItem {
product_id: String,
quantity: u32,
configuration: Option<serde_json::Value>,
}
struct Cart {
session_id: Option<String>, // For anonymous carts
user_id: Option<String>, // For authenticated carts
items: Vec<CartItem>,
created_at: DateTime<Utc>,
updated_at: DateTime<Utc>,
}
```
---
## 8. Architecture Decision Records (Updated)
### **UX-First Development**
- **Decision**: Complete UX implementation before backend infrastructure
- **Rationale**: User experience validates product-market fit and drives API requirements
- **Implementation**: Phased approach with UX testing as validation gate
### **TFC Base Currency**
- **Decision**: TFC as base currency instead of USD
- **Rationale**: Align with ThreeFold ecosystem and user expectations
- **Implementation**: 1 TFC = 1 USD fixed rate with display preferences
## 9. Concrete UX List
- Publicly available information
- A user can consult the docs at /docs
- A user can consult the privacy policy at /privacy
- A user can consult the Terms and Conditions at /terms
- A user can read about the marketplace at /about
- A user can contact ThreeFold by consulting the page /contact
- Registration and Login
- A user can register at /register
- Once registered they can log out and then log in at /login
- Purchases
- A non-logged in user can check the marketplace at /marketplace and add items to cart
- To buy an item, the user must be logged in and have enough credits
- Purchase UX
- A user can always buy a product or service in two ways
- They can Buy Now, or Add to Cart (then proceed to checkout and complete the purchase)
- A non-logged in user will have access to the cart at /cart
- If they then log in, the cart items will be added to their account's cart
- In the /cart page, they can clear cart or delete individual items in the cart
- If they try to edit the cart, a message appears telling them to log in to edit the cart
- A logged in user will have access to the cart at /dashboard/cart
- Then they add to cart, they can see their cart at /dashboard/cart
- In the /cart page, they can clear cart or delete individual items in the cart
- As a logged in user, they can edit the cart, e.g. click + to add more of that same item, or click - to remove that item
- A user can see all their purchases at /dashboard/orders
- A user can see their recent transactions at /dashboard/wallet Recent Transactions
- Credits
- A user can buy credits at /dashboard/wallet, Buy Credits
- A user can transfer credits to another user at /dashboard/wallet, Transfer Credits
- A user can set up Auto Top-Up Settings at /dashboard/wallet Auto Top-up Settings
- Thus if a user buys an app that costs e.g. 10 USD per month, they can set an automatic wallet top-up to never go below that number, ensuring their app will never run out of credits
- Marketplace
- In the marketplace, a user can search and filter items in different categories
- They can buy compute resources (slices) at /marketplace/compute
- They can reserve a complete node (server) at /marketplace/3nodes
- They can buy Mycelium gateways at /marketplace/gateways
- They can buy applications (solutions) at /marketplace/applications
- They can buy services at /marketplace/services
- Settings
- A user can change settings at /dashboard/settings
- They can
- Update the profile information (email can't be changed)
- Name
- Country
- Time Zone
- Change the password
- Set Up SSH Public Keys
- Update notification settings
- Security alerts
- Billing notifications
- System alerts (downtime, maintenance)
- Newsletter and product updates
- Dashboard Notifications
- Show alerts in dashboard
- Show update notifications
- Update their currency preferences
- They can decide to have the prices displayed in
- USD
- TFC
- EUR
- CAD
- Delete account
- When a user deletes their account, their data is still available on the marketplace backend for security, audit and legal purposes
- Dashboard UX and Provider+Consumer Interactions
- A user can see their dashboard overview activities at /dashboard
- A user can see their user profile and activities at /dashboard/user
- A user provoding resources to the grid (aka a farmer) at /dashboard/farmer
- There they can add resources, e.g. add a node to the marketplace which will be available for purchase by other users as slices (a node is distributed as slices)
- A user can become an app provider at /dashboard/app-provider by registering new applications
- When a user register an application
- The application is displayed publicly at /marketplace/applications
- The application is shown at the app provider dashboard /dashboard/app-provider at the table My Published Applications
- When another user buys that app
- The app provider will see that info at the table /dashboard/app-provider Active Deployments
- The app purchaser will see the app info at /dashboard/user My Applications
- A user can become a service provider at /dashboard/service-provider by registering new services
- When a user register a service
- The service is displayed publicly at /marketplace/services
- The service is shown at the service provider dashboard /dashboard/service-provider at the table My Service Offerings
- When another user buys that service
- The service provider will see that info at the table /dashboard/service-provider Service Requests
- There are 3 stages to this service request from the service provider POV
- Open
- In Progress
- Completed
- The service purchaser will see the service info at /dashboard/user My Service Bookings
- A user can become a resource provider by adding nodes and thus resources to the marketplace at /dashboard/farmer
- They can Add Nodes
- Then it fetches the nodes on the ThreeFold Grid and distribute the node into slices
- Then the slices are can be seen publicly at /marketplace/compute
- Any user can then purchase slices from that farmer and access the slices
- Products
- Farmers at the core offer compute resources (slices) to the marketplace.
- On their end, farmers host nodes on the threefold grid. nodes are split into slices.
- Users can use slices for individual VMs, or for Kubernetes cluster
- The UI is intuitive
- .html_template_tests
- Apps can be self-managed and managed
- An app is at its core a slice (VM or Kubernetes cluster) with an app on top
- for self-managed node, users can set their SSH public key, they set it in /dashboard/settings SSH Keys page
- for managed node, users will have access to the credentials on their marketplace dashboard in /dashboard/user page at the section of the app/product they rent/bought
### Testing the UX
- The marketplace should have a complete test suite to confirm the frontend UX is as expected for usersT
- Thus, there should be a series of test for the UX above
- We should have the tests mentioned above in an isolated section, e.g. not shared with other tests
- This ensures that the tests are not affected by other tests
### Marketplace-Wide UX Contract Audit Plan
**Goal:** Ensure every UX flow follows the canonical API envelope and CSP-compliant frontend contract, with deterministic tests and persistence-backed verification. The complete UX should be complete and functioning.
- **Scope (frontend code):**
- `src/static/js/**/*.js` and `static/js/**/*.js` (external JS only; no inline handlers)
- Views in `src/views/**` (JSON hydration blocks only; keep CSP clean)
- **Scope (backend routes):**
- `src/routes/mod.rs` is the single source of truth for endpoint paths
- **Contract checks (apply to every fetch):**
- Unwrap ResponseBuilder: `const result = await response.json(); const data = result.data || result;`
- Render optional numbers safely: show "No limit" for null/undefined values
- Prefer server-formatted currency/amounts; avoid client recomputation
- Keep all logic in external JS; hydrate via `<script type="application/json">`
- **Priority UX areas to audit:**
- Wallet & credits: `src/static/js/dashboard_wallet.js`, `/api/wallet/*`
- Auto top-up status: `GET /api/wallet/auto-topup/status`
- SSH keys: `src/static/js/dashboard-ssh-keys.js`, `/api/dashboard/ssh-keys*`
- Settings: `/api/dashboard/settings/*` forms and responses
- Cart, orders, products: `/api/cart*`, `/api/orders*`, `/api/products*`
- **Verification workflow:**
- Console: `fetch('<endpoint>').then(r=>r.json()).then(console.log)` to confirm `{ success, data }`
- UI: ensure badges/labels and form prefills reflect `data`
- Cross-check each referenced path with `src/routes/mod.rs`
- **Test adoption:**
- Use per-step runner pattern (see `tests/frontend_ux/settings_management_ux_test.rs`)
- Persist state under `user_data/` and assert post-conditions
- Run: `cargo test --features ux_testing -- --nocapture`
- **Reporting:**
- Track per-feature audit items in this roadmap; document deviations and fixes under each UX contract subsection