tfgrid/projectmycelium
11
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs: archive old roadmap and todo files, add marketplace redesign plan

Browse Source
This commit is contained in:
mik-tf
2025-09-06 10:23:00 -04:00
parent fa50384dad
commit 70990b8a13
5 changed files with 474 additions and 552 deletions
Download Patch File Download Diff File Expand all files Collapse all files

532
docs/dev/design/archive/threefold-marketplace-design-architecture.md Normal file
View File

@@ -0,0 +1,532 @@
# Project Mycelium - Design & Architecture
**Last Updated:** 2025-08-23 10:21:27 (EDT)
**Purpose:** Comprehensive architectural vision and complete UX specification (patterns, guidelines, ADRs) for the Project Mycelium. Roadmap and status live in the separate roadmap document.
See also: [Roadmap](./projectmycelium-roadmap.md)
## 1. Architecture Vision (Target State)
### Core Design Principles
#### **User Experience as Single Source of Truth**
- Complete UX specification drives all development priorities
- All features must support the defined user workflows
- UX testing validates every implementation milestone
- API development derives from UX requirements
#### **Builder Pattern as Single Source of Truth**
- [`SessionDataBuilder`](src/services/session_data.rs), [`ConfigurationBuilder`](src/config/builder.rs), [`ResponseBuilder`](src/utils/response_builder.rs), [`ServiceFactory`](src/services/factory.rs) centralize construction and lifecycles
- All HTTP endpoints return via [`ResponseBuilder`](src/utils/response_builder.rs) with consistent JSON envelopes
#### **ResponseBuilder Envelope & Frontend Contract**
```json
{
"success": true|false,
"data": { ... },
"error"?: { ... }
}
```
- Frontend must always unwrap: `const data = result.data || result;` before accessing fields
#### **CSP-Compliant Frontend: External JS + JSON Hydration**
- **Zero inline scripts/handlers** - code lives in [`src/static/js/*.js`](src/static/js/) and is served under `/static/js/*`
- **Data hydration** via `<script type="application/json" id="...">` blocks
- **Individual field encoding**: Each field is JSON-encoded individually (e.g., via server-side serializer)
- **Template structure**: Templates expose `{% block scripts %}` and `{% block head %}` at top level only
- **Static mapping**: Actix serves static assets at `/static/*` from `./src/static` (see `src/main.rs`)
- **Hydration safety**: Mark embedded JSON as safe in templates (e.g., Tera `| safe`) to avoid HTML entity escaping that breaks `JSON.parse`
#### **Persistent-Data-Only Runtime**
- **No mock data** in production code
- **Canonical user data model** persisted per user under [`./user_data/{email_encoded}.json`](user_data/) via [`UserPersistence`](src/services/user_persistence.rs)
- **All operations** (orders, services, wallet, products) read/written through persistence services
#### **Product Model: User-Owned SOT + Derived Catalog**
- **Products owned by provider users** in their persistent files
- **[`ProductService`](src/services/product.rs)** aggregates derived marketplace catalog
- **Category ID normalization** with optional dev TTL cache (disabled by default in prod)
#### **Currency System (Updated)**
- **TFC as base currency** with display currencies: TFC/USD/EUR/CAD (extensible)
- **TFC credits settle** at 1 TFC = 1 USD (fixed exchange rate)
- **Server-formatted amounts**: Server returns formatted display amounts and currency code
- **Frontend renders** without recomputing
- **User preference**: Users can choose display currency in settings
#### **Unified Insufficient-Balance Contract**
- **Target**: HTTP 402 Payment Required for all insufficient funds cases
- **Canonical error payload** with `error.details` containing currency-aware amounts and deficit
---
## 2. Complete User Experience Specification
### **Public Access (No Authentication Required)**
#### **Information Pages**
- `/docs` - Complete marketplace documentation
- `/privacy` - Privacy policy and data handling
- `/terms` - Terms and conditions
- `/about` - Marketplace information and mission
- `/contact` - Contact ThreeFold support
#### **Marketplace Browsing**
- `/marketplace` - Full marketplace access for browsing
- **Anonymous cart functionality** - Add items without login
- **Search and filtering** across all categories
- **Product details** and pricing information
### **Authentication & Registration**
#### **User Registration**
- `/register` - New user registration flow
- **GitEa OAuth integration** with seamless account creation
- **Profile setup** during registration
#### **User Login**
- `/login` - User authentication
- **Session management** with secure token handling
- **Cart migration** from anonymous to authenticated session
### **Purchase Workflows**
#### **Shopping Cart Management**
- **Anonymous users**: `/cart`
- Add/remove items
- View cart contents
- Login prompt for checkout
- **Cart preservation** during login transition
- **Authenticated users**: `/dashboard/cart`
- Full cart editing (quantity adjustment)
- Item removal and cart clearing
- **Enhanced cart management** with user preferences
#### **Purchase Options**
- **Buy Now** - Direct purchase flow
- **Add to Cart** - Traditional shopping cart workflow
- **Checkout process** with credit validation
- **Order confirmation** and invoice generation
### **Marketplace Categories**
#### **Compute Resources**
- `/marketplace/compute` - Virtual machine slices
- **VM configuration** options (CPU, RAM, storage)
- **Kubernetes cluster** deployment options
- **Self-managed** resource mode
- Users set SSH public key to access VM
#### **Infrastructure**
- `/marketplace/3nodes` - Complete server reservations
- `/marketplace/gateways` - Mycelium gateway services
#### **Applications & Services**
- `/marketplace/applications` - Published applications
- `/marketplace/services` - Professional services
### **User Dashboard & Profile Management**
#### **Dashboard Overview**
- `/dashboard` - Activity overview and notifications
- **Recent transactions** and order status
- **Quick actions** and shortcuts
#### **User Profile**
- `/dashboard/user` - Profile and activity management
- **My Applications** - Purchased and deployed apps
- **My Service Bookings** - Active service engagements
- **Deployment management** for purchased resources
#### **Wallet & Credits**
- `/dashboard/wallet` - Complete wallet management
- **Buy Credits** - Credit purchase functionality
- **Transfer Credits** - User-to-user transfers
- **Auto Top-Up Settings** - Automated balance management
- **Recent Transactions** - Complete transaction history
#### **Orders & History**
- `/dashboard/orders` - Complete order history
- **Order details** and status tracking
- **Invoice access** and download
### **Settings & Preferences**
#### **Profile Settings**
- `/dashboard/settings` - Comprehensive settings management
- **Profile Information**:
- Name (editable)
- Email (read-only)
- Country
- Time Zone
- **Password Management** - Secure password updates
- **Account Deletion** - Data retention policy compliance
#### **SSH Key Management**
- `/dashboard/settings` - SSH Keys section
- **Public key upload** for self-managed resources
- **Key management** (add, remove, edit)
- **Multiple SSH keys per user** with duplicate prevention
- **Real-time validation** with security level indicators
- **Integration** with VM and cluster deployments
##### SSH Keys UX Contract — Canonical Pattern
- Endpoints (ResponseBuilder envelope; always unwrap `const data = result.data || result;`):
- `GET /api/dashboard/ssh-keys`
- `POST /api/dashboard/ssh-keys`
- `PUT /api/dashboard/ssh-keys/{id}`
- `DELETE /api/dashboard/ssh-keys/{id}`
- `POST /api/dashboard/ssh-keys/{id}/set-default`
- `GET /api/dashboard/ssh-keys/{id}`
- Frontend implementation:
- CSP-compliant JS in `src/static/js/dashboard-ssh-keys.js`
- Views in `src/views/dashboard/settings.html` (no inline JS)
- Read/write via fetch; unwrap `data` before accessing fields
- Verification workflow:
- Use browser console to call each endpoint and confirm `{ success, data }` envelope
- In UI: add/edit/delete keys, set default; verify list updates and validation messages
- Regression tests:
- Reference: `tests/frontend_ux/settings_management_ux_test.rs` (per-step runner pattern)
- Ensure tests assert ResponseBuilder unwrapping and persistence via user_data
#### **Notification Preferences**
- **Security Alerts** - Account security notifications
- **Billing Notifications** - Payment and credit alerts
- **System Alerts** - Downtime and maintenance
- **Newsletter** - Product updates and news
- **Dashboard Notifications** - In-app alert preferences
- **Message Notifications** - Real-time message alerts and desktop notifications
#### **Currency Preferences**
- **Display Currency Selection**:
- TFC (base currency)
- USD
- EUR
- CAD
- **Real-time conversion** display
##### Settings UX Contract — Canonical Pattern
- Endpoints (ResponseBuilder envelope; always unwrap `const data = result.data || result;`):
- Profile: `POST /api/dashboard/settings/profile`
- Password: `POST /api/dashboard/settings/password`
- Verify Password: `POST /api/dashboard/settings/verify-password`
- Notifications: `POST /api/dashboard/settings/notifications`
- Delete Account: `POST /api/dashboard/settings/delete-account`
- Billing History: `GET /api/dashboard/settings/billing-history`
- Currency Preference: `GET /api/user/currency`, `POST /api/user/currency`
- Frontend contract:
- Forms or fetch-based actions must read from `data` and render messages/status accordingly
- Avoid recomputation; use server-formatted values when provided
- Keep CSP compliance (no inline JS); hydrate via JSON or fetch
- Verification workflow:
- Console: test endpoints and confirm `{ success, data }` envelope
- UI: submit profile/password/notifications; verify success toasts and field updates
- Currency: change preference and verify display reflects new currency
- Regression tests:
- Covered by `tests/frontend_ux/settings_management_ux_test.rs` (validated passing)
- Use per-step runner with persistence checks in `user_data/`
### **Communication & Messaging**
#### **Message Notification System**
- **Industry-standard notifications** across all platform touchpoints
- **Navbar integration** - Messages item in user dropdown with unread counter
- **Sidebar integration** - Enhanced message counter in dashboard navigation
- **Real-time updates** - 15-second polling with visual feedback
- **Desktop notifications** - Browser notifications for new messages (with permission)
- **Progressive disclosure** - Badge → dropdown preview → full messaging interface
- **Cross-platform consistency** - Unified notification badges and counters
##### Messaging UX Contract — Canonical Pattern
- **Core API endpoints** (ResponseBuilder envelope; always unwrap `const data = result.data || result;`):
- `GET /api/messages/threads` - Thread list with unread counts and last messages
- `POST /api/messages/threads` - Create new conversation thread (requires `Content-Type: application/json`)
- `GET /api/messages/threads/{id}/messages` - Get messages for specific thread
- `POST /api/messages/threads/{id}/messages` - Send message to thread
- `PUT /api/messages/threads/{id}/read` - Mark thread as read (resets unread count)
- **Frontend implementation**:
- `src/static/js/messaging-system.js` - Core messaging functionality and modal interface
- `src/static/js/notification-system.js` - Cross-platform notification management (15s polling)
- `src/static/js/dashboard-messages.js` - Full-page messaging interface at `/dashboard/messages`
- Real-time badge updates via polling and custom event system
- **Thread creation workflow**:
- Contact Provider buttons redirect to `/dashboard/messages?recipient=email&context=type&booking_id=id`
- System checks for existing thread by `recipient_email`, `context_type`, and `context_id`
- If no thread exists, creates new thread via POST with proper headers
- Auto-selects newly created thread by matching both `recipient_email` AND `context_id`
- **Notification locations**:
- Navbar user dropdown (between Wallet and Settings) with unread counter badge
- Dashboard sidebar (Communication section) with red danger badge and pulse animation
- Document title updates showing unread count
- Desktop notifications (with user permission) for new messages
- **Visual standards**:
- Red danger badges for unread counts with 99+ cap for high counts
- Pulse animation for new notifications
- Consistent styling with cart counter patterns
- Toast notifications (top-right, error-only for cleaner UX)
#### **Full-Page Messaging Interface**
- **Route**: `/dashboard/messages` - Dedicated messaging dashboard page
- **Controller**: `DashboardController::messages_page()` with proper context setup
- **Template**: `src/views/dashboard/messages.html` - Industry-standard messaging layout
- **Architecture**:
- **Left panel**: Conversation list with unread indicators and context badges
- **Right panel**: Message view with fixed header, scrollable messages, anchored input
- **Flexbox layout**: `calc(100vh - 200px)` height with proper viewport utilization
- **Message input**: Fixed at bottom using `card-footer` with `flex-shrink-0`
- **UX Standards**:
- **Toast notifications**: Top-right positioning, error-only (success removed for cleaner UX)
- **Real-time updates**: 15-second polling with immediate message display
- **Character validation**: 1000 character limit with live counter
- **Responsive design**: Mobile and desktop optimized
- **CSP compliance**: External JS with JSON hydration pattern
#### **Message Thread Management**
- **Thread Structure**: Each thread connects two users (`user_a_email`, `user_b_email`) with context
- **Context Types**: `service_booking`, `slice_rental`, `general` - determines thread purpose
- **Context ID**: Optional identifier (e.g., booking ID) for specific business objects
- **Unread Tracking**: Per-user unread counters (`user_a_unread_count`, `user_b_unread_count`)
- **Persistence**: Stored in user data files under `message_threads` and `messages` arrays
- **Thread Creation**: Automatic via Contact Provider buttons or manual via messaging interface
- **Thread Selection**: Matches by `recipient_email` + `context_id` for precise targeting
#### **Integration Points**
- **Service Bookings**: Contact Provider buttons in `/dashboard/user` My Service Bookings
- **Slice Rentals**: Contact Provider functionality for compute resource support
- **General Messaging**: Direct user-to-user communication without specific context
- **Provider Dashboards**: Incoming message notifications for service/app/farmer providers
- **Notification System**: Cross-platform badges, desktop notifications, document title updates
### **Provider Dashboards**
#### **Farmer Dashboard**
- `/dashboard/farmer` - Resource provider management
- **Add Nodes** - Grid node registration
- **Node Management** - Active node monitoring
- **Slice Distribution** - Resource allocation management
- **Revenue Tracking** - Earnings and analytics
#### **App Provider Dashboard**
- `/dashboard/app-provider` - Application provider tools
- **Register Applications** - New app publishing
- **My Published Applications** - App management table
- **Active Deployments** - Customer deployment tracking
- **Revenue Analytics** - App earnings dashboard
#### **Service Provider Dashboard**
- `/dashboard/service-provider` - Professional service management
- **Register Services** - New service offerings
- **My Service Offerings** - Service catalog management
- **Service Requests** - Customer request pipeline:
- Open
- In Progress
- Completed
- **Client Management** - Customer relationship tools
---
## 5. Technical Implementation Guidelines
### **Currency System Implementation**
```rust
// Updated currency constants
const BASE_CURRENCY: &str = "TFC";
const TFC_TO_USD_RATE: f64 = 1.0;
// Currency display preferences
struct CurrencyPreference {
user_id: String,
display_currency: String, // TFC, USD, EUR, CAD, where USD is default
created_at: DateTime<Utc>,
}
```
### **SSH Key Management**
```rust
// SSH key storage structure
struct SSHKey {
id: String,
user_id: String,
name: String,
public_key: String,
fingerprint: String,
created_at: DateTime<Utc>,
}
```
### **Cart Management Pattern**
```rust
// Cart persistence and migration
struct CartItem {
product_id: String,
quantity: u32,
configuration: Option<serde_json::Value>,
}
struct Cart {
session_id: Option<String>, // For anonymous carts
user_id: Option<String>, // For authenticated carts
items: Vec<CartItem>,
created_at: DateTime<Utc>,
updated_at: DateTime<Utc>,
}
```
---
## 8. Architecture Decision Records (Updated)
### **UX-First Development**
- **Decision**: Complete UX implementation before backend infrastructure
- **Rationale**: User experience validates product-market fit and drives API requirements
- **Implementation**: Phased approach with UX testing as validation gate
### **TFC Base Currency**
- **Decision**: TFC as base currency instead of USD
- **Rationale**: Align with ThreeFold ecosystem and user expectations
- **Implementation**: 1 TFC = 1 USD fixed rate with display preferences
## 9. Concrete UX List
- Publicly available information
- A user can consult the docs at /docs
- A user can consult the privacy policy at /privacy
- A user can consult the Terms and Conditions at /terms
- A user can read about the marketplace at /about
- A user can contact ThreeFold by consulting the page /contact
- Registration and Login
- A user can register at /register
- Once registered they can log out and then log in at /login
- Purchases
- A non-logged in user can check the marketplace at /marketplace and add items to cart
- To buy an item, the user must be logged in and have enough credits
- Purchase UX
- A user can always buy a product or service in two ways
- They can Buy Now, or Add to Cart (then proceed to checkout and complete the purchase)
- A non-logged in user will have access to the cart at /cart
- If they then log in, the cart items will be added to their account's cart
- In the /cart page, they can clear cart or delete individual items in the cart
- If they try to edit the cart, a message appears telling them to log in to edit the cart
- A logged in user will have access to the cart at /dashboard/cart
- Then they add to cart, they can see their cart at /dashboard/cart
- In the /cart page, they can clear cart or delete individual items in the cart
- As a logged in user, they can edit the cart, e.g. click + to add more of that same item, or click - to remove that item
- A user can see all their purchases at /dashboard/orders
- A user can see their recent transactions at /dashboard/wallet Recent Transactions
- Credits
- A user can buy credits at /dashboard/wallet, Buy Credits
- A user can transfer credits to another user at /dashboard/wallet, Transfer Credits
- A user can set up Auto Top-Up Settings at /dashboard/wallet Auto Top-up Settings
- Thus if a user buys an app that costs e.g. 10 USD per month, they can set an automatic wallet top-up to never go below that number, ensuring their app will never run out of credits
- Marketplace
- In the marketplace, a user can search and filter items in different categories
- They can buy compute resources (slices) at /marketplace/compute
- They can reserve a complete node (server) at /marketplace/3nodes
- They can buy Mycelium gateways at /marketplace/gateways
- They can buy applications (solutions) at /marketplace/applications
- They can buy services at /marketplace/services
- Settings
- A user can change settings at /dashboard/settings
- They can
- Update the profile information (email can't be changed)
- Name
- Country
- Time Zone
- Change the password
- Set Up SSH Public Keys
- Update notification settings
- Security alerts
- Billing notifications
- System alerts (downtime, maintenance)
- Newsletter and product updates
- Dashboard Notifications
- Show alerts in dashboard
- Show update notifications
- Update their currency preferences
- They can decide to have the prices displayed in
- USD
- TFC
- EUR
- CAD
- Delete account
- When a user deletes their account, their data is still available on the marketplace backend for security, audit and legal purposes
- Dashboard UX and Provider+Consumer Interactions
- A user can see their dashboard overview activities at /dashboard
- A user can see their user profile and activities at /dashboard/user
- A user provoding resources to the grid (aka a farmer) at /dashboard/farmer
- There they can add resources, e.g. add a node to the marketplace which will be available for purchase by other users as slices (a node is distributed as slices)
- A user can become an app provider at /dashboard/app-provider by registering new applications
- When a user register an application
- The application is displayed publicly at /marketplace/applications
- The application is shown at the app provider dashboard /dashboard/app-provider at the table My Published Applications
- When another user buys that app
- The app provider will see that info at the table /dashboard/app-provider Active Deployments
- The app purchaser will see the app info at /dashboard/user My Applications
- A user can become a service provider at /dashboard/service-provider by registering new services
- When a user register a service
- The service is displayed publicly at /marketplace/services
- The service is shown at the service provider dashboard /dashboard/service-provider at the table My Service Offerings
- When another user buys that service
- The service provider will see that info at the table /dashboard/service-provider Service Requests
- There are 3 stages to this service request from the service provider POV
- Open
- In Progress
- Completed
- The service purchaser will see the service info at /dashboard/user My Service Bookings
- A user can become a resource provider by adding nodes and thus resources to the marketplace at /dashboard/farmer
- They can Add Nodes
- Then it fetches the nodes on the ThreeFold Grid and distribute the node into slices
- Then the slices are can be seen publicly at /marketplace/compute
- Any user can then purchase slices from that farmer and access the slices
- Products
- Farmers at the core offer compute resources (slices) to the marketplace.
- On their end, farmers host nodes on the threefold grid. nodes are split into slices.
- Users can use slices for individual VMs, or for Kubernetes cluster
- The UI is intuitive
- .html_template_tests
- Apps can be self-managed and managed
- An app is at its core a slice (VM or Kubernetes cluster) with an app on top
- for self-managed node, users can set their SSH public key, they set it in /dashboard/settings SSH Keys page
- for managed node, users will have access to the credentials on their marketplace dashboard in /dashboard/user page at the section of the app/product they rent/bought
### Testing the UX
- The marketplace should have a complete test suite to confirm the frontend UX is as expected for usersT
- Thus, there should be a series of test for the UX above
- We should have the tests mentioned above in an isolated section, e.g. not shared with other tests
- This ensures that the tests are not affected by other tests
### Marketplace-Wide UX Contract Audit Plan
**Goal:** Ensure every UX flow follows the canonical API envelope and CSP-compliant frontend contract, with deterministic tests and persistence-backed verification. The complete UX should be complete and functioning.
- **Scope (frontend code):**
- `src/static/js/**/*.js` and `static/js/**/*.js` (external JS only; no inline handlers)
- Views in `src/views/**` (JSON hydration blocks only; keep CSP clean)
- **Scope (backend routes):**
- `src/routes/mod.rs` is the single source of truth for endpoint paths
- **Contract checks (apply to every fetch):**
- Unwrap ResponseBuilder: `const result = await response.json(); const data = result.data || result;`
- Render optional numbers safely: show "No limit" for null/undefined values
- Prefer server-formatted currency/amounts; avoid client recomputation
- Keep all logic in external JS; hydrate via `<script type="application/json">`
- **Priority UX areas to audit:**
- Wallet & credits: `src/static/js/dashboard_wallet.js`, `/api/wallet/*`
- Auto top-up status: `GET /api/wallet/auto-topup/status`
- SSH keys: `src/static/js/dashboard-ssh-keys.js`, `/api/dashboard/ssh-keys*`
- Settings: `/api/dashboard/settings/*` forms and responses
- Cart, orders, products: `/api/cart*`, `/api/orders*`, `/api/products*`
- **Verification workflow:**
- Console: `fetch('<endpoint>').then(r=>r.json()).then(console.log)` to confirm `{ success, data }`
- UI: ensure badges/labels and form prefills reflect `data`
- Cross-check each referenced path with `src/routes/mod.rs`
- **Test adoption:**
- Use per-step runner pattern (see `tests/frontend_ux/settings_management_ux_test.rs`)
- Persist state under `user_data/` and assert post-conditions
- Run: `cargo test --features ux_testing -- --nocapture`
- **Reporting:**
- Track per-feature audit items in this roadmap; document deviations and fixes under each UX contract subsection

510
docs/dev/design/archive/threefold-marketplace-roadmap.md
View File

@@ -1,245 +1,349 @@
# Project Mycelium - Complete Vision & Roadmap
# Project Mycelium - Roadmap & Status
**Last Updated:** 2025-08-23 10:21:27 (EDT)
**Purpose:** Prioritized roadmap and implementation status for the Project Mycelium. For system design and architecture, see the linked document below.
See also: [Design & Architecture](./projectmycelium-design-architecture.md)
## 1. Post-UX Development (Future Phases)
**Last Updated:** 2025-08-20 21:25:00 (EST)
### **Database & Backend Infrastructure**
*To be prioritized after UX completion*
- PostgreSQL integration
- Payment processing enhancement
- Grid deployment automation
- Advanced analytics and monitoring
**Purpose:** Complete architectural vision, current state, and actionable roadmap for finalizing the Project Mycelium.
## 🎯 Project Status: Ready for Feature Development
**Compilation Status:** ✅ Zero errors achieved - codebase compiles successfully
**Data Migration:** ✅ Transitioned from mock to persistent data architecture
**Next Phase:** Runtime testing, feature completion, and production readiness
### **Production Readiness**
*Final phase after UX validation*
- Security hardening
- Performance optimization
- Monitoring and observability
- Production deployment
---
## 1. Architecture Vision (Target State)
## 2. Error Debugging Methodology
### Core Design Principles
### **Proven Systematic Methodology Excellence:**
#### **Builder Pattern as Single Source of Truth**
- [`SessionDataBuilder`](src/services/session_data.rs), [`ConfigurationBuilder`](src/config/builder.rs), [`ResponseBuilder`](src/utils/response_builder.rs), [`ServiceFactory`](src/services/factory.rs) centralize construction and lifecycles
- All HTTP endpoints return via [`ResponseBuilder`](src/utils/response_builder.rs) with consistent JSON envelopes
The systematic approach established in previous phases continued exceptional effectiveness:
#### **ResponseBuilder Envelope & Frontend Contract**
```json
{
"success": true|false,
"data": { ... },
"error"?: { ... }
```bash
# Error tracking pipeline (proven highly effective)
cargo check 2>&1 | grep "error\[" | wc -l # Progress monitoring
cargo check 2>&1 | grep "error\[" | sort | uniq -c | sort -nr # Pattern analysis
```
**Execution Strategy Validated:**
1. **Progress Tracking**: Regular error count measurements for monitoring reduction
2. **Pattern Analysis**: Target highest-count error categories for maximum impact
3. **Systematic Fixes**: Apply multiple related fixes in single operations
4. **Type Safety**: Maintain architectural integrity throughout
5. **Builder Consistency**: Unified patterns across codebase
---
## 3. SSH Key Management Implementation Status ✅ **FULLY OPERATIONAL**
### **Implementation Summary (2025-08-22 - FINAL)**
**Current Status:** ✅ **SSH Key Management System FULLY OPERATIONAL** - Production ready with comprehensive testing. This can be useful when enhancing other parts of the marketplace.
#### **Completed Components:**
1. **Data Model & Persistence** ✅
- [`SSHKey`](src/models/ssh_key.rs) model with validation error types
- [`SSHKeyBuilder`](src/models/builders.rs:3301-3384) following established builder pattern
- User persistent data integration in [`UserPersistentData`](src/services/user_persistence.rs)
2. **Service Layer** ✅
- [`SSHKeyService`](src/services/ssh_key_service.rs) with validation and management
- [`SSHKeyServiceBuilder`](src/services/ssh_key_service.rs) following architectural patterns
- Multiple SSH keys per user with duplicate prevention within user accounts
- SSH key format validation (Ed25519, ECDSA, RSA)
- SHA256 fingerprint generation for key identification
3. **API Endpoints** ✅
- 6 new SSH key API endpoints in [`dashboard.rs`](src/controllers/dashboard.rs:7214-7390)
- [`ResponseBuilder`](src/utils/response_builder.rs) pattern integration for consistent JSON responses
- Routes integrated in [`mod.rs`](src/routes/mod.rs:197-202)
4. **Frontend UI** ✅
- SSH Keys tab in [`settings.html`](src/views/dashboard/settings.html)
- Bootstrap modals for add/edit/delete operations
- Real-time validation with security level indicators
- CSP-compliant implementation with external JavaScript
5. **JavaScript Implementation** ✅
- [`dashboard-ssh-keys.js`](src/static/js/dashboard-ssh-keys.js) - CSP-compliant external file
- JSON hydration for data transfer (no inline scripts)
- Real-time SSH key format validation
- AJAX integration with error handling
6. **Module Integration** ✅
- [`ssh_key`](src/models/mod.rs) module export added
- [`ssh_key_service`](src/services/mod.rs) module export added
- Full architectural integration following established patterns
#### **Technical Implementation Details:**
- **Architecture Compliance**: Follows builder pattern, ResponseBuilder envelope, user persistent data architecture
- **Security Features**: SHA256 fingerprints, format validation, duplicate prevention, reasonable key limits (20 per user)
- **User Experience**: Multiple key support, default key selection, intuitive management interface
- **CSP Compliance**: External JavaScript files, JSON hydration, no inline scripts or handlers
#### **Current Phase Requirements:**
**IMMEDIATE NEXT STEPS (Required before manual testing):**
1. **Error Fixing Phase** - Apply methodology 10 systematic error resolution:
```bash
cargo check 2>&1 | grep "error\[" | wc -l # Progress monitoring
cargo check 2>&1 | grep "error\[" | sort | uniq -c | sort -nr # Pattern analysis
```
2. **Manual Testing Phase** - Comprehensive SSH key functionality testing:
- SSH key addition, editing, deletion workflows
- Format validation testing (Ed25519, ECDSA, RSA)
- Duplicate prevention validation
- UI/UX testing across browsers
- Integration testing with settings page
3. **Documentation Phase** - Complete technical documentation:
- API endpoint documentation
- User guide for SSH key management
- Integration guides for VM/cluster deployments
#### **Pending Integration:**
- **VM/Cluster Deployment Integration**: Connect SSH keys to actual deployment workflows
- **Advanced Security Features**: Rate limiting, audit logging, enhanced validation
- **Production Hardening**: Performance optimization, monitoring integration
#### **SSH Key System Architecture:**
```rust
// Core data structure (implemented)
struct SSHKey {
id: String,
name: String,
public_key: String,
key_type: SSHKeyType,
fingerprint: String,
is_default: bool,
created_at: DateTime<Utc>,
}
```
- Frontend must always unwrap: `const data = result.data || result;` before accessing fields
#### **CSP-Compliant Frontend: External JS + JSON Hydration**
- **Zero inline scripts/handlers** - code lives in [`src/static/js/*.js`](src/static/js/) and is served under `/static/js/*`
- **Data hydration** via `<script type="application/json" id="...">` blocks
- **Individual field encoding**: Each field is JSON-encoded individually (server-side serializer)
- **Template structure**: Templates expose `{% block scripts %}` and `{% block head %}` at top level only
- **Static mapping**: Actix serves `/static/*` from `./src/static` (see `src/main.rs`)
#### **Persistent-Data-Only Runtime**
- **No mock data** in production code
- **Canonical user data model** persisted per user under [`./user_data/{email_encoded}.json`](user_data/) via [`UserPersistence`](src/services/user_persistence.rs)
- **All operations** (orders, services, wallet, products) read/written through persistence services
#### **Product Model: User-Owned SOT + Derived Catalog**
- **Products owned by provider users** in their persistent files
- **[`ProductService`](src/services/product.rs)** aggregates derived marketplace catalog
- **Category ID normalization** with optional dev TTL cache (disabled by default in prod)
#### **Currency System**
- **USD as base currency** with display currencies: USD/TFC/EUR/CAD (extensible)
- **TFC credits settle** at 1 TFC = 1 USD
- **Server-formatted amounts**: Server returns formatted display amounts and currency code
- **Frontend renders** without recomputing
#### **Unified Insufficient-Balance Contract**
- **Target**: HTTP 402 Payment Required for all insufficient funds cases
- **Canonical error payload** with `error.details` containing currency-aware amounts and deficit
---
## 2. Current Implementation State
### ✅ Completed Foundation
- **CSP externalization** complete across marketplace and dashboard
- **ResponseBuilder integration** applied across all controllers (100% coverage)
- **Orders & invoices** persisted under user data with HTML invoice view
- **Currency system** working with multi-currency display
- **Insufficient-funds responses** unified to HTTP 402 with canonical error envelope
- **Mock data elimination** completed - persistent-only architecture established
### ✅ Core Services Architecture
- **Authentication**: GitEa OAuth integration with user creation
- **Data Persistence**: [`UserPersistence`](src/services/user_persistence.rs) as single source of truth
- **Product Management**: [`ProductService`](src/services/product.rs) with category normalization
- **Order Processing**: Complete order lifecycle with invoice generation
- **Wallet Operations**: Multi-currency support with transaction history
### 🔧 Architecture Patterns Established
#### **Data Flow Pattern**
```rust
// Persistent data access pattern
let persistent_data = UserPersistence::load_user_data(&user_email)?;
// Direct usage of persistent data
persistent_data.wallet_balance_usd
```
#### **Transaction Pattern**
```rust
Transaction {
id: transaction_id,
user_id: user_email,
transaction_type: TransactionType::Purchase { product_id },
amount: amount_to_add,
currency: Some("USD".to_string()),
timestamp: chrono::Utc::now(),
status: TransactionStatus::Completed,
// Validation and management (implemented)
struct SSHKeyService {
// Validation, fingerprint generation, format checking
// Integration with UserPersistentData
}
```
#### **ResponseBuilder Pattern**
```rust
ResponseBuilder::success()
.data(json!({ "products": products }))
.build()
**Summary:** SSH Key Management system is **FULLY OPERATIONAL** with all 4 core operations working perfectly. Ready for VM/cluster deployment integration and UX testing framework.
---
## 4. SSH Feature Deep Implementation & Debugging Methodology
### **Complete SSH Key Management Feature Documentation**
The SSH Key Management system represents a comprehensive implementation showcasing the Project Mycelium's architectural patterns and demonstrates a systematic approach to complex feature development.
#### **Feature Overview & UX Possibilities**
**Core SSH Key Operations (All Working):**
1. **Create SSH Key** - Upload and validate public keys with real-time feedback
2. **Set Default SSH Key** - Designate primary key for deployments
3. **Edit SSH Key** - Modify key names and default status
4. **Delete SSH Key** - Remove keys with confirmation workflow
**UX Possibilities Enabled:**
- **Self-Managed VM Access** - Users can SSH into their virtual machines
- **Kubernetes Cluster Management** - Direct kubectl access to deployed clusters
- **Development Workflows** - Git repository access and CI/CD integration
- **Multi-Key Management** - Different keys for different environments (dev/staging/prod)
- **Team Collaboration** - Shared access keys for team-managed resources
- **Security Best Practices** - Key rotation and secure access patterns
#### **Architecture & System Interaction Analysis**
**Frontend-Backend Data Flow:**
```mermaid
graph TD
A[HTML Template] --> B[JavaScript Event Handlers]
B --> C[AJAX API Calls]
C --> D[Rust Controller]
D --> E[SSH Key Service]
E --> F[UserPersistence]
F --> G[JSON File Storage]
G --> F
F --> E
E --> D
D --> H[ResponseBuilder]
H --> C
C --> I[DOM Updates]
```
---
**Key Architectural Components:**
## 3. Immediate Development Priorities
1. **HTML Template Layer** ([`settings.html`](src/views/dashboard/settings.html))
- Bootstrap modal structure for user interactions
- Data attributes for JavaScript-HTML bridge (`data-key-id`)
- CSP-compliant template with no inline scripts
- JSON hydration blocks for data transfer
### **A. Runtime Testing & Validation (Week 1)**
1. **Critical User Flows**
- User registration and authentication via GitEa OAuth
- Product browsing, search, and filtering functionality
- Wallet operations (top-up, balance checks, transactions)
- Order placement and payment processing
- Provider product creation and management
2. **JavaScript Layer** ([`dashboard-ssh-keys.js`](src/static/js/dashboard-ssh-keys.js))
- Event delegation with null-safe programming
- Data attribute management for DOM-JavaScript bridge
- AJAX API integration with error handling
- Real-time validation and user feedback
2. **Integration Testing**
- End-to-end purchase workflows
- Multi-currency display and calculations
- Insufficient funds handling (HTTP 402 responses)
- Invoice generation and viewing
3. **Backend Service Layer** ([`ssh_key_service.rs`](src/services/ssh_key_service.rs))
- SSH key validation (Ed25519, ECDSA, RSA support)
- SHA256 fingerprint generation
- Duplicate prevention and user limits
- Auto-default logic for first key
3. **Data Persistence Validation**
- User data creation, updates, and retrieval
- Transaction history accuracy
- Product catalog aggregation performance
4. **Controller Integration** ([`dashboard.rs`](src/controllers/dashboard.rs))
- ResponseBuilder pattern for consistent JSON responses
- Session authentication and user validation
- Error handling with user-friendly messages
### **B. Feature Completion (Week 2)**
1. **Payment Integration**
- Complete TFC payment flow implementation
- Payment method management
- Auto-topup functionality
- Exchange rate handling
#### **Critical Frontend-Backend Integration Debugging**
2. **Provider Features**
- Service provider dashboard completion
- App provider deployment tools
- Farmer node management interface
- Revenue tracking and analytics
**Root Cause Identified & Solved:**
- **Issue**: Backend services worked perfectly (100% test success) but frontend buttons failed
- **Problem**: JavaScript was setting `data-key-id` on wrong DOM element during template cloning
- **Solution**: Fixed element targeting in [`dashboard-ssh-keys.js`](src/static/js/dashboard-ssh-keys.js:225)
- **Template Fix**: Added `data-key-id=""` placeholder to HTML template
3. **Marketplace Features**
- Advanced search and filtering
- Product recommendations
- Category browsing optimization
- Featured products management
**Debugging Process:**
1. **Backend Isolation**: Confirmed all 6 API endpoints working via service tests
2. **Frontend Simulation**: Identified disconnect between frontend and backend
3. **Data Flow Analysis**: Traced JavaScript data attribute handling
4. **DOM Inspection**: Found incorrect element targeting during cloning
5. **Systematic Fix**: Corrected both JavaScript logic and HTML template
### **C. Security & Production Readiness (Week 3+)**
1. **Security Hardening**
- CSRF protection implementation
- Rate limiting configuration
- Session security hardening
- Input validation and sanitization
2. **Performance Optimization**
- Catalog aggregation caching
- Database query optimization
- Asset optimization and CDN integration
- Response time monitoring
3. **Monitoring & Observability**
- Health check endpoints
- Metrics collection and alerting
- Error tracking and logging
- Performance monitoring
**Key Learning**: Frontend-backend integration issues often involve data attribute management and DOM element targeting rather than API functionality.
---
## 4. Development Guidelines for AI Coders
## 5. UX Testing Framework Development (Section 13 Implementation - 2025-08-22)
### **Code Organization**
- **Controllers**: [`src/controllers/`](src/controllers/) - HTTP request handling with ResponseBuilder
- **Services**: [`src/services/`](src/services/) - Business logic and data operations
- **Models**: [`src/models/`](src/models/) - Data structures and builders
- **Utils**: [`src/utils/`](src/utils/) - Shared utilities and helpers
##### Checkout & Orders Contract — Implemented 2025-08-23
### **Key Patterns to Follow**
1. **Always use [`ResponseBuilder`](src/utils/response_builder.rs)** for HTTP responses
2. **Persistent data only** - no mock data in production code
3. **CSP compliance** - external JS files only, no inline scripts
4. **Builder patterns** for complex object construction
5. **Error handling** with proper HTTP status codes (especially 402 for insufficient funds)
- **Template hydration**: `<script type="application/json" id="checkout-hydration">{{ hydration_json | safe }}</script>`; client reads via `document.getElementById('checkout-hydration').textContent` and parses.
- **Frontend request**: `POST /api/orders` with body:
- `payment_method`: `{ method_type: 'wallet', details: { source: 'usd_credits' } }`
- `currency`: e.g., `USD` (server also supports user preference)
- `cart_items`: ignored by server (order is constructed from session cart; field retained for fwd-compat)
- **Auth**: Requires authenticated session; anonymous users are redirected to login via UI.
- **Responses**:
- `200 OK`: `{ success, data: { order_id, confirmation_number } }` or `{ order_id, confirmation }` depending on legacy envelope; client tolerates both via unwrapping and key aliasing
- `400 Bad Request`: Validation or unsupported payment method; envelope includes `error` details
- `402 Payment Required`: Insufficient funds; standardized payload with currency-aware deficit
- `401 Unauthorized`: No session
- **Client UX**: Shows toast, best-effort clears server cart (`DELETE /api/cart`), refreshes navbar/cart/orders, then redirects to `/orders/{order_id}/confirmation[?confirmation=...]`.
### **Testing Strategy**
- **Unit tests** for service layer logic
- **Integration tests** for controller endpoints
- **End-to-end tests** for critical user workflows
- **Performance tests** for catalog aggregation and search
- **Manual validation (2025-08-23)**: user0 created a service; user1 executed Buy Now and Add to Cart successfully; orders appear under `/dashboard/orders`.
- **Remaining**: Validate `tests/frontend_ux/purchase_cart_ux_test.rs` with `--features ux_testing` for regression coverage.
### **Configuration Management**
- **Environment-based** configuration via [`ConfigurationBuilder`](src/config/builder.rs)
- **Feature flags** for dev/prod differences
- **Database connection** management
- **External service** integration settings
###### Frontend API Standardization — `window.apiJson` + 402 Interceptor
---
- **Global 402 handler** (`src/static/js/base.js`): wraps `window.fetch` to detect HTTP 402 and invoke `window.Errors.handleInsufficientFundsResponse(responseClone, text)` (throttled to prevent duplicate modals).
- **`window.apiJson` helper** (`src/static/js/base.js`):
- Sets `Accept: application/json`, defaults `credentials: 'same-origin'`.
- JSON-encodes plain object bodies when `Content-Type: application/json`.
- Reads text, parses JSON, and unwraps standardized envelopes: `const data = parsed.data ?? parsed`.
- On non-OK, throws `Error` with `.status`, `.errors`, `.data`, `.metadata`, `.body`.
- Returns `null` for 204/empty bodies.
- **Adoption**: `src/static/js/checkout.js` now uses `apiJson` for `POST /api/orders`. Keep migrating modules to ensure consistent headers, envelope handling, and centralized errors.
## 5. Success Criteria
Next Steps
- Optional: Audit other open JS modules you mentioned (`src/static/js/cart.js`, `src/static/js/checkout.js`, `src/static/js/dashboard.js`, and any legacy `static/js/dashboard.js`) for any remaining direct `fetch` usage and refactor to `apiJson` for consistency.
### **Functional Requirements**
- [ ] Complete user registration and authentication flow
- [ ] Full product browsing and purchase workflow
- [ ] Working payment processing with TFC integration
- [ ] Provider dashboards for all user types (service, app, farmer)
- [ ] Real-time wallet and transaction management
### **UX Testing Framework Implementation Status** ⚡ **MAJOR PROGRESS**
### **Technical Requirements**
- [ ] Zero compilation errors (✅ Achieved)
- [ ] All tests passing with >90% coverage
- [ ] Performance benchmarks met (sub-second page loads)
- [ ] Security audit passed
- [ ] Production deployment ready
#### **Completed & Validated Tests**
1. **SSH Key UX Tests** ✅ **ORIGINAL WORKING TEMPLATE**
- File: [`tests/frontend_ux/ssh_key_frontend_ux_test.rs`](tests/frontend_ux/ssh_key_frontend_ux_test.rs)
- Status: Fully functional reference implementation
- Pattern: Direct service calls, persistent data, simple cleanup
### **User Experience Goals**
- [ ] Intuitive navigation and product discovery
- [ ] Clear pricing and payment flow
- [ ] Responsive design across devices
- [ ] Comprehensive provider management tools
- [ ] Real-time updates and notifications
2. **Public Access UX Tests** ✅ **RECENTLY VALIDATED**
- File: [`tests/frontend_ux/public_access_ux_test.rs`](tests/frontend_ux/public_access_ux_test.rs)
- Status: Passes all tests (2 passed; 0 failed)
- Validates: Documentation pages, privacy, terms, about, contact access
---
#### **Rewritten Tests (Pending Final Validation)**
3. **Settings Management UX Tests** - [`tests/frontend_ux/settings_management_ux_test.rs`](tests/frontend_ux/settings_management_ux_test.rs)
4. **Credits Wallet UX Tests** - [`tests/frontend_ux/credits_wallet_ux_test.rs`](tests/frontend_ux/credits_wallet_ux_test.rs)
5. **Purchase Cart UX Tests** - [`tests/frontend_ux/purchase_cart_ux_test.rs`](tests/frontend_ux/purchase_cart_ux_test.rs)
6. **Authentication UX Tests** - [`tests/frontend_ux/authentication_ux_test.rs`](tests/frontend_ux/authentication_ux_test.rs)
7. **Marketplace Categories UX Tests** - [`tests/frontend_ux/marketplace_categories_ux_test.rs`](tests/frontend_ux/marketplace_categories_ux_test.rs)
8. **Provider Dashboards UX Tests** - [`tests/frontend_ux/provider_dashboards_ux_test.rs`](tests/frontend_ux/provider_dashboards_ux_test.rs)
## 6. Architecture Decision Records
### **Technical Breakthrough: SSH Key Template Pattern**
### **Data Architecture**
- **Decision**: Persistent file-based user data storage
- **Rationale**: Simplicity, portability, and direct user ownership
- **Implementation**: [`UserPersistence`](src/services/user_persistence.rs) service layer
#### **What Works (Proven Pattern)**
```rust
// Direct service instantiation with builder pattern
let ssh_service = SSHKeyService::builder().build()?;
### **Frontend Architecture**
- **Decision**: CSP-compliant external JS with JSON hydration
- **Rationale**: Security, maintainability, and separation of concerns
- **Implementation**: [`src/static/js/`](src/static/js/) modules with data hydration
// Persistent data operations (no session mocking)
let user_data = UserPersistence::load_user_data(user_email).unwrap_or_default();
### **API Design**
- **Decision**: Consistent JSON envelope via [`ResponseBuilder`](src/utils/response_builder.rs)
- **Rationale**: Predictable frontend integration and error handling
- **Implementation**: All controllers use ResponseBuilder pattern
// Direct service method calls
let result = ssh_service.add_ssh_key(user_email, &ssh_key)?;
---
// Simple cleanup without complex mocking
UserPersistence::delete_user_data(user_email)?;
```
This roadmap provides the complete vision and current state for an AI coder to continue development. The foundation is solid with zero compilation errors, established architectural patterns, and clear next steps for feature completion and production readiness.
#### **What Caused 89 Compilation Errors**
- **Session Mocking Complexity**: `MockActixSession` vs actual `Session` type mismatches
- **Currency Service Integration**: Method signature changes (`convert_usd_to_target_currency` vs `convert_usd_to_display_currency`)
- **Builder Pattern Compliance**: Inconsistent service construction patterns
#### **Solution Applied**
- **Removed all session mocking** from UX tests
- **Adopted persistent data approach** using [`UserPersistence`](src/services/user_persistence.rs)
- **Standardized service construction** using `.builder().build()` pattern
- **Fixed currency service calls** and removed where inappropriate (public access without sessions)
### **UX Testing Framework Architecture**
#### **Test Organization**
- **Directory**: [`tests/frontend_ux/`](tests/frontend_ux/)
- **Module Configuration**: [`tests/frontend_ux/mod.rs`](tests/frontend_ux/mod.rs)
- **Test Runner**: [`tests/frontend_ux/test_runner.rs`](tests/frontend_ux/test_runner.rs)
- **Cargo Feature**: Tests require `--features="ux_testing"` flag
#### **Test Execution Pattern**
```bash
# Individual test execution
cargo test --test public_access_ux --features="ux_testing"
# Full suite execution (when ready)
cargo test --features="ux_testing" frontend_ux
```
#### **Data Persistence Architecture**
- **User Data Storage**: [`user_data/{email}.json`](user_data/) files
- **No Mock Dependencies**: Real service implementations with persistent data
- **Cross-Reference Testing**: Manual testing validates automated results
### **Key Discoveries & Lessons Learned**
#### **Session-Free Testing Approach**
- **Persistent data testing** eliminates complex session mocking issues
- **Service-based testing** more reliable than HTTP endpoint testing
- **Builder pattern consistency** essential for successful compilation
#### **Real Application Issues Identified**
- **Password Change Bug**: Cross-reference testing revealed "undefined" error in password change functionality
- **Currency Service Integration**: Method signature mismatches fixed
- **Data Attribute Issues**: Frontend-backend integration patterns validated

623
docs/dev/design/archive/threefold-marketplace-todo.md Normal file
View File

@@ -0,0 +1,623 @@
# Project Mycelium TODO
> See also: [Design & Architecture](./projectmycelium-design-architecture.md)
> See also: [Roadmap](./projectmycelium-roadmap.md)
## Status
### Completed
- apiJson helper wired globally in `src/static/js/base.js` and fetch 402 interceptor enabled.
- Refactors to use `apiJson`:
- `src/static/js/base.js`: navbar dropdown and cart count loaders.
- `src/static/js/dashboard-ssh-keys.js`: list/add/update/delete/set-default flows.
- Inventory of remaining direct `fetch` usage created and prioritized.
- Backend observability & safety (2025-08-23):
- Structured logging with `req_id` and timings in `DashboardController::add_grid_nodes()` (`src/controllers/dashboard.rs`).
- Lock wait timing logged under target `concurrency`.
- Added timings to `UserPersistence::{load_user_data, save_user_data}` and created async wrappers `load_user_data_locked`/`save_user_data_locked` (`src/services/user_persistence.rs`).
- Per-user write serialization enforced in controller using existing per-user async lock via `UserPersistence::get_user_lock`.
- FarmerService multi-node add paths refactored to batch save once at the end to reduce write contention (`src/services/farmer.rs`).
- Wallet controller and pools (2025-08-23):
- Refactored `WalletController` to use `load_user_data_locked`/`save_user_data_locked` everywhere (no direct persistence calls).
- Propagated `req_id` through wallet handlers and helper `credit_recipient()`; added missing `.await` at call sites.
- Fixed `wallet_page` to call `load_user_with_persistent_data(.., req_id)` and await it.
- Updated `PoolController` to call `WalletController::load_user_with_session_data(.., req_id).await` (old 1-arg sync calls fixed).
- Build is green after refactor; warnings remain to be cleaned up.
- SSH keys and session manager (2025-08-23):
- Added async variants to `SSHKeyService` using per-user locked persistence wrappers with `req_id` propagation; kept sync methods for compatibility.
- Updated Dashboard SSH key endpoints to use the new async methods and pass/await `req_id`.
- Patched remaining synchronous `SessionManager::save_user_session_data` call in `src/controllers/pool.rs` to `save_user_session_data_async(..., req_id).await`.
- `cargo check` passed after these changes.
- Farmer dashboard: node groups API migration (2025-08-25):
- Refactored `src/static/js/dashboard-farmer.js` to use `apiJson` unwrapped responses for node groups (list/create/delete/assign).
- Removed legacy `result.success`/`data.success` checks; success inferred from resolved promises and errors handled via exceptions.
- Defensive response handling for `/api/dashboard/node-groups/api` supporting `Array` or `{ groups: [...] }` shapes.
- Preserved notifications, modal flows, and table refreshes.
- Grep verification: no remaining `result.success`/`data.success` checks and no direct `fetch(` usage in `dashboard-farmer.js`.
- Farmer dashboard node details modal (2025-08-24):
- Implemented `showNodeDetailsModal(node)` in `src/static/js/dashboard-farmer.js` and wired it to the "View details" action.
- Displays General, Capacity, Pricing, and SLA sections with defensive fallbacks, plus a collapsible Raw JSON data block.
- Fixed raw JSON rendering bug by precomputing `JSON.stringify(node, null, 2)` and interpolating, instead of passing a function to `safe()`.
- Ensures only one modal instance by removing any existing `#nodeDetailsModal` before appending.
- Farmer dashboard modal fixes (2025-08-25 22:35 local → 2025-08-26):
- Fixed malformed `loadExistingGroups()` in `src/static/js/dashboard-farmer.js` (closed braces, removed stray return, deduplicated function).
- Restored `viewNodeDetails(nodeId)` to fetch fresh data and added stale-guard: `AbortController` + sequence check and `{ cache: 'no-store' }`.
- Addressed "Create Custom Group" modal darken-only issue by preventing default delegated click handling and programmatically showing the modal.
- Moved `#createCustomNodeGroupModal` element under `document.body` before showing to avoid stacking/overflow contexts.
- Added CSS overrides to raise modal/backdrop z-index above navbar/sidebar (`.modal{z-index:1080}`, `.modal-backdrop{z-index:1070}`).
- Fixed "Manage Group" modal open failure by loading `/api/dashboard/node-groups` and selecting the group client-side (fallback since GET by id is not defined).
- Updated `viewNodeGroupDetails()` to use the list API and select by id client-side to avoid 404s.
- Added immediate UX feedback for grid sync: 'Syncing with ThreeFold Grid...' info toast on click, then success toast on completion.
- Note: Verification pending on dev instance to confirm visual display across layouts.
- Farmer dashboard: group selects consistency (2025-08-26 local):
- Updated `loadExistingGroups()` in `src/static/js/dashboard-farmer.js` to use detailed endpoint `/api/dashboard/node-groups/api`.
- Defensive response handling for `{ groups: [...] }` or array responses; no reliance on legacy `success` flags.
- Populates `#existingGroup` with default groups first, then custom groups; adds `(Custom)` label and displays node counts (from `stats.total_nodes` when present).
- Error handling: concise console error on fetch failure; leaves a safe default option intact.
- Slice statistics path fix (2025-08-24):
- Corrected frontend URL to `/api/dashboard/farmer/slice-statistics` in `src/static/js/dashboard-farmer.js` (previously missing `/farmer`).
- Backend handler `DashboardController::get_slice_statistics()` returns `success: true` with fields: `total_base_slices`, `allocated_base_slices`, `available_base_slices`, `slice_utilization_percentage`.
- Verified response e.g.: `{ "success": true, "data": { "statistics": { ... }, "success": true } }`.
- Service provider dashboard apiJson migration (2025-08-26 local):
- Refactored `src/static/js/dashboard-service-provider.js` to use `window.apiJson` for all JSON API calls.
- Converted legacy `.then/.catch` chains to `async/await` where needed; made `saveProgressUpdate()` async; `viewCompletedRequest()` now awaits `apiJson`.
- Kept raw `fetch()` only for invoice text/PDF downloads: `GET /api/dashboard/service-requests/{id}/invoice` with `Accept: text/plain` (generate) and `Accept: application/pdf` (download).
- Removed legacy `result.success`/`data.success` checks in this module; rely on exceptions from `apiJson` and show toasts accordingly.
- Verified by grep: no remaining JSON `fetch(` calls in this file; only invoice binary/text fetches remain by design.
- Dashboard pools apiJson migration (2025-08-27 local):
- Refactored `src/static/js/dashboard_pools.js` to use `window.apiJson` for all JSON API calls (pools data, analytics, buy/sell/stake flows).
- Removed legacy `success` flag checks; rely on exceptions thrown by `apiJson` with standardized toast handling.
- Catch blocks ignore HTTP 402 to rely on the global interceptor opening the credit modal (avoids duplicate error toasts).
- Added `{ cache: 'no-store' }` to `GET /api/pools` and `GET /api/pools/analytics` to prevent stale UI.
- Preserved UI flows (toasts, modal closing, input resets) and CSP compliance.
- Frontend apiJson migrations (2025-08-27 local):
- Migrated `src/static/js/dashboard-user.js`, `src/static/js/marketplace-integration.js`, `src/static/js/marketplace_dashboard.js`, and `src/static/js/services.js` to use `window.apiJson` for JSON APIs.
- Removed legacy `success` flag checks; standardized error handling and toasts; rely on global interceptor for HTTP 402.
- Added `{ cache: 'no-store' }` to applicable GET requests to avoid stale UI where relevant.
- Migrated `src/static/js/marketplace-compute.js`, `src/static/js/dashboard.js`, and `src/static/js/dashboard_layout.js`:
- Replaced JSON `fetch` calls with `window.apiJson`.
- Standardized error handling; rely on global 402 interceptor. In `marketplace-compute.js`, 401 triggers `showAuthenticationModal()` and successes use toasts when available.
- Kept UI/UX flows identical (button states, redirects). `dashboard.js` and `dashboard_layout.js` now use `apiJson` for navbar balance and cart count.
- Inline script migration to external JS files (2025-08-27 local):
- **COMPLETED**: Migrated all remaining inline scripts in HTML templates to external JS files for CSP compliance and maintainability.
- Created external JS files:
- `src/static/js/marketplace-category.js` - Handles add-to-cart for applications, gateways, and three_nodes pages
- `src/static/js/products-page.js` - Manages product listing view toggles and add-to-cart
- `src/static/js/product-detail-step2.js` - Handles quantity controls and add-to-cart on product detail pages
- `src/static/js/dashboard-settings.js` - Manages all dashboard settings forms and account operations
- `src/static/js/slice-rental-form.js` - Handles slice rental form submissions
- `src/static/js/cart-marketplace.js` - Manages cart operations
- `src/static/js/shared-handlers.js` - Provides centralized error handling utilities
- Updated templates to use external JS:
- `src/views/marketplace/applications.html` - Inline scripts removed, external JS referenced
- `src/views/marketplace/gateways.html` - Inline scripts removed, external JS referenced
- `src/views/marketplace/three_nodes.html` - Inline scripts removed, external JS referenced
- `src/views/marketplace/products.html` - Inline scripts removed, external JS referenced
- `src/views/marketplace/product_detail_step2.html` - Inline scripts removed, external JS referenced
- `src/views/dashboard/settings.html` - Extensive inline scripts removed, external JS referenced
- `src/views/slice_rental_form.html` - Inline fetch migrated to apiJson
- `src/views/cart.html` - All fetch calls migrated to apiJson
- All external JS files use `window.apiJson` with consistent error handling and CSP compliance.
- Global HTTP 402 interceptor handles credit modal; authentication errors show login modal.
- **Status**: 100% complete - zero remaining inline scripts or direct fetch calls in HTML templates.
- Add to cart functionality fixes (2025-08-27 local):
- **COMPLETED**: Fixed inconsistent request body formatting causing 400 Bad Request errors from product detail pages.
- **COMPLETED**: Fixed duplicate checkmark icons in success state by removing manual icon from `product-detail-step2.js`.
- **COMPLETED**: Standardized all add-to-cart API calls to use object body format (not JSON.stringify).
- **COMPLETED**: Fixed Tera template parsing error in `cart.html` by removing extra `{% endblock %}` tag.
- **Status**: Add to cart now works consistently from both marketplace listings and product detail pages.
- Code cleanup and optimization (2025-08-27 local):
- **COMPLETED**: Removed unused fields from `UserServiceConfig` struct: `include_metrics`, `cache_enabled`, `real_time_updates`.
- **COMPLETED**: Removed unused methods from `UserServiceBuilder` and `UserService` classes.
- **COMPLETED**: Fixed compilation errors by updating `DashboardController::user_data_api` to use simplified builder pattern.
- **Status**: Reduced dead code warnings and improved maintainability.
### Pending
- **COMPLETED**: All inline script migrations to external JS files with `apiJson` integration.
- **COMPLETED**: All fetch call migrations to `window.apiJson` across the codebase.
- **COMPLETED**: Add to cart functionality standardized and working across all pages.
- **COMPLETED**: Smoke test navbar/cart/SSH key flows after refactors - all core functionality verified working (navbar, login, cart, SSH keys).
- **COMPLETED**: Enhanced `apiJson` with comprehensive API handling capabilities (2025-08-27 local):
- **COMPLETED**: Added `apiFormData` helper for consistent FormData handling with automatic error handling.
- **COMPLETED**: Added `apiText` helper for non-JSON responses (PDFs, plain text, etc.) with performance logging.
- **COMPLETED**: Added `apiBlob` helper for binary downloads with consistent error handling.
- **COMPLETED**: Added request deduplication via `apiJsonDeduped` to prevent double submissions.
- **COMPLETED**: Added intelligent retry logic for GET requests (2 retries with exponential backoff).
- **COMPLETED**: Added comprehensive performance and error logging for development debugging.
- **COMPLETED**: Enhanced error handling with structured error objects containing status, body, and parsed data.
- **Status**: All API helpers are backward compatible and provide best-in-class API handling with logging, retries, and deduplication.
- TFC as base currency instead of USD (1 TFC = 1 USD display mapping).
- Farmer adds node → distributed into compute slices → appears in marketplace → purchasable → appears in orders.
- Verify all UX flows end-to-end.
- Slice statistics UI visualization:
- Frontend: render cards/charts using returned fields; consider sparklines for utilization and trend charts for history when available.
- Add graceful empty state and loading skeletons.
- **Critical JSON parsing errors** (2025-08-27 local):
- **COMPLETED**: Fixed `provider_instant_at_example_com.json` - corrected `availability` field type from string to boolean, added missing `created_at` and `updated_at` fields to service object.
- **COMPLETED**: Resolved `instant_customer_at_example_com.json` and `provider_instant_at_example_com.json` parsing issues by removing problematic files (clean slate approach).
- **COMPLETED**: Verified `products.json` already has required `last_updated` field.
- **Result**: All JSON parsing errors resolved - marketplace now loads without errors for guest users.
- **Status**: Marketplace functionality fully restored for both authenticated and guest users.
- Persistence & logging follow-ups:
- Audit remaining call sites to use `*_locked` wrappers (avoid double-lock): `wallet.rs` DONE; `session_manager.rs` DONE; `ssh_key_service.rs` DONE; `pool.rs` key paths patched. Verify other controllers/services.
- Ensure controllers propagate `req_id` into `SessionManager` and `SSHKeyService` methods for consistent correlation/timing (most updated; continue auditing).
- Audit findings (updated 2025-08-27 local):
- **COMPLETED**: All JSON `fetch(` usages migrated to `window.apiJson`. All inline scripts in HTML templates migrated to external JS files.
- Expected/intentional `fetch(` usage:
- `src/static/js/base.js` inside the `apiJson` helper implementation (by design).
- `src/static/js/dashboard-service-provider.js` for invoice text/PDF only (binary/text endpoints).
- **COMPLETED**: All inline scripts under `src/views/**` migrated to external JS files using `apiJson`; shared error handler utility created in `shared-handlers.js`.
- **COMPLETED**: CSP compliance fix (2025-08-27 local):
- **COMPLETED**: Removed remaining inline JavaScript from `src/views/dashboard/settings.html`.
- **COMPLETED**: All JavaScript functionality moved to external `src/static/js/dashboard-settings.js` file.
- **COMPLETED**: Settings template now uses only external JS file references and JSON hydration blocks.
- **Status**: 100% CSP compliant - zero inline scripts across entire codebase.
- **COMPLETED**: Dashboard settings comprehensive testing & fixes (2025-08-27 local):
- **COMPLETED**: Fixed critical `window.apiJson()` usage pattern errors in `dashboard-settings.js`:
- **Issue**: Multiple forms calling `response.json()` on `window.apiJson()` result (which already returns parsed JSON)
- **Fix**: Removed redundant `.json()` calls in profile, password, notifications, and account deletion forms
- **Impact**: All settings forms now work correctly without "Unexpected server response" errors
- **COMPLETED**: Fixed notifications form boolean serialization:
- **Issue**: Backend expects boolean values but form sent "on"/"null" strings from checkboxes
- **Fix**: Convert checkbox values to proper booleans before sending to `/api/dashboard/settings/notifications`
- **COMPLETED**: Fixed currency preference form:
- **Issue**: Form field name mismatch (`currency` vs `display_currency`) and missing Content-Type header
- **Fix**: Corrected field name and added `application/json` header for `/api/user/currency` endpoint
- **COMPLETED**: Removed non-functional "Notification Settings" tab to clean up UI
- **COMPLETED**: All settings functionality validated:
- ✅ Profile information updates (name, country)
- ✅ Password changes with validation
- ✅ SSH key management (add, edit, delete, set default)
- ✅ Currency preferences (USD, TFC, EUR, CAD)
- ✅ Account deletion flow (password verification, confirmation modal, soft delete, redirect)
- **Critical pattern identified for codebase audit**:
- **Pattern**: `window.apiJson().then(response => response.json())` or `await (await window.apiJson()).json()`
- **Root cause**: Misunderstanding that `window.apiJson()` returns parsed JSON, not raw Response object
- **Search targets**: All `.js` files in `src/static/js/` that use `window.apiJson()`
- **Priority**: High - this pattern causes "Unexpected server response" errors across the application
### Next TODOs
## Phase 1: Testing & Quality Assurance (Immediate)
- **COMPLETED**: Critical apiJson pattern audit (2025-08-27 local):
- **COMPLETED**: Audited all 26 JavaScript files using `window.apiJson()` (138+ total calls)
- **Result**: Zero additional problematic patterns found - all other files use `apiJson` correctly
- **Conclusion**: The "Unexpected server response" errors were isolated to `dashboard-settings.js` only
- **Status**: API layer is solid across the entire codebase
- **COMPLETED**: Checkout form JSON serialization fix (2025-08-27 local):
- **Issue**: Checkout sending raw JavaScript object as `body` instead of JSON string + missing Content-Type header
- **Error**: 500 Internal Server Error on `/api/orders` POST request
- **Fix**: Added `JSON.stringify(body)` and `Content-Type: application/json` header in `checkout.js`
- **Impact**: Checkout process now works correctly - order placement successful
- **COMPLETED**: Widespread JSON serialization issues fixed (2025-08-27 local):
- **Scope**: Fixed 25+ files with `body: { object }` instead of `body: JSON.stringify({ object })`
- **Files fixed**: `dashboard_cart.js`, `marketplace_dashboard.js`, `product-detail-step2.js`, `cart.js`, `dashboard_pools.js`, `dashboard-user.js`, `marketplace-compute.js`, `dashboard-app-provider.js`, `dashboard_wallet.js`, `dashboard-service-provider.js`
- **Changes**: Added `JSON.stringify()` and `Content-Type: application/json` headers to all API calls
- **Impact**: Prevents 500 errors across marketplace, wallet, cart, and dashboard operations
## 📋 Pending Features
### Core Marketplace
- 📋 **Product catalog** - Browse and search available products/services
- 📋 **Service booking** - Book and manage service appointments
- 📋 **Provider profiles** - Detailed provider information and ratings
- 📋 **Search functionality** - Advanced search and filtering options
### E-commerce Features
- 📋 **Shopping cart** - Add/remove items, quantity management
- 📋 **Checkout process** - Secure payment processing
- 📋 **Order management** - Track orders and delivery status
- 📋 **Payment integration** - Multiple payment method support
### Advanced Features
- 📋 **Rating system** - User reviews and ratings for services
- 📋 **Recommendation engine** - Personalized product suggestions
- 📋 **Analytics dashboard** - Usage statistics and insights
- 📋 **Multi-language support** - Internationalization features
## 🔧 Technical Notes
### Messaging System Implementation
- **Message ownership logic**: Uses `message.sender_email !== thread.recipient_email` to determine if message belongs to current user
- **No user detection required**: Thread structure provides recipient information, eliminating need for complex current user detection
- **Email-based identification**: Shows complete email addresses for message senders (except "You" for current user)
- **Thread-based conversations**: Each conversation is a thread with two participants (user_a and user_b)
- **API endpoints**: `/api/messages/threads` (list), `/api/messages/threads/{id}/messages` (messages), `/api/messages/threads/{id}/read` (mark read)
### Messaging System Implementation (2025-08-29)
- **COMPLETED**: Industry-standard message notification system with real-time unread count management
- **COMPLETED**: Cross-platform notification badges (navbar, sidebar, dropdown) with synchronized updates
- **COMPLETED**: Message thread management with proper read/unread state handling
- **COMPLETED**: Real-time polling system with desktop notifications and document title updates
- **COMPLETED**: Modal thread list with instant UI updates when marking messages as read
- **COMPLETED**: Backend unread count increment/decrement logic with proper thread participant handling
- **COMPLETED**: Navbar message badge positioned optimally near username (visible without dropdown)
- **Architecture**: Uses ResponseBuilder envelope pattern and CSP-compliant frontend standards
- **Integration**: Seamlessly integrated with existing messaging-system.js via custom events
- **Comprehensive testing of remaining functionality**:
- ~~Test dashboard settings~~ - **COMPLETED**
- ~~Fix checkout process~~ - **COMPLETED**
- ~~Scan marketplace for similar JSON serialization issues~~ - **COMPLETED**
{{ ... }}
- ~~Fix widespread JSON serialization issues~~ - **COMPLETED**
- **READY FOR TESTING**: All critical API serialization issues resolved
- Smoke test all marketplace flows: browse products, add to cart, complete checkout
- Verify slice rental form submission and cart operations
- Test authentication flows and credit modal triggers (HTTP 402)
- Validate farmer dashboard and service provider flows
- Test wallet operations and credit management
- **Performance & UX validation**:
- Monitor console for JavaScript errors or warnings
- Verify CSP compliance (no inline script violations) - **COMPLETED**
- Test error handling consistency across all migrated components
- Validate loading states and user feedback mechanisms
## Phase 2: Feature Development & Enhancement (Short-term)
### Messaging System Enhancement (2025-08-29)
**✅ COMPLETED: Dashboard Page Integration**
- ✅ Moved messaging from modal-based interface to dedicated dashboard page (`/dashboard/messages`)
- ✅ Created new route and controller endpoint following existing patterns (`/dashboard/wallet`, `/dashboard/user`)
- ✅ Implemented full-page messaging interface with better UX and more screen real estate
- ✅ Added proper navigation integration in dashboard sidebar with active state highlighting
**✅ COMPLETED: UI/UX Enhancement**
- ✅ Enhanced messaging UI design to match existing dashboard components (wallet, cart, modals)
- ✅ Implemented modern conversation interface with:
- ✅ Message bubbles with proper sender/recipient styling
- ✅ Timestamp formatting and message status indicators
- ✅ Real-time character count and input validation
- ✅ Loading states and empty state handling
- ✅ Toast notifications for success/error feedback (error only - success removed for cleaner UX)
- ✅ Followed design patterns from existing dashboard components
- ✅ Implemented responsive design for mobile and desktop views
- ✅ CSP-compliant implementation with external JS and JSON hydration
**✅ COMPLETED: Layout & UX Improvements (2025-08-29)**
- ✅ Fixed message input box cropping issue - now fully visible at bottom
- ✅ Implemented industry-standard messaging layout with proper viewport height calculations
- ✅ Restructured layout using flexbox with fixed header, scrollable messages, and anchored input
- ✅ Moved toast notifications from bottom-right to top-right to avoid blocking message input
- ✅ Removed success toast for message sending (message appearing in chat provides sufficient feedback)
- ✅ Added proper `gitea_enabled` configuration to dashboard controller context
**📋 PENDING: Advanced Features (Low Priority)**
- File attachment support and media preview
- Message search and filtering capabilities
- Message threading and reply functionality
- Typing indicators and read receipts
- Message reactions and emoji support
- Conversation archiving and organization
**Current Status**: Full-page messaging interface complete and production-ready with industry-standard UX. All core functionality implemented following marketplace design patterns and technical standards. Layout optimized for full-screen usage with proper message input visibility.
- **Slice statistics UI visualization**:
- Frontend: render cards/charts using returned fields; consider sparklines for utilization and trend charts for history when available.
- Add graceful empty state and loading skeletons.
- **Currency system enhancement**:
- TFC as base currency instead of USD (1 TFC = 1 USD display mapping).
- **Core marketplace workflow**:
- Farmer adds node → distributed into compute slices → appears in marketplace → purchasable → appears in orders.
## Phase 3: Advanced Features & Polish (Medium-term)
- **Farmer dashboard enhancements**:
- Align `updateNodeGroupSelects()` to the same normalization as `loadExistingGroups()` so `#nodeGroup` and `#nodeGroupSelect` include custom groups with consistent labels/counts.
- Extract a shared `normalizeGroups()` helper to centralize group_type handling (object|string), default/custom detection, and totals; reuse in both functions.
- Trigger select refresh after create/delete (call `updateNodeGroupSelects()` and `loadExistingGroups()` on success handlers).
- QA: Create/delete a custom group; verify Add Nodes and Edit Node modals refresh lists and labels; confirm counts.
- Improve empty/error states for group fetches (placeholder item on failure/empty; concise error logs).
- Sort groups alphabetically within Default and Custom sections.
- **API enhancements**:
- Review/extend `apiJson` for FormData and non-JSON/text endpoints if needed.
## Archived Work Plans (Historical Reference)
- Work plan: Group selects (2025-08-25 23:25 local)
- Step 1: Implement `normalizeGroups()` in `src/static/js/dashboard-farmer.js` (handles object|string `group_type`, default/custom detection, totals, labels).
- Step 2: Refactor `updateNodeGroupSelects()` to use `normalizeGroups()`; populate `#nodeGroup` and `#nodeGroupSelect` with defaults then customs; preserve "Single (No Group)".
- Step 3: Wire post-actions to refresh selects after create/delete (call `updateNodeGroupSelects()` and `loadExistingGroups()` on success).
- Step 4: QA: create/delete custom group; verify both Add Nodes and Edit Node modals update lists, counts, labels; check no console errors.
- Step 5: Polish: empty/error states and alphabetical sort within sections.
- Acceptance criteria:
- Custom groups appear in `#existingGroup`, `#nodeGroup`, and `#nodeGroupSelect` with "(Custom)" label and node counts where available.
- Default groups listed before custom groups; each section sorted alphabetically.
- Selects refresh automatically after create/delete without page reload.
- No direct `fetch(` or legacy `success` checks in the touched code paths; only `apiJson` with thrown errors.
- Work plan: API migrations (2025-08-25 23:26 local)
- Step 1: Migrated `src/static/js/dashboard-service-provider.js` to `apiJson`; remove `result.success`/`data.success` checks; centralize error toasts.
- Step 2: Migrated `src/static/js/dashboard_pools.js` and `src/static/js/dashboard-app-provider.js` (2025-08-27 local), plus `src/static/js/dashboard-user.js`, `src/static/js/marketplace-integration.js`, `src/static/js/marketplace_dashboard.js`, and `src/static/js/services.js` (2025-08-27 local) to `apiJson`.
- Step 3: Audit templates under `src/views/**` for inline `fetch` and swap to `apiJson` via a small injected helper or module import.
- Step 4: Add minimal shared error handler (utility) to standardize toast messages and 402 handling remains in interceptor.
- Acceptance criteria:
- No remaining `fetch(` calls in migrated files; all use `apiJson`.
- No boolean success flag checks; errors are thrown and surfaced to UI.
- 402 flows open the credit modal automatically via the global interceptor.
- Status (2025-08-27 local):
- **COMPLETED**: All JavaScript migration work finished successfully.
- API migrations Step 1 completed: `src/static/js/dashboard-service-provider.js` migrated to `apiJson`.
- API migrations Step 2 completed: `src/static/js/dashboard_pools.js` and `src/static/js/dashboard-app-provider.js` migrated.
- Additional migrations completed: `src/static/js/dashboard-user.js`, `src/static/js/marketplace-integration.js`, `src/static/js/marketplace_dashboard.js`, and `src/static/js/services.js`.
- Completed remaining JSON `fetch` migrations: `src/static/js/marketplace-compute.js`, `src/static/js/dashboard.js`, and `src/static/js/dashboard_layout.js` now use `apiJson`.
- **COMPLETED**: All inline scripts under `src/views/**` migrated to external JS files with `apiJson` integration.
- **COMPLETED**: Created 7 new external JS files with consistent error handling and CSP compliance.
- **COMPLETED**: Updated 8 HTML templates to remove all inline scripts and reference external JS files.
- **MILESTONE**: Zero remaining inline scripts or direct fetch calls in HTML templates across the entire codebase.
- QA plan: Farmer dashboard node groups (2025-08-25 23:26 local)
- Create custom group → appears in `#existingGroup`, `#nodeGroup`, `#nodeGroupSelect` with label/counts.
- Delete custom group → removed from all selects; affected nodes default to “Single (No Group)”.
- Assign node to custom group in Edit modal → persists and reflects in table and selects.
- Add nodes with selected group → nodes show in that group after add; counts update.
- Manage group view → opens and shows stats; no 404s; client-side selection continues to work.
- Error paths: simulate API failure; placeholder items render and no JS exceptions.
- Audit remaining parts of `src/static/js/dashboard-farmer.js` post-migration (modal state reset for View Details, Sync with Grid UX feedback, residual direct fetches if any).
- Add a small shared error handler utility to standardize toast messages; keep HTTP 402 handling in the global interceptor.
- Smoke/UX tests: navbar balance/cart count, add-to-cart, rent/purchase flows after recent apiJson migrations.
- Farmer Dashboard UX polish:
- Add loading states/spinners to group modals and node details fetch.
- Consider adding backend `GET /api/dashboard/node-groups/{id}` for direct access (optional; current client-side selection works).
- Improve error toasts with more context (group id, node id).
- Audit inline scripts under `src/views/**` and migrate to `apiJson` via an injected helper/module (ensure CSP compliance).
## Node Addition Reliability & JSON Persistence Hardening
- Deprecate legacy endpoint and handler
- Disable/remove route `POST /api/dashboard/add-nodes-automatic` in `src/routes/mod.rs` (legacy flow).
- Keep only `POST /api/dashboard/grid-nodes/add` → `DashboardController::add_grid_nodes`.
- If needed, hide behind a feature flag for quick rollback.
- Frontend duplication prevention
- In `src/static/js/dashboard-farmer.js`:
- No-op or remove `addValidatedNodes()` (legacy) and any bindings to it.
- Ensure only new handlers are attached: `validateGridNodes`, `addGridNodes`.
- Keep dataset flags in `initializeAddNodeForm()` to prevent duplicate listener attachment.
- Disable Add/Validate buttons during in-flight calls and re-enable on completion.
- Batch saves in FarmerService
- In `src/services/farmer.rs`, for multi-node add methods:
- `add_multiple_grid_nodes()`
- `add_multiple_grid_nodes_with_config()`
- `add_multiple_grid_nodes_with_comprehensive_config()`
- `add_multiple_grid_nodes_with_individual_pricing()`
- Accumulate node additions in memory and perform a single `UserPersistence::save_user_data(...)` after all nodes are appended to reduce write contention.
- Status: DONE (2025-08-23). Implemented single save after accumulation for multi-node paths.
- Per-user write serialization
- Introduce a per-user async lock (e.g., `tokio::sync::Mutex`) keyed by `user_email` (global registry using `once_cell` + `DashMap`/`Mutex`).
- Wrap load → modify → save sequences so only one write path per user runs at a time.
- Implement in persistence or a small write coordinator used by FarmerService and similar writers.
- Status: DONE (2025-08-23). Using global per-user async lock in `UserPersistence` and controller-level lock in `add_grid_nodes()`. Added `*_locked` persistence wrappers.
- Structured logging
- Add entry/exit logs and request IDs for `DashboardController::add_grid_nodes()` in `src/controllers/dashboard.rs`.
- Add logs around `load_user_data`/`save_user_data` in `src/services/user_persistence.rs` including user email, operation, and timing.
- Status: DONE (2025-08-23). Includes lock wait durations and total elapsed timings.
- Ops & remediation
- Keep `scripts/fix_user_data.py` documented as a repair tool, but aim for it to be unnecessary post-fix.
## What shipped in this pass (2025-08-23)
- Frontend
- `dashboard-farmer.js`: wiring continued to prefer new add flow and guard against duplicate submits.
- Backend
- Structured logging + timings with `req_id` in `add_grid_nodes()`; lock wait metrics under `concurrency` target.
- Persistence timing instrumentation; added `load_user_data_locked`/`save_user_data_locked` wrappers.
- FarmerService: multi-node add paths batch-save once.
- WalletController: replaced all direct persistence calls with `*_locked` and propagated `req_id`; fixed remaining async call sites and added await.
- PoolController: updated to call `WalletController::load_user_with_session_data(.., req_id).await` in handlers.
- General cleanup: consistent ID parsing and option handling in relevant areas.
## Build status (cargo check 2025-08-23 14:27 local)
- Result: cargo check passed (no errors). Binary built with warnings.
- Warnings: 346 (212 duplicates). Continue staged cleanup.
- Note: Structured logs now available under targets `api.dashboard`, `user_persistence`, and `concurrency`.
## Build status (cargo check 2025-08-25 15:26 local)
- Result: cargo check passed (no errors). Binary built with warnings.
- Warnings: 345 (210 duplicates). Continue cleanup, especially dead code in `slice_calculator.rs`, `slice_rental.rs`, `ssh_key_service.rs`, and `user_persistence.rs` noted by compiler.
- Note: JS changes (apiJson migrations) do not affect Rust build; proceed to warning cleanup after UI fixes.
## Next steps
- Warning cleanup across the codebase (prefix unused vars with `_` or remove; tighten imports).
- Audit persistence call sites and switch to `*_locked` wrappers where no outer controller lock exists:
- Prioritize: `src/services/session_manager.rs`, `src/services/ssh_key_service.rs`, and `src/controllers/pool.rs` persistence paths.
- Propagate `req_id` into `SessionManager` and `SSHKeyService`; audit controllers that call them to pass/await.
- Tests & manual verification:
- Concurrency test: two simultaneous add requests for the same user; expect serialized writes and correct persistence.
- Concurrency test: concurrent wallet transfers to same user; ensure per-user lock serialization and no corruption.
- Smoke test pool exchange/stake flows after we migrate pool persistence to `*_locked`.
- Smoke tests for navbar/cart/SSH keys after `apiJson` refactors.
- Documentation:
- Ensure references use `POST /api/dashboard/grid-nodes/add` (deprecated legacy endpoint removed/hidden).
## Learnings (2025-08-23)
- The E0061/E0308 errors were from outdated call sites after making helper methods async and adding `req_id`. Fixes required passing `req_id` and adding `.await` consistently.
- Keep a single locking discipline: use `UserPersistence::*_locked` for per-user JSON updates and avoid layering another lock around them.
- Generate `req_id` at handler boundaries and propagate into all persistence/service calls for traceable logs and timings.
- Stagger refactors module-by-module and run `cargo check` frequently to catch signature drifts early (especially in controllers consuming services).
## Details
### apiJson
- `window.apiJson` helper (`src/static/js/base.js`).
- Adoption status:
- `src/static/js/checkout.js` uses `apiJson` for `POST /api/orders`.
- `src/static/js/base.js` navbar and cart loaders now use `apiJson`.
- `src/static/js/dashboard-ssh-keys.js` fully migrated to `apiJson`.
- `src/static/js/dashboard-service-provider.js` fully migrated to `apiJson` (invoice downloads continue to use `fetch` for binary/text).
- `src/static/js/dashboard_pools.js` fully migrated to `apiJson` (GETs set `cache: 'no-store'`; toasts standardized; HTTP 402 handled by interceptor).
- `src/static/js/dashboard-app-provider.js` fully migrated to `apiJson` (GETs set `cache: 'no-store'` for dashboard/app list/deployment details; toasts standardized; HTTP 402 handled by interceptor).
- `src/static/js/dashboard-user.js` fully migrated to `apiJson`.
- `src/static/js/marketplace-integration.js` fully migrated to `apiJson`.
- `src/static/js/marketplace_dashboard.js` fully migrated to `apiJson`.
- `src/static/js/services.js` fully migrated to `apiJson`.
- `src/static/js/dashboard_wallet.js` uses `apiJson` for wallet endpoints.
- `src/static/js/marketplace-compute.js` fully migrated to `apiJson` (rent, purchase, add-to-cart; 402 handled by interceptor; 401 triggers auth modal).
- `src/static/js/dashboard.js` uses `apiJson` for navbar dropdown balance.
- `src/static/js/dashboard_layout.js` uses `apiJson` for dashboard cart count.
- Audit targets for migration (`fetch` → `apiJson`):
- Inline scripts under `src/views/**`.
## UI Progress (2025-08-25 15:26 local)
- Add Nodes: Works via the new flow (`POST /api/dashboard/grid-nodes/add`). Form resets and UI refresh triggers are in place; good progress.
- Delete Node: Works. Clicking Delete removes the node and the UI updates accordingly.
- View Details (Slice table action): Fixed stale/wrong data via abort + sequence guard and `no-store` caching; modal is rebuilt per open.
- Slice Statistics: Works. Endpoint `/api/dashboard/farmer/slice-statistics` returns stats and UI updates the counters.
- Refresh Calculations: Works (confirmed via notification and page reload).
- Sync with Grid: Shows immediate 'Syncing...' toast, then success; backend takes time, UX now communicates progress.
- Note: Re-test node group operations end-to-end after `apiJson` migration in `dashboard-farmer.js` (list/create/delete/assign). Monitor console/network for envelope-free responses and interceptor behavior.
## Next TODO: Farmer Dashboard UX Fixes (2025-08-25 22:35 local)
- Create Custom Group: Fixed (rely on data attributes only).
- Manage (Node Group): Fixed (load list and select by id client-side).
- Slice Management:
- Refresh Calculations: Works (✅). Leave as-is.
- Sync with Grid: Consider adding a subtle spinner on the button while in-flight (optional; toasts already added).
- Slice table Actions (View Details): Fixed; nodeId binding works and race/stale cases guarded.
- Capacity Analytics: No data or graphs are displayed.
- Earnings Monitoring: No data or graphs are displayed.
Notes:
- Verify button handlers and modal initialization in `src/static/js/dashboard-farmer.js`.
- Confirm API paths for refresh/sync actions match backend routes in `src/routes/mod.rs` (`/api/dashboard/refresh-slice-calculations`, `/api/dashboard/sync-with-grid`).
- Inspect console for errors and ensure `window.apiJson` calls resolve and update UI accordingly.
## ✅ RESOLVED: Messaging Interface Integration (2025-08-29)
### Problem Summary
The "Contact Provider" flow for service bookings had two critical issues:
1. Thread creation failing with 400 Bad Request due to missing Content-Type header
2. New threads not being auto-selected after creation due to incorrect matching logic
### Final Status - ALL ISSUES RESOLVED
- **Thread matching logic**: ✅ WORKING - correctly identifies no existing thread for new booking ID
- **New thread creation**: ✅ FIXED - Added missing Content-Type header to POST `/api/messages/threads`
- **Thread auto-selection**: ✅ FIXED - Updated matching logic to use both recipient_email AND context_id
- **Enhanced logging**: ✅ ADDED - both frontend and backend logging in place
### Technical Details
**Frontend (dashboard-messages.js):**
- URL parameter handling works correctly
- Thread search logic properly matches by `recipient_email` and `context_id` (booking ID)
- ✅ FIXED: Added missing `Content-Type: application/json` header to thread creation request
- Using correct endpoint: `/api/messages/threads` (not `/api/messages`)
**Backend (messaging.rs):**
- Added validation logging for `CreateThreadRequest`
- Validates: `recipient_email`, `context_type`, `subject` not empty
- Route registration confirmed correct in `src/routes/mod.rs` line 239
**API Structure:**
```rust
// Expected by /api/messages/threads
pub struct CreateThreadRequest {
pub recipient_email: String,
pub context_type: String,
pub context_id: Option<String>,
pub subject: String,
}
```
**Frontend Request Data:**
```javascript
{
recipient_email: "user0@example.com",
context_type: "service_booking",
context_id: "req-104ecd70-1699cf5a",
subject: "Service Booking #req-104ecd70-1699cf5a"
}
```
### Resolution Summary
**Root Cause:** Missing `Content-Type: application/json` header in frontend thread creation request caused server to return 400 Bad Request due to JSON deserialization failure.
**Fixes Applied:**
1. **Thread Creation Fix** - Added proper Content-Type header:
```javascript
const response = await apiJson('/api/messages/threads', {
method: 'POST',
headers: {
'Content-Type': 'application/json' // ← Added this
},
body: JSON.stringify(requestData)
});
```
2. **Thread Auto-Selection Fix** - Updated matching logic:
```javascript
// OLD: Only matched by recipient (selected wrong thread)
const newThread = this.threads.find(thread =>
thread.recipient_email === recipient
);
// NEW: Match by both recipient AND context_id (selects correct thread)
const newThread = this.threads.find(thread =>
thread.recipient_email === recipient &&
thread.context_id === bookingId
);
```
### Files Modified
- ✅ `src/static/js/dashboard-messages.js` - Added Content-Type header + fixed thread selection logic
- `src/controllers/messaging.rs` - Enhanced validation logging (already in place)
- `src/static/js/dashboard-user.js` - Contact Provider redirect (already working)
### Verified Working Flow
When clicking "Contact Provider" for a service booking:
1. ✅ Redirect to `/dashboard/messages` with URL parameters
2. ✅ Find no existing thread for booking ID
3. ✅ Create new thread via POST `/api/messages/threads` (FIXED - Content-Type header)
4. ✅ Reload conversations and auto-select new thread (FIXED - matching logic)
5. ✅ Display conversation interface ready for messaging
**Status: FULLY RESOLVED** - Contact Provider flow works end-to-end with proper thread auto-selection