tfgrid/projectmycelium

Fork 0

Files

mik-tf 70990b8a13 docs: archive old roadmap and todo files, add marketplace redesign plan

2025-09-06 10:23:00 -04:00

43 KiB

Raw Permalink Blame History

Project Mycelium TODO

See also: Design & Architecture

See also: Roadmap

Status

Completed

apiJson helper wired globally in src/static/js/base.js and fetch 402 interceptor enabled.
Refactors to use apiJson:
- src/static/js/base.js: navbar dropdown and cart count loaders.
- src/static/js/dashboard-ssh-keys.js: list/add/update/delete/set-default flows.
Inventory of remaining direct fetch usage created and prioritized.
Backend observability & safety (2025-08-23):
- Structured logging with req_id and timings in DashboardController::add_grid_nodes() (src/controllers/dashboard.rs).
- Lock wait timing logged under target concurrency.
- Added timings to UserPersistence::{load_user_data, save_user_data} and created async wrappers load_user_data_locked/save_user_data_locked (src/services/user_persistence.rs).
- Per-user write serialization enforced in controller using existing per-user async lock via UserPersistence::get_user_lock.
- FarmerService multi-node add paths refactored to batch save once at the end to reduce write contention (src/services/farmer.rs).
Wallet controller and pools (2025-08-23):
- Refactored WalletController to use load_user_data_locked/save_user_data_locked everywhere (no direct persistence calls).
- Propagated req_id through wallet handlers and helper credit_recipient(); added missing .await at call sites.
- Fixed wallet_page to call load_user_with_persistent_data(.., req_id) and await it.
- Updated PoolController to call WalletController::load_user_with_session_data(.., req_id).await (old 1-arg sync calls fixed).
- Build is green after refactor; warnings remain to be cleaned up.
SSH keys and session manager (2025-08-23):
- Added async variants to SSHKeyService using per-user locked persistence wrappers with req_id propagation; kept sync methods for compatibility.
- Updated Dashboard SSH key endpoints to use the new async methods and pass/await req_id.
- Patched remaining synchronous SessionManager::save_user_session_data call in src/controllers/pool.rs to save_user_session_data_async(..., req_id).await.
- cargo check passed after these changes.
Farmer dashboard: node groups API migration (2025-08-25):
- Refactored src/static/js/dashboard-farmer.js to use apiJson unwrapped responses for node groups (list/create/delete/assign).
- Removed legacy result.success/data.success checks; success inferred from resolved promises and errors handled via exceptions.
- Defensive response handling for /api/dashboard/node-groups/api supporting Array or { groups: [...] } shapes.
- Preserved notifications, modal flows, and table refreshes.
- Grep verification: no remaining result.success/data.success checks and no direct fetch( usage in dashboard-farmer.js.
Farmer dashboard node details modal (2025-08-24):
- Implemented showNodeDetailsModal(node) in src/static/js/dashboard-farmer.js and wired it to the "View details" action.
- Displays General, Capacity, Pricing, and SLA sections with defensive fallbacks, plus a collapsible Raw JSON data block.
- Fixed raw JSON rendering bug by precomputing JSON.stringify(node, null, 2) and interpolating, instead of passing a function to safe().
- Ensures only one modal instance by removing any existing #nodeDetailsModal before appending.
Farmer dashboard modal fixes (2025-08-25 22:35 local → 2025-08-26):
- Fixed malformed loadExistingGroups() in src/static/js/dashboard-farmer.js (closed braces, removed stray return, deduplicated function).
- Restored viewNodeDetails(nodeId) to fetch fresh data and added stale-guard: AbortController + sequence check and { cache: 'no-store' }.
- Addressed "Create Custom Group" modal darken-only issue by preventing default delegated click handling and programmatically showing the modal.
- Moved #createCustomNodeGroupModal element under document.body before showing to avoid stacking/overflow contexts.
- Added CSS overrides to raise modal/backdrop z-index above navbar/sidebar (.modal{z-index:1080}, .modal-backdrop{z-index:1070}).
- Fixed "Manage Group" modal open failure by loading /api/dashboard/node-groups and selecting the group client-side (fallback since GET by id is not defined).
- Updated viewNodeGroupDetails() to use the list API and select by id client-side to avoid 404s.
- Added immediate UX feedback for grid sync: 'Syncing with ThreeFold Grid...' info toast on click, then success toast on completion.
- Note: Verification pending on dev instance to confirm visual display across layouts.
Farmer dashboard: group selects consistency (2025-08-26 local):
- Updated loadExistingGroups() in src/static/js/dashboard-farmer.js to use detailed endpoint /api/dashboard/node-groups/api.
- Defensive response handling for { groups: [...] } or array responses; no reliance on legacy success flags.
- Populates #existingGroup with default groups first, then custom groups; adds (Custom) label and displays node counts (from stats.total_nodes when present).
- Error handling: concise console error on fetch failure; leaves a safe default option intact.
Slice statistics path fix (2025-08-24):
- Corrected frontend URL to /api/dashboard/farmer/slice-statistics in src/static/js/dashboard-farmer.js (previously missing /farmer).
- Backend handler DashboardController::get_slice_statistics() returns success: true with fields: total_base_slices, allocated_base_slices, available_base_slices, slice_utilization_percentage.
- Verified response e.g.: { "success": true, "data": { "statistics": { ... }, "success": true } }.
Service provider dashboard apiJson migration (2025-08-26 local):
- Refactored src/static/js/dashboard-service-provider.js to use window.apiJson for all JSON API calls.
- Converted legacy .then/.catch chains to async/await where needed; made saveProgressUpdate() async; viewCompletedRequest() now awaits apiJson.
- Kept raw fetch() only for invoice text/PDF downloads: GET /api/dashboard/service-requests/{id}/invoice with Accept: text/plain (generate) and Accept: application/pdf (download).
- Removed legacy result.success/data.success checks in this module; rely on exceptions from apiJson and show toasts accordingly.
- Verified by grep: no remaining JSON fetch( calls in this file; only invoice binary/text fetches remain by design.
Dashboard pools apiJson migration (2025-08-27 local):
- Refactored src/static/js/dashboard_pools.js to use window.apiJson for all JSON API calls (pools data, analytics, buy/sell/stake flows).
- Removed legacy success flag checks; rely on exceptions thrown by apiJson with standardized toast handling.
- Catch blocks ignore HTTP 402 to rely on the global interceptor opening the credit modal (avoids duplicate error toasts).
- Added { cache: 'no-store' } to GET /api/pools and GET /api/pools/analytics to prevent stale UI.
- Preserved UI flows (toasts, modal closing, input resets) and CSP compliance.
Frontend apiJson migrations (2025-08-27 local):
- Migrated src/static/js/dashboard-user.js, src/static/js/marketplace-integration.js, src/static/js/marketplace_dashboard.js, and src/static/js/services.js to use window.apiJson for JSON APIs.
- Removed legacy success flag checks; standardized error handling and toasts; rely on global interceptor for HTTP 402.
- Added { cache: 'no-store' } to applicable GET requests to avoid stale UI where relevant.
- Migrated src/static/js/marketplace-compute.js, src/static/js/dashboard.js, and src/static/js/dashboard_layout.js:
  - Replaced JSON fetch calls with window.apiJson.
  - Standardized error handling; rely on global 402 interceptor. In marketplace-compute.js, 401 triggers showAuthenticationModal() and successes use toasts when available.
  - Kept UI/UX flows identical (button states, redirects). dashboard.js and dashboard_layout.js now use apiJson for navbar balance and cart count.
Inline script migration to external JS files (2025-08-27 local):
- COMPLETED: Migrated all remaining inline scripts in HTML templates to external JS files for CSP compliance and maintainability.
- Created external JS files:
  - src/static/js/marketplace-category.js - Handles add-to-cart for applications, gateways, and three_nodes pages
  - src/static/js/products-page.js - Manages product listing view toggles and add-to-cart
  - src/static/js/product-detail-step2.js - Handles quantity controls and add-to-cart on product detail pages
  - src/static/js/dashboard-settings.js - Manages all dashboard settings forms and account operations
  - src/static/js/slice-rental-form.js - Handles slice rental form submissions
  - src/static/js/cart-marketplace.js - Manages cart operations
  - src/static/js/shared-handlers.js - Provides centralized error handling utilities
- Updated templates to use external JS:
  - src/views/marketplace/applications.html - Inline scripts removed, external JS referenced
  - src/views/marketplace/gateways.html - Inline scripts removed, external JS referenced
  - src/views/marketplace/three_nodes.html - Inline scripts removed, external JS referenced
  - src/views/marketplace/products.html - Inline scripts removed, external JS referenced
  - src/views/marketplace/product_detail_step2.html - Inline scripts removed, external JS referenced
  - src/views/dashboard/settings.html - Extensive inline scripts removed, external JS referenced
  - src/views/slice_rental_form.html - Inline fetch migrated to apiJson
  - src/views/cart.html - All fetch calls migrated to apiJson
- All external JS files use window.apiJson with consistent error handling and CSP compliance.
- Global HTTP 402 interceptor handles credit modal; authentication errors show login modal.
- Status: 100% complete - zero remaining inline scripts or direct fetch calls in HTML templates.
Add to cart functionality fixes (2025-08-27 local):
- COMPLETED: Fixed inconsistent request body formatting causing 400 Bad Request errors from product detail pages.
- COMPLETED: Fixed duplicate checkmark icons in success state by removing manual icon from product-detail-step2.js.
- COMPLETED: Standardized all add-to-cart API calls to use object body format (not JSON.stringify).
- COMPLETED: Fixed Tera template parsing error in cart.html by removing extra {% endblock %} tag.
- Status: Add to cart now works consistently from both marketplace listings and product detail pages.
Code cleanup and optimization (2025-08-27 local):
- COMPLETED: Removed unused fields from UserServiceConfig struct: include_metrics, cache_enabled, real_time_updates.
- COMPLETED: Removed unused methods from UserServiceBuilder and UserService classes.
- COMPLETED: Fixed compilation errors by updating DashboardController::user_data_api to use simplified builder pattern.
- Status: Reduced dead code warnings and improved maintainability.

Pending

COMPLETED: All inline script migrations to external JS files with apiJson integration.
COMPLETED: All fetch call migrations to window.apiJson across the codebase.
COMPLETED: Add to cart functionality standardized and working across all pages.
COMPLETED: Smoke test navbar/cart/SSH key flows after refactors - all core functionality verified working (navbar, login, cart, SSH keys).
COMPLETED: Enhanced apiJson with comprehensive API handling capabilities (2025-08-27 local):
- COMPLETED: Added apiFormData helper for consistent FormData handling with automatic error handling.
- COMPLETED: Added apiText helper for non-JSON responses (PDFs, plain text, etc.) with performance logging.
- COMPLETED: Added apiBlob helper for binary downloads with consistent error handling.
- COMPLETED: Added request deduplication via apiJsonDeduped to prevent double submissions.
- COMPLETED: Added intelligent retry logic for GET requests (2 retries with exponential backoff).
- COMPLETED: Added comprehensive performance and error logging for development debugging.
- COMPLETED: Enhanced error handling with structured error objects containing status, body, and parsed data.
- Status: All API helpers are backward compatible and provide best-in-class API handling with logging, retries, and deduplication.
TFC as base currency instead of USD (1 TFC = 1 USD display mapping).
Farmer adds node → distributed into compute slices → appears in marketplace → purchasable → appears in orders.
Verify all UX flows end-to-end.
Slice statistics UI visualization:
- Frontend: render cards/charts using returned fields; consider sparklines for utilization and trend charts for history when available.
- Add graceful empty state and loading skeletons.
Critical JSON parsing errors (2025-08-27 local):
- COMPLETED: Fixed provider_instant_at_example_com.json - corrected availability field type from string to boolean, added missing created_at and updated_at fields to service object.
- COMPLETED: Resolved instant_customer_at_example_com.json and provider_instant_at_example_com.json parsing issues by removing problematic files (clean slate approach).
- COMPLETED: Verified products.json already has required last_updated field.
- Result: All JSON parsing errors resolved - marketplace now loads without errors for guest users.
- Status: Marketplace functionality fully restored for both authenticated and guest users.
Persistence & logging follow-ups:
- Audit remaining call sites to use *_locked wrappers (avoid double-lock): wallet.rs DONE; session_manager.rs DONE; ssh_key_service.rs DONE; pool.rs key paths patched. Verify other controllers/services.
- Ensure controllers propagate req_id into SessionManager and SSHKeyService methods for consistent correlation/timing (most updated; continue auditing).
Audit findings (updated 2025-08-27 local):
- COMPLETED: All JSON fetch( usages migrated to window.apiJson. All inline scripts in HTML templates migrated to external JS files.
- Expected/intentional fetch( usage:
  - src/static/js/base.js inside the apiJson helper implementation (by design).
  - src/static/js/dashboard-service-provider.js for invoice text/PDF only (binary/text endpoints).
- COMPLETED: All inline scripts under src/views/** migrated to external JS files using apiJson; shared error handler utility created in shared-handlers.js.
COMPLETED: CSP compliance fix (2025-08-27 local):
- COMPLETED: Removed remaining inline JavaScript from src/views/dashboard/settings.html.
- COMPLETED: All JavaScript functionality moved to external src/static/js/dashboard-settings.js file.
- COMPLETED: Settings template now uses only external JS file references and JSON hydration blocks.
- Status: 100% CSP compliant - zero inline scripts across entire codebase.
COMPLETED: Dashboard settings comprehensive testing & fixes (2025-08-27 local):
- COMPLETED: Fixed critical window.apiJson() usage pattern errors in dashboard-settings.js:
  - Issue: Multiple forms calling response.json() on window.apiJson() result (which already returns parsed JSON)
  - Fix: Removed redundant .json() calls in profile, password, notifications, and account deletion forms
  - Impact: All settings forms now work correctly without "Unexpected server response" errors
- COMPLETED: Fixed notifications form boolean serialization:
  - Issue: Backend expects boolean values but form sent "on"/"null" strings from checkboxes
  - Fix: Convert checkbox values to proper booleans before sending to /api/dashboard/settings/notifications
- COMPLETED: Fixed currency preference form:
  - Issue: Form field name mismatch (currency vs display_currency) and missing Content-Type header
  - Fix: Corrected field name and added application/json header for /api/user/currency endpoint
- COMPLETED: Removed non-functional "Notification Settings" tab to clean up UI
- COMPLETED: All settings functionality validated:
  - ✅ Profile information updates (name, country)
  - ✅ Password changes with validation
  - ✅ SSH key management (add, edit, delete, set default)
  - ✅ Currency preferences (USD, TFC, EUR, CAD)
  - ✅ Account deletion flow (password verification, confirmation modal, soft delete, redirect)
Critical pattern identified for codebase audit:
- Pattern: window.apiJson().then(response => response.json()) or await (await window.apiJson()).json()
- Root cause: Misunderstanding that window.apiJson() returns parsed JSON, not raw Response object
- Search targets: All .js files in src/static/js/ that use window.apiJson()
- Priority: High - this pattern causes "Unexpected server response" errors across the application

Next TODOs

Phase 1: Testing & Quality Assurance (Immediate)

COMPLETED: Critical apiJson pattern audit (2025-08-27 local):
- COMPLETED: Audited all 26 JavaScript files using window.apiJson() (138+ total calls)
- Result: Zero additional problematic patterns found - all other files use apiJson correctly
- Conclusion: The "Unexpected server response" errors were isolated to dashboard-settings.js only
- Status: API layer is solid across the entire codebase
COMPLETED: Checkout form JSON serialization fix (2025-08-27 local):
- Issue: Checkout sending raw JavaScript object as body instead of JSON string + missing Content-Type header
- Error: 500 Internal Server Error on /api/orders POST request
- Fix: Added JSON.stringify(body) and Content-Type: application/json header in checkout.js
- Impact: Checkout process now works correctly - order placement successful
COMPLETED: Widespread JSON serialization issues fixed (2025-08-27 local):
- Scope: Fixed 25+ files with body: { object } instead of body: JSON.stringify({ object })
- Files fixed: dashboard_cart.js, marketplace_dashboard.js, product-detail-step2.js, cart.js, dashboard_pools.js, dashboard-user.js, marketplace-compute.js, dashboard-app-provider.js, dashboard_wallet.js, dashboard-service-provider.js
- Changes: Added JSON.stringify() and Content-Type: application/json headers to all API calls
- Impact: Prevents 500 errors across marketplace, wallet, cart, and dashboard operations

📋 Pending Features

Core Marketplace

📋 Product catalog - Browse and search available products/services
📋 Service booking - Book and manage service appointments
📋 Provider profiles - Detailed provider information and ratings
📋 Search functionality - Advanced search and filtering options

E-commerce Features

📋 Shopping cart - Add/remove items, quantity management
📋 Checkout process - Secure payment processing
📋 Order management - Track orders and delivery status
📋 Payment integration - Multiple payment method support

Advanced Features

📋 Rating system - User reviews and ratings for services
📋 Recommendation engine - Personalized product suggestions
📋 Analytics dashboard - Usage statistics and insights
📋 Multi-language support - Internationalization features

🔧 Technical Notes

Messaging System Implementation

Message ownership logic: Uses message.sender_email !== thread.recipient_email to determine if message belongs to current user
No user detection required: Thread structure provides recipient information, eliminating need for complex current user detection
Email-based identification: Shows complete email addresses for message senders (except "You" for current user)
Thread-based conversations: Each conversation is a thread with two participants (user_a and user_b)
API endpoints: /api/messages/threads (list), /api/messages/threads/{id}/messages (messages), /api/messages/threads/{id}/read (mark read)

Messaging System Implementation (2025-08-29)

COMPLETED: Industry-standard message notification system with real-time unread count management
COMPLETED: Cross-platform notification badges (navbar, sidebar, dropdown) with synchronized updates
COMPLETED: Message thread management with proper read/unread state handling
COMPLETED: Real-time polling system with desktop notifications and document title updates
COMPLETED: Modal thread list with instant UI updates when marking messages as read
COMPLETED: Backend unread count increment/decrement logic with proper thread participant handling
COMPLETED: Navbar message badge positioned optimally near username (visible without dropdown)
Architecture: Uses ResponseBuilder envelope pattern and CSP-compliant frontend standards
Integration: Seamlessly integrated with existing messaging-system.js via custom events
Comprehensive testing of remaining functionality:
- ~~Test dashboard settings~~ - COMPLETED
- ~~Fix checkout process~~ - COMPLETED
- ~~Scan marketplace for similar JSON serialization issues~~ - COMPLETED {{ ... }}
- ~~Fix widespread JSON serialization issues~~ - COMPLETED
- READY FOR TESTING: All critical API serialization issues resolved
- Smoke test all marketplace flows: browse products, add to cart, complete checkout
- Verify slice rental form submission and cart operations
- Test authentication flows and credit modal triggers (HTTP 402)
- Validate farmer dashboard and service provider flows
- Test wallet operations and credit management
Performance & UX validation:
- Monitor console for JavaScript errors or warnings
- Verify CSP compliance (no inline script violations) - COMPLETED
- Test error handling consistency across all migrated components
- Validate loading states and user feedback mechanisms

Phase 2: Feature Development & Enhancement (Short-term)

Messaging System Enhancement (2025-08-29)

✅ COMPLETED: Dashboard Page Integration

✅ Moved messaging from modal-based interface to dedicated dashboard page (/dashboard/messages)
✅ Created new route and controller endpoint following existing patterns (/dashboard/wallet, /dashboard/user)
✅ Implemented full-page messaging interface with better UX and more screen real estate
✅ Added proper navigation integration in dashboard sidebar with active state highlighting

✅ COMPLETED: UI/UX Enhancement

✅ Enhanced messaging UI design to match existing dashboard components (wallet, cart, modals)
✅ Implemented modern conversation interface with:
- ✅ Message bubbles with proper sender/recipient styling
- ✅ Timestamp formatting and message status indicators
- ✅ Real-time character count and input validation
- ✅ Loading states and empty state handling
- ✅ Toast notifications for success/error feedback (error only - success removed for cleaner UX)
✅ Followed design patterns from existing dashboard components
✅ Implemented responsive design for mobile and desktop views
✅ CSP-compliant implementation with external JS and JSON hydration

✅ COMPLETED: Layout & UX Improvements (2025-08-29)

✅ Fixed message input box cropping issue - now fully visible at bottom
✅ Implemented industry-standard messaging layout with proper viewport height calculations
✅ Restructured layout using flexbox with fixed header, scrollable messages, and anchored input
✅ Moved toast notifications from bottom-right to top-right to avoid blocking message input
✅ Removed success toast for message sending (message appearing in chat provides sufficient feedback)
✅ Added proper gitea_enabled configuration to dashboard controller context

📋 PENDING: Advanced Features (Low Priority)

File attachment support and media preview
Message search and filtering capabilities
Message threading and reply functionality
Typing indicators and read receipts
Message reactions and emoji support
Conversation archiving and organization

Current Status: Full-page messaging interface complete and production-ready with industry-standard UX. All core functionality implemented following marketplace design patterns and technical standards. Layout optimized for full-screen usage with proper message input visibility.

Slice statistics UI visualization:
- Frontend: render cards/charts using returned fields; consider sparklines for utilization and trend charts for history when available.
- Add graceful empty state and loading skeletons.
Currency system enhancement:
- TFC as base currency instead of USD (1 TFC = 1 USD display mapping).
Core marketplace workflow:
- Farmer adds node → distributed into compute slices → appears in marketplace → purchasable → appears in orders.

Phase 3: Advanced Features & Polish (Medium-term)

Farmer dashboard enhancements:
- Align updateNodeGroupSelects() to the same normalization as loadExistingGroups() so #nodeGroup and #nodeGroupSelect include custom groups with consistent labels/counts.
- Extract a shared normalizeGroups() helper to centralize group_type handling (object|string), default/custom detection, and totals; reuse in both functions.
- Trigger select refresh after create/delete (call updateNodeGroupSelects() and loadExistingGroups() on success handlers).
- QA: Create/delete a custom group; verify Add Nodes and Edit Node modals refresh lists and labels; confirm counts.
- Improve empty/error states for group fetches (placeholder item on failure/empty; concise error logs).
- Sort groups alphabetically within Default and Custom sections.
API enhancements:
- Review/extend apiJson for FormData and non-JSON/text endpoints if needed.

Archived Work Plans (Historical Reference)

Work plan: Group selects (2025-08-25 23:25 local)
- Step 1: Implement normalizeGroups() in src/static/js/dashboard-farmer.js (handles object|string group_type, default/custom detection, totals, labels).
- Step 2: Refactor updateNodeGroupSelects() to use normalizeGroups(); populate #nodeGroup and #nodeGroupSelect with defaults then customs; preserve "Single (No Group)".
- Step 3: Wire post-actions to refresh selects after create/delete (call updateNodeGroupSelects() and loadExistingGroups() on success).
- Step 4: QA: create/delete custom group; verify both Add Nodes and Edit Node modals update lists, counts, labels; check no console errors.
- Step 5: Polish: empty/error states and alphabetical sort within sections.
- Acceptance criteria:
  - Custom groups appear in #existingGroup, #nodeGroup, and #nodeGroupSelect with "(Custom)" label and node counts where available.
  - Default groups listed before custom groups; each section sorted alphabetically.
  - Selects refresh automatically after create/delete without page reload.
  - No direct fetch( or legacy success checks in the touched code paths; only apiJson with thrown errors.
Work plan: API migrations (2025-08-25 23:26 local)
- Step 1: Migrated src/static/js/dashboard-service-provider.js to apiJson; remove result.success/data.success checks; centralize error toasts.
- Step 2: Migrated src/static/js/dashboard_pools.js and src/static/js/dashboard-app-provider.js (2025-08-27 local), plus src/static/js/dashboard-user.js, src/static/js/marketplace-integration.js, src/static/js/marketplace_dashboard.js, and src/static/js/services.js (2025-08-27 local) to apiJson.
- Step 3: Audit templates under src/views/** for inline fetch and swap to apiJson via a small injected helper or module import.
- Step 4: Add minimal shared error handler (utility) to standardize toast messages and 402 handling remains in interceptor.
- Acceptance criteria:
  - No remaining fetch( calls in migrated files; all use apiJson.
  - No boolean success flag checks; errors are thrown and surfaced to UI.
  - 402 flows open the credit modal automatically via the global interceptor.
Status (2025-08-27 local):
- COMPLETED: All JavaScript migration work finished successfully.
- API migrations Step 1 completed: src/static/js/dashboard-service-provider.js migrated to apiJson.
- API migrations Step 2 completed: src/static/js/dashboard_pools.js and src/static/js/dashboard-app-provider.js migrated.
- Additional migrations completed: src/static/js/dashboard-user.js, src/static/js/marketplace-integration.js, src/static/js/marketplace_dashboard.js, and src/static/js/services.js.
- Completed remaining JSON fetch migrations: src/static/js/marketplace-compute.js, src/static/js/dashboard.js, and src/static/js/dashboard_layout.js now use apiJson.
- COMPLETED: All inline scripts under src/views/** migrated to external JS files with apiJson integration.
- COMPLETED: Created 7 new external JS files with consistent error handling and CSP compliance.
- COMPLETED: Updated 8 HTML templates to remove all inline scripts and reference external JS files.
- MILESTONE: Zero remaining inline scripts or direct fetch calls in HTML templates across the entire codebase.
QA plan: Farmer dashboard node groups (2025-08-25 23:26 local)
- Create custom group → appears in #existingGroup, #nodeGroup, #nodeGroupSelect with label/counts.
- Delete custom group → removed from all selects; affected nodes default to “Single (No Group)”.
- Assign node to custom group in Edit modal → persists and reflects in table and selects.
- Add nodes with selected group → nodes show in that group after add; counts update.
- Manage group view → opens and shows stats; no 404s; client-side selection continues to work.
- Error paths: simulate API failure; placeholder items render and no JS exceptions.
Audit remaining parts of src/static/js/dashboard-farmer.js post-migration (modal state reset for View Details, Sync with Grid UX feedback, residual direct fetches if any).
Add a small shared error handler utility to standardize toast messages; keep HTTP 402 handling in the global interceptor.
Smoke/UX tests: navbar balance/cart count, add-to-cart, rent/purchase flows after recent apiJson migrations.
Farmer Dashboard UX polish:
Add loading states/spinners to group modals and node details fetch.
Consider adding backend GET /api/dashboard/node-groups/{id} for direct access (optional; current client-side selection works).
Improve error toasts with more context (group id, node id).
Audit inline scripts under src/views/** and migrate to apiJson via an injected helper/module (ensure CSP compliance).

Node Addition Reliability & JSON Persistence Hardening

Deprecate legacy endpoint and handler
- Disable/remove route POST /api/dashboard/add-nodes-automatic in src/routes/mod.rs (legacy flow).
- Keep only POST /api/dashboard/grid-nodes/add → DashboardController::add_grid_nodes.
- If needed, hide behind a feature flag for quick rollback.
Frontend duplication prevention
- In src/static/js/dashboard-farmer.js:
  - No-op or remove addValidatedNodes() (legacy) and any bindings to it.
  - Ensure only new handlers are attached: validateGridNodes, addGridNodes.
  - Keep dataset flags in initializeAddNodeForm() to prevent duplicate listener attachment.
  - Disable Add/Validate buttons during in-flight calls and re-enable on completion.
Batch saves in FarmerService
- In src/services/farmer.rs, for multi-node add methods:
  - add_multiple_grid_nodes()
  - add_multiple_grid_nodes_with_config()
  - add_multiple_grid_nodes_with_comprehensive_config()
  - add_multiple_grid_nodes_with_individual_pricing()
- Accumulate node additions in memory and perform a single UserPersistence::save_user_data(...) after all nodes are appended to reduce write contention.
- Status: DONE (2025-08-23). Implemented single save after accumulation for multi-node paths.
Per-user write serialization
- Introduce a per-user async lock (e.g., tokio::sync::Mutex) keyed by user_email (global registry using once_cell + DashMap/Mutex).
- Wrap load → modify → save sequences so only one write path per user runs at a time.
- Implement in persistence or a small write coordinator used by FarmerService and similar writers.
- Status: DONE (2025-08-23). Using global per-user async lock in UserPersistence and controller-level lock in add_grid_nodes(). Added *_locked persistence wrappers.
Structured logging
- Add entry/exit logs and request IDs for DashboardController::add_grid_nodes() in src/controllers/dashboard.rs.
- Add logs around load_user_data/save_user_data in src/services/user_persistence.rs including user email, operation, and timing.
- Status: DONE (2025-08-23). Includes lock wait durations and total elapsed timings.
Ops & remediation
- Keep scripts/fix_user_data.py documented as a repair tool, but aim for it to be unnecessary post-fix.

What shipped in this pass (2025-08-23)

Frontend
- dashboard-farmer.js: wiring continued to prefer new add flow and guard against duplicate submits.
Backend
- Structured logging + timings with req_id in add_grid_nodes(); lock wait metrics under concurrency target.
- Persistence timing instrumentation; added load_user_data_locked/save_user_data_locked wrappers.
- FarmerService: multi-node add paths batch-save once.
- WalletController: replaced all direct persistence calls with *_locked and propagated req_id; fixed remaining async call sites and added await.
- PoolController: updated to call WalletController::load_user_with_session_data(.., req_id).await in handlers.
- General cleanup: consistent ID parsing and option handling in relevant areas.

Build status (cargo check 2025-08-23 14:27 local)

Result: cargo check passed (no errors). Binary built with warnings.
Warnings: 346 (212 duplicates). Continue staged cleanup.
Note: Structured logs now available under targets api.dashboard, user_persistence, and concurrency.

Build status (cargo check 2025-08-25 15:26 local)

Result: cargo check passed (no errors). Binary built with warnings.
Warnings: 345 (210 duplicates). Continue cleanup, especially dead code in slice_calculator.rs, slice_rental.rs, ssh_key_service.rs, and user_persistence.rs noted by compiler.
Note: JS changes (apiJson migrations) do not affect Rust build; proceed to warning cleanup after UI fixes.

Next steps

Warning cleanup across the codebase (prefix unused vars with _ or remove; tighten imports).
Audit persistence call sites and switch to *_locked wrappers where no outer controller lock exists:
- Prioritize: src/services/session_manager.rs, src/services/ssh_key_service.rs, and src/controllers/pool.rs persistence paths.
Propagate req_id into SessionManager and SSHKeyService; audit controllers that call them to pass/await.
Tests & manual verification:
- Concurrency test: two simultaneous add requests for the same user; expect serialized writes and correct persistence.
- Concurrency test: concurrent wallet transfers to same user; ensure per-user lock serialization and no corruption.
- Smoke test pool exchange/stake flows after we migrate pool persistence to *_locked.
Smoke tests for navbar/cart/SSH keys after apiJson refactors.
Documentation:
- Ensure references use POST /api/dashboard/grid-nodes/add (deprecated legacy endpoint removed/hidden).

Learnings (2025-08-23)

The E0061/E0308 errors were from outdated call sites after making helper methods async and adding req_id. Fixes required passing req_id and adding .await consistently.
Keep a single locking discipline: use UserPersistence::*_locked for per-user JSON updates and avoid layering another lock around them.
Generate req_id at handler boundaries and propagate into all persistence/service calls for traceable logs and timings.
Stagger refactors module-by-module and run cargo check frequently to catch signature drifts early (especially in controllers consuming services).

Details

apiJson

window.apiJson helper (src/static/js/base.js).
Adoption status:
- src/static/js/checkout.js uses apiJson for POST /api/orders.
- src/static/js/base.js navbar and cart loaders now use apiJson.
- src/static/js/dashboard-ssh-keys.js fully migrated to apiJson.
- src/static/js/dashboard-service-provider.js fully migrated to apiJson (invoice downloads continue to use fetch for binary/text).
- src/static/js/dashboard_pools.js fully migrated to apiJson (GETs set cache: 'no-store'; toasts standardized; HTTP 402 handled by interceptor).
- src/static/js/dashboard-app-provider.js fully migrated to apiJson (GETs set cache: 'no-store' for dashboard/app list/deployment details; toasts standardized; HTTP 402 handled by interceptor).
- src/static/js/dashboard-user.js fully migrated to apiJson.
- src/static/js/marketplace-integration.js fully migrated to apiJson.
- src/static/js/marketplace_dashboard.js fully migrated to apiJson.
- src/static/js/services.js fully migrated to apiJson.
- src/static/js/dashboard_wallet.js uses apiJson for wallet endpoints.
- src/static/js/marketplace-compute.js fully migrated to apiJson (rent, purchase, add-to-cart; 402 handled by interceptor; 401 triggers auth modal).
- src/static/js/dashboard.js uses apiJson for navbar dropdown balance.
- src/static/js/dashboard_layout.js uses apiJson for dashboard cart count.
Audit targets for migration (fetch → apiJson):
- Inline scripts under src/views/**.

UI Progress (2025-08-25 15:26 local)

Add Nodes: Works via the new flow (POST /api/dashboard/grid-nodes/add). Form resets and UI refresh triggers are in place; good progress.
Delete Node: Works. Clicking Delete removes the node and the UI updates accordingly.
View Details (Slice table action): Fixed stale/wrong data via abort + sequence guard and no-store caching; modal is rebuilt per open.
Slice Statistics: Works. Endpoint /api/dashboard/farmer/slice-statistics returns stats and UI updates the counters.
Refresh Calculations: Works (confirmed via notification and page reload).
Sync with Grid: Shows immediate 'Syncing...' toast, then success; backend takes time, UX now communicates progress.
Note: Re-test node group operations end-to-end after apiJson migration in dashboard-farmer.js (list/create/delete/assign). Monitor console/network for envelope-free responses and interceptor behavior.

Next TODO: Farmer Dashboard UX Fixes (2025-08-25 22:35 local)

Create Custom Group: Fixed (rely on data attributes only).
Manage (Node Group): Fixed (load list and select by id client-side).
Slice Management:
- Refresh Calculations: Works (✅). Leave as-is.
- Sync with Grid: Consider adding a subtle spinner on the button while in-flight (optional; toasts already added).
Slice table Actions (View Details): Fixed; nodeId binding works and race/stale cases guarded.
Capacity Analytics: No data or graphs are displayed.
Earnings Monitoring: No data or graphs are displayed.

Notes:

Verify button handlers and modal initialization in src/static/js/dashboard-farmer.js.
Confirm API paths for refresh/sync actions match backend routes in src/routes/mod.rs (/api/dashboard/refresh-slice-calculations, /api/dashboard/sync-with-grid).
Inspect console for errors and ensure window.apiJson calls resolve and update UI accordingly.

✅ RESOLVED: Messaging Interface Integration (2025-08-29)

Problem Summary

The "Contact Provider" flow for service bookings had two critical issues:

Thread creation failing with 400 Bad Request due to missing Content-Type header
New threads not being auto-selected after creation due to incorrect matching logic

Final Status - ALL ISSUES RESOLVED

Thread matching logic: ✅ WORKING - correctly identifies no existing thread for new booking ID
New thread creation: ✅ FIXED - Added missing Content-Type header to POST /api/messages/threads
Thread auto-selection: ✅ FIXED - Updated matching logic to use both recipient_email AND context_id
Enhanced logging: ✅ ADDED - both frontend and backend logging in place

Technical Details

Frontend (dashboard-messages.js):

URL parameter handling works correctly
Thread search logic properly matches by recipient_email and context_id (booking ID)
✅ FIXED: Added missing Content-Type: application/json header to thread creation request
Using correct endpoint: /api/messages/threads (not /api/messages)

Backend (messaging.rs):

Added validation logging for CreateThreadRequest
Validates: recipient_email, context_type, subject not empty
Route registration confirmed correct in src/routes/mod.rs line 239

API Structure:

// Expected by /api/messages/threads
pub struct CreateThreadRequest {
    pub recipient_email: String,
    pub context_type: String, 
    pub context_id: Option<String>,
    pub subject: String,
}

Frontend Request Data:

{
    recipient_email: "user0@example.com",
    context_type: "service_booking", 
    context_id: "req-104ecd70-1699cf5a",
    subject: "Service Booking #req-104ecd70-1699cf5a"
}

Resolution Summary

Root Cause: Missing Content-Type: application/json header in frontend thread creation request caused server to return 400 Bad Request due to JSON deserialization failure.

Fixes Applied:

Thread Creation Fix - Added proper Content-Type header:

const response = await apiJson('/api/messages/threads', {
    method: 'POST',
    headers: {
        'Content-Type': 'application/json'  // ← Added this
    },
    body: JSON.stringify(requestData)
});

Thread Auto-Selection Fix - Updated matching logic:

// OLD: Only matched by recipient (selected wrong thread)
const newThread = this.threads.find(thread => 
    thread.recipient_email === recipient
);

// NEW: Match by both recipient AND context_id (selects correct thread)
const newThread = this.threads.find(thread => 
    thread.recipient_email === recipient && 
    thread.context_id === bookingId
);

Files Modified

✅ src/static/js/dashboard-messages.js - Added Content-Type header + fixed thread selection logic
src/controllers/messaging.rs - Enhanced validation logging (already in place)
src/static/js/dashboard-user.js - Contact Provider redirect (already working)

Verified Working Flow

When clicking "Contact Provider" for a service booking:

✅ Redirect to /dashboard/messages with URL parameters
✅ Find no existing thread for booking ID
✅ Create new thread via POST /api/messages/threads (FIXED - Content-Type header)
✅ Reload conversations and auto-select new thread (FIXED - matching logic)
✅ Display conversation interface ready for messaging

Status: FULLY RESOLVED - Contact Provider flow works end-to-end with proper thread auto-selection

43 KiB Raw Permalink Blame History