init projectmycelium

2025-09-01 21:37:01 -04:00
commit b41efb0e99
319 changed files with 128160 additions and 0 deletions
--- a/docs/dev/design/archive/SLICE_PLAN.md
+++ b/docs/dev/design/archive/SLICE_PLAN.md
@@ -0,0 +1,469 @@
+# Project Mycelium - Slice Management Enhancement Plan
+## Focused Enhancement of Existing Architecture
+
+---
+
+## 🎯 **Project Overview**
+
+**Goal**: Add dynamic slice management to the existing Project Mycelium by enhancing current architecture rather than rebuilding it.
+
+**Key Principle**: A slice is the minimum unit: **1 vCPU, 4GB RAM, 200GB storage**
+
+**Enhancement Strategy**: Leverage existing [`GridService`](projectmycelium/src/services/grid.rs:1), [`FarmNode`](projectmycelium/src/models/user.rs:164), and [`NodeMarketplaceService`](projectmycelium/src/services/node_marketplace.rs:1) to add slice functionality.
+
+---
+
+## 🏗️ **Current Architecture Analysis**
+
+### **✅ What Already Exists and Works:**
+
+1. **[`GridService`](projectmycelium/src/services/grid.rs:1)** - Robust gridproxy integration
+   - Fetches node data from ThreeFold gridproxy API
+   - Converts to internal [`GridNodeData`](projectmycelium/src/models/user.rs:788) format
+   - Has mock data fallback for testing
+   - Uses builder pattern consistently
+
+2. **[`FarmNode`](projectmycelium/src/models/user.rs:164)** - Comprehensive node model
+   - Has `capacity: NodeCapacity` (total resources)
+   - Has `used_capacity: NodeCapacity` (currently used)
+   - Has `grid_node_id` and `grid_data` for grid integration
+   - Has `rental_options` for configuration
+   - Already stored in [`./user_data/`](projectmycelium/user_data/user1_at_example_com.json:1) JSON files
+
+3. **[`NodeMarketplaceService`](projectmycelium/src/services/node_marketplace.rs:1)** - Node-to-marketplace conversion
+   - Scans all farmer data from `./user_data/` directory
+   - Converts [`FarmNode`](projectmycelium/src/models/user.rs:164) to [`Product`](projectmycelium/src/models/product.rs:1)
+   - Has filtering by location, price, CPU, RAM, storage
+   - Calculates capacity statistics
+
+4. **[`MarketplaceController`](projectmycelium/src/controllers/marketplace.rs:1)** - Marketplace endpoints
+   - `/marketplace/compute` endpoint exists
+   - Has pagination and filtering
+   - Uses currency conversion
+   - Integrates with [`NodeMarketplaceService`](projectmycelium/src/services/node_marketplace.rs:1)
+
+### **🔧 What Needs Enhancement:**
+
+1. **Slice calculation** - Add automatic slice calculation from node capacity
+2. **Slice allocation** - Add atomic slice rental to prevent conflicts
+3. **Marketplace display** - Show slices instead of full nodes
+4. **Dashboard tracking** - Track slice rentals for farmers and users
+
+---
+
+## 🔄 **User Experience Flow (Same Goals)**
+
+### **Farmer Journey**
+1. **Farmer adds node** → Uses existing node addition, system fetches from gridproxy
+2. **System calculates slices** → NEW: Automatic calculation based on node capacity
+3. **Farmer sets slice price** → NEW: Price per slice within platform limits
+4. **Slices appear in marketplace** → ENHANCED: `/marketplace/compute` shows slices
+5. **Farmer monitors rentals** → ENHANCED: Dashboard shows slice rental income
+
+### **User Journey**
+1. **User browses marketplace** → ENHANCED: `/marketplace/compute` shows available slices
+2. **User selects slice** → NEW: Sees slice specs, node SLA, location, price
+3. **User rents slice** → NEW: Atomic allocation prevents conflicts
+4. **User sees rental** → ENHANCED: Dashboard shows slice rentals
+
+---
+
+## 🛠️ **Implementation Plan**
+
+### **Phase 1: Enhance FarmNode Model**
+**File**: [`src/models/user.rs`](projectmycelium/src/models/user.rs:164) (ADD fields to existing struct)
+
+```rust
+// ADD these fields to existing FarmNode struct around line 164
+impl FarmNode {
+    // ... keep ALL existing fields ...
+    
+    // NEW: Slice management fields
+    #[serde(default)]
+    pub calculated_slices: Vec<CalculatedSlice>,
+    #[serde(default)]
+    pub allocated_slices: Vec<AllocatedSlice>,
+    #[serde(default = "default_slice_price")]
+    pub slice_price_per_hour: rust_decimal::Decimal,
+    #[serde(default)]
+    pub price_locked: bool,
+    #[serde(default = "default_min_uptime")]
+    pub min_uptime_sla: f32,
+    #[serde(default = "default_min_bandwidth")]
+    pub min_bandwidth_mbps: i32,
+}
+
+// NEW: Slice structures
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct CalculatedSlice {
+    pub id: String,
+    pub cpu_cores: i32,    // Always 1
+    pub memory_gb: i32,    // Always 4
+    pub storage_gb: i32,   // Always 200
+    pub status: SliceStatus,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub enum SliceStatus {
+    Available,
+    Reserved,
+    Allocated,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct AllocatedSlice {
+    pub slice_id: String,
+    pub renter_email: String,
+    pub rental_start: DateTime<Utc>,
+    pub rental_end: Option<DateTime<Utc>>,
+    pub monthly_cost: rust_decimal::Decimal,
+}
+```
+
+### **Phase 2: Create Slice Calculator Service**
+**File**: `src/services/slice_calculator.rs` (NEW)
+
+```rust
+use crate::models::user::{FarmNode, CalculatedSlice, SliceStatus};
+
+pub struct SliceCalculatorService;
+
+impl SliceCalculatorService {
+    pub fn calculate_available_slices(node: &FarmNode) -> Vec<CalculatedSlice> {
+        let available_cpu = node.capacity.cpu_cores - node.used_capacity.cpu_cores;
+        let available_ram = node.capacity.memory_gb - node.used_capacity.memory_gb;
+        let available_storage = node.capacity.storage_gb - node.used_capacity.storage_gb;
+        
+        // Algorithm: Max slices = min(CPU/1, RAM/4, Storage/200)
+        let max_slices_by_cpu = available_cpu / 1;
+        let max_slices_by_ram = available_ram / 4;
+        let max_slices_by_storage = available_storage / 200;
+        
+        let max_slices = std::cmp::min(
+            std::cmp::min(max_slices_by_cpu, max_slices_by_ram),
+            max_slices_by_storage
+        );
+        
+        (0..max_slices)
+            .map(|i| CalculatedSlice {
+                id: format!("{}_slice_{}", node.id, i + 1),
+                cpu_cores: 1,
+                memory_gb: 4,
+                storage_gb: 200,
+                status: SliceStatus::Available,
+            })
+            .collect()
+    }
+}
+```
+
+### **Phase 3: Enhance GridService**
+**File**: [`src/services/grid.rs`](projectmycelium/src/services/grid.rs:1) (ADD method to existing)
+
+```rust
+// ADD this method to existing GridService implementation
+impl GridService {
+    pub async fn fetch_node_with_slices(&self, node_id: u32) -> Result<FarmNode, String> {
+        // Use existing fetch_node_data method
+        let grid_data = self.fetch_node_data(node_id).await?;
+        
+        // Use existing conversion logic
+        let mut farm_node = self.convert_grid_data_to_farm_node(&grid_data)?;
+        
+        // NEW: Calculate slices
+        farm_node.calculated_slices = SliceCalculatorService::calculate_available_slices(&farm_node);
+        farm_node.allocated_slices = Vec::new();
+        farm_node.slice_price_per_hour = rust_decimal::Decimal::from_str("0.50").unwrap();
+        
+        Ok(farm_node)
+    }
+}
+```
+
+### **Phase 4: Create Slice Allocation Service**
+**File**: `src/services/slice_allocation.rs` (NEW)
+
+```rust
+use crate::services::user_persistence::UserPersistence;
+use std::fs::OpenOptions;
+use std::io::{Seek, SeekFrom};
+
+pub struct SliceAllocationService;
+
+impl SliceAllocationService {
+    pub fn rent_slice_atomic(
+        node_id: &str,
+        slice_id: &str,
+        user_email: &str,
+        duration_months: u32,
+    ) -> Result<(), String> {
+        // Find farmer who owns the node
+        let farmer_email = self.find_node_owner(node_id)?;
+        
+        // Lock farmer's data file
+        let file_path = format!("./user_data/{}.json", farmer_email.replace("@", "_at_"));
+        let _lock_file = OpenOptions::new()
+            .create(true)
+            .write(true)
+            .open(format!("{}.lock", file_path))
+            .map_err(|e| format!("Failed to acquire lock: {}", e))?;
+        
+        // Load farmer data
+        let mut farmer_data = UserPersistence::load_user_data(&farmer_email)
+            .ok_or("Farmer data not found")?;
+        
+        // Find node and slice
+        let node = farmer_data.nodes.iter_mut()
+            .find(|n| n.id == node_id)
+            .ok_or("Node not found")?;
+        
+        let slice = node.calculated_slices.iter_mut()
+            .find(|s| s.id == slice_id)
+            .ok_or("Slice not found")?;
+        
+        // Check if still available
+        if slice.status != SliceStatus::Available {
+            return Err("Slice no longer available".to_string());
+        }
+        
+        // Allocate slice
+        slice.status = SliceStatus::Allocated;
+        let allocation = AllocatedSlice {
+            slice_id: slice_id.to_string(),
+            renter_email: user_email.to_string(),
+            rental_start: chrono::Utc::now(),
+            rental_end: Some(chrono::Utc::now() + chrono::Duration::days(duration_months as i64 * 30)),
+            monthly_cost: node.slice_price_per_hour * rust_decimal::Decimal::from(24 * 30),
+        };
+        node.allocated_slices.push(allocation);
+        
+        // Save farmer data
+        UserPersistence::save_user_data(&farmer_email, &farmer_data)?;
+        
+        // Add rental to user data
+        let mut user_data = UserPersistence::load_user_data(user_email)
+            .unwrap_or_default();
+        user_data.node_rentals.push(/* create NodeRental */);
+        UserPersistence::save_user_data(user_email, &user_data)?;
+        
+        Ok(())
+    }
+}
+```
+
+### **Phase 5: Enhance NodeMarketplaceService**
+**File**: [`src/services/node_marketplace.rs`](projectmycelium/src/services/node_marketplace.rs:1) (ADD method to existing)
+
+```rust
+// ADD this method to existing NodeMarketplaceService implementation
+impl NodeMarketplaceService {
+    pub fn get_all_available_slices(&self) -> Vec<SliceMarketplaceInfo> {
+        let mut available_slices = Vec::new();
+        
+        // Use existing logic to scan user_data directory
+        if let Ok(entries) = std::fs::read_dir("./user_data/") {
+            for entry in entries.flatten() {
+                if let Some(filename) = entry.file_name().to_str() {
+                    if filename.ends_with(".json") && !filename.starts_with('.') {
+                        let farmer_email = filename.replace("_at_", "@").replace(".json", "");
+                        
+                        if let Some(farmer_data) = UserPersistence::load_user_data(&farmer_email) {
+                            for node in farmer_data.nodes {
+                                for slice in node.calculated_slices {
+                                    if slice.status == SliceStatus::Available {
+                                        available_slices.push(SliceMarketplaceInfo {
+                                            slice,
+                                            node_info: NodeInfo {
+                                                id: node.id.clone(),
+                                                name: node.name.clone(),
+                                                location: node.location.clone(),
+                                                uptime_sla: node.min_uptime_sla,
+                                                bandwidth_sla: node.min_bandwidth_mbps,
+                                                price_per_hour: node.slice_price_per_hour,
+                                            },
+                                            farmer_email: farmer_email.clone(),
+                                        });
+                                    }
+                                }
+                            }
+                        }
+                    }
+                }
+            }
+        }
+        
+        available_slices
+    }
+}
+
+#[derive(Debug, Clone)]
+pub struct SliceMarketplaceInfo {
+    pub slice: CalculatedSlice,
+    pub node_info: NodeInfo,
+    pub farmer_email: String,
+}
+
+#[derive(Debug, Clone)]
+pub struct NodeInfo {
+    pub id: String,
+    pub name: String,
+    pub location: String,
+    pub uptime_sla: f32,
+    pub bandwidth_sla: i32,
+    pub price_per_hour: rust_decimal::Decimal,
+}
+```
+
+### **Phase 6: Enhance MarketplaceController**
+**File**: [`src/controllers/marketplace.rs`](projectmycelium/src/controllers/marketplace.rs:1) (MODIFY existing method)
+
+```rust
+// ENHANCE existing compute_resources method around line 150
+impl MarketplaceController {
+    pub async fn compute_resources(tmpl: web::Data<Tera>, session: Session, query: web::Query<std::collections::HashMap<String, String>>) -> Result<impl Responder> {
+        // Keep existing setup code...
+        
+        // REPLACE product service logic with slice marketplace logic
+        let node_marketplace_service = NodeMarketplaceService::builder()
+            .currency_service(currency_service.clone())
+            .build()
+            .map_err(|e| actix_web::error::ErrorInternalServerError(e))?;
+        
+        // Get available slices instead of products
+        let available_slices = node_marketplace_service.get_all_available_slices();
+        
+        // Apply existing filtering logic to slices
+        let filtered_slices = self.apply_slice_filters(&available_slices, &query);
+        
+        // Keep existing pagination and currency conversion logic...
+        
+        ctx.insert("slices", &slice_products);
+        render_template(&tmpl, "marketplace/compute.html", &ctx)
+    }
+    
+    // NEW: Slice rental endpoint
+    pub async fn rent_slice(request: web::Json<RentSliceRequest>, session: Session) -> Result<impl Responder> {
+        let user_email = session.get::<String>("user_email")
+            .map_err(|_| actix_web::error::ErrorUnauthorized("Not logged in"))?
+            .ok_or_else(|| actix_web::error::ErrorUnauthorized("Not logged in"))?;
+        
+        match SliceAllocationService::rent_slice_atomic(
+            &request.node_id,
+            &request.slice_id,
+            &user_email,
+            request.duration_months,
+        ) {
+            Ok(_) => Ok(HttpResponse::Ok().json(serde_json::json!({
+                "success": true,
+                "message": "Slice rented successfully"
+            }))),
+            Err(e) => Ok(HttpResponse::BadRequest().json(serde_json::json!({
+                "success": false,
+                "message": e
+            }))),
+        }
+    }
+}
+```
+
+### **Phase 7: Enhance Frontend Template**
+**File**: [`templates/marketplace/compute.html`](projectmycelium/src/views/marketplace/compute_resources.html:1) (MODIFY existing)
+
+```html
+<!-- ENHANCE existing compute.html template -->
+<!-- Replace product display with slice display -->
+
+<div class="slice-marketplace">
+    <h3>🍰 Available Compute Slices</h3>
+    
+    <div class="table-responsive">
+        <table class="table table-striped">
+            <thead>
+                <tr>
+                    <th>Node</th>
+                    <th>Location</th>
+                    <th>Resources</th>
+                    <th>SLA</th>
+                    <th>Price</th>
+                    <th>Action</th>
+                </tr>
+            </thead>
+            <tbody>
+                {{#each slices}}
+                <tr>
+                    <td>{{node_info.name}}</td>
+                    <td>{{node_info.location}}</td>
+                    <td>
+                        <span class="badge badge-secondary">1 vCPU</span>
+                        <span class="badge badge-secondary">4GB RAM</span>
+                        <span class="badge badge-secondary">200GB Storage</span>
+                    </td>
+                    <td>
+                        <div>🔄 {{node_info.uptime_sla}}% uptime</div>
+                        <div>🌐 {{node_info.bandwidth_sla}} Mbps</div>
+                    </td>
+                    <td>{{formatted_price}}</td>
+                    <td>
+                        <button class="btn btn-success btn-sm rent-slice-btn" 
+                                data-slice-id="{{slice.id}}" 
+                                data-node-id="{{node_info.id}}">
+                            Rent Slice
+                        </button>
+                    </td>
+                </tr>
+                {{/each}}
+            </tbody>
+        </table>
+    </div>
+</div>
+
+<script>
+$('.rent-slice-btn').click(function() {
+    const sliceId = $(this).data('slice-id');
+    const nodeId = $(this).data('node-id');
+    
+    $.ajax({
+        url: '/api/marketplace/rent-slice',
+        method: 'POST',
+        contentType: 'application/json',
+        data: JSON.stringify({
+            node_id: nodeId,
+            slice_id: sliceId,
+            duration_months: 1
+        }),
+        success: function(response) {
+            if (response.success) {
+                alert('Slice rented successfully!');
+                location.reload();
+            } else {
+                alert('Error: ' + response.message);
+            }
+        }
+    });
+});
+</script>
+```
+
+---
+
+## 🎯 **Key Benefits of This Approach**
+
+1. **Leverages Existing Work** - Builds on robust [`GridService`](projectmycelium/src/services/grid.rs:1) and [`NodeMarketplaceService`](projectmycelium/src/services/node_marketplace.rs:1)
+2. **Minimal Changes** - Adds fields to existing [`FarmNode`](projectmycelium/src/models/user.rs:164), doesn't rebuild
+3. **Consistent Patterns** - Uses existing builder patterns and JSON persistence
+4. **Same UX Goals** - Achieves all the same user experience objectives
+5. **Incremental Implementation** - Can be implemented phase by phase
+
+---
+
+## 📋 **Implementation Checklist**
+
+- [ ] **Phase 1**: Add slice fields to [`FarmNode`](projectmycelium/src/models/user.rs:164)
+- [ ] **Phase 2**: Create `SliceCalculatorService`
+- [ ] **Phase 3**: Enhance [`GridService`](projectmycelium/src/services/grid.rs:1) with slice calculation
+- [ ] **Phase 4**: Create `SliceAllocationService` with atomic operations
+- [ ] **Phase 5**: Enhance [`NodeMarketplaceService`](projectmycelium/src/services/node_marketplace.rs:1) for slices
+- [ ] **Phase 6**: Modify [`MarketplaceController`](projectmycelium/src/controllers/marketplace.rs:1) compute endpoint
+- [ ] **Phase 7**: Update marketplace template for slice display
+
+**Estimated Implementation**: 2-3 days (vs weeks for a complete rewrite)