projectmycelium/docs/dev/design/archive/SLICE_PLAN.md

Project Mycelium - Slice Management Enhancement Plan

Focused Enhancement of Existing Architecture

---

🎯 Project Overview

Goal: Add dynamic slice management to the existing Project Mycelium by enhancing current architecture rather than rebuilding it.

Key Principle: A slice is the minimum unit: 1 vCPU, 4GB RAM, 200GB storage

Enhancement Strategy: Leverage existing GridService, FarmNode, and NodeMarketplaceService to add slice functionality.

---

🏗️ Current Architecture Analysis

✅ What Already Exists and Works:

1. GridService - Robust gridproxy integration

   • Fetches node data from ThreeFold gridproxy API
   • Converts to internal GridNodeData format
   • Has mock data fallback for testing
   • Uses builder pattern consistently

2. FarmNode - Comprehensive node model

   • Has capacity: NodeCapacity (total resources)
   • Has used_capacity: NodeCapacity (currently used)
   • Has grid_node_id and grid_data for grid integration
   • Has rental_options for configuration
   • Already stored in ./user_data/ JSON files

3. NodeMarketplaceService - Node-to-marketplace conversion

   • Scans all farmer data from ./user_data/ directory
   • Converts FarmNode to Product
   • Has filtering by location, price, CPU, RAM, storage
   • Calculates capacity statistics

4. MarketplaceController - Marketplace endpoints

   • /marketplace/compute endpoint exists
   • Has pagination and filtering
   • Uses currency conversion
   • Integrates with NodeMarketplaceService

🔧 What Needs Enhancement:

1. Slice calculation - Add automatic slice calculation from node capacity
2. Slice allocation - Add atomic slice rental to prevent conflicts
3. Marketplace display - Show slices instead of full nodes
4. Dashboard tracking - Track slice rentals for farmers and users

---

🔄 User Experience Flow (Same Goals)

Farmer Journey

1. Farmer adds node → Uses existing node addition, system fetches from gridproxy
2. System calculates slices → NEW: Automatic calculation based on node capacity
3. Farmer sets slice price → NEW: Price per slice within platform limits
4. Slices appear in marketplace → ENHANCED: /marketplace/compute shows slices
5. Farmer monitors rentals → ENHANCED: Dashboard shows slice rental income

User Journey

1. User browses marketplace → ENHANCED: /marketplace/compute shows available slices
2. User selects slice → NEW: Sees slice specs, node SLA, location, price
3. User rents slice → NEW: Atomic allocation prevents conflicts
4. User sees rental → ENHANCED: Dashboard shows slice rentals

---

🛠️ Implementation Plan

Phase 1: Enhance FarmNode Model

File: src/models/user.rs (ADD fields to existing struct)

// ADD these fields to existing FarmNode struct around line 164
impl FarmNode {
    // ... keep ALL existing fields ...
    
    // NEW: Slice management fields
    #[serde(default)]
    pub calculated_slices: Vec<CalculatedSlice>,
    #[serde(default)]
    pub allocated_slices: Vec<AllocatedSlice>,
    #[serde(default = "default_slice_price")]
    pub slice_price_per_hour: rust_decimal::Decimal,
    #[serde(default)]
    pub price_locked: bool,
    #[serde(default = "default_min_uptime")]
    pub min_uptime_sla: f32,
    #[serde(default = "default_min_bandwidth")]
    pub min_bandwidth_mbps: i32,
}

// NEW: Slice structures
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CalculatedSlice {
    pub id: String,
    pub cpu_cores: i32,    // Always 1
    pub memory_gb: i32,    // Always 4
    pub storage_gb: i32,   // Always 200
    pub status: SliceStatus,
}

#[derive(Debug, Clone, Serialize, Deserialize)]
pub enum SliceStatus {
    Available,
    Reserved,
    Allocated,
}

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AllocatedSlice {
    pub slice_id: String,
    pub renter_email: String,
    pub rental_start: DateTime<Utc>,
    pub rental_end: Option<DateTime<Utc>>,
    pub monthly_cost: rust_decimal::Decimal,
}

Phase 2: Create Slice Calculator Service

File: src/services/slice_calculator.rs (NEW)

use crate::models::user::{FarmNode, CalculatedSlice, SliceStatus};

pub struct SliceCalculatorService;

impl SliceCalculatorService {
    pub fn calculate_available_slices(node: &FarmNode) -> Vec<CalculatedSlice> {
        let available_cpu = node.capacity.cpu_cores - node.used_capacity.cpu_cores;
        let available_ram = node.capacity.memory_gb - node.used_capacity.memory_gb;
        let available_storage = node.capacity.storage_gb - node.used_capacity.storage_gb;
        
        // Algorithm: Max slices = min(CPU/1, RAM/4, Storage/200)
        let max_slices_by_cpu = available_cpu / 1;
        let max_slices_by_ram = available_ram / 4;
        let max_slices_by_storage = available_storage / 200;
        
        let max_slices = std::cmp::min(
            std::cmp::min(max_slices_by_cpu, max_slices_by_ram),
            max_slices_by_storage
        );
        
        (0..max_slices)
            .map(|i| CalculatedSlice {
                id: format!("{}_slice_{}", node.id, i + 1),
                cpu_cores: 1,
                memory_gb: 4,
                storage_gb: 200,
                status: SliceStatus::Available,
            })
            .collect()
    }
}

Phase 3: Enhance GridService

File: src/services/grid.rs (ADD method to existing)

// ADD this method to existing GridService implementation
impl GridService {
    pub async fn fetch_node_with_slices(&self, node_id: u32) -> Result<FarmNode, String> {
        // Use existing fetch_node_data method
        let grid_data = self.fetch_node_data(node_id).await?;
        
        // Use existing conversion logic
        let mut farm_node = self.convert_grid_data_to_farm_node(&grid_data)?;
        
        // NEW: Calculate slices
        farm_node.calculated_slices = SliceCalculatorService::calculate_available_slices(&farm_node);
        farm_node.allocated_slices = Vec::new();
        farm_node.slice_price_per_hour = rust_decimal::Decimal::from_str("0.50").unwrap();
        
        Ok(farm_node)
    }
}

Phase 4: Create Slice Allocation Service

File: src/services/slice_allocation.rs (NEW)

use crate::services::user_persistence::UserPersistence;
use std::fs::OpenOptions;
use std::io::{Seek, SeekFrom};

pub struct SliceAllocationService;

impl SliceAllocationService {
    pub fn rent_slice_atomic(
        node_id: &str,
        slice_id: &str,
        user_email: &str,
        duration_months: u32,
    ) -> Result<(), String> {
        // Find farmer who owns the node
        let farmer_email = self.find_node_owner(node_id)?;
        
        // Lock farmer's data file
        let file_path = format!("./user_data/{}.json", farmer_email.replace("@", "_at_"));
        let _lock_file = OpenOptions::new()
            .create(true)
            .write(true)
            .open(format!("{}.lock", file_path))
            .map_err(|e| format!("Failed to acquire lock: {}", e))?;
        
        // Load farmer data
        let mut farmer_data = UserPersistence::load_user_data(&farmer_email)
            .ok_or("Farmer data not found")?;
        
        // Find node and slice
        let node = farmer_data.nodes.iter_mut()
            .find(|n| n.id == node_id)
            .ok_or("Node not found")?;
        
        let slice = node.calculated_slices.iter_mut()
            .find(|s| s.id == slice_id)
            .ok_or("Slice not found")?;
        
        // Check if still available
        if slice.status != SliceStatus::Available {
            return Err("Slice no longer available".to_string());
        }
        
        // Allocate slice
        slice.status = SliceStatus::Allocated;
        let allocation = AllocatedSlice {
            slice_id: slice_id.to_string(),
            renter_email: user_email.to_string(),
            rental_start: chrono::Utc::now(),
            rental_end: Some(chrono::Utc::now() + chrono::Duration::days(duration_months as i64 * 30)),
            monthly_cost: node.slice_price_per_hour * rust_decimal::Decimal::from(24 * 30),
        };
        node.allocated_slices.push(allocation);
        
        // Save farmer data
        UserPersistence::save_user_data(&farmer_email, &farmer_data)?;
        
        // Add rental to user data
        let mut user_data = UserPersistence::load_user_data(user_email)
            .unwrap_or_default();
        user_data.node_rentals.push(/* create NodeRental */);
        UserPersistence::save_user_data(user_email, &user_data)?;
        
        Ok(())
    }
}

Phase 5: Enhance NodeMarketplaceService

File: src/services/node_marketplace.rs (ADD method to existing)

// ADD this method to existing NodeMarketplaceService implementation
impl NodeMarketplaceService {
    pub fn get_all_available_slices(&self) -> Vec<SliceMarketplaceInfo> {
        let mut available_slices = Vec::new();
        
        // Use existing logic to scan user_data directory
        if let Ok(entries) = std::fs::read_dir("./user_data/") {
            for entry in entries.flatten() {
                if let Some(filename) = entry.file_name().to_str() {
                    if filename.ends_with(".json") && !filename.starts_with('.') {
                        let farmer_email = filename.replace("_at_", "@").replace(".json", "");
                        
                        if let Some(farmer_data) = UserPersistence::load_user_data(&farmer_email) {
                            for node in farmer_data.nodes {
                                for slice in node.calculated_slices {
                                    if slice.status == SliceStatus::Available {
                                        available_slices.push(SliceMarketplaceInfo {
                                            slice,
                                            node_info: NodeInfo {
                                                id: node.id.clone(),
                                                name: node.name.clone(),
                                                location: node.location.clone(),
                                                uptime_sla: node.min_uptime_sla,
                                                bandwidth_sla: node.min_bandwidth_mbps,
                                                price_per_hour: node.slice_price_per_hour,
                                            },
                                            farmer_email: farmer_email.clone(),
                                        });
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
        
        available_slices
    }
}

#[derive(Debug, Clone)]
pub struct SliceMarketplaceInfo {
    pub slice: CalculatedSlice,
    pub node_info: NodeInfo,
    pub farmer_email: String,
}

#[derive(Debug, Clone)]
pub struct NodeInfo {
    pub id: String,
    pub name: String,
    pub location: String,
    pub uptime_sla: f32,
    pub bandwidth_sla: i32,
    pub price_per_hour: rust_decimal::Decimal,
}

Phase 6: Enhance MarketplaceController

File: src/controllers/marketplace.rs (MODIFY existing method)

// ENHANCE existing compute_resources method around line 150
impl MarketplaceController {
    pub async fn compute_resources(tmpl: web::Data<Tera>, session: Session, query: web::Query<std::collections::HashMap<String, String>>) -> Result<impl Responder> {
        // Keep existing setup code...
        
        // REPLACE product service logic with slice marketplace logic
        let node_marketplace_service = NodeMarketplaceService::builder()
            .currency_service(currency_service.clone())
            .build()
            .map_err(|e| actix_web::error::ErrorInternalServerError(e))?;
        
        // Get available slices instead of products
        let available_slices = node_marketplace_service.get_all_available_slices();
        
        // Apply existing filtering logic to slices
        let filtered_slices = self.apply_slice_filters(&available_slices, &query);
        
        // Keep existing pagination and currency conversion logic...
        
        ctx.insert("slices", &slice_products);
        render_template(&tmpl, "marketplace/compute.html", &ctx)
    }
    
    // NEW: Slice rental endpoint
    pub async fn rent_slice(request: web::Json<RentSliceRequest>, session: Session) -> Result<impl Responder> {
        let user_email = session.get::<String>("user_email")
            .map_err(|_| actix_web::error::ErrorUnauthorized("Not logged in"))?
            .ok_or_else(|| actix_web::error::ErrorUnauthorized("Not logged in"))?;
        
        match SliceAllocationService::rent_slice_atomic(
            &request.node_id,
            &request.slice_id,
            &user_email,
            request.duration_months,
        ) {
            Ok(_) => Ok(HttpResponse::Ok().json(serde_json::json!({
                "success": true,
                "message": "Slice rented successfully"
            }))),
            Err(e) => Ok(HttpResponse::BadRequest().json(serde_json::json!({
                "success": false,
                "message": e
            }))),
        }
    }
}

Phase 7: Enhance Frontend Template

File: templates/marketplace/compute.html (MODIFY existing)

<!-- ENHANCE existing compute.html template -->
<!-- Replace product display with slice display -->

<div class="slice-marketplace">
    <h3>🍰 Available Compute Slices</h3>
    
    <div class="table-responsive">
        <table class="table table-striped">
            <thead>
                <tr>
                    <th>Node</th>
                    <th>Location</th>
                    <th>Resources</th>
                    <th>SLA</th>
                    <th>Price</th>
                    <th>Action</th>
                </tr>
            </thead>
            <tbody>
                {{#each slices}}
                <tr>
                    <td>{{node_info.name}}</td>
                    <td>{{node_info.location}}</td>
                    <td>
                        <span class="badge badge-secondary">1 vCPU</span>
                        <span class="badge badge-secondary">4GB RAM</span>
                        <span class="badge badge-secondary">200GB Storage</span>
                    </td>
                    <td>
                        <div>🔄 {{node_info.uptime_sla}}% uptime</div>
                        <div>🌐 {{node_info.bandwidth_sla}} Mbps</div>
                    </td>
                    <td>{{formatted_price}}</td>
                    <td>
                        <button class="btn btn-success btn-sm rent-slice-btn" 
                                data-slice-id="{{slice.id}}" 
                                data-node-id="{{node_info.id}}">
                            Rent Slice
                        </button>
                    </td>
                </tr>
                {{/each}}
            </tbody>
        </table>
    </div>
</div>

<script>
$('.rent-slice-btn').click(function() {
    const sliceId = $(this).data('slice-id');
    const nodeId = $(this).data('node-id');
    
    $.ajax({
        url: '/api/marketplace/rent-slice',
        method: 'POST',
        contentType: 'application/json',
        data: JSON.stringify({
            node_id: nodeId,
            slice_id: sliceId,
            duration_months: 1
        }),
        success: function(response) {
            if (response.success) {
                alert('Slice rented successfully!');
                location.reload();
            } else {
                alert('Error: ' + response.message);
            }
        }
    });
});
</script>

---

🎯 Key Benefits of This Approach

1. Leverages Existing Work - Builds on robust GridService and NodeMarketplaceService
2. Minimal Changes - Adds fields to existing FarmNode, doesn't rebuild
3. Consistent Patterns - Uses existing builder patterns and JSON persistence
4. Same UX Goals - Achieves all the same user experience objectives
5. Incremental Implementation - Can be implemented phase by phase

---

📋 Implementation Checklist

• Phase 1: Add slice fields to FarmNode
• Phase 2: Create SliceCalculatorService
• Phase 3: Enhance GridService with slice calculation
• Phase 4: Create SliceAllocationService with atomic operations
• Phase 5: Enhance NodeMarketplaceService for slices
• Phase 6: Modify MarketplaceController compute endpoint
• Phase 7: Update marketplace template for slice display

Estimated Implementation: 2-3 days (vs weeks for a complete rewrite)