projectmycelium/docs/dev/design/archive/ThreeFold_Slice_Management_Implementation_Plan.md

Project Mycelium Slice Management Implementation Plan

Overview

This document outlines the implementation plan for the new slice management system in the Project Mycelium. The system replaces the current manual slice configuration with an automatic slice calculation system based on node capacity fetched from gridproxy.

Core Requirements

Slice Definition

• Base Slice Unit: 1 vCPU, 4GB RAM, 200GB storage
• Automatic Calculation: System calculates maximum slices from node capacity
• Dynamic Combinations: All possible slice combinations are available in marketplace
• Real-time Availability: Slice availability updates when rented/released

Key Principles

• No Manual Slice Configuration: Farmers cannot create custom slices
• No Full Node Rental: Only slice-based rentals are supported
• Grid-based Resource Data: All capacity data comes from gridproxy API
• Slice-only Marketplace: Focus exclusively on the new slice system
• Node-Slice Inheritance: Slices inherit node characteristics (uptime, bandwidth, location) while having their own resource specs (CPU, RAM, storage)

Architecture Overview

graph TD
    A[Farmer Adds Node ID] --> B[GridService Fetches Real Data]
    B --> C[SliceCalculatorService]
    C --> D[Calculate Base Slices from Capacity]
    D --> E[Generate All Slice Combinations]
    E --> F[Create Marketplace Products]
    F --> G[Update User Data]
    
    H[User Rents Slice] --> I[Update Node Allocation]
    I --> J[Recalculate Available Combinations]
    J --> K[Update Marketplace Availability]
    
    L[Marketplace Browse] --> M[Real-time Availability Check]
    M --> N[Display Current Slice Options]

Slice Calculation Logic

We present an example of the logic on how to set up a node with slices.

Base Calculation

Node Capacity: 4 vCPU, 16GB RAM, 800GB Storage
Base Slice: 1 vCPU, 4GB RAM, 200GB Storage
Maximum Base Slices: min(4/1, 16/4, 800/200) = min(4, 4, 4) = 4 slices

Available Combinations

From 4 base slices, the marketplace shows:

• 1x Large: 4 vCPU, 16GB RAM, 800GB Storage (uses 4 base slices)
• 2x Medium: 2 vCPU, 8GB RAM, 400GB Storage (uses 2 base slices each)
• 4x Small: 1 vCPU, 4GB RAM, 200GB Storage (uses 1 base slice each)

All slices inherit node characteristics:

• Uptime: 99.8% (from parent node)
• Bandwidth: 1000 Mbps (shared from parent node)
• Location: Belgium (from parent node)
• Certification: Certified (from parent node)

Dynamic Availability

After user rents 1x Medium (2 vCPU, 8GB RAM, 400GB Storage):

• Remaining: 2 base slices available
• New Options:
  • 1x Medium: 2 vCPU, 8GB RAM, 400GB Storage (still inherits 99.8% uptime, 1000 Mbps bandwidth)
  • 2x Small: 1 vCPU, 4GB RAM, 200GB Storage (still inherits 99.8% uptime, 1000 Mbps bandwidth)

Implementation Components

1. SliceCalculatorService

/// Core service for slice calculations following builder pattern
pub struct SliceCalculatorService {
    base_slice: SliceUnit,
    pricing_calculator: PricingCalculator,
}

/// Base slice unit definition
pub struct SliceUnit {
    pub cpu_cores: u32,     // 1
    pub memory_gb: u32,     // 4  
    pub storage_gb: u32,    // 200
}

/// Calculated slice combination
pub struct SliceCombination {
    pub id: String,
    pub multiplier: u32,           // How many base slices this uses
    pub cpu_cores: u32,            // Slice-specific resource
    pub memory_gb: u32,            // Slice-specific resource
    pub storage_gb: u32,           // Slice-specific resource
    pub quantity_available: u32,   // How many of this combination available
    pub price_per_hour: Decimal,
    pub base_slices_required: u32,
    
    // Inherited from parent node
    pub node_uptime_percentage: f64,
    pub node_bandwidth_mbps: u32,
    pub node_location: String,
    pub node_certification_type: String,
}

impl SliceCalculatorService {
    /// Calculate maximum base slices from node capacity
    pub fn calculate_max_base_slices(&self, capacity: &NodeCapacity) -> u32;
    
    /// Generate all possible slice combinations
    pub fn generate_slice_combinations(&self, max_base_slices: u32, allocated_slices: u32) -> Vec<SliceCombination>;
    
    /// Update availability after rental
    pub fn update_availability_after_rental(&self, node: &mut FarmNode, rented_combination: &SliceCombination) -> Result<(), String>;
}

2. Enhanced Data Models

/// Enhanced FarmNode with slice allocation tracking
pub struct FarmNode {
    // Existing fields...
    pub id: String,
    pub name: String,
    pub location: String,
    pub status: NodeStatus,
    pub capacity: NodeCapacity,
    pub grid_node_id: Option<u32>,
    pub grid_data: Option<GridNodeData>,
    
    // Node characteristics that slices inherit
    pub uptime_percentage: f64,
    pub bandwidth_mbps: u32,
    pub certification_type: String,
    pub farming_policy_id: u32,
    
    // NEW: Slice management fields
    pub total_base_slices: u32,
    pub allocated_base_slices: u32,
    pub slice_allocations: Vec<SliceAllocation>,
    pub available_combinations: Vec<SliceCombination>,
    pub slice_pricing: SlicePricing,
}

/// Track individual slice rentals
pub struct SliceAllocation {
    pub allocation_id: String,
    pub slice_combination_id: String,
    pub renter_email: String,
    pub base_slices_used: u32,
    pub rental_start: DateTime<Utc>,
    pub rental_end: Option<DateTime<Utc>>,
    pub status: AllocationStatus,
}

/// Pricing configuration for node slices
pub struct SlicePricing {
    pub base_price_per_hour: Decimal,  // Price for 1 base slice per hour
    pub currency: String,
    pub pricing_multiplier: Decimal,   // Farmer can adjust pricing (0.5x - 2.0x)
}

pub enum AllocationStatus {
    Active,
    Expired,
    Cancelled,
}

3. Updated UserPersistentData

/// Remove old slice system, focus on new slice allocations
pub struct UserPersistentData {
    // Existing fields...
    pub user_email: String,
    pub nodes: Vec<FarmNode>,
    
    // REMOVED: slice_products field (old system)
    // NEW: Slice rental tracking
    pub slice_rentals: Vec<SliceRental>,
}

pub struct SliceRental {
    pub rental_id: String,
    pub node_id: String,
    pub farmer_email: String,
    pub slice_allocation: SliceAllocation,
    pub total_cost: Decimal,
    pub payment_status: PaymentStatus,
}

4. Grid Integration Enhancement

impl FarmerService {
    /// Enhanced node addition with automatic slice calculation
    pub async fn add_node_from_grid(&self, user_email: &str, grid_node_id: u32) -> Result<FarmNode, String> {
        // 1. Fetch real node data from gridproxy
        let grid_data = self.grid_service.fetch_node_data(grid_node_id).await?;
        
        // 2. Calculate slice capacity
        let slice_calculator = SliceCalculatorService::builder().build()?;
        let max_base_slices = slice_calculator.calculate_max_base_slices(&grid_data.capacity);
        
        // 3. Generate initial slice combinations
        let available_combinations = slice_calculator.generate_slice_combinations(max_base_slices, 0);
        
        // 4. Create node with slice data
        let node = FarmNode {
            // ... basic node data from grid_data ...
            total_base_slices: max_base_slices,
            allocated_base_slices: 0,
            slice_allocations: Vec::new(),
            available_combinations,
            slice_pricing: SlicePricing::default(),
        };
        
        // 5. Save and create marketplace products
        self.save_node_and_create_products(user_email, node).await
    }
}

5. Marketplace Integration

impl NodeMarketplaceService {
    /// Get all available slice products from all farmer nodes
    pub fn get_available_slice_products(&self) -> Vec<Product> {
        let mut slice_products = Vec::new();
        
        // Iterate through all farmer nodes
        for user_data in self.get_all_user_data() {
            for node in &user_data.nodes {
                // Convert each available slice combination to marketplace product
                for combination in &node.available_combinations {
                    if combination.quantity_available > 0 {
                        let product = self.create_slice_product(node, combination, &user_data.user_email);
                        slice_products.push(product);
                    }
                }
            }
        }
        
        slice_products
    }
    
    /// Create marketplace product from slice combination
    fn create_slice_product(&self, node: &FarmNode, combination: &SliceCombination, farmer_email: &str) -> Product {
        let mut attributes = HashMap::new();
        
        // Slice-specific attributes
        attributes.insert("cpu_cores".to_string(), ProductAttribute {
            key: "cpu_cores".to_string(),
            value: serde_json::Value::Number(combination.cpu_cores.into()),
            attribute_type: AttributeType::Number,
            is_searchable: true,
            is_filterable: true,
            display_order: Some(1),
        });
        
        attributes.insert("memory_gb".to_string(), ProductAttribute {
            key: "memory_gb".to_string(),
            value: serde_json::Value::Number(combination.memory_gb.into()),
            attribute_type: AttributeType::Number,
            is_searchable: true,
            is_filterable: true,
            display_order: Some(2),
        });
        
        attributes.insert("storage_gb".to_string(), ProductAttribute {
            key: "storage_gb".to_string(),
            value: serde_json::Value::Number(combination.storage_gb.into()),
            attribute_type: AttributeType::Number,
            is_searchable: true,
            is_filterable: true,
            display_order: Some(3),
        });
        
        // Inherited node characteristics
        attributes.insert("uptime_percentage".to_string(), ProductAttribute {
            key: "uptime_percentage".to_string(),
            value: serde_json::Value::Number(serde_json::Number::from_f64(node.uptime_percentage).unwrap()),
            attribute_type: AttributeType::Number,
            is_searchable: true,
            is_filterable: true,
            display_order: Some(4),
        });
        
        attributes.insert("bandwidth_mbps".to_string(), ProductAttribute {
            key: "bandwidth_mbps".to_string(),
            value: serde_json::Value::Number(node.bandwidth_mbps.into()),
            attribute_type: AttributeType::Number,
            is_searchable: true,
            is_filterable: true,
            display_order: Some(5),
        });
        
        attributes.insert("location".to_string(), ProductAttribute {
            key: "location".to_string(),
            value: serde_json::Value::String(node.location.clone()),
            attribute_type: AttributeType::Text,
            is_searchable: true,
            is_filterable: true,
            display_order: Some(6),
        });
        
        attributes.insert("certification_type".to_string(), ProductAttribute {
            key: "certification_type".to_string(),
            value: serde_json::Value::String(node.certification_type.clone()),
            attribute_type: AttributeType::Text,
            is_searchable: true,
            is_filterable: true,
            display_order: Some(7),
        });
        
        Product {
            id: format!("slice_{}_{}", node.id, combination.id),
            name: format!("{} vCPU, {}GB RAM, {}GB Storage - {}% Uptime",
                         combination.cpu_cores, combination.memory_gb, combination.storage_gb, node.uptime_percentage),
            category_id: "compute".to_string(),
            description: format!("Slice from node {} in {} - {}% uptime, {} Mbps bandwidth",
                               node.name, node.location, node.uptime_percentage, node.bandwidth_mbps),
            base_price: combination.price_per_hour,
            base_currency: "USD".to_string(),
            attributes,
            provider_id: farmer_email.to_string(),
            provider_name: farmer_email.to_string(),
            availability: if combination.quantity_available > 0 {
                ProductAvailability::Available
            } else {
                ProductAvailability::Unavailable
            },
            metadata: ProductMetadata {
                tags: vec![
                    "compute".to_string(),
                    "slice".to_string(),
                    node.location.clone(),
                    format!("{}vcpu", combination.cpu_cores),
                    format!("{}gb-ram", combination.memory_gb),
                ],
                location: Some(node.location.clone()),
                rating: None,
                review_count: 0,
                custom_fields: HashMap::new(),
            },
            created_at: Utc::now(),
            updated_at: Utc::now(),
        }
    }
}

Current Marketplace Integration Analysis

Existing Product Attributes System

The current marketplace uses a generic ProductAttribute system that needs to be enhanced for slice inheritance:

// Current system supports these attribute types:
pub enum AttributeType {
    Text,
    Number,
    SliceConfiguration,  // Already exists!
    Boolean,
    Select(Vec<String>),
    // ... others
}

Required Marketplace Refactoring

1. Product Search and Filtering

Current: Users can filter by category, location, price range Enhanced: Add slice-specific filters with inherited characteristics

// Enhanced search criteria for slices
pub struct SliceSearchCriteria {
    // Slice-specific filters
    pub min_cpu_cores: Option<u32>,
    pub max_cpu_cores: Option<u32>,
    pub min_memory_gb: Option<u32>,
    pub max_memory_gb: Option<u32>,
    pub min_storage_gb: Option<u32>,
    pub max_storage_gb: Option<u32>,
    
    // Inherited node characteristics filters
    pub min_uptime_percentage: Option<f64>,
    pub min_bandwidth_mbps: Option<u32>,
    pub locations: Option<Vec<String>>,
    pub certification_types: Option<Vec<String>>,
    
    // Existing filters
    pub price_range: Option<(Decimal, Decimal)>,
    pub availability: Option<ProductAvailability>,
}

2. Marketplace Controller Updates

Current: MarketplaceController::compute_resources() shows generic compute products Enhanced: Show slice combinations with inherited characteristics

impl MarketplaceController {
    pub async fn compute_resources(/* ... */) -> Result<impl Responder> {
        // Get slice products instead of generic compute products
        let node_marketplace_service = NodeMarketplaceService::builder().build()?;
        let slice_products = node_marketplace_service.get_available_slice_products();
        
        // Apply slice-specific filtering
        let filtered_slices = self.apply_slice_filters(&slice_products, &query);
        
        // Group slices by node characteristics for better UX
        let grouped_slices = self.group_slices_by_node_characteristics(&filtered_slices);
        
        ctx.insert("slice_products", &grouped_slices);
        ctx.insert("slice_filters", &self.get_slice_filter_options());
    }
}

3. Template Updates

Current: Generic product display templates Enhanced: Slice-specific templates showing inheritance

<!-- Enhanced slice product card -->
<div class="slice-product-card">
    <h3>{{cpu_cores}} vCPU, {{memory_gb}}GB RAM, {{storage_gb}}GB Storage</h3>
    
    <!-- Slice-specific specs -->
    <div class="slice-specs">
        <span class="spec">{{cpu_cores}} vCPU</span>
        <span class="spec">{{memory_gb}}GB RAM</span>
        <span class="spec">{{storage_gb}}GB Storage</span>
    </div>
    
    <!-- Inherited node characteristics -->
    <div class="node-characteristics">
        <span class="uptime">{{uptime_percentage}}% Uptime</span>
        <span class="bandwidth">{{bandwidth_mbps}} Mbps</span>
        <span class="location">{{location}}</span>
        <span class="certification">{{certification_type}}</span>
    </div>
    
    <div class="pricing">
        {{formatted_price}}/hour
    </div>
</div>

4. Enhanced Phase 4 Implementation

Updated Phase 4 tasks to include inheritance-aware marketplace integration:

• Update NodeMarketplaceService for slice products with inheritance
• Modify marketplace controllers to show slice + node characteristics
• Update compute resources page templates with inheritance display
• Add slice-specific filtering (CPU, RAM, storage) + inherited filters (uptime, bandwidth, location)
• Real-time availability updates with inheritance preservation
• Group slices by node characteristics for better user experience

Implementation Phases

Phase 1: Core Slice Calculator (Week 1)

• Create SliceCalculatorService with builder pattern
• Implement base slice calculation algorithm
• Implement slice combination generation
• Add comprehensive unit tests
• Integration with existing GridService

Phase 2: Enhanced Data Models (Week 2)

• Update FarmNode structure with slice fields
• Remove old slice_products from UserPersistentData
• Add new slice allocation tracking
• Update JSON serialization/deserialization
• Database migration scripts

Phase 3: Grid Integration (Week 3)

• Enhance FarmerService::add_node() to use gridproxy
• Integrate SliceCalculatorService into node addition
• Remove manual slice configuration options
• Update farmer dashboard to show calculated slices
• Add node validation and error handling

Phase 4: Marketplace Integration (Week 4)

• Update NodeMarketplaceService for slice products
• Modify marketplace controllers for slice display
• Update compute resources page templates
• Add slice filtering and search capabilities
• Real-time availability updates

Phase 5: Rental System (Week 5)

• Implement slice rental functionality
• Add slice allocation tracking
• Real-time availability recalculation
• Rental management and tracking
• Payment integration for slice rentals

Phase 6: Testing & Optimization (Week 6)

• End-to-end testing of slice system
• Performance optimization for large node counts
• User interface testing and refinement
• Documentation and farmer onboarding
• Production deployment preparation

Technical Specifications

Slice Calculation Algorithm

fn calculate_max_base_slices(capacity: &NodeCapacity) -> u32 {
    let cpu_slices = capacity.cpu_cores / BASE_SLICE_CPU;
    let memory_slices = capacity.memory_gb / BASE_SLICE_MEMORY;
    let storage_slices = capacity.storage_gb / BASE_SLICE_STORAGE;
    
    // Return the limiting factor
    std::cmp::min(std::cmp::min(cpu_slices, memory_slices), storage_slices)
}

fn generate_combinations(max_base_slices: u32) -> Vec<SliceCombination> {
    let mut combinations = Vec::new();
    
    // Generate all divisors of max_base_slices
    for multiplier in 1..=max_base_slices {
        if max_base_slices % multiplier == 0 {
            let quantity = max_base_slices / multiplier;
            combinations.push(SliceCombination {
                multiplier,
                cpu_cores: BASE_SLICE_CPU * multiplier,
                memory_gb: BASE_SLICE_MEMORY * multiplier,
                storage_gb: BASE_SLICE_STORAGE * multiplier,
                quantity_available: quantity,
                base_slices_required: multiplier,
                // ... other fields
            });
        }
    }
    
    combinations
}

Real-time Availability Updates

impl SliceCalculatorService {
    pub fn update_availability_after_rental(&self, node: &mut FarmNode, rented_slices: u32) -> Result<(), String> {
        // Update allocated count
        node.allocated_base_slices += rented_slices;
        
        // Recalculate available combinations
        let available_base_slices = node.total_base_slices - node.allocated_base_slices;
        node.available_combinations = self.generate_slice_combinations(available_base_slices, 0);
        
        Ok(())
    }
}

Pricing Strategy

• Base Price: Farmer sets price per base slice per hour
• Linear Scaling: Larger combinations cost proportionally more
• Market Dynamics: Farmers can adjust pricing multiplier (0.5x - 2.0x)
• Location Premium: Optional location-based pricing adjustments

Migration Strategy

Complete System Replacement

• Remove old slice system entirely - no backward compatibility needed
• Convert existing nodes to use new slice calculation
• Migrate active rentals to new slice allocation system
• Update all farmer dashboards to show new slice system only

Data Migration Steps

1. Backup existing data before migration
2. Calculate slices for existing nodes using gridproxy data
3. Convert active slice rentals to new allocation format
4. Remove old slice_products fields from user data
5. Update marketplace products to use new slice system

Success Metrics

Technical Metrics

• Slice Calculation Performance: < 100ms per node
• Marketplace Load Time: < 2s for slice product listing
• Real-time Updates: < 500ms availability refresh
• System Reliability: 99.9% uptime for slice calculations

Business Metrics

• Farmer Adoption: 90% of farmers successfully add nodes
• User Experience: Intuitive slice selection interface
• Resource Utilization: Improved node capacity utilization
• Marketplace Activity: Increased slice rental transactions

Risk Mitigation

Technical Risks

• GridProxy Dependency: Implement fallback mechanisms for API failures
• Performance Scaling: Optimize for large numbers of nodes and combinations
• Data Consistency: Ensure slice allocation accuracy across concurrent rentals

Business Risks

• Farmer Resistance: Clear communication about benefits of new system
• User Confusion: Intuitive UI design for slice selection
• Migration Issues: Thorough testing and gradual rollout

Conclusion

This implementation plan provides a comprehensive roadmap for replacing the current manual slice system with an automatic, grid-based slice calculation system. The new architecture ensures:

• Automatic slice calculation from real node capacity
• Dynamic marketplace products with real-time availability
• Simplified farmer experience with no manual configuration
• Improved resource utilization through standardized slicing
• Scalable architecture following existing patterns

The phased implementation approach minimizes risk while delivering a robust, user-friendly slice management system that aligns with ThreeFold's vision of decentralized compute resources.