tfgrid/projectmycelium
11
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

init projectmycelium

Browse Source
This commit is contained in:
mik-tf
2025-09-01 21:37:01 -04:00
commit b41efb0e99
319 changed files with 128160 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

600
docs/dev/design/archive/COMPREHENSIVE_SLICE_PLAN.md Normal file
View File

@@ -0,0 +1,600 @@
# Project Mycelium - Comprehensive Slice Management Plan
## Complete UX Vision & Implementation Strategy
---
## 🌟 **Vision & End Goals**
### **The Big Picture**
Transform the Project Mycelium from a **node-centric** to a **slice-centric** compute marketplace, where users can rent standardized compute units instead of entire nodes. This democratizes access to ThreeFold's decentralized infrastructure by making it more affordable and accessible.
### **Core Value Proposition**
- **For Users**: Rent exactly what you need (1 vCPU, 4GB RAM, 200GB storage) instead of entire nodes
- **For Farmers**: Maximize node utilization by selling multiple slices per node
- **For ThreeFold**: Increase marketplace liquidity and accessibility
### **Key Success Metrics**
1. **Increased marketplace activity** - More rentals due to lower entry barriers
2. **Higher node utilization** - Farmers earn more from partially-used nodes
3. **Better user experience** - Clear, standardized compute units
4. **Reduced complexity** - No more guessing about resource allocation
---
## 🎯 **Complete User Experience Design**
### **🚜 Farmer Journey: From Node to Slices**
#### **Current State (Node-based)**
```
Farmer adds 32 vCPU, 128GB RAM, 2TB storage node
→ Sets price for entire node ($200/month)
→ Node sits empty until someone needs ALL resources
→ Low utilization, missed revenue
```
#### **Target State (Slice-based)**
```
Farmer adds 32 vCPU, 128GB RAM, 2TB storage node
→ System automatically calculates: 32 slices available
(min(32/1, 128/4, 2000/200) = min(32, 32, 10) = 10 slices)
→ Farmer sets price per slice ($25/month per slice)
→ 10 different users can rent individual slices
→ Maximum utilization, predictable revenue
```
#### **Detailed Farmer UX Flow**
**Step 1: Node Addition (Enhanced)**
- Farmer navigates to "Add Node" in dashboard
- Enters ThreeFold Grid node ID
- System fetches real-time capacity from gridproxy API
- **NEW**: System automatically shows "This node can provide X slices"
- **NEW**: Farmer sees potential monthly revenue: "Up to $X/month if all slices rented"
**Step 2: Slice Configuration (New)**
- Farmer sets price per slice (within platform min/max limits)
- **NEW**: Farmer sets node-level SLA commitments:
- Minimum uptime guarantee (e.g., 99.5%)
- Minimum bandwidth guarantee (e.g., 100 Mbps)
- **NEW**: System shows competitive analysis: "Similar nodes in your region charge $X-Y"
- Farmer confirms and publishes slices to marketplace
**Step 3: Slice Management Dashboard (Enhanced)**
- **NEW**: Real-time slice allocation view:
```
Node: farmer-node-001 (Belgium)
├── Slice 1: 🟢 Rented by user@example.com (expires: 2025-02-15)
├── Slice 2: 🟢 Rented by dev@startup.com (expires: 2025-03-01)
├── Slice 3: 🟡 Available
└── Slice 4: 🟡 Available
```
- **NEW**: Revenue tracking: "This month: $150 earned, $100 pending"
- **NEW**: Performance monitoring: "Uptime: 99.8% (above SLA)"
**Step 4: Ongoing Management (Enhanced)**
- Farmer receives notifications when slices are rented/released
- **NEW**: Automatic price suggestions based on demand
- **NEW**: SLA monitoring alerts if uptime/bandwidth drops below commitments
### **👤 User Journey: From Complex to Simple**
#### **Current State (Node-based)**
```
User needs 1 vCPU for small app
→ Must rent entire 32 vCPU node ($200/month)
→ Wastes 31 vCPUs, overpays by 3100%
→ Complex resource planning required
```
#### **Target State (Slice-based)**
```
User needs 1 vCPU for small app
→ Rents 1 slice with exactly 1 vCPU, 4GB RAM, 200GB storage
→ Pays fair price ($25/month)
→ Simple, predictable resources
```
#### **Detailed User UX Flow**
**Step 1: Marketplace Discovery (Enhanced)**
- User visits `/marketplace/compute`
- **NEW**: Clear slice-based interface:
```
🍰 Available Compute Slices
Each slice provides:
• 1 vCPU core
• 4GB RAM
• 200GB SSD storage
[Filter by Location] [Filter by Price] [Filter by SLA]
```
**Step 2: Slice Selection (New)**
- **NEW**: Standardized slice cards showing:
```
📍 Belgium • Node: farmer-node-001
💰 $25/month • 🔄 99.8% uptime • 🌐 150 Mbps
Resources (guaranteed):
🖥️ 1 vCPU 💾 4GB RAM 💿 200GB Storage
[Rent This Slice]
```
- **NEW**: No complex resource calculations needed
- **NEW**: Clear SLA commitments visible upfront
**Step 3: Rental Process (New)**
- User clicks "Rent This Slice"
- **NEW**: Simple rental modal:
```
Rent Compute Slice
Duration: [1 month ▼] [3 months] [6 months] [12 months]
Total cost: $25/month
You'll get:
• Dedicated 1 vCPU, 4GB RAM, 200GB storage
• Access credentials within 5 minutes
• 99.8% uptime SLA guarantee
[Confirm Rental - $25]
```
**Step 4: Instant Access (Enhanced)**
- **NEW**: Atomic allocation prevents double-booking
- **NEW**: Immediate access credentials provided
- **NEW**: Clear resource boundaries (no sharing confusion)
**Step 5: Management Dashboard (Enhanced)**
- **NEW**: Simple slice tracking:
```
My Compute Slices
slice-001 (Belgium)
├── Status: 🟢 Active
├── Resources: 1 vCPU, 4GB RAM, 200GB storage
├── Expires: 2025-02-15
├── Cost: $25/month
└── [Extend Rental] [Access Console]
```
---
## 🏗️ **Technical Architecture & Implementation**
### **System Design Principles**
1. **Leverage Existing Infrastructure**: Build on robust [`GridService`](projectmycelium/src/services/grid.rs:1), [`FarmNode`](projectmycelium/src/models/user.rs:164), [`NodeMarketplaceService`](projectmycelium/src/services/node_marketplace.rs:1)
2. **Atomic Operations**: Prevent slice double-booking with file locking
3. **Clear Separation of Concerns**:
- **Node-level**: SLA characteristics (uptime, bandwidth, location)
- **Slice-level**: Compute resources (1 vCPU, 4GB RAM, 200GB storage)
4. **Backward Compatibility**: Existing node rentals continue working
### **Core Algorithm: Slice Calculation**
```rust
// The heart of the system: How many slices can a node provide?
fn calculate_max_slices(node: &FarmNode) -> i32 {
let available_cpu = node.capacity.cpu_cores - node.used_capacity.cpu_cores;
let available_ram = node.capacity.memory_gb - node.used_capacity.memory_gb;
let available_storage = node.capacity.storage_gb - node.used_capacity.storage_gb;
// Each slice needs: 1 vCPU, 4GB RAM, 200GB storage
let max_by_cpu = available_cpu / 1;
let max_by_ram = available_ram / 4;
let max_by_storage = available_storage / 200;
// Bottleneck determines maximum slices
std::cmp::min(std::cmp::min(max_by_cpu, max_by_ram), max_by_storage)
}
```
**Example Calculations**:
- **High-CPU node**: 32 vCPU, 64GB RAM, 1TB storage → `min(32, 16, 5) = 5 slices`
- **Balanced node**: 16 vCPU, 64GB RAM, 4TB storage → `min(16, 16, 20) = 16 slices`
- **Storage-heavy node**: 8 vCPU, 32GB RAM, 10TB storage → `min(8, 8, 50) = 8 slices`
### **Data Model Enhancement**
#### **Enhanced FarmNode Structure**
```rust
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct FarmNode {
// ... ALL existing fields remain unchanged ...
// NEW: Slice management
pub calculated_slices: Vec<CalculatedSlice>,
pub allocated_slices: Vec<AllocatedSlice>,
pub slice_price_per_hour: Decimal,
// NEW: Node-level SLA (inherited by all slices)
pub min_uptime_sla: f32, // e.g., 99.5%
pub min_bandwidth_mbps: i32, // e.g., 100 Mbps
pub price_locked: bool, // Prevent farmer price manipulation
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CalculatedSlice {
pub id: String, // e.g., "node123_slice_1"
pub cpu_cores: i32, // Always 1
pub memory_gb: i32, // Always 4
pub storage_gb: i32, // Always 200
pub status: SliceStatus, // Available/Reserved/Allocated
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AllocatedSlice {
pub slice_id: String,
pub renter_email: String,
pub rental_start: DateTime<Utc>,
pub rental_end: Option<DateTime<Utc>>,
pub monthly_cost: Decimal,
}
```
### **Service Architecture**
#### **1. SliceCalculatorService** (New)
- **Purpose**: Calculate available slices from node capacity
- **Key Method**: `calculate_available_slices(node: &FarmNode) -> Vec<CalculatedSlice>`
- **Algorithm**: Uses min(CPU/1, RAM/4, Storage/200) formula
#### **2. SliceAllocationService** (New)
- **Purpose**: Atomic slice rental with conflict prevention
- **Key Method**: `rent_slice_atomic(node_id, slice_id, user_email, duration)`
- **Features**: File locking, double-booking prevention, automatic billing
#### **3. Enhanced GridService**
- **New Method**: `fetch_node_with_slices(node_id) -> FarmNode`
- **Enhancement**: Automatically calculates slices when fetching from gridproxy
#### **4. Enhanced NodeMarketplaceService**
- **New Method**: `get_all_available_slices() -> Vec<SliceMarketplaceInfo>`
- **Enhancement**: Scans all farmer data, returns available slices instead of nodes
### **Marketplace Integration**
#### **Enhanced `/marketplace/compute` Endpoint**
```rust
// Transform from node-centric to slice-centric
pub async fn compute_resources() -> Result<impl Responder> {
// Get all available slices from all farmers
let available_slices = node_marketplace_service.get_all_available_slices();
// Apply filters (location, price, SLA requirements)
let filtered_slices = apply_slice_filters(&available_slices, &query);
// Paginate and format for display
let slice_products = format_slices_for_display(&filtered_slices);
ctx.insert("slices", &slice_products);
render_template(&tmpl, "marketplace/compute.html", &ctx)
}
```
#### **New `/api/marketplace/rent-slice` Endpoint**
```rust
pub async fn rent_slice(request: web::Json<RentSliceRequest>) -> Result<impl Responder> {
match SliceAllocationService::rent_slice_atomic(
&request.node_id,
&request.slice_id,
&user_email,
request.duration_months,
) {
Ok(_) => Ok(HttpResponse::Ok().json({
"success": true,
"message": "Slice rented successfully",
"access_info": "Credentials will be provided within 5 minutes"
})),
Err(e) => Ok(HttpResponse::BadRequest().json({
"success": false,
"message": e
}))
}
}
```
---
## 🎨 **Frontend User Interface Design**
### **Marketplace Display (Enhanced)**
#### **Before: Node-centric Display**
```
Available Compute Nodes
Node: farmer-node-001 (Belgium)
Resources: 32 vCPU, 128GB RAM, 2TB storage
Price: $200/month
[Rent Entire Node]
```
#### **After: Slice-centric Display**
```html
🍰 Available Compute Slices
<div class="slice-marketplace">
<div class="slice-info-banner">
<h4>What's a slice?</h4>
<p>Each slice provides exactly <strong>1 vCPU, 4GB RAM, 200GB storage</strong> - perfect for small to medium applications.</p>
</div>
<div class="table-responsive">
<table class="table table-striped">
<thead>
<tr>
<th>Location</th>
<th>Node</th>
<th>Resources</th>
<th>SLA Guarantee</th>
<th>Price</th>
<th>Action</th>
</tr>
</thead>
<tbody>
{{#each slices}}
<tr>
<td>
<span class="location-badge">📍 {{node_info.location}}</span>
</td>
<td>{{node_info.name}}</td>
<td>
<div class="resource-badges">
<span class="badge badge-primary">🖥️ 1 vCPU</span>
<span class="badge badge-primary">💾 4GB RAM</span>
<span class="badge badge-primary">💿 200GB SSD</span>
</div>
</td>
<td>
<div class="sla-info">
<div>🔄 {{node_info.uptime_sla}}% uptime</div>
<div>🌐 {{node_info.bandwidth_sla}} Mbps</div>
</div>
</td>
<td>
<div class="price-info">
<strong>{{formatted_price}}</strong>
<small>/month</small>
</div>
</td>
<td>
<button class="btn btn-success btn-sm rent-slice-btn"
data-slice-id="{{slice.id}}"
data-node-id="{{node_info.id}}"
data-price="{{node_info.price_per_hour}}">
Rent Slice
</button>
</td>
</tr>
{{/each}}
</tbody>
</table>
</div>
</div>
```
### **Rental Modal (New)**
```html
<div class="modal fade" id="rentSliceModal">
<div class="modal-dialog">
<div class="modal-content">
<div class="modal-header">
<h5>🍰 Rent Compute Slice</h5>
</div>
<div class="modal-body">
<div class="slice-summary">
<h6>You're renting:</h6>
<ul>
<li>🖥️ 1 dedicated vCPU core</li>
<li>💾 4GB dedicated RAM</li>
<li>💿 200GB SSD storage</li>
<li>📍 Located in <span id="slice-location"></span></li>
<li>🔄 <span id="slice-uptime"></span>% uptime guarantee</li>
</ul>
</div>
<div class="rental-options">
<label>Rental Duration:</label>
<select id="rental-duration" class="form-control">
<option value="1">1 month - $<span class="monthly-price"></span></option>
<option value="3">3 months - $<span class="quarterly-price"></span> (5% discount)</option>
<option value="6">6 months - $<span class="biannual-price"></span> (10% discount)</option>
<option value="12">12 months - $<span class="annual-price"></span> (15% discount)</option>
</select>
</div>
<div class="access-info">
<h6>Access Information:</h6>
<p>After rental confirmation, you'll receive:</p>
<ul>
<li>SSH access credentials</li>
<li>IP address and connection details</li>
<li>Setup instructions</li>
</ul>
<p><small>⏱️ Access typically provided within 5 minutes</small></p>
</div>
</div>
<div class="modal-footer">
<button type="button" class="btn btn-secondary" data-dismiss="modal">Cancel</button>
<button type="button" class="btn btn-success" id="confirm-rental">
Confirm Rental - $<span id="total-cost"></span>
</button>
</div>
</div>
</div>
</div>
```
### **Dashboard Enhancements**
#### **Farmer Dashboard: Slice Management**
```html
<div class="farmer-slice-dashboard">
<h3>🚜 My Nodes & Slices</h3>
{{#each nodes}}
<div class="node-card">
<div class="node-header">
<h5>{{name}} ({{location}})</h5>
<span class="utilization-badge">
{{allocated_slices.length}}/{{calculated_slices.length}} slices rented
</span>
</div>
<div class="slice-grid">
{{#each calculated_slices}}
<div class="slice-item {{#if (eq status 'Allocated')}}allocated{{else}}available{{/if}}">
<div class="slice-id">Slice {{@index}}</div>
<div class="slice-status">
{{#if (eq status 'Allocated')}}
🟢 Rented
<small>by {{renter_email}}</small>
{{else}}
🟡 Available
{{/if}}
</div>
</div>
{{/each}}
</div>
<div class="node-stats">
<div class="stat">
<strong>${{monthly_revenue}}</strong>
<small>Monthly Revenue</small>
</div>
<div class="stat">
<strong>{{utilization_percentage}}%</strong>
<small>Utilization</small>
</div>
<div class="stat">
<strong>{{uptime_percentage}}%</strong>
<small>Uptime</small>
</div>
</div>
</div>
{{/each}}
</div>
```
#### **User Dashboard: My Slices**
```html
<div class="user-slice-dashboard">
<h3>👤 My Compute Slices</h3>
{{#each rented_slices}}
<div class="slice-rental-card">
<div class="slice-header">
<h5>{{slice_id}}</h5>
<span class="status-badge status-{{status}}">{{status}}</span>
</div>
<div class="slice-details">
<div class="detail-row">
<span>📍 Location:</span>
<span>{{node_location}}</span>
</div>
<div class="detail-row">
<span>🖥️ Resources:</span>
<span>1 vCPU, 4GB RAM, 200GB storage</span>
</div>
<div class="detail-row">
<span>💰 Cost:</span>
<span>${{monthly_cost}}/month</span>
</div>
<div class="detail-row">
<span>📅 Expires:</span>
<span>{{rental_end}}</span>
</div>
</div>
<div class="slice-actions">
<button class="btn btn-primary btn-sm" onclick="accessSlice('{{slice_id}}')">
Access Console
</button>
<button class="btn btn-success btn-sm" onclick="extendRental('{{slice_id}}')">
Extend Rental
</button>
</div>
</div>
{{/each}}
</div>
```
---
## 🔄 **Complete Implementation Workflow**
### **Phase 1: Foundation (Day 1)**
1. **Enhance FarmNode model** - Add slice fields to existing [`FarmNode`](projectmycelium/src/models/user.rs:164)
2. **Create SliceCalculatorService** - Implement slice calculation algorithm
3. **Test slice calculation** - Verify algorithm works with existing node data
### **Phase 2: Core Services (Day 2)**
1. **Enhance GridService** - Add slice calculation to node fetching
2. **Create SliceAllocationService** - Implement atomic rental with file locking
3. **Test allocation logic** - Verify no double-booking possible
### **Phase 3: Marketplace Integration (Day 3)**
1. **Enhance NodeMarketplaceService** - Add slice marketplace methods
2. **Modify MarketplaceController** - Update compute endpoint for slices
3. **Add rental endpoint** - Implement `/api/marketplace/rent-slice`
### **Phase 4: Frontend Enhancement (Day 4)**
1. **Update marketplace template** - Replace node display with slice display
2. **Add rental modal** - Implement slice rental interface
3. **Enhance dashboards** - Add slice management for farmers and users
### **Phase 5: Testing & Polish (Day 5)**
1. **End-to-end testing** - Full farmer and user journeys
2. **Performance optimization** - Ensure slice calculations are fast
3. **Documentation** - Update user guides and API docs
---
## 🎯 **Success Criteria & Validation**
### **Technical Validation**
- [ ] Slice calculation algorithm works correctly for all node types
- [ ] Atomic allocation prevents double-booking under concurrent load
- [ ] Marketplace displays slices instead of nodes
- [ ] Rental process completes in under 30 seconds
- [ ] Dashboard shows real-time slice allocation status
### **User Experience Validation**
- [ ] New users can understand slice concept without explanation
- [ ] Farmers can see potential revenue increase from slice model
- [ ] Rental process requires no technical knowledge
- [ ] Users receive access credentials within 5 minutes
- [ ] Both farmers and users prefer slice model over node model
### **Business Impact Validation**
- [ ] Increased number of rentals (lower barrier to entry)
- [ ] Higher average node utilization (multiple slices per node)
- [ ] Reduced support tickets (clearer resource allocation)
- [ ] Positive farmer feedback (increased revenue potential)
---
## 🚀 **Long-term Vision**
### **Phase 2 Enhancements (Future)**
1. **Auto-scaling slices** - Automatically adjust slice resources based on usage
2. **Slice networking** - Allow users to connect multiple slices
3. **Slice templates** - Pre-configured slices for specific use cases (web hosting, databases, etc.)
4. **Advanced SLA monitoring** - Real-time uptime and performance tracking
5. **Slice marketplace analytics** - Demand forecasting and pricing optimization
### **Integration Opportunities**
1. **ThreeFold Grid integration** - Direct deployment to rented slices
2. **Kubernetes support** - Deploy containerized applications to slices
3. **Monitoring integration** - Built-in monitoring and alerting for slices
4. **Backup services** - Automated backup and restore for slice data
This comprehensive plan transforms the Project Mycelium into a modern, accessible, slice-based compute platform while leveraging all existing infrastructure and maintaining backward compatibility.