469 lines
18 KiB
Markdown
469 lines
18 KiB
Markdown
# Project Mycelium - Slice Management Enhancement Plan
|
|
## Focused Enhancement of Existing Architecture
|
|
|
|
---
|
|
|
|
## 🎯 **Project Overview**
|
|
|
|
**Goal**: Add dynamic slice management to the existing Project Mycelium by enhancing current architecture rather than rebuilding it.
|
|
|
|
**Key Principle**: A slice is the minimum unit: **1 vCPU, 4GB RAM, 200GB storage**
|
|
|
|
**Enhancement Strategy**: Leverage existing [`GridService`](projectmycelium/src/services/grid.rs:1), [`FarmNode`](projectmycelium/src/models/user.rs:164), and [`NodeMarketplaceService`](projectmycelium/src/services/node_marketplace.rs:1) to add slice functionality.
|
|
|
|
---
|
|
|
|
## 🏗️ **Current Architecture Analysis**
|
|
|
|
### **✅ What Already Exists and Works:**
|
|
|
|
1. **[`GridService`](projectmycelium/src/services/grid.rs:1)** - Robust gridproxy integration
|
|
- Fetches node data from ThreeFold gridproxy API
|
|
- Converts to internal [`GridNodeData`](projectmycelium/src/models/user.rs:788) format
|
|
- Has mock data fallback for testing
|
|
- Uses builder pattern consistently
|
|
|
|
2. **[`FarmNode`](projectmycelium/src/models/user.rs:164)** - Comprehensive node model
|
|
- Has `capacity: NodeCapacity` (total resources)
|
|
- Has `used_capacity: NodeCapacity` (currently used)
|
|
- Has `grid_node_id` and `grid_data` for grid integration
|
|
- Has `rental_options` for configuration
|
|
- Already stored in [`./user_data/`](projectmycelium/user_data/user1_at_example_com.json:1) JSON files
|
|
|
|
3. **[`NodeMarketplaceService`](projectmycelium/src/services/node_marketplace.rs:1)** - Node-to-marketplace conversion
|
|
- Scans all farmer data from `./user_data/` directory
|
|
- Converts [`FarmNode`](projectmycelium/src/models/user.rs:164) to [`Product`](projectmycelium/src/models/product.rs:1)
|
|
- Has filtering by location, price, CPU, RAM, storage
|
|
- Calculates capacity statistics
|
|
|
|
4. **[`MarketplaceController`](projectmycelium/src/controllers/marketplace.rs:1)** - Marketplace endpoints
|
|
- `/marketplace/compute` endpoint exists
|
|
- Has pagination and filtering
|
|
- Uses currency conversion
|
|
- Integrates with [`NodeMarketplaceService`](projectmycelium/src/services/node_marketplace.rs:1)
|
|
|
|
### **🔧 What Needs Enhancement:**
|
|
|
|
1. **Slice calculation** - Add automatic slice calculation from node capacity
|
|
2. **Slice allocation** - Add atomic slice rental to prevent conflicts
|
|
3. **Marketplace display** - Show slices instead of full nodes
|
|
4. **Dashboard tracking** - Track slice rentals for farmers and users
|
|
|
|
---
|
|
|
|
## 🔄 **User Experience Flow (Same Goals)**
|
|
|
|
### **Farmer Journey**
|
|
1. **Farmer adds node** → Uses existing node addition, system fetches from gridproxy
|
|
2. **System calculates slices** → NEW: Automatic calculation based on node capacity
|
|
3. **Farmer sets slice price** → NEW: Price per slice within platform limits
|
|
4. **Slices appear in marketplace** → ENHANCED: `/marketplace/compute` shows slices
|
|
5. **Farmer monitors rentals** → ENHANCED: Dashboard shows slice rental income
|
|
|
|
### **User Journey**
|
|
1. **User browses marketplace** → ENHANCED: `/marketplace/compute` shows available slices
|
|
2. **User selects slice** → NEW: Sees slice specs, node SLA, location, price
|
|
3. **User rents slice** → NEW: Atomic allocation prevents conflicts
|
|
4. **User sees rental** → ENHANCED: Dashboard shows slice rentals
|
|
|
|
---
|
|
|
|
## 🛠️ **Implementation Plan**
|
|
|
|
### **Phase 1: Enhance FarmNode Model**
|
|
**File**: [`src/models/user.rs`](projectmycelium/src/models/user.rs:164) (ADD fields to existing struct)
|
|
|
|
```rust
|
|
// ADD these fields to existing FarmNode struct around line 164
|
|
impl FarmNode {
|
|
// ... keep ALL existing fields ...
|
|
|
|
// NEW: Slice management fields
|
|
#[serde(default)]
|
|
pub calculated_slices: Vec<CalculatedSlice>,
|
|
#[serde(default)]
|
|
pub allocated_slices: Vec<AllocatedSlice>,
|
|
#[serde(default = "default_slice_price")]
|
|
pub slice_price_per_hour: rust_decimal::Decimal,
|
|
#[serde(default)]
|
|
pub price_locked: bool,
|
|
#[serde(default = "default_min_uptime")]
|
|
pub min_uptime_sla: f32,
|
|
#[serde(default = "default_min_bandwidth")]
|
|
pub min_bandwidth_mbps: i32,
|
|
}
|
|
|
|
// NEW: Slice structures
|
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
|
pub struct CalculatedSlice {
|
|
pub id: String,
|
|
pub cpu_cores: i32, // Always 1
|
|
pub memory_gb: i32, // Always 4
|
|
pub storage_gb: i32, // Always 200
|
|
pub status: SliceStatus,
|
|
}
|
|
|
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
|
pub enum SliceStatus {
|
|
Available,
|
|
Reserved,
|
|
Allocated,
|
|
}
|
|
|
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
|
pub struct AllocatedSlice {
|
|
pub slice_id: String,
|
|
pub renter_email: String,
|
|
pub rental_start: DateTime<Utc>,
|
|
pub rental_end: Option<DateTime<Utc>>,
|
|
pub monthly_cost: rust_decimal::Decimal,
|
|
}
|
|
```
|
|
|
|
### **Phase 2: Create Slice Calculator Service**
|
|
**File**: `src/services/slice_calculator.rs` (NEW)
|
|
|
|
```rust
|
|
use crate::models::user::{FarmNode, CalculatedSlice, SliceStatus};
|
|
|
|
pub struct SliceCalculatorService;
|
|
|
|
impl SliceCalculatorService {
|
|
pub fn calculate_available_slices(node: &FarmNode) -> Vec<CalculatedSlice> {
|
|
let available_cpu = node.capacity.cpu_cores - node.used_capacity.cpu_cores;
|
|
let available_ram = node.capacity.memory_gb - node.used_capacity.memory_gb;
|
|
let available_storage = node.capacity.storage_gb - node.used_capacity.storage_gb;
|
|
|
|
// Algorithm: Max slices = min(CPU/1, RAM/4, Storage/200)
|
|
let max_slices_by_cpu = available_cpu / 1;
|
|
let max_slices_by_ram = available_ram / 4;
|
|
let max_slices_by_storage = available_storage / 200;
|
|
|
|
let max_slices = std::cmp::min(
|
|
std::cmp::min(max_slices_by_cpu, max_slices_by_ram),
|
|
max_slices_by_storage
|
|
);
|
|
|
|
(0..max_slices)
|
|
.map(|i| CalculatedSlice {
|
|
id: format!("{}_slice_{}", node.id, i + 1),
|
|
cpu_cores: 1,
|
|
memory_gb: 4,
|
|
storage_gb: 200,
|
|
status: SliceStatus::Available,
|
|
})
|
|
.collect()
|
|
}
|
|
}
|
|
```
|
|
|
|
### **Phase 3: Enhance GridService**
|
|
**File**: [`src/services/grid.rs`](projectmycelium/src/services/grid.rs:1) (ADD method to existing)
|
|
|
|
```rust
|
|
// ADD this method to existing GridService implementation
|
|
impl GridService {
|
|
pub async fn fetch_node_with_slices(&self, node_id: u32) -> Result<FarmNode, String> {
|
|
// Use existing fetch_node_data method
|
|
let grid_data = self.fetch_node_data(node_id).await?;
|
|
|
|
// Use existing conversion logic
|
|
let mut farm_node = self.convert_grid_data_to_farm_node(&grid_data)?;
|
|
|
|
// NEW: Calculate slices
|
|
farm_node.calculated_slices = SliceCalculatorService::calculate_available_slices(&farm_node);
|
|
farm_node.allocated_slices = Vec::new();
|
|
farm_node.slice_price_per_hour = rust_decimal::Decimal::from_str("0.50").unwrap();
|
|
|
|
Ok(farm_node)
|
|
}
|
|
}
|
|
```
|
|
|
|
### **Phase 4: Create Slice Allocation Service**
|
|
**File**: `src/services/slice_allocation.rs` (NEW)
|
|
|
|
```rust
|
|
use crate::services::user_persistence::UserPersistence;
|
|
use std::fs::OpenOptions;
|
|
use std::io::{Seek, SeekFrom};
|
|
|
|
pub struct SliceAllocationService;
|
|
|
|
impl SliceAllocationService {
|
|
pub fn rent_slice_atomic(
|
|
node_id: &str,
|
|
slice_id: &str,
|
|
user_email: &str,
|
|
duration_months: u32,
|
|
) -> Result<(), String> {
|
|
// Find farmer who owns the node
|
|
let farmer_email = self.find_node_owner(node_id)?;
|
|
|
|
// Lock farmer's data file
|
|
let file_path = format!("./user_data/{}.json", farmer_email.replace("@", "_at_"));
|
|
let _lock_file = OpenOptions::new()
|
|
.create(true)
|
|
.write(true)
|
|
.open(format!("{}.lock", file_path))
|
|
.map_err(|e| format!("Failed to acquire lock: {}", e))?;
|
|
|
|
// Load farmer data
|
|
let mut farmer_data = UserPersistence::load_user_data(&farmer_email)
|
|
.ok_or("Farmer data not found")?;
|
|
|
|
// Find node and slice
|
|
let node = farmer_data.nodes.iter_mut()
|
|
.find(|n| n.id == node_id)
|
|
.ok_or("Node not found")?;
|
|
|
|
let slice = node.calculated_slices.iter_mut()
|
|
.find(|s| s.id == slice_id)
|
|
.ok_or("Slice not found")?;
|
|
|
|
// Check if still available
|
|
if slice.status != SliceStatus::Available {
|
|
return Err("Slice no longer available".to_string());
|
|
}
|
|
|
|
// Allocate slice
|
|
slice.status = SliceStatus::Allocated;
|
|
let allocation = AllocatedSlice {
|
|
slice_id: slice_id.to_string(),
|
|
renter_email: user_email.to_string(),
|
|
rental_start: chrono::Utc::now(),
|
|
rental_end: Some(chrono::Utc::now() + chrono::Duration::days(duration_months as i64 * 30)),
|
|
monthly_cost: node.slice_price_per_hour * rust_decimal::Decimal::from(24 * 30),
|
|
};
|
|
node.allocated_slices.push(allocation);
|
|
|
|
// Save farmer data
|
|
UserPersistence::save_user_data(&farmer_email, &farmer_data)?;
|
|
|
|
// Add rental to user data
|
|
let mut user_data = UserPersistence::load_user_data(user_email)
|
|
.unwrap_or_default();
|
|
user_data.node_rentals.push(/* create NodeRental */);
|
|
UserPersistence::save_user_data(user_email, &user_data)?;
|
|
|
|
Ok(())
|
|
}
|
|
}
|
|
```
|
|
|
|
### **Phase 5: Enhance NodeMarketplaceService**
|
|
**File**: [`src/services/node_marketplace.rs`](projectmycelium/src/services/node_marketplace.rs:1) (ADD method to existing)
|
|
|
|
```rust
|
|
// ADD this method to existing NodeMarketplaceService implementation
|
|
impl NodeMarketplaceService {
|
|
pub fn get_all_available_slices(&self) -> Vec<SliceMarketplaceInfo> {
|
|
let mut available_slices = Vec::new();
|
|
|
|
// Use existing logic to scan user_data directory
|
|
if let Ok(entries) = std::fs::read_dir("./user_data/") {
|
|
for entry in entries.flatten() {
|
|
if let Some(filename) = entry.file_name().to_str() {
|
|
if filename.ends_with(".json") && !filename.starts_with('.') {
|
|
let farmer_email = filename.replace("_at_", "@").replace(".json", "");
|
|
|
|
if let Some(farmer_data) = UserPersistence::load_user_data(&farmer_email) {
|
|
for node in farmer_data.nodes {
|
|
for slice in node.calculated_slices {
|
|
if slice.status == SliceStatus::Available {
|
|
available_slices.push(SliceMarketplaceInfo {
|
|
slice,
|
|
node_info: NodeInfo {
|
|
id: node.id.clone(),
|
|
name: node.name.clone(),
|
|
location: node.location.clone(),
|
|
uptime_sla: node.min_uptime_sla,
|
|
bandwidth_sla: node.min_bandwidth_mbps,
|
|
price_per_hour: node.slice_price_per_hour,
|
|
},
|
|
farmer_email: farmer_email.clone(),
|
|
});
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
available_slices
|
|
}
|
|
}
|
|
|
|
#[derive(Debug, Clone)]
|
|
pub struct SliceMarketplaceInfo {
|
|
pub slice: CalculatedSlice,
|
|
pub node_info: NodeInfo,
|
|
pub farmer_email: String,
|
|
}
|
|
|
|
#[derive(Debug, Clone)]
|
|
pub struct NodeInfo {
|
|
pub id: String,
|
|
pub name: String,
|
|
pub location: String,
|
|
pub uptime_sla: f32,
|
|
pub bandwidth_sla: i32,
|
|
pub price_per_hour: rust_decimal::Decimal,
|
|
}
|
|
```
|
|
|
|
### **Phase 6: Enhance MarketplaceController**
|
|
**File**: [`src/controllers/marketplace.rs`](projectmycelium/src/controllers/marketplace.rs:1) (MODIFY existing method)
|
|
|
|
```rust
|
|
// ENHANCE existing compute_resources method around line 150
|
|
impl MarketplaceController {
|
|
pub async fn compute_resources(tmpl: web::Data<Tera>, session: Session, query: web::Query<std::collections::HashMap<String, String>>) -> Result<impl Responder> {
|
|
// Keep existing setup code...
|
|
|
|
// REPLACE product service logic with slice marketplace logic
|
|
let node_marketplace_service = NodeMarketplaceService::builder()
|
|
.currency_service(currency_service.clone())
|
|
.build()
|
|
.map_err(|e| actix_web::error::ErrorInternalServerError(e))?;
|
|
|
|
// Get available slices instead of products
|
|
let available_slices = node_marketplace_service.get_all_available_slices();
|
|
|
|
// Apply existing filtering logic to slices
|
|
let filtered_slices = self.apply_slice_filters(&available_slices, &query);
|
|
|
|
// Keep existing pagination and currency conversion logic...
|
|
|
|
ctx.insert("slices", &slice_products);
|
|
render_template(&tmpl, "marketplace/compute.html", &ctx)
|
|
}
|
|
|
|
// NEW: Slice rental endpoint
|
|
pub async fn rent_slice(request: web::Json<RentSliceRequest>, session: Session) -> Result<impl Responder> {
|
|
let user_email = session.get::<String>("user_email")
|
|
.map_err(|_| actix_web::error::ErrorUnauthorized("Not logged in"))?
|
|
.ok_or_else(|| actix_web::error::ErrorUnauthorized("Not logged in"))?;
|
|
|
|
match SliceAllocationService::rent_slice_atomic(
|
|
&request.node_id,
|
|
&request.slice_id,
|
|
&user_email,
|
|
request.duration_months,
|
|
) {
|
|
Ok(_) => Ok(HttpResponse::Ok().json(serde_json::json!({
|
|
"success": true,
|
|
"message": "Slice rented successfully"
|
|
}))),
|
|
Err(e) => Ok(HttpResponse::BadRequest().json(serde_json::json!({
|
|
"success": false,
|
|
"message": e
|
|
}))),
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### **Phase 7: Enhance Frontend Template**
|
|
**File**: [`templates/marketplace/compute.html`](projectmycelium/src/views/marketplace/compute_resources.html:1) (MODIFY existing)
|
|
|
|
```html
|
|
<!-- ENHANCE existing compute.html template -->
|
|
<!-- Replace product display with slice display -->
|
|
|
|
<div class="slice-marketplace">
|
|
<h3>🍰 Available Compute Slices</h3>
|
|
|
|
<div class="table-responsive">
|
|
<table class="table table-striped">
|
|
<thead>
|
|
<tr>
|
|
<th>Node</th>
|
|
<th>Location</th>
|
|
<th>Resources</th>
|
|
<th>SLA</th>
|
|
<th>Price</th>
|
|
<th>Action</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody>
|
|
{{#each slices}}
|
|
<tr>
|
|
<td>{{node_info.name}}</td>
|
|
<td>{{node_info.location}}</td>
|
|
<td>
|
|
<span class="badge badge-secondary">1 vCPU</span>
|
|
<span class="badge badge-secondary">4GB RAM</span>
|
|
<span class="badge badge-secondary">200GB Storage</span>
|
|
</td>
|
|
<td>
|
|
<div>🔄 {{node_info.uptime_sla}}% uptime</div>
|
|
<div>🌐 {{node_info.bandwidth_sla}} Mbps</div>
|
|
</td>
|
|
<td>{{formatted_price}}</td>
|
|
<td>
|
|
<button class="btn btn-success btn-sm rent-slice-btn"
|
|
data-slice-id="{{slice.id}}"
|
|
data-node-id="{{node_info.id}}">
|
|
Rent Slice
|
|
</button>
|
|
</td>
|
|
</tr>
|
|
{{/each}}
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
|
|
<script>
|
|
$('.rent-slice-btn').click(function() {
|
|
const sliceId = $(this).data('slice-id');
|
|
const nodeId = $(this).data('node-id');
|
|
|
|
$.ajax({
|
|
url: '/api/marketplace/rent-slice',
|
|
method: 'POST',
|
|
contentType: 'application/json',
|
|
data: JSON.stringify({
|
|
node_id: nodeId,
|
|
slice_id: sliceId,
|
|
duration_months: 1
|
|
}),
|
|
success: function(response) {
|
|
if (response.success) {
|
|
alert('Slice rented successfully!');
|
|
location.reload();
|
|
} else {
|
|
alert('Error: ' + response.message);
|
|
}
|
|
}
|
|
});
|
|
});
|
|
</script>
|
|
```
|
|
|
|
---
|
|
|
|
## 🎯 **Key Benefits of This Approach**
|
|
|
|
1. **Leverages Existing Work** - Builds on robust [`GridService`](projectmycelium/src/services/grid.rs:1) and [`NodeMarketplaceService`](projectmycelium/src/services/node_marketplace.rs:1)
|
|
2. **Minimal Changes** - Adds fields to existing [`FarmNode`](projectmycelium/src/models/user.rs:164), doesn't rebuild
|
|
3. **Consistent Patterns** - Uses existing builder patterns and JSON persistence
|
|
4. **Same UX Goals** - Achieves all the same user experience objectives
|
|
5. **Incremental Implementation** - Can be implemented phase by phase
|
|
|
|
---
|
|
|
|
## 📋 **Implementation Checklist**
|
|
|
|
- [ ] **Phase 1**: Add slice fields to [`FarmNode`](projectmycelium/src/models/user.rs:164)
|
|
- [ ] **Phase 2**: Create `SliceCalculatorService`
|
|
- [ ] **Phase 3**: Enhance [`GridService`](projectmycelium/src/services/grid.rs:1) with slice calculation
|
|
- [ ] **Phase 4**: Create `SliceAllocationService` with atomic operations
|
|
- [ ] **Phase 5**: Enhance [`NodeMarketplaceService`](projectmycelium/src/services/node_marketplace.rs:1) for slices
|
|
- [ ] **Phase 6**: Modify [`MarketplaceController`](projectmycelium/src/controllers/marketplace.rs:1) compute endpoint
|
|
- [ ] **Phase 7**: Update marketplace template for slice display
|
|
|
|
**Estimated Implementation**: 2-3 days (vs weeks for a complete rewrite) |