init projectmycelium

docs/dev/guides/data-validation-guide.md

@@ -0,0 +1,152 @@
+# Data Validation Guide
+
+## Overview
+
+This guide documents the data validation tools and processes for the Project Mycelium application. These tools help maintain data integrity and resolve JSON parsing errors when user data schemas evolve.
+
+## Problem Context
+
+The application stores user data in JSON files that must match Rust struct definitions for serde deserialization. When schemas change or data becomes corrupted, parsing errors occur that prevent users from accessing dashboard features.
+
+## Validation Tools
+
+### 1. Rust Data Validator (`src/utils/data_validator.rs`)
+
+A comprehensive validation utility that:
+- Validates JSON structure against expected Rust schemas
+- Provides detailed error reporting with line numbers
+- Handles schema migration and repair
+- Supports batch validation of multiple user files
+
+**Usage:**
+```rust
+use crate::utils::data_validator::DataValidator;
+
+let validator = DataValidator::new();
+let result = validator.validate_user_file("user_data/user1_at_example_com.json");
+```
+
+### 2. Python Validation Script (`fix_user_data.py`)
+
+A standalone Python script that:
+- Fixes common schema mismatches automatically
+- Handles missing fields, invalid enum variants, and null values
+- Processes all user data files in batch
+- Provides detailed change reporting
+
+**Usage:**
+```bash
+cd projectmycelium
+python3 scripts/fix_user_data.py
+```
+
+## Common Schema Issues and Solutions
+
+### Missing Fields
+**Problem:** JSON missing required fields like `minimum_deployment_duration`, `preferred_regions`
+**Solution:** Add default values based on field type and context
+
+### Invalid Enum Variants
+**Problem:** Old enum values like `ServiceProgress`, `AppDeployment` 
+**Solution:** Map to current valid variants: `ServiceCreated`, `Deployment`
+
+### Incorrect Object Structure
+**Problem:** Objects not matching expected struct definitions
+**Solution:** Restructure to match current schema requirements
+
+### Example: UsageStatistics Structure
+```json
+{
+  "usage_statistics": {
+    "total_deployments": 85,
+    "active_services": 12,
+    "total_spent": "2450.25",
+    "favorite_categories": ["compute", "storage", "networking"],
+    "usage_trends": [
+      {
+        "period": "month",
+        "metric": "deployments", 
+        "value": 25.5,
+        "change_percentage": 12.3
+      }
+    ],
+    "login_frequency": 4.2,
+    "preferred_regions": ["us-east", "eu-west"],
+    "account_age_days": 150,
+    "last_activity": "2025-06-18T20:30:00Z"
+  }
+}
+```
+
+## Troubleshooting Process
+
+### 1. Identify Parsing Errors
+Look for serde deserialization errors in application logs:
+```
+Failed to parse user data for user@example.com: missing field `field_name` at line X column Y
+```
+
+### 2. Run Validation Tools
+Use the Python script for quick fixes:
+```bash
+python3 scripts/fix_user_data.py
+```
+
+Or use the Rust validator for detailed analysis:
+```rust
+let validator = DataValidator::new();
+validator.validate_all_user_files();
+```
+
+### 3. Manual Schema Updates
+For complex schema changes, manually update JSON structure to match current Rust struct definitions in `src/models/user.rs`.
+
+### 4. Verify Resolution
+Test the application to ensure parsing errors are resolved:
+```bash
+cargo run
+```
+
+## Best Practices
+
+### Schema Evolution
+1. **Update validation tools** when adding new required fields to Rust structs
+2. **Provide migration logic** for existing data
+3. **Test with all user data files** before deploying schema changes
+4. **Document breaking changes** in this guide
+
+### Data Integrity
+1. **Run validation regularly** as part of development workflow
+2. **Backup user data** before applying automated fixes
+3. **Validate after manual edits** to ensure correctness
+4. **Monitor application logs** for new parsing errors
+
+### Tool Maintenance
+1. **Keep validation tools updated** with current schema requirements
+2. **Add new validation rules** for new data structures
+3. **Test validation tools** with various error scenarios
+4. **Document new validation patterns** in this guide
+
+## File Locations
+
+- **Rust Validator:** `src/utils/data_validator.rs`
+- **Python Script:** `scripts/fix_user_data.py`
+- **User Data:** `user_data/*.json`
+- **Schema Definitions:** `src/models/user.rs`
+- **Utils Module:** `src/utils/mod.rs`
+
+## Integration
+
+The validation tools are integrated into the utils module and can be used throughout the application for:
+- Development-time validation
+- Runtime error recovery
+- Data migration during updates
+- Quality assurance testing
+
+## Future Enhancements
+
+Consider adding:
+- Automated validation in CI/CD pipeline
+- Real-time validation during data updates
+- Schema versioning and automatic migration
+- Web interface for data validation and repair

docs/dev/guides/troubleshooting-json-errors.md

@@ -0,0 +1,57 @@
+# Quick Troubleshooting: JSON Parsing Errors
+
+## 🚨 Emergency Fix
+
+If you see JSON parsing errors in logs, run this immediately:
+
+```bash
+cd projectmycelium
+python3 scripts/fix_user_data.py
+cargo run
+```
+
+## Common Error Patterns
+
+### Missing Field Errors
+```
+missing field `field_name` at line X column Y
+```
+**Quick Fix:** Run `python3 scripts/fix_user_data.py` - it handles most missing fields automatically.
+
+### Invalid Enum Variant
+```
+unknown variant `OldVariantName`
+```
+**Quick Fix:** The Python script maps old variants to new ones automatically.
+
+### Structure Mismatch
+```
+invalid type: map, expected a sequence
+```
+**Manual Fix Required:** Check the data structure against `src/models/user.rs` definitions.
+
+## Step-by-Step Resolution
+
+1. **Stop the application** if it's running
+2. **Run the fix script:**
+   ```bash
+   python3 scripts/fix_user_data.py
+   ```
+3. **Check the output** for what was fixed
+4. **Test the application:**
+   ```bash
+   cargo run
+   ```
+5. **If errors persist,** check the detailed guide: [`data-validation-guide.md`](./data-validation-guide.md)
+
+## Prevention
+
+- Run `python3 scripts/fix_user_data.py` after pulling schema changes
+- Monitor application logs for new parsing errors
+- Backup user data before making manual edits
+
+## Need Help?
+
+1. Check [`data-validation-guide.md`](./data-validation-guide.md) for detailed information
+2. Look at `src/models/user.rs` for current schema requirements
+3. Use the Rust validator in `src/utils/data_validator.rs` for detailed analysis