projectmycelium/docs/dev/guides/data-validation-guide.md

Data Validation Guide

Overview

This guide documents the data validation tools and processes for the Project Mycelium application. These tools help maintain data integrity and resolve JSON parsing errors when user data schemas evolve.

Problem Context

The application stores user data in JSON files that must match Rust struct definitions for serde deserialization. When schemas change or data becomes corrupted, parsing errors occur that prevent users from accessing dashboard features.

Validation Tools

1. Rust Data Validator (src/utils/data_validator.rs)

A comprehensive validation utility that:

• Validates JSON structure against expected Rust schemas
• Provides detailed error reporting with line numbers
• Handles schema migration and repair
• Supports batch validation of multiple user files

Usage:

use crate::utils::data_validator::DataValidator;

let validator = DataValidator::new();
let result = validator.validate_user_file("user_data/user1_at_example_com.json");

2. Python Validation Script (fix_user_data.py)

A standalone Python script that:

• Fixes common schema mismatches automatically
• Handles missing fields, invalid enum variants, and null values
• Processes all user data files in batch
• Provides detailed change reporting

Usage:

cd projectmycelium
python3 scripts/fix_user_data.py

Common Schema Issues and Solutions

Missing Fields

Problem: JSON missing required fields like minimum_deployment_duration, preferred_regions Solution: Add default values based on field type and context

Invalid Enum Variants

Problem: Old enum values like ServiceProgress, AppDeployment Solution: Map to current valid variants: ServiceCreated, Deployment

Incorrect Object Structure

Problem: Objects not matching expected struct definitions Solution: Restructure to match current schema requirements

Example: UsageStatistics Structure

{
  "usage_statistics": {
    "total_deployments": 85,
    "active_services": 12,
    "total_spent": "2450.25",
    "favorite_categories": ["compute", "storage", "networking"],
    "usage_trends": [
      {
        "period": "month",
        "metric": "deployments", 
        "value": 25.5,
        "change_percentage": 12.3
      }
    ],
    "login_frequency": 4.2,
    "preferred_regions": ["us-east", "eu-west"],
    "account_age_days": 150,
    "last_activity": "2025-06-18T20:30:00Z"
  }
}

Troubleshooting Process

1. Identify Parsing Errors

Look for serde deserialization errors in application logs:

Failed to parse user data for user@example.com: missing field `field_name` at line X column Y

2. Run Validation Tools

Use the Python script for quick fixes:

python3 scripts/fix_user_data.py

Or use the Rust validator for detailed analysis:

let validator = DataValidator::new();
validator.validate_all_user_files();

3. Manual Schema Updates

For complex schema changes, manually update JSON structure to match current Rust struct definitions in src/models/user.rs.

4. Verify Resolution

Test the application to ensure parsing errors are resolved:

cargo run

Best Practices

Schema Evolution

1. Update validation tools when adding new required fields to Rust structs
2. Provide migration logic for existing data
3. Test with all user data files before deploying schema changes
4. Document breaking changes in this guide

Data Integrity

1. Run validation regularly as part of development workflow
2. Backup user data before applying automated fixes
3. Validate after manual edits to ensure correctness
4. Monitor application logs for new parsing errors

Tool Maintenance

1. Keep validation tools updated with current schema requirements
2. Add new validation rules for new data structures
3. Test validation tools with various error scenarios
4. Document new validation patterns in this guide

File Locations

• Rust Validator: src/utils/data_validator.rs
• Python Script: scripts/fix_user_data.py
• User Data: user_data/*.json
• Schema Definitions: src/models/user.rs
• Utils Module: src/utils/mod.rs

Integration

The validation tools are integrated into the utils module and can be used throughout the application for:

• Development-time validation
• Runtime error recovery
• Data migration during updates
• Quality assurance testing

Future Enhancements

Consider adding:

• Automated validation in CI/CD pipeline
• Real-time validation during data updates
• Schema versioning and automatic migration
• Web interface for data validation and repair