152 lines
4.6 KiB
Markdown
152 lines
4.6 KiB
Markdown
# Data Validation Guide
|
|
|
|
## Overview
|
|
|
|
This guide documents the data validation tools and processes for the Project Mycelium application. These tools help maintain data integrity and resolve JSON parsing errors when user data schemas evolve.
|
|
|
|
## Problem Context
|
|
|
|
The application stores user data in JSON files that must match Rust struct definitions for serde deserialization. When schemas change or data becomes corrupted, parsing errors occur that prevent users from accessing dashboard features.
|
|
|
|
## Validation Tools
|
|
|
|
### 1. Rust Data Validator (`src/utils/data_validator.rs`)
|
|
|
|
A comprehensive validation utility that:
|
|
- Validates JSON structure against expected Rust schemas
|
|
- Provides detailed error reporting with line numbers
|
|
- Handles schema migration and repair
|
|
- Supports batch validation of multiple user files
|
|
|
|
**Usage:**
|
|
```rust
|
|
use crate::utils::data_validator::DataValidator;
|
|
|
|
let validator = DataValidator::new();
|
|
let result = validator.validate_user_file("user_data/user1_at_example_com.json");
|
|
```
|
|
|
|
### 2. Python Validation Script (`fix_user_data.py`)
|
|
|
|
A standalone Python script that:
|
|
- Fixes common schema mismatches automatically
|
|
- Handles missing fields, invalid enum variants, and null values
|
|
- Processes all user data files in batch
|
|
- Provides detailed change reporting
|
|
|
|
**Usage:**
|
|
```bash
|
|
cd projectmycelium
|
|
python3 scripts/fix_user_data.py
|
|
```
|
|
|
|
## Common Schema Issues and Solutions
|
|
|
|
### Missing Fields
|
|
**Problem:** JSON missing required fields like `minimum_deployment_duration`, `preferred_regions`
|
|
**Solution:** Add default values based on field type and context
|
|
|
|
### Invalid Enum Variants
|
|
**Problem:** Old enum values like `ServiceProgress`, `AppDeployment`
|
|
**Solution:** Map to current valid variants: `ServiceCreated`, `Deployment`
|
|
|
|
### Incorrect Object Structure
|
|
**Problem:** Objects not matching expected struct definitions
|
|
**Solution:** Restructure to match current schema requirements
|
|
|
|
### Example: UsageStatistics Structure
|
|
```json
|
|
{
|
|
"usage_statistics": {
|
|
"total_deployments": 85,
|
|
"active_services": 12,
|
|
"total_spent": "2450.25",
|
|
"favorite_categories": ["compute", "storage", "networking"],
|
|
"usage_trends": [
|
|
{
|
|
"period": "month",
|
|
"metric": "deployments",
|
|
"value": 25.5,
|
|
"change_percentage": 12.3
|
|
}
|
|
],
|
|
"login_frequency": 4.2,
|
|
"preferred_regions": ["us-east", "eu-west"],
|
|
"account_age_days": 150,
|
|
"last_activity": "2025-06-18T20:30:00Z"
|
|
}
|
|
}
|
|
```
|
|
|
|
## Troubleshooting Process
|
|
|
|
### 1. Identify Parsing Errors
|
|
Look for serde deserialization errors in application logs:
|
|
```
|
|
Failed to parse user data for user@example.com: missing field `field_name` at line X column Y
|
|
```
|
|
|
|
### 2. Run Validation Tools
|
|
Use the Python script for quick fixes:
|
|
```bash
|
|
python3 scripts/fix_user_data.py
|
|
```
|
|
|
|
Or use the Rust validator for detailed analysis:
|
|
```rust
|
|
let validator = DataValidator::new();
|
|
validator.validate_all_user_files();
|
|
```
|
|
|
|
### 3. Manual Schema Updates
|
|
For complex schema changes, manually update JSON structure to match current Rust struct definitions in `src/models/user.rs`.
|
|
|
|
### 4. Verify Resolution
|
|
Test the application to ensure parsing errors are resolved:
|
|
```bash
|
|
cargo run
|
|
```
|
|
|
|
## Best Practices
|
|
|
|
### Schema Evolution
|
|
1. **Update validation tools** when adding new required fields to Rust structs
|
|
2. **Provide migration logic** for existing data
|
|
3. **Test with all user data files** before deploying schema changes
|
|
4. **Document breaking changes** in this guide
|
|
|
|
### Data Integrity
|
|
1. **Run validation regularly** as part of development workflow
|
|
2. **Backup user data** before applying automated fixes
|
|
3. **Validate after manual edits** to ensure correctness
|
|
4. **Monitor application logs** for new parsing errors
|
|
|
|
### Tool Maintenance
|
|
1. **Keep validation tools updated** with current schema requirements
|
|
2. **Add new validation rules** for new data structures
|
|
3. **Test validation tools** with various error scenarios
|
|
4. **Document new validation patterns** in this guide
|
|
|
|
## File Locations
|
|
|
|
- **Rust Validator:** `src/utils/data_validator.rs`
|
|
- **Python Script:** `scripts/fix_user_data.py`
|
|
- **User Data:** `user_data/*.json`
|
|
- **Schema Definitions:** `src/models/user.rs`
|
|
- **Utils Module:** `src/utils/mod.rs`
|
|
|
|
## Integration
|
|
|
|
The validation tools are integrated into the utils module and can be used throughout the application for:
|
|
- Development-time validation
|
|
- Runtime error recovery
|
|
- Data migration during updates
|
|
- Quality assurance testing
|
|
|
|
## Future Enhancements
|
|
|
|
Consider adding:
|
|
- Automated validation in CI/CD pipeline
|
|
- Real-time validation during data updates
|
|
- Schema versioning and automatic migration
|
|
- Web interface for data validation and repair |