doctree/impl_plan.md

# DocTree WebBuilder Implementation Plan

## Overview

This document outlines the implementation plan for the WebBuilder component of the DocTree project. The WebBuilder is designed to process hjson configuration files (like those in `examples/doctreenew/sites/demo1/`) and generate a `webmeta.json` file that can be used by a browser-based website generator.

## Current Status

### What's Implemented:

1. **DocTree Core Functionality**:
   - The main DocTree library with functionality for scanning directories, managing collections, processing includes, and converting markdown to HTML
   - Redis storage backend for storing document metadata
   - Command-line interface (doctreecmd) for interacting with collections

2. **Example Structure for the New Approach**:
   - Example hjson configuration files in `examples/doctreenew/sites/demo1/`
   - This includes `main.hjson`, `header.hjson`, `footer.hjson`, `collection.hjson`, and `pages/mypages1.hjson`

3. **Specification Document**:
   - Detailed specification in `webbuilder/src/builder/specs.md`
   - Example output format in `webbuilder/src/builder/webmeta.json`

### What's Not Yet Implemented:

1. **WebBuilder Implementation**:
   - The actual Rust code for the webbuilder component

2. **Hjson Parsing**:
   - Code to parse the hjson files in the doctreenew directory

3. **Git Repository Integration**:
   - Functionality to download referenced collections from Git repositories

4. **IPFS Export**:
   - Complete functionality to export assets to IPFS

5. **Browser-Based Generator**:
   - The browser-based website generator that would use the webmeta.json file

## Implementation Plan

### Phase 1: Core WebBuilder Implementation (2-3 weeks)

1. **Setup Project Structure**:
   - Create necessary modules and files in `webbuilder/src/`
   - Define main data structures and traits

2. **Implement Hjson Parsing**:
   - Add hjson crate dependency
   - Create parsers for each hjson file type (main, header, footer, collection, pages)
   - Implement validation for hjson files

3. **Implement Site Structure Builder**:
   - Create a module to combine parsed hjson data into a cohesive site structure
   - Implement navigation generation based on page definitions

4. **Implement WebMeta Generator**:
   - Create functionality to generate the webmeta.json file
   - Ensure all required metadata is included

### Phase 2: Git Integration and Collection Processing (2 weeks)

1. **Implement Git Repository Integration**:
   - Add git2 crate dependency
   - Create functionality to clone/pull repositories based on collection.hjson
   - Implement caching to avoid unnecessary downloads

2. **Integrate with DocTree Library**:
   - Create an adapter to use DocTree functionality with hjson-defined collections
   - Implement processing of includes between documents

3. **Implement Content Processing**:
   - Create functionality to process markdown content
   - Handle special directives or custom syntax

### Phase 3: IPFS Integration (2 weeks)

1. **Enhance IPFS Integration**:
   - Complete the IPFS export functionality in DocTree
   - Create a module to handle IPFS uploads

2. **Implement Asset Management**:
   - Create functionality to track and process assets (images, CSS, etc.)
   - Ensure proper IPFS linking

3. **Implement Content Hashing**:
   - Add Blake hash calculation for content integrity verification
   - Store hashes in webmeta.json

### Phase 4: CLI and Testing (1-2 weeks)

1. **Implement Command-Line Interface**:
   - Create a CLI for the webbuilder
   - Add commands for building, validating, and deploying sites

2. **Write Comprehensive Tests**:
   - Unit tests for each component
   - Integration tests for the full workflow
   - Test with example sites

3. **Documentation**:
   - Update README with usage instructions
   - Create detailed API documentation
   - Add examples and tutorials

### Phase 5: Browser-Based Generator (Optional, 3-4 weeks)

1. **Design Browser Component**:
   - Create a JavaScript/TypeScript library to consume webmeta.json
   - Design component architecture

2. **Implement Content Rendering**:
   - Create components to render markdown content
   - Implement navigation and site structure

3. **Implement IPFS Integration**:
   - Add functionality to fetch content from IPFS
   - Implement content verification using Blake hashes

4. **Create Demo Site**:
   - Build a demo site using the browser-based generator
   - Showcase features and capabilities

## Technical Details

### Key Dependencies

- **hjson**: For parsing hjson configuration files
- **git2**: For Git repository integration
- **ipfs-api**: For IPFS integration
- **blake3**: For content hashing
- **clap**: For command-line interface
- **tokio**: For async operations

### Data Flow

1. Parse hjson files from input directory
2. Download referenced Git repositories
3. Process content with DocTree
4. Export assets to IPFS
5. Generate webmeta.json
6. (Optional) Upload webmeta.json to IPFS

### Key Challenges

1. **Git Integration**: Handling authentication, rate limits, and large repositories
2. **IPFS Performance**: Optimizing IPFS uploads for large sites
3. **Content Processing**: Ensuring proper handling of includes and special syntax
4. **Browser Compatibility**: Ensuring the browser-based generator works across different browsers

## Milestones and Timeline

1. **Core WebBuilder Implementation**: Weeks 1-3
2. **Git Integration and Collection Processing**: Weeks 4-5
3. **IPFS Integration**: Weeks 6-7
4. **CLI and Testing**: Weeks 8-9
5. **Browser-Based Generator (Optional)**: Weeks 10-13

## Resources Required

1. **Development Resources**:
   - 1-2 Rust developers
   - 1 Frontend developer (for browser-based generator)

2. **Infrastructure**:
   - IPFS node for testing
   - Git repositories for testing
   - CI/CD pipeline

## Conclusion

This implementation plan provides a roadmap for developing the WebBuilder component of the DocTree project. By following this plan, we can transform the current specification and example files into a fully functional system for generating websites from hjson configuration files and markdown content.