doctree/doctree_implementation_plan.md

DocTree Implementation Plan

Overview

The DocTree library will be a Rust implementation of the Go reference, maintaining the core functionality while improving the API design to be more idiomatic Rust. We'll use Redis as the storage backend and implement a minimal CLI example to demonstrate usage.

Architecture

classDiagram
    class DocTree {
        +collections: HashMap<String, Collection>
        +default_collection: Option<String>
        +new() DocTreeBuilder
        +add_collection(path, name) Result<&Collection>
        +get_collection(name) Result<&Collection>
        +delete_collection(name) Result<()>
        +list_collections() Vec<String>
        +page_get(collection, page) Result<String>
        +page_get_html(collection, page) Result<String>
        +file_get_url(collection, file) Result<String>
    }

    class DocTreeBuilder {
        -collections: HashMap<String, Collection>
        -default_collection: Option<String>
        +with_collection(path, name) DocTreeBuilder
        +with_default_collection(name) DocTreeBuilder
        +build() Result<DocTree>
    }

    class Collection {
        +path: String
        +name: String
        +new(path, name) CollectionBuilder
        +scan() Result<()>
        +page_get(name) Result<String>
        +page_set(name, content) Result<()>
        +page_delete(name) Result<()>
        +page_list() Result<Vec<String>>
        +file_get_url(name) Result<String>
        +file_set(name, content) Result<()>
        +file_delete(name) Result<()>
        +file_list() Result<Vec<String>>
        +page_get_html(name) Result<String>
    }

    class CollectionBuilder {
        -path: String
        -name: String
        +build() Result<Collection>
    }

    class RedisStorage {
        +client: redis::Client
        +new(url) Result<RedisStorage>
        +store_collection_entry(collection, key, value) Result<()>
        +get_collection_entry(collection, key) Result<String>
        +delete_collection_entry(collection, key) Result<()>
        +list_collection_entries(collection) Result<Vec<String>>
        +delete_collection(collection) Result<()>
    }

    class IncludeProcessor {
        +process_includes(content, collection, doctree) Result<String>
    }

    DocTree --> DocTreeBuilder : creates
    DocTree --> "0..*" Collection : contains
    Collection --> CollectionBuilder : creates
    DocTree --> RedisStorage : uses
    Collection --> RedisStorage : uses
    DocTree --> IncludeProcessor : uses

Implementation Steps

1. Project Setup and Dependencies

1. Update the Cargo.toml files with necessary dependencies:
   • redis (for Redis client)
   • walkdir (for directory traversal)
   • pulldown-cmark (for Markdown to HTML conversion)
   • thiserror (for error handling)
   • clap (for CLI argument parsing in doctreecmd)

2. Core Library Structure

1. Error Module

   • Create a custom error type using thiserror
   • Define specific error variants for different failure scenarios

2. Storage Module

   • Implement the RedisStorage struct to handle Redis operations
   • Provide methods for storing, retrieving, and deleting collection entries
   • Implement connection pooling for efficient Redis access

3. Utils Module

   • Implement utility functions like name_fix (equivalent to tools.NameFix in Go)
   • Implement markdown to HTML conversion using pulldown-cmark

3. Collection Implementation

1. Collection Module

   • Implement the Collection struct to represent a collection of documents
   • Implement the CollectionBuilder for creating Collection instances
   • Implement methods for scanning directories, managing pages and files

2. Collection Builder Pattern

   • Create a builder pattern for Collection creation
   • Allow configuration of Collection properties before building

4. DocTree Implementation

1. DocTree Module

   • Implement the DocTree struct to manage multiple collections
   • Implement the DocTreeBuilder for creating DocTree instances
   • Implement methods for managing collections and accessing documents

2. DocTree Builder Pattern

   • Create a builder pattern for DocTree creation
   • Allow adding collections and setting default collection before building

5. Include Processor Implementation

1. Include Module
   • Implement the IncludeProcessor to handle include directives
   • Implement parsing of include directives
   • Implement recursive processing of includes

6. CLI Example

1. Update doctreecmd
   • Implement a minimal CLI interface using clap
   • Provide commands for basic operations:
     • Scanning a directory
     • Listing collections
     • Getting page content
     • Getting HTML content

Detailed Module Breakdown

Error Module (src/error.rs)

use thiserror::Error;

#[derive(Error, Debug)]
pub enum DocTreeError {
    #[error("IO error: {0}")]
    IoError(#[from] std::io::Error),
    
    #[error("Redis error: {0}")]
    RedisError(#[from] redis::RedisError),
    
    #[error("Collection not found: {0}")]
    CollectionNotFound(String),
    
    #[error("Page not found: {0}")]
    PageNotFound(String),
    
    #[error("File not found: {0}")]
    FileNotFound(String),
    
    #[error("Invalid include directive: {0}")]
    InvalidIncludeDirective(String),
    
    #[error("No default collection set")]
    NoDefaultCollection,
    
    #[error("Invalid number of arguments")]
    InvalidArgumentCount,
}

pub type Result<T> = std::result::Result<T, DocTreeError>;

Storage Module (src/storage.rs)

use redis::{Client, Commands, Connection};
use crate::error::{DocTreeError, Result};

pub struct RedisStorage {
    client: Client,
}

impl RedisStorage {
    pub fn new(url: &str) -> Result<Self> {
        let client = Client::open(url)?;
        Ok(Self { client })
    }
    
    pub fn get_connection(&self) -> Result<Connection> {
        Ok(self.client.get_connection()?)
    }
    
    pub fn store_collection_entry(&self, collection: &str, key: &str, value: &str) -> Result<()> {
        let mut conn = self.get_connection()?;
        let collection_key = format!("collections:{}", collection);
        conn.hset(collection_key, key, value)?;
        Ok(())
    }
    
    pub fn get_collection_entry(&self, collection: &str, key: &str) -> Result<String> {
        let mut conn = self.get_connection()?;
        let collection_key = format!("collections:{}", collection);
        let value: String = conn.hget(collection_key, key)?;
        Ok(value)
    }
    
    // Additional methods for Redis operations
}

Utils Module (src/utils.rs)

use pulldown_cmark::{Parser, Options, html};

pub fn name_fix(name: &str) -> String {
    // Implementation of name_fix similar to tools.NameFix in Go
    // Normalize the name by converting to lowercase, replacing spaces with hyphens, etc.
}

pub fn markdown_to_html(markdown: &str) -> String {
    let mut options = Options::empty();
    options.insert(Options::ENABLE_TABLES);
    options.insert(Options::ENABLE_FOOTNOTES);
    options.insert(Options::ENABLE_STRIKETHROUGH);
    
    let parser = Parser::new_ext(markdown, options);
    let mut html_output = String::new();
    html::push_html(&mut html_output, parser);
    
    html_output
}

Collection Module (src/collection.rs)

use std::path::{Path, PathBuf};
use walkdir::WalkDir;
use crate::error::Result;
use crate::storage::RedisStorage;
use crate::utils::name_fix;

pub struct Collection {
    pub path: PathBuf,
    pub name: String,
    storage: RedisStorage,
}

pub struct CollectionBuilder {
    path: PathBuf,
    name: String,
    storage: Option<RedisStorage>,
}

impl Collection {
    pub fn builder<P: AsRef<Path>>(path: P, name: &str) -> CollectionBuilder {
        CollectionBuilder {
            path: path.as_ref().to_path_buf(),
            name: name_fix(name),
            storage: None,
        }
    }
    
    pub fn scan(&self) -> Result<()> {
        // Implementation of scanning directory and storing in Redis
    }
    
    pub fn page_get(&self, page_name: &str) -> Result<String> {
        // Implementation of getting page content
    }
    
    // Additional methods for Collection
}

impl CollectionBuilder {
    pub fn with_storage(mut self, storage: RedisStorage) -> Self {
        self.storage = Some(storage);
        self
    }
    
    pub fn build(self) -> Result<Collection> {
        let storage = self.storage.ok_or_else(|| {
            std::io::Error::new(std::io::ErrorKind::Other, "Storage not provided")
        })?;
        
        let collection = Collection {
            path: self.path,
            name: self.name,
            storage,
        };
        
        Ok(collection)
    }
}

DocTree Module (src/doctree.rs)

use std::collections::HashMap;
use std::path::Path;
use crate::collection::{Collection, CollectionBuilder};
use crate::error::{DocTreeError, Result};
use crate::storage::RedisStorage;

pub struct DocTree {
    collections: HashMap<String, Collection>,
    default_collection: Option<String>,
    storage: RedisStorage,
}

pub struct DocTreeBuilder {
    collections: HashMap<String, Collection>,
    default_collection: Option<String>,
    storage: Option<RedisStorage>,
}

impl DocTree {
    pub fn builder() -> DocTreeBuilder {
        DocTreeBuilder {
            collections: HashMap::new(),
            default_collection: None,
            storage: None,
        }
    }
    
    pub fn add_collection<P: AsRef<Path>>(&mut self, path: P, name: &str) -> Result<&Collection> {
        // Implementation of adding a collection
    }
    
    // Additional methods for DocTree
}

impl DocTreeBuilder {
    pub fn with_storage(mut self, storage: RedisStorage) -> Self {
        self.storage = Some(storage);
        self
    }
    
    pub fn with_collection<P: AsRef<Path>>(mut self, path: P, name: &str) -> Result<Self> {
        // Implementation of adding a collection during building
        Ok(self)
    }
    
    pub fn with_default_collection(mut self, name: &str) -> Self {
        self.default_collection = Some(name.to_string());
        self
    }
    
    pub fn build(self) -> Result<DocTree> {
        let storage = self.storage.ok_or_else(|| {
            std::io::Error::new(std::io::ErrorKind::Other, "Storage not provided")
        })?;
        
        let doctree = DocTree {
            collections: self.collections,
            default_collection: self.default_collection,
            storage,
        };
        
        Ok(doctree)
    }
}

Include Module (src/include.rs)

use crate::doctree::DocTree;
use crate::error::Result;

pub fn process_includes(content: &str, collection_name: &str, doctree: &DocTree) -> Result<String> {
    // Implementation of processing include directives
}

fn parse_include_line(line: &str) -> Result<(Option<String>, Option<String>)> {
    // Implementation of parsing include directives
}

fn handle_include(page_name: &str, collection_name: &str, doctree: &DocTree) -> Result<String> {
    // Implementation of handling include directives
}

Main Library File (src/lib.rs)

mod error;
mod storage;
mod utils;
mod collection;
mod doctree;
mod include;

pub use error::{DocTreeError, Result};
pub use storage::RedisStorage;
pub use collection::{Collection, CollectionBuilder};
pub use doctree::{DocTree, DocTreeBuilder};
pub use include::process_includes;

CLI Example (doctreecmd/src/main.rs)

use clap::{App, Arg, SubCommand};
use doctree::{DocTree, RedisStorage};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let matches = App::new("DocTree CLI")
        .version("0.1.0")
        .author("Your Name")
        .about("A tool to manage document collections")
        .subcommand(
            SubCommand::with_name("scan")
                .about("Scan a directory and create a collection")
                .arg(Arg::with_name("path").required(true))
                .arg(Arg::with_name("name").required(true)),
        )
        .subcommand(
            SubCommand::with_name("list")
                .about("List collections"),
        )
        .subcommand(
            SubCommand::with_name("get")
                .about("Get page content")
                .arg(Arg::with_name("collection").required(true))
                .arg(Arg::with_name("page").required(true)),
        )
        .get_matches();

    // Implementation of CLI commands
    
    Ok(())
}

Example Usage

Here's how the library would be used with the builder pattern:

use doctree::{DocTree, RedisStorage};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create a Redis storage instance
    let storage = RedisStorage::new("redis://localhost:6379")?;
    
    // Create a DocTree instance using the builder pattern
    let mut doctree = DocTree::builder()
        .with_storage(storage.clone())
        .with_collection("path/to/collection", "my-collection")?
        .with_default_collection("my-collection")
        .build()?;
    
    // Get page content
    let content = doctree.page_get("my-collection", "page-name")?;
    println!("Page content: {}", content);
    
    // Get HTML content
    let html = doctree.page_get_html("my-collection", "page-name")?;
    println!("HTML content: {}", html);
    
    Ok(())
}

Testing Strategy

1. Unit Tests

   • Test individual components in isolation
   • Mock Redis for testing storage operations
   • Test utility functions

2. Integration Tests

   • Test the interaction between components
   • Test the builder pattern
   • Test include processing

3. End-to-End Tests

   • Test the complete workflow with real files
   • Test the CLI interface

Timeline

1. Project Setup and Dependencies: 1 day
2. Core Library Structure: 2 days
3. Collection Implementation: 2 days
4. DocTree Implementation: 2 days
5. Include Processor Implementation: 1 day
6. CLI Example: 1 day
7. Testing and Documentation: 2 days

Total estimated time: 11 days