Hero Atlas - Document Management System

A Rust-based document collection management system with CLI, library, and web interfaces for processing markdown-based documentation with support for cross-collection references, link validation, and export to self-contained directories.

Project Structure

This is a single Rust package (hero_atlas) with multiple entry points:

• Library (src/lib.rs) - Core functionality for document collection management
• CLI (src/bin/cli.rs) - Command-line interface (atlas binary)
• Web Server (src/bin/web.rs) - HTTP API and web interface (atlas-web binary)
• Modules:
  • doctree - Document tree management and validation
  • website - Website configuration and metadata
  • ebook - Ebook parsing and configuration
  • web - HTTP server handlers and routes
  • cli - CLI command definitions and handlers

Module Documentation

• DocTree Module - Complete document collection management system

  • Collection scanning and indexing
  • Link parsing and validation
  • Include directive processing
  • Access control management
  • Export to self-contained directories
  • Read-only client API

• Website Module - Metadata-driven website definitions

  • Website configuration (Docusaurus-style)
  • Navigation bar with dropdown menus
  • Sidebar navigation with multiple sidebars
  • Footer with link columns
  • Page-level metadata and SEO
  • Theme and styling configuration
  • Social links and custom fields

Quick Start

# Build the project
./build.sh

# Run the CLI
./target/release/atlas --help

# Start the web server
./target/release/atlas-web

# Or use the build script which handles everything
./run.sh

Features

• Collection scanning: Automatically discover collections marked with .collection files
• Cross-collection references: Link between pages in different collections using collection:page syntax
• Include directives: Embed content from other pages with !!include collection:page
• Link validation: Detect broken links to pages, images, and files
• Export: Generate self-contained directories with all dependencies
• Access control: Group-based ACL via .group files
• Git integration: Automatically detect repository URLs

Installation

Build from source

./build.sh

Binaries will be at:

• target/release/atlas - CLI application
• target/release/atlas-web - Web server

Install to PATH

./install.sh

Installs both binaries to ~/.local/bin/:

• ~/.local/bin/atlas - CLI application
• ~/.local/bin/atlas-web - Web server

Architecture & Concepts

Separation of Concerns

Hero Atlas separates content from presentation:

DocTree (Content):

• Manages markdown collections and pages
• Validates links and references
• Tracks files and images
• Processes include directives
• Enforces access control

Website (Presentation):

• Defines navigation structure
• Configures sidebars and menus
• Manages theming and styling
• Handles SEO metadata
• Provides plugin architecture

This separation allows flexible website layouts without changing content.

Key Concepts

Collections: Directories of markdown pages marked with .collection file

• Each collection is independently managed
• Collections can reference each other
• Access control per collection via ACL files

Pages: Individual markdown files with:

• Extracted title (from H1 heading)
• Description (from first paragraph)
• Parsed internal links and includes
• Optional front matter metadata

Links: References to pages, images, or files:

• Same collection: [text](page_name)
• Cross-collection: [text](collection:page)
• External: Automatic detection of HTTP(S) URLs
• Images: Identified by extension

Groups: Access control lists defining user membership

• Grant read/write access to collections
• Support wildcards for email patterns
• Support group inclusion (nested groups)

Export: Self-contained read-only directory:

• Pages and files organized by collection
• JSON metadata for each collection
• Suitable for static hosting or archival

Data Flow

Directory Scan
    ↓
Find Collections (.collection files)
    ↓
Parse Pages (extract metadata, parse links)
    ↓
Validate Links (check references exist)
    ↓
Process Includes (expand !!include directives)
    ↓
Enforce ACL (check group membership)
    ↓
Export (write to structured directory)
    ↓
Read Client (query exported collections)

Refactoring Notes

This project was refactored from a multi-package workspace into a single unified package following Rust best practices:

Previous Structure (workspace with 3 crates):

• lib/ - atlas-lib library
• atlas/ - atlas CLI binary
• web/ - atlas-web server

New Structure (single package with multiple binaries):

• Single hero_atlas package in Cargo.toml
• Two binaries via src/bin/:
  • cli.rs → atlas binary
  • web.rs → atlas-web binary
• Modular organization in src/:
  • cli/ - CLI command definitions
  • doctree/ - Core document management
  • ebook/ - Ebook parsing
  • web/ - HTTP server handlers
  • website/ - Website configuration

Benefits:

• Simpler dependency management
• Unified build system
• Easier code sharing between CLI and web server
• Cleaner project organization
• Aligned with Rust conventions for monolithic applications

CLI Usage

Scan collections

atlas scan --path /path/to/docs
atlas scan --path /path/to/docs --ignore node_modules --ignore .git

Export collections

# Export to default location (/tmp/atlas)
atlas export --path /path/to/docs

# Export with reset (clear destination first)
atlas export --path /path/to/docs --reset

# Export to custom location
atlas export --path /path/to/docs --destination /path/to/export

# Process include directives during export
atlas export --path /path/to/docs --include

List collections and pages

atlas list --path /path/to/docs

Show validation errors

atlas errors --path /path/to/docs

Get page content

atlas page --key collection_name:page_name --path /path/to/docs

Read from exported doctree

# List collections
atlas read collections

# List pages in a collection
atlas read pages --collection mycolection

# Get page content
atlas read page --collection mycollection --page mypage

# List images
atlas read images --collection mycollection

# List files
atlas read files --collection mycollection

Directory Structure

Source Structure

docs/
├── collection1/
│   ├── .collection           # Marks as collection (optional: name:custom_name)
│   ├── read.acl              # Optional: group names for read access
│   ├── write.acl             # Optional: group names for write access
│   ├── page1.md
│   ├── subdir/
│   │   └── page2.md
│   └── img/
│       └── logo.png
├── collection2/
│   ├── .collection
│   └── intro.md
└── groups/                   # Special collection for ACL groups
    ├── .collection
    ├── admins.group
    └── editors.group

Export Structure

/tmp/atlas/
├── content/
│   └── collection_name/
│       ├── page1.md          # Pages at root of collection dir
│       ├── page2.md
│       ├── img/              # All images in img/ subdirectory
│       │   └── logo.png
│       └── files/            # All other files in files/ subdirectory
│           └── document.pdf
└── meta/
    └── collection_name.json  # Collection metadata

File Formats

.collection

name:custom_collection_name

If empty or name not specified, uses directory name.

.group

// Comments start with //
user@example.com
*@company.com
include:other_group

ACL files (read.acl, write.acl)

admins
editors

One group name per line.

Link Syntax

Page links

[text](page_name)           # Same collection
[text](collection:page)     # Cross-collection

Image links

![alt](img/image.png)              # Same collection
![alt](collection:img/image.png)   # Cross-collection

Include directives

!!include page_name
!!include collection:page_name

Name Normalization

Page and collection names are normalized:

1. Convert to lowercase
2. Replace - with _
3. Replace / with _
4. Remove .md extension
5. Strip numeric prefix (e.g., 03_page → page)
6. Remove special characters

Supported Image Extensions

• .png, .jpg, .jpeg, .gif, .svg, .webp, .bmp, .tiff, .ico

Development

Building

# Build release binaries
./build.sh

# Build with debug info
cargo build

# Run tests
cargo test

# Generate documentation
cargo doc --no-deps --open

Testing

# Run all tests
cargo test

# Run specific module tests
cargo test doctree
cargo test website

# Run with output
cargo test -- --nocapture

Code Organization

src/
├── lib.rs              # Library exports
├── bin/
│   ├── cli.rs         # CLI entry point (atlas binary)
│   └── web.rs         # Web server entry point (atlas-web binary)
├── cli/                # CLI command implementation
├── doctree/            # Document management (1,975 LOC)
├── ebook/              # Ebook parsing
├── web/                # HTTP API handlers
└── website/            # Website configuration (1,522 LOC)

Adding New Features

1. New DocTree functionality: Add to src/doctree/
2. New Website config: Add to src/website/
3. New CLI commands: Add to src/cli/mod.rs
4. New API endpoints: Add to src/web/mod.rs

Library Usage

use doctree::{DocTree, ExportArgs};

fn main() -> doctree::Result<()> {
    // Create and scan
    let mut doctree = DocTree::new("mydocs");
    doctree.scan(Path::new("/path/to/docs"), &[])?;
    doctree.init_post()?;  // Validate links

    // Access pages
    let page = doctree.page_get("collection:page")?;
    let content = page.content()?;

    // Export
    doctree.export(ExportArgs {
        destination: PathBuf::from("/tmp/atlas"),
        reset: true,
        include: false,
    })?;

    Ok(())
}

DocTreeClient (for reading exports)

use doctree::DocTreeClient;

fn main() -> doctree::Result<()> {
    let client = DocTreeClient::new(Path::new("/tmp/atlas"))?;

    // List collections
    let collections = client.list_collections()?;

    // Get page content
    let content = client.get_page_content("collection", "page")?;

    // Check existence
    if client.page_exists("collection", "page") {
        println!("Page exists!");
    }

    Ok(())
}

Directory Structure

atlasserver_rust/
├── Cargo.toml           # Package configuration
├── build.sh             # Build script
├── install.sh           # Install script
├── run.sh               # Run script
├── README.md            # This file
├── src/
│   ├── lib.rs           # Library entry point and module declarations
│   ├── bin/
│   │   ├── cli.rs       # CLI binary entry point
│   │   └── web.rs       # Web server entry point
│   ├── cli/
│   │   └── mod.rs       # CLI commands and handlers
│   ├── doctree/         # Document tree management
│   │   ├── mod.rs
│   │   ├── core.rs
│   │   ├── collection.rs
│   │   ├── page.rs
│   │   ├── group.rs
│   │   ├── link.rs
│   │   ├── client.rs
│   │   ├── types.rs
│   │   ├── error.rs
│   │   └── utils.rs
│   ├── ebook/           # Ebook parsing
│   │   └── mod.rs
│   ├── web/             # HTTP API routes and handlers
│   │   └── mod.rs
│   └── website/         # Website configuration
│       ├── mod.rs
│       ├── config.rs
│       ├── navbar.rs
│       ├── sidebar.rs
│       ├── footer.rs
│       ├── page.rs
│       └── theme.rs
└── target/
    └── release/
        ├── atlas        # CLI binary
        └── atlas-web    # Web server binary