...
This commit is contained in:
24
webbuilder/Cargo.toml
Normal file
24
webbuilder/Cargo.toml
Normal file
@@ -0,0 +1,24 @@
|
||||
[package]
|
||||
name = "doctree"
|
||||
version = "0.1.0"
|
||||
edition = "2024"
|
||||
[lib]
|
||||
path = "src/lib.rs"
|
||||
|
||||
|
||||
[dependencies]
|
||||
walkdir = "2.3.3"
|
||||
pulldown-cmark = "0.9.3"
|
||||
thiserror = "1.0.40"
|
||||
lazy_static = "1.4.0"
|
||||
toml = "0.7.3"
|
||||
serde = { version = "1.0", features = ["derive"] }
|
||||
redis = { version = "0.23.0", features = ["tokio-comp"] }
|
||||
tokio = { version = "1.28.0", features = ["full"] }
|
||||
sal = { git = "https://git.ourworld.tf/herocode/sal.git", branch = "main" }
|
||||
chacha20poly1305 = "0.10.1"
|
||||
blake3 = "1.3.1"
|
||||
csv = "1.1"
|
||||
rand = "0.9.1"
|
||||
ipfs-api-backend-hyper = "0.6"
|
||||
ipfs-api = { version = "0.17.0", default-features = false, features = ["with-hyper-tls"] }
|
||||
87
webbuilder/src/builder/specs.md
Normal file
87
webbuilder/src/builder/specs.md
Normal file
@@ -0,0 +1,87 @@
|
||||
# Web Builder Specification
|
||||
|
||||
This document describes the process of building web metadata and exporting assets for a website, resulting in a `webmeta.json` file that can be used by a browser-based website generator.
|
||||
|
||||
## Overview
|
||||
|
||||
The web building process starts with a directory containing the site's Hjson configuration files, such as the example directory `/Users/despiegk/code/git.ourworld.tf/herocode/doctree/examples/doctreenew/sites/demo1`. These Hjson files define the structure and content of the entire site and may reference external collections. The Hjson configuration sits "on top" of the collections it utilizes. Using the metadata defined in these Hjson files, the necessary collection data is downloaded from Git repositories (if referenced). The `doctree` is then used to process the relevant data, identify pages and images, and prepare them for export to IPFS. Finally, a `webmeta.json` file is generated containing all the necessary information, including IPFS keys and Blake hashes for content verification, allowing a browser-based tool to render the website by fetching assets from IPFS. Optionally, the generated `webmeta.json` file can also be uploaded to IPFS, and its IPFS URL returned.
|
||||
|
||||
## Process Steps
|
||||
|
||||
1. **Start from Hjson Directory:**
|
||||
* The process begins with a designated directory containing the site's Hjson configuration files. This directory serves as the single input for the web building process.
|
||||
|
||||
2. **Parse Site Metadata (Hjson):**
|
||||
* Locate and parse all `.hjson` files within the input directory and its subdirectories (e.g., `pages`). These files collectively define the site's structure, content, and configuration, and may include references to external collections.
|
||||
|
||||
3. **Download Referenced Collections from Git:**
|
||||
* If the Hjson metadata references external collections hosted in Git repositories, download these collections using a separate tool or crate responsible for Git interactions. The Hjson files provide the necessary information (e.g., repository URLs, branch names) to perform these downloads.
|
||||
|
||||
4. **Process Site Content and Collections with Doctree:**
|
||||
* Utilize the `doctree` library to process the parsed site metadata and the content of any downloaded collections.
|
||||
* `doctree` will build the document tree based on the Hjson structure and identify relevant assets such as pages (e.g., Markdown files) and images referenced within the site configuration or collections.
|
||||
|
||||
5. **Export Assets to IPFS:**
|
||||
* Export the identified assets (pages, images, etc.) to IPFS.
|
||||
* For each exported asset, obtain its IPFS key (CID) and calculate its Blake hash for content integrity verification.
|
||||
|
||||
6. **Generate `webmeta.json`:**
|
||||
* Create a single `webmeta.json` file that consolidates all the necessary information for the browser-based generator.
|
||||
* This file should include:
|
||||
* Site-level metadata (from Hjson).
|
||||
* Structure of the website (pages, navigation, etc.).
|
||||
* For each page, include:
|
||||
* Page metadata (from Hjson).
|
||||
* The IPFS key of the page content.
|
||||
* The Blake hash of the page content.
|
||||
* Information about other assets (images, etc.), including their IPFS keys.
|
||||
|
||||
7. **Optional: Upload `webmeta.json` to IPFS:**
|
||||
* Optionally, upload the generated `webmeta.json` file to IPFS.
|
||||
* If uploaded, the IPFS URL of the `webmeta.json` file is returned as the output of the web building process.
|
||||
|
||||
8. **Utilize `webmeta.json` in Browser:**
|
||||
* The generated `webmeta.json` file (either locally or fetched from IPFS) serves as the single configuration entry point for a browser-based website generator.
|
||||
* The browser tool reads `webmeta.json`, uses the IPFS keys to fetch the content and assets from the IPFS network, and renders the website dynamically. The Blake hashes can be used to verify the integrity of the downloaded content.
|
||||
|
||||
## `webmeta.json` Structure (Example)
|
||||
|
||||
```json
|
||||
{
|
||||
"site_metadata": {
|
||||
// Consolidated data from site-level Hjson files (collection, header, footer, main, etc.)
|
||||
"name": "demo1",
|
||||
"title": "Demo Site 1",
|
||||
"description": "This is a demo site for doctree",
|
||||
"keywords": ["demo", "doctree", "example"],
|
||||
"header": { ... },
|
||||
"footer": { ... }
|
||||
},
|
||||
"pages": [
|
||||
{
|
||||
"id": "mypages1",
|
||||
"title": "My Pages 1",
|
||||
"ipfs_key": "Qm...", // IPFS key of the page content
|
||||
"blakehash": "sha256-...", // Blake hash of the page content
|
||||
"sections": [
|
||||
{ "type": "text", "content": "..." } // Potentially include some inline content or structure
|
||||
],
|
||||
"assets": [
|
||||
{
|
||||
"name": "image1.png",
|
||||
"ipfs_key": "Qm..." // IPFS key of an image used on the page
|
||||
}
|
||||
]
|
||||
}
|
||||
// Other pages...
|
||||
],
|
||||
"assets": {
|
||||
// Global assets not tied to a specific page, e.g., CSS, global images
|
||||
"style.css": {
|
||||
"ipfs_key": "Qm..."
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This structure is a suggestion and can be adapted based on the specific needs of the browser-based generator. The key is to include all necessary information (metadata, IPFS keys, hashes) to allow the browser to fetch and render the complete website.
|
||||
43
webbuilder/src/builder/webmeta.json
Normal file
43
webbuilder/src/builder/webmeta.json
Normal file
@@ -0,0 +1,43 @@
|
||||
{
|
||||
"site_metadata": {
|
||||
"name": "demo1",
|
||||
"title": "Demo Site 1",
|
||||
"description": "This is a demo site for doctree",
|
||||
"keywords": ["demo", "doctree", "example"],
|
||||
"header": {
|
||||
"logo": "/images/logo.png",
|
||||
"nav": [
|
||||
{ "text": "Home", "url": "/" },
|
||||
{ "text": "About", "url": "/about" }
|
||||
]
|
||||
},
|
||||
"footer": {
|
||||
"copyright": "© 2023 My Company",
|
||||
"links": [
|
||||
{ "text": "Privacy Policy", "url": "/privacy" }
|
||||
]
|
||||
}
|
||||
},
|
||||
"pages": [
|
||||
{
|
||||
"id": "mypages1",
|
||||
"title": "My Pages 1",
|
||||
"ipfs_key": "QmPlaceholderIpfsKey1",
|
||||
"blakehash": "sha256-PlaceholderBlakeHash1",
|
||||
"sections": [
|
||||
{ "type": "text", "content": "This is example content for My Pages 1." }
|
||||
],
|
||||
"assets": [
|
||||
{
|
||||
"name": "image1.png",
|
||||
"ipfs_key": "QmPlaceholderImageIpfsKey1"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"assets": {
|
||||
"style.css": {
|
||||
"ipfs_key": "QmPlaceholderCssIpfsKey1"
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user