Expose docusaurus generation over OpenRPC as async jobs #92

New issue

Closed

opened 2026-04-19 10:48:48 +00:00 by mahmoud · 4 comments

mahmoud commented 2026-04-19 10:48:48 +00:00

Owner

Copy link

Goal

Bring docusaurus site generation into hero_books_server so it can be invoked over JSON-RPC from any Hero service or UI, not only from the standalone hero_docs CLI. Long-running generation must be async — the call returns a job ID immediately, and a separate method reports status.

Motivation

Today hero_docs is a standalone binary that blocks on generation. Making this a server capability means:

• Other services and UIs can trigger a build without spawning a CLI.
• Multiple books can be generated in parallel as background jobs.
• The same cache-by-hash pattern used by books.pdf can apply to docusaurus output.

Scope

Dependencies

• Add hero_books_docusaurus to crates/hero_books_server/Cargo.toml.

New OpenRPC methods

Add to crates/hero_books_server/openrpc.json:

• docs.new — scaffold a new docusaurus site. Params: name, path, optional force. Returns: { job_id }.
• docs.generate — generate a docusaurus site from an existing book (by book id or path). Returns: { job_id }.
• docs.jobStatus — poll a job. Params: { job_id }. Returns: { state: pending|running|done|failed, output_path?, error? }.

Job registry

• Simple in-process registry on the server's app state: Arc<Mutex<HashMap<JobId, JobState>>>.
• Each job spawns via tokio::spawn and updates state on completion or failure.
• JobId is a UUID string.
• Cache docusaurus output by input hash (mirror books.pdf caching) so repeated calls with identical inputs are cheap.

Handlers

• Implement handlers in crates/hero_books_server/src/web/rpc.rs alongside handle_books_pdf (~line 1249). Dispatch entries go in the same switch block.

The standalone CLI

• Keep hero_docs as-is. It calls hero_books_docusaurus directly — the server is an additional entry point, not a replacement.

Acceptance criteria

• docs.new and docs.generate return a job id within milliseconds.
• docs.jobStatus reflects running while work is in flight and done with output_path on success.
• Failures surface as state: failed with an error string; the server does not panic.
• Two concurrent docs.generate calls for different inputs run in parallel.
• Two concurrent calls for the same input either share a job or return the cached result — document which.
• OpenRPC spec and examples are updated in openrpc.json.
• Smoke test added that exercises docs.new → docs.jobStatus polling loop.

Non-goals

• Persisting job state across server restarts. In-memory is fine for now.
• A separate job worker process. One tokio::spawn per job is enough at this stage.
• Authentication/authorization changes — the method follows whatever auth pattern the existing books.* methods use.

Dependency order

Lands after #1 (fix --path nesting) so the server exposes correct scaffold behavior. Can land in parallel with #3 in hero_skills (installer work).

## Goal Bring docusaurus site generation into `hero_books_server` so it can be invoked over JSON-RPC from any Hero service or UI, not only from the standalone `hero_docs` CLI. Long-running generation must be async — the call returns a job ID immediately, and a separate method reports status. ## Motivation Today `hero_docs` is a standalone binary that blocks on generation. Making this a server capability means: - Other services and UIs can trigger a build without spawning a CLI. - Multiple books can be generated in parallel as background jobs. - The same cache-by-hash pattern used by `books.pdf` can apply to docusaurus output. ## Scope ### Dependencies - [ ] Add `hero_books_docusaurus` to `crates/hero_books_server/Cargo.toml`. ### New OpenRPC methods Add to `crates/hero_books_server/openrpc.json`: - [ ] `docs.new` — scaffold a new docusaurus site. Params: `name`, `path`, optional `force`. Returns: `{ job_id }`. - [ ] `docs.generate` — generate a docusaurus site from an existing book (by book id or path). Returns: `{ job_id }`. - [ ] `docs.jobStatus` — poll a job. Params: `{ job_id }`. Returns: `{ state: pending|running|done|failed, output_path?, error? }`. ### Job registry - [ ] Simple in-process registry on the server's app state: `Arc<Mutex<HashMap<JobId, JobState>>>`. - [ ] Each job spawns via `tokio::spawn` and updates state on completion or failure. - [ ] JobId is a UUID string. - [ ] Cache docusaurus output by input hash (mirror `books.pdf` caching) so repeated calls with identical inputs are cheap. ### Handlers - [ ] Implement handlers in `crates/hero_books_server/src/web/rpc.rs` alongside `handle_books_pdf` (~line 1249). Dispatch entries go in the same switch block. ### The standalone CLI - [ ] Keep `hero_docs` as-is. It calls `hero_books_docusaurus` directly — the server is an additional entry point, not a replacement. ## Acceptance criteria - [ ] `docs.new` and `docs.generate` return a job id within milliseconds. - [ ] `docs.jobStatus` reflects `running` while work is in flight and `done` with `output_path` on success. - [ ] Failures surface as `state: failed` with an `error` string; the server does not panic. - [ ] Two concurrent `docs.generate` calls for different inputs run in parallel. - [ ] Two concurrent calls for the same input either share a job or return the cached result — document which. - [ ] OpenRPC spec and examples are updated in `openrpc.json`. - [ ] Smoke test added that exercises `docs.new` → `docs.jobStatus` polling loop. ## Non-goals - Persisting job state across server restarts. In-memory is fine for now. - A separate job worker process. One `tokio::spawn` per job is enough at this stage. - Authentication/authorization changes — the method follows whatever auth pattern the existing `books.*` methods use. ## Dependency order Lands after #1 (fix `--path` nesting) so the server exposes correct scaffold behavior. Can land in parallel with #3 in `hero_skills` (installer work).

fatmaebrahim commented 2026-04-20 10:43:59 +00:00

Member

Copy link

Implementation Spec for Issue #92

Objective

Add three new JSON-RPC methods (docs.new, docs.generate, docs.jobStatus) to hero_books_server that expose hero_books_docusaurus scaffolding and site generation as asynchronous background jobs. Callers receive a job ID immediately and poll for completion, mirroring the existing import_jobs() pattern.

Requirements

• docs.new accepts name, path, and optional force; spawns scaffold + full generation in background; returns { job_id } within milliseconds
• docs.generate accepts path (heroscript path or book identifier); spawns docusaurus generation in background; returns { job_id }
• docs.jobStatus accepts { job_id }; returns { state: "pending"|"running"|"done"|"failed", output_path?, error? }
• Jobs run via std::thread::spawn (docusaurus APIs are synchronous/CPU-bound, dispatch is synchronous)
• Two concurrent jobs for different inputs run in parallel
• Two concurrent jobs for the same input hash share the existing job (return existing job_id)
• Failures captured as state: "failed" with error string; server never panics
• Cache docusaurus output by input hash under .docusaurus_cache/
• OpenRPC spec updated with three new method definitions
• Smoke test exercises docs.new -> docs.jobStatus polling

Files to Modify

File	Action	Description
crates/hero_books_server/Cargo.toml	Modify	Add hero_books_docusaurus dependency
crates/hero_books_server/src/web/server.rs	Modify	Add DocsJobStatus enum, DocsJob struct, docs_jobs() registry, cache dir helper, input hash computation, running job dedup
crates/hero_books_server/src/web/rpc.rs	Modify	Add dispatch entries and three handler functions
crates/hero_books_server/src/web/mod.rs	Modify	Re-export new public items
crates/hero_books_server/openrpc.json	Modify	Add docs.new, docs.generate, docs.jobStatus method definitions
crates/hero_books_server/src/web/rpc_spec.rs	Modify	Add typed request/response structs, update inline OpenRPC schema

Implementation Plan

Step 1: Add hero_books_docusaurus dependency

Files: crates/hero_books_server/Cargo.toml

• Add hero_books_docusaurus = { path = "../hero_books_docusaurus" } to [dependencies]
  Dependencies: none

Step 2: Add docs job registry and cache helpers to server.rs

Files: crates/hero_books_server/src/web/server.rs

• Add DocsJobStatus enum (Pending/Running/Done/Failed)
• Add DocsJob struct (state, output_path, error, input_hash)
• Add docs_jobs() global registry (follows import_jobs() pattern)
• Add get_docusaurus_cache_dir() (follows get_pdf_cache_dir() pattern)
• Add calculate_docs_input_hash() and find_running_job_for_hash() helpers
  Dependencies: Step 1

Step 3: Add dispatch entries and handler functions to rpc.rs

Files: crates/hero_books_server/src/web/rpc.rs

• Add dispatch entries for docs.new, docs.generate, docs.jobStatus in the match block
• Implement handle_docs_new(): parse params, compute hash, dedup check, spawn thread for scaffold + generate
• Implement handle_docs_generate(): parse params, compute hash, dedup check, spawn thread for generation
• Implement handle_docs_job_status(): look up job, return state
  Dependencies: Step 2

Step 4: Update mod.rs re-exports

Files: crates/hero_books_server/src/web/mod.rs

• Add new public items to the pub use server::{ ... } block
  Dependencies: Step 2

Step 5: Update openrpc.json with new method definitions

Files: crates/hero_books_server/openrpc.json

• Add three method entries with params and result schemas
  Dependencies: none

Step 6: Add typed structs to rpc_spec.rs and update inline schema

Files: crates/hero_books_server/src/web/rpc_spec.rs

• Add request/response structs for docs methods
• Update get_openrpc_schema() inline JSON
  Dependencies: none

Step 7: Add smoke test

Files: crates/hero_books_server/src/web/server.rs, crates/hero_books_server/src/web/rpc.rs

• Test docs job registry mechanics (insert, state transitions)
• Integration test: docs.new -> docs.jobStatus via handle_rpc_request
  Dependencies: Steps 2, 3

Acceptance Criteria

• hero_books_docusaurus is a dependency of hero_books_server
• docs.new returns { job_id } within milliseconds (non-blocking)
• docs.generate returns { job_id } within milliseconds (non-blocking)
• docs.jobStatus returns correct state transitions (pending -> running -> done/failed)
• Failures surface as { state: "failed", error: "..." }; server does not panic
• Two concurrent calls for different inputs run in parallel
• Two concurrent calls for same input hash share a job (return existing job_id)
• openrpc.json contains three new method definitions
• Smoke test passes
• cargo build succeeds with no new warnings
• Existing tests continue to pass

Notes

• The handle_rpc_request function is synchronous (called inside spawn_blocking). Job handlers use std::thread::spawn for background work.
• Cache by input hash under .docusaurus_cache/{hash}/. If cache exists with build/ subdirectory, skip regeneration.
• Jobs are never evicted from the in-memory registry (matches import_jobs() behavior).
• Uses parking_lot::Mutex<HashMap<...>> for thread safety (same as import_jobs()).
• uuid crate with v4 feature will be added for job ID generation.

## Implementation Spec for Issue #92 ### Objective Add three new JSON-RPC methods (`docs.new`, `docs.generate`, `docs.jobStatus`) to `hero_books_server` that expose `hero_books_docusaurus` scaffolding and site generation as asynchronous background jobs. Callers receive a job ID immediately and poll for completion, mirroring the existing `import_jobs()` pattern. ### Requirements - `docs.new` accepts `name`, `path`, and optional `force`; spawns scaffold + full generation in background; returns `{ job_id }` within milliseconds - `docs.generate` accepts `path` (heroscript path or book identifier); spawns docusaurus generation in background; returns `{ job_id }` - `docs.jobStatus` accepts `{ job_id }`; returns `{ state: "pending"|"running"|"done"|"failed", output_path?, error? }` - Jobs run via `std::thread::spawn` (docusaurus APIs are synchronous/CPU-bound, dispatch is synchronous) - Two concurrent jobs for different inputs run in parallel - Two concurrent jobs for the same input hash share the existing job (return existing job_id) - Failures captured as `state: "failed"` with `error` string; server never panics - Cache docusaurus output by input hash under `.docusaurus_cache/` - OpenRPC spec updated with three new method definitions - Smoke test exercises `docs.new` -> `docs.jobStatus` polling ### Files to Modify | File | Action | Description | |------|--------|-------------| | `crates/hero_books_server/Cargo.toml` | Modify | Add `hero_books_docusaurus` dependency | | `crates/hero_books_server/src/web/server.rs` | Modify | Add `DocsJobStatus` enum, `DocsJob` struct, `docs_jobs()` registry, cache dir helper, input hash computation, running job dedup | | `crates/hero_books_server/src/web/rpc.rs` | Modify | Add dispatch entries and three handler functions | | `crates/hero_books_server/src/web/mod.rs` | Modify | Re-export new public items | | `crates/hero_books_server/openrpc.json` | Modify | Add `docs.new`, `docs.generate`, `docs.jobStatus` method definitions | | `crates/hero_books_server/src/web/rpc_spec.rs` | Modify | Add typed request/response structs, update inline OpenRPC schema | ### Implementation Plan #### Step 1: Add `hero_books_docusaurus` dependency Files: `crates/hero_books_server/Cargo.toml` - Add `hero_books_docusaurus = { path = "../hero_books_docusaurus" }` to `[dependencies]` Dependencies: none #### Step 2: Add docs job registry and cache helpers to `server.rs` Files: `crates/hero_books_server/src/web/server.rs` - Add `DocsJobStatus` enum (Pending/Running/Done/Failed) - Add `DocsJob` struct (state, output_path, error, input_hash) - Add `docs_jobs()` global registry (follows `import_jobs()` pattern) - Add `get_docusaurus_cache_dir()` (follows `get_pdf_cache_dir()` pattern) - Add `calculate_docs_input_hash()` and `find_running_job_for_hash()` helpers Dependencies: Step 1 #### Step 3: Add dispatch entries and handler functions to `rpc.rs` Files: `crates/hero_books_server/src/web/rpc.rs` - Add dispatch entries for `docs.new`, `docs.generate`, `docs.jobStatus` in the match block - Implement `handle_docs_new()`: parse params, compute hash, dedup check, spawn thread for scaffold + generate - Implement `handle_docs_generate()`: parse params, compute hash, dedup check, spawn thread for generation - Implement `handle_docs_job_status()`: look up job, return state Dependencies: Step 2 #### Step 4: Update `mod.rs` re-exports Files: `crates/hero_books_server/src/web/mod.rs` - Add new public items to the `pub use server::{ ... }` block Dependencies: Step 2 #### Step 5: Update `openrpc.json` with new method definitions Files: `crates/hero_books_server/openrpc.json` - Add three method entries with params and result schemas Dependencies: none #### Step 6: Add typed structs to `rpc_spec.rs` and update inline schema Files: `crates/hero_books_server/src/web/rpc_spec.rs` - Add request/response structs for docs methods - Update `get_openrpc_schema()` inline JSON Dependencies: none #### Step 7: Add smoke test Files: `crates/hero_books_server/src/web/server.rs`, `crates/hero_books_server/src/web/rpc.rs` - Test docs job registry mechanics (insert, state transitions) - Integration test: `docs.new` -> `docs.jobStatus` via `handle_rpc_request` Dependencies: Steps 2, 3 ### Acceptance Criteria - [ ] `hero_books_docusaurus` is a dependency of `hero_books_server` - [ ] `docs.new` returns `{ job_id }` within milliseconds (non-blocking) - [ ] `docs.generate` returns `{ job_id }` within milliseconds (non-blocking) - [ ] `docs.jobStatus` returns correct state transitions (pending -> running -> done/failed) - [ ] Failures surface as `{ state: "failed", error: "..." }`; server does not panic - [ ] Two concurrent calls for different inputs run in parallel - [ ] Two concurrent calls for same input hash share a job (return existing job_id) - [ ] `openrpc.json` contains three new method definitions - [ ] Smoke test passes - [ ] `cargo build` succeeds with no new warnings - [ ] Existing tests continue to pass ### Notes - The `handle_rpc_request` function is synchronous (called inside `spawn_blocking`). Job handlers use `std::thread::spawn` for background work. - Cache by input hash under `.docusaurus_cache/{hash}/`. If cache exists with `build/` subdirectory, skip regeneration. - Jobs are never evicted from the in-memory registry (matches `import_jobs()` behavior). - Uses `parking_lot::Mutex<HashMap<...>>` for thread safety (same as `import_jobs()`). - `uuid` crate with `v4` feature will be added for job ID generation.

fatmaebrahim commented 2026-04-20 11:25:30 +00:00

Member

Copy link

Test Results

• Total: 11
• Passed: 11
• Failed: 0
• Ignored: 1 (doc-test, pre-existing)

All tests pass, including the 2 new docs job tests:

• test_docs_job_registry -- verifies job state transitions (Pending -> Running -> Done)
• test_docs_job_dedup -- verifies duplicate job detection by input hash

Build compiles cleanly with no warnings.

## Test Results - Total: 11 - Passed: 11 - Failed: 0 - Ignored: 1 (doc-test, pre-existing) All tests pass, including the 2 new docs job tests: - `test_docs_job_registry` -- verifies job state transitions (Pending -> Running -> Done) - `test_docs_job_dedup` -- verifies duplicate job detection by input hash Build compiles cleanly with no warnings.

fatmaebrahim commented 2026-04-20 11:27:19 +00:00

Member

Copy link

Implementation Summary

Changes Made

Modified files:

• crates/hero_books_server/Cargo.toml -- Added hero_books_docusaurus (path dependency) and uuid (v1, v4 feature) dependencies.
• crates/hero_books_server/src/web/server.rs -- Added DocsJobState enum, DocsJob struct, docs_jobs() global registry (follows import_jobs() pattern), get_docusaurus_cache_dir(), calculate_docs_input_hash(), and find_running_docs_job() helpers. Added 2 unit tests.
• crates/hero_books_server/src/web/rpc.rs -- Added dispatch entries for docs.new, docs.generate, docs.jobStatus. Implemented handle_docs_new(), handle_docs_generate(), and handle_docs_job_status() handler functions with async job spawning via std::thread::spawn.
• crates/hero_books_server/src/web/mod.rs -- Added re-exports for all new public items.
• crates/hero_books_server/openrpc.json -- Added three new method definitions with params and result schemas.
• crates/hero_books_server/src/web/rpc_spec.rs -- Added DocsNewRequest, DocsGenerateRequest, DocsJobStatusRequest, DocsJobStatusResponse structs. Updated inline get_openrpc_schema() with the three new methods.

Key Design Decisions

• Jobs use std::thread::spawn since handle_rpc_request is synchronous (runs inside spawn_blocking).
• Job dedup: concurrent calls for the same input hash return the existing job ID.
• Cache: output is cached under .docusaurus_cache/{hash}/. If a cached build/ directory exists, a synthetic "done" job is returned immediately.
• Jobs are never evicted from the in-memory registry, matching import_jobs() behavior.
• UUID v4 used for job IDs.

Test Results

• 11 tests passed, 0 failed
• 2 new tests: test_docs_job_registry, test_docs_job_dedup
• Build compiles cleanly with no warnings

## Implementation Summary ### Changes Made **Modified files:** - `crates/hero_books_server/Cargo.toml` -- Added `hero_books_docusaurus` (path dependency) and `uuid` (v1, v4 feature) dependencies. - `crates/hero_books_server/src/web/server.rs` -- Added `DocsJobState` enum, `DocsJob` struct, `docs_jobs()` global registry (follows `import_jobs()` pattern), `get_docusaurus_cache_dir()`, `calculate_docs_input_hash()`, and `find_running_docs_job()` helpers. Added 2 unit tests. - `crates/hero_books_server/src/web/rpc.rs` -- Added dispatch entries for `docs.new`, `docs.generate`, `docs.jobStatus`. Implemented `handle_docs_new()`, `handle_docs_generate()`, and `handle_docs_job_status()` handler functions with async job spawning via `std::thread::spawn`. - `crates/hero_books_server/src/web/mod.rs` -- Added re-exports for all new public items. - `crates/hero_books_server/openrpc.json` -- Added three new method definitions with params and result schemas. - `crates/hero_books_server/src/web/rpc_spec.rs` -- Added `DocsNewRequest`, `DocsGenerateRequest`, `DocsJobStatusRequest`, `DocsJobStatusResponse` structs. Updated inline `get_openrpc_schema()` with the three new methods. ### Key Design Decisions - Jobs use `std::thread::spawn` since `handle_rpc_request` is synchronous (runs inside `spawn_blocking`). - Job dedup: concurrent calls for the same input hash return the existing job ID. - Cache: output is cached under `.docusaurus_cache/{hash}/`. If a cached `build/` directory exists, a synthetic "done" job is returned immediately. - Jobs are never evicted from the in-memory registry, matching `import_jobs()` behavior. - UUID v4 used for job IDs. ### Test Results - 11 tests passed, 0 failed - 2 new tests: `test_docs_job_registry`, `test_docs_job_dedup` - Build compiles cleanly with no warnings

fatmaebrahim referenced this issue from a commit 2026-04-20 11:31:23 +00:00

feat(server): expose docusaurus generation over OpenRPC as async jobs

fatmaebrahim referenced this issue 2026-04-20 11:32:06 +00:00

feat(server): expose docusaurus generation over OpenRPC as async jobs #94

fatmaebrahim commented 2026-04-20 11:32:37 +00:00

Member

Copy link

Pull request opened: #94

This PR implements the changes discussed in this issue.

Pull request opened: https://forge.ourworld.tf/lhumina_code/hero_books/pulls/94 This PR implements the changes discussed in this issue.

fatmaebrahim referenced this issue from a commit 2026-04-20 11:59:14 +00:00

fix(server): improve docs job dedup to cover completed jobs

mahmoud referenced this issue 2026-04-20 12:09:34 +00:00

feat(server): expose docusaurus generation over OpenRPC as async jobs #94

fatmaebrahim referenced this issue from a commit 2026-04-20 12:23:07 +00:00

fix(server): address review feedback on docs API

fatmaebrahim referenced this issue from a commit 2026-04-20 12:37:28 +00:00

fix(server): address review feedback and add comprehensive tests

fatmaebrahim referenced this issue from a commit 2026-04-20 12:49:46 +00:00

style(server): apply cargo fmt formatting

mahmoud self-assigned this 2026-04-20 15:48:09 +00:00

fatmaebrahim was assigned by mahmoud 2026-04-20 15:48:14 +00:00

mahmoud added this to the ACTIVE project 2026-04-20 15:48:17 +00:00

mahmoud added this to the now milestone 2026-04-20 15:48:20 +00:00

mahmoud referenced this issue from a commit 2026-04-20 15:55:33 +00:00

feat(server): expose docusaurus generation over OpenRPC as async jobs (#94)

mahmoud closed this issue 2026-04-20 15:55:33 +00:00

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

development

development_docs_rpc_methods

development_mik_nu_demo

development_expose_docusaurus_openrpc_async_jobs

development_mik2

main

v0.1.5

v0.1.4

v0.1.3

v0.1.2

v0.1.1

v0.1.0

v0.0.1-rc1

Labels

Clear labels   

prio_critical

  

prio_low

  

type_bug

  

type_contact

  

type_issue

  

type_lead

  

type_question

  

type_story

  

type_task

No labels

prio_critical

prio_low

type_bug

type_contact

type_issue

type_lead

type_question

type_story

type_task

Milestone

Clear milestone

No items

No milestone

now

Projects

Clear projects

No items

No project

ACTIVE

Assignees

Clear assignees

No assignees

mahmoud

fatmaebrahim

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

lhumina_code/hero_books#92

No description provided.