lhumina_code/hero_books
21
0
Fork 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity Actions

HERO Books Refactor #74

New issue
Closed
opened 2026-03-01 15:15:52 +00:00 by despiegk · 3 comments
despiegk commented 2026-03-01 15:15:52 +00:00
Owner
Copy link

Meeting Minutes

Project: HERO Services / Books Refactor
Date: [Insert Date]
Participants:

  • Speaker 1: Kristof
  • Speaker 2: Mahmoud

1. ZInit & Service Management

1.1 Development Branch Setup

  • All services must be started via ZInit.

  • No more screen, tmux, or manual background processes.

  • Everything runs on Unix domain sockets only (no TCP ports), except:

    • hero_inspector
    • init itself

1.2 Embedder Shutdown Issue

Problem:

  • Embedder frequently fails with:

    “failed to open embedding cache – database already open, cannot acquire lock”

  • Root cause: unclean shutdown, leaving DB locked.

Action Required:

  • Ensure init stop performs:

    • Proper graceful shutdown
    • Sufficient timeout for embedded database flush

  • Prevent corrupted databases.

2. OpenRPC Architecture Standardization

2.1 Core Principle

All services must follow this structure:

LIB → SERVER → SDK → UI / Viewer / CLI

2.2 Layer Responsibilities

1️⃣ Lib (Core Logic)

  • Pure functionality only

  • No networking

  • Examples:

    • Books management
    • Collections
    • PDF generation
    • Embedder integration
    • Ontology
    • Import/export
    • HeroScript parsing

2️⃣ Server

  • Uses Lib

  • Exposes manual OpenRPC over HTTP on Unix socket

  • Single source of truth

  • Must NOT:

    • Bind TCP ports
    • Expose HTTP UI
    • Use raw sockets (must be HTTP-over-socket)

3️⃣ SDK

  • Uses OpenRPC client macro
  • Generated client + optional convenience functions
  • Talks only to Server

4️⃣ UI / Viewer

  • Uses SDK only
  • Must not call Lib directly
  • Must not bypass Server
  • Uses OpenRPC proxy macro for automatic routing

5️⃣ CLI

  • Uses SDK
  • For testing and automation

3. OpenRPC Proxy Macro

3.1 Objective

Avoid manual proxy implementations.

3.2 Requirement

  • Use OpenRPC client macro

  • Auto-generate proxy code

  • Inspector must:

    • Read OpenRPC schema
    • Display methods
    • Auto-expose UI

3.3 Current Status

  • Macro implementation improved
  • Some services still not exposing OpenRPC correctly
  • Inspector still partially broken

4. Removal of Old Schema-Generated Framework

Decision:

  • Remove oschema-generated and old auto-generation approach
  • Use manual OpenRPC servers
  • Too many breaking changes otherwise

5. Hero Books Refactor (Major Focus)

5.1 Current State

  • Large architectural mess
  • Mixed responsibilities
  • UI directly calling libraries
  • Generated server artifacts mixed into logic
  • Viewer replaced accidentally by admin UI

5.2 Required Target Structure

hero_books_lib
hero_books_server
hero_books_sdk
hero_books_ui_admin
hero_books_viewer
hero_books_cli

5.3 Functional Requirements

Book Management

Server must expose OpenRPC methods for:

  • Scan book
  • List books
  • Delete book
  • Add page
  • Resolve pages
  • Access metadata (TOML)
  • Generate PDF

Collection Management

  • List collections

  • Add collection

  • Remove collection

  • Add/remove pages

  • Generate/export collection

  • Scan from:

    • Path
    • URL

Context Model

  • Library

    • Contains Books
    • Contains Collections

  • Inspired by earlier structure (Library of Alexandria concept)

5.4 Viewer Restoration

Problem:

  • Previous rich books viewer lost.
  • Replaced with minimal admin interface.

Decision:

  • Restore original viewer functionality.

  • Rebuild using:

    • SDK
    • OpenRPC
    • Clean architecture

Admin UI = OpenRPC control panel
Viewer UI = End-user interface

6. Inspector Improvements

  • Automatically detects OpenRPC methods
  • Can auto-open website UI
  • Creates temporary port mappings automatically
  • Eliminates port management complexity

7. Cleanup Actions Required

  • Remove broken / obsolete crates

  • Move functionality properly into lib

  • Remove generated artifacts

  • Ensure:

    • No TCP ports
    • No direct library calls from UI
    • No raw socket RPC

8. Immediate Action Plan

Mahmoud

  • Refactor Hero Books architecture
  • Restore full viewer functionality
  • Implement complete OpenRPC surface
  • Clean Lib structure
  • Provide update by end of day (if possible)

Kristof

  • Update skills
  • Share extracted meeting transcript
  • Continue OpenRPC macro improvements

Final Architecture Goal

Unix Sockets Only
Manual OpenRPC
Macro-based Client Generation
Single Source of Truth in Server
Clean Lib Core
SDK as Gateway
UI & Viewer strictly via SDK
ZInit for all service lifecycle management
## Meeting Minutes **Project:** HERO Services / Books Refactor **Date:** [Insert Date] **Participants:** * **Speaker 1:** Kristof * **Speaker 2:** Mahmoud --- ## 1. ZInit & Service Management ### 1.1 Development Branch Setup * All services must be started via **ZInit**. * No more `screen`, `tmux`, or manual background processes. * Everything runs on **Unix domain sockets only** (no TCP ports), except: * `hero_inspector` * `init` itself ### 1.2 Embedder Shutdown Issue **Problem:** * Embedder frequently fails with: > “failed to open embedding cache – database already open, cannot acquire lock” * Root cause: **unclean shutdown**, leaving DB locked. **Action Required:** * Ensure `init stop` performs: * Proper graceful shutdown * Sufficient timeout for embedded database flush * Prevent corrupted databases. --- ## 2. OpenRPC Architecture Standardization ### 2.1 Core Principle All services must follow this structure: ``` LIB → SERVER → SDK → UI / Viewer / CLI ``` ### 2.2 Layer Responsibilities #### 1️⃣ Lib (Core Logic) * Pure functionality only * No networking * Examples: * Books management * Collections * PDF generation * Embedder integration * Ontology * Import/export * HeroScript parsing #### 2️⃣ Server * Uses Lib * Exposes **manual OpenRPC over HTTP on Unix socket** * Single source of truth * Must NOT: * Bind TCP ports * Expose HTTP UI * Use raw sockets (must be HTTP-over-socket) #### 3️⃣ SDK * Uses OpenRPC client macro * Generated client + optional convenience functions * Talks only to Server #### 4️⃣ UI / Viewer * Uses SDK only * Must not call Lib directly * Must not bypass Server * Uses OpenRPC proxy macro for automatic routing #### 5️⃣ CLI * Uses SDK * For testing and automation --- ## 3. OpenRPC Proxy Macro ### 3.1 Objective Avoid manual proxy implementations. ### 3.2 Requirement * Use OpenRPC client macro * Auto-generate proxy code * Inspector must: * Read OpenRPC schema * Display methods * Auto-expose UI ### 3.3 Current Status * Macro implementation improved * Some services still not exposing OpenRPC correctly * Inspector still partially broken --- ## 4. Removal of Old Schema-Generated Framework Decision: * Remove `oschema-generated` and old auto-generation approach * Use **manual OpenRPC servers** * Too many breaking changes otherwise --- ## 5. Hero Books Refactor (Major Focus) ### 5.1 Current State * Large architectural mess * Mixed responsibilities * UI directly calling libraries * Generated server artifacts mixed into logic * Viewer replaced accidentally by admin UI ### 5.2 Required Target Structure ``` hero_books_lib hero_books_server hero_books_sdk hero_books_ui_admin hero_books_viewer hero_books_cli ``` --- ### 5.3 Functional Requirements #### Book Management Server must expose OpenRPC methods for: * Scan book * List books * Delete book * Add page * Resolve pages * Access metadata (TOML) * Generate PDF #### Collection Management * List collections * Add collection * Remove collection * Add/remove pages * Generate/export collection * Scan from: * Path * URL #### Context Model * Library * Contains Books * Contains Collections * Inspired by earlier structure (Library of Alexandria concept) --- ### 5.4 Viewer Restoration **Problem:** * Previous rich books viewer lost. * Replaced with minimal admin interface. **Decision:** * Restore original viewer functionality. * Rebuild using: * SDK * OpenRPC * Clean architecture Admin UI = OpenRPC control panel Viewer UI = End-user interface --- ## 6. Inspector Improvements * Automatically detects OpenRPC methods * Can auto-open website UI * Creates temporary port mappings automatically * Eliminates port management complexity --- ## 7. Cleanup Actions Required * Remove broken / obsolete crates * Move functionality properly into `lib` * Remove generated artifacts * Ensure: * No TCP ports * No direct library calls from UI * No raw socket RPC --- ## 8. Immediate Action Plan ### Mahmoud * Refactor Hero Books architecture * Restore full viewer functionality * Implement complete OpenRPC surface * Clean Lib structure * Provide update by end of day (if possible) ### Kristof * Update skills * Share extracted meeting transcript * Continue OpenRPC macro improvements --- # Final Architecture Goal ``` Unix Sockets Only Manual OpenRPC Macro-based Client Generation Single Source of Truth in Server Clean Lib Core SDK as Gateway UI & Viewer strictly via SDK ZInit for all service lifecycle management ```
👍 1
despiegk added this to the ACTIVE project 2026-03-01 15:16:06 +00:00
mahmoud commented 2026-03-01 23:14:58 +00:00
Owner
Copy link

Quick update on the hero_books refactor — made solid progress today:

  • Created hero_books_lib as a proper crate (all pure logic: ai, book, doctree, pdf, embedder, vectors)
  • Extracted all server infrastructure into hero_books_server — now depends on hero_books_lib only, no more root monolith
  • Added 7 missing OpenRPC methods (promoted books.list, books.get, books.getPages, search.query from legacy + new books.pdf, collections.add, collections.remove)
  • Gutted the root crate to a thin re-export
  • Cleaned up ~56k lines of dead code and clutter from the repo

All 7 workspace crates compile cleanly. Work is pushed to development.

Still in progress, though — there may be some rough edges and potential bugs, especially around the new RPC handlers. Will continue tomorrow.

Let me know if anything looks off on your end!

Quick update on the hero_books refactor — made solid progress today: - Created hero_books_lib as a proper crate (all pure logic: ai, book, doctree, pdf, embedder, vectors) - Extracted all server infrastructure into hero_books_server — now depends on hero_books_lib only, no more root monolith - Added 7 missing OpenRPC methods (promoted books.list, books.get, books.getPages, search.query from legacy + new books.pdf, collections.add, collections.remove) - Gutted the root crate to a thin re-export - Cleaned up ~56k lines of dead code and clutter from the repo All 7 workspace crates compile cleanly. Work is pushed to development. Still in progress, though — there may be some rough edges and potential bugs, especially around the new RPC handlers. Will continue tomorrow. Let me know if anything looks off on your end!
mik-tf commented 2026-03-02 14:36:59 +00:00
Owner
Copy link

Previous UI

  • Top navbar

    • Libraries
    • Tools
    • Admin

  • Tools

    • Import (as is now)
    • Markdown to PDF (via link or file)

  • Libraries

    • click on it, then see all the libraries
    • click on a library
      • then see the books one after the other
      • can ask the librarian (aka search): vocally or in writing
      • then it shows the AI summary, and the files where it found the info, with relevancy (0 to 100)
      • on the right we have the filters, can filter by categories, or by books

  • The books list had extensive options:

    • Download pdf, view pdf, view markdown, check versions (based on hash)

Hints

  • Go back in previous commits of hero_books to see the UI with these features
  • OR check archipelagos island hero_books, we have all this UI in wasm. But it might be in a branch if my PR were not merged yet.
# Previous UI - Top navbar - Libraries - Tools - Admin - Tools - Import (as is now) - Markdown to PDF (via link or file) - Libraries - click on it, then see all the libraries - click on a library - then see the books one after the other - can ask the librarian (aka search): vocally or in writing - then it shows the AI summary, and the files where it found the info, with relevancy (0 to 100) - on the right we have the filters, can filter by categories, or by books - The books list had extensive options: - Download pdf, view pdf, view markdown, check versions (based on hash) # Hints - Go back in previous commits of hero_books to see the UI with these features - OR check archipelagos island hero_books, we have all this UI in wasm. But it might be in a branch if my PR were not merged yet.
mahmoud commented 2026-03-09 10:14:49 +00:00
Owner
Copy link

Done: The work has been merged into development. Please reopen if you still have concerns

Done: The work has been merged into development. Please reopen if you still have concerns
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
development
development_docs_rpc_methods
development_mik_nu_demo
development_expose_docusaurus_openrpc_async_jobs
development_mik2
main
v0.1.5
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
v0.0.1-rc1
Labels
Clear labels   
prio_critical

   
prio_low

   
type_bug

   
type_contact

   
type_issue

   
type_lead

   
type_question

   
type_story

   
type_task
No labels
prio_critical
prio_low
type_bug
type_contact
type_issue
type_lead
type_question
type_story
type_task
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
ACTIVE
Assignees
Clear assignees
No assignees
mahmoud
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
lhumina_code/hero_books#74
No description provided.