weynandkuijpers/docs_localcluster
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
docs_localcluster/requirements.md
Weynand Kuijpers 5ea754d048 update
2025-10-27 15:48:07 +04:00

11 KiB
Raw Permalink Blame History

Product Specification Document: ThreeFold Zero-OS 3-Node Redundant Cluster (WIP)

1. Document Overview

1.1 Purpose

This document outlines the requirements, functional specifications, and technical specifications for a 3-node redundant cluster product built on ThreeFold's zero-OS. The cluster leverages k3s (lightweight Kubernetes) for orchestration and redundancy, SeaweedFS for distributed storage, and integrates application services including Nextcloud for file management, Matrix for instant messaging, LiveKit for video conferencing, and a custom paraphrasing AI for local data processing and storage. Nodes are interconnected via the ThreeFold Mycelium network for secure, decentralized communication.

The product targets small-to-medium organizations or edge deployments requiring self-sovereign, privacy-focused collaboration tools with built-in redundancy and local AI capabilities.

1.2 Scope

  • In Scope: Hardware/software requirements, cluster architecture, service integrations, deployment procedures, and basic monitoring.
  • Out of Scope: Custom hardware fabrication, advanced AI model training (uses pre-trained models), integration with external cloud providers, and ongoing maintenance/support contracts.
  • Assumptions: Users have access to ThreeFold Grid for node provisioning; basic Linux/Kubernetes knowledge for initial setup.

1.3 Version History

Version Date Author Changes
0.1 2025-10-26 Weynand Initial draft

2. Requirements

2.1 Business Requirements

  • BR-01: Provide a turnkey, self-hosted collaboration platform that ensures data sovereignty and redundancy across 3 nodes to minimize single points of failure.
  • BR-02: Support up to 50 concurrent users for file storage/sharing, instant messaging, and video conferencing without external dependencies.
  • BR-03: Enable local AI-driven insights (e.g., paraphrasing/summarization) from video conferences, storing all data on-cluster for privacy compliance (e.g., GDPR).
  • BR-04: Integrate seamlessly with the ThreeFold ecosystem for scalable, secure node networking via Mycelium.

2.2 User Requirements

  • UR-01: Users can upload, share, and manage files via a web-based Nextcloud interface with automatic redundancy.
  • UR-02: Enable real-time text/voice chat via Matrix clients (e.g., Element) with end-to-end encryption.
  • UR-03: Host video conferences (up to 10 participants) using LiveKit, with AI-generated summaries/paraphrases available post-session.
  • UR-04: Administrators can monitor cluster health, scale storage, and backup data via a unified dashboard.

2.3 System Requirements

2.3.1 Hardware Requirements (Per Node)

Component Minimum Specification Recommended
CPU 4-core ARM64/x86_64 8-core
RAM 8 GB 16 GB
Storage 128 GB NVMe SSD (for OS) + 1 TB HDD/SSD (for data) 256 GB SSD + 2 TB NVMe
Network 1 Gbps Ethernet, Wi-Fi 6 optional 10 Gbps
Power Standard 5V/12V DC, UPS recommended Standard 5V/12V DC, UPS
  • Total Cluster: 2 identical nodes; minimum aggregate: 12 cores, 24 GB RAM, 3 TB storage. One mode powerfull node on which the local AI model can run and paraphrasing and summarization.
  • Redundancy: Seaweed FS provides erasure coding based datasecurity across nodes for critical data; power/network failover via k3s.

2.3.2 Software Requirements

  • OS: ThreeFold zero-OS v3.x (container-native, minimal footprint).
  • Orchestration: k3s v1.28+ (lightweight Kubernetes for edge).
  • Storage: SeaweedFS v3.5+ (distributed file system with erasure coding).
  • Applications:
    • Nextcloud v28+.
    • Matrix Synapse v1.100+.
    • LiveKit v1.4+.
    • Paraphrasing AI: Custom container using Hugging Face Transformers (e.g., T5 model for summarization) + Whisper for transcription.
  • Networking: ThreeFold Mycelium (VPN-like overlay for secure P2P).
  • Dependencies: Docker (embedded in zero-OS), Helm for k3s charts.

2.3.3 Network Requirements

  • Connectivity: Nodes must be on the ThreeFold Grid; Mycelium provides zero-trust encryption.
  • Ports: Expose 80/443 (HTTPS), 8443 (Matrix/LiveKit), 19302 (WebRTC UDP).
  • Bandwidth: Minimum 100 Mbps symmetric per node; QoS for video traffic.

2.3.4 Non-Functional Requirements

  • Performance: <5s file upload (1 GB), <2s video join time, 99.9% uptime.
  • Security: TLS 1.3 everywhere; E2EE for Matrix/LiveKit; RBAC via k3s.
  • Scalability: Horizontal scaling to 5 nodes; storage auto-expansion via SeaweedFS.
  • Reliability: Erasue coding based storage for data; self-healing via k3s.
  • Compliance: Data stored locally; audit logs for all services. Optional blockchain to record log entries.

3. Functional Specification

3.1 High-Level Features

The cluster provides a unified platform for collaboration, with redundancy ensured by k3s scheduling across nodes.

Feature ID Feature Name Description Priority
F-01 Redundant File Storage Files stored via SeaweedFS; accessible via Nextcloud UI. High
F-02 File Management CRUD operations (create, read, update, delete) files/folders; sharing links. High
F-03 Instant Messaging Matrix-based chat rooms, 1:1 messaging, file attachments; bridges to other mail and instant messaging services (TBD). Medium
F-04 Video Conferencing LiveKit rooms for audio/video calls; screen sharing; recording (local only). High
F-05 AI Paraphrasing Post-conference: Transcribe audio → Paraphrase/summarize → Store metadata locally. High
F-06 Secure Node Connectivity Mycelium overlay for inter-node traffic; auto-discovery on ThreeFold Grid. High
F-07 Admin Dashboard k3s-integrated UI for monitoring pods, storage usage, and logs. Low

3.2 User Workflows

3.2.1 File Storage and Sharing (Nextcloud + SeaweedFS)

  1. User logs into Nextcloud web/app.
  2. Upload file → SeaweedFS distributes chunks across 3 nodes (erasure coding).
  3. Share link/email → Recipients access via authenticated URL.
  4. Delete → Automatic cleanup with audit log.

3.2.2 Instant Messaging (Matrix)

  1. Join/create room via Element client.
  2. Send messages/files → Synapse federates internally (no external federation by default).
  3. Notifications via WebSocket.

3.2.3 Video Conferencing (LiveKit)

  1. Create room via web UI.
  2. Join with token → WebRTC peer-to-peer with SFU fallback.
  3. End session → Trigger AI processing.

3.2.4 Paraphrasing AI

  1. Post-session: LiveKit recording ingested into AI pod.
  2. Whisper transcribes → T5 paraphrases key points.
  3. Output: JSON summary stored in SeaweedFS; accessible via Nextcloud.

3.3 Data Flows

  • Ingress: User traffic → Mycelium → Load balancer (k3s Ingress) → Service pods.
  • Storage: All data → SeaweedFS volumes (persistent via k3s PVs).
  • AI Processing: Event-driven (Kubernetes CronJob) on node-local storage.

4. Technical Specification

4.1 Architecture Overview

  • Topology: 3-node k3s cluster (1 master, 2 workers) with etcd (TBD) embedded for HA.
  • Layers:
    • OS Layer: zero-OS (immutable, container-only; no systemd).
    • Orchestration Layer: k3s.
    • Storage Layer: SeaweedFS master + volume servers (1 per node).
    • Network Layer: Mycelium for secure communication.
    • Application Layer: Helm-deployed charts for Nextcloud, Synapse, LiveKit, AI.
  • Diagram Description (Textual; visualize as a layered stack): 
    [Users (Web/Mobile)] --> [Mycelium] --> [k3s Ingress (Traefik)]
                                    |
[Node 1 (Master)] -- [Node 2 (Worker)] -- [Node 3 (Worker)]
    |                   |                   |
[zero-OS]          [zero-OS]          [zero-OS]
    |                   |                   |
[k3s Control Plane] [SeaweedFS Vol]   [LiveKit Pod]
    |                   |                   |
[Nextcloud Pod]     [Matrix Pod]      [AI Pod (Whisper/T5)]
    |                   |                   |
[SeaweedFS PVs (Shared)] <-- Replication -->

4.2 Component Details

4.2.1 Orchestration (k3s)

  • myceliumcloud install?

4.2.2 Storage (SeaweedFS)

  • Deployment: StatefulSet with 1 master pod (replicated) + 3 volume pods.
  • Configuration: Erasure coding (EC:3:1); filer for Nextcloud mount.
  • Integration: Mount as /mnt/seaweed in Nextcloud/Matrix pods.

4.2.3 Applications

  • Nextcloud: Helm chart; external DB (SQLite for simplicity); SeaweedFS as primary storage.
  • Matrix Synapse: Docker image; PostgreSQL backend (k3s PVC); federation disabled.
  • LiveKit: Server pod per node; Redis for coordination; Webhook to AI on session end.
  • Paraphrasing AI:
    • Container: Python + Torch; models from Hugging Face (offline download).
    • Workflow: Kubernetes Job triggered by LiveKit event → Process recording → Output to SeaweedFS.
    • Resources: 4 GB RAM, GPU optional (CPU fallback).

4.2.4 Networking (Mycelium)

  • Setup: zero-OS native; provision nodes via ThreeFold Grid dashboard.
  • Security: Automatic key exchange; traffic encrypted at L4.

4.3 Deployment Procedures

All presented deployment processes have not been tested. WIP.

  1. Node Provisioning:

    • Acquire 3 nodes on premise and connect them on ThreeFold Grid; boot zero-OS.

  2. k3s Installation:

  3. Storage Setup:

    • Deploy SeaweedFS Helm chart: helm install seaweedfs seaweedfs/seaweedfs --set volume.replicas=3.

  4. Application Deployment:

    • Nextcloud: helm install nextcloud bitnami/nextcloud --set persistence.storageClass=seaweedfs.
    • Matrix: Custom YAML for Synapse.
    • LiveKit: helm install livekit livekit/livekit.
    • AI: Custom Job YAML with env vars for model paths.

  5. Verification:

    • kubectl get pods → All running.
    • Test endpoints: Nextcloud login, Matrix room creation, LiveKit join.

4.4 Monitoring and Maintenance

  • Tools: Prometheus + Grafana (k3s addon).
  • Logs: Centralized via Fluentd to SeaweedFS.
  • Updates: Rolling upgrades via kubectl rollout; zero-OS atomic updates.

4.5 Risks and Mitigations

Risk ID Risk Description Mitigation
R-01 Node failure k3s auto-rescheduling; 2/3 quorum.
R-02 Storage desync SeaweedFS replication checks.
R-03 AI model inference latency Off-peak scheduling; model quantization.

5. Appendices

  • Glossary: zero-OS (minimal OS for containers), Mycelium (decentralized network), etc.
  • References: ThreeFold Docs (threefold.io), k3s.io, seaweedfs.org.
  • Approval: [Space for signatures].