merge branches and cleanup db

_archive/instructions.md

@@ -0,0 +1,153 @@
+# HeroDB: ACL Layer Implementation
+
+## Project Overview
+
+Create a new module that implements an Access Control List (ACL) layer on top of the existing `ourdb` and `tst` databases. This module will manage permissions and access control for data stored in the database system.
+
+call this module: acldb
+
+implement in acldb
+
+remark: there is no dependency on herodb
+
+## Architecture
+
+- The module will sit as a layer between client applications and the underlying `ourdb` & `tst` databases
+- ACLs are defined at the circle level and stored in a special topic called "acl"
+- Data in `ourdb` is stored at path: `~/hero/var/ourdb/$circleid/$topicid`
+- `tst` is used to create mappings between keys and IDs in `ourdb`
+
+## ACL Structure
+
+Each ACL contains:
+- A unique name (per circle)
+- A list of public keys with associated permissions
+- Rights are hierarchical: read → write → delete → execute → admin (each right includes all rights to its left)
+
+
+## factory
+
+- returns an object with name ACLDB and is linked to 1 directory which represents a circle, 
+- the argument to open an ACLDB is circleid, the dir is on ~/hero/var/ourdb/$circleid
+- the ACLDB has acl methods directly implemented on this one
+- we have a getter on ACLDB which returns ACLDBTOPIC which is a DB isntance per topic
+- on ACLDBTOPIC we have the set,get,delte, prefix, ... 
+- the ACLDBTOPIC knows how to get ACL's from its parent and use them
+
+## Core Methods
+
+### ACL Management
+
+#### aclupdate
+Updates or creates an ACL with specified permissions.
+
+**Parameters:**
+- `callerpubkey`: Public key of the requesting user
+- `circleid`: ID of the circle where the ACL exists
+- `name`: Unique name for the ACL within the circle
+- `pubkeys`: Array of public keys to grant permissions to
+- `right`: Permission level (enum: read/write/delete/execute/admin)
+#### aclremove
+Removes specific public keys from an existing ACL.
+
+**Parameters:**
+- `callerpubkey`: Public key of the requesting user
+- `circleid`: ID of the circle where the ACL exists
+- `name`: Name of the ACL to modify
+- `pubkeys`: Array of public keys to remove from the ACL
+
+#### acldel
+Deletes an entire ACL.
+
+**Parameters:**
+- `callerpubkey`: Public key of the requesting user
+- `circleid`: ID of the circle where the ACL exists
+- `name`: Name of the ACL to delete
+
+### Data Operations
+
+#### set
+Stores or updates data in the database with optional ACL protection.
+
+**Parameters:**
+- `callerpubkey`: Public key of the requesting user
+- `circleid`: ID of the circle where the data belongs
+- `topic`: String identifier for the database category (e.g., "customer", "product")
+- `key`: Optional string key for the record
+- `id`: Optional numeric ID for direct access
+- `value`: Binary blob of data to store
+- `aclid`: ID of the ACL to protect this record (0 for public access)
+
+**Behavior:**
+- If only `key` is provided, use `tst` to map the key to a new or existing ID
+- If `id` is specified or derived from an existing key, update the corresponding record
+- Returns the ID of the created/updated record
+
+#### del
+Marks a record as deleted.
+
+**Parameters:**
+- `callerpubkey`: Public key of the requesting user
+- `circleid`: ID of the circle where the data belongs
+- `topic`: String identifier for the database category
+- `id` or `key`: Identifier for the record to delete
+
+**Behavior:**
+- Deletes the mapping in `tst` if a key was used
+- Marks the record as deleted in `ourdb` (not physically removed)
+
+#### get
+Retrieves data from the database.
+
+**Parameters:**
+- `callerpubkey`: Public key of the requesting user
+- `circleid`: ID of the circle where the data belongs
+- `topic`: String identifier for the database category
+- `id` or `key`: Identifier for the record to retrieve
+
+**Returns:**
+- The binary data stored in the record if the caller has access
+
+## Implementation Details
+
+### ACL Storage Format
+- ACLs are stored in a special topic named "acl" within each circle
+- Each ACL has a unique numeric ID within the circle
+
+### Record ACL Protection
+- When a record uses ACL protection, the first 4 bytes of the stored data contain the ACL ID
+- A new constructor in `ourdb` should be created to handle ACL-protected records
+- Records with ACL ID of 0 are accessible to everyone
+
+## RPC Interface
+
+The module should expose its functionality through an RPC interface:
+
+1. Client sends:
+   - Method name (e.g., "del", "set", "get")
+   - JSON-encoded arguments
+   - Cryptographic signature of the JSON data
+
+2. Server:
+   - Verifies the signature is valid
+   - Extracts the caller's public key from the signature
+   - Checks permissions against applicable ACLs
+   - Executes the requested operation if authorized
+   - Returns appropriate response
+
+## Security Considerations
+
+- All operations must validate the caller has appropriate permissions
+- ACL changes should be logged for audit purposes
+- Consider implementing rate limiting to prevent abuse
+
+## THE SERVER
+
+- create actix webserver
+- make a router that handles the rpc interface
+- use openapi spec
+- embed swagger interface
+- implement a queuing mechanism, so internal we don't have to implement locks, but we do 1 request after the other per circle, so we know we never have conflicting changes in 1 circle
+- create a logger which gives us good overview of what happened when
+
+