WIP3 implemeting lancedb
This commit is contained in:
Maxime Van Hees
444
docs/lance.md
Normal file
444
docs/lance.md
Normal file
@@ -0,0 +1,444 @@
|
||||
# Lance Vector Backend (RESP + JSON-RPC)
|
||||
|
||||
This document explains how to use HeroDB’s Lance-backed vector store. It is text-first: users provide text, and HeroDB computes embeddings server-side (no manual vectors). It includes copy-pasteable RESP (redis-cli) and JSON-RPC examples for:
|
||||
|
||||
- Creating a Lance database
|
||||
- Embedding provider configuration (OpenAI, Azure OpenAI, or deterministic test provider)
|
||||
- Dataset lifecycle: CREATE, LIST, INFO, DROP
|
||||
- Ingestion: STORE text (+ optional metadata)
|
||||
- Search: QUERY with K, optional FILTER and RETURN
|
||||
- Delete by id
|
||||
- Index creation (currently a placeholder/no-op)
|
||||
|
||||
References:
|
||||
- Implementation: [src/lance_store.rs](src/lance_store.rs), [src/cmd.rs](src/cmd.rs), [src/rpc.rs](src/rpc.rs), [src/server.rs](src/server.rs), [src/embedding.rs](src/embedding.rs)
|
||||
|
||||
Notes:
|
||||
- Admin DB 0 cannot be Lance (or Tantivy). Only databases with id >= 1 can use Lance.
|
||||
- Permissions:
|
||||
- Read operations (SEARCH, LIST, INFO) require read permission.
|
||||
- Mutating operations (CREATE, STORE, CREATEINDEX, DEL, DROP, EMBEDDING CONFIG SET) require readwrite permission.
|
||||
- Backend gating:
|
||||
- If a DB is Lance, only LANCE.* and basic control commands (PING, ECHO, SELECT, INFO, CLIENT, etc.) are permitted.
|
||||
- If a DB is not Lance, LANCE.* commands return an error.
|
||||
|
||||
Storage layout and schema:
|
||||
- Files live at: <base_dir>/lance/<db_id>/<dataset>.lance
|
||||
- Records schema:
|
||||
- id: Utf8 (non-null)
|
||||
- vector: FixedSizeList<Float32, dim> (non-null)
|
||||
- text: Utf8 (nullable)
|
||||
- meta: Utf8 JSON (nullable)
|
||||
- Search is an L2 KNN brute-force scan for now (lower score = better). Index creation is a no-op placeholder to be implemented later.
|
||||
|
||||
Prerequisites:
|
||||
- Start HeroDB with RPC enabled (for management calls):
|
||||
- See [docs/basics.md](./basics.md) for flags. Example:
|
||||
```bash
|
||||
./target/release/herodb --dir /tmp/herodb --admin-secret mysecret --port 6379 --enable-rpc
|
||||
```
|
||||
|
||||
|
||||
## 0) Create a Lance-backed database (JSON-RPC)
|
||||
|
||||
Use the management API to create a database with backend "Lance". DB 0 is reserved for admin and cannot be Lance.
|
||||
|
||||
Request:
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 1,
|
||||
"method": "herodb_createDatabase",
|
||||
"params": [
|
||||
"Lance",
|
||||
{ "name": "vectors-db", "storage_path": null, "max_size": null, "redis_version": null },
|
||||
null
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
- Response contains the allocated db_id (>= 1). Use that id below (replace 1 with your actual id).
|
||||
|
||||
Select the database over RESP:
|
||||
```bash
|
||||
redis-cli -p 6379 SELECT 1
|
||||
# → OK
|
||||
```
|
||||
|
||||
|
||||
## 1) Configure embedding provider (server-side embeddings)
|
||||
|
||||
HeroDB embeds text internally at STORE/SEARCH time using a per-dataset EmbeddingConfig sidecar. Configure provider before creating a dataset to choose dimensions and provider.
|
||||
|
||||
Supported providers:
|
||||
- openai (standard OpenAI or Azure OpenAI)
|
||||
- testhash (deterministic, CI-friendly; no network)
|
||||
|
||||
Environment variables for OpenAI:
|
||||
- Standard OpenAI: export OPENAI_API_KEY=sk-...
|
||||
- Azure OpenAI: export AZURE_OPENAI_API_KEY=...
|
||||
|
||||
RESP examples:
|
||||
```bash
|
||||
# Standard OpenAI with default dims (model-dependent, e.g. 1536)
|
||||
redis-cli -p 6379 LANCE.EMBEDDING CONFIG SET myset PROVIDER openai MODEL text-embedding-3-small
|
||||
|
||||
# OpenAI with reduced output dimension (e.g., 512) when supported
|
||||
redis-cli -p 6379 LANCE.EMBEDDING CONFIG SET myset PROVIDER openai MODEL text-embedding-3-small PARAM dim 512
|
||||
|
||||
# Azure OpenAI (set env: AZURE_OPENAI_API_KEY)
|
||||
redis-cli -p 6379 LANCE.EMBEDDING CONFIG SET myset PROVIDER openai MODEL text-embedding-3-small \
|
||||
PARAM use_azure true \
|
||||
PARAM azure_endpoint https://myresource.openai.azure.com \
|
||||
PARAM azure_deployment my-embed-deploy \
|
||||
PARAM azure_api_version 2024-02-15 \
|
||||
PARAM dim 512
|
||||
|
||||
# Deterministic test provider (no network, stable vectors)
|
||||
redis-cli -p 6379 LANCE.EMBEDDING CONFIG SET myset PROVIDER testhash MODEL any
|
||||
```
|
||||
|
||||
Read config:
|
||||
```bash
|
||||
redis-cli -p 6379 LANCE.EMBEDDING CONFIG GET myset
|
||||
# → JSON blob describing provider/model/params
|
||||
```
|
||||
|
||||
JSON-RPC examples:
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 2,
|
||||
"method": "herodb_lanceSetEmbeddingConfig",
|
||||
"params": [
|
||||
1,
|
||||
"myset",
|
||||
"openai",
|
||||
"text-embedding-3-small",
|
||||
{ "dim": "512" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 3,
|
||||
"method": "herodb_lanceGetEmbeddingConfig",
|
||||
"params": [1, "myset"]
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
## 2) Create a dataset
|
||||
|
||||
Choose a dimension that matches your embedding configuration. For OpenAI text-embedding-3-small without dimension override, typical dimension is 1536; when `dim` is set (e.g., 512), use that. The current API requires an explicit DIM.
|
||||
|
||||
RESP:
|
||||
```bash
|
||||
redis-cli -p 6379 LANCE.CREATE myset DIM 512
|
||||
# → OK
|
||||
```
|
||||
|
||||
JSON-RPC:
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 4,
|
||||
"method": "herodb_lanceCreate",
|
||||
"params": [1, "myset", 512]
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
## 3) Store text documents (server-side embedding)
|
||||
|
||||
Provide your id, the text to embed, and optional META fields. The server computes the embedding using the configured provider and stores id/vector/text/meta in the Lance dataset. Upserts by id are supported via delete-then-append semantics.
|
||||
|
||||
RESP:
|
||||
```bash
|
||||
redis-cli -p 6379 LANCE.STORE myset ID doc-1 TEXT "Hello vector world" META title "Hello" category "demo"
|
||||
# → OK
|
||||
```
|
||||
|
||||
JSON-RPC:
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 5,
|
||||
"method": "herodb_lanceStoreText",
|
||||
"params": [
|
||||
1,
|
||||
"myset",
|
||||
"doc-1",
|
||||
"Hello vector world",
|
||||
{ "title": "Hello", "category": "demo" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
## 4) Search with a text query
|
||||
|
||||
Provide a query string; the server embeds it and performs KNN search. Optional: FILTER expression and RETURN subset of fields.
|
||||
|
||||
RESP:
|
||||
```bash
|
||||
# K nearest neighbors for the query text
|
||||
redis-cli -p 6379 LANCE.SEARCH myset K 5 QUERY "greetings to vectors"
|
||||
# → Array of hits: [id, score, [k,v, ...]] pairs, lower score = closer
|
||||
|
||||
# With a filter on meta fields and return only title
|
||||
redis-cli -p 6379 LANCE.SEARCH myset K 3 QUERY "greetings to vectors" FILTER "category = 'demo'" RETURN 1 title
|
||||
```
|
||||
|
||||
JSON-RPC:
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 6,
|
||||
"method": "herodb_lanceSearchText",
|
||||
"params": [1, "myset", "greetings to vectors", 5, null, null]
|
||||
}
|
||||
```
|
||||
|
||||
With filter and selected fields:
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 7,
|
||||
"method": "herodb_lanceSearchText",
|
||||
"params": [1, "myset", "greetings to vectors", 3, "category = 'demo'", ["title"]]
|
||||
}
|
||||
```
|
||||
|
||||
Response shape:
|
||||
- RESP over redis-cli: an array of hits [id, score, [k, v, ...]].
|
||||
- JSON-RPC returns an object containing the RESP-encoded wire format string or a structured result depending on implementation. See [src/rpc.rs](src/rpc.rs) for details.
|
||||
|
||||
|
||||
## 5) Create an index (placeholder)
|
||||
|
||||
Index creation currently returns OK but is a no-op. It will integrate Lance vector indices in a future update.
|
||||
|
||||
RESP:
|
||||
```bash
|
||||
redis-cli -p 6379 LANCE.CREATEINDEX myset TYPE "ivf_pq" PARAM nlist 100 PARAM pq_m 16
|
||||
# → OK (no-op for now)
|
||||
```
|
||||
|
||||
JSON-RPC:
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 8,
|
||||
"method": "herodb_lanceCreateIndex",
|
||||
"params": [1, "myset", "ivf_pq", { "nlist": "100", "pq_m": "16" }]
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
## 6) Inspect datasets
|
||||
|
||||
RESP:
|
||||
```bash
|
||||
# List datasets in current Lance DB
|
||||
redis-cli -p 6379 LANCE.LIST
|
||||
|
||||
# Get dataset info
|
||||
redis-cli -p 6379 LANCE.INFO myset
|
||||
```
|
||||
|
||||
JSON-RPC:
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 9,
|
||||
"method": "herodb_lanceList",
|
||||
"params": [1]
|
||||
}
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 10,
|
||||
"method": "herodb_lanceInfo",
|
||||
"params": [1, "myset"]
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
## 7) Delete and drop
|
||||
|
||||
RESP:
|
||||
```bash
|
||||
# Delete by id
|
||||
redis-cli -p 6379 LANCE.DEL myset doc-1
|
||||
# → OK
|
||||
|
||||
# Drop the entire dataset
|
||||
redis-cli -p 6379 LANCE.DROP myset
|
||||
# → OK
|
||||
```
|
||||
|
||||
JSON-RPC:
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 11,
|
||||
"method": "herodb_lanceDel",
|
||||
"params": [1, "myset", "doc-1"]
|
||||
}
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 12,
|
||||
"method": "herodb_lanceDrop",
|
||||
"params": [1, "myset"]
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
## 8) End-to-end example (RESP)
|
||||
|
||||
```bash
|
||||
# 1. Select Lance DB (assume db_id=1 created via RPC)
|
||||
redis-cli -p 6379 SELECT 1
|
||||
|
||||
# 2. Configure embedding provider (OpenAI small model at 512 dims)
|
||||
redis-cli -p 6379 LANCE.EMBEDDING CONFIG SET myset PROVIDER openai MODEL text-embedding-3-small PARAM dim 512
|
||||
|
||||
# 3. Create dataset
|
||||
redis-cli -p 6379 LANCE.CREATE myset DIM 512
|
||||
|
||||
# 4. Store documents
|
||||
redis-cli -p 6379 LANCE.STORE myset ID doc-1 TEXT "The quick brown fox jumps over the lazy dog" META title "Fox" category "animal"
|
||||
redis-cli -p 6379 LANCE.STORE myset ID doc-2 TEXT "A fast auburn fox vaulted a sleepy canine" META title "Fox paraphrase" category "animal"
|
||||
|
||||
# 5. Search
|
||||
redis-cli -p 6379 LANCE.SEARCH myset K 2 QUERY "quick brown fox" RETURN 1 title
|
||||
|
||||
# 6. Dataset info and listing
|
||||
redis-cli -p 6379 LANCE.INFO myset
|
||||
redis-cli -p 6379 LANCE.LIST
|
||||
|
||||
# 7. Delete and drop
|
||||
redis-cli -p 6379 LANCE.DEL myset doc-2
|
||||
redis-cli -p 6379 LANCE.DROP myset
|
||||
```
|
||||
|
||||
|
||||
## 9) End-to-end example (JSON-RPC)
|
||||
|
||||
Assume RPC server on port 8080. Replace ids and ports as needed.
|
||||
|
||||
1) Create Lance DB:
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 100,
|
||||
"method": "herodb_createDatabase",
|
||||
"params": ["Lance", { "name": "vectors-db", "storage_path": null, "max_size": null, "redis_version": null }, null]
|
||||
}
|
||||
```
|
||||
|
||||
2) Set embedding config:
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 101,
|
||||
"method": "herodb_lanceSetEmbeddingConfig",
|
||||
"params": [1, "myset", "openai", "text-embedding-3-small", { "dim": "512" }]
|
||||
}
|
||||
```
|
||||
|
||||
3) Create dataset:
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 102,
|
||||
"method": "herodb_lanceCreate",
|
||||
"params": [1, "myset", 512]
|
||||
}
|
||||
```
|
||||
|
||||
4) Store text:
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 103,
|
||||
"method": "herodb_lanceStoreText",
|
||||
"params": [1, "myset", "doc-1", "The quick brown fox jumps over the lazy dog", { "title": "Fox", "category": "animal" }]
|
||||
}
|
||||
```
|
||||
|
||||
5) Search text:
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 104,
|
||||
"method": "herodb_lanceSearchText",
|
||||
"params": [1, "myset", "quick brown fox", 2, null, ["title"]]
|
||||
}
|
||||
```
|
||||
|
||||
6) Info/list:
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 105,
|
||||
"method": "herodb_lanceInfo",
|
||||
"params": [1, "myset"]
|
||||
}
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 106,
|
||||
"method": "herodb_lanceList",
|
||||
"params": [1]
|
||||
}
|
||||
```
|
||||
|
||||
7) Delete/drop:
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 107,
|
||||
"method": "herodb_lanceDel",
|
||||
"params": [1, "myset", "doc-1"]
|
||||
}
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 108,
|
||||
"method": "herodb_lanceDrop",
|
||||
"params": [1, "myset"]
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
## 10) Operational notes and troubleshooting
|
||||
|
||||
- If using OpenAI and you see “missing API key env”, set:
|
||||
- Standard: `export OPENAI_API_KEY=sk-...`
|
||||
- Azure: `export AZURE_OPENAI_API_KEY=...` and pass `use_azure true`, `azure_endpoint`, `azure_deployment`, `azure_api_version`.
|
||||
- Dimensions mismatch:
|
||||
- Ensure the dataset DIM equals the provider’s embedding dim. For OpenAI text-embedding-3 models, set `PARAM dim 512` (or another supported size) and use that same DIM for `LANCE.CREATE`.
|
||||
- DB 0 restriction:
|
||||
- Lance is not allowed on DB 0. Use db_id >= 1.
|
||||
- Permissions:
|
||||
- Read operations (SEARCH, LIST, INFO) require read permission.
|
||||
- Mutations (CREATE, STORE, CREATEINDEX, DEL, DROP, EMBEDDING CONFIG SET) require readwrite permission.
|
||||
- Backend gating:
|
||||
- On Lance DBs, only LANCE.* commands are accepted (plus basic control).
|
||||
- Current index behavior:
|
||||
- `LANCE.CREATEINDEX` returns OK but is a no-op. Future versions will integrate Lance vector indices.
|
||||
- Implementation files for reference:
|
||||
- [src/lance_store.rs](src/lance_store.rs), [src/cmd.rs](src/cmd.rs), [src/rpc.rs](src/rpc.rs), [src/server.rs](src/server.rs), [src/embedding.rs](src/embedding.rs)
|
||||
Reference in New Issue
Block a user