herodb/docs/lance_vector_db.md

Lance Vector Database Operations

HeroDB includes a powerful vector database integration using Lance, enabling high-performance vector storage, search, and multimodal data management. By default, it uses Ollama for local text embeddings, with support for custom external embedding services.

Overview

The Lance vector database integration provides:

• High-performance vector storage using Lance's columnar format
• Local Ollama integration for text embeddings (default, no external dependencies)
• Custom embedding service support for advanced use cases
• Text embedding support (images via custom services)
• Vector similarity search with configurable parameters
• Scalable indexing with IVF_PQ (Inverted File with Product Quantization)
• Redis-compatible command interface

Architecture

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   HeroDB        │    │  External        │    │   Lance         │
│   Redis Server  │◄──►│  Embedding       │    │   Vector Store  │
│                 │    │  Service         │    │                 │
└─────────────────┘    └──────────────────┘    └─────────────────┘
         │                       │                       │
         │                       │                       │
    Redis Protocol          HTTP API              Arrow/Parquet
    Commands                JSON Requests         Columnar Storage

Key Components

1. Lance Store: High-performance columnar vector storage
2. Ollama Integration: Local embedding service (default)
3. Custom Embedding Service: Optional HTTP API for advanced use cases
4. Redis Command Interface: Familiar Redis-style commands
5. Arrow Schema: Flexible schema definition for metadata

Configuration

Default Setup (Ollama)

HeroDB uses Ollama by default for text embeddings. No configuration is required if Ollama is running locally:

# Install Ollama (if not already installed)
# Visit: https://ollama.ai

# Pull the embedding model
ollama pull nomic-embed-text

# Ollama automatically runs on localhost:11434
# HeroDB will use this by default

Default Configuration:

• URL: http://localhost:11434
• Model: nomic-embed-text
• Dimensions: 768 (for nomic-embed-text)

Custom Embedding Service (Optional)

To use a custom embedding service instead of Ollama:

# Set custom embedding service URL
redis-cli HSET config:core:aiembed url "http://your-embedding-service:8080/embed"

# Optional: Set authentication if required
redis-cli HSET config:core:aiembed token "your-api-token"

Embedding Service API Contracts

Ollama API (Default)

HeroDB calls Ollama using this format:

POST http://localhost:11434/api/embeddings
Content-Type: application/json

{
  "model": "nomic-embed-text",
  "prompt": "Your text to embed"
}

Response:

{
  "embedding": [0.1, 0.2, 0.3, ...]
}

Custom Service API

Your custom embedding service should accept POST requests with this JSON format:

{
  "texts": ["text1", "text2"],           // Optional: array of texts
  "images": ["base64_image1", "base64_image2"], // Optional: base64 encoded images
  "model": "your-model-name"             // Optional: model specification
}

And return responses in this format:

{
  "embeddings": [[0.1, 0.2, ...], [0.3, 0.4, ...]],  // Array of embedding vectors
  "model": "model-name",                               // Model used
  "usage": {                                          // Optional usage stats
    "tokens": 100,
    "requests": 2
  }
}

Commands Reference

Dataset Management

LANCE CREATE

Create a new vector dataset with specified dimensions and optional schema.

LANCE CREATE <dataset> DIM <dimension> [SCHEMA field:type ...]

Parameters:

• dataset: Name of the dataset
• dimension: Vector dimension (e.g., 384, 768, 1536)
• field:type: Optional metadata fields (string, int, float, bool)

Examples:

# Create a simple dataset for 384-dimensional vectors
LANCE CREATE documents DIM 384

# Create dataset with metadata schema
LANCE CREATE products DIM 768 SCHEMA category:string price:float available:bool

LANCE LIST

List all available datasets.

LANCE LIST

Returns: Array of dataset names

LANCE INFO

Get information about a specific dataset.

LANCE INFO <dataset>

Returns: Dataset metadata including name, version, row count, and schema

LANCE DROP

Delete a dataset and all its data.

LANCE DROP <dataset>

Data Operations

LANCE STORE

Store multimodal data (text/images) with automatic embedding generation.

LANCE STORE <dataset> [TEXT <text>] [IMAGE <base64>] [key value ...]

Parameters:

• dataset: Target dataset name
• TEXT: Text content to embed
• IMAGE: Base64-encoded image to embed
• key value: Metadata key-value pairs

Examples:

# Store text with metadata
LANCE STORE documents TEXT "Machine learning is transforming industries" category "AI" author "John Doe"

# Store image with metadata
LANCE STORE images IMAGE "iVBORw0KGgoAAAANSUhEUgAA..." category "nature" tags "landscape,mountains"

# Store both text and image
LANCE STORE multimodal TEXT "Beautiful sunset" IMAGE "base64data..." location "California"

Returns: Unique ID of the stored item

Search Operations

LANCE SEARCH

Search using a raw vector.

LANCE SEARCH <dataset> VECTOR <vector> K <k> [NPROBES <n>] [REFINE <r>]

Parameters:

• dataset: Dataset to search
• vector: Comma-separated vector values (e.g., "0.1,0.2,0.3")
• k: Number of results to return
• NPROBES: Number of partitions to search (optional)
• REFINE: Refine factor for better accuracy (optional)

Example:

LANCE SEARCH documents VECTOR "0.1,0.2,0.3,0.4" K 5 NPROBES 10

LANCE SEARCH.TEXT

Search using text query (automatically embedded).

LANCE SEARCH.TEXT <dataset> <query_text> K <k> [NPROBES <n>] [REFINE <r>]

Parameters:

• dataset: Dataset to search
• query_text: Text query to search for
• k: Number of results to return
• NPROBES: Number of partitions to search (optional)
• REFINE: Refine factor for better accuracy (optional)

Example:

LANCE SEARCH.TEXT documents "artificial intelligence applications" K 10 NPROBES 20

Returns: Array of results with distance scores and metadata

Embedding Operations

LANCE EMBED.TEXT

Generate embeddings for text without storing.

LANCE EMBED.TEXT <text1> [text2] [text3] ...

Example:

LANCE EMBED.TEXT "Hello world" "Machine learning" "Vector database"

Returns: Array of embedding vectors

Index Management

LANCE CREATE.INDEX

Create a vector index for faster search performance.

LANCE CREATE.INDEX <dataset> <index_type> [PARTITIONS <n>] [SUBVECTORS <n>]

Parameters:

• dataset: Dataset to index
• index_type: Index type (currently supports "IVF_PQ")
• PARTITIONS: Number of partitions (default: 256)
• SUBVECTORS: Number of sub-vectors for PQ (default: 16)

Example:

LANCE CREATE.INDEX documents IVF_PQ PARTITIONS 512 SUBVECTORS 32

Usage Patterns

1. Document Search System

# Setup
LANCE CREATE documents DIM 384 SCHEMA title:string content:string category:string

# Store documents
LANCE STORE documents TEXT "Introduction to machine learning algorithms" title "ML Basics" category "education"
LANCE STORE documents TEXT "Deep learning neural networks explained" title "Deep Learning" category "education"
LANCE STORE documents TEXT "Building scalable web applications" title "Web Dev" category "programming"

# Create index for better performance
LANCE CREATE.INDEX documents IVF_PQ PARTITIONS 256

# Search
LANCE SEARCH.TEXT documents "neural networks" K 5

2. Image Similarity Search

# Setup
LANCE CREATE images DIM 512 SCHEMA filename:string tags:string

# Store images (base64 encoded)
LANCE STORE images IMAGE "iVBORw0KGgoAAAANSUhEUgAA..." filename "sunset.jpg" tags "nature,landscape"
LANCE STORE images IMAGE "iVBORw0KGgoAAAANSUhEUgBB..." filename "city.jpg" tags "urban,architecture"

# Search by image
LANCE STORE temp_search IMAGE "query_image_base64..."
# Then use the returned ID to get embedding and search

3. Multimodal Content Management

# Setup
LANCE CREATE content DIM 768 SCHEMA type:string source:string

# Store mixed content
LANCE STORE content TEXT "Product description for smartphone" type "product" source "catalog"
LANCE STORE content IMAGE "product_image_base64..." type "product_image" source "catalog"

# Search across all content types
LANCE SEARCH.TEXT content "smartphone features" K 10

Performance Considerations

Vector Dimensions

• 384: Good for general text (e.g., sentence-transformers)
• 768: Standard for BERT-like models
• 1536: OpenAI text-embedding-ada-002
• Higher dimensions: Better accuracy but slower search

Index Configuration

• More partitions: Better for larger datasets (>100K vectors)
• More sub-vectors: Better compression but slower search
• NPROBES: Higher values = better accuracy, slower search

Best Practices

1. Create indexes for datasets with >1000 vectors
2. Use appropriate dimensions based on your embedding model
3. Configure NPROBES based on accuracy vs speed requirements
4. Batch operations when possible for better performance
5. Monitor embedding service response times and rate limits

Error Handling

Common error scenarios and solutions:

Embedding Service Errors

# Error: Embedding service not configured
ERR Embedding service URL not configured. Set it with: HSET config:core:aiembed url <YOUR_EMBEDDING_SERVICE_URL>

# Error: Service unavailable
ERR Embedding service returned error 404 Not Found

Solution: Ensure embedding service is running and URL is correct.

Dataset Errors

# Error: Dataset doesn't exist
ERR Dataset 'mydata' does not exist

# Error: Dimension mismatch
ERR Vector dimension mismatch: expected 384, got 768

Solution: Create dataset first or check vector dimensions.

Search Errors

# Error: Invalid vector format
ERR Invalid vector format

# Error: No index available
ERR No index available for fast search

Solution: Check vector format or create an index.

Integration Examples

With Python

import redis
import json

r = redis.Redis(host='localhost', port=6379)

# Create dataset
r.execute_command('LANCE', 'CREATE', 'docs', 'DIM', '384')

# Store document
result = r.execute_command('LANCE', 'STORE', 'docs', 
                          'TEXT', 'Machine learning tutorial',
                          'category', 'education')
print(f"Stored with ID: {result}")

# Search
results = r.execute_command('LANCE', 'SEARCH.TEXT', 'docs', 
                           'machine learning', 'K', '5')
print(f"Search results: {results}")

With Node.js

const redis = require('redis');
const client = redis.createClient();

// Create dataset
await client.sendCommand(['LANCE', 'CREATE', 'docs', 'DIM', '384']);

// Store document
const id = await client.sendCommand(['LANCE', 'STORE', 'docs', 
                                   'TEXT', 'Deep learning guide',
                                   'category', 'AI']);

// Search
const results = await client.sendCommand(['LANCE', 'SEARCH.TEXT', 'docs',
                                        'deep learning', 'K', '10']);

Monitoring and Maintenance

Health Checks

# Check if Lance store is available
LANCE LIST

# Check dataset health
LANCE INFO mydataset

# Test embedding service
LANCE EMBED.TEXT "test"

Maintenance Operations

# Backup: Use standard Redis backup procedures
# The Lance data is stored separately in the data directory

# Cleanup: Remove unused datasets
LANCE DROP old_dataset

# Reindex: Drop and recreate indexes if needed
LANCE DROP dataset_name
LANCE CREATE dataset_name DIM 384
# Re-import data
LANCE CREATE.INDEX dataset_name IVF_PQ

This integration provides a powerful foundation for building AI-powered applications with vector search capabilities while maintaining the familiar Redis interface.