# Lance Vector Database Operations HeroDB includes a powerful vector database integration using Lance, enabling high-performance vector storage, search, and multimodal data management. By default, it uses Ollama for local text embeddings, with support for custom external embedding services. ## Overview The Lance vector database integration provides: - **High-performance vector storage** using Lance's columnar format - **Local Ollama integration** for text embeddings (default, no external dependencies) - **Custom embedding service support** for advanced use cases - **Text embedding support** (images via custom services) - **Vector similarity search** with configurable parameters - **Scalable indexing** with IVF_PQ (Inverted File with Product Quantization) - **Redis-compatible command interface** ## Architecture ``` ┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐ │ HeroDB │ │ External │ │ Lance │ │ Redis Server │◄──►│ Embedding │ │ Vector Store │ │ │ │ Service │ │ │ └─────────────────┘ └──────────────────┘ └─────────────────┘ │ │ │ │ │ │ Redis Protocol HTTP API Arrow/Parquet Commands JSON Requests Columnar Storage ``` ### Key Components 1. **Lance Store**: High-performance columnar vector storage 2. **Ollama Integration**: Local embedding service (default) 3. **Custom Embedding Service**: Optional HTTP API for advanced use cases 4. **Redis Command Interface**: Familiar Redis-style commands 5. **Arrow Schema**: Flexible schema definition for metadata ## Configuration ### Default Setup (Ollama) HeroDB uses Ollama by default for text embeddings. No configuration is required if Ollama is running locally: ```bash # Install Ollama (if not already installed) # Visit: https://ollama.ai # Pull the embedding model ollama pull nomic-embed-text # Ollama automatically runs on localhost:11434 # HeroDB will use this by default ``` **Default Configuration:** - **URL**: `http://localhost:11434` - **Model**: `nomic-embed-text` - **Dimensions**: 768 (for nomic-embed-text) ### Custom Embedding Service (Optional) To use a custom embedding service instead of Ollama: ```bash # Set custom embedding service URL redis-cli HSET config:core:aiembed url "http://your-embedding-service:8080/embed" # Optional: Set authentication if required redis-cli HSET config:core:aiembed token "your-api-token" ``` ### Embedding Service API Contracts #### Ollama API (Default) HeroDB calls Ollama using this format: ```bash POST http://localhost:11434/api/embeddings Content-Type: application/json { "model": "nomic-embed-text", "prompt": "Your text to embed" } ``` Response: ```json { "embedding": [0.1, 0.2, 0.3, ...] } ``` #### Custom Service API Your custom embedding service should accept POST requests with this JSON format: ```json { "texts": ["text1", "text2"], // Optional: array of texts "images": ["base64_image1", "base64_image2"], // Optional: base64 encoded images "model": "your-model-name" // Optional: model specification } ``` And return responses in this format: ```json { "embeddings": [[0.1, 0.2, ...], [0.3, 0.4, ...]], // Array of embedding vectors "model": "model-name", // Model used "usage": { // Optional usage stats "tokens": 100, "requests": 2 } } ``` ## Commands Reference ### Dataset Management #### LANCE CREATE Create a new vector dataset with specified dimensions and optional schema. ```bash LANCE CREATE DIM [SCHEMA field:type ...] ``` **Parameters:** - `dataset`: Name of the dataset - `dimension`: Vector dimension (e.g., 384, 768, 1536) - `field:type`: Optional metadata fields (string, int, float, bool) **Examples:** ```bash # Create a simple dataset for 384-dimensional vectors LANCE CREATE documents DIM 384 # Create dataset with metadata schema LANCE CREATE products DIM 768 SCHEMA category:string price:float available:bool ``` #### LANCE LIST List all available datasets. ```bash LANCE LIST ``` **Returns:** Array of dataset names #### LANCE INFO Get information about a specific dataset. ```bash LANCE INFO ``` **Returns:** Dataset metadata including name, version, row count, and schema #### LANCE DROP Delete a dataset and all its data. ```bash LANCE DROP ``` ### Data Operations #### LANCE STORE Store multimodal data (text/images) with automatic embedding generation. ```bash LANCE STORE [TEXT ] [IMAGE ] [key value ...] ``` **Parameters:** - `dataset`: Target dataset name - `TEXT`: Text content to embed - `IMAGE`: Base64-encoded image to embed - `key value`: Metadata key-value pairs **Examples:** ```bash # Store text with metadata LANCE STORE documents TEXT "Machine learning is transforming industries" category "AI" author "John Doe" # Store image with metadata LANCE STORE images IMAGE "iVBORw0KGgoAAAANSUhEUgAA..." category "nature" tags "landscape,mountains" # Store both text and image LANCE STORE multimodal TEXT "Beautiful sunset" IMAGE "base64data..." location "California" ``` **Returns:** Unique ID of the stored item ### Search Operations #### LANCE SEARCH Search using a raw vector. ```bash LANCE SEARCH VECTOR K [NPROBES ] [REFINE ] ``` **Parameters:** - `dataset`: Dataset to search - `vector`: Comma-separated vector values (e.g., "0.1,0.2,0.3") - `k`: Number of results to return - `NPROBES`: Number of partitions to search (optional) - `REFINE`: Refine factor for better accuracy (optional) **Example:** ```bash LANCE SEARCH documents VECTOR "0.1,0.2,0.3,0.4" K 5 NPROBES 10 ``` #### LANCE SEARCH.TEXT Search using text query (automatically embedded). ```bash LANCE SEARCH.TEXT K [NPROBES ] [REFINE ] ``` **Parameters:** - `dataset`: Dataset to search - `query_text`: Text query to search for - `k`: Number of results to return - `NPROBES`: Number of partitions to search (optional) - `REFINE`: Refine factor for better accuracy (optional) **Example:** ```bash LANCE SEARCH.TEXT documents "artificial intelligence applications" K 10 NPROBES 20 ``` **Returns:** Array of results with distance scores and metadata ### Embedding Operations #### LANCE EMBED.TEXT Generate embeddings for text without storing. ```bash LANCE EMBED.TEXT [text2] [text3] ... ``` **Example:** ```bash LANCE EMBED.TEXT "Hello world" "Machine learning" "Vector database" ``` **Returns:** Array of embedding vectors ### Index Management #### LANCE CREATE.INDEX Create a vector index for faster search performance. ```bash LANCE CREATE.INDEX [PARTITIONS ] [SUBVECTORS ] ``` **Parameters:** - `dataset`: Dataset to index - `index_type`: Index type (currently supports "IVF_PQ") - `PARTITIONS`: Number of partitions (default: 256) - `SUBVECTORS`: Number of sub-vectors for PQ (default: 16) **Example:** ```bash LANCE CREATE.INDEX documents IVF_PQ PARTITIONS 512 SUBVECTORS 32 ``` ## Usage Patterns ### 1. Document Search System ```bash # Setup LANCE CREATE documents DIM 384 SCHEMA title:string content:string category:string # Store documents LANCE STORE documents TEXT "Introduction to machine learning algorithms" title "ML Basics" category "education" LANCE STORE documents TEXT "Deep learning neural networks explained" title "Deep Learning" category "education" LANCE STORE documents TEXT "Building scalable web applications" title "Web Dev" category "programming" # Create index for better performance LANCE CREATE.INDEX documents IVF_PQ PARTITIONS 256 # Search LANCE SEARCH.TEXT documents "neural networks" K 5 ``` ### 2. Image Similarity Search ```bash # Setup LANCE CREATE images DIM 512 SCHEMA filename:string tags:string # Store images (base64 encoded) LANCE STORE images IMAGE "iVBORw0KGgoAAAANSUhEUgAA..." filename "sunset.jpg" tags "nature,landscape" LANCE STORE images IMAGE "iVBORw0KGgoAAAANSUhEUgBB..." filename "city.jpg" tags "urban,architecture" # Search by image LANCE STORE temp_search IMAGE "query_image_base64..." # Then use the returned ID to get embedding and search ``` ### 3. Multimodal Content Management ```bash # Setup LANCE CREATE content DIM 768 SCHEMA type:string source:string # Store mixed content LANCE STORE content TEXT "Product description for smartphone" type "product" source "catalog" LANCE STORE content IMAGE "product_image_base64..." type "product_image" source "catalog" # Search across all content types LANCE SEARCH.TEXT content "smartphone features" K 10 ``` ## Performance Considerations ### Vector Dimensions - **384**: Good for general text (e.g., sentence-transformers) - **768**: Standard for BERT-like models - **1536**: OpenAI text-embedding-ada-002 - **Higher dimensions**: Better accuracy but slower search ### Index Configuration - **More partitions**: Better for larger datasets (>100K vectors) - **More sub-vectors**: Better compression but slower search - **NPROBES**: Higher values = better accuracy, slower search ### Best Practices 1. **Create indexes** for datasets with >1000 vectors 2. **Use appropriate dimensions** based on your embedding model 3. **Configure NPROBES** based on accuracy vs speed requirements 4. **Batch operations** when possible for better performance 5. **Monitor embedding service** response times and rate limits ## Error Handling Common error scenarios and solutions: ### Embedding Service Errors ```bash # Error: Embedding service not configured ERR Embedding service URL not configured. Set it with: HSET config:core:aiembed url # Error: Service unavailable ERR Embedding service returned error 404 Not Found ``` **Solution:** Ensure embedding service is running and URL is correct. ### Dataset Errors ```bash # Error: Dataset doesn't exist ERR Dataset 'mydata' does not exist # Error: Dimension mismatch ERR Vector dimension mismatch: expected 384, got 768 ``` **Solution:** Create dataset first or check vector dimensions. ### Search Errors ```bash # Error: Invalid vector format ERR Invalid vector format # Error: No index available ERR No index available for fast search ``` **Solution:** Check vector format or create an index. ## Integration Examples ### With Python ```python import redis import json r = redis.Redis(host='localhost', port=6379) # Create dataset r.execute_command('LANCE', 'CREATE', 'docs', 'DIM', '384') # Store document result = r.execute_command('LANCE', 'STORE', 'docs', 'TEXT', 'Machine learning tutorial', 'category', 'education') print(f"Stored with ID: {result}") # Search results = r.execute_command('LANCE', 'SEARCH.TEXT', 'docs', 'machine learning', 'K', '5') print(f"Search results: {results}") ``` ### With Node.js ```javascript const redis = require('redis'); const client = redis.createClient(); // Create dataset await client.sendCommand(['LANCE', 'CREATE', 'docs', 'DIM', '384']); // Store document const id = await client.sendCommand(['LANCE', 'STORE', 'docs', 'TEXT', 'Deep learning guide', 'category', 'AI']); // Search const results = await client.sendCommand(['LANCE', 'SEARCH.TEXT', 'docs', 'deep learning', 'K', '10']); ``` ## Monitoring and Maintenance ### Health Checks ```bash # Check if Lance store is available LANCE LIST # Check dataset health LANCE INFO mydataset # Test embedding service LANCE EMBED.TEXT "test" ``` ### Maintenance Operations ```bash # Backup: Use standard Redis backup procedures # The Lance data is stored separately in the data directory # Cleanup: Remove unused datasets LANCE DROP old_dataset # Reindex: Drop and recreate indexes if needed LANCE DROP dataset_name LANCE CREATE dataset_name DIM 384 # Re-import data LANCE CREATE.INDEX dataset_name IVF_PQ ``` This integration provides a powerful foundation for building AI-powered applications with vector search capabilities while maintaining the familiar Redis interface.