Vector DB Testing Guide
This document provides comprehensive information about testing the Vector DB integration, including unit tests, integration tests, and performance benchmarks.
Overview
The Vector DB integration includes comprehensive test coverage across multiple layers:
- Unit Tests: Test individual components in isolation
- Integration Tests: Test components working together with real Home Assistant instances
- Performance Tests: Benchmark operations with large datasets
Test Structure
tests/
├── unit/
│ ├── test_vectordb_backend.py # Abstract backend interface tests
│ ├── test_vectordb_chroma_backend.py # Chroma backend implementation tests
│ ├── test_vectordb_config.py # Configuration tests
│ ├── test_vectordb_config_enhanced.py # Enhanced configuration tests
│ ├── test_vectordb_embeddings.py # Embedding model tests
│ ├── test_vectordb_manager.py # Manager singleton tests
│ ├── test_vectordb_indexing.py # Entity indexing tests
│ ├── test_vectordb_search.py # Semantic search tests
│ ├── test_vectordb_classification.py # Query classification tests
│ ├── test_vectordb_history.py # Query history tests
│ ├── test_vectordb_relationships.py # Relationship graph tests
│ └── test_vectordb_description.py # Description generation tests
├── integration/
│ ├── test_vectordb_indexing.py # Integration tests for indexing
│ ├── test_vectordb_search.py # Integration tests for search
│ ├── test_vectordb_query_processing.py # Integration tests for query processing
│ └── test_vectordb_entity_suggestions.py # Integration tests for suggestions
└── performance/
└── test_vectordb_performance.py # Performance benchmarks
Running Tests
Unit Tests
Run all unit tests:
Run specific test file:
Run with coverage:
Integration Tests
Integration tests require a running Home Assistant instance. Set environment variables:
Run integration tests:
Performance Tests
Run performance benchmarks:
Test Coverage
The Vector DB codebase aims for >90% test coverage. Current coverage includes:
- Backend Interface: 100% coverage
- Chroma Backend: ~58% coverage (needs improvement)
- Configuration: ~95% coverage
- Embeddings: ~63% coverage (needs improvement)
- Manager: ~63% coverage (needs improvement)
- Indexing: ~79% coverage
- Search: ~80% coverage
- Classification: ~99% coverage
- History: ~79% coverage
- Relationships: ~80% coverage
- Description: ~87% coverage
New Test Suites
The following new test suites have been added to improve coverage:
- Error Handling Tests (
test_vectordb_error_handling.py): Comprehensive tests for error handling and fallback behavior - Manager initialization errors
- Backend health check failures
- Search fallback to keyword search
- Disabled vector DB handling
- Empty results handling
-
Runtime errors for uninitialized manager
-
Batch Operations Tests (
test_vectordb_batch_operations.py): Tests for batch operations - Batch entity indexing (success, partial failure, all failures)
- Empty batch handling
- Large batch processing (100+ entities)
- Batch embedding generation
- Batch vector addition
- Vector DB disabled handling
-
Embedding errors
-
Backend Fallback Tests (
test_vectordb_backend_fallbacks.py): Tests for backend fallback behavior - Unsupported backends (Qdrant, Weaviate, Pinecone)
- Invalid backend handling
- Chroma backend import errors
- Chroma backend initialization errors
- Embedding model fallbacks
- Missing API keys (OpenAI, Cohere)
- Sentence-transformers import errors
Unit Test Details
Backend Tests
Tests for the abstract VectorDBBackend interface ensure all implementations follow the contract:
def test_abstract_methods():
"""Test that all abstract methods are defined."""
# Ensures all backends implement required methods
Chroma Backend Tests
Comprehensive tests for the Chroma implementation:
- Initialization (success and failure cases)
- Health checks
- Collection operations (create, exists, stats)
- Vector operations (add, search, update, delete)
- Error handling
Configuration Tests
Tests for configuration loading and validation:
- Default configuration
- Environment variable overrides
- Config file loading (JSON/YAML)
- Configuration validation
- Error handling for invalid configurations
Embedding Model Tests
Tests for embedding model initialization and usage:
- Sentence-transformers initialization
- OpenAI initialization
- Cohere initialization
- Embedding generation
- Error handling for missing dependencies
Manager Tests
Tests for the VectorDBManager singleton:
- Initialization (success, disabled, unsupported backend)
- Health checks
- Embedding generation
- Vector operations
- Singleton pattern enforcement
Indexing Tests
Tests for entity indexing:
- Single entity indexing
- Batch entity indexing
- Entity description generation
- Metadata generation
- Update and removal operations
- Error handling
Search Tests
Tests for semantic search:
- Basic semantic search
- Search with filters (domain, area)
- Similarity threshold filtering
- Hybrid search (semantic + keyword)
- Result ranking and boosting
- Error handling and fallbacks
Classification Tests
Tests for query intent classification:
- Intent classification (SEARCH, CONTROL, STATUS, etc.)
- Domain prediction
- Action extraction
- Entity extraction
- Parameter extraction
- Query refinement
History Tests
Tests for query history and learning:
- Storing query history
- Retrieving query history
- Query statistics
- Entity popularity calculation
- Personalized ranking
Relationship Tests
Tests for entity relationship graph:
- Building relationship graph
- Querying related entities
- Relationship statistics
- Various relationship types
Description Tests
Tests for entity description generation:
- Template-based descriptions
- Domain-specific templates
- Batch description generation
- Error handling
Integration Test Details
Indexing Integration Tests
Test entity indexing with real Home Assistant instances:
- Index single entity
- Index multiple entities
- Update entity index
- Remove entity from index
- Get indexing status
Search Integration Tests
Test semantic search with real data:
- Basic semantic search
- Search with domain filter
- Search with area filter
- Search with similarity threshold
- Hybrid search
Query Processing Integration Tests
Test query processing pipeline:
- Intent classification
- Domain prediction
- Action extraction
- Entity extraction
- Parameter extraction
- Query refinement
- Full pipeline processing
Entity Suggestions Integration Tests
Test entity suggestions with real data:
- Suggestions by area
- Suggestions by device
- Suggestions by domain
- Multiple relationship types
Performance Test Details
Indexing Performance
Benchmarks entity indexing with different batch sizes:
- Small batches (10 entities)
- Medium batches (50 entities)
- Large batches (100 entities)
- Measures entities per second
Search Performance
Benchmarks search operations:
- Different query types
- Response time measurements
- Result count analysis
Embedding Generation Performance
Benchmarks embedding generation:
- Different text lengths
- Generation time measurements
Concurrent Operations
Tests concurrent operations:
- Multiple simultaneous searches
- Performance under load
Large Entity Sets
Tests with large datasets (1000+ entities):
- Indexing performance
- Search performance
- Memory usage
Writing New Tests
Unit Test Template
"""Unit tests for module_name."""
import pytest
from unittest.mock import MagicMock, patch
from app.core.vectordb.module_name import FunctionName
class TestFunctionName:
"""Test the FunctionName class/function."""
@pytest.fixture
def mock_dependency(self):
"""Create a mock dependency."""
return MagicMock()
@pytest.mark.asyncio
async def test_success_case(self, mock_dependency):
"""Test successful operation."""
with patch("module.dependency", mock_dependency):
result = await FunctionName()
assert result is not None
@pytest.mark.asyncio
async def test_error_case(self):
"""Test error handling."""
with pytest.raises(ValueError):
await FunctionName(invalid_input)
Integration Test Template
"""Integration tests for module_name."""
import pytest
from app.core.vectordb.module_name import FunctionName
@pytest.mark.integration
@pytest.mark.asyncio
async def test_function_with_real_ha():
"""Test function with real Home Assistant."""
from app.core.vectordb.manager import get_vectordb_manager
manager = get_vectordb_manager()
if not manager.config.is_enabled():
pytest.skip("Vector DB is not enabled")
# Test with real data
result = await FunctionName()
assert result is not None
Performance Test Template
"""Performance tests for module_name."""
import time
import pytest
from app.core.vectordb.module_name import FunctionName
@pytest.mark.performance
@pytest.mark.asyncio
async def test_performance_benchmark():
"""Benchmark function performance."""
from app.core.vectordb.manager import get_vectordb_manager
manager = get_vectordb_manager()
if not manager.config.is_enabled():
pytest.skip("Vector DB is not enabled")
start_time = time.time()
result = await FunctionName()
elapsed_time = time.time() - start_time
assert result is not None
print(f"\nTime: {elapsed_time:.3f}s")
Test Best Practices
- Isolation: Each test should be independent and not rely on other tests
- Mocking: Use mocks for external dependencies
- Coverage: Aim for >90% code coverage
- Edge Cases: Test error conditions and edge cases
- Performance: Include performance tests for critical operations
- Documentation: Document test purpose and expected behavior
Troubleshooting
Tests Failing Due to Missing Dependencies
If tests fail with import errors for optional dependencies (e.g., chromadb, sentence_transformers), ensure they are installed:
Or install all vector DB dependencies:
Integration Tests Skipped
Integration tests are automatically skipped if Home Assistant is not available. To run them:
- Start Home Assistant
- Set
HA_URLandHA_TOKENenvironment variables - Run tests with
-m integrationmarker
Performance Tests Slow
Performance tests may take longer to run. Use pytest markers to skip them during regular test runs:
Continuous Integration
Tests are automatically run in CI/CD pipelines:
- Unit tests run on every commit
- Integration tests run on pull requests (if HA available)
- Performance tests run on scheduled basis
Contributing
When adding new Vector DB features:
- Write unit tests first (TDD approach)
- Add integration tests for real-world scenarios
- Add performance tests for critical operations
- Ensure >90% code coverage
- Update this documentation