Add embeddings setup documentation
This commit is contained in:
ifedan-ed
parent
1687647964
commit
9740650fbf
1 changed files with 268 additions and 0 deletions
268
EMBEDDINGS_SETUP.md
Normal file
268
EMBEDDINGS_SETUP.md
Normal file
|
|
@ -0,0 +1,268 @@
|
|||
# Embeddings & Semantic Search Setup
|
||||
|
||||
This guide explains how to set up and use the new vector-based semantic search for the Learning Hub.
|
||||
|
||||
## 🎯 What's New
|
||||
|
||||
- **Semantic search** - Find content by meaning, not just keywords
|
||||
- **3 search modes**:
|
||||
- **Keyword** (`/api/learning/search`) - Traditional text matching
|
||||
- **Semantic** (`/api/learning/search/semantic`) - AI-powered vector similarity
|
||||
- **Hybrid** (`/api/learning/search/hybrid`) - Combines both for best results
|
||||
- **Auto-embedding** - Content is automatically vectorized when created/updated
|
||||
- **HIPAA-compliant** - Uses Vertex AI embeddings (BAA available)
|
||||
|
||||
## 📋 Prerequisites
|
||||
|
||||
### 1. Install pgvector Extension
|
||||
|
||||
The database needs the `pgvector` extension for vector operations:
|
||||
|
||||
```bash
|
||||
# For PostgreSQL 16 on Ubuntu/Debian
|
||||
sudo apt-get install postgresql-16-pgvector
|
||||
|
||||
# For PostgreSQL 15
|
||||
sudo apt-get install postgresql-15-pgvector
|
||||
|
||||
# For Docker (add to Dockerfile or docker-compose)
|
||||
# The postgres:16-alpine base image doesn't include pgvector by default
|
||||
# You'll need to use a custom image or install at runtime
|
||||
```
|
||||
|
||||
**For Docker deployments**, use this postgres image instead:
|
||||
```yaml
|
||||
postgres:
|
||||
image: pgvector/pgvector:pg16
|
||||
# ... rest of your config
|
||||
```
|
||||
|
||||
### 2. Configure Embedding Provider
|
||||
|
||||
Add to your `.env` file:
|
||||
|
||||
```bash
|
||||
# Option 1: Vertex AI (HIPAA-eligible, recommended)
|
||||
EMBEDDING_MODEL=vertex_ai/text-embedding-005
|
||||
EMBEDDING_DIMENSIONS=768
|
||||
VERTEX_PROJECT=your-gcp-project-id
|
||||
VERTEX_LOCATION=us-central1
|
||||
GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
|
||||
|
||||
# Option 2: LiteLLM Proxy (routes to any provider)
|
||||
LITELLM_API_BASE=http://localhost:4000
|
||||
LITELLM_API_KEY=your-key
|
||||
EMBEDDING_MODEL=text-embedding-005 # LiteLLM will route to configured provider
|
||||
|
||||
# Option 3: OpenAI (NOT HIPAA-eligible, fallback only)
|
||||
OPENAI_API_KEY=sk-your-key
|
||||
# Uses text-embedding-3-small automatically
|
||||
```
|
||||
|
||||
## 🚀 Available Vertex AI Embedding Models
|
||||
|
||||
Tested and working via LiteLLM:
|
||||
|
||||
| Model | Dimensions | Use Case | HIPAA |
|
||||
|-------|-----------|----------|-------|
|
||||
| **vertex_ai/text-embedding-005** | 768 | English + code (recommended) | ✅ Yes |
|
||||
| **vertex_ai/gemini-embedding-001** | 768-3072 | Multilingual + code, best quality | ✅ Yes |
|
||||
| **vertex_ai/text-multilingual-embedding-002** | 768 | Multilingual focus | ✅ Yes |
|
||||
|
||||
## 🔧 Setup Steps
|
||||
|
||||
### 1. Database Migration
|
||||
|
||||
The database will automatically:
|
||||
- Enable the `pgvector` extension
|
||||
- Add `embedding vector(768)` column to `learning_content`
|
||||
- Create IVFFLAT index for fast similarity search (after 10+ embeddings)
|
||||
|
||||
Just restart your server after installing pgvector.
|
||||
|
||||
### 2. Generate Embeddings for Existing Content
|
||||
|
||||
Two options:
|
||||
|
||||
**Option A: Admin API (recommended)**
|
||||
```bash
|
||||
curl -X POST http://localhost:3000/api/admin/learning/embeddings/generate \
|
||||
-H "Authorization: Bearer YOUR_JWT_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"regenerateAll": false}'
|
||||
```
|
||||
|
||||
**Option B: Via Admin Panel**
|
||||
- Go to Admin → Learning Hub → Settings
|
||||
- Click "Generate Embeddings" button
|
||||
- Check status at `/api/admin/learning/embeddings/status`
|
||||
|
||||
### 3. Verify Setup
|
||||
|
||||
Check embedding status:
|
||||
```bash
|
||||
curl http://localhost:3000/api/admin/learning/embeddings/status \
|
||||
-H "Authorization: Bearer YOUR_JWT_TOKEN"
|
||||
```
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"enabled": true,
|
||||
"total": 50,
|
||||
"withEmbeddings": 50,
|
||||
"missing": 0,
|
||||
"model": "vertex_ai/text-embedding-005",
|
||||
"dimensions": 768
|
||||
}
|
||||
```
|
||||
|
||||
## 🔍 Using Semantic Search
|
||||
|
||||
### Keyword Search (existing)
|
||||
```bash
|
||||
GET /api/learning/search?q=pneumonia
|
||||
```
|
||||
Returns exact text matches in title/subject/body.
|
||||
|
||||
### Semantic Search (new)
|
||||
```bash
|
||||
GET /api/learning/search/semantic?q=childhood breathing problems&limit=10&threshold=0.5
|
||||
```
|
||||
Returns content similar by **meaning** (e.g., finds "pediatric asthma" articles).
|
||||
|
||||
**Parameters:**
|
||||
- `q` (required) - Search query
|
||||
- `limit` (optional, default 10, max 50) - Max results
|
||||
- `threshold` (optional, default 0.5) - Similarity threshold (0-1, higher = more similar)
|
||||
- `contentType` (optional) - Filter by type: article, quiz, pearl, presentation
|
||||
|
||||
### Hybrid Search (recommended)
|
||||
```bash
|
||||
GET /api/learning/search/hybrid?q=fever management
|
||||
```
|
||||
Combines keyword + semantic for best results. Automatically deduplicates and ranks by relevance.
|
||||
|
||||
## 🔬 How It Works
|
||||
|
||||
1. **Content Creation/Update**:
|
||||
- Text is extracted from `title`, `subject`, and `body` (HTML stripped)
|
||||
- Sent to embedding model (Vertex AI)
|
||||
- Returns 768-dimensional vector
|
||||
- Stored in `learning_content.embedding` column
|
||||
|
||||
2. **Semantic Search**:
|
||||
- Query text → embedding vector
|
||||
- PostgreSQL pgvector computes cosine similarity
|
||||
- Returns top N most similar documents
|
||||
- Similarity score 0-1 (1 = identical, 0 = unrelated)
|
||||
|
||||
3. **Hybrid Search**:
|
||||
- Runs both keyword + semantic searches in parallel
|
||||
- Merges results (semantic first for quality)
|
||||
- Deduplicates by content ID
|
||||
- Sorts by relevance score
|
||||
|
||||
## 💰 Cost Estimate (Vertex AI)
|
||||
|
||||
**Titan Text Embeddings (AWS) pricing:**
|
||||
- ~$0.10 per 1M tokens
|
||||
- Average article: 2,000 words (~2,700 tokens) = $0.00027
|
||||
- 1,000 articles: ~**$0.27 one-time**
|
||||
- Search queries: ~500 tokens = $0.00005 per query
|
||||
|
||||
**Google Vertex AI pricing:**
|
||||
- text-embedding-005: $0.025 per 1M characters
|
||||
- Average article: 10,000 chars = $0.00025
|
||||
- 1,000 articles: ~**$0.25 one-time**
|
||||
- Search queries: ~$0.0000125 per query
|
||||
|
||||
## 🐛 Troubleshooting
|
||||
|
||||
### "pgvector extension not available"
|
||||
- Install: `apt-get install postgresql-16-pgvector`
|
||||
- For Docker: Use `pgvector/pgvector:pg16` image
|
||||
|
||||
### "Embeddings not configured"
|
||||
- Verify `.env` has `VERTEX_PROJECT` or `LITELLM_API_BASE` or `OPENAI_API_KEY`
|
||||
- Check service account credentials: `GOOGLE_APPLICATION_CREDENTIALS`
|
||||
- Test: `curl http://localhost:3000/api/admin/learning/embeddings/status`
|
||||
|
||||
### "Embedding generation failed"
|
||||
- Check logs for API errors
|
||||
- Verify Vertex AI API is enabled in GCP
|
||||
- Verify service account has `aiplatform.endpoints.predict` permission
|
||||
- Check content isn't empty (skips empty bodies)
|
||||
|
||||
### "No results from semantic search"
|
||||
- Check if embeddings exist: `/api/admin/learning/embeddings/status`
|
||||
- Lower threshold: `?threshold=0.3` (default 0.5)
|
||||
- Verify pgvector index exists: `\di` in psql
|
||||
|
||||
## 📊 Performance
|
||||
|
||||
- **Embedding generation**: ~500ms per article (Vertex AI)
|
||||
- **Search latency**:
|
||||
- Keyword: 10-50ms
|
||||
- Semantic: 20-100ms (with IVFFLAT index)
|
||||
- Hybrid: 30-150ms
|
||||
- **Index build time**: ~1-5 seconds per 1,000 articles
|
||||
|
||||
## 🔐 Security & Compliance
|
||||
|
||||
- **HIPAA-eligible**: Vertex AI supports BAA (Business Associate Agreement)
|
||||
- **Data retention**: Embeddings stored in your database only
|
||||
- **No PHI**: Only article content (not patient data) is embedded
|
||||
- **Encryption**: TLS in transit, at-rest encryption via PostgreSQL
|
||||
|
||||
## 🎓 Example Queries
|
||||
|
||||
**Before (keyword):**
|
||||
```
|
||||
Query: "fever in babies"
|
||||
Results: Only articles with exact words "fever" or "babies"
|
||||
```
|
||||
|
||||
**After (semantic):**
|
||||
```
|
||||
Query: "fever in babies"
|
||||
Results:
|
||||
- Infant hyperthermia management (similarity: 0.89)
|
||||
- Pediatric fever evaluation (similarity: 0.87)
|
||||
- Febrile seizures in toddlers (similarity: 0.82)
|
||||
- Neonatal temperature regulation (similarity: 0.78)
|
||||
```
|
||||
|
||||
**Hybrid (best):**
|
||||
```
|
||||
Query: "asthma"
|
||||
Results:
|
||||
- Childhood asthma management (keyword + semantic: 1.0)
|
||||
- Pediatric breathing difficulties (semantic: 0.91)
|
||||
- Reactive airway disease (semantic: 0.86)
|
||||
- Bronchiolitis vs asthma (keyword: 1.0)
|
||||
```
|
||||
|
||||
## 📚 API Reference
|
||||
|
||||
### Admin Endpoints
|
||||
|
||||
- `POST /api/admin/learning/embeddings/generate` - Backfill embeddings
|
||||
- `GET /api/admin/learning/embeddings/status` - Check status
|
||||
- `GET /api/admin/learning/stats` - Includes embedding count
|
||||
|
||||
### User Endpoints
|
||||
|
||||
- `GET /api/learning/search` - Keyword search
|
||||
- `GET /api/learning/search/semantic` - Semantic search
|
||||
- `GET /api/learning/search/hybrid` - Hybrid search (recommended)
|
||||
|
||||
All endpoints require authentication (JWT token).
|
||||
|
||||
---
|
||||
|
||||
**Questions?** Check logs for detailed error messages, or review the code in:
|
||||
- `/src/utils/embeddings.js` - Core embedding logic
|
||||
- `/src/routes/learningHub.js` - Search endpoints
|
||||
- `/src/routes/learningAdmin.js` - Admin management
|
||||
Loading…
Reference in a new issue