519f2e572a
Search - Retrieval was hybrid in name only: the keyword filter was applied to the SQL query, so results were the *intersection* of the two rankers. A question that matched the meaning but not the literal string could never be returned. It is now a union, fused with Reciprocal Rank Fusion (a text rank and a cosine distance are not on comparable scales, so RRF uses only their orderings). - Added a generated `search_vector` tsvector + GIN index, so the lexical half is ranked full text rather than ILIKE substring matching. - Chose Postgres + pgvector over OpenSearch/Elasticsearch: a search cluster would add a second datastore to keep in sync and a JVM on this host, to replace an index Postgres maintains inside the same transaction. - Removed the keyword-only mode. It looks precise but silently drops the question that asks the same thing in different words. Embeddings — measured on 500 real questions, using each question's own explanation as a paraphrase query (known answer, no hand labelling): bge-small (local CPU, 384d) R@1 0.840 R@5 0.953 186ms/query bge-m3 (LiteLLM proxy, 1024d) R@1 0.847 R@5 0.973 93ms/query BGE-M3 wins on both quality and latency and needs no extra credential, since llm.danvics.com already serves `openrouter-bge-m3`. Three gaps this exposed, all fixed: - Nothing recorded which model produced a stored vector, so changing models silently mixed incomparable spaces. `embedding_model` / `embedded_at` now stamp every vector, `GET /admin/embedding/health` reports current vs stale vs missing, and regeneration defaults to stale-only. - The generator read the model from env while the stamp read a Redis override, so a vector could be labelled with a model that did not produce it. Both now resolve through one function, with a regression test. - Embedding at creation is best effort, and a failure left a question invisible to semantic search forever. `retry_missing_embeddings` runs every 15 minutes via Celery beat and backfills missing or stale rows. - Query embeddings are cached in Redis per model, so typing is not a network round-trip per keystroke. `dimensions` is only sent to OpenAI's embedding-3 family; BGE-M3 rejects it. Tests: 8 new backend tests (union not intersection, fusion ordering, per-ranker failure degradation, provenance stamping, stale/missing accounting, generator and stamp agreement). Full suites green: 95 backend, 127 frontend, build clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yhHB8Pc7oQqyqn2Vo9DXA
29 lines
978 B
Python
29 lines
978 B
Python
from celery import Celery
|
|
|
|
from app.config import settings
|
|
from app.logging_config import setup_logging
|
|
|
|
# Configure structured JSON logging for Celery workers
|
|
setup_logging(settings.LOG_LEVEL)
|
|
|
|
celery_app = Celery(
|
|
"quiz_tasks",
|
|
broker=settings.REDIS_URL,
|
|
backend=settings.REDIS_URL,
|
|
include=["app.tasks.pdf_tasks", "app.tasks.quiz_tasks"],
|
|
)
|
|
celery_app.conf.task_serializer = "json"
|
|
celery_app.conf.result_serializer = "json"
|
|
celery_app.conf.accept_content = ["json"]
|
|
celery_app.conf.worker_hijack_root_logger = False # Don't override our JSON logging
|
|
celery_app.conf.broker_connection_retry_on_startup = True
|
|
|
|
# Questions whose embedding failed at creation would otherwise never be
|
|
# searchable semantically; this sweeps them up. It normally finds nothing.
|
|
celery_app.conf.beat_schedule = {
|
|
"retry-missing-embeddings": {
|
|
"task": "retry_missing_embeddings",
|
|
"schedule": 900.0, # every 15 minutes
|
|
},
|
|
}
|
|
celery_app.conf.timezone = "UTC"
|