Yiorgis Gozadinos
7ccabf4fb9
Log to console with logfire only in development
2025-12-11 10:47:18 +02:00
Yiorgis Gozadinos
5edae13cc0
Run logfire regardless of environment
2025-12-11 10:43:25 +02:00
Yiorgis Gozadinos
2161727716
Add auto_vacuum config option to control automatic vacuuming
2025-12-10 11:42:14 +02:00
Yiorgis Gozadinos
ed809759d5
Fix mxbai rerank default model
2025-12-10 11:39:33 +02:00
Yiorgis Gozadinos
68457f251e
Do not generate base64 images by default
2025-12-09 18:42:56 +02:00
Yiorgis Gozadinos
156bba3359
Infinite scroll lazy load document list in inspector
2025-12-09 17:48:27 +02:00
Yiorgis Gozadinos
5a593e928e
Set max_context_items to 10
2025-12-09 12:47:42 +02:00
Yiorgis Gozadinos
802f058205
Move search related config under SearchConfig
2025-12-09 12:28:28 +02:00
Yiorgis Gozadinos
4da50873bd
Make search limit configurable
2025-12-09 11:46:43 +02:00
Yiorgis Gozadinos
a56e1ba67c
Simplify import_document, update_document
2025-12-08 18:19:12 +02:00
Yiorgis Gozadinos
67684c799b
Rebuild in batches to avoid creating multiple lancedb versions
2025-12-08 15:56:41 +02:00
Yiorgis Gozadinos
1b96937ec6
Update dependencies
2025-12-08 15:56:22 +02:00
Yiorgis Gozadinos
06b4f6a224
Add format parameter for text-to-DoclingDocument conversion
2025-12-08 15:56:21 +02:00
Yiorgis Gozadinos
1b14a4643d
Remove unused bounding box calculations
2025-12-08 15:56:21 +02:00
Yiorgis Gozadinos
9a5f8e4368
Add type-aware context expansion for search results
2025-12-08 15:56:21 +02:00
Yiorgis Gozadinos
dab04668bd
Dependency updates
2025-12-08 15:56:02 +02:00
Yiorgis Gozadinos
4f0f214f26
Remove unecessary tests, add no coverage pragmas where appropriate
2025-12-08 15:56:02 +02:00
Yiorgis Gozadinos
2572125804
Move embedding logic from ChunkRepository to client._ensure_chunks_embedded()
2025-12-08 15:56:02 +02:00
Yiorgis Gozadinos
e97d235c98
Unify update_document, update_document_fields
2025-12-08 15:56:02 +02:00
Yiorgis Gozadinos
16ba3a9962
Remove create_chunks_for_document from repositories
2025-12-08 15:56:01 +02:00
Yiorgis Gozadinos
17c7147a49
Remove pre-processor, no longer needed
2025-12-08 15:56:01 +02:00
Yiorgis Gozadinos
69b1afa534
Remove _create_and_chunk & _update_and_rechunk, keep storage independent of chunking
2025-12-08 15:55:31 +02:00
Yiorgis Gozadinos
0ab43d62cd
Refactor create_document(), create_document_from_source() to use primitives
2025-12-08 15:55:30 +02:00
Yiorgis Gozadinos
e299aed788
contextualize() and embed_chunks() utilities in embeddings
2025-12-08 15:55:30 +02:00
Yiorgis Gozadinos
4fbb4609b6
HaikuRag.chunk() utility method
2025-12-08 15:55:30 +02:00
Yiorgis Gozadinos
ace915473c
HaikuRag.convert() utility method
2025-12-08 15:55:30 +02:00
Yiorgis Gozadinos
ab57c55eaa
Drop ChunkWithMetadata, use good old Chunk
2025-12-08 15:55:30 +02:00
Yiorgis Gozadinos
33e6a36290
Make document import/update handling consistent for DoclingDocument
2025-12-08 15:55:29 +02:00
Yiorgis Gozadinos
60ab864fc1
Add docling_document_json to update_document_fields() and
2025-12-08 15:55:29 +02:00
Yiorgis Gozadinos
8e30f67266
Introduce import_document(), remove chunks from create_document()
2025-12-08 15:55:29 +02:00
Yiorgis Gozadinos
de14bb9f90
Avoid citation meta inside LLM context
2025-12-08 15:55:28 +02:00
Yiorgis Gozadinos
a57203065a
Fix test
2025-12-08 15:55:28 +02:00
Yiorgis Gozadinos
84f25f0627
Visual grounding in ag-ui-example
2025-12-08 15:55:28 +02:00
Yiorgis Gozadinos
6e31ea38a8
Remove bounding boxes from citations, pass sources meta info from ResearchContext
2025-12-08 15:55:28 +02:00
Yiorgis Gozadinos
75a086bae0
Visualize chunk command, format citations to show chunk id
2025-12-08 15:55:28 +02:00
Yiorgis Gozadinos
910d382748
Introduce visualize_chunk(), refactor inspector to use it
2025-12-08 15:55:27 +02:00
Yiorgis Gozadinos
bfdaa4652a
Configure logging regardsless of the env
2025-12-08 15:55:27 +02:00
Yiorgis Gozadinos
9382a6b132
expose citations in ag-ui-example
2025-12-08 15:55:27 +02:00
Yiorgis Gozadinos
d808c6c425
Simplify citations in qa & graph agents
2025-12-08 15:55:27 +02:00
Yiorgis Gozadinos
16a97cc140
Properly expand context in the case of docling documents and merge chunks and meta
2025-12-08 15:55:26 +02:00
Yiorgis Gozadinos
ac439b6c69
Store raw text (do not contextualize) in chunk.text. Embed with contextualized text.
2025-12-08 15:55:26 +02:00
Yiorgis Gozadinos
0d8691342a
Show bounding boxes for chunks in inspector
2025-12-08 15:55:26 +02:00
Yiorgis Gozadinos
6ab990b2ed
Always generate page images, default in docling-serve
2025-12-08 15:54:57 +02:00
Yiorgis Gozadinos
c69b934eac
Incorporate additional docling meta in search(), expand_context() and relevant qa/research agents
2025-12-08 15:54:56 +02:00
Yiorgis Gozadinos
fe4ac530be
DB migration
2025-12-08 15:54:56 +02:00
Yiorgis Gozadinos
d267b2c433
LRU cache for DoclingDocument
2025-12-08 15:54:31 +02:00
Yiorgis Gozadinos
a5d8af07e9
Resolve DoclingDocument
2025-12-08 15:54:31 +02:00
Yiorgis Gozadinos
84aa46a10b
Update rebuild to handle DoclingDocument json
2025-12-08 15:54:31 +02:00
Yiorgis Gozadinos
ad4e79ab5d
Store DoclingDocument json
2025-12-08 15:54:31 +02:00
Yiorgis Gozadinos
343272d6fb
Chunkers return chunks with metadata, i.e. (refs, labels, headings, page_numbers)
2025-12-08 15:54:31 +02:00
Yiorgis Gozadinos
44e8d7b340
Update Document & Chunk schemas to include DoclingDocument references.
2025-12-08 15:54:30 +02:00
Yiorgis Gozadinos
80447c9024
Make download_models() a HaikuRAG client method. Update app to show progress of what happens.
2025-12-04 16:30:26 +02:00
Yiorgis Gozadinos
78f1d0cef5
No warnings, just info on db upgrade
2025-12-03 08:51:25 +02:00
Yiorgis Gozadinos
33b254ae3d
Always upgrade database on access. Explicitly create db by running init or HaikuRAG(path, create=True)
2025-12-02 16:49:55 +02:00
Yiorgis Gozadinos
3525fae625
Use EmbeddingModelConfig similar to ModelConfig for embeddings
2025-12-02 11:55:31 +02:00
Yiorgis Gozadinos
f07590bd5b
vb
2025-12-01 17:58:47 +02:00
Yiorgis Gozadinos
e548a51ff7
Optimized rebuild --embed-only to use batch updates via LanceDB merge_insert instead of individual chunk updates
2025-12-01 17:53:27 +02:00
Yiorgis Gozadinos
3e18a5f2ff
Update project description
2025-11-29 11:18:53 +02:00
Yiorgis Gozadinos
10aa2a7cf2
vb
2025-11-28 10:15:59 +02:00
Yiorgis Gozadinos
88a4edb537
--embed-only, --rechunk options in rebuild command
2025-11-28 10:11:52 +02:00
Yiorgis Gozadinos
8a0cc04181
Add opencv-python-headless to docling optional dependency, required for TableStructureModel which uses TableFormer for table detection.
2025-11-27 16:22:37 +02:00
Yiorgis Gozadinos
eee29578ba
vb
2025-11-27 15:24:39 +02:00
Yiorgis Gozadinos
b4602b0cab
use OcrAutoOptions for automatic OCR engine selection.
2025-11-27 15:23:17 +02:00
Yiorgis Gozadinos
51104ddb5a
Use httpx for docling-serve
2025-11-27 15:06:13 +02:00
Yiorgis Gozadinos
3959ce39f7
vb
2025-11-27 13:06:04 +02:00
Yiorgis Gozadinos
ed53ec461a
Convert prefetch_models() to async
2025-11-26 18:08:48 +02:00
Yiorgis Gozadinos
e40c352026
Switch from requests to httpx.AsyncClient for docling-serve
2025-11-26 18:00:29 +02:00
Yiorgis Gozadinos
14bdb7cb27
async versions for convert_file() convert_text()
2025-11-26 17:46:25 +02:00
Yiorgis Gozadinos
ca2445d869
vb
2025-11-26 10:33:46 +02:00
Yiorgis Gozadinos
0303e6a8cd
Support LMStudio
2025-11-26 10:31:40 +02:00
Yiorgis Gozadinos
283c44867d
Generate the config from the AppConfig model directly, add tests
2025-11-26 10:17:22 +02:00
Yiorgis Gozadinos
c22a5bb0ce
vb
2025-11-25 13:44:48 +02:00
Yiorgis Gozadinos
e83978f88b
Revert embeddings to use flat config
2025-11-25 12:51:58 +02:00
Yiorgis Gozadinos
ba4dddeeaf
Make thinking false by default
2025-11-25 12:35:38 +02:00
Yiorgis Gozadinos
a9701616c8
Rename model to name under model
2025-11-25 12:28:43 +02:00
Yiorgis Gozadinos
1bb7b6d5bd
Add support for per-model configuration settings including thinking, temperature and max_tokens
2025-11-25 12:14:24 +02:00
Yiorgis Gozadinos
13d3a366e8
Enhanced ActivitySnapshot events with richer structured data
2025-11-24 16:18:27 +02:00
Yiorgis Gozadinos
03c1f356b2
Update qa agent search limit to 5
2025-11-24 15:01:31 +02:00
Yiorgis Gozadinos
0b92eae75f
Docs
2025-11-22 17:46:21 +02:00
Yiorgis Gozadinos
2c6efc55c6
Make search a screen
2025-11-21 17:06:30 +02:00
Yiorgis Gozadinos
0d126d894e
Keep documents when moving from main view to search and back
2025-11-21 16:48:35 +02:00
Yiorgis Gozadinos
59eba31267
Handle missing textual dependencies
2025-11-21 16:48:35 +02:00
Yiorgis Gozadinos
8450c07085
Fix markdown
2025-11-21 16:48:35 +02:00
Yiorgis Gozadinos
82831c8941
Search in inspector
2025-11-21 16:48:34 +02:00
Yiorgis Gozadinos
2c1c530f3a
Fix tabbing
2025-11-21 16:48:34 +02:00
Yiorgis Gozadinos
89a2668c25
inspector command that runs a textual app to browse documents and chunks in the database
2025-11-21 16:48:34 +02:00
Yiorgis Gozadinos
ada2b66d5b
vb
2025-11-21 16:38:52 +02:00
Yiorgis Gozadinos
8c02e865c7
Change default embedding model to qwen3-embedding:4b
2025-11-21 16:30:41 +02:00
Yiorgis Gozadinos
6d8ab2575d
Clarify refine_factor, set default to 30
2025-11-21 15:53:24 +02:00
Yiorgis Gozadinos
2f54ee7d59
Replace allow_create with cleaner read_only. Elaborate on the test_info tests
2025-11-21 14:37:33 +02:00
Yiorgis Gozadinos
fdcf9a1a12
Show index stats in cli info command
2025-11-21 13:54:04 +02:00
Yiorgis Gozadinos
6a2f33b464
Create vector index from CLI, use configurable refine_factor
2025-11-21 13:54:01 +02:00
Yiorgis Gozadinos
c45d507213
Allow custom config when building research/deep ask graphs. Closes #149
2025-11-21 12:37:41 +02:00
Yiorgis Gozadinos
5c4a32f1c8
Fix ACTIVITY_SNAPSHOT payloads. Closes #150
2025-11-21 12:27:54 +02:00
Yiorgis Gozadinos
09900f8e8b
vb
2025-11-19 13:08:37 +02:00
Yiorgis Gozadinos
30a0bbcff2
Accept both single chunks and lists for batch insertion in ChunkRepository.create()
2025-11-19 12:35:02 +02:00
Yiorgis Gozadinos
5089ea4203
New update_document_fields() method for partial document updates
2025-11-19 11:37:36 +02:00
Yiorgis Gozadinos
0ea5f1e10e
Update dependencies
2025-11-19 10:24:37 +02:00
Yiorgis Gozadinos
826986708d
vb
2025-11-18 15:52:52 +02:00
Yiorgis Gozadinos
0bb3f4301c
Additional options for document conversion with docling
2025-11-18 15:44:04 +02:00
Yiorgis Gozadinos
2dca72d670
Revert "Handle client-side tool calls in graph ag-ui emitter"
...
This reverts commit b1d19651d5
2025-11-18 15:19:28 +02:00
Yiorgis Gozadinos
cc09ce87c2
Increase reranking candidate retrieval multiplier from 3x to 10x for improved result quality
2025-11-18 14:35:57 +02:00
Yiorgis Gozadinos
b1d19651d5
Handle client-side tool calls in graph ag-ui emitter
2025-11-18 14:10:21 +02:00
Yiorgis Gozadinos
efbd5b92f7
vb
2025-11-17 16:30:39 +02:00
Yiorgis Gozadinos
91de82ec14
docling-serve chunker implementation. Lacks tests
2025-11-17 13:05:06 +02:00
Yiorgis Gozadinos
75cbdb47a8
Add chunker configuration options
...
- Add `chunker_type` config field to choose between "hybrid" (default) and "hierarchical" chunking strategies
- Add `chunking_merge_peers` config field for HybridChunker (default: true)
- Add `chunking_use_markdown_tables` config field to control table serialization format (default: false, matching docling's default)
2025-11-17 13:05:06 +02:00
Yiorgis Gozadinos
de5bb117fe
Chunker abstraction
2025-11-17 13:05:06 +02:00
Yiorgis Gozadinos
541552215e
Switch to using HuggingFace tokenizers
2025-11-17 13:05:05 +02:00
Yiorgis Gozadinos
fe7d0c5c24
Implement docling-serve converter
2025-11-17 13:05:05 +02:00
Yiorgis Gozadinos
92def1c6a5
Rebase after haiku.rag-slim migration
2025-11-17 13:05:05 +02:00
Yiorgis Gozadinos
eb2b65135a
Lazy load docling & friends
2025-11-17 13:05:05 +02:00
Yiorgis Gozadinos
7ba1c2376e
Introduce converters for supporting more than local docling document conversion. Transform existing FileReader and utils to "docling-local" converter
2025-11-17 13:05:05 +02:00
Yiorgis Gozadinos
636d5261e6
vb
2025-11-14 22:29:37 +02:00
Yiorgis Gozadinos
c8999a89b8
Update and lock pydantic-ai-slim
2025-11-14 11:26:09 +02:00
Yiorgis Gozadinos
6eb0d59a28
vb
2025-11-13 15:00:57 +02:00
Yiorgis Gozadinos
4599dd1481
Drop disable_auto_create flag, simply create only on write operations
2025-11-13 13:53:11 +02:00
Yiorgis Gozadinos
8e802d4a91
Default vacuum_retention_seconds increased from 60 seconds to 86400 seconds (1 day) for better version retention in typical workflows
2025-11-13 13:31:53 +02:00
Yiorgis Gozadinos
2c43f034de
Refactor ag-ui-research. Drop human-in-the-loop, use MemoryObjectSendStream to merge the graph and agent streams together
2025-11-13 13:22:53 +02:00
Yiorgis Gozadinos
4a623934b2
Log additional ag-ui events in the cli renderer
2025-11-13 13:22:52 +02:00
Yiorgis Gozadinos
0c8d2838b2
Better typing
2025-11-13 13:22:52 +02:00
Yiorgis Gozadinos
68d1127e46
Compute and emit state deltas instead of full state snapshots by default
2025-11-13 13:22:52 +02:00
Yiorgis Gozadinos
657245f686
Simplify InsightRecord, GapRecord
2025-11-13 13:22:52 +02:00
Yiorgis Gozadinos
2060898018
Use a node factor for common nodes
2025-11-13 13:22:52 +02:00
Yiorgis Gozadinos
032625b0bb
Refactor everything graph-related under the graph module
2025-11-13 13:22:03 +02:00
Yiorgis Gozadinos
8f0597e89e
Update deep ask graph
2025-11-13 13:22:03 +02:00
Yiorgis Gozadinos
d5b6ad544d
Bring back verbose
2025-11-13 13:22:02 +02:00
Yiorgis Gozadinos
60ef8fe04c
AGUI starlette server
2025-11-13 13:22:02 +02:00
Yiorgis Gozadinos
2c1f79614a
Use StateSnapshot updates
2025-11-13 13:22:02 +02:00
Yiorgis Gozadinos
663a7fac9a
Replace console/stream with AG-UI event protocol
2025-11-13 13:22:01 +02:00
Yiorgis Gozadinos
da77e0f2f9
Convert ResearchState from dataclass to Pydantic model
2025-11-13 13:22:01 +02:00
Yiorgis Gozadinos
765ebb698f
AG-UI Event-emitter for graphs. Compute state-deltas from from BaseModel pydantic state models
2025-11-13 13:22:01 +02:00
Yiorgis Gozadinos
908296e155
Do not create unecessary DoclingDocument when chunks are provided. Rename private _create_with_docling & _update_with_docling to _create_and_chunk & _update_and_rechunk
2025-11-13 13:16:15 +02:00
Yiorgis Gozadinos
942ad2818f
Filereader error messages now include both original exception details and file path for easier debugging
2025-11-10 18:26:25 +02:00
Yiorgis Gozadinos
34f50f37a0
vb
2025-11-07 14:44:58 +02:00
Yiorgis Gozadinos
b6650d0486
Introduce delete_orphans in monitor (default=false) to delete db entries that do not resolve in the filesystem
2025-11-07 14:40:13 +02:00
Yiorgis Gozadinos
31713e5fb6
Properly skip (and log as such) documents without changing updated_at when the md5 hash is unchanged
2025-11-07 14:00:05 +02:00
Yiorgis Gozadinos
9ee64a33e0
Remove A2A from haiku.rag, turn it into an example
2025-11-07 13:31:51 +02:00
Yiorgis Gozadinos
0b696d9629
Remove deprecated use of env variables for configuration
2025-11-07 10:54:32 +02:00
Yiorgis Gozadinos
b51283c647
Use haiku.rag.yaml as the default config filename everywhere
2025-11-07 10:42:57 +02:00
Yiorgis Gozadinos
3c557d5bfe
Make --config a global CLI parameter, add it as a param to HaikuRAGApp
2025-11-07 10:31:55 +02:00
Yiorgis Gozadinos
0e3f2ebd67
vb
2025-11-06 16:13:53 +02:00
Yiorgis Gozadinos
ce6dcb8b64
Pin critical packages
2025-11-06 16:09:31 +02:00
Yiorgis Gozadinos
782fbafb3d
Make graph settings part of config for both deep ask and research graphs
2025-11-06 14:31:22 +02:00
Yiorgis Gozadinos
17d6a1dfee
Bring back max_concurrency by means of asyncio.Semaphore
2025-11-06 11:02:32 +02:00
Yiorgis Gozadinos
6858760bca
Remove max_concurrency, not supported (yet) when we .map() in the beta Graph API
2025-11-06 11:02:32 +02:00
Yiorgis Gozadinos
38861da549
Go back and separate common patters in graph_common
2025-11-06 11:02:32 +02:00
Yiorgis Gozadinos
9075dac23a
Remove graph common and move its content to deep QA, research.
2025-11-06 11:02:32 +02:00
Yiorgis Gozadinos
bc71ad6fb0
Refactor research, deep ask to work with new beta pyndtic AI graph
2025-11-06 11:02:32 +02:00
Yiorgis Gozadinos
b8bc6d9b68
Let ruff know about our package structure
2025-11-05 17:47:27 +02:00
Yiorgis Gozadinos
21bb7b5aab
Version bump & Changelog
2025-11-05 13:16:03 +02:00
Yiorgis Gozadinos
74d9c60ba0
Update docling
2025-11-05 13:11:30 +02:00
Yiorgis Gozadinos
ba8e503cb1
Depend on pydantic-ai-slim with extras
2025-11-05 11:37:06 +02:00
Yiorgis Gozadinos
f06dac7102
Update READMEs
2025-11-05 10:47:00 +02:00
Yiorgis Gozadinos
be2d8ea8f4
Catch import errors seperately to raise errors about missing dependencies.
2025-11-05 10:31:07 +02:00
Yiorgis Gozadinos
5cd951a5f3
When checking for updates, use haiku.rag-slim as the package, since it is always installed
2025-11-05 10:30:59 +02:00
Yiorgis Gozadinos
b02514dc82
Make docling optional, depend on docling-core
2025-11-04 18:15:04 +02:00
Yiorgis Gozadinos
2f9c907031
Restructure into uv workspace to support minimal and full installations
2025-11-04 17:59:12 +02:00
