Merge pull request #303 from ggozad/feat/param-tune

Set appropriate temperature and max_tokens defaults

CHANGELOG.md

@@ -3,6 +3,10 @@
 
 ### Changed
 
+- **Default model temperatures**: Set task-appropriate temperature defaults — 0.3 for QA, research, and title generation; 0.0 for RLM and picture description. Previously unset (provider defaults, typically 0.7–1.0).
+- **QA thinking enabled by default**: `enable_thinking` now defaults to `True` for QA agent, improving answer quality with reasoning models.
+- **Default title max_tokens**: Set `max_tokens=100` for title generation model to keep titles concise
+- **Evaluation judge**: Set `temperature=0.0` and `enable_thinking=True` for deterministic, higher-quality judging. Removed unused judge config from retrieval benchmarks.
 - **Test suite cleanup**: Removed stale VCR cassettes, dead fixtures, orphaned directories, and redundant tests. Strengthened weak assertions across search, context enhancement, and converter tests. Relocated misplaced `SearchResult._get_primary_label` test to `test_search.py`
 - **Parallel test execution**: Added `pytest-xdist` and enabled parallel test runs by default (`-n auto`), reducing test suite time from ~3.5 min to ~2 min
 

docs/configuration/index.md

@@ -44,7 +44,7 @@ qa:
   model:
     provider: ollama
     name: gpt-oss
-    enable_thinking: false
+    enable_thinking: true
 ```
 
 ## Complete Configuration Example
@@ -84,7 +84,8 @@ qa:
   model:
     provider: ollama
     name: gpt-oss
-    enable_thinking: false
+    enable_thinking: true
+    temperature: 0.3
   max_iterations: 2
   max_concurrency: 1
 
@@ -93,6 +94,7 @@ research:
     provider: ""  # Empty to use qa settings
     name: ""
     enable_thinking: false
+    temperature: 0.3
   max_iterations: 3
   max_concurrency: 1
 
@@ -122,6 +124,8 @@ processing:
     provider: ollama
     name: gpt-oss
     enable_thinking: false
+    temperature: 0.3
+    max_tokens: 100
   conversion_options:
     do_ocr: true
     force_ocr: false
@@ -156,7 +160,7 @@ custom_config = AppConfig(
         model=ModelConfig(
             provider="openai",
             name="gpt-4o",
-            temperature=0.7
+            temperature=0.3
         )
     ),
     embeddings=EmbeddingsConfig(

docs/configuration/processing.md

@@ -120,6 +120,7 @@ conversion_options:
     model:
       provider: ollama             # ollama, openai, or custom
       name: ministral-3            # VLM model name
+      temperature: 0.0             # Default: 0.0 (factual descriptions)
     timeout: 90                    # Request timeout in seconds
     max_tokens: 200                # Maximum tokens in response
 ```

docs/configuration/providers.md

@@ -16,17 +16,17 @@ qa:
   model:
     provider: ollama
     name: gpt-oss
-    temperature: 0.7
+    temperature: 0.3
     max_tokens: 500
 ```
 
 **Available options:**
 
-- **temperature**: Sampling temperature (0.0-1.0+)
+- **temperature**: Sampling temperature (0.0-1.0+). Defaults vary by task: 0.3 for QA, research, and title generation; 0.0 for RLM and picture description.
   - Lower (0.0-0.3): Deterministic, focused responses
   - Medium (0.4-0.7): Balanced
   - Higher (0.8-1.0+): Creative, varied responses
-- **max_tokens**: Maximum tokens in response
+- **max_tokens**: Maximum tokens in response. Default: unset (provider default), except title generation (100).
 - **enable_thinking**: Control reasoning behavior (see below)
 - **base_url**: Custom endpoint for OpenAI-compatible servers (vLLM, LM Studio, etc.)
 
@@ -37,7 +37,7 @@ The `enable_thinking` setting controls whether models use explicit reasoning ste
 ```yaml
 qa:
   model:
-    enable_thinking: false  # Faster responses
+    enable_thinking: true   # Better grounded answers
 
 research:
   model:
@@ -63,8 +63,8 @@ See the [Pydantic AI thinking documentation](https://ai.pydantic.dev/thinking/)
 - **LM Studio**: Models supporting reasoning (gpt-oss, etc.)
 
 **When to use:**
-- Disable for simple queries, RAG workflows, speed-critical applications
-- Enable for complex reasoning, mathematical problems, research tasks
+- Enable for QA, research, complex reasoning, and mathematical problems
+- Disable for speed-critical applications, title generation, and simple tasks
 
 ## Embedding Providers
 

docs/configuration/qa-research.md

@@ -31,7 +31,8 @@ qa:
   model:
     provider: ollama
     name: gpt-oss
-    enable_thinking: false
+    enable_thinking: true
+    temperature: 0.3          # Default: 0.3
   max_iterations: 2     # Maximum search iterations
   max_concurrency: 1    # Concurrent search operations
 ```
@@ -50,6 +51,7 @@ research:
     provider: ""            # Empty to use qa settings
     name: ""               # Empty to use qa model
     enable_thinking: false
+    temperature: 0.3        # Default: 0.3
   max_iterations: 3
   max_concurrency: 1
 ```
@@ -69,6 +71,7 @@ rlm:
   model:
     provider: anthropic
     name: claude-sonnet-4-20250514
+    temperature: 0.0        # Default: 0.0 (deterministic for code generation)
   code_timeout: 60.0      # Max seconds for code execution
   max_output_chars: 50000 # Truncate output after this many chars
 ```

evaluations/evaluations/benchmark.py

@@ -38,10 +38,10 @@ def build_experiment_metadata(
     dataset_key: str,
     test_cases: int,
     config: AppConfig,
-    judge_config: ModelConfig,
+    judge_config: ModelConfig | None = None,
 ) -> dict[str, Any]:
     """Build experiment metadata for Logfire tracking."""
-    return {
+    metadata: dict[str, Any] = {
         "dataset": dataset_key,
         "test_cases": test_cases,
         "embedder_provider": config.embeddings.model.provider,
@@ -61,12 +61,18 @@ def build_experiment_metadata(
         "qa_temperature": config.qa.model.temperature,
         "qa_max_tokens": config.qa.model.max_tokens,
         "qa_enable_thinking": config.qa.model.enable_thinking,
-        "judge_provider": judge_config.provider,
-        "judge_model": judge_config.name,
-        "judge_temperature": judge_config.temperature,
-        "judge_max_tokens": judge_config.max_tokens,
-        "judge_enable_thinking": judge_config.enable_thinking,
     }
+    if judge_config is not None:
+        metadata.update(
+            {
+                "judge_provider": judge_config.provider,
+                "judge_model": judge_config.name,
+                "judge_temperature": judge_config.temperature,
+                "judge_max_tokens": judge_config.max_tokens,
+                "judge_enable_thinking": judge_config.enable_thinking,
+            }
+        )
+    return metadata
 
 
 async def populate_db(
@@ -220,14 +226,10 @@ async def run_retrieval_benchmark(
 
         eval_name = name if name is not None else f"{spec.key}_retrieval_evaluation"
 
-        judge_config = ModelConfig(
-            provider="ollama", name="gpt-oss", enable_thinking=False
-        )
         experiment_metadata = build_experiment_metadata(
             dataset_key=spec.key,
             test_cases=len(cases),
             config=config,
-            judge_config=judge_config,
         )
 
         report = await dataset.evaluate(
@@ -275,7 +277,9 @@ async def run_qa_benchmark(
         for index, doc in enumerate(corpus, start=1)
     ]
 
-    judge_config = ModelConfig(provider="ollama", name="gpt-oss", enable_thinking=False)
+    judge_config = ModelConfig(
+        provider="ollama", name="gpt-oss", enable_thinking=False, temperature=0.0
+    )
     judge_model = get_model(judge_config, config)
 
     evaluation_dataset = EvalDataset[str, str, dict[str, str]](

evaluations/evaluations/evaluators/judge.py

@@ -36,7 +36,9 @@ class LLMJudge:
     """LLM-as-judge for evaluating answer equivalence using Pydantic AI."""
 
     def __init__(self, model: str = "gpt-oss", config: AppConfig | None = None):
-        model_config = ModelConfig(provider="ollama", name=model, enable_thinking=False)
+        model_config = ModelConfig(
+            provider="ollama", name=model, enable_thinking=True, temperature=0.0
+        )
         model_obj = get_model(model_config, config)
 
         # Create Pydantic AI agent

haiku_rag_slim/haiku/rag/config/models.py

@@ -75,7 +75,8 @@ class QAConfig(BaseModel):
         default_factory=lambda: ModelConfig(
             provider="ollama",
             name="gpt-oss",
-            enable_thinking=False,
+            enable_thinking=True,
+            temperature=0.3,
         )
     )
     max_iterations: int = 2
@@ -88,6 +89,7 @@ class ResearchConfig(BaseModel):
             provider="ollama",
             name="gpt-oss",
             enable_thinking=False,
+            temperature=0.3,
         )
     )
     max_iterations: int = 3
@@ -100,6 +102,7 @@ class RLMConfig(BaseModel):
             provider="ollama",
             name="gpt-oss",
             enable_thinking=False,
+            temperature=0.0,
         )
     )
     code_timeout: float = 60.0
@@ -114,6 +117,7 @@ class PictureDescriptionConfig(BaseModel):
         default_factory=lambda: ModelConfig(
             provider="ollama",
             name="ministral-3",
+            temperature=0.0,
         )
     )
     timeout: int = 90
@@ -162,6 +166,8 @@ class ProcessingConfig(BaseModel):
             provider="ollama",
             name="gpt-oss",
             enable_thinking=False,
+            temperature=0.3,
+            max_tokens=100,
         )
     )