From e55125763ffffa7f0ae4e13ad96b59b4570e8d3c Mon Sep 17 00:00:00 2001 From: Yiorgis Gozadinos Date: Thu, 21 May 2026 10:25:48 +0300 Subject: [PATCH] Document setting thinking on OpenAI compat providers --- docs/configuration/providers.md | 20 +++++++++++++++++++- docs/configuration/qa.md | 3 +++ 2 files changed, 22 insertions(+), 1 deletion(-) diff --git a/docs/configuration/providers.md b/docs/configuration/providers.md index 0bd5ca31..63aea039 100644 --- a/docs/configuration/providers.md +++ b/docs/configuration/providers.md @@ -56,13 +56,16 @@ See the [Pydantic AI thinking documentation](https://ai.pydantic.dev/thinking/) - **Groq**: Models with reasoning capabilities - **Bedrock**: Claude, OpenAI, and Qwen models - **Ollama**: Models supporting reasoning (gpt-oss, etc.) -- **vLLM**: Models supporting reasoning (gpt-oss, etc.) +- **vLLM**: Models with a pydantic-ai reasoning profile (gpt-oss). Qwen3, Gemma, and similar templates ignore the OpenAI `reasoning_effort` that `enable_thinking` translates to — use [`extra_body`](#raw-provider-pass-through) to drive them. - **LM Studio**: Models supporting reasoning (gpt-oss, etc.) **When to use:** - Enable for QA, complex reasoning, and mathematical problems - Disable for speed-critical applications, title generation, and simple tasks +!!! note "vLLM-served models without a reasoning profile" + On `provider: openai` with a custom `base_url`, `enable_thinking` only takes effect for models whose pydantic-ai profile advertises reasoning support (o-series, gpt-5, gpt-oss). For other vLLM-served models (Qwen3, Gemma family, …) the field is a silent no-op. Reach the chat template's thinking switch directly via [`extra_body`](#raw-provider-pass-through). + ### Raw Provider Pass-through The `extra_body` setting takes a dict that haiku.rag forwards verbatim to the underlying model SDK as `ModelSettings.extra_body`. Use it to reach provider-specific keys that haiku.rag does not model with a dedicated field. @@ -82,6 +85,21 @@ qa: vLLM serves Qwen3 chat templates that read their thinking switch from `chat_template_kwargs.enable_thinking`. The high-level `enable_thinking` setting on the openai provider maps to vLLM's `reasoning_effort` parameter, which Qwen3 templates ignore, so the field is a no-op for this combination. `extra_body` reaches the chat template directly and disables thinking. With it off, Qwen3 returns the answer in `content` immediately instead of emitting a hidden reasoning trace first. +**Example: enable Gemma-family thinking on vLLM:** + +```yaml +qa: + model: + provider: openai + name: nvidia/Gemma-4-26B-A4B-NVFP4 + base_url: http://localhost:11432/v1 + extra_body: + chat_template_kwargs: + enable_thinking: true +``` + +Same mechanism, opposite direction. Without `extra_body` the Gemma-4 chat template defaults to non-thinking and dumps a verbose answer straight into `content`. With it on, vLLM (started with `--reasoning-parser`) populates the parsed `reasoning` field and leaves `content` as the concise final answer. + **Provider support:** honored by openai, ollama, anthropic, and groq via pydantic-ai's `ModelSettings.extra_body`. Silently ignored by gemini and bedrock. ## Embedding Providers diff --git a/docs/configuration/qa.md b/docs/configuration/qa.md index e2da510e..f8f76f7f 100644 --- a/docs/configuration/qa.md +++ b/docs/configuration/qa.md @@ -37,6 +37,9 @@ qa: - **model.vision**: Set to `true` for vision-capable models (`qwen2.5vl`, `qwen3.6`, `gpt-4o`, `claude-sonnet`, …). The skill's `search` tool only attaches picture bytes (`BinaryContent`) to its `ToolReturn` when this is `true`, otherwise picture bytes are withheld. See [Pictures × embedder × QA model](processing.md#pictures-embedder-qa-model-how-the-pieces-compose) for the full matrix. - **max_searches**: Maximum number of search tool calls the rag skill can make per question (default: 3) +!!! note "Thinking on vLLM" + `enable_thinking` only applies to models with a pydantic-ai reasoning profile (o-series, gpt-5, gpt-oss). For other vLLM-served models such as Qwen3 or the Gemma family, the field is a silent no-op — set the chat template switch via [`extra_body`](providers.md#raw-provider-pass-through) instead. + ## Analysis Configuration Configure the analysis skill: