Two-phase extraction for end-of-document answer keys (PREP 2013); scroll fix; bug audit

Two-phase extraction:
- Detects end-of-document answer key format by scanning last 40 pages for
  "Preferred Response:" (PREP 2013, 2014 etc use this vs PREP 2012 inline "Correct Answer:")
- Phase 1: Extract questions with item_number field, allow null correct_answer
- Phase 2: Extract answer key (item_number → letter) from last 40% of document
- Phase 3: Match questions to answers by item number, resolve letter → full option text
- Unmatched questions go to skipped list with reason shown in Jobs page
- Standard inline format (PREP 2012) unchanged

Updated extraction prompts:
- item_number field added to all extractions for cross-referencing
- Image content rule: "Item CXXXB" figure references must NOT be treated as new questions
- Recognises both "Correct Answer: X" and "Preferred Response: X"
- ANSWER_KEY_PROMPT: dedicated prompt for extracting answer key tables

Quiz navigation scroll:
- Clicking Next, Previous, or question number now scrolls the question card
  into view (smooth scroll to start of question-card div)

Code: extract_questions_no_answers(), extract_answer_key(), _call_model() added to ai_service.py

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

backend/app/services/ai_service.py

@@ -16,36 +16,52 @@ def _proxy_model(model_id: str) -> str:
 
 EXTRACTION_PROMPT = """You are extracting questions from a PREP (Pediatric Review and Education Program) exam PDF.
 
-These PDFs follow a strict format:
+These PDFs have questions with a clinical vignette, followed by five answer options A-E.
-1. A numbered question with a clinical vignette (patient scenario)
+Some PDFs include "Correct Answer: X" or "Preferred Response: X" right after the options.
-2. Five answer options labeled A, B, C, D, E
+Some PDFs have the correct answers only at the END of the document — in that case correct_answer will be null.
-3. A line "Correct Answer: X" where X is the letter of the correct option
-4. An explanation paragraph
-5. A "Critique:" section with detailed reasoning
-6. A "Content Specifications:" section listing the learning objectives
 
-Your task: extract every question and return ONLY a JSON object in this exact format:
+Return ONLY a JSON object:
 
 {{"questions": [
   {{
-    "question_text": "<full question stem including any patient vignette>",
+    "item_number": "<item number like '193' or '21' — digits only, null if not found>",
+    "question_text": "<full question stem including the clinical vignette>",
     "question_type": "mcq",
     "options": ["<option A text>", "<option B text>", "<option C text>", "<option D text>", "<option E text>"],
-    "correct_answer": "<full text of the correct option, NOT the letter>",
+    "correct_answer": "<full text of the correct option, NOT the letter — or null if not found>",
-    "explanation": "<explanation paragraph>\\n\\nCritique: <critique section verbatim>\\n\\nContent Specifications: <content spec section verbatim>",
+    "explanation": "<explanation/critique/content specs if present, else empty string>",
     "page_reference": {page_ref}
   }}
 ]}}
 
-CRITICAL RULES — the correct_answer field is the most important:
+RULES:
-- Find the line "Correct Answer: X" after each question. X is the letter (A, B, C, D, or E).
+- A new question starts when you see "Item NNN" or "ltem NNN" (OCR artifact for Item).
-- Look up the full text of option X from the options list and store it as correct_answer.
+- Extract question_text as the full vignette + the "Of the following..." or "Which of the following..." stem.
-- Example: if options are ["alpha","beta","gamma","delta","epsilon"] and Correct Answer is C, then correct_answer = "gamma"
+- Options are labeled A. B. C. D. E. — extract just the text, not the letter.
-- NEVER store just a letter like "C" — always store the FULL OPTION TEXT.
+- If "Correct Answer: X" or "Preferred Response: X" appears after the options, resolve it to full option text.
-- If you cannot find "Correct Answer:" for a question, set correct_answer to null (do not guess).
+- CRITICAL: Content immediately after an image reference (e.g. "Item C123A", "Item C123B", figure captions,
-- Do not omit any question — extract ALL questions even if partially cut off.
+  table data) is NOT a new question. Skip it and wait for the next "Item NNN" number.
-- Preserve ALL text verbatim — do not summarize or paraphrase anything.
+- If no correct answer line is found for a question, set correct_answer to null — do NOT guess.
-- Return ONLY the JSON — no markdown, no preamble, no explanation.
+- Do NOT omit questions — extract all questions found in the content, even if partial.
+- Return ONLY the JSON — no markdown, no preamble.
+
+Content from page(s) {page_info}:
+{content}"""
+
+ANSWER_KEY_PROMPT = """Extract the answer key from this PREP exam content.
+
+The answer key lists items with their correct answer letters, like:
+  "Item 193  Preferred Response: D"
+  "Item 194  Preferred Response: A"
+
+Return ONLY a JSON object mapping item numbers to correct letters:
+{{"answers": {{"193": "D", "194": "A", "211": "C"}}}}
+
+Rules:
+- "Preferred Response: X" or "Correct Answer: X" — X is the letter.
+- Item numbers may appear as "ltemXXX" (OCR artifact — l is actually I).
+- Only include items where you find a clear correct answer letter.
+- Return ONLY the JSON — no markdown, no preamble.
 
 Content from page(s) {page_info}:
 {content}"""
@@ -178,6 +194,115 @@ def extract_questions(
     raise RuntimeError(f"Failed to extract questions after 3 attempts: {last_error}")
 
 
+def _call_model(prompt: str, model_id: str | None, api_key: str | None) -> str:
+    """Call the configured LLM and return raw text response."""
+    use_model = _proxy_model(model_id or settings.LITELLM_MODEL)
+    use_key = api_key or settings.LITELLM_API_KEY
+    kwargs = {
+        "model": use_model,
+        "messages": [{"role": "user", "content": prompt}],
+        "temperature": 0.1,
+    }
+    if use_key:
+        kwargs["api_key"] = use_key
+    if settings.LITELLM_API_BASE:
+        kwargs["api_base"] = settings.LITELLM_API_BASE
+    response = litellm.completion(**kwargs)
+    return response.choices[0].message.content
+
+
+def extract_questions_no_answers(
+    content: str,
+    page_info: str = "unknown",
+    page_ref: int | None = None,
+    model_id: str | None = None,
+    api_key: str | None = None,
+) -> list[dict]:
+    """Extract questions allowing null correct_answer — for PDFs where answers are at the end."""
+    content = _truncate_content(content)
+    prompt = EXTRACTION_PROMPT.format(
+        content=content,
+        page_info=page_info,
+        page_ref=page_ref if page_ref else "null",
+    )
+    last_error = None
+    for attempt in range(3):
+        try:
+            text = _call_model(prompt, model_id, api_key)
+            text = text.strip()
+            if text.startswith("```"):
+                text = text.split("\n", 1)[1] if "\n" in text else text[3:]
+                if text.endswith("```"):
+                    text = text[:-3]
+                text = text.strip()
+            data = json.loads(text)
+            questions = []
+            if isinstance(data, list):
+                questions = data
+            elif isinstance(data, dict):
+                for key in ("questions", "items", "results", "data"):
+                    if isinstance(data.get(key), list):
+                        questions = data[key]
+                        break
+                else:
+                    if "question_text" in data:
+                        questions = [data]
+
+            result = []
+            for q in questions:
+                if "question_text" not in q:
+                    continue
+                qtype = q.get("question_type", "mcq")
+                if qtype not in ("mcq", "true_false", "fill_blank"):
+                    qtype = "mcq"
+                result.append({
+                    "item_number": str(q.get("item_number") or "").strip().lstrip("0") or None,
+                    "question_text": q["question_text"],
+                    "question_type": qtype,
+                    "options": q.get("options"),
+                    "correct_answer": q.get("correct_answer"),  # may be null
+                    "explanation": q.get("explanation", ""),
+                    "page_reference": q.get("page_reference"),
+                })
+            if result:
+                return result
+            raise ValueError("No questions found in content")
+        except Exception as e:
+            last_error = e
+            logger.warning(f"No-answer extraction attempt {attempt + 1} failed: {e!r}")
+    raise RuntimeError(f"Failed after 3 attempts: {last_error}")
+
+
+def extract_answer_key(
+    content: str,
+    page_info: str = "unknown",
+    model_id: str | None = None,
+    api_key: str | None = None,
+) -> dict[str, str]:
+    """Extract answer key from end-of-document content. Returns {item_number: letter}."""
+    content = _truncate_content(content, max_chars=80000)
+    prompt = ANSWER_KEY_PROMPT.format(content=content, page_info=page_info)
+    last_error = None
+    for attempt in range(3):
+        try:
+            text = _call_model(prompt, model_id, api_key)
+            text = text.strip()
+            if text.startswith("```"):
+                text = text.split("\n", 1)[1] if "\n" in text else text[3:]
+                if text.endswith("```"):
+                    text = text[:-3]
+                text = text.strip()
+            data = json.loads(text)
+            answers = data.get("answers", data) if isinstance(data, dict) else {}
+            # Normalize: strip leading zeros, uppercase letters
+            return {str(k).strip().lstrip("0"): str(v).strip().upper() for k, v in answers.items() if v}
+        except Exception as e:
+            last_error = e
+            logger.warning(f"Answer key extraction attempt {attempt + 1} failed: {e!r}")
+    logger.warning(f"Answer key extraction failed: {last_error}")
+    return {}
+
+
 def generate_tts_audio(
     text: str,
     model_id: str | None = None,

backend/app/tasks/quiz_tasks.py

@@ -95,41 +95,111 @@ def extract_quiz(
         if n_chunks > 1:
             _push_step(r, job_id, "text", f"Large section: splitting into {n_chunks} chunks of up to {CHUNK_PAGES} pages each.")
 
+        # --- Detect end-of-document answer key format (e.g. PREP 2013) ---
+        last_chunk_content = vector_service.get_pages_text(
+            document_id=section.document_id,
+            start_page=max(section.end_page - 40, section.start_page),
+            end_page=section.end_page,
+        )
+        has_end_answer_key = bool(last_chunk_content and (
+            "Preferred Response:" in last_chunk_content or
+            "preferred response:" in last_chunk_content.lower()
+        ))
+
+        if has_end_answer_key:
+            _push_step(r, job_id, "ai", "Detected end-of-document answer key format (Preferred Response). Using two-phase extraction.")
+
         all_valid_questions = []
         all_skipped = []
 
-        for chunk_idx, (start_p, end_p) in enumerate(chunks, 1):
+        if has_end_answer_key:
-            if n_chunks > 1:
+            # === PHASE 1: Extract questions (allow null correct_answer) ===
-                _push_step(r, job_id, "ai", f"Chunk {chunk_idx}/{n_chunks}: pages {start_p}–{end_p} → {model_name}…")
+            raw_questions = []
-            else:
+            for chunk_idx, (start_p, end_p) in enumerate(chunks, 1):
-                _push_step(r, job_id, "ai", f"Sending pages {start_p}–{end_p} to {model_name}…")
+                _push_step(r, job_id, "ai", f"Phase 1 – Chunk {chunk_idx}/{n_chunks}: pages {start_p}–{end_p} (questions)…")
-
+                chunk_content = vector_service.get_pages_text(
-            chunk_content = vector_service.get_pages_text(
+                    document_id=section.document_id, start_page=start_p, end_page=end_p,
-                document_id=section.document_id,
-                start_page=start_p,
-                end_page=end_p,
-            )
-            if not chunk_content:
-                _push_step(r, job_id, "ai", f"  No text found for pages {start_p}–{end_p}, skipping.")
-                continue
-
-            try:
-                chunk_data = ai_service.extract_questions(
-                    chunk_content,
-                    page_info=f"{start_p}-{end_p}",
-                    page_ref=start_p,
-                    model_id=model_id,
-                    api_key=api_key,
                 )
-                chunk_skipped = []
+                if not chunk_content:
-                if chunk_data and chunk_data[0].get("skipped"):
+                    continue
-                    chunk_skipped = chunk_data[0].pop("skipped")
+                try:
-                chunk_valid = [q for q in chunk_data if q.get("correct_answer")]
+                    chunk_qs = ai_service.extract_questions_no_answers(
-                all_valid_questions.extend(chunk_valid)
+                        chunk_content, page_info=f"{start_p}-{end_p}",
-                all_skipped.extend(chunk_skipped)
+                        page_ref=start_p, model_id=model_id, api_key=api_key,
-                _push_step(r, job_id, "ai", f"  Pages {start_p}–{end_p}: {len(chunk_valid)} questions extracted{f', {len(chunk_skipped)} skipped' if chunk_skipped else ''}. Total so far: {len(all_valid_questions)}.")
+                    )
-            except Exception as e:
+                    raw_questions.extend(chunk_qs)
-                _push_step(r, job_id, "ai", f"  Pages {start_p}–{end_p} failed: {e}. Continuing…")
+                    _push_step(r, job_id, "ai", f"  Pages {start_p}–{end_p}: {len(chunk_qs)} questions found. Total: {len(raw_questions)}.")
+                except Exception as e:
+                    _push_step(r, job_id, "ai", f"  Pages {start_p}–{end_p} failed: {e}. Continuing…")
+
+            # === PHASE 2: Extract answer key from end of document ===
+            _push_step(r, job_id, "ai", f"Phase 2 – Extracting answer key from last pages…")
+            # Include larger portion for answer key (last 30-40% of doc)
+            answer_start = max(section.start_page, int(section.end_page * 0.6))
+            answer_content = vector_service.get_pages_text(
+                document_id=section.document_id,
+                start_page=answer_start,
+                end_page=section.end_page,
+            )
+            answer_key = ai_service.extract_answer_key(
+                answer_content, page_info=f"{answer_start}-{section.end_page}",
+                model_id=model_id, api_key=api_key,
+            )
+            _push_step(r, job_id, "ai", f"  Answer key extracted: {len(answer_key)} answers found.")
+
+            # === PHASE 3: Match questions to answers ===
+            _push_step(r, job_id, "ai", "Phase 3 – Matching questions to answers…")
+            for q in raw_questions:
+                item_num = q.get("item_number")
+                if not item_num:
+                    # No item number — skip
+                    all_skipped.append(q.get("question_text", "")[:120])
+                    continue
+
+                letter = answer_key.get(item_num)
+                if not letter:
+                    all_skipped.append(q.get("question_text", "")[:120])
+                    continue
+
+                # Resolve letter → full option text
+                options = q.get("options") or []
+                letter_idx = ord(letter.upper()) - ord("A")
+                if 0 <= letter_idx < len(options):
+                    q["correct_answer"] = options[letter_idx]
+                    all_valid_questions.append(q)
+                else:
+                    all_skipped.append(q.get("question_text", "")[:120])
+
+            _push_step(r, job_id, "ai", f"Matching complete: {len(all_valid_questions)} matched, {len(all_skipped)} unmatched.")
+
+        else:
+            # === STANDARD: inline correct answer format ===
+            for chunk_idx, (start_p, end_p) in enumerate(chunks, 1):
+                if n_chunks > 1:
+                    _push_step(r, job_id, "ai", f"Chunk {chunk_idx}/{n_chunks}: pages {start_p}–{end_p} → {model_name}…")
+                else:
+                    _push_step(r, job_id, "ai", f"Sending pages {start_p}–{end_p} to {model_name}…")
+
+                chunk_content = vector_service.get_pages_text(
+                    document_id=section.document_id, start_page=start_p, end_page=end_p,
+                )
+                if not chunk_content:
+                    _push_step(r, job_id, "ai", f"  No text found for pages {start_p}–{end_p}, skipping.")
+                    continue
+                try:
+                    chunk_data = ai_service.extract_questions(
+                        chunk_content, page_info=f"{start_p}-{end_p}",
+                        page_ref=start_p, model_id=model_id, api_key=api_key,
+                    )
+                    chunk_skipped = []
+                    if chunk_data and chunk_data[0].get("skipped"):
+                        chunk_skipped = chunk_data[0].pop("skipped")
+                    chunk_valid = [q for q in chunk_data if q.get("correct_answer")]
+                    all_valid_questions.extend(chunk_valid)
+                    all_skipped.extend(chunk_skipped)
+                    _push_step(r, job_id, "ai", f"  Pages {start_p}–{end_p}: {len(chunk_valid)} questions extracted{f', {len(chunk_skipped)} skipped' if chunk_skipped else ''}. Total so far: {len(all_valid_questions)}.")
+                except Exception as e:
+                    _push_step(r, job_id, "ai", f"  Pages {start_p}–{end_p} failed: {e}. Continuing…")
 
         valid_questions = all_valid_questions
         skipped = all_skipped

frontend/src/pages/QuizPage.jsx

@@ -195,7 +195,12 @@ useEffect(() => {
 
   const setAnswer = (questionId, value) => setAnswers(prev => ({ ...prev, [questionId]: value }))
 
-  const safeNavigate = (targetIdx) => setCurrentIdx(targetIdx)
+  const safeNavigate = (targetIdx) => {
+    setCurrentIdx(targetIdx)
+    setTimeout(() => {
+      document.querySelector('.question-card')?.scrollIntoView({ behavior: 'smooth', block: 'start' })
+    }, 50)
+  }
 
   const handleSubmit = useCallback(async (autoSubmit = false) => {
     if (!attemptId || submitting) return
@@ -392,7 +397,7 @@ useEffect(() => {
                 {submitting ? 'Submitting...' : 'Submit Quiz'}
               </button>
             ) : (
-              <button className="btn btn-primary" onClick={() => setCurrentIdx(i => Math.min(totalCount - 1, i + 1))}>Next →</button>
+              <button className="btn btn-primary" onClick={() => safeNavigate(Math.min(totalCount - 1, currentIdx + 1))}>Next →</button>
             )}
           </div>