danvics/pdf-quiz-generator
1
0
Fork 0
Code Issues Pull requests Projects Releases 3 Packages Wiki Activity Actions

Two-phase extraction for end-of-document answer keys (PREP 2013); scroll fix; bug audit

Browse source 
Two-phase extraction:
- Detects end-of-document answer key format by scanning last 40 pages for
  "Preferred Response:" (PREP 2013, 2014 etc use this vs PREP 2012 inline "Correct Answer:")
- Phase 1: Extract questions with item_number field, allow null correct_answer
- Phase 2: Extract answer key (item_number → letter) from last 40% of document
- Phase 3: Match questions to answers by item number, resolve letter → full option text
- Unmatched questions go to skipped list with reason shown in Jobs page
- Standard inline format (PREP 2012) unchanged

Updated extraction prompts:
- item_number field added to all extractions for cross-referencing
- Image content rule: "Item CXXXB" figure references must NOT be treated as new questions
- Recognises both "Correct Answer: X" and "Preferred Response: X"
- ANSWER_KEY_PROMPT: dedicated prompt for extracting answer key tables

Quiz navigation scroll:
- Clicking Next, Previous, or question number now scrolls the question card
  into view (smooth scroll to start of question-card div)

Code: extract_questions_no_answers(), extract_answer_key(), _call_model() added to ai_service.py

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Daniel 2026-04-01 03:22:26 +02:00
parent 287d3a72f4
commit e5e31f6eba
3 changed files with 253 additions and 53 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

165
backend/app/services/ai_service.py
View file

@ -16,36 +16,52 @@ def _proxy_model(model_id: str) -> str:
EXTRACTION_PROMPT = """You are extracting questions from a PREP (Pediatric Review and Education Program) exam PDF.
These PDFs follow a strict format:
1. A numbered question with a clinical vignette (patient scenario)
2. Five answer options labeled A, B, C, D, E
3. A line "Correct Answer: X" where X is the letter of the correct option
4. An explanation paragraph
5. A "Critique:" section with detailed reasoning
6. A "Content Specifications:" section listing the learning objectives
These PDFs have questions with a clinical vignette, followed by five answer options A-E.
Some PDFs include "Correct Answer: X" or "Preferred Response: X" right after the options.
Some PDFs have the correct answers only at the END of the document — in that case correct_answer will be null.
Your task: extract every question and return ONLY a JSON object in this exact format:
Return ONLY a JSON object:
{{"questions": [
{{
"question_text": "<full question stem including any patient vignette>",
"item_number": "<item number like '193' or '21' — digits only, null if not found>",
"question_text": "<full question stem including the clinical vignette>",
"question_type": "mcq",
"options": ["<option A text>", "<option B text>", "<option C text>", "<option D text>", "<option E text>"],
"correct_answer": "<full text of the correct option, NOT the letter>",
"explanation": "<explanation paragraph>\\n\\nCritique: <critique section verbatim>\\n\\nContent Specifications: <content spec section verbatim>",
"correct_answer": "<full text of the correct option, NOT the letter — or null if not found>",
"explanation": "<explanation/critique/content specs if present, else empty string>",
"page_reference": {page_ref}
}}
]}}
CRITICAL RULES — the correct_answer field is the most important:
- Find the line "Correct Answer: X" after each question. X is the letter (A, B, C, D, or E).
- Look up the full text of option X from the options list and store it as correct_answer.
- Example: if options are ["alpha","beta","gamma","delta","epsilon"] and Correct Answer is C, then correct_answer = "gamma"
- NEVER store just a letter like "C" — always store the FULL OPTION TEXT.
- If you cannot find "Correct Answer:" for a question, set correct_answer to null (do not guess).
- Do not omit any question — extract ALL questions even if partially cut off.
- Preserve ALL text verbatim — do not summarize or paraphrase anything.
- Return ONLY the JSON — no markdown, no preamble, no explanation.
RULES:
- A new question starts when you see "Item NNN" or "ltem NNN" (OCR artifact for Item).
- Extract question_text as the full vignette + the "Of the following..." or "Which of the following..." stem.
- Options are labeled A. B. C. D. E. — extract just the text, not the letter.
- If "Correct Answer: X" or "Preferred Response: X" appears after the options, resolve it to full option text.
- CRITICAL: Content immediately after an image reference (e.g. "Item C123A", "Item C123B", figure captions,
table data) is NOT a new question. Skip it and wait for the next "Item NNN" number.
- If no correct answer line is found for a question, set correct_answer to null — do NOT guess.
- Do NOT omit questions — extract all questions found in the content, even if partial.
- Return ONLY the JSON — no markdown, no preamble.
Content from page(s) {page_info}:
{content}"""
ANSWER_KEY_PROMPT = """Extract the answer key from this PREP exam content.
The answer key lists items with their correct answer letters, like:
"Item 193 Preferred Response: D"
"Item 194 Preferred Response: A"
Return ONLY a JSON object mapping item numbers to correct letters:
{{"answers": {{"193": "D", "194": "A", "211": "C"}}}}
Rules:
- "Preferred Response: X" or "Correct Answer: X" — X is the letter.
- Item numbers may appear as "ltemXXX" (OCR artifact — l is actually I).
- Only include items where you find a clear correct answer letter.
- Return ONLY the JSON — no markdown, no preamble.
Content from page(s) {page_info}:
{content}"""
@ -178,6 +194,115 @@ def extract_questions(
raise RuntimeError(f"Failed to extract questions after 3 attempts: {last_error}")
def _call_model(prompt: str, model_id: str | None, api_key: str | None) -> str:
"""Call the configured LLM and return raw text response."""
use_model = _proxy_model(model_id or settings.LITELLM_MODEL)
use_key = api_key or settings.LITELLM_API_KEY
kwargs = {
"model": use_model,
"messages": [{"role": "user", "content": prompt}],
"temperature": 0.1,
}
if use_key:
kwargs["api_key"] = use_key
if settings.LITELLM_API_BASE:
kwargs["api_base"] = settings.LITELLM_API_BASE
response = litellm.completion(**kwargs)
return response.choices[0].message.content
def extract_questions_no_answers(
content: str,
page_info: str = "unknown",
page_ref: int | None = None,
model_id: str | None = None,
api_key: str | None = None,
) -> list[dict]:
"""Extract questions allowing null correct_answer — for PDFs where answers are at the end."""
content = _truncate_content(content)
prompt = EXTRACTION_PROMPT.format(
content=content,
page_info=page_info,
page_ref=page_ref if page_ref else "null",
)
last_error = None
for attempt in range(3):
try:
text = _call_model(prompt, model_id, api_key)
text = text.strip()
if text.startswith("```"):
text = text.split("\n", 1)[1] if "\n" in text else text[3:]
if text.endswith("```"):
text = text[:-3]
text = text.strip()
data = json.loads(text)
questions = []
if isinstance(data, list):
questions = data
elif isinstance(data, dict):
for key in ("questions", "items", "results", "data"):
if isinstance(data.get(key), list):
questions = data[key]
break
else:
if "question_text" in data:
questions = [data]
result = []
for q in questions:
if "question_text" not in q:
continue
qtype = q.get("question_type", "mcq")
if qtype not in ("mcq", "true_false", "fill_blank"):
qtype = "mcq"
result.append({
"item_number": str(q.get("item_number") or "").strip().lstrip("0") or None,
"question_text": q["question_text"],
"question_type": qtype,
"options": q.get("options"),
"correct_answer": q.get("correct_answer"), # may be null
"explanation": q.get("explanation", ""),
"page_reference": q.get("page_reference"),
})
if result:
return result
raise ValueError("No questions found in content")
except Exception as e:
last_error = e
logger.warning(f"No-answer extraction attempt {attempt + 1} failed: {e!r}")
raise RuntimeError(f"Failed after 3 attempts: {last_error}")
def extract_answer_key(
content: str,
page_info: str = "unknown",
model_id: str | None = None,
api_key: str | None = None,
) -> dict[str, str]:
"""Extract answer key from end-of-document content. Returns {item_number: letter}."""
content = _truncate_content(content, max_chars=80000)
prompt = ANSWER_KEY_PROMPT.format(content=content, page_info=page_info)
last_error = None
for attempt in range(3):
try:
text = _call_model(prompt, model_id, api_key)
text = text.strip()
if text.startswith("```"):
text = text.split("\n", 1)[1] if "\n" in text else text[3:]
if text.endswith("```"):
text = text[:-3]
text = text.strip()
data = json.loads(text)
answers = data.get("answers", data) if isinstance(data, dict) else {}
# Normalize: strip leading zeros, uppercase letters
return {str(k).strip().lstrip("0"): str(v).strip().upper() for k, v in answers.items() if v}
except Exception as e:
last_error = e
logger.warning(f"Answer key extraction attempt {attempt + 1} failed: {e!r}")
logger.warning(f"Answer key extraction failed: {last_error}")
return {}
def generate_tts_audio(
text: str,
model_id: str | None = None,

132
backend/app/tasks/quiz_tasks.py
View file

@ -95,41 +95,111 @@ def extract_quiz(
if n_chunks > 1:
_push_step(r, job_id, "text", f"Large section: splitting into {n_chunks} chunks of up to {CHUNK_PAGES} pages each.")
# --- Detect end-of-document answer key format (e.g. PREP 2013) ---
last_chunk_content = vector_service.get_pages_text(
document_id=section.document_id,
start_page=max(section.end_page - 40, section.start_page),
end_page=section.end_page,
)
has_end_answer_key = bool(last_chunk_content and (
"Preferred Response:" in last_chunk_content or
"preferred response:" in last_chunk_content.lower()
))
if has_end_answer_key:
_push_step(r, job_id, "ai", "Detected end-of-document answer key format (Preferred Response). Using two-phase extraction.")
all_valid_questions = []
all_skipped = []
for chunk_idx, (start_p, end_p) in enumerate(chunks, 1):
if n_chunks > 1:
_push_step(r, job_id, "ai", f"Chunk {chunk_idx}/{n_chunks}: pages {start_p}–{end_p} → {model_name}…")
else:
_push_step(r, job_id, "ai", f"Sending pages {start_p}–{end_p} to {model_name}…")
chunk_content = vector_service.get_pages_text(
document_id=section.document_id,
start_page=start_p,
end_page=end_p,
)
if not chunk_content:
_push_step(r, job_id, "ai", f" No text found for pages {start_p}–{end_p}, skipping.")
continue
try:
chunk_data = ai_service.extract_questions(
chunk_content,
page_info=f"{start_p}-{end_p}",
page_ref=start_p,
model_id=model_id,
api_key=api_key,
if has_end_answer_key:
# === PHASE 1: Extract questions (allow null correct_answer) ===
raw_questions = []
for chunk_idx, (start_p, end_p) in enumerate(chunks, 1):
_push_step(r, job_id, "ai", f"Phase 1 – Chunk {chunk_idx}/{n_chunks}: pages {start_p}–{end_p} (questions)…")
chunk_content = vector_service.get_pages_text(
document_id=section.document_id, start_page=start_p, end_page=end_p,
)
chunk_skipped = []
if chunk_data and chunk_data[0].get("skipped"):
chunk_skipped = chunk_data[0].pop("skipped")
chunk_valid = [q for q in chunk_data if q.get("correct_answer")]
all_valid_questions.extend(chunk_valid)
all_skipped.extend(chunk_skipped)
_push_step(r, job_id, "ai", f" Pages {start_p}–{end_p}: {len(chunk_valid)} questions extracted{f', {len(chunk_skipped)} skipped' if chunk_skipped else ''}. Total so far: {len(all_valid_questions)}.")
except Exception as e:
_push_step(r, job_id, "ai", f" Pages {start_p}–{end_p} failed: {e}. Continuing…")
if not chunk_content:
continue
try:
chunk_qs = ai_service.extract_questions_no_answers(
chunk_content, page_info=f"{start_p}-{end_p}",
page_ref=start_p, model_id=model_id, api_key=api_key,
)
raw_questions.extend(chunk_qs)
_push_step(r, job_id, "ai", f" Pages {start_p}–{end_p}: {len(chunk_qs)} questions found. Total: {len(raw_questions)}.")
except Exception as e:
_push_step(r, job_id, "ai", f" Pages {start_p}–{end_p} failed: {e}. Continuing…")
# === PHASE 2: Extract answer key from end of document ===
_push_step(r, job_id, "ai", f"Phase 2 – Extracting answer key from last pages…")
# Include larger portion for answer key (last 30-40% of doc)
answer_start = max(section.start_page, int(section.end_page * 0.6))
answer_content = vector_service.get_pages_text(
document_id=section.document_id,
start_page=answer_start,
end_page=section.end_page,
)
answer_key = ai_service.extract_answer_key(
answer_content, page_info=f"{answer_start}-{section.end_page}",
model_id=model_id, api_key=api_key,
)
_push_step(r, job_id, "ai", f" Answer key extracted: {len(answer_key)} answers found.")
# === PHASE 3: Match questions to answers ===
_push_step(r, job_id, "ai", "Phase 3 – Matching questions to answers…")
for q in raw_questions:
item_num = q.get("item_number")
if not item_num:
# No item number — skip
all_skipped.append(q.get("question_text", "")[:120])
continue
letter = answer_key.get(item_num)
if not letter:
all_skipped.append(q.get("question_text", "")[:120])
continue
# Resolve letter → full option text
options = q.get("options") or []
letter_idx = ord(letter.upper()) - ord("A")
if 0 <= letter_idx < len(options):
q["correct_answer"] = options[letter_idx]
all_valid_questions.append(q)
else:
all_skipped.append(q.get("question_text", "")[:120])
_push_step(r, job_id, "ai", f"Matching complete: {len(all_valid_questions)} matched, {len(all_skipped)} unmatched.")
else:
# === STANDARD: inline correct answer format ===
for chunk_idx, (start_p, end_p) in enumerate(chunks, 1):
if n_chunks > 1:
_push_step(r, job_id, "ai", f"Chunk {chunk_idx}/{n_chunks}: pages {start_p}–{end_p} → {model_name}…")
else:
_push_step(r, job_id, "ai", f"Sending pages {start_p}–{end_p} to {model_name}…")
chunk_content = vector_service.get_pages_text(
document_id=section.document_id, start_page=start_p, end_page=end_p,
)
if not chunk_content:
_push_step(r, job_id, "ai", f" No text found for pages {start_p}–{end_p}, skipping.")
continue
try:
chunk_data = ai_service.extract_questions(
chunk_content, page_info=f"{start_p}-{end_p}",
page_ref=start_p, model_id=model_id, api_key=api_key,
)
chunk_skipped = []
if chunk_data and chunk_data[0].get("skipped"):
chunk_skipped = chunk_data[0].pop("skipped")
chunk_valid = [q for q in chunk_data if q.get("correct_answer")]
all_valid_questions.extend(chunk_valid)
all_skipped.extend(chunk_skipped)
_push_step(r, job_id, "ai", f" Pages {start_p}–{end_p}: {len(chunk_valid)} questions extracted{f', {len(chunk_skipped)} skipped' if chunk_skipped else ''}. Total so far: {len(all_valid_questions)}.")
except Exception as e:
_push_step(r, job_id, "ai", f" Pages {start_p}–{end_p} failed: {e}. Continuing…")
valid_questions = all_valid_questions
skipped = all_skipped

9
frontend/src/pages/QuizPage.jsx
View file

@ -195,7 +195,12 @@ useEffect(() => {
const setAnswer = (questionId, value) => setAnswers(prev => ({ ...prev, [questionId]: value }))
const safeNavigate = (targetIdx) => setCurrentIdx(targetIdx)
const safeNavigate = (targetIdx) => {
setCurrentIdx(targetIdx)
setTimeout(() => {
document.querySelector('.question-card')?.scrollIntoView({ behavior: 'smooth', block: 'start' })
}, 50)
}
const handleSubmit = useCallback(async (autoSubmit = false) => {
if (!attemptId || submitting) return
@ -392,7 +397,7 @@ useEffect(() => {
{submitting ? 'Submitting...' : 'Submit Quiz'}
</button>
) : (
<button className="btn btn-primary" onClick={() => setCurrentIdx(i => Math.min(totalCount - 1, i + 1))}>Next →</button>
<button className="btn btn-primary" onClick={() => safeNavigate(Math.min(totalCount - 1, currentIdx + 1))}>Next →</button>
)}
</div>