中級篇 · 第 2 章Intermediate · Chapter 2

結構化回答與出處檢查Structured answers and citations

如果 AI 回傳的是一段自由文字，你的程式只能原封不動地顯示它，沒辦法檢查。改成固定格式之後，程式就能問：「你說找到答案了，出處呢？那個出處真的是我給你的嗎？」

If the AI returns free text, your code can only display it as-is; it can't check anything. With a fixed format, code can ask: "You say you found the answer. Where's the source? Is that source really one I gave you?"

⏱ 20 分鐘min🧪 schema.py · bot.py

這章要加的功能What this chapter adds

機器人用 JSON 回答：found、answer、citations。
用 Pydantic 檢查三條規則，特別是「出處必須是這次真的搜尋到的段落」。
不合格時，把錯誤告訴 AI，讓它重試一次。

The bot answers in JSON: found, answer, citations.
Pydantic checks three rules, above all "citations must be passages actually retrieved this time".
On failure, tell the AI what was wrong and let it retry once.

回答的格式The answer format

範例Example

{"found": true, "answer": "請特休需在三個工作天前提出申請。", "citations": ["H01"]}

found 是這個格式最重要的欄位。它讓 AI 有一個正式的方式說「我不知道」，程式也能據此決定要不要顯示「請洽人資」的按鈕。

found is the most important field. It gives the AI an official way to say "I don't know", and lets code decide whether to show a "contact HR" button.

兩層檢查：結構交給 API，規則交給你Two layers: structure by the API, rules by you

我們把一份 JSON Schema 交給 API，它會確保輸出一定是合法的 JSON、欄位一定齊全（這叫 structured output）。但 API 不知道你的業務規則，例如「出處必須是這次提供的段落」。所以規則寫在 Pydantic 模型裡，由你的程式檢查。

We hand the API a JSON Schema, and it guarantees valid JSON with every field present (this is called structured output). But the API doesn't know your business rules, such as "citations must be passages provided this time". Those live in a Pydantic model that your code checks.

labs/intermediate/schema.py

# 送給 API 的 JSON Schema：只放結構（型別、必填）
ANSWER_SCHEMA = {
    "type": "object",
    "properties": {
        "found": {"type": "boolean", "description": "手冊或工具結果裡有沒有這個問題的答案"},
        "answer": {"type": "string", "description": "給員工看的回答，繁體中文，簡潔"},
        "citations": {"type": "array", "items": {"type": "string"}, "description": "引用的段落編號，例如 H01"},
    },
    "required": ["found", "answer", "citations"],
    "additionalProperties": False,
}


class BotAnswer(BaseModel):
    found: bool
    answer: str = Field(min_length=1)
    citations: list[str] = Field(default_factory=list)

    @model_validator(mode="after")
    def check_citations(self, info: ValidationInfo) -> "BotAnswer":
        ctx = info.context or {}
        retrieved = set(ctx.get("retrieved_ids", []))
        used_tool = ctx.get("used_tool", False)

        if self.found and not self.citations and not used_tool:
            raise ValueError("found 為 true 時，citations 不能是空的：請附上段落編號")
        if not self.found and self.citations:
            raise ValueError("found 為 false 時，不應該有 citations")
        unknown = [c for c in self.citations if c not in retrieved]
        if unknown:
            raise ValueError(f"引用了這次沒有提供的段落：{', '.join(unknown)}；只能引用 {sorted(retrieved)}")
        return self

重點Key point

context 是 Pydantic 的功能：驗證時可以把「這次的情境」傳進去（這次搜尋到哪些段落、有沒有用工具）。同一個模型，每次驗證的標準可以不同。

context is a Pydantic feature: you can pass the situation into validation (which passages were retrieved, whether a tool was used). The same model applies different standards each time.

動手試：哪些回答會被擋下？Try it: which answers get rejected?

demo

回答檢查（這次搜尋到 H01、H03）Answer check (this time H01 and H03 were retrieved)

這次有使用工具（工具查到的答案可以沒有手冊出處）A tool was used this time (tool-based answers may lack handbook citations)

不合格就重試一次Retry once on failure

檢查失敗時，我們把 AI 的回答和具體的錯誤訊息加進對話，請它修正。注意兩個細節：重試次數有上限（MAX_RETRIES = 1）；超過上限就回報錯誤，不要偷偷顯示不合格的回答。

When validation fails, we append the AI's answer and the specific error to the conversation and ask for a fix. Two details: retries are capped (MAX_RETRIES = 1), and when the cap is hit we report an error rather than quietly showing a rejected answer.

labs/intermediate/bot.py

used_tool = bool(turn.tool_calls)
for attempt in range(config.MAX_RETRIES + 1):
    turn.attempts = attempt + 1
    try:
        turn.answer = parse_answer(reply.text, retrieved, used_tool)
        break
    except ValidationError as err:
        if attempt == config.MAX_RETRIES:
            turn.error = "回答格式檢查失敗：" + "; ".join(e["msg"] for e in err.errors())
            return self._finish(turn)
        feedback = "你上一次的回答沒有通過檢查：\n" + "\n".join(f"- {e['msg']}" for e in err.errors()) + \
                   "\n請修正後重新回答，只輸出 JSON。"
        messages += [{"role": "assistant", "content": reply.content}, {"role": "user", "content": feedback}]
        reply = self._call(messages, turn, tools=False)

測試裡用 FakeLLM(mistakes=["fake_citation"]) 讓第一次回答引用不存在的 H99，確認機器人會重試並得到正確的 H03；連續犯錯三次，則確認它會放棄並回報錯誤。

The tests use FakeLLM(mistakes=["fake_citation"]) to make the first answer cite a non-existent H99, checking that the bot retries and gets H03; with three consecutive mistakes, they check that it gives up and reports an error.

它擋不住什麼What it can't catch

這些檢查只能確認「格式對、出處存在」。如果 AI 引用了 H01，但把「三個工作天」說成「五個工作天」，這裡是抓不到的。那要靠第 6 章的測試題，以及進階篇第 04、07 章的方法。

These checks only confirm "correct format, real source". If the AI cites H01 but says "five working days" instead of "three", nothing here catches it. That takes chapter I6's test questions, and the methods in core chapters 04 and 07.

小測驗Quiz

1. 這次搜尋到 H01、H03，AI 回答引用了 H07。應該怎麼處理？1. H01 and H03 were retrieved, but the AI cites H07. What should happen? 判定不合格，把錯誤告訴 AI 重試；仍失敗就回報錯誤。Reject it, tell the AI and retry; if it still fails, report an error. 接受，H07 也是手冊裡的段落。Accept it; H07 is a real handbook section. 把 H07 從回答裡刪掉，其餘照樣顯示。Strip H07 and show the rest.

AI 沒有看到 H07 的內容，引用它代表答案可能是憑空來的。刪掉出處只是把問題藏起來。

The AI never saw H07, so citing it suggests the answer came from nowhere. Stripping the citation just hides the problem.

2. 為什麼不把所有規則都寫進送給 API 的 JSON Schema？2. Why not put every rule into the JSON Schema sent to the API? 因為 JSON Schema 太難寫。JSON Schema is too hard to write. 有些規則取決於這次的情境（搜尋到哪些段落），而且 API 不一定支援所有 schema 關鍵字。Some rules depend on this request's situation (which passages were retrieved), and APIs don't support every schema keyword. 因為放進 schema 會比較貴。It costs more to put them in the schema.

schema 是固定的，但「可以引用哪些段落」每次都不同，只能在你的程式裡檢查。

The schema is fixed, but which passages may be cited changes every time, so only your code can check it.