結構化輸出與資料契約Structured output and contracts
只要模型的輸出會被程式讀取，它就是一個 API。這個 API 的另一端是一個機率模型，所以你要用對待外部輸入的方式對待它：先驗證，再使用。
As soon as code reads a model's output, that output is an API, and the other side of that API is a probabilistic model. Treat it the way you treat external input: validate first, then use.
讀完這章你能夠After this chapter you can
- 分辨四種輸出錯誤，並知道每一種該由哪一層擋下。
- 用 provider 的 structured output 確保結構，用 Pydantic 確保業務規則。
- 實作有上限、有紀錄的驗證修復迴圈。
- 說明 schema 驗證擋不住什麼，以及要靠什麼補上。
- Tell apart four kinds of output error and which layer should stop each one.
- Use provider structured output for structure and Pydantic for business rules.
- Implement a bounded, logged validate-and-repair loop.
- Explain what schema validation cannot catch and what covers the gap.
四種輸出錯誤Four kinds of output error
| 錯誤Error | 例子Example | 由誰擋下Stopped by |
|---|---|---|
| 不是合法 JSONNot valid JSON | 外面包了 ```json、輸出被截斷Wrapped in ```json, truncated output |
structured output（受限解碼）Structured output (constrained decoding) |
| 結構不對Wrong shape | 少了欄位、型別錯、enum 外的值Missing field, wrong type, value outside the enum | structured output + schema 驗證Structured output + schema validation |
| 結構對，值不合規則Right shape, rule-breaking values | ID 格式錯、修復建議寫「TBD」、引用了沒檢索到的文件Bad ID format, "TBD" as the fix, citing a document that wasn't retrieved | 你的程式（Pydantic validator）Your code (Pydantic validators) |
| 完全合規，但內容是錯的Fully valid, factually wrong | 風險等級判斷錯、理由不成立Wrong risk level, reasoning that doesn't hold | 評測（第 7 章）與人工審核Evaluation (chapter 7) and human review |
「用了 structured output，輸出就保證正確。」它保證的是形狀，不是內容。一個欄位齊全、型別正確、但結論錯誤的 JSON，比一個解析失敗的 JSON 更危險，因為它會安靜地流進下游。
"With structured output, the output is guaranteed correct." It guarantees shape, not content. A JSON object with every field present and typed correctly but a wrong conclusion is more dangerous than one that fails to parse, because it flows silently downstream.
分層防護Defense in layers
第 1 層：provider 的 structured outputLayer 1: provider structured output
主流 API 都提供某種形式的 structured output：你給一份 JSON Schema，模型在產生 token 時就被限制在符合 schema 的範圍內，所以不會再出現「多了一段說明文字」或「括號沒關」的情況。不過各家支援的 schema 關鍵字不同。以 Claude 為例，型別、enum、required 會被強制執行，但數值範圍（minimum/maximum）與字串長度（minLength）不在解碼時強制；官方 SDK 會把這些限制從送出的 schema 移除，改在本機驗證。
Major APIs offer some form of structured output: you supply a JSON Schema and token generation is constrained to it, so "an extra sentence of explanation" or "an unclosed brace" can no longer happen. Supported schema keywords differ by provider, though. On Claude, for example, types, enum and required are enforced, while numeric ranges (minimum/maximum) and string lengths (minLength) are not enforced during decoding; the official SDKs strip them from the schema sent to the API and validate them locally.
結論：不管用哪家 provider，本機的 Pydantic 驗證都不能省。
Conclusion: whatever the provider, local Pydantic validation is not optional.
第 2、3 層：Pydantic 型別與業務規則Layers 2–3: Pydantic types and business rules
下面是 lab 裡的契約。注意最後一個 validator：它用這次請求實際檢索到的文件來檢查引用，這是抓捏造出處最有效的方法之一，而且 schema 本身做不到。
Below is the contract from the lab. Note the last validator: it checks citations against the documents actually retrieved for this request, one of the most effective ways to catch fabricated sources, and something a schema alone cannot express.
PLACEHOLDER_PATTERN = re.compile(
r"\b(tbd|todo|fixme|n/a|contact (the )?admin|check later|look at (the )?logs)\b",
re.IGNORECASE,
)
class ClauseRiskAssessment(BaseModel):
clause_id: str = Field(pattern=r"^CLAUSE-\d{3,4}$")
title: str = Field(min_length=3, max_length=120)
risk_level: Literal["CRITICAL", "HIGH", "MEDIUM", "LOW"]
# 模型自報的 confidence 沒有校準，只能當參考 / self-reported, not calibrated
confidence: float = Field(ge=0.0, le=1.0)
findings: list[str] = Field(min_length=1)
mitigation: str = Field(min_length=25)
cited_clause_ids: list[str] = Field(default_factory=list)
@field_validator("mitigation")
@classmethod
def mitigation_is_concrete(cls, v: str) -> str:
match = PLACEHOLDER_PATTERN.search(v)
if match:
raise ValueError(f"mitigation contains placeholder text '{match.group(0)}'; "
"give a concrete, actionable step")
return v
@field_validator("cited_clause_ids")
@classmethod
def citations_were_retrieved(cls, v: list[str], info: ValidationInfo) -> list[str]:
allowed = (info.context or {}).get("retrieved_ids")
if allowed is not None:
unknown = [c for c in v if c not in allowed]
if unknown:
raise ValueError(f"cites clauses that were not retrieved: {', '.join(unknown)}")
return v
# 使用時傳入 context / pass the context at validation time
ClauseRiskAssessment.model_validate_json(raw, context={"retrieved_ids": {"CLAUSE-003", "CLAUSE-004"}})舊版用子字串比對 "todo" in text，會把「更新 todos 資料表」也擋掉。\b 讓規則只比對完整的詞。驗證規則和任何程式碼一樣需要測試，lab 裡有專門的測試案例。
A substring check like "todo" in text also rejects "update the todos table". \b limits the rule to whole words. Validation rules need tests like any other code; the lab includes cases for this.
第 4 層：修復迴圈Layer 4: the repair loop
驗證失敗時，把具體的錯誤訊息當成下一輪的使用者訊息回給模型。三個要點：次數有上限、每次嘗試都記錄（之後分析失敗模式用）、最後失敗時有去處（人工佇列或明確的錯誤），而不是回傳一個預設值。
When validation fails, send the specific error messages back to the model as the next user turn. Three essentials: a cap on attempts, a log entry per attempt (for analysing failure patterns later), and somewhere for final failures to go (a human queue or an explicit error) rather than a silent default.
def generate_validated(llm, prompt, model_cls=ClauseRiskAssessment, max_attempts=3, context=None):
messages = [{"role": "user", "content": prompt}]
attempts = []
for attempt in range(1, max_attempts + 1):
raw = llm(messages)
try:
obj = model_cls.model_validate_json(raw, context=context)
except ValidationError as err:
attempts.append({"attempt": attempt, "ok": False,
"errors": err.errors(include_url=False, include_context=False)})
messages += [
{"role": "assistant", "content": raw},
{"role": "user", "content": format_errors_for_model(err) + "\nReturn corrected JSON only."},
]
continue
attempts.append({"attempt": attempt, "ok": True})
return obj, attempts
raise RepairExhausted(attempts)接上真正的模型Plugging in a real model
llm 只是一個「訊息進、文字出」的函式，所以換 provider 不影響驗證邏輯。下面是用 Anthropic Python SDK 的寫法。送給 API 的 schema 只放它會強制的關鍵字；其餘規則由上面的 Pydantic 模型負責。
llm is just a messages-in, text-out function, so switching providers doesn't touch the validation logic. Here is the Anthropic Python SDK version. The schema sent to the API contains only keywords it enforces; the Pydantic model above handles every other rule.
import anthropic
from labs.core.contracts import generate_validated
client = anthropic.Anthropic() # 讀取 ANTHROPIC_API_KEY
OUTPUT_SCHEMA = {
"type": "object",
"properties": {
"clause_id": {"type": "string"},
"title": {"type": "string"},
"risk_level": {"type": "string", "enum": ["CRITICAL", "HIGH", "MEDIUM", "LOW"]},
"confidence": {"type": "number"},
"findings": {"type": "array", "items": {"type": "string"}},
"mitigation": {"type": "string"},
"cited_clause_ids": {"type": "array", "items": {"type": "string"}},
},
"required": ["clause_id", "title", "risk_level", "confidence",
"findings", "mitigation", "cited_clause_ids"],
"additionalProperties": False,
}
def claude(messages: list[dict]) -> str:
response = client.messages.create(
model="claude-opus-5-5",
max_tokens=16000,
system="You assess contract clauses for operational risk. "
"Cite only clause ids that appear in the provided context.",
messages=messages,
output_config={"format": {"type": "json_schema", "schema": OUTPUT_SCHEMA}},
)
# refusal 或 max_tokens 時，輸出不一定符合 schema
if response.stop_reason in ("refusal", "max_tokens"):
raise RuntimeError(f"unusable response: {response.stop_reason}")
return next(b.text for b in response.content if b.type == "text")
retrieved = {"CLAUSE-003": "...", "CLAUSE-004": "..."}
prompt = "<context>\n" + "\n".join(f"[{k}] {v}" for k, v in retrieved.items()) + \
"\n</context>\nAssess CLAUSE-004 against the attached migration script."
assessment, attempts = generate_validated(claude, prompt, context={"retrieved_ids": set(retrieved)})如果不需要自訂修復迴圈，Anthropic SDK 也提供 client.messages.parse(output_format=YourModel)，會自動轉換 schema 並在本機驗證，回傳 response.parsed_output。另外，舊教學常見「強制 tool_choice 來取得 JSON」的寫法；目前較新的模型（例如 claude-opus-5-5）不支援強制 tool_choice，應改用 structured output。
If you don't need a custom repair loop, the Anthropic SDK also offers client.messages.parse(output_format=YourModel), which converts the schema, validates locally and returns response.parsed_output. Older tutorials often force tool_choice to get JSON; newer models such as claude-opus-5-5 reject forced tool_choice, so use structured output instead.
動手試：驗證器Try it: the validator
ClauseRiskAssessment 驗證ClauseRiskAssessment validation
和 Python lab 相同的規則（JS 版）。選一個情境，或直接編輯 JSON。注意「捏造引用」這個情境：只有開啟右邊的選項，用檢索結果當 context 時才抓得到。
The same rules as the Python lab, in JS. Pick a scenario or edit the JSON. Note the "fabricated citation" case: it is only caught when the option on the right uses the retrieval results as context.
設計 schema 的建議Schema design tips
- 能用 enum 就不要用自由文字。下游的
if判斷需要固定的值。 - 允許「不知道」。如果資料不足，給模型一個明確的出口（例如
"risk_level": "UNKNOWN"或可為 null 的欄位），否則它會被迫猜一個答案。 - 欄位描述寫給模型看。schema 的 description 會影響輸出品質，寫清楚格式與例子。
- 不要把 confidence 當機率。模型自報的信心分數通常沒有校準。如果需要可靠的信心，用評測資料去校準，或用其他訊號（檢索分數、多次取樣的一致性）。
- 版本化。schema 改了，下游的解析程式、測試集與儲存的資料都會受影響，當成 API 版本管理。
- Prefer enums to free text. Downstream
ifstatements need fixed values. - Allow "don't know". When information is missing, give the model an explicit exit (e.g.
"risk_level": "UNKNOWN"or a nullable field), or it will be forced to guess. - Write field descriptions for the model. Schema descriptions affect output quality; state formats and give examples.
- Don't treat confidence as probability. Self-reported confidence is usually uncalibrated. If you need reliable confidence, calibrate against eval data or use other signals (retrieval scores, agreement across samples).
- Version it. A schema change affects downstream parsers, test sets and stored data; manage it like an API version.