更好的搜尋，並量出差別Better search, measured

機器人的回答品質，上限由搜尋決定：沒找到的段落，AI 不可能引用。這一章把入門篇 B4 的陽春搜尋改進兩次，而且每一次都用同一組測試題量出效果，不靠感覺。

Search sets the ceiling on answer quality: a passage that isn't found can't be cited. This chapter improves the basic search from B4 twice, and measures each step against the same test questions instead of going by feel.

切段：一個主題一段Chunking: one topic per chunk

手冊用 ## H01 特休假 這樣的標題分段。我們就照標題切，每段帶上標題，讓「特休假：……」單獨拿出來也看得懂：

The handbook is divided by headings like ## H01 特休假. We split on those headings and keep the title on each chunk, so a chunk reads sensibly on its own:

def load_chunks(path: Path = HANDBOOK_PATH) -> list[Chunk]:
    """依「## H01 標題」切段，每段一個 Chunk。"""
    text = path.read_text(encoding="utf-8")
    chunks = []
    for m in re.finditer(r"^## (H\d+) (.+?)\n(.*?)(?=^## |\Z)", text, re.S | re.M):
        chunks.append(Chunk(m.group(1), m.group(2).strip(), " ".join(m.group(3).split())))
    return chunks

三個版本Three versions

v1：數相同的片段count shared pieces

就是入門篇 B4 的做法。問題：「公司」「提供」這種到處都有的片段，和「特休」這種很有資訊量的片段，算一樣多分。所以「公司有提供免費午餐嗎？」會被配到設備那段。

The B4 approach. Problem: everywhere-pieces like 公司 (company) and 提供 (provide) score the same as informative ones like 特休 (annual leave). So "does the company provide free lunch?" gets matched to the equipment section.

v2：罕見的片段比較重要，加上門檻rare pieces matter more, plus a threshold

一個片段出現在越少段落裡，代表它越能區分段落，權重就越高（這個概念叫 IDF）。再加上一條規則：總分低於門檻就當作「找不到」，讓機器人誠實說手冊沒寫。

The fewer chunks a piece appears in, the better it distinguishes them, so it gets more weight (this idea is called IDF). Add one rule: if the total score is below a threshold, treat it as "not found" so the bot honestly says the handbook doesn't cover it.

# 每個片段出現在幾段裡；越少見，權重越高（IDF 的概念）
n = len(self.chunks)
df = {}
for ps in self.chunk_pieces.values():
    for p in ps:
        df[p] = df.get(p, 0) + 1
self.weight = {p: math.log((n + 1) / (f + 0.5)) for p, f in df.items()}

...
score = sum(self.weight[p] for p in shared)          # v2、v3
...
hits = [h for h in hits if h.score >= self.min_score] # 低於門檻就不要

v3：同義詞表a synonym table

員工會說「年假」「筆電」「在家上班」，手冊寫的是「特休」「筆記型電腦」「遠端工作」。最直接的做法是維護一張同義詞表，問題裡出現左邊的詞，就把右邊的詞也加進去搜尋：

Employees say 年假, 筆電 and 在家上班; the handbook says 特休, 筆記型電腦 and 遠端工作. The most direct fix is a synonym table: when a left-hand word appears in the question, add the right-hand word to the search:

SYNONYMS = {
    "筆電": "筆記型電腦",
    "年假": "特休",
    "在家上班": "遠端工作",
    "報銷": "報帳",
    "辭職": "離職",
    "進修": "教育訓練",
    ...
}

量出差別Measure the difference

測試題共 18 題，每題標了類型。執行 python -m labs.intermediate.evals --search 的實際結果：

There are 18 test questions, each tagged by type. Actual output of python -m labs.intermediate.evals --search:

只看總分，v1 和 v2 一樣是 11 題。但分類看就很清楚：

On the total alone, v1 and v2 tie at 11. Broken down by type, the picture is clear:

• v2 解決了「亂配」（手冊沒寫的 3 題全對），但門檻也擋掉了一些分數不高的正確結果，同義詞題反而變差。這就是取捨：門檻越高，越不會亂答，但也越容易漏掉。
• v3 補回了同義詞題，總分提升到 15。
• 這就是為什麼要分類：如果只看總分，你會以為 v2 沒有用。

• v2 fixes false matches (all 3 not-in-handbook questions pass), but the threshold also blocks some low-scoring correct results, so synonym questions get worse. That's the trade-off: a higher threshold means fewer wrong answers but more misses.
• v3 wins back the synonym questions, lifting the total to 15.
• This is why you tag questions: from the total alone you'd conclude v2 was useless.

動手試：三個版本並排比較Try it: three versions side by side

剩下的 3 題The remaining three

我們沒有為了讓這 3 題通過而修改題目或硬調參數。誠實的測試集應該反映真正的弱點。要解決「換句話說」，需要能理解意思的搜尋方法：embedding（向量）搜尋，並和關鍵字搜尋合併使用。這是進階篇第 05 章的主題。

We did not reword questions or force parameters to make these pass. An honest test set reflects real weaknesses. Paraphrases need search that understands meaning: embedding (vector) search, combined with keyword search. That's the subject of core chapter 05.

練習Exercises

1. 把 min_score 改成 2.0 和 4.0，各跑一次 --search。哪一類題目變好、哪一類變差？
2. 在 SYNONYMS 加一條，讓「年假要提前多久講？」通過，然後確認其他題目沒有變差。
3. 在 eval_set.jsonl 加 3 題你自己想的問題，至少一題是手冊沒寫的。

1. Set min_score to 2.0 and 4.0 and run --search for each. Which question types improve and which get worse?
2. Add one synonym entry so 「年假要提前多久講？」 passes, then confirm nothing else regressed.
3. Add 3 questions of your own to eval_set.jsonl, at least one not covered by the handbook.

小測驗Quiz