自動測試整個機器人Testing the whole bot

機器人有了搜尋、格式檢查、工具、記憶。每改一個地方，都可能影響其他地方。這一章把入門篇 B5 的「考卷」升級成兩種測試，讓你每次修改後幾秒內就知道有沒有弄壞東西。

The bot now has search, validation, tools and memory, and changing one part can affect the others. This chapter upgrades the B5 "exam" into two kinds of test that tell you within seconds whether a change broke something.

兩種測試，回答兩種問題Two kinds of test, two questions

單元測試：測「會出事」的地方Unit tests: test what can go wrong

好的單元測試不是只測「正常情況會成功」，而是測那些出事時會很嚴重的地方。以下是 tests/test_intermediate.py 的幾個例子：

Good unit tests don't just check the happy path; they target what would be serious if it broke. Some examples from tests/test_intermediate.py:

def test_tools_take_no_employee_id():
    # 權限：沒有任何工具能讓 AI 指定要查誰
    for tool in TOOLS:
        assert "employee_id" not in tool["input_schema"]["properties"]


def test_retry_fixes_fake_citation():
    # 第一次故意捏造出處，確認會重試並修正
    t = HandbookBot(FakeLLM(mistakes=["fake_citation"])).ask("事假一年可以請幾天？")
    assert t.attempts == 2 and t.answer.citations == ["H03"]


class LoopingLLM:
    """永遠要求用工具的模型，用來測試迴圈上限。"""
    def create(self, system, messages, tools=None, output_schema=None):
        call = ToolCall("t1", "get_today", {})
        return Reply([...], "", [call], "tool_use", 10, 10)


def test_tool_rounds_are_capped():
    # 成本：模型一直要工具時，迴圈一定會停
    t = HandbookBot(LoopingLLM()).ask("今天幾號？")
    assert "超過" in t.error and len(t.tool_calls) == config.MAX_TOOL_ROUNDS

注意 LoopingLLM：為了測一個很少發生、但發生了很貴的情況，我們寫了一個「故意壞掉」的模型。這就是第 1 章把 LLM 包成介面的回報。

Note LoopingLLM: to test a rare but expensive situation, we wrote a deliberately broken model. That's the payoff from wrapping the LLM in an interface in chapter I1.

評測集：每題檢查四件事The eval set: four checks per question

{"id": "wfh-synonym", "question": "一週可以在家上班幾天？", "relevant": ["H05"], "must_include": ["兩天"], "tags": ["synonym"]}
{"id": "no-parking", "question": "公司有員工停車位嗎？", "relevant": [], "expect_not_found": true, "tags": ["not-in-handbook"]}
{"id": "tool-balance", "question": "我今年還剩幾天特休？", "relevant": ["H01"], "must_include": ["4"], "expect_tool": "get_leave_balance", "tags": ["tool"]}

✓ annual-notice    請特休要提前幾天申請？
✗ annual-synonym   年假要提前多久講？
      搜尋：找到 []，應該是 ['H01']
      內容：答案沒有「三個工作天」
✗ core-hours       最晚幾點要到公司？
      搜尋：找到 []，應該是 ['H04']
      內容：答案沒有「十點」
✓ laptop-synonym   筆電壞了可以自己拿去修嗎？
✓ no-parking       公司有員工停車位嗎？
✓ tool-balance     我今年還剩幾天特休？
...
{"cases": 18, "passed": 15, "pass_rate": 0.8333}

失敗訊息直接指出是哪一步出問題：這兩題都是「搜尋」沒找到，所以要改的是搜尋（第 3 章），不是 prompt。分開檢查每一步，才知道該修哪裡。

The failure messages point straight at the broken step: both failed at search, so the fix belongs in search (chapter I3), not the prompt. Checking each step separately tells you where to look.

基準與 CIBaseline and CI

確認結果合理後，執行 --update-baseline 把分數存成基準。之後只要分數低於基準，evals.py 就會以失敗結束，CI 會擋下這次修改。這個專案的 GitHub Actions 已經設定好，每次推送都會跑全部測試。

Once the results look right, run --update-baseline to save the score. From then on, a lower score makes evals.py exit with failure and CI blocks the change. This project's GitHub Actions already runs every test on each push.

python -m pytest labs/intermediate                    # 單元測試
python -m labs.intermediate.evals                     # 評測（假模型），比較基準
python -m labs.intermediate.evals --update-baseline   # 人確認過後才更新基準
python -m labs.intermediate.evals --real              # 真模型（要 API key、會花錢、分數會浮動）
python -m labs.intermediate.evals --ollama            # 本機開源模型（免費但較慢，見第 08 章）

假模型與真模型的分工Fake vs. real model

中級篇完成了You've finished the intermediate track

下一章 I7 會把它部署成別人能用的網站。到這裡，你做出了一個完整的小型 AI 應用：會搜尋、會檢查自己的回答、會用工具、記得對話、知道花了多少錢，而且每個部分都有測試。這已經是很多公司內部工具的實際樣貌。

The next chapter, I7, deploys it as a website others can use. At this point you've built a complete small AI application: it searches, checks its own answers, uses tools, remembers the conversation, knows what it costs, and every part is tested. That's what many real internal tools actually look like.

進階篇處理的是「把這種系統放到客戶的正式環境」時會遇到的問題：

The core course tackles what happens when you put systems like this into a customer's production environment:

小測驗Quiz