實作練習與延伸資源Labs and further reading
三個練習都放在 labs/,只需要 Python 與兩個套件,完全離線、不需要 API key。每個練習都有測試,改完程式碼跑一次就知道有沒有弄壞東西。
All three labs live in labs/ and need only Python plus two packages. They run fully offline with no API key. Every lab has tests, so one run tells you whether a change broke something.
安裝Setup
# Python 3.10 以上 / Python 3.10+
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -r requirements.txt
python -m pytest # 105 passed
入門篇練習Basics exercises
對應入門篇 B1 到 B5。每個檔案都很短、沒有 class,可以直接讀懂。
Covers basics chapters B1–B5. Each file is short, has no classes, and reads top to bottom.
| 檔案File | 內容Contents | 執行Run |
|---|---|---|
next_word.py | 只讀過六句話的「猜下一個字」模型(B1)A next-character model trained on six sentences (B1) | python -m labs.beginner.next_word |
first_call.py | 第一次呼叫 API,看對話歷史怎麼累積(B2)A first API call; watch the history accumulate (B2) | python -m labs.beginner.first_call |
handbook_qa.py | 搜尋員工手冊並組合 prompt(B4)Search a handbook and assemble a prompt (B4) | python -m labs.beginner.handbook_qa "問題" |
check_answers.py | 會自動閱卷的考卷(B5)An exam that grades itself (B5) | python -m labs.beginner.check_answers |
延伸作業Extension exercises
- 在
next_word.py的訓練資料加入五句你自己寫的話,觀察生成結果怎麼改變。 - 在
handbook_qa.py加一個「同義詞表」(例如「筆電」→「筆記型電腦」),讓考卷的第 5 題通過。 - 在
check_answers.py加三題考題,其中一題是陷阱題。
- Add five sentences of your own to the training data in
next_word.pyand watch how the output changes. - Add a synonym table to
handbook_qa.py(e.g. 筆電 → 筆記型電腦) so exam question 5 passes. - Add three questions to
check_answers.py, including one trick question.
中級篇專案:員工手冊機器人Intermediate project: the handbook bot
對應中級篇 I1 到 I6。完整的小型應用,每個功能都有測試;沒有 API key 時自動使用離線假模型。
Covers intermediate chapters I1–I6. A complete small application with tests for every feature; without an API key it uses the offline fake model.
python -m labs.intermediate.chat --fake # 聊天(I1–I5)
python -m labs.intermediate.chat --ollama # 改用本機開源模型(需安裝 Ollama,見第 08 章)
uvicorn labs.intermediate.web:app --reload # 網頁版(I7),打開 http://127.0.0.1:8000
python -m labs.intermediate.evals --search # 比較三個搜尋版本(I3)
python -m labs.intermediate.evals # 評測整個機器人(I6)
python -m pytest labs/intermediate # 單元測試
每章的練習寫在各章頁面最後。修改手冊、同義詞或價格之後,執行 python scripts/build_demo_data.py 更新網頁示範的資料。
Each chapter's exercises are at the end of its page. After changing the handbook, synonyms or prices, run python scripts/build_demo_data.py to refresh the web demos' data.
練習 1:核心模式Lab 1: core patterns
對應第 4 到 7 章。每個模組都可以單獨執行,看看它在做什麼。
Covers chapters 4–7. Each module runs on its own so you can see what it does.
| 檔案File | 內容Contents | 執行Run |
|---|---|---|
contracts.py | Pydantic 契約、引用檢查、修復迴圈(第 4 章)Pydantic contract, citation check, repair loop (ch. 4) | python -m labs.core.contracts |
retrieval.py | BM25、3-gram 向量、RRF、權限過濾、指標(第 5 章)BM25, 3-gram vectors, RRF, ACL filter, metrics (ch. 5) | python -m labs.core.retrieval |
agent.py | 狀態機、checkpoint、中斷與核准、唯讀連線(第 6 章)State machine, checkpoints, interrupt and approval, read-only connection (ch. 6) | python -m labs.core.agent |
evals.py | 測試集、程式評分、baseline 比較(第 7 章)Test set, code graders, baseline gate (ch. 7) | python -m labs.core.evals |
延伸作業Extension exercises
- 允許「不知道」:在
risk_level加入UNKNOWN,並規定此時findings必須說明缺少什麼資訊。寫測試。 - 接上真正的 embedding 模型:實作
Embedder介面(例如用 sentence-transformers 或任一 embedding API),傳給HybridRetriever。重跑評測:semantic-oncall應該會通過。確認沒有其他案例退步後,更新 baseline。 - 核准期限:在
Agent.resume加入檢查,等待超過指定時間的核准一律拒絕,並寫測試。 - 你自己的測試集:把語料換成你工作領域的 10 份文件,寫 15 題測試案例,其中至少 3 題是權限案例、1 題是已知會失敗的案例。
- Allow "don't know": add
UNKNOWNtorisk_level, and requirefindingsto state what information is missing in that case. Write tests. - Plug in a real embedding model: implement the
Embedderprotocol (e.g. with sentence-transformers or any embedding API) and pass it toHybridRetriever. Re-run the evals:semantic-oncallshould now pass. Once nothing else regressed, update the baseline. - Approval expiry: make
Agent.resumereject approvals that arrive after a configurable delay, with tests. - Your own test set: replace the corpus with 10 documents from your domain and write 15 test cases, including at least 3 permission cases and 1 known failure.
練習 2:AI code reviewerLab 2: AI code reviewer
對應第 3、4 章。用 Python 的 ast 模組做確定性的靜態檢查(SQL injection、寫死的金鑰、缺少逾時、eval、在程式中 DROP TABLE),可以選擇加上 LLM 補充語意問題。重點在設計:LLM 只提出發現,分數與結論由程式計算;LLM 回報的行號不存在就丟棄。
Covers chapters 3 and 4. Deterministic static checks with Python's ast module (SQL injection, hard-coded secrets, missing timeouts, eval, DROP TABLE in app code), with an optional LLM pass for semantic issues. The point is the design: the LLM only proposes findings; score and verdict are computed in code, and findings on non-existent lines are dropped.
python -m labs.code_reviewer.cli # 預設檢查 samples/vulnerable_api.py
python -m labs.code_reviewer.cli labs/code_reviewer/samples/clean_api.py
python -m labs.code_reviewer.cli path/to/your_file.py --rules team_rules.md
# 選用:加上 LLM 補充(需要 pip install google-genai)
export GEMINI_API_KEY=...
延伸作業Extension exercises
- 新增規則:偵測
subprocess.run(..., shell=True)搭配非常數參數。先寫測試,包含「常數字串不應該被報告」的案例。 - 在你自己的程式庫上執行,記錄誤報,然後修正規則並把誤報寫成測試。
- 把 LLM 補充改成使用你偏好的 provider(例如第 4 章的 Claude 寫法),保持
validate_llm_issues與merge_issues不變。
- Add a rule detecting
subprocess.run(..., shell=True)with non-constant arguments. Write tests first, including "constant strings are not reported". - Run it on your own codebase, record the false positives, then fix the rules and turn each false positive into a test.
- Switch the LLM pass to your preferred provider (e.g. the Claude pattern from chapter 4), leaving
validate_llm_issuesandmerge_issuesunchanged.
練習 3:MCP SRE 服務Lab 3: MCP SRE service
對應第 5、6、8 章。一個最小的 MCP server(stdio)與 client,加上一個「診斷 → 查 runbook → 提出修改 → 人工核准 → 執行 → 驗證」的 workflow。雲端環境是模擬的(cloud_sim.py),不會呼叫任何真正的 GCP API。
Covers chapters 5, 6 and 8. A minimal MCP server (stdio) and client, plus a "diagnose → search runbooks → propose change → human approval → apply → verify" workflow. The cloud is simulated (cloud_sim.py); no real GCP API is called.
python -m labs.mcp_sre.cli # 診斷 payment-gateway,修改前會詢問 y/N
python -m labs.mcp_sre.cli --stdio # 以子行程啟動 server,走真正的 stdio
python -m labs.mcp_sre.server # 單獨啟動 server,從 stdin 讀 JSON-RPC
這個 server 只實作 MCP 的一小部分(initialize、tools/list、tools/call、ping),目的是讓你看懂協定本身。正式專案請使用官方的 MCP SDK,它會處理完整的協定、傳輸與相容性。
This server implements only a small part of MCP (initialize, tools/list, tools/call, ping) so you can see the protocol itself. For real projects use an official MCP SDK, which handles the full protocol, transports and compatibility.
延伸作業Extension exercises
- 新增唯讀工具
get_recent_deploys(name),讓診斷流程在修改擴容設定前,先檢查是不是最近的部署造成的。 - 新增一筆「修改後狀態沒有改善」的模擬情境,讓 workflow 回報失敗並建議回滾。
- 把 server 改用官方 MCP Python SDK 實作,保留相同的工具與 annotations,讓現有測試繼續通過。
- Add a read-only tool
get_recent_deploys(name)so the diagnosis checks for a recent deploy before touching scaling. - Add a simulated scenario where the change doesn't help, and make the workflow report failure and recommend rollback.
- Re-implement the server with the official MCP Python SDK, keeping the same tools and annotations, so the existing tests still pass.
下一步:作品集專案Next: a portfolio project
練習做完之後,挑一個作品集專案,用自己的資料從頭做一次,並照著頁面上的 README 範本與檢查表包裝成作品。
Once the labs are done, pick a portfolio project, build it from scratch with your own data, and present it using the README template and checklist on that page.
延伸閱讀Further reading
| 資料Resource | 對應章節與重點Chapters and why |
|---|---|
| Anthropic — Building effective agents | 第 6 章。workflow 與 agent 的區分、常見模式,以及「從簡單開始」。Ch. 6. Workflows vs agents, common patterns, and "start simple". |
| Hamel Husain — Your AI Product Needs Evals | 第 7 章。從錯誤分析出發的評測方法。Ch. 7. Evaluation grounded in error analysis. |
| Simon Willison — prompt injection | 第 8 章。提示注入的持續追蹤,包括「致命三要素」。Ch. 8. Ongoing coverage of prompt injection, including the lethal trifecta. |
| OWASP GenAI Security Project | 第 8 章。LLM 應用的風險清單,可用來和客戶的資安團隊對話。Ch. 8. Risk lists for LLM applications, useful when talking to customer security teams. |
| Model Context Protocol | 第 6 章、練習 3。協定規格與官方 SDK。Ch. 6, lab 3. Specification and official SDKs. |
| pgvector | 第 5 章。索引類型、參數與過濾查詢的注意事項。Ch. 5. Index types, parameters and filtered-query caveats. |
| Pydantic | 第 4 章。validator、validation context、JSON Schema 產生。Ch. 4. Validators, validation context, JSON Schema generation. |
| Chip Huyen — AI Engineering (O'Reilly, 2025) | 全課程。以基礎模型建構應用的完整工程觀點。Whole course. A full engineering view of building on foundation models. |
| Martin Kleppmann — Designing Data-Intensive Applications | 第 5、8 章。AI 系統的很多問題其實是資料系統的問題:一致性、索引、快取。Ch. 5, 8. Many AI-system problems are data-system problems: consistency, indexing, caching. |
| Cormack, Clarke & Büttcher — Reciprocal Rank Fusion outperforms Condorcet and individual rank learning methods (SIGIR 2009) | 第 5 章。RRF 的原始論文,很短。Ch. 5. The original RRF paper; short. |
關於證照About certifications
對 FDE 這類職位,證照的參考價值有限。比較有說服力的是:你做過、上線過、能講清楚取捨的系統,加上一份公開的作品或技術文章。如果客戶主要在某一家雲端平台上,該平台的架構類證照能幫你更快和客戶的基礎架構團隊建立共同語言,這是實際的用處。不要為了收集證照而犧牲做專案的時間。
For FDE-type roles, certifications carry limited weight. What persuades is systems you built and shipped and can explain the trade-offs of, plus public work or technical writing. If your customers mostly run on one cloud, that cloud's architecture certification helps you share vocabulary with their infrastructure teams faster; that is its practical value. Don't trade project time for collecting certificates.
接下來What next
挑一個你工作上真實存在的小問題,照第 2 章的方式寫下成功條件,用第 4 到 7 章的方法做出來,並在第一週就建立測試集。做完一次完整的循環,比讀完所有教材更有收穫。
Pick a small, real problem from your own work, write success criteria as in chapter 2, build it with the methods from chapters 4–7, and start the test set in week one. One full loop teaches more than any amount of reading.