從需求到上線的交付流程The engagement lifecycle
大多數 AI 專案失敗不是因為技術做不到，而是因為一開始沒講清楚要解決什麼、怎樣算成功。這一章談的是在寫程式之前與之後，必須做好的那些事。
Most AI projects fail not because the technology can't do it, but because nobody agreed up front on what problem is being solved and what success means. This chapter covers what has to happen before and after the code.
讀完這章你能夠After this chapter you can
- 主持一場需求探索，問出範圍、錯誤代價與決策者。
- 和客戶一起寫下可量測、有人負責判定的成功條件。
- 說出每個階段應該交出什麼，以及最常卡住的地方。
- Run a discovery session that surfaces scope, cost of errors and decision makers.
- Write measurable success criteria with the customer, each with a named owner.
- State what each phase must deliver and where projects usually stall.
五個階段Five phases
下圖是本課程使用的階段劃分。實際上各階段會重疊、會來回，時間長短也依專案差異很大，不要把它當成固定的天數表。
The diagram shows the phases this course uses. In practice they overlap and loop, and durations vary widely between projects, so don't treat it as a fixed schedule.
階段 1：釐清問題Phase 1: Discovery
目標是把「我們想用 AI 做 X」變成一段可以驗收的問題描述。最有效的方法是看使用者實際操作，而不是只聽主管描述。
The goal is to turn "we want to use AI for X" into a problem statement that can be accepted or rejected. The most effective method is to watch users do the work, not just listen to their manager describe it.
一定要問的問題Questions to always ask
| 問題Question | 為什麼重要Why it matters |
|---|---|
| 現在誰在做這件事？每天做多少次？每次花多久？Who does this today? How often? How long each time? | 決定價值上限，也提供改善前的基準。Sets the ceiling on value and gives you a before-baseline. |
| 做錯一次的代價是什麼？誰會發現？What does one mistake cost? Who notices? | 決定要多少人工審核，以及能不能全自動。Decides how much human review is needed and whether full automation is possible. |
| 需要的資料在哪裡？誰能核准存取？Where is the data? Who can approve access? | 資料存取常是時程上最長的一段。Data access is often the longest item on the schedule. |
| 誰決定上線？誰會被這個系統影響？Who decides go-live? Who is affected by the system? | 找出真正的決策者與可能的反對者。Finds the real decision maker and potential blockers. |
| 有哪些法規、資安或資料留存的限制？What regulatory, security or retention constraints apply? | 可能直接決定能用哪個模型、部署在哪裡（見第 08 章：雲端 API 還是自己架）。May dictate which models you can use and where they run (see chapter 08: cloud API or self-hosted). |
| 之前試過什麼？為什麼沒成功？What has been tried before? Why didn't it work? | 避免重蹈覆轍，也揭露組織內部的期待。Avoids repeating failures and reveals internal expectations. |
先確認：這真的需要 LLM 嗎？First check: does this need an LLM at all?
如果輸入格式固定、規則明確，一般的程式或規則引擎更便宜、更快、更好測試。如果是找資料，好的搜尋可能就夠了。LLM 適合的是輸入是自然語言、變化大，而且可以容忍並偵測錯誤的工作。能說出「這部分不需要 AI」，會讓客戶更信任你其他的建議。
If inputs have a fixed format and clear rules, ordinary code or a rules engine is cheaper, faster and easier to test. If the job is finding information, good search may be enough. LLMs fit work where inputs are natural language, highly variable, and errors can be tolerated and detected. Being able to say "this part doesn't need AI" makes the customer trust your other recommendations more.
交付物：成功條件Deliverable: success criteria
每一條都要有：量什麼、怎麼量、門檻、誰判定。下表是範例，數字要和你的客戶一起定。
Each criterion needs: what is measured, how, the threshold, and who judges it. The table is an example; agree the numbers with your customer.
| 條件Criterion | 量測方式Measured by | 門檻（範例）Threshold (example) | 判定者Owner |
|---|---|---|---|
| 答案正確Answer correctness | 200 題真實問題，由兩位資深客服評分200 real questions graded by two senior agents | ≥ 85% | 客服主管Support lead |
| 不亂答No fabrication | 答案中沒有出處支持的陳述比例Share of answers containing unsupported claims | ≤ 2% | 客服主管Support lead |
| 權限Permissions | 權限測試案例，回傳無權限文件的次數Permission test cases returning unauthorized documents | 0 | 資安Security |
| 回應時間Latency | 正式環境 P95Production P95 | ≤ 8 s | IT 維運IT operations |
| 業務成果Business outcome | 每張工單平均處理時間Average handling time per ticket | 比導入前減少 20%20% lower than before | 營運主管Operations head |
把「不能出現的錯誤」和「希望變好的指標」分開寫。權限外洩、對外承諾錯誤價格這類錯誤是零容忍，應該用程式擋，而不是用平均分數衡量。
Separate "errors that must never happen" from "metrics we want to improve". Leaking restricted data or quoting wrong prices externally are zero-tolerance; block them in code rather than averaging them into a score.
階段 2：資料與權限Phase 2: Data and access
- 第一天就送出存取申請。帳號、VPN、資料庫唯讀帳號、模型 API 的使用核准，都可能要等好幾週。
- 做一份資料清單：來源、格式、更新頻率、擁有者、誰能看。最後一欄會直接變成檢索的權限過濾條件（第 5 章）。
- 開始蒐集真實問題。從工單、信件、聊天紀錄中挑出 50 到 100 題，請業務專家寫下正確答案。這就是測試集的雛形（第 7 章）。
- 確認個資處理方式：哪些欄位要遮蔽、資料能不能送到外部 API、紀錄要保留多久。
- File access requests on day one. Accounts, VPN, read-only DB users and approval to use a model API can each take weeks.
- Build a data inventory: source, format, update frequency, owner, who may see it. That last column becomes your retrieval permission filter (chapter 5).
- Start collecting real questions. Pull 50–100 from tickets, email or chat logs and have domain experts write the correct answers. That is your seed test set (chapter 7).
- Settle PII handling: which fields to mask, whether data may go to an external API, how long logs are kept.
階段 3：原型Phase 3: Prototype
做一條從輸入到輸出完整打通的細線，而不是把某一層做到完美。用真實資料、交給真實使用者，觀察他們卡在哪裡。這個階段 coding agent 可以大幅加速（第 3 章），但要記得原型的目的是學習，不是交付。
Build a thin slice that runs end to end, rather than perfecting one layer. Use real data, hand it to real users and watch where they get stuck. Coding agents speed this phase up considerably (chapter 3), but remember the prototype exists to learn, not to ship.
原型 demo 成功後，客戶常會問「下週可以上線嗎？」。事先講清楚原型缺了什麼：評測、監控、權限、錯誤處理。把第 4 階段的工作列成清單給對方看。
After a successful demo, customers often ask "can we go live next week?". Say up front what the prototype lacks: evals, monitoring, permissions, error handling. Show them the phase-4 list.
階段 4：強化Phase 4: Hardening
這是把原型變成正式系統的階段，也是第 4 到 8 章的主要內容：
This turns the prototype into a production system, and is what chapters 4–8 cover:
- 模型輸出經過 schema 與業務規則驗證（第 4 章）。
- 檢索有權限過濾，並量過 recall（第 5 章）。
- 會改資料的操作需要核准，並留下紀錄（第 6 章）。
- 測試集在每次改動時自動執行，有退步就擋下（第 7 章）。
- 有追蹤、告警、成本監控與降級方案（第 8 章）。
- Model output passes schema and business-rule validation (chapter 4).
- Retrieval filters by permission and has measured recall (chapter 5).
- Data-changing actions need approval and are logged (chapter 6).
- The test set runs on every change and blocks regressions (chapter 7).
- Tracing, alerting, cost monitoring and a degraded mode exist (chapter 8).
階段 5：上線與交接Phase 5: Launch and handover
- 分階段上線：先讓一小群使用者用，或先「只建議、不執行」，確認指標正常再擴大。
- 寫 runbook：常見告警代表什麼、怎麼查、怎麼回滾、要找誰。
- 明確的擁有者：誰負責模型升級？誰維護測試集？誰付 API 費用？
- 回頭對照成功條件：用第 1 階段寫下的條件做驗收，而不是用感覺。
- Roll out in stages: start with a small group, or in "suggest, don't act" mode, and widen once metrics hold.
- Write a runbook: what each alert means, how to investigate, how to roll back, whom to call.
- Name owners: who handles model upgrades, who maintains the test set, who pays for the API.
- Review against the success criteria written in phase 1, not against impressions.
範本：架構決策紀錄Template: architecture decision record
重要的技術選擇寫成一頁 ADR。半年後有人問「為什麼不用向量資料庫？」時，你（或接手的人）會感謝現在的自己。
Write important technical choices as one-page ADRs. Six months from now, when someone asks "why didn't we use a vector database?", you or your successor will be glad you did.
# ADR 0003: Use PostgreSQL + pgvector for retrieval
Status: accepted (2026-03-14)
## Context
- ~40k documents, ~200k chunks; growth < 10%/year.
- Customer already runs PostgreSQL 16 with backups, monitoring and an on-call team.
- Queries often contain exact part numbers (e.g. "PR-9821-X").
- Documents carry per-department access rules.
## Decision
Store chunks, embeddings and a tsvector column in the existing PostgreSQL cluster.
Run vector and full-text search in one query and fuse with RRF (k=60).
Apply access rules in the SQL WHERE clause.
## Consequences
+ No new system for the customer to operate; access rules live next to the data.
+ Exact-ID queries handled by full-text search.
- Index build time grows with data; revisit if chunks exceed ~5M.
- Re-embedding after a model change needs a backfill job (see runbook §4).
## Alternatives considered
- Managed vector DB: extra vendor review, duplicated access rules.
- Vector-only search: failed 9 of 20 exact-ID test queries in the spike.常見風險與早期訊號Common risks and early signals
| 風險Risk | 早期訊號Early signal | 對策Mitigation |
|---|---|---|
| 資料拿不到No data access | 第二週還在用假資料Still on fake data in week two | 升級到客戶的專案負責人；先用匿名化樣本推進Escalate to the customer's sponsor; proceed with an anonymised sample |
| 範圍蔓延Scope creep | 每次會議都多一個「順便」Every meeting adds a "while you're at it" | 新需求放進待辦清單，對照成功條件決定優先順序Put new asks on a backlog and prioritise against the success criteria |
| 沒人在用Nobody uses it | 原型上線兩週，使用次數個位數Single-digit usage two weeks after the prototype ships | 回到現場觀察；問題可能是流程而不是模型Go back and observe; the problem may be workflow, not the model |
| 驗收標準變動Moving goalposts | 「感覺還不夠好」"It doesn't feel good enough yet" | 拿出書面成功條件與測試集結果討論Discuss using the written criteria and test-set results |