對話記憶與成本Conversation memory and cost

員工問完「特休要提前幾天？」，接著問「那病假呢？」。機器人要記得前一題，才看得懂「那」。但記得越多，每次送出的 token 越多，費用也越高。這一章在兩者之間找平衡。

An employee asks how far ahead to request annual leave, then "what about sick leave?". The bot must remember the previous question to understand "what about". But the more it remembers, the more tokens each request sends and the more it costs. This chapter balances the two.

歷史要存什麼What to keep in history

入門篇 B2 說過：API 不會記得你，每次都要把對話重新送出去。但每一輪實際送出的內容，除了問題，還有好幾段手冊（可能是問題的十幾倍長）。如果把這些全部存進歷史，對話越長就越貴，而且舊的手冊段落多半和新問題無關。

Basics chapter B2 showed that the API doesn't remember you, so you resend the conversation each time. But each turn sends not just the question, but several handbook passages, often many times longer. Storing all of that in history makes long chats expensive, and old passages are mostly irrelevant to new questions.

所以我們只存「問題」與「最終回答」。新問題的手冊段落，每次重新搜尋：

So we keep only questions and final answers, and search afresh for each new question:

recent = self.history[-2 * self.max_history_turns:] if self.max_history_turns else []
messages = recent + [{"role": "user", "content": build_user_message(question, hits)}]
...
# 回答通過檢查後，只把「問題」與「回答文字」存進歷史
self.history += [{"role": "user", "content": question},
                 {"role": "assistant", "content": turn.answer.answer}]

記錄用量與費用Tracking usage and cost

每次 API 回應都會告訴你這次用了多少輸入與輸出 token。把它記下來，就能算出費用。不要等到月底收到帳單才知道花了多少。

Every API response reports the input and output tokens used. Record them and you can compute cost. Don't wait for the monthly bill to find out what you spent.

@dataclass
class Usage:
    calls: int = 0
    input_tokens: int = 0
    output_tokens: int = 0

    def add(self, reply) -> None:
        self.calls += 1
        self.input_tokens += reply.input_tokens
        self.output_tokens += reply.output_tokens

    def cost_usd(self, model: str = config.MODEL) -> float:
        price_in, price_out = config.PRICES_PER_MTOK[model]
        return (self.input_tokens * price_in + self.output_tokens * price_out) / 1_000_000

注意：一個問題可能呼叫 API 不只一次（用工具時兩次、重試時再多一次），所以「每個問題的成本」要把這些都加起來。chat.py 輸入 /usage 可以看到目前累計。

Note that one question may call the API more than once (twice with a tool, once more on a retry), so cost per question must add them all up. Type /usage in chat.py to see the running total.

動手試：對話越長，每次送出越多Try it: longer chats send more each time

保留全部歷史時，每次請求的輸入會一直線性變長，總成本則是越聊越快地增加。只保留最近幾輪，每次的輸入很快就會穩定下來。

Keeping all history makes each request's input grow linearly, so total cost grows faster and faster. Keeping only recent turns makes per-request input level off quickly.

再省一點：prompt cachingSaving more: prompt caching

我們的 system prompt 每次都一模一樣。很多 API 提供 prompt caching：如果 prompt 的開頭和之前的請求完全相同，這段可以用比較便宜的價格計費，也比較快。要用好它只有一個原則：不變的內容放前面，會變的放後面。如果你在 system prompt 開頭放了「現在時間：14:03:22」，每次都不一樣，快取就永遠不會生效。詳細做法見進階篇第 08 章。

Our system prompt is identical every time. Many APIs offer prompt caching: if the beginning of a prompt matches an earlier request exactly, that part is billed at a lower rate and processed faster. One rule makes it work: stable content first, changing content last. Put "current time: 14:03:22" at the top of the system prompt and it differs every time, so the cache never hits. Details in core chapter 08.

練習Exercises

1. 用 chat.py --fake 連續問 6 個問題，每問一題就輸入 /usage，觀察輸入 token 怎麼變化。再把 MAX_HISTORY_TURNS 改成 0 重做一次。
2. 在 Turn 裡加上「這一題的費用」，並讓 chat.py 在每個回答後面顯示。

1. With chat.py --fake, ask 6 questions in a row, typing /usage after each, and watch input tokens change. Then set MAX_HISTORY_TURNS to 0 and repeat.
2. Add "cost of this question" to Turn and have chat.py show it after each answer.

小測驗Quiz