LLM answer cache keyed only on the input keeps serving old answers after you change the prompt or model
A batch job cached each model answer under sha256(url + normalized page text), so an unchanged page was never asked again. After the extraction instructions were rewritten to fix a wrong answer, the next run reused last month's answer for every unchanged page, and the fix never reached them. Put the model id and a fingerprint of the fixed instructions into the key (the instructions rendered with placeholders in place of the per-item input), and treat entries written without a fingerprint as stale.
Setup
A monthly job fetches ~150 web pages, asks an LLM to extract structured facts from each one (no tools, quotes checked verbatim against the page), and caches each answer in a JSON file so unchanged pages cost nothing the next month.
Cache key: sha256(url + "\n" + normalize(page_text)).
What went wrong
- One page's answer was wrong because of a gap in the instructions: for a usual time given as a range ("late April or early May"), the model returned no month.
- The instructions were fixed so that a range gives its first month. A unit test against a fresh answer passed.
- On the next real run, that page had not changed, so its key was unchanged. The old answer, made under the old instructions, was reused, and the wrong result came back.
Nothing errored. The run reported most pages as "unchanged since last time, not asked again", which looked like the cache working as intended.
Fix
// The fixed instructions, rendered with placeholders where the per-item input goes
const instructions = () => buildPrompt({ url: "<url>", text: "<page>", today: "<today>" });
const FINGERPRINT = sha256(`${MODEL}\n${instructions()}`);
const key = sha256(`${MODEL}\n${FINGERPRINT}\n${url}\n${normalize(text)}`);
// store `fingerprint` in each entry; an entry with none (old format) counts as a miss
- Change any word of the instructions, or the model: every item is asked again once on the next run (here ~150 calls instead of ~50). Unchanged instructions plus unchanged input still cost 0 calls.
- Render the template with placeholders, not the real input: otherwise the fingerprint changes with every item and with today's date, and the cache never hits.
- Per-item context that is part of the input (here, the list entry being re-checked) belongs in the input half of the key, not in the fingerprint.
Check that the test can fail
Three fixture tests:
- An old-format entry is asked again.
- Two runs with the same instructions: the second makes 0 calls.
- Changing one word of the instructions means the page is asked again.
With the fingerprint removed from the key, test 1 or 3 fails. The next real run after the fix asked every page again, and the previously stuck page came out right.
General point
An LLM answer is a function of (model, instructions, input). A cache keyed on input alone silently pins every past answer to the instructions it was made with. The cost shows up only on the next real run, after the fix looked done.
Conditions
- node
- 24
- os
- Ubuntu 24.04
- model_call
- headless CLI, no tools
- cache
- JSON file, atomic write
- observed
- 2026-10-15