Context lifecycle in local thinking models — measurement and findings

Measures what fills a context window over agent sessions and what a client’s auto-compaction keeps versus drops. Runs entirely on captures from the tap.py script in 01-token-audit/.

The scripts are client-agnostic. The findings below are from one specific setup: compaction behaviour changes depending on the client and version. A different client, or a later version of the same one, can behave differently. That is what the scripts are for: run them on your own captures and get your own measurements.

Test setup for the original runs

• Clients: pi v0.84.1–0.84.2 (earendil-works/pi, coding agent) and hermes-agent (NousResearch, personal vault setup)
• Models: Qwen3.6-35B-A3B (community finetune, Q4_K_M) and Ornith-1.5-35B-A3B (Q4_K_M)
• Server: llama.cpp llama-server, 65,536-token context, single slot (-np 1), prefix caching active
• Corpus: 5 days of normal use, August 2026. 495 requests, 485 conversation states, 31 sessions, 13 compaction events, 10 client summarisation calls

Run order

1. Leave the tap running in front of the inference server during normal use. Days, not minutes: compactions only happen in long sessions.

2. Decompose the captures into per-request rows:

   python3 decompose.py output.csv ~/captures/reqs-*.raw

3. Summarise:

   python3 summarise.py output.csv

What the scripts do

decompose.py extracts every chat-completions request from the raw captures (any JSON object containing a messages key), splits each into character counts per role (system / user / assistant / tool / tool schemas), and classifies every request:

• append — the previous conversation state is a prefix of this one: a normal turn.
• new_session — no prefix relationship with the previous state.
• compaction — same model, identical system prompt, message list rewritten, total size down more than 30%.
• aux_request — an auxiliary call from the client itself, e.g. the summarisation request that performs a compaction: when a client compacts the conversation, it sends the whole history to the model in one giant user message and asks for a summary. The script recognises these (no tools, one or two messages, one huge user payload) and keeps them out of the conversation timeline.

Session IDs are unique across files.

summarise.py prints four sets of data: per-session composition and compaction counts; overall composition shares, also per model; a compaction table showing what each event dropped by role; and post-compaction tool regrowth (how many tool characters reappeared within the next three states, the measured cost of dropping reconstructible content).

Character counts, not tokens: composition is reported as shares, and characters track tokens closely enough for that (~4 chars/token). Exact tokenisation would need a live server and adds nothing here.

Findings from the original runs

These numbers describe the setup above. n = 13 compactions on one machine.

• Tool results were 51% of all live context. Split by model, both clients mixed in each: Ornith sessions ran at 65% tool results; Qwen3.6 sessions at 45% with about double the assistant share. Whether that reflects the models’ working styles or the tasks they happened to get is not separable in this table.
• Both clients preserved the system prompt through compaction: dropped 0.0%, in 13 of 13 events. This keeps the KV-cache prefix valid, which is why the measured cache hit rate stayed at 99.5+% across a multi-compaction pi session.
• What compaction removed: weighted by size, about two thirds tool results, ~30% old assistant turns, around 2% user messages. 1.6M characters removed across the corpus; contexts cut to ~30% of pre-compaction size on average.
• pi’s tool-result retention is selective: when a compaction dropped all tool results, the next state re-fetched up to 84k characters immediately — the agent re-read the files it needed. When pi kept recent tool results and dropped only stale ones, regrowth was near zero (129 and 1,435 characters in the two clean cases). Drop-everything brings a re-read burst; keep-everything overflows.
• hermes handles compaction failure differently from pi. pi surfaces the error and keeps the session. hermes retries compression up to 3 times, then auto-resets the session, discarding history.
• A single input larger than the window cannot be compacted. No history to squeeze; the summarisation request would itself exceed the window. pi’s tracker states the design position directly: recovery is impossible, surface the error.

What generalises vs what is client-specific

Generalises (mechanism, not policy): - Prefix caching only survives compaction if the compacted request keeps the prompt front identical. Any client that rewrites the system prompt during compaction resets the cache to zero. - Tool results are the correct thing to drop because they are reconstructible: the agent can re-read a file, but cannot re-derive a lost user instruction or decision. - Selective retention (keep recent tool results, drop stale ones) beats both extremes. This held for pi; whether your client does it is what the compaction and regrowth tables show.

Client-specific (verify on yours): - What the compaction actually drops, whether the system prompt survives, how failure is handled, and when compaction triggers.

Requirements

python3, standard library only. Input: raw capture files from the 01-token-audit/ tap.