Token audit for local agent stacks — method and findings

Notes from an audit of my setup: Hermes agent + qwen3.6-35b-a3b via llama.cpp, on an i9 / 32GB RAM / 8GB NVIDIA laptop GPU. The method works on any OpenAI-compatible endpoint.

Principle

The agent layer assembles each request from: custom instructions, tool definitions, injected context, and conversation history. The full assembly is sent to the model on every request, before the user’s question. On local hardware, prefill cost scales with request size.

Config files and framework docs describe intended behavior. Network traffic shows actual behavior. Measure the traffic. In my case the difference was ~16,000 tokens.

Method

1. Intercept the traffic

Put a logging proxy between agent and inference server. Move the server to a different port (example: 8081). Run the proxy on the original port. The agent config stays unchanged.

# tap.py — logs client->server traffic, forwards both directions
import socket, threading, sys

LISTEN = int(sys.argv[1]) if len(sys.argv) > 1 else 8080
TARGET = int(sys.argv[2]) if len(sys.argv) > 2 else 8081
LOGFILE = sys.argv[3] if len(sys.argv) > 3 else "reqs.raw"

def pipe(src, dst, log=None):
    while True:
        try:
            data = src.recv(65536)
        except OSError:
            break
        if not data:
            break
        if log:
            log.write(data)
            log.flush()
        try:
            dst.sendall(data)
        except OSError:
            break

srv = socket.socket()
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
srv.bind(("127.0.0.1", LISTEN))
srv.listen(50)
print(f"tap listening on {LISTEN} -> {TARGET}, logging requests to {LOGFILE}")
while True:
    c, _ = srv.accept()
    u = socket.socket()
    u.connect(("127.0.0.1", TARGET))
    log = open(LOGFILE, "ab")
    threading.Thread(target=pipe, args=(c, u, log), daemon=True).start()
    threading.Thread(target=pipe, args=(u, c), daemon=True).start()

(Shown hardcoded for clarity; the repo version takes ports and logfile as arguments.)

Send one simple message through the agent (“hi” is enough). You want the assembled request, not a complex task.

If your stack routes requests through a gateway (LiteLLM, for example), it may have built-in logging/callback hooks that can capture full request bodies. Those can be used instead. Approaches that did not work, so you do not repeat them: llama.cpp /slots hides prompt content by default. Verbosity -lv 4 logs requests but not request bodies. socat -v reformats the payload and breaks JSON extraction. The raw Python proxy worked.

2. Extract the request

Pull the largest JSON object out of the capture: the largest object in an agent’s traffic is the main chat-completions request. Script: extract.py (mechanical brace-matching, ~30 lines).

3. Split the request into components and count tokens

# list message roles and sizes
jq -c '.messages[] | {role, len: (.content|tostring|length)}' /tmp/req.json
jq '.tools | length' /tmp/req.json

# extract system prompt and tool schemas
jq -r '.messages[0].content' /tmp/req.json > /tmp/system.txt
jq '.tools' /tmp/req.json > /tmp/tools.json

# count tokens with the server tokenizer
jq -Rs '{content: .}' < /tmp/system.txt | curl -s localhost:8081/tokenize -d @- | jq '.tokens | length'
jq -Rs '{content: .}' < /tmp/tools.json | curl -s localhost:8081/tokenize -d @- | jq '.tokens | length'

# rank tools by schema size
jq -r '.tools[] | "\(.function.name) \(.function|tostring|length)"' /tmp/req.json | sort -k2 -rn

Output: tokens per component (system prompt, tool schemas, context). Then decide which components are necessary.

Findings on my stack

Tool schemas were ~73% of the request. 16,182 tokens of tool definitions. 5,873 tokens of system prompt (instructions, identity, memories). The request contained 30 tools. The vault agent uses about 5. The largest single tool schema was 8.8KB, for a feature the agent never uses. I cut the enabled toolsets (the framework groups tools into toolsets) from 17 to 5 in config. The static prompt went from ~22k to ~12.5k tokens. Relevance for an MCP architecture: every MCP server adds its tool schemas to every request that can reach it. The cost applies to every request, whether the tools are used or not.

The framework made background calls to the main model. It auto-generated session titles with the same model. Each call blocked the request queue for ~30 seconds: a reasoning model, asked for a short title, generates reasoning tokens until the framework timeout cancels it, then the framework retries. Each call also truncated the KV cache of the conversation. The next real request then re-processed 15-20k tokens. Detection: server logs show a repeating pattern of small prompt, 1,000+ generated tokens, client-side cancel. Fix: enabled: false on the feature, or route auxiliary tasks to a separate small model.

The inference server defaulted to multi-user mode. llama.cpp default is 4 parallel slots. Each slot has its own KV cache. With one user, requests land on whichever slot is free, often one with no cached prefix. That slot re-processes the full context. Fix: -np 1. One slot means every request is compared against the same cache. A request that arrives while another runs waits in a queue. In my tests the queue was also faster: one request alone decoded at 33 t/s; three concurrent requests got 6-16 t/s each. The GPU divides its compute across concurrent requests, so parallelism gains nothing for a single user.

Ollama has equivalent settings with the same effects: OLLAMA_NUM_PARALLEL controls concurrent requests per model, OLLAMA_KEEP_ALIVE controls how long the model stays in memory after use. If the model unloads, the next request pays load time plus full re-prefill, because the cache is lost with the model. Verify both values on the deployment.

Batch size controlled prefill throughput. llama.cpp specific: with MoE expert weights in system RAM, prefill cost depends on how often the CPU-side tensors are read. Each ubatch reads all expert tensors once. I raised ubatch from 512 to 4096 (-b 4096 -ub 4096). Prefill went from ~300 t/s to ~1,190 t/s on the same hardware. On a machine without a dedicated GPU this tuning has larger effect, because all prefill runs on system RAM bandwidth.

Result on my machine: cold start went from 3+ minutes to ~12 seconds. Follow-up requests take a few seconds. Hardware unchanged.

Caching

A correctly configured server does prefix caching: it keeps the KV state of the processed prompt. The next request only processes tokens after the first point of difference from the cache. Within a session, a follow-up should cost hundreds of tokens of prefill, not tens of thousands. My logs show 250-900 new tokens per turn on top of a 20k+ cached conversation. Important to verify this directly on your deployment.

Three properties:

  1. The cache only protects appends. It matches from the first token and stops at the first difference. Any change early or mid-prompt invalidates everything after the change point: a timestamp near the top, reordered tool definitions, a document swapped by retrieval, an edited or cancelled message, a history-rewriting summarization pass. Consequence for prompt structure: static content first (identity, instructions, tool schemas), variable content last (retrieved documents, dates, the current question).
  2. The cache is per slot. Multiple slots fragment it. See -np above.
  3. Verify it in the logs. llama.cpp logs show selected slot by LCP similarity, f_keep = 0.99: 99% of the cache survived, only the tail was computed. f_keep = 0.17 after a background call: the conversation cache was destroyed. General check for any serving layer: compare tokens processed on turn two against turn one of a session. If they are similar, caching is not working, and the server is re-processing the full context on every turn.

Pre-deployment check

A useful check before putting a local agent stack into real use: capture one request per client (Open WebUI, Claude Code, each agent) and record five values. Total tokens per request. Tokens per component: instructions, tool schemas per MCP server, injected context. Prefill time on a cold start. Prefill time on a follow-up, to verify caching works. Background calls observed in the logs during a 30-minute session. This turns the token cost per request into a measured number. Keeping the recorded values also gives a reference point: when a new MCP server is added later, measure again and compare, because each server increases the size of every request.