local-llm-measurement

Measurement tooling and findings for local LLM inference.

GitHub repo: local-llm-measurement
Last update: 2026/08/15

Description

local-llm-measurement is a set of bash/python harnesses for measuring what happens between an agent and a local inference server, measuring the network traffic. Three questions, one directory each: what enters the model per request (token audit), what persists between requests (KV cache behavior), what exits (reasoning overhead). Written against llama.cpp’s llama-server, but every tool takes a base URL and works on any OpenAI-compatible endpoint. The repository contains:

• Token audit: a logging TCP proxy captures the agent’s requests; an extractor pulls the assembled request; an itemizer splits it into system prompt, tool schemas, and context, token-counts each component via the server’s /tokenize, and ranks every tool by schema size.
• Reasoning-overhead harness: runs a fixed prompt set against the server under different thinking configurations (unrestricted, disabled, budget-capped), recording reasoning and answer tokens separately per response, plus decode time. An analyzer summarizes per condition and tier, with an optional hand-scored quality column.
• Scoring methodology included: a reference-key guide for writing your own domain prompt set and scoring answers 0–2 against ground truth — including why an LLM must not assign the scores.
• Findings from the original runs: 73% of every agent request was tool schemas for unused tools; framework background calls silently destroyed the conversation’s KV cache; reasoning was ~90% of decode with no measurable quality gain on most tasks; two of four documented thinking-disable mechanisms did nothing.

Running these tools requires no dependencies beyond the basics: bash, curl, jq, python3.

Technologies

• Python: raw TCP logging proxy, largest-JSON-object request extractor.
• Bash + curl + jq: test runner, per-component tokenization, portable-awk results analyzer.
• llama.cpp (/v1/chat/completions, /tokenize): reference target; any OpenAI-compatible server works.

Write-ups

Findings from the original runs, one article per measurement:

1. Token audit — what enters the model per request
2. Context lifecycle — what persists between requests
3. Reasoning overhead — what exits the model

Why this project exists

I built these because my local agent took over three minutes to answer a simple question, and nothing in any config file explained why. Measuring the wire did: most of the prompt was tool definitions the agent never used, and most of the decode was reasoning nobody reads. Same hardware, zero money spent, cold start down to ~12 seconds. The tools are generic because the problem is the same across all agents and harnesses: every local stack pays for tokens its documentation doesn’t mention, and the only way to actually know what yours pays is to measure it.