Reasoning overhead in local thinking models — measurement and findings

Subject: how much of decode time a thinking model spends on reasoning, whether that reasoning buys answer quality, and which control mechanisms actually work. Test system: qwen3.6-35b-a3b (Q4_K_M) via llama.cpp on an i9 / 32GB RAM / 8GB NVIDIA laptop GPU. Method and harness are reusable against any OpenAI-compatible endpoint.

Why this matters locally

A reasoning model generates thinking tokens before the answer. With API this means an increase in cost. On local hardware what is impacted is decode time: at ~33 tokens/s, every 1,000 thinking tokens is 30 seconds before the answer starts. The thinking is also invisible in normal chat UIs, so the cost does not look like anything, and the model just seems slow.

Method

8 fixed prompts in 3 tiers: trivial (one-sentence factual/code questions), mechanical (reformat, extract, transform), hard (a math derivation, a debugging scenario, a systems design task). Each prompt sent as a single message with no system prompt and no history, via a bash harness against /v1/chat/completions. The harness records completion_tokens, reasoning tokens and answer tokens separately (reasoning_content vs content, tokenized via /tokenize), and decode time.

Three conditions, same server flags otherwise, fresh server per condition:

Answers scored 0-2 by hand against a written reference key (0 = unusable, 1 = usable with defects, 2 = correct and complete), with adversarial claim-checking of factual statements. Scoring rule: reasoning traces leaked into the answer content = 0, regardless of what follows them.

The harness (bash + curl + jq, ~100 lines, base-url as argument) is published in local-llm-measurement, directory 03-reasoning-overhead/, together with the scoring guide.

Finding 1: reasoning is ~90% of decode under default settings

Condition A, measured: reasoning share of completion tokens was 95% / 94% / 89% for tiers 1/2/3. A one-sentence question about a 4-line C function generated ~1,200 thinking tokens for a ~30-token answer. Total for the 8 prompts: 20,098 tokens, 9.5 minutes of decode. The same 8 prompts with thinking disabled: 2,923 tokens, 1.3 minutes. Factor roughly 7 in both tokens and wall-clock.

Finding 2: thinking bought no measurable quality on this task mix

Quality totals (of 16 per condition): A = 13, B = 12, C = 9.

Per tier, B (no thinking) scored better than A on tier 1 (2.00 vs 1.50) and equal on tier 2 (1.67) — on simple and mechanical tasks, removing thinking improved or matched quality while cutting tokens by a factor of 15-25. B lost tier 3 only through one prompt (p8, systems design), where the no-thinking answer contained visible self-argument and false claims. Interpretation, marked as such: denied a think block, the model did its deliberation, including its confusion, in the answer stream. On this evidence, thinking’s main function for this model is not better conclusions but a private place to be wrong; open-ended design tasks are where that matters. Notable single result on the other side: the hardest math prompt (gyro bias error growth derivation, two regimes) was answered fully correctly with zero thinking tokens.

Caveats stated plainly: n = 8 prompts, one scorer, temperature 1.0 (single sample per cell), one finetuned model. Directional, not definitive. Portability: the method transfers to any OpenAI-compatible endpoint unchanged (the harness takes a base-url argument); the ~90% overhead order of magnitude is consistent with the Qwen3 family generally; the quality tie and especially the toggle verdicts below may be finetune-specific — community finetunes routinely alter template behavior — and the base model was not tested. Rerunning the same harness against the base model, or against your own deployment’s model, is one evening and answers the portability question for your stack directly.

Finding 3: most thinking controls do not work on this model — and the one that half-works corrupts output

Four mechanisms tested:

  1. /no_think in the prompt (Qwen soft toggle): ignored. Reasoning unchanged, sometimes longer.
  2. --reasoning-budget 0 (llama.cpp hard off): ignored. Full reasoning anyway.
  3. --chat-template-kwargs '{"enable_thinking":false}' (template-level toggle): works. Reasoning exactly 0 on all prompts.
  4. --reasoning-budget 800 (cap): caps the accounting, not the behavior. Reasoning tokens report exactly 799, but on heavy prompts the model continued deliberating in the content stream after the forced close, then emitted its own </think> and the final answer. Result: raw thinking traces inside the answer the user sees. 3 of 8 prompts affected.

Consequence of (4): condition C’s answer_tokens are contaminated (leaked thinking counted as answer), and its deliverables on heavy prompts are unusable as-is. The cap flag should be treated as broken on this model family until verified otherwise. This also means an asymmetry worth knowing: the mechanism can truncate a think block but not prevent one from opening or continuing.

Policy adopted

--chat-template-kwargs '{"enable_thinking":false}' as the permanent default server flag. No budget cap, ever, on this model. Thinking re-enabled deliberately (separate server invocation) only for tasks where deliberation is explicitly wanted — with the p8 result as a warning that its absence hurts most on open-ended design tasks, and its presence costs ~7x on everything.

Verified after adoption on real agent (Hermes) tasks: thinking dropped from 30-90s of deliberation per answer to 0-4s residual thinking per turn; total turn time 10-20s, the rest being file reads and prefill of tool results. Vault operation quality unchanged: multi-file retrieval and synthesis tasks completed correctly.

General checks for any local thinking-model deployment

  1. Measure the reasoning share once: one hard prompt, compare reasoning_content tokens to answer tokens. Above ~70%, the thinking toggle greatly affects performance as well, not just quality output.
  2. Verify the toggle you use actually works — request one response and check reasoning tokens are 0. Two of the four documented mechanisms did nothing on this model.
  3. If using a reasoning budget cap, inspect the raw content of a heavy response for leaked deliberation or stray </think> markers before trusting it in any pipeline.
  4. Score quality with and without thinking on your own task mix before paying the overhead by default. The assumption that thinking helps is exactly that, just an assumption. Here it meant no significant drop in output quality for most tasks, but a 7x increase in cost.