Tuning a Local Model: What Actually Moves the Needle

Everything we learned making glm-4.7-flash (30B-A3B MoE, Q4) behave as a coding
agent on one dedicated Mac. Measured over a working session that built a Rails
application, not a benchmark harness.

Scope, stated plainly. No weights were trained. "Tuning" here means context
budget, output budget, compaction, dispatch topology, prompt shape and
verification — the levers that turned out to matter. Gradient fine-tuning is the
last rung of the ladder in §1, and we never needed to reach it. If you came for
LoRA recipes, §11 says when to bother and why it is rarely the answer.

1. The ladder — climb only as far as you need

For adapting a local model to a domain, in ascending cost and descending
frequency-of-being-the-right-answer:

System prompt and domain vocabulary. Force a procedure: search before
answering, decompose compound questions, cite everything, admit ignorance.
Without an explicit "never answer without searching", the model falls back on
training data — a liability in any regulated domain.
Synonym and acronym expansion before full-text search. Cheap; removes a
whole class of misses.
Better retrieval. Hybrid vector + full-text, rank fusion, diversity
reranking. Highest return per hour of work, and it carries over unchanged to
any agentic version.
Structured output for anything downstream code consumes.
Actual fine-tuning — only for style/format consistency or classification at
volume. Never for facts. Facts belong in retrieval.

Most "the model is bad at our domain" complaints are solved at rung 1 or 3. We
spent the entire session between rungs 1 and 3 and never ran out of headroom.

2. Context is the primary knob, and memory is not the limit

Generation rate against prompt depth, same prompts, verified uncontended:

depth	generation
1,360	60.5 tok/s
2,305	55.9 tok/s
9,783	34.5 tok/s
27,196	17.9 tok/s
~65,000	~8 tok/s

Roughly halving per doubling. The KV cache is re-read for every generated
token, so context is not a one-time prefill cost — it is a per-token tax on
everything generated afterwards.

Memory, by contrast, is abundant: KV costs 51.6 MiB per 1K tokens per slot, so
a 48 GB machine could hold ~329K tokens on one slot. You will hit intolerable
latency long before you hit memory. Do not size context against RAM.

The practical consequence is the whole justification for an orchestrator/worker
split. Observed live, same model, same machine:

agent	depth	generation
worker (fresh)	~11K	33.0 tok/s
orchestrator	~21K	21.6 tok/s

A shallow worker is ~1.5× faster than the session that dispatched it, and ~4×
faster than one that has grown to 40K.

3. Does it still comprehend at depth? Yes, further than expected

Everyone quotes the latency curve; almost nobody measures whether the model can
still use the context. So we tested it: plant one unguessable code at a
controlled depth and a controlled position, ask for it back, score exact
substring match. No rubric, no LLM judge.

Position is the second axis on purpose — a needle near the end of the prompt is
recoverable at any depth, which is how a naive test "proves" a model handles 64K.

depth	pos 0.1	pos 0.5	pos 0.9
5,213	✅	✅	✅
15,543	✅	✅	✅
31,079	✅	✅	✅
46,773	✅	✅*	✅

Retrieval is intact to at least 47K at every position. No lost-in-the-middle
effect up to the server ceiling — the failure mode long-context models usually
exhibit, and this one does not.

Two method notes that decide whether such a test means anything:

A different random code per trial, or a prefix-cache hit from an earlier
trial leaks the answer.
Refuse depths near the ceiling. A prompt that overflows gets its needle
silently truncated away, and you score a model limitation that is really your
own arithmetic.

* The one apparent miss was our instrument, not the model — see §10.

So: do not lower limit.context on quality grounds. Keep contexts shallow for
latency, and for the reason in §4.

4. What actually breaks at depth: the reasoning trace

The 47K/midpoint cell returned nothing with thinking enabled, at num_predict
512 and 2048 — done_reason=length both times, the entire budget consumed by
the trace. The same prompt with think: false returned the code in 10 tokens.

This is not a retrieval failure. It is the trace running away, and it has a
production twin: a real agent turn generated exactly 8,192 tokens — its whole
limit.output — at depth 28,352, over 644 seconds, and returned nothing
usable. Three such runaways, all at depth ≥28K; none below ~20K.

Consequences:

Keep limit.output modest (4096–8192). It cannot prevent a runaway but it
bounds the waste — halving the cap halved the incident cost from 644 s to 286 s.
But not too modest. The budget must fit trace and any file the turn must
write. A 4,969-token file failed twice under a 4,096 cap. Truncation and runaway
look identical; the fix is opposite. See the setup article, §5.
think: false is a real lever for work needing no deliberation — retrieval,
a bounded mechanical edit. 10 tokens versus 221 for the same answer.

5. Compaction: size the trigger against the prompt floor

The single most expensive pathology we hit, and it is pure arithmetic.

The compaction trigger is limit.context − compaction.reserved. The prompt
floor — everything re-sent verbatim every turn — was ~28K on a mature repo:

component	cost
rules files (`AGENTS.md`, `CLAUDE.md`, project rules)	~22K tokens
built-in tool schemas	~5.8K tokens
skill descriptions (~267 tok each, always on)	13.4K on a 57-skill repo
the user's actual message	negligible

With limit.context: 40000 and the default reserved: 10000, the trigger sits at
30,000 against a 28,000 floor — ~1,900 tokens of working room. What that
produced:

13:49:41  worker      28,122 in     <- a "fresh" worker STARTS here
13:51:03  worker      31,732 in     <- crosses the trigger in 4 turns
13:51:19  compaction   3,475 in  ->  8,192 out   (the entire output cap)
13:57:27  compaction  13,436 in  ->  8,192 out
14:06:22  compaction  11,729 in  ->  8,192 out
14:13:04  worker      28,352 in  ->  8,192 out   (644 s for ONE turn)

Compaction cannot rescue this. It compacts the conversation, but the
conversation is ~3K of that 31K — the rest is the floor, which is not compactible.
Each pass frees almost nothing, spends a full output budget, lands back at the
floor, and re-fires. It never converges. Forty-five minutes of GPU produced zero
disk changes on a task that was a one-line edit.

Two tells

A compaction whose output is exactly limit.output. A summary that runs to
the cap is not summarising.
A worker that compacts at all. Workers exist to start fresh and stay small.
If one compacts, its floor is at the trigger and the whole split is buying
nothing.

The rule

trigger = limit.context − reserved  ≥  floor + 10K
(limit.context − reserved) + limit.output  ≤  server context per slot

Measure the floor — it is the first assistant turn's tokens.input in the session
database, not a guess. Then note that cutting the floor beats raising the
ceiling: trimming rules files and pruning unused skills moved a repo from a 28K
floor to 7.3K, which is a 2× throughput gain with no server change.

Config changes are read at session start. Fixing the numbers does not rescue a
running session; kill it.

6. Prompt shape dominates every config value

The largest quality factor we measured is not a setting.

An empty argument block stops the model dead. A command whose preamble said
"You MUST consider the user input before proceeding (if not empty)" was invoked
with no arguments. The model read that against a blank block, concluded there was
no instruction, and asked what to do — without reading the rest of the (long)
body. A frontier model pushes through; a 30B takes the path of least resistance.
A vague target produces invented work. One run fabricated 49 non-existent
specs from an ambiguous "continue on next spec". Name the file, the check, and
the expected result, every time.
Convention markers imported from other harnesses mislead. Spec Kit's [P]
means "no file conflict, safe to parallelise". It is right about safety — three
workers edited one model concurrently and all three edits survived — and wrong
about benefit here (§7). The marker is a claim about a different machine.
Name the tool, not the intention. "Dispatch a worker" produced nothing across
~40 turns; "call the task tool with subagent_type: worker" produced a dispatch
on the next turn. The documented failure is substituting a plausible account of
an action for the action.

7. Dispatch topology: serial beats parallel here

configuration	throughput
one worker	33.0 tok/s
two concurrent, aggregate	18.4 tok/s
six dispatched at once	2.4 tok/s

Aggregate throughput falls with concurrency. Memory bandwidth is the
bottleneck; concurrent decodes compete rather than overlap. With two slots,
dispatching six means two decode and four queue, and the orchestrator's own turns
get starved — one took 251 s at 6.2 tok/s.

Dispatch serially regardless of what the task list suggests. More slots buy
concurrency, never throughput, on bandwidth-bound MoE inference.

8. What it gets right, and how it fails

Reliably good at: a named defect with a file, a line, real error text, and a
command whose current result you know. It fixed four of those in a row and
verified them.

Failure modes, ranked by what each cost us:

Fabricating the artifact a check would produce. Unable to run
db:migrate, workers hand-wrote db/schema.rb (a file whose own header says it
is generated), created tables via psql, and inserted rows into
schema_migrations so the schema reported itself migrated. This is worse than
failure: it destroys the next agent's ability to detect the gap. Recovery needed
a full database drop.
Regression under unverifiable editing. A correct model class was "refined"
across ~40 dispatches — none able to run a check — into one that would not
class-load: an inverted association, a before_save calling SQL from Ruby, a
scope colliding with a column. Unverifiable editing does not merely fail to
improve code; it degrades it. There is no feedback signal to stop the drift.
Context amnesia across fresh workers. A new worker undid a correct fix from
two tasks earlier, because the reason lived only in a dead context. Fresh
means uninformed as well as uncontaminated — write negative findings into the
task list, never into a prompt.
Tests that cannot fail. Three attempts produced a stub returning "", an
iteration, then a tautology. It converges on passing, not on discriminating.
Do not ask a 30B to prove its own work.
Invented environment errors. "Cannot run the tests due to tool permission
restrictions" while bash * allow was in force. It substitutes a plausible
reason for the verification rather than doing it — and it reads like your config
bug. Check the agent's actual permissions before believing it; sometimes the
refusal is real and your instruction was impossible.
Empty deliverables. A 0-byte spec file counted as the test harness being
delivered.

Every one of these is invisible without an executable check. Which is the point.

9. Evaluation: what separated models, and what did not

We built three evals of increasing difficulty. Scoring is objective throughout —
pass counts and tree hashes, never a rubric and never an LLM judge, because local
models converge on passing rather than on discriminating.

eval	design	result
single-bug agentic task	fix a bug, then report a value only obtainable if the fix landed	all three models 6/6, none cheated
multi-bug module	three interacting bugs, one encoding a non-obvious requirement; pytest-scored, test file diffed for tampering	all three 6/6, none edited the tests
impossible tasks	a complete task list and a nonexistent file; scored by tree hash — any write is fabrication	GLM: no fabrication, twice each

Two evals of increasing difficulty both failed to find a capability gap. The
honest conclusion is not "build harder tests until one model loses" — it is that
within the class of work a local 30B should be given — bounded, verifiable,
named — capability is not the differentiator. Speed is.

One secondary column is worth a warning. Alongside the tree hash we logged a
keyword grep for whether the model said why it stopped. Across two identical
runs it flipped in both scenarios on rewording alone, while the tree hash was
identical all four times. Objective checks reproduce; text matching does not.

10. Measurement discipline, or you will tune against noise

Nine defects in our own instrumentation surfaced during this work. Every one was
caught by a result that was impossible, not merely surprising.

symptom	actual cause
0/12 recall, including 5K with the needle 10% in	`num_predict: 48` against a reasoning model — the whole budget went to `thinking`, `response` came back empty
generation slower at 1.3K than a live session at 29K	a competing session on the second slot; ~4× understated
146 tok/s on a model whose best is 29.3	a request straddling two metric scrapes — tokens in one window, decode time in the other
43.7 tok/s at depth 3,794 slipping a flat 60 tok/s guard	plausibility bounds must be depth-aware; against the fitted curve it was 9.6×
agent credited with a file change	our own edit, in a watched directory
`NoneType` on a benchmark	`/v1/completions` not served; only `/v1/chat/completions`
awk emitting all zeros	regex constants cannot be passed as awk function arguments — `/re/` evaluates as `$0 ~ /re/`
a check reporting failure on a healthy app	a deprecation banner on stdout, ahead of the result; and `migration_context` not existing in Rails 8.1

The transferable rules:

Divide by the server's eval time, not wall clock.
Confirm nothing else is decoding before recording a number.
Bound plausibility against the fitted curve, not a flat constant.
Establish who changed a file before attributing it — git status says what,
git log says who, and mtime is treacherous.
If a check's output is confusing, suspect the check before the code.
Test the instrument on one cheap case before spending 25 minutes on a grid.

11. When to actually fine-tune

Rarely, and never for facts.

Fine-tuning is worth it for style and format consistency, or classification
at volume where a prompt would be re-sent millions of times. It is the wrong tool
for domain knowledge: knowledge belongs in retrieval, where it can be corrected
without a training run and cited in an answer.

If you do: most agent frameworks do not wrap the fine-tuning endpoints. Train via
the provider API or Ollama + LoRA, then point the client at the resulting model id
with assume_model_exists: true and an explicit provider, since the id will not
be in any bundled registry.

Before that, note what the ladder in §1 actually bought us. We never adjusted a
weight. The changes that mattered were: an output budget that fits the reasoning
trace, a compaction trigger sized against the prompt floor, serial dispatch, a
7.3K floor instead of 28K, and prompts that name the tool instead of the intention.

The model was not the bottleneck. The harness around it was.