Serving DeepSeek V4 Flash to a Local Coding Agent

When to use it: tuning a locally-served LLM for agentic coding, debugging a
context window that compacts instantly, sizing a KV cache, or deciding whether a
disappointing local agent is limited by hardware or by configuration.

Abstract

We document the configuration of DeepSeek-V4-Flash served via vLLM across two
NVIDIA DGX Spark (GB10) nodes, driven by the opencode CLI agent. The system
initially exhibited three user-visible failures: immediate context compaction on
every session, apparent non-termination during generation, and an unusable 8,192-token
context. We show that all three were configuration defects rather than hardware
limits, and that the corrective work required no additional memory, no model
change, and no hardware.

The most consequential finding is counterintuitive: an explicit --kv-cache-memory
allocation, intended to give precise control, silently disabled vLLM's memory
profiler and produced a KV cache less than half the size the profiler derived
unaided (18,272 vs 47,930 tokens). Removing the flag — deleting configuration, not
adding it — was the single largest capacity gain in the study.

We further show that the client-side token budget is governed by an arithmetic
constraint frequently violated in practice, prompt + max_tokens ≤ max_model_len,
and that violating it by a single token reproduced the original symptom exactly.

Final configuration serves 24,576 tokens of context with 1.95x KV cache headroom,
against an initial working budget of 192 usable tokens.

---

0. Relation to prior work

This paper is a companion to Tuning a Local Model: What Actually Moves the Needle,
which covers the same problem class on different hardware — a 30B-A3B MoE at Q4 on a
single Mac, driven through a session that built a Rails application. That document is
the authority for the general levers: the adaptation ladder (§1), context as a
per-token latency tax (§2), long-context retrieval behaviour (§3), reasoning-trace
runaway (§4), the compaction-trigger arithmetic (§5), prompt shape (§6), dispatch
topology (§7), and measurement discipline (§10).

We do not restate those results. This paper contributes what a two-node
tensor-parallel deployment exposes that a single-node one cannot:

• KV cache sizing as a startup precondition rather than a latency tax (§2.1, §3.1)
• Engine-side memory profiling and the cost of overriding it (§3.1)
• Rank asymmetry — configuration validated against the weakest node (§3.3)
• Serialisation as a defect that conceals its own diagnosis (§3.2)
• A transport-layer field-name mismatch that mimics non-termination (§3.4)
• Configuration propagation through a reconciled state registry (§3.6)

Where our measurements contradict the companion document, §5.2 says so explicitly.

---

1. System under test

Component	Value
Hardware	2 × NVIDIA DGX Spark (GB10), unified memory, ~119 GB/node
Interconnect	ConnectX-7, RDMA, dedicated cluster interface
Orchestration	Ray, head + worker, tensor_parallel_size=2
Serving engine	vLLM, vllm/vllm-openai:nightly-aarch64
Model	deepseek-ai/DeepSeek-V4-Flash, FP8 weights, MLA attention, MoE
KV cache dtype	fp8
Client	opencode 1.18.15, @ai-sdk/openai-compatible provider
Ingress	Caddy reverse proxy, path-routed per model

GB10 is a unified-memory architecture: nvidia-smi reports [N/A] for total and
used device memory, so all memory accounting in this study derives from vLLM's own
boot-time reporting rather than from device queries.

---

2. Background: what the KV cache is and why context costs memory

A transformer produces each token by attending over every preceding token.
Recomputing that attention from scratch at every step is quadratic in sequence
length. The standard mitigation caches the Key and Value projections of each token
after it is first processed; generating token n then requires computing one new
K/V pair and attending over the cached remainder.

This cache is resident in accelerator memory. Its size is linear in tokens and is
incurred per concurrent sequence.

The companion document's §2 develops the latency consequence of this — the cache is
re-read for every generated token, so depth taxes everything generated afterwards, at
roughly half the generation rate per doubling. We take that as given and develop the
orthogonal consequence, which dominates on a multi-node deployment:

Context length is not a preference. It is a memory allocation.

Memory partitions three ways:

total = model weights + activations + KV cache

Weights are fixed by model and quantization; activations are approximately fixed by
batch shape. The KV cache receives the remainder, and that remainder determines
the maximum serviceable context.

2.1 Measured cost per token

We measured KV cache capacity directly across three explicit allocations, with all
other parameters held constant (fp8 KV dtype, TP=2):

--kv-cache-memory	GPU KV cache size
4 GiB (4294967296)	10,434 tokens
7 GiB (7516192768)	18,272 tokens
14 GiB (15032385536)	36,555 tokens

The relationship is linear at ≈2,610 tokens per GiB, or roughly 400 KiB per
token — a large figure reflecting the model's layer count and MoE width.

vLLM enforces the obvious constraint: KV cache capacity must be ≥ max_model_len,
or the engine refuses to start. It reports the resulting ratio at boot:

GPU KV cache size: 47,930 tokens, Maximum concurrency for 24,576 tokens per request: 1.95x

That ratio is not decoration. It is the number of full-length conversations the
cache can hold simultaneously, and it governs whether prefix caching has anywhere
to live.

---

3. Findings

3.1 Explicit KV allocation disables the memory profiler

vLLM can size the KV cache by profiling actual free memory at startup, or accept an
explicit byte count via --kv-cache-memory. These are mutually exclusive, and the
engine states so plainly at boot:

reserved 14.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config
and skipped memory profiling. This does not respect the gpu_memory_utilization config.

Two consequences follow, both of which we observed:

1. A gpuMemoryUtilization setting coexisting with --kv-cache-memory is inert.
   Ours was 0.75 and had no effect for the duration of the misconfiguration.
2. Sizing becomes a manual estimate against unmeasured memory. Our estimates were
   poor, and produced two out-of-memory failures.

Removing the flag entirely and permitting the profiler to operate:

Method	KV cache	Headroom @ 16,384
Explicit 7 GiB	18,272 tokens	1.12x
Profiled	33,932 tokens	2.07x

Deleting one configuration line nearly doubled capacity. The generalisable point
is that an engine measuring its own runtime environment will routinely outperform an
operator estimating it in advance, and that "explicit control" can be a net loss when
it silently disables measurement.

3.1.1 The cost: nondeterminism

Profiling is dynamic. Across four boots of identical configuration we observed KV
cache sizes of 26,890 / 33,932 / 34,887 / 47,930 tokens — a 1.78x spread. The
variance originates in a second tenant's processes (§3.3) occupying worker memory.

This has a direct operational implication: max_model_len must be chosen against
the worst observed profile, not the best. At the time of writing, the highest
observation (47,930) would have accommodated 32,768 tokens of context, but the
lowest (26,890) would not, and a service that intermittently fails to start is
worse than one with modestly less context. We selected 24,576 accordingly.

3.2 Serialisation via maxNumSeqs

The deployment specified maxNumSeqs: 1. The endpoint therefore served exactly one
request at a time; all others queued:

Running: 1 reqs, Waiting: 1 reqs

This is severe for agentic workloads, which issue concurrent tool calls, and it has
a compounding diagnostic cost: it prevented instrumentation. Attempts to
characterise the generation behaviour of §3.4 returned no data at all, because every
probe queued behind the very request under investigation. The defect concealed
itself.

Raising the limit to 4 was verified against three simultaneous requests:

req1 http=200 time=10.54s
req2 http=200 time=10.63s
req3 http=200 time=10.62s
Running: 3 reqs, Waiting: 0 reqs

3.3 Node asymmetry and multi-tenancy

The two nodes did not present equal memory. vLLM reports free memory per rank at
startup:

Node	Initial free memory
Head (192.168.100.11)	108.76 – 108.86 GiB
Worker (192.168.100.10)	88.14 – 91.45 GiB

A consistent deficit of roughly 20 GiB. The cause was a second user's processes on
the worker — approximately 20 opencode instances totalling 17.56 GB RSS, with swap
at 13–14 of 15 GB.

Consequently a configuration accepted by the head still failed:

[rank1]: memory allocation failed with OOM on device 0 while trying to allocate 15032385536 bytes
ERROR ... ray_executor_v2.py:514] RayWorkerProc rank=[1] died unexpectedly, shutting down executor.

Size for the weakest node. Head-node headroom is not a usable signal in a
tensor-parallel deployment; the minimum across ranks governs.

We note the processes were owned by another user and attached to live TTYs.
Reclaiming that memory was neither technically available to us (kill returned
operation not permitted) nor appropriate. This is a coordination problem, not an
engineering one, and it remains the dominant source of capacity variance.

3.3.1 Saturation is not failure

During the study the worker briefly reached a load average of 278 while
sustaining repeated 182 GB checkpoint loads. SSH banner exchange timed out, and the
cluster launcher — correctly — abandoned the attempt and removed its partial
resources. The node was never down; it was busy. Load decayed to 16 within minutes
of the last load cycle, and to 0.05 on the head shortly after.

Diagnosis should distinguish unreachable from saturated; the remedy differs
entirely.

3.4 Reasoning separation and a field-name mismatch

The deployment configured --tool-call-parser deepseek_v4 but no reasoning parser,
while a sibling model in the same registry correctly specified one. Without it,
chain-of-thought was emitted as ordinary content:

content: 'We are asked: "Reply with exactly: OK". So the response should be exactly "OK".'

Beyond being wrong output, this consumes context on every turn and corrupts tool-call
parsing. Adding --reasoning-parser deepseek_v4 produced clean separation —
content: '391' for an arithmetic prompt, with reasoning accounted separately as
reasoning_tokens: 26.

The client, however, still displayed indefinite "thinking" and almost no output.
Inspection of the raw SSE stream identified the cause:

delta keys seen: {'role': 1, 'content': 2, 'reasoning': 65}
finish_reason: stop

The model terminated correctly. vLLM's deepseek_v4 parser emits reasoning under the
key reasoning, whereas the OpenAI-compatible convention the client implements
expects reasoning_content. Sixty-five chunks were therefore invisible to the
client and two characters were not.

The apparent non-termination was a rendering artifact of a field-name mismatch.
We resolved it client-side by declaring the model non-reasoning, retaining the
server-side parser for its content-cleaning effect.

3.5 A one-token failure, and independent confirmation of a known rule

The companion document derives the governing constraint in its §5 and states it as:

trigger = limit.context − reserved  ≥  floor + 10K
(limit.context − reserved) + limit.output  ≤  server context per slot

We reproduce it here only as a validation case, because we violated the second line
and recovered the predicted symptom exactly.

Our measured prompt floor is 8,193 tokens — system instructions plus tool schemas,
obtained not by estimation but from the server's own rejection message. The initial
client configuration declared context: 8192 with reserved: 8000, giving 192
usable tokens against that floor. The client compacted on turn one of every session,
and the observed "compaction at 0 tokens of context" was the client correctly
predicting an inadmissible request and attempting to shrink it.

During remediation we raised limit.output from 4,096 to 8,192, reasoning that a
larger allowance permits larger single-turn edits. Against a 16,384-token model this
leaves exactly 8,192 tokens for the prompt:

This model's maximum context length is 16384 tokens. However, you requested 8192
output tokens and your prompt contains at least 8193 input tokens, for a total of at
least 16385 tokens. (parameter=input_tokens, value=8193)

A one-token overflow reproduced the original symptom exactly. The failure was
invisible to small synthetic probes (max_tokens: 20–400) and manifested only under
the real workload — a testing gap worth generalising: validate against the actual
prompt profile, not a convenient one.

3.6 A configuration-propagation trap

The management tool reads a reconciled state registry, not the file an operator
edits:

SOURCE_REGISTRY = ~/.config/vllm/models.json      # edited by hand
REGISTRY        = ~/.local/state/vllm/registry.json  # read by start/recreate

start and recreate do not synchronise these; only update does, and update
refuses while managed containers exist. Editing the source file and restarting
therefore silently replays the previous configuration. Two full boot cycles were
consumed before the divergence was detected by comparing the two files directly.

The correct sequence is rm → update → verify registry → start, and the
verification step is not optional.

A related hazard: the launcher arms a cleanup trap (trap cleanup_armed_start EXIT)
that tears down the deployment if the process exits before the trap is disarmed. A
backgrounded launch whose parent session terminated triggered exactly this, producing
a clean Exited (0) container and an upstream 502 at the proxy. Running the launcher
under tmux, so it owns its session, eliminated the failure mode.

---

4. Results

Parameter	Initial	Final
max_model_len	8,192	24,576
KV cache	10,434 tok (hand-sized)	47,930 tok (profiled)
Concurrency headroom	1.12x	1.95x
maxNumSeqs	1	4
maxNumBatchedTokens	8,192	8,192
Reasoning parser	absent	deepseek_v4
CUDA graphs	breakable (auto)	regular
Client usable input	192 tokens	20,480 tokens
First-message context load	compaction	44%

Observed serving behaviour after remediation: prefill to 1,086 tok/s, decode
24.7 tok/s, prefix cache hit rate rising 0% → 24.1% within a session, zero
rejections.

---

5. Discussion

5.1 What a server-side deployment adds to the priority ordering

The companion document establishes that harness configuration, not model capability,
dominates outcomes. We concur without qualification. Two factors specific to a
batched multi-node server deserve adding to its ordering, both absent on a
single-slot deployment:

• Prefix cache headroom. Agents resend a near-identical prefix each turn; cache
  headroom converts that pathological pattern into an affordable one. Measured
  0% → 24.1% hit rate within a single session. This is the reason we preferred
  24,576 tokens at 1.95x headroom over 32,768 at approximately 1.0x — the larger
  window would have left nothing to cache with.
• Concurrency admission. maxNumSeqs is a hard gate before any throughput
  consideration applies. At 1, the endpoint is not slow; it is unavailable to the
  second caller.

5.2 Where our measurements diverge from the companion document

The companion document's §2 concludes that memory is not the binding constraint —
"you will hit intolerable latency long before you hit memory" — at a measured cost
of 51.6 MiB per 1K tokens.

Our figure is ~400 MiB per 1K tokens, roughly 8× higher, and memory was the
binding constraint throughout. The engine refuses to start when KV capacity is below
max_model_len, and we took two out-of-memory kills before reaching a serviceable
configuration.

We read this as a difference in scope rather than a contradiction. The claim holds
for a quantised model on single-node unified memory, where the tax is paid in
latency. It does not hold for an FP8 MLA model under tensor parallelism, where KV
capacity is a startup precondition set by the weakest rank (§3.3). Both documents
should be read as scoped to their architecture.

A second divergence is weaker and we state it as such. The companion §7 recommends
serial dispatch "regardless of what the task list suggests", on measurements showing
aggregate throughput falling with concurrency. Our maxNumSeqs: 1 was actively
harmful, and three concurrent requests each returned in ~10.6 s. We did not measure
a serial baseline on identical prompts, so we cannot claim a refutation — only that
the recommendation appears to be scoped to bandwidth-bound single-slot inference
rather than to continuous-batching servers.

5.3 Remediation as a source of defects

Three of the defects in this study were introduced or prolonged by remediation
attempts: the explicit KV sizing (§3.1), the raised output limit (§3.5), and the
backgrounded launcher (§3.6). Changes made confidently and without measurement were,
in aggregate, more costly than the original misconfiguration.

This is the operational counterpart to the companion document's §10. Where that
section treats instrumentation defects, we observe the same failure applied to
corrective changes: an unmeasured fix is an unvalidated hypothesis, and deploying
several at once forecloses attribution.

---

6. Limitations and remaining work

Decode throughput is 3–4x below the published reference. We measure 24.7 tok/s;
a public recipe for identical hardware reports 74–96 tok/s. Three unadopted
differences are candidates, none verified by us:

• MTP-5 speculative decoding, absent from our deployment entirely. The likeliest
  single contributor.
• A hardware-specific image (ghcr.io/anemll/dspark-vllm-gx10:0.1.1) carrying
  GB10-tuned kernels, versus our stock vllm-openai:nightly-aarch64.
• An unpinned checkpoint, versus a pinned revision with a checkpoint-supplied
  encoder.

Context remains an order of magnitude below the reference. The same public
recipe serves 1,048,576 tokens on this hardware using nvfp4_ds_mla — a 4-bit MLA
KV cache — against our fp8. Halving bytes per token doubles context for identical
memory; this is the highest-value unadopted change and it is gated on the image, not
on hardware.

We did not verify whether the current image supports nvfp4_ds_mla; a single
inconclusive probe was run and should not be treated as evidence either way.

Multi-tenancy remains unresolved. The worker's memory contention and exhausted
swap are the proximate cause of KV cache nondeterminism and constrain max_model_len
below what the hardware would otherwise permit.

---

7. Conclusion

A local coding agent that appeared unusable was, in every observed particular, a
correctly-functioning model behind incorrect arithmetic. The corrective work consisted
of deleting one flag, correcting two integers, adding one parser, and raising a
concurrency limit from 1 to 4.

The specific contribution of a multi-node deployment is that the engine measures its
own environment better than the operator can. The profiler outperformed our estimate
by a factor of two, and every corrective signal in this study — the skipped-profiling
warning, the OOM rank identifier, the token counts in the rejection message, the
streaming delta keys — was already present in the engine's own output before we
looked for it.

Read what the system reports before deciding what it needs.

For the general practice of tuning a local model — prompt shape, dispatch, evaluation
design, and instrumentation — see Tuning a Local Model: What Actually Moves the Needle.

---

References

• Tuning a Local Model: What Actually Moves the Needle — companion study on
  single-node quantised inference. Authority for the adaptation ladder, compaction
  arithmetic, prompt shape, dispatch topology, evaluation design, and measurement
  discipline. Referenced throughout rather than restated.
• MiaAI-Lab, DeepSeek-v4-Flash-DSpark-2x-DGX-Spark, PR #14 — two-node 1M-token
  serving profile, NVFP4 MLA KV cache, MTP-5 speculation, CUDA-graph benchmark.
  https://github.com/MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark/pull/14
• vLLM, kv_cache_utils.py / gpu_worker.py — boot-time KV sizing and memory
  profiling reports.
• Local operational notes: .claude/skills/vllm-context-tuning/SKILL.md.

---

Draft. Measurements taken 2026-08-26 on a two-node DGX Spark cluster. Figures for
comparison systems are cited from the referenced PR and were not independently
reproduced.