Showing version 1 bot legacy · api · 2026-08-26T11:52:56Z

Running a Local Coding Model: opencode + GLM on Apple Silicon

Running a Local Coding Model: opencode + GLM on Apple Silicon

A complete, measured account of putting a 30B coding model on a dedicated Mac and
driving it from opencode over a network. Every number here was read off the
running system, not estimated. Where an earlier estimate was wrong, the correction
is kept and labelled — the wrong figures are as instructive as the right ones.

Posture: one machine, one resident model, loopback-only, verified by a script
rather than by belief.

1. Hardware and the real ceiling
Host dedicated Mac, 48 GB unified memory, Apple Silicon
Usable for inference ~37.4 GiB, not 48 — macOS wires roughly 75% for the GPU
Server Ollama 0.32.15, loopback only
Model glm-4.7-flash:latest — 30B-A3B MoE, Q4_K_M
Client opencode on a laptop, over an SSH tunnel

The 48-vs-37.4 gap is the first thing that catches people. Budget against 37.4 GiB
or you will be debugging an eviction loop.

Measured memory, read from the load log

load_tensors:  MTL0 model buffer size = 17958.41 MiB
llama_kv_cache: size = 5076.00 MiB ( 49152 cells, 47 layers, 2/2 seqs )
sched_reserve:  MTL0 compute buffer size = 213.21 MiB  (+ 56.01 CPU)
  • Weights: 17,958 MiB (17.5 GiB)
  • KV cache: 51.6 MiB per 1K tokens per slot — 5,076 MiB ÷ 49,152 cells ÷ 2
    sequences. An earlier note in our own docs said "~0.1 GB per 1K"; that was 2×
    pessimistic     and led to under-provisioning context.
  • Resident total: 23,304 MiB ≈ 22.8 GiB, which matches Ollama's reported
    24.38 GB (decimal).

Registry size ≠ resident size. Resident is weights + KV at the loaded context

  • compute graph. Read size from /api/ps; never infer it from the download.

What memory permits, versus what is useful

With ~11.6 GiB free:

slots max context/slot
1 ~329K
2 ~164K
4 ~82K

Memory is almost never the binding constraint. Latency is — see §7.

2. Installation on macOS

The published install.sh is Linux only and will fail. Use the headless
tarball; nothing needs sudo and nothing lands outside $HOME.

# the tarball is FLAT — extract into a dedicated dir or it spills ~40 files
mkdir -p ~/.local/ollama-dist && tar -xzf ollama-darwin.tgz -C ~/.local/ollama-dist
ln -sf ~/.local/ollama-dist/ollama ~/.local/bin/ollama

The binary loads its dylibs as siblings, so the directory cannot be split up.

launchd, not a shell background job

<!-- ~/Library/LaunchAgents/com.llmmac.ollama.plist -->
<key>EnvironmentVariables</key>
<dict>
  <key>OLLAMA_HOST</key>           <string>127.0.0.1:11434</string>
  <key>OLLAMA_CONTEXT_LENGTH</key> <string>49152</string>
  <key>OLLAMA_KEEP_ALIVE</key>     <string>24h</string>
  <key>OLLAMA_NUM_PARALLEL</key>   <string>2</string>
</dict>
<key>KeepAlive</key><true/>
<key>RunAtLoad</key><true/>

Bootstrap into gui/$UID so it survives ssh logout and starts at login:

launchctl bootstrap gui/$UID ~/Library/LaunchAgents/com.llmmac.ollama.plist

Why each variable matters

variable value reason
OLLAMA_HOST 127.0.0.1:11434 Ollama has no authentication. Never 0.0.0.0.
OLLAMA_CONTEXT_LENGTH 49152 Clients cannot set num_ctx over /v1. This is the only place context can be pinned.
OLLAMA_KEEP_ALIVE 24h opencode does not propagate keep_alive (opencode#2979), so the model would otherwise unload between turns and pay a 54–84 s cold load.
OLLAMA_NUM_PARALLEL 2 Two slots at 49,152 each. See §7 for why more is not better.

3. Network: an SSH tunnel, and two traps

opencode → 127.0.0.1:11435 (laptop) → ssh → 127.0.0.1:11434 (host) → Ollama

The local port is 11435, not 11434, deliberately. A local Ollama would own
11434, and a silently-failed forward would route the agent to the local model
while you believed you were testing the remote one. Different port, loud failure.

Trap 1 — the forward must not multiplex

Put the forward on its own host alias with its own connection. If the keeper
multiplexes onto a shared ControlMaster, its forward inherits ControlPersist
and dies with it — a tunnel that evaporates after ten idle minutes, presenting as
Cannot connect to API with a perfectly healthy server.

Trap 2 — ExitOnForwardFailure kills ordinary ssh

Once the keeper holds 11435, every ssh host that also requests the forward
fails with Address already in use — and with ExitOnForwardFailure yes the whole
session dies, breaking log tails and health scripts. Split it:

Host llmmac-tunnel          # tunnel only; MUST come first (first value wins)
  ControlMaster no
  ControlPath none
  LocalForward 11435 127.0.0.1:11434
  ExitOnForwardFailure yes

Host llmmac llmmac-tunnel   # everything else, shared
  HostName 10.x.x.x
  ControlMaster auto
  ControlPersist 10m

The launchd keeper targets llmmac-tunnel; humans and scripts use llmmac.
Verify with the only honest check:

ssh -G llmmac        | grep -c localforward   # want 0
ssh -G llmmac-tunnel | grep -c localforward   # want 1

Then prove KeepAlive works rather than assuming: kill the keeper's PID, wait
past ThrottleInterval, confirm a new PID owns the port.

4. The opencode provider

{
  "provider": {
    "llmmac": {
      "npm": "@ai-sdk/openai-compatible",
      "options": { "baseURL": "http://127.0.0.1:11435/v1" },
      "models": {
        "glm-4.7-flash:latest": {
          "tool_call": true,          // false or unset => tools never fire
          "reasoning": true,
          "limit": { "context": 46000, "output": 8192 }
        }
      }
    }
  },
  "compaction": { "auto": true, "prune": true, "reserved": 8000 }
}

tool_call: true must be declared per model. Without it the model narrates what
it would have done and never calls a tool — a failure that looks like
incompetence rather than configuration.

The sizing inequality

num_ctx on the server covers prompt plus generation, so:

(limit.context − compaction.reserved) + limit.output  ≤  server n_ctx_slot

Worked against a 49,152 slot:

context reserved output trigger worst case verdict
48000 8000 8192 40,000 48,192 valid, 960 margin — too thin
46000 8000 8192 38,000 46,192 2,960 margin
48000 8000 4096 40,000 44,096 valid, but see §5

Exceed it and the request is accepted; the server truncates the oldest tokens
and the model behaves as though it simply forgot
(ollama#14259). No error, no
truncation field, HTTP 200.

5. The output budget is not a safety valve

Reflexively halving limit.output to bound runaway generations is wrong, and cost
us two failed attempts to generate one file.

GLM writes its reasoning trace into the output budget. A task producing a
~5,000-token file therefore needs room for trace and file:

attempt limit.output generated result
1 4096 4,096 (capped) nothing written
2 4096 4,096 (capped) nothing written
3 8192 4,969 file written

Truncation and runaway present identically — the turn hits the cap, burns
~290 s, produces nothing. The distinguishing signal: a runaway hits the cap on
turns with nothing large to produce; truncation hits it on the one turn with a big
legitimate file. Same number, opposite fix.

The undersized cap also silently degraded compaction, whose summary is just
another large output:

output cap compaction result
4096 44K → 22.6K (weak, summary truncated)
8192 36K → 9.3K (clean)

6. Verification: eleven layers, in order

Debugging by guessing which layer broke wastes afternoons. A doctor.sh that
checks in dependency order and names the failure does not:

  1. ssh config resolves the host
  2. passwordless login works
  3. key file ownership
  4. connection multiplexing alive, connect time sane
  5. terminfo present on the remote
  6. port 11435 bound locally
  7. Ollama reachable through the tunnel
  8. server refuses the LAN address (still loopback-only)
  9. expected models installed
  10. residency — exactly one heavy model, with its context
  11. client limit.context ≤ server context, per model

Step 11 is the one that catches silent truncation before it costs you a session.

7. Results: what the thing actually does

Solo, verified uncontended, same prompts at each depth:

prompt depth prefill generation
1,360 1,183 tok/s 60.5 tok/s
2,305 1,017 tok/s 55.9 tok/s
2,933 926 tok/s 53.1 tok/s
~9,800 — 34.5 tok/s
~27,200 — 17.9 tok/s
~65,000 — ~8 tok/s

Generation halves for roughly every doubling of depth, because the KV cache is
re-read for every generated token. Context is not a one-time prefill cost; it is a
tax on every token you subsequently generate.

The second slot is not free

OLLAMA_NUM_PARALLEL=2 gives concurrency, not throughput:

configuration throughput
one worker, ~11K depth 33.0 tok/s
two concurrent, aggregate 18.4 tok/s
six dispatched at once 2.4 tok/s

Aggregate throughput falls. Two decodes complete less total work per second
than running them sequentially — memory bandwidth is the bottleneck and the KV
reads do not share. Parallel agent dispatch on this hardware is net-negative.

The prefix cache buys TTFT, not decode

Four distinct ~16K prompts stayed cached simultaneously (n_slots = 1 limits
concurrency, not retention), each returning in 0.1–0.2 s after the others had
been served. But at identical depth, cache hit rates of 18% and 90% produced
20.7 and 21.2 tok/s — indistinguishable. The cache skips recomputing prefill;
the KV is still re-read per generated token.

8. Failure modes that produce no error

The ones worth memorising, because nothing in the stack reports them.

symptom cause
"The model forgot what I sent." client limit.context > server OLLAMA_CONTEXT_LENGTH. HTTP 200, normal finish_reason.
"Every message takes a minute." The resident model is not the one being requested; each turn evicts 22–25 GB and reloads.
"Slow after I ran two things." Two heavy models competing. Only one fits.
"First message hangs." Cold load, 54–84 s. Normal. Warm it.
Cannot connect to API, server healthy The tunnel died — see §3.
Turns cost 620–860 s for 50 tokens A context shift. Unrecoverable; restart the session.

Measuring correctly

  • Divide by the server's eval time, not wall clock. An early attempt divided
    completion tokens by total wall time and produced "2.21 tok/s at 27K" — nonsense;
    prefill dominated the clock. The log prints prompt eval time and eval time
    separately.
  • Check for a competing session. With two slots, another client silently halves
    your numbers. Confirm all slots are idle and that only your task ids interleave.
  • Distrust impossible orderings. A contaminated benchmark once read slower at
    1.3K than a live session was getting at 29K    . Generation cannot improve with
    depth; if it appears to, you measured contention, not throughput.

9. What this setup is good for

GLM at 46K context, 2 slots, on one dedicated Mac, reaches roughly 50–60 tok/s
on shallow contexts and 17 tok/s at 29K, with retrieval that stays reliable to
at least 47K. That is a usable coding assistant for bounded, well-specified work,
and it costs nothing per token.

It is not a substitute for a frontier model on ambiguous work, and the discipline
it demands — shallow contexts, one model resident, serial dispatch, verified
configuration — is the price of running it at all.

The companion article, Tuning a Local Model, covers what to change when it is
not behaving.