Slide deck
BackLocal LLMs and Buying a Mac — A Technical Report
On a Mac, an agent's feel comes from KV-cache size and prefill speed, not context length — Gemma 4's 26B-A4B design, with only 3.8B parameters active, is the local sweet spot. Apple has raised Mac mini prices twice since April 2026.
Technical report · record to September 2026
The real bottleneck is KV cache and prefill, not the context limit
On a Mac, an agent's feel comes from KV-cache size and prefill speed, not context length — Gemma 4's 26B-A4B design, with only 3.8B parameters active, is the local sweet spot.
Local LLMs and buying a Mac1 / 20
In four numbers
The figures that carry the report
- 3.8Bactive parameters in the 26B-A4B model recommended below, out of 25.2B total
- 256Kmaximum context on Gemma 4's three larger models — 12B, 26B-A4B and 31B
- +35%rise in Taiwan's entry Mac mini price since April 2026
- 614 GB/sunified-memory bandwidth on Apple's M5 Max, at 128GB
Local LLMs and buying a Mac2 / 20
What happened · prices
Apple raised Mac mini prices twice
Price Before After Entry, effective US price (May 2026) $599 $799 M4 Pro starting price, US (June 2026) $1,399 $1,599 Entry M4 (16GB/256GB), Taiwan NT$19,900 (October 2024) NT$26,900 (25 August 2026) About the prices
These are the M4 line's prices; a new Mac mini, with M6 or M5 Pro, went on sale on 22 September 2026. Check the current figure before buying.
Local LLMs and buying a Mac3 / 20
What happened
What 2026 has changed
Graded by how it's confirmed, strongest first
Standing What happened Confirmed Gemma 4 replaces Gemma 3's naming Confirmed Apple raised Mac mini prices twice Confirmed Both major coding CLIs now reach local models Confirmed, one model size only A flash-attention hang hits one Gemma 4 size, not the one recommended here Confirmed, still open Ollama's Qwen 3.5 27B tool calling remains broken Reported, not independently verified M5 Max is faster for local inference, but by how much is unclear Local LLMs and buying a Mac4 / 20
Timeline · background
Four Years in Chips and Models
- 2023Apple cuts the M3 Pro's memory bandwidth to 150GB/s, from 200GB/s on the M2 Pro.
- 2024.10Mac mini M4 and M4 Pro launch; Taiwan's entry configuration (16GB/256GB) costs NT$19,900.
- 2025.03Google DeepMind releases Gemma 3, in 1B/4B/12B/27B sizes.
- 2026.01Ollama ships a native, Anthropic-compatible API (v0.14), letting Claude Code run directly against a local model.
- 2026.03Apple announces the M5 Pro and M5 Max, with unified-memory bandwidth up to 614GB/s.
Local LLMs and buying a Mac5 / 20
Timeline · April to June 2026
The year this report covers
- 2026.04.02Google DeepMind releases Gemma 4 in four sizes — E2B, E4B, 26B-A4B, 31B.
- 2026.04Ollama users on CUDA and, separately, on Apple Silicon report Gemma 4's 31B Dense model hanging under Flash Attention on long prompts.
- 2026.05.01Apple discontinues the Mac mini's cheapest configuration; the entry price effectively rises from $599 to $799.
- 2026.06.03Google adds a fifth Gemma 4 size, the 12B “Unified”, which has no separate image or audio encoder.
Local LLMs and buying a Mac6 / 20
Timeline · June to September 2026
Prices up, and new Macs on sale
- 2026.06.17GitHub Copilot CLI adds bring-your-own-key support for local and external models.
- 2026.06.25Apple raises the Mac mini M4 Pro's starting price from $1,399 to $1,599, and reinstates the discontinued entry configuration at the new, higher price.
- 2026.08.25Apple's Taiwan store lists the Mac mini from NT$26,900, and its M4 Pro configurations at NT$54,900 and NT$61,900.
- 2026.09.22Apple puts a new Mac mini, with M6 or M5 Pro, and a Mac Studio with M5 Max or M5 Ultra on sale.
Local LLMs and buying a Mac7 / 20
The argument
Why context length is the wrong number to watch
Prefill and KV cache, not the token limit
Every turn
An agentic CLI tool re-sends its entire working state — a system prompt, a tool schema, and the running transcript — on every single turn.
Prefill
Re-processing that state, called prefill, is what a Mac's memory bandwidth actually has to keep up with.
Context length
Context length only bounds how long a session can run; it says nothing about how that session feels, turn by turn.
Local LLMs and buying a Mac8 / 20
The argument
What sets the feel: prefill speed and the KV cache
What sets that feel is prefill speed, and how much memory the KV cache — the running memory of everything already processed — occupies alongside the model's own weights.
- 1,024tokens: the sliding-window size bounding most of Gemma 4's attention layers, per the model's own published specification.
- ×0.5Quantising the KV cache from 16-bit floating point to 8-bit integers halves its memory footprint outright — a definitional fact about those formats, not a measured benchmark.
Local LLMs and buying a Mac9 / 20
Gemma 4 is not 1B/4B/12B/27B
The naming everyone gets wrong
Every row is read off Google's own model cards; none is a like-for-like replacement for its Gemma 3 counterpart.
Commonly said Gemma 4 SKU Parameters Max context 1B E2B 2.3B effective (5.1B with embeddings) 128K 4B E4B 4.5B effective (8B with embeddings) 128K 12B 12B Unified (dense) 11.95B (dense, all active) 256K No Gemma 3 counterpart 26B-A4B (mixture of experts) 3.8B active of 25.2B total 256K 27B 31B dense 30.7B (dense, all active) 256K Local LLMs and buying a Mac10 / 20
The argument · naming
What “E”, “A” and “Unified” stand for
E = effective
Each layer carries its own per-layer embedding table, large but used only for quick look-ups, so the effective count (2.3B, 4.5B) is much smaller than the total with embeddings (5.1B, 8B).
A = active
3.8B of 26B-A4B's 25.2B parameters work on each token, through eight of a hundred and twenty-eight experts plus one always-on shared expert.
Unified
The 12B keeps a Gemma 3 size but not its design: it has no separate image or audio encoder, and feeds both straight into the language model.
Local LLMs and buying a Mac11 / 20
Known frictions · 31B Dense only
The flash-attention hang
At least one report states that the 26B-A4B model recommended here handles the same long prompts without issue; the bug is specific to 31B Dense.
Hardware Reported hang Nvidia CUDA (an RTX 3090) prompts over roughly 3,000–4,000 tokens Apple Silicon (an M5 Max) prompts over roughly 500 tokens The reported root cause
Gemma 4 mixes roughly fifty sliding-window attention layers with ten global-attention layers, and the two use different head dimensions (256 versus 512) that Ollama's Flash Attention implementation did not originally handle.
Local LLMs and buying a Mac12 / 20
Known frictions
Two more frictions on Ollama
Real, and each narrower than it sounds
Not yet accelerated: Gemma 4 runs without MLX on Apple Silicon
One report found Gemma 4 falling back to the slower llama.cpp backend, at roughly 15 tokens per second on 31B Dense and roughly 75 on 26B-A4B. There is as yet no MLX-accelerated figure to set beside these.
Still open: Qwen 3.5 27B tool calling
Three faults were reported together. The sampling-penalty bug was patched in v0.17.5; Ollama's correct tool-calling renderer, wired only to the model name “qwen3-coder,” not “qwen3.5,” was still open when last reported.
Local LLMs and buying a Mac13 / 20
Three years of the same trade-off
From the M3 Pro to the M5 Max
A newer chip number does not automatically mean more bandwidth.
- 150GB/sM3 Pro, October 2023 — a 25% cut from the M2 Pro's 200GB/s
- 614GB/sM5 Max, March 2026 — unified-memory bandwidth at the top 128GB configuration
- 4×more than four times the previous generation's GPU compute, on Apple's own figures
Tokens per second: estimated only
A benchmark blog's own, self-labelled “Estimated” testing puts the M5 Max ahead of the M5 Pro and of a reference RTX 4090 on several model sizes; Apple's own announcement states no tokens-per-second figure for any model.
Local LLMs and buying a Mac14 / 20
Different protocols, different timelines
Two CLIs, two ways in
Item Claude Code Copilot CLI Native local-model support Yes, since Ollama v0.14 (Jan 2026) Yes, bring-your-own-key, since June 2026 Protocol Anthropic Messages API OpenAI-compatible / external providers Set-up Point ANTHROPIC_BASE_URLat a local Ollama serverConfigure a local or external model in the CLI's model picker Local LLMs and buying a Mac15 / 20
Conclusion · fit, not a winner
Which Mac, for what
Entry point: learning the ropes on a small model
For an 8B-class dense model: Apple's M4 entry Mac mini (16GB/256GB, NT$26,900 at Apple Taiwan in August 2026) is quiet, sips power, and needs nothing beyond stock Ollama. A new Mac mini, with M6 or M5 Pro, went on sale on 22 September 2026.
The sweet spot: Gemma 4's 26B-A4B, comfortably
More unified memory (24GB or more) is what actually buys agent responsiveness here: it lets the model's active parameters and its KV cache sit alongside the OS without swapping.
Trade-off, for both
Apple prices build-to-order memory and storage through its configurator rather than as a listed figure, so budget a premium over the entry tier rather than a fixed number.
Local LLMs and buying a Mac16 / 20
Conclusion · fit, not a winner
Beyond the Mac mini: M5 Max, NVIDIA or the cloud
Maximum local capability: M5 Max, if the budget stretches
Up to 614GB/s of unified-memory bandwidth, on Apple's own figures. Apple introduced M5 Max in March 2026 for the MacBook Pro, a laptop; a Mac Studio with M5 Max went on sale only on 22 September 2026, from US$2,499 in the US.
Not on a Mac at all: an NVIDIA machine, or the cloud
An NVIDIA card's VRAM is dedicated to the model rather than shared with the OS, and a cloud API removes version and configuration management entirely. But a cloud API is neither private nor free per token, the way a machine the buyer already owns is.
Local LLMs and buying a Mac17 / 20
Trade-off: power
Power: one card against a whole computer
NVIDIA also asks for an 850W system power supply.
Rating RTX 4090, the card alone (NVIDIA) Base Mac Studio with M5 Max, whole computer at the wall (Apple) Idle 19W 7W Maximum 450W total graphics power 200W Local LLMs and buying a Mac18 / 20
What could not be verified
“Sustained inference risks thermal throttling”
Unconfirmed report
Said of the M5 Max in a MacBook Pro chassis, and not asserted here. A published troubleshooting guide says every laptop chassis eventually slows under continuous inference, but no published measurement of throttling on an M5 Max MacBook Pro was found when searched for on 28 Sep 2026.
Local LLMs and buying a Mac19 / 20
Bottom line
Which model to run has not changed; what it costs has
- KV cache and prefill still matter more than context length.
- Gemma 4's 26B-A4B, with only 3.8B parameters active, is the local sweet spot; the flash-attention hang is specific to 31B Dense.
- Apple raised Mac mini prices twice in May and June 2026, and prices build-to-order memory upgrades only through its configurator.
- Check the current price for yourself before buying. None of this is purchasing advice.
Local LLMs and buying a Mac20 / 20