Tratopedia
繁中
Settings

Text size

Language

Theme

High contrast

Version

v1.230.2

The release this page was built from. It is what the service worker caches under.

Slide deck

Back

Local LLMs and Buying a Mac — A Technical Report

On a Mac, an agent's feel comes from KV-cache size and prefill speed, not context length — Gemma 4's 26B-A4B design, with only 3.8B parameters active, is the local sweet spot. Apple has raised Mac mini prices twice since April 2026.

  1. Technical report · record to September 2026

    The real bottleneck is KV cache and prefill, not the context limit

    On a Mac, an agent's feel comes from KV-cache size and prefill speed, not context length — Gemma 4's 26B-A4B design, with only 3.8B parameters active, is the local sweet spot.

    Local LLMs and buying a Mac1 / 20

  2. In four numbers

    The figures that carry the report

    • 3.8Bactive parameters in the 26B-A4B model recommended below, out of 25.2B total
    • 256Kmaximum context on Gemma 4's three larger models — 12B, 26B-A4B and 31B
    • +35%rise in Taiwan's entry Mac mini price since April 2026
    • 614 GB/sunified-memory bandwidth on Apple's M5 Max, at 128GB

    Local LLMs and buying a Mac2 / 20

  3. What happened · prices

    Apple raised Mac mini prices twice

    PriceBeforeAfter
    Entry, effective US price (May 2026)$599$799
    M4 Pro starting price, US (June 2026)$1,399$1,599
    Entry M4 (16GB/256GB), TaiwanNT$19,900 (October 2024)NT$26,900 (25 August 2026)

    About the prices

    These are the M4 line's prices; a new Mac mini, with M6 or M5 Pro, went on sale on 22 September 2026. Check the current figure before buying.

    Local LLMs and buying a Mac3 / 20

  4. What happened

    What 2026 has changed

    Graded by how it's confirmed, strongest first

    StandingWhat happened
    ConfirmedGemma 4 replaces Gemma 3's naming
    ConfirmedApple raised Mac mini prices twice
    ConfirmedBoth major coding CLIs now reach local models
    Confirmed, one model size onlyA flash-attention hang hits one Gemma 4 size, not the one recommended here
    Confirmed, still openOllama's Qwen 3.5 27B tool calling remains broken
    Reported, not independently verifiedM5 Max is faster for local inference, but by how much is unclear

    Local LLMs and buying a Mac4 / 20

  5. Timeline · background

    Four Years in Chips and Models

    1. 2023Apple cuts the M3 Pro's memory bandwidth to 150GB/s, from 200GB/s on the M2 Pro.
    2. 2024.10Mac mini M4 and M4 Pro launch; Taiwan's entry configuration (16GB/256GB) costs NT$19,900.
    3. 2025.03Google DeepMind releases Gemma 3, in 1B/4B/12B/27B sizes.
    4. 2026.01Ollama ships a native, Anthropic-compatible API (v0.14), letting Claude Code run directly against a local model.
    5. 2026.03Apple announces the M5 Pro and M5 Max, with unified-memory bandwidth up to 614GB/s.

    Local LLMs and buying a Mac5 / 20

  6. Timeline · April to June 2026

    The year this report covers

    1. 2026.04.02Google DeepMind releases Gemma 4 in four sizes — E2B, E4B, 26B-A4B, 31B.
    2. 2026.04Ollama users on CUDA and, separately, on Apple Silicon report Gemma 4's 31B Dense model hanging under Flash Attention on long prompts.
    3. 2026.05.01Apple discontinues the Mac mini's cheapest configuration; the entry price effectively rises from $599 to $799.
    4. 2026.06.03Google adds a fifth Gemma 4 size, the 12B “Unified”, which has no separate image or audio encoder.

    Local LLMs and buying a Mac6 / 20

  7. Timeline · June to September 2026

    Prices up, and new Macs on sale

    1. 2026.06.17GitHub Copilot CLI adds bring-your-own-key support for local and external models.
    2. 2026.06.25Apple raises the Mac mini M4 Pro's starting price from $1,399 to $1,599, and reinstates the discontinued entry configuration at the new, higher price.
    3. 2026.08.25Apple's Taiwan store lists the Mac mini from NT$26,900, and its M4 Pro configurations at NT$54,900 and NT$61,900.
    4. 2026.09.22Apple puts a new Mac mini, with M6 or M5 Pro, and a Mac Studio with M5 Max or M5 Ultra on sale.

    Local LLMs and buying a Mac7 / 20

  8. The argument

    Why context length is the wrong number to watch

    Prefill and KV cache, not the token limit

    Every turn

    An agentic CLI tool re-sends its entire working state — a system prompt, a tool schema, and the running transcript — on every single turn.

    Prefill

    Re-processing that state, called prefill, is what a Mac's memory bandwidth actually has to keep up with.

    Context length

    Context length only bounds how long a session can run; it says nothing about how that session feels, turn by turn.

    Local LLMs and buying a Mac8 / 20

  9. The argument

    What sets the feel: prefill speed and the KV cache

    What sets that feel is prefill speed, and how much memory the KV cache — the running memory of everything already processed — occupies alongside the model's own weights.

    • 1,024tokens: the sliding-window size bounding most of Gemma 4's attention layers, per the model's own published specification.
    • ×0.5Quantising the KV cache from 16-bit floating point to 8-bit integers halves its memory footprint outright — a definitional fact about those formats, not a measured benchmark.

    Local LLMs and buying a Mac9 / 20

  10. Gemma 4 is not 1B/4B/12B/27B

    The naming everyone gets wrong

    Every row is read off Google's own model cards; none is a like-for-like replacement for its Gemma 3 counterpart.

    Commonly saidGemma 4 SKUParametersMax context
    1BE2B2.3B effective (5.1B with embeddings)128K
    4BE4B4.5B effective (8B with embeddings)128K
    12B12B Unified (dense)11.95B (dense, all active)256K
    No Gemma 3 counterpart26B-A4B (mixture of experts)3.8B active of 25.2B total256K
    27B31B dense30.7B (dense, all active)256K

    Local LLMs and buying a Mac10 / 20

  11. The argument · naming

    What “E”, “A” and “Unified” stand for

    E = effective

    Each layer carries its own per-layer embedding table, large but used only for quick look-ups, so the effective count (2.3B, 4.5B) is much smaller than the total with embeddings (5.1B, 8B).

    A = active

    3.8B of 26B-A4B's 25.2B parameters work on each token, through eight of a hundred and twenty-eight experts plus one always-on shared expert.

    Unified

    The 12B keeps a Gemma 3 size but not its design: it has no separate image or audio encoder, and feeds both straight into the language model.

    Local LLMs and buying a Mac11 / 20

  12. Known frictions · 31B Dense only

    The flash-attention hang

    At least one report states that the 26B-A4B model recommended here handles the same long prompts without issue; the bug is specific to 31B Dense.

    HardwareReported hang
    Nvidia CUDA (an RTX 3090)prompts over roughly 3,000–4,000 tokens
    Apple Silicon (an M5 Max)prompts over roughly 500 tokens

    The reported root cause

    Gemma 4 mixes roughly fifty sliding-window attention layers with ten global-attention layers, and the two use different head dimensions (256 versus 512) that Ollama's Flash Attention implementation did not originally handle.

    Local LLMs and buying a Mac12 / 20

  13. Known frictions

    Two more frictions on Ollama

    Real, and each narrower than it sounds

    Not yet accelerated: Gemma 4 runs without MLX on Apple Silicon

    One report found Gemma 4 falling back to the slower llama.cpp backend, at roughly 15 tokens per second on 31B Dense and roughly 75 on 26B-A4B. There is as yet no MLX-accelerated figure to set beside these.

    Still open: Qwen 3.5 27B tool calling

    Three faults were reported together. The sampling-penalty bug was patched in v0.17.5; Ollama's correct tool-calling renderer, wired only to the model name “qwen3-coder,” not “qwen3.5,” was still open when last reported.

    Local LLMs and buying a Mac13 / 20

  14. Three years of the same trade-off

    From the M3 Pro to the M5 Max

    A newer chip number does not automatically mean more bandwidth.

    • 150GB/sM3 Pro, October 2023 — a 25% cut from the M2 Pro's 200GB/s
    • 614GB/sM5 Max, March 2026 — unified-memory bandwidth at the top 128GB configuration
    • 4×more than four times the previous generation's GPU compute, on Apple's own figures

    Tokens per second: estimated only

    A benchmark blog's own, self-labelled “Estimated” testing puts the M5 Max ahead of the M5 Pro and of a reference RTX 4090 on several model sizes; Apple's own announcement states no tokens-per-second figure for any model.

    Local LLMs and buying a Mac14 / 20

  15. Different protocols, different timelines

    Two CLIs, two ways in

    ItemClaude CodeCopilot CLI
    Native local-model supportYes, since Ollama v0.14 (Jan 2026)Yes, bring-your-own-key, since June 2026
    ProtocolAnthropic Messages APIOpenAI-compatible / external providers
    Set-upPoint ANTHROPIC_BASE_URL at a local Ollama serverConfigure a local or external model in the CLI's model picker

    Local LLMs and buying a Mac15 / 20

  16. Conclusion · fit, not a winner

    Which Mac, for what

    Entry point: learning the ropes on a small model

    For an 8B-class dense model: Apple's M4 entry Mac mini (16GB/256GB, NT$26,900 at Apple Taiwan in August 2026) is quiet, sips power, and needs nothing beyond stock Ollama. A new Mac mini, with M6 or M5 Pro, went on sale on 22 September 2026.

    The sweet spot: Gemma 4's 26B-A4B, comfortably

    More unified memory (24GB or more) is what actually buys agent responsiveness here: it lets the model's active parameters and its KV cache sit alongside the OS without swapping.

    Trade-off, for both

    Apple prices build-to-order memory and storage through its configurator rather than as a listed figure, so budget a premium over the entry tier rather than a fixed number.

    Local LLMs and buying a Mac16 / 20

  17. Conclusion · fit, not a winner

    Beyond the Mac mini: M5 Max, NVIDIA or the cloud

    Maximum local capability: M5 Max, if the budget stretches

    Up to 614GB/s of unified-memory bandwidth, on Apple's own figures. Apple introduced M5 Max in March 2026 for the MacBook Pro, a laptop; a Mac Studio with M5 Max went on sale only on 22 September 2026, from US$2,499 in the US.

    Not on a Mac at all: an NVIDIA machine, or the cloud

    An NVIDIA card's VRAM is dedicated to the model rather than shared with the OS, and a cloud API removes version and configuration management entirely. But a cloud API is neither private nor free per token, the way a machine the buyer already owns is.

    Local LLMs and buying a Mac17 / 20

  18. Trade-off: power

    Power: one card against a whole computer

    NVIDIA also asks for an 850W system power supply.

    RatingRTX 4090, the card alone (NVIDIA)Base Mac Studio with M5 Max, whole computer at the wall (Apple)
    Idle19W7W
    Maximum450W total graphics power200W

    Local LLMs and buying a Mac18 / 20

  19. What could not be verified

    “Sustained inference risks thermal throttling”

    Unconfirmed report

    Said of the M5 Max in a MacBook Pro chassis, and not asserted here. A published troubleshooting guide says every laptop chassis eventually slows under continuous inference, but no published measurement of throttling on an M5 Max MacBook Pro was found when searched for on 28 Sep 2026.

    Local LLMs and buying a Mac19 / 20

  20. Bottom line

    Which model to run has not changed; what it costs has

    • KV cache and prefill still matter more than context length.
    • Gemma 4's 26B-A4B, with only 3.8B parameters active, is the local sweet spot; the flash-attention hang is specific to 31B Dense.
    • Apple raised Mac mini prices twice in May and June 2026, and prices build-to-order memory upgrades only through its configurator.
    • Check the current price for yourself before buying. None of this is purchasing advice.

    Local LLMs and buying a Mac20 / 20