The 2026 local AI lab: 16GB CUDA vs 64/96GB Mac vs planned 128GB boxes

Bottom Line The lab can measure three machines today. The RTX 4080 16GB is the fast CUDA path for small and mid-size models; 27B Q4 is tight. M1 Max 64GB and M2 Max 96GB unified memory carry long context and large weights. AMD Ryzen AI Max+ 395 128GB (Strix Halo) and Gigabyte DGX Spark 128GB are still on order (not arrived); after they land they will take 70B-class work and local agents. This article does not invent tok/s we have not measured.

Local-LLM writeups often treat a card’s VRAM and a unified-memory box as the same number. In a real lab they bottleneck differently. 16GB CUDA is compute-fast, but weights plus KV cache are trapped on the card. 64/96GB Apple UMA has room, but bandwidth and runtimes are not CUDA. The 128GB boxes are here to close that gap. They are not machines we have already benched.

As of 30 August 2026, AI Tech Engine writes down what we hold and what is still planned, separately.

Current fleet (on site)

Spec sources are vendor pages. M1 Max 64GB / 400GB/s: Apple Newsroom (2021-10). M2 Max 96GB / 400GB/s: Apple Newsroom (2023-01). RTX 4080 16GB GDDR6X: NVIDIA GeForce RTX 4080 Family.

Planned fleet (not arrived)

The two machines below are not in this lab yet. They are next measurement targets.

Sources: AMD Ryzen AI Max+ 395, AMD Ryzen AI Halo Developer Platform, NVIDIA DGX Spark, GIGABYTE AI TOP ATOM.

NVIDIA’s DGX Spark page claims up to 200B-parameter inference and 70B fine-tunes. That is NVIDIA’s claim, not a number from this lab. Gigabyte writes 200B on one box and 405B with two boxes over ConnectX-7. We do not treat those nameplates as our bench before the machines arrive.

Runtimes we actually use

Three host paths, day to day:

• Ollama — the fast “put it up” server on Mac and CUDA. A GGUF-shaped interface.
• llama.cpp — when we want to touch quant, context, and layer offload. CUDA / Metal / (planned) Vulkan · ROCm.
• MLX — Apple Silicon only. One of the main ways we keep weights in unified memory on the M1 Max and M2 Max.

vLLM and SGLang can land on the CUDA box later. Qwen’s official card recommends those two for production serving. Daily loop here is still Ollama / llama.cpp / MLX. We do not claim vLLM runs 27B “with room” on a 4080 16GB.

What is realistic at each tier

The numbers answer “do the weights fit?”, not how it feels. KV cache, vision projectors, and agent tool loops eat memory on top of weights. That is why the same 27B Q4 feels different in 16GB VRAM and 96GB UMA.

7–8B class

All three machines are fine. This is the 4080 16GB’s day job. Q4 weights are a few GB, so a fairly open context still fits on the card. Macs run the same size on MLX / llama.cpp when we want power, noise, or travel. At this tier you pick a runtime, not a capacity.

14–32B class (including 27B)

This is where the boxes split.

• RTX 4080 16GB: 14B Q4 is a realistic daily load. 27B Q4 is tight. Unsloth’s public Qwen3.8-27B GGUFs already sit around 14.3–17.6GB at 4-bit (UD-IQ4_XS 14.3GB, UD-Q4_K_S 15.4GB, Q4_0 16.1GB, UD-Q4_K_M 16.5GB, UD-Q4_K_XL 17.6GB, unsloth/Qwen3.8-27B-GGUF). Unsloth’s own 4-bit note is 16–19GB total. Drop Q4 on a 16GB card and there is almost no room for KV or a multimodal projector. Honest expectations: short context, a lower quant (Q3), or “it loads but skip the agent loop.”
• M1 Max 64GB / M2 Max 96GB: the current home for 27–32B Q4. Weights are not cropped by 16GB VRAM, and unified memory can open more context. Bandwidth may lose to 4080 GDDR6X; whether it loads is the question these boxes answer. The 96GB Studio has more slack than the 64GB MacBook. Open long context, vision, and a tool loop together and 64GB gets tight fast.
• Planned 128GB: not “barely loading 27B,” but holding context and agent state at the same time. Plan, not measurement.

70B class

16GB CUDA does not work. Q4 70B weights alone are already high 30GB, then KV.

• M1 Max 64GB: close to the floor where you can try Q4 70B. If it loads, opening context, batch, and vision together is still hard.
• M2 Max 96GB: the most realistic box in the current fleet for parking 70B Q4 locally. Still not an “agent runtime with slack.”
• Planned 128GB (Strix Halo / DGX Spark): the reason we are bringing these in. We do not quote a 200B nameplate as our result before they arrive.

How we split the boxes

Day-to-day:

1. 4080 16GB — short, repetitive work. 7–8B / 14B, CUDA kernel checks, llama.cpp CUDA builds, “does this quant even load.” 27B is smoke-test only.
2. M1 Max 64GB MacBook — travel, drafts, MLX, first-pass long context. 14–32B off power.
3. M2 Max 96GB Mac Studio — the always-on local endpoint. Around 32B, and 70B Q4 only when needed. The Apple-path reference machine.
4. Planned Strix Halo 128GB — x86 unified-memory control. Same 70B GGUF, compared with Mac UMA on bandwidth and ROCm.
5. Planned Gigabyte DGX Spark 128GB — NVIDIA unified memory plus the CUDA software stack. Agent harnesses, local packages that assume Spark (for example Portable Computer). Spec mapping lives in the open-weights article.

We do not squash three machines (now) or five (with planned) into one “lab TPS.” Different memory layouts, different bottlenecks.

Honest limits

• Unmeasured speed. Meta published Muse Glimmer Q4 tok/s on RTX 5090 / M4 Max / M5 Max. Unsloth published NVFP4 throughput on B200. Not our fleet. We do not copy those numbers. The only speed table we have is M1 Max 64GB + Qwen3.8-27B in the 2026-08-30 measurement.
• What 16GB means. August 2026 open-weight 27B/30B Q4 files often exceed 16GB, or fit with no KV left. In this lab the 4080 is not “the 27B card.” It is “the card that shows how tight 27B Q4 is.”
• What unified memory means. 64/96GB Macs are the long-context / large-weight path because of capacity, not because we are bragging about speed.
• 128GB is not a data point yet. After they land we time the same prompt, same quant, same context across three layouts (16GB CUDA / Apple UMA / 128GB UMA).

What we will measure when the two 128GB boxes arrive

Only this, and only once they sit on the desk. Until then the cells stay empty.

1. Does the same weight load — Qwen3.8-27B Q4, Muse Glimmer 30B Q4, and one 70B-class Q4. Load vs fail, memory used, how far context opens.
2. Runtime matrix — same model on Ollama / llama.cpp / (Mac) MLX / (Spark) CUDA serving. Same prompt set when we can.
3. Agent loops — not a one-shot chat. Multi-turn tool calls, including retries, as wall-clock. Not tok/s alone.
4. Strix Halo vs DGX Spark — both 128GB, different bandwidth (AMD 256GB/s nameplate vs NVIDIA 273GB/s) and software. Same 70B Q4, compare headroom and stability.
5. 16GB control — settings that failed on the 4080 (long context, vision+text, with a drafter) — do they open on 128GB? If they do, that is why we bought 128GB.

Until a speed table exists, one sentence is enough. Today, 16GB CUDA is the floor for 27B Q4, 64/96GB Mac is the long-context path, and 128GB is the planned expansion for 70B and agents.