Benchmarks
Magnitude vs Ollama on the same hardware, across 11 popular open models
Magnitude vs Ollama (MLX)Mac M4 Pro 48 GB
1.32x
faster prefill on average
2.19x
faster decode on average
Magnitude vs Ollama (llama.cpp)NVIDIA DGX Spark
1.34x
faster prefill on average
2.17x
faster decode on average
Magnitude vs Ollama (MLX)Mac M4 Pro 48 GB
| Model | Prefill tok/s | Speedup | Decode tok/s | Speedup | Drafters (Ollama / Magnitude) |
|---|---|---|---|---|---|
| Qwen 3.5 4BOllama: GGUF Q4_K_M | 422 → 762 | 1.81x | 45.2 → 134 | 2.97x | MTP / DFlash |
| Qwen 3.5 9BOllama: MLX NVFP4 | 324 → 532 | 1.64x | 41.8 → 97.4 | 2.33x | MTP / DFlash |
| Qwen 3.8 27BOllama: MLX NVFP4 | 98.5 → 128 | 1.30x | 21.0 → 34.3 | 1.63x | MTP / DFlash2 |
| Qwen 3.6 35BOllama: MLX NVFP4 | 497 → 613 | 1.23x | 47.2 → 122 | 2.59x | MTP / DFlash |
| Gemma 4 E2BOllama: MLX NVFP4 | 2,503 → 2,281 | 0.91x | 118 → 164 | 1.39x | MTP / MTP |
| Gemma 4 E4BOllama: MLX NVFP4 | 961 → 1,276 | 1.33x | 37.5 → 111 | 2.96x | MTP / MTP |
| Gemma 4 12BOllama: MLX NVFP4 | 208 → 300 | 1.44x | 30.7 → 41.5 | 1.35x | MTP / MTP |
| Gemma 4 26BOllama: MLX NVFP4 | 516 → 522 | 1.01x | 38.7 → 82.4 | 2.13x | MTP / MTP |
| Gemma 4 31BOllama: MLX NVFP4 | 77.5 → 81.9 | 1.06x | 8.6 → 30.2 | 3.51x | MTP / MTP |
| Nemotron 3.5Ollama: MLX NVFP4 | 585 → 641 | 1.10x | 49.1 → 63.4 | 1.29x | MTP / DFlash |
| Muse GlimmerOllama: MLX NVFP4 | 111 → 185 | 1.67x | 17.5 → 34.1 | 1.95x | DFlash / DFlash |
| Average | 1.32x | 2.19x |
Magnitude vs Ollama (llama.cpp)NVIDIA DGX Spark
| Model | Prefill tok/s | Speedup | Decode tok/s | Speedup | Drafters (Ollama / Magnitude) |
|---|---|---|---|---|---|
| Qwen 3.5 4BOllama: GGUF Q4_K_M | 2,925 → 4,314 | 1.48x | 61.1 → 296 | 4.84x | MTP / DFlash |
| Qwen 3.5 9BOllama: GGUF Q4_K_M | 2,065 → 2,952 | 1.43x | 53.9 → 214 | 3.96x | MTP / DFlash |
| Qwen 3.8 27BOllama: GGUF Q4_K_M | 663 → 840 | 1.27x | 34.0 → 66.7 | 1.96x | MTP / DFlash2 |
| Qwen 3.6 35BOllama: GGUF Q4_K_M | 2,285 → 2,528 | 1.11x | 85.8 → 216 | 2.52x | MTP / DFlash |
| Gemma 4 E2BOllama: GGUF Q4_K_M | 4,409 → 9,718 | 2.20x | 177 → 268 | 1.51x | MTP / MTP |
| Gemma 4 E4BOllama: GGUF Q4_K_M | 2,998 → 5,369 | 1.79x | 111 → 183 | 1.65x | MTP / MTP |
| Gemma 4 12BOllama: GGUF Q4_K_M | 1,420 → 1,421 | 1.00x | 45.4 → 62.8 | 1.38x | MTP / MTP |
| Gemma 4 26BOllama: GGUF Q4_K_M | 2,368 → 2,270 | 0.96x | 79.7 → 113 | 1.42x | MTP / MTP |
| Gemma 4 31BOllama: GGUF Q4_K_M | 448 → 487 | 1.09x | 25.1 → 50.7 | 2.02x | MTP / MTP |
| Nemotron 3.5Ollama: GGUF Q4_K_M | 2,626 → 2,693 | 1.03x | 94.9 → 72.2 | 0.76x | MTP / DFlash |
| Muse GlimmerOllama: GGUF Q4_K_M | 734 → 1,018 | 1.39x | 37.9 → 68.2 | 1.80x | DFlash / DFlash |
| Average | 1.34x | 2.17x |
Methodology
- Engines
- Magnitude and Ollama v0.35.1, each with its default settings
- Hardware
- Mac mini with M4 Pro and 48 GB on macOS 15.7. NVIDIA DGX Spark (GB10) with 128 GB
- Models
- The same model at about 4-bit on both engines. Ollama used its MLX NVFP4 tags on Mac and GGUF Q4_K_M tags on DGX Spark
- Speculative decoding
- On for both. Ollama used the drafter that ships with each tag, and Magnitude used its catalog drafter
- Workload
- One chat request with about 64k tokens of context and 256 output tokens: passages from Moby-Dick, with the model asked to repeat the latest one. Greedy sampling, thinking off, and the identical prompt for both engines
- Measurement
- A fresh engine process and an empty prompt cache for every request, on an idle machine that passed a thermal check. Rates come from each engine’s own timing counters. Each result is the median of two requests, except some Ollama results, which are single requests from an earlier run of the same harness
- Averages
- The arithmetic mean of each model’s speedup, so every model counts equally.
Notes
- On Mac, Ollama’s MLX runner drafted very few tokens for Gemma 4 E4B, 26B, and 31B at 64k, even with its drafter loaded. That is Ollama’s default behavior, but those decode gains partly compare speculative decoding against almost none.
- Where Magnitude is slower: Nemotron 3.5 decode on DGX Spark, Gemma 4 E2B prefill on Mac, and Gemma 4 26B prefill on DGX Spark. All are included in the averages.
- The engines don’t run identical weights, templates, or numerics, so outputs differ. Every request produced valid text for the full 256 tokens. Answer quality isn’t scored.
Reproduce
The benchmark harness is open source. From inference/ in the Magnitude repository, one model on Magnitude:
uv run --project evaluation --package magnitude-bench magnitude-bench run \
--mode prose-repeat --model qwen3.6-35b-a3b:gguf:q4 --context 64k \
--engine magnitude --passes 2 --fresh-tuningOllama runs use the same command with --engine ollama-mlx (Mac) or --engine ollama-registry (DGX Spark) and the Ollama tag as the artifact.