> ## Documentation Index
> Fetch the complete documentation index at: https://docs.heylinger.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Benchmarks

> Compare model speed and quality on real hardware before you install.

These model results describe specific experimental configurations. They do not describe the model that setup selects today. Check [Hardware compatibility](https://docs.heylinger.com/hardware/compatibility.md) for current model choices and available-memory requirements.

## How do the local models score?

Compare a benchmark with your plan only when the model, quantization, engine, GPU, context, and concurrency match. A model name and memory capacity alone do not establish the same writing speed.

The scores below illustrate the third-party AIHOT scale, where 50 is the average of 123 reference models. Scores within 3 points count as the same level. A candidate fallback such as Spark-X2.6-4B has no installed SimplexAI score. Vendor-reported results do not become an installed score.

Each machine tier below lists the model it gets, the measured writing speed and the installed score.

| Machine | Model | Quant | Context | Decode, tok/s | SimplexAI | Compares with |
| - | - | - | - | - | - | - |
| Mac 16 GB | Spark-X2.6-4B | Q4\_K\_M | 64K | 32.99–37.24 for the slower of two concurrent sessions with short prompts; 24.61–26.23 after one 56,978-token prompt | Not ranked | — |
| Mac 24 GB | Qwen3.8-27B | UD-IQ3\_S | 64K | 46.06 single session (median of 3); 48.43 for 2 sessions together | 51.9 | ≈ Claude Opus 4.5 |
| Mac 32 GB | Qwen3.8-27B | UD-Q4\_K\_M | 128K | 45.44 single session (median of 3) | 54.0 | ≈ Claude Sonnet 4.6 |
| Linux NVIDIA 8 GB + 32 GB | Qwen3.6-35B-A3B | UD-Q3\_K\_M | 64K | 121–146 short; 88–103 after a 62K prompt | 35.4 | ≈ Gemini 2.5 Pro |
| Linux NVIDIA 12 GB + 32 GB | Qwen3.8-Flash-Next† | GSQ-RCO Q2\_0 | 64K | 21.3 short (19.8–25.4); 34.1 after a 62K prompt (12.0–37.3), first token after about 3 minutes | 57.3 | ≈ Claude Opus 4.6 |
| Linux NVIDIA 24 GB + 64 GB | Qwen3.8-27B | UD-Q4\_K\_M | 128K | 32.5–33.2 single session at 130K; 28–41 per session with two sessions at about 8K | 54.0 | ≈ Claude Sonnet 4.6 |

Decode speeds were measured in our lab, mostly on a simulated tier (a larger card held to the tier's memory budget, or a larger Mac planned as the tier), so a real machine of the tier can be slower. SimplexAI scores sit on the third-party AIHOT scale, where 50 is the reference average, and scores within 3 points count as the same level.

Test conditions:

* Mac 16 GB: An Apple M6 Mac mini with 24 GB of unified memory, not a 16 GB machine. llama.cpp b12167 Metal with q8\_0 KV, a 64K window shared by two slots, and the RAM prompt cache disabled.
* Mac 24 GB: Simulated 24 GB tier on an M5 Ultra with 256 GB: only the planner's memory figures were lowered, the physical memory and page cache were not; installed with a 128K window shared by two sessions, KV q4\_0, short prompts; the installer now plans 64K; the chips in common 24 GB Macs are much slower.
* Mac 32 GB: Short prompts on an M5 Ultra with 256 GB, using the 48 GB tier's setup (200K window, two sessions, KV q8\_0); the 32 GB setup (128K, KV q4\_0) is not measured yet; common 32 GB Macs are slower.
* Linux NVIDIA 8 GB + 32 GB: Simulated: a 96 GB card limited to an 8 GB budget, memory capped at 28 GB for a 32 GB machine; strata-poly poly.7, one request per start; not a real 8 GB card.
* Linux NVIDIA 12 GB + 32 GB: Simulated: a 96 GB card limited to a 12 GB budget, memory capped at 28 GB; Strata pages the experts from the model file; 3 requests from one start; not a real 12 GB card.
* Linux NVIDIA 24 GB + 64 GB: A real RTX 4090 D 24 GB card; the two sessions share one 128K window (about 8K each when both ran); llama.cpp r3 build of 2026-10-25, KV q8\_0/q4\_0 and q8\_0/q8\_0 (the ranges cover both); the two sessions overlapped only in part, so per-session numbers are approximate; not a rerun of the current installer recipe.

Not ranked (Mac 16 GB): AIHOT has no third-party results for Spark-X2.6-4B. Our own self-test: SciCode subproblems 28.2% (82/291), read on AIHOT's SciCode scale as 22.1 (12.5–30.8). Self-test, not a SimplexAI score: Q4\_K\_M on an M5 Ultra with 256 GiB (llama.cpp b11370, 4 sessions of 192K, not the 16 GB setup), thinking on, 16,384 output tokens per turn (25 of 65 problems hit the cap and count as failed), one pass\@1; the interval covers resampling the problems only; the vendor reports 36.8% unquantized.

† Linux NVIDIA 12 GB + 32 GB: the table shows Qwen3.8-Flash-Next GSQ-RCO Q2\_0; the default recommendation for this machine is Qwen3.6-35B-A3B UD-Q4\_K\_M.

## Where does the SimplexAI score come from?

The SimplexAI score rates how capable a model is after it is installed on your machine. It borrows the scale of the [AIHOT leaderboard](https://aihot.news/leaderboard), so a local model and a cloud model sit on one scale.

The score starts from the base score: the score AIHOT publishes for the full-precision model. AIHOT builds that score in three steps, described on its [method page](https://aihot.news/leaderboard/rules):

1. **Count** only results that a third-party evaluator ran itself. Vendor-reported results do not score.
2. **Convert** each result to one scale, where the score is 50 plus 15 for each standard deviation above the reference average. The 123 reference models are frozen, so the scale does not drift when new models arrive.
3. **Weight** the results by domain into one overall score.

| Domain | Weight | Example evaluations |
| - | - | - |
| Coding | 38% | Terminal-Bench 4.0, LiveBench Coding, SciCode |
| Research | 19% | HLE, FrontierMath, LiveBench Reasoning |
| Professional work | 19% | GDPval-AA, AutomationBench-AA, TaxAgentBench |
| Knowledge | 14% | AA-LCR, SimpleQA Verified |
| Human preference | 10% | LMArena blind votes |

A domain that nobody has tested for a model takes the model's overall level. It adds no points and removes none.

A score is not a percent correct, and 50 is not half marks. It means average among the reference models.

## How does quantization change the score?

A quantized model loses some quality, so its installed score is lower than the base score. Each pass rate of the full-precision model is multiplied by a retention rate, then read on the AIHOT scale again.

The retention rate comes from the best source that exists:

1. **Measured task scores.** The quantized model's task score divided by the full-precision score, from a lab run or from the publisher of the quantization.
2. **A band by bits per weight (bpw).** When no task score exists, the size of the file sets the rate.

| Bits per weight | 7.5 or more | 5.5 to 7.5 | 4.5 to 5.5 | 3.8 to 4.5 | 3.2 to 3.8 | 2.6 to 3.2 | Below 2.6 |
| - | - | - | - | - | - | - | - |
| Retention | 1.00 | 0.99 | 0.97 | 0.95 | 0.92 | 0.88 | 0.83 |

The table shows how each score in the tier table is made.

| Model and quantization | Base score | Retention | Source of retention | Installed score |
| - | - | - | - | - |
| Qwen3.8-27B UD-IQ3\_S | 55.3 | 0.92 | Band for 3.52 bpw | 51.9 |
| Qwen3.8-27B UD-Q4\_K\_M | 55.3 | 0.97 | Band for 4.88 bpw | 54.0 |
| Qwen3.6-35B-A3B UD-Q3\_K\_M | 37.1 | 0.95 | Band for 3.91 bpw | 35.4 |
| Qwen3.8-Flash-Next GSQ-RCO Q2\_0 | 59.6 | 0.957 | Publisher's task scores | 57.3 |

Retention applies to pass rates, not to the score. That is why 55.3 at 0.97 gives 54.0 and not 53.6.

## How do I read a score?

Compare a score with the cloud models on the same scale, and ignore gaps under 3 points. AIHOT finds that the order of two models is stable only when they differ by 3 points or more.

| Reference model | AIHOT score |
| - | - |
| Claude Opus 5.5 | 73.9 |
| GPT-5.5 | 62.7 |
| Claude Opus 4.6 | 58.7 |
| Claude Sonnet 4.6 | 53.6 |
| Claude Opus 4.5 | 53.1 |
| Gemini 2.5 Pro | 34.9 |
| Claude Haiku 4.5 | 34.3 |

Treat local-model scores within 3 points of a reference as the same level.

Do not divide scores. A score has no zero point, so "points per gigabyte" means nothing.

## What does the score not tell me?

The score rates the model alone, at one point in time. Keep these limits in mind:

* **Scores move.** These use the AIHOT round of November 4, 2026, method version `2026.11-capability-scale-v24`. A new round can change them.
* **The score is not a task result.** It does not show how well an agent finishes a task on your machine.
* **The score is not a speed.** A model with a higher score can write more slowly.

## How fast is PolyStrata on each model?

PolyStrata runs more models than setup plans. The table shows writing speed in tokens per second, at the size the PolyStrata installer takes for each model.

| Model | Laptop RTX 4070, 8 GB | RTX 5090, 32 GB |
| - | - | - |
| LFM2.5-24B-A2B | 70–73 | 480–485 |
| Flint 40B-A2.5B | 61–63 | 251–252 |
| Qwen3.6-35B-A3B | 94–100 | 394–395 |
| Ornith 1.5 | 76–82 | 316–319 |
| Gemma 4 26B-A4B | 92–93 | 353–356 |
| Tiel Coder 35B-A3B | 83 | 337–345 |
| MiniCPM-V 4.7 35B-A3B | 64 | 308–311 |

The laptop has a Core i7-14700HX and 32 GB of RAM. The RTX 5090 machine has 90 to 120 GB of RAM. Every size, the comparison with llama.cpp, and the method are in the [PolyStrata results](https://github.com/simplexai-labs/PolyStrata#how-fast-is-it).

## How well do agents finish real tasks?

Writing speed and quality scores measure the model alone. Agent task results, such as research and computer use on each machine, are not published yet.

## Keep learning

> [card] [Hardware compatibility](https://docs.heylinger.com/hardware/compatibility.md)
>
> Check whether your machine can run local models.

> [card] [PolyStrata](https://docs.heylinger.com/models/polystrata.md)
>
> Learn how a large model fits into limited memory.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.