Skip to main content
PolyStrata is the local inference engine that Linger uses by default. It fits a large mixture-of-experts (MoE) model into the limited memory of a consumer GPU (graphics processing unit) or Mac. PolyStrata is open source under the MIT license. It is a separate project, and Linger runs its local models on it. The source is at github.com/simplexai-labs/PolyStrata.

How does it fit a large model into limited memory?

An MoE model holds many sub-networks called experts, and each token uses only a few of them. Qwen3.8-Flash-Next, for example, uses 10 of its 24,576 experts for each token. PolyStrata keeps the most-used experts next to the compute, and the rest wait one level further away.
On an NVIDIA PC, CUDA memory holds the most-used experts and the CPU computes the rest from system RAM at the same time; on Apple silicon, Metal unified memory holds the model when it fits and the SSD serves the rest, with the next layer read ahead; the model can also draft a few tokens that one full pass checks, and a follow-up reuses the conversation cache.

Each token needs only a few experts, so the most-used ones stay next to the compute.

  • On an NVIDIA PC, GPU memory holds the most-used experts. The CPU (central processing unit) computes the rest from system RAM at the same time.
  • On an Apple silicon Mac, unified memory holds the model when it fits. When memory runs out, the SSD serves the rest, and the engine reads the next layer ahead.
Two more methods make replies faster. Multi-token prediction (MTP) drafts up to three tokens that one full pass checks. The conversation cache keeps earlier turns, so a follow-up reads only what is new. With these methods, Qwen3.8-Flash-Next runs on a 12 GB card. Its 3-bit model file is 55 GB, and the model has 125B parameters. Without PolyStrata, the whole model must sit on the card, which needs a GPU such as an H100 with 80 GB.

How fast is it?

On an RTX 3060 with 12 GB and 64 GB of RAM, Qwen3.8-Flash-Next writes 45.5 tokens per second. On an RTX 5090 with 32 GB, it writes 179. A token is about three quarters of a word, and 60 tokens per second is faster than you can read. A token that needs an expert from RAM or the SSD takes longer than a token that stays on the GPU. So speed depends on how much of the model fits on your card. The PolyStrata project measured these numbers with its own benchmark. On a laptop with an 8 GB card, PolyStrata read prompts 2.7 to 3.5 times as fast as llama.cpp, and wrote 1.5 to 1.8 times as fast. See Benchmarks for each model, and the PolyStrata results for how they were measured.

Does PolyStrata change the quality score?

PolyStrata does not change a model’s quality score, but it changes which model your machine can run. The SimplexAI score follows the model and its quantization, not the engine. The gain is indirect. Without PolyStrata, a 24 GB card holds a 27B model, which scores 54.0. With PolyStrata and 64 GB of RAM, the same card runs Qwen3.8-Flash-Next, which scores 59.3. Speed and score are separate numbers. The speeds on this page are measurements from the PolyStrata project. The scores come from third-party test results, adjusted for quantization. See how the score is built.

Keep learning

Local models

See which local model Linger plans for your hardware.

Benchmarks

Compare speed and quality on real hardware.