Qwen 3.8 Flash Next on 2 RTX 3090s + 128 GB of RAM: QEngine, Strata, vLLM or llama.cpp?

In 30 seconds

On my machine, QEngine remains the default engine, and Strata is installed alongside it for large contexts.

Why this model changes the comparison

QFN is not a dense model that you quantize and squeeze into the card. It is a MoE of about 125 billion parameters (512 experts, 10 activated per token, i.e. ~6 billion active), plus an n-gram table of 51.2 billion: 176 billion in total. That doesn't fit in my 48 GB of VRAM. So part of the experts and the table live in the machine's RAM, and the engines differ in how they move them around.

If you're building a machine for this model, RAM and the PCIe generation matter more than the exact card model. VRAM per card remains the entry requirement, not the deciding factor.

A few terms to know for what follows:

• tok/s: tokens generated per second (decoding). Human reading speed is around 6 tok/s: the difference is only really felt on long generations and agents.
• Prefill: the speed at which the engine reads your prompt. It determines the wait before the first word on a large context.
• TTFT: the time before the first response token.
• Drafter / MTP: a small model proposes several tokens ahead, the big one validates them. This speeds up decoding.
• P2P: direct transfer from one card to the other, without going through RAM. My cards don't have it, which matters later.

My machine

2 RTX 3090s with 24 GB each (one of them a Ti), an AMD EPYC 7401P with 24 cores, 125 GB of DDR4, PCIe 3.0 on both cards, and no P2P between them: the driver refuses it.

The most flattering published figures come from PCIe 4 rigs with P2P enabled. Without P2P, all communication between cards goes through the machine's memory, which is costly for a MoE that calls on thousands of experts per step. That's why my figures are lower than those on the spec sheets: QEngine advertises 165 tok/s in narration on 2×3090 with P2P, and I measure 136.

The 4 engines in one table

The fastest engines of the moment (capy-flash, Basalt, Infernix) target the RTX 50 series and NVFP4, which a 3090 can't decode: they're out of the running on Ampere.

QEngine vs Strata: same model, same machine, same scripts

Before talking speed, both passed the same functional tests: tool calling, streaming (including streamed tool calls), reasoning mode, vision 4/4, and a context filled up to 240K tokens (91% of 262,144), with the needle found at every step. The weights are close in fidelity: about 89% agreement with the unquantized model according to the spec sheets (BF16 reference on one side, Q8_0 on the other). Nothing in my tests points to a quality gap: it's a trade-off between speed and memory.

Gaps are expressed from Strata's point of view. QEngine here is in the default Docker configuration: its prefill gains about 17% by fixing the memory placement, which brings the two engines to the same level at 10K (I explain how in the next article).

Decoding. QEngine wins on a short prompt (+38%) and at 87K (92 vs 79 tok/s), but the two are tied at 10K (84 vs 85). In real-world use, at 80-100K tokens of context, I observe 78 to 127 tok/s under QEngine, with 99% of the cache reused.

Long context. Strata wins, and not only at 90K. On the 94K to 240K scale, it holds around 3,100 tok/s of prefill at every step, whereas QEngine fluctuates between 1,600 and 2,500. At 240K, only ~120 MiB of free VRAM per card remains under QEngine, versus a constant 434 MB under Strata. This is the regime where QEngine documents a graph instantiate out of memory.

RAM and concurrency. Strata reads its n-gram table (28.8 GB) directly from the NVMe instead of pinning it in RAM (51.8 GB with QEngine): 31 GB more RAM available. The price: only one request at a time, because its drafter only handles one request. An agent that launches sub-agents in parallel has to queue, whereas QEngine serves 8. This isn't a setting, it's structural.

The license. QEngine is under PolyForm Noncommercial 1.0.0: look at it closely if your service goes beyond personal use. Strata is MIT.

In one sentence: one generates text much faster (136 tok/s vs 99 in narration), the other swallows a 90K context much faster (29 s of waiting vs 50 s). The right choice therefore depends on what you do with the model, not just its advertised speed.

The other 2, briefly

vLLM. DominikBucko's recipe (pinned and patched vLLM, albucino W4A16-FP8PLE checkpoint of 116 GB) holds 262K of context at 80-89 tok/s. It's reliable and validated, but about 2 times slower, with a startup around 550 s and --disable-custom-all-reduce for lack of P2P. A lesson that holds even if you stay elsewhere: on another rig, without P2P, pipeline parallel beats tensor parallel (117/163 tok/s vs 45/63).

llama.cpp. Support merged upstream on 27/08 (b10675), GGUF from Unsloth and ISTA. On a profile close to mine (PCIe 3.0, DDR4), the published series climb from 17 to 37-41 tok/s with UD-Q4_K_XL, an expert cache of 150 slots and MTP-3, and up to 49 in code. Moving the expert cache off the GPU during prefill multiplies the latter by 2.2 to 2.5. It's 3 to 4 times slower than QEngine or Strata, but often the only choice on a single-GPU machine.

The pitfalls I ran into

No P2P: the real ceiling. QEngine's log says it: no peer access: collectives through pinned host memory. QEngine has handled this case since 1.3.4 (announced loss of about 4% in decoding), but the gap between rigs with and without P2P remains in the order of 12 to 18%. The leads are in the BIOS: Above-4G Decoding, ReBAR, ACS and IOMMU. Check with nvidia-smi topo -m.

Memory placement. On my 4-NUMA-node EPYC, pinned memory was spread out at random, with 51% on nodes without a card. Interleaving it across the 4 nodes gave +17% prefill and finally reproducible startups. Everything is detailed in the next article.

Strata's installer forces --ple-io ram. As soon as RAM reaches model + table + 16 GB, the installer adds this flag itself on an SSD, and STRATA_PLE_AUTO_RAM=0 doesn't prevent it. The fix: set user_args: ["--ple-io","direct"] in the pack's launch JSON file. It takes precedence over args and survives restarts. Measured effect: RAM used from 99 to 72 GB, available RAM from 29 to 57 GB. To be redone after any change to CONTEXT or KV, which relaunches the installer.

Very long context. On the Strata side, the constraint is input + max_tokens ≤ max-context: beyond that, it's an HTTP 400, with no truncation. On the QEngine side, cap the served context at ~200K, or set FLASH_EC_MARGIN_MB=1024 if the graph instantiate out of memory appears (the margin costs about 10% of decoding, so I prefer the cap).

Let it warm up. At startup, the expert cache starts at 20-30% hit rate and climbs to 63% after about 700 decoding steps. The bench therefore does 3 warm-ups before the measurements: on Strata, the first code run comes out at 81 tok/s, then the warm-ups bring it up to 111-116.

My QEngine settings

To copy as-is if you start from the preset:

FLASH_MAX_SEQS=8
ENABLE_THINKING=true
REASONING_EFFORT=low
VISION=1
KV_OFFLOAD_DISK=0

KV_OFFLOAD_DISK=0 is the right choice as long as the n-gram table is pinned in RAM. I also keep the dashboard and metrics on loopback only: it has no authentication and exposes a POST /api/reset.

Sources and method

Everything "measured" comes from my bench: bench.sh for throughput and prefill (3 warm-ups then 5 measurements), verify-full.sh and verify-stress.sh for the functional tests and the context scale, with the same scripts on both engines. Strata was at 0.2.12 (0.2.13 came out at the same time).

The "published" figures come from the BENCHMARKS.md of the QEngine presets repository, DominikBucko's recipe for vLLM, llama.cpp PR #27742 and the published bench series around llama.cpp, as well as discussions on r/LocalLLaMA, V2EX, Juejin and CSDN. A throughput without quantization or context length is worthless: always compare at equal weights and context.