Serving Qwen3.8-27B on One DGX Spark

I wanted one NVIDIA DGX Spark to serve two versions of Qwen3.8-27B:

qwen3.8-27b             clean mixed-NVFP4 base
qwen3.8-27b-mode-b      optional per-request adapter

The obvious implementation worked functionally and failed architecturally. An always-on runtime projection hook made every request—including a nominally clean request with projection strength set to zero—enter a modified CUDA graph. Single-request throughput fell from roughly 45 to 30 output tokens per second.

The eventual fix was not another conditional branch in the runtime. It was to rewrite the projection as a standard rank-1 PEFT LoRA and let vLLM’s native per-request LoRA machinery do the routing. A request with no adapter now stays entirely on the clean path. A request selecting the second model ID loads an 8.3 MiB adapter inside the same process.

The result on one DGX Spark was:

• clean base through the previous always-on hook: 30.32 output tok/s;
• clean base through the native-LoRA server: 45.41 output tok/s;
• adapter request through that same server: 31.92 output tok/s;
• clean fast-path recovery: +49.8% relative to the previous hook.

This article explains the algebra, quantized-weight conversion, DFlash2 configuration, cache boundary, benchmark protocol, and failures behind those four numbers. The complete sanitized implementation is available in liuzl/qwen38-dgx-spark-lab.

This is an independent community experiment, not an official Qwen, NVIDIA, vLLM, SGLang, RadixArk, or Z Lab project. It publishes code and aggregate measurements—not model weights, draft weights, source transformation artifacts, generated adapters, credentials, or private evaluation records.

Why one process matters on a DGX Spark

The DGX Spark combines a GB10 Grace Blackwell system with 128 GB of coherent unified memory. That is enough to make a 27B dense model practical on a desktop, but it is not enough to be careless about duplication.

The measured vLLM process used about 46 GiB:

target checkpoint       20.42 GiB
DFlash2 drafter          3.58 GiB
FP8 KV cache             16.00 GiB
CUDA graphs               2.06 GiB
runtime workspaces        ~4.0 GiB

The adapter itself was only 8.3 MiB. Running two complete serving engines merely to expose two serving modes would duplicate the expensive target, draft model, KV pools, CUDA graphs, and runtime workspaces. Native LoRA makes the second identity almost free by comparison.

The reference stack uses:

• RadixArk/Qwen3.8-27B-NVFP4 as the target;
• z-lab/Qwen3.8-27B-DFlash2 as the speculative drafter;
• vLLM commit f94666b60;
• DFlash2 probabilistic sampling with draft depth 7;
• a 131,072-token maximum context and a 16 GiB FP8 E4M3 KV cache.

Everything is pinned because a checkpoint name, a nightly container tag, or a compiler cache is not an experimental identity. Revisions and hashes are.

The failed implementation: a clean request on a dirty graph

The optional transform starts as an output-space projection applied after selected linear sublayers:

Here $r$ is a direction in residual space, $c$ is the coefficient for a particular module, and $\lambda$ controls the strength of the edit. The direction set used in this experiment covers 128 residual-writing modules:

• 48 Gated DeltaNet linear_attn.out_proj modules;
• 16 full-attention self_attn.o_proj modules;
• 64 MLP down_proj modules.

The first per-request implementation exposed a scalar adapter control through the three API surfaces and included it in the cache salt. Functionally, it did what it was supposed to do: base and adapter outputs were stable, the aliases did not cross-contaminate their prefix cache, invalid values returned HTTP 400, and real Claude/Codex-style tool loops worked.

But the implementation allocated the projection buffer and used the projected graph for all requests. Setting $\lambda=0$ removed the numerical edit; it did not restore the clean execution path. In the matched C1 workload:

That was the key diagnosis: a zero coefficient is not the same thing as no adapter path. The clean model identity needed to bypass the modified graph, not merely multiply its modification by zero.

Turning the projection into a rank-1 LoRA

For a linear map $y = Wx$ and fixed $\lambda=1$:

Therefore:

$B$ has one column and $A$ has one row. This is exactly a rank-1 LoRA. The converter emits one $A/B$ pair for each edited module and packages them as a standard PEFT adapter.

The algebra is simple. The checkpoint is not.

The measured RadixArk model is mixed precision rather than uniformly four-bit:

The conversion code reads static-FP8 output weights directly. For packed NVFP4 MLP projections it decodes E2M1 nibbles, applies per-block FP8 scales and the global scale, and accumulates $r^T W$ in row chunks. It never materializes an entirely dequantized 27B model in memory.

There is also an important numerical qualification. A post-linear projection and a weight-space LoRA are identical for an ordinary linear map. Quantized activation kernels add rounding, so the two implementations are not assumed to be bit-identical. Algebra proves the form of the transform; private adapter and capability evaluation still have to qualify the artifact.

Final serving topology

The final topology is deliberately boring:

1. one vLLM process loads the mixed-NVFP4 Qwen3.8-27B target;
2. the same process loads the DFlash2 draft checkpoint;
3. an 8.3 MiB PEFT adapter is registered as a second model ID;
4. base requests carry no LoRA mapping;
5. adapter requests carry vLLM’s native LoRA identity.

This matters for two independent reasons. First, a base request never enters an adapter graph, which restores the clean CUDA path. Second, vLLM includes native LoRA identity in the prefix-cache key, which gives cache isolation without a custom salt protocol.

Why probabilistic K7, not the apparent defaults

The DFlash2 configuration was not an incidental detail. Early testing with greedy drafting and depth 10 achieved only 25.62 tok/s, with an acceptance length of 2.97. Explicitly switching to probabilistic drafting and depth 7 produced 45.49 tok/s and an acceptance length of 5.15 on the same single-request workload—a 77.6% difference.

Depth 10 remained in the low-acceptance regime even after switching the draft sampler. More speculative tokens are useful only when the target accepts them. Draft depth is a workload- and checkpoint-dependent search problem, not a monotonic speed knob.

The adapter path also reveals the cost of distribution mismatch. The target is changed by LoRA while the drafter is not. Its C1 acceptance length falls from 5.13 to 3.74, and throughput follows it from 45.41 to 31.92 tok/s. The adapter’s arithmetic is tiny; the expensive part is accepting fewer drafted tokens.

Measured performance

The C1 workload requested 512 random input tokens, rendered as 526 actual tokens, and generated 2,048 tokens with ignore-EOS, concurrency 1, temperature 0, and thinking disabled. Five repetitions established the clean baseline. The native-LoRA table reports a matched post-warmup qualification run.

The C8 workload sent 32 random requests at concurrency 8. Each requested 1,024 input tokens and 256 output tokens; the tokenizer produced approximately 1,036 input tokens per request.

Three interpretation rules matter:

1. The +49.8% headline compares the native-LoRA base request with the previous always-on hook. It is not a claim that this server is 49.8% faster than vanilla vLLM, SGLang, every DGX Spark recipe, or a newer checkpoint.
2. C1 is single-request output throughput. C8 is aggregate output throughput across eight concurrent requests. They should not be mixed.
3. The C8 native-LoRA values are single qualification runs, not confidence intervals. The adapter result being higher than base at C8 should not be generalized without more repetitions and workload variation.

Cache isolation: performance is not the only fast path

A shared prefix cache is useful only if semantically different model identities never reuse one another’s KV blocks. The validator sent the same long prefix across both aliases and measured vLLM’s cache-hit counters:

The first cross-identity adapter request received zero hits even though the base prefix had just been cached. Repetition within the adapter identity then hit the expected 3,296 tokens. This is the behavior we want: reuse inside an identity, isolation across identities.

API and agent validation

Serving a model that wins a local throughput benchmark but breaks clients is not a deployment. The public validator qualifies both aliases across:

• OpenAI Chat Completions;
• OpenAI Responses;
• Anthropic Messages;
• forced tool calls on all three APIs.

The internal qualification also exercised real CLI-agent tool loops. This matters because protocol shims fail in subtle ways: a chat endpoint can return text correctly while losing a forced tool call, a reasoning block, a streaming field, or an Anthropic compatibility requirement.

Private adapter qualification

The optional adapter passed the internal compatibility and regression gates defined for this experiment, with no empty responses or request errors in the binding run. Public material deliberately exposes only a coarse pass signal; it does not describe the private suite composition, category breakdown, rules, or raw outputs.

That result is not a general capability certificate. Any transformed model still needs normal task, tool-use, domain, privacy, reliability, and application-specific evaluation before deployment.

Failures that shaped the final design

The clean result emerged from several failures worth retaining:

• Compile cache identity was incomplete. Upstream DFlash/DSpark cache keys did not include speculative depth, so different K values could reuse incompatible compiled artifacts. Each experimental shape received its own cache root, and the public image applies a fail-closed overlay.
• DFlash2 construction needed a pinned patch. The tested nightly’s shared constructor selected an ordinary DFlash layer and omitted attention_conv. Pinning the container digest and overlay anchors made drift visible instead of silently producing a different runtime.
• Autotuning can consume the last safe memory margin. A large automatically allocated KV cache allowed loading but caused FlashInfer autotune to hit driver OOM. The stable reference uses an explicit 16 GiB KV cache and disables FlashInfer autotune.
• Old services are real memory leaks. A forgotten llama-server still occupied roughly 21.7 GiB. Later, simultaneous SGLang and vLLM processes pushed allocation close to 100 GiB and triggered an NVRM OOM. Process inventory is part of model deployment.
• A parameter equal to zero is not a bypass. This was the decisive architectural lesson. If clean-path performance matters, prove that the clean request does not enter the modified graph.

A new BF16 lm_head checkpoint

On the same day this reference was published, RadixArk released Qwen3.8-27B-NVFP4-BF16-LMHead. It replaces only the quantized lm_head with the original BF16 weights; all other tensors are reported identical to the source NVFP4 checkpoint.

That is a different experimental identity. The public model card currently reports GSM8K 96.13% for the BF16-head checkpoint versus 96.36% for the source checkpoint under its stated sampling protocol, describing the difference as within single-run noise. Other workloads may behave differently. I therefore make no accuracy or throughput claim for it here.

Before updating this recipe, it needs a fresh end-to-end qualification:

1. converter compatibility and hashes;
2. DFlash2 acceptance across K values;
3. C1/C8 throughput and TTFT;
4. capability and tool-use evaluation;
5. private adapter qualification;
6. cache isolation and a soak run.

This is precisely why the article and repository name exact revisions rather than treating a Hugging Face repository name as immutable.

Reproducing the reference

The public repository contains a pinned ARM64 Docker build, the mixed-quantization converter, the serving script, API and cache validators, benchmark harness, and sanitized JSON results.

At a high level:

git clone https://github.com/liuzl/qwen38-dgx-spark-lab
cd qwen38-dgx-spark-lab

docker build -t qwen38-vllm-dflash2:lab docker/

# Obtain and review the target, drafter, and compatible source artifact.
# Then convert the source transform into a native PEFT adapter.

cp configs/qwen38-spark.env.example .env
$EDITOR .env
set -a; source .env; set +a
scripts/serve-native-lora.sh

python3 scripts/validate-apis.py \
  --base-url http://127.0.0.1:18102 \
  --base-model qwen3.8-27b \
  --adapter-model <adapter-model-id>

python3 scripts/validate-cache-isolation.py \
  --base-url http://127.0.0.1:18102 \
  --base-model qwen3.8-27b \
  --adapter-model <adapter-model-id>

The repository does not redistribute weights, draft checkpoints, source transformation artifacts, or generated adapters. Repository-authored code is Apache-2.0; third-party model and artifact licenses remain their own. A generated adapter is derived from both the base checkpoint and its transformation input, so anyone redistributing one must independently establish that right.

What I would carry to the next deployment

This experiment is nominally about Qwen3.8, DFlash2, and DGX Spark. The reusable lessons are more general:

• Fast-path preservation is an architectural property. Measure it; do not infer it from a zero-valued parameter.
• Speculative decoding is governed by acceptance, not draft depth alone. Report acceptance length alongside throughput.
• Model identity must include adapters, quantization, draft model, sampler, software commit, and cache state. A model name is not enough.
• Cache isolation is correctness, not merely optimization. Validate first-cross-arm misses and same-arm reuse.
• An 8 MiB adapter can avoid duplicating tens of GiB of serving state. The scheduling abstraction matters more than adapter arithmetic.
• Publish limits with the benchmark. One machine, one pinned stack, no 24-hour soak, and single-run C8 figures are part of the result.

The useful outcome is not simply “45 tokens per second.” It is a service whose clean path remains genuinely clean, whose second serving mode has a native identity, and whose performance and isolation claims are reproducible rather than implicit.

Links

• Code and sanitized results: liuzl/qwen38-dgx-spark-lab
• Base model: Qwen/Qwen3.8-27B
• Measured target: RadixArk/Qwen3.8-27B-NVFP4
• DFlash2 drafter: z-lab/Qwen3.8-27B-DFlash2
• New checkpoint requiring separate qualification: RadixArk/Qwen3.8-27B-NVFP4-BF16-LMHead