What I learned serving DeepSeek-V4-Flash as an RL rollout engine on 2 DGX Spark machines.

Two-panel throughput chart for DeepSeek V4 Flash FP8 on 2 DGX Spark. The left panel shows accepted serving settings rising from a 31 tok/s baseline to 731.5 tok/s sustained on a distinct-prompt benchmark, with an 843.9 tok/s peak. The right panel shows the shared-prompt RL workload after a decode-workspace patch scales concurrency past the old 256-sequence cap to a 1,106 tok/s peak at 768 sequences.

Figure 1. The left panel shows aggregate throughput after each accepted serving setting on the distinct-prompt benchmark, from a 31 tok/s baseline to 731.5 sustained. The right panel shows the shared-prompt RL workload after the decode-workspace patch, where concurrency scales past the old 256-sequence cap to 1,106 tokens per second at 768 sequences.

When DeepSeek-V4-Flash and DeepSpec landed in late June, I wanted to see how far I could push the hardware I had on hand.

That hardware is a local cluster of 2 DGX Spark machines. The test was serving a 284B-parameter MoE as an RL rollout engineA rollout is one generated training sample. In agentic RL, it can also mean a longer trajectory with tool calls, retrieval steps and multiple model turns. with no cloud compute.

The whole thing boils down to throughput, the rollout tokens per second the cluster can generate. In RL the model trains on data it produces itself, so a faster rollout engine is a faster learner. I wanted to find the ceiling on this hardware, then see how much of what I learned getting there still holds when you scale up.

Why rollout throughput is the bottleneck

Start with the loop. In RL post-training, the model generates its own training data. It samples a batch of responses. A verifier scores them. Then a gradient step updates the policy.

Generation is the slow part. On reasoning and agentic workloads, it is often the largest cost in the loop.

Agentic RL makes this worse. A long task can call the model many times across tool use, retrieval and web steps. A single trajectory can run to tens of thousands of decoded tokens.

Every one of those tokens is paid at generation speed. The longer the task, the more the run waits on the sampler.

Generation speed only helps if the samples stay valid for training. In RL, the policy learns from its own samples. If you distort those samples, you distort the gradient.

If you think of RL progress as a product, it decomposes into the product below.

learning speed = effectiveness × throughput

Effectiveness is how much learning signal you get from each rollout. Throughput is how much rollout and training work the system does per unit time.

Most speedups buy throughput by spending effectiveness. Asynchronous execution trains on slightly stale policy samples. Off-policy replay reuses old trajectories. Lower-precision rollouts shift the sampling distribution. Each one trades some learning quality for speed.

Speculative decoding is the exception. A small draft model proposes tokens. The large target model checks them. A rejection step throws out any token the target would not have produced.

With exact rejection sampling, the accepted tokens follow the target model’s distribution. That makes speculative decoding the first speedup to try, before anything that changes the rollout distribution.

Throughput is a systems problem. On fixed compute, it is the only term left to move. Faster throughput means faster learning.

Compute is fixed at 2 DGX Spark, 128 GB of unified memory each. What I’m optimizing for here is rollout tokens per second from DeepSeek-V4-Flash on that hardware, at the temperature I would actually train at.

What this builds on

DeepSpec is DeepSeek’s codebase for training and evaluating draft models for speculative decoding. DeepSeek released it on 26 June 2026. It packages 3 draft algorithms, DSpark, DFlash and Eagle3.

This recipe is the serving side of the same idea. I use DeepSeek-V4-Flash’s native multi-token-prediction layer as a draft head inside the model. That is the design direction DeepSpec studies with DSpark.

The framing above comes from NVIDIA’s NeMo-RL speculative-decoding report,Iso et al., Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decoding, NVIDIA, 2026. which measures speculative decoding for RL rollouts on datacenter Blackwell. The report credits the throughput decomposition to earlier work by Piché and colleagues.

I measure the same idea on consumer Blackwell, the GB10 in DGX Spark. One result comes out the opposite way.

Thinking about hardware

One constraint shapes every choice in this recipe. On this hardware, generation speed is set by memory bandwidth, not compute.

Each DGX Spark has 4 relevant constraints.

  • a GB10 chip, NVIDIA’s consumer-class Blackwell
  • 128 GB of unified memory, about 119 GB usable for serving
  • 273 GB/s of memory bandwidth
  • a 200 GbE RoCE link to the other machine

Every decode step reads the model weights from memory. On GB10, that read is the expensive part.

If you decode one token, you pay the full read for one token. If you batch 256 sequences, one weight read serves 256 next-token decisions.

That reduces the problem to one question. How many tokens can you place behind each weight read before something breaks?

This is the memory-bound side of the roofline model.A roofline model asks what limits performance first, moving data or doing math. Here the limit is moving weights from memory, not running the arithmetic. Below a critical batch size, the chip sits idle waiting for weights. Adding more concurrent work is close to free.

You leave that regime only when you pack enough tokens behind each read to saturate compute. On GB10 with a model this size, you never get close. The machine stays memory-bound across the useful range. That is why aggregate throughput is the number that moves.

That also explains the single-stream ceiling. At batch 1, you pay a full weight read per token. Decode tops out in the tens of tokens per second.

My best single-stream config for DeepSeek-V4-Flash reached 38.3 tokens per second. You cannot beat that per stream. You can only stack more streams behind each read.

Unified memory adds the second rule. The CPU and GPU share one 128 GB pool, with no separate video memory. Weights, KV cacheThe KV cache stores attention keys and values from earlier tokens. It lets the model avoid recomputing the whole prompt on every new token, but it grows with sequence count and context length. and activations all draw from it.

Every extra sequence costs KV cache. KV cache competes with the weights for the same space. Concurrency is not free. You buy it from memory headroom.

The 2 machines are not optional. DeepSeek-V4-Flash is 149 GB across 46 shards. It does not fit in one machine’s 119 GB.

I split the weights across both machines with tensor parallelism of 2.Tensor parallelism splits one model across multiple devices. Each device holds part of the weights, and the devices exchange activations while generating each token. Each machine holds about 75 GB of weights, which leaves room for KV cache.

Once you split the model, the network becomes part of every token. Each decoded token triggers a cross-machine exchange. The link between the 2 machines matters as much as the chips. Moving that link from a plain network socket to RoCE is one of the largest wins in the sequence.

One more fact explains the first failure. GB10 is consumer-class Blackwell, compute capability SM121. The Blackwell in a datacenter B200 is SM100.

A low-precision checkpoint is not portable between them by default. Its kernels are compiled for a specific tensor-core path. GB10 has less shared memory, lacks the datacenter tensor-core instructions, and has no instruction for converting to 4-bit floats.

So a checkpoint quantized for datacenter Blackwell can refuse to run here. I treat that as a compatibility check before optimization.

Last, the model. DeepSeek-V4-Flash is a 284B-parameter mixture-of-experts model.A mixture-of-experts model has many expert networks but routes each token through only a few of them. That keeps active compute lower than the total parameter count suggests. Only about 13B parameters are active for each token.

Each token routes to 6 of 256 experts plus 1 shared expert. That means each token reads about 13B of the 284B weights. This is why decode lands in the tens of tokens per second rather than far lower.

The model also ships one multi-token-prediction layer, a small draft head.A multi-token-prediction head tries to predict future tokens beyond the immediate next token. That makes it useful as an internal draft model for speculative decoding. I use that head for speculative decoding later.

The serving levers

Table 1. Throughput after each serving lever on 2 DGX Spark. Single-stream throughput measures the per-token path at low concurrency. Aggregate throughput measures concurrent rollout serving. The first row is the first configuration that served, so the 31 to 843.9 change is an end-to-end rollout-server improvement, not a tuned-baseline speedup.

Lever Measured configuration Single-stream (tok/s) Aggregate (tok/s)
Fit the model across both machines native FP8/FP4 checkpoint, TP=2, default MoE kernel, TCP transport 12.6 31
Use the GB10 MoE kernel and RoCE transport B12X MoE kernel, NCCL over RoCE 18.7 39.5
Use the model’s draft head MTP-1 speculative decoding 33.4 85
Fill the batch with rollout work concurrency 128, greedy decoding - 555
Re-test at RL sampling temperature concurrency 256, temperature 0.8, MTP-1 - 843.9 peak / 731.5 sustained

The table has 2 regimes. The first 3 rows tune the per-token path. The last 2 rows tune the rollout engine. That split matters because single-stream speed and aggregate throughput answer different questions.

Single-stream speed tells you how fast one sequence advances. Aggregate throughput tells you how much training data the server produces per second. RL cares about the second number once you have enough parallel rollouts.

Fit the model with tensor parallelism

The next question is how to fit 149 GB into 119 GB. You cannot, on one machine.

Tensor parallelism solves that by splitting one model across devices. Each machine holds a shard of the weights. During generation, the machines exchange activations and partial results so the sharded model behaves like one model.

In vLLM, I run this as a 2-node job with --tensor-parallel-size 2 and --nnodes 2.vLLM recommends tensor parallelism when a model does not fit on one GPU or one node. Multi-node tensor parallelism needs fast communication because the workers exchange data during each forward pass. One Spark runs the API server. The other runs headless as the worker. Each machine holds about 75 GB of weights.

This is the first config that ran. It served at 12.6 tokens per second single-stream, 31 aggregate. That is the floor. Everything after makes the same model faster without touching a weight.

Make the MoE kernel match the chip

The model serves, and it is slow. Which part is slow?

The expert matrix multiplications are the expensive part. A mixture-of-experts layer first routes tokens to experts, then runs the expert matmuls, then gathers the outputs. That route-pack-compute-gather path is where kernel engineering matters.

vLLM picks its MoE kernel automatically. On GB10 it picked MARLIN, a general fallback. The serving image also has a GB10-specific B12X path. One environment variable switches it on, VLLM_USE_B12X_MOE=1.

B12X does not change the model. It changes the low-level GPU program that runs the experts. Same weights, same math, different execution path.

On a new chip, check the kernel choice before you tune higher-level settings. Auto-selection optimizes for “runs everywhere,” not “fast here.”

B12X alone did not show the full gain because tensor parallelism made the network part of every token. Over a TCP socket, the faster expert kernel waited on communication.

Move cross-node traffic onto RoCE

A faster expert kernel barely moved aggregate throughput. Why?

At tensor parallelism of 2, every decoded token crosses the network. The 2 machines exchange activations on every step. A token cannot finish until that exchange finishes.

My traffic first ran over a plain TCP socket. A faster kernel does not help if its tokens wait on a slow link.

The 2 machines have a 200 GbE RoCE link for this.RoCE is RDMA over Converged Ethernet. NCCL is NVIDIA’s communication library for multi-GPU and multi-node jobs. NCCL can use that RDMA transport instead of falling back to a TCP socket.

On GB10, GPU Direct RDMA Disabled in the logs is expected.GPUDirect-RDMA lets a network adapter access GPU memory directly on systems that support it. GB10 can still benefit from RoCE transport even when GPUDirect-RDMA is disabled. The win is the RoCE transport path, not GPUDirect-RDMA.

B12X and RoCE together took the pair from 31 to 39.5 aggregate. RoCE is the largest infrastructure win in the per-token path.

RoCE inside a container needs each RDMA device passed with --device, plus --cap-add=IPC_LOCK. I started with a bind-mount of /dev/infiniband. It looks right. The devices appear inside the container.

Docker’s device cgroup still blocks the process from opening them.A device cgroup controls which host devices a container process may open. A bind mount can show the files without granting permission to use them. NCCL falls back to TCP. The logs can look healthy because the model still generates.

You have erased the win with no serving error. Read the NCCL log and confirm it reports the RoCE HCA, not No device found. A clean GB10 run can still print GPU Direct RDMA Disabled. That line is not the failure.

Use the model’s draft head

Now the model fits, the expert kernel matches the chip and the cross-node link is on RoCE. The remaining per-token question is speculative decoding.

DeepSeek-V4-Flash has one multi-token-prediction layer built in. I use it as the draft head. With speculative decoding on, that head proposes future tokens. The main model verifies them in one pass.

Accepted tokens are nearly free. Rejected ones still cost a verify. The question is how many tokens the draft head should propose.

I swept k, the number of proposed tokens, over 1, 2 and 3. One MTP layer means accuracy falls off fast down the sequence. The first proposed token is accepted about 79% of the time, the second 48%, the third 16%.

That decay sets the choice. k=2 wins single-stream latency, because a second accepted token shortens one sequence’s path.

k=1 wins aggregate. Under concurrency, every proposed token competes for a batch slot. A token that lands 16% of the time wastes that slot. k=3 and up lose everywhere.

An RL rollout engine is an aggregate-throughput problem, so k=1 wins.

MTP-1 took single-stream to 33.4 tokens per second. MTP-2 set the single-stream best at 38.3. Aggregate at low concurrency reached 85. At that point, the per-token path stopped being the limiting question. Concurrency became the next lever.

Concurrency and the speculative-decoding surprise

Raise concurrency to the wall

Once the per-token path was tuned, I had to answer the concurrency question. How many sequences can the machines hold before something breaks?

Each added sequence shares the same weight read. Aggregate throughput rises as more rollout work sits behind each read.

Two settings mattered. I set --max-num-batched-tokens 8192, because the speculative-decoding default of 2048 starves the batch. I set --gpu-memory-utilization 0.9 to give KV cache room.

I served at 128k context, not the model’s full 1M. That choice left memory for more concurrent sequences.

Past 256 sequences, the SM120 decode kernel crashes in _get_decode_scratch. It reserves a 289 MB workspace, then asks for more and hits a lock. At the time I read that as a hard kernel limit and capped concurrency at 256.

At 128 sequences with greedy decoding, aggregate throughput reached 555 tokens per second.

Re-decide at the temperature you train at

Every number so far used greedy decoding. RL does not. The policy samples at a temperature, often near 0.8, and that changes the speculative-decoding math.

So I re-ran the comparison that matters. Speculative decoding on versus off, at temperature 0.8, at both sequence caps.

Table 2. Speculative decoding at RL sampling temperature. Rows are grouped by --max-num-seqs. The benchmark used 160 client requests for the 128-sequence server and 224 client requests for the 256-sequence server.

Sequence cap Speculative decoding Sustained (tok/s) Peak (tok/s) Mean accepted length
256 MTP-1 731.5 843.9 1.81
256 off 587.9 662.4 -
128 MTP-1 434.9 590.4 1.79
128 off 339.1 498.8 -

Speculative decoding wins at both sequence caps. At 256 sequences it adds 24% sustained and 27% peak. At 128 it adds 28% sustained and 18% peak.

The winning config is MTP-1 at 256 sequences, temperature 0.8. It reached 843.9 tokens per second peak and 731.5 sustained.

This is the result I did not expect. The usual rule says speculative decoding should hurt at high concurrency. Verifying drafted tokens costs compute. At high concurrency, compute should be scarce, so verification should not pay.

That rule assumes a compute-bound GPU. On this DGX Spark setup, decode waits on weight reads. The draft’s verify tokens ride along on weights already in flight, so the extra compute is nearly free.

The same bandwidth limit that caps single-stream decode keeps speculation paying at high concurrency. On new hardware, run the comparison at the temperature and concurrency you will actually train at.

Adding DSpark

DeepSeek trained a heavier drafter for exactly this, DSparkDSpark is DeepSeek’s trained draft model for DeepSeek-V4-Flash, from the DeepSpec release. It runs natively in vLLM from PR #46995. Published checkpoint DeepSeek-V4-Pro-DSpark.. It proposes a block of tokens in parallel with non-causal sliding-window attention, then verifies them in one pass. On a large-scale cluster it reaches mean accepted length near 5, 12 to 42% higher acceptance than the MTP head across draft depths. Acceptance is the term the roofline rewards, since each accepted token rides one weight read, so a drafter that accepts 5 where the built-in head accepts 1.8 should win here too.

Running it on GB10 was the first problem. Mainline vLLM’s DSpark path disables the sparse-MLA backend on SM121, and the one community overlay that implements DSpark targets x86 RTX Pro, not aarch64 GB10. So I ported the drafter onto the working GB10 image. Serving it took disabling the FlashInfer autotuner, whose per-rank trial count desynced the ranks, and fixing a base vLLM decode-split bug that only parallel-drafting methods hit.

Once it served, I benchmarked it graphed against MTP-1 at the same concurrency.

Table 3. DSpark against MTP-1, graphed, at concurrency 224. Sustained tokens per second, with mean accepted length in parentheses.

Workload MTP-1 DSpark k=4
Distinct-prompt 748 (1.80) 418 (1.66)
Shared 1,100 prefix 512 (1.90) 400 (2.05)
Shared 4,096 prefix 512 (1.90) 417 (2.07)

DSpark runs 19 to 44% slower than the free MTP head, and the accepted-length column says why. Throughput is accepted length times decode steps per second. DSpark’s accepted length on GB10 is 1.7 to 2.1, not the 5 it reaches at scale, while k=4 makes the 284B target verify 5 positions per step against MTP-1’s 2. On distinct prompts MTP-1 runs 416 decode steps per second and DSpark 252, so the DSpark step costs 1.65x more, and it runs 1.3 to 1.65x more across the three workloads. A drafter that keeps 2 of the 4 tokens it proposes pays the full verify cost for all 4. That trade closes only when acceptance reaches 4 or 5. Here it sits at 2.

Graphed and eager DSpark measure the same, 418 either way, so a CUDA graph has no launch overhead to remove. The cost is compute per step. You also cannot dial k down to recover it. DSpark is trained for a 5-token block, and running it at k=2 hangs the multi-node draft-sample path, so a shorter block needs a retrained drafter, not a flag.

Acceptance is the whole game on a bandwidth-bound machine, and DSpark’s advantage does not transfer to GB10. MTP-1 stays the right draft head here.

Shared prompts and prefix caching

The distinct-prompt sweep sends a different prompt to every request. Real RL rollouts do the opposite. To estimate an advantage, the trainer samples many completions from one prompt, so a batch of rollouts shares a long prefix, the system prompt, the tool schemas and the task.

That shared prefix changes the serving math. The engine can compute it once and reuse the cached attention state for every completion in the group. vLLM does this automatically, and the distinct-prompt benchmark hides all of it, because no two requests share anything.

I re-ran the temp-0.8 comparison on the realistic workload, one shared agent prompt of about 1,100 tokens, 224 completions, the same winning config, prefix caching on versus off.

Table 4. Prefix caching on a shared-prompt rollout workload. Same prompt, same completions, caching toggled.

Prefix caching GPU cache hit Sustained (tok/s)
on 77.7% 495.7
off 0% 311.1

Prefix caching lands a 77.7% hit and 1.6x sustained throughput, 495.7 against 311.1 tokens per second. Without caching, every completion recomputes the 1,100-token prompt. With caching, the group pays for it once.

That makes 731.5 an upper bound. On the workload RL actually runs, the same server sustains about 496 tokens per second because the shared prompt is real work the distinct-prompt benchmark skips.

I measured one prompt length and one group size. The logs confirm the mechanism. Prefix-cache queries drop to zero with caching off. Longer prompts and larger groups should make this lever stronger, but this run only proves one point on that curve.

Patch the decode workspace

The 256-sequence cap set the aggregate ceiling, so before accepting it I wanted to know what the kernel was hitting.

The serving image includes 4 optional GB10-specific code paths, all disabled by default. They are a faster FP8 GEMM for the dense projections, a fused output projection, a sparse-attention indexer and a multi-head path. I turned each on at the winning config to see what moved. The output projection gained 1.4%, inside the run-to-run noise. The sparse indexer cost 3%. The FP8 GEMM and the multi-head path crashed at load.

That is the bandwidth limit again. All four optimize compute, and decode here waits on weight reads, so a faster kernel meets the same memory wall.

So I read the crash instead of routing around it.

AssertionError: Workspace is locked but allocation from
'sm120.py:_get_decode_scratch' requires 325.27 MB,
current size is 289.12 MB.
Workspace growth is not allowed after locking.

vLLM sizes a scratch workspace during warmup, then locks it so the CUDA graphs see stable allocations. The decode scratch grows with concurrency, and warmup had locked it at the 256-sequence size of 289 MB. At 288 sequences the kernel asked for 325 MB and hit the lock, 36 MB short with about 45 GB of memory free. A warmup helper had reserved the workspace for 64 tokens, the count it uses to pick the small-batch kernel, instead of the concurrency the server runs at.

The patch reserves the workspace for the concurrency you serve, behind an environment variable so the default stays as it was. It is 12 lines, and it changes a buffer size rather than any math.

With the workspace sized for the concurrency, throughput climbs past 256. On the shared-prompt workload at a 4,096-token prefix, the cap had held 905.8 tokens per second. The patch takes that to a peak of 1,106 at 768 sequences, 22% higher. Past 768 it turns over. The per-sequence attention and scheduling cost grows faster than the weight read amortizes, and the 273 GB/s roofline is close. For this workload, 768 is the operating point.

The patch is numerically free. I compared the model’s token distributions before and after the change, at fixed positions. The gap matches what I get comparing the unpatched server to a second run of itself, the floor that batched non-determinism sets.

I validated the patch at 128k context. At 262k the decode path fails a new way, a workspace assertion on the first request that a larger reservation does not fix. Longer contexts need their own debugging pass.

The recipe

The runnable scripts live in the companion repo, inference-recipes/inference. This post keeps the decisions and reasoning. The repo keeps the exact image tag and flags, which can drift.

You need 2 DGX Spark machines on a working RoCE link. You also need the official deepseek-ai/DeepSeek-V4-Flash checkpoint on both machines and a GB10 vLLM image with the B12X MoE path. The repo README lists the rest.

Find your network values

RoCE is the easiest thing to get silently wrong, so confirm your values first. Run preflight.sh on each machine before launch. It prints the values you need.

ibv_devices        # the RDMA device name          -> ROCE_HCA
rdma link show     # device-to-netdev, link state   -> NET_IF
show_gids          # the GID table; the RoCE v2 row -> GID_INDEX
ls /dev/infiniband # the devices passed into the container

The settings that matter

GB10-scoped settings carry to other large MoEs on this hardware. Model-scoped settings change per checkpoint.

Setting Value Scope Why it matters
VLLM_USE_B12X_MOE 1 GB10 use the GB10 MoE kernel, not the MARLIN fallback
RDMA --device and --cap-add=IPC_LOCK pass through GB10 let NCCL open the RoCE devices inside Docker
NCCL_IB_HCA, NCCL_IB_GID_INDEX your HCA, the v2 GID GB10 point NCCL at the RoCE device
--tensor-parallel-size, --nnodes 2, 2 GB10 split 149 GB of weights across both machines
--max-num-seqs 256 GB10 the stock cap; the decode-workspace patch raises the useful ceiling to 768
--max-num-batched-tokens 8192 GB10 avoid starving speculative decoding batches
--kv-cache-dtype, --block-size, --gpu-memory-utilization fp8, 256, 0.9 GB10 leave enough unified memory for KV cache
--enable-prefix-caching on GB10 reuse a shared prompt prefix across a rollout group
IMAGE, MODEL_DIR, --served-model-name the model files model the model and its GB10 serving build
--tokenizer-mode, --trust-remote-code deepseek_v4 model the model’s own tokenizer and code
--speculative-config mtp, k=1 model use the draft head; k=1 wins aggregate
VLLM_SPARSE_INDEXER_MAX_LOGITS_MB 256 model V4 hybrid attention only

Run it

git clone https://github.com/jbarnes850/inference-recipes
cd inference-recipes/inference
cp config.env.example config.env   # edit for your machines
./preflight.sh                     # print RoCE values; repeat on the worker
./up.sh                            # launch both nodes, ~5 min cold start
./rl_sweep.sh                      # run the temp-0.8 comparison
./down.sh                          # stop and free the machines when done

Cold start is about 5 minutes, limited by reading the weights off disk. Serving holds about 120 GB on each machine at 0% idle use, so tear it down between runs.

What changes when the trainer is live

Everything so far measured a static server. The weights never moved. A live async RL trainer changes that because the policy updates while the server runs.

Recent open-weight systems lean on asynchronous generationAsynchronous generation means the sampler keeps producing rollouts while training continues. The learner may train on slightly stale samples, but the hardware spends less time idle. to keep the sampler busy while the learner trains. Luke Huang’s survey of frontier async RL maps that pattern.

Three constraints start to matter. None of them break the recipe, but you have to design around them.

The draft head drifts

The 843.9 peak from the distinct-prompt sweep used the model’s frozen MTP head, measured against the base policy. RL changes the main model step by step. The head was trained to predict the old policy, so its guesses match less often as training goes on.

Acceptance falls, and the speculative-decoding win shrinks with it.

Watch mean accepted length in the vLLM logs. It starts near 1.8. Below about 1.4, verifying drafted tokens costs more than it saves. At that point, drop to no speculation.

The better fix is to keep the head in sync. Update the MTP head alongside the main weights at each resync. NVIDIA’s report calls this online draft adaptation. Then acceptance can hold through training.

Each rollout has a floor

On the shared-prompt workload, about 496 tokens per second across 256 sequences is roughly 1.9 tokens per second per rollout.

That is fine for many short rollouts in parallel. It is slow for a long multi-turn trajectory that needs tens of thousands of tokens in one sequence.

Set concurrency to your fanout, not to the maximum. The 256 cap maximizes aggregate throughput. It does not maximize any single rollout.

If your rollouts are long and few, run fewer sequences so each one gets more of the machine. Tune --max-num-seqs to how many trajectories you sample at once.

Weights have to hot-swap

vLLM supports this through an in-place weight update, with the engine briefly paused rather than restarted. That update path has to include the MTP head.

How I’d approach the next model

On GB10, throughput came down to a few simple levers.

  1. Match the MoE kernel to the chip before tuning anything else.
  2. Put the cross-node link on RoCE, and read the NCCL log to confirm it.
  3. If the model ships a draft head, turn on speculative decoding and sweep k.
  4. Raise concurrency until throughput stops climbing. If the kernel crashes instead, read the crash before you treat it as the ceiling.
  5. Re-run the comparison at the temperature you’ll actually train at.
  6. Benchmark on shared-prompt rollouts with prefix caching on, the workload RL actually runs.

The numbers move with the model and the chip. The order is what carries.

What this shows

The climb from 31 to 843.9 was systems work, not a model change.

The 12-line workspace patch matters because it moved the real shared-prompt workload from a 905.8 tok/s ceiling to 1,106 tok/s at 768 sequences.

The lesson I take forward is to tune kernels, network, speculation and concurrency against the workload the trainer actually runs.

The next recipe builds the other half, the async RL engine that runs this server and keeps trainer, sampler, and weight sync coherent.

Prompt-lookup drafting (added 2026-07-18)

vLLM ships ngram speculative decoding. It matches the last few generated tokens against the context and proposes the tokens that followed the last match. Drafting is a string search. No draft model, no extra weights.

Table 5. Prompt-lookup ngram against MTP-1 at temperature 0.8. Sustained tokens per second, mean accepted length in parentheses. Ranges span replicate runs of identical configs. Rebuilt harness, not comparable to the tables above. Shared-prompt rows are warm repeat bursts, 768 completions over a ~3,400-token shared prompt.

Workload MTP-1 ngram k=3
Distinct-prompt 745 to 761 (1.8) 518 (1.86)
Shared prompt, warm 784 to 931 (1.8) 949 and 976 (1.7)

On the rollout workload the worst ngram run beat the best MTP-1 run. On distinct prompts ngram loses a third of throughput. The DSpark arithmetic cuts the other way here. Verifying 4 positions at accepted length near 1.9 loses on novel prompts and wins when completions echo a long shared prompt.

For RL the bigger property is that there is no drafter to drift. MTP acceptance decays as training moves the policy away from the frozen head.The Qwen team measured the decay directly. MTP acceptance is bounded by policy entropy, which rises during RL. Breaking Entropy Bounds, 2026. A string search has nothing to go stale, so the win holds through training.