Let the small model read
KV-cache transfer from small to big models: which layers, which architectures, how long a context
In How do we transfer context between models?, OneTriangle explore an interesting idea: a smaller model like Qwen3.5-4B can handle prefill for a bigger model. Since the smaller model needs less compute, this can save a lot of prefill time.
That got me thinking: if a smaller model can handle prefill, we could do that work with a smaller compute budget and fit in more requests. That could help boost throughput for a given amount of compute.
I wanted to see whether this idea works in practice, so I started by reproducing the recipe. From there, I tried a few different model pairs and ways of sharing the work:
- Reproduce the recipe on a simpler Qwen model pair and see whether it works
- Can this work with a hybrid DeltaNet model?
- What happens if I transfer only some layers?
- What if the big model reads the end of the prompt itself?
- Does it hold up on long contexts?
- Can a dense model prefill for a mixture of experts?
- What happens across model families and tokenizers?
Before getting into those experiments, here is a quick primer on the KV cache and what gets transferred.
A quick primer on KV cache
Generating a response has two parts: prefill and decode.
During prefill, the model processes the prompt tokens. In each attention layer, it computes queries, keys and values. Each token uses its query to attend to the keys and values of the tokens it is allowed to see. The model saves the keys and values for the prompt in a KV cache, and uses the prefill output to predict the first response token.
During decode, the model generates the rest of the response one token at a time. At each step, it processes the latest token and computes its query, key and value. It uses the new query to attend to the cached keys and values from the prompt and earlier generated tokens, then adds the new key and value to the cache. That way it can use the earlier context without recomputing its keys and values at every step.
For a long prompt, prefill can take a while because the model has to process all those prompt tokens before it starts generating. A bigger model generally needs more compute for that work. The idea behind cache transfer is to let a smaller model do that reading, then use a learned mapper to turn its states into a cache the bigger model can use.
The recipe: a small model reads for a big model
OneTriangle describe this recipe in their cache-transfer post. To reproduce it, we freeze both models and train a mapper: a set of linear maps that turns the small model’s states into keys and values for the big model’s cache. We build separate maps for the keys and values of each target layer. The layer-selection method and settings below describe our implementation.
Stage 1: fit a direct map
First, we run both models on the same text and collect the keys and values their layers produce. For each big-model layer, we test how well each small-model layer can predict its keys. We use ridge regression to fit a linear map: it looks for a close match while keeping the map’s weights from becoming too large. We select the three small-model layers with the lowest prediction error, combine their outputs, and fit a new map using all three. We repeat the process separately for values.
The picture below follows one of those maps. Each block represents a layer; the counts are illustrative. The highlighted small-model layers supply the input for one big-model layer. We repeat this for every target layer to build its cache.
These steps build the initial ridge maps. The numbers match the comments in the Code tab.
- Collect examples. Run both frozen models on the same training text. From each layer, collect the keys before positional encoding (RoPE) and the values. These give us examples of what the small model produces and what the big model’s cache should contain.
- Set up the fit. Ridge regression finds a linear map that predicts the big model’s keys or values from the small model’s corresponding keys or values. It also penalizes large weights to make the fit more stable. The Code tab shows how we set the strength of that penalty.
- Choose what to predict. Start with the keys of one big-model layer. Follow steps 4–6 to build its key map, then repeat those steps for its values and for every other big-model layer.
- Test each small-model layer. Fit a separate map from that layer’s keys or values to the corresponding keys or values of the big-model layer. Measure how far the prediction is from the big model’s actual keys or values, relative to their size. We calculate this error on the same text used to fit the map.
- Combine the best three. Pick the three small-model layers with the lowest errors. Put their keys or values together and fit a new map that uses all three to predict the big-model layer’s corresponding keys or values.
- Keep the result. Save the selected layer numbers and the new map for this big-model layer. When we use the mapper later, those numbers tell us which small-model outputs to combine.
The Code tab follows these same steps. The experiment code computes the fit in batches to avoid keeping every model output in memory.
Illustrative Python: collect_kv runs the frozen model and returns per-layer matrices; the regression and selection below mirror the experiment.
import numpy as np
# 1. Run both models on the same text. Collect each layer’s keys
# before RoPE and values, keeping the corresponding token outputs aligned.
source = collect_kv(small_model, training_tokens)
target = collect_kv(big_model, training_tokens)
# 2. Ridge fit with the experiment's scaled regularization.
def ridge_fit(X, Y):
n = X.shape[0]
gram = X.T @ X / n
cross = X.T @ Y / n
strength = 0.001 * np.diag(gram).mean()
return np.linalg.solve(gram + strength * np.eye(gram.shape[0]), cross)
maps = {}
# 3. Build separate maps for keys and values at every target layer.
for kind in ("keys", "values"):
for t in range(len(target[kind])):
Y = target[kind][t]
errors = []
# 4. Fit and score each small-model layer on the same tokens.
for X in source[kind]:
W = ridge_fit(X, Y)
error = np.sum((Y - X @ W) ** 2) / np.sum(Y ** 2)
errors.append(error)
# 5. Combine the three best layers and fit a new joint map.
selected = sorted(np.argsort(errors)[:3].tolist())
X_selected = np.concatenate([source[kind][s] for s in selected], axis=1)
W0 = ridge_fit(X_selected, Y)
# 6. Retain both the source indices and weights for inference.
maps[(kind, t)] = {"source_layers": selected, "weights": W0}
We can use this ridge map as it is. In the Qwen3-0.6B → Qwen3-8B reproduction below, it reaches a KL of 0.42, compared with 0.73 when the big model gets no prefix. That is a useful starting point, but there is still room to improve.
Stage 2: train a correction
Next, we improve the ridge map by adding a trainable correction: W = W₀ + A·B. The two smaller matrices A and B let us adjust the map without training a full new matrix. Their product has rank 64, and B starts at zero, so training begins with the ridge map.
We start with documents from FineWeb-Edu and tokenize them before training. We keep documents long enough for the largest split: at least 2,176 tokens. At each training step, we choose a prefix length between 512 and 2,048 tokens and sample documents from the training set. For each document, we take that many tokens from the beginning as the prefix, then take the next 128 tokens as the continuation. The prefix gives the models context, and the continuation gives us a way to compare their predictions.
The teacher is the big model processing both parts with its own cache. For the student, the small model reads the prefix and the mapper builds a cache for it; the big model then processes the continuation using that mapped cache. We compare what the teacher and student predict at each continuation position, and train the correction to bring those predictions closer using KL divergence. Both language models remain frozen.
We use AdamW at a learning rate of 2e-4, with 8 documents per step for 1,000–1,500 steps. At inference, the maps use bf16. In the same Qwen reproduction, this second stage brings KL down from 0.42 to 0.13.
The numbered steps match the comments in the Code tab.
- Start from the ridge map. Keep W₀ fixed and add a rank-64 correction A·B. Initialize A randomly and B to zero, so the correction initially changes nothing. Freeze both language models.
- Prepare documents. Tokenize the FineWeb-Edu documents and keep those with at least 2,176 tokens. At each step, sample training documents and choose a prefix length between 512 and 2,048 tokens. Take that many tokens from the start of each document, followed by the next 128 tokens as the continuation.
- Run the teacher. Let the big model read the full document and record its predictions over the continuation. We do not update the teacher.
- Run the student. Let the small model read the prefix and use the mapper to build the big model’s cache. Then let the same big model read the continuation using that mapped cache and record its predictions.
- Compare and update. Measure the KL divergence between the teacher’s and student’s continuation predictions. Backpropagate through the student and mapper, and update only A and B with AdamW. Repeat for the training steps.
Illustrative PyTorch based on the training loop. The named helper functions represent model-specific cache handling; the real code also batches documents and schedules the learning rate.
import torch
import torch.nn.functional as F
# 1. Freeze the models and ridge weights; train only A and B.
freeze(small_model, big_model, mapper.W0)
initialize_random(mapper.A)
initialize_zero(mapper.B) # W = W0 + A @ B starts at W0
optimizer = torch.optim.AdamW([mapper.A, mapper.B], lr=2e-4, weight_decay=0)
for document in training_documents:
# 2. Sample tokenized training documents. Choose a prefix length;
# take that many tokens from the start and the next 128 as continuation.
prefix, continuation = split_document(document, prefix_length=(512, 2048),
continuation_length=128)
# 3. Teacher: big model reads prefix + continuation.
with torch.no_grad():
teacher_logits = continuation_logits(big_model, prefix, continuation)
# 4. Student: small model reads the prefix; mapper builds the cache.
# Keep gradients through the mapper and the frozen big model.
mapped_cache = map_prefix_cache(small_model, mapper, prefix)
student_logits = continuation_logits(big_model, continuation,
past_key_values=mapped_cache)
# 5. Compare distributions over continuation tokens; update A and B.
teacher_prob = F.softmax(teacher_logits, dim=-1)
student_log_prob = F.log_softmax(student_logits, dim=-1)
loss = F.kl_div(student_log_prob, teacher_prob, reduction="none")
loss = loss.sum(dim=-1).mean()
optimizer.zero_grad()
loss.backward()
optimizer.step()
For clarity, mapper.A and mapper.B stand for all trainable correction matrices across the maps.
Every experiment below is measured the same way:
- Fidelity: How closely the big model’s predictions with the transferred cache match its predictions with its own cache. We report KL divergence (lower is closer) and top-1 agreement (higher is closer), on 64 held-out FineWeb-Edu documents with a 2,048-token prefix and a 128-token continuation.
- Tasks: SQuAD (300 questions, paragraph hidden among distractors, ~4.2k tokens), HotpotQA (300, distractor setting padded to ~4.5k), and GSM8K (400, 8-shot chain of thought, ~1.1k tokens). The documents or exemplars are the transferred prefix. The question header is fed to the big model normally.
- Speed: wall-clock prefill at batch 1, best of 3, unless stated otherwise.
Experiment 0: reproducing the recipe on Qwen3-0.6B → Qwen3-8B
I started by reproducing the method with Qwen3-0.6B and Qwen3-8B. The ridge map alone gets a KL of 0.42; giving the 8B no prefix at all gives 0.73. Adding the KL-trained low-rank correction brings it down to 0.13. That second step makes a big difference.
Rounds 1–3 ran on NVIDIA RTX PRO 6000; round 4 ran on a different machine. Fidelity used 128 held-out documents there and 64 in round 4. Task prompts and sample counts (300 / 300 / 400) are identical across every experiment in this post.
"Task post-training" continues the KL training on task-shaped prompts (no labels: the teacher is still the 8B). "Closed book" is the 8B answering with no context at all, which is the floor any transferred cache must beat.
A few things stood out from this first set of runs:
- Extractive QA is easy; reasoning is not. SQuAD survives transfer almost perfectly. Multi-hop QA and GSM8K, where the answer is computed rather than copied, lose a quarter or more.
- The baseline that matters is “small model alone”, not “closed book”. With Qwen3-4B as the reader, the 4B on its own scores 0.915 on GSM8K and the transfer 0.877. The comparison only makes sense when the small model is clearly worse than the big one at the task.
- Values, not keys, are the bottleneck here. Giving the student the exact keys changes nothing (0.135 → 0.134), while exact values drop KL to 0.094. OneTriangle report the opposite for their pairs, so this is probably pair-dependent.
On the new machine I re-ran the exact round-3 command. The fidelity KL came back at 0.129 against the original 0.128. Task scores moved by ±0.02–0.04 between independent training runs, which is the noise floor to keep in mind for every table below.
Experiment 1: partial-layer transfer
The reproduction worked, but the gap on reasoning made me wonder if the 8B needed to do some of the reading itself. Qwen3.5 mixes attention layers with linear-attention layers, which gave me a place to start: do we need to transfer every layer? Maybe the big model can compute some layers itself and use the small model’s cache for the rest. That would let us trade some prefill speed for better quality.
To try this, I let the big model run over the prompt and replace each transferred layer with a stand-in:
- The stand-in writes mapped K, V into the cache and skips the layer’s compute.
- A run of transferred layers followed by a native layer leaves a hole in the residual stream. A second mapped term, Δ, fills it: a prediction of how much the skipped block would have changed the hidden state.
- Each plan gets its own ridge init and its own 1,500 steps of KL training, exactly as in Experiment 0.
Plan names describe which of the 8B's 36 layers are transferred:
last:Ntransfers the top N; the 8B computes its own bottom 36−N.first:Ntransfers the bottom N.every:Ktransfers every K-th layer, so layer 0 stays native.keep:Kkeeps every K-th layer native, so layer 0 is transferred.range:A-Btransfers layers A to B−1./nodeltadrops the Δ term, which turns the plan into pure layer skipping.
Speedup is the 8B's own prefill time divided by (0.6B prefill + mapping + the 8B's native layers), in HF, bf16, at batch 1. Each plan is ridge-initialised, then KL-trained for 1,500 steps with rank 64, and evaluated on the same 300 / 300 / 400 prompts.
last:N from 6 to 36 transferred layers. Squares are first:N, triangles are interleaved plans, hollow circles have no Δ. Click any point to inspect it.Here is what I found:
- Bottom layers must stay native; top layers transfer well. Along
last:N, GSM8K climbs smoothly as more of the bottom is computed natively: 69% → 76% → 88% → 94% → 97% of the 8B for 30 / 21 / 18 / 15 / 12 transferred layers. Speedup falls 2.4× → 1.55× → 1.35× → 1.23× → 1.14× along the way. Transferring the bottom half instead (first:18) is a disaster on GSM8K: 13%. - The trade-off looks most useful around 15–18 native layers out of 36. Above that you get most of the quality back. Below about 9 native layers, nothing beats transferring everything: you pay for the 8B’s layers and still sit near the all-transferred quality.
- Interleaving works only if layer 0 is native.
every:2(layer 0 native) lands right on thelast:18point, 87% GSM8K at 1.31×.keep:2(layer 0 transferred) drops to 59%. Layer 0 is where the attention sink lives, and errors there propagate through everything above it. - Δ is not optional. Without the mapped residual, interleaved plans sit around 50% on every task. That makes them worse than transferring everything, at a similar speed.
One caveat: these speed numbers are for 0.6B → 8B in HuggingFace. Here, the small model and mapper make up a sizeable part of the cost. The trade-off may look different with a bigger target, where each native layer costs more.
Experiment 1b: letting the big model read the tail
Transferring fewer layers helped, but it also gave up some speed. That made me think about splitting the work another way: by tokens. Instead of transferring the whole prompt, the big model can read the last part itself, using the mapped cache for everything before it. In round 2, letting the 8B prefill the last 512 tokens brought KL down from 0.135 to 0.047. I combined this with partial-layer transfer: 6 layer plans and 5 tail sizes, evaluated on the same 300 / 300 / 400 prompts.
The tail is set as a fraction of the prompt (none, 1/32, 1/16, 1/8, 1/4), not a token count, so its cost means the same thing on a 1.1k-token GSM8K prompt and a 4.5k-token HotpotQA one. Nothing is retrained: these are the Experiment 1 mappers, which never saw a tail during training. We choose the tail size when running the model; we do not train the mapper again.
- Grid: plans
all,last:30,last:24,last:18,last:12,every:2(the trained r64 mappers from Experiment 1) × a native tail of none, 1/32, 1/16, 1/8 or 1/4 of the transferred prompt. - Tail: the mapped cache covers the prompt minus its tail; the 8B then prefills the tail through all of its layers, attending to the mapped cache. The question header after the prompt is fed to the 8B as before.
- Speedup: the 8B's own prefill time divided by (0.6B prefill + maps + the 8B's native layers + the native tail), HF, bf16, batch 1, best of 3.
- Tail 0 was re-run inside the grid rather than reused from Experiment 1, so every cell comes from the same run.
The two choices help in different ways:
- On GSM8K the tail is the dial that matters. With every layer transferred, a tail of just 1/16 of the prompt takes GSM8K from 70% to 96% of the 8B, while prefill is still 2.5× faster at 8k tokens. For comparison, the best layer-only plan at similar quality,
last:12, is barely faster than the 8B itself (1.12×). In this setup the math question is the last ~5% of the transferred prompt (median 5.3%), so a short tail means the 8B reads the question itself while the eight exemplars stay transferred. A 1/16 tail covers the whole question in two thirds of the prompts; even 1/32, which covers it fully in only 6%, already gets GSM8K to 87%. - On HotpotQA the tail does nothing.
allsits at 74% for every tail size. The question is already read natively there; what transfer loses is multi-hop reasoning over the documents, and only native layers bring that back (89% atlast:12, up to 94% with a tail on top). - SQuAD barely moves (91–95% everywhere): extraction survives transfer whatever the setting.
- A tail costs more at long contexts. The tail’s tokens attend to the whole mapped prefix, so
all+ 1/4 is 1.75× at 8k but 1.40× at 32k. At 2k tokens any tail drops the speedup to 0.8–1.0×, slower than no transfer at all in some cells, because my implementation runs the tail as a second forward pass with its own fixed overhead; a serving engine would fuse it into the same pass.
The practical rule: put the part that must be exact at the end, and let the big model read it. For a prompt that ends in a question or an instruction, a native tail of a few percent buys back most of the quality at most of the speedup. For reasoning spread across a long context, it has to be native layers instead.
Experiment 2: a hybrid big model (Qwen3.5-4B → Qwen3.8-27B)
So far I had been using a small Qwen3 pair to explore these choices. Next I wanted to try the pair from the OneTriangle post: Qwen3.5-4B and Qwen3.8-27B. Both are hybrid models. Three out of four layers use Gated DeltaNet, a linear-attention layer with a recurrent state and a short convolution. The remaining layers use full attention.
A DeltaNet layer has no per-token KV cache to map. Its “cache” is a fixed-size recurrent state that summarises the whole prefix. Mapping that state directly would mean predicting a large matrix from a sequence, so I don’t. Instead, for each DeltaNet layer the mapper predicts the layer’s input. The 27B’s own DeltaNet kernel then runs over that input in a state-only mode: it builds the recurrent and convolution state and skips the output projection. Attention layers get mapped K and V as usual.
Hover a layer to see what is mapped there. The plans other than all are ridge-only (no KL training), with speedups at 8k tokens and state-only DeltaNet stand-ins, measured in HF bf16 at batch 1.
Only all was KL-trained: rank 64, 1,000 steps, 8 docs per step, DDP across 2 GPUs. Ridge KL 0.191 → 0.069, top-1 agreement 0.884. Everything was implemented functionally, so the DeltaNet fill is differentiable and the mapper trains end to end through it.
keep:4) is the natural partial plan for a hybrid model, but it only buys 1.42×. Ridge KL is noticeably better for the partial plans, and none of them were trained.On the short-context tasks, transfer works, but the 4B model on its own is a strong baseline:
| method | SQuAD F1 | HotpotQA F1 | GSM8K |
|---|---|---|---|
| Qwen3.8-27B itself | 0.929 | 0.775 | 0.907 |
| Qwen3.5-4B prefill → transfer | 0.797 (86%) | 0.717 (93%) | 0.868 (96%) |
| Qwen3.5-4B alone | 0.897 (97%) | 0.720 (93%) | 0.855 (94%) |
| 27B, closed book | 0.301 | 0.329 | – |
Qwen3.5-4B does well on these roughly 4k-token QA tasks. Letting it answer on its own is as good as transfer or better. So I next tried longer, harder contexts, where the smaller model might have more to gain from the 27B.
Experiment 3: long-context prefill, 8k to 248k tokens
The short-context results left me with another question: would transfer be more useful when the prompt gets really long? Long contexts are also where saving prefill time could matter most. I wanted to see how much time transfer saves as the prompt grows, and whether it still works beyond the mapper’s training length of at most 2k tokens.
I built a dedicated benchmark (make_long_data.py) with two parts:
- Needle in a haystack: 20 prompts at each of 8k, 16k, 32k, 64k, 128k and 248k tokens, each with 4 keys to retrieve. This checks retrieval.
- LongBench v2: 100 questions of up to 120k tokens, sampled with seed 0. These are 4-way multiple-choice questions that need reasoning over the whole document, so they actually separate strong models from weak ones.
For timing I measured two things. The first is the full transfer pipeline in HuggingFace, end to end. The second is native time to first token in SGLang for both models. SGLang has no transfer path yet, so its 27B ÷ 4B ratio is the ceiling an optimised implementation could approach.
- HF: transformers 5.23, bf16, one GPU, batch 1, best of 3. The prefill is unchunked up to 128k; the 27B runs out of memory at 248k without chunking, and chunked HF prefill distorts the timing, so there is no HF point at 248k. "4B + transfer" is the 4B prefill, plus the trained r64 maps, plus the 27B's state-only DeltaNet fill.
- SGLang: v0.5.22, radix cache off, chunked prefill 16k,
max_new_tokens=1, one GPU per model. - Prompts are the NIAH haystacks at each length (exact token counts in the tooltips).
Most of the gap between the measured 2.2–2.4× and the 4–6× ceiling is the DeltaNet state fill. The 27B still runs its 48 DeltaNet kernels over the full prompt, just without their outputs. That is why the transfer pipeline costs about 1.75× the 4B prefill alone (8.4 s vs 4.8 s at 128k).
The quality results were better than I expected:
| method | LongBench v2 (100 q) | NIAH, all lengths 8k–248k (20 per length) |
|---|---|---|
| Qwen3.8-27B itself | 0.52 | 100% |
| Qwen3.5-4B prefill → transfer | 0.50 | 100% at every length |
| Qwen3.5-4B alone | 0.39 | 100% |
Each square is one LongBench v2 question, ordered by prompt length from 10k (top left) to 120k tokens (bottom right). The 27B and the 4B alone were run greedily in SGLang; transfer ran in HF with the trained r64 mapper. With n = 100 the standard error is about ±0.05.
A few things stood out:
- On long-context reasoning, transfer recovers most of the 27B’s advantage. It scores 0.50 against the 27B’s 0.52, while the 4B alone scores 0.39. The difference from the 27B is within noise; the difference from the 4B is not.
- Retrieval survives at lengths 100× beyond training. A mapper trained on prefixes of at most 2k tokens produced a cache from which the 27B found every needle at 248k tokens.
- The needle test does not distinguish the models here, though: the 4B alone also gets 100%. It shows that transfer doesn’t break retrieval; it says nothing about whether transfer helps. LongBench v2 is the discriminative number, and the length breakdown suggests that training on longer prefixes is the obvious next step.
Experiment 4: dense → MoE (Qwen3.5-4B → Qwen3.5-35B-A3B)
The hybrid model worked, so I wanted to see if transfer would also work when the larger model was a mixture of experts (MoE). I tried transferring from a dense model to an MoE. The same recipe worked here too: KL dropped from 0.175 to 0.066 after training, top-1 agreement reached 0.889, and task scores were the best of any pair I tried:
| method | SQuAD F1 | HotpotQA F1 | GSM8K |
|---|---|---|---|
| Qwen3.5-35B-A3B itself | 0.918 | 0.747 | 0.922 |
| Qwen3.5-4B prefill → transfer | 0.906 (98.7%) | 0.705 (94.5%) | 0.838 (90.8%) |
| Qwen3.5-4B alone | 0.897 | 0.720 | 0.855 |
But for this pair, transfer does not help much with speed. The MoE activates about 3B parameters per token, fewer than the dense 4B that would replace it in prefill. At 8k tokens the 35B-A3B prefills in 0.23 s and the 4B in 0.18 s. Once the maps and the DeltaNet fill are added, transfer is 0.88–1.03× the native speed: slower at every length except 2k. This is a feasibility result: the cache of a sparse model can be predicted from a dense one. For speed, you would pair a much larger MoE with a much smaller reader.
I also tried google/gemma-4-E4B → gemma-4-26B-A4B, and it is blocked by a library bug. In transformers 5.23, Gemma-4’s cached decoding path disagrees with its own full forward pass (KL 0.10 on a continuation even with the exact cache), so any transfer number would measure the bug, not the mapper. I stopped there and kept a small script that reproduces the mismatch.
Experiment 5: across model families and tokenizers
Finally, I tried transferring between model families. This adds a problem: the two models use different tokenizers, so they may split the same text differently. I tried Qwen3-4B → Gemma3-12B, and then Gemma3-4B → Qwen3-14B.
The small model reads the same text with its own tokenizer, and every big-model position needs a small-model position to read from. The rule I settled on is character-based: each big token takes the small token that covers its last character. On top of that:
- the big model’s own token embedding is added as an extra input to the mapper, which is cheap since it is a lookup;
- the
<bos>attention sink is taken exactly from the big model, a one-token prefill; - source layers are matched by relative depth.
These are the real tokenizations of one sentence by the Qwen3 and Gemma3 tokenizers, with character offsets. On ordinary English the two mostly agree, but they split rare words differently ("Kil|iman|j|aro" vs "Kilim|anj|aro"), and only Gemma prepends a <bos> token.
Each cross-family mapper was ridge-fitted on hidden states (not K, V), then KL-trained at rank 64 for 1,500 steps, and evaluated on the same 300 / 300 / 400 prompts.
The results are mixed:
| pair | fidelity KL, ridge → trained | SQuAD F1 | HotpotQA F1 | GSM8K | big model closed book |
|---|---|---|---|---|---|
| Qwen3-4B → Gemma3-12B | 0.373 → 0.181 | 0.577 (72%) | 0.449 (69%) | 0.325 (35%) | 0.233 / 0.266 |
| Gemma3-4B → Qwen3-14B | 0.873 → 0.135 | 0.380 (48%) | 0.104 (16%) | 0.242 (26%) | 0.276 / 0.270 |
For reference, the big models on their own score 0.806 / 0.646 / 0.917 (Gemma3-12B) and 0.789 / 0.655 / 0.945 (Qwen3-14B).
In the Qwen → Gemma direction, the translated cache demonstrably carries the document: QA scores sit far above closed book. But Qwen3-4B on its own already scores 0.738 on this SQuAD setup, so the transfer is worse than not transferring at all. The Gemma → Qwen direction mostly fails: HotpotQA lands below closed book, which means the cache actively misleads the decoder. A curiosity: that direction reaches the better fidelity KL (0.135) and the worse task scores. Matching the next-token distribution on web text is not the same as preserving what a QA prompt needs.
What I learned
- Treat transfer as a dial, not a switch. The most useful result here is the partial-layer one. Let the big model compute its bottom layers, map the top ones, and choose the split from the quality you need. It turns “is cache transfer good enough?” into an engineering trade-off.
- The case for transfer is long context. On 4k-token QA a strong 4B can just answer by itself. On LongBench v2 it can’t, and the transferred cache recovers nearly all of the 27B’s advantage while cutting prefill by 2.2–2.4×, with headroom to roughly 4–6× in a real serving stack.
- Hybrid models need a new trick, and it works. Predict the DeltaNet layers’ inputs and let the big model build its own recurrent state. That fill is also the remaining bottleneck.
- Pick pairs by active compute, not parameter count. Dense → MoE transfers beautifully and saves nothing.
- Cross-family transfer is not there yet. A character-level alignment is enough to carry content, but not enough to beat the small model alone.
There are a few things I’d like to try next:
- train the 27B mapper on long prefixes (16k–64k) to close the gap past 64k tokens;
- harder long-context tasks than NIAH;
- partial-layer plans on the hybrid 27B with KL training;
- a transfer path inside SGLang, to find out how much of the 4–6× ceiling is real.
How these runs were made reproducible
- Hardware and software: rounds 1–3 used NVIDIA RTX PRO 6000; round 4 used a different machine. Software: torch 2.15, transformers 5.23, flash-linear-attention 0.6, SGLang 0.5.22 in Docker.
- Job runner: every job is one line in a single job file, executed by a small GPU scheduler. It handles FIFO order, GPU locks and
.donemarkers, and launches each job withPYTHONHASHSEED=0. - Evaluation: identical sample counts in every experiment: KL on 64 docs, SQuAD 300, HotpotQA 300, GSM8K 400, LongBench v2 100, NIAH 20 per length.
- Logs: every job’s log is kept alongside its result JSON.
- Smoke test: each new pair went through an end-to-end check (ridge → a few training steps → eval on a handful of prompts) before its real run.
- Rerun: the round-3 headline result was re-run on the new hardware with the identical command, and reproduced (KL 0.128 vs 0.129).