Paging experts: in offloaded MoE serving, the cheap win is eviction, not routing

A mixture-of-experts (MoE) model is cheap to run per token and expensive to hold in memory. When its experts do not all fit on the GPU, a serving system keeps some of them in CPU memory and copies each one over the PCIe link when the router asks for it, and decoding waits for the copy. That turns expert routing into a paging problem, the same one operating systems have studied since the 1960s. This post measures what the copies cost, shows why routing tricks that hold experts in place trade quality for speed, and explains why the headroom lies in the cache's eviction rule: a clairvoyant evictor cuts transfers 2.1 times without changing a single output, and sixteen tokens of foresight recover 96% of that gain. The paper will be presented at AXIOM, the NeurIPS 2026 workshop on the foundations of efficient deep learning, in Paris.

Anatomy of an offloaded MoE layer

An MoE layer runs only a few of its expert networks on each token. OLMoE-1B-7B, the model we used, selects 8 of 64 experts in each of its 16 MoE layers, so about 1B of its 7B parameters are active per token. Once the weights are offloaded, these routing choices become memory requests: an expert must be resident on the GPU before the layer can run it, and a cache decides which experts stay.

To measure this precisely, we built a serving harness in plain PyTorch on stock Hugging Face Transformers. In eight of OLMoE's sixteen MoE layers, all 64 experts live in pinned host memory as packed gate, up and down projections, 12.6 MB per expert in bf16, and a pool of 32 GPU slots holds the resident ones. The integration is a single attribute swap: a Hugging Face decoder layer ends with hidden = self.mlp(hidden), so replacing mlp with our block puts every residency decision inside the model's own forward pass. Each decision sees the actual hidden state produced by the policy's earlier decisions, which matters as soon as a policy changes what the model computes. Eviction is bookkeeping, not data movement: the victim's slot is simply overwritten by the next copy, and because the expert matmuls are queued after the copies on the same CUDA stream, ordering alone guarantees correctness.

Every policy is evaluated with teacher-forced, batch-1 decoding on sixteen held-out sequences of up to 384 tokens, so all policies see identical inputs. Quality is next-token negative log-likelihood (NLL), and throughput comes from paired device events. Policies that preserve every routing choice match the full-capacity reference NLL in every run.

The cost model

Because copies block decoding in this setting, the time per token decomposes into a compute term and a transfer term:

where \(c\) is the computation, \(\rho\) the number of expert copies per token across all offloaded layers, and \(\tau\) the time to copy one expert. The last factor is a property of the machine, not the model: on our L40S the pinned host-to-device copy loop runs at 23.0 GiB/s, so \(\tau \approx 12.6\ \text{MB} / 23.0\ \text{GiB/s} \approx 0.51\ \text{ms}\). Across the GPUs we measured, \(\tau\) varies almost fivefold:

With every expert resident, decoding runs at 49.0 tokens per second, or 20.4 ms per token. With a standard least-recently-used (LRU) cache it runs at 38.3 tokens per second, 26.1 ms per token, while making 9.6 copies per token. At 0.51 ms each, those copies account for about 4.9 ms of the 5.7 ms difference. Decode is waiting on PCIe.

The formula also says how to pick a policy. Given a tolerance \(\varepsilon\) on quality loss, the fastest admissible policy is

which is piecewise constant in \(\tau\): policies \(i\) and \(j\) trade places only at \(\tau_{ij} = (c_j - c_i)/(\rho_i - \rho_j)\). One startup measurement of \(c\) and \(\tau\) on the target machine is enough to choose. If copies overlap with computation, as in an asynchronous pipeline, the cost tends toward \(\max(c, \rho\tau)\) instead of the sum, which shrinks every policy's advantage on fast links.

Switching is not copying

One popular way to reduce copies is to change the routing itself: keep one set of experts for several tokens, and decide when to replace it. Shen et al. recently proposed learning these two decisions with the option-critic architecture, which I developed with Jean Harb and Doina Precup during my PhD: a termination head decides when the current set expires, a selection head picks the next one, and a deliberation-cost regularizer pushes the switch rate from over 50% of tokens to under 5%. In that work's released evaluation, every expert stays on the GPU and the chosen set only masks the router, so no weights move and transfers are not measured. We wanted to know what happens when they are.

The first observation is that switch rate is the wrong metric. Switches and copies are different events:

The second is that holding a set fixed changes what the model computes. When the router asks for an expert outside the held set, the layer must use the best resident expert instead, and prediction quality can drop. We compared lossless LRU against several set-holding policies: hold-16 and hold-64 keep the pool for 16 or 64 tokens and then refill it with the most demanded experts; freeze profiles 64 tokens and then never changes; and learned switching is our implementation of the option-critic controller, whose termination threshold trades switch rate against fidelity. Both learned heads run on the timed path, so their computation is charged too.

* Estimated from the cost model at this policy's copy count. All other throughputs are measured.

At this copy cost, every set-holding policy that runs faster than LRU pays at least 0.067 nats per token of extra loss. Holding the set is a trade of quality for speed, not a free improvement. Across six termination thresholds, with mean dwell times from 2.9 to 73 tokens, the learned controller was dominated up to threshold 0.7: some simpler rule matched its quality with fewer copies, or its copies with better quality. At threshold 0.5 its quality is close to hold-16 (+0.076 against +0.067 nats) but it moves 2.7 times as many experts. Its selection head is good at short horizons, costing only +0.030 nats when refreshed about every three tokens, but at a 73-token dwell its loss rises to +0.350 nats, against +0.110 for a frozen 64-token profile. The pattern suggests the selection head predicts near-term demand well while the termination head holds that prediction too long.

The ranking does flip as \(\tau\) grows, exactly as the cost model predicts. We slowed transfers 10 and 50 times; at both settings, the startup profile picked the fastest of six policies. In the model, at 50 times the L40S copy cost, hold-16 and hold-64 run 2.0 to 2.5 times faster than LRU and the most aggressive learned threshold is the fastest restricted policy. On slow links, holding expert sets can be worth its quality cost; on a PCIe 4 server, it is not.

Two families of policies

The comparison above mixes two different kinds of policy, and separating them clarifies what can be gained for free. Fix one layer, its capacity \(C\), and the router's unrestricted choices \(S_1, \dots, S_T\), each a set of \(k\) experts. A schedule is a sequence of resident sets \(R_t\) with \(|R_t| \le C\), and its traffic is the number of insertions,

A schedule is exact when \(S_t \subseteq R_t\) for every token: everything the router asked for is resident, so the model computes exactly what it would with unlimited memory. For exact schedules, quality leaves the problem and only traffic remains,

Because an exact schedule never restricts the router, the demand sequence does not depend on the schedule, and \(B^\star\) is a property of one recorded trace. Every exact policy, causal or clairvoyant, with or without prefetching, can be scored by replaying that trace. Set-holding policies belong to the other, lossy family: they trade an increase in loss \(\Delta L\) for traffic below what exactness requires, and the best achievable trade is a frontier \(Q_C(\rho) = \inf\{\Delta L : \rho(\text{policy}) \le \rho\}\).

Belady's rule, applied to experts

For an exact schedule, one decision remains. When the router requests an expert that is not resident, it must be fetched, and if the pool is full, one resident expert that the current token does not need must leave. LRU evicts the one used least recently. In 1966, Laszlo Belady showed that for a cache serving one request at a time, the optimal victim is the item whose next use lies farthest in the future.

Applied to the recorded routing decisions, farthest-reuse eviction cuts traffic from 9.6 copies per token to 4.6, 2.1 times fewer, while the model computes exactly what it computed before. Belady's optimality proof does not carry over directly, because an MoE token requests several experts that must be resident at the same time. A three-token example shows the difference. With \(C = 3\) and requests \(\{a,b\}, \{c,d\}, \{a,b\}\) from an empty cache, the flattened sequence \(a, b, c, d, a, b\) can be served with four copies by evicting \(c\) right after its use. A real token cannot do that: \(c\) and \(d\) must coexist during the second token, so one of \(a\) or \(b\) has to go, and the true minimum is five.

Flattening is still useful, because it can only make the problem easier. It yields a sandwich:

where \(B_{\text{flat}}\) is the classical single-request optimum of the flattened trace and \(B_{\text{greedy}}\) is the traffic of the farthest-reuse schedule. Rather than proving the greedy schedule optimal, we certify it on the actual traces. On our sixteen sequences, \(14{,}587 \le B^\star \le 15{,}087\), so farthest reuse is within 3.5% of the best exact schedule. The two bounds coincide on 14% of the 128 (layer, sequence) streams, where greedy is exactly optimal, and the median per-stream gap is 2.9%. On 42,000 random instances small enough for exhaustive dynamic programming over all schedules, including uniform, Zipf-skewed and bursty demand, greedy matched the optimum every time.

How far ahead does an evictor need to see?

Farthest reuse needs the future, so it is a target rather than a policy. The practical question is how much of the future. Truncating its foresight to \(h\) tokens, so that experts whose next use lies beyond the horizon look identical and the least recently used of them is evicted, gives a sharp answer:

Sixteen tokens of exact foresight recover 96% of the gap between LRU and the clairvoyant schedule, and thirty-two recover essentially all of it. The rest of an expert's future barely matters. Rules that see only the past get part of the way. Evicting the expert with the lowest exponentially decayed use frequency recovers 27% of the gap on the full trace, about 14% fewer copies than LRU, with no learning and no change in output. A measured run of this rule in the harness reproduced the replay within 0.6%.

We also trained small scorers to imitate the clairvoyant victim from causal features, following the Hawkeye and Parrot line of work on learning from Belady's rule. Given only each expert's access history, a two-layer network recovered 25% of the gap on held-out sequences, against −136% for a control trained on shuffled labels. The linear version put nearly all of its weight on frequency features, with the sign that evicts rare experts: imitation rediscovered frequency-based eviction. Adding the router's own probabilities as inputs, trained on a separate 48-sequence trace, lifted the scorer to 32 to 33% on the four held-out sequences, against 29% for the hand-built frequency rule; across all sixteen evaluation sequences, the learned and hand-built rules tie at 25 to 27%.

Two negative results are as informative as the positive ones. A per-expert prediction of the mean time between requests did worse than LRU, by 17.5% of the gap, because demand is locally clustered and penalizing recently used experts backfires. Evicting by the current router logit was 46% worse than LRU in the harness: the instantaneous logit evicts experts that were just used and the cache thrashes. Signals that integrate the past help; one-shot signals hurt.

Why LRU and LFU keep winning: an index view

The appearance of recency and frequency rules is not an accident, and a restless-bandit relaxation explains it. Treat each expert as an arm. Replace the hard capacity by a rent \(\nu\) paid per token by each resident expert, Whittle's Lagrangian relaxation, and assume that an absent expert, when demanded, can either be fetched at cost \(\tilde\tau\), the copy time expressed in loss units, or substituted at a penalty \(\delta_e\). If \(h_e(a)\) is the probability that expert \(e\) is demanded at the next token given the \(a\) tokens since its last use, one more token of residency costs \(\nu\) and saves, in expectation, \(h_e(a)\min(\delta_e, \tilde\tau)\). The one-step rule is to keep the expert while

This is a heuristic comparison, not an optimal-control result: it ignores the option value of future fetches, and the paper states the assumptions it needs. Its value is that each classical rule becomes a choice of estimator for \(h_e\). If demand is memoryless, \(h_e\) is a constant rate and the rule ranks by frequency. If every expert shares a hazard that decreases with idle time, the rule ranks by recency, which is LRU. If each expert switches between hot and cold regimes, the posterior demand rate behaves approximately like an exponentially forgetting average of past use, which is decayed frequency. With known future demand, the indifference rent becomes \(\tilde\tau/d_e\), where \(d_e\) is the time to the next request, and evicting the lowest index is exactly farthest reuse. The imitation result above fits this reading: a model trained to mimic the clairvoyant teacher from access history alone has to estimate a hazard, and it learned decayed frequency.

The relaxation also explains why the learned controller struggled. Retraining its heads with the traffic penalty priced at the measured \(\tau\), times an exchange rate \(e\), moves it along a smooth curve (one seed, matched steps and data): at \(e = 0\) it refreshes almost every token, 73 copies per token at full quality; at \(e = 0.02\) its threshold sweeps from 62 down to 4.8 copies per token; and \(e = 0.08\) reproduces the hand-tuned controller. No price reached the frontier: at 4.0 copies per token it paid +0.16 nats, against +0.085 for hold-64 at 3.9 and zero for farthest reuse at 4.6. Pricing the objective correctly moves a policy along its own curve; it does not fix a weak demand estimate.

Try it on a trace

Everything above about lossless policies can be checked by replaying a recorded routing trace. This is the core of that replay, with the three eviction rules discussed here:

INF = float("inf")

def next_use(e, trace, t):
    """First token after t that requests expert e (INF if never)."""
    return next((u for u in range(t + 1, len(trace)) if e in trace[u]), INF)

def last_use(e, trace, t):
    return max((u for u in range(t + 1) if e in trace[u]), default=-1)

def decayed_freq(e, trace, t, beta=0.95):
    return sum(beta ** (t - u) for u in range(t + 1) if e in trace[u])

RULES = {  # each rule picks a victim among the candidates
    "lru":      lambda c, tr, t: min(c, key=lambda e: last_use(e, tr, t)),
    "freq":     lambda c, tr, t: min(c, key=lambda e: decayed_freq(e, tr, t)),
    "farthest": lambda c, tr, t: max(c, key=lambda e: next_use(e, tr, t)),
}

def copies(trace, C, rule):
    """Lossless serving: every requested expert is made resident before the
    token runs. Returns the number of CPU-to-GPU copies."""
    pool, n = set(), 0
    for t, demand in enumerate(trace):
        for e in sorted(demand - pool):
            if len(pool) == C:
                victims = pool - demand        # never evict a co-requested expert
                pool.remove(RULES[rule](victims, trace, t))
            pool.add(e)
            n += 1
    return n

On the three-token example, all three rules need five copies with \(C = 3\), the true optimum. On real traces they separate quickly, and replaying your own model's routing decisions is the cheapest experiment you can run before touching a serving engine. An open vLLM pull request (#37190) adds expert offloading with a fixed GPU cache and exposes exactly this victim-selection point.

What to take away

• Measure \(\tau\) and \(c\) on the target machine before choosing a residency policy. The best policy is piecewise constant in \(\tau\), and the crossover points follow from the cost formula.
• At the L40S's measured copy cost, every set-holding policy that beat LRU paid at least 0.067 nats per token; on faster links the case for holding sets only weakens. Report physical transfers, not switch rates.
• Change the eviction rule first. It preserves every routing decision, needs no retraining and no new quality evaluation, and decayed frequency already takes about 14% of the copies away.
• Treat eviction as a forecasting problem with a known horizon: predicting which experts the next sixteen tokens will request is enough to recover almost all of the 2.1 times headroom.

Limits

We measured one model, OLMoE-1B-7B, with a router fine-tuned to favor reusing experts. The model actually fits on an L40S; we imposed the memory budget so that policies could be compared under controlled conditions. Decoding was batch-1 and teacher-forced on fixed text, sequences were at most 384 tokens, and copies were not overlapped with compute. Batching, longer contexts and asynchronous copy pipelines may change both how often experts are reused and how much restricted routing costs in quality, and overlap in particular narrows the gains on fast links.

Why we care

CarbonForge tunes how models are served for a given workload and GPU. This paper is a compact example of how we work: measure the hardware cost first, separate what changes the model's output from what does not, and prefer the changes that do not, because they can go to production without a new round of quality evaluation. A finding only counts for us if it is valid and deployable. This paper is about deployability. Our EvidenceTree paper, accepted at the NeurIPS 2026 workshop on e-values, is about validity.

The paper

Fewer Expert Transfers Without Changing Routing: Expert Eviction for MoE Offloading. Pierre-Luc Bacon, Diego Calanzone and Jean-Maxime Larouche. AXIOM: Foundations of Efficient Deep Learning, NeurIPS 2026 workshop, Paris, December 2026. OpenReview

References

• L. A. Belady (1966). A study of replacement algorithms for a virtual-storage computer. IBM Systems Journal.
• R. L. Mattson, J. Gecsei, D. R. Slutz and I. L. Traiger (1970). Evaluation techniques for storage hierarchies. IBM Systems Journal.
• P. Whittle (1988). Restless bandits: activity allocation in a changing world. Journal of Applied Probability.
• P.-L. Bacon, J. Harb and D. Precup (2017). The option-critic architecture. AAAI.
• J. Harb, P.-L. Bacon, M. Klissarov and D. Precup (2018). When waiting is not an option: learning options with a deliberation cost. AAAI.
• A. Jain and C. Lin (2016). Back to the future: leveraging Belady's algorithm for improved cache replacement. ISCA.
• E. Z. Liu, M. Hashemi, K. Swersky, P. Ranganathan and J. Ahn (2020). An imitation learning approach for cache replacement. ICML.
• A. Eliseev and D. Mazur (2023). Fast inference of mixture-of-experts language models with offloading. arXiv.
• N. Muennighoff et al. (2024). OLMoE: open mixture-of-experts language models. arXiv.

How to get started

Serving an MoE model on hardware where it does not quite fit? We tune serving configurations for your workload, model and GPU, and we can measure what offloading costs on your machines.