Paging experts: in offloaded MoE serving, the cheap win is eviction, not routing
A mixture-of-experts (MoE) model is cheap to run per token and expensive to hold in memory. When its experts do not all fit on the GPU, a serving system keeps s …
Food for thought: how the 20W brain compares to a GPU. Measured in poutine.
The human brain is amazing for many reasons, and one of the biggest is its efficiency. It runs on 20 watts.
First Fable, then NVIDIA and HuggingFace. Is it time for Owned Intelligence?
It seems that we are at an inflection point, perhaps even a tipping point. Companies are re-evaluating their AI strategy and asking an important question: is it …
GPU waste, tokens per dollar, and the impact on inference businesses
The cost of AI is in the spotlight, and every part of the AI industry has at least some focus on improving efficiency to reduce that cost.Model builders, improv …
