GPU waste, tokens per dollar, and the impact on inference businesses

The cost of AI is in the spotlight, and every part of the AI industry has at least some focus on improving efficiency to reduce that cost. Model builders, improving models; hardware vendors, improving performance; infrastructure vendors, improving power and cooling; orchestration platforms, improving utilization.

The problem, however, is more fundamental. It exists at the level of the chip – the GPUs powering AI infrastructure.

Most AI infrastructure is wasting compute because GPUs do not process workloads at the level the silicon is capable of. They are not delivering maximum intelligence per dollar invested in the GPU.

Where does this really bite hard? Inference workloads, for companies with processes or entire business models based on serving inference. There is a direct relationship between inference performance - tokens per dollar - and business performance. In this post we’ll explain why, and what you can do about it.

What causes poor inference performance?

Inference performance depends on the interaction between the model, the workload, the serving configuration, and the GPU hardware. The same model can require different batching, scheduling, sharding, memory management, and frequency settings depending on the workload it’s running - chat, coding or agentic. Each use case can have different context lengths, request arrival patterns and latency requirements.

Modern inference is often constrained by data movement rather than compute. Model weights and activations must repeatedly move through the memory hierarchy, and the cost of that movement can dominate computation.

This is even more apparent in long-context and agentic workloads. Longer generations reduce serving concurrency, lower arithmetic intensity, and can push execution further into a memory-bound regime.

When compute is starved of data, the GPU is not being used in an optimal way, and service level objectives suffer: crucial measures like time to first token (TTFT) and time per output token (TPOT).

The industry’s answer to this, to date, has largely been “buy more GPU” – but that doesn’t help companies serving inference, whose GPU costs grow with usage.

Why don't companies just... fix it?

They try. Inference serving engines such as vLLM exist to optimize GPU compute and memory usage, but they only ship with a default, one-size-fits-all configuration per model. The best optimizations are often custom and specific to the use case. Companies can tune the engine to those specific requirements.

However, that is a time-consuming’ manual process that requires skilled engineers, complex iterative testing, access to testing systems and yet more GPU time – and meanwhile, models evolve, hardware changes, and so do a company’s service level objectives.

Each additional combination creates a new optimization problem, containing the following parameters:

Only the largest labs have the time and resources to optimize models for the increasingly complex mix of GPU, model and workload type. They tend to keep the results to themselves.

Everyone else makes do with default engine configurations or spends time and money tuning limited aspects of the problem.

How we solve this at CarbonForge

CarbonForge solves this problem by going back to where we began at the start of this post: the silicon. In our own lab at Mila, we measure silicon under load at the most fundamental level - power - and find optimal efficient configurations in a high-dimensional and unbounded search space, adjusting inference engine knobs, performing kernel engineering, hardware control, request management and model topology tuning.

The results are delivered as ready-to-deploy containers for a wide range of open-source models. Most CarbonForge customers aim to increase tokens delivered per dollar; or the reverse, reduce cost per million tokens; or they want to improve inference latency, throughput or accuracy.

With CarbonForge you can achieve any of these outcomes with a simple switch of the container your model runs in, and without impacting the quality of intelligence delivered by the model.

In a following post, we’ll dive into more detail about how this actually works. In the meantime, head to the CarbonForge Library to get started – you’ll find a wide range of inference containers with models optimized for different workload types, GPUs and GPU counts.