Contact us Download containers
Contact us Download containers

How CarbonForge containers work

A CarbonForge container is a serving image built for a specific combination of hardware, model and workload type.

The image carries the model and the inference engine, plus two control layers: the CarbonForge Scheduler and the CarbonForge Controller.

Stock vLLM
vLLM GPU
one default configuration
CarbonForge container 2 layers added
requests in Scheduler vLLM GPU tokens out Controller

The Scheduler is in the path of every request. The Controller watches the live server and moves the operating point on the GPU, against your target.

CARBONFORGE SCHEDULER

Reorders and groups the requests waiting in the enginequeue, to hold the latency of the queue tail and yourservice level target. Same requests, same GPU, sametotal compute: only the position in the queue changes.Pure software, so no hardware setting is touched andno elevated privileges are needed.

CARBONFORGE CONTROLLER

An on-host agent that monitors the live server andadapts the operating point for the load the server iscarrying, against your TTFT and TPOT target.

What the Optimization Engine searches

vLLM ships one default configuration per model. It is a reasonable starting point
and almost never the best one for a specific model on a specific GPU under aspecific workload.

Your engine does not search for a better one, because searching is not its job. That is what the CarbonForge Optimization Engine does, in our lab, on a number of axes at once.

icon-check
Precision path by GPU class
Precision path by GPU class
icon-check
Parallelism degree, measured per model ratherthan defaulted
Parallelism degree, measured per model ratherthan defaulted
icon-check
Batch policy and saturation point
Batch policy and saturation point
icon-check
Kernel selection
Kernel selection
icon-check
KV cache sizing at target concurrency, and prefix caching
KV cache sizing at target concurrency, and prefix caching
icon-check
Speculative decoding, compared both ways
Speculative decoding, compared both ways

LOCKED, THEN RE-LOCKED

IN THE LAB

Search, then lock

The Engine runs on the model, the GPU class andthe workload profile, and the result is locked intothe image.

IN PRODUCTION

Adjust in bounds

The Controller moves inside the lockedconfiguration to follow your load. It does notsearch on your cluster.

ON CHANGE

Search again

New model, new GPU, new traffic shape. TheEngine runs again and the container is re-locked.

Every published figure is measured in cost per million tokens against stock vLLM
at the same latency target, and names its GPU, its model, its precision and its vLLM version.

Read the methodology

Questions

What has to change in my environment?

Nothing. The container runs next to vLLM in your own cloud account, with your API, your code and theopen models you already serve. Nothing about your workload leaves your environment either: noweights, no prompts, no outputs. The one thing that changes is the serving configuration inside theimage, and that is the whole product.

What does the container send back?

An operating log: what the Scheduler and the Controller decided, and what the hardware did. Noweights, no prompts, no outputs, nothing that describes what your workload contains. The log iswritten where you can read it. Sending it back is part of running a CarbonForge container, because itis how the gain gets measured and how your next container gets better.

Does output quality move?

The weights do not change. What changes is how the engine schedules and executes work aroundthem. Where a configuration touches numerics, precision path for instance, quality is evaluatedbefore the configuration is locked, and a configuration that fails the evaluation is not shipped. If youhave your own evaluation suite, run it: the container is a normal image and you own the environmentit runs in.

The Scheduler reorders requests. What happens to long prompts?

Reordering a queue is a change in behaviour and it deserves a straight answer. Every policy thatreorders carries an aging rule, so a long request moves up the queue the longer it waits and cannotbe starved by a stream of short ones. First-come-first-served is one of the available policies andstays the reference baseline. Which policy ships in a given container is a decision made during thesearch, not a switch left for you to find.

Does it work on my fine-tune?

The search operates on the serving path, not on the weights, so a fine-tune of a supported basemodel should behave like the base model. The measured gain will differ from the published one,because your checkpoint and your traffic are not the ones we measured. That is what a baseline onyour own workload is for.

A new model ships. Then what?

A new model means a new search. The container library is where new containers appear, and eachcard names the GPU, the precision and the workload it was built for. If the model you run is not in thelibrary, tell us and we will measure it.

How is it billed?

Containers are licensed by GPU class. On consumer hardware they are free to pull and run. On datacenter hardware the price follows the gain measured on your workload, so the bill tracks the resultrather than the seat count. The details are worth a conversation rather than a pricing page, becausethey depend on what you run and how much of it.

Browse the container library