How Can the Same Task Use Less Energy?
Reusing loaded weights, reducing memory transfers, and changing execution can lower energy use. The savings depend on the workload and must be measured.
Load a set of model weights, use it for one request, and discard it from nearby working storage. Load the same weights again for another request. If the hardware could instead use the first load for both requests, some data movement could be avoided.
That is a physical change in the execution. The transistor has acquired no new law of nature. The system has arranged its work differently. This is how an implementation can become more energy-efficient while the underlying model remains recognizably the same.
Scope note: this article separates several efficiency mechanisms. A faster method, a smaller number format, or a larger batch is not by itself proof of lower energy or equivalent output quality.
Share a weight load across requests
Processing several sequences together is called batching. In a simplified dense-model example, suppose a set of weights occupies seventy gigabytes and is read once for each batch’s generation step. With one sequence in the batch, that is seventy gigabytes of weight traffic per generated token. With thirty-two sequences, the same load can contribute to thirty-two generated tokens, giving about 2.19 gigabytes per token.
The calculation divides seventy by thirty-two. It describes shared weight traffic under the stated assumption, not the entire system’s memory traffic. Each sequence still needs its own arithmetic and, for unrelated contexts, its own attention state. Hardware caches and the serving implementation can also make actual transfer patterns more complicated.
There are practical limits. Larger batches require more memory and can change response times, while a lightly used service may not have enough simultaneous requests to fill them without making people wait for work that has not yet arrived. A benchmark can deliberately supply continuous work; a real service has to cope with arrivals, departures, and requests of different lengths.
This makes “the model’s energy per token” incomplete as a description. The same weights can be used in an efficient, continuously occupied configuration or in a lightly occupied one. To compare the results honestly, keep batch size and workload alongside the model name.
The ML.ENERGY benchmark treats serving conditions as part of the experiment. Its approach is useful precisely because a name and a nominal processor power rating cannot reconstruct the executed work.
Do the same attention calculation with fewer transfers
The attention calculation discussed in the previous article can create large intermediate results. Writing those results out to high-bandwidth memory and reading them back costs time and energy.
Tri Dao and colleagues developed FlashAttention to reduce that traffic. The method processes the calculation in blocks that make better use of fast on-chip memory. It computes exact attention in the mathematical sense used by the paper, rather than replacing attention with a rough approximation; ordinary finite-precision implementation differences still exist.
The useful distinction is between the mathematical operation and its execution order. Two implementations can calculate the same operation while making different numbers of transfers between memory layers. An operation count alone will not reveal that saving.
It would still be a mistake to take a speedup from the paper and announce an identical percentage reduction in electricity. Energy depends on power over the duration of the actual run. The method provides a reason to expect reduced transfer work; measurement establishes the electrical result on the chosen system.
Store each number differently
A model’s weights can be represented with different numerical formats. Reducing the number of bits per weight can shrink storage and memory traffic. Some processors also execute lower-precision arithmetic at higher rates.
The change has consequences beyond file size. A less precise representation may alter the model’s outputs, and additional conversion or scaling work may be needed during execution, with hardware support determining whether the smaller format brings the expected benefit on the selected processor. A format that performs well on one processor may bring little benefit on another.
For that reason, an efficiency comparison needs a quality condition as well as an energy condition. If one configuration uses less electricity but fails the task more often, energy per generated token has improved while energy per useful result may not have. Repeated attempts belong in the latter calculation.
Chung and colleagues’ inference-energy study found that precision, batch size, and hardware utilization interact. Lower precision was not uniformly more energy-efficient across all tested conditions. That finding argues for recording the configuration, not for dismissing the potential savings.
Generate proposals, then check them
Ordinary autoregressive generation has a sequential dependency: the selected token becomes part of the context for the next one. Speculative decoding changes how that dependency is handled. A smaller model proposes several tokens, and a target model checks them. Accepted proposals advance the sequence; rejected proposals have consumed work without becoming delivered tokens.
Charlie Chen and colleagues’ speculative-sampling paper describes a method that preserves the target sampling distribution under its algorithmic assumptions. This is more specific than asking a cheap model to answer and hoping the result resembles the expensive model’s answer.
The energy calculation must include both participants. Add the energy used to propose the tokens and the energy used to verify them, then divide by the number of committed tokens. Counting only accepted work would erase a real electrical expenditure from the experiment.
Speculation can improve the execution, but its energy benefit depends on acceptance rates, the proposal model, verification costs, and hardware behavior. A shorter wait for the user is valuable in its own right. It should still be measured separately from energy saved.
State what stayed the same
These examples do not all preserve the same things. Batching primarily changes how requests share execution. FlashAttention reorganizes a mathematical operation. Lower numerical precision can change results. Speculative sampling can preserve a target distribution while adding a different pattern of work.
I would therefore ask an efficiency claim to name its invariant. Was the model fixed? Was output quality checked? Did the response-time requirement stay the same? Were all participating devices counted? Without those details, two attractive energy figures may describe different services.
The physical laws remain consistent throughout. The implementation can reduce transfers, improve reuse, or shorten powered execution. What is needed next is an experiment that records those changes rather than inferring them from a faster demonstration.
