How Do You Measure a Token's Energy?

Suppose two people report the energy used by the same AI model. One records the processor’s own sensor. The other measures the entire server at its power supply. Their numbers differ, and both measurements may be correct.

Before comparing them, ask what each instrument included. A precise number with an unspecified boundary is difficult to use. Adding more decimal places will not recover the missing information.

The basic experiment is straightforward: measure electrical consumption while specified work runs, then divide by a specified count. The care belongs in defining the work, the equipment, the interval, and the count before starting.

Scope note: the published examples below are GPU-energy measurements. The hypothetical arithmetic demonstrates a method; it is not a new hardware experiment performed for this article.

Choose where the measurement begins and ends

A GPU measurement follows the accelerator. A server measurement can include its CPUs, memory, storage, fans, and power-conversion losses, depending on where the sensor sits. A facility measurement can include cooling and other supporting equipment. These boundaries answer different questions.

Google’s full-stack AI-serving study accounts for active accelerators, host systems, idle capacity, and data-centre overhead. That wider accounting is a reason to label a GPU-only result accurately, not a reason to reject it. The smaller boundary may be exactly what an engineer needs to compare two GPU execution methods.

There is also a distinction between total energy during a run and the additional energy above an already-powered baseline. The latter requires a matched no-computation measurement that is appropriate to the experiment. Subtracting an arbitrary idle wattage from a shared system can attribute other activity to the request or remove energy that the request actually required.

I would report the directly measured run total first, with its boundary, then describe any baseline subtraction separately. A reader should be able to reconstruct both quantities without guessing what was removed.

Measure watts through time

Power can change while the model works. The required energy is the accumulated power over the measurement interval, summed across all participating devices. In mathematical notation this is an integral. In practice, a meter may supply cumulative energy directly, or a measurement system may collect power samples and estimate the area under the resulting curve.

For a simple hypothetical run, suppose each of two accelerators averages 400 watts for ten seconds. Together they use 8,000 joules. If the measured interval produces 20,000 generated tokens, the output-normalized result is 0.4 joules per token.

Every number in that example has a job. Count only one accelerator and half the energy disappears. Include tokens generated before measurement began and the denominator becomes too large. Stop measuring before outstanding work finishes and the interval becomes too short. These are ordinary experimental errors with considerable effects on the answer.

Sampling also needs enough resolution for the workload. A short request can be poorly represented by a slow sensor. Longer repeated runs, cumulative-energy counters where available, and synchronized device measurements can make the result more useful, but their suitability should be checked against the equipment rather than assumed.

Decide which tokens are being counted

A request can involve processing input and generating output. Input processing is often called prefill; generation is often called decode. Their execution differs, and the electricity used for one should not vanish merely because a reported metric divides by the other.

An output-normalized metric can legitimately include input-processing energy in its numerator. Its label should say so. “Joules per generated token during steady serving” does not mean “energy spent exclusively on the final token-generation operation.” It means all included energy was divided by that output count.

Internal reasoning tokens matter too. If a system generates them before the visible answer, they are part of the computation even when the interface does not display them. Dividing by all generated tokens and dividing by visible answer tokens will produce different numbers.

The same rule applies to speculative decoding. Rejected proposals remain in the energy total. They do not belong in a denominator described as committed output. The label must identify the count actually used.

What a published measurement looks like

Jae-Won Chung, Ruofan Wu, Jeff J. Ma, and Mosharaf Chowdhury measured inference across models and serving configurations in Where Do the Joules Go?. For language models, they measured GPU energy during steady operation and divided by tokens generated during that interval.

Their Qwen 3 32B results on one B200 include:

Those are measurements of specified configurations. They establish that energy per token can change with workload and batching on the same model and accelerator. They do not provide a universal coefficient for Qwen, every B200 installation, or a proprietary chatbot.

For this series, the useful evidence is the reproducible relationship between a declared experiment and its result. The reader can inspect which hardware was used, what the system generated, and what the researchers counted. The subsequent dollar conversion does not require access to the provider’s selling price.

Leave enough information for someone else

A repeatable record identifies the model revision, hardware, software, numerical precision, input and output lengths, batch settings, and generation method. It records the measurement boundary and interval, how warm-up was handled, and whether reported energy includes input processing or idle capacity.

Repeat the run rather than treating one reading as permanent. Report the spread as well as a central value. If the experiment seeks the lowest-energy configuration, retain the response-time and output-quality conditions; otherwise a slow or ineffective configuration may appear to win a comparison it was never intended to enter.

For a publicly runnable model, those steps permit a direct measurement. An external researcher examining a proprietary model may lack access to its hardware, architecture, or serving configuration. A conditional estimate is still possible when enough information is disclosed, but its assumptions must remain attached. A product name alone does not provide the missing execution record.

Once the measurement is established, the difficult physical accounting is largely behind the calculation. The final article converts the result into money and asks exactly what that comparison can establish about a quoted token price.

Continue: How Do Joules Become Dollars?