Why Does a Long Conversation Change the Work?
A model retains numerical state from earlier text. Longer context increases the storage and attention work involved in producing the next token.
Paste a long report into a conversation, ask a question, and then ask another. The second question may be only six words long. The system can still have a substantial amount of earlier material to consider before it produces its reply.
The visible length of the new message is therefore a poor guide to the work required. What matters is the context actually supplied to the model, including any retained conversation and documents. An application may shorten, retrieve, or summarize that material, so the conversation displayed on screen is not always identical to the model’s active context.
Scope note: the worked example uses conventional full attention and the published dimensions of Llama 3.1 70B. It estimates arithmetic and storage, not measured electrical consumption.
What the computer keeps
A transformer does not need to repeat every earlier calculation each time it generates a token. It can retain numerical state from preceding positions, usually called the key-value cache. Keys and values are numerical representations used by attention, the mechanism through which the current position combines information from earlier positions.
The words “key” and “value” sound more ordinary than the mathematics. For the present purpose, the important fact is that both sets of numbers occupy memory. A new token’s query is compared with retained keys, and the resulting weights determine how retained values contribute to the calculation.
Keeping these results avoids recalculating their projections from scratch. It does not remove the need to access them during subsequent attention. As context grows, more retained positions may participate in that work. The original transformer paper describes the underlying attention operation; the cache is part of making autoregressive execution efficient.
This is computational memory in a specific sense. It should not be confused with a person’s memory of a conversation, an application’s long-term user profile, or a database where selected facts have been stored for later retrieval. Each arrangement has its own representation and cost. Calling all of them “memory” can conceal which one is actually using the hardware.
Give the stored context a size
Meta’s published architecture definitions give Llama 3.1 70B eighty layers, eight key-value heads, and 128 dimensions per head. A head is a subdivision of the attention calculation. This model shares key-value heads across query heads, an arrangement called grouped-query attention.
For a simple storage estimate, assume every retained number uses two bytes. For each context position, count two sets of numbers, keys and values, across eighty layers, eight key-value heads, and 128 dimensions. Multiply by two bytes:
2 × 80 × 8 × 128 × 2 = 327,680 bytes per context position.
At 8,192 positions, that is about 2.68 gigabytes. At 131,072 positions, it is about 42.95 gigabytes. These are decimal gigabytes, calculated for one sequence under the stated representation. They exclude model weights and other runtime storage.
That is a considerable difference produced without changing a single model weight. Sixteen times as much retained context requires sixteen times as much cache in this example, although a different cache precision, attention architecture, or compression method would change the estimate and must be named rather than silently assumed.
The model version matters here. Llama 3.1 supports the long-context case in this example; the original Llama 3 release should not be used interchangeably with it. Meta’s Llama 3.1 model card describes the expanded context window.
The next token does more attention work
There is also an arithmetic estimate. For conventional full attention during single-token generation, a useful leading count is four times the number of layers, times the combined query-head width, times the context length. Two main operations account for it: comparing the query with keys and combining the values, each with multiplication and addition.
Using eighty layers and a width of 8,192 gives the following rounded totals. The linear-transformation term comes from the previous article’s illustrative seventy-billion-weight estimate.
| Context | Attention | Total with linear work |
|---|---|---|
| 8,192 tokens | 21.5 billion | 161.5 billion |
| 131,072 tokens | 343.6 billion | 483.6 billion |
The attention term grows sixteenfold. The combined estimate grows about threefold, because the linear term remains roughly constant. Neither figure should be described as a measured energy ratio. Storage access, implementation, hardware utilization, and other operations still affect the electrical result.
The distinction between the two ratios is worth retaining. “Sixteen times more context” does not automatically mean “sixteen times more total computation,” and neither phrase establishes “sixteen times more electricity.” Each step requires its own calculation or measurement.
Other requests share the available memory
A deployed service often processes several sequences together. Their model weights may be shared, but unrelated conversations usually require their own retained context. A long sequence therefore uses memory that could otherwise accommodate other sequences or a different execution arrangement.
Suppose thirty-two unrelated conversations each require the shorter 2.68-gigabyte cache from the example. Their combined cache alone is around 86 gigabytes, before weights and other storage. The arithmetic is simple; the operational consequence is substantial. The server cannot choose a batch size without considering how much context it must retain for each member.
This helps explain why a workload can change energy per token indirectly. More context means more attention work and memory traffic, but it can also reduce the number of simultaneous sequences that fit, which in turn affects how effectively the hardware reuses its loaded weights across requests.
For a person using an AI service, the immediate lesson is descriptive rather than prescriptive. A long report may be essential to the task. Its presence simply needs to remain visible when comparing energy results. An answer produced from a short prompt and an answer produced from a full dossier are different experiments, even when both answers contain fifty tokens.
Next comes the other side of that relationship: how shared work and a different execution order can reduce energy without changing the task being requested.
