Is Attention a Budget or a Curve?

Attention budget and attention curve both describe the same problem: a model gets less reliable as its context grows. It's the "lost in the middle" problem, and it's one of the things contributing to what's often referred to as context rot.

What I'm really interested in is how those two words create different mental models in our heads. I've obsessed about language in software for a long time, and this is no different.

What attention budget implies

An attention budget suggests there's only so much attention to go around. As the context grows, more things compete for it, and the model becomes less reliable at picking out what matters.

It's a good word to describe what's going on.

A budget is finite, and it runs out. Every tool call and every dead end you debugged is still sitting in there, costing you attention on every turn.

But budget doesn't capture how attention is distributed. In fact, it makes attention sound as though an important instruction carries the same weight wherever it appears in the context.

But that's not true.

There is some maths behind the budget metaphor. In standard transformer attention, the model scores how relevant different parts of the context are, then those scores are turned into weights that add up to a fixed total.

Essentially, giving more weight to one thing means giving less to something else.

So budget does a good job at describing how attention is finite and how there's only so much to go around. But it doesn't tell us how that attention is allocated.

What attention curve implies

The word curve communicates how attention isn't spread evenly across the context.

Models tend to pay more attention to the beginning and the end, with information in the middle being easier to lose.

Researchers call this lost in the middle, and Chroma's research shows that models still don't use long contexts uniformly. Where something appears in the context can affect how reliably the model uses it, and the curve describes the shape of that effect.

It's also why subagents help us manage our context window. They keep work inside the "smart zone", near the start of the window.

Budget and curve describe different things

Using the word budget to describe attention makes us think about it in terms of capacity. Using curve makes us think about how that attention gets distributed across the context.

I was writing something else when I caught myself switching between the two. I started wondering whether I was using the terms inconsistently, or whether I'd misunderstood something, so I stopped to write this and clear up my own thinking.

I'd assumed I was using one of the terms incorrectly and that I needed to pick one. But once I actually sat down and looked at what each word was doing, I realised they weren't competing terms at all.

They were describing different aspects of the same problem.

I'm not yet sure if a single word can describe an agent's attention as both a budget and a curve. Attention degradation might be the closest term I know that implies both. But, I'm not sure, yet. Heck, in six months, these descriptions might be irrelevant, anyways.

So here's where I've landed for now. When I'm talking about attention as a capacity, I'll say budget. When I'm talking about how it's distributed, I'll say curve. I'll keep using both and aim to be deliberate about which one I choose.

You might think I've got that the wrong way round, or have the perfect description. If you do, I'd like to hear from you.

Either way, both words push toward the same habit: keeping the window small.

I make videos about building software, where I spend most of my time hunting for the right metaphor for things like this.