Context windows keep getting bigger and bigger these days. As a result, more and more people seem to be confusing it with a database when they stuff an entire document into their prompt. I get the desire to skip a retrieval pipeline because it keeps things simple when you don’t need a vector database, a chunking strategy, or infrastructure overhead but…it’s a bad idea.

It can work for some scenarios but in the long run it always seems to catch up with you and when it does…something goes terribly wrong in production.

As I said, I get it, it makes sense. Simple seems to be the better route at first. Retrieval pipelines add overhead both from a design and management perspective, and mean that you have to make decisions that you may not know how to make:

What embedding model do I use? How do you manage a vector database? How do you chunk documents? How to tune it all so that it runs smoothly?

It doesn’t help the situation that vendors make it seem like you don’t need to think about any of this because they make it seem like the 1 million token context is a silver bullet at first, and the demos wow you. And honestly, for a lot of use cases they may not be wrong such as tools with a small document set, prototypes, and applications with a stable document corpus overall that has low traffic.

It’s a slippery slope.

The failure modes are predictable enough that you can roughly order them by when teams tend to hit them — first on the bill, then on quality, then in a post-incident review.

Cost and latency at scale

A million context token window is impressive but it’s also expensive to process. Running a million tokens through a model will cost you per request, and as you scale it quickly adds up.

Also, larger context windows take longer to process, which is fine and dandy when you are testing, but will become noticeable for your users as they sit and wait. This is typically the first failure mode teams hit and is the easiest one to avoid if you pay attention to your monitoring.

Attention degradation

Models don’t read context the way you do. There’s a well-documented phenomenon where content buried in the middle of a long context gets underweighted, meaning the model pays more attention to what’s at the beginning and the end. So you can stuff tons of documents in there and the model will silently ignore large chunks of them.

The problem is that nothing breaks. The system keeps running. It just gives worse answers in ways that are hard to pin down unless you know exactly where to look. Most teams don’t. They spend weeks tweaking prompts and questioning the model before they finally figure out what’s actually happening.

Stale data, no update path

When you stuff documents into a prompt, you’re stuffing a snapshot. That’s fine until documents change, and they always do. Policies get updated. Products get discontinued. Procedures get revised.

With a retrieval pipeline, you have something to update. Re-index the document, refresh the embedding, set an expiry. With context stuffing, you have none of that. The model just keeps using the old version and nothing in the system flags it. Users just get wrong answers delivered.

No observability

This one usually shows up after the first production incident (trust me, it will).

When something goes wrong, and it will, you want to know why. Which documents did the model actually use? What did it draw on? With a retrieval pipeline, you at least have a record. With context stuffing, you have nothing. The whole context was available, the model did something with it, and now you’re debugging by guessing.

That’s a bad place to be.

Before you commit to a final architecture, ask yourself four questions:

How often do your documents change? Rarely, with a clear refresh process? Stuffing is probably fine. Frequently, or by multiple teams? You want a retrieval layer with proper invalidation.

How much traffic are you expecting? Internal tool with light usage, the cost math may work. User-facing at any real scale, even a few hundred requests a day, run the numbers before you commit. The total cost may surprise you.

Do you need to audit what the model used? If you’ll ever need to debug a production incident, the answer is yes. Regulated industries make it non-negotiable (financial institutions, etc.).

How many documents are you actually working with? Ten is different from one hundred. The attention problem is manageable at a small-ish scale. It becomes a real problem as that number grows.

Stable documents. Light traffic. No audit requirements. Small document set. Check all four and context stuffing is probably the right call, at least to start. Can’t check all four? You’re going to want a retrieval pipeline sooner than you think.

Context stuffing isn’t wrong. It’s a reasonable starting point that a lot of teams are going to outgrow. The ones that do it well are the ones who went in knowing what they were trading away.

Know the failure modes. Run the numbers before you scale. Build the retrieval layer before you need it. Fin.