Building Token-Efficient AI Architecture for Applications

A product team can spend weeks building an AI feature that works perfectly during testing. The demo looks good, the team is happy with the results and the feature is ready for real users. The problem often appears later, when thousands of people start using it and the AI bill begins to grow much faster than expected. 

This happens because AI costs are not determined only by the price of the model you choose. Every request sent to an AI model uses tokens and those tokens add up as more people use the application. The amount of information you send to the model, how often you call it, how much information it returns and which model handles each task can all affect your final cost. A small amount of unnecessary usage may not seem important during testing but at production scale, the same waste can repeat thousands or millions of times.
That is why token efficiency should be considered while you are designing an AI application rather than treated as a problem to solve after the bills start increasing. The right architecture can help you control costs without reducing the usefulness of the application or making users work harder.

Coinbase offers a useful real-world example. As the company's use of AI grew, so did its AI spending. Instead of simply asking employees to use AI less, the company changed how its AI systems handled requests. It improved how requests were directed to different models, made better use of repeated information and used lower-cost models for tasks that did not require the most powerful option. These changes helped Coinbase reduce its AI spending by about half while its overall AI usage continued to grow.

The example shows why token efficiency is not simply about cutting AI costs after launch. It is about making better architectural decisions from the beginning so your AI application can grow without unnecessary costs.

Tokens Are the Hidden Unit Economics of AI

Every large language model prices its work in tokens, the small chunks of text a model reads and generates. That single unit quietly sets three things engineering and business leaders both care about: what a transaction costs, how fast a response comes back and how much information a model can hold in mind at once before it starts dropping context. Get token usage wrong, and all three degrade together. A bloated prompt does not just cost more; it also takes longer to process and eats into the context budget you need for the actual task.

What makes this different from a typical cloud cost problem is that token waste is per request. A cloud instance that is slightly oversized wastes money sitting idle. A prompt that is slightly bloated wastes money on every single call, and that waste multiplies by every user and every interaction the moment the feature goes live.

According to a recent Forbes report, companies are becoming more careful about how they use AI models as their usage grows. Instead of using the most expensive and powerful model for every task, businesses are choosing less expensive models for simpler tasks and saving the more advanced models for work that actually needs them. This shows why controlling token usage has become an important part of managing the overall cost of an AI application.

The irony is that falling prices have not translated into falling bills. Gartner's forecast puts global AI spending at $2.6 trillion this year, a 47% year-over-year jump, and most of that growth is not coming from higher per-token prices. It is coming from how much more of the application stack now runs through a model at all and how many times each user action quietly triggers a model call behind the scenes.

Where AI Tokens Actually Leak

Before trying to reduce AI costs, it is important to understand where the extra usage is coming from. In many applications, the problem is not that the model is too expensive. The problem is that the application keeps sending more information to the model than it actually needs.

This can happen in several ways. A prompt may become unnecessarily long, a conversation may carry months of old messages, an AI system may receive information about tools it never uses, or the model may produce a much longer answer than the application needs. Each of these may seem like a small issue on its own, but at thousands of requests a day, the extra usage can add up quickly.

Here are five common places where AI applications tend to use more tokens than necessary.

1. Bloated system prompts and instruction sprawl

The instructions given to an AI model often start simple. As the product develops, teams keep adding new rules, examples and special cases whenever they want the AI to handle another situation. Over time, a prompt that was once a few lines can become several pages long.

The problem is that much of this information may have nothing to do with a particular request, yet the application can still send the entire set of instructions every time someone uses it. If an AI assistant handles thousands of requests each day, repeatedly sending information that is not needed can add significantly to its overall usage.

For example, imagine a customer service assistant that has been built to handle returns, refunds, delivery issues, product questions, account changes and complaints. A customer asking, “Where is my order?” does not need all the instructions related to refunds, account changes or complaints. Yet if the application sends all of those instructions with every request, the AI is processing a lot of information that has no role in answering that particular question.

A better approach is to keep the basic instructions focused and provide additional information only when a particular request actually needs it. This keeps the AI focused on the task while also avoiding unnecessary usage.

2. Context stuffing instead of retrieval

When an AI application needs information from company documents or a large collection of files, one simple approach is to send a large amount of information to the model and let it find the answer.

That can work when there are only a few documents. It becomes much less practical when the amount of information grows. A customer asking about the return policy does not need an entire product catalogue, company handbook and years of customer records included in the request. A better approach is to first find the information that is relevant to the question and then give that information to the AI.

For example, imagine an online retailer with thousands of product documents. If a customer asks whether a particular laptop can be returned within 30 days, the AI does not need information about every product the company sells. It only needs the relevant return policy and perhaps the details of that particular product.

This is one of the main ideas behind RAG, where an application searches its information first and gives the AI only the useful parts.

3. Replaying full conversation history

Long conversations create another easy-to-miss source of unnecessary usage.

Imagine a customer has exchanged twenty messages with an AI assistant. When the customer asks the twenty-first question, the application may send all twenty earlier messages along with it so the AI knows what has been discussed. The same thing happens again with the twenty-second question, then the twenty-third and so on. The conversation becomes longer, but the customer may still be asking very simple questions.

Consider a travel assistant helping someone plan a two-week holiday. Early in the conversation, the customer may discuss their budget, preferred destinations, hotel choices and travel dates. Several messages later, they may simply ask, "What time does the train leave?" Sending the entire conversation again may not be necessary to answer that question.

A better approach is to keep the recent conversation available while summarising older information or storing important details separately. The AI can then use those details when they are actually relevant instead of receiving the entire conversation every time.

4. Tool schemas loaded on every call

Modern AI applications can connect to many different services. An assistant might be able to search company documents, check an order, update a customer account, schedule a meeting or look up information in a database. The problem occurs when the application gives the AI information about all of these options for every request, even when most of them are irrelevant.

For example, if a customer asks, "Where is my order?", the AI may only need access to the order system. There is little reason to give it information about the company's calendar, employee directory or meeting system for that particular question.

As AI assistants become capable of handling more tasks, this can become a bigger issue. The application should give the AI access to the tools it is likely to need for the current task rather than presenting every possible option each time. This is especially important for AI agents that work across several business systems.

5. Unconstrained, verbose model output

When people ask an AI model an open-ended question, it often provides a detailed answer with explanations, introductions and additional information. That can be useful when someone wants to learn about a topic. It is less useful when the AI is working inside another application and the software only needs a small piece of information.

For example, an insurance application may ask an AI system to identify whether a claim is related to a vehicle accident, theft or property damage. The application may only need the category "Vehicle accident." There is little value in having the AI produce several paragraphs explaining why it selected that category. Those extra words increase usage without improving what the application actually needs.

The same principle applies to business applications that ask AI to extract names, dates, prices or order numbers from documents. If the next step only needs those specific details, the AI does not need to produce a long explanation around them.

The goal is not to make every AI response as short as possible. A customer who asks for a detailed explanation should receive one. The goal is to make the response as detailed as the task requires, without adding information that serves no purpose.

Six Approaches for Developing Token-Efficient Architecture

The problems above are usually not caused by one major mistake. They come from small design decisions that become expensive when an AI application starts handling a large number of users and requests. The good news is that you can address many of these problems while designing the application rather than waiting until the costs become difficult to control.

Here are six practical approaches you can consider when building an AI application that needs to remain affordable and reliable as it grows.

1. Retrieval over injection

When an AI application needs information from company documents, databases or other sources, you do not need to give the model everything you have. You can first search for information related to the user's question, then provide only the most useful parts to the AI.

This approach is commonly known as retrieval-augmented generation (RAG). Instead of sending all available information to the AI, RAG helps the application find the most relevant information first and use it to answer the user's question. Microsoft's guide to Retrieval-Augmented Generation explains how this approach can help AI applications work with large amounts of business data while keeping the information sent to the model focused and relevant.

For example, imagine you are building an AI assistant for an insurance company that has thousands of policy documents. If a customer asks whether their policy covers damage caused by flooding, there is no reason to send all those documents to the AI. Your application can first find the policy and section related to flood damage and then give that information to the model.

This approach can make the response more focused while reducing the amount of information the AI has to process. You can make it even more useful by organising documents according to their meaning rather than simply cutting every document into pieces of the same size.

2. Prompt caching

Many AI applications send the same information with every request. This could include the application's basic instructions, company information, examples of how the AI should respond or other information that rarely changes. Instead of processing all of that information from the beginning every time, you can use prompt caching to reuse information that has already been provided. This is particularly useful when your application receives a large number of similar requests.

A good recent example comes from ProjectDiscovery, whose security-focused AI system was sending the same 4,000-token instructions and tool information with each request. Its cache was initially being used for only 7% of the relevant information. After changing how the prompt was organised, the cache hit rate increased to 74%, and the reported monthly AI bill fell by 59%.

For example, if you build an AI customer support assistant, its basic instructions and company policies may remain the same for thousands of conversations. You can reuse that information instead of treating it as completely new information every time.

The more often the same information is used, the more valuable this approach becomes.

3. Model cascading and routing

You do not need to use the most powerful and expensive AI model for every request. Some tasks are simple enough for a smaller and less expensive model, while more difficult tasks may need a more capable one. You can therefore design your application to start with a less expensive model for straightforward requests and use a more advanced model only when the task actually requires it. This is often called model routing.

For example, imagine you are building an AI customer service assistant. A question such as "What are your delivery charges?" is relatively simple. A smaller model may be able to answer it accurately. A request such as "Compare these three insurance policies and explain which one is most suitable for my situation" may require a more capable model.

Research behind FrugalGPT demonstrated how combining different models can significantly reduce costs while maintaining strong results. Another approach, RouteLLM, explored the automatic selection of which model should handle a request.

The important point is that you do not have to choose between quality and cost for the entire application. You can make that decision based on the type of work being done.

4. Deferred tool loading

An AI agent can be connected to many different tools. It might be able to search documents, check an order, send an email, schedule a meeting or update a customer record. You do not necessarily need to give the AI information about all of those tools every time it receives a request. You can make the relevant tools available when they are actually needed.

For example, suppose you have built an AI assistant for an online retailer. It can check orders, process returns, answer product questions and contact delivery partners. If a customer asks, "What is your return policy?", the assistant does not need to load the tools used for checking delivery status or contacting a shipping company. This approach becomes particularly useful as you add more capabilities to an AI agent. Instead of giving the AI a long list of everything it can do, you can let it access the right group of tools for the task.

This is no longer just a theoretical idea. Tools such as OpenAI's Tool Search allow AI agents to find and load tools when they need them, while Docker's agent documentation describes loading tools only when an agent actually needs to use them.

For you, the benefit is straightforward: the more tools your AI application has, the more important it becomes to avoid sending information about every tool with every request.

5. Memory compression

AI assistants can accumulate a lot of information during a long conversation. If you send the complete conversation back to the model every time the user asks something new, the amount of information being processed keeps growing. A better approach is to keep the details that are still useful and summarise or store older information separately.

Consider an AI travel assistant helping someone plan a two-week trip. Early in the conversation, the user may explain their budget, preferred hotels, travel dates and places they want to visit. After several days of planning, there is little reason to send every message from the beginning just to answer a question about tomorrow's train.

You could instead keep a summary containing the important details, such as the travel dates, budget and chosen destinations. The assistant can then use those details when they are relevant without carrying the entire conversation every time. This becomes even more important for AI agents that work on long tasks.

The goal is not to forget everything that happened earlier. It is to remember what matters without making the AI carry the entire history for every new step.

6. Constrained generation

AI models are good at producing detailed answers but your application may not always need one. If you are asking an AI to extract a customer's name, order number and delivery date, you do not need three paragraphs explaining the answer. You need those three pieces of information in a format your application can use. This is where structured responses can help. Instead of asking the AI to respond however it wants, you can tell it exactly what information you need and how that information should be organised.

For example, imagine an insurance company uses AI to read claim forms. Instead of asking the AI to "summarise this claim," you could ask it to return the customer's name, claim number, incident date, claim type and estimated amount in specific fields.

This makes the result easier for your software to use and can also prevent unnecessary text from being generated. There is a growing focus on this approach in production AI systems. For example, Amazon Bedrock introduced structured outputs in 2026, allowing businesses to specify the format they want the AI to return rather than relying on the model to produce correctly formatted information on its own.

The principle is simple, if your application needs five pieces of information, ask the AI for those five pieces rather than an entire essay about them.

6 Ways to Measure Token Usage

You can't optimize what you don't measure.  Once your application is running, the monthly AI bill only tells you how much you spent. It does not tell you why the cost increased, which feature is responsible, or whether the money you are spending is actually helping your users. To keep token usage under control, you need to measure a few key numbers regularly, such as:

1. Tokens Used Per Successful Task

One of the most useful numbers to track is how many tokens your application needs to complete a task successfully.

For example, imagine your customer support assistant uses 7,000 tokens to resolve a customer request. If another workflow can resolve a similar request using 3,000 tokens while providing the same quality of answer, the difference is worth investigating.

This metric also prevents you from focusing only on reducing token usage. If using fewer tokens results in poor answers or causes the user to ask the same question again, you have not really improved the system. The goal is to use a reasonable amount of information to achieve a useful result.

2. Cost Per Resolved Request

Token usage becomes more meaningful when you connect it to the result you want from the AI. If you use AI for customer support, you can measure the cost of successfully resolving one customer request. If you use AI to process documents, you can measure the cost of processing one document successfully. This gives you a better understanding of whether your AI spending is delivering value.

For example, your total AI bill may increase because the number of customers using your application has doubled. That is not necessarily a problem if the cost of resolving each request has remained stable. However, if the number of users has stayed roughly the same while the cost of each resolved request keeps increasing, you have a clear reason to investigate.

3. Input and Output Tokens

You should also track how much information your application sends to the AI and how much information the AI sends back. A sudden increase in input tokens may mean that your application is sending longer instructions, too much company information or unnecessary conversation history. An increase in output tokens may mean that the AI is producing longer responses than your application actually needs.

For example, if a customer support request normally uses around 2,000 input tokens and 500 output tokens but gradually increases to 6,000 input tokens and 1,500 output tokens, you should be able to investigate what changed before that increase becomes a much larger cost.

4. Cost by Model

If your application uses different AI models, track how much each model costs and what type of work it handles. You may find that a large number of simple requests are being sent to a more expensive model even though a less expensive model could handle them just as well. In that case, changing which model handles different types of requests could reduce your costs without changing the experience for the user.

This is why model selection should not be treated as a one-time decision. As your application grows, you should regularly check whether each model is still being used for the right type of work.

5. Cost by Feature or Workflow

Your total AI bill can hide where most of your spending is actually coming from.

Suppose your application has an AI search feature, a document processing feature and an automated customer support assistant. Looking only at the total bill will not tell you which one is responsible for most of the spending.

Tracking spending by feature or workflow can reveal that one part of your application is using far more AI than the others. You can then look at that particular workflow and find out whether it is sending too much information, making too many AI requests or using a more expensive model than necessary.

This is where data and ML engineering can play an important role. The right data systems can help you connect AI usage with individual features and business results instead of treating the entire application as one large expense.

6. Cost Per Agent Step

This metric becomes especially important when you are building AI agents. A user may see one action on the screen but the agent could be performing several steps behind the scenes. It may search for information, check a database, use a tool, review the result and then prepare the final answer. Each of these steps can involve another AI request.

If one step consistently uses much more than the others, you have found a place where the application may be wasting tokens.

For example, an AI agent helping an insurance company process claims might use one step to read the claim, another to check the policy and another to prepare a response. If the policy-checking step is responsible for most of the token usage, you can investigate whether it is receiving more information than necessary or whether it needs such a powerful model.

This level of visibility becomes increasingly important when you are building AI/ML applications or autonomous AI agents.

Watch How These Numbers Change Over Time

Measuring these numbers once is not enough. You need to watch them as your application, user base and AI workflows grow. A gradual increase in token usage for the same task can reveal a problem before it becomes expensive. Perhaps your instructions have become longer. Maybe more information is being added to every request. Your AI agent may also be making more calls than it did when the application was first launched.

For this reason, token usage should sit alongside the other production numbers you already monitor, such as response time, errors and system availability. Your monitoring and observability setup should help you spot these changes early rather than waiting for the monthly invoice.

The goal is not to reduce every number as much as possible. A lower token count is not useful if it leads to poorer answers or a worse customer experience. What matters is understanding how much you are spending, where that spending is happening and whether it is producing the result you need.

Building AI Applications That Can Scale

Token efficiency is ultimately not about using less AI. It is about making sure you are using AI in the right way. When you design an application, decisions about how much information to send, which model to use, how often the model should be called and how much information it should return can have a direct impact on its cost and performance. These decisions become even more important when an application moves from a small proof of concept to thousands or millions of real-world interactions.

Retrieval can help you avoid sending unnecessary information. Caching can reduce repeated work. Choosing the right model can prevent you from paying more for simple tasks. Loading tools only when they are needed can keep AI agents focused, while better memory management can prevent long conversations from becoming unnecessarily expensive. Measuring the right metrics then helps you understand whether these decisions are actually working.

For organizations planning LLM-powered applications or AI agents for real-world use, token efficiency should therefore be part of the AI strategy from the beginning rather than something considered after costs start rising.

Ready to build an AI application that can scale without letting unnecessary AI costs scale with it?

Daffodil Software helps organizations plan, build, and improve production-ready AI applications with the right combination of AI models, data, architecture, and engineering practices. Talk to our AI experts about your AI roadmap and explore how you can build an AI system that delivers the right results while keeping cost, performance, and reliability in balance.