.jpg)
Moving Beyond the Knowledge Base: What to Do When Your AI Agent Stalls
When content stops moving your resolution rates forward, the next gains come from data, actions and process.
By Lorcan McGeough, CX Optimization Specialist, Otonomee
Alongside our core services, Otonomee helps companies on their AI journey in customer support. As a certified Fin partner, we deploy and optimize AI agents for clients across a range of industries, and I work on those deployments in our CX Optimization team.
If you lead a support team and have invested in an AI agent to help manage everything, there's a good chance you'll recognise this problem. You invested in your Knowledge Base, and watched the AI agent resolution rate climb, then stop.
The Knowledge Base is usually the first place teams look to improve AI agent performance, but once the basics are in place, there's a lot more you can do to ensure it hits peak potential.
Consider the AI agent plateau phase a milestone, not a failure
Most successful AI agent installations will start with several quick wins. These will include the questions any good Knowledge Base can answer, such as returns policies or how a specific feature should work. Get your knowledge content right, and resolution on these queries can reach close to 90%.
But then it flattens. Resolution rates stall or even begin to falter as the agent struggles to deal with even slightly more complex or unpredictable customer queries.
The plateau can have more than one cause. Your agent may not be deployed on every channel, or there may be a product line or customer segment it can’t reach yet. The first step to get past this stage is analysis. You need to know exactly why the agent has stalled in your particular system. One of the most common causes is that the AI agent is hitting the limit of what knowledge alone can resolve.
For example, say 40% of your customer queries are informational, and your AI agent resolves nearly all of them. Well done. You've done the hard work on content. This leaves the other 60%, however. Those interactions need something a Knowledge Base can't give, such as specific customer data, or the ability to take an action and see it through to conclusion.
If this is where you’ve found yourself, it means your basic setup is working and you're ready for what comes next. Here are five lessons we’ve learned from our recent AI agent deployments and CX optimization projects.
1. Know which queries you're dealing with
You can't plan the next stage of support until you know two things: what your customers contact you about, and what exactly each query needs to be resolved.
Our process with clients starts with a taxonomy analysis. We take around eight weeks of customer conversations and sort each one into four types:
When this piece is done, you’ll have a clear idea of where to go next. I think of it as a two-layer taxonomy. The first is what to solve for (the topics your customers raise). The second is how to solve it (which of the four types each topic falls into).
Next step is to prioritize topics by three things: unresolved volume, time to value and business need.
Time to value matters more than teams often expect. Ease of build is rarely the only factor, and a process that returns high value in a short time should go ahead of one with huge potential but comes with complex requirements. Dependencies like new endpoints could stretch out the time before you see any meaningful return.
2. Put governance in place before you build
If your agent has plateaued on informational queries, you already know that keeping resolution rates up is a challenge, even on content-based answers. Achieving it takes strong organizational governance, especially for fast-moving products.
Connecting your AI agent to your business systems for personalized and action-based queries amplifies the need to have good governance in place. The agent can now read data and take actions on customers' behalf, meaning ungoverned changes to either a process or its underlying data can have an immediate impact on customers with high-stakes queries.
Incorrect answers can be easy to fix. Incorrect actions can have a costly effect on customer accounts, and problems scale as fast as the automation does. An AI agent acting on bad information or out-of-date processes causes problems even when it's only answering questions.
Once that same agent can take actions, clear ownership becomes even more critical. The first step is simple: make sure someone is accountable for your content and strategy, and that both are reviewed regularly.
3. Map the process before you build
Personalized queries look simple from the outside, but they take a bit of thought to ensure they’re correctly dealt with. “Where's my order?” is a question, but resolving it satisfactorily is a process.
Human agents bring a lot of context to customer interactions and often have the experience and good judgment to make quick decisions. The AI agent needs to confirm who the customer is, find the right order and pull its status. It needs to know what to say when the order is delayed, lost or already refunded. And crucially, it needs to know exactly when to hand over to a human.
Before building anything, then, it’s vital to ask three key questions:
- Does the data exist? Is there a source that holds what the agent needs?
- Can the agent reach it? Is it set up to access the right sources securely, or does the process or information actually live in someone's head? If the answer relies on what an experienced agent just knows, it’s not yet ready to automate.
- Is every step mapped? Do you know each point in the customer journey, including the exceptions?
The more a process depends on individual knowledge, the more mapping needs to be done first.
4. Build the requirements and the business case
This is where your AI strategy meets both your technical team and your business leaders.
Once you know which topics to target and what each one needs, that list becomes a set of requirements. For example, to resolve subscription queries, the agent needs customer-specific subscription data. If there's no existing API in place to retrieve it, building those endpoints needs to be factored into your roadmap.
That is a business decision, not just a technical one. What will it cost to build? How much volume will it resolve? What's the likely return?
(The taxonomy analysis and prioritisation from lesson 1 above will give you the detail you need to answer those questions.)
In many businesses, the role that owns this process is often called an AI Architect. Their job is to translate customer demand into clear requirements, so the team can focus on building the processes that will actually improve resolution rates.
5. Measure what “good” looks like for you
As you move into personalized and action-based queries, out-of-the-box metrics stop telling the full story. Start with three numbers, and look at how they connect:
People often confuse these rates, and then end up trying to fix the wrong thing. Say, for example, that voice makes up half of your customer contact volume, and your AI agent is only set up to run in chat and email. Therefore, its involvement rate is 50%. Then it resolves 80% of the conversations it handles, which sounds strong, but your overall automation rate is only 40% — because the agent never gets to see the queries of half your customers.
So one possible fix may not be more content or more connectors. It may just be deploying your agent into a new channel, and optimizing what you've already built for each one.
That's why I recommend putting your AI agent in every conversation from the start, even where it hands straight over to a human. Fin — Otonomee is a Fin Certified Partner — calls this a “Fin first” workflow. If you choose to expand topic by topic, and then channel by channel, it could mean rebuilding workflows again and again. If instead you implement one agent across every channel, it means you can build once and then keep adding to it.
Be aware, too, that tools define resolution differently. Some count several resolutions in a single conversation, so always make sure to check what's being counted.
Rather than working from vendor-defined success metrics, the strongest teams we’ve worked with make the effort to define what good looks like for them, and then measure performance accordingly. One of our clients built their own measure of total customer experience, tailored to its customers, which gave them a clear guiding light: are we doing well everywhere we need to, and if not, what do we need to improve?
Where to start
You don't need a full transformation to get things moving. Here are first three steps I'd suggest:
- Measure your volume. Sort a recent sample of conversations into informational, personalized, action-based and investigative, and rank topics by unresolved volume.
- Check your governance. Make sure someone owns your content and strategy and reviews them regularly, before you move into higher-stakes query types.
- Pick one personalized topic. Choose the one with the highest share of unresolved volume and map it end to end: data, access, steps and exceptions. Then work out the investment needed to resolve that share.
Summary
A plateau in resolution rates does not have to be a dead end. It could point to untapped channels or segments, or it could be a sign you've already mastered the basics and can now take a leap forward into more meaningful automations. Your initial analysis will tell you which.
One thing we’ve seen repeatedly is that the businesses that create the biggest impact are never those who have just invested in expensive new technology. They're those who have asked the right questions so that they know exactly what their customers need, and then build their CX optimization strategy around that.
.jpg)
