Analytics in Conversational AI – Key metrics to track customer success

Technology advancements have enabled AI-powered conversational bots to strike conversation like human agents to address and resolve customer queries efficiently.

Why should you measure a bot’s performance?

For any business, no matter what technology is used, the end goal is to give customers the correct information quickly.

While quantitative metrics like calls per hour and the cost incurred per call as essential to track, it is equally important to focus on the other side of the coin – the qualitative metrics.

Qualitative metrics, namely customer satisfaction scores, help retain customers and improve customer retention.

Conversational AI bots are one such technology businesses adopt that plays a massive role in driving qualitative metrics by providing instant customer service. Each conversation made by the AI bot has a wealth of information to make a business impact. To check whether the bot is aligned with the organization’s goals and make necessary interactions with customers, it is imperative to monitor the bot interactions and analyze the data to measure its performance.

How to measure a bot’s performance?

To measure a bot’s performance, you must first understand what metrics to track and how to follow them. Each use case requires a different set of metrics to monitor.

Choosing the right set of performance metrics is key to measuring business objectives. Depending on the type of use case and the industry, different sets of metrics gain importance.

Top 3 critical metrics to measure your bot’s performance :

1. Engagement metrics: help determine the level of engagement the bot has with the caller. Higher engagement should translate to better customer experience.

The engagement metrics include the following:

• Retention Rate – The percentage of repeated callers in a specific period of time. This indicates what percentage of callers are getting help from the bot and are returning to talk to it.
• Bounce Rate – The percentage of callers who start interacting with the bot but abandon the conversation. This tells us whether the bot can keep the caller engaged in the conversation and solve caller queries—the lower the bounce rate, the better. If the bounce rate is high, check the logs and discover why callers are abandoning the conversation. It may be either because the bot needs help understanding what the caller is saying or is not yet trained to handle the particular query.
• Goal Completion Rate (GCR) – The percentage of conversations in which the bot solved caller queries. A higher GCR indicates a higher success rate for the bot.
• Fallback Rate – The percentage of conversations in which the bot couldn’t understand the caller’s query. A higher Fallback Rate means that the bot is unable to understand the caller and hence unable to solve the question.
• Human Takeover Rate – The percentage of conversations that had to be transferred to a live agent. A higher Human Takeover Rate would indicate that the bot is either unable to understand the caller or isn’t equipped enough to solve incoming caller queries.

2. Quality metrics: The quality metrics determine the average conversation quality. The metrics include the following:

• Customer Satisfaction Score – This is a score calculated from caller responses to an exit survey at the end of each conversation.
• First Call Resolution Rate – The percentage of conversations where the bot could resolve caller query in the first call itself without needing a follow-up call.

3. Caller metrics: The caller metrics help understand the profile of the callers interacting with the bot. The metrics include the following:

• Total number of callers – The total number of callers that have used the bot
• Active callers – The number of callers that have interacted with the bot in a specific time interval
• New callers – The number of callers who have used the bot for the first time in a particular time interval

The data guide to calculating bot metrics

Like tiny drops make an ocean, capturing the following raw data points facilitates accurate measurement of the metrics as mentioned earlier to make strategic business decisions:

These are the attributes that are associated with the caller. These include:

1. Phone number
2. IP address (for VOIP or webchat)
3. Geographic information

The details about the call that needs to be captured include the following:

1. Duration of the call
2. Start and end time
3. Date

Information about the conversation between the caller and the bot is captured here:

1. Conversation logs at a turn level. Each log should have the following:

• Identified topic and category
• Dialog state – waiting for user input, capturing information, fulfill, etc
• Sentiment score of the caller’s statement

2. Dialog state of termination

System information is to do with the performance of the system while serving the customer:

1. Time to process a query
2. Time taken for the backend system to respond
3. Instances of system failure

While the data mentioned above has to be captured in real-time, the metrics can be calculated with a delay. A delay in metrics calculation is acceptable as the objective is to derive meaning and make correct decisions, not quick ones. Therefore, a data pipeline with two ends can be built. The first end is where the real-time data is dumped, and the second end is where the processed data is presented for analysis.

Conclusion

As bot adoption continues to grow, it’s important to remember the fundamentals of measurement for businesses to optimize and leverage AI-powered bots. The tracked metrics vary according to business type, targeted industries, and how a company wants its bot’s performance to be evaluated. The right metrics in place help understand how customers are served and find areas for improvement.

Authors

Harish Lanka,
Jaya Prakash (JP), Solution Architect, Smart bots