Cassi: AI Chatbot Feedback Redesign

After delivering the main updates for i-Exchange, I was on the hunt for new problems to solve.

While conducting user research with a frontline colleague, I noticed they were using a tool called Colleague Assist. The project looked really interesting and I was looking for a new challenge, so I offered my help to improve it.

What's Colleague Assist?

Colleague Assist is an AI chatbot that provides Santander colleagues with guidance on how to serve customers. It's a bit like ChatGPT for banking. It uses i-Exchange, Santander's internal knowledge base, as its source of information to provide answers.

Joining the team

After speaking to the Product Owner I found out they were having trouble getting users to select their business area (audience) and only a fraction of responses were receiving feedback. They were happy for me to join the team. However there was a big problem.

There was no money left!

The developers agreed that front end changes such as copy and styles would count as business as usual and not incur additional costs, but this limited my options for updating the tool. In addition, there was a long change freeze coming up in three weeks, so it was a race against time.

Do what you can

Functional changes in the back end were out of the question, so I focused on what we could do. My knowledge of front end development was enough to understand what low cost changes I could make, so I focused on the UI instead of trying to create new journeys or improve the quality of responses directly.

In a bid to drive engagement with colleagues, I announced a competition to design a new logo and posted a banner on i-Exchange to promote it.

There was no time to plan workshops or even build a proper research plan, but there were plenty of colleagues using the platform who I could speak to.

Town hall meeting

I wanted to understand why users weren't giving feedback or selecting their audience. By putting a town hall meeting in place, I quickly understood user needs and learned more about their pain points.

What did the town hall reveal?

Heuristic analysis

No guidance

It wasn't clear that users should select an audience or how important it was to give feedback.

Simple errors

The audience selection was out of place, the feedback icons were wedged between responses, and don't get me started on colour contrast.

Unclear feedback journey

The icons were there, but users weren't clicking them. The start of the feedback journey wasn't clear enough.

The feedback journey only allowed discrete feedback with three vague options, causing users to drop off. The audience selection was poorly positioned and carried no explanation of why it mattered. Neither problem existed in isolation. At the root of both was a complete lack of guidance and context within the tool itself. Users didn't know why feedback mattered, didn't understand what audience selection did, and had no reason to engage with either.

The feedback problem

Users wanted to leave feedback about why the chatbot was wrong, but they only had three options which were too broad. This meant they didn't get to share their true thoughts and feelings, so they dropped off the journey.

Cassi uses feedback to improve its responses, guard railing those that receive multiple thumbs down. This is really important to reduce risk, so feedback is vital. I needed to look at how we received feedback to inform the model.

Additionally, the thumbs up and down were not clearly signalled, so people were likely missing the start of the feedback journey.

The audience problem

Users were not selecting their audience regularly enough. This caused responses to be guard railed for being wrong when really they were just shown to the wrong audience. The layout of the tool did not make sense, there were no prompts to choose an audience and no guidance for new users.

There needed to be a clearer journey and guidance for users to select their audience. Audience selection needed to become second nature, as all queries should be filtered by audience.

The Root Cause: Understanding

I was riding solo for this ideation phase. There were only three days before I needed to hand over designs to developers, so I had to move quickly and get mock ups ready super fast!

If only I had somewhere users could leave their feedback

I had built a good relationship with the Community Forum team, so I enquired about creating a new feedback forum for Colleague Assist. They said yes! It was agreed that Community moderators would monitor the forum and triage feedback to the Colleague Assist team. The discrete feedback options were gone, and even better, the Community forum was free!

Plugging the knowledge gap

I had noticed a complete lack of guidance on the tool. When colleagues were onboarded they were given training, but due to knowledge fade some might not recall the basics. Tooltips are commonly used to provide context where a user needs support, so I used one to keep things concise and provide key information.

When users first opened the tool there was a big empty space in the window. Could we use this to provide some initial guidance?

What about the audience filters?

The positioning of the menu did not clearly signal that an audience needed to be selected. The audience filter menu was positioned at the top of the screen, far away from where users entered their query. Would moving it closer be enough? The law of proximity would indicate yes, so I tried it out.

A clearer starting point for feedback

Adjustments to the thumbs up and down UI made the feedback journey stand out more.

A new tooltip

By providing key information in a tooltip, colleagues always had what they needed to use Colleague Assist effectively.

A new introduction

The empty space was now welcoming and provided quick guidance on best practice when opening the tool.

A new home for audience filters

Putting the audience selection directly above the query box made it more obvious that the user should select their audience before entering a query.

Just in case, I provided a fallback prompt if users entered a query without selecting an audience.

I was on my final day before handover, so I quickly put in some group calls with users to check my design choices. Users thought it was a big upgrade and they really liked the new option for feedback.

Last minute changes

Colleague Assist was always a web based tool, but I learned that many users were minimising their screen to take up a slim column on the right hand side. This meant they could use other windows while serving customers. The screen needed a smaller breakpoint. By reducing the size of the logo in the top bar and reducing spacing lower down, there was space to fit the chat window. I made sure larger screens were not impacted, but it went down to the wire.

Toast or overlay?

I created two ways of displaying the feedback journey. One was an overlay that the user would need to dismiss. The other was a toast that would appear for 10 seconds at the top of the chat window.

Users were far more keen on the toast. They made an excellent point that they would usually be speaking to a customer live, so overlay pop ups would really get in the way of providing good service.

Feedback

Users had a clearer path to mark responses and share their thoughts. The new toast went down a treat!

Rebrand

By engaging with our community, we had a new logo, new name and better brand for Cassi.

Tooltip

A simple tooltip held guidance for new users without getting in the way of colleagues who knew what they were doing.

Audience

Moving the audience field and adding a prompt made it much clearer that it needed to be selected.

The changes were delivered just before the change freeze and received plenty of positive feedback on the new Community forum page. A month after release, audience selections were close to 100% adherence and the feedback count was higher than ever. Best of all, I achieved this with no additional development cost.

Community support

By providing a dedicated forum for colleagues, they started sharing their thoughts and we were able to increase feedback engagement. Experienced colleagues were helping new starters and we gained more context for bad responses. Often the feedback was about a process on i-Exchange, so i-Exchange managers had the information they needed to improve processes.

The message was clear

The new thumbs up and down UI clearly signalled the start of the feedback journey. The tooltip and toast provided colleagues with the guidance they needed. We also linked up with training teams to communicate this further. Combined with the new forum, these changes meant that feedback rates increased by 30%.

Adherence improved

Prior to the update, 1 in 3 guard railed responses did not have an audience selected. In the first month following Cassi's deployment, there were 30 guard railed responses. In all cases, the user had selected their audience. This was a marked improvement and meant the team were spending more time reviewing the responses that really needed their attention.

And the winner is...

The competition to design a new logo went down a hit. We received dozens of submissions and even a new name suggestion! The designs were shown to a panel of judges. Aishat Arowosegbe won the competition by designing the Cassi logo.