AI Enablement at Canadian Bank's Call Centre
Building AI tools that make sense to agent's workflows.
Background
A Canadian-based Bank wanted to test AI capabilities within certain areas of their organization before doing any kind of large-scale AI transformation and investment.
Their internal Call Centre is selected to pilot two different AI-led features, through two 3 to 4 week pilots, meant to decrease agent's average handle time (AHT) and increase their productivity during calls:
- AI Summaries (UC1): An AI-generated post-call summary that alliviates the need of Call Centre agents to type their own notes, offloading this task and freeing them to take more calls during the day.
- Magic Autofill (UC2): A browser API extension that allows agents to copy information between platforms and websites easily, removing the need for them to copy and paste information line-by-line.
How research ensured AI product adoption & fit
As AI features were deployed through two different pilot rounds at the Call Centre, my goal as a lead researcher for this project was to study agent's behaviour to ensure product adoption and proper fit.
In detail, research goals were to:
- Study AI features so they could be continuously iterated on to ensure successful adoption and fit from agents – how well (or not) do AI use cases fit the current ways in which agents work? How are agents adapting to a new way of working with AI features?
- Capture behavioural or cultural frictions that might impact adoption of AI features – how are current Call Centre rules agents live by change or not when introducing AI features into their workflows?
- Once pilot’s completed, provide a series of recommendations that set the Bank up for success if deciding to implement AI features across Call Centre agent platforms.
To do this, I selected the below methods:
Shadowing & IDI Sessions
- Shadowing sessions during real customer calls observing how AI features are used by agents IRL
- Open ended questions on experience using AI features – what works vs. what doesn't
- Moderated walkthrough of AI features through simulated tasks (e.g., completing a fraud form with Magic Autofill)
Agents and Sample Size
First research round, 13 agents in total:
- 5 agents using only UC1
- 5 agents using only UC2
- 3 agents with no AI features
Second research round, 14 agents in total:
- 5 seasoned agents using both UC1 & UC2
- 5 newly hired agents using both UC1 & UC2
- 4 agents with no AI features
Observational Insights shaping AI Product Decisions
Round #1 of IDIs and observational research allowed us to get an early understanding of the behavioural and cultural barriers at the Call Centre impacting adoption, which let us know how to iterate on the AI Product next.
UX Burden vs. Seamless Nature
- Magic Autofill behaves in an understated and invisible manner within agent's call platforms – seamlessly integrated within their workflows not needing extra steps to work.
- AI Summaries introduced extra steps to a previous 2-step process: AI Summary is generated → Agent goes to main Call platform → Opens ‘AI Summary’ tab → Agent copies AI summary → Agent goes back to Customer Platform → Agent opens ‘Notes’ tab → Agent pastes summary → Agent hits ‘Submit’.
Agents at the centre vs. not
- Magic Autofill elicited the least improvement comments from agents. It was designed with them in mind – solving for the annoyance of copying information line-by-line – so they embrace it from the start.
- Because AI Summaries replaces an agent-led task, typing a post-call summary, they have more to say about the quality of the notes: ‘not concise or succinct enough’, ‘misinterpreting charges’, ‘I need to proof-read it.
First round of research findings led to changes to both the AI product and approach on how pilots would be delivered in the Call Centre.
- Notification system added: UX feedback (e.g., pop-ups) were deployed as a result of seeing agents forgetting or being unsure AI features were 'on' and avaible while shadowing them.
- AI prompt iterated: Per agent feedback, the AI Summaries prompt was re-designed so post-call summaries could be more concise and succinct while focusing on more important parts of the call.
- Change management audience broadened: From frictions and cultural norms observed during research, decision was made to not only target agents during change management sessions but to also involve other stakeholders (like Managers) in the process.
Round #2 helped the team get a clearer understanding of the agent pushback towards AI Summaries, and the deep love and key behind Magic Autofill’s success.
AI and the risk of penalization
- Although improved from its’ first iteration, AI Summaries misses important 'good note' rubric: procedures performed by agent during call, which all agents regardless of tenure agree is a must have.
- Agents work harder with it: proof-reading summary, adding additional notes if needed, saying procedures out-loud so AI includes it in the summary, etc.
- For agents, trusting AI Summaries fully means risking potential penalization as procedures capture is how their performance is rated.
Alleviating the pressure of work
- Magic Autofill continues to ellicit little to no complaints from agents, some claiming 'they wouldn't change a thing'.
- Feature has become indispensable for agent's day to day tasks, and when not there its' absence is deeply felt: 'I had a technicall issue with it... I need it to work again!'
- Autofill’s success is attributed to eliminating a major mental stress in agents day-to-day, prone to happening when copying information from one platform to another: making mistakes with sensitive client data.
Final Product Recommendations
After two rounds of research and two separate 3-4 weeks pilots, we were pretty clear on why only one feature was being recommended.
✓ Yes to Magic Autofill
Magic Autofill solves for agents problems while tackling the org's business goal – agents take an avg. of ~5 secs to do tasks that would take an avg. of 39.75 secs before. It has integrated seamlesly into their processes. Productization is recommended to reap benefits across Call Centre.
✗ No to AI Summaries (for now)
Agents go faster with AI Summaries (copying a generated summary in ~3 secs avg.) but AHT still suffers (agents spend an extra ~15 secs avg. inspecting the note). They distrust it as current output risks their job security and performance metrics. Productization, with its current version, can't be recommended yet.
In order to continue testing AI Summaries and investigate its potential benefits, if wanted, Bank first needed to work on a couple of things:
- Notes Strategy: AI Summaries was a test in defining a non-standardized process – ‘good’ can mean many things to agents when it comes to a post-call note. Before deploying a future ‘good’ version of summaries (AI or not), Call Centre should: 1) co-define that version with agents 2) understand implications on work 3) deploy through pilots.
- Solve for inconsistencies or shop for new vendor: In-field research captured bugs and/or inconsistencies with product (e.g., french version containing gibberish). This could warrant a deeper dive into other vendors with better AI summarization capabilities.
- Run a longer pilot: Once product has been improved and standarization of notes has been put in place, a new pilot study (+4 weeks) can be run testing new version of AI Summaries to see if a change in Call Centre cultural norms affects acceptance of product.
Collaborators & Credits
Project was made while at Koru, a Venture Studio / Consultancy in Toronto, ON, Canada.
Team members included: Maryam Nabavi (Project Manager), Shawn Cole (Design), Rishika Negi (Product), Khaled Eniba (Biz Ops), Juan Musleh (Engineering), and Dan Whiffing (Dev).
Magic Autofill and AI Summaries design language and UX belongs to Shawn Cole who led Design and Change Management. Development and implementation of features and notifications belongs to Dan Whiffing.
Protected
Enter the password to view this project.
Incorrect password. Please try again.