​

I cannot believe that's already 22nd of April.

It's been a hectic month in a good way. For example,

• In my role as UK partnership lead for European Women on Boards, I've been preparing two virtual events to discuss how women can get on commercial boards as well as how to transition from board member to chairperson.
• This Thursday, I'll deliver the talk "Beyond the Offer Letter: Dispelling 7 Myths About the Gender Pay Gap" at the virtual Women in Tech Global Conference 2024. The keynote is based on this post.
• I felt many "sudden" AI experts were misleading people about AI. So I created an AI Readiness Checklist to set the record straight.

And I've been writing the article below.

It has taken me a lot of time because

1. It's about a topic I care deeply about - the intersection of customer support and AI - and I wanted to do a thorough review of the literature before writing the piece.
2. Chatbots are everywhere and it was important to me to show the impact beyond customer support.
3. I had so much material that it was hard to let go of many other angles - I have left out enough material for at least a second blog.

Without further ado...

​

AI Chatbots in Customer Support: Breaking Down the Myths

I’m a Director of Scientific Support for a tech corporation that develops software for engineers and scientists. One of the aspects that makes us unique is that we deliver amazing customer service.

We have records that attest to 98% customer satisfaction back-to-back for 14+ years. Moreover, many of our support representatives have been with us for more than a decade — some even three! — and we have people retiring with us each year.

That’s an unbelievable feat for a sector that is often seen as an operational cost, with high employee turnover, and a sink of one’s career. The worst? Support representatives are often portrayed as mindless robots repeating tasks without a deep understanding of the products and services they support.

It’s that last assumption that has spearheaded the assumption that one of the best uses of AI — and Generative AI in particular — is substituting support agents with an army of chatbots.

The rationale? We’re told they are cheaper, more efficient, and improve customer satisfaction.

But, is that true?

In this article, I review

• The gap between outstanding and remedial support
• Lessons from 60 years of chatbots
• The reality underneath the AI chatbot hype
• The unsustainability of support bots

​

Customer support: Champions vs Firefighters

I’ve delivered services all my commercial career in tech: Training, Contract Research, and now for more than a decade, Scientific Support.

I’ve found that of the three services — training customers, delivering projects, and providing support —the last one creates the deepest connection between a tech company and its clients.

But not all support is created equal so, what does great support look like?

And more importantly, what’s disguised under the “customer support” banner but it's a proxy for something else?

​

Customer support as an enabler

Customer service is the department with the mission to empower customers to make the most out of their purchases.

Whilst on the surface that may look like simply answering clients’ questions, outstanding customer service is delivered when the representative has been given the agency and tools to become the ambassador between the client and the organisation.

What does that mean in practice?

• The support representative doesn't patronise the customer, diminishes their issue, or downplays its negative impact. Instead, they focus on understanding both the problem and its impact on the client. This creates a personalised experience
• The agent doesn’t overpromise or disguise the bad news. Instead, they build trust by communicating on roadblocks and suggesting possible alternatives. This builds trust.
• The support staff takes ownership of providing a resolution to the issue, no matter the number of iterations necessary or how many colleagues they need to involve in the case. This builds loyalty.

Over and over, I've seen this kind of customer support transform users into advocates, even for ordinary products and services.

Unfortunately, customer support is often misunderstood and misused.

​

Customer support as a stopgap

Rather than seeing support as a way to build the kind of relationship that ensures product and service renewals and increases the business footprint, many organisations see support as

• A cost center
• A way to make up for deficient — or inexistent — product documentation
• A remedy for poorly designed user experience
• A shield to protect product managers’ valuable time from "irrelevant" customer feedback
• A catch-all for lousy and inaccessible institutional websites
• An outlet for customers to vent

In that context, it’s obvious why most organisations believe that swapping human support representatives for chatbots is a no-brainer.

And this is not a new idea, as some want us to believe.

​

A short history of chatbots

Eliza the therapist

​The first chatbot was created in 1966 and played the role of a psychotherapist. She was named Eliza, after Eliza Doolittle in the play Pygmalion. The rationale was that through changing how she spoke, the fictional character created the illusion that she was a duchess.

Eliza didn’t provide any solution. Instead, it asked questions and repeated users’ replies. Below an except of an interaction between Eliza and a user:

​

​

Eliza's creator — computer scientist Joseph Weizenbaum — was very surprised to observe that people would treat the chatbot as a human and would elicit emotional responses even through extremely short interactions with the chatbot

​

​

We now have a name for this kind of behaviour

​

​

Through the years, other chatbots have become famous too.

​

Tay, the zero chill chatbot

In 2016, Microsoft released the chatbot Tay on X (aka Twitter). Tay’s image profile was that of a “female” and it was “designed to mimic the language patterns of a 19-year-old American girl, and to learn from interacting with human users of Twitter.”

The bot’s social media profile was an open invitation to engage in conversation. It read: “The more you talk the smarter Tay gets.”

​

What could go wrong? Trolls.

They “taught” Tay racist and sexually charged content that the chatbot adopted. For example​

​

​

After a couple of attempts to “fix” Tay, the chatbot was finally shut down seven days later.

​

Chatbot disaster at the NGO

The helpline of the US National Eating Disorder Association (NEDA) served nearly 70,000 people and families in 2022.

Then, they replaced their 6 paid staff and 200 volunteers with chatbot Tessa.

The bot was developed based on decades of research conducted by experts on eating disorders. Still, somehow it was reported to offer dieting advice to vulnerable people reaching for help.

The result? Under the mediatic pressure of the chatbot’s repeated potentially harmful responses, the NEDA shut down the helpline. Now, 70,000 people were left without either chatbots or humans to help them.

​

Lessons learned?

Throughout these and other negative experiences with chatbots around the world, we may have thought that we understood the security and performance limitations of chatbots as well as how easy is for our brains to "humanise" them.

However, the advent of ChatGPT has made us forget all the lessons learned and instead has enticed us to believe that they’re a suitable replacement for entire customer support departments.

​

The AI chatbot hype

CEOs boasting about people being replaced by chatbots

If you think that companies would be wary of advertising that they are replacing people with chatbots, you are wrong.

In July 2023, Summit Shah — CEO of the e-commerce company Dukaan — bragged that they had replaced 90% of their customer support staff with a chatbot developed in-house on the social media platform X.

​

​

Note the use of the word “necessary” as a way to exculpate the organisation from the layoffs. I wonder about the feelings of loyalty and trust of the remainder 10% support team towards their employer.

And Shah is not the only one.

Last February, Klarna's CEO — Sebastian Siemiatkowski — gloated on X that their AI can do the work of 700 people.

​

​

In summary

• Klarna wants us to believe that the company is releasing this AI assistant for the benefit of others — clients, their developers, and shareholders — but that their core concern is about the future of work.
• Siemiatkowski only sees layoffs as a problem when it affects his direct employees. Partners’ workers are not his problem.
• He frames the negative impacts of replacing humans with chatbots as an “individual” problem.
• Klarna deflects any responsibility for the negative impacts to the "decision makers worldwide."

Shah and Siemiatkowski are birds of a feather: Business leaders shamelessly profiting from the AI chatbot hype.

​

When chatbots disguise process improvements

In some organisations, customer service agents are seen as jacks of all trades — their work is akin to a Whac-A-Mole game where the goal is to make up for all the clunky and disconnected internal workflows.

The Harvard Business Review article “Your Organization Isn’t Designed to Work with GenAI” provides a great example of this organisational dysfunction.

The piece presents a framework developed to “derive” value from GenAI. It’s called Design for Dialogue. To warm us up, the article showers us with a deluge of anthropomorphic language signaling that both humans and AI are in this “together”

​

or

​

Then, the authors offer us an example of what’s possible

​

​

Sounds amazing, doesn’t it?

However, if you’re thinking that it was a case of simply “swamping” humans with chatbots, let me burst your bubble — it takes a village.

Reading the article we uncover the details underneath that “transformation”

• They broke down the customer service agent’s role into multiple knowledge domains and tasks.
• They discovered that there are points in the AI–customer interaction when matters need to be escalated to the agent, who then takes the lead, so they designed interaction protocols to transfer the inquiry to a human agent.
• AI chatbots conduct the laborious hunt for information and suggest the agent a course of action.
• Engineers review failures daily and adjust the system to correct them.

In other words,

• Customer support agents used to be flooded with a hodgepodge of requests without any filtering between domains and tasks.
• As part of the makeover, they put in place mechanisms to parse and route the support requests depending on the topic and the action. That is, they upgraded their support ticketing system from an amateur "team" inbox to a professional call center.
• We also learn that customer representatives use the bots to retrieve information, hinting that all data — service requests, sales quotes, licenses, marketing datasheets — are collected in a generic bucket instead of being classified in a structured searchable way, i.e. a knowledge base.

And despite all that progress

• They designed the chatbots to pass the “hot potatoes” to agents
• The system requires daily monitoring by humans.

If you don’t still believe that this is about improving operations rather than AI chatbots, let me share with you the end of the article

​

​

In addition of process improvements, chatbots can also disguise human labor.

​

AI washing or the new Mechanical Turk

​

Historically, machines have often been used to provide a veneer of novelty to work performed by humans.

The Mechanical Turk was a fraudulent chess-playing machine constructed in 1770 by Wolfgang von Kempelen. It was a mechanical illusion that allowed a human chess master hiding inside to operate the machine. It defeated politicians such as Napoleon Bonaparte and Benjamin Franklin.

Chatbots are no different.

In April, Amazon announced that they’d be removing their “Just Walk Out” technology which allowed shoppers to skip the check-out line. In theory, the technology was fully automated thanks to computer vision.

In practice, about 1,000 workers in India reviewed what customers picked up and walked out with from the stores.

​

​

That is, Amazon wanted us to believe that whilst the technology was launched in 2018 — branded as “Amazon Go” — they still needed about 1,000 workers in India to train the model in 2022.

Still, no matter if the technology was “untrainable” or needed an army of humans to deliver the work, it’s not surprising that Amazon phased it out. It didn’t live up to its hype.

And they were not the only ones.

Last August, Presto Automation — a company that provides drive-thru systems — claimed on its website that its AI could take over 95 percent of drive-thru orders “without any human intervention.”

Later, they admitted in filings with the US Securities and Exchange Commission that they employed “off-site agents in countries like the Philippines who help its Presto Voice chatbots in over 70 percent of customer interactions.”

The fix? To change their claims. They now advertise the technology as “95 percent without any restaurant or staff intervention.”

Amazon and Presto Automation cases suggest that, in addition to making clear when chatbots use AI, we may also need to label some tech applications as “powered by humans.”

Of course, there is a final use case for AI chatbots: As scapegoats.

​

Blame it on the algorithm

Last February, Air Canada made the headlines when it was ordered to pay compensation after its chatbot gave a customer inaccurate information that led him to miss a reduced fare ticket. Quick summary below

• A customer interacted with a chatbot on the Air Canada website. More precisely, asked for reimbursement information about a flight.
• The chatbot provided inaccurate information.
• The customer got their reimbursement claim rejected by Air Canada because it didn’t follow the policies on their website, even when the customer shared a screenshot of his written exchange with the chatbot.
• The customer took Air Canada to court and won.

At a high level, nothing appears to look different from the case where a human support representative would have provided inaccurate information but, as always, the devil is in the details.

During the trial, Air Canada argued that they were not liable because their chatbot “was responsible for its own actions” when giving wrong information about the fare.

Fortunately, the court ordered Air Canada to reimburse the customer but this opens a can of worms.

• What if Air Canada had terms and conditions similar to ChatGPT or Google Gemini that “absolved” them from the chatbot’s replies?
• Does Air Canada also defect their responsibility when a support representative makes a mistake or is it only for AI systems?

We’d be naïve to think that this attempt at using an AI chatbot for dodging responsibility is a one-off.

​

The planetary costs of chatbots

​

Tech companies keep trying to convince us that the current glitches with GenAI are “growing pains” and that we “just” need bigger models and more powerful computer chips.

And what's the upside to enduring those teething problems? The promise of the massive efficiencies chatbots will bring to the table. Once the technology is “perfect”, no more need for workers to perform or remediate the half-cooked bot work. Bottomless savings in terms of time and staff.

But is that true?

The reality is that those productivity gains come from exploiting both people and the planet.

​

The people

Many of us are used to hearing the recorded message “this call may be recorded for training purposes” when we phone a support hotline. But how far can that "training" go?

Customer support chatbots are being developed using data from millions of exchanges between support representatives and clients. How are all those “creators” being compensated? Or should we now assume that any interaction with support can be collected, analysed, and repurposed to build organisations' AI systems?

Moreover, the models underneath those AI chatbots need to be trained and sanitized for toxic content; however, that's not a highly rewarded job. Let’s remember that OpenAI used Kenyan workers paid ​less than $2 per hour​ to make ChatGPT less toxic.

And it’s not only about the humans creating and curating that content. There are also humans behind the appliances we use to access those chatbots.

For example, cobalt is a critical mineral for every Lithium-Ion battery and the Democratic Republic of Congo provides at least 50% of the world’s supply. It’s mined by 40,000 children who are paid $1–2 for working up to 12 hours a day and inhaling toxic cobalt dust.

And 80% of electronic waste in the US and most of other countries is transported to Asia. Workers on e-waste sites are paid an average of $1.50 per day, with women frequently having the lowest-tier jobs. They are exposed to harmful materials, chemicals, and acids as they pick and separate the electronic equipment into their components, which in turn negatively affect their morbidity, mortality, and fertility.

​

The planet

The terminology and imagery used by Big Tech to refer to the infrastructure underpinning artificial intelligence has misled us into believing that AI is ethereal and cost-free.

Nothing is farthest from the truth. AI is rooted in material objects: datacentres, servers, smartphones, and laptops. Moreover, training and using AI models demand energy and water and generate CO2.

Let’s crack some numbers

• ​Luccioni and co-workers estimated that the training of GPT-3 — a GenAI model that has underpinned the development of many chatbots — emitted about 500 metric tons of carbon, roughly equivalent to over a million miles driven by an average gasoline-powered car. It also required the evaporation of 700,000 litres (185,000 gallons) of fresh water to cool down Microsoft’s high-end datacentres.
• It’s estimated that using GPT-3 requires about 500 ml (16 ounces) of water for every 10–50 responses.
• A new report from the International Energy Agency (IEA) forecasts that the AI industry could burn through ten times as much electricity in 2026 as it did in 2023.
• Counterintuitively, many datacentres are built in desertic areas like the US Southwest. Why? It’s easier to remove the heat generated inside the datacentre in a dry environment. Moreover, that region has access to cheap and reliable non-renewable energy from the largest nuclear plant in the country.
• Coming back to e-waste, we generate around 40 million tons of electronic waste every year worldwide and only 12.5% is recycled.

In summary, the scale efficiencies that chatbots are supposed to bring in appear to be based on exploitative labor, stolen content, and depletion of natural resources.

​

For reflection

Organisations — including NGOs and governments — are falling under the spell of the AI chatbot mirage. They see it as a magic weapon to cut costs, increase efficiency, and boost productivity.

Unfortunately, when things don’t go as planned, rather than questioning what’s wrong with using a parrot to do the work of a human, they want us to believe that the solution is sending the parrot to Harvard.

That approach prioritises the short-term gains of a few — the chatbot sellers and purchasers — to the detriment of the long-term prosperity of people and the planet.

My perspective as a tech employee?

I don’t feel proud when I hear a CEO bragging about how AI is replacing workers. Neither do I enjoy seeing a company claiming that chatbots provide the same customer experience as humans do. Nor do I appreciate organisations obliterating the materiality of artificial intelligence.

Instead, I feel moral injury.

And you, how do YOU feel?

Inclusion is a practice, not a certificate.
​
​patriciagestoso.com​

If you were forwarded this newsletter and you like it, you can subscribe here.

​​​​​

​Unsubscribe · Preferences​