Back in March we unveiled our very own Facebook Messenger chatbot, lovingly created to help promote our forthcoming Supercharged event which is all about how AI is impacting marketing.

In the intervening months close to 400 people have spoken to our little bot and in general I think it’s coped quite well.

I detailed the aims of our chatbot and the design process in a previous article, but in a nutshell the plan was to:

  • Provide a proof of concept that ties to the event’s themes.
  • Try to do something new and interesting alongside our usual event marketing.
  • Learn about this new technology and write about it for our audience.

The bot was built by our friends over at Byte London, who will also be speaking at Supercharged on July 4 (you can see the full agenda here).

We’re currently in the process of updating the bot with new content such as updated speaker and sponsor information, and a few days before Supercharged we’ll make further changes so the bot is ready to help answer questions on the day of the event. 

But what have we learned about chatbots so far?

1. People have high expectations of chatbots

As with any new tech people have different expectations of what our chatbot should be able to do.

We were very upfront about the fact that our bot offers simple functionality and doesn’t use AI. It is built to offer information on our Supercharged event and nothing more. As such it works off a decision tree, offering information based on the user’s answer to pre-programmed multiple choice questions.

A majority of people went with the flow and were happy to navigate the bot using the menu options.

However, some people understandably want to test the boundaries of what the tech can do (which is one of the reasons we built it). This led to a decent proportion of users asking free text questions to see if the bot could process natural language. More than one person asked the bot if it was AI.

We’d programmed a few responses to common free text questions and updated this based on the bot’s interactions with users.

Even so, a few people got very frustrated with the bot’s limitations and its inability to understand natural language. I’m really not quite sure what they were expecting – if we’d created one of the world’s first AIs capable of having a conversation on any topic you desire, then we’d probably have made a bit more fanfare about it. 

That said, I can see why people would question why we’ve built a chatbot that can’t actually chat.

2. They probably shouldn’t be called chatbots

In relation to my previous point, I think the name ‘chatbot’ is quite misleading. In fact, Facebook has said that they should be referred to simply as ‘bots’.

Speaking to Recode, Facebook’s VP of messaging products, David Marcus, said that brands shouldn't necessarily be rushing to create conversations with users in Messenger. Instead they should be trying to build simple experiences that help the user achieve their goal.

“The reality is, if you can tap on a button that has a word in it, rather than typing it, you’d much rather do that,” he said.

So menus and multiple choice questions are actually the way forward, rather than trying to create some ground-breaking AI within Messenger. Thomas Claburn over at The Register might well have uncovered Facebook’s real plan:

"Let's call a spade a spade. Facebook is trying to cram apps into its mobile chat software."

3. Broadly it worked very well

Overall we’ve been quite pleased with the response to our chatbot. Almost 400 people have tried it out, with only a minority of people getting frustrated by the experience.

In general people seemed willing to interact using the menu buttons and were happy just to use the bot for its stated purpose – to tell users about Supercharged. 15 people even signed up for future alerts via the bot.

As long as you're able to manage people's expectations and keep your bot's functionality focused on specific tasks, then Messenger bots definitely have their uses.

4. You get some really interesting data from chatbots

Byte London were able to provide a lot of data from our bot’s interactions (I’ll shy away from calling them conversations), all pulled from Facebook Analytics. For example, we can see that users were mainly men aged 25-34. 

Nobody aged under 25 spoke to the bot and the most common operating system was Windows, which tallies with the fact that people will mainly be interacting with the bot via their work computer.

This chart shows the peaks that occurred when we sent out marketing comms or blogged about the bot.

One of the conversation routes within the chatbot also resulted in users answering a number of questions. As only 77 people answered the questions and we haven’t done any weighting the results aren’t in any way robust, but they are also quite interesting.

For example, when asked what they thought was the best use case for chatbots, 51 people said customer service compared to just 5 entertainment and 2 for commerce.

5. Some people are mean...

Just finally, though most people were polite to our poor little bot, others were less so.

I’ll leave you with this screenshot from what turned out to be a very long and at times quite edgy interaction between our bot and Parry Malm, who is one of Econsultancy's favourite guest bloggers and also CEO of AI marketing firm Phrasee (one of Supercharged's sponsors). 

Parry quickly twigged that the bot's responses to free text worked by picking up certain keywords...

Don’t forget to buy your tickets for Supercharged on July 4 to find out how AI will impact marketing. Speakers include ASOS, JustEat, Shop Direct, BT and First Utility.

David Moth

Published 8 June, 2017 by David Moth

David Moth is Editor and Head of Social at Econsultancy. You can follow him on Twitter or connect via LinkedIn

1720 more posts from this author

You might be interested in

Comments (3)

Pete Austin

Pete Austin, CINO at Fresh Relevance

Re: "Even so, a few people got very frustrated with the bot’s limitations and its inability to understand natural language. I’m really not quite sure what they were expecting"

Personally I expected something as good as Eliza, from the 1960s (50 years ago). That could chat just fine by recognizing keywords and combining scripted responses with extracts from what you had said previously. Or maybe a text adventure game from the 1980s.
https://en.wikipedia.org/wiki/ELIZA

But it was actually worse than Eliza, and understood literally nothing I said, including when I echoed back its initial suggestion for how to start. It didn't even help out by suggesting questions that it would understand, or respond usefully to one-word entries such as "help".

In brief, if this is typical, then I agree they shouldn't be called chatbots.

6 months ago

Michael Thomas

Michael Thomas, Managing Director at Rolling Star Ltd

So, at a basic level without the AI piece, is this another form of a phone menu, "press 2 for support, press 3 for sales...'?

5 months ago

Avatar-blank-50x50

Engtai Bot, Bot at Engati

David moth very well written. I agree with most of the tenets and the conclusions that you have drawn. Am also an avid blogger in this space and tend to look at it more from an evolutionary platform perspective, Read more about it https://goo.gl/NCW5rR

5 months ago

Comment
No-profile-pic
Save or Cancel
Daily_pulse_signup_wide

Enjoying this article?

Get more just like this, delivered to your inbox.

Keep up to date with the latest analysis, inspiration and learning from the Econsultancy blog with our free Digital Pulse newsletter. You will receive a hand-picked digest of the latest and greatest articles, as well as snippets of new market data, best practice guides and trends research.