In the intervening months close to 400 people have spoken to our little bot and in general I think it’s coped quite well.
I detailed the aims of our chatbot and the design process in a previous article, but in a nutshell the plan was to:
- Provide a proof of concept that ties to the event’s themes.
- Try to do something new and interesting alongside our usual event marketing.
- Learn about this new technology and write about it for our audience.
We’re currently in the process of updating the bot with new content such as updated speaker and sponsor information, and a few days before Supercharged we’ll make further changes so the bot is ready to help answer questions on the day of the event.
But what have we learned about chatbots so far?
1. People have high expectations of chatbots
As with any new tech people have different expectations of what our chatbot should be able to do.
We were very upfront about the fact that our bot offers simple functionality and doesn’t use AI. It is built to offer information on our Supercharged event and nothing more. As such it works off a decision tree, offering information based on the user’s answer to pre-programmed multiple choice questions.
A majority of people went with the flow and were happy to navigate the bot using the menu options.
However, some people understandably want to test the boundaries of what the tech can do (which is one of the reasons we built it). This led to a decent proportion of users asking free text questions to see if the bot could process natural language. More than one person asked the bot if it was AI.
We’d programmed a few responses to common free text questions and updated this based on the bot’s interactions with users.
Even so, a few people got very frustrated with the bot’s limitations and its inability to understand natural language. I’m really not quite sure what they were expecting – if we’d created one of the world’s first AIs capable of having a conversation on any topic you desire, then we’d probably have made a bit more fanfare about it.
That said, I can see why people would question why we’ve built a chatbot that can’t actually chat.
2. They probably shouldn’t be called chatbots
In relation to my previous point, I think the name ‘chatbot’ is quite misleading. In fact, Facebook has said that they should be referred to simply as ‘bots’.
Speaking to Recode, Facebook’s VP of messaging products, David Marcus, said that brands shouldn’t necessarily be rushing to create conversations with users in Messenger. Instead they should be trying to build simple experiences that help the user achieve their goal.
“The reality is, if you can tap on a button that has a word in it, rather than typing it, you’d much rather do that,” he said.
So menus and multiple choice questions are actually the way forward, rather than trying to create some ground-breaking AI within Messenger. Thomas Claburn over at The Register might well have uncovered Facebook’s real plan:
“Let’s call a spade a spade. Facebook is trying to cram apps into its mobile chat software.”
3. Broadly it worked very well
Overall we’ve been quite pleased with the response to our chatbot. Almost 400 people have tried it out, with only a minority of people getting frustrated by the experience.
In general people seemed willing to interact using the menu buttons and were happy just to use the bot for its stated purpose – to tell users about Supercharged. 15 people even signed up for future alerts via the bot.
As long as you’re able to manage people’s expectations and keep your bot’s functionality focused on specific tasks, then Messenger bots definitely have their uses.
4. You get some really interesting data from chatbots
Byte London were able to provide a lot of data from our bot’s interactions (I’ll shy away from calling them conversations), all pulled from Facebook Analytics. For example, we can see that users were mainly men aged 25-34.
Nobody aged under 25 spoke to the bot and the most common operating system was Windows, which tallies with the fact that people will mainly be interacting with the bot via their work computer.
This chart shows the peaks that occurred when we sent out marketing comms or blogged about the bot.
One of the conversation routes within the chatbot also resulted in users answering a number of questions. As only 77 people answered the questions and we haven’t done any weighting the results aren’t in any way robust, but they are also quite interesting.
For example, when asked what they thought was the best use case for chatbots, 51 people said customer service compared to just 5 entertainment and 2 for commerce.
5. Some people are mean…
Just finally, though most people were polite to our poor little bot, others were less so.
I’ll leave you with this screenshot from what turned out to be a very long and at times quite edgy interaction between our bot and Parry Malm, who is one of Econsultancy’s favourite guest bloggers and also CEO of AI marketing firm Phrasee (one of Supercharged’s sponsors).
Parry quickly twigged that the bot’s responses to free text worked by picking up certain keywords…
Don’t forget to buy your tickets for Supercharged on July 4 to find out how AI will impact marketing. Speakers include ASOS, JustEat, Shop Direct, BT and First Utility.