WHY BOT ANALYTICS MATTER – WHY LOG BOTY STUFF ?
You’ve got a bot running. Or maybe you don’t? Then you should! Great ! But now what ? When you launch your bot into the wild, be it a commercial one or not, you usually want to know how users interact with it, what they gain from it and how you can make the experience better. But that certainly isn’t an easy task : who uses my bot, why are they using it, did they get what they expected, are my business objectives completed… Indeed, those are high level questions that can’t be answered by just taking a look at conversations held: you must dive into your bot analytics.
This is where data comes in. Data that will help you analyse what is going on, that will allow you to set well defined KPIs, to know your users and their expectations better, and maybe find out new use-cases and business opportunities. The evolution over time of your metrics also makes it possible to value the changes you make to your bot or to your marketing strategy.
WHAT BOT ANALYTICS DO I LOG ?
For starters, it is always useful to know when new users start a conversation with your bot. That will allow you to track the number of new users everyday. Logging messages makes it possible to know how many are sent everyday, as well as the average number of messages per conversation per day, and the number of messages per conversation. You can also know the number of returning users, you might say that they are the ones sending messages on a different day than the day the conversation started for example.
You can also log business goals. You haven’t set business goals for your bot? Check this tribune explaining why it’s necessary. For a pizza booking bot, it could be the selection of a pizza and it’s payment. For a support bot, it could be every time the user is satisfied with your help, either implicitly by matching intents and sentiment analysis, or explicitly with a satisfaction survey. In general, clicking on a link, asking for more information, asking questions outside the scope of the bot or finishing a conversation flow are useful things to log.
HOW TO LOG BOTY STUFF ?
In this section, we will show an example of a stack allowing us to register raw data, structure it and display it on a visualization dashboard. We will use Logstash, Elasticsearch and Kibana, but you can use anything you like, including a custom script fetching data from a simple database. We think a dashboard has a lot of advantages though : it’s easier to read, much easier to display graphs, it can be shown to colleagues, managers, business partners, it updates itself… It makes your bot analytics easier to read and process.
We will use “Movie Bot” as an example, you can chat with him here : link, he’s quite the nice chap.
So let’s get to the meat of it all. Or maybe tofu, that’s up to you.
On macOS, you can just do :
On other systems, you can download them on their maker’s website : elasticsearch, logstash, kibana. For each project, you will find the binary you are looking for in the
Make sure you have a Java8 JRE or JDK on your system.
By default elasticsearch will listen on port 9200.
Logstash will allow us to make bulk inserts into elasticsearch, set some fields (like the timestamp) automatically for us, have multiple input sources (files, HTTP APIs, other DBs…) and more.
In this example, we want a HTTP API we can call from anywhere, from any platform or language we might be using. Here is config file for that :
This config tells logstash to accept requests on port 4242. Here are some other input sources you might be interested in : link. By default, the
http input adds a
headers and a
host field, which I am not interested in, but you might be. We also tell logstash where to find our elasticsearch DB (no port specified = default elasticsearch port), and in which index to put our data. In this case, we want to have the freedom to specify the index to use as an argument to the API, so that’s what we do. You can find more output plugins here.
You can launch logstash like so :
$ ./bin/logstash -f logstash.config
INSERT A FIRST DOCUMENT
To allow Kibana to know about your indexes and the fields you are going to be using, make a first request to your newly created API. You might use
curl for example :
Or you might directly want to have some code doing it, you’ll probably be using that anyway later on. Here are some snippets :
To start kibana :
Once kibana is running, visit http://localhost:5601, or whatever port kibana tells you it is listening on, 5601 is the default. You’ll be asked to configure an index pattern, enter the name of your index and click on ‘Create’.
Then, you will be able to see the inserted documents in the ‘Discover’ tab.
You’re pretty much ready to go ! The next step is to create visualizations and dashboards for your bots analytics. After some tweaking, you will be able to have something along those lines :
In the top left, the number of conversations taking place right now. In the top right, a visualization of the most common intents of the bot. We also have the number of conversations, of messages, the number of messages per length (in words), as well as the percentage of movie searches in one of the 6 most frequent languages, and the raw number of searches. One very cool thing is that we can click on a day, or on an intent, and watch the entire dashboard change as a result (everything becomes filtered based on what you clicked on). Here is a small demo :
Finally as a last piece of advice, don’t forget to refresh fields when you add new ones, otherwise Kibana won’t know about them :
PS: If you wonder where to find this cute wooden robot pencil sharpener featured in the cover, check here.
🤖 Happy bot building 🤖
Have an issue with your bot? Is something unclear in this tutorial? Feel free to comment or join our Slack to discuss it.
Also published on Medium.