Because "chatbot" today are highly specialized, Google is working on developing an almost universal conversational agent. The goal : ensure it is able to discuss everything and anything. In this year, Meena approximates human.
Researchers from the teamBrain, specialized in (deep learning), have just presented , a able to conduct conversations in a much more natural way than other chatbots. They call it a conversational model which has been trained from start to finish with 2,6 billion settings.
Meena is built on architecture Evolved Transformer, and consists of a single encoding block that allows him to understand the sentences of his interlocutor and the subject of the discussion, and of 13 decoding blocks to formulate its response. The researchers used 341 GB of text from conversations on for driving the chatbot. Their goal was to reduce the perplexity, ie the uncertainty in predicting the next word in the conversation.
A score close exchanges between humans
One problem in evaluating the performance of Meena was no reliable metric. The researchers therefore created a new measure called Sensibleness and Specificity Average (IN). To do this, they collected a hundred conversations for each agent, composed of 1.600 at 2.400 tours (or answers). Volunteers then had to rate each response, if it seemed reasonable and credible in the context, and if it was context specific and not a catch-all answer like "yes" or "me too". The SSA is the average of the scores on these two parameters.
This measure, Meena largely beat the others , with a score of 79 % against 56 % for the best. He approaches even human, rated 86 %. The researchers also discovered that there is an inverse correlation between perplexity and SSA, meaning that the perplexity could be used to automatically evaluate the performance of conversational agents, to accelerate their development.