Rasa Developer Summit

The Rasa Developer Summit is an annual event that brings together developers and members of the Rasa community from around the world to share their experiences and learnings.
This year’s summit took place in San Francisco and included a diverse lineup of presenters and speakers from academia and the enterprise, startup, and open-source communities. Each of the contributors use Rasa in a different way—some use it to enable conversational AI through the framework; others use it to develop tooling to help understand and analyze the usage of conversational platforms. Finn AI attended the summit to learn about the use cases being explored with Rasa Core and natural-language understanding (NLU).

Leveraging NLP and Conversational AI for Search

A compelling use case and demo was presented by Adobe, addressing the difficulty of finding specific images amongst millions of potential stock photographs. The application using Rasa was a natural language processing(NLP)-enabled search to understand conversational descriptions of images. They utilize word embeddings to configure dozens of different filtering criteria including colors, positioning, and layout. The system sits on top of additional models for classifying over 16,000 objects like “man” or “accordion,” and shape-based positioning of these objects within an image. This allows for queries like, “Show me a man in a coffee shop drinking coffee.” This populates dozens of relevant images satisfying this description. The user can then filter by saying additional things like, “Show me more coffee,” or “move the man to the right of the image.”

 

Adobe-Coffee

 

This workflow works well for creatives as they search for images required for their publications. The use of voice-enabled NLP for discovery is more efficient than advanced search with scrolling for the “more” or “less” <object/property> that’s currently used by less than 10% of users. This approach to search is in beta phase of release and, to date, Adobe is focussed on getting the functionality right, rather than the underlying NLU.

How to Measure Success

At the summit, the CEOs and/or CPOs of several bot analytics platforms including Dashbot, Botanalytics, Botmock, and Pandorabots spoke on a panel about the challenges involved—and the tooling available—for measuring the performance and return on investment (ROI) of chatbots. The common message from all speakers was to keep measuring real-world deployments. They advised getting to production as soon as possible and iterating on the deployment from there. Try different designs, measure, iterate and take lots of calculated risks. Above all, don’t wait until your bot is perfect—because it will never be.

One of the biggest challenges identified by the speakers was ensuring their target audience and internal stakeholders know the bot exists. Organizations that build their own bots analyze usage patterns in Excel spreadsheets. However, as spreadsheets are shared across business functions, the templates for measurements change. This results in inconsistent analytics, inefficient processes, and/or a complete lack of process.

ROI is hard to measure at the best of times. This inconsistency compounds the issue.

The overall advice from the Rasa speakers was to collaborate cross-functionally to build a shared understanding of bot performance and integrate how this relates to the wider business needs.

Engaging with Stakeholders

An influential speaker from a large US health company—spoke about the three major challenges of creating a bot in-house: stakeholder buy-in, design, and ROI.

To address stakeholder buy-in, it is necessary to educate the company about AI and NLP. A lack of understanding coupled with the uncertainty of the approach can result in a lack of understanding about the benefits of an approach like this. Call center staff can be particularly unsure about the changes required to interact with and integrate AI systems. They have to invest in change management and training to see projects through to production.

Design is also an issue as AI presents a new way of designing content and experiences that doesn’t always lend itself to ideal user experiences. An entirely new way of thinking is required to design content and experiences within a conversational interface.

The final and biggest challenge is the measurement and ROI justification for AI innovation. There is no standard way to measure success or anticipate the ROI of many of these technologies. At Finn, we focus on task completion and conversation containment to help address these concerns. Can we enable the user to complete their goals within our platform? Can we integrate with backend APIs to execute? And can we escalate highly personalized needs to qualified customer representatives, when appropriate?

Content Management for Financial Data

N26, a rapidly-growing challenger bank presented their chatbot that has been built using the Rasa framework. Their chatbot’s raison d’etre is to reduce the need for expansion of their customer service team as the company grows. They support five languages across three distinct banks in the US, UK, and EU—all with subtle differences. They built a common set of core entities across each of these regions/languages.

To manage their content, they create stories and forms for fixed and smart flows, which are configured by content and data science teams. It is difficult to educate content writers on the underlying technology and what everything means in conversational AI, but they have devised a pseudocode way of defining these stories which teams have become accustomed to. This method empowers their teams to iterate on stories to enable end-users to achieve their goals.

Overall the Rasa Developer Summit 2019 was a wonderful learning experience and it was a pleasure to attend, learn from peers, share ideas, and reaffirm that Finn AI is on the cutting edge of many of the hot topics under discussion.

To learn more about the appropriate conversational AI models to consider for your bank, download our paper Basic, Build, or Buy: Conversational AI for Banks.

Kenneth Conroy
Dr. Kenneth Conroy is the Vice-President of Data Science at Finn AI. He leads the development of our proprietary NLP system and leverages machine learning to enable intelligent communication through turn-based, conversational flow. When he is not busy leading the team of data scientists, Ken enjoys speaking about the application of AI at events, and taking his new-born Boston Terrier on long walks on the beach.