Acronym search for enterprises

Microsoft • 2018 • User Experience Design

Design of an AI-based acronym search for enterprises

An employee of a medium to large-scale enterprise spends a significant portion of their time looking for documents, asking for files from a colleague, looking for subject matter experts, information on company portals, etc.

Microsoft Search enables employees to search their work-related information like documents, meetings, people, locations, intranet and cloud portals, meanings and acronyms, etc. through any search surfaces in the Office suite.

A new employee or an employee onboarding onto a different team/project needs to get up to speed with the project and vocabulary of the team. It’s not uncommon to find an employee spending a significant amount of time looking for documents, people, or the meaning of internal technical terms.

The Microsoft search system consists of 3 key stakeholders

  1. Company admin
    Can add new acronyms and moderate existing results.
  2. Employee
    Employees of the enterprise search and view acronyms. They can also give feedback on search results and suggest additional acronyms.
  3. Development team
    Microsoft is the service provider that builds and moderates the AI results based on inputs from other stakeholders.

Designing search answers powered by AI

Design principles for search answers

  1. Limit distractions
  2. Be credible and trustworthy

Designing for different confidence levels

Unlike traditional software, systems with AI/ML components are probabilistic in nature and tend to change over time. Their design needs to account for this probabilistic and changing nature.

A confidence level is a statistical measurement that indicates how confident the AI system is about a particular outcome. Design of the enterprise acronym answer adapted to the confidence of the AI system.

Answers that were added by admins (human counterparts) were given the highest confidence followed by the AI-mined answers. Results with very low confidence were not shown at all.

For the purpose of understanding, I’ve categorized the AI confidence levels into 4 types:

  1. Very high confidence
  2. High confidence
  3. Low confidence
  4. Very low confidence

Very high confidence answer

High confidence answer

Low confidence answer

Very low confidence answer

Feedback mechanisms

User feedback is the communication channel between your users, your product, and your team. Leveraging feedback is a powerful and scalable way to improve your technology, provide personalized content, and enhance the user experience.

The AI system is guided by a reward function that helps the model determine whether the results are correct or incorrect (or a percentage value of how correct it was). The feedback mechanism was designed keeping in mind the goal of improving the AI model.

Explicit feedback on an answer through a thumbs-up / down mechanism
General feedback
Enabling users to provide negative feedback
Enabling users to suggest missing acronyms

Privacy and trust

The enterprise search answers appear alongside web search results. This makes it imperative to build trust with users being shown sometimes what could be sensitive data. For the system, trust and privacy were established in two ways:

  1. Reiterating that their data is secure
  2. Building explainability into the design

Reiterating that their data is secure

Telling users that their search results are private.

The copy indicates that the search results are private

Building explainability into the design

Ensuring that users are aware of the source of a mined acronym to help them make a conscious choice. Acronyms that were published by the company admin were given prominence and relevant attribution.

Attribution under the acronym definition indicates how the answers were mined along with its source


Adding missing acronyms

If an acronym is not automatically extracted, an administrator can add new acronyms from the company’s O365 admin portal.

Onboarding new users

The team made a conscious choice of only introducing the feature to new users when they typed a related query. For e.g. Searching for a word definition might be the right place to introduce this feature.

The first run experience of acronym search occurs when users look for a related query

Due to the confidential nature of unreleased projects and products, much has been left out.