How Tribe AI Built a Model on GCP That Increased Security Questionnaire Auditor Efficiency by 55%

A software leader that provides end-to-end compliance and audit management for modern companies sought to leverage ML and NLP applications to automate a highly manual process typically required for establishing compliance practices. 

Challenge

The company had reached a standstill on their AI-driven security product when the accuracy results plateaued around 70%. The internal team needed an NLP specialist to optimize the existing model, but did not have the in-house expertise. 
‍

Tribe was brought in to evaluate the company’s approach, optimize their existing model, and investigate alternative models to increase the speed at which their audit team can deliver to their customers (“audit efficiency”). 
‍

Team

Tribe started by assembling a team with deep NLP experience, including:

• Alex, a former Salesforce ML engineer with deep expertise in NLP and building production-ready NLP and recommender systems 
• Haggai, an ML Ops engineer who led the prediction team at Cruise and built tools to automate ML ops at Airbnb
• Saurabh, an experienced head of product who focused on technical product integrations at Zapier and built the Gsuite Marketplace and Gsuite add-ons product at Google
  ‍

Project

The team started by evaluating the model design, including:

• Model approach – the current model was based on STS, but the team investigated whether Q&A would perform better
• Model choice – evaluating several models against benchmarks like inference time
• Training data – with a focus on how to reduce training time. 
  ‍

In addition, the team uncovered several existing issues in model workflow, which were increasing the effort in iterating and maintaining models. The Tribe team developed a two-phase, iterative approach to improving the accuracy of the core model and updating the ML infrastructure that would improve model workflow. 
‍

Solution

In ~2 months, using advanced natural language processing and working with a very limited data set, the Tribe team was able to increase the accuracy of the company’s core model by 15%.  This included both selection of an alternative approach, developing an NLP Q&A model, and training the model. 
‍

The increase in accuracy greatly increased the throughput of the audit team, with team efficiency increasing by 55%.
‍

In addition to the model improvements, Tribe AI also built the foundation of a ML infrastructure that would allow the team to rapidly draw in new data, retrain modes, and deploy at scale.