A software leader that provides end-to-end compliance and audit management for modern companies sought to leverage ML and NLP applications to automate a highly manual process typically required for establishing compliance practices.
The company had reached a standstill on their AI-driven security product when the accuracy results plateaued around 70%. The internal team needed an NLP specialist to optimize the existing model, but did not have the in-house expertise.
Tribe was brought in to evaluate the company’s approach, optimize their existing model, and investigate alternative models to increase the speed at which their audit team can deliver to their customers (“audit efficiency”).
Tribe started by assembling a team with deep NLP experience, including:
The team started by evaluating the model design, including:
In addition, the team uncovered several existing issues in model workflow, which were increasing the effort in iterating and maintaining models. The Tribe team developed a two-phase, iterative approach to improving the accuracy of the core model and updating the ML infrastructure that would improve model workflow.
In ~2 months, using advanced natural language processing and working with a very limited data set, the Tribe team was able to increase the accuracy of the company’s core model by 15%. This included both selection of an alternative approach, developing an NLP Q&A model, and training the model.
The increase in accuracy greatly increased the throughput of the audit team, with team efficiency increasing by 55%.
In addition to the model improvements, Tribe AI also built the foundation of a ML infrastructure that would allow the team to rapidly draw in new data, retrain modes, and deploy at scale.