From PoC to Production: Scaling Bright’s Training Simulations with Tribe AI & AWS Bedrock

Rania El-Ghezzaoui

To learn about the initial PoC, read this case study.

Introduction

Bright is a learning services platform that empowers companies like Nike and Aetna to train their teams through immersive, scenario-based simulations. These simulations are essential for preparing customer support agents to handle complex situations, such as managing order discrepancies or navigating insurance approvals. For instance, Nike's customer support agents use Bright's platform to engage in simulated conversations with virtual customers, addressing issues like incorrect shoe sizes or missing items in bulk orders. 

The platform allows agents to practice accessing internal systems like Salesforce or order management systems, identify root causes, and follow proper procedures. Moreover, it trains agents to handle a variety of customer personalities, from friendly inquirers to irate customers, ensuring they respond empathetically and in alignment with the brand's voice and customer-facing ethos. 

The Problem

The existing system at Bright required Learning and Development Managers at companies like Nike and Aetna to manually create detailed, if-else decision trees for various customer personalities and scenarios. This process was used to simulate a wide range of customer interactions, from angry customers inquiring about order updates to calm individuals seeking insurance pre-authorization. However, this manual approach was extremely labor-intensive and often fell short of covering the full spectrum of possible interactions, limiting the effectiveness and adaptability of the training programs. 

For instance, a manager might need to create scenarios for: 

An angry customer asking about an order update 

A happy customer inquiring about an order status 

An irritated customer trying to complete an insurance pre-authorization 

A confused customer lacking proper insurance or doctor information for pre-authorization 

The complexity of creating these simulations with extensive if-then-else branching was not only time consuming but also challenging. Managers had to provide verbatim responses for each potential interaction, which proved to be an inefficient method of preparing customer service representatives for the diversity of real-world scenarios they might encounter.

Challenge 

The core challenge lay in the inefficiency and limitations of manually constructing these conversation simulations. Learning and Development Managers needed to define every potential interaction, which involved crafting specific responses for various customer moods and scenarios. This approach required managers to anticipate and script out numerous possible conversation paths, considering factors such as: 

Customer emotions (e.g., happy, angry, confused, calm) 

Types of inquiries (e.g., order updates, insurance pre-authorizations) 

Customer demographics (e.g., age, geographical location) 

Communication styles and preferences 

This method was not only time-consuming but also failed to encompass the true diversity of real-world customer interactions. It was nearly impossible to create an exhaustive set of conditions to tackle the various responses that a learner might encounter in actual customer service situations. 

Solution 

To address these challenges, Bright developed a next-generation platform that revolutionizes the creation of customer personalities and interaction scenarios. The new system automates the process, making it significantly more efficient and comprehensive. Here's how the solution works: 

Natural Language Profile Creation: Instead of manual scripting, Learning and Development Managers can now describe customer profiles in natural language. They can specify traits such as: Age 

  • Personality style 
  • Geographical location 
  • Communication preferences 
  • Specific scenario details (e.g., seeking a prescription refill) 

Dynamic Interaction Simulation: These profiles are used to generate sophisticated bots that simulate these customer profiles. For example, it can create "Eva," an elderly customer seeking a prescription refill from Aetna. 


Comprehensive Scenario Coverage: The system allows Learning and Development Managers to focus on key scenario fork points in the conversation tree without needing to cover every single detail. Generative AI and Large Language Models (LLMs) augment and fill in the gaps of the conversation tree, providing comprehensive coverage that managers might not have time to create

manually. This approach ensures a more thorough and diverse range of training scenarios while significantly reducing the time and effort required from managers. 


Integrated Evaluation Engine: Tribe also developed a rating and evaluation engine as part of the solution. This engine assesses the trainee's performance across various metrics, like:

  • Empathy levels 
  • Adherence to company policies 
  • Following the optimal path along the conversation tree 
  • Ability to handle varying personalities for the same scenario 

This comprehensive solution enables companies like Nike and Aetna to create more realistic, diverse, and effective training simulations for their customer-facing teams, significantly improving their ability to handle a wide range of real-world customer interactions. 

AI / LLM Challenge addressed in the solution 

While leveraging LLMs to fill in gaps and handle unspecified paths in the decision tree greatly enhanced the system's flexibility, it also introduced new challenges that Tribe AI had to address: 

1. Consistency and Comparability: The use of LLMs introduced variability in bot responses, which posed a challenge for fair evaluation of trainees. It was crucial for companies like Nike or Aetna to be able to compare the performance of multiple employees who underwent training, rating who did well and who didn't. This required a certain form of "apples to apples" comparison between the paths taken, the responses provided by bots like Eva, and how human trainees responded to those responses. Tribe AI developed mechanisms to maintain a delicate balance between variability and consistency in bot responses, allowing for meaningful comparisons between trainees' performances while still providing diverse, realistic interactions.

2. Customization and Control: After numerous training sessions with a bot like Eva, Learning and Development Managers often gathered feedback about the bot's quirks and behaviors. This necessitated a way to fine-tune bot personalities without breaking critical conversation points or paths. Tribe AI engineered a system that allows for this level of steerability and control over the LLM, ensuring that critical conversation points remain intact while allowing for adjustments to enhance the training experience. This was achieved without making it too burdensome for learning managers to specify everything in great detail.

3. Integration of Real-world Data: Companies like Aetna and Nike often wanted to incorporate their own past data from real-world customer interactions to create more authentic bot personalities. To address this need, Tribe AI designed the system to incorporate actual customer interaction data, allowing for the creation of bot personalities based on real-world customer profiles and behaviors. This feature significantly enhanced the realism and relevance of the training scenarios.

4. Latency and Real-time Interaction: A major challenge with a purely GenAI system is latency, especially in scenarios where conversations between trainees and bots could last for hours. In such cases, the GenAI system needs to keep track of the conversation's progress, covered and uncovered topics, and generate responses in real-time. To address this complexity, Tribe AI implemented advanced indexing and smart retrieval systems. These innovations enable the GenAI system to quickly access relevant information and generate timely responses, even supporting voice-to-voice conversations between trainees and bots like Eva. 

Implementation 

The implementation leveraged AWS services and Claude 3.5 Sonnet from Anthropic via AWS Bedrock. The primary component of the solution was the Bot Interaction Engine. 

Bot Interaction System 

This system facilitates real-time online processing. It employs Python and FastAPI, along with AWS ECS Fargate, to respond to trainee interactions in real-time. The system's complexity necessitated the development of custom indexing and AI-powered retrieval engines (backed by AWS RDS and pgvector) by Tribe AI engineers to support the use case. The diagram below illustrates how the bot interaction system functions within the FastAPI service, which is run on AWS ECS Fargate and sits inside a VPC. 

Architecture 

Architecture Details 

The Bright Learning Platform utilizes a modern cloud-native architecture built on AWS, consisting of four main components: 

Frontend Layer 

  • This is a AWS Amplify hosted React web application. This is part of their existing platform. No changes / development was done here as part of the Tribe AI engagement.

API Gateway Layer 

  • Leveraged Bright's existing AWS API Gateway infrastructure, adding new routes specifically for the conversation bot functionality 
  • The existing gateway continues to handle request routing, composition, and protocol translation 
  • Utilizes the platform's established security, monitoring, and rate limiting capabilities 

AI Processing Engine 

  • Core AI processing logic implemented as a Python FastAPI application 
  • Deployed on Bright's existing AWS ECS Fargate cluster within their VPC, utilizing their established auto-scaling capabilities 
  • Leverages Bright's existing Amazon RDS infrastructure for storing AI chat session state and management data alongside their core application data 
  • Establishes connection to Anthropic's Claude 3.5 Sonnet model via AWS Bedrock for real-time conversation processing and response generation

The entire system is contained within AWS Cloud, with the application components running in a dedicated VPC for security. This architecture ensures scalability, reliability, and secure handling of training simulations while maintaining low latency for real-time interactions between trainees and AI powered conversation bots.

Related Stories

Applied AI

Navigating the Generative AI Landscape: Opportunities and Challenges for Investors

Applied AI

How AI Enhances Real-Time Credit Risk Assessment in Lending

Applied AI

AI Diagnostics in Healthcare: How Artificial Intelligence Streamlines Patient Care

Applied AI

8 Ways AI for Healthcare Is Revolutionizing the Industry

Applied AI

Scalability in AI Projects: Strategies, Types & Challenges

Applied AI

A Gentle Introduction to Structured Generation with Anthropic API

Applied AI

No labels are all you need – how to build NLP models using little to no annotated data

Applied AI

AI in Private Equity: A Guide to Smarter Investing

Applied AI

The Hitchhiker’s Guide to Generative AI for Proteins

Applied AI

How to Build a Data-Driven Culture With AI in 6 Steps

Applied AI

5 machine learning engineers predict the future of self-driving

Applied AI

AI and Predictive Analytics in Investment

Applied AI

How AI is Cutting Healthcare Costs and Streamlining Operations

Applied AI

AI Consulting in Insurance Industry: Key Considerations for 2024 and Beyond

Applied AI

AI in Finance: Common Challenges and How to Solve Them

Applied AI

Using data to drive private equity with Drew Conway

Applied AI

Key Takeaways from Tribe AI’s LLM Hackathon

Applied AI

The Secret to Successful Enterprise RAG Solutions

Applied AI

How to Improve Sales Efficiency Using AI Solutions

Applied AI

AI Security: How to Use AI to Ensure Data Privacy in Finance Sector

Applied AI

How AI for Fraud Detection in Finance Bolsters Trust in Fintech Products

Applied AI

How to Reduce Costs and Maximize Efficiency With AI in Insurance

Applied AI

AI in Portfolio Management

Applied AI

Tribe welcomes data science legend Drew Conway as first advisor 🎉

Applied AI

Everything you need to know about generative AI

Applied AI

AI and Blockchain Integration: How They Work Together

Applied AI

AI Implementation in Healthcare: How to Keep Data Secure and Stay Compliant

Applied AI

How AI Improves Knowledge Process Automation

Applied AI

A Deep Dive Into Machine Learning Consulting: Case Studies and FAQs

Applied AI

Tribe's First Fundraise

Applied AI

A primer on generative models for music production

Applied AI

Top 8 Generative AI Trends Businesses Should Embrace

Applied AI

Making the moonshot real – what we can learn from a CTO using ML to transform drug discovery

Applied AI

Best Practices for Integrating AI in Healthcare Without Disrupting Workflows

Applied AI

How to Measure ROI on AI Investments

Applied AI

AI and Predictive Analytics in the Cryptocurrency Market

Applied AI

How to build a highly effective data science program

Applied AI

AI for Cybersecurity: How Online Safety is Enhanced by Artificial Intelligence

Applied AI

Self-Hosting Llama 3.1 405B (FP8): Bringing Superintelligence In-House

Applied AI

Common Challenges of Applying AI in Insurance and Solutions

Applied AI

7 Key Benefits of AI in Software Development

Applied AI

AI Implementation: Ultimate Guide for Any Industry

Applied AI

How to Seamlessly Integrate AI in Existing Finance Systems

Applied AI

Top 5 AI Solutions for the Construction Industry

Applied AI

An Actionable Guide to Conversational AI for Customer Service

Applied AI

Announcing Tribe AI’s new CRO!

Applied AI

What the OpenAI Drama Taught us About Enterprise AI

Applied AI

Advanced AI Analytics: Strategies, Types and Best Practices

Applied AI

A Guide to AI in Insurance: Use Cases, Examples, and Statistics

Applied AI

Current State of Enterprise AI Adoption, A Tale of Two Cities

Applied AI

What our community of 200+ ML engineers and data scientist is reading now

Applied AI

AI in Construction in 2024 and Beyond: Use Cases and Benefits

Applied AI

How the U.S. can accelerate AI adoption: Tribe AI + U.S. Department of State

Applied AI

Generative AI: Powering Business Growth across 7 Key Operations

Applied AI

10 ways to succeed at ML according to the data superstars

Applied AI

How AI Enhances Hospital Resource Management and Reduces Operational Costs

Applied AI

How to Measure and Present ROI from AI Initiatives

Applied AI

Key Generative AI Use Cases From 10 Industries

Applied AI

AI Consulting in Healthcare: The Complete Guide

Applied AI

How data science drives value for private equity from deal sourcing to post-investment data assets

Applied AI

10 Common Mistakes to Avoid When Building AI Apps

Applied AI

7 Effective Ways to Simplify AI Adoption in Your Company

Applied AI

7 Prerequisites for AI Tranformation in Healthcare Industry

Applied AI

How to Optimize Supply Chains with AI

Applied AI

AI-Driven Digital Transformation

Applied AI

Why do businesses fail at machine learning?

Applied AI

AI in Banking and Finance: Is It Worth The Risk? (TL;DR: Yes.)

Applied AI

10 Expert Tips to Improve Patient Care with AI

Applied AI

Top 10 Common Challenges in Developing AI Solutions (and How to Overcome Them)

Applied AI

How to Reduce Costs and Maximize Efficiency With AI in Finance

Applied AI

Welcome to Tribe House New York 👋

Applied AI

How to Evaluate Generative AI Opportunities – A Framework for VCs

Applied AI

How 3 Companies Automated Manual Processes Using NLP

Applied AI

Leveraging Data Science – From Fintech to TradFi with Christine Hurtubise

Applied AI

AI in Customer Relationship Management

Applied AI

8 Prerequisites for AI Transformation in Insurance Industry

Applied AI

AI in Construction: How to Optimize Project Management and Reducing Costs

Applied AI

10 AI Techniques to Improve Developer Productivity

Applied AI

How to Enhance Data Privacy with AI

Applied AI

7 Strategies to Improve Customer Care with AI

Applied AI

AI Consulting in Finance: Benefits, Types, and What to Consider

Applied AI

Segmenting Anything with Segment Anything and FiftyOne

Applied AI

3 things we learned building Tribe and why project-based work will change AI

Applied AI

Write Smarter, Not Harder: AI-Powered Prompts for Every Product Manager

Applied AI

Top 9 Criteria for Evaluating AI Talent

Applied AI

How to Use Generative AI to Boost Your Sales

Applied AI

Thoughts from AWS re:Invent

Applied AI

Understanding MLOps: Key Components, Benefits, and Risks

Applied AI

Machine Learning in Healthcare: 7 real-world use cases

Get started with Tribe

Companies

Find the right AI experts for you

Talent

Join the top AI talent network

Close
Partnerships
Rania El-Ghezzaoui