Introduction

I built a human-in-the-loop (HITL) evaluation framework for WEX Inc. that revealed why traditional metrics were hiding real usability failures in their Account Manager workflows. The framework reduced operator errors by 35% and increased AI feature adoption from 34% to 61%.

Role: Senior UX Researcher | Timeline: 9 months

Problem Statement

WEX's eManager platform showed 85%+ task completion and low error rates in automated metrics, but Account Managers reported significant friction, workarounds, and wasted time. Automated logging was missing what mattered most: workflow efficiency and operator confidence in AI-assisted decisions.

Approach

• Conducted audit of 12+ WEX products to identify gaps between metrics and user experience
• Built human-in-the-loop evaluation framework combining automated data with qualitative observation
• Conducted discovery interviews and task-based usability testing with Account Managers
• Performed SQL analysis of behavioral data (override rates, alert fatigue patterns)
• Triangulated findings: What users SAY + What users DO + What metrics SHOW

Solution

Developed and validated a human-in-the-loop evaluation methodology that:

• Captured workflow inefficiencies that metrics couldn't see (workarounds, confidence erosion, time waste)
• Prioritized improvements by severity × frequency based on user feedback and behavioral data
• Provided business case for AI fraud detection investments with quantified impact
• Established frameworks for ongoing evaluation of AI features across WEX's product portfolio

Impact