DarbyDarby's AI agent uses Anthropic's Large Language Model to determine whether Home Medical Equipment (HME) orders are eligible for insurance coverage by analyzing incoming PDF orders. The engineering team was shipping changes to the eligibility service without any systematic way to measure accuracy improvements. Manual spot-checking in staging was slow, inconsistent, and created a bottleneck. The team couldn't confidently deploy to production without repeatable benchmarks. An initial test harness existed but ran only locally on one engineer's machine and was tightly coupled to one microservice, limiting the entire team's ability to iterate and assess changes.
I built a decoupled Angular evaluation dashboard backed by a Spring Boot microservice that enables the entire team to benchmark AI performance against our live staging environment. The system allows users to upload test PDF orders and golden dataset answer keys (curated by our in-house DME expert) to S3, then executes full eligibility analyses by sending orders to the AI agent one at a time. As each order is processed, the backend compares the AI's JSON output, including eligibility decisions and supporting evidence, against expected results, streaming real-time progress updates to the frontend. I implemented defensive deserialization logic with custom ObjectMapper configuration to handle the inherent inconsistency of LLM-generated JSON (missing fields, type mismatches), and designed a multi-threaded architecture with a single publisher thread and multiple worker threads to process 65+ PDFs without blocking the UI or introducing race conditions. The system generates a confusion matrix across key eligibility dimensions, produces an LLM-generated summary of performance, and stores downloadable test results in S3 with user notifications.
The dashboard democratized evaluation. Any team member, not just engineers, could now independently run tests and assess changes to Darby's AI agent in staging, meaningfully reducing bottlenecks during iteration cycles. Initial benchmark results revealed specific areas for improvement: 80% accuracy identifying the correct Local Coverage Determinations (LCDs), but only 13% for primary items, 7% for secondary items, 40% for ICD-10 codes, and 57% overall eligibility accuracy across 65 orders spanning PAP, O2, and CGM devices. These metrics provided the team with actionable insights and a repeatable framework to confidently iterate on LLM prompts and models, measure impact, and prevent regressions before production deployment.
