DarbyWhen I joined Darby, our AI agent could only process eligibility determinations for a narrow range of medical devices, specifically CPAP machines and oxygen equipment, because policy files were cached in memory with extremely limited capacity. This architecture couldn't scale to handle the volume and variety of CMS data needed to serve our customer base, especially since some NCD/LCD datasets contain up to half a million records. We also lacked any archiving capability, meaning if CMS changed data formats or lost historical records, we had no proof of the evidence used in past eligibility decisions. The business needed a robust database solution that could store diverse policy data types (NCDs, LCDs, articles, HCPCS codes) while providing auditability and compliance.
I owned the end-to-end design and implementation of the CMS data replication and archiving system. After analyzing the range of file formats and data structures we needed to support, I made the strategic decision to build this as a standalone microservice rather than embedding it in an existing service, given the file sizes involved. I designed a flexible database schema to accommodate both CSV and Excel formats, coordinating closely with peer interns to align on field naming conventions for parallel workstreams. The core architectural innovation was introducing an enum-based dispatch pattern that mapped each file type to its corresponding parser, validator, and entity classes, allowing a single constant from the client to resolve the entire processing pipeline through a generic CRUD API. I leveraged Spring Boot, Spring Data JPA, and Hibernate to avoid writing explicit SQL and ensure long-term database portability. I also implemented validation layers to catch malformed data before database insertion and built dual-table structures (main and archive) for each data shape to support historical compliance requirements. The trickiest part was that the challenges were not obvious upfront. Early on, frequent OOM errors in production led me to discover that loading full files at once created up to four simultaneous in-memory copies of the same data: raw file bytes, parsed OpenCSV objects, domain model objects, and Hibernate entities. I had to extend OpenCSV to stream records incrementally, use Spring's TransactionTemplate to batch writes at a configurable per-file-type chunk size, and work around Apache POI's streaming API only supporting legacy XLS, not XLSX. Each fix surfaced the next problem.
After deployment, the system proved its extensibility immediately: we scaled from 2 to 12 product categories in just three months without requiring engineering intervention for each new addition. The AI agent's eligibility coverage improved by 6x, and critically, our operations team gained the ability to onboard new policy files independently, removing a significant engineering bottleneck and accelerating our go-to-market capability.
