Drove Pivot to Vision-Language Models for Home-Inventory App

As Founding Engineer on a 3-person team at VaultlyAI, I owned the object-detection pipeline end to end for a home-inventory insurance app that lets users scan their belongings to auto-generate a structured, claim-ready inventory. Because accurate detection was the foundation of the entire product - a miscounted or misidentified item directly undermines an insurance claim - I was responsible not just for building the pipeline, but for validating its accuracy before anything reached production, drawing on my SDET background. For the MVP, I started with a fine-tuned YOLO model. Since public datasets like COCO, Objects365, Open Images, and LVIS poorly covered our long-tail household item classes, I built training data manually: gathering 200–300 images per class, using SAM (Segment Anything) to generate masks, and annotating/converting everything into YOLO format to produce a 10K+ image training set - cutting manual annotation effort by roughly 60% versus labeling from scratch. As I scaled this approach, three critical issues emerged: it was too slow to expand across new classes, accuracy fell short of the bar, and worst of all it produced duplicate detections that corrupted the generated inventory, since insurance claims can't list the same item twice. This led me to drive a pivot to Vision-Language Models. Rather than choosing a model on instinct, I built a benchmarking dashboard to evaluate several Gemini VLMs side by side on detection count, precision, latency, and cost, then integrated the winning model into the Python backend and connected it end-to-end with the React Native app. The result solved both problems at once: the chosen VLM outperformed the fine-tuned YOLO baseline on detection accuracy while running at essentially free cost at our volume - an unusual win on two axes that typically trade off against each other. Most critically, the duplicate-detection problem that had made the manual pipeline unusable was eliminated entirely. This pipeline was validated internally when the CEO and CTO reviewed the benchmarking results and confirmed the duplicate-detection issue was resolved, greenlighting it for production. That decision made the feature launch-ready, and the app shipped on the Apple App Store and Google Play. The app has been live for about a month, and while broader user feedback is still early-stage (I transitioned off the startup shortly after launch due to visa timing), the internal validation from the founding team confirmed the pivot had solved the core reliability problem that stood between the product and launch.