Drove Pivot to Vision-Language Models for Home-Inventory App

PMPujitha Sai Tulasi Muddhu·Founding Engineer at Stealth Startup

As Founding Engineer on a 3-person team at VaultlyAI, I owned the object-detection pipeline end to end for a home-inventory insurance app that lets users scan their belongings to auto-generate a structured, claim-ready inventory. Because accurate detection was the foundation of the entire product - a miscounted or misidentified item directly undermines an insurance claim - I was responsible not just for building the pipeline, but for validating its accuracy before anything reached production, drawing on my SDET background.

For the MVP, I started with a fine-tuned YOLO model. Since public datasets like COCO, Objects365, Open Images, and LVIS poorly covered our long-tail household item classes, I built training data manually: gathering 200–300 images per class, using SAM (Segment Anything) to generate masks, and annotating/converting everything into YOLO format to produce a 10K+ image training set - cutting manual annotation effort by roughly 60% versus labeling from scratch. As I scaled this approach, three critical issues emerged: it was too slow to expand across new classes, accuracy fell short of the bar, and worst of all it produced duplicate detections that corrupted the generated inventory, since insurance claims can't list the same item twice.

This led me to drive a pivot to Vision-Language Models. Rather than choosing a model on instinct, I built a benchmarking dashboard to evaluate several Gemini VLMs side by side on detection count, precision, latency, and cost, then integrated the winning model into the Python backend and connected it end-to-end with the React Native app. The result solved both problems at once: the chosen VLM outperformed the fine-tuned YOLO baseline on detection accuracy while running at essentially free cost at our volume - an unusual win on two axes that typically trade off against each other. Most critically, the duplicate-detection problem that had made the manual pipeline unusable was eliminated entirely.

This pipeline was validated internally when the CEO and CTO reviewed the benchmarking results and confirmed the duplicate-detection issue was resolved, greenlighting it for production. That decision made the feature launch-ready, and the app shipped on the Apple App Store and Google Play. The app has been live for about a month, and while broader user feedback is still early-stage (I transitioned off the startup shortly after launch due to visa timing), the internal validation from the founding team confirmed the pivot had solved the core reliability problem that stood between the product and launch.

Questions

JQAsked by James Quinlan

Were the custom data you assembled still used after deciding to use VLMs instead? Or were they only used for fine-tuning the YOLO models, and thus not necessary to achieve the end result? Also, were any gains from the VLMs attributable to internal reasoning of the models, or more likely just other factors, like the pre-training data sets simply being far larger? I have a layman's understanding of deep learning and vision models, so apologies if my questions don't even make sense! I'm just wicked interested in this space :)

PMAnswered by Pujitha Sai Tulasi Muddhu

Great questions - and they make total sense, no need to apologize! On the custom dataset: you're right that it was originally built to fine-tune the YOLO model. Once we moved to the VLMs, we used them off the shelf rather than training on my data, so that dataset didn't feed the final production model directly. But it wasn't wasted - it did two important things. It was what let me prove the manual/YOLO path couldn't scale or hit the accuracy bar in the first place, and it became my benchmark set for evaluating the VLMs, since I needed labeled ground-truth images to fairly compare how many objects each model detected and how precisely. So it shifted from being training data to being evaluation data. On where the gains came from: your instinct is right - I'd attribute most of it to the pre-training rather than any clever reasoning. My YOLO model only ever "saw" the few hundred images per class I could hand-label, so its world was narrow. The VLMs are pre-trained on massive, web-scale image-text data, so they already recognized a huge range of everyday household objects without me teaching them anything. That broad prior coverage is really what fixed both the accuracy and the duplicate-counting problems. I wouldn't overclaim any deep "reasoning" beyond that general visual understanding. Really appreciate you digging in on this - it's a fun space to nerd out on :)

Validators

Prudhvi Vuda

Direct Manager

Validated

She was a great engineer. I am very happy to work with her again in the future.

Questions

JQAsked by James Quinlan

PMAnswered by Pujitha Sai Tulasi Muddhu