Drove Pivot to Vision-Language Models for Home-Inventory App

PMPujitha Sai Tulasi Muddhu·Founding Engineer at Stealth Startup

As Founding Engineer on a 3-person team at VaultlyAI, I owned the object-detection pipeline end to end for a home-inventory insurance app that lets users scan their belongings to auto-generate a structured, claim-ready inventory. Because accurate detection was the foundation of the entire product - a miscounted or misidentified item directly undermines an insurance claim - I was responsible not just for building the pipeline, but for validating its accuracy before anything reached production, drawing on my SDET background.

For the MVP, I started with a fine-tuned YOLO model. Since public datasets like COCO, Objects365, Open Images, and LVIS poorly covered our long-tail household item classes, I built training data manually: gathering 200–300 images per class, using SAM (Segment Anything) to generate masks, and annotating/converting everything into YOLO format to produce a 10K+ image training set - cutting manual annotation effort by roughly 60% versus labeling from scratch. As I scaled this approach, three critical issues emerged: it was too slow to expand across new classes, accuracy fell short of the bar, and worst of all it produced duplicate detections that corrupted the generated inventory, since insurance claims can't list the same item twice.

This led me to drive a pivot to Vision-Language Models. Rather than choosing a model on instinct, I built a benchmarking dashboard to evaluate several Gemini VLMs side by side on detection count, precision, latency, and cost, then integrated the winning model into the Python backend and connected it end-to-end with the React Native app. The result solved both problems at once: the chosen VLM outperformed the fine-tuned YOLO baseline on detection accuracy while running at essentially free cost at our volume - an unusual win on two axes that typically trade off against each other. Most critically, the duplicate-detection problem that had made the manual pipeline unusable was eliminated entirely.

This pipeline was validated internally when the CEO and CTO reviewed the benchmarking results and confirmed the duplicate-detection issue was resolved, greenlighting it for production. That decision made the feature launch-ready, and the app shipped on the Apple App Store and Google Play. The app has been live for about a month, and while broader user feedback is still early-stage (I transitioned off the startup shortly after launch due to visa timing), the internal validation from the founding team confirmed the pivot had solved the core reliability problem that stood between the product and launch.

Questions

EYAsked by Eric Yu

Amazing what VLMs can do these days! Super curious around what you mentioned with the duplicate detection issue you had with YOLO - what kinds of behavior were you seeing? I’ve used YOLO before for small experiments too and remember finding it annoying to have to tune its NMS settings so was curious if you ran into the same thing (and also if you maybe tried other model architectures that take a different approach to object detection)?

PMAnswered by Pujitha Sai Tulasi Muddhu

Good question - though it's a bit different from the NMS case you're picturing. The clearest example: for an IKEA Alex drawer unit, YOLO gave multiple entries - basically one per drawer, so a single piece of furniture got logged as several separate items. It was detecting the visually distinct sub-parts rather than recognizing the whole unit as one object. The VLM, on the other hand, returned it as a single entry - it understood the drawer unit as one item, which is what we actually wanted for an inventory. And on your other question - I haven't explored other detector architectures much. We went fairly directly from the fine-tuned YOLO path to VLMs, mostly because our specific home-inventory classes weren't well covered in the public datasets and the manual annotation wasn't scaling, so that's where we focused. Really enjoying these questions :)

JQAsked by James Quinlan

Were the custom data you assembled still used after deciding to use VLMs instead? Or were they only used for fine-tuning the YOLO models, and thus not necessary to achieve the end result? Also, were any gains from the VLMs attributable to internal reasoning of the models, or more likely just other factors, like the pre-training data sets simply being far larger? I have a layman's understanding of deep learning and vision models, so apologies if my questions don't even make sense! I'm just wicked interested in this space :)

PMAnswered by Pujitha Sai Tulasi Muddhu

Great questions - and they make total sense, no need to apologize! On the custom dataset: you're right that it was originally built to fine-tune the YOLO model. Once we moved to the VLMs, we used them off the shelf rather than training on my data, so that dataset didn't feed the final production model directly. But it wasn't wasted - it did two important things. It was what let me prove the manual/YOLO path couldn't scale or hit the accuracy bar in the first place, and it became my benchmark set for evaluating the VLMs, since I needed labeled ground-truth images to fairly compare how many objects each model detected and how precisely. So it shifted from being training data to being evaluation data. On where the gains came from: your instinct is right - I'd attribute most of it to the pre-training rather than any clever reasoning. My YOLO model only ever "saw" the few hundred images per class I could hand-label, so its world was narrow. The VLMs are pre-trained on massive, web-scale image-text data, so they already recognized a huge range of everyday household objects without me teaching them anything. That broad prior coverage is really what fixed both the accuracy and the duplicate-counting problems. I wouldn't overclaim any deep "reasoning" beyond that general visual understanding. Really appreciate you digging in on this - it's a fun space to nerd out on :)

Validators

Prudhvi Vuda

Direct Manager

Validated

Pujitha was a founding engineer at VaultlyAI and played a key role in shaping both the product and the engineering direction. She contributed across product research, technical design, full-stack development, and infrastructure, taking ownership from ideation through production. Her ability to combine product thinking with strong engineering execution made her an invaluable part of the team. She also led the evolution of our AI object recognition pipeline, driving the transition from a traditional YOLO-based approach to Vision-Language Models to improve accuracy and reliability. Pujitha consistently demonstrated ownership, initiative, and a bias for solving the right problems, and I would confidently recommend her for any engineering or AI-focused role.

Ezra Golani

CEO

Direct Manager

Validated

I had the pleasure of working with Pujitha, and she consistently exceeded expectations. She was exceptionally diligent, a quick learner, and an efficient worker who tackled every task with professionalism and attention to detail. She consistently delivered outstanding results, often going above and beyond what was expected. Any team would be fortunate to have her.

Questions

EYAsked by Eric Yu

PMAnswered by Pujitha Sai Tulasi Muddhu

JQAsked by James Quinlan

PMAnswered by Pujitha Sai Tulasi Muddhu