A consultant's advice on AI training data almost cost me a client

Last spring, a data consultant told me to dump all my old customer emails into a custom AI model, saying it would give me better chatbot answers. I spent about 40 hours cleaning and uploading that data, and in June I ran a test. The responses were full of outdated pricing from 2021 and even some angry rants from disgruntled buyers. My client saw the demo and almost pulled the contract, that was a rough Tuesday. I went back to a smaller, curated dataset with just 500 recent interactions and the quality jumped way up. Have any of you found that less data actually works better for fine-tuning your models?

2 comments

2 Comments

annaw731mo ago

The 40 hours you sunk into that is exactly where the trap sits. I ran a similar test last year with about 3,000 support tickets from 2019 to 2022, and the model kept quoting a discontinued return policy like it was gospel. Trimming down to the 400 most recent, on topic exchanges fixed it almost overnight. The issue isn't that more data is bad, it's that old data carries noise, and noise scales worse than silence. Your 500 curated ones probably have a way better signal to junk ratio than that big pile ever did. Yeah, I'm with you on the bigger picture, but I'd argue the real fix is having someone actually read what's going in, not just dumping a decade of files into a blender. Quality beats quantity every single time, and your Tuesday proves it.

jake9861mo ago

...which is funny because I remember doing something similar way back with a different project, except I went the opposite direction. I had like a hundred emails, super clean, hand picked, and the thing still kept hallucinating a shipping deadline that only existed for like two weeks during the holiday rush that year. Drove me nuts. But what finally fixed it wasn't trimming more, it was actually just adding a bunch of really boring, repetitive tickets about password resets and account locks. That noise somehow grounded it, weirdest thing. So I get your point about curation, but sometimes the junk is what keeps the model from spinning off into fantasyland, you know? Anyways, your Tuesday test sounds like it worked, so maybe I just got lucky with my weird fix.