So I’ve been trying to fine-tune a small model on some pretty niche data for a personal project, and I’m hitting a wall where it just starts generating plausible but completely made-up details. I’m not sure if it’s my data being too sparse, my approach to reinforcement learning from human feedback being off, or something else entirely. Has anyone else run into this kind of thing when the model seems to confidently invent things?
I hear you hallucinations are a stubborn knot. When the data is niche the model fills gaps with made up details that still sound convincing. Maybe the problem is not just the data but how the model treats uncertainty.
From a math angle the issue shows up as overconfident priors and distribution shift. The tiny model latches onto surface cues in the data and then hallucinations dominate to stay coherent.
I might be missing something but it feels like you are chasing a perfect ledger of facts while the model is asked to generalize. It reads like learning to tell a story with true facts sprinkled in but it leans to fiction when pressed and the hallucinations can pop up.
I'm skeptical the issue is not only data The model may be delivering what you asked for not what is true hallucinations show up there too Maybe the prompt is over praising fluency and not truth Could be a mistake in how you measure accuracy
Sometimes the issue is not the fine tuning so much as how you evaluate success Maybe reframe the goal toward stable responses from a reference rather than creating new facts to reduce hallucinations
On the practical side I found that adding a small vector store and a guard that cites sources helped reduce hallucinations. It slows generation but makes results feel more trustworthy
Short and reflective The tension between creativity and truth is never tidy. Hallucinations are part of the domain but you can loosen their grip with guard rails and better evaluation without killing imagination