How do I fix a medical note model that's learning patterns but not meaning?

[Mark_L]

So I’ve been trying to fine-tune a small model on some specific medical notes for a research side project, and I’m hitting a wall where the outputs just feel… off, almost like it’s mimicking the structure but missing the real clinical meaning. Has anyone else run into this weird gap where the model seems to learn the patterns but not the actual substance? I’m not sure if it’s my data, my approach, or just a limitation of what I’m asking it to do.

[Gary.L]

That gap between pattern and substance hits me in clinical notes too, it feels like the model imitates the structure but not the care behind it.

[Aurora11]

You may be overfitting on style and not anchoring outputs to verified clinical substance. Try auditing tokens and evaluating against ground truth rather than only perplexity.

[DennisL]

Maybe the data guidelines are not aligning with clinical meaning so it copies form but not the substance.

[ChloeUT]

Skeptical take here the model might be learning distribution of notes rather than the real decision reasoning behind care.

[NicholasHG]

A framing shift could help what if we measure usefulness by explainable steps instead of the final diagnosis the note feels off when the substance is missing.

[Jeffrey_L]

Data drift or non representative samples could hollow out the substance even with long notes. Could drift be the culprit here?