Persistent memory leak in a long-running Java service processing a Kafka queue

[Ella.H]

I'm debugging a persistent memory leak in a long-running Java service that processes a high volume of messages from a Kafka queue. The heap usage slowly climbs over several days until it triggers an alert, but standard profiling with tools like VisualVM hasn't clearly pinpointed the root cause, as the heap dumps just show a general increase in object counts without an obvious culprit. For developers who've tackled similar elusive leaks in JVM applications, what advanced profiling techniques or tools would you recommend to trace the source of these gradual allocations, and are there common patterns in modern frameworks or libraries that might be causing these kinds of hidden retention issues?

[JacobL]

You're fighting a classic elusive leak. Start with low-overhead production profiling using Java Flight Recorder (JFR) and Mission Control (or VisualVM if you must). Record around steady-state throughput to see allocation rates and retention. Then load the heap dump into Eclipse MAT; use the Dominator Tree and Leak Suspects to spot growth hotspots. For Kafka-heavy apps, pay attention to in-flight records, history stores, and consumer/producer metadata. Common culprits include caches and data structures that grow without bounds, event listeners or buffers not released, thread-locals holding onto context, and libraries that pool objects (Netty ByteBuf, DB client pools). If a particular class keeps growing, trace the retention path with MAT’s object graph view to GC roots and traces. A practical workflow is: enable JFR, let the system run under load for a while, generate a heap dump during a peak, and compare to a clean-state dump. Then purge/limit caches or add evictions based on findings.