Diagnosing a sporadic race condition in a Go distributed system under heavy load

[NathanR]

I'm a mid-level software engineer working on a complex distributed system written in Go, and I've been stuck for days debugging a sporadic race condition that only manifests under heavy load in production. My usual techniques of adding log statements and using the debugger aren't capturing the intermittent nature of the bug. For developers who have tackled similar elusive concurrency issues, what advanced debugging techniques or specialized tools have you found most effective for diagnosing race conditions and deadlocks in live, high-throughput environments?