Diagnosing peak-hour latency in a Python Flask thumbnail and metadata service

[JerryC]

I'm a senior backend engineer at a streaming media company, and our team is currently grappling with a significant performance bottleneck in our content delivery pipeline. Specifically, our service responsible for generating personalized thumbnails and metadata for user homepages is experiencing severe latency spikes during peak evening hours, sometimes taking over two seconds to respond. This service is built on a Python and Flask stack, relies heavily on a Redis cache, and queries several microservices for user preferences and content metadata. We've already optimized database queries and increased cache sizes, but the problem persists, and we suspect it might be related to synchronous I/O calls blocking the event loop or inefficient serialization/deserialization of JSON data between services. We're considering a few different paths: rewriting the service in a more performant language like Go, implementing an asynchronous framework within Python, or re-architecting the entire pipeline to use a message queue for decoupling. Before we commit to a major rewrite, I wanted to see if others have tackled similar scaling issues in a media or personalization context. What profiling tools or techniques proved most valuable in identifying the true root cause of latency in a service with many external dependencies? And if you did undertake a language migration for performance, what were the biggest unforeseen challenges in terms of developer onboarding, inter-service communication, or maintaining feature parity during the transition?