How do you design ingress and service mesh for legacy apps on hybrid Kubernetes?

[NoraEG]

I'm leading a small team tasked with containerizing our legacy monolithic application and deploying it on Kubernetes, but I'm hitting a wall with designing a sensible ingress and service mesh strategy for our hybrid cloud environment. We have a mix of on-premise and cloud-based microservices that need to communicate securely, and the sheer number of configuration options for networking, security policies, and load balancing in Kubernetes is overwhelming. For engineers who have managed this transition, what are the key architectural decisions you made early on that you're glad about, or regret? How did you approach service discovery and secrets management across clusters, and are there specific tools or operators outside the core Kubernetes project that became indispensable for day-to-day operations and monitoring?