I'm working on some comparison reviews for different productivity apps, and I'm struggling with how to actually measure software performance in a meaningful way. Some apps feel fast on my machine but users with different hardware report issues. Other apps have great benchmark numbers but feel sluggish in actual use.
What performance metrics do you find most useful for evaluating software performance? Do you focus more on technical metrics like response times and memory usage, or do you prioritize user-reported experiences? And how do you account for the performance variance across different systems?
I focus on a mix of technical metrics and user experience metrics. For technical stuff, I look at response time percentiles (95th and 99th percentile are way more important than averages), error rates under load, and resource utilization patterns. But I also track things like time to interactive" for web apps or "perceived performance" scores from user testing.
The performance variance issue is real. What I do is test on three different system tiers: a budget setup, a mid-range setup, and a high-end setup. Then I report results for all three. It's more work, but it gives users a much better idea of what to expect on their specific hardware.
I've started paying more attention to consistency metrics rather than just peak performance. A system that delivers consistent 60fps is often better than one that averages 80fps but drops to 40fps during intense scenes. For productivity apps, I look at things like typing latency consistency and scroll smoothness.
One metric I've found really useful is time to value" - how long does it take for a user to accomplish their main task in the app? This combines technical performance with UX design. A fast app with confusing navigation might have worse time to value than a slightly slower app with better UX.