At Hotstar, I was leading a backend team responsible for handling content playback services during major live events like cricket matches, where traffic spikes were extremely high and unpredictable. We were facing recurring latency issues during peak loads, especially when millions of users were trying to start streams simultaneously. The existing architecture was heavily dependent on synchronous service calls, which was becoming a bottleneck. At that point, I had to make a strategic decision on whether to continue optimizing the current system or shift towards a more scalable, event-driven architecture, which involved significant changes and some risk given the upcoming tournament timelines.
After analyzing production metrics, failure patterns, and traffic projections, I decided to move forward with introducing an event-driven approach using asynchronous processing for non-critical workflows, such as user session tracking and recommendation triggers, while keeping critical playback flows optimized and lightweight. I discussed the approach with stakeholders, clearly outlining the trade-offs, including short-term development effort versus long-term scalability and reliability benefits. I then broke down the transition into phases so we could incrementally adopt the new architecture without disrupting ongoing releases. The team was aligned around clear ownership, and we also added fallback mechanisms to ensure system stability during rollout.
As a result, during the next major live event, we were able to handle peak traffic much more efficiently, reducing system latency by around 40% and significantly lowering failure rates. More importantly, this decision laid the foundation for a more scalable platform that could support future growth without major rework. This experience taught me that strategic decisions often require balancing immediate delivery pressures with long-term system health, and it’s important to take calculated risks backed by data and phased execution.
After analyzing production metrics, failure patterns, and traffic projections, I decided to move forward with introducing an event-driven approach using asynchronous processing for non-critical workflows, such as user session tracking and recommendation triggers, while keeping critical playback flows optimized and lightweight. I discussed the approach with stakeholders, clearly outlining the trade-offs, including short-term development effort versus long-term scalability and reliability benefits. I then broke down the transition into phases so we could incrementally adopt the new architecture without disrupting ongoing releases. The team was aligned around clear ownership, and we also added fallback mechanisms to ensure system stability during rollout.
As a result, during the next major live event, we were able to handle peak traffic much more efficiently, reducing system latency by around 40% and significantly lowering failure rates. More importantly, this decision laid the foundation for a more scalable platform that could support future growth without major rework. This experience taught me that strategic decisions often require balancing immediate delivery pressures with long-term system health, and it’s important to take calculated risks backed by data and phased execution.