System Design Interview Prep: How to Build a System to Handle Long Running Job
How do you design a YouTube or a Netflix? Have you ever wondered how to build a system that handle long running task like uploading and processing videos to platform like YouTube or Netflix. Unlike most of the transactions you see in normal websites, these video process takes minutes to hours to finish. What software components you will need? What design pattern will you apply?
The problem
In terms of workload, it can be categorized as long running and short running. For those load that consumes little compute power and get the result soon, it is easy to provide nice user experience since the users will not need to wait for long time. The application can simply request and wait synchronously for the response. The the UI can render properly based on the returned results. This type of system design is simple request-response with consideration of timeout scenario.
Want to read this story later? Save it in Journal.
However, for those systems that involved expensive workload, like video processing which is IO intensive, you will need different system design so that the long computing workload will run in the background. With this design, your users will not be blocked and continue to use your system’s other features until they get…
