Goal

System design interviews are key interviews for most of the candidates. In fact, for every candidate, these interviews are the ones that differentiate a good fit from a great fit. Unfortunately, the topic to study is so vast that you can feel lost. And unlike coding, there are no right or wrong answers (besides the most obvious ones). There are only tradeoffs.

The good news though is that you do not need to know everything about everything. You need to know enough to make a tradeoff decision. And the impact of tradeoff increases as you keep leading larger services

Structure

While I have used experience years as a unit, it depends on the job responsibilities you are applying for

  • Fundamental - Required for all.
  • Senior engineers - Anyone with 5+ years of experience would be expected to know these
  • Staff software engineers (Team/tech leads) - Anyone with 8+ years of experience would be expected to know these

Have you checked out our discord group? We host live sessions for coding and system design every week on the server, have study kits that are based on multiple different patterns and topics. No sign up required, just join the discord


Fundamentals

Let’s say you have a simple website serving traffic to your customers. In its simplest form, the architecture would look like below
imbitious_load_balancer_system_design_interview-3
Now, let’s assume a scenario. The machine that your web server is hosted on just died (disk failure, data center failure, etc.). So how do you make service more reliable? Simply by adding another machine.

Both of the above machines will be on different IP addresses. So how does the client know which one to connect to? Let’s say we add a simple piece of code where the client is randomly connecting to one of the machines. This will distribute the load as well. This is known as load balancing
imbitious_client_load_balancer_system_design_interview-2

This can get very complicated once you start adding more machines in different aspects like regions, datacenters and geo. What happens if one of them goes down? What happens if the new one that just came up has a different IP address than before?

Fortunately, there is a simpler solution which is using a load balancer.

What is a load balancer?

Load balancer efficiently distributes incoming network traffic across a group of backend servers. It makes sure that no single server bears too much load/demand. This in turn makes the application more responsive with increased availability. The load balancer makes sure that requests are routed to responsive servers.
imbitious_load_balancer_system_design_interview_final-3

💡
Whenever you have any service design questions, add a Load balancer. 1 application server is not going to cut it.

Have you checked out our discord group? We host live sessions for coding and system design every week on the server, have study kits that are based on multiple different patterns and topics. No sign up required, just join the discord


Senior engineers

Let’s dive a little bit more into how LBs work

How does load balancing work?

To distribute the load, LBs have some sort of routing algorithm. Few of the algorithms are listed below (details)

  • Least connection method
  • Least response time method
  • Round robin method
  • IP Hash

For the purpose of the interview, “round-robin” is the easiest one to explain. Unless you are designing a load balancer, chances are less the interviewer would jump deeper

Load balancers also operate at different levels. If you are not aware of the 7 layers of OSI models, I recommend giving it a glance.

Not diving into too many details, here are the two types

  • Application load balancer - Operates at the seventh layer or application layer in the OSI model. This is mostly your HTTP and HTTPS traffic. The routing rules can be set on HTTP payload (like content type, cookie, headers). For example, we may have a website that services high-quality and low-quality pictures. We can route high-quality jpeg requests to one set of servers and low quality to another set of servers.
  • Network load balancer - Operates at 4th layer. You can read packet headers but not HTTP headers. It checks if "server" is available (health ping).

(More deep dive here)

💡
When asked, stick with round-robin as your preferred algorithm. And if designing services that depend on TCP/IP only or HTTP/HTTPS use an application load balancer.

Lead Engineers

For lead engineers, you have to think of global scale and multi-microservice architectures. This is when your design will combine other services like CDNs to ensure high availability and reliability. In addition, load balancers can be used for internal services which face high traffic.

imbitious_load_balancer_system_design_interview_multi_lb-3

Below are some paper napkin numbers to give you an idea of throughput and cost.

Note that these numbers are to give you an idea. It is very hard to get an estimate without having the entire design in perspective. Since load balancers are just forwarding requests, most of the time the limiting factor is underlying network bandwidth.

  • 1 Load Balancer can handle around 1 million requests with 8 core and 16 thread instances and around 1150 concurrent connections. (Source)
    imbitious_ha-proxy-load

  • Cost of LB: Hosting the above LB would cost the same as hosting an m5.2xlarge (AWS ec2 machine type). This costs 0.384 per hour. So the total cost for the day would be ~$10/day or $300/month

  • You would need at least two LBs to support active/passive or active/active state. So $20 per region. For each region (for example west coast, east coast) there are more than one zones each representing different data centers. You want machines in different zones in case one data center goes down.

  • If available globally, you would want to scale to multiple different regions in the above fashion.

While hosting your LB is definitely an option, another cheaper and more convenient option is hosting a managed Load Balancer like Application Load Balancer on AWS or Azure load balancer. While the calculation is a little bit more complicated, a basic load balancer with 5 rules on azure is around $23.25.

imbitious_load_balancer_azure_cost-1

💡
Rarely does anyone face a question of “Design a load balancer” unless you are interviewing for that particular role. However, in today’s modern apps, having a load balancer is a must once you start scaling.

Have you checked out our discord group? We host live sessions for coding and system design every week on the server, have study kits that are based on multiple different patterns and topics. No sign up required, just join the discord