System Design Fundamentals: How Scalable Applications Work Behind the Scenes

Introduction

When we build a small application, the architecture may be very simple.

A frontend application sends a request to a backend server, the backend communicates with a database, and the response is returned to the user.

A simple architecture may look like this:

User
  ↓
Frontend
  ↓
Backend API
  ↓
Database

This can work perfectly for hundreds or even thousands of users.

But what happens when your application starts receiving hundreds of thousands or millions of requests?

A single server may no longer be enough.

The application may become slow, the database may become overloaded, and one server failure could make the entire application unavailable.

This is where system design becomes important.

System design is about deciding how different components of an application should communicate with each other so that the application remains:

• Scalable

• Reliable

• Available

• Secure

• Maintainable

• Fast

In this article, we will understand a common scalable architecture step by step.

The architecture we will discuss looks roughly like this:

Users
  ↓
Web Application
  ↓
Load Balancer
  ↓
API Gateway
  ↓
Backend Services
  ├── Cache
  ├── Message Queue
  └── Database

Let's understand what each component does.

1. Users

Everything starts with the user.

A user may interact with your application using:

• Web browser

• Mobile application

• Desktop application

• Another API

• Third-party integration

For example, imagine we are building an e-commerce application.

Users may:

• Browse products

• Search products

• Add products to a cart

• Place orders

• Make payments

• View previous orders

Every action eventually creates a request to our system.

For example:

GET /api/products

or:

POST /api/orders

The architecture must be able to handle these requests efficiently.

2. Web Application or Client

The client is normally the first application layer the user interacts with.

For a modern web application, this could be built using technologies such as:

React
Next.js
Vue
Angular

For mobile applications, it could be:

React Native
Flutter
Native Android
Native iOS

The client is responsible for displaying the interface and communicating with backend APIs.

For example:

const response = await fetch("/api/products");

const products = await response.json();

The frontend should usually not communicate directly with the database.

Instead, it communicates with backend services through APIs.

This separation improves security, maintainability, and scalability.

3. Load Balancer

Suppose our application originally has one backend server.

User
 ↓
Server

This works until traffic becomes too high.

Imagine the server can comfortably handle:

1,000 requests per second

But suddenly your application receives:

5,000 requests per second

Instead of replacing the server with one extremely powerful machine, we can run multiple servers.

             ┌── Server 1
Users → Load Balancer ── Server 2
             └── Server 3

The load balancer distributes incoming traffic between multiple application servers.

For example:

Request 1 → Server A
Request 2 → Server B
Request 3 → Server C
Request 4 → Server A

This is an example of horizontal scaling.

Instead of continuously increasing the power of one server, we add more servers.

Why do we need a load balancer?

A load balancer can help with:

• Traffic distribution

• High availability

• Horizontal scaling

• Failover

• Health checks

If Server B becomes unavailable, the load balancer can stop sending requests to it.

Server A → Healthy
Server B → Unhealthy
Server C → Healthy

Traffic can then be routed only to the healthy servers.

Popular load balancing solutions include:

• AWS Application Load Balancer

• Nginx

• HAProxy

• Cloudflare

• Google Cloud Load Balancing

4. API Gateway

After the request reaches our backend infrastructure, it may go through an API Gateway.

An API Gateway acts as the main entry point for backend APIs.

Instead of allowing clients to directly communicate with many backend services, requests first go through the gateway.

Client
  ↓
API Gateway
  ├── User Service
  ├── Product Service
  └── Order Service

The API Gateway can handle common responsibilities such as:

• Authentication

• Authorization

• Rate limiting

• Request routing

• Logging

• API versioning

• Request validation

For example:

/api/users   → User Service
/api/products → Product Service
/api/orders   → Order Service

This keeps many common concerns outside individual services.

5. Rate Limiting

One important responsibility of an API Gateway can be rate limiting.

Imagine a user or automated bot sends:

10,000 requests per second

Without protection, these requests could overload your system.

A rate limiter can apply a rule such as:

Maximum 100 requests per minute per user.

If the limit is exceeded, the API may return:

HTTP 429 Too Many Requests

Rate limiting helps protect applications from:

• API abuse

• Bots

• Accidental request loops

• Resource exhaustion

• Some denial-of-service scenarios

6. Backend Services

As an application grows, putting everything inside one huge backend application can become difficult to maintain.

One possible architecture is to separate business functionality into multiple services.

For example:

User Service
Product Service
Order Service
Payment Service
Notification Service

Each service has a specific responsibility.

User Service

Responsible for things such as:

User registration
Login
Profiles
Account management

Product Service

Responsible for:

Product information
Categories
Inventory
Search

Order Service

Responsible for:

Creating orders
Order status
Order history
Order processing

This approach is commonly associated with microservices architecture.

However, microservices are not always necessary.

For small or medium applications, a well-structured monolithic application can often be simpler and more practical.

A good system design should solve the actual problem rather than automatically choosing the most complicated architecture.

7. Database

Most applications need persistent storage.

This is where databases come into the architecture.

Common relational databases include:

PostgreSQL
MySQL
SQL Server

Popular NoSQL databases include:

MongoDB
DynamoDB
Cassandra

For example, an order might be stored as:

{
  "orderId": 1024,
  "userId": 55,
  "total": 3500,
  "status": "pending"
}

The database provides persistent storage, which means the data remains available even after the application server restarts.

8. Why the Database Can Become a Bottleneck

Imagine that every page request performs several database queries.

If one million users visit the application, the database may receive millions of repeated queries.

For example:

SELECT * FROM products WHERE id = 10;

If thousands of users request the same popular product, repeatedly querying the database is inefficient.

This is one of the problems caching can solve.

9. Cache

A cache stores frequently accessed data temporarily so that the application does not need to query the main database every time.

A common cache technology is:

Redis

Without caching:

User
 ↓
Backend
 ↓
Database

With caching:

User
 ↓
Backend
 ↓
Cache
 ↓
Database

The application first checks the cache.

If the data exists in the cache, it returns it immediately.

This is called a cache hit.

Request
 ↓
Cache
 ↓
Data found
 ↓
Return response

If the data does not exist, we get a cache miss.

Request
 ↓
Cache
 ↓
Not found
 ↓
Database
 ↓
Store result in cache
 ↓
Return response

Pseudo-code may look like this:

const cachedProduct = await redis.get(`product:${productId}`);

if (cachedProduct) {
  return JSON.parse(cachedProduct);
}

const product = await database.getProduct(productId);

await redis.set(`product:${productId}`, JSON.stringify(product));

return product;

Caching can dramatically reduce database load and improve response times.

However, caching also introduces challenges, especially cache invalidation.

If data changes in the database, the cached copy must eventually be updated or removed.

10. Message Queue

Not every operation needs to happen immediately.

Imagine a user places an order.

The system may need to:

1. Save the order.

2. Send a confirmation email.

3. Send an SMS.

4. Update analytics.

5. Notify the warehouse.

6. Generate an invoice.

If all these operations happen during the same HTTP request, the user may wait several seconds.

Instead, some operations can be processed asynchronously.

User places order
       ↓
Order Service
       ↓
Message Queue
       ↓
Background Worker
       ↓
Send Email

The Order Service can quickly return:

Order placed successfully.

Meanwhile, background services process other tasks.

Popular message queue technologies include:

RabbitMQ
Apache Kafka
AWS SQS
Google Pub/Sub

Message queues help systems become more:

• Responsive

• Scalable

• Fault tolerant

• Loosely coupled

11. Synchronous vs Asynchronous Processing

Understanding this difference is important in system design.

Synchronous Processing

The user waits until the operation finishes.

User
 ↓
Server
 ↓
Process
 ↓
Response

Example:

Login request

The user needs an immediate response.

Asynchronous Processing

The system accepts the task and processes it later.

User
 ↓
Server
 ↓
Queue
 ↓
Worker

Examples include:

• Sending emails

• Generating reports

• Processing images

• Sending notifications

• Updating analytics

• Video processing

Choosing which operations should be synchronous and which should be asynchronous can significantly improve application performance.

12. Example: E-commerce Request Flow

Let's put all the components together.

Imagine a user opens a product page.

Step 1

The user visits:

https://example.com/products/123

Step 2

The web application requests product information.

GET /api/products/123

Step 3

The request reaches the load balancer.

The load balancer forwards it to an available backend server.

Step 4

The request reaches the API Gateway.

The gateway may check:

Authentication
Rate limits
Request validity

Step 5

The request is routed to the Product Service.

API Gateway
     ↓
Product Service

Step 6

The Product Service checks Redis.

Product Service
     ↓
Redis Cache

If the product exists in the cache, it is returned immediately.

Otherwise:

Redis Cache
     ↓
Cache Miss
     ↓
Database

The Product Service reads the product from the database and stores a copy in Redis.

Step 7

The response is returned to the user.

The complete flow may look like:

User
 ↓
Web App
 ↓
Load Balancer
 ↓
API Gateway
 ↓
Product Service
 ↓
Cache
 ↓
Database

13. What Happens When the User Places an Order?

Now imagine the same user places an order.

User
 ↓
Web App
 ↓
Load Balancer
 ↓
API Gateway
 ↓
Order Service

The Order Service writes the order to the database.

Order Service
 ↓
Database

Then it publishes an event.

OrderCreated

to a message queue.

Order Service
 ↓
Message Queue

Different services can consume that event.

                    ┌→ Email Service
OrderCreated → Queue ├→ Inventory Service
                    ├→ Analytics Service
                    └→ Notification Service

This means the Order Service does not need to wait for every operation to finish.

This is an example of an event-driven architecture.

14. Scalability

Scalability means the system can handle increased traffic without becoming unusable.

There are two common types of scaling.

Vertical Scaling

Increase the power of one server.

For example:

4 GB RAM → 16 GB RAM
2 CPU → 8 CPU

This is simple but has limits.

Horizontal Scaling

Add more servers.

Server 1
Server 2
Server 3
Server 4

A load balancer distributes traffic between them.

Modern large-scale applications commonly use horizontal scaling because capacity can be increased by adding more instances.

15. Availability

Availability describes how accessible your application is.

If your system depends on one application server:

User
 ↓
Server

and that server fails, your application becomes unavailable.

With multiple servers:

        ┌→ Server A
User → Load Balancer
        └→ Server B

one server can fail while the other continues serving users.

High availability often involves:

• Multiple servers

• Multiple availability zones

• Health checks

• Database replication

• Failover strategies

• Backups

16. Reliability

A scalable system is not automatically reliable.

Reliability means the system continues behaving correctly even when failures occur.

Failures are normal in distributed systems.

Servers can crash.

Networks can fail.

Database connections can timeout.

External APIs can become unavailable.

Therefore, system designers need to think about mechanisms such as:

Retries
Timeouts
Circuit breakers
Replication
Backups
Message queues
Monitoring

A good system should expect failures instead of assuming every component will always work.

17. Monitoring and Logging

Once a system contains multiple services, understanding what is happening becomes more difficult.

Monitoring becomes essential.

You may track:

CPU usage
Memory usage
Response time
Error rate
Requests per second
Database connections
Cache hit ratio
Queue size

Logs can help developers understand errors.

For example:

2026-09-27 14:31:22
ERROR
OrderService
Payment request timed out
orderId=1024

Popular monitoring and observability tools include:

AWS CloudWatch
Prometheus
Grafana
Datadog
New Relic
Elastic Stack

18. Avoid Overengineering

One of the most important lessons in system design is:

Do not build for millions of users when you currently have only a few hundred unless there is a real reason to do so.

You probably do not need:

20 microservices
Kafka
Kubernetes
5 databases
Multiple cache clusters

for a simple application.

You may initially need only:

Frontend
Backend
Database

As traffic and business requirements grow, you can introduce:

Load Balancer
Caching
Background Jobs
Message Queues
Multiple Services
Database Replication

System design is not about using as many technologies as possible.

It is about choosing the simplest architecture that reliably solves the problem while leaving reasonable room for growth.

19. A Practical Evolution of an Application

Many applications evolve gradually.

Stage 1 : Simple Application

Client
 ↓
Server
 ↓
Database

Stage 2 — Add Load Balancing

          ┌→ Server
Client → Load Balancer
          └→ Server
               ↓
            Database

Stage 3 : Add Caching

Client
 ↓
Load Balancer
 ↓
Application
 ├→ Cache
 └→ Database

Stage 4 : Separate Services

Client
 ↓
API Gateway
 ├→ User Service
 ├→ Product Service
 └→ Order Service

Stage 5 : Add Asynchronous Processing

Services
 ↓
Message Queue
 ↓
Background Workers

Architecture should evolve with the needs of the application.

20. Questions to Ask During System Design

Before choosing technologies, ask questions such as:

1. How many users will the application have?

2. How many requests per second should it support?

3. What kind of data will we store?

4. How much data will be generated?

5. Does the application require real-time updates?

6. What happens if a server fails?

7. Can some operations run asynchronously?

8. Which data is accessed frequently?

9. Where can caching help?

10. What security requirements exist?

11. How important is consistency?

12. How much downtime is acceptable?

These questions are often more important than immediately choosing technologies.

Conclusion

System design can look complicated when you first see architecture diagrams containing load balancers, caches, queues, gateways, services, and databases.

However, each component usually exists to solve a specific problem.

The core architecture discussed in this article can be summarized as:

Users
  ↓
Web Application
  ↓
Load Balancer
  ↓
API Gateway
  ↓
Backend Services
  ├── Cache
  ├── Message Queue
  └── Database

The load balancer helps distribute traffic.

The API Gateway manages and routes API requests.

The backend services handle business logic.

The cache improves performance and reduces database load.

The message queue enables asynchronous processing.

The database provides persistent storage.

Most importantly, good system design is not about creating the most complicated architecture.

It is about understanding the problem, identifying potential bottlenecks, and introducing the right component when it is actually needed.

Once these fundamentals are clear, advanced topics such as database replication, sharding, CDNs, distributed caching, event-driven architecture, fault tolerance, and microservices become much easier to understand.