Designing a Scalable Notification System: Email, SMS, Push & In-App Notifications

Introduction

Almost every modern application needs notifications.

Examples include:

• Order confirmation

• Password reset

• Payment confirmation

• New message alerts

• Delivery updates

• Security alerts

• Marketing campaigns

• System announcements

Notifications can be delivered through different channels:

Email
SMS
Push Notification
In-App Notification

At first, sending a notification looks simple.

For example:

await sendEmail(user.email, "Your order was confirmed");

But this becomes difficult when your application grows.

Imagine that your application has:

1,000,000 users

and you need to send:

500,000 notifications

within a short period.

If your API waits for each notification provider, several problems appear:

• Slow API responses

• Provider timeouts

• Failed notifications

• Duplicate messages

• Rate limits

• Server overload

• Difficult retry logic

This is why notification systems are usually designed asynchronously.

A scalable architecture may look like this:

Application
     ↓
Notification API
     ↓
PostgreSQL
     ↓
Redis Queue
     ↓
Notification Workers
  ┌────┼────┬────┐
  ↓    ↓    ↓    ↓
Email SMS  Push In-App

In this article, we will build a simplified version of this architecture and test every part locally.

What We Will Build

By the end of this article, we will have a working notification system that supports:

1. Creating notifications

2. Email notifications

3. SMS notifications

4. Push notifications

5. In-app notifications

6. Background processing

7. Redis-based queues

8. Retry handling

9. Failed notification tracking

10. Notification status

11. User notification preferences

12. Worker-based processing

The system will work like this:

Client
  ↓
POST /notifications
  ↓
Notification Service
  ↓
Save notification to PostgreSQL
  ↓
Add job to Redis Queue
  ↓
Worker processes job
  ↓
Provider sends notification
  ↓
Update database status

Architecture

Our high-level architecture will be:

                        ┌─────────────────┐
                        │     Client      │
                        └────────┬────────┘
                                 │
                                 ↓
                        ┌─────────────────┐
                        │ Notification API│
                        └────────┬────────┘
                                 │
                    ┌────────────┴────────────┐
                    ↓                         ↓
             ┌─────────────┐           ┌─────────────┐
             │ PostgreSQL  │           │ Redis Queue │
             └─────────────┘           └──────┬──────┘
                                              │
                                              ↓
                                      ┌──────────────┐
                                      │    Worker    │
                                      └──────┬───────┘
                                             │
                   ┌─────────────────────────┼────────────────────────┐
                   ↓                         ↓                        ↓
              Email Provider            SMS Provider            Push Provider
                   │                         │                        │
                   ↓                         ↓                        ↓
               MailHog                  Mock Console             Mock Console

Step 1: Create the Project

Create a new folder:

mkdir notification-system
cd notification-system

Initialize Node.js:

npm init -y

Install dependencies:

npm install express pg bullmq ioredis nodemailer dotenv

Install development dependencies:

npm install -D nodemon

Our main dependencies are:

express     → REST API
pg          → PostgreSQL
bullmq      → Background jobs
ioredis     → Redis connection
nodemailer  → Email sending
dotenv      → Environment variables

Step 2: Create the Project Structure

Create the following folders:

notification-system/
│
├── src/
│   ├── config/
│   │   ├── db.js
│   │   └── redis.js
│   │
│   ├── controllers/
│   │   └── notification.controller.js
│   │
│   ├── queues/
│   │   └── notification.queue.js
│   │
│   ├── providers/
│   │   ├── email.provider.js
│   │   ├── sms.provider.js
│   │   └── push.provider.js
│   │
│   ├── routes/
│   │   └── notification.routes.js
│   │
│   ├── workers/
│   │   └── notification.worker.js
│   │
│   └── server.js
│
├── docker-compose.yml
├── .env
└── package.json

Step 3: Start PostgreSQL, Redis, and MailHog

Instead of installing everything manually, we will use Docker.

Create:

docker-compose.yml

Add:

version: "3.9"

services:
  postgres:
    image: postgres:16
    container_name: notification-postgres
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: notifications
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data

  redis:
    image: redis:7
    container_name: notification-redis
    ports:
      - "6379:6379"

  mailhog:
    image: mailhog/mailhog
    container_name: notification-mailhog
    ports:
      - "1025:1025"
      - "8025:8025"

volumes:
  postgres_data:

Start everything:

docker compose up -d

Verify:

docker ps

You should see:

notification-postgres
notification-redis
notification-mailhog

Step 4: Test Redis

Run:

docker exec -it notification-redis redis-cli

Then:

PING

Expected response:

PONG

Exit:

exit

Redis is working.

Step 5: Test PostgreSQL

Connect:

docker exec -it notification-postgres psql -U postgres -d notifications

Run:

SELECT NOW();

You should receive the current database time.

Exit:

\q

Step 6: Test MailHog

Open:

http://localhost:8025

You should see the MailHog inbox.

MailHog acts like a fake SMTP server.

Instead of sending real emails, our application sends them to MailHog.

This means we can test email notifications without:

• Gmail

• AWS SES

• SendGrid

• Mailgun

Step 7: Environment Variables

Create:

.env

Add:

PORT=3000

DATABASE_URL=postgresql://postgres:postgres@localhost:5432/notifications

REDIS_HOST=127.0.0.1
REDIS_PORT=6379

SMTP_HOST=localhost
SMTP_PORT=1025

Step 8: Create the Database Table

Connect again:

docker exec -it notification-postgres psql -U postgres -d notifications

Create the notifications table:

CREATE TABLE notifications (
    id SERIAL PRIMARY KEY,
    user_id INTEGER NOT NULL,
    channel VARCHAR(20) NOT NULL,
    recipient VARCHAR(255),
    title VARCHAR(255),
    message TEXT NOT NULL,
    status VARCHAR(20) DEFAULT 'pending',
    attempts INTEGER DEFAULT 0,
    error_message TEXT,
    created_at TIMESTAMP DEFAULT NOW(),
    sent_at TIMESTAMP
);

Check the table:

\d notifications

You should see all columns.

Step 9: Create User Preferences

Notification preferences are important.

A user may want:

Email = Enabled
SMS = Disabled
Push = Enabled

Create another table:

CREATE TABLE notification_preferences (
    user_id INTEGER PRIMARY KEY,
    email_enabled BOOLEAN DEFAULT TRUE,
    sms_enabled BOOLEAN DEFAULT TRUE,
    push_enabled BOOLEAN DEFAULT TRUE,
    in_app_enabled BOOLEAN DEFAULT TRUE
);

Insert a test user:

INSERT INTO notification_preferences (
    user_id,
    email_enabled,
    sms_enabled,
    push_enabled,
    in_app_enabled
)
VALUES (
    1,
    TRUE,
    TRUE,
    TRUE,
    TRUE
);

Verify:

SELECT * FROM notification_preferences;

Step 10: Database Connection

Create:

src/config/db.js

Add:

const { Pool } = require("pg");

const pool = new Pool({
  connectionString: process.env.DATABASE_URL,
});

module.exports = pool;

Step 11: Redis Connection

Create:

src/config/redis.js

Add:

const IORedis = require("ioredis");

const redisConnection = new IORedis({
  host: process.env.REDIS_HOST,
  port: Number(process.env.REDIS_PORT),
  maxRetriesPerRequest: null,
});

module.exports = redisConnection;

Step 12: Create the Notification Queue

Create:

src/queues/notification.queue.js

Add:

const { Queue } = require("bullmq");
const redisConnection = require("../config/redis");

const notificationQueue = new Queue("notifications", {
  connection: redisConnection,
});

module.exports = notificationQueue;

Now notifications can be placed into Redis.

Step 13: Build the Email Provider

Create:

src/providers/email.provider.js

Add:

const nodemailer = require("nodemailer");

const transporter = nodemailer.createTransport({
  host: process.env.SMTP_HOST,
  port: Number(process.env.SMTP_PORT),
  secure: false,
});

async function sendEmail({ recipient, title, message }) {
  const result = await transporter.sendMail({
    from: "notifications@example.com",
    to: recipient,
    subject: title,
    text: message,
  });

  return result;
}

module.exports = {
  sendEmail,
};

Step 14: Test Email Provider Separately

Create temporarily:

test-email.js

Add:

require("dotenv").config();

const { sendEmail } = require("./src/providers/email.provider");

sendEmail({
  recipient: "developer@example.com",
  title: "Test Email",
  message: "Notification system is working.",
})
  .then(() => {
    console.log("Email sent");
    process.exit();
  })
  .catch(console.error);

Run:

node test-email.js

Open:

http://localhost:8025

You should see the email.

Now email sending is confirmed before integrating it into the full system.

Step 15: Create the SMS Provider

For local development, we do not need Twilio.

Instead, create a mock SMS provider.

Create:

src/providers/sms.provider.js

Add:

async function sendSMS({ recipient, message }) {
  console.log("----- MOCK SMS -----");
  console.log(`To: ${recipient}`);
  console.log(`Message: ${message}`);
  console.log("--------------------");

  return {
    success: true,
  };
}

module.exports = {
  sendSMS,
};

When we later move to production, this provider can be replaced by:

Twilio
AWS SNS
Vonage
MessageBird

Step 16: Create the Push Provider

Create:

src/providers/push.provider.js

Add:

async function sendPush({ recipient, title, message }) {
  console.log("----- MOCK PUSH -----");
  console.log(`Device: ${recipient}`);
  console.log(`Title: ${title}`);
  console.log(`Message: ${message}`);
  console.log("---------------------");

  return {
    success: true,
  };
}

module.exports = {
  sendPush,
};

In production, this could later connect to:

Firebase Cloud Messaging
Expo Notifications
Apple Push Notification Service

Step 17: Create the Notification API Controller

Create:

src/controllers/notification.controller.js

Add:

const db = require("../config/db");
const notificationQueue = require("../queues/notification.queue");

async function createNotification(req, res) {
  try {
    const {
      userId,
      channel,
      recipient,
      title,
      message,
    } = req.body;

    if (!userId || !channel || !message) {
      return res.status(400).json({
        message: "userId, channel and message are required",
      });
    }

    const allowedChannels = [
      "email",
      "sms",
      "push",
      "in_app",
    ];

    if (!allowedChannels.includes(channel)) {
      return res.status(400).json({
        message: "Invalid notification channel",
      });
    }

    const result = await db.query(
      `
      INSERT INTO notifications
      (user_id, channel, recipient, title, message)
      VALUES ($1, $2, $3, $4, $5)
      RETURNING *
      `,
      [
        userId,
        channel,
        recipient,
        title,
        message,
      ]
    );

    const notification = result.rows[0];

    await notificationQueue.add(
      "send-notification",
      {
        notificationId: notification.id,
      },
      {
        attempts: 3,
        backoff: {
          type: "exponential",
          delay: 2000,
        },
      }
    );

    return res.status(201).json({
      message: "Notification queued",
      notification,
    });
  } catch (error) {
    console.error(error);

    return res.status(500).json({
      message: "Internal server error",
    });
  }
}

module.exports = {
  createNotification,
};

Notice:

attempts: 3

This means BullMQ retries failed jobs.

The backoff:

delay: 2000

starts with a two-second delay.

Step 18: Add Notification Routes

Create:

src/routes/notification.routes.js

Add:

const express = require("express");

const {
  createNotification,
} = require("../controllers/notification.controller");

const router = express.Router();

router.post("/", createNotification);

module.exports = router;

Step 19: Create the Server

Create:

src/server.js

Add:

require("dotenv").config();

const express = require("express");
const notificationRoutes = require("./routes/notification.routes");

const app = express();

app.use(express.json());

app.get("/health", (req, res) => {
  res.json({
    status: "ok",
  });
});

app.use("/notifications", notificationRoutes);

const PORT = process.env.PORT || 3000;

app.listen(PORT, () => {
  console.log(`API running on port ${PORT}`);
});

Update:

package.json

Add:

{
  "scripts": {
    "dev": "nodemon src/server.js",
    "start": "node src/server.js",
    "worker": "node src/workers/notification.worker.js"
  }
}

Step 20: Start the API

Run:

npm run dev

Test:

curl http://localhost:3000/health

Expected:

{
  "status": "ok"
}

Step 21: Create the Notification Worker

Now we create the most important component.

Create:

src/workers/notification.worker.js

Add:

require("dotenv").config();

const { Worker } = require("bullmq");

const db = require("../config/db");
const redisConnection = require("../config/redis");

const { sendEmail } = require("../providers/email.provider");
const { sendSMS } = require("../providers/sms.provider");
const { sendPush } = require("../providers/push.provider");

const worker = new Worker(
  "notifications",

  async (job) => {
    const { notificationId } = job.data;

    const result = await db.query(
      `
      SELECT *
      FROM notifications
      WHERE id = $1
      `,
      [notificationId]
    );

    const notification = result.rows[0];

    if (!notification) {
      throw new Error("Notification not found");
    }

    const preferencesResult = await db.query(
      `
      SELECT *
      FROM notification_preferences
      WHERE user_id = $1
      `,
      [notification.user_id]
    );

    const preferences = preferencesResult.rows[0];

    if (preferences) {
      const preferenceMap = {
        email: preferences.email_enabled,
        sms: preferences.sms_enabled,
        push: preferences.push_enabled,
        in_app: preferences.in_app_enabled,
      };

      if (!preferenceMap[notification.channel]) {
        await db.query(
          `
          UPDATE notifications
          SET status = 'skipped'
          WHERE id = $1
          `,
          [notification.id]
        );

        return;
      }
    }

    await db.query(
      `
      UPDATE notifications
      SET
        status = 'processing',
        attempts = attempts + 1
      WHERE id = $1
      `,
      [notification.id]
    );

    if (notification.channel === "email") {
      await sendEmail(notification);
    }

    if (notification.channel === "sms") {
      await sendSMS(notification);
    }

    if (notification.channel === "push") {
      await sendPush(notification);
    }

    if (notification.channel === "in_app") {
      console.log(
        `In-app notification created for user ${notification.user_id}`
      );
    }

    await db.query(
      `
      UPDATE notifications
      SET
        status = 'sent',
        sent_at = NOW(),
        error_message = NULL
      WHERE id = $1
      `,
      [notification.id]
    );
  },

  {
    connection: redisConnection,
  }
);

worker.on("completed", (job) => {
  console.log(`Job ${job.id} completed`);
});

worker.on("failed", async (job, error) => {
  console.error(
    `Job ${job?.id} failed:`,
    error.message
  );

  if (!job) {
    return;
  }

  await db.query(
    `
    UPDATE notifications
    SET
      status = 'failed',
      error_message = $1
    WHERE id = $2
    `,
    [
      error.message,
      job.data.notificationId,
    ]
  );
});

console.log("Notification worker started");

Step 22: Start the Worker

Open another terminal.

Run:

npm run worker

You now have:

Terminal 1
API

Terminal 2
Worker

Docker is already running:

PostgreSQL
Redis
MailHog

Step 23: Send Your First Email Notification

Run:

curl -X POST http://localhost:3000/notifications \
-H "Content-Type: application/json" \
-d '{
  "userId": 1,
  "channel": "email",
  "recipient": "developer@example.com",
  "title": "Order Confirmed",
  "message": "Your order #1001 has been confirmed."
}'

Expected API response:

{
  "message": "Notification queued"
}

Notice something important.

The API does not wait for the email provider.

It only:

Creates notification
↓
Queues notification
↓
Returns response

The worker handles the actual sending.

Step 24: Verify the Email

Open:

http://localhost:8025

You should see:

Subject:
Order Confirmed

Open it.

You should see:

Your order #1001 has been confirmed.

The full notification pipeline is working.

Step 25: Verify Database Status

Run:

docker exec -it notification-postgres psql -U postgres -d notifications

Then:

SELECT
    id,
    channel,
    status,
    attempts,
    sent_at
FROM notifications;

Expected:

id | channel | status | attempts
--------------------------------
1  | email   | sent   | 1

Step 26: Test SMS

Run:

curl -X POST http://localhost:3000/notifications \
-H "Content-Type: application/json" \
-d '{
  "userId": 1,
  "channel": "sms",
  "recipient": "+8801700000000",
  "title": "OTP",
  "message": "Your OTP is 123456."
}'

Look at the worker terminal.

You should see:

----- MOCK SMS -----
To: +8801700000000
Message: Your OTP is 123456.
--------------------

Check database:

SELECT id, channel, status
FROM notifications;

Expected:

sms | sent

Step 27: Test Push Notification

Run:

curl -X POST http://localhost:3000/notifications \
-H "Content-Type: application/json" \
-d '{
  "userId": 1,
  "channel": "push",
  "recipient": "device-token-abc123",
  "title": "Delivery Update",
  "message": "Your order is out for delivery."
}'

Worker output:

----- MOCK PUSH -----
Device: device-token-abc123
Title: Delivery Update
Message: Your order is out for delivery.
---------------------

Step 28: Test In-App Notification

Run:

curl -X POST http://localhost:3000/notifications \
-H "Content-Type: application/json" \
-d '{
  "userId": 1,
  "channel": "in_app",
  "title": "Welcome",
  "message": "Welcome to the application."
}'

Worker output:

In-app notification created for user 1

Step 29: Improve In-App Notifications

For a real system, in-app notifications should remain visible until the user reads them.

Add:

ALTER TABLE notifications
ADD COLUMN is_read BOOLEAN DEFAULT FALSE;

Now create an endpoint:

GET /notifications/user/:userId

This allows the frontend to show a notification center.

Example query:

async function getUserNotifications(req, res) {
  const { userId } = req.params;

  const result = await db.query(
    `
    SELECT *
    FROM notifications
    WHERE
      user_id = $1
      AND channel = 'in_app'
    ORDER BY created_at DESC
    `,
    [userId]
  );

  res.json(result.rows);
}

Step 30: Add the Route

Update routes:

router.get(
  "/user/:userId",
  getUserNotifications
);

Test:

curl http://localhost:3000/notifications/user/1

You should receive:

[
  {
    "id": 4,
    "channel": "in_app",
    "message": "Welcome to the application.",
    "is_read": false
  }
]

Step 31: Mark Notification as Read

Add:

PATCH /notifications/:id/read

Controller:

async function markAsRead(req, res) {
  const { id } = req.params;

  const result = await db.query(
    `
    UPDATE notifications
    SET is_read = TRUE
    WHERE id = $1
    RETURNING *
    `,
    [id]
  );

  res.json(result.rows[0]);
}

Test:

curl -X PATCH \
http://localhost:3000/notifications/4/read

Now query:

SELECT id, is_read
FROM notifications
WHERE id = 4;

Expected:

true

Step 32: Test Notification Preferences

Now disable SMS.

Run:

UPDATE notification_preferences
SET sms_enabled = FALSE
WHERE user_id = 1;

Send another SMS:

curl -X POST http://localhost:3000/notifications \
-H "Content-Type: application/json" \
-d '{
  "userId": 1,
  "channel": "sms",
  "recipient": "+8801700000000",
  "message": "This SMS should not be sent."
}'

Check database:

SELECT id, channel, status
FROM notifications
ORDER BY id DESC
LIMIT 1;

Expected:

sms | skipped

No SMS should appear in the worker logs.

This demonstrates preference-based notification delivery.

Step 33: Simulate Provider Failure

Now we need to test retry behavior.

Modify:

src/providers/sms.provider.js

Temporarily replace it with:

async function sendSMS() {
  console.log("SMS provider failed");

  throw new Error(
    "Temporary SMS provider error"
  );
}

module.exports = {
  sendSMS,
};

Enable SMS again:

UPDATE notification_preferences
SET sms_enabled = TRUE
WHERE user_id = 1;

Restart the worker.

Now create an SMS notification.

You should see several attempts.

BullMQ will retry because we configured:

attempts: 3

The delays increase because we configured:

type: "exponential"

Conceptually:

Attempt 1
↓
Fail
↓
Wait
↓
Attempt 2
↓
Fail
↓
Wait longer
↓
Attempt 3

This is much safer than retrying immediately.

Step 34: Why Exponential Backoff Matters

Imagine a provider like Twilio becomes unavailable.

If 50,000 jobs instantly retry, you may create even more load.

Instead:

Attempt 1
↓
2 seconds

Attempt 2
↓
4 seconds

Attempt 3
↓
8 seconds

This gives the external provider time to recover.

Step 35: Dead Letter Queue Concept

Some notifications may fail permanently.

For example:

Invalid email address
Invalid phone number
Expired push token
Provider rejection

Instead of retrying forever, a production system should eventually move failed messages into a separate queue.

Conceptually:

Notification Queue
       ↓
     Worker
       ↓
   Retry 3 times
       ↓
Still Failed
       ↓
Dead Letter Queue

You could create:

const failedQueue = new Queue(
  "failed-notifications",
  {
    connection: redisConnection,
  }
);

Then after permanent failure:

await failedQueue.add(
  "failed-notification",
  {
    notificationId,
    error: error.message,
  }
);

This allows operations teams to inspect failed messages later.

Step 36: Prevent Duplicate Notifications

Duplicate notifications are dangerous.

For example, a customer should not receive:

Payment received
Payment received
Payment received

because an API request was retried.

One solution is an idempotency key.

Add:

ALTER TABLE notifications
ADD COLUMN idempotency_key VARCHAR(255);

Create a unique index:

CREATE UNIQUE INDEX
unique_notification_idempotency
ON notifications(idempotency_key);

Now the client can send:

{
  "idempotencyKey": "order-1001-confirmation"
}

If the same request arrives twice, the database rejects the duplicate.

Step 37: Example Idempotent Request

Request:

{
  "userId": 1,
  "channel": "email",
  "recipient": "developer@example.com",
  "title": "Payment Received",
  "message": "Payment completed.",
  "idempotencyKey": "payment-1001-email"
}

If another identical request arrives:

payment-1001-email

the system can return the existing notification instead of creating another one.

Step 38: Add Notification Templates

Large systems should not hardcode every message.

Instead, store templates.

Create:

CREATE TABLE notification_templates (
    id SERIAL PRIMARY KEY,
    name VARCHAR(100) UNIQUE NOT NULL,
    channel VARCHAR(20) NOT NULL,
    title_template TEXT,
    message_template TEXT NOT NULL
);

Example:

INSERT INTO notification_templates (
    name,
    channel,
    title_template,
    message_template
)
VALUES (
    'order_confirmed',
    'email',
    'Order {{orderId}} Confirmed',
    'Hello {{name}}, your order {{orderId}} has been confirmed.'
);

Application data:

{
  "name": "Rahim",
  "orderId": "1001"
}

Rendered result:

Hello Rahim, your order 1001 has been confirmed.

Step 39: Why Templates Matter

Without templates:

Order Service contains email HTML
Payment Service contains SMS text
Shipping Service contains push text

This becomes difficult to maintain.

With templates:

Business Service
     ↓
Notification System
     ↓
Template Engine
     ↓
Provider

Notification content becomes centralized.

Step 40: Add Bulk Notifications

Imagine an admin wants to notify 100,000 users.

Do not process all notifications inside one HTTP request.

A better design:

Admin
 ↓
Create Campaign
 ↓
Campaign Queue
 ↓
Batch Generator
 ↓
Notification Queue
 ↓
Workers

Instead of:

100,000 notifications

at once, process batches:

Batch 1 = 1,000
Batch 2 = 1,000
Batch 3 = 1,000

This protects the database and providers.

Step 41: Worker Concurrency

BullMQ workers can process multiple jobs simultaneously.

Example:

const worker = new Worker(
  "notifications",
  processor,
  {
    connection: redisConnection,
    concurrency: 10,
  }
);

This means:

10 notifications

can be processed at the same time.

Be careful.

If you set:

concurrency: 500

but your email provider allows only:

100 requests per second

your provider may start rejecting requests.

Concurrency must match provider capacity.

Step 42: Rate Limiting Workers

For production, you can configure limits.

Example:

const worker = new Worker(
  "notifications",
  processor,
  {
    connection: redisConnection,
    concurrency: 10,
    limiter: {
      max: 100,
      duration: 1000,
    },
  }
);

This means approximately:

100 jobs per second

This can prevent provider rate-limit violations.

Step 43: Separate Queues by Channel

Our example uses one queue.

For larger systems, separate them:

email-notifications

sms-notifications

push-notifications

in-app-notifications

Architecture:

Notification Service
       │
       ├── Email Queue
       │      ↓
       │  Email Workers
       │
       ├── SMS Queue
       │      ↓
       │  SMS Workers
       │
       ├── Push Queue
       │      ↓
       │  Push Workers
       │
       └── In-App Queue
              ↓
          In-App Workers

This has a major advantage.

If the SMS provider becomes slow, email notifications continue normally.

Step 44: Priority Notifications

Not all notifications have equal importance.

For example:

Password reset

should have higher priority than:

Weekly marketing newsletter

BullMQ supports priorities.

Example:

await notificationQueue.add(
  "send-notification",
  data,
  {
    priority: 1,
  }
);

A marketing notification might use:

priority: 10

Conceptually:

Priority 1
Security Alert

Priority 2
Payment Alert

Priority 5
Order Update

Priority 10
Marketing

Step 45: Schedule Notifications

Some notifications need to be sent later.

Example:

Appointment reminder tomorrow at 9 AM

BullMQ supports delayed jobs.

Example:

await notificationQueue.add(
  "send-notification",
  data,
  {
    delay: 60 * 60 * 1000,
  }
);

That schedules the job approximately one hour later.

Step 46: Real Production Providers

After everything works locally, replace mocks.

Email

Local:

MailHog

Production:

AWS SES
SendGrid
Mailgun
Postmark

SMS

Local:

Console Mock

Production:

Twilio
AWS SNS
Vonage

Push

Local:

Console Mock

Production:

Firebase Cloud Messaging
Expo Notifications
Apple Push Notification Service

The advantage of our provider architecture is that the worker does not need major changes.

Only provider implementations change.

Step 47: Handling Provider Timeouts

External providers can hang.

Never wait forever.

Conceptually:

Worker
 ↓
Provider
 ↓
Timeout after 5 seconds
 ↓
Retry

For example, using an HTTP client:

axios.post(url, payload, {
  timeout: 5000,
});

This prevents workers from becoming permanently blocked.

Step 48: Monitoring the System

A production notification system should track:

Notifications created

Notifications sent

Notifications failed

Average processing time

Queue length

Retry count

Provider latency

Email bounce rate

SMS delivery rate

Push failures

Example metrics:

Email Sent: 99,500

Email Failed: 500

SMS Sent: 45,000

SMS Failed: 250

Queue Pending: 1,250

Average Processing Time: 180 ms

Step 49: Useful Logs

Logs should contain enough information to debug failures.

Example:

notificationId=541

userId=120

channel=email

status=failed

provider=ses

attempt=3

error=connection_timeout

Avoid logging secrets such as:

Passwords
Authentication tokens
API keys
Full OTP values

Step 50: Scaling the Notification System

Our local system:

API
 ↓
Redis
 ↓
Worker
 ↓
Provider

can scale by adding more workers.

For example:

Redis Queue
   │
   ├── Worker 1
   ├── Worker 2
   ├── Worker 3
   └── Worker 4

BullMQ distributes jobs between workers.

This is horizontal scaling.

Step 51: Production Architecture

A larger production architecture may look like:

                    Applications
                         ↓
                  Load Balancer
                         ↓
                Notification API
                         ↓
                   PostgreSQL
                         │
                         ↓
                      Redis
                         │
       ┌─────────────────┼─────────────────┐
       │                 │                 │
       ↓                 ↓                 ↓
 Email Queue         SMS Queue         Push Queue
       ↓                 ↓                 ↓
Email Workers       SMS Workers       Push Workers
       ↓                 ↓                 ↓
 Email Provider      SMS Provider      Push Provider

In-app notifications may be stored directly in:

PostgreSQL

and delivered to connected clients using:

WebSocket
Server-Sent Events
Polling

Step 52: Example Real-World Flow

Suppose a customer places an order.

The Order Service publishes:

{
  "event": "ORDER_CONFIRMED",
  "userId": 100,
  "orderId": 5001
}

The notification system receives it.

It looks up user preferences:

Email = Yes
SMS = No
Push = Yes

It creates:

Email Notification
Push Notification

but skips:

SMS Notification

Both jobs go into their queues.

Workers process them independently.

The customer may receive:

Email:
Your order #5001 has been confirmed.

Push:
Order confirmed!

Step 53: Complete Local Test Checklist

Before calling the system complete, test every scenario.

Test 1

Health endpoint:

curl http://localhost:3000/health

Expected:

200 OK

Test 2

Email notification.

Expected:

Mail appears in MailHog.

Test 3

SMS notification.

Expected:

SMS appears in worker console.

Test 4

Push notification.

Expected:

Push appears in worker console.

Test 5

In-app notification.

Expected:

Notification appears through GET API.

Test 6

Disable SMS preference.

Expected:

status = skipped

Test 7

Force provider failure.

Expected:

Worker retries job.

Test 8

Allow all retries to fail.

Expected:

status = failed

Test 9

Send duplicate idempotency key.

Expected:

Duplicate notification is prevented.

Test 10

Stop the worker.

Create notification.

Expected:

Notification remains queued.

Restart the worker.

Expected:

Queued notification is processed.

This last test demonstrates one of the biggest advantages of asynchronous architecture.

The API and notification worker do not need to be available at exactly the same moment.

Key System Design Lessons

This implementation demonstrates several important system design concepts.

Asynchronous Processing

The API does not wait for external providers.

Instead:

API
 ↓
Queue
 ↓
Worker

This keeps user requests fast.

Loose Coupling

The application creating the notification does not need to know how email, SMS, or push providers work.

Retries

Temporary failures are handled automatically.

Idempotency

Duplicate notifications can be prevented.

Horizontal Scaling

More workers can be added when traffic increases.

User Preferences

Users control which notification channels they receive.

Rate Limiting

Workers can respect provider limits.

Fault Isolation

Separate queues prevent one provider from blocking another channel.

Avoid Overengineering

You do not need the complete production architecture on day one.

A small application may start with:

Application
 ↓
Database
 ↓
Single Worker

As traffic increases, add:

Redis Queue

Then:

Multiple Workers

Then:

Separate Channel Queues

Eventually:

Multiple Regions
Provider Failover
Dedicated Analytics
Advanced Monitoring

System design should grow with real requirements.

Final Architecture

The system we implemented locally looks like:

Client
  ↓
Notification API
  ↓
PostgreSQL
  ↓
Redis / BullMQ
  ↓
Notification Worker
  │
  ├── Email → MailHog
  │
  ├── SMS → Mock Provider
  │
  ├── Push → Mock Provider
  │
  └── In-App → PostgreSQL

A production version may evolve into:

Applications
     ↓
Notification API
     ↓
Database
     ↓
Message Broker
     │
 ┌───┼────┬───────┐
 ↓   ↓    ↓       ↓
Email SMS Push   In-App
 ↓    ↓    ↓       ↓
Workers Workers Workers
 ↓    ↓    ↓
External Providers

Conclusion

A notification system looks simple until you need to support multiple channels, large traffic, provider failures, retries, preferences, and guaranteed delivery.

The core principle is:

Do not send notifications directly inside the user's request whenever the operation can be processed asynchronously.

Instead, separate the system into:

Create
Queue
Process
Deliver
Track

The architecture becomes:

Application
 ↓
Notification API
 ↓
Queue
 ↓
Worker
 ↓
Provider

This design improves:

• Performance

• Reliability

• Scalability

• Maintainability

• Failure recovery

Most importantly, the architecture we built in this article can be tested locally.

You can:

Create notifications
Observe queues
Stop workers
Restart workers
Simulate provider failures
Test retries
Disable notification preferences
Inspect database states
Test emails using MailHog

That makes this more than a system design diagram.

It becomes a working implementation of the concepts.