System Design - Distributed Notification System

A modern enterprise notification system orchestrates billions of real-time transactional, security, and promotional messages across heterogeneous communication channels: mobile push notifications (iOS APNs, Android FCM), SMS (Twilio), email (SendGrid), and in-app feeds.

In system design interviews, designing a notification system evaluates your understanding of asynchronous event queues, third-party vendor rate limiting, user preference modeling, idempotency, and high-throughput prioritization.

Requirements & Scope

Functional Requirements

Multi-Channel Support: Push notifications (iOS / Android), SMS, Email, and In-App notifications.
Prioritization: High-priority alerts (e.g., OTP login codes, fraud alerts, password resets) must deliver within 5 seconds; low-priority messages (e.g., marketing promotions, weekly summaries) can be delayed or batched.
User Preferences & Opt-Outs: Users can customize notification settings per channel, category, or set "Do Not Disturb" quiet hours.
Deduplication: Prevent spamming users with identical notifications due to upstream service retries.
Rate Limiting per User: Protect users from receiving more than $X$ promotional notifications per hour or day.
Analytics & Delivery Receipts: Track dispatch status (sent, delivered, opened, bounced, unsubscribed).

Non-Functional Requirements

High Availability & Fault Tolerance: Failure of an external provider (e.g., Twilio outage) must not crash the notification pipeline.
Scalability: Capable of sending 100 million notifications per day with peak bursts of 20,000 notifications/sec.
Zero Loss for Critical Alerts: Transactional alerts must never be lost.

High-Level System Architecture

[ Microservices / Cron ] ──► [ Notification Gateway ]
                                      │
                 ┌────────────────────┼────────────────────┐
                 ▼                    ▼                    ▼
        [ User Preferences ]   [ Rate Limiter ]   [ Deduplication ]
          (PostgreSQL/Cache)    (Redis Counter)    (Redis Bloom/NX)
                 │                    │                    │
                 └────────────────────┼────────────────────┘
                                      ▼
                        [ Prioritized Message Queues ]
                      (Kafka Topics / RabbitMQ Streams)
                                      │
                 ┌────────────────────┴────────────────────┐
                 ▼                                         ▼
      [ High-Priority Workers ]                 [ Bulk/Low-Priority Workers ]
                 │                                         │
                 └────────────────────┬────────────────────┘
                                      ▼
                      [ Third-Party Provider Adapters ]
                 ┌──────────┬──────────┬──────────┬──────────┐
                 ▼          ▼          ▼          ▼          ▼
              [ APNs ]   [ FCM ]   [ Twilio ] [ SendGrid ] [ In-App ]
              (Apple)   (Google)     (SMS)     (Email)      (Redis)

Architecture Breakdown

Notification Gateway: Stateless API service validating payloads, authenticating client services, and verifying recipient credentials.
Filter & Policy Engine:
- User Preference Check: Verifies if the recipient opted out of this specific notification category or is currently in a quiet-hours window.
- Rate Limiter: Checks sliding window counters in Redis (e.g., maximum 3 marketing emails per day per user).
- Deduplication Filter: Checks a hash of (user_id, template_id, payload_hash) in Redis with a 10-minute TTL to prevent duplicate dispatches.
Prioritized Message Broker: Separates traffic into distinct Kafka topics or message queues based on criticality:
- notifications.priority.high (OTP codes, emergency security alerts).
- notifications.priority.normal (Order confirmations, chat mentions).
- notifications.priority.bulk (Marketing campaigns, digest emails).
Notification Workers: Scalable worker pools consuming messages from queues, assembling dynamic message templates, and dispatching to channel-specific adapters.
Provider Adapters: Encapsulate third-party vendor SDKs, handle provider-specific authentication, and implement circuit breakers.

Data Models & Schema Design

1. User Notification Settings

CREATE TABLE user_notification_preferences (
    user_id BIGINT NOT NULL,
    channel VARCHAR(16) NOT NULL,       -- 'PUSH', 'SMS', 'EMAIL', 'IN_APP'
    category VARCHAR(32) NOT NULL,      -- 'SECURITY', 'TRANSACTION', 'MARKETING'
    is_enabled BOOLEAN DEFAULT TRUE,
    quiet_hours_start TIME NULL,
    quiet_hours_end TIME NULL,
    timezone VARCHAR(64) DEFAULT 'UTC',
    PRIMARY KEY (user_id, channel, category)
);

2. Notification Log & Audit Trail

CREATE TABLE notification_logs (
    notification_id UUID PRIMARY KEY,
    user_id BIGINT NOT NULL,
    channel VARCHAR(16) NOT NULL,
    status VARCHAR(16) NOT NULL,        -- 'QUEUED', 'SENT', 'DELIVERED', 'FAILED'
    retry_count INT DEFAULT 0,
    error_message TEXT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX idx_user_created ON notification_logs (user_id, created_at DESC);

Step-by-Step Notification Dispatch Flow

An upstream service (e.g., Payment Service) posts an event:

{
  "recipient_id": "usr_9981",
  "event_type": "PAYMENT_SUCCESS",
  "priority": "HIGH",
  "channels": ["EMAIL", "PUSH"],
  "template_data": { "amount": "$49.99", "order_id": "ord_102" }
}

Gateway Verification: The Gateway authenticates the caller and generates an idempotent notification_id.
Policy Evaluation:
- Queries the User Preference cache. If the user disabled push notifications for purchases, the push channel is pruned.
- Verifies the user is not in "Do Not Disturb" hours (unless overridden by priority: HIGH).
Queue Ingestion: The Gateway publishes independent work items to notifications.email and notifications.push.
Template Rendering: The assigned worker fetches the localized template (e.g., French or English based on user profile) and interpolates parameters.
Provider Dispatch: The worker invokes the vendor adapter (e.g., SendGrid API).
Delivery Tracking: When SendGrid fires an async webhook indicating successful delivery or bounce, the event is ingested into notification_logs for audit tracking.

Resilience & Production Hardening

1. Exactly-Once Delivery & Idempotency

Network timeouts between the notification worker and third-party vendors can lead to ambiguous outcomes: did the SMS send, or did the connection fail before transmission?

Upstream producers attach an idempotency_key (UUID) to every notification request.

Workers use Redis atomic operations:

SET notification:idempotency:<key> "PROCESSING" NX EX 300

If the key exists, the request is recognized as an in-flight duplicate and discarded.
When calling third-party vendors, pass the idempotency_key in request headers (supported by Stripe, Twilio, and SendGrid) so the provider avoids duplicate dispatch.

2. Dead Letter Queues (DLQ) & Exponential Backoff

If an external vendor fails (e.g., HTTP 503 or network timeout):

The worker reschedules the message using exponential backoff with jitter: $$\text{Delay} = \min(60, 2^{\text{retry_count}} \times \text{rand}(1, 1.5)) \text{ seconds}$$
After reaching a maximum retry threshold (e.g., 3 to 5 attempts), the message is moved to a Dead Letter Queue (DLQ).
Engineers monitor DLQ metrics; failed messages can be replayed after third-party outages are resolved.

3. Provider Failover & Fallbacks

What happens if Twilio experiences a major region-wide outage?

Implement the Circuit Breaker Pattern around external vendor calls.
If Twilio returns continuous 5xx errors exceeding a 10% error threshold over 60 seconds, the circuit breaker opens.
The adapter automatically routes SMS messages to a secondary provider (e.g., MessageBird or AWS SNS) until Twilio health checks recover.

4. Preventing Thundering Herds & Rate Limiting Vendors

Third-party providers enforce strict incoming API rate limits (e.g., SendGrid may limit an account to 500 requests/second):

Deploy a token-bucket rate limiter in front of each provider adapter.
If the token bucket is exhausted, workers pause consumption from the queue rather than overwhelming the provider and triggering HTTP 429 Too Many Requests.