Reducing Telegram Latency by Optimizing Message Delivery Paths

Telegram infrastructure is a mix of real-time delivery, durable storage, media handling, and multi-device synchronization. If you are working on the backend side of such a system, the fastest wins usually do not come from rewriting everything in a new language or adding more machines. They come from shaving work off the message delivery path.

This tutorial focuses on one practical performance improvement: decoupling message acknowledgment from downstream fanout and persistence work. The goal is to reduce the time between a client sending a message and the server confirming receipt, while preserving correctness.

What is usually slow in Telegram-like systems?

A message send request often triggers all of the following:

1. Authentication and rate limiting
2. Serialization and validation
3. Deduplication and ordering checks
4. Persistence to durable storage
5. Fanout to the recipient’s active sessions
6. Push notification dispatch for offline devices
7. Cross-region replication or queueing

If these steps are chained synchronously, the user-visible latency becomes the sum of many small delays. The worst part is usually not the average; it is the tail latency caused by slow storage, busy fanout workers, or transient network jitter.

The most effective improvement is to keep the critical path short and move non-essential work off it.

Define the critical path

For a send-message API, ask one question:

What must happen before the server can safely acknowledge the message?

In most systems, the answer is:

• the request is authenticated
• the message is assigned a server-side ID or sequence number
• the message is written to a durable log or durable queue

Everything else can often happen after acknowledgment, provided it is idempotent and recoverable.

That means these steps can be deferred:

• pushing to online recipient sessions
• updating secondary indexes
• sending unread counters
• generating push notifications
• analytics and tracing enrichment

Before: synchronous fanout in the request path

A common implementation looks like this:

client -> api gateway -> message handler
    -> validate
    -> write to DB
    -> fanout to online devices
    -> enqueue push notifications
    -> update counters
    -> return OK

This is simple, but every downstream dependency slows the sender. If a single recipient device session is slow, the sender may wait even though the message is already durably stored.

After: acknowledge after durability, fanout asynchronously

A better layout is:

client -> api gateway -> message handler
    -> validate
    -> append to durable log
    -> return OK

background workers:
    -> fanout to active sessions
    -> enqueue push notifications
    -> update counters
    -> replicate or index

The key is that the handler only waits for the minimum durable write. Everything else is driven by a queue, stream, or log consumer.

Why this improves performance

1. Fewer synchronous dependencies

When the request path depends on fewer services, the probability of waiting on a slow component drops sharply.

2. Better batching opportunities

Background workers can batch fanout and storage updates. Batching is much harder when each request must complete inline.

3. Lower tail latency

Tail latency often comes from rare slow sub-operations. Moving them out of the critical path protects the user-facing API.

4. Improved isolation

A spike in notification delivery no longer slows message acceptance. That separation is especially important during high-traffic events or large group chats.

Implementation pattern: durable log first

A good pattern for Telegram-like systems is:

1. Accept request
2. Validate sender state and permissions
3. Append a compact message record to a durable log
4. Return acknowledgment
5. Process the log asynchronously

The durable log can be:

• a replicated commit log
• a partitioned queue with strong persistence
• an append-only storage layer

The important property is that it is the source of truth for downstream work.

Example record shape

{
  "chat_id": 481516,
  "message_id": 992771,
  "sender_id": 1442,
  "timestamp": 1730000000,
  "payload_ref": "blob:aa91...",
  "delivery_flags": ["fanout_pending", "push_pending"]
}

Keep the record compact. Large payloads should usually be stored separately and referenced by ID.

Make workers idempotent

Once you move work off the request path, retries become normal. That means every asynchronous stage must be safe to run more than once.

For example:

• fanout worker should skip already-delivered session receipts
• notification worker should deduplicate by message ID and device ID
• counter updater should use monotonic offsets or compare-and-swap semantics

A simple rule: every downstream side effect should be traceable to a unique message record.

Avoid hidden synchronous reads

A common mistake is to move writes async but leave reads synchronous.

For example, after appending a message, your handler might still do this:

• read the recipient’s presence state
• read session routing metadata
• read unread counters for response payload

Those reads can still dominate latency. Cache hot metadata in memory or in a local shard index, and accept slightly stale non-critical data when possible.

For Telegram-style traffic, presence and session routing are usually ideal cache candidates because they are read-heavy and tolerate short staleness.

Measure the right metric

Do not stop at average request time. Track:

• P50 send latency
• P95 and P99 send latency
• durable append time
• queue lag for fanout workers
• worker retry rate
• per-chat fanout depth
• per-recipient session delivery time

If the optimization works, you should see:

• lower request latency
• stable append latency
• higher worker throughput
• bounded queue lag

If request latency drops but queue lag explodes, you have only moved the bottleneck.

Practical rollout plan

Phase 1: instrument

Add timing spans around:

• auth
• validation
• append/write
• fanout
• notification enqueue
• response serialization

This tells you where the time goes now.

Phase 2: split acknowledgment from side effects

Return success after the durable write is complete. Move fanout and notification work to background consumers.

Phase 3: add deduplication

Store a processed marker keyed by message ID plus stage name. This prevents duplicate delivery during retries.

Phase 4: batch consumers

Consume multiple message records at a time and group them by chat, recipient, or shard. This improves cache locality and reduces per-item overhead.

Phase 5: verify correctness under failure

Test these cases:

• worker crash after partial fanout
• duplicate message append attempts
• delayed replication
• recipient reconnect during delivery
• slow push provider

Your system should eventually converge without user-visible corruption.

Example of a safer request handler

def send_message(request):
    user = authenticate(request)
    validate_permissions(user, request.chat_id)

    record = build_message_record(request)
    offset = durable_log.append(record)

    # Return as soon as the message is safely stored.
    return {
        "status": "ok",
        "message_id": record.message_id,
        "log_offset": offset,
    }

And the worker:

def process_message(record):
    deliver_to_online_sessions(record)
    enqueue_push_notifications(record)
    update_unread_counters(record)
    mark_processed(record.message_id)

This structure makes the API path short and predictable.

Trade-offs to watch

This approach is not free.

Consistency becomes eventual for some fields

Unread counters and online delivery status may lag behind message acceptance.

Retry logic becomes mandatory

Background consumers must be resilient to duplicates, reordering, and restarts.

Debugging gets more complex

You need good tracing across the append log, worker queue, and delivery results.

These costs are worth it when latency matters, especially in large chats or high-volume regions.

When not to apply it

Keep some operations synchronous if they are required for correctness:

• authorization checks
• anti-abuse enforcement
• durable acceptance of the message
• safety-related content gating, if legally required in your environment

Do not defer anything that would make an unauthorized or invalid message appear accepted.

Final takeaway

For Telegram infrastructure, the fastest meaningful performance win is often to shorten the message send critical path. Acknowledge only after durable persistence, then move fanout, notification, and counter updates into idempotent asynchronous workers.

That one change usually delivers:

• lower send latency
• better tail behavior
• higher throughput under load
• cleaner service boundaries

If you are optimizing a Telegram-like backend, start here before chasing micro-optimizations in serialization or network code. The biggest gains usually come from removing work from the request path, not making every step slightly faster.