Reducing Latency in Telegram-Scale Infrastructure with Hot Path Caching and Stateful Shard Affinity

Telegram-style infrastructure has a distinctive profile: extremely high fan-out, frequent small reads, bursty writes, and strict expectations around responsiveness. The fastest systems are not usually the ones with the most hardware; they are the ones that avoid unnecessary cross-node work on the critical path.

This guide focuses on a concrete performance improvement: reducing end-to-end latency for message delivery and update retrieval by combining hot path caching with shard affinity for stateful data.

The goal is not to redesign the whole stack. It is to shave milliseconds from the paths that run millions of times per second.

The latency problem

In a Telegram-like system, a single user action often touches several internal subsystems:

• authentication and session lookup
• dialog metadata retrieval
• message fan-out or update sequencing
• storage reads for recent history
• deduplication and idempotency checks

The hidden latency cost is usually not the database itself. It is the coordination:

• extra network hops between services
• cache misses that cascade into backend reads
• shard moves caused by poor locality
• mutex contention on shared hot keys
• cross-region or cross-rack traffic on the synchronous path

If you profile the p95 or p99 latency of a message-related endpoint, you will often find that only one or two components are responsible for most of the tail.

Target architecture

The optimization works best when the system is organized around two principles:

1. State should have a stable home.
2. Hot reads should be satisfied without leaving the process whenever possible.

That leads to this layout:

• Shard-affine workers own a consistent subset of users, chats, or dialogs.
• In-process hot caches store recent metadata, sequence counters, and session state.
• Backend storage remains the source of truth, but it is consulted only on cache miss or invalidation.
• Asynchronous replication or invalidation updates secondary nodes without blocking the read path.

The important part is that the shard key and the access pattern align. If a request for user U usually lands on the worker that already owns U's active state, you eliminate a round trip before the request even reaches storage.

What to cache

Do not cache large payloads first. Cache the fields that are both small and frequently consulted.

Good candidates:

• user session metadata
• dialog last-read sequence numbers
• per-chat delivery offsets
• message existence checks
• permission bits and role flags
• recent notification state

Avoid caching:

• full message histories
• large media blobs
• rarely accessed configuration objects
• data that changes too frequently without a cheap invalidation strategy

The best hot-path cache entries are compact, immutable for short windows, and cheap to serialize.

Shard affinity in practice

Shard affinity means routing requests so that related state tends to be handled by the same worker or the same small set of workers.

A simple version uses a consistent hash on the primary entity key:

worker_id = hash(user_id) mod worker_count

That is easy to implement, but Telegram-scale workloads usually need more nuance:

• A user may belong to many chats.
• A chat may have many consumers.
• Update streams can be read-heavy but write-sensitive.
• Some users become extremely hot for short bursts.

A better strategy is a two-level affinity model:

• Primary ownership: one worker owns the authoritative in-memory state for a key.
• Read replicas or local mirrors: nearby workers hold short-lived copies for read-heavy access.

For example:

• user session state stays pinned to the primary worker
• dialog summary data is replicated to the edge of the request path
• write acknowledgements go through the owner
• reads that do not require immediate consistency can be served locally

This reduces the cost of common operations while keeping correctness constraints explicit.

The critical path for message delivery

Consider a simplified delivery flow:

1. client sends a message
2. gateway authenticates the session
3. routing layer selects the owner shard
4. owner validates permissions and sequence state
5. message is written to durable storage
6. recipient update queues are populated
7. sender receives acknowledgment

There are two places where latency often balloons:

• step 3 if routing requires an additional lookup
• step 4 if every permission or sequence check hits storage

The improvement is to keep the routing decision and the most common validation state in memory.

Before

• gateway asks a centralized directory for shard location
• worker fetches session and dialog state from database
• database round trips happen serially
• each miss adds 1–3 ms, sometimes more under load

After

• gateway uses a cached shard map
• worker already owns the user’s active session state
• permission and sequence checks are answered from the local cache
• database is touched only for write-through or cold miss

This can remove multiple synchronous hops per request.

Cache consistency strategy

Performance gains are worthless if they introduce subtle data corruption. The consistency model must be clear.

Use one of these patterns:

1. Write-through for authoritative small state

When a value is mutated:

• update the in-memory owner state
• synchronously persist the change to durable storage
• publish an invalidation or version bump

This is best for:

• counters
• last seen offsets
• auth/session flags

2. Versioned invalidation for replicated reads

Each cached item carries a monotonically increasing version.

• readers accept cached data if version matches or is recent enough
• writers increment the version on commit
• stale copies are invalidated asynchronously

This is useful for:

• dialog summaries
• participant role snapshots
• read-mostly metadata

3. Lease-based cache entries

A worker can hold a short lease on a key, guaranteeing ownership for a brief period.

• reduces repeated routing lookups
• prevents thrashing on hot keys
• enables predictable eviction behavior

This is effective when a single user or chat becomes extremely active for a short interval.

Practical implementation details

Keep cache entries small

Small entries fit better in CPU cache lines and reduce memory bandwidth pressure.

Example compact shape:

type DialogState struct {
    LastSeq   uint32
    ReadSeq   uint32
    Flags     uint16
    Version   uint64
}

If you store this alongside a session key, you can answer most checks without touching storage.

Avoid global locks

A global mutex over a hot cache will destroy tail latency. Use:

• sharded maps
• lock striping
• atomic pointers for immutable snapshots
• per-key or per-shard single-flight deduplication

If multiple goroutines frequently request the same cold key, single-flight prevents a thundering herd.

Batch backend reads

When a miss does occur, fetch related keys together.

Instead of:

• session lookup
• dialog lookup
• permission lookup
• sequence lookup

in four separate round trips, use one batched read per shard.

Batching matters because the network overhead often exceeds the storage cost for small records.

Prewarm hot keys after deploy

A rolling restart can wipe useful state from memory. Mitigate that by:

• snapshotting the hottest keys before shutdown
• restoring them on startup
• warming from recent access logs
• using staggered restarts to avoid synchronized cold starts

This is especially important for large chats, active sessions, and popular public channels.

Observability you need before changing anything

If you cannot measure the hot path, you cannot optimize it safely.

Track at minimum:

• p50, p95, p99 latency per endpoint
• cache hit ratio by key class
• shard skew and key distribution
• backend read amplification
• invalidation lag
• queue depth per worker
• lock contention on cache access

Useful breakdowns:

• by request type
• by user activity level
• by chat size
• by region
• by worker shard

A single aggregate hit rate is not enough. A 95% hit rate may still be terrible if the 5% misses are concentrated on the hottest user flows.

Common failure modes

Hot key amplification

A very active chat can overload its owner shard.

Mitigations:

• split read responsibilities from write ownership
• use replicated read state for fan-out-heavy metadata
• apply per-key admission control
• move noncritical counters off the synchronous path

Stale local state

A fast cache that serves old data is worse than a slow cache.

Mitigations:

• version every mutable record
• enforce bounded staleness windows
• invalidate on commit for critical fields
• fall back to source of truth on conflict

Over-caching

If you cache too much, memory pressure causes eviction churn, and latency gets worse.

Mitigations:

• cache only high-frequency, low-size keys
• enforce size budgets per class
• monitor eviction rate, not just hit rate

Routing instability

If shard ownership changes too often, cache locality never settles.

Mitigations:

• keep shard assignments stable
• rebalance slowly
• migrate cold keys first
• avoid reactive reassignments during traffic spikes

A rollout plan that is safe

1. Instrument the existing path. Identify the top 3 synchronous calls.
2. Introduce a read-only hot cache. Measure hit rate and staleness.
3. Pin ownership for one state class. Start with session metadata or dialog offsets.
4. Add versioned invalidation. Keep the source of truth unchanged.
5. Move one validation check off storage. Usually permissions or sequence state.
6. Compare p95 and p99 before and after. Focus on tail behavior, not averages.

Do not optimize everything at once. The value of this pattern comes from isolating a single hot path and making it locally efficient.

When this approach works best

This technique is strongest when the workload has the following traits:

• many small requests
• high read frequency
• stable key ownership
• clear primary shard keys
• tolerance for short-lived cached metadata

It is weaker when:

• writes dominate reads
• keys are uniformly cold
• strict cross-object transactions are required
• ownership changes constantly

Final takeaway

At Telegram scale, latency is usually reduced not by faster storage, but by fewer synchronous dependencies. The most effective improvement is to keep hot state close to the worker that needs it and to ensure that the common case never waits on a remote lookup.

If you preserve shard affinity for mutable session-like data and cache only the smallest, hottest pieces of metadata, you can cut multiple network hops from the critical path. That reduces p95 and p99 latency, lowers backend load, and makes the system more predictable under bursty traffic.

In short: own the hot state, cache the hot reads, and make the slow path exceptional.