Redis Internals, Probabilistic Structures & Datastore Selection

When to use it: going deeper on Redis (memory blowups, cluster, client-side caching),
reaching for probabilistic structures (Bloom/Count-Min/Top-K/t-digest), or deciding whether
to add a specialized store (search engine, OLAP warehouse, Kafka, graph, vector DB) instead
of stretching Postgres/Redis. Companion to Push Compute to the Datastore (§5 Redis).

1. Redis memory & encoding

Small collections use a compact listpack/intset encoding; past a threshold they convert to
the full pointer structure (hashtable/skiplist). OBJECT ENCODING key shows which.¹

• Fold "one entity → many keys" into one Hash. user:123:name/:email/:age each pay Redis's
  per-key overhead (~tens of bytes of dictEntry+robj); HSET user:123 name … email … pays
  it once and packs the body densely (3–5× reported saving) — if field count stays under
  hash-max-listpack-entries (512). Past it, hash ops revert from listpack's O(n) scan to
  hashtable O(1) but memory jumps.
• Measure with MEMORY USAGE key, not DBSIZE × avg_value (ignores per-key overhead +
  fragmentation, which dominate for many small keys).
• Expiration vs eviction are different. Expiration (TTL): lazy on access + active sampling.
  Eviction (memory pressure, only when maxmemory set): pick a policy (allkeys-lru/-lfu
  for caches). Set maxmemory + a policy or a cache workload either grows unbounded or hits
  OOM command not allowed under noeviction.

2. Probabilistic structures — bounded memory, bounded error

As of Redis 8 these are in core (formerly RedisBloom). Each answers a question your app would
otherwise answer by pulling the full dataset in and looping — at fixed sub-linear memory. Pick
the one whose error mode matches your risk:²

• Bloom as a cache-penetration guard: BF.EXISTS before a cache-miss DB lookup — a
  never-seen key skips Postgres entirely (blocks nonexistent-key hammering). ~9.6 bits/item at
  1% error vs a full SET's hundreds of bits/item (~17× smaller). Always confirm a positive
  against the real store; trust only negatives.
• Cuckoo when membership is revocable (coupon redeemed → CF.DEL); Bloom when append-only.
• Count-Min instead of a per-key Hash of counters when key cardinality is unbounded (URLs,
  IPs) and approximate magnitude suffices — never for billing/quotas (it overcounts).

3. RediSearch / RedisJSON, Cluster, client-side caching

• RediSearch (FT.CREATE/FT.SEARCH, core in Redis 8) adds secondary indexing + full-text
  • vector to Redis data — right when the canonical hot data already lives in Redis and now
    needs richer-than-key queries. Wrong when Postgres already holds the canonical data and
    you'd mirror it just to search (a dual-write consistency burden — use Postgres FTS, see
    Postgres Full-Text and Fuzzy Search).³
• Cluster: 16384 hash slots; CRC16(key) mod 16384. Multi-key ops (MSET, Lua, MULTI)
  require all keys in one slot or you get CROSSSLOT. Use hash tags — {user:1000}.name
  and {user:1000}.sessions co-locate — but never one giant tag (that collapses to a single
  hot node). Design keys for co-location before you need cluster mode.⁴
• Client-side caching (RESP3 CLIENT TRACKING): the server pushes an invalidation when a
  tracked key changes, enabling a self-invalidating in-process cache (no round trip on hits,
  no TTL-staleness). Best for hot small reference data (flags, config).
• Keyspace notifications ride Pub/Sub → unreliable (offline listener misses events, no
  replay). For a must-happen side effect use Streams (acks + replay), not notifications.
• Redis vs Memcached: Memcached wins only for pure multi-core GET/SET/DEL caching with no
  persistence/structures/scripting (natively multithreaded). Anything richer → Redis.

4. When Redis is the wrong tool

Not the system of record (RDB loses seconds–minutes on crash; MULTI/EXEC has no rollback;
RAM-bounded). No joins/relational integrity. No cross-slot multi-key atomicity in Cluster. Large
blobs belong in object storage with metadata in Postgres. Per-write durability at volume is a
Postgres/durable-log problem.

5. Reach for a specialized store only when the pain is real

Default: don't add a datastore until you have measured, current pain on a specific access
pattern — every new store is another backup/monitor/on-call/consistency surface and a second
source of truth (a bug generator). There's almost always a cheaper intermediate step inside
Postgres/Redis first. Prefer a managed version before self-hosting.⁵

The meta-pattern: the "graduate" signal is always a measured production regression (real
p95/ingest-ceiling/cache-miss storm), never hypothetical future scale — and you run the
specialized store as a secondary, always-rederivable-from-Postgres index, never the source
of truth.

See also

• Push Compute to the Datastore — the hub and the ladder; Redis is §5, specialized stores §7.
• Postgres Full-Text and Fuzzy Search — exhaust this before an external search engine.
• Postgres Scale-Out, Replication and Data Movement — the in-Postgres analytics escalation.