Redis Internals, Probabilistic Structures & Datastore Selection

When to use it: going deeper on Redis (memory blowups, cluster, client-side caching),
reaching for probabilistic structures (Bloom/Count-Min/Top-K/t-digest), or deciding whether
to add a specialized store (search engine, OLAP warehouse, Kafka, graph, vector DB) instead
of stretching Postgres/Redis. Companion to Push Compute to the Datastore (§5 Redis).

1. Redis memory & encoding

Small collections use a compact listpack/intset encoding; past a threshold they convert to
the full pointer structure (hashtable/skiplist). OBJECT ENCODING key shows which.¹

• Fold "one entity → many keys" into one Hash. user:123:name/:email/:age each pay Redis's
  per-key overhead (~tens of bytes of dictEntry+robj); HSET user:123 name … email … pays
  it once and packs the body densely (3–5× reported saving) — if field count stays under
  hash-max-listpack-entries (512). Past it, hash ops revert from listpack's O(n) scan to
  hashtable O(1) but memory jumps.
• Measure with MEMORY USAGE key, not DBSIZE × avg_value (ignores per-key overhead +
  fragmentation, which dominate for many small keys).
• Expiration vs eviction are different. Expiration (TTL): lazy on access + active sampling.
  Eviction (memory pressure, only when maxmemory set): pick a policy (allkeys-lru/-lfu
  for caches). Set maxmemory + a policy or a cache workload either grows unbounded or hits
  OOM command not allowed under noeviction.

2. Probabilistic structures — bounded memory, bounded error

As of Redis 8 these are in core (formerly RedisBloom). Each answers a question your app would
otherwise answer by pulling the full dataset in and looping — at fixed sub-linear memory. Pick
the one whose error mode matches your risk:²

Question	Structure	Error mode
How many distinct?	HyperLogLog (PFADD/PFCOUNT)	±% (~0.81%), ~12 KB fixed
Is X in the set? (no delete)	Bloom (BF.ADD/BF.EXISTS)	false positives only
Is X in the set? (need delete)	Cuckoo (CF.ADD/CF.DEL)	false positives only
Roughly how often did X happen? (unbounded keys)	Count-Min Sketch (CMS.INCRBY/CMS.QUERY)	overcounts only
Top-K most frequent?	Top-K (TOPK.ADD/TOPK.LIST)	transient misrank, converges
p50/p95/p99 of a stream?	t-digest (TDIGEST.ADD/.QUANTILE)	tunable via COMPRESSION

• Bloom as a cache-penetration guard: BF.EXISTS before a cache-miss DB lookup — a
  never-seen key skips Postgres entirely (blocks nonexistent-key hammering). ~9.6 bits/item at
  1% error vs a full SET's hundreds of bits/item (~17× smaller). Always confirm a positive
  against the real store; trust only negatives.
• Cuckoo when membership is revocable (coupon redeemed → CF.DEL); Bloom when append-only.
• Count-Min instead of a per-key Hash of counters when key cardinality is unbounded (URLs,
  IPs) and approximate magnitude suffices — never for billing/quotas (it overcounts).

3. RediSearch / RedisJSON, Cluster, client-side caching

• RediSearch (FT.CREATE/FT.SEARCH, core in Redis 8) adds secondary indexing + full-text
  • vector to Redis data — right when the canonical hot data already lives in Redis and now
    needs richer-than-key queries. Wrong when Postgres already holds the canonical data and
    you'd mirror it just to search (a dual-write consistency burden — use Postgres FTS, see
    Postgres Full-Text & Fuzzy Search).³
• Cluster: 16384 hash slots; CRC16(key) mod 16384. Multi-key ops (MSET, Lua, MULTI)
  require all keys in one slot or you get CROSSSLOT. Use hash tags — {user:1000}.name
  and {user:1000}.sessions co-locate — but never one giant tag (that collapses to a single
  hot node). Design keys for co-location before you need cluster mode.⁴
• Client-side caching (RESP3 CLIENT TRACKING): the server pushes an invalidation when a
  tracked key changes, enabling a self-invalidating in-process cache (no round trip on hits,
  no TTL-staleness). Best for hot small reference data (flags, config).
• Keyspace notifications ride Pub/Sub → unreliable (offline listener misses events, no
  replay). For a must-happen side effect use Streams (acks + replay), not notifications.
• Redis vs Memcached: Memcached wins only for pure multi-core GET/SET/DEL caching with no
  persistence/structures/scripting (natively multithreaded). Anything richer → Redis.

4. When Redis is the wrong tool

Not the system of record (RDB loses seconds–minutes on crash; MULTI/EXEC has no rollback;
RAM-bounded). No joins/relational integrity. No cross-slot multi-key atomicity in Cluster. Large
blobs belong in object storage with metadata in Postgres. Per-write durability at volume is a
Postgres/durable-log problem.

5. Reach for a specialized store only when the pain is real

Default: don't add a datastore until you have measured, current pain on a specific access
pattern — every new store is another backup/monitor/on-call/consistency surface and a second
source of truth (a bug generator). There's almost always a cheaper intermediate step inside
Postgres/Redis first. Prefer a managed version before self-hosting.⁵

Category	Store	You've outgrown Postgres/Redis when…
Search	Meilisearch/Typesense → Elasticsearch/OpenSearch	top-N relevance ranking at multi-million rows (ts_rank isn't indexable), faceted aggregation, CJK, or zero-tuning typo tolerance. Try ParadeDB pg_search first.
OLAP	ClickHouse / DuckDB / BigQuery / Snowflake	aggregates over billions of rows, sub-second dashboards, or analytics starving OLTP cache — after materialized views → columnar extension → read replica (see Postgres Scale-Out & Replication). DuckDB for embedded/file analytics; ClickHouse for a real-time OLAP service.
Time-series	TimescaleDB / ClickHouse / Prometheus	high-cardinality, high-ingest with retention/downsampling as first-class needs. Timescale to keep it in Postgres; ClickHouse past pure time-series; Prometheus for ops metrics only.
Streaming	Kafka / Redpanda / NATS JetStream	sustained publish>consume backpressure (SKIP-LOCKED queues fall apart), long-term replay, or many independent consumer groups. Redis Streams first if you already run Redis and volume is modest.
Graph	Neo4j	deep traversals (6+ hops), graph algorithms (PageRank/community) as a first-class need, or graph traversal is the app. Below that, WITH RECURSIVE in Postgres is fine — be skeptical.
Vector	Qdrant/Pinecone/Weaviate	tail latency climbs past single-digit-million vectors, or selective metadata pre-filtering matters (pgvector post-filters). pgvector is the default up to tens of millions if you already run Postgres.

The meta-pattern: the "graduate" signal is always a measured production regression (real
p95/ingest-ceiling/cache-miss storm), never hypothetical future scale — and you run the
specialized store as a secondary, always-rederivable-from-Postgres index, never the source
of truth.

See also

• Push Compute to the Datastore — the hub and the ladder; Redis is §5, specialized stores §7.
• Postgres Full-Text & Fuzzy Search — exhaust this before an external search engine.
• Postgres Scale-Out & Replication — the in-Postgres analytics escalation.