Diff v1 → v2

  # Redis Internals, Probabilistic Structures & Datastore Selection
  
  > **When to use it:** going deeper on Redis (memory blowups, cluster, client-side caching),
  > reaching for probabilistic structures (Bloom/Count-Min/Top-K/t-digest), or deciding whether
  > to add a *specialized* store (search engine, OLAP warehouse, Kafka, graph, vector DB) instead
- > of stretching Postgres/Redis. Companion to [[Push Compute to the Datastore]] (§5 Redis).
+ > of stretching Postgres/Redis. Companion to [[push-compute-to-datastore|Push Compute to the Datastore]] (§5 Redis).
  
  ## 1. Redis memory & encoding
  
  Small collections use a compact **listpack**/`intset` encoding; past a threshold they convert to
  the full pointer structure (`hashtable`/`skiplist`). `OBJECT ENCODING key` shows which.[^mem]
  
  - **Fold "one entity → many keys" into one Hash.** `user:123:name/:email/:age` each pay Redis's
    per-key overhead (~tens of bytes of `dictEntry`+`robj`); `HSET user:123 name … email …` pays
    it once and packs the body densely (3–5× reported saving) — *if* field count stays under
    `hash-max-listpack-entries` (512). Past it, hash ops revert from listpack's O(n) scan to
    hashtable O(1) but memory jumps.
  - **Measure with `MEMORY USAGE key`, not `DBSIZE × avg_value`** (ignores per-key overhead +
    fragmentation, which dominate for many small keys).
  - **Expiration vs eviction are different.** Expiration (TTL): lazy on access + active sampling.
    Eviction (memory pressure, only when `maxmemory` set): pick a policy (`allkeys-lru`/`-lfu`
    for caches). **Set `maxmemory` + a policy** or a cache workload either grows unbounded or hits
    `OOM command not allowed` under `noeviction`.
  
  ## 2. Probabilistic structures — bounded memory, bounded error
  
  As of **Redis 8** these are in core (formerly RedisBloom). Each answers a question your app would
  otherwise answer by pulling the full dataset in and looping — at fixed sub-linear memory. **Pick
  the one whose error mode matches your risk:**[^prob]
  
  | Question | Structure | Error mode |
  |---|---|---|
  | How many distinct? | HyperLogLog (`PFADD`/`PFCOUNT`) | ±% (~0.81%), ~12 KB fixed |
  | Is X in the set? (no delete) | Bloom (`BF.ADD`/`BF.EXISTS`) | false positives only |
  | Is X in the set? (need delete) | Cuckoo (`CF.ADD`/`CF.DEL`) | false positives only |
  | Roughly how often did X happen? (unbounded keys) | Count-Min Sketch (`CMS.INCRBY`/`CMS.QUERY`) | overcounts only |
  | Top-K most frequent? | Top-K (`TOPK.ADD`/`TOPK.LIST`) | transient misrank, converges |
  | p50/p95/p99 of a stream? | t-digest (`TDIGEST.ADD`/`.QUANTILE`) | tunable via `COMPRESSION` |
  
  - **Bloom as a cache-penetration guard**: `BF.EXISTS` before a cache-miss DB lookup — a
    never-seen key skips Postgres entirely (blocks nonexistent-key hammering). ~9.6 bits/item at
    1% error vs a full `SET`'s hundreds of bits/item (~17× smaller). Always confirm a *positive*
    against the real store; trust only negatives.
  - **Cuckoo** when membership is revocable (coupon redeemed → `CF.DEL`); Bloom when append-only.
  - **Count-Min** instead of a per-key Hash of counters when key cardinality is unbounded (URLs,
    IPs) and approximate magnitude suffices — never for billing/quotas (it overcounts).
  
  ## 3. RediSearch / RedisJSON, Cluster, client-side caching
  
  - **RediSearch** (`FT.CREATE`/`FT.SEARCH`, core in Redis 8) adds secondary indexing + full-text
    + vector to Redis data — right when the **canonical hot data already lives in Redis** and now
    needs richer-than-key queries. **Wrong** when Postgres already holds the canonical data and
    you'd mirror it just to search (a dual-write consistency burden — use Postgres FTS, see
-   [[Postgres Full-Text and Fuzzy Search]]).[^search]
+   [[postgres-full-text-search|Postgres Full-Text & Fuzzy Search]]).[^search]
  - **Cluster**: 16384 hash slots; `CRC16(key) mod 16384`. **Multi-key ops (MSET, Lua, MULTI)
    require all keys in one slot** or you get `CROSSSLOT`. Use **hash tags** — `{user:1000}.name`
    and `{user:1000}.sessions` co-locate — but never one giant tag (that collapses to a single
    hot node). Design keys for co-location before you need cluster mode.[^cluster]
  - **Client-side caching** (RESP3 `CLIENT TRACKING`): the server pushes an invalidation when a
    tracked key changes, enabling a self-invalidating *in-process* cache (no round trip on hits,
    no TTL-staleness). Best for hot small reference data (flags, config).
  - **Keyspace notifications** ride Pub/Sub → **unreliable** (offline listener misses events, no
    replay). For a must-happen side effect use **Streams** (acks + replay), not notifications.
  - **Redis vs Memcached**: Memcached wins *only* for pure multi-core GET/SET/DEL caching with no
    persistence/structures/scripting (natively multithreaded). Anything richer → Redis.
  
  ## 4. When Redis is the wrong tool
  
  Not the system of record (RDB loses seconds–minutes on crash; `MULTI/EXEC` has **no rollback**;
  RAM-bounded). No joins/relational integrity. No cross-slot multi-key atomicity in Cluster. Large
  blobs belong in object storage with metadata in Postgres. Per-write durability at volume is a
  Postgres/durable-log problem.
  
  ## 5. Reach for a specialized store only when the pain is real
  
  **Default: don't add a datastore until you have measured, current pain on a specific access
  pattern** — every new store is another backup/monitor/on-call/consistency surface and a second
  source of truth (a bug generator). There's almost always a cheaper intermediate step *inside*
  Postgres/Redis first. Prefer a managed version before self-hosting.[^stores]
  
  | Category | Store | You've outgrown Postgres/Redis when… |
  |---|---|---|
  | **Search** | Meilisearch/Typesense → Elasticsearch/OpenSearch | top-N relevance ranking at multi-million rows (`ts_rank` isn't indexable), faceted aggregation, CJK, or zero-tuning typo tolerance. Try ParadeDB pg_search first. |
- | **OLAP** | ClickHouse / DuckDB / BigQuery / Snowflake | aggregates over billions of rows, sub-second dashboards, or analytics starving OLTP cache — after materialized views → columnar extension → read replica (see [[Postgres Scale-Out, Replication and Data Movement]]). DuckDB for embedded/file analytics; ClickHouse for a real-time OLAP service. |
+ | **OLAP** | ClickHouse / DuckDB / BigQuery / Snowflake | aggregates over billions of rows, sub-second dashboards, or analytics starving OLTP cache — after materialized views → columnar extension → read replica (see [[postgres-scale-out|Postgres Scale-Out & Replication]]). DuckDB for embedded/file analytics; ClickHouse for a real-time OLAP service. |
  | **Time-series** | TimescaleDB / ClickHouse / Prometheus | high-cardinality, high-ingest with retention/downsampling as first-class needs. Timescale to keep it in Postgres; ClickHouse past pure time-series; Prometheus for ops metrics only. |
  | **Streaming** | Kafka / Redpanda / NATS JetStream | sustained publish>consume backpressure (SKIP-LOCKED queues fall apart), long-term replay, or many independent consumer groups. Redis Streams first if you already run Redis and volume is modest. |
  | **Graph** | Neo4j | deep traversals (6+ hops), graph algorithms (PageRank/community) as a first-class need, or graph traversal is *the* app. Below that, `WITH RECURSIVE` in Postgres is fine — **be skeptical**. |
  | **Vector** | Qdrant/Pinecone/Weaviate | tail latency climbs past single-digit-million vectors, or selective metadata pre-filtering matters (pgvector post-filters). pgvector is the default up to tens of millions if you already run Postgres. |
  
  The meta-pattern: the "graduate" signal is always a **measured production regression** (real
  p95/ingest-ceiling/cache-miss storm), never hypothetical future scale — and you run the
  specialized store as a **secondary, always-rederivable-from-Postgres** index, never the source
  of truth.
  
  ## See also
- - [[Push Compute to the Datastore]] — the hub and the ladder; Redis is §5, specialized stores §7.
- - [[Postgres Full-Text and Fuzzy Search]] — exhaust this before an external search engine.
- - [[Postgres Scale-Out, Replication and Data Movement]] — the in-Postgres analytics escalation.
+ - [[push-compute-to-datastore|Push Compute to the Datastore]] — the hub and the ladder; Redis is §5, specialized stores §7.
+ - [[postgres-full-text-search|Postgres Full-Text & Fuzzy Search]] — exhaust this before an external search engine.
+ - [[postgres-scale-out|Postgres Scale-Out & Replication]] — the in-Postgres analytics escalation.
  
  [^mem]: Redis memory optimization / eviction —
      <https://redis.io/docs/latest/operate/oss_and_stack/management/optimization/memory-optimization/>.
      Per-key byte figures are third-party estimates; the mechanism (per-key overhead) is documented.
  [^prob]: Redis probabilistic types — <https://redis.io/docs/latest/develop/data-types/probabilistic/>;
      core as of Redis 8 (<https://redis.io/docs/latest/operate/.../redisos-8.0-release-notes/>).
  [^search]: RediSearch/RedisJSON — <https://github.com/RediSearch/RediSearch>.
  [^cluster]: Redis cluster spec — <https://redis.io/docs/latest/operate/oss_and_stack/reference/cluster-spec/>;
      client-side caching — <https://redis.io/docs/latest/develop/reference/client-side-caching/>.
  [^stores]: Store-selection thresholds synthesize official docs + independent (non-vendor) writeups
      (GitLab's "Elasticsearch as a rederivable secondary index" pattern). Vector/latency numbers are
      2026 third-party benchmarks — directional, not settled.