Understanding Semantic Caching in LLM Systems
Semantic caching stores both the original prompt and its LLM response together with a vector embedding of the prompt. On subsequent requests, the system embeds the new prompt and performs a vector similarity search against stored embeddings. If a sufficiently similar prompt is found, the cached response is returned immediately without calling the LLM provider.
This differs from traditional key-value caching that relies on exact string matches or simple hashes. Two prompts that differ by only a few words or reordering of clauses will miss an exact cache but hit a semantic cache. The technique is especially valuable for applications with repetitive but not identical queries, such as customer support bots, internal knowledge assistants, or analytical dashboards that ask variations of the same business question.
Redis Stack (Redis 7.2 with the Search and Query module) provides both the key-value store for prompt/response pairs and the vector index in a single process, eliminating network hops between cache and vector database. The HNSW (Hierarchical Navigable Small World) index type delivers sub-millisecond queries even with tens of thousands of cached items on modest hardware.
Embedding Model Selection and Dimensionality Tradeoffs
The quality of semantic caching depends entirely on the embedding model. OpenAI's text-embedding-3-small (1536 dimensions, $0.00002 per 1k tokens) offers a good balance for most teams already paying for OpenAI API access. The newer text-embedding-3-large (3072 dimensions) improves retrieval accuracy at roughly double the cost and storage.
For teams avoiding additional vendor dependency, the open-source all-MiniLM-L6-v2 model via sentence-transformers (384 dimensions) runs locally or on CPU-only instances. It is smaller and faster but shows noticeably weaker performance on nuanced technical or domain-specific prompts compared to OpenAI models.
Lower dimensionality reduces memory usage and index build time but can increase false negatives. In practice, 384-dimensional embeddings require roughly 40% less RAM than 1536-dimensional ones for the same number of vectors. When using Redis, each vector is stored as a binary blob; a 1536-float32 vector occupies 6KB before compression. With 50,000 cached items this adds up to roughly 300MB of vector data alone.
Choose the embedding model before creating the Redis index. Changing dimensionality later requires rebuilding the entire index. Many production deployments standardize on text-embedding-3-small because the same model can also power retrieval-augmented generation pipelines, creating reuse across the system.
Creating and Configuring the Redis Vector Index
Redis Stack must be used. The official Docker image redis/redis-stack:7.2.0-v0 includes both the core server and the search module. Once running, create the index with the following commands:
FT.CREATE llm_cache_idx ON HASH PREFIX 1 "cache:"
SCHEMA
prompt TEXT
response TEXT
embedding VECTOR HNSW 6 TYPE FLOAT32 DIM 1536 DISTANCE_METRIC COSINE
The HNSW 6 parameter sets the number of neighbors per node; higher values improve recall at the cost of memory and indexing speed. For most semantic cache workloads, values between 4 and 8 work well. DISTANCE_METRIC COSINE is preferred over L2 because embedding vectors are usually normalized.
After index creation, verify with FT.INFO llm_cache_idx. The output shows number of documents, average bytes per record, and index size. Add an alias for easier future schema changes:
FT.ALIASADD llm_cache llm_cache_idx
TTL management is handled at the hash level using Redis EXPIRE rather than inside the index. A typical TTL for semantic cache entries is 604800 seconds (7 days). Business-specific facts may warrant longer retention; rapidly changing data such as stock prices or news should use shorter TTLs or be excluded from caching entirely.
Python Implementation with FastAPI and redis-py
The following example uses FastAPI, the official redis client (version 5.0+), and OpenAI SDK (1.35+). It implements a dependency that can be added to any route.
import redis
import hashlib
from openai import OpenAI
from fastapi import FastAPI, Depends
from typing import Optional
import json
import numpy as np
client = OpenAI()
r = redis.Redis(host='localhost', port=6379, decode_responses=False)
INDEX_NAME = "llm_cache_idx"
EMBEDDING_DIM = 1536
SIMILARITY_THRESHOLD = 0.12 # cosine distance; lower = stricter
def get_embedding(text: str) -> list[float]:
response = client.embeddings.create(
input=text, model="text-embedding-3-small"
)
return response.data[0].embedding
def semantic_cache_dependency(prompt: str, temperature: float = 0.0):
# Normalize prompt for basic deduplication
normalized = prompt.strip().lower()
emb = get_embedding(normalized)
emb_bytes = np.array(emb, dtype=np.float32).tobytes()
# KNN search - return top 5 closest
query = (
f"*=>[KNN 5 @embedding $vec_param AS vector_score]"
)
results = r.ft(INDEX_NAME).search(
query,
query_params={"vec_param": emb_bytes},
limit=5,
sort_by="vector_score",
return_fields=["prompt", "response", "vector_score"]
)
for doc in results.docs:
score = float(doc.vector_score)
if score < SIMILARITY_THRESHOLD:
return {
"hit": True,
"response": json.loads(doc.response),
"source": "semantic_cache"
}
# Cache miss - call LLM
llm_response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
temperature=temperature
)
answer = llm_response.choices[0].message.content
# Store in cache
cache_key = f"cache:{hashlib.md5(normalized.encode()).hexdigest()}"
r.hset(cache_key, mapping={
"prompt": prompt,
"response": json.dumps(answer),
"embedding": emb_bytes
})
r.expire(cache_key, 604800) # 7 days
return {
"hit": False,
"response": answer,
"source": "llm"
}
app = FastAPI()
@app.post("/ask")
async def ask_question(prompt: str, result = Depends(semantic_cache_dependency)):
return {"answer": result["response"], "cached": result["hit"]}
The code first attempts a vector search using the KNN syntax introduced in RediSearch 2.4. The SIMILARITY_THRESHOLD of 0.12 on cosine distance (Redis returns distance, not similarity) equates to roughly 0.88 cosine similarity. This value must be tuned per domain. The embedding step is performed even on cache hits because the cost is usually less than 1% of a full LLM completion.
Tuning Similarity Thresholds and Measuring Effectiveness
Threshold selection creates a precision-recall tradeoff. A cosine distance threshold of 0.08 delivers high precision (few incorrect cache hits) but lower hit rate. A threshold of 0.20 increases cache hits at the risk of returning slightly inaccurate answers for ambiguous queries.
Practical tuning involves creating a validation set of 200-500 prompt pairs that a human reviewer has labeled as "semantically equivalent" or "different enough to require fresh generation." Run the cache lookup against this set while varying the threshold and record precision and recall. Most teams settle between 0.10 and 0.15 for OpenAI embeddings.
Redis provides the FT.SEARCH RETURN option to retrieve the raw vector score, making A/B testing straightforward. Log both hit rate and the distribution of scores in production. A sudden drop in average hit distance can indicate embedding model drift or changing query patterns.
Hybrid search combining vector similarity with text filters is possible. For multi-tenant applications, add a tenant_id TEXT field to the schema and filter with @tenant_id:{tenant123} in every query. This prevents cross-tenant cache leakage while maintaining a single index.
Cache Invalidation and Freshness Strategies
Semantic caching introduces the problem of knowing when a cached response has become stale. Unlike exact caching where the key can be deleted when source data changes, semantic entries are harder to target because multiple prompts map to the same conceptual question.
Common strategies include:
- Short TTLs (24-48 hours) for domains with frequent factual updates
- Namespace keys by content version or dataset hash (e.g. prepend "docs_v20241015:")
- Manual invalidation via pattern scan and delete when source documents change
- Using a secondary exact-match cache for known volatile queries
For RAG systems, store the list of source document IDs alongside the response. On cache lookup, verify that none of the source documents have been updated since the cache entry was written. This check adds minimal latency when document metadata is also stored in Redis.
Production Monitoring and Observability
Track these Redis metrics specifically for the semantic cache index:
- Cache hit rate (application level, not Redis)
- Average vector search latency (exposed via
FT.INFOor slow log) - Index memory usage via
FT.INFOmemoryfield - Number of documents in the index
The Redis exporter for Prometheus surfaces redis_command_calls_total{command="FT.SEARCH"}. Combine this with OpenAI token counters to calculate exact dollar savings. A mature implementation typically logs the embedding model version, threshold used, and distance score for every cache lookup to enable retrospective analysis when quality issues appear.
Concurrency safety requires care. The example above executes the embedding and LLM call outside any lock. In high-throughput services, use a deduplication key (SHA-256 of normalized prompt) with SETNX to ensure only one instance performs the expensive LLM call while others wait or retry. Redis 7.2's LATENCY command helps identify when vector queries themselves become slow due to index fragmentation.
Limitations and Scenarios Where Semantic Caching Underperforms
Semantic caching works poorly for prompts containing current timestamps, unique identifiers, or rapidly changing external data. It also struggles with prompts that are intentionally adversarial or contain jailbreak attempts, as similarity to a previously cached safe prompt could return an inappropriate cached response.
Extremely long prompts (above 4000 tokens) increase embedding cost and reduce the likelihood of finding close matches. In these cases many teams fall back to hashing the first 500 characters plus a metadata signature.
Legal and compliance considerations matter when caching responses that contain personal data or copyrighted material. Because the cache persists both prompt and completion, it must be included in data deletion workflows and subject to the same retention policies as the primary application database.
When prompt volume is low or the embedding model produces poor domain separation, the operational complexity of maintaining a vector index may outweigh the benefits. Start with a simple exact-match cache (e.g. Redis hash of prompt + model + temperature) and add semantic caching only after measuring the exact-match hit rate and confirming it is below 15-20%.
Despite these limitations, semantic caching has become a standard optimization layer in production LLM platforms because it directly attacks the two largest costs: token spend and user-perceived latency. Teams that implement it early and instrument it well can maintain high cache hit rates even as their user base and query diversity grow.