Why Your LLM App Keeps Answering the Same Question Twice — And How Semantic Caching Fixes It

Redis built a cache that cuts LLM costs by 90%!
(and it also leads to 15x faster generation)
Production LLM apps do not receive completely new questions every time.
A customer-support assistant might receive all three of these:
- "Can I get a refund after buying the monthly plan?"
- "Is the monthly subscription refundable?"
- "Can I cancel the plan and get my money back?"
The wording is different, but the underlying question and its answer remain the same.
Yet LLM apps process every version as a new request. They assemble the prompt, send it to the model, and generate an answer that may have already been generated.
Prefix caching reduces part of these repeated calls.
When requests begin with the same system prompt or context, the model can reuse the KV states already computed for that shared prefix.
But the request still hits the LLM. The new tokens must be processed, and the complete answer must still be decoded.
So even with a prefix-cache hit, there's another generation call involved.
To solve this, instead of only caching computation inside the model, the application can cache the generated response outside it.
When another question arrives, the system embeds it and compares it with previously answered questions. If it finds a sufficiently close match, it returns the stored response without invoking the LLM again.
A cache hit removes the input tokens, output tokens, and decoding time associated with another LLM call.
In practice, it is important to decide which questions can safely share an answer since a production setup needs well-tuned similarity thresholds, expiration policies, data isolation, and monitoring for incorrect matches.
If you want to use this in practice, Redis already implements it as a managed service called Redis LangCache.
Under the hood, it generates embeddings, searches previous responses, and returns a matching answer before another model call occurs.
Redis also handles access scopes, custom filtering, TTL and eviction controls, and cache monitoring through Redis Cloud.
I built an interface to compare it against direct LLM inference. The video below shows this in action, and I worked with Redis on this post to put this together.
For the paraphrased question in my run, direct inference took 2.232 seconds and consumed 514 input tokens plus 250 output tokens.
Redis returned the earlier response in 0.37 seconds with zero LLM input or output tokens. That was roughly 6x faster in this run.
Redis reports API cost savings of up to 90% and cache-hit responses up to 15x faster. The actual result depends on how much safe repetition exists in the workload.
You can try Redis LangCache here: https://fandf.co/4ikTNO8
____
Share this with your network if you found this insightful ♻️
Find me → Avi Chawla. Every day, I share tutorials and insights on DS, ML, LLMs, and RAGs.

The core technical claims check out:

• Prefix caching vs. semantic caching distinction — accurate. Prefix/KV caching reuses computed states for shared prompt prefixes but still runs a full decode; semantic caching skips the LLM call entirely on a match.
• Redis LangCache is real — it's a Redis-managed semantic caching service (launched April 2025, still marked "preview" in official docs), confirmed via Redis's own site and docs.
• "Up to 90% cost cut" and "15x faster" — these are Redis's own published marketing figures, not independent benchmarks. Other sources report a wide range (e.g., AWS testing reported cost reductions up to 86% while maintaining 91% answer accuracy, noting results depend heavily on dataset and threshold tuning, and one third-party writeup cites 70–90% token cost cuts). So "90%" and "15x" are real vendor-stated ceilings, not guaranteed or independently verified averages — the post does hedge this correctly by saying "actual result depends on workload." Semantic caching-as-a-service with Redis LangCache - Redis +2
• The author's own demo numbers (2.232s / 514 input / 250 output tokens vs. 0.37s / 0 tokens, "~6x faster") — unverifiable, self-reported from his own test, but consistent in direction with vendor claims.
• Disclosure — the author explicitly states this was done in partnership with Redis, which is good practice and should be weighted into how "independent" the piece is.

Nothing here is false; the main issue is that it reads like sponsored content presenting vendor-supplied ceiling numbers as general expectations.