Redis LangCache: A smarter way to reduce LLM cost and latency
One of the biggest challenges in production GenAI applications is that similar questions often trigger completely new LLM calls
For example:
“Can I get a refund for my subscription?”
“Is my monthly plan refundable?”
“Can I cancel and get my money back?”
Traditional caching may treat these as three different requests.
Semantic caching is different.
Redis recently introduced LangCache, a fully managed semantic caching service designed specifically for LLM applications.
#What is Redis LangCache?
LangCache sits between your application and the LLM.
Instead of checking whether the incoming request is exactly the same, it checks whether the new request is semantically similar to something that has already been answered.
If a sufficiently similar response exists:
User Query → LangCache → Cached Response
No new LLM generation is required.
If there is no suitable match:
User Query → LangCache → LLM → Response → Store in Cache
Redis reports up to 90% API cost savings and up to 15× faster responses for cache hits, although real-world savings depend heavily on the application's cache-hit rate and traffic patterns.
#Key Features
• Semantic similarity matching — matches queries based on meaning, not exact wording
• Vector-based search — embeddings are used to find similar previous requests
• Fully managed — no need to build and maintain your own semantic-cache infrastructure
• Embedding model flexibility — use Redis's default embedding models or bring your own
• Similarity thresholds — control how closely queries must match before returning a cached response
• TTL & eviction policies — prevent stale responses from staying indefinitely
• Adaptive controls — optimize precision and recall
• REST API + SDK support — can be integrated with applications using Python, JavaScript and other languages
• LLM-provider agnostic — can sit in front of different LLM providers
• Monitoring — track cache hits, latency and savings through Redis Cloud
#Semantic Cache vs Traditional Cache
- Traditional cache
Exact Query → Exact Match → Return Response
- Semantic cache
New Query → Embedding → Similarity Search → Reuse Response
That's the important shift:
Cache the meaning, not just the string.
#Why this matters for Agentic AI
As AI agents become more complex, they can make multiple LLM calls per user request.
Reducing even a portion of those calls can improve:
- Cost
- Latency
- Scalability
- Compute efficiency
- User experience
But semantic caching also needs careful design.
A similarity threshold that is too low can return an incorrect answer, while a threshold that is too high reduces cache hits. TTL, tenant isolation, data freshness and monitoring therefore become important production considerations.
https://lnkd.in/dT_E_zVx

Redis LangCache: A Smarter Way to Reduce LLM Costs and Latency

One of the biggest challenges in production GenAI applications is that similar questions can trigger repeated LLM calls.

For example:

“Can I get a refund for my subscription?”

“Is my monthly plan refundable?”

“Can I cancel and get my money back?”

Traditional exact-match caching treats these as different requests.

Semantic caching takes a different approach.

Instead of checking whether the incoming request is identical to a previous query, semantic caching looks for previously stored responses to semantically similar prompts.

What is Redis LangCache?

Redis LangCache is a fully managed semantic caching service designed for LLM applications.

It sits between your application and the LLM, helping reuse previously generated responses when an appropriate semantic match exists.

How it works:

User Query → LangCache → Cached Response

If a suitable cached response is found, the application can return it without making another LLM call.

If no suitable match exists:

User Query → LangCache → LLM → Response → Store in Cache

The next similar query may then be served from the cache.

How much can it save?

Redis advertises up to 90% API cost savings and up to 15× faster responses in its official materials.

However, these are not guaranteed results for every application.

Actual savings depend on factors such as:

• Cache-hit rate

• Repetition of user queries

• Output-token costs

• Embedding and storage costs

• Cache configuration

• Response-generation latency

A high cache-hit rate can reduce redundant LLM calls and improve response times.

Key Features

• Semantic similarity matching — Matches prompts based on meaning rather than exact wording.

• Vector-based search — Uses embeddings to identify semantically similar prompts.

• Fully managed service — Provides a managed semantic-cache infrastructure through Redis Cloud.

• Embedding model support — Provides managed embedding options and embedding model selection.

• Similarity thresholds — Control how closely prompts must match before a cached response is returned.

• TTL and eviction controls — Help manage cache lifetime and prevent outdated entries from remaining indefinitely.

• Adaptive controls — Redis provides controls designed to improve cache retrieval precision and recall.

• REST API and SDKs — Supports integration through REST APIs and Python and JavaScript SDKs.

• LLM-provider agnostic — Can be integrated into applications using different LLM providers.

• Monitoring — Redis Cloud provides cache usage and performance monitoring capabilities.

Semantic Cache vs Traditional Cache

Traditional Cache

Exact Query → Exact Match → Return Response

Semantic Cache

New Query → Embedding → Similarity Search → Reuse Response

The important shift is:

Cache the meaning, not just the string.

Why This Matters for Agentic AI

AI agents often make multiple LLM calls during a single user request.

These calls may involve:

• Planning

• Retrieving information

• Generating responses

• Repeated sub-queries

• Multi-step workflows

Caching suitable repeated requests can help improve:

• Cost efficiency

• Response latency

• Scalability

• Compute utilization

• User experience

LangCache can also be used in AI assistants, chatbots, RAG applications, and agent workflows.

Production Considerations

Semantic caching is not simply about finding the most similar question.

A cached response must also be appropriate for the current request.

Important considerations include:

1. Similarity thresholds

A threshold that is too low may return an incorrect response. A threshold that is too high may reduce cache hits.

2. Data freshness

Responses that depend on changing information, such as account balances, subscription eligibility, or current prices, may require fresh data instead of a cached answer.

3. Tenant and user isolation

Cached responses should not expose one user's private information to another user. Applications should use appropriate access controls and response-scoping mechanisms.

4. TTL and invalidation

Cached responses should expire or be invalidated when the underlying information becomes outdated.

5. Monitoring

Track cache-hit rates, latency, cost savings, and incorrect cache matches to evaluate whether semantic caching is helping.

Conclusion

Redis LangCache demonstrates how semantic caching can improve the efficiency of LLM applications.

Instead of generating a new answer for every similar question, applications can reuse suitable previously generated responses.

For AI agents and high-traffic GenAI applications, this can reduce redundant LLM calls, lower costs, and improve response speed.

The key is to balance cache reuse with response correctness.

Semantic caching is not a replacement for good AI system design. It is another useful layer for building efficient and scalable GenAI applications.

Official documentation: https://redis.io/docs/latest/develop/ai/langcache/