Scaling LLMs: Implementing Semantic Cache-Aside with Upstash

As LLMs move from experimental prototypes to production-grade features, two challenges consistently emerge: cost and latency. Every call to a high-reasoning model like GPT-4o or Claude 3.5 Sonnet incurs a financial cost and, more importantly, a significant delay in the user experience.

In traditional web development, we solve this with caching. If a user requests a resource that hasn't changed, we serve it from a fast, in-memory store like Redis. However, LLMs present a unique challenge. In natural language, two different strings can mean the exact same thing. "How do I reset my password?" and "What is the process for a password reset?" are semantically identical but would result in a cache miss in a traditional key-value store.

This is where Semantic Cache-Aside comes in. By leveraging vector embeddings and similarity search, we can identify when a new query is 'close enough' to a previously answered one, allowing us to bypass the LLM entirely. In this article, we will explore how to implement this pattern using Upstash Vector and Redis.

The Architecture of a Semantic Cache

In a standard cache-aside pattern, the application checks the cache first. If the data is missing (a cache miss), it fetches it from the primary source and populates the cache for future requests.

In a Semantic Cache-Aside workflow, we add a step involving vector embeddings:

1. User Input: The user submits a natural language query.
2. Embedding Generation: We convert that query into a numerical vector using an embedding model (e.g., OpenAI’s text-embedding-3-small).
3. Vector Search: We query Upstash Vector to find the nearest neighbor to this new vector.
4. Similarity Evaluation: If the distance (similarity score) between the query and the best match is above a specific threshold (e.g., 0.92), we consider it a hit.
5. Cache Retrieval: We use the ID returned by the vector search to fetch the full LLM response from Upstash Redis.
6. LLM Fallback: If no similar vector is found, we call the LLM, store the response in Redis, and index the embedding in Upstash Vector.

Why Upstash Vector and Redis?

While you could technically use a single database for this, a hybrid approach using Upstash Vector and Redis offers the best performance and cost profile.

• Upstash Vector: It is purpose-built for low-latency similarity searches. It handles the heavy lifting of indexing high-dimensional data and performing cosine similarity calculations in milliseconds.
• Upstash Redis: While vector databases can store metadata, Redis is optimized for high-throughput key-value retrieval. Storing the actual (often large) LLM response strings in Redis keeps your vector index lean and fast.
• Serverless Synergy: Both are serverless, meaning you don't manage infrastructure, and you only pay for what you use—critical for applications with fluctuating traffic.

Implementation Deep Dive

Let’s walk through a TypeScript implementation of this pattern.

1. Generating the Embedding

First, we need a way to turn text into vectors. Using the OpenAI API is the industry standard for this.

import OpenAI from 'openai';

const openai = new OpenAI();

async function getEmbedding(text: string): Promise<number[]> {
  const response = await openai.embeddings.create({
    model: "text-embedding-3-small",
    input: text,
  });
  return response.data[0].embedding;
}

2. The Semantic Lookup Logic

Next, we check Upstash Vector. We aren't looking for an exact match; we're looking for proximity.

import { Index } from "@upstash/vector";
import { Redis } from "@upstash/redis";

const vectorIndex = new Index();
const redis = new Redis();

const SIMILARITY_THRESHOLD = 0.90; // Adjust based on your use case

async function getSemanticCache(query: string) {
  const queryVector = await getEmbedding(query);

  // Search for the top match
  const [match] = await vectorIndex.query({
    vector: queryVector,
    topK: 1,
    includeMetadata: true,
  });

  if (match && match.score >= SIMILARITY_THRESHOLD) {
    // We found a semantic match! Fetch the full text from Redis.
    const cachedResponse = await redis.get(`cache:${match.id}`);
    return {
      data: cachedResponse,
      hit: true,
      score: match.score
    };
  }

  return { hit: false };
}

3. Orchestrating the Flow

Now we combine the lookup with the LLM call and the "populate" step.

async function askLLM(query: string) {
  // 1. Check Cache
  const cacheResult = await getSemanticCache(query);
  if (cacheResult.hit) {
    console.log(`Semantic Hit! Score: ${cacheResult.score}`);
    return cacheResult.data;
  }

  // 2. Cache Miss - Call LLM
  console.log("Cache Miss. Calling LLM...");
  const completion = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: query }],
  });
  
  const answer = completion.choices[0].message.content;
  const cacheId = crypto.randomUUID();

  // 3. Update Cache (Parallelized for speed)
  const queryVector = await getEmbedding(query);
  await Promise.all([
    redis.set(`cache:${cacheId}`, answer, { ex: 86400 }), // 24h TTL
    vectorIndex.upsert({
      id: cacheId,
      vector: queryVector,
      metadata: { originalQuery: query }
    })
  ]);

  return answer;
}

The "Goldilocks" Problem: Setting the Threshold

One of the most critical parts of semantic caching is the SIMILARITY_THRESHOLD.

• Too High (e.g., 0.98): You will rarely get cache hits. Even minor rephrasing will result in a cache miss, making the cache nearly useless.
• Too Low (e.g., 0.75): You will get "false hits." The cache might return an answer for a different question that just happens to share some keywords.

For most RAG (Retrieval-Augmented Generation) or Q&A applications, a threshold between 0.88 and 0.94 is usually the sweet spot. However, this is highly dependent on your embedding model. Newer models like text-embedding-3-small have higher dimensionality and better separation, allowing for tighter thresholds.

Advanced Considerations

Hybrid Exact-Semantic Caching

Before running a vector search, it is often cheaper and faster to perform an exact string match in Redis. This covers the case where the user sends the exact same query twice (e.g., clicking a suggested question button).

Cache Invalidation

Unlike a standard database, invalidating a semantic cache is tricky. If your underlying data changes (e.g., your documentation is updated), a cached answer might become stale.

One strategy is to use a short Time-to-Live (TTL) in Redis. When the Redis key expires, the vector search will still find a match, but the redis.get() will return null, signaling the need for a refresh. You can then delete the stale vector or update it with the new LLM response.

Namespacing

If you are building a multi-tenant application, ensure your Upstash Vector queries are namespaced. You don't want User A's private data being served as a cached answer to User B. Upstash Vector supports namespaces natively, allowing you to isolate embeddings per user or organization.

Cost and Performance Analysis

Let’s look at the math.

• Standard LLM Call: ~$0.01 per 1k tokens (GPT-4o) and ~2,000ms to 5,000ms latency.
• Semantic Cache Hit: ~$0.00002 for the embedding + negligible Upstash costs. Latency is typically <100ms for the embedding and <50ms for the vector/Redis lookups.

By implementing this pattern, you aren't just saving pennies; you are transforming the user experience from a "loading spinner" to an "instant response." For high-traffic applications, this can result in thousands of dollars in savings per month and significantly reduced pressure on your LLM rate limits.

Conclusion

Implementing a Semantic Cache-Aside pattern is no longer a luxury—it is a necessity for production-grade AI applications. By using Upstash Vector for similarity search and Upstash Redis for data persistence, you create a robust, serverless infrastructure that handles the nuances of natural language while maintaining the performance of traditional web systems.

Actionable Next Steps:

1. Audit your LLM logs: Identify how many queries are semantically similar.
2. Start small: Implement the hybrid approach (Exact match in Redis first, then Semantic match).
3. Monitor scores: Log the similarity scores of your cache hits to fine-tune your threshold over time.