RAG (Retrieval-Augmented Generation) — Customer Support That Knows Your Docs

You've seen those AI customer-service chatbots. They're terrible. "Hi, how do I integrate Stripe?" → "Here are some general payment integration tips." Wrong product, wrong API, wrong everything. A vanilla LLM has no context — it's trained on public data and generalizes wildly. RAG (Retrieval-Augmented Generation) fixes this. Instead of asking the LLM to generate an answer from memory, you retrieve the relevant docs first, then ask Claude to answer based on what you gave it. The LLM becomes a search-results formatter, not a hallucinator. Ingest your help docs, FAQs, closed tickets, API references, and changelog — chunk them into semantic blocks, embed with pgvector, retrieve top-k matches for each question, feed to Claude. Claude answers with your own knowledge, not guesses. Support tickets drop 40%. Response time halves. Customers stop getting nonsense. Here's how to ship RAG for your SaaS in a weekend.

Why Naive Chatbots Fail

A vanilla LLM has a knowledge cutoff — it knows data up to a training date, nothing after. Ask it about your product's latest feature? Blank stare. Ask it an edge case in your API? Hallucination. The model confidently invents an answer that sounds plausible but is 100% wrong. Your customer tries it, wastes two hours, posts a bad review.

This is called "hallucination." LLMs are pattern-matching engines — they predict the next token based on patterns in training data. When they don't know something, they don't say "I don't know"; they generate something that feels right. For customer support, that's a disaster.

RAG removes the hallucination. You don't ask Claude "How do I use the new webhook feature?" You ask it "Based on these docs, how do I use the webhook feature?" You hand Claude a document that says exactly how. Claude's job is now retrieval + synthesis, not generation from memory. It can't hallucinate if the answer is in the context you gave it.

RAG Architecture in 30 Seconds

Five steps:

1. Ingest: Load help docs, FAQs, tickets, API references.
2. Chunk: Split into 300–500 word blocks.
3. Embed: Generate vectors with OpenAI or Claude.
4. Store: Save in pgvector with metadata (source, date, category).
5. Retrieve: When customer asks, find top-5 similar chunks.
6. Generate: Pass chunks + question to Claude.
7. Stream: Return Claude's response to the customer.

User question
       ↓
   Embed query
       ↓
 Search pgvector (top-5 chunks)
       ↓
 Pass to Claude with context
       ↓
 Stream response to user

That's RAG. Retrieval-augmented generation. You augment the LLM's generation with actual retrieval. Simple.

Step 1: Ingest Your Knowledge

Start with what you have:

- Help docs (Zendesk, Intercom, custom)
- API reference (generated from your OpenAPI spec)
- Closed support tickets (400+ words each, labeled with solution)
- Feature changelogs
- Pricing + billing FAQs
- Setup guides, tutorials, troubleshooting

Export as markdown or plain text. Structure: one file per topic. Total target: 100–500 docs for a mature SaaS. More is fine; you're just using vector search, not reading them sequentially.

Load into Supabase as a documents table:

create table documents (
  id uuid primary key default gen_random_uuid(),
  source_type text, -- 'faq', 'api_ref', 'ticket', 'guide'
  source_name text, -- 'password-reset-faq', 'stripe-webhook-guide'
  content text not null,
  embedding vector(1536),
  created_at timestamp default now()
);

Step 2: Chunk Strategically

Don't embed entire docs — split them into 300–500 word chunks. Why? An LLM context window is finite. If you stuff a 5,000-word guide into the prompt, you're burning tokens. Chunks let you retrieve only the relevant sections.

const chunk = (text, chunkSize = 400) => {
  const words = text.split(/\s+/);
  const chunks = [];
  for (let i = 0; i < words.length; i += chunkSize) {
    chunks.push(words.slice(i, i + chunkSize).join(' '));
  }
  return chunks;
};

// Insert chunks into Supabase
const chunks = chunk(apiReference);
for (const chunk of chunks) {
  await supabase.from('documents').insert({
    source_type: 'api_ref',
    source_name: 'stripe-webhooks',
    content: chunk,
  });
}

For structured data (API endpoints), chunk by logical block: one chunk per endpoint. For prose (guides), chunk by paragraph count or word count. The goal: each chunk is self-contained and answerable.

Step 3: Embed and Store

Generate vectors using OpenAI or Claude embeddings, store in pgvector (see pgvector + Supabase for semantic search for setup). Then batch embed your documents:

import { embed } from 'ai';
import { openai } from '@ai-sdk/openai';

const chunks = await supabase.from('documents').select('id, content');

for (const chunk of chunks) {
  const { embedding } = await embed({
    model: openai.embedding('text-embedding-3-small'),
    value: chunk.content,
  });

  await supabase
    .from('documents')
    .update({ embedding })
    .eq('id', chunk.id);
}

// Cost: ~$0.02 per 1M tokens.
// 500 docs × 400 words = ~200k tokens = $0.004

Batch this nightly or on doc upload. Don't block user requests with embedding.

Step 4: Retrieve on Query

When a customer asks a question, embed the query and find the top-5 most similar chunks:

// Customer types: "How do I handle failed webhooks?"
const query = "How do I handle failed webhooks?";

const { embedding } = await embed({
  model: openai.embedding('text-embedding-3-small'),
  value: query,
});

// Search pgvector
const { data: chunks } = await supabase.rpc('search_by_embedding', {
  query_embedding: embedding,
  match_threshold: 0.6,
  match_count: 5,
});

// chunks now = [
//   { content: "To handle failed webhooks...", source_name: "stripe-webhooks" },
//   { content: "Retry logic with exponential backoff...", ... },
//   ...
// ]

Each chunk has metadata (source, type). Use this to cite sources in your response.

Step 5: Generate with Claude + Context

Pass the retrieved chunks to Claude as system context:

const systemPrompt = `You are a helpful customer support agent.
Answer based on the provided docs. If the docs don't cover it, say so.
Always cite the source (e.g., "According to the Webhooks guide...").`;

const docsContext = chunks
  .map(c => `[${c.source_name}]\n${c.content}`)
  .join('\n\n');

const response = await anthropic.messages.create({
  model: 'claude-3-5-sonnet-20241022',
  max_tokens: 1024,
  system: `${systemPrompt}\n\nKnowledge base:\n${docsContext}`,
  messages: [
    { role: 'user', content: query },
  ],
});

// Stream to frontend
return response.content[0].text;

Claude now has context. It can synthesize an answer from your docs. No hallucination — if the docs don't cover it, Claude says so.

Six FAQs

How much does RAG cost?

Embedding: ~$0.004 for 500 docs (one-time). Query: ~$0.002 per user question (embedding the query). Claude call: ~$0.002–0.01 per response (size-dependent). Total: ~$1 per 100 customer questions. Compared to hiring support staff or SLA-backed SaaS support (Intercom + Zendesk = $500+/mo), RAG is 10–100x cheaper.

What if my docs change?

Re-embed the changed chunks. Add a webhook: when a doc updates, delete old chunks, insert new chunks, embed them. Costs are incremental. For a mature product updating 5 docs/week, that's ~$0.04/week in embeddings.

Can I use Claude instead of OpenAI embeddings?

Claude doesn't have a dedicated embedding API yet. Stick with OpenAI's text-embedding-3-small or text-embedding-3-large. For local embeddings (no API cost), use Ollama's nomic-embed-text. Quality is lower but cost is zero.

How do I prevent the chatbot from making up sources?

Instruct Claude to cite only sources you provided. In the system prompt: "Only cite sources that appear in the knowledge base. Do not invent sources." This is 90% effective. For 100% safety, post-process Claude's response and strip unverified citations.

Should I use RAG or fine-tune the model?

RAG is faster, cheaper, and more maintainable. Fine-tuning locks your knowledge into model weights — updating takes hours and retraining. RAG lets you update docs instantly. Fine-tune only if you need consistent style or tone across all responses. For knowledge retrieval, RAG wins.

Can RAG handle edge cases and complex scenarios?

Yes, if your closed tickets include them. Ticket-based RAG is powerful: "Customer asked X, we solved with Y" becomes part of the knowledge base. Complex edge cases + solutions compound. Over months, your chatbot gets smarter as your ticket backlog grows.

The Bottom Line

RAG turns your docs into a 24/7 support agent. Ingest help docs, FAQs, API refs, and closed tickets. Chunk into 300–500 word blocks. Embed with pgvector. Retrieve top-5 matches per question. Pass to Claude. Stream the response. Support load drops 40%. Customers get contextual answers. Hallucinations vanish. Cost is ~$1 per 100 questions. Start small: pick your top 10 FAQs, embed them this week, see how many support tickets drop next week. Scale to your full knowledge base once the pattern works. Ready to ship RAG? Let's build your knowledge base infrastructure or check out pgvector for semantic search to get hands-on.