Part 1: Define Your Data Structure
1.1 How to Organize Documents
Documents need to be broken into small, retrievable chunks. Each chunk should be standalone and focused.
Document + Chunk Format
One Document = a file (policy, API doc, contract)
One Chunk = a single paragraph or section (256-512 tokens)
Metadata = source, type, date, author
One Chunk = a single paragraph or section (256-512 tokens)
Metadata = source, type, date, author
Example: HR Policy Document
{
"document_id": "hr-policy-2024",
"title": "HR Policy Manual",
"source": "hr-policies/vacation.md",
"type": "policy",
"chunks": [
{
"chunk_id": "hr-policy-2024-chunk-001",
"text": "Employees receive 20 days of paid vacation annually. Unused days can carry over to the next year, up to a maximum of 5 days.",
"section": "Vacation Policy",
"metadata": {
"page": 5,
"updated": "2024-01-15"
}
},
{
"chunk_id": "hr-policy-2024-chunk-002",
"text": "Vacation requests must be submitted 2 weeks in advance...",
"section": "Vacation Policy",
"metadata": {
"page": 5,
"updated": "2024-01-15"
}
}
]
}Tip: Keep chunks between 200-500 characters. Too small = lost context. Too big = harder to retrieve exactly what's needed.
Part 2: Index and Embed Your Data
2.1 What Embedding Does
Embedding converts text into numbers (vectors) so the system can find similar text mathematically.
"Vacation policy allows 20 days" ↓ [0.12, -0.45, 0.78, 0.23, -0.91, ...] (384-dimensional vector)
Similar text has similar vectors. Your vector store finds chunks close to the user's question vector.
2.2 Simple Python Setup
from openai import OpenAI
# Initialize client
client = OpenAI(api_key="sk-...")
# Embed a chunk of text
def embed_text(text):
response = client.embeddings.create(
input=text,
model="text-embedding-3-small"
)
return response.data[0].embedding
# Example
chunk = "Vacation policy allows 20 days annually"
vector = embed_text(chunk)
print(f"Vector (first 5 values): {vector[:5]}")
# Output: Vector (first 5 values): [0.012, -0.045, ...]Pro tip: Use text-embedding-3-small for cost-efficiency, or text-embedding-3-large for higher accuracy.
Part 3: Store and Retrieve
3.1 Simple In-Memory Vector Store
For prototyping, store vectors in Python. For production, use Pinecone, Weaviate, or Milvus.
import json
from math import sqrt
class SimpleVectorStore:
def __init__(self):
self.chunks = []
def add_chunk(self, chunk_id, text, vector, metadata=None):
"""Store a chunk with its vector"""
self.chunks.append({
"chunk_id": chunk_id,
"text": text,
"vector": vector,
"metadata": metadata or {}
})
def cosine_similarity(self, vec1, vec2):
"""Calculate similarity between two vectors"""
dot_product = sum(a * b for a, b in zip(vec1, vec2))
norm1 = sqrt(sum(a ** 2 for a in vec1))
norm2 = sqrt(sum(b ** 2 for b in vec2))
return dot_product / (norm1 * norm2) if norm1 * norm2 != 0 else 0
def search(self, query_vector, top_k=3):
"""Find top K most similar chunks"""
scores = []
for chunk in self.chunks:
similarity = self.cosine_similarity(query_vector, chunk["vector"])
scores.append((chunk, similarity))
# Sort by similarity, return top K
scores.sort(key=lambda x: x[1], reverse=True)
return [chunk for chunk, _ in scores[:top_k]]
# Usage
store = SimpleVectorStore()
# Add chunks
store.add_chunk(
"hr-001",
"Vacation policy allows 20 days annually",
[0.12, -0.45, 0.78, ...], # actual embedding vector
{"source": "hr-policy", "section": "vacation"}
)
# Search for similar chunks
query_vector = embed_text("How much vacation do I get?")
results = store.search(query_vector, top_k=3)
for chunk in results:
print(chunk["text"])Part 4: Retrieve and Augment
4.1 The Retrieval Pipeline
When a user asks a question:
- Embed the question into a vector
- Search the vector store for top 3-5 relevant chunks
- Assemble context from those chunks
- Send to Claude along with the question
4.2 Retrieval Function
def retrieve_context(user_question, vector_store, top_k=3):
"""
Retrieve relevant chunks for a question
"""
# 1. Embed the question
question_vector = embed_text(user_question)
# 2. Search vector store
relevant_chunks = vector_store.search(question_vector, top_k=top_k)
# 3. Assemble context string
context = ""
for i, chunk in enumerate(relevant_chunks, 1):
source = chunk["metadata"].get("source", "unknown")
context += f"[{i}] (Source: {source})\n{chunk['text']}\n\n"
return context, relevant_chunks
# Example
question = "How much vacation do I get?"
context, chunks = retrieve_context(question, store)
print("Retrieved context:")
print(context)
# Output:
# [1] (Source: hr-policy)
# Vacation policy allows 20 days annually...
#
# [2] (Source: hr-policy)
# Unused days can carry over...Part 5: Connect to Claude
5.1 Send Context to Claude
Combine the retrieved context with the user question and send to Claude's API.
from openai import OpenAI
client = OpenAI(api_key="sk-...")
def rag_query(user_question, vector_store):
"""
Main RAG function: retrieve context and query Claude
"""
# Step 1: Retrieve relevant context
context, chunks = retrieve_context(user_question, vector_store)
# Step 2: Build the prompt
system_message = """You are a helpful assistant.
Answer questions using ONLY the provided context.
If the answer isn't in the context, say "I don't have this information."
Always cite which source [1], [2], etc. you used."""
user_message = f"""Context from our documents:
{context}
Question: {user_question}"""
# Step 3: Call Claude
response = client.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=1024,
system=system_message,
messages=[
{"role": "user", "content": user_message}
]
)
return response.content[0].text
# Test it
answer = rag_query("How much vacation do I get?", store)
print(answer)
# Output:
# Based on the HR policy [1], you receive 20 days of paid vacation annually.
# Additionally [2], up to 5 unused days can carry over into the next year.Why This Works
Claude sees the actual policy text [1] [2], so it can answer accurately. It cites sources, making the answer traceable. If your policy changes, you just update the document, no retraining needed.
Part 6: Complete RAG Flow
6.1 End-to-End Example
# SETUP: Load all documents and embed them once
documents = load_documents("policies/") # Load your docs
store = SimpleVectorStore()
for doc in documents:
for chunk in doc["chunks"]:
vector = embed_text(chunk["text"])
store.add_chunk(
chunk["chunk_id"],
chunk["text"],
vector,
chunk["metadata"]
)
print(f"Indexed {len(store.chunks)} chunks")
# RUNTIME: Answer questions using RAG
questions = [
"How much vacation do I get?",
"When do I need to request vacation?",
"Can I carry over unused vacation days?"
]
for q in questions:
print(f"\nQ: {q}")
answer = rag_query(q, store)
print(f"A: {answer}")Documents ↓ (split into chunks) Chunks ↓ (embed each) Vectors + Chunks → Vector Store ↑ User Question ↓ (embed) Query Vector ↓ (search top-k) Retrieved Chunks ↓ (assemble context) Context + Question → Claude API → Answer
Part 7: Best Practices
1
Chunk Size Matters
Too small: Loses context. Too big: Retrieval noise. Sweet spot: 200-500 characters (1-2 paragraphs).
2
Always Add Metadata
Include source, section, date, author. Helps with traceability and filtering.
3
Validate Retrieval
Test that your top-3 results are actually relevant. If not, adjust chunk size or embedding model.
4
Version Your Documents
Track which version of a policy was used. Policies change—you need to know what was current when the answer was given.
5
Use a Real Vector Store for Production
Simple in-memory stores don't scale. Use Pinecone, Weaviate, Qdrant, or Milvus for production.
Quick Start Checklist
| Step | Action | Tool/Tech |
|---|---|---|
| 1. Prepare | Gather documents (policies, docs, manuals) | Any format (PDF, MD, TXT) |
| 2. Split | Break into chunks (200-500 chars each) | LangChain, llama-index, or manual |
| 3. Embed | Convert chunks to vectors | OpenAI Embeddings API |
| 4. Store | Save vectors + metadata | Pinecone, Weaviate, or JSON |
| 5. Retrieve | Find top-k similar chunks for queries | Vector similarity search |
| 6. Augment | Add context to user question | String assembly |
| 7. Query | Send to Claude API | Claude 3.5 Sonnet |
