Evaluating RAG with RAGAS

· 4 min read

Building a Retrieval-Augmented Generation (RAG) pipeline is one thing. Knowing if it actually works—without spending hundreds of hours on manual evaluation—is the real challenge.

If you're building AI assistants (like a student support bot for KenteCode AI Academy that handles course details, discounts, and enrollment policies), you can't just rely on vibes to test your system. You need rigorous evaluation.

Enter Ragas (Retrieval Augmented Generation Assessment)—the open-source framework that turns LLM evaluation into a science.

Here is exactly how you structure your datasets and run the evaluations in code for both Single-Turn and Multi-Turn interactions.

1️⃣ Single-Turn Conversations: The Core 4 Metrics

When a user asks a standalone question, Ragas evaluates the quality of both the retrieved context and the generated response using four core metrics:

Context Precision: Did the system pull the most relevant course policy documents to the top ?

Context Recall: Did it catch all the necessary details about the 50% Africa Resident discount ?

Faithfulness: Is the answer strictly grounded in the retrieved documents, or did it hallucinate a new discount ?

Answer Relevancy: Does the response actually answer the user’s question directly, without fluff?

📊 Single-Turn Dataset Example:

{
  "question": "Do you offer a discount for Africa Residents?",
  "contexts": ["Kentecode AI Academy offers a 50% discount for Africa Residents"],
  "answer": "Yes, Kentecode AI Academy provides a 50% discount for Africa Residents",
  "ground_truth": "Yes, there is a 50% discount for Africa Residents"
}

🛠️ How to Test It in Ragas:

For standalone Q&A, Ragas uses a standard Dataset object. It maps your data columns against the core metrics using an LLM (acting as the judge) to compute a score between 0 and 1.

from langchain_openai import OpenAIEmbeddings
from ragas import evaluate
from ragas.llms import llm_factory
from ragas.embeddings import LangchainEmbeddingsWrapper
from ragas.metrics._faithfulness import Faithfulness
from ragas.metrics._context_recall import LLMContextRecall
from ragas.metrics._context_precision import LLMContextPrecisionWithReference
from ragas.metrics._answer_relevance import ResponseRelevancy
from datasets import Dataset
import os
from openai import OpenAI

CHAT_MODEL = "gpt-4o-mini"
EMBEDDING_MODEL = "text-embedding-3-small"
OPENAI_API_KEY = "Your OPENAI API KEY Here"
openai_client = OpenAI(api_key=OPENAI_API_KEY)
evaluator_llm = llm_factory(CHAT_MODEL, client=openai_client, temperature=0)
evaluator_embeddings = LangchainEmbeddingsWrapper(
    OpenAIEmbeddings(model=EMBEDDING_MODEL, api_key=OPENAI_API_KEY)
)

METRICS = [
   Faithfulness(llm=evaluator_llm),
   LLMContextRecall(llm=evaluator_llm),
   LLMContextPrecisionWithReference(llm=evaluator_llm),
   ResponseRelevancy(llm=evaluator_llm, embeddings=evaluator_embeddings),
]
# 1. Load the Kentecode single-turn data
single_turn_data = {
    "question": ["Do you offer a discount for Africa Residents?"],
    "contexts": [["Kentecode AI Academy offers a 50% discount for Africa Residents"]],
    "answer": ["Yes, Kentecode AI Academy offers 50% discount for Africa Residents."],
    "ground_truth": ["Yes, there is a 50% discount for Africa Residents"]
}
dataset = Dataset.from_dict(single_turn_data)

# 2. Run the evaluation
result = evaluate(
    dataset=dataset,
    metrics=METRICS
)
print(result)

2️⃣ Multi-Turn Conversations: Tracking the Flow

Real users don't just ask one question; they have follow-ups. Evaluating conversational AI requires analyzing the context across multiple turns using:

Coherence: Does the assistant maintain a logical, natural flow across the conversation?

Correctness: Throughout the chat history, does the AI remain factually accurate?

Aspect Critic: Need to measure something unique? Ragas lets you define custom criteria (e.g., Conciseness, Professionalism) to ensure your assistant aligns perfectly with KenteCode's brand voice.

📊 Multi-Turn Dataset Example:

{
  "user_input": "What is the refund policy?",
  "response": "You can request a full refund within the first 7 days of the course starting.",
  "reference": "Full refunds are available up to 7 days after the cohort start date.",
  "interaction_history": [
    {
      "user_input": "Hi, I'm interested in the Gen AI course.",
      "response": "Great choice! It's a 6 month program covering LLMs, RAG, Agentic AI and Evaluations."
    }
  ]
}

🛠️ How to Test It in Ragas:

Multi-turn conversations cant be treated as isolated rows. Ragas evaluates them by treating the entire chat history as a continuous thread, passing the interaction history to the metrics.

from ragas.messages import HumanMessage, AIMessage
from ragas.dataset_schema import EvaluationDataset, MultiTurnSample
from ragas.metrics import AspectCritic
from ragas.llms import llm_factory
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_openai import OpenAIEmbeddings
from ragas import evaluate
import os
from openai import OpenAI

CHAT_MODEL = "gpt-4o-mini"
EMBEDDING_MODEL = "text-embedding-3-small"
OPENAI_API_KEY = "Your OPENAI API KEY"
openai_client = OpenAI(api_key=OPENAI_API_KEY)
evaluator_llm = llm_factory(CHAT_MODEL, client=openai_client, temperature=0)
evaluator_embeddings = LangchainEmbeddingsWrapper(
    OpenAIEmbeddings(model=EMBEDDING_MODEL, api_key=OPENAI_API_KEY)
)
# 1. Represent the multi-turn chat history structurally
sample = MultiTurnSample(
    user_input=[
        HumanMessage(content="Hi, I'm interested in the Gen AI course."),
        AIMessage(content="Great choice! It's a 6 month program covering LLMs, RAG, Agentic AI and Evaluations."),
        HumanMessage(content="What is the refund policy?"),
        AIMessage(content="You can request a full refund within the first 7 days of the course starting."),
    ],
    reference="Full refunds are available up to 7 days after the cohort start date.",
)
eval_dataset = EvaluationDataset(samples=[sample])

# 2. Evaluate using multi-turn conversational metrics
coherence = AspectCritic(
    name="conversational_coherence",
    definition="Does the AI's final response logically follow from and stay consistent with the rest of the conversation history?",
    llm=evaluator_llm,
)
correctness = AspectCritic(
    name="conversational_correctness",
    definition="Is the AI's final response factually correct with respect to the reference answer?",
    llm=evaluator_llm,
)

result = evaluate(
    dataset=eval_dataset,
    metrics=[coherence, correctness],
    llm=evaluator_llm,
    embeddings=evaluator_embeddings,
)
print(result)

💡 The Takeaway

Whether you're fine-tuning a customer support bot or building a complex enterprise search engine, you can't improve what you don't measure. By integrating Ragas into your CI/CD pipeline, you gain quantitative visibility into your RAG pipelines strengths and catch regressions before a student ever sees a hallucinated discount policy.

Are you using Ragas in production yet? What custom metrics have you built using Aspect Critic?

Share this post