How to Evaluate a RAG or LLM App: a Beginner’s Guide to Evals in CI/CD

The whole picture

Five stacks work together, all on your cluster in your AWS account. You do not need all of them on day one — the Evals stack and MLflow are enough to start — but this is where each one fits.

• MLflow is the record: every evaluation run with its questions, answers and scores, and the traces of your RAG stack and LLM Gateway.
• The LLM Gateway puts your approved Amazon Bedrock models behind one OpenAI-compatible address, with a key per app, budgets, and a log and trace of every call. It can also serve the evaluation judge.
• The RAG stack is the app being tested in this guide: it indexes your documents and answers questions about them with citations.
• Notebooks (JupyterHub) is where you look at real questions in the traces, build and clean test datasets, and compare runs.
• The Evals stack asks the questions, scores the answers, compares the candidate with main and reports a verdict.

Concepts in plain words

What an eval is

An eval sends a fixed list of questions to your app and scores each answer. Model output changes from run to run, so instead of checking one exact string, an eval looks at a score across many questions: “9 of 10 answers contained the expected fact, the slowest took 2.1 seconds”.

Golden questions and production questions

Golden questions are ones you write and know the answer to. They live in your repo, are reviewed like code, and change with the code they test. Production questions come from real users through the review queue; they stay in the Evals stack’s bucket and never go to git, because they can contain personal or customer data.

Scorers

• contains — the answer contains the expected text (case and spacing ignored). The workhorse for RAG apps.
• regex, json_valid — the answer matches a pattern, or is valid JSON.
• latency_ms — how long the answer took.
• judge — a model grades each answer against a rubric you write, for example “Is the answer correct and polite? ”
• ragas.faithfulness, ragas.answer_relevancy, ragas.context_precision, ragas.context_recall — the Ragas metrics: is the answer supported by the retrieved passages, does it address the question, were the right passages retrieved.

Thresholds and the floor

A threshold turns a score into pass or fail: contains: {min: 0.8} fails a build where fewer than 80% of answers contain the expected text. The Evals stack’s thresholds are the floor for every app it watches; an app can ask for a stricter bar, never an easier one. A tolerance also fails a build whose score dropped by more than that amount compared with main.

Candidate and main

Every app has one main deployment that serves production and any number of non-main deployments. CI/CD only ever deploys to a non-main one — the candidate. The Evals stack scores the candidate and main on exactly the same questions, so the comparison is fair.

Warn and Block

Warn shows the verdict when you promote and always lets you go ahead. Block refuses to promote a candidate whose evaluation failed, is still running, or never ran; a member who may deploy can override with a written reason, which is recorded in the audit log.

Walk-through: from nothing to a gated release

Each step names what to click and what you should see. If you already have a stack from a step, skip it.

Step 1. Deploy MLflow

In your environment, open Templates, pick MLflow and deploy. When the Stack tab shows it running, open its address: you see the MLflow UI with no experiments yet. Every stack below sends its runs and traces here. Details: MLflow stack.

Step 2. Deploy an LLM Gateway

Pick LLM Gateway, choose the Amazon Bedrock models your organization approves, and turn tracing on with the MLflow stack from step 1. The Stack tab shows the gateway’s address and the model names it serves. Details: LLM Gateway.

Step 3. Deploy a RAG stack through the gateway

Pick RAG Stack. Under models, choose the LLM Gateway from step 2 as the source of chat models, pick one or more of its models, and trace to the same MLflow stack. Add a few documents and ask a question in the web UI; the answer and its trace appear in MLflow. The stack gets its own key on the gateway, limited to the models you picked. Step by step: Deploy a RAG stack in minutes.

Step 4. Open a notebook

Pick Notebooks and sign in. A notebook is the easiest place to read traces, turn them into test questions and compare runs with the MLflow Python client. Details: Notebooks stack.

Step 5. Deploy the Evals stack

Pick Evals and deploy it to the same cluster. The wizard has three steps:

1. What to evaluate — the RAG stack from step 3, and the MLflow stack from step 1 for results.
2. Judge — start with None: code checks only. You can add a judge later (see choosing a judge).
3. Suites & gate — keep the built-in suite, set Promotion gate to Warn, Instances to watch to All code instances of this organization, and GitHub status on, with a GitHub provider or a token that may write commit statuses and read contents.

SeaGit creates an API key for the stack that may only list, read and update deployments of the apps it watches. You never handle it, and it is deleted with the stack.

Step 6. Write your first golden.jsonl

The best first questions are real ones. In a notebook, read the questions people asked the RAG stack from its traces:

import json
import mlflow

# The MLflow address and the RAG stack's experiment are on their Stack tabs.
mlflow.set_tracking_uri("https://<your MLflow stack address>")
traces = mlflow.search_traces(
    experiment_names=["<the RAG stack's experiment>"], max_results=500
)

questions = []
for request in traces["request"]:
    body = json.loads(request) if isinstance(request, str) else request
    q = body.get("question") if isinstance(body, dict) else None
    if q and q not in questions:
        questions.append(q)

# Keep the questions you can answer, and write the fact a correct answer must contain.
rows = [{"input": q, "expected": ""} for q in questions[:50]]
with open("golden.jsonl", "w") as f:
    for r in rows:
        f.write(json.dumps(r, ensure_ascii=False) + "\n")

Open golden.jsonl, keep 20 to 50 questions that matter, and for each write the short fact a correct answer must contain in expected. Remove expected from a row when there is no single right fact — it is still asked and timed. Save it in your app’s repository as .seagit/evals/golden.jsonl:

{"input": "What is our refund window?", "expected": "30 days"}
{"input": "Which regions do we ship to?", "expected": "European Union"}
{"input": "Who approves a discount over 20%?", "expected": "sales director", "tags": {"topic": "pricing"}}

Step 7. Add .seagit/evals/evals.yml

You can stop at golden.jsonl. Add .seagit/evals/evals.yml only when this app needs something the Evals stack does not give every app:

# .seagit/evals/evals.yml — every key is optional
type: seagit-rag              # or openai-chat (default: the Evals stack's suite type)
include_shared: true          # also ask the Evals stack's shared questions
scorers:
  - ragas.faithfulness        # added to the stack's scorers (needs a gateway judge)
thresholds:
  contains: {min: 0.9}        # stricter than the stack's floor of 0.8: used
  ragas.faithfulness: {min: 0.7}

Set include_shared: false to ask only this app’s questions, for example when the stack’s built-in questions do not fit your domain.

Step 8. Push a change and watch the gate

Commit both files and push. Your CI/CD builds the commit and deploys it to a non-main deployment. When that deployment is running, SeaGit asks the Evals stack to evaluate it — there is no schedule to wait for. Shortly after, you should see:

• the deployment’s evaluation go from running to pass, warn or fail;
• a seagit/evals status on the commit in GitHub, first pending, then the result.

Step 9. Read the scores in MLflow

Open the evaluation’s MLflow link. The candidate run shows each question, the answer, and every score, next to the main run on the same questions. The run is tagged with both commits and with sg.dataset_sha256, a fingerprint of the exact question set, so two runs with the same fingerprint are directly comparable. In a notebook, mlflow.search_runs puts the history of a metric into a table.

Step 10. Turn Warn into Block

After a few pushes, when the verdicts match what you would decide yourself, edit the Evals stack and set the gate to Block. From then on, Promote refuses a candidate that failed or was never evaluated. If an evaluation was missed — for example the Evals stack was restarting — the Promote dialog offers Run evaluation.

Step 11. Grow your tests from the review queue

Set Live traffic to Review queue. Answers people rate thumbs-down in the RAG web UI wait for review; when you write the correct answer and accept one, it becomes a new question in the Evals stack’s dataset. Review queue and online scoring also scores a sample of real answers on a schedule. The more real failures you capture, the more the next release is tested against what users actually ask.

Recipes

Evaluate a RAG app

Use contains with an expected fact per question and a floor around 0.8, plus latency_ms. Add ragas.faithfulness with an LLM Gateway judge when you also want to catch answers that are not supported by the retrieved documents.

Evaluate a chat app behind the LLM Gateway

Choose An LLM Gateway stack under What to evaluate, or set type: openai-chat and model in the app’s evals.yml. Without retrieved passages, Ragas faithfulness does not apply; use contains and a judge rubric instead.

Choose a judge: none, Bedrock or Gateway

• None — code checks only. No model calls beyond your app’s own answers. Start here.
• Amazon Bedrock — a rubric judge calls an approved Bedrock model directly: one judge call per question for the candidate, and one for main when main is scored. Ragas metrics are not available with this judge.
• LLM Gateway — the judge goes through your gateway with its own key, so its calls show up in the gateway’s logs and budgets. Required for the Ragas metrics, which make several judge calls per question.

Main is scored once per main commit and reused for up to a day, so most gates only pay for the candidate.

Gate several apps with one Evals stack

One Evals stack per organization is usually enough. With All code instances it watches every app built from code, including apps added later; with Selected instances it watches only the ones you pick. Each app keeps its own questions in its own repository, and each GitHub status goes to that app’s repository.

A stricter bar for one app

Raise the threshold in that app’s evals.yml, for example contains: {min: 0.95}. Other apps keep the stack’s floor.

Reference

evals.yml keys

golden.jsonl rows

One JSON object per line: input (required, the question), expected (the text a correct answer contains), contexts (reference passages, kept with the question) and tags (any labels, kept with the question).

How the questions and rules are merged

• Questions: the stack’s shared questions (unless include_shared: false), then the app’s golden.jsonl, then the app’s production questions if there are any.
• Scorers: the stack’s plus the app’s.
• Thresholds: per metric, the stricter of the stack’s and the app’s.
• Files are read at the candidate’s commit, so a pull request that changes the tests is evaluated with its own tests.

Limits

golden.jsonl: at most 500 rows and 1 MB. evals.yml: at most 16 KB. The repository is read with a 10-second timeout. An invalid or unreadable file makes the verdict fail with the file and the reason.

Troubleshooting

Frequently asked questions

Next

• Evals stack reference — every wizard field and output.
• The MLOps platform — how the stacks fit together.
• LLM Gateway — keys, budgets and call logs.