Demo: an Evaluated RAG Release, End to End

A small, real RAG service — it answers questions about SeaGit’s own documentation — deployed from GitHub with MLflow and an Evals stack. A change that looks like a harmless performance tweak passes every unit test, and the promotion gate stops it anyway. Follow along with the seagits/rag-demo repository.

You push a commit to an app that is built from code.
Your CI/CD (SeaGit’s own build, Argo, or an external pipeline) builds it and deploys it to a non-main deployment — never to main.
The deployment turns started: it is running and answering.
The platform asks every Evals stack that watches the app to evaluate that deployment. Nothing polls.
The Evals stack asks the same questions of the candidate and of the main deployment: the stack’s shared questions, the app’s own .seagit/evals/golden.jsonl from that commit, and production questions if there are any. An optional judge (Amazon Bedrock or your LLM Gateway) grades the answers.
Every run, score and verdict is recorded in MLflow.
The verdict appears on the deployment in SeaGit and, if you turn it on, as the seagit/evals status on the commit in GitHub.
When you promote the deployment to main, Warn shows the result; Block refuses a candidate that failed or was never evaluated (a member who may deploy can override with a reason, which is audited).

How the pieces connect

From	To	How
GitHub (rag-demo)	Your cluster	SeaGit clones the commit and builds the image with kaniko in your cluster, pushes it to ECR, and starts a non-main deployment.
SeaGit	Evals stack	When a watched deployment turns started, SeaGit asks the Evals stack to evaluate it, inside the cluster. Nothing polls.
Evals stack	GitHub	Reads .seagit/evals at the candidate’s commit, and posts the seagit/evals status on that commit.
Evals stack	Candidate and main	Asks both the same questions at their in-cluster addresses (POST /v1/query).
Evals stack	MLflow stack	Records each run: every question, answer and score, tagged with the commit and the exact question set.
Evals stack	SeaGit API	Reads deployments and writes the verdict with its own API key, which may only act on the instances it watches.
rag-demo (optional)	LLM Gateway stack	Set LLM_BASE_URL and LLM_MODEL to answer with a model; without them the app answers with the retrieved passages, at no cost.

What is in the repository

A FastAPI service: section-aware chunks of the docs, BM25 retrieval, and POST /v1/query returning the answer and the passages it used. Next to the code, its tests for answer quality:

.seagit/evals/golden.jsonl — 26 questions with the expected answer, for example:

{"input": "Which vector database does the RAG stack use?", "expected": "Qdrant", "tags": {"topic": "rag"}}
{"input": "Which Postgres version does the LLM Gateway database run?", "expected": "Postgres 17", "tags": {"topic": "gateway"}}
{"input": "What path does the Auth Proxy redirect URL end with?", "expected": "/oauth2/callback", "tags": {"topic": "auth"}}

.seagit/evals/evals.yml — which scorers run and a threshold stricter than the stack’s floor:

# .seagit/evals/evals.yml
type: seagit-rag          # the app speaks the Seagit RAG API
include_shared: true      # also ask the Evals stack's shared questions
scorers:
  - contains              # the expected text appears in the answer
  - latency_ms
thresholds:
  contains: {min: 0.85}   # stricter than the stack's floor of 0.8: used
  latency_ms: {max: 2000}

Run it

Prepare an environment

Step 1. Use an environment with a domain and a native (sgt-k8s) cluster. Everything below runs on that one cluster, in one space.

Deploy MLflow

Step 2. Create an MLflow stack on the cluster. Every evaluation run, score and verdict is recorded there.

Deploy the app from GitHub and mark it main

Step 3. Create an application from the rag-demo repository (port 8080, health check /healthz, not exposed). Deploy the main branch, then mark that deployment as main: it is the baseline every change is scored against.

Add the Evals stack with a Block gate

Step 4. Create an Evals stack on the same cluster. Evaluate a Seagit RAG API at the main deployment’s in-cluster address, record results in the MLflow stack, set the promotion gate to Block, watch only this app, and post the result as a GitHub status.

The stack’s own questions are the organization-wide baseline every watched app also answers. rag-demo needs no API key, so the suite names none:

# Evals stack → Suites → Write your own (the organization's baseline)
suites:
  - name: docs-qa
    target:
      type: seagit-rag
      url: '${TARGET_URL}'
    dataset:
      rows:
        - {input: "Which vector database holds the RAG stack index?", expected: "Qdrant"}
        - {input: "What does the Evals stack record each evaluation in?", expected: "MLflow"}
    scorers: [contains, latency_ms]
    thresholds:
      contains: {min: 0.8}
      latency_ms: {max: 10000}

Push a change

Step 5. Deploy a new commit. SeaGit builds it in your cluster and starts it as a non-main deployment; main is never overwritten.

Read the verdict

Step 6. Seconds after the candidate starts, the Evals stack scores it and main on the same questions. The verdict appears on the deployment, as the seagit/evals check on the commit, and as a run in MLflow.

Fix it and promote

Step 7. Promote refuses the failing candidate. Push a fix; when its verdict passes, promote it to main.

What happened

Measured on a SeaGit cluster. Each candidate and main were asked the same 31 questions: the stack’s 5 shared ones and the app’s 26.

Commit	Retrieval	contains	Verdict	Promote
main Ask the SeaGit docs: an evaluated RAG service	800-char chunks, overlap 150, top 3	0.935 (29/31)	baseline	—
perf perf: halve the prompt context (400-char chunks, top 2)	400-char chunks, overlap 50, top 2	0.806 (25/31)	fail	Refused: “Promotion blocked by evaluation”
fix fix: 500-char chunks, keep three passages	500-char chunks, overlap 100, top 3	0.935 (29/31)	pass	Promoted to main

The failing candidate’s check on GitHub read docs-qa: failed - contains: 0.8065 < min 0.85. The shared questions alone passed all three commits: the regression was only visible in the app’s own golden set.

FAQ

Why did unit tests not catch the regression?

The perf commit only changed retrieval settings. Every unit test still passed; the answers simply lost the facts users ask for. Only scoring real questions shows that, which is what the gate does.

Why does the app need its own golden questions?

In this run the organization’s five shared questions passed for all three commits. The regression only showed in the app’s own 26 questions in .seagit/evals/golden.jsonl, which travel with the code and are read at the candidate’s commit.

Does the demo need an LLM?

No. Without a model the app answers with the passages it retrieved, and the gate uses code checks (contains, latency). Point LLM_BASE_URL at an LLM Gateway stack to answer with a model, and add a judge to the Evals stack for Ragas and rubric scores.

What happens when a deployment is never evaluated?

Under Block, Promote refuses it until it is evaluated. The Promote dialog has a Run evaluation button that sends the same request again.

How to evaluate a RAG or LLM app — the concepts behind this demo, step by step.
Evals stack reference — every wizard field and the gate rules.
MLflow stack and LLM Gateway.