Evals Stack

What the Evals Stack is

The template deploys one service into a Kubernetes cluster in your AWS account. It reads questions from datasets in the stack’s S3 bucket, sends them to the application you chose, and scores each answer with code checks, Ragas metrics and a rubric judge. Each evaluation is an MLflow run in your MLflow stack, with the score per question kept as a table on the run. Your data, prompts and answers stay in your cloud account.

When to use it

• You change prompts, models, chunking or retrieval and want to know whether answers got better or worse before users see them.
• Each pull request or branch gets its own ephemeral deployment, and you want it scored against main before anyone promotes it.
• You want a sample of real traffic scored continuously, and the answers people dislike turned into test cases.

The wizard estimates about 4 minutes. It needs a native (sgt-k8s) cluster and an MLflow stack on the same cluster. The stack runs on AWS.

How it fits together

Evals stack (one service in your cluster, no ingress)
  |
  |  datasets/<suite>.jsonl  (stack bucket, S3 in your account)
  v
target: RAG stack | LLM Gateway stack | an address
  |  answers (+ retrieved passages)
  v
scorers: code checks | Ragas | rubric judge --> judge model via an LLM Gateway stack
  |
  v
MLflow stack: one run per suite and target, scores per row
  |
  +--> promotion gate --> deployment result in SeaGit --> Promote: warn or block
  +--> GitHub commit status  seagit/evals

RAG / LLM Gateway traces in MLflow --> online scoring, review queue --> dataset rows

The service has no ingress. It reaches MLflow, the target and the judge model by their in-cluster addresses, and reads and writes datasets in the stack bucket through an IAM role, without keys.

Deploy it

1. Deploy an MLflow stack and the application to evaluate, for example a RAG stack or an LLM Gateway, on the same cluster.
2. Open Applications → Templates and choose Evals. Pick the environment and cluster.
3. Choose the MLflow stack and what to evaluate. Pick an LLM Gateway stack as the judge if you want Ragas and rubric scores.
4. Choose the promotion gate mode and, for Warn or Block, the instance to watch and a Seagit API key.
5. Deploy, then upload a dataset to datasets/rag-smoke.jsonl (or chat-smoke.jsonl) in the bucket shown under Datasets in the Stack tab.

Wizard fields

Targets

A RAG or LLM Gateway stack must run on the same cluster, because the service calls it by its in-cluster address. Its key is handed to the service as a secret at create; it never appears in the Evals stack’s settings or outputs. Each call times out after 60 seconds; a failed call scores 0 on every scorer for that row and counts as an error.

Datasets

A dataset is a JSON Lines file, one question per line. Keep it in the stack bucket under datasets/, or inline it in a suite.

{"input": "What is the refund window?", "expected": "30 days"}
{"input": "Who approves travel over $5,000?", "expected": "the CFO", "tags": {"area": "finance"}}
{"input": "Summarise the security policy.", "contexts": ["Passwords rotate every 90 days."]}

• input is required. expected is needed by exact_match, contains and ragas.context_recall. contexts is optional reference material for the rubric judge. tags are kept with the results.
• An invalid line fails the run and names the line number. A dataset with no rows fails the run.
• Rows added by online scoring and by accepted reviews are written to datasets/<suite>/from-traces.jsonl and datasets/<suite>/reviewed.jsonl. Point a suite at one of them, or merge them into your dataset.

aws s3 cp rag-smoke.jsonl s3://<stack bucket>/datasets/rag-smoke.jsonl

Scorers

Ragas and the rubric judge call the judge model through the LLM Gateway stack you chose. Without a judge model only code checks run: the built-in suite then leaves out the Ragas and judge scorers. A scorer whose inputs are missing, for example faithfulness on a chat answer with no retrieved passages, is reported as skipped, never as 0. The versions of MLflow, Ragas and the judge model are recorded with every run so scores stay comparable.

Suites

A suite names a target, a dataset, its scorers, and the thresholds the promotion gate checks. With Built-in smoke suite, the stack writes it for you from your choices:

suites:
  - name: rag-smoke                 # chat-smoke for a chat target
    target:
      type: seagit-rag              # openai-chat for a chat target (+ model)
      url: "<the target you chose>"
      api_key_env: TARGET_API_KEY
    dataset:
      path: s3://<stack bucket>/datasets/rag-smoke.jsonl
    scorers:
      - contains
      - latency_ms
      # with a judge model:
      - ragas.faithfulness          # RAG targets only
      - judge:
          rubric: "Is the answer correct and grounded in the retrieved context?"
    thresholds:
      contains: {min: 0.8}
      latency_ms: {max: 10000}
      # with a judge model:
      ragas.faithfulness: {min: 0.7, tolerance: 0.05}
      judge: {min: 0.5, tolerance: 0.1}

With Write your own (YAML) you write one or more suites. The first one is used by the promotion gate and online scoring. Every key:

suites:
  - name: support-bot               # letters, digits, . _ -; unique
    target:
      type: seagit-rag              # seagit-rag | openai-chat | http-json
      url: "${TARGET_URL}"
      api_key_env: TARGET_API_KEY   # optional: the variable holding the key
      # openai-chat: model: "${TARGET_MODEL}"
      # http-json:   request: '{"q": "{{input}}"}'
      #              answer_path: $.answer
      #              contexts_path: $.sources[*].text
    dataset:
      path: s3://${DATASET_BUCKET}/datasets/support-bot.jsonl
      # or inline:  rows: [{input: "What is the refund window?", expected: "30 days"}]
    scorers:
      - contains
      - latency_ms
      - regex: {pattern: "^[A-Z]"}
      - ragas.faithfulness
      - judge:
          rubric: "Is the answer polite, correct and under 100 words?"
          model: nova-pro           # optional; the judge model by default
    thresholds:                     # what the promotion gate checks
      contains: {min: 0.8}
      ragas.faithfulness: {min: 0.7, tolerance: 0.05}
      latency_ms: {max: 5000}
      errors: {max: 0}

• ${TARGET_URL}, ${TARGET_MODEL} (chat targets) and ${DATASET_BUCKET} are filled in with the target, model and stack bucket you chose; the target’s key is in TARGET_API_KEY. A variable that is not set stops the service at start with its name.
• min and max are absolute limits. tolerance is the drop allowed against main; leave it out to compare with the limits only. For a metric with only a max, such as latency, a rise counts as the drop.
• errors and rows can be given thresholds too.

The promotion gate

With Warn or Block, the service checks the instance you chose every minute. When a deployment that is not main reaches started with a commit it has not scored, it scores that deployment against the main deployment:

1. Main first. It reuses the latest run of the suite for main’s commit from the last 24 hours, or scores main now.
2. Then the deployment. Its scores are compared with the thresholds and with main’s scores.
3. Then the result is reported to SeaGit, to MLflow and, with a token, to GitHub.

The overall verdict is the worst one, with the reasons listed. The deployment’s row in Deployments shows it as pass, warn or fail (or running while it is scored), linked to the MLflow run.

Gate modes

The gate reads the latest result for the deployment’s commit; a short commit id on either side matches the full one. When the service starts scoring a deployment it reports it as running, and Block refuses a running evaluation the same way as a failed one, so a deployment cannot be promoted while its score is still being computed. A deployment with no result at all has nothing for the gate to check, and the Promote dialog says so. Deployments without a commit, such as an image or Helm chart deployed directly, are not scored automatically; call POST /v1/gate from your CI to score them. Their result stops counting once the deployment is deployed again.

GitHub commit status

With a repository and a token that may write commit statuses, each result is posted on the commit with the context seagit/evals: pass and warn as success (a warn mentions its first reason), fail as failure, and a gate that could not run as error. The status links to the MLflow run, so you can require seagit/evals in a branch protection rule.

Seagit API key

Create an account API key that may list, read and update deployments. The service uses it to find new deployments of the one instance you chose and to record each result. It is stored encrypted and never shown again. A failed report is retried three times and never changes the verdict. Only an API key can record a result: members see results in Deployments but cannot post them, and each result records the prefix of the key that reported it.

Online scoring and the review queue

Turn on tracing in the RAG or LLM Gateway stack you evaluate (its Trace answers in MLflow or Trace calls in MLflow setting), then choose Live traffic in the Evals stack. The target must trace to the same MLflow stack you chose under Results, because the service reads its traces from that MLflow stack. The Evals stack is refused when the target sends no traces, or sends them to another MLflow stack.

• Online scoring reads new traces on the interval you set and scores the share you chose. The same answer is always in or out of the sample. Only scorers that need no expected answer run, and latency is always measured. Scores are attached to each trace in MLflow, as judge or code feedback. An answer below a min or above a max is added to datasets/<suite>/from-traces.jsonl.
• Review queue. Answers people rate thumbs-down in the RAG web UI are listed by GET /v1/review. Accept one with the answer you expected and it becomes a row in datasets/<suite>/reviewed.jsonl; dismiss it and it leaves the queue.

TOKEN=...   # the token you set on the evals application (see Calling the API)
EVALS=http://<release>-application.<namespace>.svc.cluster.local:8080

curl -s -H "Authorization: Bearer $TOKEN" $EVALS/v1/review
curl -s -X POST -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"expected": "30 days from delivery"}' $EVALS/v1/review/<trace_id>/accept

Results in MLflow

Each run of a suite is an MLflow run in the experiment evals/<suite>. It holds the mean of each scorer, the number of rows and errors, and a results table with the question, answer, retrieved passages, scores and latency of every row. Tags say what the run was: sg.role (baseline for main, candidate for a deployment, adhoc for a run you started), sg.verdict, sg.deployment_id and sg.commit_sha. Compare two runs in the MLflow UI to see which questions changed.

Calling the API

The service listens on port 8080 inside the cluster only. Every /v1 call needs Authorization: Bearer and the service’s API token, which is generated when the stack is created and kept as a secret. To call the API from your own pods or CI, set your own value for the evals application’s secret EVALS_API_TOKEN and redeploy it.

When a run or gate names another address, the target’s key is sent only to the target’s own host, to the deployments the gate watches, and to the hosts you listed in the wizard.

Outputs

Limits

• The MLflow stack and the RAG or LLM Gateway stacks it uses must run on the same cluster as the Evals stack.
• One service and one watched instance per stack. Results are kept by the service for its recent runs; MLflow keeps them all.
• Live traffic needs the evaluated stack to trace to the same MLflow stack as the Evals stack.
• AWS only.

Related

• MLflow Stack
• RAG Stacks and LLM Gateway
• Deployments (ephemeral deployments and Promote to main)
• Private MLflow with single sign-on