MLflow Stack

What the MLflow Stack is

The template deploys MLflow from a single form. Experiments, runs, metrics and the model registry are kept in Postgres. Artifacts go to an S3 bucket in your account and are served through MLflow (--serve-artifacts), so clients need no S3 credentials; the server reaches the bucket through an IAM role, not keys. Your data does not leave your cloud account.

When to use it

• A team needs shared experiment tracking and a model registry that it hosts itself. MLflow is Apache-2.0 and a Linux Foundation project.
• You evaluate LLM or RAG applications: MLflow GenAI features such as traces and mlflow.genai.evaluate with code-based scorers are supported, and the Evals stack scores answers with Ragas metrics, a rubric judge and code checks.
• Your data scientists already use the Notebooks Stack and want to log from there.

The wizard estimates about 8 minutes. It needs a native (sgt-k8s) cluster, the EBS CSI add-on if you create a new database, and an environment with a domain.

How it fits together

Browser
  |  https://mlflow.<domain>
  v
ingress-nginx  --- asks --->  Auth Proxy (auth.<domain>)
  |                                |
  |                                +--> your identity provider (sign-in)
  |                                |       |
  |                                +<------+  cookie on .<domain>
  v
MLflow UI

Notebooks / training jobs / CI
  |  in-cluster tracking URI
  v
MLflow ---> Postgres (metadata)
       `--> S3 bucket in your account (artifacts)

Evals stack (service in the cluster)
  |  datasets -> calls the RAG API / app under test
  |  -> Ragas metrics, rubric judge through the LLM Gateway, code checks
  v
results logged to MLflow --> commit status / promotion gate

RAG API, LLM Gateway --- traces ---> MLflow <--- online scoring, feedback

People reach the UI through the sign-in page. Notebooks, training jobs and CI use the in-cluster address and do not sign in. Evaluation runs and their results are described in Evaluation and the promotion gate.

Deploy it

1. Open Applications → Templates and choose MLflow.
2. First step: name, environment and cluster, and the Database choice, new or existing.
3. Choose the sign-in method. Set a hostname if you want a name you control. With a sign-in, choose the access control; set a retention period if old runs should be deleted.
4. Deploy. The Stack tab then shows the tracking URIs and a Python snippet to copy.

Wizard fields

Sign-in

Attaching to a shared Auth Proxy

Deploy an Auth Proxy stack in the same environment first. In the MLflow wizard, pick Use existing Auth Proxy stack (it is preselected when one exists) and choose the stack. Optionally restrict MLflow to groups, for example only ml-team. A person who has signed in to any attached app is signed in to MLflow too, because the cookie is set on the environment domain. A shared Auth Proxy needs the environment’s own domain; it is refused on SeaGit’s shared domain.

Access control

Per experiment needs a sign-in. MLflow reads the person’s email and groups from the sign-in page (New sign-in or a shared Auth Proxy). With No sign-in the option is not offered, and the API refuses a stack that asks for it.

Grant access from the MLflow UI or API, as with any MLflow server that has permissions turned on. Admin rights follow the lists: a person removed from Admin emails or from an admin group stops being an admin at their next request.

Notebooks and jobs that use the in-cluster address do not sign in. Under Per experiment they all act as the same service identity, so runs from them are not attributed to a person.

Who gets which permission. Under Per experiment, people who sign in through the Auth Proxy get their own permissions. In-cluster callers that send no identity header get the in-cluster permission. Anything that can reach MLflow on the cluster network can send identity headers itself and act as any user, so keep untrusted workloads away from it, for example in a separate namespace or behind network rules. The MLflow ingress must never enable underscore headers (enable-underscores-in-headers), because identity headers would then be spoofable from outside.

Retention

Delete runs and traces after (days) sets a retention period. 0, the default, keeps everything. With a period set, a daily pass inside the MLflow server deletes runs that ended, and traces created, more than that many days ago, then removes the deleted runs’ artifacts from the bucket.

To keep something past the period, tag it:

• A run tagged sg.retain=true is kept.
• An experiment tagged sg.retain=true keeps all its runs and traces. Traces are kept by the experiment tag only.

import mlflow

mlflow.set_experiment_tag("sg.retain", "true")   # the whole experiment
with mlflow.start_run():
    mlflow.set_tag("sg.retain", "true")          # one run

Database backups are separate: Keep database backups for (days) controls how long the nightly Postgres backups stay in the bucket.

Audit log

Every request that changes something (POST, PUT, PATCH, DELETE) is written to the MLflow container’s standard output as one JSON line, in both access-control modes:

{"audit": true, "ts": "...", "user": "ana@acme.io", "method": "POST", "path": "/api/2.0/mlflow/runs/create", "status": 200}

Ship the container logs with your usual log pipeline and filter on "audit": true. Requests without a signed-in identity are logged with the service identity. Gate overrides are recorded separately in SeaGit’s own audit log, on every route to main: Promote, a Push onto the main deployment (from a deployment, or an instance’s own commit or settings) and Mark as main.

Outputs

Log from a notebook or job

Copy Tracking URI (in-cluster, no sign-in) from the Stack tab. The Connect panel shows this snippet with the address filled in.

import mlflow

mlflow.set_tracking_uri("http://<release>-application.<namespace>.svc.cluster.local:5000")
mlflow.set_experiment("my-experiment")

with mlflow.start_run():
    mlflow.log_param("model", "baseline")
    mlflow.log_metric("accuracy", 0.91)

Open the public Tracking URI in a browser, sign in, and the run is listed under the experiment. Artifacts you log are uploaded through MLflow to the bucket; no AWS keys are needed in the notebook.

Evaluation and the promotion gate

MLflow is where evaluation results live, and SeaGit connects the rest of the loop to it:

• Evals stack. A service in your cluster runs datasets against your RAG or LLM endpoint and scores the answers with Ragas metrics (faithfulness, answer relevancy, context precision and recall), a rubric judge that calls a model through the LLM Gateway, and code checks (exact match, contains, regex, valid JSON, latency). Each evaluation is an MLflow run.
• Promotion gate. Every ephemeral deployment is scored against main when it starts. The verdict (pass, warn or fail) shows on the deployment, is posted as a GitHub commit status, and, depending on the gate mode, warns or blocks Promote to main. A member with deploy permission can override a block with a reason, which is audited.
• Tracing and online scoring. The RAG API and the LLM Gateway send a trace per request to MLflow when you link them to an MLflow stack. The Evals stack scores a sample of recent traces on a schedule; traces that fail become dataset rows. LLM Gateway traces record prompts and responses as sent, without redaction; the RAG API redacts its own traces.
• Human feedback. Thumbs up and down in rag-web are stored on the trace. Down-votes go to a review queue; accepted reviews become dataset rows.

The Evals Stack reference covers datasets, scorers, the suite file, the gate modes and the review queue. You can also score in your own scripts: the server supports mlflow.genai.evaluate with code-based scorers.

Limits

• The in-cluster tracking address has no sign-in, by design. Anything running in the cluster can use it. Under Per experiment, callers that send no identity header get the in-cluster permission, and a caller on the cluster network can send identity headers itself and act as any user. Keep untrusted workloads away from MLflow.
• Linked RAG, LLM Gateway and Notebooks stacks use the in-cluster address, so they must run on the same cluster as MLflow.
• With No sign-in, anyone with the link can read and change experiments.
• A shared Auth Proxy needs the environment’s own domain.
• AWS only. One MLflow server per stack, on one cluster.

Related

• Auth Proxy Stack
• Evals Stack
• Notebooks Stack, LLM Gateway and RAG Stacks
• Private MLflow with single sign-on