Private MLflow with single sign-on on your cluster

What is private MLflow on SeaGit?

It is an MLflow 3.x tracking server that SeaGit deploys into a Kubernetes cluster in your own AWS account from a template. Runs and the model registry are stored in Postgres, artifacts in an S3 bucket in your account, and people open the UI through a sign-in page that uses your identity provider (OIDC, GitHub or Google). One shared Auth Proxy can protect MLflow and other stacks in the same environment, with optional group rules per app. Notebooks, training jobs and CI reach MLflow by its in-cluster address and do not sign in. MLflow is Apache-2.0 and a Linux Foundation project, and your data stays in your cloud account.

How to set it up

1. 1

   Deploy an Auth Proxy (optional)

   Pick your identity provider and who may sign in. It gets an address such as https://auth.<your domain>. Skip it if you want MLflow to carry its own sign-in.

2. 2

   Deploy MLflow from the template

   Choose a new Postgres or an existing one, then the sign-in: new for this stack, your Auth Proxy, or none. Set a hostname if you want one. The wizard estimates about 8 minutes.

3. 3

   Log from notebooks and jobs

   Copy the in-cluster tracking URI from the stack’s outputs. It works in a Notebooks-stack notebook, a training job or CI without credentials.

4. 4

   Evaluate and gate

   Each ephemeral deployment is scored against main; the result is logged to MLflow and can warn or block Promote to main.

What you get

• →Experiments, runs, metrics and the model registry in Postgres: created for you with storage and backup retention you choose, or your own Postgres URL.
• →Artifacts in an S3 bucket in your account, served through MLflow so clients need no cloud keys.
• →One sign-in for many apps: attach MLflow and other stacks to one Auth Proxy, with group rules per app. The identity reaches apps as X-Auth-Request-User, -Email and -Groups headers.
• →Optional Redis-backed sessions for instant sign-out, no oversized cookies with many groups, and the refresh token kept server-side. Session lifetime defaults to 8 hours.
• →Machine access: in-cluster services skip the proxy, and with Custom OIDC a bearer JWT from your provider is accepted.
• →MLflow GenAI support: traces, and mlflow.genai.evaluate with code-based scorers.

Evaluation, gating and feedback in one loop

• →An Evals stack in your cluster runs datasets against your RAG or LLM endpoint and scores the answers with Ragas metrics, a rubric judge that calls a model through the LLM Gateway, and code checks. Each evaluation is an MLflow run.
• →Every ephemeral deployment is scored against main. The verdict is shown on the deployment, posted as a GitHub commit status, and warns or blocks Promote to main; overriding a block needs deploy permission and a reason, and is audited.
• →The RAG API and the LLM Gateway send a trace per request to MLflow. A sample of recent traces is scored on a schedule, and failing traces become dataset rows.
• →Thumbs up and down in rag-web are stored on the trace; down-votes go to a review queue, and accepted reviews become dataset rows.
• →Per-experiment permissions from the signed-in identity, retention for runs and traces, and an audit line for every change.

Datasets, scorers, suite files and gate modes are described in the Evals stack docs.

Good to know

A shared Auth Proxy needs the environment’s own domain, because its cookie is set on that domain. The in-cluster address has no sign-in by design, so treat the cluster network as trusted. SeaGit stacks run on AWS. See also the Notebooks stack, LLM Gateway and stack templates.

Private MLflow — FAQ

What is private MLflow on SeaGit?

It is an MLflow 3.x tracking server that SeaGit deploys into a Kubernetes cluster in your own AWS account from a template. Experiments and the model registry are kept in Postgres, artifacts in an S3 bucket in your account, and a sign-in page from your identity provider sits in front of the UI.

Which sign-in providers are supported?

Custom OIDC (Okta, Entra ID, Keycloak, Auth0 and others), GitHub (an organization, optionally one team) and Google (Google Workspace domains are required). Each Auth Proxy stack uses one provider.

Do notebooks and training jobs have to sign in?

No. Services in the same cluster use MLflow’s in-cluster address, which has no sign-in by design. People opening the UI in a browser do sign in.

Does it need my own domain?

A shared Auth Proxy does. It is refused on SeaGit’s shared domain because its cookie would reach other customers’ apps. An MLflow stack with its own sign-in has a host-only cookie.

Does it run outside AWS?

No. SeaGit stacks run on Kubernetes clusters in your own AWS account.