Employees ask HR the same questions every day, and the answers are in policies nobody reads. This guide builds an assistant that answers from those policies, with a citation for every answer, and then shows how to change it safely. We ran every step on SeaGit on 22 October 2026; the screenshots, answers and scores are from that run.
Three stacks, each from one form: MLflow (records), RAG Stack (the assistant) and Evals (the scores).
Upload policy files in the web UI and ask in plain English. Each answer cites the file it came from.
Ten golden questions, each with the fact its answer must contain, decide whether a change may go live.
In our run, a “friendlier” prompt that avoided amounts scored 0.5 and was blocked; one that kept them scored 1.0 and went live in about half a minute.
What you will build
An HR policy chatbot for a made-up company, Brightwater Labs. It uses RAG, retrieval-augmented generation: when someone asks a question, the assistant first finds the passages of your documents that match, then asks a language model to answer using only those passages. The answer cites the passage, so anyone can check it.
Building it takes about fifteen minutes of forms. The more useful half of this guide is what comes after: changing the assistant without breaking it. Every change runs on a copy next to the live one, gets scored on your questions, and only goes live if it passes.
Words you will see
Stack
A set of apps SeaGit deploys and connects for you from one template, such as the RAG Stack.
Chunk
A piece of a document. Documents are cut into chunks so the assistant can find the right paragraph.
Prompt
The instructions sent to the language model with every question: how to answer, and to use only the passages found.
Golden question
A real question plus the fact its answer must contain. Your definition of “correct”.
Candidate
A copy of the assistant with your change, running next to the live one so it can be scored first.
Gate
The rule that decides whether a candidate may go live. With Block, a candidate that scores too low cannot be adopted.
Trace
A record in MLflow of one question: what was found, the model, the prompt version and the answer.
Before you start
A SeaGit account on a paid plan. The RAG, MLflow and Evals templates are marked Paid plan.
An environment with a started cluster on Amazon EKS in your own AWS account. Models come from Amazon Bedrock, so there are no model keys to manage.
Settings changes and prompt versions are enabled per account. Ask us to turn them on before step 6.
In the SeaGit dashboard, open Applications and then Templates. You will use three cards: MLflow (keeps prompts, traces and scores), RAG Stack (the assistant itself) and Evals (scores answers against your questions). Build them in that order, because each one points at the one before.
The Templates page. Each card has a Deploy button that opens a short form.
Step 2: Create the MLflow stack
Click Deploy on the MLflow card. On Where, name it hr-mlflow and pick your environment; keep Create new Postgres (in this stack). On Configure, choose who can open MLflow in a browser. We picked Use existing Auth Proxy stack because we already run one; New sign-in for this stack works with your own identity provider. Then Review and Deploy.
Configure step of the MLflow form. The sign-in choice decides who can open MLflow in a browser; the other stacks reach it inside the cluster and never sign in.After Deploy, each part of the stack starts on its own. Click Run in background to carry on.
Step 3: Create the RAG stack
Click Deploy on the RAG Stack card and name it hr-assistant. The form has five steps. For a first assistant, the defaults are right everywhere except the last one:
Data sources: leave the list empty and keep A new S3 bucket for this stack. You will upload files in the web UI.
Ingestion: keep chunk size 800 and overlap 100. You can change them later without rebuilding anything by hand.
Answering: keep the web UI on, then set Trace answers in MLflow to Send traces to an MLflow stack in this environment and pick hr-mlflow. Every question then becomes a trace you can read later.
Where: the name, environment and cluster. Single cluster gives one ingest and query endpoint.Data sources: no source needed. Uploaded files go to a new S3 bucket in your account and are indexed when they change.Ingestion: how documents are cut into chunks and indexed. The defaults suit most documents.Answering: the chat model, how many passages each answer uses, the web UI, and tracing to the MLflow stack.The API key and the web UI access key are shown once, here. Copy both before you close the window. (We covered ours.)
In our run, all four parts (vector database, rag-api, web UI and sync) were running about a minute after Deploy. Open the application and its Stack tab: you will see the connection details, a Settings changes card, and Prompt versions with Version 1, the built-in prompt, marked Live.
Step 4: Upload the policies and ask a question
Download the four sample policies for a made-up company, Brightwater Labs: parental-leave.md, travel.md, expenses.md and time-off.md. On the Stack tab, click the link next to Web UI access, sign in with the web UI access key, and drag the files into the box. Each one says Stored — searchable within seconds. Then ask: How many weeks of paid parental leave do I get?
The Stack tab. Prompt versions starts with Version 1, the built-in prompt, kept in the MLflow stack.The real answer: “You get 16 weeks of fully paid parental leave after the birth or adoption of a child. [1]”, citing parental-leave.md.
Your intranet or Slack bot can ask the same thing in code. Use the RAG API address from the Stack tab and the API key:
bash
curl -s https://<your-rag-api-address>/v1/query \
-H "Authorization: Bearer $RAG_API_KEY" \
-H "Content-Type: application/json" \
-d '{"question": "When am I allowed to book business class?"}'
json
{
"answer": "You are allowed to book business class only when a single flight is scheduled to be longer than 6 hours. [1]",
"citations": [{"n": 1, "uri": "s3://<your-bucket>/uploads/travel.md", "source_id": "uploads", "score": 0.4301}],
"trace_id": "tr-0e0c2ee2ca30847ca5ce1960bcf5ec44",
"model": "bedrock:us.amazon.nova-lite-v1:0"
}
The response from our run. trace_id finds the same question in MLflow.
Step 5: Write golden questions and create the Evals stack
Before you change anything, write down what “correct” means. A golden question is a real question plus the fact the answer must contain. Ours are in golden.jsonl, one per line:
json
{"input": "How many weeks of paid parental leave do I get?", "expected": "16 weeks"}
{"input": "Can I expense a home-office chair, and how much can it cost?", "expected": "$300"}
{"input": "When am I allowed to book business class?", "expected": "6 hours"}
{"input": "What is the nightly hotel limit in London?", "expected": "$320"}
...six more lines
Click Deploy on the Evals card and name it hr-evals. On What to evaluate, choose A RAG stack, pick hr-assistant, and pick hr-mlflow for results. On Judge, keep None: code checks only: the check is simply whether each answer contains the expected text. On Suites & gate, choose Write your own (YAML) and paste suite.yaml, set Promotion gate to Block, choose Selected instances with none picked (the gate then covers only this RAG stack’s changes), and set GitHub status to Off.
yaml
suites:
- name: hr-golden
target:
type: seagit-rag
url: '${TARGET_URL}'
api_key_env: TARGET_API_KEY
dataset:
rows:
- {input: "How many weeks of paid parental leave do I get?", expected: "16 weeks"}
- {input: "What is the nightly hotel limit in London?", expected: "$320"}
# ...the other eight questions
scorers:
- contains
- latency_ms
thresholds:
contains: {min: 0.85}
latency_ms: {max: 15000}
contains: {min: 0.85} means at least 85% of answers must contain their expected text.
What to evaluate: the RAG stack answers the questions; every run and score is recorded in the MLflow stack.Suites & gate: your own suite, and Block, so a change that scores below the threshold cannot be adopted.
Step 6: Try a settings change
On the Stack tab, click Change settings. We tried bigger chunks: chunk size 1200 and overlap 200. Both fields say Needs reindex, and the estimate shows the cost before you start (about $0.00 for our four documents). Pick the Evals stack, add golden.jsonl, and click Start change.
Change settings. The new values run on a candidate; nothing changes for users until you adopt.The card walks through Building candidate, Reindexing, Evaluating and Passed. The MLflow run link shows every question and answer.
Both the live stack and the candidate answered all 10 questions with the expected fact (contains 1.0). Bigger chunks did not help these short policies, but they did not hurt either, so we clicked Adopt. That is a normal result: the point is that you know before your users do.
Step 7: A prompt change the gate refuses
HR asks for friendlier answers, and legal adds: do not quote amounts, point people to the policy. It sounds reasonable. Under Prompt versions, click New version from this on Version 1, change the system message, add a note, and Save version. The user message must keep {{context}} and {{question}}.
text
You are a friendly HR helper for employees. Answer warmly in one or two
short sentences, using ONLY the provided context. Do not quote exact amounts,
limits or numbers of days; tell the employee which policy covers it so they
can check the details there. Cite sources by their [n] number. If the context
does not contain the answer, say you don't know.
Version 2, the system message we saved.Then click Try this version. It opens Change settings with Version 2 picked; choose the Evals stack and start.
New prompt version: the system message, the user message with {{context}} and {{question}}, and a note that shows in the list.The evaluation failed: contains 0.5, below 0.85. Clicking Adopt shows “Adoption is blocked”. Adopting anyway needs a reason, which is recorded in the audit log.
The candidate did exactly what the prompt said, and that is the problem. These are real answers from the run:
Question
Must contain
Version 2 answered
How many weeks of paid parental leave do I get?
16 weeks
You get 16 weeks of fully paid parental leave, as per the Parental Leave Policy [1]. (passes)
Can I expense a home-office chair, and how much can it cost?
$300
Yes, you can expense a home-office chair, up to a certain amount. Check the Home Office Equipment section of the Expenses and Home Office Policy for the exact limit. [1] (fails)
What is the nightly hotel limit in London?
$320
The nightly hotel limit in London is covered under the Business Travel Policy [1]. (fails)
What is the daily meal allowance when travelling?
$75
Meals while traveling are covered up to a certain amount per day, as per the Business Travel Policy [1]. (fails)
How big is my yearly learning budget?
1,500
Your yearly learning budget is covered under the Training and Conferences policy [1]. (fails)
How many vacation days do full-time employees get?
25 days
Full-time employees get their vacation days as outlined in the Vacation and Time Off Policy [1]. (fails)
Five of ten answers lost the one thing an employee asked for. If you tried three questions by hand, you could easily have picked the ones that still worked. We clicked Discard; nobody ever saw these answers.
Step 8: A prompt change that passes
Keep the friendly tone, but ask for the amounts. Version 3:
text
You are a friendly HR helper for employees. Answer warmly in one or two
sentences, using ONLY the provided context. Always include the exact amounts,
limits and numbers of days the policy gives, and name the policy. Cite sources
by their [n] number after the sentence they support. If the context does not
contain the answer, say you don't know and cite nothing.
Save it, click Try this version on Version 3, and start the change.
Every version stays in the list with its note, so you can see what was tried and why.Version 3 passed: all 10 answers contained their fact (contains 1.0). Version 3 is marked Candidate until you adopt it.After Adopt, the live assistant answers with Version 3: “The nightly hotel limit in London is up to $320. [1]”
In our run, Version 3 was live about half a minute after Adopt. The RAG API’s /healthz then reports prompt_ref, the version and a fingerprint of its text, and every MLflow trace of a question carries the same value. If someone asks “what did the assistant say last Tuesday, and why?”, you can answer both halves.
Step 9: Roll back
Changed your mind? On the adopted change, click Roll back, type a reason, and confirm. The card records Prompt: version 3 → built-in with your reason. The roll-back window is about 23 hours after you adopt.
After the roll-back, Version 1 is Live again. Version 2 and 3 stay in the list, ready to try again.
What we measured
The score is the share of the 10 golden questions whose answer contained the expected text. The gate needed at least 0.85. Answers took about a second each.
What we tried
Score
Result
Live stack, built-in prompt, chunk size 800
1.0
Baseline
Chunk size 1200, overlap 200
1.0
Passed, adopted
Prompt version 2: friendlier, no amounts
0.5
Failed, adoption blocked, discarded
Prompt version 3: friendlier, keeps amounts
1.0
Passed, adopted, then rolled back
Two lessons carry over to any assistant. A change that sounds harmless can quietly remove the answer people came for. And ten questions you wrote down beforehand catch that in about two minutes, every time, without anyone spot-checking by hand.
Clean up
Delete the stacks in reverse order: hr-evals, then hr-assistant, then hr-mlflow. On each Stack tab, click Delete stack. Tick Also delete my uploaded documents (or Also delete my backups for MLflow) if you want the S3 bucket gone too; it is kept by default so you never lose data by accident.
Frequently asked questions
HR policy chatbot: FAQ
Do I need to know Python or Kubernetes?
No. Every step in this guide is a form in the SeaGit dashboard. The only text you write is the golden questions file and, if you want, a prompt. The curl example is optional.
Can I upload PDFs and Word files?
Yes. The web UI accepts .txt, .md, .pdf, .docx, .html, .csv and .json files. This guide uses Markdown files so you can open them and see exactly what the assistant reads.
What happens if a prompt change makes answers worse?
It never reaches your users. The new prompt runs on a candidate next to the live stack, the Evals stack scores it, and with the gate set to Block, Adopt is refused while the score is below your threshold. Overriding needs a written reason, which goes to the audit log.
How long can I roll back?
After you adopt a change, the card offers Roll back for about 23 hours. Rolling back puts the previous settings and prompt back on the live stack.
Where are the prompt versions stored?
In the MLflow stack’s prompt registry, one named prompt per RAG stack. You can also add a version from a notebook with the MLflow client; it then appears in the panel too.
Is this available on every plan?
The RAG, MLflow and Evals stacks need a paid plan. Settings changes and prompt versions are enabled per account; ask us to turn them on. Stacks run in Amazon EKS clusters in your own AWS account, with Amazon Bedrock models.
