Migrate Ingress from nginx to Envoy Gateway

Gradually shift traffic from the nginx ingress controller to Kubernetes Gateway API (Envoy Gateway), one step at a time. You control how much traffic moves, and you can roll back at every step until you remove nginx.

Overview

SeaGit supports migrating your cluster's ingress controller from nginx to Envoy Gateway (a production-grade Kubernetes Gateway API implementation). This guide uses the public API to:

Prepare: Install Envoy Gateway, copy your Ingress routes, and run both controllers simultaneously
Test: Send a fraction of traffic to Envoy (10%, 25%, 50%) while nginx serves the rest
Validate: Check error rates, latency, and compatibility
Cutover: Switch DNS and remove nginx after a soak period

AWS/EKS only. This API is available today on AWS. Azure and GCP support is on the roadmap.

Prerequisites

AWS EKS cluster running on SeaGit (on-prem Kubernetes not supported yet)
API access: An API key (sgt_…) or Firebase ID token
Your cluster IDs: Organization ID and cluster ID (visible in the SeaGit dashboard under Clusters)
No blockers: All Ingress annotations must be compatible with Envoy Gateway (see "What is an Ingress blocker?" below)

The Migration Workflow

Goal	Request Body	What Happens	Allowed When
0. Set up the gateway	`{"action":"prepare"}`	Envoy Gateway + its own load balancer installed. All your Ingress routes are copied over. 100% of traffic still on nginx.	Phase: `none`
1. Send 10% to the gateway	`{"action":"set_weight","weight":10}`	nginx forwards 10% of requests to Envoy. DNS stays pointing at nginx (no customer-facing change yet).	Phase: `prepared` or `shifting`
2a. Roll back to 100% nginx	`{"action":"set_weight","weight":0}`	All traffic back on nginx in under a minute. Envoy Gateway stays installed so you can try again.	Phase: `shifting`
2b. Remove the gateway entirely	`{"action":"abort"}`	Envoy Gateway, its routes, and its load balancer are deleted. Phase resets to `none`.	Phase: `prepared` or `shifting`
2c. Undo DNS switch (if needed)	`{"action":"revert_dns"}` then `set_weight` to 0	DNS reverts to the nginx load balancer, then return all traffic to nginx. Useful if you switched DNS and then discovered a problem.	Phase: `soak`
3. Fully switch to Envoy	`{"action":"set_weight","weight":100}` then `{"action":"cutover","soak_seconds":86400}`	Set weight to 100%, then switch DNS to point at Envoy's load balancer. nginx is kept as a fallback during the 1-day soak period.	Weight is 100 AND no blockers
4. Remove nginx	`{"action":"finish"}`	Verifies Envoy serves every route and DNS is switched, then uninstalls nginx and deletes its load balancer. No way back after this.	Phase: `soak`, after soak period

Note: set_weight always needs a weight: 0, 10, 25 or 50, or 100. Any other value is rejected with 400.

Check the Migration Status

Poll the migration status at any time to see the current phase, traffic weight, DNS addresses, and any errors:

GET /org/{orgId}/clusters/{clusterId}/ingress-migration?accid={accountId}

Response fields you can use to monitor progress:

phase: none → prepared → shifting → soak → done
weight: percentage of traffic on Envoy (0–100)
envoy_address: DNS name of Envoy's load balancer (set by you after cutover)
nginx_address: DNS name of nginx's load balancer (customer-facing until switch)
blockers[]: Ingress annotations preventing cutover (e.g. nginx.ingress.kubernetes.io/auth-url)
pending: step still running? Timestamp when it will complete
last_action_error: why the last step was refused (empty when it succeeded)

What is an Ingress blocker?

Some nginx Ingress annotations are not supported by Envoy Gateway. If you have any of these, you must either:

Remove the annotation from your Ingress (if the functionality is not critical)
Use Envoy Gateway's equivalent feature if one exists

The API response includes blockers[] listing which annotations are blocking your migration. Common ones:

nginx.ingress.kubernetes.io/auth-url — external auth
nginx.ingress.kubernetes.io/proxy-body-size — body size limits
nginx.ingress.kubernetes.io/rate-limit — rate limiting
nginx.ingress.kubernetes.io/auth-type — auth type

Check the full GET response to see your specific blockers. You cannot cutover until blockers are resolved.

Deploy a new app during migration

If you deploy a new app (instance) to the cluster while a migration is in progress, it will have an Ingress that is not yet copied to Envoy. To sync it:

POST /org/{orgId}/clusters/{clusterId}/ingress-migration
{"action":"sync"}

This copies any new Ingresses to Envoy Gateway without changing the traffic weight or phase. Run this after each app deployment to keep routes in sync.

Example: Gradual migration with curl

Replace these placeholders with your values:

$ORG_ID: Your organization ID
$CLUSTER_ID: Your cluster ID
$ACCOUNT_ID: Your account ID
$API_KEY: Your API key (sgt_…)

1. Check initial status

curl -H "Authorization: $API_KEY" \
  "https://seagit.com/api/org/$ORG_ID/clusters/$CLUSTER_ID/ingress-migration?accid=$ACCOUNT_ID"

2. Prepare the gateway

curl -X POST -H "Authorization: $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"action":"prepare"}' \
  "https://seagit.com/api/org/$ORG_ID/clusters/$CLUSTER_ID/ingress-migration?accid=$ACCOUNT_ID"

3. Gradually increase traffic to Envoy

# Send 10% to Envoy
curl -X POST -H "Authorization: $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"action":"set_weight","weight":10}' \
  "https://seagit.com/api/org/$ORG_ID/clusters/$CLUSTER_ID/ingress-migration?accid=$ACCOUNT_ID"

# Wait and monitor. If good, increase to 50%
sleep 300
curl -X POST -H "Authorization: $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"action":"set_weight","weight":50}' \
  "https://seagit.com/api/org/$ORG_ID/clusters/$CLUSTER_ID/ingress-migration?accid=$ACCOUNT_ID"

4. Poll until step completes

# Poll every 10 seconds until pending is null
while true; do
  STATUS=$(curl -s -H "Authorization: $API_KEY" \
    "https://seagit.com/api/org/$ORG_ID/clusters/$CLUSTER_ID/ingress-migration?accid=$ACCOUNT_ID")
  PENDING=$(echo "$STATUS" | jq -r '.pending')
  if [ "$PENDING" = "null" ]; then
    echo "Step complete!"
    echo "$STATUS" | jq .
    break
  fi
  echo "Still pending... (will complete at $PENDING)"
  sleep 10
done

Rolling back

Before cutover (phase prepared or shifting): send {"action":"set_weight","weight":0} to put all traffic back on nginx (about a minute). DNS never moved. To also remove Envoy Gateway, send {"action":"abort"}.
After cutover (phase soak): first send {"action":"revert_dns"} so your hostnames point at nginx again, then {"action":"set_weight","weight":0}.
After finish (phase done): nginx is gone and there is no rollback step.

Troubleshooting

`400 Bad Action`

The action, weight, or soak_seconds value is invalid. Allowed actions: prepare, set_weight, cutover, revert_dns, finish, abort, sync. Allowed weights: 0, 10, 25, 50, 100. soak_seconds must be a whole number from 0 to 604800 (7 days).

`409 Step not allowed`

The step is not allowed in the current phase, another step is still running, the cluster is not Started, or this is not an AWS (EKS) cluster. The error message includes which actions are allowed now.

Envoy Gateway pod not starting

prepare fails and GET shows the reason in last_action_error. The most common cause is a cluster without room for two more pods: add capacity to the node group, then send prepare again (it is safe to repeat).

Ingress blockers preventing cutover

Call GET to see which annotations are blocking you. Update your Ingress manifests to remove or replace the unsupported annotation, redeploy the app, then call{"action":"sync"} to refresh Envoy Gateway's routes.

`last_action_error` is not null

The last step was accepted but could not be carried out, for example finish while a hostname still resolves to nginx. Nothing was changed by the failed step. Fix the cause and send the step again.

Next Steps

Explore the full API at /docs/swagger
Read about cluster management and networking
Troubleshooting guide for common issues