Kubernetes troubleshooting
HTTPRoute Not Working in Gateway API: Status Checks
An HTTPRoute that does not route traffic is almost always rejected by its status, not by the data plane. Gateway API writes the reason into the route status, so the fastest fix starts with two conditions: Accepted and ResolvedRefs.
By Muhammad Soliman, Founder at SeaGit
·
Key takeaways
- Read status.parents on the HTTPRoute first. Accepted=False means the Gateway refused the route. ResolvedRefs=False means a backend could not be used.
- A Gateway listener allows routes only from its own namespace unless allowedRoutes says otherwise. Routes in another namespace are rejected by default.
- A route hostname must match the listener hostname. A route with no matching hostname is accepted by no listener.
- A backend in another namespace needs a ReferenceGrant in the backend namespace.
Step 1: read the route status
The route status lists one entry per parent Gateway. Each entry has its own Accepted and ResolvedRefs conditions. Read them before you change anything else.
# Confirm the Gateway API CRDs are installed
kubectl get crd | grep gateway.networking.k8s.io
kubectl get httproute my-app -n prod -o jsonpath='{range .status.parents[*]}{.parentRef.name}{" "}{range .conditions[*]}{.type}={.status}({.reason}) {end}{"\n"}{end}'
# Full detail, with the message text
kubectl describe httproute my-app -n prod# Illustrative status, not copied from a cluster
Accepted=False Reason: NotAllowedByListeners
Message: No listeners on the parent Gateway allow this route from namespace "staging".
ResolvedRefs=False Reason: BackendNotFound
Message: Service "my-api-v2" not foundStep 2: check the Gateway itself
A route can be accepted while the Gateway has no address yet. Check the Gateway conditions and the address the controller assigned:
kubectl get gatewayclass
kubectl get gateway -A
kubectl describe gateway public -n gateway-system | sed -n '/Status:/,$p'
# The address that DNS should point to
kubectl get gateway public -n gateway-system -o jsonpath='{.status.addresses[*].value}{"\n"}'If the GatewayClass is not accepted, the controller that owns it is not running. Fix the controller first, because no route under that class will work until it is.
Step 3: fix the common causes
parentRefs points at the wrong Gateway or section
parentRefs must name the Gateway by name and namespace. A sectionName that does not match a listener name leaves the route unattached. A typo here is the most common cause of a route that is silently ignored.
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: my-app
namespace: prod
spec:
parentRefs:
- name: public
namespace: gateway-system
sectionName: https # must match a listener name on the Gateway
hostnames:
- app.example.com
rules:
- backendRefs:
- name: my-app
port: 8080The Gateway does not allow routes from this namespace
allowedRoutes defaults to the Gateway namespace only. A route in prod is rejected with NotAllowedByListeners unless the listener allows it. Allow the namespaces you need, and keep the rule narrow:
listeners:
- name: https
protocol: HTTPS
port: 443
hostname: "*.example.com"
allowedRoutes:
namespaces:
from: Selector
selector:
matchLabels:
gateway-access: "public" # label the namespaces that may attachThe hostname does not match the listener
The route hostnames must overlap the listener hostname. A route for app.example.com cannot attach to a listener limited to api.example.com. The reason in this case is NoMatchingListenerHostname.
The backend service or port is wrong
ResolvedRefs=False with BackendNotFound means the Service name is wrong or the Service is in another namespace. Check the Service and its port. The backendRef port must be a port the Service exposes.
kubectl get svc -n prod
kubectl get endpoints my-app -n prod # empty means no ready pods behind the ServiceThe backend is in another namespace
A backendRef to a Service in another namespace needs a ReferenceGrant in the namespace that owns the Service. Without it the route reports RefNotPermitted:
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
name: allow-prod-routes
namespace: backend
spec:
from:
- group: gateway.networking.k8s.io
kind: HTTPRoute
namespace: prod
to:
- group: ""
kind: ServiceStep 4: test the route before you move DNS
Test the route against the Gateway address with the Host header, so DNS does not need to change first:
GW=$(kubectl get gateway public -n gateway-system -o jsonpath='{.status.addresses[0].value}')
curl -sSI -H "Host: app.example.com" "http://$GW/" | head -5
# HTTPS listener: keep the hostname for SNI, and resolve it to the gateway
# address with --resolve (use the IP of $GW if it is a hostname that resolves)
curl -sSI --resolve "app.example.com:443:$(dig +short "$GW" | head -1)" https://app.example.com/ | head -5- 404 from the Gateway: no route matched. Check hostnames and path matches.
- 503 from the Gateway: the route is attached, but the backend has no ready endpoints.
- Connection refused or timeout: the load balancer or its security group does not accept the port.
How SeaGit handles this
SeaGit can run the Envoy Gateway addon in your cluster in your AWS account, and the docs include a guide for moving from ingress-nginx to Gateway API. The migration tooling in cluster-worker converts existing ingress settings into Gateway API resources as one of its steps. Check the status conditions above on the routes it creates, the same way you would for routes you write by hand.
SeaGit does not configure your listeners' allowedRoutes or your ReferenceGrants for you beyond the migration. If a route is rejected by a rule in your own Gateway, the fix is in that Gateway. The migration guide describes the steps and their order: see the ingress migration guide.