Kubernetes troubleshooting
AWS Certificate Manager on EKS with ALB
With the AWS Load Balancer Controller, an Ingress that uses an ALB can terminate TLS with a certificate from AWS Certificate Manager (ACM). The certificate lives in ACM, not in a Kubernetes Secret, so the load balancer serves it and no pod handles the private key. This guide covers the request, the Ingress annotations and the checks that catch most failures.
By Muhammad Soliman, Founder at SeaGit
·
Key takeaways
- The ACM certificate must be ISSUED, in the same region as the load balancer, and cover the hostname in the Ingress.
- Set alb.ingress.kubernetes.io/certificate-arn to the certificate ARN, and listen on HTTPS 443.
- An ACM public certificate cannot be exported, so it cannot be copied into a pod. That is by design.
- The ALB, not cert-manager, serves this certificate. An ingress class of alb means ACM issues the certificate, not Let's Encrypt.
Step 1: request the certificate in the cluster region
Request the certificate in the same region as the EKS cluster. An ALB can use only a certificate from its own region. DNS validation is the simplest option when the zone is in Route 53.
CERT_ARN=$(aws acm request-certificate \
--domain-name app.example.com \
--subject-alternative-names api.example.com \
--validation-method DNS \
--region eu-west-1 \
--query CertificateArn --output text)
echo "$CERT_ARN"
# The CNAME record that proves domain ownership
aws acm describe-certificate --certificate-arn "$CERT_ARN" --region eu-west-1 \
--query 'Certificate.DomainValidationOptions[].ResourceRecord'A wildcard such as *.example.com covers one label, so it does not cover app.internal.example.com. Add each name you serve as a subject alternative name, or request a second certificate.
Step 2: confirm the certificate is ISSUED
An ALB listener cannot use a certificate that is still PENDING_VALIDATION. Check the status before you deploy the Ingress:
aws acm describe-certificate --certificate-arn "$CERT_ARN" --region eu-west-1 \
--query 'Certificate.{Status:Status,Domains:SubjectAlternativeNames,NotAfter:NotAfter}'Step 3: the Ingress with the certificate ARN
The AWS Load Balancer Controller reads the annotations and creates the ALB and its listeners. The ingressClassName selects the controller:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-api
namespace: prod
annotations:
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
alb.ingress.kubernetes.io/ssl-redirect: '443'
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:eu-west-1:111122223333:certificate/REPLACE-ME
spec:
ingressClassName: alb
rules:
- host: app.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: my-api
port:
number: 8080ssl-redirect sends plain HTTP requests to HTTPS. The HTTP listener is still needed for that redirect, which is why listen-ports includes both. Remove HTTP from listen-ports if you want HTTPS only.
Step 4: verify the certificate the client sees
Check the certificate that the ALB serves for the hostname, not the certificate in ACM. The subject, issuer and expiry must match:
openssl s_client -connect app.example.com:443 -servername app.example.com </dev/null 2>/dev/null \
| openssl x509 -noout -subject -issuer -dates
# The ALB address and the Ingress status
kubectl get ingress my-api -n prod -o jsonpath='{.status.loadBalancer.ingress[0].hostname}{"\n"}'Common failures
The ALB is created, but HTTPS shows the wrong certificate
The certificate ARN is missing, misspelled, or from another region. Compare the annotation with the ARN from aws acm list-certificates in the cluster region.
The Ingress has no address or its events show an error
Read the ingress events first. The controller writes the reason there, for example when a certificate is not ISSUED or the ARN cannot be found. Then check the controller logs:
kubectl describe ingress my-api -n prod | sed -n '/Events:/,$p'
kubectl logs -n kube-system deploy/aws-load-balancer-controller --tail=100The controller cannot read the certificate
The controller needs permission to describe ACM certificates. The policy it uses for the install includes acm:DescribeCertificate and acm:ListCertificates. A custom policy without them fails with an access error in the controller log.
The name resolves to the wrong place
The DNS record for app.example.com must point at the ALB, as an alias or a CNAME to its hostname. A record that still points at an old load balancer serves the old certificate, even when the new ALB is correct.
- Certificate is PENDING_VALIDATION: the CNAME for validation is missing or in the wrong zone.
- Certificate region differs from the ALB region: request a new certificate in the cluster region.
- Hostname not in the certificate: add it as a subject alternative name, or request a new certificate.
How SeaGit handles this
On SeaGit, the ingress class you choose decides who issues the certificate. An nginx path uses cert-manager with Let's Encrypt. An ALB path uses AWS ACM, and that path needs the DNS zone to be in Route 53, where the address records are created as aliases. The domains docs lay out the matrix.
A certificate ARN that you pin on an ALB ingress is kept as you set it, and SeaGit does not replace it with a generated one. The certificate itself still lives in your AWS account, so the request and validation steps in this guide still apply.
SeaGit runs on AWS only, so this path applies to AWS clusters.