CreateContainerConfigError: Causes and How to Fix

Read the event first

Describe the pod and read the Events section at the bottom. The reason appears in the container state, and the message says what the kubelet could not find.

kubectl describe pod my-api-6d8f9 -n prod | sed -n '/Containers:/,/Conditions:/p'

kubectl describe pod my-api-6d8f9 -n prod | sed -n '/Events:/,$p'

# Just the waiting reason and message for each container
kubectl get pod my-api-6d8f9 -n prod \
  -o jsonpath='{range .status.containerStatuses[*]}{.name}{": "}{.state.waiting.reason}{" "}{.state.waiting.message}{"\n"}{end}'

The message decides the cause. Match the text to one of the three sections below. The lines are illustrative; copy the exact text from your own events.

# Secret referenced by env or envFrom does not exist
Error: secret "db-credentials" not found

# ConfigMap referenced by env or envFrom does not exist
Error: configmap "app-settings" not found

# Object exists but the key inside it does not
Error: couldn't find key DB_PASSWORD in Secret prod/db-credentials

# Security context the image cannot satisfy
container has runAsNonRoot and image has non-numeric user (app), cannot verify user is non-root

Cause 1: the Secret or ConfigMap does not exist

A pod reads environment variables from a Secret or ConfigMap with secretKeyRef, configMapKeyRef or envFrom. The reference names an object and a namespace. The object must exist in that namespace before the pod starts. Check the object from the pod namespace:

# Every object the pod names, by kind
kubectl get pod my-api-6d8f9 -n prod -o json \
  | jq -r '.spec.containers[] | (.envFrom[]? | (.secretRef.name // .configMapRef.name // empty)), (.env[]? | (.valueFrom.secretKeyRef.name // .valueFrom.configMapKeyRef.name // empty))' \
  | sort -u

# Check each one exists in the same namespace
kubectl get secret db-credentials -n prod
kubectl get configmap app-settings -n prod

The usual mistakes are a Secret created in the default namespace, a name with a different suffix from the chart release, and a Secret that a GitOps sync has not applied yet. Create the object in the same namespace, or correct the reference.

Cause 2: the object exists but the key does not

When a Secret or ConfigMap exists but a key named in a secretKeyRef or configMapKeyRef is not in its data, the kubelet cannot build the environment and reports CreateContainerConfigError. Compare the keys the pod expects with the keys the object holds. Secret data is base64 in the API, so read the key names, not the values:

# Key names only, values are not printed
kubectl get secret db-credentials -n prod -o jsonpath='{.data}' | jq 'keys'

kubectl get configmap app-settings -n prod -o jsonpath='{.data}' | jq 'keys'

# The key the pod asks for
kubectl get pod my-api-6d8f9 -n prod -o jsonpath='{.spec.containers[0].env[*].valueFrom.secretKeyRef.key}'

Fix it by adding the key or correcting the reference

Add the missing key to the object, or change the reference to the key that exists. If the value is optional for your app, mark that single reference optional: true so the pod starts without it, and make sure the app handles the absence.

env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: db-credentials
        key: password
        optional: false   # the default: the pod fails if the key is missing

Cause 3: the image user conflicts with runAsNonRoot

If the pod sets runAsNonRoot: true and the image sets its user by name rather than by number, the kubelet cannot prove the user is not root and refuses to start the container. The message reads "image has non-numeric user". This is the cause to check when no Secret or ConfigMap is missing.

securityContext:
  runAsNonRoot: true
  runAsUser: 10001   # a numeric UID, so the kubelet can verify it
  runAsGroup: 10001

The kubernetes.io security-context page describes runAsNonRoot and runAsUser. Check the image metadata with docker image inspect or your registry UI before you pick a UID, because the UID must match the files the app writes.

Verify the fix

After you fix the object or the spec, the kubelet retries the container. Watch the pod leave the waiting state and confirm the container is running:

kubectl get pod my-api-6d8f9 -n prod -w

kubectl get pod my-api-6d8f9 -n prod \
  -o jsonpath='{.status.containerStatuses[0].ready}{" "}{.status.containerStatuses[0].state}{"\n"}'

If the pod is managed by a Deployment and the spec did not change, the ReplicaSet may keep the old pod. Delete the stuck pod, or roll the Deployment with a new annotation so a fresh pod starts.

Frequently asked questions