DevOpsJuly 17, 202612 views0 comments

Building CI on Argo Workflows: Kaniko Builds, Exit Handlers, and Reliable Status Reporting

By SeaGit

#argo#ci-cd#kaniko#kubernetes#gitops

Running CI inside the Kubernetes cluster that will eventually run the workload sounds circular until you try it. The build executes next to the registry and the deployment target, secrets stay inside the cluster boundary instead of being exported to a SaaS runner, and scaling build capacity is the same autoscaling you already operate. Argo Workflows is the engine that makes this practical: each pipeline run is a Workflow CRD, each step is a pod, and the whole thing is declarative YAML you can template and version.

The event path matters as much as the build path

A push lands on a webhook receiver, which validates the signature and hands the event to Argo Events; a sensor matches the repository and branch and instantiates a Workflow from a WorkflowTemplate with the commit SHA, repo URL, and image tag as parameters. Keeping the templates in-cluster (applied from a catalog at addon install time) instead of inlined in the sensor means pipeline changes are a template upgrade, not a sensor redeploy — and the sensor stays a thin trigger with no business logic to break.

Why Kaniko for the image build

For the image build step, Kaniko earns its place by not needing a Docker daemon. It runs as an ordinary unprivileged pod, executes the Dockerfile, and pushes directly to the registry. Three settings pay for themselves immediately:

Registry-backed layer caching (--cache=true with a cache repo) turns repeat builds of a warm Dockerfile from minutes into seconds.
A dedicated, short-lived push credential mounted as a projected secret keeps the blast radius small.
Explicit resource requests on the build pod stop one heavy npm install from starving the neighbours — builds are bursty, and the scheduler can only protect you if you tell it the truth.

The cache-scope bug that caused an outage-shaped afternoon

One caching lesson cost us a real outage-shaped afternoon: never let two different pipelines share one cache scope. We had CI (test-and-build) and CD (build-and-deploy) workflows writing to a single shared cache key namespace, and under concurrent runs the second workflow would fail with failed to reserve cache — a lock collision that looks like an infrastructure flake and is actually a design bug. The fix was boring and permanent: prefix every cache scope with the workflow kind (ci- and cd-) and treat cache reservation errors as ignorable, because a cache miss is a slowdown, not a failure.

Exit handlers: report every terminal state

The part most home-grown CI systems get wrong is status reporting, because they report success at the end of the happy path and report nothing when the workflow dies halfway. Argo's exit handlers fix the shape of the problem: an onExit template is guaranteed to run whether the workflow succeeded, failed, or got killed, and it receives the final phase. Ours does two things — posts the commit status back to GitHub (with a details link pointing at the Argo UI for that exact workflow run) and calls an acknowledgement endpoint so the deployment queue knows the build finished and can proceed or mark the deployment errored. If your exit handler only reports failures you will eventually ship a green build nobody recorded; report every terminal state.

Security boundaries that survived contact with reality

Treat the webhook receiver as the only component that trusts the outside world — it verifies HMAC signatures before anything else runs.
Maintain an explicit allow-list of repository owners whose workflow customizations are honoured; a fork's .argo/ci.yml must never execute with the trusted pipeline's credentials.
Give repos with no CI config a sane default pipeline (checkout, build, notify) generated server-side, so the zero-config path is also the safe path.

When in-cluster CI is the wrong tool

Where this approach stops being the right tool: massive monorepo test fan-out with thousands of parallel shards, or builds needing exotic hardware. For the common case — container images built from application repos, deployed to the same platform — in-cluster Argo CI is less moving machinery than any external CI wired back into the cluster through service accounts and long-lived tokens.

On SeaGit this pipeline is the Argo CI add-on: enable it and pushes build with Kaniko in your own cluster, report status to GitHub, and hand off to the deployment queue — no external runners, no exported secrets.

Frequently asked questions

How do I build container images in Kubernetes without Docker?

Use Kaniko. It runs as an unprivileged pod, executes your Dockerfile, and pushes to the registry with no Docker daemon required. Enable registry-backed caching and set resource requests so bursty builds don't starve other pods.

What causes "failed to reserve cache" in Kaniko/Argo?

Two workflows sharing one cache scope collide on the reservation lock under concurrency. Give each workflow kind its own cache prefix (ci-, cd-) and treat cache-reservation errors as ignorable — a cache miss only slows a build, it doesn't fail it.

How do I make CI status always report back to GitHub?

Use an Argo onExit handler. It runs on success, failure, or kill and receives the final phase, so you can post the commit status and acknowledge the deployment for every terminal state — not just the happy path.

Is running CI inside the cluster secure with pull requests from forks?

Only with an allow-list. The webhook receiver verifies HMAC signatures, and a fork's .argo/ci.yml must never run with the trusted pipeline's credentials. Repos without config get a safe server-side default pipeline.