Argo Workflows on Kubernetes: The Object Model, the Gotchas, and When to Use Airflow Instead

Argo Workflows is a container-native workflow engine where every step in your pipeline is a Kubernetes pod. That single design decision explains everything good and everything bad about it. Steps get real isolation, arbitrary images, and the full resource model — requests, limits, node selectors, GPUs. They also get pod scheduling latency, which is a few seconds you pay per step whether the step takes an hour or a millisecond.

Before anything else: Argo Workflows is not Argo CD. They come from the same project family and are otherwise unrelated. Argo CD reconciles a cluster toward a git repository. Argo Workflows runs DAGs. People conflate them constantly, install one expecting the other, and end up confused about why there is no "sync" button.

They compose well and are deployed independently. You can run Workflows without ever touching Argo CD.

The object model

Everything is a CRD under argoproj.io/v1alpha1:

A Workflow has a spec.entrypoint naming which template to start with, and a spec.templates list containing them. A template here is a unit of work or a unit of orchestration — the word carries both meanings, which trips people up.

The template types worth knowing:

• container — run an image. The workhorse.
• script — run an inline script; the output is captured as a result parameter.
• dag — orchestrate other templates with explicit dependencies.
• steps — orchestrate other templates in ordered phases.
• resource — create or patch an arbitrary Kubernetes object and wait on a condition.
• suspend — pause until resumed, manually or after a duration. This is how you build approval gates.

DAG: the form you should default to

1apiVersion: argoproj.io/v1alpha1
2kind: Workflow
3metadata:
4  generateName: dag-diamond-
5spec:
6  entrypoint: diamond
7  templates:
8  - name: diamond
9    dag:
10      tasks:
11      - name: A
12        template: echo
13        arguments:
14          parameters: [{name: message, value: A}]
15      - name: B
16        depends: "A"
17        template: echo
18        arguments:
19          parameters: [{name: message, value: B}]
20      - name: C
21        depends: "A"
22        template: echo
23        arguments:
24          parameters: [{name: message, value: C}]
25      - name: D
26        depends: "B && C"
27        template: echo
28        arguments:
29          parameters: [{name: message, value: D}]
30
31  - name: echo
32    inputs:
33      parameters:
34      - name: message
35    container:
36      image: busybox
37      command: [echo, "{{inputs.parameters.message}}"]

B and C both depend on A, so they run in parallel once A finishes. D waits for both.

Two notes on depends. It takes a boolean expression — "B && C", and you can also express "B.Succeeded || B.Failed" to run a task regardless of outcome, which is how you build cleanup tasks. An older field, dependencies, takes a plain list and cannot express anything but AND-of-all. Plenty of blog posts and Stack Overflow answers still show dependencies; prefer depends.

generateName rather than name is deliberate — each submission gets a unique suffix, so you can submit the same workflow repeatedly without collisions.

Steps: the syntax that catches everyone

steps is the other orchestration form, and its syntax is genuinely surprising:

1spec:
2  entrypoint: hello-hello-hello
3  templates:
4  - name: hello-hello-hello
5    steps:
6    - - name: hello1
7        template: print-message
8        arguments:
9          parameters: [{name: message, value: "hello1"}]
10    - - name: hello2a
11        template: print-message
12        arguments:
13          parameters: [{name: message, value: "hello2a"}]
14      - name: hello2b
15        template: print-message
16        arguments:
17          parameters: [{name: message, value: "hello2b"}]

Look at the dashes. steps is a list of lists. The outer list runs sequentially; each inner list runs in parallel.

So hello1 completes first. Then hello2a and hello2b run at the same time, because they are two entries in the same inner list.

Getting this wrong is the most common beginner error in Argo Workflows, and the failure is silent — write - name: where you meant - - name: and your pipeline still works, just fully sequential and much slower. Nothing errors.

Use dag unless you specifically want phase semantics. DAG expresses the same graphs more clearly and does not have a whitespace trap.

Passing data between steps

Two mechanisms, and the distinction matters.

Parameters are small strings, passed in the workflow object itself. Fine for a filename, a commit SHA, a count.

Artifacts are files, passed through external storage:

1  - name: artifact-example
2    steps:
3    - - name: generate-artifact
4        template: hello-world-to-file
5    - - name: consume-artifact
6        template: print-message-from-file
7        arguments:
8          artifacts:
9          - name: message
10            from: "{{steps.generate-artifact.outputs.artifacts.hello-art}}"
11
12  - name: hello-world-to-file
13    container:
14      image: busybox
15      command: [sh, -c]
16      args: ["sleep 1; echo hello world | tee /tmp/hello_world.txt"]
17    outputs:
18      artifacts:
19      - name: hello-art
20        path: /tmp/hello_world.txt
21
22  - name: print-message-from-file
23    inputs:
24      artifacts:
25      - name: message
26        path: /tmp/message
27    container:
28      image: busybox
29      command: [sh, -c]
30      args: ["cat /tmp/message"]

Here is the part that is not obvious from the YAML: artifact passing requires a configured artifact repository. Steps are separate pods, potentially on separate nodes, with no shared filesystem. When generate-artifact finishes, Argo tars its output path and uploads it to object storage — S3, GCS, Azure Blob, MinIO — and the consuming pod downloads it before starting.

If you have not configured an artifact repository, artifact passing fails at runtime, not at submission. Every Argo Workflows installation that is going to do anything real needs a bucket wired up first. Budget for the object-storage round trip in your step timings, too: a 5 GB intermediate dataset is an upload and a download, not a filesystem move.

Reuse: WorkflowTemplate and templateRef

Copy-pasting template definitions across workflows gets old immediately. WorkflowTemplate stores them once:

1apiVersion: argoproj.io/v1alpha1
2kind: WorkflowTemplate
3metadata:
4  name: workflow-template-print-message
5spec:
6  entrypoint: print-message
7  templates:
8  - name: print-message
9    inputs:
10      parameters:
11      - name: message
12    container:
13      image: busybox
14      command: [echo]
15      args: ["{{inputs.parameters.message}}"]

And templateRef calls into it from any workflow:

1    steps:
2    - - name: hello1
3        templateRef:
4          name: workflow-template-print-message
5          template: print-message
6        arguments:
7          parameters:
8          - name: message
9            value: "hello1"

templateRef takes both the name of the WorkflowTemplate and the template inside it. This is your library mechanism — put your standard build step, your standard notification step, and your standard cleanup step in ClusterWorkflowTemplate objects, and let teams reference them.

Retries are per template, not global:

    retryStrategy:
      limit: 10

Set this on the templates that touch flaky external systems rather than everywhere. A blanket retry on a non-idempotent step is how you get duplicate writes.

Scheduling with CronWorkflow

1apiVersion: argoproj.io/v1alpha1
2kind: CronWorkflow
3metadata:
4  name: hello-world
5spec:
6  schedules:
7    - "* * * * *"
8  timezone: "America/Los_Angeles"   # Default to local machine timezone
9  startingDeadlineSeconds: 0
10  concurrencyPolicy: "Replace"      # Default to "Allow"
11  successfulJobsHistoryLimit: 4     # Default 3
12  failedJobsHistoryLimit: 4         # Default 1
13  suspend: false                    # Set to "true" to suspend scheduling
14  workflowSpec:
15    entrypoint: hello-world-with-time
16    templates:
17      - name: hello-world-with-time
18        container:
19          image: busybox
20          command: [echo]
21          args: ["🕓 hello world. Scheduled on: {{workflow.scheduledTime}}"]

schedules is a list. Older examples show a singular schedule: field; current versions take schedules: with one or more cron expressions, which lets a single object cover "every weekday at 9" and "Sundays at 3" without duplication. If you copied a manifest from an old tutorial and the workflow never fires, check this first.

concurrencyPolicy defaults to Allow, which will happily start a second run while the first is still going. For anything that writes to shared state, set Forbid or Replace deliberately.

The operational realities

Every step is a pod. For steps that do real work — process a dataset, train a model, run an integration suite — the isolation is worth the few seconds of scheduling. For a pipeline of two hundred tasks that each take 50 ms, you will spend all your time scheduling pods. Argo Workflows is the wrong tool for fine-grained task graphs.

Workflow objects accumulate in etcd. A completed Workflow holds the full status of every node in its status field. Run a thousand workflows with two hundred steps each and you have a serious etcd problem. Configure the workflow garbage collector (ttlStrategy) and enable the workflow archive so history goes to a database instead of the API server. This is the single most common way a healthy Argo Workflows install becomes an unhealthy cluster.

RBAC is per service account, per workflow. The pods your workflow runs use a service account you nominate. Give it the minimum it needs — a workflow that runs untrusted code with a permissive service account is a cluster takeover waiting to happen. See Kubernetes RBAC in practice for how to scope it.

Resource requests matter more than usual. Because every step is a pod, every step is a scheduling decision. Steps without requests land wherever and get throttled unpredictably; see requests, limits and QoS.

Argo Workflows vs Airflow vs Tekton

These get compared constantly and they are optimised for different things.

Choose Airflow when your DAG structure itself needs to be computed — hundreds of tables where the graph is derived from a catalogue — or when your team is data engineers who think in Python and your sources are mostly databases and APIs rather than containers.

Choose Argo Workflows when the steps are containers, you are already on Kubernetes, and you want the pod-level resource model — GPUs for a training step, high memory for one aggregation, spot instances for the rest. This is why it dominates ML pipelines and underpins Kubeflow Pipelines.

Choose Tekton if the only thing you want is CI/CD and you want a Kubernetes-native way to do it. It is narrower than Argo Workflows by design.

An honest note on the CI use case: running your CI on Argo Workflows is entirely viable and some teams are very happy with it, but you are building the ergonomics yourself. GitHub Actions or GitLab CI give you the pull-request integration, the log UI, and the marketplace for free.

When not to use it

• You are not on Kubernetes. It is a set of CRDs and a controller. There is no standalone mode.
• Your tasks are small and numerous. Pod-per-step is a bad trade below roughly a second of real work per step.
• Your team writes Python and thinks in DataFrames. Argo will feel like YAML programming, because it is. That friction is real and it does not go away.
• You only need scheduled jobs. If the requirement is "run this container nightly," a Kubernetes CronJob does it with no new components — see Jobs and CronJobs in production. Reach for CronWorkflow when you need a graph on a schedule.

Frequently asked questions

Is Argo Workflows the same as Argo CD?

No. They share a project family and a naming convention, nothing else. Argo CD continuously reconciles cluster state toward a git repository. Argo Workflows executes container-based DAGs. Installing one gives you nothing of the other, and they are commonly run together and independently.

Should I use dag or steps?

Default to dag. It expresses dependencies explicitly with depends, supports boolean expressions like "B && C", and has no whitespace trap. Use steps only when you genuinely want sequential phases — and remember it is a list of lists, where the outer level is sequential and the inner level is parallel.

Why is artifact passing failing?

Almost always because no artifact repository is configured. Steps run as separate pods with no shared filesystem, so artifacts move through object storage — S3, GCS, Azure Blob, or MinIO. Without a bucket configured, the upload has nowhere to go and the failure surfaces at runtime rather than at submission.

Why did my CronWorkflow never run?

Check whether you wrote schedule: instead of schedules:. Current versions take a list of cron expressions under schedules, and manifests copied from older tutorials use the singular form. Also check suspend is not true, and that startingDeadlineSeconds is not so small that a briefly-busy controller causes runs to be skipped.

How do I stop workflows filling up etcd?

Set a ttlStrategy so completed workflows are deleted, and enable the workflow archive so history is persisted to a database rather than living in the API server. Without both, a busy installation steadily grows the etcd dataset until cluster performance degrades — and it will look like a cluster problem rather than an Argo problem.

Can I use it for CI/CD instead of GitHub Actions or GitLab CI?

Yes, and some teams do — the primitives are all there. What you give up is the integration layer: pull-request status checks, the log viewer, secret management, and a marketplace of ready-made steps. You will rebuild some of that. It is a reasonable choice when your builds are unusual enough that generic CI systems fight you, and a poor one when they are not.

Does it support approval gates?

Yes, via the suspend template type. A suspended workflow pauses until it is resumed — either manually through the UI or CLI, or automatically after a specified duration. Combined with depends, this gives you a manual gate in the middle of a graph without an external system.

Getting started sensibly

Install the controller, configure an artifact repository before you write anything real, then build one three-step DAG that passes a parameter forward and an artifact backward. That exercises the two data mechanisms and the failure mode that catches everyone.

After that, the productive move is to put your common steps into ClusterWorkflowTemplate objects early. Teams that skip that step end up with the same twenty-line container template copy-pasted into forty workflows, and no way to fix a bug in it once.