Airflow Interview Questions 2026: 16 Answers That Prove You've Run Pipelines in Production

A DAG (Directed Acyclic Graph) is Airflow's unit of a pipeline: a set of tasks plus the dependencies between them, defined in Python, with no cycles allowed. Cron gives you time-based triggers and nothing else. A DAG adds dependency ordering (task B waits for task A), per-task retries, backfill over historical dates, data-interval awareness, parallelism controls, and a UI that shows exactly which run of which task failed and why. The acyclic constraint matters because the scheduler must resolve a topological order — a cycle would mean no valid execution order exists. A good answer also notes that DAG files are parsed repeatedly by the scheduler, so top-level code must be cheap: no API calls or database queries at module level, or you slow down scheduling for every DAG in the deployment.

An operator is a template for one unit of work — BashOperator runs a shell command, PythonOperator calls a function. When you instantiate an operator inside a DAG it becomes a task. A sensor is a special operator whose only job is to wait for a condition to become true — a file landing in S3, a partition appearing in a table, another DAG's task finishing (ExternalTaskSensor) — and only then let downstream tasks proceed. A hook is a reusable interface to an external system (Postgres, S3, an HTTP API) that manages the connection and credentials through Airflow Connections; hooks are used inside operators and sensors and are never scheduled on their own. A clean mental model to state in the interview: hooks talk to systems, operators do work, sensors wait for the world to be ready.

XCom ("cross-communication") is Airflow's mechanism for passing small pieces of data between tasks. A task pushes a value — explicitly with xcom_push or implicitly by returning one — and a downstream task pulls it with xcom_pull. With the TaskFlow API this becomes invisible: return a value from one @task function, pass it as an argument to another, and Airflow wires the XCom for you. Values are serialized and stored in the metadata database, keyed by DAG id, task id, and run. That storage location is the key constraint: XCom is for metadata — a row count, a filename, an S3 key — not for datasets. Passing a DataFrame through XCom bloats the metadata database and pressures the scheduler; pass a path or table name and let the downstream task fetch the data itself.

Every task can declare retries (attempts after the first failure), retry_delay (wait between attempts), and optionally retry_exponential_backoff with max_retry_delay so waits grow instead of hammering a struggling service. These are usually set in default_args at the DAG level and overridden per task. When an attempt fails, the task moves to up_for_retry until attempts run out, then becomes failed, and that failure propagates downstream according to each task's trigger rule. Two related settings interviewers like: execution_timeout, which kills an attempt that runs too long — a retry cannot fire until the hung attempt actually ends — and callbacks such as on_retry_callback and on_failure_callback for alerting. The catch worth stating: retries only help if the task is idempotent. Retrying a non-idempotent append can double-write data, which turns a transient failure into silent corruption.

Five core pieces. The scheduler parses DAG files, creates DAG runs when their schedule is due, and hands ready tasks to the executor. The executor decides where task work runs — local subprocesses, Celery workers, or Kubernetes pods. Workers execute the task code (with LocalExecutor the scheduler's machine plays this role). The webserver serves the UI for monitoring, triggering, and clearing tasks. The metadata database — usually Postgres or MySQL — stores all state: DAG runs, task instances, XComs, connections, variables. Since Airflow 2.2 there is also the triggerer, a process running the async event loop for deferrable operators, letting thousands of waiting tasks park without occupying worker slots. Knowing which component owns which failure mode — UI up but nothing running usually means a scheduler problem — is what interviewers actually probe.

Both create runs for past data intervals; the difference is who initiates. With catchup=True, the scheduler automatically creates a run for every interval between start_date and now that has not run yet — deploy a daily DAG with a start_date six months back and roughly 180 runs queue instantly. Backfill is explicit and on-demand: airflow dags backfill -s 2026-06-01 -e 2026-06-07 my_dag creates or re-runs runs only for that range, regardless of the catchup setting. In practice most teams set catchup=False so deploys and un-pauses do not flood the cluster, then run targeted backfills when they genuinely need history. A strong answer names the guardrails: max_active_runs to cap concurrency during the backfill, depends_on_past when intervals must process in order, and idempotent tasks so re-running a range is safe.

Because Airflow schedules by data intervals, not wall-clock trigger times. A daily run "for" 2026-07-21 covers the interval from 2026-07-21 00:00 to 2026-07-22 00:00, and it only starts after that interval ends — only then is Monday's data complete. The run's logical_date (historically execution_date, now a deprecated alias) is the start of that interval, not the moment of execution. This is the single most common Airflow misunderstanding, so interviewers use it as a filter question. Practical implications: templates like ds, data_interval_start, and data_interval_end should drive all date logic in your tasks; the first run of a new DAG happens one full interval after start_date; and code that uses datetime.now() instead of the interval breaks the moment you retry or backfill. If you can whiteboard the interval timeline, you clear this question.

An idempotent task produces the same end state no matter how many times it runs for the same data interval. It matters because Airflow's whole value — retries, backfills, clearing failed tasks — assumes re-running is safe. Techniques: key every read and write to the templated interval (write to a path like s3://bucket/table/ds=2026-07-21/); prefer overwrite semantics — delete-then-insert for the partition, or MERGE/upsert — over blind appends; never derive dates from datetime.now(); make DDL steps use IF NOT EXISTS; and guard side effects like emails or API posts with state checks. Classic non-idempotent bug: a task that appends "yesterday's" rows computed from the current clock — retry it a day late and it loads the wrong window, twice. Interviewers often show exactly that snippet and ask you to spot the problem.

The executor defines where task processes run. SequentialExecutor runs one task at a time against SQLite — demos only. LocalExecutor runs tasks as parallel subprocesses on the scheduler's machine: the simplest production setup, fine until one box cannot hold your concurrency or you need isolation. CeleryExecutor distributes tasks to a static worker fleet through a message broker (Redis or RabbitMQ): good steady-state throughput and fast task start, but you run always-on workers and every worker needs all DAG dependencies installed. KubernetesExecutor launches a fresh pod per task: strong isolation, per-task resource requests and images, scale-to-zero — at the cost of pod start-up latency and Kubernetes operational overhead. Rule of thumb: single team with modest load, LocalExecutor; high sustained volume of short tasks, Celery; spiky load or conflicting dependencies across teams, Kubernetes.

The TaskFlow API (Airflow 2.0+) lets you define tasks as plain Python functions with the @task decorator and DAGs with @dag. Its main win is data passing: return a value from one task, pass it as an argument to the next, and Airflow creates the XComs and the dependency edge automatically — no xcom_pull with string task ids, no PythonOperator boilerplate with python_callable and op_kwargs. DAG code reads like normal Python, which kills the classic bug class of misspelled task-id strings. Two limits worth stating: values passed this way still travel through XCom, so the small-data rule still applies; and TaskFlow covers Python work — for transfer or vendor operators (S3, BigQuery, Spark submit) you still instantiate classic operators. Mixing both styles in one DAG is fully supported and is what real codebases do.

In poke mode a sensor occupies a worker slot for its entire wait, sleeping between checks — a hundred sensors waiting on files burn a hundred slots doing nothing. In reschedule mode the sensor releases its slot between checks: it enters up_for_reschedule and the scheduler re-queues it at the next poke interval, so slots are held only during the actual check. Deferrable operators (Airflow 2.2+) go further: the task defers to a trigger — an async coroutine run by the triggerer process — so thousands of waits share one event loop at near-zero worker cost; when the event fires, the task resumes on a worker. Rule of thumb: short waits with tight polling, poke; long waits with coarse polling, reschedule; large-scale or very long waits where slot economics matter, deferrable. Naming the triggerer by name signals real 2.x experience.

An SLA in Airflow 2.x is a timedelta on a task: if the task has not completed within that window measured from when the DAG run starts, the scheduler records an SLA miss, surfaces it in the UI, and can send an email or invoke the sla_miss_callback. The gotchas are what interviewers actually want. SLA misses are only evaluated for scheduled runs — manually triggered runs skip them entirely. An SLA miss does not stop or fail the task; a hard cap is execution_timeout, which kills the attempt. The callback is defined at the DAG level, not per task, and receives misses in batches. Because of these rough edges, many teams implement latency alerting outside Airflow instead, and SLA handling has been reworked in newer Airflow releases — saying that out loud signals real-world experience rather than doc-reading.

Dynamic task mapping (Airflow 2.3+) creates a variable number of parallel task instances at runtime from data an upstream task produced. You call .expand() with the varying argument — often a prior task's output — and .partial() for constants: process.partial(table="events").expand(file=list_files_output). Each mapped instance is scheduled, retried, and logged independently, and the Grid view shows them individually. Contrast that with dynamic DAG generation, where a parse-time Python loop stamps out tasks: that shape is frozen when the file is parsed and cannot react to "how many files landed today". Constraints worth naming: the mapped input travels through XCom, so map over lists of keys or paths, never payloads; there is a configurable cap on mapped instances (max_map_length); and you can bound the fan-out's parallelism with max_active_tis_per_dag so five hundred mapped copies do not starve the rest of the cluster.

Start with the contract: any interval can be re-run at any time, alone or in bulk, and produce correct results. Concretely: every task derives its window strictly from data_interval_start and data_interval_end templates — zero datetime.now(); writes are partition-scoped overwrites or MERGEs so a re-run replaces rather than duplicates; intermediate artifacts live at deterministic, interval-keyed paths. Then control the blast radius of the backfill itself: cap max_active_runs so queued runs do not starve everything else; use pools to protect a rate-limited source API; set depends_on_past only if intervals genuinely depend on prior state, because it serializes the whole backfill. Call out the semantic trap: today's DAG code runs against historical intervals, so if business logic changed mid-year, a naive backfill silently rewrites history under new rules. Finally, rehearse on a two-day range and verify counts before running the real thing.

Default XCom serializes values into the metadata database, which caps practical size at what your DB tolerates and, worse, puts data volume on the component every scheduler query touches. Three escape hatches. First, pass references by convention — S3 keys, table names, run ids — and let downstream tasks fetch the data themselves; this is the most common production pattern. Second, implement a custom XCom backend: subclass BaseXCom and override its serialize/deserialize hooks so large values are transparently written to object storage with only a reference stored in the database — task code stays unchanged, which is the selling point. Third, restructure so the handoff is the storage layer itself: task A writes a partition, task B reads that partition, and nothing crosses XCom but the partition key. Interviewers want you to reject "just XCom the DataFrame" and articulate exactly why it fails.

Walk it as an ordered checklist, most likely first. Is the DAG paused? New DAGs deploy paused by default unless configured otherwise. Does it parse? An import error keeps it out of scheduling entirely — check the UI banner or airflow dags list-import-errors. Then scheduling semantics: the first run fires only after the first full data interval ends, so start_date of today with a daily schedule produces nothing until tomorrow; and with catchup=False you only get intervals after the DAG appears. Timezone next: a naive start_date is treated as UTC, so a "9 AM" expectation in IST drifts by 5:30. Then capacity: is a stuck previous run holding max_active_runs, or is the pool exhausted? Finally the platform itself: is the scheduler alive and heartbeating, and did the new DAG file actually reach every component? Naming even four of these, in order, reads as production experience.