Databricks Interview Questions and Answers for Data Engineers (2026)

The lakehouse keeps a single copy of data in open formats (Parquet under Delta Lake) on cheap object storage, then layers on what historically forced a separate warehouse: ACID transactions, schema enforcement, fine-grained governance through Unity Catalog, and BI-grade performance via Photon, caching, and data skipping. The problem it solves is the two-copy architecture — a lake feeding a warehouse via ETL — which doubles storage, adds sync jobs that drift, and leaves ML training on stale extracts. The trade-off interviewers probe: object storage is high-latency, so the model depends on table-format maintenance (compaction, clustering, statistics) to actually hit warehouse-class performance.

Delta Lake is an open table format: Parquet data files plus a _delta_log directory of ordered JSON commit files — with periodic Parquet checkpoints so readers need not replay the whole log — recording every add/remove action, schema change, and protocol upgrade. A write is atomic because it becomes visible only when its numbered commit file lands; readers always see a consistent snapshot as of one version, never a half-finished write. Concurrent writers use optimistic concurrency control: each transaction validates at commit that no conflicting commit appeared since the version it read, and fails with a Concurrent*Exception rather than corrupting data. The same log powers schema evolution, time travel, and Change Data Feed.

Bronze stores raw data exactly as ingested — append-only, minimally transformed, cheap insurance that lets you replay history without touching source systems again. Silver holds cleaned, deduplicated, conformed entities with enforced schemas and quality expectations; Gold holds business-level aggregates and marts consumed by BI and ML. Each hop is incremental, so a logic bug is fixed by rebuilding from the layer below rather than re-extracting from sources. The nuance interviewers want: it is a quality contract, not a rigid rule of three — teams add or collapse layers — but the invariant is that raw stays immutable and every derived layer is reproducible from the one beneath it.

All-purpose (interactive) clusters are long-lived, shared by people in notebooks, and billed at a materially higher DBU rate; job clusters are created by a job run, exist only for that run, and terminate when it finishes, at a cheaper rate and with per-run isolation — one job's library or config change cannot break another's. The production rule: humans on all-purpose or serverless, every scheduled workload on job clusters or serverless jobs compute. The follow-up interviewers probe is cost hygiene: an always-on interactive cluster running nightly ETL is the classic red flag, and auto-termination plus cluster policies are how you prevent it organisation-wide rather than team by team.

Every commit creates a new table version, so you can query the past with SELECT ... VERSION AS OF n or TIMESTAMP AS OF, audit changes with DESCRIBE HISTORY, and roll back with RESTORE TABLE — which is itself a new commit, so history is preserved rather than rewritten. Reach is bounded by two retention settings: delta.logRetentionDuration (default 30 days) for commit history and delta.deletedFileRetentionDuration (default 7 days) for the removed data files that old versions still reference. The trap interviewers test: VACUUM permanently deletes files past that retention, so aggressive vacuuming silently breaks time travel — and can break lagging streaming readers that still reference those files.

Auto Loader is the cloudFiles Structured Streaming source. It discovers new files by directory listing or, at scale, cloud file-notification mode, and records processed files in a RocksDB-backed checkpoint so each file is ingested exactly once even across restarts. It infers the schema, evolves it as new columns appear, and shunts unparseable values into the _rescued_data column instead of dropping rows. COPY INTO is the simpler idempotent SQL alternative, fine for occasional batch loads of thousands of files; Auto Loader scales to millions and runs continuously or as a cheap incremental batch with Trigger.AvailableNow. The follow-up: notification mode avoids re-listing huge directories, which is exactly what kills listing-based ingestion at scale.

Unity Catalog centralises governance across every workspace attached to it: one metastore (typically per region) containing catalogs, then schemas, then securables — tables, views, volumes, functions, and models — addressed by the three-level namespace catalog.schema.table. Access control is ANSI GRANT/REVOKE on those objects, inherited down the hierarchy, with row filters and column masks for fine-grained control. It also provides automatic column-level lineage, audit logs, and Delta Sharing for cross-organisation sharing. The contrast interviewers want: it replaces per-workspace Hive metastores and cluster-scoped ACLs, so permissions and discovery are defined once, centrally, instead of being re-implemented per workspace and per cluster.

Photon is Databricks' native vectorised execution engine, written in C++, that transparently replaces JVM execution for supported Spark SQL and DataFrame operators — no code changes, with seamless fallback for unsupported ones. It shines on SQL-heavy work — scans, joins, aggregations over Delta and Parquet — and is the engine inside Databricks SQL warehouses. It does not accelerate RDD code or per-row Python UDF logic, so a UDF-heavy pipeline sees little gain. The cost nuance interviewers probe: Photon compute carries a higher DBU rate, so the honest answer is that it usually lowers total cost by finishing faster — but you verify with cost per run from billing data rather than assuming, and skip it where unsupported operations dominate.

For batch, MERGE INTO target USING source ON key handles inserts, updates, and deletes in one atomic statement. For streaming, run the MERGE per micro-batch inside foreachBatch — MERGE is not a native streaming sink — or use DLT's APPLY CHANGES INTO, which orders out-of-sequence events via a SEQUENCE BY column and maintains SCD Type 1 or Type 2 targets declaratively. To propagate row-level changes downstream, enable Change Data Feed (delta.enableChangeDataFeed) and consume inserts, updates, and deletes with readChangeFeed. The follow-ups interviewers probe: deduplicate the source batch first, since duplicate keys make MERGE fail on multiple matches, and align the MERGE predicate with partitioning or clustering to avoid concurrent-write conflicts.

Writers use optimistic concurrency: each transaction records the table version it read, does its work, then validates at commit that no conflicting commit landed in between — if one did, it fails with ConcurrentAppendException or ConcurrentDeleteReadException. Mitigations: make concurrent operations touch disjoint data and express that disjointness in the MERGE or UPDATE predicate (include the partition or clustering column so Delta can prove non-overlap), serialise jobs that write the same keys, and add application-level retries for the residual cases. Partitioning or liquid clustering aligned with writer boundaries removes most conflicts; also know the isolation trade-off — WriteSerializable (default) permits some read-write anomalies that strict Serializable forbids.

Start in the Spark UI: find the longest stage and check for skew (one straggler task), shuffle spill to disk, and scans over huge numbers of small files. Typical fixes: let AQE coalesce shuffle partitions and split skewed joins, broadcast the small side of joins, restructure logic to cut shuffles, and compact inputs with OPTIMIZE or auto compaction so scans read fewer, larger files. On infrastructure: right-size worker type and count, enable autoscaling, turn on Photon for SQL-heavy stages, and move the job off any all-purpose cluster onto job or serverless compute. Then prove the win with cost per run from system.billing.usage before and after — interviewers want the measurement step, not just a list of knobs.

Auto Loader with file notifications ingests raw files into a Bronze streaming table, with _rescued_data capturing malformed records instead of dropping them; Silver applies expectations, watermark-based deduplication, and APPLY CHANGES or foreachBatch MERGE for CDC; Gold builds materialised aggregates for BI. Implement it in DLT so the DAG, retries, and incremental compute are managed — choosing continuous mode for low latency or Trigger.AvailableNow on a schedule when minutes of latency are acceptable and cost matters. Checkpoints make each hop exactly-once, Unity Catalog supplies lineage and grants, and VACUUM retention is tuned so streaming readers never lose files they still reference. Close with monitoring: the DLT event log feeds quality and latency alerts.