Databricks Interview Questions and Answers for Data Engineers (2026)

A lakehouse keeps all data in open formats on cheap object storage — like a data lake — while adding warehouse capabilities: ACID transactions, schema enforcement, fine-grained governance, and BI-grade performance through Delta Lake and engines like Photon. It removes the classic two-copy architecture where a lake feeds a separate warehouse, eliminating duplicate storage, sync jobs, and stale marts.

Delta Lake is an open table format: Parquet data files plus a _delta_log directory of ordered JSON commit files, with periodic Parquet checkpoints, recording every add and remove action. A transaction is atomic because it becomes visible only when its commit file is written; concurrent writers use optimistic concurrency control with conflict detection at commit time. The same log enables schema enforcement, schema evolution, and time travel.

Bronze holds raw data exactly as ingested — append-only and replayable; Silver holds cleaned, deduplicated, conformed data with enforced schemas; Gold holds business-level aggregates and marts serving BI and ML. Each hop is incremental, so a logic bug can be fixed and replayed from the layer below without re-ingesting source systems.

All-purpose (interactive) clusters are long-lived, shared for notebook development, and billed at a higher DBU rate; job clusters are created by a job run and terminated when it finishes, at a significantly cheaper rate. The production rule: humans on all-purpose or serverless, scheduled workloads always on job clusters. Running production jobs on an interactive cluster is a classic cost red flag interviewers probe for.

Every commit creates a new table version, so you can query history with SELECT ... VERSION AS OF or TIMESTAMP AS OF, inspect it with DESCRIBE HISTORY, and roll back with RESTORE TABLE. Reach is bounded by retention: delta.logRetentionDuration (default 30 days) for the log and delta.deletedFileRetentionDuration (default 7 days) for data files. Once VACUUM removes old files, those versions are unrecoverable.

OPTIMIZE compacts many small files into roughly 1 GB files, fixing the small-file problem caused by streaming and frequent small writes; ZORDER BY additionally co-locates related values so per-file min/max statistics can skip files on high-cardinality filter columns like customer_id. Run OPTIMIZE on a schedule after heavy write windows, Z-ORDERing on your top query-filter columns. On newer runtimes, liquid clustering (CLUSTER BY) replaces both hive-style partitioning and Z-ORDER and re-clusters incrementally.

Auto Loader is the cloudFiles Structured Streaming source: it discovers new files via directory listing or cloud file-notification mode and tracks processed files in a RocksDB-backed checkpoint for exactly-once ingestion. It supports schema inference and evolution, capturing unparseable data in the _rescued_data column. COPY INTO suits occasional batch loads of thousands of files; Auto Loader scales to millions and runs continuously or incrementally with Trigger.AvailableNow.

Unity Catalog is centralised governance across all workspaces in a region: one metastore containing catalogs, then schemas, then tables, views, volumes, functions, and models — the three-level namespace catalog.schema.table. It provides ANSI GRANT-based access control, automatic column-level lineage, audit logs, row filters and column masks, and Delta Sharing. It replaces per-workspace Hive metastores, so permissions and discovery are defined once rather than per cluster.

Photon is Databricks' native C++ vectorised execution engine that transparently replaces JVM execution for supported Spark SQL and DataFrame operators — no code changes required. It typically delivers 2–3x speedups on SQL-heavy workloads (scans, joins, aggregations over Delta and Parquet) and powers Databricks SQL warehouses. Photon compute carries a higher DBU rate, but shorter runtimes usually reduce total cost; it does not accelerate RDD code or arbitrary Python UDF logic.

DLT — now Lakeflow Declarative Pipelines — lets you declare streaming tables and materialised views in SQL or Python (@dlt.table), while the framework builds the dependency DAG and manages compute, retries, and incremental processing. Expectations are declarative data-quality rules like CONSTRAINT valid_id EXPECT (id IS NOT NULL) with ON VIOLATION DROP ROW or FAIL UPDATE, and violation metrics land in the pipeline event log. It is the standard answer for medallion pipelines without hand-rolled orchestration.

For batch, MERGE INTO target USING updates ON key handles inserts, updates, and deletes in one atomic statement. For streaming, run a MERGE per micro-batch inside foreachBatch, or in DLT use APPLY CHANGES INTO, which orders out-of-sequence events via a SEQUENCE BY column and supports SCD Type 1 and Type 2 targets. To propagate changes downstream, enable Change Data Feed (delta.enableChangeDataFeed) and consume it with readChangeFeed.

Managed tables have files owned and placed by Unity Catalog in the catalog or schema's managed storage location; DROP TABLE deletes the data (files are cleaned up after a retention window) and features like predictive optimization apply. External tables register data at an explicit cloud path through an external location and storage credential; DROP removes only metadata. Default to managed for new data; use external when other tools read or write the same files, or during hive_metastore migration with SYNC.

Writers use optimistic concurrency: each transaction records the table version it read, does its work, then validates at commit that no conflicting commit landed in between — if one did, it fails with ConcurrentAppendException or ConcurrentDeleteReadException. Mitigations: make concurrent operations touch disjoint data and express that disjointness in the MERGE or UPDATE predicate (include the partition or clustering column), serialise jobs writing the same keys, and add application-level retries. Partitioning or liquid clustering aligned with writer boundaries removes most conflicts; know the WriteSerializable (default) vs Serializable isolation trade-off.

Start in the Spark UI: find the longest stage and check for skew (one straggler task), shuffle spill, and small-file scans. Typical fixes: let AQE coalesce shuffle partitions and handle skew joins, broadcast the small side of joins, restructure to cut shuffles, and compact inputs with OPTIMIZE or auto compaction so scans read fewer, larger files. On infrastructure: right-size worker type and count, enable autoscaling, turn on Photon for SQL-heavy stages, and move the job off any all-purpose cluster. Prove the win with cost per run from system.billing.usage before and after.

Auto Loader with file notifications ingests raw files into a Bronze streaming table, with _rescued_data capturing malformed records; Silver applies expectations, watermark-based deduplication, and APPLY CHANGES or MERGE for CDC; Gold builds materialised aggregates for BI. Implement it in DLT so dependencies, retries, and incremental compute are managed, choosing continuous mode for low latency or Trigger.AvailableNow on a schedule for cost. Checkpoints make each hop exactly-once, Unity Catalog provides lineage and grants, and VACUUM retention is tuned so streaming readers never lose referenced files.

Enforce cluster policies that limit node types, autoscaling ranges, and mandate auto-termination; require job clusters or serverless for scheduled work, with spot workers plus on-demand fallback for fault-tolerant stages. Tag clusters and jobs by team and project, then attribute spend through system tables — system.billing.usage joined with list_prices — into a chargeback dashboard. Hunt the classics: always-on interactive clusters, oversized drivers, continuous streams that could run Trigger.AvailableNow, and unoptimised tables forcing full scans.