GCP Data Engineer Interview Questions 2026: Service Selection, Pipelines, and Answers That Hold Up

A production GCP data stack usually has six pieces. Cloud Storage is the object store — the landing zone for raw files and the data-lake layer. BigQuery is the serverless, columnar warehouse where analytics SQL runs. Dataflow is the managed Apache Beam runner used for both batch and streaming transformations. Pub/Sub is the global messaging service that ingests events and decouples producers from consumers. Dataproc is managed Spark and Hadoop, used mainly when you already have Spark or Hive code. Cloud Composer is managed Apache Airflow and handles orchestration — scheduling, dependencies, retries. A typical flow: events arrive through Pub/Sub or files land in Cloud Storage, Dataflow cleans and enriches them, results land in BigQuery, and Composer coordinates the whole pipeline. In an interview, attach a "when I'd use it" to each service, not just a definition.

Cloud Storage offers four classes that trade storage price against access cost. Standard has no minimum storage duration or retrieval fee — use it for hot data that pipelines read constantly. Nearline is cheaper per GB but carries a 30-day minimum and a retrieval charge, suiting data touched roughly monthly, like recent backups. Coldline has a 90-day minimum for data accessed maybe quarterly, and Archive has a 365-day minimum for compliance and long-term retention. Two points interviewers like: all four classes serve reads through the same API with low latency — only the economics change, unlike tape-style archive products — and Object Lifecycle Management rules can transition objects automatically, for example raw files moving Standard to Nearline to Archive as partitions age. Picking a colder class for frequently read data actually costs more, because retrieval fees dominate.

Pub/Sub is Google Cloud's fully managed, globally available messaging service. Producers publish messages to a topic; each subscription attached to that topic receives its own copy of the stream, so multiple downstream systems can consume independently — that fan-out is a key design feature. It solves the coupling problem: without a buffer, a slow or crashed consumer forces producers to drop data or block. Pub/Sub absorbs traffic spikes, retains unacknowledged messages (seven days by default), and redelivers anything not acknowledged before the ack deadline. Delivery is at-least-once by default, so consumers must handle duplicates idempotently; exactly-once delivery is available on pull subscriptions. A typical data engineering use: application events or CDC streams published to one topic, with one subscription feeding a Dataflow streaming job and another archiving raw events to Cloud Storage.

A user account represents a human authenticated through Google sign-in; a service account is an identity for workloads — a Dataflow job, a Composer task, a VM — so pipelines can call GCP APIs without human credentials. You grant IAM roles to a service account exactly as you would to a user, and the workload authenticates as it. Best practices interviewers listen for: create a dedicated service account per pipeline or component rather than sharing one broad identity; grant the narrowest predefined role that works — for example BigQuery Data Editor on one dataset, not project-wide Editor; and avoid downloading long-lived JSON keys — prefer service accounts attached to the compute resource or Workload Identity, so credentials stay short-lived and never sit in a repo. Service accounts are also resources: access to impersonate them must itself be controlled.

Both process data at scale, but they target different situations. Dataproc is managed open source — Spark, Hadoop, Hive — where Google provisions clusters but you still think in cluster terms (machine types, autoscaling policies), though the serverless Spark option removes much of that. Choose it when you already have Spark or Hive code, depend on the open-source ecosystem, or are migrating a Hadoop estate and want minimal rewriting. Dataflow is a fully managed runner for Apache Beam: no clusters to size, automatic horizontal scaling, and one programming model for both batch and streaming with strong event-time windowing support. Choose it for new pipelines, especially streaming ones. The honest interview answer: existing Spark assets point to Dataproc; greenfield unified batch-and-streaming points to Dataflow; and pure SQL transformations often need neither — just BigQuery ELT.

Start with the flow, then justify each hop. Files land in a Cloud Storage bucket partitioned by date. A Cloud Composer DAG scheduled daily — or triggered on file arrival via a sensor — validates that expected files exist and checks basics like row counts and schema. For the load, prefer a BigQuery load job into a date-partitioned staging table: load jobs are free and avoid streaming costs; you only need Dataflow in the middle if transformations are too complex for SQL or files need heavy parsing. Then run ELT: SQL transforms from staging into curated, partitioned, clustered tables. Make every step idempotent — write each day's data with a partition overwrite or a MERGE keyed on business keys, so reruns cannot duplicate rows. Finish with freshness checks and alerting on DAG failure. Interviewers reward the cost note and the idempotency note more than the service names.

Beam's core abstraction is the PCollection, which can be bounded (a fixed dataset — batch) or unbounded (a stream). The same pipeline code — ParDo transforms, GroupByKey, aggregations — runs against either; only the source changes. For unbounded data, Beam separates event time (when the event happened) from processing time (when it arrives), and you group events with windows: fixed, sliding, or session. Since a stream never finishes, the watermark is the runner's estimate of event-time progress — when the watermark passes the end of a window, that window is considered complete and results can be emitted. Triggers control early or repeated emissions, and allowed lateness defines how long to accept stragglers after the watermark, with late results refining earlier ones. Dataflow executes this model with autoscaling and checkpointing. Explaining watermarks correctly is usually the differentiator on this question.

Default Pub/Sub delivery is at-least-once: if a subscriber fails to acknowledge within the ack deadline — because it crashed, ran slow, or the ack was lost — the message is redelivered. Duplicates are therefore normal, and the baseline defence is idempotent consumers: dedupe on a message ID or business key before applying effects. Pub/Sub also offers exactly-once delivery on pull subscriptions, which prevents redelivery of already-acknowledged messages, but that guarantee ends at the subscriber — it does not make downstream writes exactly-once. In practice, correctness is layered: Dataflow deduplicates redelivered Pub/Sub messages by message ID, or by a custom ID attribute you set at publish time to also catch publisher retries, and the final sink is made idempotent — MERGE on keys in BigQuery, or Storage Write API offsets. Mention ordering keys if per-entity ordering matters; they order delivery per key, not globally.

Cloud Composer is managed Apache Airflow, and the interview signal is knowing what belongs in Airflow versus outside it. A good DAG is a thin control plane: operators submit work to BigQuery, Dataflow, or Dataproc and wait for completion; Airflow workers never crunch the data themselves, because they are not sized for it. Structure DAGs around the execution date so every run is parameterised and rerunnable — a backfill then becomes re-triggering old dates. Use sensors or event-driven triggers for upstream availability rather than guessing with fixed delays; set retries with exponential backoff; make tasks idempotent so a retry cannot double-write. Keep dependencies explicit so partial failures resume from the failed task, not from the start. Add SLA or freshness alerts so a silently stuck DAG pages someone. Anti-patterns worth naming: giant Python transforms inside tasks, hidden cross-DAG dependencies, non-idempotent loads.

Two pricing models exist. On-demand bills per TB scanned, so cost control means scanning less: partition large tables by date and require partition filters, cluster on frequent filter columns, never SELECT * on wide tables, and use dry runs or the bytes-scanned estimate before running expensive queries. Capacity pricing through BigQuery editions buys slot capacity — dedicated compute with optional autoscaling — which makes spend predictable and usually wins once query volume is high and steady; reservations also isolate workloads, keeping ad-hoc analyst queries from starving scheduled pipelines. Platform-level guardrails matter too: custom quotas capping per-user or per-project daily scan volume, and monitoring job metadata in INFORMATION_SCHEMA to find the most expensive queries and the worst offenders. A strong answer picks a pricing model based on workload shape and names at least one guardrail — not just "use partitioning".

Least privilege, applied with structure. Separate projects per environment — dev, staging, prod — so a mistake in dev cannot touch prod data. Give each pipeline its own service account, granted the narrowest predefined roles: BigQuery Data Editor on the specific target dataset plus Job User to run jobs, Storage Object Viewer on the source bucket — never project-level Editor or Owner, and treat basic roles as a red flag in any design. Grant human access through Google Groups mapped to job functions rather than individual bindings, so onboarding and offboarding become group membership changes. In BigQuery, control access at dataset level, use authorized views to expose subsets, apply policy tags for column-level security on PII, and add row-level policies where teams should see different slices. Keep audit logs enabled so access is provable. Interviewers specifically listen for per-pipeline service accounts and dataset-scoped grants.

Four realistic options, ordered by complexity. Pub/Sub BigQuery subscriptions write topic messages directly into a table with no code — the right choice when raw events just need landing, untransformed. The Storage Write API is the modern high-throughput ingestion API applications can call directly; with committed streams and offsets it supports exactly-once semantics, and it supersedes the legacy insertAll streaming inserts, which cost more and offer only best-effort deduplication. Dataflow sits in the middle when the stream needs work before landing: parsing, enrichment via lookups, deduplication, windowed aggregation, or routing bad records to a dead-letter destination — its BigQuery sink uses the Storage Write API underneath. Decision rule for interviews: no transformation, use a BigQuery subscription; app-controlled writes, use the Storage Write API; any real processing, event-time logic, or dedup, use Dataflow. Naming the legacy option and why to avoid it shows currency.

Ingest clickstream events into a Pub/Sub topic. Two consumers: a Dataflow streaming job, plus an archive path — a second subscription or a side output — writing raw events to partitioned storage in Cloud Storage or a raw BigQuery table as the immutable source of truth. The Dataflow job parses and validates events, sends malformed ones to a dead-letter topic, then aggregates in event-time windows — say one-minute fixed windows keyed by page — with defined allowed lateness, emitting refinements as late events arrive. Aggregates land in a partitioned BigQuery table via the Storage Write API, and the dashboard queries that table. Data arriving after allowed lateness is why the reconciliation leg exists: a nightly Composer-scheduled batch job recomputes yesterday's aggregates from the raw archive, which has everything however late, and MERGEs corrections into the serving table. Real-time numbers are approximately right immediately, exactly right by morning — state that tradeoff explicitly.

Within the pipeline, Dataflow's streaming engine gives exactly-once processing: state and shuffle are checkpointed, and when failed work is retried, the runner deduplicates so each element affects state once. At the edges the guarantee erodes. On the source side, Pub/Sub redelivers unacknowledged messages; Dataflow deduplicates these by Pub/Sub message ID — or by a custom ID attribute attached at publish time, which also catches publisher-side retries that create distinct messages. The sink is the classic leak: if the pipeline writes to an external system and the write succeeds but its confirmation is lost, the retry double-writes. Mitigations: idempotent sinks such as MERGE on business keys, the BigQuery Storage Write API with committed streams and offsets, or accepting at-least-once into a raw table and deduplicating in a downstream view. Strong candidates explicitly separate "exactly-once processing" from "exactly-once end to end" — the latter needs source and sink cooperation.

Sequence it in phases and justify each. Phase one is lift-and-shift to Dataproc: run existing Spark and Hive jobs mostly unchanged, but replace HDFS with Cloud Storage through the GCS connector so storage and compute decouple. That single change enables ephemeral, job-scoped clusters that spin up, run, and delete — which is where most of the cost saving comes from versus an always-on cluster. Phase two re-platforms selectively: Hive and SQL-heavy workloads usually map cleanly to BigQuery ELT; Spark ETL either stays on serverless Spark or gets rewritten to Dataflow when streaming or Beam's model adds value; cron and Oozie-style orchestration moves to Composer. Decide per workload on effort versus payoff — rewrite the pipelines that run most often and cost the most, keep the long tail on Dataproc. De-risk with parallel runs comparing outputs before cutover, and migrate data before compute. Never propose a big-bang rewrite.

Treat additive change as the default and everything else as a migration. BigQuery lets you add NULLABLE columns and relax REQUIRED to NULLABLE in place; renames and type changes are not in-place operations, so plan them as new-column-plus-backfill or new-table-plus-swap. Protect consumers by having them read views, not tables — the view is the contract, so you can restructure the underlying table and adjust the view without breaking dashboards. For streaming pipelines, version the event schema — Avro or Protobuf with a schema-registry pattern — so old and new producers coexist, and make parsers tolerant of unknown fields. Backfills should reprocess from the immutable raw layer in Cloud Storage, partition by partition, into a shadow table; validate row counts and key aggregates against production, then swap the view atomically. Keeping the raw layer immutable is what makes any of this recoverable — say that explicitly in the interview.