Hadoop Interview Questions 2026: HDFS, MapReduce, YARN & Migration-Era Answers

Hadoop is an open-source framework for storing and processing very large datasets across clusters of commodity machines. It has three core components plus a support layer: HDFS, the distributed file system that splits files into large blocks and replicates them across nodes; YARN, the resource manager that allocates CPU and memory to applications; and MapReduce, the original batch processing model. Hadoop Common provides shared libraries and utilities. Two design ideas matter more than the component list: computation moves to where the data lives instead of pulling data across the network, and failure is handled in software — replicated blocks and re-run tasks — because commodity hardware is expected to fail. A good answer also notes that the wider ecosystem (Hive, HBase, Spark on YARN) sits on top of these primitives.

The NameNode is the master that stores the filesystem namespace: the directory tree, file permissions, and the mapping of each file to its blocks. It keeps this metadata in RAM for fast access, persisting it as an fsimage plus an edit log. Block locations are not persisted — they are rebuilt at startup from block reports sent by DataNodes. DataNodes are the workers: they store actual block data on local disks, serve read and write requests, and send heartbeats and block reports to the NameNode. Crucially, the NameNode is not in the data path — a client asks it for block locations, then streams data directly to or from DataNodes. That is why one modest master can coordinate petabytes. In Hadoop 1 the NameNode was a single point of failure; Hadoop 2+ added high availability with a standby.

The default block size is 128 MB (64 MB in very old Hadoop 1 clusters), enormous compared with disk filesystems. Three reasons. First, seek overhead: with large sequential blocks, time spent seeking is tiny relative to transfer time, so throughput approaches disk bandwidth. Second, NameNode memory: every block consumes roughly 150 bytes of metadata in the NameNode heap, so fewer, bigger blocks let a single master track petabytes. Third, scheduling: one map task typically processes one block, and larger blocks mean fewer tasks and less startup overhead per job. There is a ceiling — blocks that are too large reduce parallelism because fewer tasks can run concurrently. One detail interviewers like: a file smaller than the block size occupies only its actual size on disk. The block is a logical unit, not preallocated space.

Replication factor is the number of copies HDFS keeps of each block; the default is 3. When a client writes a block, it is pipelined to three DataNodes, so the cluster can lose a node — or, with rack-aware placement, an entire rack — without losing data. The NameNode continuously compares expected versus reported replicas; if a DataNode dies, under-replicated blocks are re-copied from surviving replicas automatically, with no operator action. Replication also helps reads: clients fetch from the nearest replica, and popular data has more sources. The cost is 3x storage overhead, which is why Hadoop 3 introduced erasure coding as an alternative for cold data at roughly 1.5x. Replication is set per file, not just per cluster, so teams often keep hot datasets at 3 and archives lower or erasure-coded.

YARN (Yet Another Resource Negotiator) is Hadoop's cluster resource management layer, introduced in Hadoop 2. Hadoop 1's JobTracker did two jobs at once — scheduling cluster resources and tracking every MapReduce task — which limited scale and locked the cluster to a single processing model. YARN split those roles. The ResourceManager is the global authority: its scheduler (Capacity or Fair) hands out containers, and its ApplicationsManager accepts job submissions. Each node runs a NodeManager that launches and monitors containers locally. Every application gets its own ApplicationMaster, a small process that negotiates containers and drives execution for that one job. The consequences: cluster scale improved because per-job bookkeeping moved out of the central daemon, and the cluster became engine-agnostic — MapReduce, Spark, Tez, and Hive can all share the same hardware under one scheduler.

Start from input: the InputFormat computes splits (usually one per HDFS block) and a RecordReader turns bytes into key-value pairs. The map function processes each record and writes output to an in-memory circular buffer (100 MB by default). When the buffer passes a threshold (around 80%), a background thread spills it to local disk — during the spill, records are partitioned by reducer and sorted by key, and the optional combiner runs. Spill files are merged into one partitioned, sorted file per map task. Note this lands on local disk, not HDFS. In the shuffle phase, each reducer pulls its partition from every completed map task over HTTP, merges the sorted streams, and calls reduce once per key with all its values. Finally, the OutputFormat writes results to HDFS. Interviewers listen for three checkpoints: where sorting happens, where the combiner runs, and that map output never touches HDFS.

HDFS is built for a modest number of huge files, and the NameNode keeps an object in heap memory for every file, directory, and block — roughly 150 bytes each. Ten million small files means tens of millions of objects, and NameNode heap becomes the cluster's scaling wall long before disk does. Processing suffers too: each small file typically becomes its own split, so a job over a million 1 MB files launches a million short-lived tasks dominated by startup overhead. Fixes, in order of preference: stop the problem at ingestion by batching writes into larger files; run scheduled compaction jobs that rewrite small files into large ORC, Parquet, or Avro files; use CombineFileInputFormat so one task reads many files; pack legacy data into HAR archives or SequenceFiles; and for genuinely small records, use HBase rather than raw HDFS files. In interviews, lead with the NameNode-memory root cause, not just the fix list.

Every DataNode sends a heartbeat to the NameNode every 3 seconds. If heartbeats stop, the node is first treated as stale (reads and writes deprioritize it), and after roughly 10 minutes — derived from the recheck-interval settings — it is declared dead. The NameNode then walks the list of blocks that node held and schedules re-replication: healthy replicas are copied to other DataNodes until every block is back at its target replication factor, throttled so recovery traffic does not starve real work. The long timeout is deliberate: re-replicating an entire node's data is expensive, so HDFS avoids reacting to short network blips. Corruption is handled separately: clients verify checksums on every read, and a corrupt replica is reported, discarded, and replaced from a good copy. The takeaway an interviewer wants: failure handling is automatic, replica-driven, and the NameNode orchestrates but never carries the data itself.

Rack awareness means the NameNode knows which rack each DataNode belongs to, via a configured topology script or class. The default placement policy for replication factor 3: the first replica goes on the writer's own DataNode (or a random node if the client is outside the cluster); the second goes to a node on a different rack; the third goes to a different node on the same rack as the second. The reasoning is a bandwidth-versus-safety compromise. Losing a whole rack — a top-of-rack switch failure is the classic case — still leaves a live replica, because two racks always hold the data. But the write pipeline crosses the rack boundary only once (between replicas one and two), since replicas two and three are rack-local to each other. On reads, HDFS serves from the topologically closest replica. Mentioning the switch-failure scenario shows you understand why the policy exists.

The partitioner decides which reducer receives each map output key — the default HashPartitioner takes hash(key) mod the number of reducers. You write a custom one for things like total ordering or routing related keys together; it is also where key-skew problems surface. The combiner is a local mini-reduce that runs on map output before it is shuffled, shrinking data sent across the network — for a word count, a mapper that saw 'the' 10,000 times ships one pair instead of 10,000. The catch: the framework may run the combiner zero, one, or many times, so correctness must not depend on it running. That restricts combiners to commutative, associative operations — sum, count, min, max. The classic trap is average: averaging averages is wrong. The fix is to make the combiner emit (sum, count) pairs and let the reducer do the final division.

This is a deliberate trap: the Secondary NameNode is not a backup and cannot take over if the NameNode dies. Its job is checkpointing. The NameNode persists metadata as an fsimage snapshot plus an ever-growing edit log of changes; replaying a huge edit log makes restarts painfully slow. The Secondary NameNode periodically fetches both files, merges the edits into a new fsimage on its own hardware (which is why it needs comparable memory), and ships the result back. Metadata loss is still possible between checkpoints if the NameNode's disks are lost. Actual failover requires an HA deployment: an Active and a Standby NameNode sharing edits through a quorum of JournalNodes, with ZooKeeper Failover Controllers handling automatic failover and fencing preventing split-brain. In HA clusters the Standby performs checkpointing itself, so no Secondary NameNode runs at all. Stating that last fact cleanly usually ends the question.

They sit at different layers, so 'versus' is partly a category error. Hadoop is a platform: HDFS for storage, YARN for resource management, MapReduce as its original compute engine. Spark is only a compute engine — and in most enterprises it runs on YARN, reading data from HDFS. The real comparison is MapReduce versus Spark. MapReduce materializes intermediate results to disk between every job, executes rigid map-then-reduce stages, and starts fresh JVMs per task. Spark builds a DAG of the whole computation, pipelines stages, keeps executors warm, and can cache datasets in memory — which transforms iterative workloads like ML and interactive queries. Spark also bundles SQL, streaming, and ML APIs. By 2026, essentially no one writes new MapReduce jobs, but Hadoop's storage and scheduling layers survive underneath Spark in many on-prem clusters. Saying 'Spark replaced MapReduce, not Hadoop' captures it precisely.

The slogan answer — 'Spark is in-memory' — is incomplete. Spark's real advantages: it builds a DAG of the entire computation and pipelines operations within a stage, so intermediate results between chained jobs are not written to HDFS the way MapReduce requires; executors are long-lived JVMs that run many tasks, avoiding MapReduce's per-task JVM startup; explicit caching lets iterative algorithms reuse a dataset across passes; and Catalyst plus whole-stage code generation optimize SQL workloads. But the gap narrows in specific cases. Spark shuffles still write to local disk, and when a shuffle far exceeds cluster memory, Spark spills heavily and behaves much like MapReduce — with added GC and memory-management tuning burden. For a single-pass ETL job that reads once, transforms, and writes once, there is no intermediate data to keep in memory, so the win comes mainly from pipelining and JVM reuse, not caching. Demonstrating you know when the advantage shrinks is what makes this a senior answer.

The client asks the NameNode to create the file and allocate a block; the NameNode returns a pipeline of DataNodes chosen by the placement policy — three for default replication. The client streams the block as small packets (around 64 KB), which flow through the pipeline: client to DN1, DN1 forwards to DN2, DN2 to DN3. Acks travel back up the pipeline, and the client keeps unacknowledged packets in an ack queue. If a DataNode fails mid-write, the pipeline is torn down, the failed node is removed, the block gets a new generation stamp on the surviving nodes so stale replicas are detectable, and writing resumes through the shorter pipeline; the NameNode restores full replication later. hflush guarantees data has reached every DataNode in the pipeline's memory; hsync forces it to disk. On close, the NameNode commits the file once minimum replication is acknowledged. This question separates people who read the architecture from people who memorized definitions.

Erasure coding, added in Hadoop 3, cuts the storage cost of fault tolerance. With the common Reed-Solomon RS-6-3 policy, data is striped into six data cells plus three parity cells across nine DataNodes; any three can be lost and the data is still reconstructable. Overhead drops from 3x with replication to 1.5x — a massive saving on large clusters. The costs are real, though. Reconstruction after a failure requires reading from multiple surviving nodes and doing CPU-heavy decoding, versus simply copying one replica. The striped layout removes data locality, so tasks read remotely by design. Small files are handled poorly because striping needs enough data to fill cells. EC files also do not support append or hflush/hsync. The practical policy interviewers want: keep hot, frequently read data on 3x replication; move cold, archival data to erasure coding. It is a storage-economics decision, not a blanket upgrade.

This is the question that distinguishes candidates in 2026, since many enterprises and GCCs are mid-migration. Object stores like S3, ADLS, and GCS are not filesystems, and several HDFS assumptions break. Rename is the big one: on HDFS it is an atomic metadata operation, but on object stores it is a copy-then-delete proportional to data size — so job-commit protocols that rename temporary output into place need replacements like the S3A committers or manifest committers. Directory listings are emulated and slow at scale, hurting listing-heavy jobs. Data locality disappears entirely — compute and storage are separated, which is a feature for elasticity but changes performance tuning. HDFS permissions and ACLs map awkwardly onto cloud IAM. Hardcoded hdfs:// URIs and Hive metastore locations need rewriting, and bulk transfer typically runs through DistCp. Mention that many teams pair the move with a table format like Iceberg, Delta, or Hudi to restore atomic commits.