This guide is a collection or a summary on how to install and use a footprint of Apache Hadoop. I tried to follow an old version 2.7.1 guide that I created few years ago and adjusted this to use the latest version. Apache Hadoop 3.5.0 is used below; check the Apache releases page before future
installations.
These instructions target Linux (Ubuntu/Debian) for development or testing.
Production clusters need Kerberos, network controls, encryption, monitoring,
backups, and an upgrade plan. Do not expose HDFS or YARN ports to the internet.
Native single-node installation
Prerequisites
sudo apt-get update sudo apt-get install -y openjdk-17-jdk openssh-client openssh-server pdsh curl tar java -version Hadoop requires Java and SSH; pdsh is recommended by the current Apache
single-node documentation. Find JAVA_HOME if needed:
readlink -f "$(command -v java)" | sed 's:/bin/java::' Download and install
Pin the version for repeatable installs and verify Apache’s SHA-512 checksum:
export HADOOP_VERSION=3.5.0 cd /tmp curl -fLO "https://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz" curl -fLO "https://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz.sha512" sha512sum -c "hadoop-${HADOOP_VERSION}.tar.gz.sha512" sudo tar -xzf "hadoop-${HADOOP_VERSION}.tar.gz" -C /opt sudo ln -sfn "/opt/hadoop-${HADOOP_VERSION}" /opt/hadoop sudo chown -R "$USER":"$USER" "/opt/hadoop-${HADOOP_VERSION}" Add this to ~/.bashrc, adjusting JAVA_HOME if necessary:
export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 export HADOOP_HOME=/opt/hadoop export HADOOP_CONF_DIR="$HADOOP_HOME/etc/hadoop" export HADOOP_HDFS_HOME="$HADOOP_HOME" export HADOOP_YARN_HOME="$HADOOP_HOME" export HADOOP_MAPRED_HOME="$HADOOP_HOME" export PATH="$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin" Then load and verify it:
source ~/.bashrc sed -i "s|^# export JAVA_HOME=.*|export JAVA_HOME=${JAVA_HOME}|" "$HADOOP_HOME/etc/hadoop/hadoop-env.sh" hadoop version Configure pseudo-distributed mode
mkdir -p /opt/hadoop-data/{name,data} cat > "$HADOOP_HOME/etc/hadoop/core-site.xml" <<'EOF' <configuration> <property><name>fs.defaultFS</name><value>hdfs://localhost:9000</value></property> </configuration> EOF cat > "$HADOOP_HOME/etc/hadoop/hdfs-site.xml" <<'EOF' <configuration> <property><name>dfs.replication</name><value>1</value></property> <property><name>dfs.namenode.name.dir</name><value>file:///opt/hadoop-data/name</value></property> <property><name>dfs.datanode.data.dir</name><value>file:///opt/hadoop-data/data</value></property> </configuration> EOF cat > "$HADOOP_HOME/etc/hadoop/mapred-site.xml" <<'EOF' <configuration> <property><name>mapreduce.framework.name</name><value>yarn</value></property> <property><name>mapreduce.application.classpath</name><value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value></property> </configuration> EOF cat > "$HADOOP_HOME/etc/hadoop/yarn-site.xml" <<'EOF' <configuration> <property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property> <property><name>yarn.nodemanager.env-whitelist</name><value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME,HADOOP_HOME,HADOOP_MAPRED_HOME,PATH,LANG,TZ</value></property> </configuration> EOF The data paths may be moved to a separate disk. Do not use /tmp for production
data.
Enable local SSH and start Hadoop
sudo systemctl enable --now ssh test -f ~/.ssh/id_ed25519 || ssh-keygen -t ed25519 -N '' -f ~/.ssh/id_ed25519 cat ~/.ssh/id_ed25519.pub >> ~/.ssh/authorized_keys chmod 700 ~/.ssh chmod 600 ~/.ssh/authorized_keys ssh localhost exit Formatting destroys NameNode metadata. Run this only for a new test cluster:
hdfs namenode -format start-dfs.sh start-yarn.sh Web interfaces:
- NameNode: http://localhost:9870/
- ResourceManager: http://localhost:8088/
Run a test MapReduce job:
hdfs dfs -mkdir -p "/user/$USER" input hdfs dfs -put "$HADOOP_HOME/etc/hadoop"/*.xml input hadoop jar "$HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-${HADOOP_VERSION}.jar" grep input output 'dfs[a-z.]+' hdfs dfs -cat output/* Check and stop services:
jps hdfs dfsadmin -report stop-yarn.sh stop-dfs.sh Logs are normally in $HADOOP_HOME/logs.
Docker installation
Docker avoids installing Java and Hadoop on the host. Apache publishes
apache/hadoop:3.5.0:
docker pull apache/hadoop:3.5.0 docker run --rm -it --name hadoop --hostname hadoop apache/hadoop:3.5.0 bash For HDFS/YARN as multiple containers, use Apache’s Compose setup. It is
intended for local testing and can be scaled with DataNodes:
git clone --depth 1 --branch branch-3.5 https://github.com/apache/hadoop.git cd hadoop mvn clean install -Dmaven.javadoc.skip=true -DskipTests -DskipShade -Pdist,src cd hadoop-dist/target/hadoop-3.5.0/compose/hadoop docker compose up -d --scale datanode=3 docker compose ps docker compose exec datanode hdfs dfs -mkdir -p /test docker compose exec datanode hdfs dfs -ls / docker compose down The Compose setup is preferable for testing a real NameNode/DataNode topology.
Use explicit persistent volumes for NameNode metadata and DataNode storage if
data must survive container recreation. A single disposable container is not a
production cluster.
Resource sizing
The PDF’s example for 4 cores, 4 GB RAM, and two disks reserves 1 GB for the
system and calculates two 512 MB containers using:
containers = min(2 * cores, 1.8 * disks, available RAM / minimum container size) This is only a planning example. Do not configure more YARN memory than the
machine or Docker VM has; leave room for Hadoop JVMs and the operating system.
Common abbreviations in the Hadoop guide:
| Term | Meaning |
|---|---|
| HDFS | Hadoop Distributed File System |
| YARN | Yet Another Resource Negotiator |
| SSH | Secure Shell |
| JVM | Java Virtual Machine |
| XML | Extensible Markup Language |
| RPC | Remote Procedure Call |
| UI | User Interface |
| HTTP | Hypertext Transfer Protocol |
| CPU | Central Processing Unit |
| RAM | Random Access Memory |
| OS | Operating System |
| VM | Virtual Machine |
| PPA | Personal Package Archive |
| SHA-512 | Secure Hash Algorithm with a 512-bit digest |
Some Hadoop components are names rather than abbreviations:
- MapReduce — Hadoop’s distributed data-processing model.
- NameNode — manages HDFS metadata.
- DataNode — stores HDFS data blocks.
- ResourceManager — manages cluster resources.
- NodeManager — manages resources on an individual node.