{"id":4031,"date":"2026-08-02T04:03:57","date_gmt":"2026-08-02T04:03:57","guid":{"rendered":"https:\/\/tucumandevelopers.com\/index.php\/2026\/08\/02\/apache-hadoop-installation\/"},"modified":"2026-08-02T04:03:57","modified_gmt":"2026-08-02T04:03:57","slug":"apache-hadoop-installation","status":"publish","type":"post","link":"https:\/\/tucumandevelopers.com\/index.php\/2026\/08\/02\/apache-hadoop-installation\/","title":{"rendered":"Apache Hadoop Installation"},"content":{"rendered":"<div>\n<div>\n<div data-article-id=\"4290886\" id=\"article-body\">\n<p>This guide is a collection or a summary on how to install and use a footprint of Apache Hadoop. I tried to follow an old version 2.7.1 guide that I created few years ago and adjusted this to use the latest version. Apache Hadoop 3.5.0 is used below; check the <a href=\"https:\/\/hadoop.apache.org\/releases\/\" target=\"_blank\" rel=\"noopener noreferrer\">Apache releases page<\/a> before future<br \/> installations.<\/p>\n<p>These instructions target Linux (Ubuntu\/Debian) for development or testing.<br \/> Production clusters need Kerberos, network controls, encryption, monitoring,<br \/> backups, and an upgrade plan. Do not expose HDFS or YARN ports to the internet.<\/p>\n<h2> <a name=\"native-singlenode-installation\" href=\"#native-singlenode-installation\"> <\/a> Native single-node installation <\/h2>\n<h3> <a name=\"prerequisites\" href=\"#prerequisites\"> <\/a> Prerequisites <\/h3>\n<div>\n<pre><code>sudo apt-get update sudo apt-get install -y openjdk-17-jdk openssh-client openssh-server pdsh curl tar java -version <\/code><\/pre>\n<div>\n<\/p><\/div>\n<\/p><\/div>\n<p>Hadoop requires Java and SSH; pdsh is recommended by the current Apache<br \/> single-node documentation. Find JAVA_HOME if needed:<\/p>\n<div>\n<pre><code>readlink -f \"$(command -v java)\" | sed 's:\/bin\/java::' <\/code><\/pre>\n<div>\n<\/p><\/div>\n<\/p><\/div>\n<h3> <a name=\"download-and-install\" href=\"#download-and-install\"> <\/a> Download and install <\/h3>\n<p>Pin the version for repeatable installs and verify Apache&#8217;s SHA-512 checksum:<\/p>\n<div>\n<pre><code>export HADOOP_VERSION=3.5.0 cd \/tmp curl -fLO \"https:\/\/archive.apache.org\/dist\/hadoop\/common\/hadoop-${HADOOP_VERSION}\/hadoop-${HADOOP_VERSION}.tar.gz\" curl -fLO \"https:\/\/archive.apache.org\/dist\/hadoop\/common\/hadoop-${HADOOP_VERSION}\/hadoop-${HADOOP_VERSION}.tar.gz.sha512\" sha512sum -c \"hadoop-${HADOOP_VERSION}.tar.gz.sha512\" sudo tar -xzf \"hadoop-${HADOOP_VERSION}.tar.gz\" -C \/opt sudo ln -sfn \"\/opt\/hadoop-${HADOOP_VERSION}\" \/opt\/hadoop sudo chown -R \"$USER\":\"$USER\" \"\/opt\/hadoop-${HADOOP_VERSION}\" <\/code><\/pre>\n<div>\n<\/p><\/div>\n<\/p><\/div>\n<p>Add this to ~\/.bashrc, adjusting JAVA_HOME if necessary:<\/p>\n<div>\n<pre><code>export JAVA_HOME=\/usr\/lib\/jvm\/java-17-openjdk-amd64 export HADOOP_HOME=\/opt\/hadoop export HADOOP_CONF_DIR=\"$HADOOP_HOME\/etc\/hadoop\" export HADOOP_HDFS_HOME=\"$HADOOP_HOME\" export HADOOP_YARN_HOME=\"$HADOOP_HOME\" export HADOOP_MAPRED_HOME=\"$HADOOP_HOME\" export PATH=\"$PATH:$HADOOP_HOME\/bin:$HADOOP_HOME\/sbin\" <\/code><\/pre>\n<div>\n<\/p><\/div>\n<\/p><\/div>\n<p>Then load and verify it:<\/p>\n<div>\n<pre><code>source ~\/.bashrc sed -i \"s|^# export JAVA_HOME=.*|export JAVA_HOME=${JAVA_HOME}|\" \"$HADOOP_HOME\/etc\/hadoop\/hadoop-env.sh\" hadoop version <\/code><\/pre>\n<div>\n<\/p><\/div>\n<\/p><\/div>\n<h3> <a name=\"configure-pseudodistributed-mode\" href=\"#configure-pseudodistributed-mode\"> <\/a> Configure pseudo-distributed mode <\/h3>\n<div>\n<pre><code>mkdir -p \/opt\/hadoop-data\/{name,data} cat &gt; \"$HADOOP_HOME\/etc\/hadoop\/core-site.xml\" &lt;&lt;'EOF' &lt;configuration&gt; &lt;property&gt;&lt;name&gt;fs.defaultFS&lt;\/name&gt;&lt;value&gt;hdfs:\/\/localhost:9000&lt;\/value&gt;&lt;\/property&gt; &lt;\/configuration&gt; EOF cat &gt; \"$HADOOP_HOME\/etc\/hadoop\/hdfs-site.xml\" &lt;&lt;'EOF' &lt;configuration&gt; &lt;property&gt;&lt;name&gt;dfs.replication&lt;\/name&gt;&lt;value&gt;1&lt;\/value&gt;&lt;\/property&gt; &lt;property&gt;&lt;name&gt;dfs.namenode.name.dir&lt;\/name&gt;&lt;value&gt;file:\/\/\/opt\/hadoop-data\/name&lt;\/value&gt;&lt;\/property&gt; &lt;property&gt;&lt;name&gt;dfs.datanode.data.dir&lt;\/name&gt;&lt;value&gt;file:\/\/\/opt\/hadoop-data\/data&lt;\/value&gt;&lt;\/property&gt; &lt;\/configuration&gt; EOF cat &gt; \"$HADOOP_HOME\/etc\/hadoop\/mapred-site.xml\" &lt;&lt;'EOF' &lt;configuration&gt; &lt;property&gt;&lt;name&gt;mapreduce.framework.name&lt;\/name&gt;&lt;value&gt;yarn&lt;\/value&gt;&lt;\/property&gt; &lt;property&gt;&lt;name&gt;mapreduce.application.classpath&lt;\/name&gt;&lt;value&gt;$HADOOP_MAPRED_HOME\/share\/hadoop\/mapreduce\/*:$HADOOP_MAPRED_HOME\/share\/hadoop\/mapreduce\/lib\/*&lt;\/value&gt;&lt;\/property&gt; &lt;\/configuration&gt; EOF cat &gt; \"$HADOOP_HOME\/etc\/hadoop\/yarn-site.xml\" &lt;&lt;'EOF' &lt;configuration&gt; &lt;property&gt;&lt;name&gt;yarn.nodemanager.aux-services&lt;\/name&gt;&lt;value&gt;mapreduce_shuffle&lt;\/value&gt;&lt;\/property&gt; &lt;property&gt;&lt;name&gt;yarn.nodemanager.env-whitelist&lt;\/name&gt;&lt;value&gt;JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME,HADOOP_HOME,HADOOP_MAPRED_HOME,PATH,LANG,TZ&lt;\/value&gt;&lt;\/property&gt; &lt;\/configuration&gt; EOF <\/code><\/pre>\n<div>\n<\/p><\/div>\n<\/p><\/div>\n<p>The data paths may be moved to a separate disk. Do not use \/tmp for production<br \/> data.<\/p>\n<h3> <a name=\"enable-local-ssh-and-start-hadoop\" href=\"#enable-local-ssh-and-start-hadoop\"> <\/a> Enable local SSH and start Hadoop <\/h3>\n<div>\n<pre><code>sudo systemctl enable --now ssh test -f ~\/.ssh\/id_ed25519 || ssh-keygen -t ed25519 -N '' -f ~\/.ssh\/id_ed25519 cat ~\/.ssh\/id_ed25519.pub &gt;&gt; ~\/.ssh\/authorized_keys chmod 700 ~\/.ssh chmod 600 ~\/.ssh\/authorized_keys ssh localhost exit <\/code><\/pre>\n<div>\n<\/p><\/div>\n<\/p><\/div>\n<p>Formatting destroys NameNode metadata. Run this only for a new test cluster:<\/p>\n<div>\n<pre><code>hdfs namenode -format start-dfs.sh start-yarn.sh <\/code><\/pre>\n<div>\n<\/p><\/div>\n<\/p><\/div>\n<p>Web interfaces:<\/p>\n<ul>\n<li>NameNode: <a href=\"http:\/\/localhost:9870\/\" target=\"_blank\" rel=\"noopener noreferrer\">http:\/\/localhost:9870\/<\/a> <\/li>\n<li>ResourceManager: <a href=\"http:\/\/localhost:8088\/\" target=\"_blank\" rel=\"noopener noreferrer\">http:\/\/localhost:8088\/<\/a> <\/li>\n<\/ul>\n<p>Run a test MapReduce job:<\/p>\n<div>\n<pre><code>hdfs dfs -mkdir -p \"\/user\/$USER\" input hdfs dfs -put \"$HADOOP_HOME\/etc\/hadoop\"\/*.xml input hadoop jar \"$HADOOP_HOME\/share\/hadoop\/mapreduce\/hadoop-mapreduce-examples-${HADOOP_VERSION}.jar\" grep input output 'dfs[a-z.]+' hdfs dfs -cat output\/* <\/code><\/pre>\n<div>\n<\/p><\/div>\n<\/p><\/div>\n<p>Check and stop services:<\/p>\n<div>\n<pre><code>jps hdfs dfsadmin -report stop-yarn.sh stop-dfs.sh <\/code><\/pre>\n<div>\n<\/p><\/div>\n<\/p><\/div>\n<p>Logs are normally in $HADOOP_HOME\/logs.<\/p>\n<h2> <a name=\"docker-installation\" href=\"#docker-installation\"> <\/a> Docker installation <\/h2>\n<p>Docker avoids installing Java and Hadoop on the host. Apache publishes<br \/> apache\/hadoop:3.5.0:<\/p>\n<div>\n<pre><code>docker pull apache\/hadoop:3.5.0 docker run --rm -it --name hadoop --hostname hadoop apache\/hadoop:3.5.0 bash <\/code><\/pre>\n<div>\n<\/p><\/div>\n<\/p><\/div>\n<p>For HDFS\/YARN as multiple containers, use Apache&#8217;s Compose setup. It is<br \/> intended for local testing and can be scaled with DataNodes:<\/p>\n<div>\n<pre><code>git clone --depth 1 --branch branch-3.5 https:\/\/github.com\/apache\/hadoop.git cd hadoop mvn clean install -Dmaven.javadoc.skip=true -DskipTests -DskipShade -Pdist,src cd hadoop-dist\/target\/hadoop-3.5.0\/compose\/hadoop docker compose up -d --scale datanode=3 docker compose ps docker compose exec datanode hdfs dfs -mkdir -p \/test docker compose exec datanode hdfs dfs -ls \/ docker compose down <\/code><\/pre>\n<div>\n<\/p><\/div>\n<\/p><\/div>\n<p>The Compose setup is preferable for testing a real NameNode\/DataNode topology.<br \/> Use explicit persistent volumes for NameNode metadata and DataNode storage if<br \/> data must survive container recreation. A single disposable container is not a<br \/> production cluster.<\/p>\n<h2> <a name=\"resource-sizing\" href=\"#resource-sizing\"> <\/a> Resource sizing <\/h2>\n<p>The PDF&#8217;s example for 4 cores, 4 GB RAM, and two disks reserves 1 GB for the<br \/> system and calculates two 512 MB containers using:<\/p>\n<div>\n<pre><code>containers = min(2 * cores, 1.8 * disks, available RAM \/ minimum container size) <\/code><\/pre>\n<div>\n<\/p><\/div>\n<\/p><\/div>\n<p>This is only a planning example. Do not configure more YARN memory than the<br \/> machine or Docker VM has; leave room for Hadoop JVMs and the operating system.<\/p>\n<h2> <a name=\"common-abbreviations-in-the-hadoop-guide\" href=\"#common-abbreviations-in-the-hadoop-guide\"> <\/a> Common abbreviations in the Hadoop guide: <\/h2>\n<div>\n<table>\n<thead>\n<tr>\n<th>Term<\/th>\n<th>Meaning<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>HDFS<\/td>\n<td>Hadoop Distributed File System<\/td>\n<\/tr>\n<tr>\n<td>YARN<\/td>\n<td>Yet Another Resource Negotiator<\/td>\n<\/tr>\n<tr>\n<td>SSH<\/td>\n<td>Secure Shell<\/td>\n<\/tr>\n<tr>\n<td>JVM<\/td>\n<td>Java Virtual Machine<\/td>\n<\/tr>\n<tr>\n<td>XML<\/td>\n<td>Extensible Markup Language<\/td>\n<\/tr>\n<tr>\n<td>RPC<\/td>\n<td>Remote Procedure Call<\/td>\n<\/tr>\n<tr>\n<td>UI<\/td>\n<td>User Interface<\/td>\n<\/tr>\n<tr>\n<td>HTTP<\/td>\n<td>Hypertext Transfer Protocol<\/td>\n<\/tr>\n<tr>\n<td>CPU<\/td>\n<td>Central Processing Unit<\/td>\n<\/tr>\n<tr>\n<td>RAM<\/td>\n<td>Random Access Memory<\/td>\n<\/tr>\n<tr>\n<td>OS<\/td>\n<td>Operating System<\/td>\n<\/tr>\n<tr>\n<td>VM<\/td>\n<td>Virtual Machine<\/td>\n<\/tr>\n<tr>\n<td>PPA<\/td>\n<td>Personal Package Archive<\/td>\n<\/tr>\n<tr>\n<td>SHA-512<\/td>\n<td>Secure Hash Algorithm with a 512-bit digest<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>Some Hadoop components are names rather than abbreviations:<\/p>\n<ul>\n<li>MapReduce \u2014 Hadoop\u2019s distributed data-processing model.<\/li>\n<li>NameNode \u2014 manages HDFS metadata.<\/li>\n<li>DataNode \u2014 stores HDFS data blocks.<\/li>\n<li>ResourceManager \u2014 manages cluster resources.<\/li>\n<li>NodeManager \u2014 manages resources on an individual node.<\/li>\n<\/ul><\/div>\n<\/div>\n<\/div>\n<\/div>\n<p>Fuente: <a href=\"https:\/\/dev.to\/limacon23\/apache-hadoop-installation-1ep6\">Art\u00edculo original<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>This guide is a collection or a summary on how to install and use a footprint of Apache Hadoop. I tried to follow an old version 2.7.1 guide that I created few years ago and adjusted this to use the latest version. Apache Hadoop 3.5.0 is used below; check the Apache releases page before future [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":2648,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[41],"tags":[],"class_list":["post-4031","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devto"],"jetpack_publicize_connections":[],"_links":{"self":[{"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/posts\/4031","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/comments?post=4031"}],"version-history":[{"count":0,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/posts\/4031\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/media\/2648"}],"wp:attachment":[{"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/media?parent=4031"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/categories?post=4031"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/tags?post=4031"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}