Hadoop Install Guide

Prerequirement

OS: Linux Memory: 4G

package requirement

Hadoop require openssh-server, ssh, pdsh

$apt update
$apt install openssh-server ssh pdsh -y

Java prepare

Download java from Orcale direct link

wget https://download.oracle.com/java/21/latest/jdk-21_linux-x64_bin.deb
sudo dpkg -i jdk-21_linux-x64_bin.deb

Hadoop prepare

Download Hadoop here direct link

wget https://dlcdn.apache.org/hadoop/common/hadoop-3.5.0/hadoop-3.5.0.tar.gz
tar -zxvf hadoop-3.5.0.tar.gz

Configure Hadoop

cd hadoop-3.5.0
export JAVA_HOME=/usr/java/jdk-21-oracle-x64

Configure remote ssh access

Connect to server first

ssh localhost

Generate ssh key

$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ chmod 0600 ~/.ssh/authorized_keys

Configure Hadoop environment

# hadoop-3.5.0/etc/hadoop/hadoop-env.sh
JAVA_HOME=/usr/java/jdk-21-oracle-x64

# hadoop-3.5.0/etc/hadoop/core-site.xml
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

# hadoop-3.5.0/etc/hadoop/hdfs-site.xml
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

# hadoop-3.5.0/etc/hadoop/mapred-site.xml
<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.application.classpath</name>
        <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
    </property>
</configuration>

# hadoop-3.5.0/etc/hadoop/yarn-site.xml
<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME</value>
    </property>
</configuration>

Format hadoop

bin/hdfs namenode -format

Startup all hadoop services.

sbin/start-all.sh

2.5 KiB Raw Permalink Blame History