Hadoop安装与配置（超详细，适合新手）_hadoop安装配置指令及配置文件内容

首先需要先下载一个VmWare虚拟机，接下来的是进行Hadoop的安装与配置。

1、修改网络配置内容

vi /etc/sysconfig/network-scripts/ifcfg-ens33

修改内容为：

（此处需要根据你具体的IP进行更改）

IPADDR=”192.168.127.104”

GATEWAY=”192.168.127.1”

DNS1=8.8.8.8

DNS2=192.168.188.2

2、重启网卡

systemctl restart network

3、关闭selinux(内核)

vi /etc/sysconfig/selinux

修改：SELINUX=disabled

4、关闭防火墙

systemctl stop firewalld.service

5、禁用防火墙

systemctl disable firewalld.service

6、修改hosts配置

vi /etc/hosts

添加：

（此处需要根据你具体的IP进行更改）

192.168.127.104 node1

192.168.127.105 node2

192.168.127.106 node3

测试：ping node1

7、时间同步

yum install ntp -y

[如果有报错，可以尝试以下代码： ps -ef | grep “yum”    kill 进程号  yum install ntp -y]

ststemctl enable ntpd

ln -sf /usr/share/zoreinfo/Asia/Shanghai/etc/localtime

ntpdate -u ntp1.aliyun.com

测试：date

8、设置主机名和网卡

hostnamectl set-hostname node1

hostnamectl set-hostname node2

hostnamectl set-hostname node3

9、服务器之间设置免密登录（免交互）

ssh -keygen -t rsa

cat /root/.ssh/id-rsa.pub

ssh -copy -id -i node2

ssh -copy -id -i node3

10、在笔记本上，输入Win+R,接着输入PFSTP(PSFTP是进行远程传输文件的工具)

在PFSTP窗口，输入open root@192.168.127.104（此处进入的是node1，需要根据你具体的IP进行更改）

接着，再上传hadoop的文件压缩包。

put 文件路径（hadoop -3.3.4.tar.gz/jdk -8u151 -linux -x64.tar.gz）

11、在node1结点下进行解压Hadoop压缩包。

mkdir program

mv hadoop -3.3.4.tar.gz program

mv jdk -8u151 -linux -x64.tar.gz program

cd /program

tar -zxvf hadoop -3.3.4.tar.gz

tar -zxvf jdk -8u151 -linux -x64.tar.gz

12、将Hadoop加入环境变量

vi /etc/profile

HADOOP_HOME=/hadoop

JAVA_HOME=/jdk1.8.0_151

PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$JAVA_HOME/bin

export PATH HADOOP_HOEM JAVA_HOME

13、环境变量生效

source /etc/profile

14、将jdk加入环境变量

vi /program/hadoop-3.3.4/etc/hadoop/hadoop-env.sh

export JAVA_HOME=/jdk1.8.0_151

15、将pragram下的所有文件全部移动到根目录下面。

mv program/* /

16、修改hadoop安装目录 /hadoop-3.3.4/etc/hadoop下的core-site.xml

vi core-site.xml

<configuration>

       <property>

         <name>fs.defaultFS</name>

         <value>hdfs://node1:9002</value>

     </property>

     <property>

         <name>hadoop.tmp.dir</name>

         <value>/data/hadoop/tmp</value>

     </property>

</configuration>

17、修改hadoop安装目录 /hadoop-3.3.4/etc/hadoop下的hdfs-site.xml\

vi hdfs-site.xml

<property>

       <name>dfs.namenode.http-address</name>

       <value>node1:9870</value>

</property>

 <property>

        <name>dfs.namenode.secondary.http-address</name>

        <value>node3:50090</value>

   </property>

18、配置slaves

vim /data/hadoop/etc/hadoop/slaves

添加内容：

node1

node2

node3

19、配置yarn-site.xml文件

vi yarn-site.xml

配置内容如下：

 <property>

                <name>yarn.nodemanager.aux-services</name>

                <value>mapreduce_shuffle</value>

        </property>

        <property>

                <name>yarn.resourcemanager.hostname</name>

                <value>node2</value>

        </property>

<property>

         <name>yarn.nodemanager.env-whitelist</name>       

<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME

</value>

</property>

<property>

                <name>yarn.application.classpath</name>

                <value>

/data/hadoop/etc/hadoop/*,

                        /data/hadoop/share/hadoop/common/lib/*,

                        /data/hadoop/share/hadoop/common/*,

                        /data/hadoop/share/hadoop/hdfs/*,

                        /data/hadoop/share/hadoop/hdfs/lib/*,

                        /data/hadoop/share/hadoop/hdfs/*,

                        /data/hadoop/share/hadoop/mapreduce/*,

                        /data/hadoop/share/hadoop/yarn/*,

                        /data/hadoop/share/hadoop/yarn/lib/*,

                        /data/hadoop/share/hadoop/yarn/*

</value>

        </property>

<property>

                <name>yarn.log-aggregation-enable</name>

                <value>true</value>

        </property>

        <property>

                <name>yarn.log.server.url</name>

                <value>http://node2:19888/jobhistory/logs</value>

        </property>

        <property>

                <name>yarn.log-aggregation.retain-seconds</name>

                <value>604800</value>

        </property>

        <property>

                <name>yarn.nodemanager.vmem-check-enabled</name>

                <value>false</value>

        </property>

20、配置mapReaduce

vi mapred-site.xml

添加内容：

  <property>

                <name>mapreduce.framework.name</name>

                <value>yarn</value>

        </property>

        <property>

<name>mapreduce.application.classpath</name>

<value>/data/hadoop/share/hadoop/mapreduce/*:/data/hadoop /share/hadoop/mapreduce/lib/*</value>

        </property>

        <property>

                <name>mapreduce.jobhistory.address</name>

                <value>node1:10020</value>

        </property>

      

        <property>

                <name>mapreduce.jobhistory.webapp.address</name>

                <value>node1:19888</value>

        </property>

21、在其他两台机子上也分发Hadoop配置

scp -r /hadoop-3.3.4/* node2:/

scp -r /hadoop-3.3.4/* node3:/

scp -r /jdk1.8.0_151 node2:/

scp -r /jdk1.8.0_151 node3:/

scp -r /etc/profile node2:/

scp -r /etc/profile node3:/

22、在三台虚拟机中各自的/etc/profile文件中添加内容

在各节点添加/etc/profile文件中添加内容：

export HDFS_NAMENODE_USER=root

export HDFS_DATANODE_USER=root

export HDFS_SECONDARYNAMENODE_USER=root

export YARN_RESOURCEMANAGER_USER=root

export YARN_NODEMANAGER_USER=root

1)在node1启动HDFS，命令：/data/hadoop/sbin/start-dfs.sh。

2)在node2启动YARN, 命令：/data/hadoop/sbin/start-yarn.sh。

3)在node2启动ResourceManager, 命令：/data/hadoop/bin/yarn --daemon start resourcemanager。

4)在node1上启动日志服务器，命令：/data/hadoop/bin/mapred --daemon start historyserver。

23、设置免密登录

ssh -copy -id -i /root/ssh.id_rsa.pub root@localhost

24、进行三个结点的网址访问

（此处需要根据自己具体的ip）

http://192.168.127.104:9870

http://192.168.127.105:8088

http://192.168.127.106:50090