Hadoop之YARN安装部署

版本信息：

Hadoop 2.3.0-cdh5.0.0

节点分布：

NameNode：compute-50-04 
SecondaryNameNode：compute-50-04  
ResourceManager ：compute-50-03 
NodeManager ：
compute-28-16
compute-28-17
compute-50-00
compute-50-03
compute-50-04
DataNode：
compute-28-16
compute-28-17
compute-50-00
compute-50-03
compute-50-04

部署：

1.增加hadoop用户，组也是hadoop组，详情参见“linux之用户组分配”

2.配置节点之间的双向ssh免密登陆，参见博客“linux免密钥SSH登陆配置”

3.官方下载hadoop 2.3.0-cdh5.0.0包，解压路径为：/home/hadoop/hadoop 2.3.0-cdh5.0.0

4.配置core-site.xml:

 

<configuration> 
  <property> 
    <name>fs.default.name</name>  
    <value>hdfs://compute-50-04:9000</value> 
  </property>  
  <property> 
    <name>hadoop.tmp.dir</name>  
    <value>/home/hadoop/data/tmp</value> 
  </property>  
  <property> 
    <name>ha.zookeeper.quorum</name>  
    <value>compute-28-16:2181,compute-28-17:2181,compute-50-00:2181</value> 
  </property>  
  <property> 
    <name>hadoop.proxyuser.hduser.hosts</name>  
    <value>*</value> 
  </property>  
  <property> 
    <name>hadoop.proxyuser.hduser.groups</name>  
    <value>*</value> 
  </property> 
</configuration>

5.配置hdfs-site.xml：

<configuration> 
  <property> 
    <name>dfs.replication</name>  
    <value>3</value> 
  </property>  
  <property> 
    <name>dfs.namenode.secondary.http-address</name>  
    <value>compute-50-03:9001</value> 
  </property>  
  <property> 
    <name>dfs.ha.fencing.methods</name>  
    <value>shell(/bin/true)</value> 
  </property>  
  <property> 
    <name>dfs.namenode.name.dir</name>  
    <value>/home/hadoop/data/dfs/nn</value> 
  </property>  
  <property> 
    <name>dfs.data.dir</name>  
    <value>/home/hadoop/data/dfs/dn</value> 
  </property>  
  <property> 
    <name>dfs.datanode.failed.volumes.tolerated</name>  
    <value>0</value> 
  </property>  
  <property> 
    <name>ipc.client.ping</name>  
    <value>false</value> 
  </property>  
  <property> 
    <name>ipc.ping.interval</name>  
    <value>60000</value> 
  </property>  
  <property> 
    <name>dfs.webhdfs.enabled</name>  
    <value>true</value> 
  </property>  
  <property> 
    <name>dfs.client.read.shortcircuit</name>  
    <value>false</value> 
  </property>  
  <property> 
    <name>dfs.permissions.enabled</name>  
    <value>false</value> 
  </property>  
  <property> 
    <name>dfs.domain.socket.path</name>  
    <value>${hadoop.tmp.dir}/sockets/dn._PORT</value> 
  </property> 
</configuration>

6.配置mapred-site.xml：

<configuration> 
  <property> 
    <name>mapreduce.framework.name</name>  
    <value>yarn</value> 
  </property>  
  <property> 
    <name>mapreduce.jobhistory.address</name>  
    <value>compute-50-00:10020</value> 
  </property>  
  <property> 
    <name>mapreduce.jobhistory.webapp.address</name>  
    <value>compute-50-00:19888</value> 
  </property>  
  <property> 
    <name>mapreduce.jobhistory.intermediate-done-dir</name>  
    <value>/data2/data/mr/history-tmp</value> 
  </property>  
  <property> 
    <name>mapreduce.jobhistory.done-dir</name>  
    <value>/data2/data/mr/history-done</value> 
  </property>  
  <property> 
    <name>yarn.app.mapreduce.am.staging-dir</name>  
    <value>/user</value> 
  </property>  
  <property> 
    <name>mapreduce.map.memory.mb</name>  
    <value>2048</value> 
  </property>  
  <property> 
    <name>mapreduce.map.speculative</name>  
    <value>false</value> 
  </property>  
  <property> 
    <name>mapreduce.job.queuename</name>  
    <value>default</value> 
  </property>  
  <!-- acl -->  
  <property> 
    <name>mapreduce.cluster.acls.enabled</name>  
    <value>false</value> 
  </property>  
  <property> 
    <name>mapreduce.job.acl-view-job</name>  
    <value></value> 
  </property>  
  <property> 
    <name>mapreduce.job.acl-modify-job</name>  
    <value></value> 
  </property>  
</configuration>

7.配置yarn-site.xml：

<configuration> 
  <property> 
    <name>yarn.nodemanager.aux-services</name>  
    <value>mapreduce_shuffle</value> 
  </property>  
  <property> 
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>  
    <value>org.apache.hadoop.mapred.ShuffleHandler</value> 
  </property>  
  <property> 
    <name>yarn.resourcemanager.hostname</name>  
    <value>compute-50-04</value> 
  </property>  
  <property> 
    <name>yarn.nodemanager.local-dirs</name>  
    <value>/data2/data/yarn/local</value> 
  </property>  
  <property> 
    <name>yarn.nodemanager.vmem-pmem-ratio</name>  
    <value>10</value> 
  </property>  
  <property> 
    <name>yarn.nodemanager.container-executor.class</name>  
    <value>org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor</value> 
  </property>  
  <property> 
    <name>yarn.log-aggregation-enable</name>  
    <value>false</value> 
  </property>  
  <property> 
    <name>yarn.resourcemanager.scheduler.class</name>  
    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value> 
  </property>  
  <property> 
    <name>yarn.nodemanager.vmem-check-enabled</name>  
    <value>false</value> 
  </property>  
  <property> 
    <name>yarn.resourcemanager.scheduler.monitor.enable</name>  
    <value>true</value> 
  </property>  
  <property> 
    <name>yarn.resourcemanager.scheduler.monitor.policies</name>  
    <value>org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy</value> 
  </property>  
  <property> 
    <name>yarn.resourcemanager.monitor.capacity.preemption.observe_only</name>  
    <value>false</value> 
  </property>  
  <property> 
    <name>yarn.nodemanager.resource.memory-mb</name>  
    <value>20720</value> 
  </property>  
  <property> 
    <name>yarn.acl.enable</name>  
    <value>false</value> 
  </property>  
  <property> 
    <name>yarn.admin.acl</name>  
    <value>yarn,hadoop</value> 
  </property>  
  <property> 
    <name>yarn.nodemanager.resource.cpu-vcores</name>  
    <value>8</value> 
  </property> 
</configuration>

8.配置slaves：

compute-28-16
compute-28-17
compute-50-00
compute-50-03
compute-50-04

 9.配置信息已经完成，那么建立相应的目录：

mkdir -p yourpath

10.分发到各个节点上去：

scp   -r /home/hadoop/hadoop-2.3.0-cdh5.0.0  hadoop@hostxxxxx:/home/hadoop/
......

11.格式化namenode：

bin/hadoop namenode -format 

12.启动：

sbin/start-all.sh

 

其他问题：

1.启动的时候报错：Error: JAVA_HOME is not set and could not be found

修复方式：

在etc/hadoop/yarn-env.sh 和etc/hadoop/hadoop-env.sh中显示指定 

export JAVA_HOME=/usr/java/jdk1.6.0_21

 

其他说明：

1.NameNode基于SecondaryNameNode没有HA

2.ResourceManager为单节点没有HA