用途
此文档描述了如何安装、配置和维护一个重大集群从几个节点到上千节点。
初次接触hadoop建议先从单节点集群开始。
前提
从Apache 上下载了稳定的版本。
安装
安装hadoop集群通常需要在所有的节点上解压软件或者prm安装。
通常集群中的某一个节点被当做NameNode,其他节点作为ResourceManager,这些是主控节点。其他节点被当做DataNode和NodeManager,这些是从节点。
非安全模式启动Hadoop
接下来的章节将会阐述如何配置hadoop集群。
配置文件
hadoop中的配置文件有两大类型:
只读型默认配置:core-default.xml, hdfs-default.xml, yarn-default.xml and mapred-default.xml
定制化配置:conf/core-site.xml, conf/hdfs-site.xml, conf/yarn-site.xml and conf/mapred-site.xml.
此外:你可以自己操作hadoop的脚本,在bin目录下可以找到,还有一些配置的环境变量在conf/hadoop-env.sh and yarn-env.sh中。
站点配置:
配置hadoop集群你首先要配置hadoop守护进程执行的环境。
hadoop的守护进程包括NameNode/DataNode and ResourceManager/NodeManager.
hadoop守护进程环境配置:
管理员需要使用conf/hadoop-env.sh and conf/yarn-env.sh脚本对hadoop守护进程做环境配置。
首先你要验证JAVA_HOME在所有的节点上是否正确
有时候你需要 HADOOP_PID_DIR and HADOOP_SECURE_DN_PID_DIR目录只能被启动守护进程的用户执行写操作。否则就会出现软连接攻击。
管理员可以利用配置项单独配置进程,配置项如下:
NameNode | HADOOP_NAMENODE_OPTS |
DataNode | HADOOP_DATANODE_OPTS |
Secondary NameNode | HADOOP_SECONDARYNAMENODE_OPTS |
ResourceManager | YARN_RESOURCEMANAGER_OPTS |
NodeManager | YARN_NODEMANAGER_OPTS |
WebAppProxy | YARN_PROXYSERVER_OPTS |
Map Reduce Job History Server | HADOOP_JOB_HISTORYSERVER_OPTS |
如要配置Namenode 为parallelGC,那么可以添加如下到hadoop-env.sh中:
export HADOOP_NAMENODE_OPTS="-XX:+UseParallelGC ${HADOOP_NAMENODE_OPTS}"
其他有用的可定制化参数包括:
HADOOP_LOG_DIR / YARN_LOG_DIR :进程日志目录,如果不存在会自动创建。
HADOOP_HEAPSIZE / YARN_HEAPSIZE:内存堆大小默认单位为M,如果变量设置成1000 那么堆内存会设置成1000M,默认为1000,如果你需要配置他那么你可以为每个节点单独配置。
ResourceManager | YARN_RESOURCEMANAGER_HEAPSIZE |
NodeManager | YARN_NODEMANAGER_HEAPSIZE |
WebAppProxy | YARN_PROXYSERVER_HEAPSIZE |
Map Reduce Job History Server | HADOOP_JOB_HISTORYSERVER_HEAPSIZE |
hadoop守护进程非安全模式配置:
此章节是比较重要的参数配置,涉及信息如下:
conf/core-site.xml
fs.defaultFS | NameNode URI | hdfs://host:port/ |
io.file.buffer.size | 131072 | SequenceFiles的读/写缓冲区大小 |
conf/hdfs-site.xml
NameNode的配置:
dfs.namenode.name.dir | Path on the local filesystem where the NameNode stores the namespace and transactions logs persistently. | If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy. |
dfs.namenode.hosts /dfs.namenode.hosts.exclude | List of permitted/excluded DataNodes. | If necessary, use these files to control the list of allowable datanodes. |
dfs.blocksize | 268435456 | HDFS blocksize of 256MB for large file-systems. |
dfs.namenode.handler.count | 100 | More NameNode server threads to handle RPCs from large number of DataNodes. |
DataNode配置:
dfs.datanode.data.dir | Comma separated list of paths on the local filesystem of a DataNode where it should store its blocks. | If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. |
conf/yarn-site.xml
ResourceManager和NodeManager配置:
yarn.acl.enable | true /false | Enable ACLs? Defaults to false. |
yarn.admin.acl | Admin ACL | ACL to set admins on the cluster. ACLs are of for comma-separated-usersspacecomma-separated-groups. Defaults to special value of * which means anyone. Special value of just space means no one has access. |
yarn.log-aggregation-enable | false | Configuration to enable or disable log aggregation |
ResourceManager配置:
yarn.resourcemanager.address | ResourceManager host:port for clients to submit jobs. | host:port |
yarn.resourcemanager.scheduler.address | ResourceManager host:port for ApplicationMasters to talk to Scheduler to obtain resources. | host:port |
yarn.resourcemanager.resource-tracker.address | ResourceManager host:port for NodeManagers. | host:port |
yarn.resourcemanager.admin.address | ResourceManager host:port for administrative commands. | host:port |
yarn.resourcemanager.webapp.address | ResourceManager web-ui host:port. | host:port |
yarn.resourcemanager.scheduler.class | ResourceManager Scheduler class. | CapacityScheduler (recommended), FairScheduler(also recommended), or FifoScheduler |
yarn.scheduler.minimum-allocation-mb | Minimum limit of memory to allocate to each container request at the Resource Manager. | In MBs |
yarn.scheduler.maximum-allocation-mb | Maximum limit of memory to allocate to each container request at the Resource Manager. | In MBs |
yarn.resourcemanager.nodes.include-path /yarn.resourcemanager.nodes.exclude-path | List of permitted/excluded NodeManagers. | If necessary, use these files to control the list of allowable NodeManagers. |
NodeManager配置:
yarn.nodemanager.resource.memory-mb | Resource i.e. available physical memory, in MB, for givenNodeManager | Defines total available resources on the NodeManager to be made available to running containers |
yarn.nodemanager.vmem-pmem-ratio | Maximum ratio by which virtual memory usage of tasks may exceed physical memory | The virtual memory usage of each task may exceed its physical memory limit by this ratio. The total amount of virtual memory used by tasks on the NodeManager may exceed its physical memory usage by this ratio. |
yarn.nodemanager.local-dirs | Comma-separated list of paths on the local filesystem where intermediate data is written. | Multiple paths help spread disk i/o. |
yarn.nodemanager.log-dirs | Comma-separated list of paths on the local filesystem where logs are written. | Multiple paths help spread disk i/o. |
yarn.nodemanager.log.retain-seconds | 10800 | Default time (in seconds) to retain log files on the NodeManager Only applicable if log-aggregation is disabled. |
yarn.nodemanager.remote-app-log-dir | /logs | HDFS directory where the application logs are moved on application completion. Need to set appropriate permissions. Only applicable if log-aggregation is enabled. |
yarn.nodemanager.remote-app-log-dir-suffix | logs | Suffix appended to the remote log dir. Logs will be aggregated to ${yarn.nodemanager.remote-app-log-dir}/${user}/${thisParam} Only applicable if log-aggregation is enabled. |
yarn.nodemanager.aux-services | mapreduce_shuffle | Shuffle service that needs to be set for Map Reduce applications. |
运行历史配置:
yarn.log-aggregation.retain-seconds | -1 | How long to keep aggregation logs before deleting them. -1 disables. Be careful, set this too small and you will spam the name node. |
yarn.log-aggregation.retain-check-interval-seconds | -1 | Time between checks for aggregated log retention. If set to 0 or a negative value then the value is computed as one-tenth of the aggregated log retention time. Be careful, set this too small and you will spam the name node. |
conf/mapred-site.xml
MapReduce应用配置:
mapreduce.framework.name | yarn | Execution framework set to Hadoop YARN. |
mapreduce.map.memory.mb | 1536 | Larger resource limit for maps. |
mapreduce.map.java.opts | -Xmx1024M | Larger heap-size for child jvms of maps. |
mapreduce.reduce.memory.mb | 3072 | Larger resource limit for reduces. |
mapreduce.reduce.java.opts | -Xmx2560M | Larger heap-size for child jvms of reduces. |
mapreduce.task.io.sort.mb | 512 | Higher memory-limit while sorting data for efficiency. |
mapreduce.task.io.sort.factor | 100 | More streams merged at once while sorting files. |
mapreduce.reduce.shuffle.parallelcopies | 50 | Higher number of parallel copies run by reduces to fetch outputs from very large number of maps. |
MapReduce 执行历史服务配置:
mapreduce.jobhistory.address | MapReduce JobHistory Server host:port | Default port is 10020. |
mapreduce.jobhistory.webapp.address | MapReduce JobHistory Server Web UIhost:port | Default port is 19888. |
mapreduce.jobhistory.intermediate-done-dir | /mr-history/tmp | Directory where history files are written by MapReduce jobs. |
mapreduce.jobhistory.done-dir | /mr-history/done | Directory where history files are managed by the MR JobHistory Server. |
Hadoop机架感知
HDFS和YARN服务可机架感知的
NameNode 和ResourceManager通过调用api来获取集群中每个从节点的机架信息。
api以dns名称(或ip)作为一个机架id
这个模块也是可配置的,通过topology.node.switch.mapping.impl来配置,可以通过命令行参数topology.script.file.name来配置,如果topology.script.file.name没有配置那么默认其ip为机架id。
相关推荐
本手册主要整理了Hadoop2.2.0的安装,包括QJM实现HA的详细配置,以及Hdfs-site的配置详解。 不包括Hbase、Hive安装
资源名称:CentOS 6.5 x64下安装19实体节点Hadoop 2.2.0集群配置指南内容简介: CentOS 6.5 x64下安装19实体节点Hadoop 2.2.0集群配置指南主要讲述的是CentOS 6.5 x64下安装19实体节点Hadoop 2.2.0集群配置指南;...
hadoop2.2.0集群搭建
Hadoop2.2.0安装配置手册!完全分布式Hadoop集群搭建过程 按照文档中的操作步骤,一步步操作就可以完全实现hadoop2.2.0版本的完全分布式集群搭建过程
Hadoop2.2.0安装配置手册,新手安装和配置
Hadoop 2.2.0版本中在64为linux系统上运行所需要的native库文件。需要时用此native文件夹覆盖Hadoop 2.2.0中native文件夹即可。
Hadoop 2.2.0 配置文件 在4台CentOs 6.4版本下运行成功
Hadoop 2.2.0 分布式安装指导
Hadoop2.2.0版本 - 虚拟机VMWare - Linux(ubuntu) ,多节点伪分布环境搭建详细操作手册,可以直接用于完全分布。每一步都有自己详细操作记录和相关说明,几夜风雨的辛劳搭建和测试。 此操作手册搭建的集群详细说明: ...
自己配了一遍hadoop2.2.0,写给新手,绝对详细,后面还有一个配好了的测试文档,在我的资源里找
本文档主要详细介绍了Hadoop 2.2.0版本的编译安装过程。
Hadoop2.2.0安装配置手册!完全分布式Hadoop集群搭建过程.
hadoop2.2.0安装指南
hadoop2.2.0/2.6.0/2.7.0/2.7.1 64位安装包。
Hadoop官网上下载的hadoop-2.2.0安装包是32位的,直接运行在64位的linux系统上会有问题,所以需要自己将hadoop-2.2.0安装包编译为64位。
在CentSO_64bit集群搭建,hadoop2.2(64位)编译 新版亮点: 基于yarn计算框架和高可用性DFS的第一个稳定版本。 注1:官网只提供32位release版本, 若机器为64位,需要手动编译。 环境配置是个挺烦人的活,麻烦不说还...