安装配置Hadoop+Zookeeper+HBase 环境简单说明.docx
- 文档编号:5751725
- 上传时间:2022-12-31
- 格式:DOCX
- 页数:17
- 大小:455.90KB
安装配置Hadoop+Zookeeper+HBase 环境简单说明.docx
《安装配置Hadoop+Zookeeper+HBase 环境简单说明.docx》由会员分享,可在线阅读,更多相关《安装配置Hadoop+Zookeeper+HBase 环境简单说明.docx(17页珍藏版)》请在冰豆网上搜索。
安装配置Hadoop+Zookeeper+HBase环境简单说明
安装配置Hadoop+Zookeeper+HBase环境简单说明(2013)
一、机器:
集群环境有6台服务器:
1个Master,5个Slave,配置节点IP分配如下:
Hostname
IP
备注
Node0
192,168.7.80
Master,namenode,JobTracker,Hmaster
Node1
192,168.7.234
Slave,datanode,TaskTracker,Hregionserver,zookeeper
Node2
192,168.7.248
slave,datanode,TaskTracker,Hregionserver,zookeeper
Node3
192,168.7.249
slave,datanode,TaskTracker,Hregionserver,zookeeper
Node4
192,168.7.77
slave,datanode,TaskTracker,Hregionserver,zookeeper
Node5
192,168.7.84
slave,datanode,TaskTracker,Hregionserver,zookeeper
二、软件
1操作系统:
CentOSrelease5.3
2相关软件:
java-1.6.0-openjdk-1.6.0.0(CENTOS5.3随机)
hadoop-1.2.0.tar.gz
hbase-0.95.1-hadoop1-bin.tar.gz
zookeeper-3.4.5.tar.gz
三、编辑/etc/hosts(所有机器)
Node0192.168.7.80
Node1192.168.7.234
Node2192.168.7.248
Node3192.168.7.249
Node4192.168.7.77
Node5192.168.7.84
所有机器建立hadoop用户名和相同的密码
useraddhadoop
passwdhadoop
关闭防火墙
重启所有机器
四、无密码验证配置(在node0机器上)
#ssh-keygen–trsa–P‘’–f/root/.ssh/id_rsa
#cat/root/.ssh/id_rsa.pub>>/root/.ssh/authorized_keys
将id_rsa.pub拷贝到所有机器上
scp/root/.ssh/id_rsa.pubnode1:
/root/.ssh/authorized_keys
scp/root/.ssh/id_rsa.pubnode2:
/root/.ssh/authorized_keys
scp/root/.ssh/id_rsa.pubnode3:
/root/.ssh/authorized_keys
scp/root/.ssh/id_rsa.pubnode4:
/root/.ssh/authorized_keys
scp/root/.ssh/id_rsa.pubnode5:
/root/.ssh/authorized_keys
ssh所有机器第一次需要确认一下密码,以后就不用再确认密码了。
五、安装所有软件(在node0机器上)
6个节点均使用centos5.3系统,为了便于维护,集群环境使用相同用户、密码、安装路径,安装过程及路径见下:
[root@node0hadoop]#cd/home
[root@node0home]#cdhadoop
[root@node0hadoop]#pwd
/home/hadoop/
安装hadoop-1.2.0
tar-zxvfhadoop-1.2.0.tar.gz
安装hbase-0.95.1-hadoop1
tar-zxvfhbase-0.95.1-hadoop1-bin.tar.gz
安装zookeeper-3.4.5
tar-zxvfzookeeper-3.4.5.tar.gz
建立hadoop-1.2.0工作目录
mkdirtmp
建立hbase-0.95.1-hadoop1工作目录
mkdirhbase
建立zookeeper-3.4.5工作目录
mkdirzookeeper
查看软件目录
[root@node0hadoop]#dir-l
total40
drwxr-xr-x15rootroot4096Aug1322:
41hadoop-1.2.0
drwxr-xr-x3rootroot4096Aug2214:
39tmp
drwxr-xr-x8rootroot4096Aug1618:
23hbase-0.95.1-hadoop1
drwxr-xr-x3rootroot4096Aug2214:
44hbase
drwxr-xr-x10rootroot4096Nov52012zookeeper-3.4.5
drwxr-xr-x2rootroot4096Aug2213:
35zookeeper
将所有的软件及个工作目录拷贝到其它五台机器上
scp-rhadoop-1.2.0tmphbase*zookeeper*node1:
/home/hadoop
scp-rhadoop-1.2.0tmphbase*zookeeper*node2:
/home/hadoop
scp-rhadoop-1.2.0tmphbase*zookeeper*node3:
/home/hadoop
scp-rhadoop-1.2.0tmphbase*zookeeper*node4:
/home/hadoop
scp-rhadoop-1.2.0tmphbase*zookeeper*node5:
/home/hadoop
配置本机的环境变量
Vi/etc/profile,追加以下内容:
HADOOP_HOME=/home/hadoop/hadoop-1.2.0
exportJAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0/jre
exportCLASSPATH=.:
$JAVA_HOME/lib/dt.jar:
$JAVA_HOME/lib/tools.jar
exportPATH=$PATH:
$JAVA_HOME/bin
配置其它五台机器上的环境变量
scp/etc/profilenode1:
/etc/
scp/etc/profilenode2:
/etc/
scp/etc/profilenode3:
/etc/
scp/etc/profilenode4:
/etc/
scp/etc/profilenode5:
/etc/
使所有的机器上环境变量立即生效
在所有的机器上运行source/etc/profile或重启机器
六、配置hadoop-1.2.0(在node0机器上)和格式化hadoop文件系统
Vi/home/hadoop/hadoop-1.2.0/conf/hadoop-env.sh,追加以下内容
exportJAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0/jre
Vi/home/hadoop/hadoop-1.2.0/conf/hdfs-site.xml
Vi/home/hadoop/hadoop-1.2.0/conf/mapred-site.xml
9001
Vi/home/hadoop/hadoop-1.2.0/conf/core-site.xml
//node0:
9000
Vi/home/hadoop/hadoop-1.2.0/conf/masters
node0
Vi/home/hadoop/hadoop-1.2.0/conf/slaves
node1
node2
node3
node4
node5
将node0的/home/hadoop/hadoop-1.2.0/conf目录中的以上文件拷贝到其它五台机器上
Scp*.shhdfs*core*mapred*mas*sla*node1:
/home/hadoop/hadoop-1.2.0/conf
Scp*.shhdfs*core*mapred*mas*sla*node2:
/home/hadoop/hadoop-1.2.0/conf
Scp*.shhdfs*core*mapred*mas*sla*node3:
/home/hadoop/hadoop-1.2.0/conf
Scp*.shhdfs*core*mapred*mas*sla*node4:
/home/hadoop/hadoop-1.2.0/conf
Scp*.shhdfs*core*mapred*mas*sla*node5:
/home/hadoop/hadoop-1.2.0/conf
至此,所有hadoop需要安装和配置的都已经完成,接下来就要格式化hadoop文件系统.
Cd/home/hadoop/hadoop-1.2.0/bin
./hadoopnamenode-format
七、配置zookeeper-3.4.5(在node0机器上)
Cd/home/hadoop/zookeeper-3.4.5/conf/
Cpzoo_sample.cfgzoo.cfg
Vi/home/hadoop/zookeeper-3.4.5/conf/zoo.cfg,追加以下内容
dataDir=/home/hadoop/zookeeper
server.1=node1:
2888:
3888
server.2=node2:
2888:
3888
server.3=node3:
2888:
3888
server.4=node4:
2888:
3888
server.5=node5:
2888:
3888
将node0的/home/hadoop/zookeeper-3.4.5/conf/zoo.cfg拷贝到其它五台机器上
Scpzoo.cfgnode1:
/home/hadoop/zookeeper-3.4.5/conf/
Scpzoo.cfgnode2:
/home/hadoop/zookeeper-3.4.5/conf/
Scpzoo.cfgnode3:
/home/hadoop/zookeeper-3.4.5/conf/
Scpzoo.cfgnode4:
/home/hadoop/zookeeper-3.4.5/conf/
Scpzoo.cfgnode5:
/home/hadoop/zookeeper-3.4.5/conf/
八、配置hbase-0.95.1-hadoop1(在node0机器上)
Vi/home/hadoop/hbase-0.95.1-hadoop1/conf/hbase-env.sh,追加以下内容:
exportJAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0/jre
exportHBASE_CLASSPATH=/home/hadoop/hadoop-1.2.0/conf
exportHBASE_HEAPSIZE=2048
exportHBASE_MANAGES_ZK=false
exportHADOOP_HOME=/home/hadoop/hadoop-1.2.0
exportHBASE_HOME=/home/hadoop/hbase-0.95.1-hadoop1
exportPATH=$PATH:
$HADOOP_HOME/bin:
HBASE_HOME/bin
Vi/home/hadoop/hbase-0.95.1-hadoop1/conf/hbase-site.xml
//node0:
9000/hbase
//node0:
60000
Vi/home/hadoop/hbase-0.95.1-hadoop1/conf/regionservers
node1
node2
node3
node4
node5
将本机的/home/hadoop/hbase-0.95.1-hadoop1/conf/中的以上文件拷贝到其它五台机器上
Scphbase-env*regionservershbase-site*node1:
/home/hadoop/hbase-0.95.1-hadoop1/conf
Scphbase-env*regionservershbase-site*node2:
/home/hadoop/hbase-0.95.1-hadoop1/conf
Scphbase-env*regionservershbase-site*node3:
/home/hadoop/hbase-0.95.1-hadoop1/conf
Scphbase-env*regionservershbase-site*node4:
/home/hadoop/hbase-0.95.1-hadoop1/conf
Scphbase-env*regionservershbase-site*node5:
/home/hadoop/hbase-0.95.1-hadoop1/conf
九、服务启动
至此,所有需要安装和配置的都已经完成,接下来就要启动服务,测试配置是否成功。
启动hadoop->启动zookeeper>启动hbase
关闭hbase->关闭zookeeper->关闭hadoop
启动hadoop
Cd/home/hadoop/hadoop-1.2.0/bin
./start-all.sh(要在namenode上面启动)
启动zookeeper(要在所有Datanodes上面启动)
sshnode1"cd/home/hadoop/zookeeper-3.4.5/bin;./zkServer.shstart;exit"
sshnode2"cd/home/hadoop/zookeeper-3.4.5/bin;./zkServer.shstart;exit"
sshnode3"cd/home/hadoop/zookeeper-3.4.5/bin;./zkServer.shstart;exit"
sshnode4"cd/home/hadoop/zookeeper-3.4.5/bin;./zkServer.shstart;exit"
sshnode5"cd/home/hadoop/zookeeper-3.4.5/bin;./zkServer.shstart;exit"
启动Hbase
Cd/home/hadoop/hbase-0.95.1-hadoop1/bin
./start-hbase.sh(要在namenode上面启动)
如果启动成功,需要进入hbaseshell才能使用hbase命令
./hbaseshell(进入hbaseshell命令,exit是退出shell命令,详见相关资料)
关闭hbase
Cd/home/hadoop/hbase-0.95.1-hadoop1/bin
./stop-hbase.sh(要在namenode上面关闭)
关闭zookeeper(要在所有Datanodes上面关闭)
sshnode1"cd/home/hadoop/zookeeper-3.4.5/bin;./zkServer.shstop;exit"
sshnode2"cd/home/hadoop/zookeeper-3.4.5/bin;./zkServer.shstop;exit"
sshnode3"cd/home/hadoop/zookeeper-3.4.5/bin;./zkServer.shstop;exit"
sshnode4"cd/home/hadoop/zookeeper-3.4.5/bin;./zkServer.shstop;exit"
sshnode5"cd/home/hadoop/zookeeper-3.4.5/bin;./zkServer.shstop;exit"
关闭hadoop
Cd/home/hadoop/hadoop-1.2.0/bin
./stop-all.sh(要在namenode上面关闭)
十、查看hadoop、zookeeper、hbase状态
如果都启动成功可以看到以下状态
用jps命令查看hadoop和hbase(HMaster)进程,NameNode上的显示状态如下:
5873HMaster(hbase)
5505JobTracker
5222NameNode
5399SecondaryNameNode
11498Jps
十一、用web查看hadoop集群状态
http:
//192.168.7.80:
50070
http:
//192.168.7.80:
50030
十二、用hadoopdfsadmin–report查看hadoop的状态
[root@node0~]#/home/hadoop/hadoop-1.2.0/bin/hadoopdfsadmin–report
Master服务器的状态:
ConfiguredCapacity:
2938674708480(2.67TB)
PresentCapacity:
2280312115200(2.07TB)
DFSRemaining:
2280311431168(2.07TB)
DFSUsed:
684032(668KB)
DFSUsed%:
0%
Underreplicatedblocks:
0
Blockswithcorruptreplicas:
0
Missingblocks:
0
Slave服务器的状态:
Datanodesavailable:
5(5total,0dead)
Name:
192.168.7.77:
50010
DecommissionStatus:
Normal
ConfiguredCapacity:
614868582400(572.64GB)
DFSUsed:
131072(128KB)
NonDFSUsed:
134909444096(125.64GB)
DFSRemaining:
479959007232(447GB)
DFSUsed%:
0%
DFSRemaining%:
78.06%
Lastcontact:
FriAug2313:
13:
10CST2013
Name:
192.168.7.248:
50010
DecommissionStatus:
Normal
ConfiguredCapacity:
614868582400(572.64GB)
DFSUsed:
98304(96KB)
NonDFSUsed:
141611208704(131.89GB)
DFSRemaining:
- 配套讲稿:
如PPT文件的首页显示word图标,表示该PPT已包含配套word讲稿。双击word图标可打开word文档。
- 特殊限制:
部分文档作品中含有的国旗、国徽等图片,仅作为作品整体效果示例展示,禁止商用。设计者仅对作品中独创性部分享有著作权。
- 关 键 词:
- 安装配置Hadoop+Zookeeper+HBase 环境简单说明 安装 配置 Hadoop Zookeeper HBase 环境 简单 说明