SSH 配置


生成 SSH密钥,并复制密钥给 slaves1 slaves2

1
2
3
$ ssh-keygen
$ ssh-copy-id root @172.16.1.3
$ ssh-copy-id root @172.16.1.4

安装 Hadoop 和各类组件


解压 JDK、Hadoop、Sqoop、zookeeper、hive、hbase、Flume 至 /usr/local/src/

1
2
3
4
5
6
$ tar -zxvf hadoop-2.6.0.tar.gz -C /usr/local/src/
$ tar -zxvf sqoop-1.4.7.bin_hadoop-2.6.0.tar.gz -C /usr/local/src/
$ tar -zxvf zookeeper-3.4.5.tar.gz -C /usr/local/src/
$ tar -zxvf apache-hive-1.1.0-bin.tar.gz -C /usr/local/src/
$ tar -zxvf hbase-1.2.3-bin.tar.gz -C /usr/local/src/
$ tar -zxvf apache-flume-1.6.0-bin.tar.gz -C /usr/local/src/
  • 修改 Sqoop 解压后的文件夹名称为 sqoop
  • 修改 flume-env.sh.template 为 flume-env.sh
  • 修改 flume-env.ps1.template 为 flume-env.ps1
  • 修改 flume-conf.properties.template 为 flume-conf.properties

复制 jdbc 至 sqoop 中

1
$ cp -R /h3cu/mysql-connector-java-5.1.47.jar /usr/local/src/sqoop/lib

配置环境变量(仅 root 用户生效)

1
$ vi /root/.bash_profile
1
2
3
4
5
6
7
8
9
10
11
12
export JAVA_HOME=/usr/local/src/jdk1.8.0_144
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export HADOOP_HOME=/usr/local/src/hadoop-2.6.0
export SQOOP_HOME=/usr/local/src/sqoop
export ZOOKEEPER_HOME=/usr/local/src/zookeeper-3.4.5
export ZOOCFGDIR=/usr/local/src/zookeeper-3.4.5/conf
export HIVE_HOME=/usr/local/src/apache-hive-1.1.0-bin
export HBASE_HOME/usr/local/src/hbase-1.2.3
export FLUME_HOME/usr/local/src/flume
export FLUME_CONF_DIR=$FLUME_HOME/conf
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin:$SQOOP_HOME/bin:$ZOOKEEPER_HOME/bin:$HIVE_HOME/bin:$HBASE_HOME/bin:$HADOOP_HOME/sbin:$FLUME_HOME/bin
export PATH

编辑 Flume 相关文件


[ flume-env.sh ]

1
$ vi /usr/local/src/apache-flume-1.6.0-bin
1
2
export JAVA_HOME=/usr/local/src/jdk1.8.0_144
export JAVA_POTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote"

编辑 Hadoop 相关文件


在Hadoop根目录下新建文件夹 tmp
修改 mapred-site.xml.template 为 mapred-site.xml
编辑 Hadoop 相关文件(hadoop-env-sh、core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml、slaves)

[ hadoop-env.sh ]

1
$ vi /usr/local/src/hadoop-2.6.0/etc/hadoop/hadoop-env.sh
1
2
export JAVA_HOME=/usr/local/src/jdk1.8.0_144
export HADOOP_CONF_DIR=/usr/local/src/hadoop-2.6.0/etc/hadoop/

[ core-site.xml ]

1
$ vi /usr/local/src/hadoop-2.6.0/etc/hadoop/core-site.xml
1
2
3
4
5
6
7
8
9
10
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://172.16.1.2:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/src/hadoop-2.6.0/tmp</value>
</property>
</configuration>

[ hdfs-site.xml ]

1
$ vi /usr/local/src/hadoop-2.6.0/etc/hadoop/hdfs-site.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
<configuration>
<property>
<name>dfs.datanode.ipc.address</name>
<value>0.0.0.0:50020</value>
</property>
<property>
<name>dfs.datanode.http.address</name>
<value>0.0.0.0:50075</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
</configuration>

[ mapred-site.xml ]

1
$ vi /usr/local/src/hadoop-2.6.0/etc/hadoop/mapred-site.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
</configuration>

[ yarn-site.xml ]

1
$ vi /usr/local/src/hadoop-2.6.0/etc/hadoop/yarn-site.xml
1
2
3
4
5
6
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>

[ mapred-env.sh ]

1
$ vi /usr/local/src/hadoop-2.6.0/etc/hadoop/mapred-env.sh
1
export JAVA_HOME=/usr/local/src/jdk1.8.0_144

[ slaves ]

1
$ vi /usr/local/src/hadoop-2.6.0/etc/hadoop/slaves
1
2
172.16.1.3
172.16.1.4

复制 Hadoop 和环境变量至 slaves1 和 slaves2

1
2
3
4
scp -r /usr/local/src/hadoop-2.6.0 root@172.16.1.3:usr/local/src/
scp -r /usr/local/src/hadoop-2.6.0 root@172.16.1.4:/usr/local/src/
scp -r /root/.bash_profile root@172.16.1.3:/root/
scp -r /root/.bash_profile root@172.16.1.4:/root/

初始化 Hadoop


1
$ hdfs namenode -format

启动 Hadoop


1
2
$ start-all.sh
$ mr-jobhistory-daemon.sh start historyserver

查看 JPS 进程(如果三台主机有 datanode 和 namenode)则已成功。

1
$ jps

Sqoop 连接测试


使用 Sqoop 连接并列出所有数据库

1
$ sqoop list-databases --connect jdbc:mysql://[ip地址]:3306 --username [数据库用户名] --password [数据库密码]