리눅스 관련 게시판

리눅스 관련 게시판

2015/07/03(08:52) from 58.125.30.140
작성자 : 주인장	조회수 : 8582 , 줄수 : 299

[Hadoop] 1. 설치 가이드 (완전분산모드 - Fully Distributed Mode) - 1주일만에 성공

1. hadoop download

$ wget https://archive.apache.org/dist/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz

[root@hadoop01 hadoop]# cat /etc/hosts
127.0.0.1      localhost
@ 아랫것땜에 1주일 고생(아랫것처럼 하면 안된다. 쓸데없는거 다 삭제하고 윗줄과 같이 한다.)
#127.0.0.1       hadoop01 localhost

210.93.6.175    hadoop01
210.93.6.176    hadoop02
210.93.6.177    hadoop03
210.93.6.178    hadoop04

---------------------------------------------------------------------------------------------------

@ node 설명 ( 총 4개 )
hadoop01 : namenode
hadoop02 : secondary namenode, datanode
hadoop03 : datanode
hadoop04 : datanode

@ 설치방법
1. 단독작업모드 : Standalone Operation
2. 가상분산모드 : Pseudo-Distributed Operation
3. 완전분산모드 : Fully-Distributed Operation

2. ssh 설정

@ hadoop01에서만 실행

[root@hadoop01 ~] ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
d2:06:44:a1:79:58:6c:81:30:e7:01:65:08:b3:93:f8 root@hadoop01
The key's randomart image is:

[root@hadoop01 ~] cd .ssh
[root@hadoop01 .ssh] cp id_rsa.pub authorized_keys

@ 각노드로 배포

[root@hadoop01 .ssh] ssh-copy-id -i root@hadoop02
[root@hadoop01 .ssh] ssh-copy-id -i root@hadoop03
[root@hadoop01 .ssh] ssh-copy-id -i root@hadoop04


각 노드에 명령어를 날려본다. 암호를 물어보지 않고 실행되면 성공적임

[root@hadoop01 .ssh] ssh hadoop02 hostname
hadoop02
[root@hadoop01 .ssh] ssh hadoop03 hostname
hadoop03
[root@hadoop01 .ssh] ssh hadoop04 hostname
hadoop04

3. hadoop 실행

[root@hadoop01 bin] pwd
/usr/local/hadoop/bin

[root@hadoop01 bin] ./start-all.sh

각 노드에서 $ ps -ef | grep java로 데몬 실행여부 확인 (java 프로세스가 떠있어야)

4. hadoop01에 hadoop 설치

모든 노드(hadoop01, hadoop02, hadoop03, hadoop04)에서 아래 실행

[root@hadoop01 .ssh]# su - hadoop
[hadoop@hadoop01 ~]$ mkdir -p hdfs/data
[hadoop@hadoop01 ~]$ mkdir -p hdfs/temp
[hadoop@hadoop01 ~]$ mkdir -p hdfs/name

[root@hadoop01 hdfs]# pwd
/home/hadoop/hdfs

@ 모든 node의 data directory는 chmod 755 data (아주 중요)

[root@hadoop01 hdfs]# ls -l
total 16
drwxr-xr-x. 6 hadoop hadoop 4096 Jul 5 01:04 data
drwxr-xr-x. 7 root   root   4096 Jul 5 01:16 mapred
drwxrwxr-x. 5 hadoop hadoop 4096 Jul 5 01:04 name
drwxrwxr-x. 3 hadoop hadoop 4096 Jul 5 01:17 temp

[root@hadoop01 local]# pwd
/usr/local
[root@hadoop01 local]# ls -l
lrwxrwxrwx. 1 root root         12 Jul 3 02:04 hadoop -> hadoop-1.2.1
drwxr-xr-x. 16 root root       4096 Jul 3 02:07 hadoop-1.2.1
-rw-r--r--. 1 root root   63851630 Nov 6 2014 hadoop-1.2.1.tar.gz

[hadoop@hadoop01 ~]$ cd /usr/local/hadoop/conf
[hadoop@hadoop01 conf]$ vi core-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>



<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://hadoop01:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/hdfs/temp<value>
</property>
</configuration>

[hadoop@hadoop01 conf]$ vi hdfs-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>



<configuration>
       <property>
               <name>dfs.replication</name>
               <value>1</value>
       </property>
       <property>
               <name>dfs.name.dir</name>
               <value>/home/hadoop/hdfs/name</value>
       </property>
       <property>
               <name>dfs.data.dir</name>
               <value>/home/hadoop/hdfs/data</value>
       </property>
</configuration>

[root@hadoop01 conf]# vi mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>



<configuration>
<property>
<name>mapred.job.tracker</name>
<value>hadoop01:9001</value>
</property>
<property>
<name>mapred.local.dir</name>
<value>/home/hadoop/hdfs/mapred</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>/home/hadoop/hdfs/mapred</value>
</property>
</configuration>

@ Secondarynode 실행할 서버를 설정
[root@hadoop01 conf]# vi masters
hadoop02

@ 데이터 노드를 실행할 서버를 설정
[root@hadoop01 conf]# vi slaves
hadoop02
hadoop03
hadoop04

[root@hadoop01 conf]# vi hadoop-env.sh 에서 java 경로 설정
# The java implementation to use. Required.
# export JAVA_HOME=/usr/lib/j2sdk1.5-sun
export JAVA_HOME=/usr

# jkkang's add
export HADOOP_HOME_WARN_SUPPRESS=1

5. hadoop02, hadoop03, hadoop04에 hadoop 배포로 설치

[root@hadoop01 local]# tar cvf hadoop.tar hadoop-1.2.1
[root@hadoop01 local]# scp -rp hadoop.tar hadoop02:/usr/local
[root@hadoop01 local]# scp -rp hadoop.tar hadoop03:/usr/local
[root@hadoop01 local]# scp -rp hadoop.tar hadoop04:/usr/local

[root@hadoop02 local]# tar xvf hadoop.tar
[root@hadoop03 local]# tar xvf hadoop.tar
[root@hadoop04 local]# tar xvf hadoop.tar

[root@hadoop01 ~]# pwd
/root

hadoop01 ~ hadoop04 까지 작업

[root@hadoop01 ~]# vi .bash_profile

PATH=$PATH:$HOME/bin:/usr/local/hadoop/bin

추가

6. namenode 포맷 (hadoop01에서만 실행)

root@hadoop01 ~]# hadoop namenode -format
15/07/03 09:39:37 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = hadoop01/127.0.0.1
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 1.2.1
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG:   java = 1.7.0_79
************************************************************/
Re-format filesystem in /home/hadoop/hdfs/name ? (Y or N) y
Format aborted in /home/hadoop/hdfs/name
15/07/03 09:39:39 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hadoop01/127.0.0.1
************************************************************/

7. hadoop 실행 (hadoop01에서)

[root@hadoop01 ~]# ./start-all.sh
starting namenode, logging to /usr/local/hadoop-1.2.1/libexec/../logs/hadoop-root-namenode-hadoop01.out
hadoop04: starting datanode, logging to /usr/local/hadoop-1.2.1/libexec/../logs/hadoop-root-datanode-hadoop04.out
hadoop03: starting datanode, logging to /usr/local/hadoop-1.2.1/libexec/../logs/hadoop-root-datanode-hadoop03.out
hadoop02: starting datanode, logging to /usr/local/hadoop-1.2.1/libexec/../logs/hadoop-root-datanode-hadoop02.out
hadoop01: starting datanode, logging to /usr/local/hadoop-1.2.1/libexec/../logs/hadoop-root-datanode-hadoop01.out
hadoop02: starting secondarynamenode, logging to /usr/local/hadoop-1.2.1/libexec/../logs/hadoop-root-secondarynamenode-hadoop02.out
jobtracker running as process 2973. Stop it first.
hadoop01: tasktracker running as process 3111. Stop it first.
hadoop03: tasktracker running as process 1490. Stop it first.
hadoop04: tasktracker running as process 1604. Stop it first.
hadoop02: tasktracker running as process 1521. Stop it first.

[root@hadoop01 ~]# jps 실행

$ yum install java-1.7.0-openjdk-devel.x86_64 -y   를 설치하면 jps를 사용가능

[root@hadoop01 ~]# jps
6769 Jps
6444 NameNode
3081 JobTracker

[root@hadoop02 ~]# jps
6399 Jps
6357 SecondaryNameNode
6509 DataNode
3081 TaskTracker

[root@hadoop03 ~]# jps
6395 Jps
6337 DataNode
3081 TaskTracker

[root@hadoop04 ~]# jps
6179 Jps
6252 DataNode
3081 TaskTracker

8. 웹에서 실행화면 보기

http://hadoop01_ip:50070/

@ Live Nodes 가 Slave Node수와 같아야 한다.
용량은 합해서 나온다.

Cluster Summary
Safe mode is ON. The reported blocks 5 has reached the threshold 0.9990 of total blocks 5. Safe mode will be turned off automatically in 22 seconds.
20 files and directories, 5 blocks = 25 total. Heap Size is 57.5 MB / 889 MB (6%)

Configured Capacity : 37.36 GB
DFS Used : 384 KB
Non DFS Used : 8.57 GB
DFS Remaining : 28.79 GB
DFS Used% : 0 %
DFS Remaining% : 77.06 %
Live Nodes   : 3
Dead Nodes   : 0
Decommissioning Nodes   : 0
Number of Under-Replicated Blocks : 0