¸®´ª½º °ü·Ã °Ô½ÃÆÇ |
---|
2015/07/03(08:52) from 58.125.30.140 | |
ÀÛ¼ºÀÚ : ÁÖÀÎÀå | Á¶È¸¼ö : 8669 , ÁÙ¼ö : 299 |
[Hadoop] 1. ¼³Ä¡ °¡À̵å (¿ÏÀüºÐ»ê¸ðµå - Fully Distributed Mode) - 1ÁÖÀϸ¸¿¡ ¼º°ø |
---|
1. hadoop download $ wget https://archive.apache.org/dist/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz [root@hadoop01 hadoop]# cat /etc/hosts 127.0.0.1 localhost @ ¾Æ·§°Í¶«¿¡ 1ÁÖÀÏ °í»ý(¾Æ·§°Íó·³ ÇÏ¸é ¾ÈµÈ´Ù. ¾µµ¥¾ø´Â°Å ´Ù »èÁ¦ÇÏ°í ÀÁÙ°ú °°ÀÌ ÇÑ´Ù.) #127.0.0.1 hadoop01 localhost 210.93.6.175 hadoop01 210.93.6.176 hadoop02 210.93.6.177 hadoop03 210.93.6.178 hadoop04 --------------------------------------------------------------------------------------------------- @ node ¼³¸í ( ÃÑ 4°³ ) hadoop01 : namenode hadoop02 : secondary namenode, datanode hadoop03 : datanode hadoop04 : datanode @ ¼³Ä¡¹æ¹ý 1. ´Üµ¶ÀÛ¾÷¸ðµå : Standalone Operation 2. °¡»óºÐ»ê¸ðµå : Pseudo-Distributed Operation 3. ¿ÏÀüºÐ»ê¸ðµå : Fully-Distributed Operation 2. ssh ¼³Á¤ @ hadoop01¿¡¼¸¸ ½ÇÇà [root@hadoop01 ~] ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Created directory '/root/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: d2:06:44:a1:79:58:6c:81:30:e7:01:65:08:b3:93:f8 root@hadoop01 The key's randomart image is: [root@hadoop01 ~] cd .ssh [root@hadoop01 .ssh] cp id_rsa.pub authorized_keys @ °¢³ëµå·Î ¹èÆ÷ [root@hadoop01 .ssh] ssh-copy-id -i root@hadoop02 [root@hadoop01 .ssh] ssh-copy-id -i root@hadoop03 [root@hadoop01 .ssh] ssh-copy-id -i root@hadoop04 °¢ ³ëµå¿¡ ¸í·É¾î¸¦ ³¯·Áº»´Ù. ¾ÏÈ£¸¦ ¹°¾îº¸Áö ¾Ê°í ½ÇÇàµÇ¸é ¼º°øÀûÀÓ [root@hadoop01 .ssh] ssh hadoop02 hostname hadoop02 [root@hadoop01 .ssh] ssh hadoop03 hostname hadoop03 [root@hadoop01 .ssh] ssh hadoop04 hostname hadoop04 3. hadoop ½ÇÇà [root@hadoop01 bin] pwd /usr/local/hadoop/bin [root@hadoop01 bin] ./start-all.sh °¢ ³ëµå¿¡¼ $ ps -ef | grep java·Î µ¥¸ó ½ÇÇà¿©ºÎ È®ÀÎ (java ÇÁ·Î¼¼½º°¡ ¶°ÀÖ¾î¾ß) 4. hadoop01¿¡ hadoop ¼³Ä¡ ¸ðµç ³ëµå(hadoop01, hadoop02, hadoop03, hadoop04)¿¡¼ ¾Æ·¡ ½ÇÇà [root@hadoop01 .ssh]# su - hadoop [hadoop@hadoop01 ~]$ mkdir -p hdfs/data [hadoop@hadoop01 ~]$ mkdir -p hdfs/temp [hadoop@hadoop01 ~]$ mkdir -p hdfs/name [root@hadoop01 hdfs]# pwd /home/hadoop/hdfs @ ¸ðµç nodeÀÇ data directory´Â chmod 755 data (¾ÆÁÖ Áß¿ä) [root@hadoop01 hdfs]# ls -l total 16 drwxr-xr-x. 6 hadoop hadoop 4096 Jul 5 01:04 data drwxr-xr-x. 7 root root 4096 Jul 5 01:16 mapred drwxrwxr-x. 5 hadoop hadoop 4096 Jul 5 01:04 name drwxrwxr-x. 3 hadoop hadoop 4096 Jul 5 01:17 temp [root@hadoop01 local]# pwd /usr/local [root@hadoop01 local]# ls -l lrwxrwxrwx. 1 root root 12 Jul 3 02:04 hadoop -> hadoop-1.2.1 drwxr-xr-x. 16 root root 4096 Jul 3 02:07 hadoop-1.2.1 -rw-r--r--. 1 root root 63851630 Nov 6 2014 hadoop-1.2.1.tar.gz [hadoop@hadoop01 ~]$ cd /usr/local/hadoop/conf [hadoop@hadoop01 conf]$ vi core-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>fs.default.name</name> <value>hdfs://hadoop01:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/home/hadoop/hdfs/temp<value> </property> </configuration> [hadoop@hadoop01 conf]$ vi hdfs-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.name.dir</name> <value>/home/hadoop/hdfs/name</value> </property> <property> <name>dfs.data.dir</name> <value>/home/hadoop/hdfs/data</value> </property> </configuration> [root@hadoop01 conf]# vi mapred-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>mapred.job.tracker</name> <value>hadoop01:9001</value> </property> <property> <name>mapred.local.dir</name> <value>/home/hadoop/hdfs/mapred</value> </property> <property> <name>mapred.system.dir</name> <value>/home/hadoop/hdfs/mapred</value> </property> </configuration> @ Secondarynode ½ÇÇàÇÒ ¼¹ö¸¦ ¼³Á¤ [root@hadoop01 conf]# vi masters hadoop02 @ µ¥ÀÌÅÍ ³ëµå¸¦ ½ÇÇàÇÒ ¼¹ö¸¦ ¼³Á¤ [root@hadoop01 conf]# vi slaves hadoop02 hadoop03 hadoop04 [root@hadoop01 conf]# vi hadoop-env.sh ¿¡¼ java °æ·Î ¼³Á¤ # The java implementation to use. Required. # export JAVA_HOME=/usr/lib/j2sdk1.5-sun export JAVA_HOME=/usr # jkkang's add export HADOOP_HOME_WARN_SUPPRESS=1 5. hadoop02, hadoop03, hadoop04¿¡ hadoop ¹èÆ÷·Î ¼³Ä¡ [root@hadoop01 local]# tar cvf hadoop.tar hadoop-1.2.1 [root@hadoop01 local]# scp -rp hadoop.tar hadoop02:/usr/local [root@hadoop01 local]# scp -rp hadoop.tar hadoop03:/usr/local [root@hadoop01 local]# scp -rp hadoop.tar hadoop04:/usr/local [root@hadoop02 local]# tar xvf hadoop.tar [root@hadoop03 local]# tar xvf hadoop.tar [root@hadoop04 local]# tar xvf hadoop.tar [root@hadoop01 ~]# pwd /root hadoop01 ~ hadoop04 ±îÁö ÀÛ¾÷ [root@hadoop01 ~]# vi .bash_profile PATH=$PATH:$HOME/bin:/usr/local/hadoop/bin Ãß°¡ 6. namenode Æ÷¸Ë (hadoop01¿¡¼¸¸ ½ÇÇà) root@hadoop01 ~]# hadoop namenode -format 15/07/03 09:39:37 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = hadoop01/127.0.0.1 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 1.2.1 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013 STARTUP_MSG: java = 1.7.0_79 ************************************************************/ Re-format filesystem in /home/hadoop/hdfs/name ? (Y or N) y Format aborted in /home/hadoop/hdfs/name 15/07/03 09:39:39 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at hadoop01/127.0.0.1 ************************************************************/ 7. hadoop ½ÇÇà (hadoop01¿¡¼) [root@hadoop01 ~]# ./start-all.sh starting namenode, logging to /usr/local/hadoop-1.2.1/libexec/../logs/hadoop-root-namenode-hadoop01.out hadoop04: starting datanode, logging to /usr/local/hadoop-1.2.1/libexec/../logs/hadoop-root-datanode-hadoop04.out hadoop03: starting datanode, logging to /usr/local/hadoop-1.2.1/libexec/../logs/hadoop-root-datanode-hadoop03.out hadoop02: starting datanode, logging to /usr/local/hadoop-1.2.1/libexec/../logs/hadoop-root-datanode-hadoop02.out hadoop01: starting datanode, logging to /usr/local/hadoop-1.2.1/libexec/../logs/hadoop-root-datanode-hadoop01.out hadoop02: starting secondarynamenode, logging to /usr/local/hadoop-1.2.1/libexec/../logs/hadoop-root-secondarynamenode-hadoop02.out jobtracker running as process 2973. Stop it first. hadoop01: tasktracker running as process 3111. Stop it first. hadoop03: tasktracker running as process 1490. Stop it first. hadoop04: tasktracker running as process 1604. Stop it first. hadoop02: tasktracker running as process 1521. Stop it first. [root@hadoop01 ~]# jps ½ÇÇà $ yum install java-1.7.0-openjdk-devel.x86_64 -y ¸¦ ¼³Ä¡Çϸé jps¸¦ »ç¿ë°¡´É [root@hadoop01 ~]# jps 6769 Jps 6444 NameNode 3081 JobTracker [root@hadoop02 ~]# jps 6399 Jps 6357 SecondaryNameNode 6509 DataNode 3081 TaskTracker [root@hadoop03 ~]# jps 6395 Jps 6337 DataNode 3081 TaskTracker [root@hadoop04 ~]# jps 6179 Jps 6252 DataNode 3081 TaskTracker 8. À¥¿¡¼ ½ÇÇàÈ¸é º¸±â http://hadoop01_ip:50070/ @ Live Nodes °¡ Slave Node¼ö¿Í °°¾Æ¾ß ÇÑ´Ù. ¿ë·®Àº ÇÕÇؼ ³ª¿Â´Ù. Cluster Summary Safe mode is ON. The reported blocks 5 has reached the threshold 0.9990 of total blocks 5. Safe mode will be turned off automatically in 22 seconds. 20 files and directories, 5 blocks = 25 total. Heap Size is 57.5 MB / 889 MB (6%) Configured Capacity : 37.36 GB DFS Used : 384 KB Non DFS Used : 8.57 GB DFS Remaining : 28.79 GB DFS Used% : 0 % DFS Remaining% : 77.06 % Live Nodes : 3 Dead Nodes : 0 Decommissioning Nodes : 0 Number of Under-Replicated Blocks : 0 |