CentOS7x86_64编译hadoop2.4

主要涉及到工具有：hadoop-2.4.0-src.tar.gz、Ant、Maven、JDK、GCC、CMake、openssl

第一步升级系统相关编译所需的软件（升级最新版）：

yum install lzo-devel zlib-devel gcc autoconf automake libtool ncurses-devel openssl-devel

wget http://mirrors.cnnic.cn/apache/hadoop/common/hadoop-2.4.0/hadoop-2.4.0-src.tar.gz （源代版）

tar -zxvf hadoop-2.4.0-src.tar.gz

wget http://apache.fayea.com/apache-mirror//ant/binaries/apache-ant-1.9.4-bin.tar.gz

tar -xvf apache-ant-1.9.4-bin.tar.gz

wget http://apache.fayea.com/apache-mirror/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.tar.gz

tar -xvf apache-maven-3.0.5-bin.tar.gz

vi /etc/profile

export JAVA_HOME=/usr/java/jdk1.7.0_55

export JAVA_BIN=/usr/java/jdk1.7.0_55/bin

export ANT_HOME=/home/hadoop/ant

export MVN_HOME=/home/hadoop/maven

export FINDBUGS_HOME=/home/hadoop/findbugs-2.0.3

export PATH=$PATH:$JAVA_HOME/bin:$ANT_HOME/bin:$MVN_HOME/bin:$FINDBUGS_HOME/bin

生产配置文件：

source /etc/profile

验证是否配置成功

ant –version

mvn -version

findbugs –version

验证结果：

安装protobuf（以root用户登录）

wget https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz

tar zxf protobuf-2.5.0.tar.gz

cd protobuf-2.5.0

./configure

make

make install

protoc --version

安装cmake（以root用户登录）

wget http://www.cmake.org/files/v2.8/cmake-2.8.12.2-Linux-i386.tar.gz

./bootstrap

make

make install

cmake –version

（别一种方法，直接用yum install cmake）

 

编译Hadoop

mvn package -DskipTests -Pdist,native –Dtar

此时在下载maven依赖所有包及插件

慢慢等待中……

编译成功，检查nativelib 是否编译成功

[root@master hadoop-2.4.1-src]# cd hadoop-dist/target/hadoop-2.4.1/lib/native/
[root@master native]# file libhadoop.so.1.0.0
libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0xba68c7f46259525c3aae4ebd99e1faf3b6c7e7a6, not stripped

 

代表编译成功

 

或者结果如下， 也表示编译成功。

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................ SUCCESS [5.731s]
[INFO] Apache Hadoop Project POM ......................... SUCCESS [4.215s]
[INFO] Apache Hadoop Annotations ......................... SUCCESS [5.122s]
[INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.548s]
[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [4.271s]
[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [8.020s]
[INFO] Apache Hadoop MiniKDC ............................. SUCCESS [7.431s]
[INFO] Apache Hadoop Auth ................................ SUCCESS [7.517s]
[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [4.727s]
[INFO] Apache Hadoop Common .............................. SUCCESS [2:53.800s]
[INFO] Apache Hadoop NFS ................................. SUCCESS [16.696s]
[INFO] Apache Hadoop Common Project ...................... SUCCESS [0.042s]
[INFO] Apache Hadoop HDFS ................................ SUCCESS [6:07.368s]
[INFO] Apache Hadoop HttpFS .............................. SUCCESS [48.810s]
[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [20.154s]
[INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [9.709s]
[INFO] Apache Hadoop HDFS Project ........................ SUCCESS [0.049s]
[INFO] hadoop-yarn ....................................... SUCCESS [0.138s]
[INFO] hadoop-yarn-api ................................... SUCCESS [2:00.295s]
[INFO] hadoop-yarn-common ................................ SUCCESS [1:00.256s]
[INFO] hadoop-yarn-server ................................ SUCCESS [0.076s]
[INFO] hadoop-yarn-server-common ......................... SUCCESS [21.974s]
[INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [28.986s]
[INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [6.791s]
[INFO] hadoop-yarn-server-applicationhistoryservice ...... SUCCESS [13.558s]
[INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [28.431s]
[INFO] hadoop-yarn-server-tests .......................... SUCCESS [2.644s]
[INFO] hadoop-yarn-client ................................ SUCCESS [12.729s]
[INFO] hadoop-yarn-applications .......................... SUCCESS [0.102s]
[INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [4.878s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [3.103s]
[INFO] hadoop-yarn-site .................................. SUCCESS [0.055s]
[INFO] hadoop-yarn-project ............................... SUCCESS [6.390s]
[INFO] hadoop-mapreduce-client ........................... SUCCESS [0.211s]
[INFO] hadoop-mapreduce-client-core ...................... SUCCESS [39.919s]
[INFO] hadoop-mapreduce-client-common .................... SUCCESS [34.197s]
[INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [5.716s]
[INFO] hadoop-mapreduce-client-app ....................... SUCCESS [18.761s]
[INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [17.226s]
[INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [7.617s]
[INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [3.211s]
[INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [12.571s]
[INFO] hadoop-mapreduce .................................. SUCCESS [6.483s]
[INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [10.180s]
[INFO] Apache Hadoop Distributed Copy .................... SUCCESS [15.514s]
[INFO] Apache Hadoop Archives ............................ SUCCESS [5.243s]
[INFO] Apache Hadoop Rumen ............................... SUCCESS [12.533s]
[INFO] Apache Hadoop Gridmix ............................. SUCCESS [8.247s]
[INFO] Apache Hadoop Data Join ........................... SUCCESS [6.091s]
[INFO] Apache Hadoop Extras .............................. SUCCESS [5.339s]
[INFO] Apache Hadoop Pipes ............................... SUCCESS [13.666s]
[INFO] Apache Hadoop OpenStack support ................... SUCCESS [14.356s]
[INFO] Apache Hadoop Client .............................. SUCCESS [14.354s]
[INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [0.145s]
[INFO] Apache Hadoop Scheduler Load Simulator ............ SUCCESS [39.951s]
[INFO] Apache Hadoop Tools Dist .......................... SUCCESS [8.662s]
[INFO] Apache Hadoop Tools ............................... SUCCESS [0.035s]
[INFO] Apache Hadoop Distribution ........................ SUCCESS [1:45.654s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 23:27.228s
[INFO] Finished at: Tue Sep 16 23:09:45 HKT 2014
[INFO] Final Memory: 67M/179M
[INFO] ------------------------------------------------------------------------

 

 

 

错误1

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 46.796s

[INFO] Finished at: Wed Jun 04 13:28:37 CST 2014

[INFO] Final Memory: 36M/88M

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal on project hadoop-common: Could not resolve dependencies for project org.apache.hadoop:hadoop-common:jar:2.4.0: Failure to find org.apache.commons:commons-compress:jar:1.4.1 in https://repository.apache.org/content/repositories/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of apache.snapshots.https has elapsed or updates are forced -> [Help 1]

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

[ERROR]

[ERROR] After correcting the problems, you can resume the build with the command

[ERROR] mvn <goals> -rf :hadoop-common

解决方法：

根据上面日志提示说找不到“org.apache.commons:commons-compress:jar:1.4.1”,

直接将本地（Windows）包复制到Linux系统中，解决了。

错误2

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 2:16.693s

[INFO] Finished at: Wed Jun 04 13:56:31 CST 2014

[INFO] Final Memory: 48M/239M

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "cmake" (in directory "/home/hadoop/hadoop-2.4.0-src/hadoop-common-project/hadoop-common/target/native"): error=2, 没有那个文件或目录

[ERROR] around Ant part ...<exec dir="/home/hadoop/hadoop-2.4.0-src/hadoop-common-project/hadoop-common/target/native" executable="cmake" failonerror="true">... @ 4:133 in /home/hadoop/hadoop-2.4.0-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml

[ERROR] -> [Help 1]

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

[ERROR]

[ERROR] After correcting the problems, you can resume the build with the command

[ERROR] mvn <goals> -rf :hadoop-common

解决方法：

是没有安装cmake导致的，再重新安装cmake；参考《5.3.1编译环境准备》

错误3

错误提示是找不到相应的文件和不能创建目录，在网上没有相关错误（根据自己经验修改目录权限为：775，让目录有创建文件或文件夹的权限，另外最好保证hadoop编译目录有2.5G至4G的空间）

chmod -Rf 775 ./ hadoop-2.4.0-src

main:

[mkdir] Created dir: /data/hadoop/hadoop-2.4.0-src/hadoop-tools/hadoop-pipes/target/test-dir

[INFO] Executed tasks

[INFO]

[INFO] --- maven-antrun-plugin:1.7:run (make) @ hadoop-pipes ---

[INFO] Executing tasks

错误3

main:

[mkdir] Created dir: /data/hadoop/hadoop-2.4.0-src/hadoop-tools/hadoop-pipes/target/native

[exec] -- The C compiler identification is GNU 4.4.7

[exec] -- The CXX compiler identification is GNU 4.4.7

[exec] -- Check for working C compiler: /usr/bin/cc

[exec] -- Check for working C compiler: /usr/bin/cc -- works

[exec] -- Detecting C compiler ABI info

[exec] -- Detecting C compiler ABI info - done

[exec] -- Check for working CXX compiler: /usr/bin/c++

[exec] -- Check for working CXX compiler: /usr/bin/c++ -- works

[exec] -- Detecting CXX compiler ABI info

[exec] -- Detecting CXX compiler ABI info - done

[exec] CMake Error at /usr/local/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:108 (message):

[exec] Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the

[exec] system variable OPENSSL_ROOT_DIR (missing: OPENSSL_LIBRARIES

[exec] OPENSSL_INCLUDE_DIR)

[exec] Call Stack (most recent call first):

[exec] /usr/local/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:315 (_FPHSA_FAILURE_MESSAGE)

[exec] /usr/local/share/cmake-2.8/Modules/FindOpenSSL.cmake:313 (find_package_handle_standard_args)

[exec] CMakeLists.txt:20 (find_package)

[exec]

[exec]

[exec] -- Configuring incomplete, errors occurred!

[exec] See also "/data/hadoop/hadoop-2.4.0-src/hadoop-tools/hadoop-pipes/target/native/CMakeFiles/CMakeOutput.log".

[INFO] ------------------------------------------------------------------------

[INFO] Reactor Summary:

[INFO]

[INFO] Apache Hadoop Main ................................ SUCCESS [13.745s]

[INFO] Apache Hadoop Project POM ......................... SUCCESS [5.538s]

[INFO] Apache Hadoop Annotations ......................... SUCCESS [7.296s]

[INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.568s]

[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [5.858s]

[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [8.541s]

[INFO] Apache Hadoop MiniKDC ............................. SUCCESS [8.337s]

[INFO] Apache Hadoop Auth ................................ SUCCESS [7.348s]

[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [4.926s]

[INFO] Apache Hadoop Common .............................. SUCCESS [2:35.956s]

[INFO] Apache Hadoop NFS ................................. SUCCESS [18.680s]

[INFO] Apache Hadoop Common Project ...................... SUCCESS [0.059s]

[INFO] Apache Hadoop HDFS ................................ SUCCESS [5:03.525s]

[INFO] Apache Hadoop HttpFS .............................. SUCCESS [38.335s]

[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [23.780s]

[INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [8.769s]

[INFO] Apache Hadoop HDFS Project ........................ SUCCESS [0.159s]

[INFO] hadoop-yarn ....................................... SUCCESS [0.134s]

[INFO] hadoop-yarn-api ................................... SUCCESS [2:07.657s]

[INFO] hadoop-yarn-common ................................ SUCCESS [1:10.680s]

[INFO] hadoop-yarn-server ................................ SUCCESS [0.165s]

[INFO] hadoop-yarn-server-common ......................... SUCCESS [24.174s]

[INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [27.293s]

[INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [5.177s]

[INFO] hadoop-yarn-server-applicationhistoryservice ...... SUCCESS [11.399s]

[INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [28.384s]

[INFO] hadoop-yarn-server-tests .......................... SUCCESS [1.346s]

[INFO] hadoop-yarn-client ................................ SUCCESS [12.937s]

[INFO] hadoop-yarn-applications .......................... SUCCESS [0.108s]

[INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [5.303s]

[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [3.212s]

[INFO] hadoop-yarn-site .................................. SUCCESS [0.050s]

[INFO] hadoop-yarn-project ............................... SUCCESS [8.638s]

[INFO] hadoop-mapreduce-client ........................... SUCCESS [0.135s]

[INFO] hadoop-mapreduce-client-core ...................... SUCCESS [43.622s]

[INFO] hadoop-mapreduce-client-common .................... SUCCESS [36.329s]

[INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [6.058s]

[INFO] hadoop-mapreduce-client-app ....................... SUCCESS [20.058s]

[INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [16.493s]

[INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [11.685s]

[INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [3.222s]

[INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [12.656s]

[INFO] hadoop-mapreduce .................................. SUCCESS [8.060s]

[INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [8.994s]

[INFO] Apache Hadoop Distributed Copy .................... SUCCESS [15.886s]

[INFO] Apache Hadoop Archives ............................ SUCCESS [6.659s]

[INFO] Apache Hadoop Rumen ............................... SUCCESS [15.722s]

[INFO] Apache Hadoop Gridmix ............................. SUCCESS [11.778s]

[INFO] Apache Hadoop Data Join ........................... SUCCESS [5.953s]

[INFO] Apache Hadoop Extras .............................. SUCCESS [6.414s]

[INFO] Apache Hadoop Pipes ............................... FAILURE [3.746s]

[INFO] Apache Hadoop OpenStack support ................... SKIPPED

[INFO] Apache Hadoop Client .............................. SKIPPED

[INFO] Apache Hadoop Mini-Cluster ........................ SKIPPED

[INFO] Apache Hadoop Scheduler Load Simulator ............ SKIPPED

[INFO] Apache Hadoop Tools Dist .......................... SKIPPED

[INFO] Apache Hadoop Tools ............................... SKIPPED

[INFO] Apache Hadoop Distribution ........................ SKIPPED

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 19:43.155s

[INFO] Finished at: Wed Jun 04 17:40:17 CST 2014

[INFO] Final Memory: 79M/239M

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-pipes: An Ant BuildException has occured: exec returned: 1

[ERROR] around Ant part ...<exec dir="/data/hadoop/hadoop-2.4.0-src/hadoop-tools/hadoop-pipes/target/native" executable="cmake" failonerror="true">... @ 5:123 in /data/hadoop/hadoop-2.4.0-src/hadoop-tools/hadoop-pipes/target/antrun/build-main.xml

[ERROR] -> [Help 1]

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

[ERROR]

[ERROR] After correcting the problems, you can resume the build with the command

根据网上提示（ 下面需要再安装openssl-devel,安装命令yum install openssl-devel,此步不做的话会报如下错误

[exec] CMake Error at /usr/share/cmake/Modules/FindOpenSSL.cmake:66 (MESSAGE): 
[exec]   Could NOT find OpenSSL 
[exec] Call Stack (most recent call first): 
[exec]   CMakeLists.txt:20 (find_package) 
[exec] 
[exec] 
[exec] -- Configuring incomplete, errors occurred! 
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (make) on project hadoop-pipes: An Ant BuildException has occured: exec returned: 1 -> [Help 1] 
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 
[ERROR] Re-run Maven using the -X switch to enable full debug logging. 
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles: 
[ERROR] [Help 1] http://cwiki.apache.org/confluen ... oExecutionException 
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command 
[ERROR]   mvn <goals> -rf :hadoop-pipes

）

错误连接：http://f.dataguru.cn/thread-189176-1-1.html

原因是：在安装openssl-devel,少写一个l,重新安装一下

解决方法：重新安装openssl-devel

yum install openssl-devel

5.3.3 编译总结

1、 必须安装（yum install lzo-devel zlib-devel gcc autoconf automake libtool ncurses-devel openssl-devel）

2、 必须安装（protobuf，CMake）编译工具

3、 必须配置（ANT、MAVEN、FindBugs）

4、 将maven库指向开源中国，这样就可以加快编译速度，即加快下载依赖jar包速度

5、 编译出错需求详细观察出错日志，根据错误日志分析原因再结束百度和Google解决错误；