[分享] hadoop native lib 编译_Hadoop,ERP及大数据讨论区_Weblogic技术|Tuxedo技术|中间件技术|Oracle论坛|JAVA论坛|Linux/Unix技术|hadoop论坛_联动北方技术论坛  
网站首页 | 关于我们 | 服务中心 | 经验交流 | 公司荣誉 | 成功案例 | 合作伙伴 | 联系我们 |
联动北方-国内领先的云技术服务提供商
»  游客             当前位置:  论坛首页 »  自由讨论区 »  Hadoop,ERP及大数据讨论区 »
总帖数
1
每页帖数
101/1页1
返回列表
0
发起投票  发起投票 发新帖子
查看: 406 | 回复: 0   主题: [分享] hadoop native lib 编译        上一篇   下一篇 
eink
注册用户
等级:上尉
经验:764
发帖:54
精华:0
注册:2012-11-13
状态:离线
发送短消息息给eink 加好友    发送短消息息给eink 发消息
发表于: IP:您无权察看 2016-4-14 14:21:25 | [全部帖] [楼主帖] 楼主

编辑源代码 native libraries

安装maven

 

Wget –c http://mirror.bit.edu.cn/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz

tar -zxvf apache-maven-3.3.9-bin.tar.gz -C /home/hadoop

Vim /etc/profile

export MAVEN_HOME=/home/hadoop/apache-maven-3.3.9

export PATH= 添加 :$MAVEN_HOME/bin

 

maven 编译的时候需要jdk是1.7版本,所以需要在安装一个1.7版本的jdk

 

tar –zxvf jdk-7u55-linux-x64.tar.gz –C /usr/java/

Vim /etc/profile

export JAVA_HOME=/usr/java/jdk1.7.0_55

 

之后开始mvn编译

 

Tar –zxvf hadoop-2.6.0-cdh5.5.2-src.tar.gz

Cd hadoop-2.6.0-cdh5.5.2

yum install –y *gcc* ncurses-devel openssl-devel cmake autoconfautomake libtool bzip2-devel g++ autoconf automake libtool cmake zlib1g-dev pkg-config

mvn package -Pdist,native -DskipTests –Dtar

 

编译成功之后,把 native 目录下的lib包copy到 hadoop的安装目录下迟到lib目录
cp –a hadoop-dist/target/hadoop-2.7.2/lib/native/*  $HADOOP_HOME/lib/native/

 

报错问题整理:

hadoop-common —

[WARNING] [protoc, –version] failed:java.io.IOException: Cannot run program "protoc": error=2, No suchfile or directory

[ERROR] stdout: []

解决办法:

Tar –zxvf protobuf-2.5.0.tar.gz

Cd protobuf-2.5.0

./configure

Make

Make install

protoc –version
protoc: error while loading shared libraries: libprotobuf.so.8: cannot open shared object file: No such file or directory

ls /usr/local/lib

echo "/usr/local/lib"  > /etc/ld.so.conf.d/protocbuf.conf

Ldconfig

之后在执行 protoc –version
libprotoc 2.5.0

出现这个就表示好了, 之后在继续编译

 

编译也有可能出现以下错误:

 

[INFO] BUILD FAILURE
[INFO] ————————————————————————
[INFO] Total time: 4:34.649s
[INFO] Finished at: Tue Jun 03 10:18:25 CST 2014
[INFO] Final Memory: 39M/584M
[INFO] ————————————————————————
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile (default-testCompile) on project hadoop-auth: Compilation failure: Compilation failure:
[ERROR] /opt/linshuai/tardir/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[88,11] error: cannot access AbstractLifeCycle
[ERROR] class file for org.mortbay.component.AbstractLifeCycle not found
[ERROR] /tardir/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[96,29] error: cannot access LifeCycle
[ERROR] class file for org.mortbay.component.LifeCycle not found
[ERROR] /tardir/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[98,10] error: cannot find symbol
[ERROR] symbol: method start()
[ERROR] location: variable server of type Server
[ERROR] /tardir/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[104,12] error: cannot find symbol
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :hadoop-auth

问题解决 https://issues.apache.org/jira/browse/HADOOP-10110:

Index: hadoop-common-project/hadoop-auth/pom.xml

===================================================================

— hadoop-common-project/hadoop-auth/pom.xml                    (revision 1543124)

+++ hadoop-common-project/hadoop-auth/pom.xml                 (working copy)

@@ -54,6 +54,11 @@

     </dependency>

     <dependency>

       <groupId>org.mortbay.jetty</groupId>

+      <artifactId>jetty-util</artifactId>

+      <scope>test</scope>

+    </dependency>

+    <dependency>

+      <groupId>org.mortbay.jetty</groupId>

       <artifactId>jetty</artifactId>

       <scope>test</scope>

     </dependency>

然后mvn package -Pdist -DskipTests -rf :hadoop-auth

这里-rf 是只编译失败的项目。

 

 

最后检查,copy之后,可以用下面命令检查hadoop支持的格式

hadoop checknative –a

可以查看 支持的压缩属性





赞(0)    操作        顶端 
总帖数
1
每页帖数
101/1页1
返回列表
发新帖子
请输入验证码: 点击刷新验证码
您需要登录后才可以回帖 登录 | 注册
技术讨论