感谢支持
我们一直在努力

Ubuntu 搭建Hadoop源码阅读环境

JDK,Eclipse的安装请参考:Ubuntu 安装 搭建 J2EE 开发环境 http://www.linuxidc.com/Linux/2013-11/92526.htm

Ant安装:

下载ant:http://ant.apache.org/bindownload.cgi

解压Ant:

tar -xvzf apache-ant-1.9.2-bin.tar.gz

移动重命名:

  mv apache-ant-1.9.2/ ~/opt/
  mv apache-ant-1.9.2/ ant

配置环境变量,添加ANT_HOME,和$ANT_HOME/bin:

sudo vim /etc/profile

  …………………..
  export ANT_HOME=/opt/ant
  export PATH=$PATH:$ANT_HOME/bin
 …………………..

source /etc/profile

测试是否生效:

 ant -version
Apache Ant(TM) version 1.9.2 compiled on July 8 2013

Hadoop Eclipse 插件安装:

Hadoop的eclipse插件需要自行安装编译,也很简单,用ant进行编译打包

cd /opt/hadoop-1.2.1/src/contrib/eclipse-plugin

参考这篇文章:http://www.linuxidc.com/Linux/2013-11/92528.htm

此处贴出build.xml全部和META-INFO改动的:

build.xml

<?xml version=”1.0″ encoding=”UTF-8″ standalone=”no”?> 

<!– 
Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license agreements.  See the NOTICE file distributed with 
this work for additional information regarding copyright ownership. 
The ASF licenses this file to You under the Apache License, Version 2.0 
(the “License”); you may not use this file except in compliance with 
the License.  You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0 

Unless required by applicable law or agreed to in writing, software 
distributed under the License is distributed on an “AS IS” BASIS, 
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 
   See the License for the specific language governing permissions and 
   limitations under the License. 
   –> 

   <!– build.xml –> 
   <project default=”jar” name=”eclipse-plugin”> 

   <import file=”../build-contrib.xml”/> 

   <property name=”eclipse.home” location=”/opt/eclipse”/> 
   <property name=”version” value=”1.2.1″/> 

   <path id=”eclipse-sdk-jars”> 
   <fileset dir=”${eclipse.home}/plugins/”> 
   <include name=”org.eclipse.ui*.jar”/> 
   <include name=”org.eclipse.jdt*.jar”/> 
   <include name=”org.eclipse.core*.jar”/> 
   <include name=”org.eclipse.equinox*.jar”/> 
   <include name=”org.eclipse.debug*.jar”/> 
   <include name=”org.eclipse.osgi*.jar”/> 
   <include name=”org.eclipse.swt*.jar”/> 
   <include name=”org.eclipse.jface*.jar”/> 

   <include name=”org.eclipse.team.cvs.ssh2*.jar”/> 
   <include name=”com.jcraft.jsch*.jar”/> 
   </fileset> 
   </path> 

   <!– Override classpath to include Eclipse SDK jars –> 
   <path id=”classpath”> 
   <pathelement location=”${build.classes}”/> 
   <pathelement location=”${hadoop.root}/build/classes”/> 
   <fileset dir=”${hadoop.root}”> 
   <include name=”**/*.jar” /> 
   </fileset> 
   <path refid=”eclipse-sdk-jars”/> 
   </path> 

   <!– Skip building if eclipse.home is unset. –> 
   <target name=”check-contrib” unless=”eclipse.home”> 
   <property name=”skip.contrib” value=”yes”/> 
   <echo message=”eclipse.home unset: skipping eclipse plugin”/> 
   </target> 

   <target name=”compile” depends=”init, ivy-retrieve-common” unless=”skip.contrib”> 
   <echo message=”contrib: ${name}”/> 
   <javac 
   encoding=”${build.encoding}” 
   srcdir=”${src.dir}” 
   includes=”**/*.java” 
   destdir=”${build.classes}” 
   debug=”${javac.debug}” 
   deprecation=”${javac.deprecation}” 
   includeantruntime=”on”> 
   <classpath refid=”classpath”/> 
   </javac> 
   </target> 

   <!– Override jar target to specify manifest –> 
   <target name=”jar” depends=”compile” unless=”skip.contrib”> 
   <mkdir dir=”${build.dir}/lib”/> 
   <!–  <copy file=”${hadoop.root}/build/hadoop-core-${version}.jar” tofile=”${build.dir}/lib/hadoop-core.jar” verbose=”true”/> –> 
   <!–  <copy file=”${hadoop.root}/build/ivy/lib/Hadoop/common/commons-cli-${commons-cli.version}.jar”  todir=”${build.dir}/lib” verbose=”true”/>–> 
   <copy file=”${hadoop.root}/hadoop-core-${version}.jar” tofile=”${build.dir}/lib/hadoop-core.jar” verbose=”true”/> 
   <copy file=”${hadoop.root}/lib/commons-cli-${commons-cli.version}.jar”  tofile=”${build.dir}/lib/commons-cli.jar” verbose=”true”/> 
   <copy file=”${hadoop.root}/lib/commons-configuration-1.6.jar”  tofile=”${build.dir}/lib/commons-configuration.jar” verbose=”true”/> 
   <copy file=”${hadoop.root}/lib/commons-httpclient-3.0.1.jar”  tofile=”${build.dir}/lib/commons-httpclient.jar” verbose=”true”/> 
   <copy file=”${hadoop.root}/lib/commons-lang-2.4.jar”  tofile=”${build.dir}/lib/commons-lang.jar” verbose=”true”/> 
   <copy file=”${hadoop.root}/lib/jackson-core-asl-1.8.8.jar”  tofile=”${build.dir}/lib/jackson-core-asl.jar” verbose=”true”/> 
   <copy file=”${hadoop.root}/lib/jackson-mapper-asl-1.8.8.jar”  tofile=”${build.dir}/lib/jackson-mapper-asl.jar” verbose=”true”/> 

   <jar 
   jarfile=”${build.dir}/hadoop-${name}-${version}.jar” 
   manifest=”${root}/META-INF/MANIFEST.MF”> 
   <fileset dir=”${build.dir}” includes=”classes/ lib/”/> 
   <fileset dir=”${root}” includes=”resources/ plugin.xml”/> 
   </jar> 
   </target> 

   </project> 

更多详情见请继续阅读下一页的精彩内容: http://www.linuxidc.com/Linux/2013-11/92527p2.htm

相关阅读

Ubuntu 13.04上搭建Hadoop环境  http://www.linuxidc.com/Linux/2013-06/86106.htm

Ubuntu上搭建Hadoop环境(单机模式+伪分布模式) http://www.linuxidc.com/Linux/2013-01/77681.htm

Ubuntu下Hadoop环境的配置 http://www.linuxidc.com/Linux/2012-11/74539.htm

单机版搭建Hadoop环境图文教程详解 http://www.linuxidc.com/Linux/2012-02/53927.htm

搭建Hadoop环境(在Winodws环境下用虚拟机虚拟两个Ubuntu系统进行搭建)  http://www.linuxidc.com/Linux/2011-12/48894.htm

META-INFO:

Bundle-ClassPath: classes/,lib/commons-cli.jar,lib/commons-httpclient.jar,lib/Hadoop-core.jar,lib/jackson-mapper-asl.jar,lib/commons-configuration.jar,lib/commons-lang.jar,lib/jackson-core-asl.jar

生成的jar包位于:/opt/hadoop-1.2.1/build/contrib/eclipse-plugin/hadoop-eclipse-plugin-1.2.1.jar,从ant的输出信息可以看到

复制插件,重启Eclipse

sudo mv hadoop-eclipse-plugin-1.2.1.jar /opt/eclipse/plugins/

设置Eclipse:

 

windows>show views>other>mapreduce location,新建一个,如图设这

 

启动hadoop,查看project explorer,如图所示

 

OK~~!插件安装成功。

 

导入hadoop源码,太简单,不解释。。。

 

用Eclips的插件建立新文件夹,上传文件到目录中

把example下的wordcount搞过来运行一下吧,报错:

 

13/11/08 18:20:39 ERROR security.UserGroupInformation: PriviledgedActionException as:dat cause:org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: file:/user/dat/input

很明显,权限问题

试了半天,很明显,我又误导你了,可能使权限问题,也可能不是,我这边这个就不是权限问题

Run Configuration参数需要配置完整uri路径

Arguments配置如下:

hdfs://localhost:9000/user/dat/input hdfs://localhost:9000/user/dat/output

输出路径hadoop帮你建立,不能已经存在

继续run on hadoop ,ok!!没有错了是不是?!

留图纪念

调试Hadoop源代码:

启动Hadoop调试模式:

调试JobTracker为例:

 

[plain] view plaincopyprint?

  1. export HADOOP_JOBTRACKER_OPTS=”-Xdebug -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=y”
export HADOOP_JOBTRACKER_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=y"

然后再启动start-all.sh,调试启动成功会看到如下信息,别的进程正常启动,JobTracker处于监听状态,如下图所示。

 

进入Eclipse进行调试,设置参数

Run>Debug Configuration,配置参数如下

 

debug开始,ok

可以尽情调试查看了!!有木有哈哈!!顺便吐槽一句:HP的Elite Book系列真心不适合程序员啊,那几个F1-F12连个空格都没有。。。以后还是选小黑吧。

 

这只是调试JobTracker的,还有别的,以后补上,debug模式和run模式来回切换很一大川参数很麻烦有木有,写个脚本。

更多Hadoop相关信息见Hadoop 专题页面 http://www.linuxidc.com/topicnews.aspx?tid=13

赞(0) 打赏
转载请注明出处:服务器评测 » Ubuntu 搭建Hadoop源码阅读环境
分享到: 更多 (0)

听说打赏我的人,都进福布斯排行榜啦!

支付宝扫一扫打赏

微信扫一扫打赏