JDK,Eclipse的安装请参考:Ubuntu 安装 搭建 J2EE 开发环境 http://www.linuxidc.com/Linux/2013-11/92526.htm
Ant安装:
下载ant:http://ant.apache.org/bindownload.cgi
解压Ant:
tar -xvzf apache-ant-1.9.2-bin.tar.gz
移动重命名:
mv apache-ant-1.9.2/ ~/opt/
mv apache-ant-1.9.2/ ant
配置环境变量,添加ANT_HOME,和$ANT_HOME/bin:
sudo vim /etc/profile
…………………..
export ANT_HOME=/opt/ant
export PATH=$PATH:$ANT_HOME/bin
…………………..
source /etc/profile
测试是否生效:
ant -version
Apache Ant(TM) version 1.9.2 compiled on July 8 2013
Hadoop Eclipse 插件安装:
Hadoop的eclipse插件需要自行安装编译,也很简单,用ant进行编译打包
cd /opt/hadoop-1.2.1/src/contrib/eclipse-plugin
参考这篇文章:http://www.linuxidc.com/Linux/2013-11/92528.htm
此处贴出build.xml全部和META-INFO改动的:
build.xml
<?xml version=”1.0″ encoding=”UTF-8″ standalone=”no”?>
<!–
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the “License”); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an “AS IS” BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
–>
<!– build.xml –>
<project default=”jar” name=”eclipse-plugin”>
<import file=”../build-contrib.xml”/>
<property name=”eclipse.home” location=”/opt/eclipse”/>
<property name=”version” value=”1.2.1″/>
<path id=”eclipse-sdk-jars”>
<fileset dir=”${eclipse.home}/plugins/”>
<include name=”org.eclipse.ui*.jar”/>
<include name=”org.eclipse.jdt*.jar”/>
<include name=”org.eclipse.core*.jar”/>
<include name=”org.eclipse.equinox*.jar”/>
<include name=”org.eclipse.debug*.jar”/>
<include name=”org.eclipse.osgi*.jar”/>
<include name=”org.eclipse.swt*.jar”/>
<include name=”org.eclipse.jface*.jar”/>
<include name=”org.eclipse.team.cvs.ssh2*.jar”/>
<include name=”com.jcraft.jsch*.jar”/>
</fileset>
</path>
<!– Override classpath to include Eclipse SDK jars –>
<path id=”classpath”>
<pathelement location=”${build.classes}”/>
<pathelement location=”${hadoop.root}/build/classes”/>
<fileset dir=”${hadoop.root}”>
<include name=”**/*.jar” />
</fileset>
<path refid=”eclipse-sdk-jars”/>
</path>
<!– Skip building if eclipse.home is unset. –>
<target name=”check-contrib” unless=”eclipse.home”>
<property name=”skip.contrib” value=”yes”/>
<echo message=”eclipse.home unset: skipping eclipse plugin”/>
</target>
<target name=”compile” depends=”init, ivy-retrieve-common” unless=”skip.contrib”>
<echo message=”contrib: ${name}”/>
<javac
encoding=”${build.encoding}”
srcdir=”${src.dir}”
includes=”**/*.java”
destdir=”${build.classes}”
debug=”${javac.debug}”
deprecation=”${javac.deprecation}”
includeantruntime=”on”>
<classpath refid=”classpath”/>
</javac>
</target>
<!– Override jar target to specify manifest –>
<target name=”jar” depends=”compile” unless=”skip.contrib”>
<mkdir dir=”${build.dir}/lib”/>
<!– <copy file=”${hadoop.root}/build/hadoop-core-${version}.jar” tofile=”${build.dir}/lib/hadoop-core.jar” verbose=”true”/> –>
<!– <copy file=”${hadoop.root}/build/ivy/lib/Hadoop/common/commons-cli-${commons-cli.version}.jar” todir=”${build.dir}/lib” verbose=”true”/>–>
<copy file=”${hadoop.root}/hadoop-core-${version}.jar” tofile=”${build.dir}/lib/hadoop-core.jar” verbose=”true”/>
<copy file=”${hadoop.root}/lib/commons-cli-${commons-cli.version}.jar” tofile=”${build.dir}/lib/commons-cli.jar” verbose=”true”/>
<copy file=”${hadoop.root}/lib/commons-configuration-1.6.jar” tofile=”${build.dir}/lib/commons-configuration.jar” verbose=”true”/>
<copy file=”${hadoop.root}/lib/commons-httpclient-3.0.1.jar” tofile=”${build.dir}/lib/commons-httpclient.jar” verbose=”true”/>
<copy file=”${hadoop.root}/lib/commons-lang-2.4.jar” tofile=”${build.dir}/lib/commons-lang.jar” verbose=”true”/>
<copy file=”${hadoop.root}/lib/jackson-core-asl-1.8.8.jar” tofile=”${build.dir}/lib/jackson-core-asl.jar” verbose=”true”/>
<copy file=”${hadoop.root}/lib/jackson-mapper-asl-1.8.8.jar” tofile=”${build.dir}/lib/jackson-mapper-asl.jar” verbose=”true”/>
<jar
jarfile=”${build.dir}/hadoop-${name}-${version}.jar”
manifest=”${root}/META-INF/MANIFEST.MF”>
<fileset dir=”${build.dir}” includes=”classes/ lib/”/>
<fileset dir=”${root}” includes=”resources/ plugin.xml”/>
</jar>
</target>
</project>
更多详情见请继续阅读下一页的精彩内容: http://www.linuxidc.com/Linux/2013-11/92527p2.htm
相关阅读:
Ubuntu 13.04上搭建Hadoop环境 http://www.linuxidc.com/Linux/2013-06/86106.htm
Ubuntu上搭建Hadoop环境(单机模式+伪分布模式) http://www.linuxidc.com/Linux/2013-01/77681.htm
Ubuntu下Hadoop环境的配置 http://www.linuxidc.com/Linux/2012-11/74539.htm
单机版搭建Hadoop环境图文教程详解 http://www.linuxidc.com/Linux/2012-02/53927.htm
搭建Hadoop环境(在Winodws环境下用虚拟机虚拟两个Ubuntu系统进行搭建) http://www.linuxidc.com/Linux/2011-12/48894.htm
META-INFO:
Bundle-ClassPath: classes/,lib/commons-cli.jar,lib/commons-httpclient.jar,lib/Hadoop-core.jar,lib/jackson-mapper-asl.jar,lib/commons-configuration.jar,lib/commons-lang.jar,lib/jackson-core-asl.jar
生成的jar包位于:/opt/hadoop-1.2.1/build/contrib/eclipse-plugin/hadoop-eclipse-plugin-1.2.1.jar,从ant的输出信息可以看到
复制插件,重启Eclipse
sudo mv hadoop-eclipse-plugin-1.2.1.jar /opt/eclipse/plugins/
设置Eclipse:
windows>show views>other>mapreduce location,新建一个,如图设这
启动hadoop,查看project explorer,如图所示
OK~~!插件安装成功。
导入hadoop源码,太简单,不解释。。。
用Eclips的插件建立新文件夹,上传文件到目录中
把example下的wordcount搞过来运行一下吧,报错:
13/11/08 18:20:39 ERROR security.UserGroupInformation: PriviledgedActionException as:dat cause:org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: file:/user/dat/input
很明显,权限问题
试了半天,很明显,我又误导你了,可能使权限问题,也可能不是,我这边这个就不是权限问题
Run Configuration参数需要配置完整uri路径
Arguments配置如下:
hdfs://localhost:9000/user/dat/input hdfs://localhost:9000/user/dat/output
输出路径hadoop帮你建立,不能已经存在
继续run on hadoop ,ok!!没有错了是不是?!
留图纪念
调试Hadoop源代码:
启动Hadoop调试模式:
调试JobTracker为例:
[plain] view plaincopyprint?
- export HADOOP_JOBTRACKER_OPTS=”-Xdebug -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=y”
export HADOOP_JOBTRACKER_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=y"
然后再启动start-all.sh,调试启动成功会看到如下信息,别的进程正常启动,JobTracker处于监听状态,如下图所示。
进入Eclipse进行调试,设置参数
Run>Debug Configuration,配置参数如下
debug开始,ok
可以尽情调试查看了!!有木有哈哈!!顺便吐槽一句:HP的Elite Book系列真心不适合程序员啊,那几个F1-F12连个空格都没有。。。以后还是选小黑吧。
这只是调试JobTracker的,还有别的,以后补上,debug模式和run模式来回切换很一大川参数很麻烦有木有,写个脚本。
更多Hadoop相关信息见Hadoop 专题页面 http://www.linuxidc.com/topicnews.aspx?tid=13