sqoop遇到的问题 - yingziaiai/SetupEnv GitHub Wiki

http://blog.csdn.net/liyaohhh/article/details/50678135 hive入门们学习：orcFile和parquet存储格式简介 http://blog.selfup.cn/1510.html Sqoop2初步使用感受

java.lang.UnsatisfiedLinkError: no snappyjava in java.library.path Error http://crunchify.com/how-to-fix-java-lang-unsatisfiedlinkerror-no-snappyjava-in-java-library-path-error/ http://blog.csdn.net/stark_summer/article/details/47361603 http://www.th7.cn/system/lin/201508/128600.shtml

按照测试方法： 依赖： org.xerial.snappy snappy-java 1.1.1.4 jar compile 源码： public class TestSnappy { public static void main(String[] args) throws Exception{

    String input = "Hello snappy-java! Snappy-java is a JNI-based wrapper of "
            + "Snappy, a fast compresser/decompresser.";
    byte[] compressed = Snappy.compress(input.getBytes("UTF-8"));
    byte[] uncompressed = Snappy.uncompress(compressed);

    String result = new String(uncompressed, "UTF-8");
    System.out.println(result);
}

} 结果测试没有错误，于是换成以下版本，结果报错了，于是将版本snappy-java-1.0.0.4替换成1.1.1.4，然后重新启动hadoop,再使用sqoop的--as-paquet就不报错了。

CDH5.3.6下的hadoop2.5.0中的/HADOOP_HOME/share/hadoop/common/lib下的snappy版本是snappy-java-1.0.0.4

文件压缩格式讲解： http://blog.csdn.net/bigdatahappy/article/details/42087317

在学习sqoop进行文件压缩以支持snappy（即将数据压缩存储在hdfs,以及将hdfs上的压缩文件导出到hive表中），依旧输入命令测试压缩时报错如下： bin/sqoop import --connect jdbc:mysql://localhost:3306/test --username root --password woshiwo --table my_user --target-dir /user/fuying/sqoop/import_my_user_snappy --num-mappers 1 --compress --compression-codec org.apache.hadoop.io.compress.SnappyCodec

2016-10-11 21:47:45,568 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method) at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63) at org.apache.hadoop.io.compress.SnappyCodec.createCompressor(SnappyCodec.java:143) at org.apache.hadoop.io.compress.SnappyCodec.createOutputStream(SnappyCodec.java:98) at org.apache.sqoop.mapreduce.RawKeyTextOutputFormat.getRecordWriter(RawKeyTextOutputFormat.java:102) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.(MapTask.java:644) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

于是开始编译源码以加载native库： 但在Mac下，遇到了各种各样折问题： 理想情况应该是：http://blog.csdn.net/tterminator/article/details/51779689 七、将编译出的native library复制到下载的二进制版本的Hadoop 2.6.0相应目录中

编译出的native library库的位置为 hadoop-2.6.0-src/hadoop-dist/target/hadoop-2.6.0/lib/native 拷贝到二进制版本的Hadoop 2.6.0的目录 hadoop-2.6.0/lib/native 八、修改/etc/hadoop/hadoop-env.sh配置

export HADOOP_OPTS=”$HADOOP_OPTS -Djava.net.preferIPv4Stack=true -Djava.library.path=/hadoop-2.6.0/lib/native”

可一直编译报错： 错误种类1: couldn't return protobuf version:

http://www.tuicool.com/articles/jM7Nn2/ protobuf-2.5.0.tar.gz的下载与安装

错误种类2: Execute failed: java.io.IOException: Cannot run program "cmake" (in director http://blog.csdn.net/eli00001/article/details/40082083 于是安装了cmake，但还是报错，继续查找，发现是没有安装Snappy, 于是先安装

错误种类3: http://f.dataguru.cn/thread-253905-1-1.html http://blog.csdn.net/zhulin40/article/details/38299249 一般情况下/root/soft/hadoop/hadoop-2.4.1-src/hadoop-hdfs-project/hadoop-hdfs-httpfs/downloads路径下的apache-tomcat-6.0.36.tar.gz下载不完全，

正常大小应该为6.47M左右，如果不对请官网手动下载，路径为http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.36/bin/apache-tomcat-6.0.36.tar.gz

错误种类4: http://rockyfeng.me/hadoop_native_library_mac.html Exception in thread "main" java.lang.AssertionError: Missing tools.jar at: /Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/Classes/classes.jar. Expression: file.exists()

解决办法,

sudo mkdir $JAVA_HOME/Classes sudo ln -sf $JAVA_HOME/lib/tools.jar $JAVA_HOME/Classes/classes.jar

错误种类5: 一直报此问题 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: exec returned: 1 [ERROR] around Ant part ...... @ 4:138 in /usr/local/hadoop/hadoop-2.4.0-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml [ERROR] -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: exec returned: 1 around Ant part ...<exec 有人说要安装ziplb, openssl

但Mac上根据装不上：

于是单独编译snappy http://www.linuxidc.com/Linux/2014-05/101091.htm Hadoop Snappy 压缩的安装和配置

https://github.com/electrum/hadoop-snappy

但依旧遇到错误5, http://blog.csdn.net/bruceleexiaokan/article/details/8145838 看过发现无解

于是想可不可以使用别人编译好的native包： http://blog.csdn.net/tterminator/article/details/51779689 https://github.com/mehikmat/hadoop-install/blob/cdh5/native-libs/native.tar.gz

又一个问题： bin/sqoop import --connect jdbc:mysql://localhost:3306/test --username root --password woshiwo --table my_user --target-dir /user/fuying/sqoop/imp_my_user_direct --num-mappers 1 --direct --delete-target-dir

运行--direct参数时会出现： 2016-10-13 22:40:40,216 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: Cannot run program "mysqldump": error=2, No such file or directory at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047) at java.lang.Runtime.exec(Runtime.java:617) at java.lang.Runtime.exec(Runtime.java:485) at org.apache.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:405) at org.apache.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:49) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.io.IOException: error=2, No such file or directory at java.lang.UNIXProcess.forkAndExec(Native Method) at java.lang.UNIXProcess.(UNIXProcess.java:185) at java.lang.ProcessImpl.start(ProcessImpl.java:130) at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028) ... 12 more

http://www.server110.com/mysql/201402/5659.html mysqldump使用法 http://www.cnblogs.com/JiangLe/p/5682795.html http://blog.csdn.net/joomlaer/article/details/9171203

⚠️ **GitHub.com Fallback** ⚠️