Sqoop无法将MySQL导入Hadoop，抛出ClassNotFoundException

Question

我安装了Sqoop 1.4.6，并尝试使用以下命令将一些数据从MySQL 5.6.29导入到Hadoop2.7.2版本中：

sqoop import --connect jdbc:mysql://<server_name>:3306/<dbname> --username <user-name> --password <password> --m 1 --table <table-name> --target-dir <hadoop-target-folder>

然后，我得到以下错误：

java.lang.Exception: java.lang.RuntimeException: 
java.lang.ClassNotFoundException: Class <ClassName> not found 
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class <ClassName> not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)
at org.apache.sqoop.mapreduce.db.DBConfiguration.getInputClass(DBConfiguration.java:403)
at org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.createDBRecordReader(DataDrivenDBInputFormat.java:237)
at org.apache.sqoop.mapreduce.db.DBInputFormat.createRecordReader(DBInputFormat.java:263)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:515)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:758)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: Class <ClassName> not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
... 12 more

可能有类似的问题，但我没有找到答案...

有什么想法吗？谢谢

Answer 1

生成jar：scoop codegen --connect "jdbc:mysql://example.com/sqoop" --username sqoop --password sqoop --table tbl

查找输出中的行:正在写入jar文件：

/tmp/sqoop-abe/compile/4182409b0f559d7259a32a950627e1aa/tbl.jar

在导入命令中使用上面的jar：

sqoop import -fs local -jt local -libjars /tmp/sqoop-abe/compile/4182409b0f559d7259a32a950627e1aa/tbl.jar --connect "jdbc:mysql://example.com/sqoop" --username sqoop --password sqoop --table tbl

Answer 2

根本原因是，当使用LocalJobRunner执行导入作业时，它无法加载由sqoop生成的jar。

解决方案1.将hadoop更改为在单节点集群模式http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html中运行。

解决方案2.首先生成jar文件，然后使用-libjars参数。