Oozie错误-当从mysql导入数据到hdfs时

Question

我试图通过sqoop将数据从mysql导入到hdfs。我能够在shell中运行sqoop，它运行得很好。但当我通过oozie调用时。它是随着错误而产生的，并且会失败。我还包括了罐子。

现在我正在我的家庭-CDH5.3虚拟盒中的单个节点上尝试，但是错误仍然是相同的。

我知道前面已经讨论过这件事了，但还是没有帮助。

主要错误是：

ERROR, reason: Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]

作业属性文件如下所示，在本地环境中。

nameNode=hdfs://localhost:8020
jobTracker=localhost:8032

oozie.use.system.libpath=true
oozie.wf.application.path=${nameNode}/user/harry/

Workflow.xml文件如下所示，在HDFS中${nameNode}/user/harry/

<workflow-app name="Trend_Analysi_Scoop_Script" xmlns="uri:oozie:workflow:0.4">
<start to="sqoop-c68d"/>
<kill name="Kill">
    <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="sqoop-c68d">
    <sqoop xmlns="uri:oozie:sqoop-action:0.2">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <arg>import</arg>
        <arg> --connect</arg>
        <arg>jdbc:mysql://localhost/oozie_training</arg>
        <arg>--username</arg>
        <arg>root</arg>
        <arg>--password</arg>
        <arg>cloudera</arg>
        <arg>--table</arg>
        <arg>user</arg>
        <arg>--target-dir</arg>
        <arg>/training/sqoop</arg>
        <arg>-m</arg>
        <arg>1</arg>
    </sqoop>
    <ok to="End"/>
    <error to="Kill"/>
</action>
<end name="End"/>

​

我已经把所有必要的罐子都列在下面的小路下面了

[cloudera@quickstart ~]$ hadoop fs -ls /user/oozie/share/lib/lib_20141218070949/sqoop
Found 75 items
-rw-r--r--   1 root  oozie     960374 2016-05-26 22:00 /user/oozie/share/lib/lib_20141218070949/sqoop/mysql-connector-java-5.1.34-bin.jar
-rw-r--r--   1 oozie oozie      16924 2014-12-18 07:09 /user/oozie/share/lib/lib_20141218070949/sqoop/oozie-sharelib-sqoop-4.0.0-cdh5.3.0.jar
-rw-r--r--   1 oozie oozie    1010790 2014-12-18 07:09 /user/oozie/share/lib/lib_20141218070949/sqoop/sqoop-1.4.5-cdh5.3.0.jar

在添加jars之后，我还重新启动了oozie服务器。

我得到的错误日志如下所示。

2016-05-24 10:37:17,994  INFO ActionStartXCommand:520 - SERVER[***]             USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@:start:] Start action [0000001-160524103133053-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-05-24 10:37:17,997  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@:start:] [***0000001-160524103133053-oozie-oozi-W@:start:***]Action status=DONE
2016-05-24 10:37:17,997  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@:start:] [***0000001-160524103133053-oozie-oozi-W@:start:***]Action updated in DB!
2016-05-24 10:37:18,164  INFO WorkflowNotificationXCommand:520 - SERVER[***] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000001-160524103133053-oozie-oozi-W@:start:
2016-05-24 10:37:18,165  INFO WorkflowNotificationXCommand:520 - SERVER[***] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000001-160524103133053-oozie-oozi-W
2016-05-24 10:37:18,188  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] Start action [0000001-160524103133053-oozie-oozi-W@sqoop-c68d] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-05-24 10:37:26,080  INFO SqoopActionExecutor:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] Trying to get job [job_1462295072181_0163], attempt [1]
2016-05-24 10:37:26,106  INFO SqoopActionExecutor:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] checking action, hadoop job ID [job_1462295072181_0163] status [RUNNING]
2016-05-24 10:37:26,108  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] [***0000001-160524103133053-oozie-oozi-W@sqoop-c68d***]Action status=RUNNING
2016-05-24 10:37:26,114  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] [***0000001-160524103133053-oozie-oozi-W@sqoop-c68d***]Action updated in DB!
2016-05-24 10:37:26,184  INFO WorkflowNotificationXCommand:520 - SERVER[***] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] No Notification URL is defined. Therefore nothing to notify for job 0000001-160524103133053-oozie-oozi-W@sqoop-c68d
2016-05-24 10:37:37,181  INFO CallbackServlet:520 - SERVER[***] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] callback for action [0000001-160524103133053-oozie-oozi-W@sqoop-c68d]
2016-05-24 10:37:37,314  INFO SqoopActionExecutor:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] Trying to get job [job_1462295072181_0163], attempt [1]
2016-05-24 10:37:37,388  INFO SqoopActionExecutor:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] action completed, external ID [job_1462295072181_0163]
2016-05-24 10:37:37,395  WARN SqoopActionExecutor:523 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]
2016-05-24 10:37:37,484  INFO ActionEndXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] ERROR is considered as FAILED for SLA
2016-05-24 10:37:37,679  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@Kill] Start action [0000001-160524103133053-oozie-oozi-W@Kill] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-05-24 10:37:37,683  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@Kill] [***0000001-160524103133053-oozie-oozi-W@Kill***]Action status=DONE
2016-05-24 10:37:37,683  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@Kill] [***0000001-160524103133053-oozie-oozi-W@Kill***]Action updated in DB!
2016-05-24 10:37:37,930  INFO WorkflowNotificationXCommand:520 - SERVER[***] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@Kill] No Notification URL is defined. Therefore nothing to notify for job 0000001-160524103133053-oozie-oozi-W@Kill
2016-05-24 10:37:37,931  INFO WorkflowNotificationXCommand:520 - SERVER[***] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000001-160524103133053-oozie-oozi-W
2016-05-24 10:37:37,931  INFO WorkflowNotificationXCommand:520 - SERVER[***] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] No Notification URL is defined. Therefore nothing to notify for job 0000001-160524103133053-oozie-oozi- W@sqoop-c68d

--我认为oozie没有获得共享/库，如果是这样的话，如果有人能在同一个上帮助我，那就太好了。

任何帮助都将不胜感激。

Answer 1

看起来，sqoop操作中的“命令”标记有一些问题。因为它已经是sqoop操作"/usr/hdp/2.4.0.0-169/sqoop/bin/sqoop“是不需要的。请参考下面链接中的示例来更正它们。再试着重新运行这份工作。

Action

Answer 2

在--query元素中使用command选项，oozie中不支持这个选项。您需要使用arg标记并单独提供所有参数。command标记是由空格分隔的，因此它不能使用--query选项。下面是示例示例：

        <arg>import</arg>
        <arg>--connect</arg>
        <arg>jdbc:mysql://host:3306/oozie</arg>
        <arg>--username</arg>
        <arg>user</arg>
        <arg>--password</arg>
        <arg>pwd</arg>
        <arg>--driver</arg>
        <arg>com.mysql.jdbc.Driver</arg>
        <arg>--query</arg>
        <arg>"select BUNDLE_ACTIONS.* from BUNDLE_ACTIONS where $CONDITIONS"</arg>
        <arg>--hcatalog-table</arg>
        <arg>bundle_actions</arg>
        <arg>--hcatalog-database</arg>
        <arg>oozie</arg>
        <arg>--skip-dist-cache</arg>
        <arg>-m</arg>
        <arg>1</arg>