未能将数据从Hive加载到ElasticSearch

Question

我目前正在尝试将数据从Hive加载到ElasticSearch。我在用cloudera CDH 5.3。我已经把2.0.2 jar添加到我的蜂巢路径中了。我已经在10.44.162.169上启动并运行了ElasticSearch 1.4.4。

现在我有了一个名为hive_cdr的表，它具有以下属性：

 traffic_type_id (big int)
 appelant (int)
 called_number (int)
 call_duration (int)
 location_number (string)
 date_heure_appel(string)

我试图在我的蜂箱中定义ES表来加载一些数据。为了做到这一点，我做了以下工作：

CREATE EXTERNAL TABLE es_hive_cdr (
traffic bigint ,
calling int ,
called int ,
duration int ,
location string ,
date string )
ROW FORMAT SERDE 'org.elasticsearch.hadoop.hive.EsSerDe'
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES (
'es.nodes'='10.44.162.169',
'es.resource'='indexCDR/typeCDR'
) ;

但是，我得到了这样一个例外，即EsStorage是不被识别的。

我删除了EsStorage行，并执行它，试图找出到底发生了什么。

现在尝试将数据从我的hive_cdr表加载到我的新表：

insert into table es_hive_cdr2
select
traffic_type_id,
appelant,
called_number,
call_duration,
location_number,
date_heure_appel
from hive_cdr;

但是它失败了，我得到了一个错误：

处理语句时出现错误:失败:执行错误，从返回代码2

阶段依赖性：

  Stage-1 is a root stage
  Stage-7 depends on stages: Stage-1 , consists of Stage-4, Stage-3, Stage-5
  Stage-4
  Stage-0 depends on stages: Stage-4, Stage-3, Stage-6
  Stage-2 depends on stages: Stage-0
  Stage-3
  Stage-5
  Stage-6 depends on stages: Stage-5

阶段计划：

  Stage: Stage-1
    Map Reduce
      Map Operator Tree:
          TableScan
            alias: hive_cdr
            Statistics: Num rows: 267130 Data size: 58768736 Basic stats: COMPLETE Column stats: NONE
            Select Operator
              expressions: traffic_type_id (type: bigint), appelant (type: int), called_number (type: int), call_duration (type: int), location_number (type: string), date_heure_appel (type: string)
              outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
              Statistics: Num rows: 267130 Data size: 58768736 Basic stats: COMPLETE Column stats: NONE
              File Output Operator
                compressed: false
                Statistics: Num rows: 267130 Data size: 58768736 Basic stats: COMPLETE Column stats: NONE
                table:
                    input format: org.apache.hadoop.mapred.TextInputFormat
                    output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                    serde: org.elasticsearch.hadoop.hive.EsSerDe
                    name: default.es_hive_cdr2

  Stage: Stage-7
    Conditional Operator

  Stage: Stage-4
    Move Operator
      files:
          hdfs directory: true
          destination: hdfs://master:8020/user/hive/warehouse/es_hive_cdr2/.hive-staging_hive_2015-03-02_14-09-08_285_4734041865540737822-2/-ext-10000

  Stage: Stage-0
    Move Operator
      tables:
          replace: false
          table:
              input format: org.apache.hadoop.mapred.TextInputFormat
              output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
              serde: org.elasticsearch.hadoop.hive.EsSerDe
              name: default.es_hive_cdr2

  Stage: Stage-2
    Stats-Aggr Operator

  Stage: Stage-3
    Map Reduce
      Map Operator Tree:
          TableScan
            File Output Operator
              compressed: false
              table:
                  input format: org.apache.hadoop.mapred.TextInputFormat
                  output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                  serde: org.elasticsearch.hadoop.hive.EsSerDe
                  name: default.es_hive_cdr2

  Stage: Stage-5
    Map Reduce
      Map Operator Tree:
          TableScan
            File Output Operator
              compressed: false
              table:
                  input format: org.apache.hadoop.mapred.TextInputFormat
                  output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                  serde: org.elasticsearch.hadoop.hive.EsSerDe
                  name: default.es_hive_cdr2

  Stage: Stage-6
    Move Operator
      files:
          hdfs directory: true
          destination: hdfs://master:8020/user/hive/warehouse/es_hive_cdr2/.hive-staging_hive_2015-03-02_14-09-08_285_4734041865540737822-2/-ext-10000

我真的需要一些帮助和指导，并且要感激和感激你！

Answer 1

尝试提供表属性。

TBLPROPERTIES('es.resource‘=’myview/myview‘，'es.nodes’=‘hostname-of-es-群集’，'es.port‘= '9200'，'es.input.json’= 'false'，'es.write.operation‘= 'index'，'es.index.auto.create’= 'yes'，'es.nodes.wan.only‘= 'true')；

还将elasticsearch.yml文件中的属性更改为

network.host：_site_