首页
学习
活动
专区
圈层
工具
发布
社区首页 >问答首页 >如何获取多条记录?

问如何获取多条记录?
EN

Stack Overflow用户
提问于 2020-11-08 22:30:58
回答 1查看 42关注 0票数 0

我正在使用JDBC和logstash从PostgreSQL查询中获取数据,并将其导出到ES v7,以下是配置文件:

代码语言:javascript
复制
input {
    jdbc {
        jdbc_connection_string => "jdbc:postgresql://ldatabase_rds_path?useSSL=true"
        jdbc_user => "username"
        jdbc_password => "password"
        jdbc_driver_library => "/home/z/Documents/postgresql-42.2.18.jar"
        jdbc_driver_class => "org.postgresql.Driver"
        tracking_column => "id"
        tracking_column_type => "numeric"
         clean_run => true
        schedule => "0 */1 * * *"
        statement => "SELECT id as id, type as type, z_id as z_id, sender_id as sender_id, receiver_id as receiver_id, status as status, amount as amount, fees as fees, created as created, metadata as metadata, funding_source_from_id as funding_source_from_id, funding_source_to_id as funding_source_to_id, is_parent as is_parent, destination_type as destination_type, source_type as source_type FROM payments_transfer"
    }
}
output {
        stdout { codec => json_lines }
        elasticsearch {
                hosts => ["localhost:9200"]
                manage_template => false
                index => "payments_transfer_data"
                document_id => "%{id}"
         }
}

从数据库中只获取一条记录需要花费很多时间!我尝试了一些解决方案,比如显式定义一个映射,所以我为数据添加了一个映射,如下所示:

代码语言:javascript
复制
PUT payments_transfer_data/_mapping/doc?include_type_name=true
{
  "properties": {
    "@timestamp": {
      "type": "date"
    },
    "@version": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword"
        }
      }
    },
    "amount": {
      "type": "float"
    },
    "created": {
      "type": "date"
    },
    "destination_type": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword"
        }
      }
    },
    "z_id": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword"
        }
      }
    },
    "fees": {
      "type": "float"
    },
    "funding_source_from_id": {
      "type": "long"
    },
    "funding_source_to_id": {
      "type": "long"
    },
    "id": {
      "type": "long"
    },
    "is_parent": {
      "type": "boolean"
    },
    "metadata": {
      "type": "keyword"
    },
    "receiver_id": {
      "type": "long"
    },
    "sender_id": {
      "type": "long"
    },
    "source_type": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword"
        }
      }
    },
    "status": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword"
        }
      }
    },
    "type": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword"
        }
      }
    }
  }
}

下面是唯一得到的记录:

代码语言:javascript
复制
    source_type:customer sender_id:3 destination_type:funding-source type:funded_transfer z_id:863a240c-2011-e911-8114-bacd823e9f1d receiver_id:65 status:processed amount:550 funding_source_from_id:332 @timestamp:Nov 8, 2020 @ 16:00:08.809 fees:5.61 is_parent:false created:Jan 5, 2019 @ 21:28:21.847 @version:1 id:2,160 funding_source_to_id: - metadata: - _id:2160 _type:doc _index:payments_transfer_data _score: -
EN

回答 1

Stack Overflow用户

发布于 2020-11-09 03:38:39

根据官方文档,您应该将tracking_column与use_column_value结合使用。使用您当前的设置,tracking_column将不会有任何影响。

您是否尝试过使用Select * from查看是否正在提取所有内容。

票数 0
EN
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接:

https://stackoverflow.com/questions/64739245

复制
相关文章

相似问题

领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档