架构未与AWS Glue爬虫正确合并

Question

目前，我正在构建一个数据集，每天运行AWS GlueJobs，以便在数据库中复制数据，并通过AWS雅典娜查询数据。因为我获取的数据的模式经常发生变化，所以我经常用Glue Crawler爬行它们。不幸的是，当我连续两天运行爬虫和模式更改时，我会得到一个关于不兼容模式的错误：

HIVE_CANNOT_OPEN_SPLIT: Error opening Hive split s3://***/raw/itemstore/parquet_flattened/v1/type=articles/year=2019/month=12/day=12/part-00012-13fc8243-cd4e-47b8-8763-56b15ea46e84-c000.snappy.parquet (offset=0, length=32745292): Schema mismatch, metastore schema for row column item__timeline.element has 10 fields but parquet schema has 9 fields

This query ran against the "***" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: ***

下面是云形成中我们的爬行器的代码：

  ItemStoreCrawler:
    Type: AWS::Glue::Crawler
    Properties:
      Name: <A STRING>
      DatabaseName: !Ref DatabaseName
      Configuration: "{\"Version\": 1.0, \"CrawlerOutput\": {\"Partitions\": {\"AddOrUpdateBehavior\": \"InheritFromTable\"},\"Tables\":{\"AddOrUpdateBehavior\":\"MergeNewColumns\"}}}"
      Role: !GetAtt CrawlerRole.Arn
      TablePrefix: String 
      Tags:
        Platform: !Ref Platform
        Maintainer: !Ref Maintainer
        ServerType: !Ref ServerType
        ServiceName: !Sub ${ProjectName}
        Environment: !Ref Environment

      Targets:
        S3Targets:
          - Path: String

我的猜测是，我的爬虫的模式合并行为在以Configuration开头的行中设置得很糟糕，但我找不到修复程序。

Answer 1

这与让它忽略列顺序有关--我强烈建议不要使用Glue Crawler --直接将Glue作为您的Hive Metastore写入雅典娜以避免这种情况。

https://docs.aws.amazon.com/athena/latest/ug/handling-schema-updates-chapter.html#summary-of-updates