"ArrowInvalid:不能统一模式与重复的字段名“从s3读取拼图文件使用dask；

Question

使用查询，我将数据从红移到拼图。

UNLOAD
('
SELECT
     delivered_at
    , flow_name
    , variant_name
    , user_id
') TO
            's3://data/raw/redshift/all_campaigns'
            IAM_ROLE 'arn:aws:iam::XYZ:role/redshift'
            FORMAT AS PARQUET
            maxfilesize 96 mb
            ALLOWOVERWRITE
            MANIFEST;

从另一端使用dask，我试着读取该数据的一部分

df = dd.read_parquet('s3://data/raw/redshift/all_campaigns0079_part_00.parquet')

() ->1df= dd.read_parquet('s3://data/raw/redshift/crm/was_clicked/all_campaigns0079_part_00.parquet')中的/1>/扇颊/Devel/NyRec/EDA.ipynb Cell 7‘

File ~/opt/miniconda3/envs/deeprec/lib/python3.9/site-packages/dask/dataframe/io/parquet/core.py:460, in read_parquet(path, columns, filters, categories, index, storage_options, engine, calculate_divisions, ignore_metadata_file, metadata_task_size, split_row_groups, chunksize, aggregate_files, parquet_file_extension, **kwargs)
    457 if index and isinstance(index, str):
    458     index = [index]
--> 460 read_metadata_result = engine.read_metadata(
    461     fs,
    462     paths,
    463     categories=categories,
    464     index=index,
    465     gather_statistics=calculate_divisions,
    466     filters=filters,
    467     split_row_groups=split_row_groups,
    468     chunksize=chunksize,
    469     aggregate_files=aggregate_files,
    470     ignore_metadata_file=ignore_metadata_file,
    471     metadata_task_size=metadata_task_size,
    472     parquet_file_extension=parquet_file_extension,
    473     **kwargs,
    474 )
    476 # In the future, we may want to give the engine the
    477 # option to return a dedicated element for `common_kwargs`.
...
File ~/opt/miniconda3/envs/deeprec/lib/python3.9/site-packages/pyarrow/error.pxi:144, in pyarrow.lib.pyarrow_internal_check_status()

File ~/opt/miniconda3/envs/deeprec/lib/python3.9/site-packages/pyarrow/error.pxi:100, in pyarrow.lib.check_status()

ArrowInvalid: Can't unify schema with duplicate field names.

我肯定我没有相同的专栏。如果问题不清楚，请帮助我提高问题的质量。

更新

dask version        : '2022.6.0'
pyarrow version     : '8.0.0'
fastparquet version : '0.8.1'

我还在他们的gir问题页面这里上提出了这个问题

Answer 1

结果发现，确实有几个同名的栏目。