将所有csv文件从s3转换为parquet

Question

我要做的是将所有文件从S3(亚马逊网络服务存储)转换为parquet格式，并将它们重新保存到s3中。

我不能从s3转换所有的文件。请帮帮我!！

import boto3
import pandas as pd
import pyarrow as pa
from s3fs import S3FileSystem
import pyarrow.parquet as pq

s3 = boto3.client('s3',region_name='us-east-2')

obj = s3.get_object(Bucket='dstest-s3', Key='dstest/movies.csv')
df = pd.read_csv(obj['Body'])

table = pa.Table.from_pandas(df)

output_file = "s3://dstest-s3/dstest/parquetconversion1.parquet"
s3 = S3FileSystem()

pq.write_to_dataset(table=table, root_path=output_file, filesystem=s3)

print("File converted from CSV to parquet completed")

Answer 1

基本上，您需要使用list_objects_v2从存储桶中获取所有密钥，并迭代这些密钥，然后下载、转换和上传。

s3 = bt3.get_client('s3')
response = s3.list_objects_v2(
        Bucket='dstest-s3',
        Prefix='dstest/'
)

for s3_obj in response['Contents']:
    obj = s3.get_object(Bucket='dstest-s3', Key=s3_obj['Key'])
    # Do your converting, and uploading here