通过vpc端点从容器读取/写入AWS S3，从而给出400个坏请求

Question

我正在尝试从容器通过vpc端点将数据读写到AWS S3，该容器位于美国地区的前提(数据中心)。下面是连接到S3的Pyspark代码：

from pyspark.conf import SparkConf
from pyspark.sql import SparkSession

conf = (
        SparkConf()
        .setAppName("PySpark S3 Example")
        .set("spark.hadoop.fs.s3a.endpoint.region", "us-east-1")
        .set("spark.hadoop.fs.s3a.endpoint","<vpc-endpoint>")
        .set("spark.hadoop.fs.s3a.access.key", "<access_key>")
        .set("spark.hadoop.fs.s3a.secret.key", "<secret_key>")
        .set("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
        .set("spark.driver.extraJavaOptions", "-Dcom.amazonaws.services.s3.enforceV4=true")
        .set("spark.executor.extraJavaOptions","-Dcom.amazonaws.services.s3.enableV4=true")
        .set("spark.executor.extraJavaOptions", "-Dcom.amazonaws.services.s3.enforceV4=true")
        .set("spark.fs.s3a.path.style.access", "true")
        .set("spark.hadoop.fs.s3a.server-side-encryption-algorithm","SSE-KMS")
        .set("spark.hadoop.fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider")
    )

spark = SparkSession.builder.config(conf=conf).getOrCreate()

data = [{"key1": "value1", "key2": "value2"}, {"key1":"val1","key2":"val2"}]

df = spark.createDataFrame(data)

df.write.format("json").mode("append").save("s3a://<bucket-name>/test/")

提出的例外情况：

py4j.protocol.Py4JJavaError: An error occurred while calling o91.save.
: org.apache.hadoop.fs.s3a.AWSBadRequestException: doesBucketExist on <bucket-name>
: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: <requestID>; 

如能提供任何帮助，将不胜感激。

Answer 1

除非您的hadoop s3a客户端是区域感知(3.3.1+)，否则设置该区域选项将无法工作。有一个aws选项"aws.region，您可以将其设置为系统属性。