通过Scala/Spark在Amazon S3中列出对象

Question

我正在尝试使用S3在亚马逊的S3FileSystem桶中获取对象列表。

val exists = s3FS.exists(new org.apache.hadoop.fs.Path("s3n://bucketname/Daily/DailyCash.TXT"))
println(exists)
val sels=s3FS.listFiles(new org.apache.hadoop.fs.Path("s3n://bucketname//Daily"),true)
println(sels)

获取以下错误：

Exception in thread "main" java.io.FileNotFoundException: File s3n://bucketname/ does not exist.
 at org.apache.hadoop.fs.s3.S3FileSystem.listStatus(S3FileSystem.java:194)
 at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1482)
 at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1522)
 at org.apache.hadoop.fs.FileSystem$4.<init>(FileSystem.java:1798)
 at org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:1797)
 at org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:1780)
 at org.apache.hadoop.fs.FileSystem$5.<init>(FileSystem.java:1842)
 at org.apache.hadoop.fs.FileSystem.listFiles(FileSystem.java:1839)

Answer 1

我会用AWS来做类似的事情，而不是混合和匹配Hadoop类，并且只用于列出对象，而不是建议加载巨大的数据集。

import com.amazonaws.auth.DefaultAWSCredentialsProviderChain
import com.amazonaws.services.s3.{AmazonS3Client, AmazonS3URI}

val location = "s3://bucketname/Daily/DailyCash.TXT"

val awsCredentials = new DefaultAWSCredentialsProviderChain()
val s3Client = new AmazonS3Client(awsCredentials)
val s3Uri = new AmazonS3URI(location)

val objects: ObjectListing = s3Client.listObjects(s3Uri.getBucket)