Heroku Postgres: AWS s3预签名URL不工作

Question

我已经犯了6小时的错误，并且发疯了。

我已经使用CLI创建了一个heroku应用程序，并成功地推动了git，但是在应用程序运行之前，我需要使用db.create_all()初始化一堆表，但是我的本地主机postgres中有6gb的数据，我希望迁移到heroku使用。

用通常的方式(create_all)实例化表将无法工作，因为索引页需要从表中获取某些内容，因此，我的表必须预先填充上述localhost数据才能工作。

我一直跟踪这里的文档，并导出了我的.dump文件，并将其上传到了桶中：https://devcenter.heroku.com/articles/heroku-postgres-import-export#import

公共共享是启用的，我已经使用setx命令在AWS中设置了ACCESS_KEY_ID和SECRET_ACCESS_KEY。我还设置了heroku config:set AWS_ACCESS_KEY_ID=matching_aws_cli_blah AWS_SECRET_ACCESS_KEY=matching_aws_cli_blah。

然后，使用命令aws s3 presign s3_url复制输出(我将在使用https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/presign.html的AWS文档中添加看起来不同的输出)

一个示例说明了我的外观(随机更改了一些数字) https://mybucketname.s3.eu-west-2.amazonaws.com/mydb.dump?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=LLLLL4C7LLLLLLLNLLLQ%2FNNNNNN08%2Feu-west-2%2Fs3%2Faws4_request&X-Amz-Date=20201208T004613Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=ll9d80nnnnc457ea83298fdnnnn4a25b0c9ll066f54e6ff8acf42dafnnnea8877，而文档提供了一个很好的整洁的AWSAccessKeyId=变量。

我只有一个桶，Block public access (bucket settings)设置为OFF，文件本身可以公开共享。对象URL初始化下载良好，包括在匿名中。

使用以下命令时(如heroku文档所示)

heroku pg:backups:restore 'https://mybucketname.s3.eu-west-2.amazonaws.com/mydb.dump?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=LLLLL4C7LLLLLLLNLLLQ%2FNNNNNN08%2Feu-west-2%2Fs3%2Faws4_request&X-Amz-Date=20201208T004613Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=ll9d80nnnnc457ea83298fdnnnn4a25b0c9ll066f54e6ff8acf42dafnnnea8877' DATABASE_URL

我得到以下输出到终端。

Restoring... !
 !    An error occurred and the backup did not finish.
 !
 !    waiting for restore to complete
 !    pg_restore finished with errors
 !    waiting for download to complete
 !    download finished with errors
 !    please check the source URL and ensure it is publicly accessible
 !
 !    Run heroku pg:backups:info r001 for more details.
'X-Amz-Credential' is not recognized as an internal or external command,
operable program or batch file.
'X-Amz-Date' is not recognized as an internal or external command,
operable program or batch file.
'X-Amz-Expires' is not recognized as an internal or external command,
operable program or batch file.
'X-Amz-SignedHeaders' is not recognized as an internal or external command,
operable program or batch file.
'X-Amz-Signature' is not recognized as an internal or external command,
operable program or batch file.

在日志文件中显示

=== Backup r001
Database:         BACKUP
Started at:       2020-12-08 00:47:33 +0000
Finished at:      2020-12-08 00:47:34 +0000
Status:           Failed
Type:             Manual
Backup Size:      0.00B (0% compression)

=== Backup Logs
2020-12-08 00:47:34 +0000 2020/12/08 00:47:34 aborting: could not write to output stream: Expected HTTP Status 200, received: "400 Bad Request"
2020-12-08 00:47:34 +0000 pg_restore: error: could not read from input file: end of file
2020-12-08 00:47:34 +0000 waiting for restore to complete
2020-12-08 00:47:34 +0000 pg_restore finished with errors
2020-12-08 00:47:34 +0000 waiting for download to complete
2020-12-08 00:47:34 +0000 download finished with errors
2020-12-08 00:47:34 +0000 please check the source URL and ensure it is publicly accessible

有人能指出我哪里出了问题吗？

谢谢!

Answer 1

诀窍是将额外的引号传递给heroku命令。

它看起来像这样：

• powershell：

$url = "https://provider/bucket/file“heroku pg:备份:还原-a -a

• bash：

url="https://provider/bucket/file“heroku pg:备份:恢复-a "$url"

Answer 2

为了调试这一点，将响应错误消息从请求中获取到AWS url可能是有用的，您可以轻松地使用curl：

curl <SIGNED AWS URL>

在我的例子中，我得到了以下信息：

Error parsing the X-Amz-Credential parameter; the region 'us-east-1' is wrong; expecting 'eu-west-3'

这意味着由于某种原因，AWS正在用错误的区域签署我的URL。如果我尝试手动将正确的区域添加到我签名的url中，那么请求就会失败，因为签名不匹配。

相反，我需要通过在presign命令中指定区域来使AWS正确地对URL签名：

aws s3 presign s3://URL --region eu-west-3

当我这样做并使用heroku pg:backups:restore命令的结果签名url时，请求是成功的，heroku能够导入我的db。