通过管道将axios请求发送到pkgcloud存储上传会导致“结束后写入”错误

Question

我正在尝试使用axios下载一个外部文件，然后使用pkgcloud将其传输到openstack对象存储容器中。上载流似乎在文件完全下载之前结束。

async function upload(url, name, extension, container) {
    const uploadStream = storage.createClient({
        ...
    }).upload({
        container: container,
        remote: name + "." + extension,
    });

    await new Promise(async (resolve) => {
        const response = await axios({
            method: "GET",
            url: url,
            responseType: "stream"
        })

        response.data.pipe(uploadStream)

        response.data.on("end", () => {
            console.log("Download finished");
        })
        response.data.on("error", (error: Error) => {
            console.log("Download error")
            console.log(error);
        })
        uploadStream.on("finish", () => {
            console.log("Upload finished");
            resolve();
        })
        uploadStream.on("error", (error: Error) => {
            console.log("Upload error");
            console.log(error);
        })
    })
}

const url = "https://picsum.photos/id/566/600/600";

(async () => {
    await upload(url, "mountain", "jpg", "dummy_container")
    console.log("Promise resolved");
})()

根据控制台输出的事件顺序如下：

Download finished
Upload finished
Promise resolved
Upload error
Error: write after end
...

Answer 1

通过使用concat-stream包将下载流式传输到缓冲区，然后将缓冲区流式传输到上传，可以解决此问题：

function upload(url, name, extension, container) {
    const uploadStream = storage.createClient({
        ...
    }).upload({
        container,
        remote: name + "." + extension,
    });

    axios({
        method: "GET",
        url,
        responseType: "stream",
    }).then((response) => {
        const concatStream = concat((buffer) => {
            const bufferStream = new Stream.PassThrough();
            bufferStream.end(buffer);
            bufferStream.pipe(uploadStream);
        })
        response.data.pipe(concatStream);
    })
}

const url = "https://picsum.photos/id/566/600/600";
upload(url, "mountain", "jpg", "dummy_container")

Answer 2

这个问题来自于pkgcloud中的一个bug，它是如何为OpenStack存储流式传输文件的。它在https://stackoverflow.com/a/60787524/4685491中有解释