网络流量FileUpload Minio

Question

我试图找出为什么通过S3端点上传到Minio (符合WebFlux规范的文档存储)的文件不能完成；我总是只将4kb的文件上传到minio。

我的端点：

    public Mono<ServerResponse> uploadFile(ServerRequest request) {
        log.info("Uploading file...");
        log.info("Content Type: {}", request.headers().contentType().orElse(MediaType.TEXT_PLAIN));

        return request.body(BodyExtractors.toMultipartData())
            .flatMap(map -> {
                Map<String, Part> parts = map.toSingleValueMap();
                return Mono.just((FilePart) parts.get("files"));
            })
            .flatMap(this::saveFile)
            .flatMap(part -> ServerResponse.ok().body(BodyInserters.fromObject(part)));
    }

    private Mono<String> saveFile(FilePart part) {
        return part.content().map(dataBuffer -> {
            try {
                log.info("Putting file {} to minio...", part.filename());
                client.putObject("files", part.filename(), dataBuffer.asInputStream(), part.headers().getContentType().getType());
            } catch(Exception e) {
                log.error("Error storing file to minio", e);
                return part.filename();
            }
        return part.filename();
        }).next();
    }

我很确定这是一个阻塞和非阻塞的问题，但是如果我尝试在其中添加一个blockFirst()调用，我就会得到一个异常，它在运行时是不允许的。

是否有一种有效地流数据的方法，或者是Minio客户端与WebFlux不兼容的情况？

我试图从一个像这样的React组件中发布：

class DataUpload extends React.Component {
    constructor(props) {
        super(props);

        this.state = {
            fileURL: '',
        };

        this.handleUploadData = this.handleUploadData.bind(this);
    }

    handleUploadData = ev => {
        ev.preventDefault()

        const data = new FormData();
        data.append('files', this.uploadInput.files[0]);
        data.append('filename', this.fileName.value);
        fetch('http://localhost:8080/requestor/upload', {
            method: 'POST',
            body: data,
        }).then((response) => {
            response.json().then((body) => {
                this.setState({ fileURL: `http://localhost:8080/${body.file}`});
            });
        });
    }

    render() {
        return (
            <form onSubmit={this.handleUploadData}>
                <div>
                    <input ref={(ref) => { this.uploadInput = ref; }} type="file" />
                </div>
                <div>
                    <input ref={(ref) => { this.fileName = ref; }} type="text" placeholder="Enter the desired name of the file" />
                </div>
                <br/>
                <div><button>Upload</button></div>
            </form>
        );
    }
}

export default DataUpload;

Answer 1

我总是只把4kb的文件输入Minio

这是因为spring块文件分成4kb的部分，您必须自己收集它们。你可以这样做：

request.body(BodyExtractors.toMultipartData())
    .map(dataBuffers -> dataBuffers.get("files"))
    .filter(Objects::nonNull)
    //get the file name and pair it with it's "Part"
    .map(partsList -> {
        List<Pair<String, Part>> pairedList = new ArrayList<>();

        for (Part part : partsList) {
            String fileName = ((FilePart) part).filename();
            pairedList.add(new Pair<>(fileName, part));
        }

        return pairedList;
    })
    .flux()
    .flatMap(Flux::fromIterable)
    //here we collect all of the file parts with the buffer operator and zip them with filename
    .flatMap(partWithName -> Mono.zip(Mono.just(partWithName.getFirst()), partWithName.getSecond().content().buffer().single()))
    .buffer()
    .single()
    .doOnNext(filePartsWithNames -> {
        //here we have a list of all uploading file parts and their names
        for (Tuple2<String, List<DataBuffer>> filePartsWithName : filePartsWithNames) {
            String fileName = filePartsWithName.getT1();
            List<DataBuffer> buffers = filePartsWithName.getT2();

            System.out.println("Filename = " + fileName);

            //"buffers" is a list of 4kb chunks of the files
            for (DataBuffer buffer : buffers) {
                System.out.println("Buffer size = " + buffer.readableByteCount());

                //here you can use buffer.asInputStream() to read the file part and
                //then save it on disk or do something else with it
            }
        }
    })

Answer 2

对于未来的读者来说，

我通过加入dataBuffers来解决这个问题，然后将其公开为inputStream，在org.springframework.core.io.buffer包中使用DataBufferUtils.join()。

   Mono<ServerResponse> uploadSingleImageToS3(ServerRequest request) {
    return request.body(toMultipartData())
            .flatMap(parts -> {
                Map<String, Part> part = parts.toSingleValueMap();
                return Mono.just((FilePart) part.get("file"));
            })
            .flatMap(this::uploadToS3Bucket)
            .flatMap(filename -> ServerResponse.ok().body(fromObject(filename)));
}

private Mono<String> uploadToS3Bucket(FilePart part) {
    return DataBufferUtils.join(part.content())
            .map(dataBuffer -> {
                String filename = UUID.randomUUID().toString();
                log.info("filename : {}", filename);

                ObjectMetadata metadata = new ObjectMetadata();
                metadata.setContentLength(dataBuffer.capacity());
                PutObjectRequest putObjectRequest = new PutObjectRequest(answerImagesBucket, filename, dataBuffer.asInputStream(), metadata);
                transferManager.upload(putObjectRequest);

                return filename;
            });
}