如何读取ORC文件列数据

Question

我已经下载了ORC c++ API并在我的Ubuntu上构建了它。现在我正在尝试批量读取它的列数据。在这个参考文献中，提到了orc::ColumnVectorBatch可以是特定列数据类型的dynamic_cast，比如：orc::Decimal64VectorBatch。但是它并没有将空指针作为动态转换结果。下面是我的代码：

// Orc Reader.

#include <memory>
#include <iostream>
#include <vector>
#include <list>
#include <fstream>

#include <orc/Reader.hh>
#include <orc/ColumnPrinter.hh>
#include <orc/Exceptions.hh>
#include <orc/OrcFile.hh>

int main(int argc, char const *argv[])
{
    std::list<uint64_t> read_cols = {4};
    std::string file_path = "~/trades_data.zlib.orc";

    std::ifstream in_file(file_path.c_str(), std::ios::binary);
    in_file.seekg(0, std::ios::end);
    int file_size = in_file.tellg();
    std::cout << "Size of the file is" << " " << file_size << " " << "bytes";

    orc::RowReaderOptions row_reader_opts;
    row_reader_opts.include(read_cols);

    orc::ReaderOptions reader_opts;
    std::unique_ptr<orc::Reader> reader;
    std::unique_ptr<orc::RowReader> row_reader;

    reader = orc::createReader(orc::readFile(file_path), reader_opts);
    row_reader = reader->createRowReader(row_reader_opts);

    std::unique_ptr<orc::ColumnVectorBatch> batch = row_reader->createRowBatch(1000);

    while (row_reader->next(*batch))
    {
        // BELOW LINE OF CODE IS GIVING NULLPOINTER.
        orc::Decimal64VectorBatch *dec_vec = dynamic_cast<orc::Decimal64VectorBatch*>(batch.get());
    }

    return 0;
}

如果有人能指出这个错误，对我来说真的是一个很大的帮助。

Answer 1

我刚才已经解决了这个问题，现在我正在为我自己的问题写一个答案。希望在代码中也有帮助。在上面的代码中，它试图将从row_reader读取的浴缸转换为orc::Decimal64VectorBatch，但是批处理应该首先被转换为orc::StructVectorBatch。然后，使用列的索引数，可以轻松地将其转换为所需的列数据。

    const int time_idx = 0; // Index of column containing time in decimal64 format.
    while (row_reader->next(*batch))
    {
        // Now batch should initially convert into StructVectorBatc.
        const auto &struct_batch = dynamic_cast<const orc::StructVectorBatch&>(*batch.get());
        // And then struct_batch can be converted into required column data format.
        const auto &dec_vec = dynamic_cast<orc::Decimal64VectorBatch&>(*(struct_batch.fields[time_idx)).values.data();
    }

Answer 2

这是我的方法，希望能对你有所帮助。

完整代码演示：演示

//double field
auto *fields = dynamic_cast<orc::StructVectorBatch *>(batch.get());
auto *col0 = dynamic_cast<orc::DoubleVectorBatch *>(fields->fields[0]);
double *buffer1 = col0->data.data();

//string field
auto *col4 = dynamic_cast<orc::StringVectorBatch *>(fields->fields[4]);
char **buffer2 = col4->data.data();
long *lengths = col4->length.data();

while (row_reader->next(*batch)) {
    for (uint32_t r = 0; r < batch->numElements; ++r) {
        std::cout << "line " << buffer1[r] << "," << std::string(buffer2[r], lengths[r]) << "\n";
    }
    //std::cout << "this batch nums" << " " << batch->numElements << " " << "lines\n";
}