ASAMMDF - MemoryError:无法为形状(2207220 )和数据类型float64的数组分配16.8 MiB

Question

我试图使用asammdf从".dat“文件中提取数据。

在使用asammdf提取数据之后，我正在尝试将数据转换为可以使用熊猫和matplotlib进行分析的数据。

下面是我用于提取数据并转换为dataframe的代码：

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import asammdf
import tkinter as tk
from tkinter.ttk import *

Data_File_01 = asammdf.MDF(r"C:\Users\hsr4ban\Desktop\16_MC Dynamic.dat")
Data_01 = Data_File_01.to_dataframe()

但是，我得到的内存错误如下：

runfile('C:/Users/hsr4ban/Desktop/untitled0.py', wdir='C:/Users/hsr4ban/Desktop')
Traceback (most recent call last):

  File ~\Desktop\untitled0.py:11 in <module>
    Data_File_01 = asammdf.MDF(r"C:\Users\hsr4ban\Desktop\16_MC Dynamic.dat").to_dataframe()

  File ~\AppData\Roaming\Python\Python39\site-packages\asammdf\mdf.py:4466 in to_dataframe
    df = pd.DataFrame(nonstrings, index=master)

  File ~\AppData\Roaming\Python\Python39\site-packages\pandas\core\frame.py:636 in __init__
    mgr = dict_to_mgr(data, index, columns, dtype=dtype, copy=copy, typ=manager)

  File ~\AppData\Roaming\Python\Python39\site-packages\pandas\core\internals\construction.py:502 in dict_to_mgr
    return arrays_to_mgr(arrays, columns, index, dtype=dtype, typ=typ, consolidate=copy)

  File ~\AppData\Roaming\Python\Python39\site-packages\pandas\core\internals\construction.py:156 in arrays_to_mgr
    return create_block_manager_from_column_arrays(

  File ~\AppData\Roaming\Python\Python39\site-packages\pandas\core\internals\managers.py:1959 in create_block_manager_from_column_arrays
    mgr._consolidate_inplace()

  File ~\AppData\Roaming\Python\Python39\site-packages\pandas\core\internals\managers.py:1685 in _consolidate_inplace
    self.blocks = tuple(_consolidate(self.blocks))

  File ~\AppData\Roaming\Python\Python39\site-packages\pandas\core\internals\managers.py:2084 in _consolidate
    merged_blocks = _merge_blocks(

  File ~\AppData\Roaming\Python\Python39\site-packages\pandas\core\internals\managers.py:2111 in _merge_blocks
    new_values = np.vstack([b.values for b in blocks])  # type: ignore[misc]

  File <__array_function__ internals>:180 in vstack

  File ~\AppData\Roaming\Python\Python39\site-packages\numpy\core\shape_base.py:282 in vstack
    return _nx.concatenate(arrs, 0)

  File <__array_function__ internals>:180 in concatenate

MemoryError: Unable to allocate 5.46 GiB for an array with shape (332, 2207220) and data type float64

我签了堆栈溢出。在“”文件的情况下，很少有人建议使用.csv文件，但在这种情况下，它是".dat“文件，而我找不到多少帮助。

有人能建议如何解决这个问题吗？

提前谢谢。

Answer 1

在使用raster时，需要使用to_dataframe参数，因为文件中有太多的单独时间戳(参见https://asammdf.readthedocs.io/en/master/api.html#asammdf.mdf.MDF.iter_to_dataframe)